Optional files still require token for `huggingface-cli` downloaded gated repo
Author: scruel
Pull request: no
Labels: name: bug

### Describe the bug

For functions `from_XXX`, it will create empty files into `.no_exist` directory if repo have some files missed, however the CLI tool `huggingface-cli download` won't do so, which caused inconsistency issues.

### Reproduction

1. `export HF_TOKEN=XXX`
2. `huggingface-cli download --resume-download meta-llama/Llama-2-7b-hf`
3. `python -c "from transformers import AutoTokenizer; AutoTokenizer.from_pretrained('meta-llama/Llama-2-7b-hf', token=False)"`

Will produce `OSError` because we are loading a gated repo, even we already requested and downloaded it via CLI tool, we won't be able to use this cached model (e.g., if we want to use it offline), until we run `from_XXX` method once to have those missed files created as empty files into `.no_exist` directory.


### Logs

```shell
...
OSError: You are trying to access a gated repo.
Make sure to request access at https://huggingface.co/meta-llama/Llama-2-7b-hf and pass a token having permission to this repo either by logging in with `huggingface-cli login` or by passing `token=<your_token>`.
```


### System info

```shell
- huggingface_hub version: 0.20.2
- Platform: Linux-6.5.0-14-generic-x86_64-with-glibc2.35
- Python version: 3.11.6
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Token path ?: /home/scruel/.cache/huggingface/token
- Has saved token ?: True
- Who am I ?: situqingyun
- Configured git credential helpers: 
- FastAI: N/A
- Tensorflow: N/A
- Torch: 2.0.1
- Jinja2: 3.1.2
- Graphviz: N/A
- Pydot: N/A
- Pillow: N/A
- hf_transfer: 0.1.4
- gradio: N/A
- tensorboard: N/A
- numpy: 1.26.0
- pydantic: N/A
- aiohttp: 3.9.1
- ENDPOINT: https://huggingface.co
- HF_HUB_CACHE: /home/scruel/.cache/huggingface/hub
- HF_ASSETS_CACHE: /home/scruel/.cache/huggingface/assets
- HF_TOKEN_PATH: /home/scruel/.cache/huggingface/token
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False
- HF_HUB_ETAG_TIMEOUT: 10
- HF_HUB_DOWNLOAD_TIMEOUT: 10
```


Comments:
@scruel it is expected that `.no_exist` is not populated when using `huggingface-cli download --resume-download meta-llama/Llama-2-7b-hf`. In fact, the "no exists" folder is only populated when `transformers` (or any script/library) tries to download a file that does not exist on the repo. `transformers` uses this information to default to another file when that's the case. When you use the `huggingface-cli` we only download existing files but we don't populate "missing files" (which wouldn't really make sense). 

So I agree this is a bug but rather in `transformers` than `huggingface_hub` itself. I think `transformers` should be able to load a model when it has already been downloaded, even if the `.no_exists` folder is not populated.

I'm therefore transferring this issue to `transformers` repo. Btw, thanks for providing a reproducible example of your problem. @amyeroberts @ArthurZucker @ydshieh would you have time to take a look at how `transformers` is loading files when already cached? 

Hi @Wauplin When I seared `no_exist` (case insensitive), there are only 4 results, and they are like `if resolved_file is not _CACHED_NO_EXIST`. 

I am happy to take the question you raised, but is it possible you share a bit more how `.no_exists` is related to `transformers`? I mean the logic of creating this is inside `huggingface_hub` right, and `transformers` only check if such path exist or not and perform different actions.

@ydshieh the `no_exist` folder is indeed populated by `huggingface_hub` each time a library calls `huggingface_hub.hf_hub_download` to download a file that doesn't exist (but the repo does!). `transformers` leverages this information when it tries to load optional files. To avoid making a HEAD call for each file each time the user do a `from_pretrained`, `transformers` looks at the no_exist folder which saves some time (actually the `no_exists` folder is only used by `transformers` as far as I know).

What `transformers` should do is that in the case of missing `no_exists` entries, if the cache is fully downloaded then the model should be loaded correctly. This is what @scruel's PR is doing if I understand correctly. My only comment is that it shouldn't be specific to gated repos but also private/missing repos as well, as long as the files are cached.

Hi @Wauplin and @scruel 

I am looking into this, but there is one minor stuff I don't get clearly.

before I do `until we run from_XXX method once to have those missed files created as empty files into .no_exist directory`, I checked with

> !ls -l /root/.cache/huggingface/hub/models--meta-llama--Llama-2-7b-hf/snapshots/8cca527612d856d7d32bd94f8103728d614eb852

and it gives 

```
total 24
lrwxrwxrwx 1 root root 52 Jan 22 14:47 config.json -> ../../blobs/34f901200fa131819b355bc4bed876c957a77a5a
lrwxrwxrwx 1 root root 52 Jan 22 14:47 generation_config.json -> ../../blobs/aa1b3d3486df56a0699ce90c33283b13556fb5a3
lrwxrwxrwx 1 root root 52 Jan 22 14:47 LICENSE.txt -> ../../blobs/51089e27e6764fb9f72c06a0f3710699fb6c9448
lrwxrwxrwx 1 root root 76 Jan 22 14:49 model-00001-of-00002.safetensors -> ../../blobs/4ec71fd53e99766de38f24753b30c9e8942630e9e576a1ba27b0ec531e87be41
lrwxrwxrwx 1 root root 76 Jan 22 14:48 model-00002-of-00002.safetensors -> ../../blobs/41780b5dac322ac35598737e99208d90bdc632a1ba3389ebedbb46a1d8385a7f
lrwxrwxrwx 1 root root 52 Jan 22 14:47 model.safetensors.index.json -> ../../blobs/8b6245796e966e50960a317e4a54aa7bf73b0186
lrwxrwxrwx 1 root root 76 Jan 22 14:51 pytorch_model-00001-of-00002.bin -> ../../blobs/ee62ed2ad7ded505ae47df50bc6c52916860dfb1c009df4715148cc4bfb50d2f
lrwxrwxrwx 1 root root 76 Jan 22 14:51 pytorch_model-00002-of-00002.bin -> ../../blobs/1fd7762035b3ca4f2d6af6bf10129689a119b7c38058025f9842511532ea02fb
lrwxrwxrwx 1 root root 52 Jan 22 14:47 pytorch_model.bin.index.json -> ../../blobs/db7264b24cac7a39947bb5fc02fe5c2d7ac9eaf4
lrwxrwxrwx 1 root root 52 Jan 22 14:47 README.md -> ../../blobs/c7579dcbcbcd699270238173303a9013135a2a7d
lrwxrwxrwx 1 root root 76 Jan 22 14:47 Responsible-Use-Guide.pdf -> ../../blobs/525dc349d71fe257fce4098c146446df6fef4247174f351381e4c3214af126f0
lrwxrwxrwx 1 root root 52 Jan 22 14:47 special_tokens_map.json -> ../../blobs/451134b2ddc2e78555d1e857518c54b4bdc2e87d
lrwxrwxrwx 1 root root 52 Jan 22 14:47 tokenizer_config.json -> ../../blobs/2ef41cbc275000b29afe157ba487f0530b8c26dc
lrwxrwxrwx 1 root root 52 Jan 22 14:47 tokenizer.json -> ../../blobs/a6e931b92caff4c79c5c56282f1e89569a0ae558
lrwxrwxrwx 1 root root 76 Jan 22 14:47 tokenizer.model -> ../../blobs/9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
lrwxrwxrwx 1 root root 52 Jan 22 14:47 USE_POLICY.md -> ../../blobs/abbcc199b2d1e4feb5d7e40c0bd67e1b0ce29e97
```
And after running 

```
AutoTokenizer.from_pretrained('meta-llama/Llama-2-7b-hf', token=token)
```
(with a real token)

The same list of entries is shown, and I can't see any `.no_exist`

But then running the following

```
AutoTokenizer.from_pretrained('meta-llama/Llama-2-7b-hf', token=False)
```
indeed works.

Could you help me to understand where the mentioned `no_exist` is?

(I am aware of the PR #28566)

@ydshieh yes so the `.no_exists` folder is separated from the `snapshot` folder. If you do
```
!ls -l /root/.cache/huggingface/hub/models--meta-llama--Llama-2-7b-hf/
```
you should see `refs/`, `blobs/`, `snapshots/` and `/no_exists` which should help the investigation :)

Thanks @Wauplin , so when I run

> until we run from_XXX method once to have those missed files created as empty files into .no_exist directory,

(with a real token), and checked

```
!ls -l /root/.cache/huggingface/hub/models--meta-llama--Llama-2-7b-hf/
```

it still shows 

```
drwxr-xr-x 2 root root 4096 Jan 22 15:04 blobs
drwxr-xr-x 2 root root 4096 Jan 22 15:00 refs
drwxr-xr-x 3 root root 4096 Jan 22 15:00 snapshots
```

but again, the subsequent call with `token=False` works. 🤔 


wait, maybe let check the huggingface_hub version

It is `0.20.2` with `transformers==4.35.2`.

@ydshieh I can confirm, I have same as both:

Both 

```py
from transformers import AutoTokenizer
AutoTokenizer.from_pretrained('meta-llama/Llama-2-7b-hf')
```

and 

```py
from transformers import AutoTokenizer
AutoTokenizer.from_pretrained('meta-llama/Llama-2-7b-hf', token=False)
```

works for me. 

Cache directory only have tokenizers files:

```
(.venv310) ➜  huggingface_hub git:(main) ✗ tree -h  ~/.cache/huggingface/hub/models--meta-llama--Llama-2-7b-hf 
[4.0K]  /home/wauplin/.cache/huggingface/hub/models--meta-llama--Llama-2-7b-hf
├── [4.0K]  blobs
│   ├── [ 776]  2ef41cbc275000b29afe157ba487f0530b8c26dc
│   ├── [ 609]  34f901200fa131819b355bc4bed876c957a77a5a
│   ├── [ 414]  451134b2ddc2e78555d1e857518c54b4bdc2e87d
│   ├── [ 26K]  8b6245796e966e50960a317e4a54aa7bf73b0186
│   ├── [488K]  9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
│   └── [1.8M]  a6e931b92caff4c79c5c56282f1e89569a0ae558
├── [4.0K]  refs
│   └── [  40]  main
└── [4.0K]  snapshots
    └── [4.0K]  8cca527612d856d7d32bd94f8103728d614eb852
        ├── [  52]  config.json -> ../../blobs/34f901200fa131819b355bc4bed876c957a77a5a
        ├── [  52]  model.safetensors.index.json -> ../../blobs/8b6245796e966e50960a317e4a54aa7bf73b0186
        ├── [  52]  special_tokens_map.json -> ../../blobs/451134b2ddc2e78555d1e857518c54b4bdc2e87d
        ├── [  52]  tokenizer_config.json -> ../../blobs/2ef41cbc275000b29afe157ba487f0530b8c26dc
        ├── [  52]  tokenizer.json -> ../../blobs/a6e931b92caff4c79c5c56282f1e89569a0ae558
        └── [  76]  tokenizer.model -> ../../blobs/9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347

4 directories, 13 files
```

@scruel are you sure, the failing command was `AutoTokenizer.from_pretrained('meta-llama/Llama-2-7b-hf', token=False)` and not `AutoModel.from_pretrained('meta-llama/Llama-2-7b-hf', token=False)`?

Yes, I'm pretty sure.
Rather than use `ls -l` and `tree -h`, you should use `ls -la` and `tree -ha`, the directory you created is not `no_exiss` but `.no_exist`, it will be treated as a hidden folder in the modern OS.

Ah yes of course :facepalm: Thanks @scruel! 

Here it is @ydshieh:
```
(.venv310) ➜  huggingface_hub git:(main) ✗ tree -ah  ~/.cache/huggingface/hub/models--meta-llama--Llama-2-7b-hf                                 
[4.0K]  /home/wauplin/.cache/huggingface/hub/models--meta-llama--Llama-2-7b-hf
├── [4.0K]  blobs
│   ├── [ 776]  2ef41cbc275000b29afe157ba487f0530b8c26dc
│   ├── [ 609]  34f901200fa131819b355bc4bed876c957a77a5a
│   ├── [ 414]  451134b2ddc2e78555d1e857518c54b4bdc2e87d
│   ├── [ 26K]  8b6245796e966e50960a317e4a54aa7bf73b0186
│   ├── [488K]  9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
│   └── [1.8M]  a6e931b92caff4c79c5c56282f1e89569a0ae558
├── [4.0K]  .no_exist
│   └── [4.0K]  8cca527612d856d7d32bd94f8103728d614eb852
│       └── [   0]  added_tokens.json
├── [4.0K]  refs
│   └── [  40]  main
└── [4.0K]  snapshots
    └── [4.0K]  8cca527612d856d7d32bd94f8103728d614eb852
        ├── [  52]  config.json -> ../../blobs/34f901200fa131819b355bc4bed876c957a77a5a
        ├── [  52]  model.safetensors.index.json -> ../../blobs/8b6245796e966e50960a317e4a54aa7bf73b0186
        ├── [  52]  special_tokens_map.json -> ../../blobs/451134b2ddc2e78555d1e857518c54b4bdc2e87d
        ├── [  52]  tokenizer_config.json -> ../../blobs/2ef41cbc275000b29afe157ba487f0530b8c26dc
        ├── [  52]  tokenizer.json -> ../../blobs/a6e931b92caff4c79c5c56282f1e89569a0ae558
        └── [  76]  tokenizer.model -> ../../blobs/9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347

6 directories, 14 files
```

Thank you both for helping me on this! Will continue this along with a review