Download models from a private hub
Author: juliensimon
Pull request: no
Labels: name: Feature request

name: private-hub

In the context of a private hub deployment, customers would like to use from_pretrained() to load models from their hub, not from the public hub. This doesn't seem to be configurable at the moment and it would be nice to add this feature.

The obvious workaround is to clone the repo first and then load it from local storage, but this adds an extra step. It'd be great to have the same experience regardless of where the hub is hosted.

The same issue exists with the datasets library and the CLI. I'm going to create issues there as well, and I'll reference them below.

Comments:
For reference:
https://github.com/huggingface/datasets/issues/3679
https://github.com/huggingface/huggingface_hub/issues/650

Correct me if I'm wrong, but this is already supported. You only need to pass an additional argument `use_auth_token=True` for private models.

This authentication token can be obtained by either doing the following in a notebook:

```
from huggingface_hub import notebook_login

notebook_login()
```
Note that you don't need to install `huggingface_hub`, it's included in the Transformers library.

Or, in case you're working with a terminal, by running the following command:
```
huggingface-cli login
```

no @NielsRogge here Julien would like to switch the endpoint to hit a private alternative to the hf.co hub (not just a private repo) e.g. like passing a custom `HF_ENDPOINT` in `dataset` (i think we may have the same here)

Maybe it can also a a param in the `from_pretrained()` calls for ease of use (we used to have something like that for mirrors at some point)

I looked around, and what we'd basically need to do is to check for an environment variable (this already works for datasets and the hub CLI):

```
hub_url = os.getenv("HF_ENDPOINT", "https://huggingface.co")
```

What would be the best place to do this once and once only? Is there a common data structure that is globally visible in the lib?

Then, we have to update all urls for config files, tokenizers and models, e.g.

```
DISTILBERT_PRETRAINED_CONFIG_ARCHIVE_MAP = {
    "distilbert-base-uncased": f"{hub_url}/distilbert-base-uncased/resolve/main/config.json",
    "distilbert-base-uncased-distilled-squad": f"{hub_url}/distilbert-base-uncased-distilled-squad/resolve/main/config.json",
    "distilbert-base-cased": f"{hub_url}/distilbert-base-cased/resolve/main/config.json",
    "distilbert-base-cased-distilled-squad": f"{hub_url}/distilbert-base-cased-distilled-squad/resolve/main/config.json",
    "distilbert-base-german-cased": f"{hub_url}/distilbert-base-german-cased/resolve/main/config.json",
    "distilbert-base-multilingual-cased": f"{hub_url}/distilbert-base-multilingual-cased/resolve/main/config.json",
    "distilbert-base-uncased-finetuned-sst-2-english": f"{hub_url}/distilbert-base-uncased-finetuned-sst-2-english/resolve/main/config.json",
}
```

What do you think @julien-c @sgugger @LysandreJik ?

Those urls are not used anywhere except internal tests, so there is no need to update them.

The endpoint just needs to be set via the env variable `HUGGINGFACE_CO_RESOLVE_ENDPOINT`, I think this is the only thing to do to use a custom endpoint.

Yep i think that's what I was alluding to when I wrote 
> (i think we may have the same here)

Maybe we can just rename (or support both env vars) `HUGGINGFACE_CO_RESOLVE_ENDPOINT` to `HF_ENDPOINT` for consistency across the different libraries? WDYT?

That works for me. Suggested in the PR linked above.

Great, thanks!