Allow private model hosting and resolution 
Author: vblagoje
Pull request: no

# 🚀 Feature request

It would be great to provide model hosting and automatic naming resolution on remote storage outside of HF hub/infra. Currently, it is possible to store a model on, say, S3 and resolve it via AutoModel and AutoConfig. However, in this case, a user has to explicitly specify the full path to the configuration file or the model's pytorch_model.bin file. 

It would be great if private remote storage could be registered with the same resolution mechanism reserved for HF so that:

`model = AutoModel.from_pretrained('my_org/my_model')` 
`config = AutoConfig.from_pretrained('my_org/my_model')` 

could be resolved to an actual remote storage path just like HF default resolution mechanism resolves config, models and tokenizers on HF hub.

## Motivation

During the model development lifecycle, organizations often produce many models for internal testing and benchmarking before producing the final model for publishing. Storing all the models during the development phase on the HF hub is sometimes impractical, and some organizations might need stricter control of model storage. 

## Your contribution

I've investigated a bit feature request's implementation scope and it doesn't seem to require a big rewrite. The name --> URL naming resolution is done in `file_utils.py`. One could follow how models are resolved for `HUGGINGFACE_CO_PREFIX`


Comments:
I think it's a brilliant idea. 

So just to validate I understood your proposal correclty, in addition to checking the usual places, it'll first check the env var `HUGGINGFACE_CO_PREFIX` and join it with `model_path` and check if it's available - and if not proceed with the normal algorithm.

So in your example using bash, it'd check `$HUGGINGFACE_CO_PREFIX/my_org/my_model`, which might be `https://some.place.on.earth/data/my_org/my_model`, right?

@LysandreJik, @julien-c - what do you think?

Yes, pretty much that's it. I think the top flat namespace where `bert-base-uncased`, `t5-base` and all other LMs "live" should never be allowed to resolve to anything else except HF hub (not just for security reasons). However, for the other 2+ level namespaces, i.e. `my_org/my_model` if users can register resolver - that would be great. In the Java world (where I come from), there are these notions of resources and resource bundles that could be dropped in predefined file locations and picked up by the library/framework. Not sure how this is done in Python, but perhaps [pkg_resources](https://setuptools.readthedocs.io/en/latest/pkg_resources.html#resourcemanager-api) could be used.  I believe this would be a better approach than registering these resolvers via some HF API. Although perhaps that should be left as an option. 

I would love to hear the opinions of others!

Hi,

I think the simplest way (and I've seen users and organizations do that) would be to extend `AutoModel` to sync your model files from your remote storage before loading it locally

Something like:

```python
class MyAutoModel:
    @classmethod
    def from_pretrained(cls, id):
        subprocess.run(f"aws s3 sync s3://mymodels/{id}/ {local_path}")
        return AutoModel.from_pretrained(local_path)
```

ie. all methods are guaranteed to work to load from local paths so pretty trivial to use them to load from anywhere

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.