Harmonize HF environment variables + other cleaning
Author: Wauplin
Pull request: yes

This PR aims at harmonizing environment variables in the HF ecosystem (following up on https://github.com/huggingface/huggingface_hub/pull/1786). I also took the time to review some integration of `huggingface_hub` into `transformers` (trying to break nothing). The goal is simply to have less duplication in the codebase. The PR is not ready to be merged as is (see comments) but can be reviewed to discuss the few points. Feedback is very welcome @LysandreJik @amyeroberts @ArthurZucker @ydshieh as I'm necesarily missing some context sometimes.

**List of changes:**
- `HF_HOME` is the preferred way to set a custom hf path. Second best solution is `HF_HUB_CACHE` (hf_hub_cache is only for model/dataset caching while hf_home also contains the token and the cached modules -when trust_remote_code-). Therefore:
  - using `PYTORCH_PRETRAINED_BERT_CACHE` is deprecated (v5) but still respected
  - using `PYTORCH_TRANSFORMERS_CACHE` is deprecated (v5) but still respected
  - using `TRANSFORMERS_CACHE` is deprecated (v5) but still respected
- `DISABLE_TELEMETRY` is deprecated in favor of [`HF_HUB_DISABLE_TELEMETRY`](https://huggingface.co/docs/huggingface_hub/package_reference/environment_variables#hfhubdisabletelemetry)
- added a deprecation warning around `get_cached_models` => looks like a legacy unused method (and not compatible with new cache system)
- removed `try_to_load_from_cache` (moved to `huggingface_hub`)
- **unsure**: _I think_ we should harmonize between `get_file_from_repo` and `cached_file` to keep only one of them. The first one raises if the file doesn't exist while the second returns None in such a case. I think having a single method and a public `raise_on_error` argument should be enough (but open to discussion). Also both methods have the exact same docstring which is misleading.
- use `huggingface_hub.send_telemetry` to send telemetry data. Result is exactly the same but the HTTP call in made in a separate thread (meaning better UX as the user is not blocked)
- (nit) in `download_url` (already deprecated): it's safer to close the file

Comments:
Thank you for the explanation. Just one more nit question, in https://github.com/huggingface/huggingface_hub/pull/1786, there is a table of the re-mapped names, like `HUGGINGFACE_HUB_CACHE` -> `HF_CACHE`. I am wondering why we still have the usage of `constants.HUGGINGFACE_HUB_CACHE`.

Thanks for the reviews @LysandreJik and @ydshieh! 

I addressed 2 points:
- do not use `use_auth_token` in deprecated code (see [here](https://github.com/huggingface/transformers/pull/27564#discussion_r1398999241))
- use `huggingface_hub.constants.HF_HUB_CACHE` instead of `huggingface_hub.constants.HUGGINGFACE_HUB_CACHE` (see [here](https://github.com/huggingface/transformers/pull/27564#issuecomment-1819011254))

About using `from huggingface_hub.utils._deprecation import _deprecate_method` I would be keen to keep it as it is for now if that's ok with you @LysandreJik. It's not ideal but still ok I think. In next release of `hfh` I'll make it more official and in any case `_deprecate_method` will be used only in the next 3 versions of `transformers` (so EoL quite soon :) ).

Finally, regarding removing `get_file_from_repo` in favor of a unified cached_file` let's do it as a separate PR as suggested by @ydshieh. It is not really related to env variable harmonization.

---

@LysandreJik @ydshieh could you re-review the last changes + message above and approve the PR? I think we are good to merge the current version if that is ok with you. Thanks in advance!

_The documentation is not available anymore as the PR was closed or merged._

I'll review tomorrow! Thanks for this refactor 😉 

Yay! 3.5 approvals, that's enough! Thanks everyone :hugs: