In attempting to install Apple's new OpenELM, huggingface's transformers appears to want us to ask for permission from FaceBook/Meta?
Author: DXXS
Pull request: no

### System Info

- `transformers` version: 4.41.0.dev0
- Platform: macOS-12.6-arm64-arm-64bit
- Python version: 3.9.15
- Huggingface_hub version: 0.22.2
- Safetensors version: 0.4.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.4.0.dev20240425 (False)
- Tensorflow version (GPU?): 2.8.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Attempted to use M1 / mps...

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

% python generate_openelm.py --model apple/OpenELM-3B-Instruct --hf_access_token XXXXXXXXXXXXXXXX --prompt 'Once upon a time there was' --generate_kwargs repetition_penalty=1.2
...
  File "/Users/ds/transformers/src/transformers/utils/hub.py", line 416, in cached_file
    raise EnvironmentError(
...
"Cannot access gated repo for url https://huggingface.co/meta-llama/Llama-2-7b-hf/resolve/main/config.json.
Access to model meta-llama/Llama-2-7b-hf is restricted and you are not in the authorized list. Visit https://huggingface.co/meta-llama/Llama-2-7b-hf to ask for access."

### Expected behavior

I don't think that installing Apple LLM's should require permission from a ~competing company!?

Comments:
Strange, their 'generate_openelm.py' comments are claiming "Args: ... tokenizer: Tokenizer instance. If model is set as a string path, the tokenizer will be loaded from the checkpoint.", however, their code does not read that way, saying     "tokenizer: Union[str, AutoTokenizer] 'meta-llama/Llama-2-7b-hf'", but won't accept that parameter in the command line, and if I explicitly change it in the code to "'apple/OpenELM-XXX'", I get an error:

ValueError: Unrecognized configuration class <class 'transformers_modules.apple.OpenELM-270M.0a49ab455190e17fcf02d2f21d15e58f6496c8d9.configuration_openelm.OpenELMConfig'> to build an AutoTokenizer.



Hi @DXXS, thanks for opening an issue! 

The reason the message is referring to `meta-llama/Llama-2-7b-hf` is because this is the [tokenizer used for the model](https://huggingface.co/apple/OpenELM-3B-Instruct/blob/56078950638e20d6a747429a5a06e3ca5ed500b1/generate_openelm.py#L21) in the generate_openelm.py script. 

It's not possible to change it to `apple/OpenELM-XXX` as these checkpoints e.g. `apple/OpenELM-3B-Instruct` don't have a [tokenizer defined](https://huggingface.co/apple/OpenELM-3B-Instruct/tree/main) and so no tokenizer can be loaded. You can see if a checkpoint has a tokenizer defined by seeing whether it has tokenizer files listed e.g. [for gpt2 we see](https://huggingface.co/openai-community/gpt2/tree/main): `tokenizer.json`, `vocab.json`, `tokenizer.config`.

To get the script working, you can either: 
* Request access to the [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf) model. This will be the easiest option.
* Use a path to a checkpoint with a different tokenizer. This is highly likely to cause issues, as the mapping between the tokens and the token_ids will not match llama-2's vocabulary, leading to strange generations. 

The 'huggingface' version of github does not appear to allow me to 'open an issue' associated with their code, for some reason, and doing a "% grep -r "Llama-2-7b" * " in the transformers code directory indicates that y'all appear to use it in dozens of places also (!?).  I'm wondering if there may be some way to cause a 'tokenizer to be defined' for said already downloaded model?

@DXXS I'm sorry, I don't fully understand your statement. 

By "The 'huggingface' version of github" are you referring to the hub: https://huggingface.co/? 

> does not appear to allow me to 'open an issue' associated with their code

Whose code? The code on `apple/OpenELM-3B-Instruct`? You can open a "discussion" [under the community tab](https://huggingface.co/apple/OpenELM-3B-Instruct/discussions)

> "% grep -r "Llama-2-7b" * " in the transformers code directory indicates that y'all appear to use it in dozens of places also (!?)

What's the reason for running this? llama 2 checkpoints are used in many places, mostly for the llama models and examples, and shouldn't be a blocker to using the library in general. 

> I'm wondering if there may be some way to cause a 'tokenizer to be defined' for said already downloaded model?

What do you mean by "cause a tokenizer to be defined"? Nothing induces the tokenizer to be defined. If you use `AutoTokenizer.from_pretrained(checkpoint)` the `AutoTokenizer` class will look for all of the tokenizer files under that checkpoint and use those to create the tokenizer class. If those tokenizer files aren't present, an error will be raised. When doing this, first the cache is checked, which will load the tokenizer from those files instead of downloading if present 

Yes, I 'opened a discussion' on the 'OpenELM-3B-Instruct', but it appears to be their 'generate_openelm.py' which is associated with all of their models which has the ~'bug'.

I have not used 'huggingface' before, but being asked to ~sign a legal agreement with an unaffiliated corporation is something I prefer not to have to do with a project claiming to be 'Open Source'.

@DXXS Hugging Face has open source projects such as transformers, which provide the code for loading and using models.
This is separate from the models themselves, which are owned by the organization or user who trained them and determine usage conditions such as the license and whether or not to add gating mechanisms, such as the one for the meta-llama models. It's of course up to you whether you wish to apply to have access. 

You can use the search functionality on the hub to find other models which are not gated e.g. [here's the results for the trending models which do text generation](https://huggingface.co/models?pipeline_tag=text-generation&sort=trending).

This is due to OpenELM using LLama's tokenizer. To use OpenELM you have to also be accepted on Metas LLama 2, which does make sense as it does access that repo license.

You can set the `tokenizer` in the generation function ([here](https://huggingface.co/apple/OpenELM-270M/blob/main/generate_openelm.py#L21)) to bypass the need for the hf_access_token.

For example, `AutoTokenizer.from_pretrained('Xenova/OpenELM-270M-Instruct')` should work (it uses the same tokenizer files).

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.