Language Modelling Example
Author: ajherman
Pull request: no
Labels: name: Examples

### System Info

I'm running the first example in the examples/pytorch/language-modelling/README.md. I have installed the requirements.txt. I consistently get the following error (sorry for the screen grab, I have issues with copy/paste in this environment):

![Screenshot from 2024-04-23 21-14-11](https://github.com/huggingface/transformers/assets/15025931/28ae6c35-161e-4551-9d93-049b4fff0a7f)

I don't understand why I am getting this key error from openai-community/gpt2. Incidentally, when I try run a script in the same environment that simply has the line model = from_pretrained('openai-community/gpt2'), I do not get any errors. So, it seems like it must be an issue in the example code? That is, the run_clm.py.



### Who can help?

ajherman

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

python run_clm.py \
    --model_name_or_path openai-community/gpt2 \
    --train_file path_to_train_file \
    --validation_file path_to_validation_file \
    --per_device_train_batch_size 8 \
    --per_device_eval_batch_size 8 \
    --do_train \
    --do_eval \
    --output_dir /tmp/test-clm

### Expected behavior

Should run the example script...

Comments:
Hi @ajherman, can you share your running environment? Run `transformers-cli env` in the terminal and copy-paste the output

I'm able to run the example from the README without issue on main:

```
python run_clm.py \
    --model_name_or_path openai-community/gpt2 \
    --dataset_name wikitext \
    --dataset_config_name wikitext-2-raw-v1 \
    --per_device_train_batch_size 8 \
    --per_device_eval_batch_size 8 \
    --do_train \
    --do_eval \
    --output_dir /tmp/test-clm
```

cc @ArthurZucker @younesbelkada 

![Screenshot from 2024-04-24 10-33-47](https://github.com/huggingface/transformers/assets/15025931/5a4908b9-d5e3-42ee-b44f-bdc5bc40cd4e)

For the last two points:

Yes, I'm using a GPU. 
I think what I'm using would be considered parallel computing? But I'm running on a remote server and I'm honestly not completely sure about the details of the set up. 

I should mention additionally, that I want to run this script with SLURM, but I get the same error whether I run through an SBATCH script or just run the command directly in the terminal. 

@ajherman Are you able to run the example I shared i.e. one with a common dataset? 

Ah, yes, the example script you provided works! Or at the very least, it begins the training loop (it hasn't gotten past 0%, but at least I'm not getting the key error anymore). 

What does this mean...?

Are you by any change passing `--model_type` in as an argument? From the error it looks like the model type is being passed in as `openai-community/gpt2` when it should be `gpt2`. 

There's a few places where this is wrong in our docs. I've opened #30480 to fix these 

Thank you! Yes, this appears to have solved the problem. As a note, the confusion came from the end of the examples/pytorch/language-modelling/README.md file. In the example command, it passes openai-community/gpt2 as the --model_type argument. I changed it to gpt2, and it now seems to work. 

The same command passes the same string to --model_tokenizer, but I did not have to modify that one. Could you give me some insight into how the naming is intended to work? What is the implied meaning of including or not including something like "openai-community" before the model name? Thanks for your help!

GPT2 is a bit special. I'll use another model first to explain the general case, and then explain why GPT2 different. 

When loading a model using a checkpoint e.g. `AutoModel.from_pretrained(checkpoint)` the checkpoint will have two parts: `organization/model-name`. The first part tells us under which organization or user the model sits under on the hub; the second part points to the specific model checkpoint. For example, with `meta-llama/Meta-Llama-3-8B` [meta-lama is the organization](https://huggingface.co/meta-llama), which can have many models, datasets and spaces under its domain. The second part specifies the [specific model checkpoint](https://huggingface.co/meta-llama/Meta-Llama-3-8B). 

Another this to note is the difference between a checkpoint and a model type. In the script `--model_name_or_path` refers to a specific checkpoint which will have this `organization/model` structure (or a local path). `--model_type` refers to a general architecture e.g. [all of the llama-2 models](https://huggingface.co/collections/meta-llama/llama-2-family-661da1f90a9d678b6f55773b) have the [model type llama](https://github.com/huggingface/transformers/blob/a98c41798cf6ed99e1ff17e3792d6e06a2ff2ff3/src/transformers/models/auto/modeling_auto.py#L136) e.g. [here in the config](https://huggingface.co/meta-llama/Llama-2-7b-hf/blob/01c7f73d771dfac7d292323805ebc428287df4f9/config.json#L13) i.e. they can all be loaded into the [transformers llama architecture](https://github.com/huggingface/transformers/blob/a98c41798cf6ed99e1ff17e3792d6e06a2ff2ff3/src/transformers/models/llama/modeling_llama.py#L4). 

Many years ago, when some models were added to to Hugging Face, we didn't have this standard, and just the model name was used e.g. `bert-base-uncased`. The same is true for `gpt2`. Rather confusingly, not only was `gpt2` the checkpoint name, but it's also the [model type](https://github.com/huggingface/transformers/blob/a98c41798cf6ed99e1ff17e3792d6e06a2ff2ff3/src/transformers/models/auto/modeling_auto.py#L111). Unlike `bert-base-uncased` whose model type is `bert`. A few months ago, we updated all the checkpoints on the hub for these old models to the canonical `organization/model` format, [and the examples were updated too](https://github.com/huggingface/transformers/pull/29001). Some of the references were mistakenly changed here for gpt2 because of this model name<->model type clash. Meaning model_type was updated to `openai-community/gpt2` when it should have been left as gpt2 (the model architecture doesn't change just the checkpoint on the hub).

Now, to avoid breaking existing scripts and code, we maintain backwards compatibility and the old checkpoints (`bert-base-uncased`, `gpt2`, ...) will STILL map to the old checkpoints i.e. I can do `AutoModel.from_pretrained('gpt2')` even though I should do `AutoModel.from_pretrained('openai-community/gpt2')`. 

So, you can't change `--model_type` to `openai-community/gpt2` as this model architecture doesn't exist (it's a checkpoints), but you can still do `--model_type gpt2 --model_path_or_name gpt2` because gpt2 is an old model and this backwards compatible behaviour. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.