--config_overrides doesn't appear to work in run_clm.py when trying to specify a larger GPT model
Author: Adrian-1234
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.10.2

- Platform: Linux-5.4.0-84-generic-x86_64-with-Ubuntu-20.04-focal

- Python version: 3.7.12

- PyTorch version (GPU?): 1.9.0+cu111 (True)
- Tensorflow version (GPU?): 2.7.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- GPT-2, GPT: @patrickvonplaten, @LysandreJik

If the model isn't in the list, ping @LysandreJik who will redirect you to the correct contributor.

Library:

Documentation: @sgugger

Model hub:

- for issues with a model, report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

For research projetcs, please ping the contributor directly. For example, on the following projects:

- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Using the following to train GPT-2 from scratch:
```
python3.7 run_clm.py \
--model_type "gpt2" \
--tokenizer_name "gpt2" \
--train_file "train_tmp.txt" \
--validation_file "eval_tmp.txt" \
--pad_to_max_length yes \
--do_train \
--do_eval \
--max_seq_length=1024 \
--per_gpu_train_batch_size 1 \
--save_steps -1 \
--num_train_epochs 10 \
--fp16_full_eval \
--output_dir=checkpoints \
--config_overrides="n_embd=1024,n_head=16,n_layer=24,n_positions=1024,n_ctx=1024,layer_norm_epsilon=1e-5,initializer_range=0.02"
```


The --config_overrides  doesn't appear to take effect:

Starting the training o/p's :
```
Model config GPT2Config {
  "activation_function": "gelu_new",
  "architectures": [
    "GPT2LMHeadModel"
  ],
  "attn_pdrop": 0.1,
  "bos_token_id": 50256,
  "embd_pdrop": 0.1,
  "eos_token_id": 50256,
  "gradient_checkpointing": false,
  "initializer_range": 0.02,
  "layer_norm_epsilon": 1e-05,
  "model_type": "gpt2",
  "n_ctx": 1024,
  "n_embd": 768,
  "n_head": 12,
  "n_inner": null,
  "n_layer": 12,
  "n_positions": 1024,
  "resid_pdrop": 0.1,
  "scale_attn_weights": true,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "task_specific_params": {
    "text-generation": {
      "do_sample": true,
      "max_length": 50
    }
  },
  "transformers_version": "4.10.2",
  "use_cache": true,
  "vocab_size": 50257
}
```


## To reproduce

Steps to reproduce the behavior:

1. Running the above training script ignore the parameters in --config_overrides


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

I was expecting the --config_overrides string to override the training parameters. Although I see documentation  suggesting that it is possible to specify something like --model_type="gpt2-medium"   this produces an error such as no such model.

Perhaps there is an alternative way to specify a medium or large GPT-2 model ?

Thanks.


Comments:
Hey @Adrian-1234, we recommend using the `model_name_or_path` parameter to specify a particular checkpoint.
cc @sgugger 

`--config_overrides` is actually an addition by @stas00 

I will try to reproduce the issue and will then follow up.

I edited the OP to add formatting.

There is no problem, other than logger misinformation. 

Please see https://github.com/huggingface/transformers/pull/14466 for details.