[Examples] create model with custom config on the fly
Author: stas00
Pull request: yes

This PR is addressing a need to:
1. be able to quickly whip up a model of any desired size for the big-science experiments.
2. be able to activate gradient checkpointing (later addition)

We already have the functionality to create a model instead of using a pretrained one, but there was no way to control its config - it would choose the defaults of the Config object, which is very doubtful is of any practical use.

This PR:

1. adds a new `PretrainedConfig` method: `update_from_string` so one can update from a string.
```
config.update_from_string("n_embd=10,n_head=5,scale_attn_weights=false,summary_type=super_cls_index")
```

   plus test.

2. adds a new `ModelArguments` arg: `--config_overrides="n_embd=1024,n_head=16,n_layer=48,n_positions=102"` which overrides the default config

3. auto-logs the resulting model size e.g.:
```
Training new model from scratch - Total size=626.69M params
```

Usage:
```
PYTHONPATH=src python examples/pytorch/language-modeling/run_clm.py --dataset_name \
"stas/openwebtext-10k" --output_dir output_dir --overwrite_output_dir --do_train --do_eval \
--max_train_samples 10000 --max_eval_samples 1000 --per_device_train_batch_size 4 \
--per_device_eval_batch_size 4 --num_train_epochs 1 --warmup_steps 8 --block_size 64 --fp16 \
--report_to none --model_type gpt2 --tokenizer_name gpt2 \
--config_overrides "n_embd=1024,n_head=16,n_layer=48,n_positions=1024"
```

Only `run_clm.py` for this experiment.

@sgugger, @LysandreJik 


Comments:
So now can we activate activation checkpointing with: `--config_overrides "gradient_checkpointing=true,use_cache=False"`

1. Should we document this somewhere? maybe `examples/pytorch/README.md` once we port this to all other examples?

2. But it's only available for non-pretrained model, should I make `config_overrides` available to any model? i.e. this change:

```
--- a/examples/pytorch/language-modeling/run_clm.py
+++ b/examples/pytorch/language-modeling/run_clm.py
@@ -286,9 +286,10 @@ def main():
     else:
         config = CONFIG_MAPPING[model_args.model_type]()
         logger.warning("You are instantiating a new config instance from scratch.")
-        if model_args.config_overrides is not None:
-            logger.info(f"Overriding config: {model_args.config_overrides}")
-            config.update_from_string(model_args.config_overrides)
+
+    if model_args.config_overrides is not None:
+        logger.info(f"Overriding config: {model_args.config_overrides}")
+        config.update_from_string(model_args.config_overrides)
```

It could invite problems for config sections which have to match the pre-trained weights, but otherwise should give users more flexibility. e.g. allow turning caching off, grad checkpointing on and perhaps do other things that aren't impacted by pretrained weights.

This option does not make any sense for pretrained models: in the best case the user will get an error of weights shape mismatch, in the worst case it will just silently yield crappy results.
Thus, the option does not make sense IMO for scripts not used for training models from scratch, have to check manually but I think it's just the scripts for language-modeling which offer that option, so in this case only document the option in their README.

> [...] have to check manually but I think it's just the scripts for language-modeling which offer that option, so in this case only document the option in their README.

by "that option" do you mean "gradient_checkpointing"? If so it's available in 30 models out of 59:
```
$ grep -Irl gradient_checkpointing src/transformers/models/*/modeling* | wc -l
30
$ ls -l src/transformers/models/*/modeling* | egrep -v '(flax|tf)'| wc -l
59
```

No I meant the option of training from scratch. I did double check, and it's only in the LM scripts.

Right, and I was talking about documenting ` --config_overrides "gradient_checkpointing=true,use_cache=False"`

which could apply to any model. (but is not coded to support that at the moment).

And you did mention elsewhere that this feature is on a todo list.

Yes, it will be a regular training argument in the future.

Excellent point, @LysandreJik.

I did both.

Though warning not, assert yes.