[Deepspeed] ZeRO-Infinity integration plus config revamp
Author: stas00
Pull request: yes
Labels: name: DeepSpeed

This PR:
- [x] integrates ZeRO-Infinity 
- [x] revamps the configuration process, instead of the confusing to users sometimes-we-override-values, sometimes-we-don't - all values are now explicit unless they are set to `auto`, then and only then the Trainer will set them to the correct or recommended values.
- [x] massively revamps the way the configuration is done. now splitting the config parsing into 2 phases - one happening at the very end of `TrainingArguments` and then a weak ref global module var is created which can then be queried by various `transformers` components w/o needing to change any APIs. The global object cleanly goes away when `TrainingArguments` goes out of scope. Users no longer need to make any special calls - just need to ensure the  `TrainingArguments` object is created before `model.from_pretrained()` is called (like we do in all examples). Phase 2 happens during `train` where we get a few variables that weren't there during `TrainingArguments`, so the config gets completed here.
- [x] ds_config is now passed to `zero.Init` in `from_pretrained` under ZeRO-3 since it now needs several configuration values - this is in preparation for fp32 and other important features.
- [x] adds new tests for ZeRO-Inf and configuration.
- [x] adds a minor fix in  `get_regression_trainer`

If you're testing this PR please make sure you install deepspeed master branch:
```
git clone https://github.com/microsoft/DeepSpeed
cd DeepSpeed
pip install -e .
```

## Important changes

Please note a major change is that now only params that are set to `auto` will get automatically overriden/set to the correct/recommended values, everything else is left as is. This is to avoid the previously confusing behavior of never being quite sure what gets overridden and what not despite the logger telling what it did override. The new behavior is completely unambiguous.

See: examples
- [zero2](https://github.com/huggingface/transformers/blob/0f221d2cce751182c455295ef2c03a2c1bd3d66b/tests/deepspeed/ds_config_zero2.json)
- [zero3](https://github.com/huggingface/transformers/blob/0f221d2cce751182c455295ef2c03a2c1bd3d66b/tests/deepspeed/ds_config_zero3.json)


It's ready to release now. 0.3.15 just has a debug print that is loud, fixed in their master.

<!--
TODO:

The following is probably best saved for the next PR as it'd probably require waiting for deepspeed==0.3.16

- [ ] may be revamp the resume to avoid first loading the model weights. can do it in another PR.

PRs waiting to be integrated before this PR can be merged:
- [ ] zero.init() ds_config arg - not yet created
- [ ] new release is needed 0.3.16
-->

@sgugger 

Comments:
> Great job @stas00! I like the solution you picked to be able to start initializing part of deepspeed inside the training arguments. Does it fully solve the chicken and egg problem you add?

Thank you!

For the current needs of `zero.Init()`, yes! As long as the user separates creating `TrainingArguments` from creating the `Trainer` and calling `from_pretrained` in between, which is what all examples do. It took quite a lot of trial and error, but I think it's pretty clean now. 

Splitting the configuration processing in 2 stages helped a lot too.

I hope that me using a weak ref global object is a good solution, since w/o it we would somehow have to make the framework aware of deepspeed in multiple places and somehow pass the config to it - most likely by sticking the DS object into the model object. The neat thing is that it being a weak ref it goes away automatically as soon as the TrainerArguments are `gc`'ed. If down the road we discover a better way nothing prevents us from switching to it.

I will clean up all the XXX's, I was originally planning to wait for 0.3.16 and include all the fixes there, but last time it took more than 10 days for them to make a new release, so I decided it'd be better for users to be able to use this code already, and will make another PR with extra changes next for deepspeed==0.3.16.


The weakref is okay by me. The only other way to achieve this would be to have some "singleton" class where all instances share the same state, but the weakref is actually more adapted in this case.