Add AdaFactor optimizer from fairseq
Author: moscow25
Pull request: yes

Tested for T5 finetuning and MLM -- reduced memory consumption compared to ADAM.

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #1256


Comments:
Hey @sshleifer -- here is belated PR for AdaFactor. Please let me know how to edit this properly, and what tests or examples we should add. Thanks!

We will integrate into examples/ in a separate PR I think.

Thanks @sshleifer -- let me try to make those changes. 

Agree that I should be able to add a single test -- appreciate the link -- and you can add examples in separate PR.

If I don't get this figure out soon, yes happy for you to make the changes yourself :-)

Hey @sshleifer -- think I got a test working finally. We can squash the commits. 

Still not sure what I need to clean up for the code standards/linter. 

Please advise, thanks!

For local style checking, you need: `pip install isort --upgrade` 
Then `make style` and `make quality` to both suggest you have no errors. 
They should autofix things or at least give error messages. My workflow is to define
```bash
sty () {
	make style
	flake8 examples templates tests src utils
}
```
and then run `sty` a lot.

Also squashing happens automatically at merge time, don't worry about that.

> For local style checking, you need: `pip install isort --upgrade`
> Then `make style` and `make quality` to both suggest you have no errors.
> They should autofix things or at least give error messages. My workflow is to define
> 
> ```shell
> sty () {
> 	make style
> 	flake8 examples templates tests src utils
> }
> ```
> 
> and then run `sty` a lot.

Hmm. Is there a way for `style` to tell me the location in offending file? Output seems pretty minimal.

if you also run the flake8 command it should just fix it.

I think I fixed the formatting, as requested. Took a sec to figure that all out...

@sshleifer  -- any idea what happened with the `black` / code quality changes overnite? I'm very confused. Seems as if the standard changed from yesterday... 

Yes they did, sorry about that. I did some cleanup on this branch.
If you are curious about the style change: I tried to future proof it here https://github.com/huggingface/transformers/pull/6748

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6722?src=pr&el=h1) Report
> Merging [#6722](https://codecov.io/gh/huggingface/transformers/pull/6722?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a75c64d80c76c3dc71f735d9197a4a601847e0cd?el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `68.23%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6722/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6722?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6722      +/-   ##
==========================================
- Coverage   78.96%   78.94%   -0.03%     
==========================================
  Files         157      157              
  Lines       28486    28571      +85     
==========================================
+ Hits        22495    22555      +60     
- Misses       5991     6016      +25     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6722?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/6722/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `99.28% <ø> (ø)` | |
| [src/transformers/optimization.py](https://codecov.io/gh/huggingface/transformers/pull/6722/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9vcHRpbWl6YXRpb24ucHk=) | `82.28% <68.23%> (-13.27%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6722/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `82.41% <0.00%> (-0.26%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6722/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `83.70% <0.00%> (+0.75%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6722?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6722?src=pr&el=footer). Last update [a75c64d...8958b9f](https://codecov.io/gh/huggingface/transformers/pull/6722?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Awesome. Thanks @sshleifer. I'll start working more on the other less mature PRs we discussed. And please ping me if/when you write tests or examples for this. Happy to contribute to that as well if you need.

I've added Adafactor to the docs and slightly changed the style of the docstrings in https://github.com/huggingface/transformers/pull/6765

Thanks! I'll add a `--adafactor` option lightning_base and trainer in 2 prs.