lr_schedulers: add get_polynomial_decay_schedule_with_warmup
Author: stas00
Pull request: yes
Labels: name: model card

this PR adds a new scheduler plus test, code is based on amalgamation of a few different implementations

I'm not sure it's 100% correct - needs more experimenting - but feedback is welcome.

For reference here are 3 different implementations of this scheduler:
1. https://github.com/pyprob/pyprob/blob/master/pyprob/nn/inference_network.py#L357
2. https://github.com/cmpark0126/pytorch-polynomial-lr-decay/blob/master/torch_poly_lr_decay/torch_poly_lr_decay.py#L5
3. https://github.com/pytorch/fairseq/blob/master/fairseq/optim/lr_scheduler/polynomial_decay_schedule.py - this one has an extra feature `--force-anneal`

@sshleifer 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6361?src=pr&el=h1) Report
> Merging [#6361](https://codecov.io/gh/huggingface/transformers/pull/6361?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/bd0eab351a338175053998ddfc059f1cb6424ab4&el=desc) will **increase** coverage by `0.48%`.
> The diff coverage is `81.05%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6361/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6361?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6361      +/-   ##
==========================================
+ Coverage   79.29%   79.77%   +0.48%     
==========================================
  Files         146      148       +2     
  Lines       26684    27214     +530     
==========================================
+ Hits        21158    21710     +552     
+ Misses       5526     5504      -22     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6361?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/6361/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190NS5weQ==) | `83.33% <0.00%> (-0.38%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_camembert.py](https://codecov.io/gh/huggingface/transformers/pull/6361/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9jYW1lbWJlcnQucHk=) | `100.00% <ø> (ø)` | |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6361/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.29% <ø> (ø)` | |
| [src/transformers/modeling\_tf\_xlm\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/6361/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl94bG1fcm9iZXJ0YS5weQ==) | `100.00% <ø> (ø)` | |
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/6361/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `81.75% <ø> (ø)` | |
| [src/transformers/tokenization\_marian.py](https://codecov.io/gh/huggingface/transformers/pull/6361/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fbWFyaWFuLnB5) | `92.17% <0.00%> (-1.64%)` | :arrow_down: |
| [src/transformers/trainer\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6361/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyX3V0aWxzLnB5) | `100.00% <ø> (+14.28%)` | :arrow_up: |
| [src/transformers/trainer\_tf.py](https://codecov.io/gh/huggingface/transformers/pull/6361/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyX3RmLnB5) | `12.37% <14.81%> (-0.08%)` | :arrow_down: |
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/6361/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `38.02% <32.25%> (-1.04%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/6361/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `64.47% <39.13%> (+0.06%)` | :arrow_up: |
| ... and [45 more](https://codecov.io/gh/huggingface/transformers/pull/6361/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6361?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6361?src=pr&el=footer). Last update [bd0eab3...6e0b1dc](https://codecov.io/gh/huggingface/transformers/pull/6361?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


All done, just need to decide whether to use the default of 1.0 for `power` as in fairseq, or 2.0 (or another value) as it actually does something polynomial.

I run fairseq as recommended [here](https://github.com/pytorch/fairseq/blob/master/examples/mbart/README.md#finetune-on-en-ro) and no, it is using the default power=1.0 at runtime.

I double checked their code, it doesn't get overriden anywhere:
```
fairseq/optim/lr_scheduler/polynomial_decay_schedule.py:        self.power = args.power
fairseq/optim/lr_scheduler/polynomial_decay_schedule.py:        parser.add_argument('--power', default=1.0, type=float)
fairseq/optim/lr_scheduler/polynomial_decay_schedule.py:        print("POWER:", self.power)
fairseq/optim/lr_scheduler/polynomial_decay_schedule.py:            lr = lr_range * pct_remaining ** (self.power) + self.end_learning_rate
```

I will open an issue there and report back. https://github.com/pytorch/fairseq/issues/2466


👍