add staggered linear LR scheduler
Author: kallewoof
Pull request: yes

The staggered linear LR scheduler uses the epoch to successively modify the learning rate, so that it is constant over each epoch, and modified at the start of each new epoch.

# What does this PR do?

This PR adds a new LR scheduler called "staggered linear LR", or "staggered scheduler with linear modifier"* to be exact. 

A lot of users use a constant learning rate with the mindset that they want to give equal weight/importance to every part of the dataset being trained on. However, it is often hit and miss on figuring out which learning rate works the best, and there is no correction happening naturally, as the LR is the exact same value the entire time.

The staggered linear LR scheduler instead keeps a constant learning rate throughout each epoch, and then modifies it (linearly) by a given modifier. For example, running a 10 epoch training run, with lr=0.2 and staggered modifier 0.01, the learning rate would be:

* 0.200 (epoch 1)
* 0.198 (epoch 2)
* 0.196
* 0.194
* 0.192
* 0.190
* 0.188
* 0.186
* 0.184
* 0.182 (epoch 10)

The idea is to train on the entire dataset with the same learning rate, then modify the learning rate and repeat.

(* This is not a scientific paper. If this name offends or confuses, I will change it.)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

- trainer: @muellerzr and @SunMarc


Comments:
Hi! Do you have any qualitative results on its improvement in your workflow/training? cc @amyeroberts 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31742). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Hi! Do you have any qualitative results on its improvement in your workflow/training? cc @amyeroberts

I don't, yet. This is something that I've heard people ask for for awhile now, though, especially in LLM fine tuning circles. I think it's a bit of a chicken/egg problem; without the feature people can't experiment with it.

> I think it's a bit of a chicken/egg problem; without the feature people can't experiment with it.

@kallewoof It should be possible to run and test from this branch without merging 

> > I think it's a bit of a chicken/egg problem; without the feature people can't experiment with it.
> 
> @kallewoof It should be possible to run and test from this branch without merging

It is. The problem is people are using software that depends on transformers (e.g. Axolotl), not transformers directly. In theory I can make a branch of the software, switch to my branch for this PR for transformers dep, add support for this scheduler, and then try to convince people to try my version.

Alternatively I could just implement the scheduler in their software directly and convince them to allow their users the option of using it. That way no transformers version trickery is involved.

@kallewoof I understand. From the transformers team perspective, we want to add in features which are highly requested, have a clear positive impact or future potential. Every feature we have has a maintenance burden, and removing features is more difficult than adding because of backwards compatibility considerations. 

If you can link to resources showing any of this (community demand / demonstrated performance) then it's an addition which we'd be happy to include. 

It should be possible to run some tests directly on this branch outside of other libraries e.g. Axolotl using `Trainer` to get some preliminary numbers. 

That makes sense, thanks.