Paged Optimizer + Lion Optimizer for Trainer
Author: TimDettmers
Pull request: yes

# What does this PR do?

This PR introduces one new optimizer (Lion) and one new feature from bitsandbytes (paged optimizers) for the trainer `--optim` variable. 

Paged optimizers is an idea that will be published in an upcoming paper where we fine-tune 65B models on a single GPU. Paged optimizers use as much GPU memory as is available, but if less GPU memory is available they automatically switch to a page-by-page transfer mode between the CPU and GPU and transfer just the optimizer states that are needed right now to perform the parameter updates. As such, they are similar to optimizers that are offloaded but Paged optimizer work well with as little as 2 MB of GPU memory and require no user interaction, no extra code, and are failsafe (you cannot do the allocation wrong). If more memory is available, they behave just like any other optimizer -- there is no difference in behavior or performance.

Paged optimizers are particularly useful for training with variable length mini-batches/sequences: if the model fits in the GPU RAM for most mini-batches and hits a mini-batch with very large context/sequence size, then the optimizer will be evicted to the CPU temporarily. Normal optimization resumes after the large mini-batch.

Since these transfers happen page-by-page and the entire system is automatic, the user does not need to do anything for memory benefits and performance considerations.

The only thing that is necessary to use paged optimizer is to pass the specific argument to the trainer: `--optim paged_adamw_32bit` or `--optim paged_lion_32bit` use standard 32-bit AdamW or Lion that are paged.

More details on the algorithm. Paged optimizers work like this:
1. Optimizer states are allocated on the CPU and mapped to a certain GPU device.
2. When an `optimizer.step()` is performed bitsandbytes prefetches the GPU memory page-by-page from the CPU buffer, thus only needing 2MB of GPU memory to perform the optimizer update. If more memory is available, then the swapped-in pages will stay in memory until ...
3. In the case new GPU memory is allocated which exceeds the total GPU RAM capacity, for example, your GPU has 11 GB of RAM and 10.5 GB are used already, and PyTorch allocates another 2 GB of tensors, then the GPU pages for the optimizer are evicted unto the CPU. This happens automatically without user interaction. As such, an out-of-memory event is prevented automatically.

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## About the implementation / Discussion 

I added tests similar to those from 8-bit Adam. I refactored all bnb optimizers into one section of the trainer to reduce bloat. 

Reviewers: @sgugger @younesbelkada  


Comments:
_The documentation is not available anymore as the PR was closed or merged._

This feature truely needed, does there any timeline on when will new release of bitesandbytes?