[Deepspeed alternative] PatrickStar
Author: flozi00
Pull request: no

### Feature request

https://github.com/Tencent/PatrickStar

Adding PatrickStar as alternative to deepspeed

### Motivation

I think it could be interesting to benchmark with deepspeed.
In their Readme they are writing it's faster then deepspeed zero 3. 
But they are also writing, gradient accumulation is not possible with the library.

pinging @stas00 for interest ?

### Your contribution

I could give it a try integrating it into the trainer

Comments:
@flozi00, 

Thank you for the heads up about this framework.

I read the overview https://github.com/Tencent/PatrickStar/blob/master/INSIDE.md and it looks like they are trying to solve problems that have been solved in Deepspeed many moons ago - perhaps they started working on this project long time ago and are referring to a really old deepspeed version? e.g. param offload has been implemented long time ago there. And of course CPU offload implements prefetching, which happens in parallel with compute.

I will try to find time to read their paper: https://arxiv.org/abs/2108.05818 to understand what innovation it has proposed with the chunked memory management.

Before doing an integration probably the first good step would be to try to reproduce their benchmarks and the current deepspeed side by side to compare the performance and see if it's indeed offering an improvement.

To update - after you shared this a few weeks ago I've requested with the deepspeed devs to look into PatrickStar and see if they could Match the performance - that way we don't need to add a complicated support for another framework. Let's see what unfolds. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@flozi00 @stas00 Thanks for your attention on PatrickStar. 
I would like to clarify that the comparison results of PatrickStar and DeepSpeed is repeatly verified by many users.
The results in the paper are still valid in May 2022 with the latest version DeepSpeed at that time. 
Also, the PatrickStar idea has already been integrated into ColossalAI and played a key role.
I strongly believe that PatrickStar's design, or part of it, will inherently accelerate the training of large models. I'm open to any help with using Patrickstar, either integrated into DeepSpeed or huggingface transformers.

Thank you for your commentary and willingness to contribute, @feifeibear 

> I strongly believe that PatrickStar's design, or part of it, will inherently accelerate the training of large models. I'm open to any help with using Patrickstar, either integrated into DeepSpeed or huggingface transformers.

I think it'd be amazing to have it integrated into Deepspeed. I'm tagging @tjruwase (Deepspeed) on this suggestion. Perhaps let's start a new DS-specific thread at https://github.com/microsoft/DeepSpeed/issues?

> Also, the PatrickStar idea has already been integrated into ColossalAI and played a key role.

Indeed, we are discussing the CAI integration here: https://github.com/huggingface/transformers/issues/18624


Thanks! I will keep an eye on the CAI issue! And feel free to contact me if I can help!