Add model sharding/gradient checkpointing from FairScale
🚀 Feature

Recently in this paper from Facebook Research they discuss making huge memory savings using checkpointing/pipe parallel and optimizer state checkpointing to train their transformer model.
We've already got integration with #4178 for optimizer state sharding (deepspeed) and with sharded data parallel, but might be worth working with the fairscale team to integrate gradient checkpointing and eventually model pipe sharding (I know Ben is working on this).
cc @williamFalcon @blefaudeux
Once the PR is merged for the initial integration, I'll outline a proposal on how we can introduce these features as configuration options to the fairscale accelerator.