ZeRO for Memory Optimizations
🚀 Feature
Incorporate the Zero Redundancy Optimizer (ZeRO) for distributed training strategies.
Motivation
Distributed training carries with it a lot of synchronization overheads. Incorporating ZeRO will help alleviate those overheads.
Pitch
ZeRO will allow users to train distributed models with greater resource utilization.
Alternatives
Incorporate all the tricks needed for full fp16 training (as done by OpenAI in GPT3) instead of MixedPrecision training (ZeRO does MP training). But MP is more trusted and I don't know how stable the full fp16 training used in GPT3 really is.