GitHub repository: nwang-ibm/fms-fsdp
Description: 🚀 Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flash attention v2.