Package: hetu-galvatron
GalvatronGalvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism.A new system framework that incorporates multiple popular parallelism dimensions and automatically finds the most efficient hybrid parallelism strategy.Environment SetupRun the following script to prepare the conda environment:condacreate-ngalvatronpython=3.8
condaactivategalvatroncdGalvatron
shprepare_env.shGalvatron System ArchitectureGalvatron is consisted of four modules, including an automatic profiler, a strategy cost estimator, parallelism optimization block, and Galvatron runtime framework. To train Transformer models over multiple GPUs using automatic parallelism with Galvatron, users only need to provide with hardware environment and the Transformer model configuration.UsageProfiling with GalvatronThe first step to use Galvatron is to profile the hardware environment and the model forward computation time. Galvatron will automatically write the tested results into config files.(1) Firstly, profile the hardward environment:For 4 GPUs on 1 node, run the following scripts:cd./test_env
shprofile_env_4gpus.shFor 8 GPUs on 1 node, run the following scripts:cd./test_env
shprofile_env_8gpus.shFor 16 GPUs on 2 nodes, first export the rightMASTER_ADDR,NCCL_SOCKET_IFNAMEandNODE_RANKinto environment for multi-node test, and then run the following scripts on both nodes:cd./test_env
shprofile_env_16gpus.sh(2) Secondly, profile the forward computation time of the chosen Transformer model.cd./model_name
shscripts/forward_profiling.shOptimizing with GalvatronAfter profiling the environments, Galvatron is able to automatically optimize the parallelism strategy for the given Transformer model. Currently, Galvatron support parallelism optimization on BERT, T5, ViT, and Swin Transformer. Given the memory budget, Galvatron provides the layerwise hybrid parallel strategy with maximum throughput. Note that the strategy cost estimator block is executed automatically before parallelism optimization.The optimized parallelism strategy will be saved in./model_name/configsfor the training. Users can train the model with the provided optimal strategy to obtain the optimal throughput.For 4 GPUs on 1 node, run the following scripts to conduct parallelim optimization:cd./model_name
shscripts/search_layerwise_hp_4gpus.shFor 8 GPUs on 1 node, run the following scripts to conduct parallelim optimization:cd./model_name
shscripts/search_layerwise_hp_8gpus.shFor 16 GPUs on 2 nodes, run the following scripts to conduct parallelim optimization:cd./model_name
shscripts/search_layerwise_hp_dist_16gpus.shThe Transformer model configuration can be assigned in the scripts above by modifying only a few arguments. Enter the corresponding directory of the model to see detailed guidance.Training with GalvatronGalvatron provides a simple way to train Transformer models in layerwise hybrid parallel fashion. We provide four models as examples in this repo, including BERT, T5, ViT, Swin Transformer, and provide scripts to train these models on single GPU or on multiple GPUs using hybrid parallel strategy.Users can either train Transformer models with the optimal parallel strategy searched by the parallelsim optimization block to obtain the optimal throughput, or use any hybrid parallel strategy as they like. Galvatron support three hybrid parallel config modes, including JSON config mode, PYTHON config mode, and GLOBAL config mode. Users can specify hybrid parallel strategies by modifying only a few arguments or a few lines of codes. Enter the corresponding directory of the model to see detailed guidance.It is fairly simple to construct a layerwise hybrid parallel model in Galvatron, which only requires a few modification to the training scripts on single GPU.In the model directorybert, t5, vit, swin, to train the model on single GPU, run:shscripts/train_xxx.shTo train the model on 4 GPUs on 1 node, run:shscripts/train_xxx_hp_layerwise_4gpus.shTo train the model on 8 GPUs on 1 node, run:shscripts/train_xxx_hp_layerwise.shTo train the model on 16 GPUs on 2 nodes, first export the rightMASTER_ADDR,NCCL_SOCKET_IFNAMEandNODE_RANKinto the environment for multi-node test, and then run:shscripts/train_xxx_hp_layerwise_dist_2nodes.sh