Before revision:  We push the model scale to over 1 trillion parameters and implement it on solely 480 NVIDIA V100-32GB GPUs, in comparison with the recent SOTA Switch Transformer on 2048 TPUs . The proposed giant model achieves substantial speedup in convergence over the same-size baseline.
After revision:  We push the model scale to over 1 trillion parameters and implement it on solely 480 NVIDIA V100-32GB GPUs, in comparison with the recent SOTA Switch Transformer on 2048 TPU cores . The proposed giant model achieves substantial speedup in convergence over the same-size baseline.