Before revision:  In this paper we discuss the SW/HW co-designed solution for high-performance distributed training of large-scale DLRMs. We introduce a high-performance scalable software stack based on PyTorch and pair it with the new evolution of   platform, namely   . We demonstrate the capability to train very large DLRMs with up to 12 Trillion parameters and show that we can attain 40\times speedup in terms of time to solution over previous systems.
After revision:  In this paper we discuss the SW/HW co-designed solution for high-performance distributed training of large-scale DLRMs. We introduce a high-performance scalable software stack based on PyTorch and pair it with the new evolution of  platform, namely  . We demonstrate the capability to train very large DLRMs with up to 12 Trillion parameters and show that we can attain 40\times speedup in terms of time to solution over previous systems.