Optimized Network Architectures for Large Language Model Training with
  Billions of Parameters
Authors: name: Weiyang Wang; name: Manya Ghobadi; name: Kayvon Shakeri; name: Ying Zhang; name: Naader Hasani
Published: 2023-07-22 21:18:41

Abstract: This paper challenges the well-established paradigm for building any-to-any
networks for training Large Language Models (LLMs). We show that LLMs exhibit a
unique communication pattern where only small groups of GPUs require
high-bandwidth any-to-any communication within them, to achieve near-optimal
training performance. Across these groups of GPUs, the communication is
insignificant, sparse, and homogeneous. We propose a new network architecture
that closely resembles the communication requirement of LLMs. Our architecture
partitions the cluster into sets of GPUs interconnected with non-blocking
any-to-any high-bandwidth interconnects that we call HB domains. Across the HB
domains, the network only connects GPUs with communication demands. We call
this network a "rail-only" connection, and show that our proposed architecture
reduces the network cost by up to 75% compared to the state-of-the-art
any-to-any Clos networks without compromising the performance of LLM training.