Optimized Network Architectures for Large Language Model Training with
  Billions of Parameters
Authors: name: Weiyang Wang, name: Manya Ghobadi, name: Kayvon Shakeri, name: Ying Zhang, name: Naader Hasani

This paper challenges the well-established paradigm for building any-to-any
networks for training Large Language Models (LLMs). We show that LLMs exhibit a
unique communication pattern where only small groups of GPUs require
high-bandwidth any-to-any communication within them, to achieve near-optimal
training performance. Across these groups of GPUs, the communication is
insignificant, sparse, and homogeneous. We propose a new network architecture
that closely resembles the communication requirement of LLMs. Our architecture
partitions the cluster into sets of GPUs interconnected with non-blocking
any-to-any high-bandwidth interconnects that we call HB domains. Across the HB
domains, the network only connects GPUs with communication demands. We call
this network a "rail-only" connection, and show that our proposed architecture
reduces the network cost by up to 75% compared to the state-of-the-art
any-to-any Clos networks without compromising the performance of LLM training.