Insights into DeepSeek-V3: Scaling Challenges and Reflections on
  Hardware for AI Architectures
Authors: name: Chenggang Zhao, name: Chengqi Deng, name: Chong Ruan, name: Damai Dai, name: Huazuo Gao, name: Jiashi Li, name: Liyue Zhang, name: Panpan Huang, name: Shangyan Zhou, name: Shirong Ma, name: Wenfeng Liang, name: Ying He, name: Yuqing Wang, name: Yuxuan Liu, name: Y. X. Wei
Organization: DeepSeek

The rapid scaling of large language models (LLMs) has unveiled critical
limitations in current hardware architectures, including constraints in memory
capacity, computational efficiency, and interconnection bandwidth. DeepSeek-V3,
trained on 2,048 NVIDIA H800 GPUs, demonstrates how hardware-aware model
co-design can effectively address these challenges, enabling cost-efficient
training and inference at scale. This paper presents an in-depth analysis of
the DeepSeek-V3/R1 model architecture and its AI infrastructure, highlighting
key innovations such as Multi-head Latent Attention (MLA) for enhanced memory
efficiency, Mixture of Experts (MoE) architectures for optimized
computation-communication trade-offs, FP8 mixed-precision training to unlock
the full potential of hardware capabilities, and a Multi-Plane Network Topology
to minimize cluster-level network overhead. Building on the hardware
bottlenecks encountered during DeepSeek-V3's development, we engage in a
broader discussion with academic and industry peers on potential future
hardware directions, including precise low-precision computation units,
scale-up and scale-out convergence, and innovations in low-latency
communication fabrics. These insights underscore the critical role of hardware
and model co-design in meeting the escalating demands of AI workloads, offering
a practical blueprint for innovation in next-generation AI systems.