Insights into DeepSeek-V3: Scaling Challenges and Reflections on
  Hardware for AI Architectures
Authors: name: Chenggang Zhao; name: Chengqi Deng; name: Chong Ruan; name: Damai Dai; name: Huazuo Gao; name: Jiashi Li; name: Liyue Zhang; name: Panpan Huang; name: Shangyan Zhou; name: Shirong Ma; name: Wenfeng Liang; name: Ying He; name: Yuqing Wang; name: Yuxuan Liu; name: Y. X. Wei
Published: 2025-05-14 12:39:03

Abstract: The rapid scaling of large language models (LLMs) has unveiled critical
limitations in current hardware architectures, including constraints in memory
capacity, computational efficiency, and interconnection bandwidth. DeepSeek-V3,
trained on 2,048 NVIDIA H800 GPUs, demonstrates how hardware-aware model
co-design can effectively address these challenges, enabling cost-efficient
training and inference at scale. This paper presents an in-depth analysis of
the DeepSeek-V3/R1 model architecture and its AI infrastructure, highlighting
key innovations such as Multi-head Latent Attention (MLA) for enhanced memory
efficiency, Mixture of Experts (MoE) architectures for optimized
computation-communication trade-offs, FP8 mixed-precision training to unlock
the full potential of hardware capabilities, and a Multi-Plane Network Topology
to minimize cluster-level network overhead. Building on the hardware
bottlenecks encountered during DeepSeek-V3's development, we engage in a
broader discussion with academic and industry peers on potential future
hardware directions, including precise low-precision computation units,
scale-up and scale-out convergence, and innovations in low-latency
communication fabrics. These insights underscore the critical role of hardware
and model co-design in meeting the escalating demands of AI workloads, offering
a practical blueprint for innovation in next-generation AI systems.