Step-3 is Large yet Affordable: Model-system Co-design for
  Cost-effective Decoding
Authors: name: StepFun; name: Bin Wang; name: Bojun Wang; name: Changyi Wan; name: Guanzhe Huang; name: Hanpeng Hu; name: Haonan Jia; name: Hao Nie; name: Mingliang Li; name: Nuo Chen; name: Siyu Chen; name: Song Yuan; name: Wuxun Xie; name: Xiaoniu Song; name: Xing Chen; name: Xingping Yang; name: Xuelin Zhang; name: Yanbo Yu; name: Yaoyu Wang; name: Yibo Zhu; name: Yimin Jiang; name: Yu Zhou; name: Yuanwei Lu; name: Houyi Li; name: Jingcheng Hu; name: Ka Man Lo; name: Ailin Huang; name: Binxing Jiao; name: Bo Li; name: Boyu Chen; name: Changxin Miao; name: Chang Lou; name: Chen Hu; name: Chen Xu; name: Chenfeng Yu; name: Chengyuan Yao; name: Daokuan Lv; name: Dapeng Shi; name: Deshan Sun; name: Ding Huang; name: Dingyuan Hu; name: Dongqing Pang; name: Enle Liu; name: Fajie Zhang; name: Fanqi Wan; name: Gulin Yan; name: Han Zhang; name: Han Zhou; name: Hanghao Wu; name: Hangyu Guo; name: Hanqi Chen; name: Hanshan Zhang; name: Hao Wu; name: Haocheng Zhang; name: Haolong Yan; name: Haoran Lv; name: Haoran Wei; name: Hebin Zhou; name: Heng Wang; name: Heng Wang; name: Hongxin Li; name: Hongyu Zhou; name: Hongyuan Wang; name: Huiyong Guo; name: Jia Wang; name: Jiahao Gong; name: Jialing Xie; name: Jian Zhou; name: Jianjian Sun; name: Jiaoren Wu; name: Jiaran Zhang; name: Jiayu Liu; name: Jie Cheng; name: Jie Luo; name: Jie Yan; name: Jie Yang; name: Jieyi Hou; name: Jinguang Zhang; name: Jinlan Cao; name: Jisheng Yin; name: Junfeng Liu; name: Junhao Huang; name: Junzhe Lin; name: Kaijun Tan; name: Kaixiang Li; name: Kang An; name: Kangheng Lin; name: Kenkun Liu; name: Lei Yang; name: Liang Zhao; name: Liangyu Chen; name: Lieyu Shi; name: Liguo Tan; name: Lin Lin; name: Lin Zhang; name: Lina Chen; name: Liwen Huang; name: Liying Shi; name: Longlong Gu; name: Mei Chen; name: Mengqiang Ren; name: Ming Li; name: Mingzhe Chen; name: Na Wang; name: Nan Wu; name: Qi Han; name: Qian Zhao; name: Qiang Zhang; name: Qianni Liu; name: Qiaohui Chen; name: Qiling Wu; name: Qinglin He; name: Qinyuan Tan; name: Qiufeng Wang; name: Qiuping Wu; name: Qiuyan Liang; name: Quan Sun; name: Rui Li; name: Ruihang Miao; name: Ruosi Wan; name: Ruyan Guo; name: Shangwu Zhong; name: Shaoliang Pang; name: Shengjie Fan; name: Shijie Shang; name: Shilei Jiang; name: Shiliang Yang; name: Shiming Hao; name: Shuli Gao; name: Siming Huang; name: Siqi Liu; name: Tiancheng Cao; name: Tianhao Cheng; name: Tianhao Peng; name: Wang You; name: Wei Ji; name: Wen Sun; name: Wenjin Deng; name: Wenqing He; name: Wenzhen Zheng; name: Xi Chen; name: Xiangwen Kong; name: Xianzhen Luo; name: Xiaobo Yang; name: Xiaojia Liu; name: Xiaoxiao Ren; name: Xin Han; name: Xin Li; name: Xin Wu; name: Xu Zhao; name: Yanan Wei; name: Yang Li; name: Yangguang Li; name: Yangshijie Xu; name: Yanming Xu; name: Yaqiang Shi; name: Yeqing Shen; name: Yi Yang; name: Yifei Yang; name: Yifeng Gong; name: Yihan Chen; name: Yijing Yang; name: Yinmin Zhang; name: Yizhuang Zhou; name: Yuanhao Ding; name: Yuantao Fan; name: Yuanzhen Yang; name: Yuchu Luo; name: Yue Peng; name: Yufan Lu; name: Yuhang Deng; name: Yuhe Yin; name: Yujie Liu; name: Yukun Chen; name: Yuling Zhao; name: Yun Mou; name: Yunlong Li; name: Yunzhou Ju; name: Yusheng Li; name: Yuxiang Yang; name: Yuxiang Zhang; name: Yuyang Chen; name: Zejia Weng; name: Zhe Xie; name: Zheng Ge; name: Zheng Gong; name: Zhenyi Lu; name: Zhewei Huang; name: Zhichao Chang; name: Zhiguo Huang; name: Zhirui Wang; name: Zidong Yang; name: Zili Wang; name: Ziqi Wang; name: Zixin Zhang; name: Binxing Jiao; name: Daxin Jiang; name: Heung-Yeung Shum; name: Xiangyu Zhang
Published: 2025-07-25 16:53:13

Abstract: Large language models (LLMs) face low hardware efficiency during decoding,
especially for long-context reasoning tasks. This paper introduces Step-3, a
321B-parameter VLM with hardware-aware model-system co-design optimized for
minimizing decoding costs. Step-3 innovates in two key dimensions: (1) A novel
Multi-Matrix Factorization Attention (MFA) mechanism that significantly reduces
both KV cache size and computation while maintaining high attention
expressiveness, and (2) Attention-FFN Disaggregation (AFD), a distributed
inference system that decouples attention and Feed-Forward Network (FFN) layers
into specialized subsystems. This co-design achieves unprecedented cost
efficiency: Step-3 significantly reduces theoretical decoding costs compared
with models like DeepSeek-V3 and Qwen3 MoE 235B, with the gains widening at
longer context. Step-3 achieves low cost while activating 38B parameters per
token (more than DeepSeek-V3 and Qwen3 MoE 235B), demonstrating that
hardware-aligned attention arithmetic intensity, MoE sparsity, and AFD are
critical to cost-effectiveness. We perform a head-to-head comparison with
DeepSeek-V3 in its favorable scenarios. Our implementation on Hopper GPUs
achieves a decoding throughput of up to 4,039 tokens per second per GPU under
50ms TPOT SLA (4K context, FP8, no MTP). It is higher than DeepSeek-V3's 2,324
in the same setup and sets a new Pareto frontier for LLM decoding.