Step-3 is Large yet Affordable: Model-system Co-design for
  Cost-effective Decoding
Authors: name: StepFun, name: Bin Wang, name: Bojun Wang, name: Changyi Wan, name: Guanzhe Huang, name: Hanpeng Hu, name: Haonan Jia, name: Hao Nie, name: Mingliang Li, name: Nuo Chen, name: Siyu Chen, name: Song Yuan, name: Wuxun Xie, name: Xiaoniu Song, name: Xing Chen, name: Xingping Yang, name: Xuelin Zhang, name: Yanbo Yu, name: Yaoyu Wang, name: Yibo Zhu, name: Yimin Jiang, name: Yu Zhou, name: Yuanwei Lu, name: Houyi Li, name: Jingcheng Hu, name: Ka Man Lo, name: Ailin Huang, name: Binxing Jiao, name: Bo Li, name: Boyu Chen, name: Changxin Miao, name: Chang Lou, name: Chen Hu, name: Chen Xu, name: Chenfeng Yu, name: Chengyuan Yao, name: Daokuan Lv, name: Dapeng Shi, name: Deshan Sun, name: Ding Huang, name: Dingyuan Hu, name: Dongqing Pang, name: Enle Liu, name: Fajie Zhang, name: Fanqi Wan, name: Gulin Yan, name: Han Zhang, name: Han Zhou, name: Hanghao Wu, name: Hangyu Guo, name: Hanqi Chen, name: Hanshan Zhang, name: Hao Wu, name: Haocheng Zhang, name: Haolong Yan, name: Haoran Lv, name: Haoran Wei, name: Hebin Zhou, name: Heng Wang, name: Heng Wang, name: Hongxin Li, name: Hongyu Zhou, name: Hongyuan Wang, name: Huiyong Guo, name: Jia Wang, name: Jiahao Gong, name: Jialing Xie, name: Jian Zhou, name: Jianjian Sun, name: Jiaoren Wu, name: Jiaran Zhang, name: Jiayu Liu, name: Jie Cheng, name: Jie Luo, name: Jie Yan, name: Jie Yang, name: Jieyi Hou, name: Jinguang Zhang, name: Jinlan Cao, name: Jisheng Yin, name: Junfeng Liu, name: Junhao Huang, name: Junzhe Lin, name: Kaijun Tan, name: Kaixiang Li, name: Kang An, name: Kangheng Lin, name: Kenkun Liu, name: Lei Yang, name: Liang Zhao, name: Liangyu Chen, name: Lieyu Shi, name: Liguo Tan, name: Lin Lin, name: Lin Zhang, name: Lina Chen, name: Liwen Huang, name: Liying Shi, name: Longlong Gu, name: Mei Chen, name: Mengqiang Ren, name: Ming Li, name: Mingzhe Chen, name: Na Wang, name: Nan Wu, name: Qi Han, name: Qian Zhao, name: Qiang Zhang, name: Qianni Liu, name: Qiaohui Chen, name: Qiling Wu, name: Qinglin He, name: Qinyuan Tan, name: Qiufeng Wang, name: Qiuping Wu, name: Qiuyan Liang, name: Quan Sun, name: Rui Li, name: Ruihang Miao, name: Ruosi Wan, name: Ruyan Guo, name: Shangwu Zhong, name: Shaoliang Pang, name: Shengjie Fan, name: Shijie Shang, name: Shilei Jiang, name: Shiliang Yang, name: Shiming Hao, name: Shuli Gao, name: Siming Huang, name: Siqi Liu, name: Tiancheng Cao, name: Tianhao Cheng, name: Tianhao Peng, name: Wang You, name: Wei Ji, name: Wen Sun, name: Wenjin Deng, name: Wenqing He, name: Wenzhen Zheng, name: Xi Chen, name: Xiangwen Kong, name: Xianzhen Luo, name: Xiaobo Yang, name: Xiaojia Liu, name: Xiaoxiao Ren, name: Xin Han, name: Xin Li, name: Xin Wu, name: Xu Zhao, name: Yanan Wei, name: Yang Li, name: Yangguang Li, name: Yangshijie Xu, name: Yanming Xu, name: Yaqiang Shi, name: Yeqing Shen, name: Yi Yang, name: Yifei Yang, name: Yifeng Gong, name: Yihan Chen, name: Yijing Yang, name: Yinmin Zhang, name: Yizhuang Zhou, name: Yuanhao Ding, name: Yuantao Fan, name: Yuanzhen Yang, name: Yuchu Luo, name: Yue Peng, name: Yufan Lu, name: Yuhang Deng, name: Yuhe Yin, name: Yujie Liu, name: Yukun Chen, name: Yuling Zhao, name: Yun Mou, name: Yunlong Li, name: Yunzhou Ju, name: Yusheng Li, name: Yuxiang Yang, name: Yuxiang Zhang, name: Yuyang Chen, name: Zejia Weng, name: Zhe Xie, name: Zheng Ge, name: Zheng Gong, name: Zhenyi Lu, name: Zhewei Huang, name: Zhichao Chang, name: Zhiguo Huang, name: Zhirui Wang, name: Zidong Yang, name: Zili Wang, name: Ziqi Wang, name: Zixin Zhang, name: Binxing Jiao, name: Daxin Jiang, name: Heung-Yeung Shum, name: Xiangyu Zhang
Published: 2025-07-25 16:53:13

Large language models (LLMs) face low hardware efficiency during decoding,
especially for long-context reasoning tasks. This paper introduces Step-3, a
321B-parameter VLM with hardware-aware model-system co-design optimized for
minimizing decoding costs. Step-3 innovates in two key dimensions: (1) A novel
Multi-Matrix Factorization Attention (MFA) mechanism that significantly reduces
both KV cache size and computation while maintaining high attention
expressiveness, and (2) Attention-FFN Disaggregation (AFD), a distributed
inference system that decouples attention and Feed-Forward Network (FFN) layers
into specialized subsystems. This co-design achieves unprecedented cost
efficiency: Step-3 significantly reduces theoretical decoding costs compared
with models like DeepSeek-V3 and Qwen3 MoE 235B, with the gains widening at
longer context. Step-3 achieves low cost while activating 38B parameters per
token (more than DeepSeek-V3 and Qwen3 MoE 235B), demonstrating that
hardware-aligned attention arithmetic intensity, MoE sparsity, and AFD are
critical to cost-effectiveness. We perform a head-to-head comparison with
DeepSeek-V3 in its favorable scenarios. Our implementation on Hopper GPUs
achieves a decoding throughput of up to 4,039 tokens per second per GPU under
50ms TPOT SLA (4K context, FP8, no MTP). It is higher than DeepSeek-V3's 2,324
in the same setup and sets a new Pareto frontier for LLM decoding.