EBench: Elemental Diagnosis of Generalist Mobile Manipulation Policies
Authors: name: Ning Gao, name: Jinliang Zheng, name: Xing Gao, name: Haoxiang Ma, name: Hanqing Wang, name: Yukai Wang, name: Jiantong Chen, name: Zanxin Chen, name: Shujie Zhang, name: Mingda Jia, name: Xuekun Jiang, name: Zihou Zhu, name: Xinyu Li, name: Shuai Wang, name: Hao Li, name: Wenzhe Cai, name: Yuqiang Yang, name: Xudong Xu, name: Zhaoyang Lyu, name: Yao Mu, name: Tai Wang, name: Jiangmiao Pang, name: Jia Zeng, name: Weinan Zhang, name: Chunhua Shen
Organization: Intern Robotics

We present EBench, a simulation benchmark that diagnoses generalist mobile manipulation policies beyond a single success-rate scalar. EBench comprises 26 diverse and challenging manipulation tasks annotated along 5 capability dimensions and 4 generalization dimensions. We evaluate state-of-the-art generalist manipulation models including π_0, π_{0.5}, XVLA, and InternVLA-A1, and reveal that models with near success rates exhibit strikingly different capability profiles: π_{0.5} achieves the highest test success rate and the best train--test retention, whereas InternVLA-A1 dominates mobile manipulation but collapses on dexterous tasks, and XVLA exhibits strengths on a disjoint set of atomic skills compared to other policies. Beyond capability profiling, EBench analyzes the generalization ability from 4 representative perspectives, identifying the impact of different distribution shift factors. The results reveal strengths and weaknesses of models behind an overall score. We hope this benchmark offers a broad set of diagnostic signals to guide iteration on generalist manipulation models.