CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?
Authors: name: Haolin Chen, name: Deon Metelski, name: Leon Qi, name: Tao Xia, name: Joonyul Lee, name: Steve Brown, name: Kevin Riley, name: Frank Wang, name: T. Y. Alvin Liu, name: Hank Capps MD, name: Zeyu Tang, name: Xiangchen Song, name: Lingjing Kong, name: Fan Feng, name: Tianyi Zeng, name: Zhiwei Liu, name: Zixian Ma, name: Hang Jiang, name: Fangli Geng, name: Yuan Yuan, name: Chenyu You, name: Qingsong Wen, name: Hua Wei, name: Yanjie Fu, name: Yue Zhao, name: Carl Yang, name: Biwei Huang, name: Kun Zhang, name: Caiming Xiong, name: Sanmi Koyejo, name: Eric P. Xing, name: Philip S. Yu, name: Weiran Yao
Organization: actAVA AI

End-to-end automation of realistic healthcare operations stresses three capabilities underrepresented in current benchmarks: policy density, decisions must be grounded in a large library of medical, insurance, and operational rules; Multi-role composition: a single task requires the agent to play multiple roles with handoffs; and multilateral interaction: intermediate workflow steps are multi-turn dialogs, such as peer-to-peer review and patient outreach. We introduce χ-Bench, a benchmark of long-horizon healthcare workflows across three domains: provider prior authorization, payer utilization management, and care management. Each task hands the agent a clinical case in a high-fidelity simulator of 20 healthcare apps exposed via 87 MCP tools, which it must drive to a terminal status through tool calls and writing the role's artifacts, guided by a 1,290+ document managed-care operations handbook skill. Across 30 agent harness/models configurations, the best agent resolves only 28.0% of tasks, no agent clears 20% on strict pass^3, and executing all tasks in a single session slumps the performance to 3.8%. These results raise the hypothesis that similar gaps are likely to surface in other policy-dense, role-composed, irreversible enterprise domains.