MSC-Bench: A Rigorous Benchmark for Multi-Server Tool Orchestration
Authors: name: Jia-Kai Dong; name: I-Wei Huang; name: Chun-Tin Wu; name: Yi-Tien Tsai
Published: 2025-10-22 09:45:11

Abstract: We introduce MSC-Bench, a large-scale benchmark for evaluating multi-hop,
end-to-end tool orchestration by LLM agents in a hierarchical Model-Context
Protocol (MCP) ecosystem. Existing benchmarks often evaluate tools in
isolation, ignoring challenges such as functional overlap and cross-server
orchestration, leading to overly optimistic assessments. MSC-Bench addresses
these gaps by constructing ground truth through 'equal function sets', allowing
objective metrics such as F1 score and reducing the dependency on
LLM-as-a-judge evaluation. Organized as a five-level curriculum, it
systematically tests agent capabilities from single-tool orchestration to
complex cross-server planning, and robustness to out-of-scope requests.
Experiments reveal that rigid hierarchies can hinder performance without
co-designed strategies, and even state-of-the-art agents exhibit systemic
weaknesses in robustness. MSC-Bench provides a diagnostic framework to expose
these limitations and guide the development of more capable and efficient
tool-using agents. The benchmark and resources are publicly available at
https://github.com/snooow1029/MSC_Bench.