Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents
Authors: name: Dhaval C. Patel, name: Kaoutar El Maghraoui, name: Shuxin Lin, name: Yusheng Li, name: Tianjun Feng, name: Chun-Yi Tsai, name: Yihan Sun, name: Wei Alexander Xin, name: Akshat Bhandari, name: Tanisha Rathod, name: Aaron Fan, name: Sanskruti Vijay Shejwal, name: Tomas Pasiecznik, name: Sagar Chethan Kumar, name: Tanmay Agarwal, name: Rohith Kanathur, name: Sam Colman, name: Amaan Sheikh, name: Dev Bahl, name: Ann Li, name: Krish Veera, name: Alimurtaza Mustafa Merchant, name: Shambhawi Baswaraj Bhure, name: Sajal Kumar Goyla, name: Chengrui Li, name: Kirthana Natarajan, name: Rui Li, name: Thomas Ajai, name: Rujing Li, name: Vivek G. Iyer, name: Sanjaii Vijayakumar, name: Yitong Bai, name: Ayal Yakobe, name: Darief Maes, name: Yassine Jebbouri, name: Tianyang Xu, name: Thai Quoc On, name: Vera Mazeeva, name: Winston Li, name: Yuval Shemla, name: Yeshitha Bhuvanesh, name: Rushin Bhatt, name: Siddharth Chethan Gowda, name: Alisha Vinod, name: Caroline Cahill, name: Shriya Aishani Rachakonda, name: Yunfeng Chen, name: Aryaman Agrawal, name: Aman Upganlawar, name: Mao Le Jonathan Ang, name: Yubin Sally Go, name: Madhav Rajkondawar, name: Yang-Jung Chen, name: Trisha Maturi, name: Ananya Kapoor, name: Andrew Li, name: Shrey Arora, name: Mana Abbaszadeh, name: Shen Li, name: Charles Xu, name: Byeolah Kwon
Organization: IBM

Agent benchmarks are growing fast, but no single benchmark touches more than four or five of the dimensions that deployment exposes. This paper aggregates the largest coordinated deep-dive of one MCP-based industrial-agent benchmark to date: fourteen parallel implementation studies covering new asset classes (including a multi-modal visual extension), alternative orchestrations, retrieval strategies, reasoning modes, infrastructure optimizations, and evaluation-methodology probes. Consolidating those studies with seven prior agent benchmarks, we argue that aggregate-score leaderboards systematically underspecify deployed-agent evaluation. Rankings derived from aggregate scores do not transfer to out-of-distribution settings; recent public-to-hidden competition retrospectives provide direct empirical evidence of this rank instability. We propose ranking configurations by predictive validity, the correlation between in-sample and out-of-sample rank, rather than in-sample mean, and report a twelve-tier measurement apparatus that exposes the deployment-relevant dimensions HELM and its agent-era successors collapse. The position is operationalized through three falsifiable out-of-distribution criteria with explicit thresholds; existing evidence partly supports it but is too thin to confirm. We close with a pre-registered pilot design and a field-level vision for what the next generation of agentic benchmarks should report.