BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language
  Models via Lens of Dynamic Interactions
Authors: name: Nan Huo; name: Xiaohan Xu; name: Jinyang Li; name: Per Jacobsson; name: Shipei Lin; name: Bowen Qin; name: Binyuan Hui; name: Xiaolong Li; name: Ge Qu; name: Shuzheng Si; name: Linheng Han; name: Edward Alexander; name: Xintong Zhu; name: Rui Qin; name: Ruihan Yu; name: Yiyao Jin; name: Feige Zhou; name: Weihao Zhong; name: Yun Chen; name: Hongyu Liu; name: Chenhao Ma; name: Fatma Ozcan; name: Yannis Papakonstantinou; name: Reynold Cheng
Published: 2025-10-06 19:31:47

Abstract: Large language models (LLMs) have demonstrated remarkable performance on
single-turn text-to-SQL tasks, but real-world database applications
predominantly require multi-turn interactions to handle ambiguous queries,
execution errors, and evolving user requirements. Existing multi-turn
benchmarks fall short by treating conversation histories as static context or
limiting evaluation to read-only operations, failing to reflect
production-grade database assistant challenges. We introduce BIRD-INTERACT, a
benchmark that restores this realism through: (1) a comprehensive interaction
environment coupling each database with a hierarchical knowledge base, metadata
files, and a function-driven user simulator, enabling models to solicit
clarifications, retrieve knowledge, and recover from errors without human
supervision; (2) two evaluation settings consisting of a pre-defined
conversational protocol (c-Interact) and an open-ended agentic setting
(a-Interact) where models autonomously decide when to query the user simulator
or explore the environment; (3) a challenging task suite covering the full CRUD
spectrum for business-intelligence and operational use cases, guarded by
executable test cases. Each task features ambiguous and follow-up sub-tasks
requiring dynamic interaction. The suite comprises BIRD-INTERACT-FULL (600
tasks, up to 11,796 interactions) for comprehensive performance assessment, and
BIRD-INTERACT-LITE (300 tasks with simplified databases) for detailed
behavioral analysis and rapid method development. Our empirical results
highlight BIRD-INTERACT's difficulty: GPT-5 completes only 8.67% of tasks in
c-Interact and 17.00% in a-Interact. Analysis via memory grafting and
Interaction Test-time Scaling validates the importance of effective interaction
for complex, dynamic text-to-SQL tasks.