Title: CoTabBench: A Real-World Benchmark for Question Answering over Weakly-Structured and Heterogeneous Tables
Primary area: datasets and benchmarks
Keywords: Table Question Answering, Large Language Models, Benchmark, Real-World Data

Abstract: Recent advancements in Large Language Models (LLMs) have significantly propelled their capabilities in table-based question answering. However, existing benchmarks predominantly feature well-structured tables, failing to address the complexities of real-world data, which is often weakly-structured and contains highly heterogeneous content. This discrepancy limits the evaluation of model robustness on diverse and challenging formats, such as tables with intricate layouts and varied data types found in scientific papers or financial reports. To bridge this gap, we introduce CoTabBench, a large-scale, multi-domain, and intricate benchmark featuring over 2,700 real-world, weakly-structured tables and more than 8,600 question-answer pairs spanning 10 distinct domains. We further propose a novel complexity assessment framework, which quantitatively validates the inherent structural and content-based challenges within CoTabBench. Furthermore, we introduce CoTabInstruct, a large-scale training corpus with over 11,000 tables, and present CoTabLLM, a 7B model trained on it that outperforms even leading models like GPT-4.1 on our benchmark. Extensive experiments reveal a significant performance degradation for state-of-the-art models on CoTabBench, highlighting its critical role in advancing robust, real-world table understanding.

TL;DR: To address the fact that LLMs fail on complex, real-world tables, we created CoTabBench: a comprehensive benchmark and dataset designed to push models beyond simple structured data and foster more robust table understanding.