LLMSQL: Upgrading WikiSQL for the LLM Era of Text-to-SQL
Authors: name: Dzmitry Pihulski, name: Karol Charchut, name: Viktoria Novogrodskaia, name: Jan Kocoń
Organization: LLMSQL

Converting natural language questions into SQL queries (Text-to-SQL) enables
non-expert users to interact with relational databases and has long been a
central task for natural language interfaces to data. While the WikiSQL dataset
played a key role in early NL2SQL research, its usage has declined due to
structural and annotation issues, including case sensitivity inconsistencies,
data type mismatches, syntax errors, and unanswered questions. We present
LLMSQL, a systematic revision and transformation of WikiSQL designed for the
LLM era. We classify these errors and implement automated methods for cleaning
and re-annotation. To assess the impact of these improvements, we evaluated
multiple large language models (LLMs), including Gemma 3, LLaMA 3.2, Mistral
7B, gpt-oss 20B, Phi-3.5 Mini, Qwen 2.5, OpenAI o4-mini, DeepSeek R1 and
others. Rather than serving as an update, LLMSQL is introduced as an LLM-ready
benchmark: unlike the original WikiSQL, tailored for pointer-network models
selecting tokens from input, LLMSQL provides clean natural language questions
and full SQL queries as plain text, enabling straightforward generation and
evaluation for modern natural language-to-SQL models.