Prompting LLMs in Italian language for Text-to-SQL translation
Authors: Federico Ranaldi, Elena Sofia Ruzzetti, Leonardo Ranaldi, Davide Venditti, Cristina Giannone, Andrea Favalli, Raniero Romagnoli, Fabio Massimo Zanzotto (Università di Roma Tor Vergata, Almawave, Idiap Research Institute, 2023)
Topic: Language Models

Abstract
Fine-tuning Large Language Models (LLMs) on tasks with instructions has demonstrated potential in enhancing zero-shot generalization to unseen tasks. Inspired by studies on the reasoning capabilities of Instruction-tuned LLMs (It-LLMs), we investigate their performance on reading comprehension, reasoning, and production across symbolic tasks. Specifically, we propose an iterative reading-comprehension and reasoning approach to tackle question-answering tasks based on structured data, such as the Text-to-SQL task. In our approach, we define a specialized procedure to extract relevant evidence from structured data and natural language queries, guiding It-LLMs to focus on both the production task and reasoning process. To facilitate this, we introduce a prompting generation procedure that allows It-LLMs to reason about the structural information and natural language queries, ultimately producing symbolic output in the form of SQL queries. Extensive zero-shot experiments with various types of structured data reveal the superhuman abilities of It-LLMs in both comprehension and production, yielding astonishingly accurate answers. However, the models also generate hallucinations and misleading answers, highlighting the limitations of instruction-tuned LLMs and their partial unreliability.

Introduction
The development of Large Language Models (LLMs) has been one of the most significant advances in Natural Language Processing (NLP). LLMs, after being pre-trained on immense corpora, have demonstrated superhuman performance on language modeling tasks. Recent advances show that LLMs are capable of zero-shot task generalization, meaning they can adapt to unknown tasks without fine-tuning. In this context, Instruction-tuning has emerged as a promising direction. Instruction-tuning enables LLMs to follow instructions across different tasks and perform well even in tasks they haven’t been explicitly trained for. Instruction-based tuning typically involves either crowd-sourced human tasks or model-generated tasks. While these methods have shown promise, they are limited in both quantity and quality. The scalability of LLMs in various dimensions has demonstrated the potential to overcome the limitations of zero-shot performance, which has led to an increasing focus on the search for high-quality and scalable Instruction-tuning tasks. Despite their success, recent work has highlighted that Instruction-tuned LLMs (It-LLMs) can sometimes generate misleading information that conflicts with factual knowledge, struggle with domain-specific knowledge, and produce hallucinated answers by stretching their generative capabilities. To address these issues, researchers like Zhou et al. have proposed efficient methods for providing optimal prompts, while Jang et al. and Arora et al. have focused on enhancing the models' understanding of the prompts. In this paper, we propose an iterative reading-comprehension and reasoning approach to solve question-answering tasks based on structured data. Specifically, we reframe the Text-to-SQL task within a prompt-based framework and define a specialized procedure for extracting relevant evidence from structured data. The query is then posed to It-LLMs in natural language. This approach directs the models to focus on understanding the prompt, reasoning with the provided information, and generating the output—the SQL code that solves the Text-to-SQL task. Our extensive experiments, conducted in zero-shot scenarios with different types of structured data, demonstrate the remarkable abilities of It-LLMs in both understanding and producing accurate responses in the presence of varying levels of information. However, we also observed errors when the provided information was limited. The results of the zero-shot experiments reveal the shortcomings of It-LLMs, showing their partial unreliability, especially when dealing with harder queries and less informative databases.

Conclusion
In this paper, we propose an iterative reading-comprehension and reasoning approach to solve question-answering challenges of the Text-to-SQL task. The results obtained from the experiments conducted in this work witness the potential of Instruction-tuned Large Language Models (It-LLMs). However, despite their promising performance, certain limitations have emerged. We discovered that even with minimal information about the database, It-LLMs can generate natural language query translations that yield correct and executable SQL queries by just prompting them. Nevertheless, it became evident that reducing the amount of information provided could lead to the generation of incorrect queries. Expanding the scope of our investigation, we believe it would be worthwhile to conduct similar experiments with other It-LLMs. Such comparisons could help determine whether the common phenomena observed in both tested models result from a coincidence or represent aspects to further investigate in studying these new technologies. In conclusion, this research underscores the substantial advancements offered by It-LLMs in the realm of Text-to-SQL translation while also highlighting the implications of choosing whether to provide more or less information during the prompting process.