Natural Language to SQL: Use it on your own database | by Param Raval | Towards Data Science

Not everyone likes or knows how to write an SQL query to search within a huge database. To a layman, it would be a nightmare to construct complex queries with SQL functions and keywords — like JOIN and GROUP BY. A system to convert simple information retrieval questions into a corresponding SQL query would come in handy especially to the banking and healthcare sectors. And with advances in NLP, this could be a reality very soon.
Though there has been substantial research to solve this problem, we are yet to get complete accurate results. Still, from work done on benchmark datasets like WikiSQL, Spider, SPaRC and ATIS, people have managed to build Text-to-SQL systems with decent results. Here, you will see how to implement such a system using the EditSQL model along with a simple workaround to use it on your own schema.
This article is structured as follows:
1. Introduction
2. Installing & Running EditSQL on SParC
3. Making Changes to the Code
4. Adding a Custom Database and Building the Vocab
5. Testing your Question
6. Conclusion
Generating SQL queries from user questions involves solving tasks more than just question-answering and machine translation. Like in question-answering, as the user interacts more, they will often put forward questions that require complex processing such as — a reference to some information mentioned previously or require a combination of several disparate schemas. As a tool, such a system helps the end users, who are often inexperienced in database querying matters, extract information from scores of complex databases.
Several attempts have been made on different benchmark datasets to address such problems in the Text-to-SQL task, especially in semantic parsing. You can check out this interesting survey [4] that introduces the task and the problem quite well.
You can also check out this interactive converter built by the AllenNLP team on the ATIS dataset. Just like several other and better performing models, they use semantic parsing and an encoder-decoder architecture to do the job.
As of July 2020, the leaderboard for Spider and SParC list the following models as some of the best performing models with an opensource code given:
RATSQL v3 + BERT [5]
IRNet + BERT [6]
EditSQL + BERT
Here, you can find the complete leaderboard for Spider and SParC. For practicality, we limit the scope of this article to exploring EditSQL especially on SParC.
While the listed models perform well on their respective datasets, the given codes do not have the option to test the model on a custom database (let alone train). In fact, even to make changes to existing queries of the dataset is a tedious and error-prone process.
However, out of the models given above we can make some changes to EditSQL and manage to run the SParC experiment on a custom SQLite database. These changes are minor and prove to be an easy workaround to test the performance on your own database.
EditSQL attempts to solve a context-dependent text to SQL query generation task and incorporates interaction histories as well as an utterance-table encoder-decoder unit to robustly understand the context of a user’s utterance (or question). To do this, they use an encoder based on BERT which avails in capturing complex database structures and relate them to the utterances. Thus, given an arbitrary question, the model will most certainly identify correctly the database schema to which the question corresponds.
Furthermore, EditSQL takes into account the relation between the user utterance and the table structures as well as recent history of encoding to correctly identify the context. As shown in the diagram above, the information gained here is then passed to a table-aware decoder that uses an attention enhanced LSTM to perform SQL generation.
However, the user often asks questions that contain information provided in a previous interaction. In a sequential generation mechanism this might lead to redundancy in processing and query generation. What gives EditSQL its name is the novel mechanism to “edit” the generated tokens of the query and take care of this problem using another Bi-LSTM paired with the attention enabled context vector.
To run this model, make sure you have a GPU enabled system. Alternatively, you can work on Google Colab.
First off, clone this repository into your system (or in Colab):
git clone https://github.com/ryanzhumich/editsql
Download the SParC dataset to run the experiment from their official page or use the gdown command as follows:
pip install gdowngdown --id 13Abvu5SUMSP3SJM-ZIj66mOkeyAquR73
Place the unzipped folder into editsql/data/sparc
Next, you can follow the instructions in their README or follow these steps:
pip install -r requirements.txt #install dependencies#Download pre-trained BERT modelgdown --id 1f_LEWVgrtZLRuoiExJa5fNzTS8-WcAX9# or download it directly from https://drive.google.com/file/d/1f_LEWVgrtZLRuoiExJa5fNzTS8-WcAX9/view?usp=sharing
And place the pretrained BERT model in the file tree as below:
If the above steps are successfully executed you can run the following bash script to begin training:
bash run_sparc_editsql.sh
The authors have saved their experimental logs at logs/logs_sparc_editsql in log.txt where you can find the details and performance of each epoch.
You will need to successfully execute at least the initial preprocessing and vocab building steps during training. This will create folders named
logs_sparc_editsql
processed_data_sparc_removefrom
processed_data_sparc_removefrom_test
You can interrupt the execution (Ctrl+C) once your output looks like shown below (right before the actual training starts). Number of examples and batch size values might differ as per your configuration.
Successful execution of these steps will create the processed data files and vocab files which are necessary for testing on the development set.
Since we haven’t really performed any “training”, we will need to download the author’s pretrained model and place it under logs/logs_sparc_editsql under the name save_31_sparc_editsql.
gdown --id 1MRN3_mklw8biUphFxmD7OXJ57yS-FkJP#or download directly from #https://drive.google.com/file/d/1MRN3_mklw8biUphFxmD7OXJ57yS-FkJP/view?usp=sharing
Now you are all set to run the bash script to predict on the dev set.
bash test_sparc_editsql.sh
The predictions are saved in a structured JSON file in logs/logs_sparc_editsql under the name dev_use_predicted_queries_predictions.json.
It can be cumbersome to verify each predicted query with its corresponding question. So you can run the following script to get a readable output stored in a file. This script will create output.txt containing the input question, predicted query and the prediction confidence for every question-query pair in the predictions file.
Make sure you have a .sqlite database file containing your SQL database. Let’s say you have a db file named “sales.sqlite”. Put this file in a new folder named “sales” as shown below.
Next, open tables.json found in data/sparc and add the description of your database schema and tables there. Use the structure described in the tables section of the README here. You can use the entries already existing in tables.json as reference and add a similar entry for your schema.
Open dev.json and dev_no_value.json, found in the same directory, and observe the input question structure as shown in the repository of Spider: right here and with examples in the file parsed_sql_examples.sql. All you have to do is replicate one complete entry (everything between the “{“ just above “database_id” and the “}” before the next “database_id” entry).
Once, you append this replicated entry, just edit “utterance” and “utterance_toks” to your desired question and “database_id” to your database name. In our example, it will be “database_id” : “sales”
Finally, add the database_id (your database name) to the list of database names in dev_db_ids.txt (found in the same directory).
During the preprocessing and vocab building step of training we get something called “interactions” written to pickle files and saved in data/sparc_data_removefrom.
These interactions are inherent in the SParC dataset which you can read about in their paper[2] and also in context of EditSQL in section 3.2 of their paper[1].
Similarly, based on the table description in tables.json and training sequences the model creates different vocabulary files for each of train, valid, and dev experiments stored at processed_data_sparc_removefrom.
For this reason, when we add a new database or edit the dev files (like we did in Step 2) we need to run the training script again. Just like in Step 1 we can terminate the script once the preprocessing steps are done.
Note: Please delete all the folders that were created by the training script before running it again. This also includes logs_sparc_editsql/args.log. These files are not updated unless the script gets to create them from scratch.
Run bash test_sparc_editsql.shto get the predicted query for your questions.
Follow the testing steps mentioned in Step 1 and generate output.txt to get a readable view of the results.
Now you can easily convert natural language questions to an SQL query on your own schema.
Thanks for reading!
[1] Zhang, Rui, et al, Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions(2019), arXiv preprint arXiv:1909.00786.
Github: https://github.com/ryanzhumich/editsql
[2] Yu, Tao, et al, Sparc: Cross-domain semantic parsing in context(2019). arXiv preprint arXiv:1906.02285.
Github: https://github.com/taoyds/sparc
[3] Yu, Tao, et al, Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task (2018), arXiv preprint arXiv:1809.08887.
Github: https://github.com/taoyds/spider
[4] Affolter, Katrin, Kurt Stockinger, and Abraham Bernstein, A comparative survey of recent natural language interfaces for databases (2019), The VLDB Journal 28.5: 793–819.
[5] Wang, Bailin, et al, Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers (2019), arXiv preprint arXiv:1911.04942.
[6] Guo, Jiaqi, et al, Towards complex text-to-sql in cross-domain database with intermediate representation (2019), arXiv preprint arXiv:1905.08205.