course5i/NatSight-t5-small-wikisql

Paper

NatSight: A framework for building domain agnostic Natural Language Interface to Databases for next-gen Augmented Analytics

Authors: Rohit Sroch, Dhiraj Patnaik, Jayachandran Ramachandran

Abstract

In modern organizations, a large volume of customer, transactional, and operational data is stored in relational database management systems (RDBMS). It provides scalability and agility for various business use cases. However, the interaction between these databases and business users is limited as users often lack the knowledge of query languages such as SQL and they need to rely on technical experts to interact with the database for curating insights & analytics. Recent advances in augmented analytics platforms have enabled business users to engage with data in natural language and consume insights in the form of data tables and charts. There are still limitations as the experience is still suboptimal.

The development of natural language interfaces to databases has long been a challenge as previous approaches rely on a considerable amount of human-labeled data for domain adaptation. Moreover, most interfaces provide a constrained environment and do not allow the users to freely and naturally interact with data.

In this work, we propose our framework for building domain-agnostic natural language (NL) interfaces to relational databases (NLIDBs) in few-shot and zero-shot scenarios. Also, recent advancements in the area of Transfer learning allowed us to leverage Transformer-based pre-trained language models (PLMs), resulting in various real-world applications in functional areas like CRM, Supply Chain, Ecommerce, Health Care, etc. for getting real-time insights. More specifically, our framework works in the following ways: First, it provides graph focused auto-suggestions to complete the natural language queries based on the graph representation of database schema, and Second, it uses an adaptive sequence-to-sequence translator model that translates natural language queries to corresponding SQL queries. Furthermore, a feedback loop is used to improve the system based on active learning.

Experiment results on benchmark datasets show that our approach achieves a state-of-the-art performance and can be effective in the few-shot and zero-shot scenarios for domain-agnostic applications.

Sroch, R. & Patnaik, D. & Ramachandran, J. (2022). NatSight: A framework for building domain agnostic Natural Language Interface to Databases for next-gen Augmented Analytics. 9th International Conference on Business Analytics and Intelligence, IIM Banglore (BAI Conf’22).

NatSight-t5-small-wikisql

For weights initialization, we used t5-small and fine-tune as sequence-to-sequence task.

Using Transformers🤗

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("course5i/NatSight-t5-small-wikisql")
model = AutoModelForSeq2SeqLM.from_pretrained("course5i/NatSight-t5-small-wikisql")

# define input
prefix = "translate English to Sql: "
raw_nat_query = "What Branding does WRKO calls use?"
query_mention_schema = "c2 | Branding <eom> c0 | calls <eom> v0 | WRKO"
table_header_schema = "c0 | Calls <eom> c1 | Frequency <eom> c2 | Branding <eom> c3 | Format <eom> c4 | Market/Rank <eom> c5 | Timeslot <eom> c6 | Group_owner"

encoder_input = prefix + raw_nat_query + " </s> " + query_mention_schema + " </s> " + table_header_schema
input_ids = tokenizer.encode(encoder_input, return_tensors="pt", add_special_tokens=True)

generated_ids = model.generate(input_ids=input_ids, num_beams=5, max_length=128)
preds = [tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=True) for g in generated_ids]
output = preds[0]

print("Output generic SQL query: {}".format(output))

# output
"SELECT c2 FROM TABLE WHERE c0 = v0"

Intended uses & limitations

More information needed

Training hyperparameters

Please take a look at the training_args.bin file

$ import torch
$ hyperparameters = torch.load(os.path.join('training_args.bin'))

Evaluation results

	NatSight-AdpSeq2Seq (T5-small)
	Dev		Test
Acc-(Logical form) %	Acc-(Execution) %	Acc-(Logical form) %	Acc-(Execution) %
82.31	85.7	83.5	85.38

Framework versions

• Transformers >=4.8.0
• Pytorch >=1.6.0
• TensorFlow >=2.5.0
• Datasets >=1.10.2
• Tokenizers >=0.11.6

If you use these models, please cite the following paper:

    ```
    @article{article, 
        author={Sroch, R. & Patnaik, D. & Ramachandran, J}, 
        title={NatSight: A framework for building domain agnostic Natural Language Interface to Databases for next-gen Augmented Analytics}, 
        journal={9th International Conference on Business Analytics and Intelligence, IIM Banglore (BAI Conf’22)},
        day={17},
        year={2022}, 
        month={Dec},
        url = {https://dcal.iimb.ac.in/baiconf2022/full_papers/2346.pdf}                                                  
    } 
    ```