Democratizing data insights on BigQuery with Data QnA
By: Anirudh Murali
Tags: NLP, Chatbots, Google Cloud Platform, Dialogflow, Bigquery

Democratizing data insights on BigQuery with Data QnA

How can we make useful insights accessible across an organization, even for those with no technical backgrounds? Let’s find out Anirudh Murali Follow Aug 27 · 4 min read

Data-driven decision making is all the rage today. Using insights on usage, performance and various other attributes, decisions can be made based on real-world statistics rather than intuition or guesswork.

Traditionally, data insights used for decision making are extracted from large volumes of data, stored in a data warehouse, by a dedicated data analysis team. These teams use software tools and scripting to extract the relevant data and present it suitably in order to understand the data.

While this is an important part of a data-driven organization, sometimes it may be necessary to get a quick overview of your data. Data QnA provides a way to do this, with no technological barrier.

Data QnA on BigQuery

Data QnA is a relatively new feature of BigQuery, which allows you to ‘ask questions’ about your data, in plain English. Under the hood, Data QnA converts your natural language queries into suitable BigQuery SQL statements, which can be executed to retrieve data from tables.

Data QnA allows users with little to no experience in scripting, analytics tools and SQL to quickly extract useful data from BigQuery. There is no technical barrier — if you know what you’re looking for, you can find it using Data QnA.

Setting up Data QnA

Data QnA needs to be enabled and set up on each BigQuery table that you intend to use with it. It is a simple, 3-step process -

In the Google Cloud console, go to the BigQuery page. Under the pull-out menu on the left, select Data QnA In the menu on the left, select Manage Click on Enable new table. Select the table you want to use

Data QnA will now index the table and prepare it for natural language queries.

Each column is referred to by one or more aliases, which are identified in your natural language queries. By default, Data QnA creates aliases for each column based on the table schema. To customize these aliases,

Under the Data QnA page, select Manage Click on the three dots to the right of your table, and select Edit setup Modify the aliases for each column, and click on Save

Data QnA will now recognize columns with any new aliases that you added.

Asking questions

You can ask questions of your data right from the BigQuery console. To get started,

Go to the BigQuery page and select a table with Data QnA enabled Click on Ask question Type out your query in plain English, and click on Generate Equivalent SQL

You can now see the SQL generated by Data QnA. To run the query, click on Open in Query Editor and select Run to execute the query and retrieve results.

While this is one way to query your data, this is mostly used to test your setup and ensure that Data QnA is generating queries as expected. The real strength of Data QnA lies in its API.

Integrating with Dialogflow

Virtual assistants have become extremely powerful in recent years. In addition to having superior natural language understanding capabilities, these assistants also allow us to perform a wide variety of actions, right from getting an update on the weather, to booking flight tickets and hotel rooms for you. Imagine a virtual assistant that could pull insights for you from your data warehouse as well!

Dialogflow is Google’s platform for building conversational agents. Using Dialogflow, we can build a natural extension to Data QnA, to create the basis for a chatbot that can be used to get data insights from BigQuery.

Supercharge your virtual assistant! [Photo by author]

Once a Dialogflow agent has been built, it can be integrated into a variety of applications to provide a natural conversation-like way to retrieve data insights, such as Google Assistant or Slack. Decision makers can get a quick pulse of their business, just by asking a question on their phones. Need to take a quick decision and looking for some data to back it up? It’s just one question away!

In order to demonstrate this, a sample application has been developed to integrate the Data QnA API with Dialogflow, to provide a chat interface for getting data insights. The project source code and setup instructions can be found here.

Don’t query your database, just ask for your data [Photo by author]

What’s Next?

Check out the Data QnA product launch blog for more information on how Data QnA is revolutionizing data insights for organizations. For more information on the technology behind it, you can read the research paper.

Since the Data QnA API is still in Alpha, it is not publicly accessible yet. If you really want to try it out right away, you can apply for access here. Otherwise, keep an eye on the BigQuery release notes to find out when the API goes public. Stay tuned!