Querying PostgreSQL data with BigQuery | by Antonio Cachuan | Towards Data Science

Query data in your Cloud SQL instance in real-time, reducing ELT development time, avoid copying and moving data thanks to BigQuery Cloud SQL federation.
As you know Cloud SQL is a fully-managed relational database service for MySQL, PostgreSQL, and SQL Server. These kinds of databases are oriented to host applications or transactional data. BigQuery on the other hand enables analytics workloads.
In a traditional world, you have to develop a data pipeline to extract the data from your application’s database. Let’s discover how to take advantage of table federation in this case from a PostgreSQL database.
In this scenario, we are responsible as Data Engineers to enable the data from a transactional database (Cloud SQL) in the analytical zone (BigQuery). This data could be used by other processes or for other roles like Data Scientists or Machine Learning Engineers.
As usual, the company wants that the latest transactional data generated could be consumed by our team immediately.
Thanks to BigQuery Cloud SQL Federation we are capable to query in real-time the data in Cloud SQL. This feature empowers new alternatives to the traditional batch ingestion. Now we can consume data from the same day or hour it was created. For example, a report on Data Studio could show the information of sales for the last hour and the best part is that if you have experience with BigQuery will be not difficult to start using it.
Before you continue
In order to follow the next steps, you a Google Cloud account. To open an account just sign up on the GCP site and automatically get $300 in credits.
In order to reproduce this case first, we create the transactional DB. In this DB, we will generate a table to host the sales from a fictional store.
Let's create the Cloud SQL instance in this case PostgreSQL
If It is the first time creating a Cloud SQL you need to enable the Compute Engine API this allows GCP to create and manage Compute Engines on your behalf. Just look in the window and click the blue button.
For this case, set the smallest values for the new instance. With these values, we are assuring the cheapest price. If you want to understand in detail the price breakdown check here.
Zone availability: Single zone
Machine type: Shared core
Storage: HDD
Creating the instance took around 10 minutes.
Then let’s connect to the instance using Cloud Shell. This is the easiest form to connect. Just remember the password you set up before. Exist other alternatives like a connection using a database client from a local machine.
After connect, create the transaction table. A simple table with a primary key and other regular fields.
Also, we insert some data into the table.
So we have our PostgreSQL database and our transactional table, keep the insert script and feel free to insert data after the next step simulating a new transaction.
We are prepared to move to our analytical zone.
BigQuery is the place where our data engineering and data science team spent a bunch of their time in order to transform and get insights from the company data.
To continue with the case, BigQuery needs to establish a connection with our Cloud SQL instance. This connection will be permitted us to query the data we've just inserted.
In your browser, open another tab and enter the BigQuery interface.
There we go to ‘ADD DATA’ and then click on ‘External data source’
If it's the first time enable the Big Query connection API.
In the right part of the window, a form asking for the external data source shows up. Add your connection id (a string you need to copy from the Cloud SQL page), database name, and user password.
Important- The federated table only works in the same project. This means your Cloud SQL instance and the Big Query Federated table need to be in the same project.- Only works with Cloud SQL instances with Public IP connectivity.
That’s all as simple as that. The best is that each change in your CloudSQL tabla will be reflected on the federated table so there is no need to run an extraction pipeline.
Finally, just run a simple query! Like you observe to query the table you can use the same BigQuery UI except for a special syntax so all the features like schedule query or save are allowed.
Another feature is that you are not limited to use only the data from Cloud SQL you could join a Federated query result with a BigQuery table. This means you could build an enrichment process that takes the transaction data from PostgreSQL and add other variables like customer info to send special promotions based on their purchase.
SELECT a.customer_id, a.name, b.first_order_dateFROM bqdataset.customers AS aLEFT OUTER JOIN EXTERNAL_QUERY(  'projects/datapth/databaseconnection',  '''SELECT * FROM public.transactions''') AS bON b.customer_id = a.customer_idGROUP BY c.customer_id, c.name, rq.first_order_date;
Pricing: You are charged for the number of bytes returned from the external query (5$ per TB).
Performance: BigQuery needs to wait for the source database to execute the external query and temporarily move data from the external data source to BigQuery.
Limitation: PostgreSQL supports many non-standard data types that are not supported in BigQuery likemoney, path, uuid or boxer.
BigQuery Cloud SQL federation is a fantastic feature to accelerate your ELT development. If you want to go further I recommend checking this article on Incremental data ingestion pipelines and then come back and rethink this small pipeline for feeding a daily reporting dashboard.
PS if you have any questions, or would like something clarified, ping me on Telegram, Facebook, or LinkedIn or I like having a data discussion 😊