5 Bigquery SQL Commands You [Probably] Didn’t Know About | by Nicholas Leong | Towards Data Science

Good Day Reader.
If you’re reading this you are probably a data practitioner, or aspiring to be one. Welcome to the world of data, get comfortable here, you are going to spend quite some time learning new things.
With the increase in job demand for the data science field, people are eager to pick up the skills needed for the profession. Python, Machine Learning, Data Visualization, the list goes on. However, the one skill that often gets ignored is ...
SQL has been the go-to querying language for all databases, and it is not going anywhere soon. All data scientists have to get their data somewhere. More often than not, it will be from a SQL-Relational Database.
Fast forward to today, we have the luxury of running SQL-Related Data Warehouses in our companies. These are OLAP (online analytical processing) databases that usually allow complex queries and collaboration with data visualization tools while not messing with the production database.
The famous ones are Google Bigquery, Amazon Redshift, Azure SQL Data Warehouse, the list goes on.
In this article, we will talk about some unique SQL commands that you probably didn’t know but will help tremendously when you hit a similar kind of brick wall.
Let’s get into it.
As shown in the illustration, this is a command we use to transform a Bigquery column that is a JSON String type into a whole table. You may encounter this when performing data engineering tasks, often from external data sources.
Here, we can use the JSON_EXTRACT_SCALAR Bigquery command.
JSON_EXTRACT_SCALAR(<JSON COLUMN>, $[<PARENT>])
In this command, JSON Strings are split into PARENT, CHILD, and SUB.The function works as such: JSON_EXTRACT_SCALAR( JSON_STRING, “$[‘PARENT_NAME’]”) will result in the CHILD value. For data engineers out there, think of it as a key-value kind of concept.
In this example,we use JSON_EXTRACT_SCALAR( saved_fields, “$[‘email’]”) which will result in a column of all emails in the JSON string. Pretty cool right.
What if the child is in a list format, like the parent language_skill in this example?
We can use JSON_EXTRACT_SCALAR(saved_fields, “$.language_skill[0]”) which will give you the first element in that list as a column.
That's pretty much it. To extract all information in this column, we use
SELECT JSON_EXTRACT_SCALAR(saved_fields, "$['email']") as email, JSON_EXTRACT_SCALAR(saved_fields, "$['name']") as name, JSON_EXTRACT_SCALAR(saved_fields, "$['phone']")as phone, JSON_EXTRACT_SCALAR(saved_fields, "$['gender']")as gender, JSON_EXTRACT_SCALAR(saved_fields, "$['nationality']")as nationality, JSON_EXTRACT_SCALAR(saved_fields, "$['birthday']")as birthday, JSON_EXTRACT_SCALAR(saved_fields, "$['experience_seeker']")as experience_seeker, JSON_EXTRACT_SCALAR(saved_fields, "$.language_skill[0]") as language_skill_1, JSON_EXTRACT_SCALAR(saved_fields, "$.language_skill[1]") as language_skill_2, JSON_EXTRACT_SCALAR(saved_fields, "$.language_skill[2]") as language_skill_3, JSON_EXTRACT_SCALAR(saved_fields, "$.language_skill[3]") as language_skill_4, JSON_EXTRACT_SCALAR(saved_fields, "$.language_skill[4]") as language_skill_5, JSON_EXTRACT_SCALAR(saved_fields, "$['highest_edu_qualification']")as highest_edu_qualification, JSON_EXTRACT_SCALAR(saved_fields, "$['job_cv']")as job_cv from  dataset.table
For the official documentation, read here.
This command is not only the most often used in this list, but it is also often brought up in data science interviews.
When speaking about data aggregations, many of the seasoned data scientists will automatically resort to Group By, which is the most commonly used command to aggregate data.
However, you may encounter situations where Group By will not suffice.The Group By function requires you to group data by columns, but what if you don’t want to group them. What if you want to keep all the existing rows while still aggregating data.
The best way I can describe it is by presenting you with an interview question:
Given customerID, date and revenue columns. For EACH date, how do I show the total revenue value for EACH customer?
Questions like these can be solved by Group By
Selectdate,customerID,sum(revenue)from datasetgroup by date, customerID
This command groups the data by date and customerID. This essentially means that for each date, we will end up with 1 row for each customerID.
Partition By comes in handy when you want to keep all the rows after performing aggregation. For the same question above :
Given customerID, date and revenue columns. For EACH date, how do I show the total revenue value for EACH customer while keeping all the rows?
You can achieve this by
Selectdate,customerID,revenue,sum(revenue) over (partition by date,customerID) as revenue_per_dayfrom dataset 
Note how all the rows are being kept after the aggregation.Here’s a more practical example that I often get.
Given customerID, date and revenue columns. For each date, how do I show the highest revenue row for each customer?
Using the partition by command, you can easily achieve this by
Selectdate,customerID,revenue,sum(1) over (partition by date order by revenue desc rows unbounded preceding) as row_numfrom dataset having row_num = 1
This command is performing:
For each date and customerID, place a row number +1 each time a row appears order by the revenue value (descending)
Only filter for the rows with the row_num=1 which will return the highest revenue row for each customer, each date
I hope by now you realize how powerful the Partition By command is.
The Having clause.If you are familiar with the Where clause, it is extremely similar.It filters for rows that return a TRUE value when they are evaluated against a boolean expression.
The only difference is that for the Having clause, it :
must have a Group By or any aggregation present in the working query
occurs after any said aggregation, before order by. The Where clause, on the other hand, occurs before any aggregation
You can already achieve this using nested where clauses:
#Looking for revenue > 100Select * from(Selectdate,customerID,sum(revenue) as revenue_sumfrom datasetgroup by 1,2) where revenue_sum > 100
But why do that, when you have the option to write more professional cool-looking code?
Selectdate,customerID,sum(revenue) as revenue_sumfrom datasetgroup by 1,2having revenue_sum > 100
Regex is short for Regular Expression, an extremely popular term in the NLP field. Regex is defined as a sequence of characters that define a search pattern. It is mainly used to process text, numbers, and punctuations.
Here’s a documentation on Regex.
You can do multiple things in Regex like look for specific texts, remove numbers or punctuation, replace certain characters, etc. Here’s how to do it in Bigquery SQL.
There are 3 main Regex commands in Bigquery:
REGEXP_CONTAINS returns TRUE if the string(value) provided is a partial match for the regular expression (regexp).
REGEXP_CONTAINS(value, regexp)
Here’s an example, we want to filter for data where the customerID starts with a ‘P’ character. We can achieve this by
#^P means string starts with the character PSelect*from dataset where REGEXP_CONTAINS(customerID, '^P')
REGEXP_REPLACE searches for a specific Regex pattern from the provided string(value) and replaces it with whatever you specified.
REGEXP_REPLACE(value, regexp, replacement)
Similar to the previous example, let’s replace all customerID that starts with the character ‘P’ with ‘Starts with P’.
select *,REGEXP_REPLACE(customerID,'^P','Starts with P')from dataset
REGEXP_EXTRACT returns the first substring from the string(value) provided that matches the Regex pattern specified .
REGEXP_EXTRACT(value, regexp)
Similar to the previous example, let’s extract the first substring that is an alphabet.
select *,REGEXP_EXTRACT(customerID,'[a-zA-Z]')from dataset
The With Clause contains subqueries that run every time a subsequent SELECT statement runs. It can be referenced anywhere of your query including joins, unions, and wheres.
The With Clause is extremely useful for readability, it also helps the data practitioner to have a high-level understanding of the query written.
WITH q1 AS (my_query)SELECT *FROM  (WITH q2 AS (SELECT * FROM q1) SELECT * FROM q2)WITH q1 AS (my_query)SELECT *FROM q2 aleft join my_query b on a.id = b.id
Note that the aliases in introduced in the same With clause must be unique, but the same alias can be used multiple times. If there are similar aliases, the With clause will automatically take the latest referenced alias.
WITH q1 AS (my_query)SELECT *FROM  (WITH q2 AS (SELECT * FROM q1),  # q1 resolves to my_query        q3 AS (SELECT * FROM q1),  # q1 resolves to my_query        q1 AS (SELECT * FROM q1),  # q1 (in the query) resolves to my_query        q4 AS (SELECT * FROM q1)   # q1 resolves to the WITH subquery                                   # on the previous line.    SELECT * FROM q1)  # q1 resolves to the third inner WITH subquery.
Well done! If you’ve made it this far, I’d imagine you’ve learned a thing or 2 about SQL. In this article, we went through these SQL commands:
JSON Column to Table
Partition By
Having
Regex
With
Keep it scarred in your brain as I swear you will use it often in the future.
We are not done yet with our data journey. Stay tuned, I am writing about more data science projects, as well as more stories, guides, and experiences in the data industry. In the meantime, feel free to check out my other articles to temporarily fill your hunger for data.As always, I end with a quote.
Learning from data is virtually universally useful. Master it and you will be welcomed anywhere. — Todd Papaioannou, CTO at Splunk
You can also support me by signing up for a medium membership through my link. You will be able to read an unlimited amount of stories from me and other incredible writers!
I am working on more stories, writings, and guides in the data industry. You can absolutely expect more posts like this. In the meantime, feel free to check out my other articles to temporarily fill your hunger for data.
Thanks for reading! If you want to get in touch with me, feel free to reach me at nickmydata@gmail.com or my LinkedIn Profile. You can also view the code for previous write-ups in my Github.