fabiochiu/medium-articles huggingface dataset:
A Simple Way to Query Table Metadata in Google BigQuery
By: Skye Tran
Tags: Big Data, Metadata, Sql, Bigquery, Tutorial

Why query table metadata in Google BigQuery?

Imagine we were given a huge dataset containing many tables in BigQuery, which one should we query? How do we identify which tables are the most updated? When was each table created? What columns are present in each table? Believe it or not, all of these questions can be answered with metadata.

And when it comes to query table metadata, there are many potential solutions, ranging from simple view in Google Cloud Console to more complex client libraries. But here are the 3 simplest and least technical solutions to begin with.

The first solution is looking at a Data Catalog, which is a collection of metadata designed to help us search through all available tables, evaluate their quality and usefulness, then access whatever tables we deem suitable for our analysis. This is probably the easiest way to figure out which specific tables we should query. But what if we don’t have a Data Catalog readily available at our fingertips. How now, brown cow?

Photo by author

The second option is readily available in Google Cloud Console as shown above. Yes, I am talking about the “Details” and “Schema” tabs related to each table under BigQuery. Nevertheless, this solution will seriously trigger headaches if we have to click through dozens of tables (or even hundreds) one by one. There must be a better way, right?

The third solution is here to save the day. Guess what? We can easily obtain all table metadata across multiple datasets using either INFORMATION_SCHEMA or TABLES meta table, all with easy SQL queries right from the familiar BigQuery interface. And that’s what will be covered in the next section. Drum Roll, everybody! Let’s dive right in.
BigQuery Authorized Views with Terraform
By: Alessandro Puccetti
Tags: Infrastructure As Code, Big Data, Gcp, Terraform, Bigquery

In one of my previous blog posts, we have seen how, at Huq Industries, we used authorised views to reduce costs, complexity, and delivery time. You can read more about it here. In this post, we will see how to implement authorised views in production and managing them as code. This solution enables us to easily manage 100s of clients, each one with unique data requirements. To better manage and automate our workflow, we source control and review all our Google Cloud Platform data infrastructure using GitHub and Terraform Cloud. This enabled us to easily communicate, track, and maintain infrastructure changes. Let’s see how to define views and authorize them in the needed datasets. First, we would need a dataset in the client project to contain a view, then we would need the view itself, and finally the authorization.

Define the dataset:

Define the view:

Lastly, we have to authorise the view to access any needed dataset (in this example just one)

After having generated and checked the plan you can apply it. The view will then be created so that each user who has access to it can run queries without having direct access to the underlying tables.

To easily scale this to 100s of clients, we can write a terraform module that implements the last 2 steps and invoke it for each client we need to provision with just different variables such as: client_project_id , client_dataset_name , view_name , view_query , and the list of project_id , datase_id , to autorise.

One of the nicest things of authorised views is that the authorisation works as a chain. At Huq, we leverage this behaviour so that we just have to authorise the clients views to the dataset that directly accesses them. We use authorised views almost everywhere so that if one of these views are used by some other views, the end user does not need extra authorisation.

WARNING

If you also manage datasets access via other terraform resources such as: bigquery_dataset_iam_policy , bigquery_dataset_iam_binding , or bigquery_dataset_iam_member . They will fight over what the policy should be with google_bigquery_dataset_access . In fact, using any of the google_bigquery_dataset_* resources will result in removing any authorised views from that dataset previously configured via google_bigquery_dataset_access .

You can read more about this in the terraform docs.

Follow or contact me on Twitter for more stories about data, infrastructure, and automation.
How to run Python code on your BigQuery table
By: Lak Lakshmanan
Tags: Bigquery, Apache Beam

How to run Python code on your BigQuery table

You can do lots of things in SQL, and SQL is undeniably convenient, but every once in a while, you will find yourself needing to run Python code on your BigQuery tables. If your data is small, you can use Pandas (and the BigQuery client library), but if your data is large, the best approach is to use Apache Beam and execute it in a serverless, autoscaled way with Cloud Dataflow.

Here’s the full code for the example in GitHub. It comes from our forthcoming book on BigQuery.

The code here is from Chapter 5 of our new book on BigQuery. You can read it in early access on Safari.

Python 3 Apache Beam + BigQuery

Here’s the key Beam code to read from BigQuery and write to BigQuery:

with beam.Pipeline(RUNNER, options = opts) as p:

(p

| 'read_bq' >> beam.io.Read(beam.io.BigQuerySource(query=query, use_standard_sql=True))

| 'compute_fit' >> beam.FlatMap(compute_fit)

| 'write_bq' >> beam.io.gcp.bigquery.WriteToBigQuery(

'ch05eu.station_stats', schema='station_id:string,ag:FLOAT64,bg:FLOAT64,cg:FLOAT64')

)

Essentially, we are running a query on a BigQuery table, running the Python method compute_fit, and writing the output to a BigQuery table.

This is my compute_fit method. As you can see, it’s just plain Python code:

def compute_fit(row):

from scipy import stats

import numpy as np

durations = row['duration_array']

ag, bg, cg = stats.gamma.fit(durations)

if np.isfinite(ag) and np.isfinite(bg) and np.isfinite(cg):

result = {}

result['station_id'] = str(row['start_station_id'])

result['ag'] = ag

result['bg'] = bg

result['cg'] = cg

yield result

Make sure to specify the Python packages that you need installed on the Dataflow workers in a requirements.txt:

%%writefile requirements.txt

numpy

scipy

Enjoy!
Using Apache Beam’s SparkRunner to load data from BigQuery to HDFS
By: Abhishek Jha
Tags: Apache Beam, Big Data, Java, Hadoop, Bigquery

Greetings visitor! Let’s start this with a bit of context.

I work in a hybrid cloud environment where we use some services from AWS and GCP alongside our own datacenter with over 200 nodes.

When this task came, I was looking for reliable and scalable ways to import data with minimal increment in infra-cost. We also had Cloudera’s distribution of Hadoop installed and running on our datacenter. This triggered a thought of using Spark to run my data import jobs.

My framework of choice was Apache Beam as I have been wrangling data with it on google cloud’s Dataflow service for a while now and yet when this requirement came, it took a while to figure things out. The good thing about Apache Beam is that when you start writing the pipeline, you don’t have to worry about where it’s going to be executed. It’s so portable end extensible that just by changing a few configurations, the pipeline can be made to run on a variety of cluster computing frameworks like Spark, Flink, Google Cloud’s Dataflow and many more.
Machine Learning with only SQL — Using BigQuery to Identify the Target Audience for Shared Bike
By: Wan Chung Huang
Tags: Towards Data Science, Data, Bigquery, Data Analysis, Data Science

Step4: Predict the result

Finally, using ML.PREDICT to come up with the result that the average duration in the 2019 dataset will be 51 minutes.

Step5: How to improve the accuracy of your model?

How to improve accuracy is one of the most interesting for data scientists.

I used to decompose the steps in a process and tackle each.

Here is my thinking process:

Data

Feature

Algorithm

Parameter

Let’s focus on fundamentals, such as data and features. We’ve done a small experiment on adjusting features. If taking a deeper look into data, there are some basic observational statistics to work with, such as observing average, maximum, and minimum to see if they make sense. For example, some data in the table don’t make sense, duration more than 30,000 hours, or a length < 0.

If we could trim out these data, the credibility of the prediction result will be more desirable.

Thoughts

BigQuery makes ML with SQL possible

Not everyone has access to Python, but what if we need to deliver anyway? For some Database Administrator or Data Analysts, the familiarity of Python may not be a “must-have” skill. The built-in ML for BigQuery let us deliver business result without mastering a new language.

Streamlining the process so you can focus on what matters the most

It’s common to see my enterprise customers running between all kinds of prerequisites before jumping into a machine learning project. As a result, it’s more time-consuming to deliver results due to data movement, security checks, access control, and networking settings. Now many solution providers integrated tools so that the process is smoother. For example, when doing data analysis, we used to download data in the database, send the data, and import it to IDE. Now, only a Mount function will do the work, just like using a USB stack to load data into a computer.

Data Warehouse is evolving to combine BI and ML as a data analytics platform

In the past, a data warehouse was simply a tool for parallel data processing and storage. As time goes by, it also combined new elements such as business intelligence and machine learning for more complex analytics jobs. Besides, many data warehouses offer real-time analysis to fulfill streaming demand in the big data scenario. The architecture below shows how to use Synapse (the Data Warehouse solution from Microsoft) for IoT devices and web click data analysis.
Copy data from Pub/Sub to BigQuery
Tags: Cloud Functions, Google Cloud Platform, Pub Sub, Dataflow, Bigquery

You want to move data from Pub/Sub topic to BigQuery table? I have noticed that most examples show how to do it using Dataflow, which is a bit of overkill if there is no transformation/enrichment going on. For just moving data around google cloud functions are great solution.

First step will be to create a BigQuery table where your data will land. You need to first figure out schema and then follow steps from here to create BQ table.

If schema is relatively simple and you already have rows of sample data, you can let BigQuery auto create schema. Save your sample data to file and use it to initialize BiqQuery table. Both JSON and CSV works fine.

Next thing is to write a function. Both Node.js and python are supported, but we notice that python functions execute much faster.

Here is an example of function that decodes message body from input event, writes it to log and saves in BigQuery table. Dataset and table names are passed in as environment variables so you can reuse same function for different topics.

Save function to a file and deploy it using gcloud cli.

You can also use GCP web console to create function. Here is how form looks like:

Pick correct topic and make sure to set env variables too:

To view function execution logs go to GCP Stackdriver logs and set this as filter

Good luck! Here are few more reference that might be useful:
Slow BigQuery results no more
By: Davide Sarra
Tags: Data Science, Python, Bigquery, Software Development, Analytics

BigQuery is generally praised for its throughput in executing queries, however most users have come to hate how long it takes to retrieve large query results. Several years, StackOverflow questions, and GitHub issues later, Google is tackling this issue with the BigQuery Storage API.

The new API doesn’t fully replace the BigQuery API, but rather it augments the classic API: instead of returning results as paginated text, the new API retrieves the results in a continuous stream of serialized binary data.

A quick benchmark shows that the BigQuery Storage API can be tens of times faster than the BigQuery API. The benefits materialize only above some threshold. In the case of this rough benchmark the new API started to outperform the classic API somewhere between 100,000 and 1 million returned rows (~3.8MB to ~38MB). For example, aside from the actual query execution time, retrieving 10 million rows was almost 20 times faster.

Pandas developers have made using the new API dead simple. Even those familiar with pandas.read_gbq might not spot the difference at first. It’s as small as passing the argument use_bqstorage_api=True .

Behind the Pandas abstraction the query continues to be executed through the classic API Python client. However, the results are fetched using a client for the new API.

The Python BigQuery Storage API library can also be used to load table contents directly into memory. Below you can find a basic example where only a subset of columns and rows are retrieved. In practice, it could be enhanced by retrieving data across multiple streams, modifiers, and more.

Perhaps the only catch with the new API is that it adds new costs. While the first 300TB are currently offered for free, subsequent operations are billed at $1.10 per TB read.

The BigQuery Storage API is still in beta, but it’s well worth a try!
How to Create an Entity Relationship Diagram (ERD) for BigQuery?
By: Donny Chen
Tags: Entity Relationship, Bigquery

How to Create an Entity Relationship Diagram (ERD) for BigQuery?

From work and personal projects, I’ve found it difficult but important for a team to document the relationships of tables (i.e., what are the foreign keys to join tables?). In this post, I’m going to quickly demonstrate how to create an Entity Relationship Diagram (ERD) for BigQuery on Google Cloud Platform (GCP) using QuickDatabaseDiagrams (QDB).

Step 1. Visit QDB at https://www.quickdatabasediagrams.com/

Try out for free without signing up!

Step 2. Retrieve the schema of BigQuery tables from BigQuery UI.

Replace <your-gcp-project-id>, <your-bigquery-dataset-name>, and <your-bigquery-table-name> for your own use case.

SELECT column_name, data_type FROM `<your-gcp-project-id>`.<your-bigquery-dataset-name>.INFORMATION_SCHEMA.COLUMNS WHERE table_name="<your-bigquery-table-name>"

2. Run the query above

3. Click “SAVE RESULTS” and then click “Copy to Clipboard”

4. Paste back to QDB, click “EDIT” and then “Untabify”

Step 3. Repeat Step 2 for the rest of your tables to be included in ERD
Partition on any field with BigQuery
By: Guillaume Blaquiere
Tags: Cloud Computing, Big Data, Data, Google Cloud Platform, Bigquery

Data are critical for companies. Many of them have more data than the capability to process them. And, sadly, a lot of values sleep in databases.

BigQuery is a Google Cloud Platform product which address this topic. BigQuery is a petabyte scale data warehouse and you can query terabytes of data in seconds and you can unlock the power of your sleeping data.

Pricing model and partitioning concern

BigQuery comes with 2 pricing model: on-demand and flat-rate

With the flat-rate model, you commit a number of “slots” (unit of computational capacity) and you know in advance the bill that you will pay. The first flat-rate commitment is quite high ($10k per month), it’s recommended for big data companies.

With the on-demand model, you are charged on the volume of data that you scan. This model is well adapted for startup, medium and small companies. Because of “more data you scan, more you pay”, for limiting the costs, the data have to be optimized for reducing the read data. For this, BigQuery allows to partition the data for narrowing the volume of data scanned.

Note: it’s important to differentiate the volume of data scanned and the volume of data returned. A limit at the end of a query limit the result not the volume scanned.

Partition and clustering

The partition and clustering are 2 features that allow you to narrow the volume of data that you scan in your database.

Until now, partitioning was only possible of date: either on a timestamp field or by ingestion time; both with a day granularity. Like this, a partition like a “sub-table”, was created for each day. When you looked for a data, simply specify the date (or the date range) for query only the interesting partition and scan only the relevant data.

Clustering is a finer grain optimization inside the partition, like composite indexes in relational database. Felipe Hoffa (Google Cloud Developer Advocate) has released a great articles on this

Integer Range Partitioning

In December 2019, Google has released a new partition capability: Integer range partitioning. This feature allow you to store all the values of a same range in the same partition

You have to define the min and max values, and the range size. That’s all, the sharding is made for you! You have user ID, zip code, geo coordinates, (…) the partitioning works for you!
New Updates on Pub/Sub to BigQuery Dataflow Templates from GCP
By: Theodore Siu
Tags: Google Cloud Platform, Pub Sub, Google Dataflow, Dataflow, Bigquery

Authors: Theodore Siu, Sameer Abhyankar

We are pleased to announce several new features to the Cloud Pub/Sub to BigQuery Template including support for subscriptions(!!!) as well as some error handling improvements. We detail these updates below.

Subscriptions y’all!

In the past, the Pub/Sub to BigQuery Dataflow template only supported reading messages from Pub/Sub topics using the parameter inputTopic . We have created a second Dataflow template that reads messages from Pub/Sub subscriptions using the parameter inputSubscription . We delineate these two templates on the Dataflow console under the CREATE JOB FROM TEMPLATE button as “Cloud Pub/Sub Subscription to BigQuery” and “Cloud Pub/Sub Topic to BigQuery”. The code for generating both of these templates can also be found on Github. Note that a caveat of using subscriptions over topics is that subscriptions are only read once while topics can be read multiple times. Therefore a subscription template cannot support multiple concurrent pipelines reading the same subscription.

Pub/Sub to BigQuery templates are now delineated between subscriptions and topics

The remaining details, once a message is read either from a subscription or a topic, remain mostly the same.

The user specifies an existing BigQuery table for the input messages to land. This is specified using the outputTableSpec parameter. As always with all pipelines, users need to specify a GCS bucket location for writing/staging temp files. (Optional) If the user wants to do further modifications to his or her messages before BigQuery insertion he or she may include a javascript transform file in Google Cloud Storage. This can be specified using the parameter - javascriptTextTransformGCSPath . The actual function name within the JavaScript file is then specified using the property- javascriptTextTransformFunctionName . A dead-letter BigQuery Table is automatically created to catch messages that fail due to various reasons including — message schemas that do not match BigQuery table schema, malformed JSON and messages which throw errors while transforming via the JavaScript function. With our latest update we more robustly catch such errors- in the past, many of these errors would retry endlessly. Users can also specify their own dead-letter table using the parameter outputDeadletterTable (see Github for the dead-letter schema).

Example template setup for a PubSub Subscription to BigQuery Dataflow pipeline with a optional javascript function for pub sub message modifications

Future Dataflow Pipelines + Features
Connecting to Google BigQuery from AWS SageMaker
By: Or Hiltch
Tags: Bigquery, Sagemaker, Google Cloud Platform

Google BigQuery is Google Cloud Platform’s serverless DWH. It is a great solution for handling huge amounts of data easily, almost in a file-system manner (there’s a strong API for manipulating tables), all in an SQL-style environment, boosted with some goodies like geospatial queries.

AWS SageMaker is Amazon Web Services’ solution for using Jupyer Notebooks in a production-grade environment, where you can achieve reproducibility, scale and enjoy the benefits of scheduling notebooks and other ML/AI related jobs.

But can they play nicely together?

Well, yes, with a few configurations and tweaks.

Here’s how you would use BigQuery from Jupyter while working on your local machine with the GCP clients configured, or on a GCP instance configured with a service account:

First, one needs to import the google.cloud.bigquery python library. This only works after you install the external library:

%load_ext google.cloud.bigquery

Then you could go:

%%bigquery total_births

SELECT

source_year AS year,

COUNT(is_male) AS birth_count

FROM `bigquery-public-data.samples.natality`

GROUP BY year

ORDER BY year DESC

LIMIT 15

The magic line is %%bigquery total_births — this trick provides you with a pandas DataFrame under the variable total_births, containing the result set in a way that could be easily manipulated later in the code.

Getting it to work from SageMaker in 2-steps

Step 1 — authenticating the BigQuery client

When running on a machine that isn’t yet authenticated (like an EC2 instance or a SageMaker backend), one has to register the credentials for the Google BigQuery client to work.

This is done by registering the environment variable GOOGLE_APPLICATION_CREDENTIALS like so:

export GOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/[FILE_NAME].json"

This is fairly easy to do on a VM, where you can create the credentials file and register environment variables.

To overcome this on SageMaker, you could apply the following steps:

Store the GOOGLE_APPLICATION_CREDENTIALS JSON file on a private S3 storage bucket Download the file from the bucket on the notebook code Register the environment variable from a cell in the notebook

Using the boto3 library, this looks roughly like so:

Step 2 — installing the external python library for bigquery

OK, so we are authenticated. But obviously, none of the SageMaker Jupyter kernels comes with the Google BigQuery client installed. How do we install it then?

Using Lifecycle configuration. Lifecycle configurations are basically entry points that you could implement in order to execute a custom script, before or after the notebook is created.

If you are using Python3 (that would be the conda_python3 kernel on SageMaker), this is how you would install the BigQuery library just for that kernel:

And… that’s it. You can start acquiring pandas DataFrames from BigQuery results in the SageMaker notebook. All that’s left now is to build something nice with it :-)
Announcing Hasura 2.0.0-beta.2
Tags: Hasura, Graphql Engine, Releases, Bigquery, GraphQL

With beta.1 available for about two weeks, and an immense amount of helpful feedback from our community, we are pleased to continue our march towards 2.0 final with the release of 2.0.0-beta.2.

What is a ‘beta’?

Let us explain what a ‘beta’ is to Hasura.

At its simplest, when a release achieves beta it means:

The code is feature frozen and no more features will be added (unless absolutely necessary)

All breaking changes are incorporated, documented in prior release notes (or addressed), and there will not be more surprises to come

The code is in a state to undergo final performance & security review

This does not, by any means, mean that the value of our community is limited to alpha stages. In fact, now is the perfect time to test the beta releases and give us your feedback as we move towards the 2.0 stable release.

A Summary of Releases

As mentioned in recent release posts, we have historically announced our releases via Discord and updates to the GitHub release page. As adoption continues to increase, we believe this is insufficient for the future.

To that end, we will begin to blog for each release so that you can track releases via blog, twitter (@HasuraHQ), Discord, and the Github Release page. In these posts, we will highlight a few specific elements of each release, but the exhaustive list of features and improvements is always in the changelog.

Overview

While we continue squashing final bugs in datasources (i.e. BigQuery) and the server, much of this release has been focused on console functionality. This includes:

Adding foreign key CRUD functionality to the MS SQL Server tables

Allowing the tracking of custom SQL functions having composite type (rowtype) input arguments

Allowing input object presets in remote schema permissions

These are, of course, just the highlights. All other fixes are included in the changelog.

Conclusion

Please use the docker target `hasura/graphql-engine:v2.0.0-beta.2`, try it out, let us know what works and what needs improvement. You can find us on Twitter (@HasuraHQ) and in Github. For ad-hoc conversation, feel free to join us on the Hasura Community Discord.

And, if you want to get started now with the latest stable release, check out the deployment simplicity of Hasura Cloud.

If you want to begin your upgrade journey with 2.0, and want to learn a bit more about how Hasura is being used in the enterprise, do feel free to contact us to set up a call with our solutions team.

Previous Posts

For previous release announcements in the 2.x series, please review:
Google BigQuery Machine Learning (BQML) on Covid-19 Data set
By: Sivaprasad Mandapati
Tags: Big Data Analytics, Bigquery Ml, Data Science, Bigquery, Data Engineering

Writing ML algorithms is tedious job which requires you to know lot many things including strong programming in Python,R,Ruby etc.

But what if we can implement complex ML algorithms using everyone's favourite, simple and easy SQL statements. Some of the new age cloud data warehousing applications are offering SQL mode ML.

Certainly Bigquery is top on this list.

BigQuery Machine Learning (BQML) enables users to create and execute machine learning models in BigQuery using SQL queries. Benefits of BQML are

Train and deploy ML models with simple SQL statements No need to move data from Bigquery Automate ML tasks

BigQuery offers ML models in Regrgression,Classification,Clustering,Recommendation/Personalization, Time Series and Deep Learning

Linear Regression

Logistic regression

Multiclass logistic regression

K-Means Clustering

Matrix Factorization

Time Series

Boosted Tree (XGBoost)

Deep Neural network

Auto ML

TensorFlow Model

Analysing COVID-19 Dataset

Got Covid-19 Data set from Kaggle (https://www.kaggle.com/arashnic/covid19-case-surveillance-public-use-dataset) which is of 10 GB size and We are trying to find out fatality rate based on age, race and ethnic group, hospital condition .

Lets take a sneak peak of data set

Covid-19 Dataset Sneak peak

From the above dataset, The aim is to predict how sex,race,age,hospital status ,ICU status and medical condition are influencing fatality rate (death_yn)

Features Set : [sex,age,age_group,race and ethinicity,current_status,hosp_yn,icu_yn,medcond_yn]

Label : death_yn

Model : Logistic regression

Steps

Load data into Bigquery Set up Training data Create Model Evaluate Model Predict Model

Load Data : The first step is to ingest the data set into Google Cloud. Since it’s 10 GB file , It’s not possible to upload the file directly into Bigquery Through UI. Alternative approach is to upload the file into Google Cloud Storage and then either create external table in Bigquery on top of the Cloud storage or load it from GCS into Bigquery as native table

Set Up Train/Test data : Once the data landed in Bigquery as Table , We are good to go with BQML. To avoid additional data costs for every SQL scan, Let’s create a View on top of table with limited columns like Features ,Label and also Split the the data set into Training,Evalution and Prediction(test) data. You can see 80% of data is taken for training and 10 each for evaluation and prediction. You can do it little better way of splitting data . But this is just to give you generic idea of how to split it. View creation does not cost any data scan.
3 Useful SQL Features You Should Start Using in 2021
By: Skye Tran
Tags: Database, Sql, Bigquery, Tutorial, Practice

3 Useful SQL Features You Should Start Using in 2021

It’s time to take your queries up a notch with Date Partition, Analytic Window Functions and WITH clause Skye Tran Jan 18·6 min read

Photo by Pietro Jeng on Unsplash

I’m starting 2021 with one of the essential New Year’s resolutions: Practice more complex SQL queries. If you’re on the same boat, join me to explore 3 useful SQL features with Google BigQuery.

You’ll learn how to:

Create tables with date partitions Aggregate over a group of rows with (analytic) window functions Break down complex queries using WITH clause

Depending on your background, these might seem like basic features that appeared in other relational databases or they may appear exotic. Either way, I have included detailed examples and approach on how I tackled each query. So let’s hop into something comfy and dive right in!

Before you begin

In Google BigQuery, you will need to be aware of the following hierarchy: Project — Dataset — Tables/ Views

A project organises all resources (e.g. data, storage, compute engine, etc.) to be used. A project can include one or multiple datasets.

organises all resources (e.g. data, storage, compute engine, etc.) to be used. A project can include one or multiple datasets. A dataset is a container to organise and control access to tables and views. A dataset can include one or many tables and/or views.

Before analysing any data, we have to create a project and a dataset. Here is how to do it in less than 3 minutes.

Create a project

Click on Select a project and select New Project

Photo by author

Upon seeing the below screen, specify the project name of your choice and click Create.

Photo by author

It will take around 10–20 seconds for project creation to complete. Notification regarding the status of completion can be viewed by clicking on the Bell icon located at the top right corner of the screen.

Create a dataset

Next, you will create a dataset to store your new tables.

Check the project name shown at the top menu next to Google Cloud Platform. If that is not newly created project, click Select a project and switch to the correct one.

Photo by author

Select the project name, then click Create Dataset

Photo by author

Upon seeing the new dataset under the new project (as shown below), you are all set and ready to get started.
Please BigQuery, can you accelerate my answers (and be cheaper)?
By: Antoine Castex
Tags: Gcp, Bigquery, Data, Cloud

#1- From the UI Interactive mode & BQ CLI Batch mode (without Cache)

Let’s do a full scan of the table, which currently contains 300 columns and 60 partitioned days.

Without anything, that means “SELECT * from mytable”.

WAIT ! I just said, that thing was clearly forbidden, and for sure the result will not be visible because it is too big, so you would have to send it to a destination table…

Let’s add at least a LIMIT 1000 , which doest not reduce the cost but the execution time.

Here is my first score 108 seconds !

#2- Same platform but last 30 days only query

Because I will never need a full table scan and my business team will most likely have to focus on the last 30 days only.

BigQuery UI

So I’m using the partitioned table feature to filter on that amount of time.

Second score is … 3.5 seconds ! This time i don’t parse 70Gb but only 30Gb, make sense and means less money. I like this =)

#3- BI Engine Enabled + last 30 days only query

BI Engine is now enabled let’s try again with the same query…

The result is 1 second, impressive !

You can see that BI Engine has been used in the log of your job with BQ CLI :

bq show --format=prettyjson -j btdp-sbx-c-antoine:EU.bquxjob_xxxxxx

"query": {

"biEngineStatistics": {

"biEngineMode": "FULL"

},

#4- DataStudio with the last 30days

Let’s try now with my first and loved Data Visualisation tool, because it’s free and because it’s so easy to use.

DataStudio UI

When i add my BigQuery datasource to build a simple chart, queries are automatically routed to BI Engine, and the score is 309ms, Wow ! That is super dope ! But one additional very interesting information is the amount of parsed data under my cache (remember i took 100GB as BI Engine reservation because my table is 70Gb and i was afraid to reach the limit) is finally only 1GB. Why ? because BI Engine also compresses data by a factor 6 or 7 and loads only the minimum amount of data necessary to my most frequent queries, crazy ! (good for my wallet too)

#5- Looker with the last 30 days

Because sometimes you need more than just a simple chart and your team is expert on the data, you should use tools like Looker.

Looker UI

We added the new datasource and after querying it we discovered the score which is this time around 1.9 seconds. Nice but not as crazy as DataStudio. This is however not so worrying since Looker has its own independent cache named “aggregated tables” and that will also reduce your next data usage (i will not test that in this article).

The good news here is the processed data which is smaller than the previous test, 700mb, nice !

#6- Connected Sheet with the last 30 days

Google introduced the Connected Sheet feature which let you access, analyse, visualise, and share billions of rows of BigQuery data from your Sheets spreadsheet.

Connected Sheet UI

Just after adding my data source with my custom query (because i need only the last 30 days), the score appears at 200ms, Wow !!!! And only 500Mb of data processed. That is simply super dope (again).
One day, I faced memory error while reading 2GB CSV file in Jupyter
By: Hubby S Hobby
Tags: Big Data, Bigquery, Shopify

One day, I faced memory error while reading 2GB CSV file in Jupyter

Problem

I have been using Jupyter Notebook to analyze data and host internal tools. Before meeting Stitch, I simply (and violently) downloaded CSV file from Shopify API with cronjob every hours. It worked somehow in two years. One day, I faced memory issue while reading giant 2GB CSV file in Jupyter.

First solution — Upgrade memory of your instance

I upgraded Google Compute Engine’s memory. This should be your first choice for the memory errors! Using this way, I saved my six months. However, memory errors became more frequent and I started looking for solutions to store Shopify data in scalable and easy way (very important)

Stitchdata.com

Solution

I use Google Cloud as much as possible. No big reason but I simply chose Google at the start. There are also many other options in Google Cloud but I decided BigQuery with this decision tree. BigQuery is scalable and supports Jupyter pandas well.

The real annoying work is to transfer Shopify API’s data to BigQuery. Shopify supports webhooks to push changes of their data but implementing all data pipeline sounds really boring (and scary). I randomly found Stitch in Google.

Stitch supports Shopify data import to Bigquery and it took 15 mins to finish setup! Super intuitive and clean user onboarding flow are must-reason all hobbyist should care.

Importing all historical data of 3 years took few hours and I finally could run query with GBQ like this in Jupyter:

sql_orders = """

SELECT * FROM `aaa.shopify.orders_view` ORDER BY created_at DESC limit 10000

"""

pd_orders = pd.read_gbq(sql_orders, project_id="aaa", credentials=credentials, dialect='standard')

Now I have no memory error and better (easy) data infra 👻👻

Tips

BigQuery does not support de-duplicate data based on a primary key meaning there will be many duplications. Use view for your simple life.

Attaching Stitch support team’s guide from the chat.
BigQuery Deduplication
By: Mark Scannell
Tags: Big Data, Bigquery

Do you ever get duplicate rows in BigQuery?

I’m going to explore some techniques for deduplication in BigQuery both for the whole table and by partition. It assumes you have the dataset transactions.

Create Your Test Data

This will create a table containing two columns (date, v). There will be 21 days of data (partitioned by date), and each day will have a million rows. The value v will be mostly unique, but there will be a number of duplicates.

Create the Data

CREATE OR REPLACE TABLE `transactions.testdata`

PARTITION BY date

AS SELECT

date,

CAST((i*2)+(2-FLOOR(RAND()*4.0)) AS INT64) AS v

FROM

UNNEST(GENERATE_DATE_ARRAY(

CURRENT_DATE(),

DATE_ADD(CURRENT_DATE(), INTERVAL 20 DAY)

)) AS date

CROSS JOIN UNNEST(GENERATE_ARRAY(1, 1000000, 1)) AS i;

Examine your Data

SELECT

date,

COUNT(DISTINCT v) AS unique_values,

COUNT(*) AS value

FROM

`transactions.testdata`

GROUP BY

1;

These are the first ten rows of output.

You can see that (depending on RAND()) there are quite a few unique_values, but always less than a full million. We want to remove the duplicates.

Deduplicate Everything — Easy Way

The easiest way is to re-create the whole table in place using DISTINCT. You need to use the same parameters (PARTITION BY) for the table.

Recreate the Table

CREATE OR REPLACE TABLE `transactions.testdata`

PARTITION BY date

AS SELECT DISTINCT * FROM `transactions.testdata`;

This processed about 320MB and consumed 2 minutes 15 seconds of slot time.

Examine your Data

Running the query to examine the data shows it is now deduplicated:

However, if you have partitions and you’re appending to your table you don’t want to deduplicate the entire table every time. Rather, you want to de-duplicate just your partition. But how?

Deduplicate Some of the Table — Merge

Rerun the query to create the table again so that it is full of duplicates. This way you can see the partial de-duplication.

Deduplicate using MERGE

MERGE `transactions.testdata` t

USING (

SELECT DISTINCT *

FROM `transactions.testdata`

WHERE date=CURRENT_DATE()

)

ON FALSE

WHEN NOT MATCHED BY SOURCE AND date=CURRENT_DATE() THEN DELETE

WHEN NOT MATCHED BY TARGET THEN INSERT ROW

This processed 15.3MB and consumed 18 seconds of slot time. As there is 21 days of data, this makes sense.

Examine your Data

One of the days — the current date as of writing — is no deduplicated, but none of the other days. It is as expected!

Conclusion

It is very easy to deduplicate rows in BigQuery across the entire table or on a subset of the table, including a partitioned subset.
BigQuery Drop or Change Column
By: Mark Scannell
Tags: Bigquery

UPDATE

BigQuery’s DDL now does support altering tables. Please see here. The rest of this article shows you some techniques you can use by replacing the table. However, replacing the table will destroy any time travel so it is recommended to use the native features rather than CREATE OR REPLACE TABLE.

OLD CONTENT — USE NATIVE FEATURES INSTEAD

BigQuery DDL doesn’t support altering tables, but it does support re-writing tables. You can use this to achieve the same effect.

Create a Table

For example, if you create a table as follows:

CREATE OR REPLACE TABLE `transactions.test_table` AS

SELECT

'' AS a,

1 AS b,

TRUE AS c

LIMIT

0;

You will have three columns (a, b, c) with respective types (STRING, INTEGER, BOOLEAN).

Change column b to an FLOAT

CREATE OR REPLACE TABLE `transactions.test_table` AS

SELECT

* EXCEPT (b),

CAST(b AS FLOAT64) AS b

FROM

`transactions.test_table`;

Drop column c

CREATE OR REPLACE TABLE `transactions.test_table` AS

SELECT

* EXCEPT (c)

FROM

`transactions.test_table`;

Be careful!

As of writing this will remove any table history so be very careful of what you are doing. Please be sure that you backup your critical data and test your SQL before running on production.
BigQuery Table Comparison
By: Mark Scannell
Tags: Bigquery

Introduction

Sometimes you need to compare data across two BigQuery tables. You may want to do this generically (match entire-row-by-row), sometimes comparing by key. This is far easier than you may think!

Sample Data

NOTE: You need the transactions dataset for this SQL.

Create a left_table with a million rows. This will contain the base table for comparison.

CREATE OR REPLACE TABLE `transactions.left_table` AS

SELECT

r AS id,

CAST(r AS STRING) AS col1,

CAST(FLOOR(RAND() * 1000) AS STRING) AS col2,

'x' AS col3,

'y' AS col4

FROM

UNNEST(GENERATE_ARRAY(1, 1000000)) r;

Create a right_table with mostly the same data, but some deletes, updates, and inserts. There is a probability used for deletes and updates, while 100 rows are simply inserted.

CREATE OR REPLACE TABLE `transactions.right_table` AS

WITH

DataChanges AS (

SELECT

* EXCEPT (col3, col4),

IF(RAND() < 0.001, 'new-x', col3) AS col3,

col4

FROM

`transactions.left_table`

WHERE

RAND() > 0.001

),

NewData AS (

SELECT

r AS id,

CAST(r AS STRING) AS col1,

CAST(FLOOR(RAND() * 1000) AS STRING) AS col2,

'x' AS col3,

'y' AS col4

FROM

UNNEST(GENERATE_ARRAY(1000000, 1000100)) r

)

SELECT * FROM DataChanges

UNION ALL

SELECT * FROM NewData;

Whole Table Comparison

A whole table comparison will generate a FARM_FINGERPRINT per row. This will allow us to do a FULL OUTER JOIN on this hash and see what values exist on one side but not the other.

Note that without a key we cannot tell if a row has been updated.

WITH

LeftData AS (

SELECT

a AS data,

FARM_FINGERPRINT(FORMAT("%T", a)) AS h

FROM

`transactions.left_table` AS a

),

RightData AS (

SELECT

b AS data,

FARM_FINGERPRINT(FORMAT("%T", b)) AS h

FROM

`transactions.right_table` AS b

)

SELECT

IF(l.h IS NULL,"New on right","New on left") AS Change,

IF(l.h IS NULL,r.data,l.data).*

FROM

LeftData l

FULL OUTER JOIN RightData r

ON l.h = r.h

WHERE

l.h IS NULL OR

r.h IS NULL

The above could be changed so that — rather than listing the rows — it could count the rows with a GROUP BY.

Key by Key Comparison

Rather than just comparing the whole table, we want to compare row-by-row. So we will use the key to determine if it’s an insert, update, or delete.

SELECT

CASE

WHEN a.id IS NULL AND b.id IS NOT NULL THEN "I"

WHEN a.id IS NOT NULL AND b.id IS NULL THEN "D"

ELSE "U"

END AS op,

IF(b.id IS NULL, a, b).*

FROM

`transactions.left_table` a

FULL OUTER JOIN `transactions.right_table` b

ON a.id = b.id

WHERE

a.id IS NULL OR

b.id IS NULL OR

FARM_FINGERPRINT(FORMAT("%T", a)) <>

FARM_FINGERPRINT(FORMAT("%T", b));

This will generate a Change-Data-Capture like log of changes between left_table and right_table. Below is the sample output from running this.

Conclusion

BigQuery can be used for creating test cases, validating your data, generating CDC logs from snapshots fairly easily. This can be done without an explicit column-by-column comparison.

Thanks to Thinh Ha for the comparison queries.
BigQuery + Cloud Functions: how to run your queries as soon as a new Google Analytics table is available
By: Marieke Pots
Tags: Google Analytics, Cloud Functions, Google Cloud Platform, Data Analysis, Bigquery

If you have your Google Analytics view linked to BigQuery, you are probably enjoying the advantages it brings. There is no sampling in your data, you can bring in third-party data, you have more control over your data and much more.

But, there was one thing bothering me. And maybe it is bothering you too. The exact time of the daily export from Google Analytics to the BigQuery table(ga_sessions_YYYYMMDD) can be somewhat unpredictable. This makes it is impossible to run scheduled queries on this dataset without taking a small risk.

You might set your scheduled queries too early. If, for some reason, the Google Analytics export is delayed, you might miss a day of data. You will have to manually correct this every time this happens. Or you might consider using the intraday table. But this might not be as ideal either, since Google states: “Data for the current day is not final until the daily import is complete. You may notice differences between intraday and daily data” (see the documentation here).

You might set your scheduled queries too late. You might be safe if the exports are delayed in this case. But, your underlying tables for your dashboards and/or possible transfers to other data systems may not be as up to date as they need to be for your decision makers.

Of course, in these uncertain times of Corona, you might call this a first-world-problem. But because it is a problem that I can help fix, I will take you through a step by step guide on how to run your scheduled queries in the exact minute that a new daily export from Google Analytics is ready. The result is a much more fail-proof set-up in which your tables and dashboards are always as up-to-date as they can possibly be.

The steps that we will be taking are:
How to query and calculate GA4 event data in BigQuery
By: Johan Van De Werken
Tags: Google Analytics, Web Analytics, Sql, Bigquery, Digital Marketing

How to query and calculate GA4 event data in BigQuery

The future of app and web analytics is here: Google announced a new way of measurement in Google Analytics. While still in beta, this is a small revolution for web and app analytics.

On top of this: there is a raw data BigQuery export available for all (even non 360) Google Analytics accounts. Let’s dive right in!

Note: do you enjoy this article? Then you’ll also like my new website GA4BigQuery.com: a digital guide with tips, ideas, example queries and tutorials on how to query Google Analytics data in BigQuery & rock your digital marketing analytics 🤘

You also might be interested in a digital course that I designed together with Mari and Simo Ahava for marketing professionals who want to dive into the world of queries, and for data professionals who need to skill up on GA4-specific knowledge:

Image by Berber Bijlsma

In this article I will focus on web data only. I will show you how to calculate new GA4 engagement metrics, but also how to replicate ‘classic’ Google Analytics dimensions and metrics that you won’t find in the GA4 user interface anymore. But beware…

This is not Universal Analytics. Try to look at the new measurement model as an opportunity to go beyond what Universal Analytics ever could, rather than just another way to populate GA’s ancient data model. (source)

App and web unified (source)

If you are new to BigQuery, I recommend reading the first part of my previous article on how to query Google Analytics (Universal Analytics) data in BigQuery.