Monitoring your BigQuery costs and reports usage with Data Studio | by 💡Mike Shakhomirov | Towards Data Science

If you are a BigQuery user and you visualise your data with Data Studio then you might want to answer the following questions:
What was the cost of each Data Studio report for yesterday?
How many times each query/report was run and who ran it?
What was the cost of queries for tables, datasets (e.g. production/staging/ analytics) with label X?
Can I be notified in case of a sudden surge in billed bytes amount?
Standard Google Billing dashboard won’t answer these questions.
According to official Google docs at the moment you can’t use labels for BigQuery jobs.
Read more here: https://cloud.google.com/bigquery/docs/filtering-labels
But what if we want to know who exactly ran the query and HOW MUCH it cost us?
In this article I will talk you through the complete process of setting up BigQuery cost monitoring dashboard.
I used standard Google Ads template from Google Data Studio. I think it looks nice and I slightly changed it for my needs.
If you want you can just copy the template with all relevant settings and widgets included. Just adjust the datsource (create your own) for each widget as shown below in bq_cost_analysis.sql
Prerequisites:
Google Cloud Platform (GCP) account
BigQuery
Data Studio
In this article I will use Cloud Audit Log for BigQuery. We’ll use these events data, export it to BigQuery and analyse it.
Google-recommended best practice is to use custom log based metrics.
Go to Logging and create a sink. Read more about creating sinks here.
This will output all BigQuery `query_job_completed` log events from Cloud Audit Log service into your BigQuery table.
In this way you can export any logs for further analysis which I find very useful.
Note that only new logs will be exported but you can also filter and narrow down the logs you are investigating in Logging console:
So now when I have BigQuery event data flowing into my logs dataset I will create a view based on the data I have:
And that’s the bit we need to calculate BigQuery costs (per query or a job):
We can simply multiply then: cost per TB processed * numbers of TB processed.
It is a custom query bq_cost_analysis.sql:
Now we have processedBytes with associated cost per user and per query nicely grouped:
You probably noticed me adding a query tag in bq_cost_analysis.sql
That’s a little trick that will help with cost groupings and let us use actual labels (which are currently not supported).
Simply adjust that SQL and add a column for your tagged query. It can be anything, a report in Data Studio, table name or a scheduled query:
substr(protopayload_auditlog.servicedata_v1_bigquery.jobCompletedEvent.job.jobConfiguration.query.query,0,30) as report
Or something like this might work even better:
select REGEXP_EXTRACT(‘ --tag: some report — tag ends/’, r’--tag: (.*) - tag ends/’) as report
Of course, you can aggregate costs by query too.
If you set the table at the bottom to display only one record (one query) per page that would make query cost analysis realy simple.
Adding sliders for hour and minute can also be very useful.
Another interesting thing. Cache. If you want to save money you should enable prefetch caching in Data Studio report you must use owner's credentials and enable Cache option (data freshness). Read more about cache here: https://cloud.google.com/bigquery/docs/cached-results
In this case if you share an access to your report with any other users they will be using your credentials to access the data and what you will see in your cost report is only your account name.
If you want to enable other users for your cost reports you should use Viewer's credentials and grant them access to underlying datasets too.
I find it really annoying but this is how it works in Data Studio.
Read more in official google docs here: https://developers.google.com/datastudio/solution/blocks/enforce-viewer-creds.
If you are using Viewer’s credential in Data Studio you will need this.
Let’s go to IAM roles and create the role called Data Studio User. For that we will need permissions to view and list tables in the datasets. You can simply add predefined BigQuery Data Viewer role.
To run reports from Data Studio we will also need BigQuery Job User role which adds bigquery.jobs.create permission. Read more about predefined roles here: https://cloud.google.com/bigquery/docs/access-control
Now you can assign this role to any user to let them use Viewer’s credentials in Data Studio. Just click on the dataset name and click Share dataset. This fits into Google best security practice when only minimum scope of permissions is granted.
I also want to be notified in case of a sudden surge in billed bytes amount.
To create this Alert first go to Dashboards, click Add chart and select Scanned Bytes Billed:
Select Aggregator and Aligner to Sum. An Aggregator describes how to aggregate data points across multiple time series and in case you have labels to use in Group By. Aligner describes how to bring the data points in each individual time series into equal periods of time. Read more about Aligner.
Now let’s go to Google Alerting and create a policy:
Let’s call it ‘Last 15 minutes BilledBytes exceed 1Gb’ so we will be getting alerts every time total amount of billed bytes exceeds 1Gb over any of 15 minute windows:
Now if we run any query/queries (including Data Studio reports) where SUM of Total Billed Bytes exceeds the threshold over the last 15 minutes the system will notify us.
Let’s run a query that exceeds this threashold and see if that worked:
You should receive an email alert. It worked:
You can also set alerts for number of slots in use and Query execution time, for example, in a similar way.
Read more here: https://cloud.google.com/monitoring/alerts/using-alerting-ui#create-policy
Now you have a beautiful cost monitoring dashboard where you can see your BigQuery costs associated to each Data Studio report allowing you to go deeper and beyond basic dataset level reporting in Google Cloud Billing.
Thanks for reading and let me know if you have any questions.