BigQuery: 3 simple ways to Audit and Monitor your Database. | by David Martins | Towards Data Science

Google BigQuery is a managed, highly scalable, serverless data warehouse, capable of querying terabytes of data in seconds. It‘s a great tool, easily accessible for Data Analysts & Scientists. Because GCP made it fully managed, you don’t have to overthink about database administration.
For years, I’ve been working with old-fashioned enterprise-grade databases (Oracle, SAP / Sybase,..). We needed Database administrators in the team, to manage the database “health”.
But when it comes to a fully managed BigQuery database, used primarily by Data Analysts and Data Scientists, it’s a classic pitfall to have nobody taking over that role.
So, here are 5 simple ways to start monitoring your BigQuery DB right away!
Metadata are data that provides information about other data.  — Wikipedia
When it comes to relational databases, metadata provides information about the database themselves: schema, tables, columns, or any other information about database elements and usage.
GCP and BigQuery provide a few convenient ways to monitor your database through metadata.
I won't go much into detail regarding BigQuery’s Metadata, since you can refer to this great article by Skye Tran. But let’s have an overview.
INFORMATION_SCHEMA is a series of views that provide information about tables, views, datasets, and more. You can accessINFORMATION_SCHEMA.TABLES, INFORMATION_SCHEMA.VIEWS, INFORMATION_SCHEMA.COLUMNS , and a few others.
Example: Get insights about the tables in a schema
-- Returns metadata for tables in a single dataset.SELECT * FROM `bigquery-public-data.census_bureau_international`.INFORMATION_SCHEMA.TABLES;
Example: Show your datasets sorted by creation date
SELECT * FROM myProject.INFORMATION_SCHEMA.SCHEMATAORDER BY creation_time;
B. __TABLES__
Each dataset has its own hidden metadata table, named __TABLES__.That one is very useful! That’s a convenient way to get info like “When were your tables last updated?” or “What’s the size of your tables?”.
Example: When were your dataset’s tables last updated?
SELECT  table_id,  # creation_time and last_modified_time to a timestamp   TIMESTAMP_MILLIS(creation_time) AS creation_time,  TIMESTAMP_MILLIS(last_modified_time) AS last_modified_time,  row_count,# convert size in MB  ROUND(size_bytes/(1024*1024),2) as size_mb,# Convert type from numerical to description   CASE    WHEN type = 1 THEN 'table'    WHEN type = 2 THEN 'view'   ELSE NULL   END AS type   FROM `bigquery-public-data.census_bureau_international`.__TABLES__ ORDER BY last_modified_time ASC
Now, you want to have more control over what happens in your database: When were your tables last accessed? How much does a user “cost”? What tables are the most heavily used?
Even if the information is not available through BigQuery’s Metadata, you can answer those questions by exporting Cloud Logging to BigQuery.
Step1: Create a new BigQuery dataset to store the logs
Since you want to receive your logs in BigQuery and analyze them with SQL, it’s better to organize your database by creating a new dataset. Choose a convenient name, such as logs or monitoring.
Step 2: Create a Cloud Logging Sink
Sinks allow you to route your logs or filtered subsets of your logs, to a selected destination. You can create it using the gcloud CLI or Google Cloud’s UI. Let’s use the second method.NB: A sink doesn’t get backfilled with events from before it was created.
Go to Google Cloud Logging, and select Logs Router.
Click on Create Sink. Give it a name.
Chose BigQuery dataset as a destination and use your newly created dataset.I recommend checking the “Use partitioned tables” option: I might improve performances and will make your dataset more readable.
Finally, you might want to use a filter to include only relevant logs.To export all records for the core BigQuery operations, use the filter:protoPayload.metadata."@type"="type.googleapis.com/google.cloud.audit.BigQueryAuditMetadata"
Step3: Query your logs from BigQuery
You will soon have your logs flowing to your dataset, into 3 tables:
cloudaudit_googleapis_com_data_access
cloudaudit_googleapis_com_activity
cloudaudit_googleapis_com_system_event
cloudaudit_googleapis_com_data_access is by far the most useful table to monitor your database. Let’s dive into it!
Analyzing access might be of great help. Now that we have the “access history” logs available in cloudaudit_googleapis_com_data_access , let’s run some queries!
Example: When were your tables last accessed?
SELECT# Dataset nameREGEXP_EXTRACT(protopayload_auditlog.resourceName,   '^projects/[^/]+/datasets/([^/]+)/tables') AS dataset,# Table nameSPLIT(REGEXP_EXTRACT(protopayload_auditlog.resourceName, '^projects/[^/]+/datasets/[^/]+/tables/(.*)$'), '$')[OFFSET(0)] AS table,# Last access dateMAX(timestamp) as last_accessed_at,# Was it a Read or Write operation ?CASE WHEN JSON_EXTRACT(protopayload_auditlog.metadataJson, "$.tableDataRead") IS NOT NULL THEN 'Read'     WHEN JSON_EXTRACT(protopayload_auditlog.metadataJson, "$.tableDataChange") IS NOT NULL THEN 'Change'END method,# Optionally: display the user who executed the operation#protopayload_auditlog.authenticationInfo.principalEmailFROM logs.cloudaudit_googleapis_com_data_accessWHERE  JSON_EXTRACT(protopayload_auditlog.metadataJson, "$.tableDataRead") IS NOT NULL  OR JSON_EXTRACT(protopayload_auditlog.metadataJson, "$.tableDataChange") IS NOT NULLGROUP BY dataset, table, method# Optionally: Select a specific dataset#HAVING dataset = 'public'ORDER BY last_accessed_at desc
Example: How much does your database cost per day/hour/month?This convenient request comes straight from the documentation.
SELECT# Adapt the period of time according to your needsTIMESTAMP_TRUNC(TIMESTAMP(JSON_EXTRACT_SCALAR(protopayload_auditlog.metadataJson, "$.jobChange.job.jobStats.endTime")), DAY) AS period,FORMAT('%9.2f',5.0 * (SUM(CAST(JSON_EXTRACT_SCALAR(protopayload_auditlog.metadataJson, "$.jobChange.job.jobStats.queryStats.totalBilledBytes") AS INT64))/POWER(2, 40))) AS Estimated_USD_CostFROM logs.cloudaudit_googleapis_com_data_accessWHEREJSON_EXTRACT_SCALAR(protopayload_auditlog.metadataJson, "$.jobChange.job.jobConfig.type") = "QUERY"GROUP BY periodORDER BY period DESC
Example: How much does your database cost per user per month?Similar to the previous request, let’s break down the cost per user per month!
SELECT# Adapt the period of time according to your needsTIMESTAMP_TRUNC(TIMESTAMP(JSON_EXTRACT_SCALAR(protopayload_auditlog.metadataJson, "$.jobChange.job.jobStats.endTime")), MONTH) AS period,# Userprotopayload_auditlog.authenticationInfo.principalEmail as principal,FORMAT('%9.2f',5.0 * (SUM(CAST(JSON_EXTRACT_SCALAR(protopayload_auditlog.metadataJson, "$.jobChange.job.jobStats.queryStats.totalBilledBytes") AS INT64))/POWER(2, 40))) AS Estimated_USD_CostFROM logs.cloudaudit_googleapis_com_data_accessWHEREJSON_EXTRACT_SCALAR(protopayload_auditlog.metadataJson, "$.jobChange.job.jobConfig.type") = "QUERY"GROUP BY period, principalORDER BY period, principal DESC
You can find more examples of requests in the documentation. Feel free to create your own!
Since cloudaudit_googleapis_com_data_access can get confusing, BigQuery provides a helper view over this table through the BigQuery utils project.
If you followed along with this article, you won’t need to recreate a log sink. Simply jump to steps 3 and 4 of the Prerequisites, execute the .sql script, and you’re all set!
Through INFORMATION_SCHEMA, __TABLES__, and Cloud Logging in BigQuery, you now have 3 easy ways to audit and monitor your database.
Because you don’t want to rewrite all those queries every time you want to check on your database health, it’s a good practice to create views based on your most-used analysis. You can also create alerts based on certain thresholds, but that’s a story for another day!