How to query and calculate Google Universal Analytics data in BigQuery | by Johan van de Werken | Towards Data Science

When I first started querying Google Analytics data in BigQuery, I had a hard time interpreting the ‘raw’ hit-level data hiding in the ga_sessions_ export tables. Because I could not find a noob-proof guide on how to calculate Google Analytics metrics in BigQuery, I decided to write one myself. I provide lots of example queries so you don’t have to reinvent the wheel and hopefully you can save yourself some valuable time.
Update: do you enjoy this article? Then you’ll also like my new website GA4BigQuery.com: a digital guide with tips, ideas, example queries and tutorials on how to query Google Analytics data in BigQuery & rock your digital marketing analytics.
I’ve migrated most example queries from this article to the new website, as it is easier to maintain all content in one place. This article will stay online to refer you to the right place.
I also published an article about querying Google Analytics 4 (previously App + Web) event data in BigQuery. This content will be migrated to GA4BigQuery.com too.
– Introduction to Google Analytics data in BigQuery– Multiple tables– User– Session– Time– Traffic Sources– Geo Network– Platform or Device– Page Tracking– Event Tracking– Goal Conversions– (Enhanced) Ecommerce (transactions)– Ecommerce (products)– Enhanced Ecommerce (products)– Custom Dimensions & Custom Metrics– Custom Channel Grouping– Intraday table– Realtime tables & view
For those of you wondering why you should use BigQuery to analyze Google Analytics data anyway, read this excellent piece. Some big advantages:
No more sampling. Ever.
Unlimited amount of dimensions
Combining different scopes in one report (not for the faint of heart!)
Calculate goal completions, build your own Channel Grouping and correct data errors, all on past data
Combine Google Analytics data with third party data sources
But let’s not become too excited. Truth is that diving into BigQuery can be quite frustrating, once you figure out a lot of the Google Analytics metrics you are used to are nowhere to be found.
What makes BigQuery interesting for Google Analytics users, specifically Premium customers, is that Google can dump raw Google Analytics data into BigQuery daily. While this enables many types of analysis that can’t be performed within the Google Analytics interface, it also doesn’t provide any basic metrics, e.g. bounce rate, to use. (source)
There are two sides to this: the tough part is that I had to calculate every ‘missing’ Google Analytics metric in my queries. The positive effect: my understanding of the metrics on a conceptual level improved considerably.
The BigQuery cookbook helped me out in some cases, but also seemed incomplete and outdated at times. Since Standard SQL syntax is the preferred BigQuery language nowadays and a lot of old Stackoverflow entries are using the (soon to be deprecated?) Legacy SQL syntax, I spent hours and hours to get my head around the SQL queries I had to write to get the reports I wanted. Apart from the calculated metrics that I needed to take care of, there was another hurdle to cross: nested and repeated fields.
Each row in the Google Analytics BigQuery dump represents a single session and contains many fields, some of which can be repeated and nested, such as the hits, which contains a repeated set of fields within it representing the page views and events during the session, and custom dimensions, which is a single, repeated field . This is one of the main differences between BigQuery and a normal database. (source)
With this article I hope to save you some trouble. I will show you how to create basic reports on session and user level and later on I will show some examples of more advanced queries that involve hit-level data (events, pageviews), combining multiple custom dimensions with different scopes, handling (enhanced) ecommerce data and joining historical data with realtime or intraday data.
No Google Cloud Billing account? Enter the BigQuery Sandbox, which allows you to use the BigQuery web UI without enabling a billing account. To set up a Google Analytics to BigQuery export you need Google Analytics 360 (part of the Google Marketing Platform).
I assume you have a basic understanding of SQL as a querying language and BigQuery as a database tool. If not, I suggest you follow a SQL introduction course first, as I will not go into details about the SQL syntax, but will focus on how to get your (custom) Google Analytics reports out of BigQuery for analysing purposes. All query examples are in Standard SQL.
In this article we will use the Google Analytics Sample dataset for BigQuery, which contains analytics data from the Google Merchandise Store.
However, I recommend you use your own Google Analytics dataset if you want to compare the results of your queries with Google Analytics, because I’ve noticed differences between the Google Merchandise Store data in Google Analytics and the sample BigQuery dataset. I tested the queries on other Google Analytics-accounts and they matched quite well.
To get a good understanding of the ga_sessions_ table in BigQuery, let’s take a look at the BigQuery Export schema, which gives us an idea of the available raw Google Analytics data fields in BigQuery.
Although you probably will recognize a lot of dimensions and metrics from the Google Analytics UI, I know this schema can be a bit overwhelming. To get a better understanding of our data set, we have to know the structure of the (nested) fields. The next picture represents two rows (= 2 sessions) from a ga_sessions_ table.
As you can see our trouble starts if you need custom dimensions, custom metrics or any data on hit-level: i.e. events, pageviews or product data. Let’s query our nested sample set:
SELECT  *FROM  `bigquery-public-data.google_analytics_sample.ga_sessions_20170801`LIMIT  2
This gives us 2 rows, which represented as a flat table would look like this:
Remember, only row 2 and 14 in this example are real rows in our table. The other ‘rows’ are in fact nested fields, in most cases NULL values. Only the hits.product columns are populated with values.
To deal with this fields and to be able to query our tables so they meet our needs, we need the UNNEST function.
The problem here is that is essentially an array (actually in BigQuery parlance it’s a “repeated record”, but you can think of it as an array). (...) This is where the UNNEST function comes in. It basically lets you take elements in an array and expand each one of these individual elements. You can then join your original row against each unnested element to add them to your table. (source)
I highly recommend reading this article which explains the UNNEST concept in detail with the Firebase Analytics sample data set as an example.
You only have to UNNEST records that contain ‘repeated fields’. In case of our Google Analytics data set these could involve:
customDimensions
hits
hits.customDimensions
hits.customMetrics
hits.product
hits.product.customDimensions
hits.product.customMetrics
To make sure you understand the structure of the BigQuery export schema, I encourage you to take look at this interactive visual representation.
OK. Enough theoretical blabber. Ready for some action? Let’s query!
Return to table of contents
Google Analytics data in BigQuery is stored per day in a table. If you only need data from one day the FROM clause in your query will look like this:
SELECT  *FROM  `bigquery-public-data.google_analytics_sample.ga_sessions_20160801`
In most cases you will need to query a larger period of time. Enter _table_suffix. More details here, but to query multiple tables of Google Analytics data you only need these examples. Note that you can combine static and dynamic dates or use only static dates or just dynamic dates for a rolling period, for the last 90 days or so. It is also possible to include the intraday table in your query.
When you perform an analysis on a static data range you should use fixed start and end dates. In this example we select August 1st 2016 to August 1st 2017.
SELECT  *FROM  `bigquery-public-data.google_analytics_sample.ga_sessions_*`WHERE  _table_suffix BETWEEN '20160801'  AND '20170801'
In this example we select period today-30 days to yesterday.
SELECT  *FROM  `bigquery-public-data.google_analytics_sample.ga_sessions_*`WHERE  _table_suffix BETWEEN FORMAT_DATE('%Y%m%d',DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY))  AND FORMAT_DATE('%Y%m%d',DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY))
We know we have 366 day tables in our data set, so we could use a fixed end date here (20170801), but usually I prefer a combination of a fixed start date and a dynamic end date (in this case: today minus one). If new data is added to our data set it is automatically included in our query. In this example we select August 1st 2016 to yesterday.
SELECT  *FROM  `bigquery-public-data.google_analytics_sample.ga_sessions_*`WHERE  _table_suffix BETWEEN '20160801'  AND FORMAT_DATE('%Y%m%d',DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY))
Return to table of contents
This example query contains all following Google Analytics dimensions and metrics. If you only need one dimension or metric, look at the -- comments in the example query and copy the part you need from the SELECT clause. Make sure that you also add any additional conditions (in the FROM, WHERE, GROUP BY and ORDER BY) that are necessary to calculate the results correctly.
User TypeCount of Sessions
UsersNew Users% New SessionsNumber of Sessions per UserHits
www.ga4bigquery.com
Return to table of contents
This example query contains all following Google Analytics dimensions and metrics. If you only need one dimension or metric, look at the # comments in the example query and copy the part you need from the SELECT clause. Make sure that you also add any additional conditions (in the FROM, WHERE, GROUP BY and ORDER BY) that are necessary to calculate the results correctly.
-
SessionsBouncesBounce RateAvg. Session Duration
www.ga4bigquery.com
Return to table of contents
This example query contains all following Google Analytics dimensions and metrics. If you only need one dimension or metric, look at the -- comments in the example query and copy the part you need from the SELECT clause. Make sure that you also add any additional conditions (in the FROM, WHERE, GROUP BY and ORDER BY) that are necessary to calculate the results correctly.
DateYearISO YearMonth of YearMonth of the yearWeek of YearWeek of the YearISO Week of the YearISO Week of ISO YearDay of the monthDay of WeekDay of Week NameHourMinuteHour of DayDate Hour and Minute
-
www.ga4bigquery.com
Return to table of contents
This example query contains all following Google Analytics dimensions and metrics. If you only need one dimension or metric, look at the -- comments in the example query and copy the part you need from the SELECT clause. Make sure that you also add any additional conditions (in the FROM, WHERE, GROUP BY and ORDER BY) that are necessary to calculate the results correctly.
Referral PathFull ReferrerDefault Channel GroupingCampaignSourceMediumSource / MediumKeywordAd ContentSocial NetworkSocial Source ReferralCampaign Code
-
www.ga4bigquery.com
Return to table of contents
This example query contains all following Google Analytics dimensions and metrics. If you only need one dimension or metric, look at the -- comments in the example query and copy the part you need from the SELECT clause. Make sure that you also add any additional conditions (in the FROM, WHERE, GROUP BY and ORDER BY) that are necessary to calculate the results correctly.
ContinentSub ContinentCountryRegionMetroCityLatitudeLongitudeNetwork DomainService ProviderCity ID
-
www.ga4bigquery.com
Return to table of contents
This example query contains all following Google Analytics dimensions and metrics. If you only need one dimension or metric, look at the -- comments in the example query and copy the part you need from the SELECT clause. Make sure that you also add any additional conditions (in the FROM, WHERE, GROUP BY and ORDER BY) that are necessary to calculate the results correctly.
BrowserBrowser VersionOperating SystemOperating System VersionMobile Device BrandingMobile Device ModelMobile Input SelectorMobile Device InfoMobile Device Marketing NameDevice CategoryBrowser SizeData Source
-
www.ga4bigquery.com
Return to table of contents
This example query contains all following Google Analytics dimensions and metrics. If you only need one dimension or metric, look at the -- comments in the example query and copy the part you need from the SELECT clause. Make sure that you also add any additional conditions (in the FROM, WHERE, GROUP BY and ORDER BY) that are necessary to calculate the results correctly.
HostnamePagePrevious Page PathPage path level 1Page path level 2Page path level 3Page path level 4Page TitleLanding PageSecond PageExit Page
EntrancesPageviewsUnique PageviewsPages / SessionExits% ExitAvg. Time on Page
www.ga4bigquery.com
Return to table of contents
This example query contains all following Google Analytics dimensions and metrics. If you only need one dimension or metric, look at the -- comments in the example query and copy the part you need from the SELECT clause. Make sure that you also add any additional conditions (in the FROM, WHERE, GROUP BY and ORDER BY) that are necessary to calculate the results correctly.
Event CategoryEvent ActionEvent Label
Total EventsUnique EventsEvent ValueAvg. ValueSessions with EventEvents / Session with Event
www.ga4bigquery.com
Return to table of contents
This example query contains all following Google Analytics dimensions and metrics. If you only need one dimension or metric, look at the -- comments in the example query and copy the part you need from the SELECT clause. Make sure that you also add any additional conditions (in the FROM, WHERE, GROUP BY and ORDER BY) that are necessary to calculate the results correctly.
Goal Completion LocationGoal Previous Step-1Goal Previous Step-2Goal Previous Step-3
Goal XX Completions
www.ga4bigquery.com
Return to table of contents
This example query contains all following Google Analytics dimensions and metrics. If you only need one dimension or metric, look at the -- comments in the example query and copy the part you need from the SELECT clause. Make sure that you also add any additional conditions (in the FROM, WHERE, GROUP BY and ORDER BY) that are necessary to calculate the results correctly.
Transaction ID
TransactionsEcommerce Conversion RateRevenueAvg. Order ValuePer Session ValueShippingTaxRevenue per UserTransactions per User
www.ga4bigquery.com
Return to table of contents
When entering the product scope you have to verify if enhanced ecommerce is enabled in Google Analytics. If so, you’re safe to use the hits.product fields. If only ‘standard’ ecommerce is measured: use the hits.item fields.
This example query contains all following Google Analytics dimensions and metrics. If you only need one dimension or metric, look at the -- comments in the example query and copy the part you need from the SELECT clause. Make sure that you also add any additional conditions (in the FROM, WHERE, GROUP BY and ORDER BY) that are necessary to calculate the results correctly.
Product SKUProductProduct Category
QuantityUnique PurchasesAvg. PriceProduct RevenueAvg. QTY
www.ga4bigquery.com
Return to table of contents
When entering the product scope you have to verify if enhanced ecommerce is enabled in Google Analytics. If so, you’re safe to use the hits.product fields. If only ‘standard’ ecommerce is measured: use the hits.item fields.
This example query contains all following Google Analytics dimensions and metrics. If you only need one dimension or metric, look at the -- comments in the example query and copy the part you need from the SELECT clause. Make sure that you also add any additional conditions (in the FROM, WHERE, GROUP BY and ORDER BY) that are necessary to calculate the results correctly.
Product SKUProductProduct Category (Enhanced Ecommerce)Product BrandProduct Variant
QuantityUnique PurchasesProduct RevenueAvg. PriceAvg. QTYBuy-to-Detail RateCart-to-Detail RateProduct Adds To CartProduct CheckoutsProduct Detail ViewsProduct RefundsProduct Removes From CartRefund Amount
www.ga4bigquery.com
Return to table of contents
This example query contains all following Google Analytics dimensions and metrics. If you only need one dimension or metric, look at the # comments in the example query and copy the part you need from the SELECT clause. Make sure that you also add any additional conditions (in the FROM, WHERE, GROUP BY and ORDER BY) that are necessary to calculate the results correctly.
Custom Dimension XX (User)Custom Dimension XX (Session)Custom Dimension XX (Hit)Custom Dimension XX (Product)
Custom Metric XX Value (Hit)Custom Metric XX Value (Product)
www.ga4bigquery.com
Return to table of contents
If you need the Default Channel Grouping, just use the channelGrouping dimension. However, there can be various reasons to build your own Channel Grouping. When you use intraday data, for instance, as the channelGrouping dimension is not available there. Or when you need to ‘repair’ data quality issues on historical data, as this is not possible with the Default Channel Grouping dimension in the Google Analytics UI.
I will show you how you can mirror the standard Google Analytics definitions of the Default Channel Grouping in BigQuery. If you use this query as a starting point it’s not so difficult anymore to create your own custom or Default Channel Grouping.
www.ga4bigquery.com
Return to table of contents
For every Google Analytics view that is exported to BigQuery, a ga_sessions_intraday_ table will be exported multiple times a day as well. Let’s see how this works:
Within each dataset, a table is imported for each day of export. Daily tables have the format “ga_sessions_YYYYMMDD”.
Intraday data is imported approximately three times a day. Intraday tables have the format “ga_sessions_intraday_YYYYMMDD”. During the same day, each import of intraday data overwrites the previous import in the same table.
When the daily import is complete, the intraday table from the previous day is deleted. For the current day, until the first intraday import, there is no intraday table. If an intraday-table write fails, then the previous day’s intraday table is preserved.
Data for the current day is not final until the daily import is complete. You may notice differences between intraday and daily data based on active user sessions that cross the time boundary of last intraday import. (source)
The easiest way to include intraday data in your query as well as historical data is use the wildcard in combination with a wider _table_suffix filter. With this condition it will include any table in the data set that starts with ga_sessions_ and contains a date with the format YYYYMMDD.
Note: make sure your data set doesn’t contain any other tables with a title that starts with ga_sessions_!
www.ga4bigquery.com
Return to table of contents
If you don’t see an intraday table, but realtime tables and view, streaming export is enabled for your Google Analytics view. To query this data and join this with historical data from the ga_sessions_ tables, another approach is needed.
For each day, streaming export creates 1 new table and 1 (BigQuery) view of that table:
Table: ga_realtime_sessions_YYYYMMDD is an internal staging table that includes all records of sessions for all activity that took place during the day. Data is exported continuously approximately every 15 minutes. Within this table are multiple records of a session when the session spans multiple export operations.
The ga_realtime_sessions_YYYYMMDD tables should not be used (and are not supported by Google Analytics technical support) for queries. Queries on these tables may yield unexpected results as they may contain duplicate records of some sessions. Query the ga_realtime_sessions_view_YYYYMMDD view instead.
View: ga_realtime_sessions_view_YYYYMMDD sits on top of the exported tables and is there to deduplicate multiple records of repeated sessions that exist across export boundaries. Query this table for deduplicated streaming data. (source)
At the time of writing this daily generated realtime view is only queryable with Legacy SQL. Any Standard SQL query will result in this error:
To be able to use Standard SQL, we have to create our own realtime view.
www.ga4bigquery.com
Return to table of contents
No rights reserved
 by the author.