How to query and calculate GA4 event data in BigQuery | by Johan van de Werken | Towards Data Science

The future of app and web analytics is here: Google announced a new way of measurement in Google Analytics. While still in beta, this is a small revolution for web and app analytics.
On top of this: there is a raw data BigQuery export available for all (even non 360) Google Analytics accounts. Let’s dive right in!
Note: do you enjoy this article? Then you’ll also like my new website GA4BigQuery.com: a digital guide with tips, ideas, example queries and tutorials on how to query Google Analytics data in BigQuery & rock your digital marketing analytics 🤘
You also might be interested in a digital course that I designed together with Mari and Simo Ahava for marketing professionals who want to dive into the world of queries, and for data professionals who need to skill up on GA4-specific knowledge:
www.teamsimmer.com
In this article I will focus on web data only. I will show you how to calculate new GA4 engagement metrics, but also how to replicate ‘classic’ Google Analytics dimensions and metrics that you won’t find in the GA4 user interface anymore. But beware...
This is not Universal Analytics. Try to look at the new measurement model as an opportunity to go beyond what Universal Analytics ever could, rather than just another way to populate GA’s ancient data model. (source)
If you are new to BigQuery, I recommend reading the first part of my previous article on how to query Google Analytics (Universal Analytics) data in BigQuery.
– What is going on?– Documentation– New engagement metrics– Google Analytics 4 BigQuery export schema– Nested fields– Get all unique events and parameters– Example queries: ‘standard’ dimensions & metrics– Example queries: calculated dimensions & metrics
Krista Seiden, former Analytics Advocate at Google, explains in a series of blogs why the new Google Analytics 4 property is a big step forward:
Currently, if you have both app and website data, you are collecting and analyzing that data separately, in Google Analytics for the web, and in Google Analytics for Firebase for apps. While the platforms differ, many of the KPIs and business needs when it comes to understanding data and reporting are the same, so it’s time to bring this data together in one Analytics property! (source)
This development has severe implications on the way data will be collected and analysed.
This new type of measurement uses the same data schema as Google Analytics for Firebase (GA4F) (...) The most important thing to know about this new data schema is that it moves away from the traditional Session + Pageview method that classic Google Analytics has used for 15+ years. Instead, it uses an Event + Parameter model. It’s different, but it opens up a whole new world of possibilities in terms of what you can track, and all of the additional details you can add to each event action via the associated parameters. (source)
Return to table of contents
If all this is new to you, please read all about GA4 properties before proceeding with the queries.
www.ga4bigquery.com
In the the Google Analytics 4 property we see some new metrics that were not available earlier:
The number of sessions that lasted longer than 10 seconds, or had a conversion event, or had 2 or more screen or page views. (source: definition in GA4 UI)
Here is some background by Krista Seiden:
The idea of an engaged session stems from wanting to remove bounces from the session count, therefore, an engaged session is only counted as such after the user has been on site for 10 seconds. If they leave before then, it will count as a bounce but not as an engaged session. (source)
And Florian Perl adds:
This is more or less opposite of Universal Analytics Bounces. An Engaged session is one with at least 2 pageviews, or a pageview + an interaction, or 1 pageview with 10-second duration. The 10-second timeout cannot be changed, for now. (source)
Number of engaged sessions per user, so if we see the same user ID coming back multiple times, this may be 3, 4, or 5 sessions per user, then averaged out over your total user count, giving you an average above 1, with the actual number dependent on how many new vs returning users (or deleted cookies) your content has. (source)
This one is perhaps my favorite new metric. Why? For years, I’ve been talking about why I hate the metrics ‘Bounce Rate’ and ‘Time on Page’ in Universal Google Analytics. There is good reason for this dislike: the metrics are flawed in how they are calculated due to how the use of the web has changed since the inception of these metrics more than 15 years ago. (...) In Google Analytics for Firebase, there is a metric called ‘Engagement time’, which I like much better because it is calculated as the actual time in foreground for the app which is much more reliable than ‘Time on Page’ in Universal Google Analytics.
For A+W, we wanted to bring a similar logic, but due to how browsers and tabs work, that is much harder. Our small team spent quite some time debating on the best way to calculate a metric that would be meaningful for analysis, and eventually came to ‘Engagement time’ for web which is meant to be synonymous with how it’s calculated in app, using some new handlers to closely follow the tab in foreground, screen sleep time, etc. Essentially, a much better calculation of time on page. (source)
The percentage of engaged sessions (Engaged Sessions divided by Sessions). (source: definition in GA4 UI)
These are not immediately useful if we mean to duplicate Universal Analytics reports in App+Web: Conversions are a count of events marked as Conversions, not deduplicated on session level (so it is not really a session-scope metric). What is more, right now there is no way to specify which conversion we want to view — they are all thrown into the same pot. The most likely use case is when we only track a single Conversion event — then the report will be more meaningful. (source)
Return to table of contents
As there is no sample data set for Google Analytics 4 properties (yet?), I will use an export data set (web only) from OrangeValley, the digital marketing agency I worked for at the time of writing. Apart from the standard events that are collected through ‘enhanced measurement’, it is possible you will see some custom event parameters in our data set, like ‘wordCount’ in event ‘page_view’. This does not impact our schema in any way.
Let’s take a look at the GA4 BigQuery export schema:
As you will notice the data set is heavily organised around event and user data, supplemented with some device, geo, app and traffic source data.
Depending on various definitions, you can expect some relatively small differences between the Google Analytics 4 user interface and the results of your BigQuery export data queries. Read about differences in user counts or dimension & metric definitions.
Return to table of contents
In the Universal Analytics export schema, every row in the data set represents a single session. As the GA4 properties are event based, every row in our data set represents an event. Every event in turn can contain multiple event parameters and corresponding values. To show you how this works in BigQuery, we query our nested sample set:
SELECT  *FROM  `ga4.analytics_1234567890.events_*`LIMIT  3
This gives us 3 rows, which represented as a flat table would look like this:
Remember, only row 2,9 and 15 in this example are real rows in our table. The other ‘rows’ are in fact nested fields, in most cases NULL values. Only the event.params columns are populated with values.
To deal with this fields and to be able to query our tables so they meet our needs, we need the UNNEST function.
The problem here is that is essentially an array (actually in BigQuery parlance it’s a “repeated record”, but you can think of it as an array). (...) This is where the UNNEST function comes in. It basically lets you take elements in an array and expand each one of these individual elements. You can then join your original row against each unnested element to add them to your table. (source)
I highly recommend reading this article which explains the UNNEST concept in detail with the Firebase Analytics sample data set as an example.
You only have to UNNEST records that contain ‘repeated fields’. In case of our Google Analytics 4 data set these could involve:
event_params
user_properties
items
Return to table of contents
As every data set can and will contain different events and event parameters with different data types, it is very helpful to query the actual tables to find out. Especially when you lack documentation about the way data is collected. To get all unique events and parameters you can use a query like this:
In case of our data set, this results in a list of all unique events and parameters with their corresponding data type. These repeated fields contain a lot of data, which we will use to calculate dimensions and metrics.
Return to table of contents
A lot of data can be derived straight from the tables without performing calculations on the data. With the help ofUNNEST and PARSE for dates or timestamps we can do the trick.
Look at the -- comments in the example query for names and definitions and copy the part you need from the SELECT clause. Make sure that you also add any additional conditions (i.e. WITH, FROM, WHERE, GROUP BY, HAVING and ORDER BY) that are necessary to calculate the results correctly.
—
user_iduser_pseudo_iduser_first_touch_timestampuser_string_valueuser_set_timestamp_microsuser_ltv_revenue currency
user_int_valueuser_float_valueuser_double_value
www.ga4bigquery.com
—
device.categorydevice.mobile_brand_namedevice.mobile_model_namedevice.mobile_marketing_namedevice.mobile_os_hardware_modeldevice.operating_systemdevice.operating_system_versiondevice.vendor_iddevice.advertising_iddevice.languagedevice.is_limited_ad_trackingdevice.time_zone_offset_secondsdevice.browserdevice.browser_versiondevice.web_info.browserdevice.web_info.browser_versiondevice.web_info.hostname
www.ga4bigquery.com
—
geo.continentgeo.sub_continentgeo.countrygeo.regiongeo.citygeo.metro
www.ga4bigquery.com
—
app_info.idapp_info.versionapp_info.install_storeapp_info.firebase_app_idapp_info.install_source
www.ga4bigquery.com
—
traffic_source.name (user)traffic_source.medium (user)traffic_source.source (user)
www.ga4bigquery.com
—
event_dateevent_timestampevent_nameevent_keyevent_previous_timestampevent_bundle_sequence_idevent_server_timestamp_offsetstream_idplatformevent_dimensions.hostname
event_int_valueevent_float_valueevent_double_valueevent_value_in_usd
www.ga4bigquery.com
Return to table of contents
ecommerce.transaction_id
ecommerce.total_item_quantityecommerce.purchase_revenue_in_usdecommerce.purchase_revenueecommerce.refund_value_in_usdecommerce.refund_valueecommerce.shipping_value_in_usdecommerce.shipping_valueecommerce.tax_value_in_usdecommerce.tax_valueecommerce.unique_items
www.ga4bigquery.com
Return to table of contents
items.item_iditems.item_nameitems.item_branditems.item_variantitems.item_categoryitems.item_category2items.item_category3items.item_category4items.item_category5items.couponitems.affiliationitems.location_iditems.item_list_iditems.item_list_nameitems.item_list_indexitems.promotion_iditems.promotion_nameitems.creative_nameitems.creative_slot
items.price_in_usditems.priceitems.quantityitems.item_revenue_in_usditems.item_revenueitems.item_refund_in_usd
www.ga4bigquery.com
Return to table of contents
Other dimensions & metrics you are used to in your Google Analytics analysing and reporting are a bit harder to access. You’ll have to calculate them, using your own queries.
As I’ve tried to sort out the most used dimensions and metrics already, I hope I can be of help with the following example queries. There are plenty other, probably even better ways to achieve these results, but here is my two cents.
These example queries (ordered by category) contain all following Google Analytics dimensions and metrics. If you only need one dimension or metric, look at the -- comments in the example query for names and definitions and copy the part you need from the SELECT clause. Make sure that you also add any additional conditions (i.e. WITH, FROM, WHERE, GROUP BY, HAVING and ORDER BY) that are necessary to calculate the results correctly.
—
User typeCount of sessions
UsersNew users% New sessionsNumber of sessions per userEvent count per user
www.ga4bigquery.com
—
SessionsEngaged sessionsEngagement rateEngagement timeBouncesBounce rateEvent count per session
www.ga4bigquery.com
—
The calculation for engaged sessions above is based on the event ‘user_engagement’. To be more specific: when parameter ‘session_engaged’ = ‘1’. The definition of engaged sessions, according to the GA4 user interface, is:
The number of sessions that lasted longer than 10 seconds, or had a conversion event, or had 2 or more screen or page views.
You can set a conversion event in the UI, but the definitions of session length or the amount of page views cannot be changed if you use the event ‘user_engagement’. But what if we want to customise the definition of an engaged session? You could use (a variation on) this advanced query.
—
DateYearISO yearMonth of yearMonth of the yearWeek of yearWeek of the yearISO week of the yearISO week of ISO yearDay of the monthDay of weekDay of week nameHourMinuteHour of dayDate hour and minute
www.ga4bigquery.com
BigQuery displays data usually in UTC. That leads to problems when using date formatting functions because dates and times can be off. Converting the datetimes prior formatting into the correct timezone solves those issues.(source & how to)
—
Default channel grouping (user or session)Source / medium (user or session)Campaign (user or session)Full referrer
www.ga4bigquery.com
—
HostnamePagePrevious page pathPage path level 1Page path level 2Page path level 3Page path level 4Page titleLanding pageSecond pageExit page
EntrancesPageviewsUnique pageviewsPages / sessionExits% Exit
www.ga4bigquery.com
—
As I’ve had no access to any ecommerce data set for GA4 so far, I’d suggest to take a look at the query examples for ecommerce transactions and products in the guide on querying GA4 data by Bounteous.
Return to table of contents
No rights reserved
 by the author.