Create Advanced Google Shopping Insights using Merchant Center BigQuery exports
By: Krisjan Oldekamp
Tags: Merchant Center, Insights, Google, Bigquery, Google Shopping Ads

Create Advanced Google Shopping Insights using Merchant Center BigQuery exports

Original Photo by Jacek Dylag on Unsplash

Product level price competitiveness and benchmark data, already available in the Merchant Center interface, can be exported to Google BigQuery. Leveraging these exports, you can create advanced insights into your product catalog that can be used for automation as well.

In this tutorial you will learn how to:

Export product and brand level market insights data, based on data from all the merchants on Google Shopping to Google BigQuery

to Google BigQuery Enrich your product feed with Merchant Center data like price competitiveness , relative demand or product ranking to create valuable and advanced insights.

, or to create valuable and advanced insights. Use the example query to combine and join the data to your product feed.

Use the exports for other practical use cases.

Connecting Merchant Center to Google BigQuery

First step, you will need a Google Cloud Project with an active billing account. If not present already, you can check out these instructions.

The official Google documentation provides all the info to set up the data transfer service between Merchant Center and Google BigQuery. When the connection is set up, Merchant Center will run a daily export of several reports.

Source: Google

The Merchant Center exports explained

Several report types can be exported by the transfer service. You don’t have to export all the reports (can be configured when setting up the export).

Be aware that the Merchant Center exports are not that small in terms of GBs. For every available Google product category and country, it will export 10.000 top products for example. So it could generate some costs in your Google Cloud project. See the tips in the last chapter of this article.

The most important exports;

Also see this overview in the official Google documentation.

Some of the most interesting properties and definition;

price_benchmark_value : The average (click-weighted) price of a specific product, based on all merchants who advertise that same product on Shopping ads. Matching products between merchants is based on GTIN.

: The average (click-weighted) price of a specific product, based on all merchants who advertise that same product on Shopping ads. Matching products between merchants is based on GTIN. rank : The popularity of the product on Shopping ads, on Google product category and country level. Popularity is based on the estimated number of products sold. The rank updates daily, but data in the exports can be delayed up to 2 days.

: The popularity of the product on Shopping ads, on Google product category and country level. Popularity is based on the estimated number of products sold. The rank updates daily, but data in the exports can be delayed up to 2 days. previous_rank : The change in rank over the previous 7 days.

: The change in rank over the previous 7 days. relative_demand.min / .max / .bucket : A product’s current estimated demand in relation to the product with the highest popularity rank in the same category and country. Also includes buckets (e.g. Very High)

: A product’s current estimated demand in relation to the product with the highest popularity rank in the same category and country. Also includes buckets (e.g. Very High) previous_relative_demand.min / .max / .bucket : Same as above, only calculated on the last 7 days.

: Same as above, only calculated on the last 7 days. price_range.min` / `price_range.max: The lower and upper pricerange without decimals. Doesn’t include shipping costs.

Practical use cases

Combining the different reports, you can think of the following use cases;

Product feed status : Get Merchant Center and Shopping errors and notifications related to your product feeds.

: Get Merchant Center and Shopping errors and notifications related to your product feeds. Product inventory optimization : Using the best seller reports, you can check which popular products are already in your inventory and what potential products could be added, based on the complete list of top products per Google product categories AND country.

: Using the best seller reports, you can check which popular products are already in your inventory and what potential products could be added, based on the complete list of top products per Google product categories AND country. Price and bid optimization: Using the price competitiveness benchmark information, you can check how you are doing in terms of price against competitors.

Using the price competitiveness benchmark information, you can check how you are doing in terms of price against competitors. Trending products and brands : Identify trending products / brands based on the changes in best seller rank over the last X days / weeks / months. Differentiate per country

: Identify trending products / brands based on the changes in best seller rank over the last X days / weeks / months. Differentiate per country Product feed enrichment : Enrich you product feed with Google market insights data

: Enrich you product feed with Google market insights data Rule based automation: Based on the product feed enrichment, pause and enable campaigns / products in channels (e.g. Google Shopping) or adjust bids. For example based on price competitiveness.

Enrich product data with Merchant Center market insights

Example query

To show you how the data can be combined, I’ve constructed a query that will join and combine data out of different Merchant Center tables on product level.

Some remarks:

Google doesn’t provide data for every product in your feed . Either a product is not in the bestsellers list or Google just doesn’t provide or calculate it (not enough data and probably some other reasons). Or you have unique products that other merchants don’t offer.

. Either a product is not in the bestsellers list or Google just doesn’t provide or calculate it (not enough data and probably some other reasons). Or you have unique products that other merchants don’t offer. Use the BestSellers_TopProducts_Inventory as a mapping table to join the BestSellers_TopProducts data with your Product data by joining on rank_id (available in both tables)

The “original” product-ID as known in your systems is not directly available in the different exports. It’s available in the product_id field but you have te split the value and select the last part since Google added additional information to it, example: online:nl:NL:<original-product-id>

The reports contain a lot of data. For example the best seller reports contains 10.000 best sellers in every Google product category and country. For efficiency reasons, try to select only the countries needed. Also important because the same product can be duplicated over different countries in the table Google provides.

The reports contain nested and repeated data, make sure you understand the table structures. For example the product errors and notifications are nested.

Example: Enrich product feed with benchmark and price competitiveness data

The following query will result in this table that you could match on your original product feed and could serve as input for automation rules (on product level)

Make sure you replace the dataset / table names with the correct values in your situation

Code is also available on GitHub

#standardSQL — Set variable to change the fetch date easily DECLARE gmc_fetch_date DATE DEFAULT DATE(“2021–09–05”); WITH — Get productfeed uploaded in GMC for specific date gmc_products AS ( SELECT _PARTITIONDATE AS date, * FROM `dataset.Products_<merchant-id>` WHERE _PARTITIONDATE = gmc_fetch_date AND target_country IN (“NL”) ), — Get mapping table to join rank (bestsellers) and productfeed gmc_product_inventory AS ( SELECT * FROM `dataset.BestSellers_TopProducts_Inventory_<merchant-id>` WHERE _PARTITIONDATE = gmc_fetch_date AND (REGEXP_CONTAINS(rank_id, “:NL:”)) ), — Get price benchmark information gmc_price_benchmarks AS ( SELECT _PARTITIONDATE AS date, * FROM `dataset.Products_PriceBenchmarks_<merchant-id>` WHERE _PARTITIONDATE = gmc_fetch_date AND country_of_sale IN (“NL”) — Specify the countrycodes, since dataset countains all countries ), — Get bestseller information (if product ranks in bestseller list -> 10000 products per Google product category and country). gmc_bestsellers AS ( SELECT _PARTITIONDATE AS date, inventory.product_id AS product_id, bestseller.rank_timestamp, bestseller.rank_id, bestseller.rank, bestseller.previous_rank, IF(inventory.product_id IS NULL,false,true) AS product_in_inventory, bestseller.ranking_country, bestseller.ranking_category AS ranking_category_id, (SELECT name FROM bestseller.ranking_category_path WHERE locale = “nl-NL”) AS ranking_category_path, (SELECT name FROM bestseller.product_title WHERE locale = “nl-NL”) AS product_title, CONCAT(gtins[SAFE_OFFSET(0)],’|’,gtins[SAFE_OFFSET(1)],’|’,gtins[SAFE_OFFSET(2)],’|’) AS gtins, bestseller.brand, bestseller.google_brand_id, bestseller.google_product_category AS google_product_category_id, (SELECT name FROM bestseller.google_product_category_path WHERE locale = “nl-NL”) AS google_category_path, bestseller.price_range.min AS price_range_min, bestseller.price_range.max AS price_range_max, bestseller.price_range.currency AS price_range_currency, bestseller.relative_demand.min AS relative_demand_min, bestseller.relative_demand.max AS relative_demand_max, bestseller.relative_demand.bucket AS relative_demand_bucket, bestseller.previous_relative_demand.min AS previous_relative_demand_min, bestseller.previous_relative_demand.max AS previous_relative_demand_max, bestseller.previous_relative_demand.bucket AS previous_relative_demand_bucket FROM `dataset.BestSellers_TopProducts_<merchant-id>` AS bestseller — Only join products when available in the productfeed and the bestsellers list INNER JOIN gmc_product_inventory AS inventory ON bestseller.rank_id = inventory.rank_id WHERE _PARTITIONDATE = gmc_fetch_date AND ranking_country IN (“NL”) ) — Join all the data on product level SELECT products.date, products.product_id AS product_id_gms, SPLIT(products.product_id, ‘:’)[SAFE_OFFSET(3)] AS product_id, products.target_country, products.title, products.price.value, benchmarks.price_benchmark_value, (CAST(benchmarks.price_benchmark_value AS FLOAT64) — CAST(products.price.value AS FLOAT64)) AS price_benchmark_diff, benchmarks.price_benchmark_currency, bestsellers.rank, bestsellers.previous_rank, bestsellers.price_range_min, bestsellers.price_range_max, bestsellers.relative_demand_min, bestsellers.relative_demand_max, bestsellers.relative_demand_bucket, bestsellers.previous_relative_demand_min, bestsellers.previous_relative_demand_max, bestsellers.previous_relative_demand_bucket FROM gmc_products AS products — Join price benchmark data LEFT JOIN gmc_price_benchmarks AS benchmarks ON products.product_id = benchmarks.product_id AND products.date = benchmarks.date — Join bestseller data LEFT JOIN gmc_bestsellers AS bestsellers ON products.product_id = bestsellers.product_id AND products.date = bestsellers.date

Closing thoughts

Using the BigQuery Merchant Center exports, you can generate insights at scale and use it for automation. But keep in mind, the exports hold a lot of data, so it could generate some costs in your Google Cloud project (and you probably won’t need all the data). The tables are date partitioned, so you could write a Python script or use Airflow to delete partitions older than X days to keep the amount of data in BigQuery (probably after you’ve created some aggregate reports).

This article was originally posted on stacktonic.com