SAP Data Analytics in the Google Cloud
Tags: Data Analytics, Google Cloud Platform, Sap, Bigquery, Data Engineering

SAP Data Analytics in the Google Cloud

How to combine SAP with the Google Cloud Platform— Provider of powerful data analytics tools such as BigQuery, Data Studio or recently looker — to gain a powerful data analytics platform and valuable insights. If you are interested in practical data analytics approaches for SAP data, this article could be also interesting for you. Christianlauer Jul 19, 2020·4 min read

The Cloud as Enabler for Data Analytics — Image by Alex Machado on Unsplash

Architecture

With SAP HANA, the SAP Data Service is already build in so you can easily export data from a SAP application or its underlying database to BigQuery. However, the combination of GCP and SAP is also possible for companies who are still living in the ERP world.

These following points show possibilities of loading data from SAP to Google Services:

Extracting data via RFC (Remote Function Call) interface with data integration tools like talend/Google Dataflow/etc.

Writing an ABAP program

3rd party connectors like fivetran

Using a SAP Replication Server

Therefore, a possible solution could look like this:

SAP to Big Query — Image from Google [1]

Realize Delta Load

Getting the data is the first step. Afterwards, you will deal with the question of how to realize a delta logic, because you probably don’t want to load all the data on a regular basis via full load. This would only come with disadvantages like heavy loading times and also having no historical data. Especially departments like controlling, internal audit, etc. often build up their data analytics based on historical data views.

While the SAP Data Service comes already build in with a CDC wizard, other data integration approaches might need some extra work. To realize this step it’s important to get to know the SAP tables and table structure very well. Realizing the delta load on an own logic can be realized especially with the help of two tables: CDHDR (Changes Header) and CDPOS (Changes Position). These tables are tracking changes in master data or transactional data.

Data Transformation

With the new era of column based data warehousing tools such as BigQuery, data transformation is often necessary due to performance but also because of different table structures or data types between the source and target system.

One example is, that in SAP you have the tables BKPF (Accounting Document Header) and BSEG (Accounting Document Segment). Due to its column based structure it will make sense to denormalize the data within the BigQuery. The solution could be to join BKPF and BSEG and, if needed other master data and reference tables — for example with the help of the build in BigQuery DTS (Data Transfer Service). As a result, you can come up with a denormalized data object architecture as shown in the figure below:

Illustration: From raw tables to a denormalized data object — Image by Author

Another transformation to gain performance and cost efficiency could be to use nested data structures. Newer data warehouse technologies like BigQuery or Amazon’s Redshift do work better with this data structure. Use cases for working with nested data are e.g header/segment tables like BKPF and BSEG or if you have cases where a table holds columns which don’t change much, like:

name

lastname

combined with historically often changeable columns like [2]:

address (a nested and repeated field and its following nested fields):

(a nested and repeated field and its following nested fields): address.address

address.city

Implement (Self Service) Reporting

The last part would be to realize reports and data analytic possibilities. With BigQuery, the data analyst already has a great tool for analytics. Data scientists can use Data Lab or one of the many ML Services within the GCP. Data Studio could be the right solution for business users for sharing KPI’s, dashboards, etc. To implement a Role/User access model within the BigQuery ,it’s a common way to use the BUKRS attribute (Company Code) — so a user would only see data of the company code assigned to him, that’s also the way SAP handles user rights.

Conclusion — the combination of SAP and GCP could lead to significant data analytics competencies and possibilities in your company. SAP systems are hosting a lot of interesting data that wants to be analyzed while GCP comes with many data analytics/scientist tools and services which will enable to do so.

Sources & Further Readings:

[1] Google, https://cloud.google.com/solutions/sap/docs/bigquery-replication-from-sap-apps?hl=de

[2] Google, https://cloud.google.com/bigquery/docs/nested-repeated

https://blogs.sap.com/2018/04/08/hana-sda-google-bigquery/

https://cloud.google.com/solutions/sap/docs/bigquery-sap-export-using-sds