Data Fusion: A Code-Free Pipeline for Your Google Cloud Data Warehouse
By: Guillaume Dupont
Tags: Data Warehouse, Cloud, Business Intelligence, Etl, Pipeline

Once the wrangling steps have been defined, the list of instructions in the recipe is applied to the transformation in the pipeline. When executed, the transformation converts input lines in its input schema into the output schema. Data Fusion does not always update the output schema in the wrangler transformation directly, but both recipes and schemas can be exported as JSON files and imported throughout Data Fusion. The recipe also shares the same language as Google Data Prep, which is a slightly more powerful tool but requires a separate license.

Data Fusion provides many other transformations which will be indispensable for more complex pipelines: Normalizing/denormalizing, XML parsing, etc. While not always as generic as Airflow equivalents, Data Fusion transformations provide comprehensive functional coverage. A transformation which will be inevitably useful is the JavaScript one. As Data Fusion is still a relatively new solution, at one point or another it will lack a feature. This is where the JavaScript transformation becomes handy. Need to add timestamps but there is no transformation to manage timestamps or file names? Not a problem, a three-line script can take care of this.

3. Big Query

Outputs of the transformations can be sent to various data stores and databases, among which Big Query which will be the foundation for the data warehouse. The Data Fusion BigQuery sink turns data produced by the transformations into rows appended in BigQuery tables. Note that BigQuery does not require explicitly defined schemas when the table is created, in which case the schema will be recorded when the first rows get inserted in the table. While convenient, it can be difficult to manage schema changes between Data Fusion and Big Query, especially in the early steps of the data warehouse creation.

Big Query data warehouse to store transformed data. Screenshot by author.

Data Fusion can in turn read from Big Query in order to transform the data and create more complex pipelines. Another alternative for relatively simple data transformations is to use Big Query Scheduled Queries: These will run at scheduled times to select data from given tables and write outputs in new tables.

4. Data studio

Data studio is Google’s SaaS data visualization tool. It is free and tightly integrated with Big Query, so that

Visualizations can instantly be created and updated with data retrieved from Big Query tables using the SQL language

Vice versa, query outputs from the Big Query GUI can be instantly exported and visualized in Data Studio

Data Studio expects relatively structured data in input and provides less filtering or aggregation capabilities than competing tools like PowerBI, hence having the right SQL scripts and in some cases preprocessed tables is important. Once this point is understood, Data Studio provides an extremely efficient framework to almost instantly retrieve, edit, publish, and export reports to PDF. In addition, parameters in the report can be directly considered within the SQL queries, allowing to easily produce reports for a variety of configurations or for each entity.