How to extend your Apache Atlas metadata to Google Data Catalog
By: Marcelo Costa
Tags: Big Data, Kafka, Compliance, Google Cloud Platform, Data Visualization

Disclaimer: All opinions expressed are my own, and represent no one but myself…. They come from the experience of participating in the development of fully operational sample connectors, available on: github.

If you missed the latest post talking about how Apache Atlas and Data Catalog structure their metadata, please check a-metadata-comparison-between-apache-atlas-and-google-data-catalog.

The Dress4Victory company

In this article, we will start by creating a fictional scenario, where we have the Dress4Victory company. They help their users getting the best deals when buying clothes, and over the years they have grown from a few servers to several hundred servers.

Dress 4 Victory company logo

This company has many analytics workloads to handle their user data and to support it, their tech stack is composed mostly of Hadoop components.

Generally speaking, their metadata management was a mess. So last year their CTO added Apache Atlas to their tech stack, to better organize their metadata, and visualize the data structures of their enterprise.

By improving their metadata management, it helped them solve many problems like:

Analysts taking a long time to find meaningful data.

Customer data spread everywhere.

Issues with access controls on their data.

Compliance requirements being ignored.

Now they are migrating some workloads to Google Cloud Platform, and he’s scared that it will be much harder to manage their metadata since Apache Atlas has starting to work for them.

He found out about Google Data Catalog, and he would love to use it, since it’s fully managed and will reduce his operational costs, but they can’t migrate everything to GCP, at the moment.

Luckily for him, he found out that there’s a connector for apache-atlas. He wants to start right away and test it out.

Full ingestion execution

Let’s help Dress4Victory and look at the Apache Atlas connector full ingestion architecture:

Full Ingestion Architecture

On each execution, it’s going to:

Scrape : connect to Apache Atlas and retrieve all the available metadata.

: connect to Apache Atlas and retrieve all the available metadata. Prepare : transform it in Data Catalog entities and create Tags with extra metadata.

: transform it in Data Catalog entities and create Tags with extra metadata. Ingest: send the Data Catalog entities to the Google Cloud project.

Currently, the connector supports the below Apache Atlas asset types:

Entity Types

Each Entity Type is converted into a Data Catalog Template with their attributes metadata. Since Google Data Catalog has pre-defined attributes, we create an extra Template to enrich the Apache Atlas metadata.

Each Entity Type is converted into a Data Catalog Template with their attributes metadata. Since Google Data Catalog has pre-defined attributes, we create an extra Template to enrich the Apache Atlas metadata. Classification Types

Each Classification Type is converted into a Data Catalog Template, so we are able to empower users to create Tags, using the same Classification they were used to work within Apache Atlas. If there are Classifications attached to Entities, the connector also migrates them as Tags.

Each Classification Type is converted into a Data Catalog Template, so we are able to empower users to create Tags, using the same Classification they were used to work within Apache Atlas. If there are Classifications attached to Entities, the connector also migrates them as Tags. Entities

Each Entity is converted into a Data Catalog Entry. Since we don’t have a Type structure in Google Data Catalog, all entries from the same type, share the same Template, so users can search in a similar way they would do in Apache Atlas.

Since even Columns are represented as Apache Atlas Entities, this connector, allows users to specify the Entity Types list as a command to be considered in the ingestion process.

At the time this was published, Data Catalog does not support Lineage, so this connector does not use the Lineage information from Apache Atlas. We might consider updating this if things change.

Running it

After setting up the connector environment, by following the instructions at the Github repo, let’s execute it using its command line args:

#Environment variables

export GOOGLE_APPLICATION_CREDENTIALS=datacatalog_credentials_file

export DATACATALOG_PROJECT_ID=google_cloud_project_id

export APACHE_ATLAS2DC_HOST=localhost

export APACHE_ATLAS2DC_PORT=21000

export APACHE_ATLAS2DC_USER=my-user

export APACHE_ATLAS2DC_PASS=my-pass google-datacatalog-apache-atlas-connector sync \

--datacatalog-project-id $DATACATALOG_PROJECT_ID \

--atlas-host $APACHE_ATLAS2DC_HOST \

--atlas-port $APACHE_ATLAS2DC_PORT \

--atlas-user $APACHE_ATLAS2DC_USER \

--atlas-pass $APACHE_ATLAS2DC_PASS

Results

Turn the subtitles on for step-by-step guidance when watching the video.