Apache Atlas and Google Data Catalog comparison
By: Marcelo Costa
Tags: Analytics, Data, Apache Atlas, Big Data, Data Science

Apache Atlas

Defines their core metadata as:

Apache Atlas Metadata mental model

Atlas allows users to define a model for the metadata objects they want to manage. The model is composed of definitions called types .

A type represents one or a collection of attributes that define the properties for the metadata object.

— Hey Marcelo, can we compare a type with any Google Data Catalog object?— Sorry! This is not a fair comparison, if you look at the mental models, the hierarchies are different! But let’s dig deeper and we will find some similarities.

There are two Composite Metatypes: Struct and Relationships that are out of the scope of this article. Google Data Catalog does not support lineage at the time of this writing, so we are not using Relationships. And if you are using struct types, I’d love to know your use cases and perhaps improve this article.

Now let’s understand each main component on that diagram.

Primitive and Enum Meta types

Think about any programming language, those are the most basic types , that you can use when creating your Entities and Classifications attributes.

Collection Meta types

This is where things get interesting, you can use arrays and maps structures composed of the primitive and enum types.

Let’s say you have a Table in Atlas, that Table will surely contain some columns. So here you would represent the columns as an array meta-type.

Composite Meta types

Here are the two most important units, Entities, and Classifications.

Entities

I told you we would find similarities, Entities in Atlas are close to what we call Entries in Google Data Catalog.

They represent an asset’s technical metadata, the difference, is that there are no pre-defined fields.

Gif from giphy

Sounds scary right? This give users a lot of flexibility, but comes with complexity, so use with care.

Lucky for us, Atlas comes with some pre-defined entity types for various Hadoop and non-Hadoop metadata, and you can even ingest sample models and data by running their quick_start.

Also, entity types can extend from other types, called superTypes , so you receive attributes from ancestors. Let’s look at one example:

Atlas Table Example

This image shows the attributes for a Table entity called customer_dim .

If we look at the Table ancestors, we would get this hierarchy: Table -> DataSet -> Asset -> Referenceable.

And the attributes we are looking at, are the combination of that hierarchy.

Bear in mind that DataSet is one of the most important types — according to Atlas documentation: “DataSet can be expected to have a Schema” — allowing us to add classifications on them later on.

Classifications

Do you remember Google Data Catalog Templates? We can say Classifications are really similar to them.

Just like Templates entities can be associated with Classifications, enabling easier discovery and management.

As Atlas Entities we have the same attributes and superTypes capabilities when creating Classifications.

To show how similar Classifications are from Google Data Catalog Templates we are going to create one named ETL Governance.

ETL Classification creation

The difference here, is we are adding the PII superType.