Package: google-datacatalog-apache-atlas-connector
google-datacatalog-apache-atlas-connectorPackage for ingesting Apache Atlas metadata into Google Cloud Data Catalog, currently
supporting below asset types:Entity Types -> Each Entity Types is converted to a Data Catalog Template with their attribute metadataClassificationDefs -> Each ClassificationDef is converted to a Data Catalog TemplateEntityDefs -> Each Entity is converted to a Data Catalog EntryEntity attributes are converted to Data Catalog Tags, in case there are Table and Columns relashionships,
Columns will be converted to Data Catalog Table schema.Since even Columns are represented as Apache Atlas Entities, this connector, allows users to specify the Entity Types list
to be considered in the ingestion process. If you don't want any type to be created as Data Catalog Entries, use the Entity Types list
arg to provide only the types the connector should sync.At this time Data Catalog does not support Lineage, so this connector does not use the Lineage information. We might
consider updating this if things change.Disclaimer: This is not an officially supported Google product.Table of Contents1. Installation1.1. Mac/Linux1.2. Windows1.3. Install from source1.3.1. Get the code1.3.2. Create and activate avirtualenv2. Environment setup2.1. Auth credentials2.1.1. Create a service account and grant it below roles2.1.2. Download a JSON key and save it as2.2. Set environment variables3. Sample Sync application entry point3.1. Run the google-datacatalog-apache-atlas-connector script3.2. Run Docker entry point4. Sample Sync Hook application entry point4.1. Run the google-datacatalog-apache-atlas-connector script4.2. Run Docker entry point5. Developer environment5.1. Install and run Yapf formatter5.2. Install and run Flake8 linter5.3. Run Tests6. Metrics7. Assumptions8. Troubleshooting1. InstallationInstall this library in avirtualenvusing pip.virtualenvis a tool to
create isolated Python environments. The basic problem it addresses is one of
dependencies and versions, and indirectly permissions.Withvirtualenv, it's possible to install this library without needing system
install permissions, and without clashing with the installed system
dependencies. Make sure you use Python 3.7+.1.1. Mac/Linuxpip3installvirtualenv
virtualenv--pythonpython3.7<your-env>source<your-env>/bin/activate
<your-env>/bin/pipinstallgoogle-datacatalog-apache-atlas-connector1.2. Windowspip3installvirtualenv
virtualenv--pythonpython3.7<your-env>
<your-env>\Scripts\activate
<your-env>\Scripts\pip.exeinstallgoogle-datacatalog-apache-atlas-connector1.3. Install from source1.3.1. Get the codegitclonehttps://github.com/GoogleCloudPlatform/datacatalog-connectors-hive.gitcddatacatalog-connectors-hive/google-datacatalog-apache-atlas-connector1.3.2. Create and activate avirtualenvpip3installvirtualenv
virtualenv--pythonpython3.7<your-env>source<your-env>/bin/activate2. Environment setup2.1. Auth credentials2.1.1. Create a service account and grant it below rolesData Catalog Admin2.1.2. Download a JSON key and save it as<YOUR-CREDENTIALS_FILES_FOLDER>/apache-atlas2dc-credentials.json2.2. Set environment variablesexportGOOGLE_APPLICATION_CREDENTIALS=datacatalog_credentials_fileexportAPACHE_ATLAS2DC_DATACATALOG_PROJECT_ID=google_cloud_project_idexportAPACHE_ATLAS2DC_HOST=localhostexportAPACHE_ATLAS2DC_PORT=21000exportAPACHE_ATLAS2DC_USER=my-userexportAPACHE_ATLAS2DC_PASS=my-pass3. Sample Sync application entry point3.1. Run the google-datacatalog-apache-atlas-connector sync scriptExecutes full scrape process in Apache Atlas and sync Data Catalog metadata creating/updating/deleting Entries and Tags.Virtualenvgoogle-datacatalog-apache-atlas-connectorsync\--datacatalog-project-id$APACHE_ATLAS2DC_DATACATALOG_PROJECT_ID\--atlas-host$APACHE_ATLAS2DC_HOST\--atlas-port$APACHE_ATLAS2DC_PORT\--atlas-user$APACHE_ATLAS2DC_USER\--atlas-pass$APACHE_ATLAS2DC_PASS\--atlas-entity-typesDB,View,Table,hbase_table,hive_db(Optional)3.2. Run Docker entry pointExecutes incremental scrape process in Apache Atlas and sync Data Catalog metadata creating/updating/deleting Entries and Tags. This options listen to event changes on Apache Atlas event bus, which is Kafka.dockerbuild--rm--tagapache-atlas2datacatalog.
dockerrun--rm--tty-v<YOUR-CREDENTIALS_FILES_FOLDER>:/data\apache-atlas2datacatalogsync\--datacatalog-project-id$APACHE_ATLAS2DC_DATACATALOG_PROJECT_ID\--atlas-host$APACHE_ATLAS2DC_HOST\--atlas-port$APACHE_ATLAS2DC_PORT\--atlas-user$APACHE_ATLAS2DC_USER\--atlas-pass$APACHE_ATLAS2DC_PASS\--atlas-entity-typesDB,View,Table,hbase_table,hive_db(Optional)4. Sample Sync Hook application entry point4.1. Run the google-datacatalog-apache-atlas-connector event-hook scriptVirtualenvgoogle-datacatalog-apache-atlas-connectorsync-event-hook\--datacatalog-project-id$APACHE_ATLAS2DC_DATACATALOG_PROJECT_ID\--atlas-host$APACHE_ATLAS2DC_HOST\--atlas-port$APACHE_ATLAS2DC_PORT\--atlas-user$APACHE_ATLAS2DC_USER\--atlas-pass$APACHE_ATLAS2DC_PASS\--event-serversmy-event-server\--event-consumer-group-idatlas-event-sync\--atlas-entity-typesDB,View,Table,hbase_table,hive_db(Optional)4.2. Run Docker entry pointdockerbuild--rm--tagapache-atlas2datacatalog.
dockerrun--rm--tty-v<YOUR-CREDENTIALS_FILES_FOLDER>:/data\apache-atlas2datacatalogsync-event-hook\--datacatalog-project-id$APACHE_ATLAS2DC_DATACATALOG_PROJECT_ID\--atlas-host$APACHE_ATLAS2DC_HOST\--atlas-port$APACHE_ATLAS2DC_PORT\--atlas-user$APACHE_ATLAS2DC_USER\--atlas-pass$APACHE_ATLAS2DC_PASS\--event-serversmy-event-server\--event-consumer-group-idatlas-event-sync\--atlas-entity-typesDB,View,Table,hbase_table,hive_db(Optional)5. Developer environment5.1. Install and run Yapf formatterpipinstall--upgradeyapf# Auto update filesyapf--in-place--recursivesrctests# Show diffyapf--diff--recursivesrctests# Set up pre-commit hook# From the root of your git project.curl-opre-commit.shhttps://raw.githubusercontent.com/google/yapf/master/plugins/pre-commit.sh
chmoda+xpre-commit.sh
mvpre-commit.sh.git/hooks/pre-commit5.2. Install and run Flake8 linterpipinstall--upgradeflake8
flake8srctests5.3. Run Testspythonsetup.pytest6. MetricsMetrics README.md7. AssumptionsThe connector uses simple authentication with user/pass provided credentials. And to provide live sync, the connector has an option to connect to the Apache Atlas backed Kafka instance, and listen to metadata event changes. It connects directly to Kafka's topic, so make sure it is executed in a secure network.For stronger security, consider using Kerberos for authentication and Apache Ranger for authorization:apache-atlas-security. If you have this kind of usage, please open a feature request.8. TroubleshootingIn the case a connector execution hits Data Catalog quota limit, an error will
be raised and logged with the following detailment, depending on the performed
operation READ/WRITE/SEARCH:status = StatusCode.RESOURCE_EXHAUSTED
details = "Quota exceeded for quota metric 'Read requests' and limit 'Read requests per minute' of service 'datacatalog.googleapis.com' for consumer 'project_number:1111111111111'."
debug_error_string = 
"{"created":"@1587396969.506556000", "description":"Error received from peer ipv4:172.217.29.42:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Quota exceeded for quota metric 'Read requests' and limit 'Read requests per minute' of service 'datacatalog.googleapis.com' for consumer 'project_number:1111111111111'.","grpc_status":8}"For more information on Data Catalog quota, please refer to:Data Catalog quota docs.