Package: google-datacatalog-hive-connector
google-datacatalog-hive-connectorLibrary for ingesting Hive metadata into Google Cloud Data Catalog.
You are able to directly connect to your Hive Metastore or Consume message events using Cloud Run.This connector is prepared to work with the Hive Metastore 2.3.0 version,
backed by a PostgreSQL or MySQL database.Disclaimer: This is not an officially supported Google product.Table of Contents1. Installation1.1. Mac/Linux1.2. Windows1.3. Install from source1.3.1. Get the code1.3.2. Create and activate avirtualenv2. Environment setup2.1. Auth credentials2.1.1. Create a service account and grant it below roles2.1.2. Download a JSON key and save it as2.2. Set environment variables to connect to your Hive Metastore3. Run entry point3.1. Run Python entry point3.2. Run Docker entry point4. Deploy Message Event Consumer on Cloud Run (Optional)4.1. Set environment variables to deploy to Cloud Run4.2. Execute the deploy script4.3. Create your Pub/Sub topic and subscription4.3.1 Set additional environment variables4.3.2 Execute pubsub config script4.3.4 Send a message to your Pub/Sub topic to test5. Tools (Optional)5.1. Clean up all entries on DataCatalog from the hive entrygroup5.2. Sample of Hive2Datacatalog Library usage6. Developer environment6.1. Install and run Yapf formatter6.2. Install and run Flake8 linter6.3. Run Tests7. Metrics8. Connector Architecture9. Troubleshooting1. InstallationInstall this library in avirtualenvusing pip.virtualenvis a tool to
create isolated Python environments. The basic problem it addresses is one of
dependencies and versions, and indirectly permissions.Withvirtualenv, it's possible to install this library without needing system
install permissions, and without clashing with the installed system
dependencies. Make sure you use Python 3.7+.1.1. Mac/Linuxpip3installvirtualenv
virtualenv--pythonpython3.7<your-env>source<your-env>/bin/activate
<your-env>/bin/pipinstallgoogle-datacatalog-hive-connector1.2. Windowspip3installvirtualenv
virtualenv--pythonpython3.7<your-env>
<your-env>\Scripts\activate
<your-env>\Scripts\pip.exeinstallgoogle-datacatalog-hive-connector1.3. Install from source1.3.1. Get the codegitclonehttps://github.com/GoogleCloudPlatform/datacatalog-connectors-hive/cddatacatalog-connectors-hive/google-datacatalog-hive-connector1.3.2. Create and activate avirtualenvpip3installvirtualenv
virtualenv--pythonpython3.7<your-env>source<your-env>/bin/activate2. Environment setup2.1. Auth credentials2.1.1. Create a service account and grant it below rolesData Catalog Admin2.1.2. Download a JSON key and save it as<YOUR-CREDENTIALS_FILES_FOLDER>/hive2dc-datacatalog-credentials.jsonPlease notice this folder and file will be required in next steps.2.2. Set environment variables to connect to your Hive MetastoreReplace below values according to your environment:exportGOOGLE_APPLICATION_CREDENTIALS=data_catalog_credentials_fileexportHIVE2DC_DATACATALOG_PROJECT_ID=google_cloud_project_idexportHIVE2DC_DATACATALOG_LOCATION_ID=us-google_cloud_location_idexportHIVE2DC_HIVE_METASTORE_DB_HOST=hive_metastore_db_serverexportHIVE2DC_HIVE_METASTORE_DB_USER=hive_metastore_db_userexportHIVE2DC_HIVE_METASTORE_DB_PASS=hive_metastore_db_passexportHIVE2DC_HIVE_METASTORE_DB_NAME=hive_metastore_db_nameexportHIVE2DC_HIVE_METASTORE_DB_TYPE=mysqlorpostgresMake sure you usemysqlonHIVE2DC_HIVE_METASTORE_DB_NAMEif you are connecting to a MySQL backed Hive Metastore orpostgresif it's a PostgreSQL backed Hive Metastore.3. Run entry point3.1. Run Python entry pointVirtualenvgoogle-datacatalog-hive-connector\--datacatalog-project-id=$HIVE2DC_DATACATALOG_PROJECT_ID\--datacatalog-location-id=$HIVE2DC_DATACATALOG_LOCATION_ID\--hive-metastore-db-host=$HIVE2DC_HIVE_METASTORE_DB_HOST\--hive-metastore-db-user=$HIVE2DC_HIVE_METASTORE_DB_USER\--hive-metastore-db-pass=$HIVE2DC_HIVE_METASTORE_DB_PASS\--hive-metastore-db-name=$HIVE2DC_HIVE_METASTORE_DB_NAME\--hive-metastore-db-type=$HIVE2DC_HIVE_METASTORE_DB_TYPE3.2. Run Docker entry pointIn case you have your Hive metastore DB running in your localhost environment, pass --network="host"dockerbuild-thive2datacatalog.
dockerrun--network="host"--rm--tty-vdata:/datahive2datacatalog--datacatalog-project-id=$HIVE2DC_DATACATALOG_PROJECT_ID--datacatalog-location-id=$HIVE2DC_DATACATALOG_LOCATION_ID--hive-metastore-db-host=$HIVE2DC_HIVE_METASTORE_DB_HOST--hive-metastore-db-user=$HIVE2DC_HIVE_METASTORE_DB_USER--hive-metastore-db-pass=$HIVE2DC_HIVE_METASTORE_DB_PASS--hive-metastore-db-name=$HIVE2DC_HIVE_METASTORE_DB_NAME--hive-metastore-db-type=$HIVE2DC_HIVE_METASTORE_DB_TYPE4. Deploy Message Event Consumer on Cloud Run (Optional)4.1. Set environment variables to deploy to Cloud RunReplace below values according to your environment:exportGOOGLE_APPLICATION_CREDENTIALS=data_catalog_credentials_fileexportHIVE2DC_DATACATALOG_PROJECT_ID=google_cloud_project_idexportHIVE2DC_DATACATALOG_LOCATION_ID=us-google_cloud_location_id4.2. Execute the deploy scriptsourcedeploy.shIf the deploy succeeded, you will be presented the Cloud Run endpoint, example:https://hive-sync-example-uc.a.run.appSave the endpoint which will be needed for the next step.4.3. Create your Pub/Sub topic and subscription4.3.1 Set additional environment variablesReplace with your Cloud Run endpoint:exportHIVE2DC_DATACATALOG_TOPIC_ID=google_cloud_topic_idexportHIVE2DC_DATACATALOG_APP_ENDPOINT=https://hive-sync-example-uc.a.run.app4.3.2 Execute pubsub config scriptsourcetools/create_pub_sub_run_invoker.sh4.3.4 Send a message to your Pub/Sub topic to testYou can look at valid message events examples on: tools/resources/*.json5. Tools (Optional)5.1. Clean up all entries on DataCatalog from the hive entrygrouprunpython tools/cleanup_datacatalog.py5.2. Sample of Hive2Datacatalog Library usagerunpython tools/hive2datacatalog_client_sample.py6. Developer environment6.1. Install and run Yapf formatterpipinstall--upgradeyapf# Auto update filesyapf--in-place--recursivesrctests# Show diffyapf--diff--recursivesrctests# Set up pre-commit hook# From the root of your git project.curl-opre-commit.shhttps://raw.githubusercontent.com/google/yapf/master/plugins/pre-commit.sh
chmoda+xpre-commit.sh
mvpre-commit.sh.git/hooks/pre-commit6.2. Install and run Flake8 linterpipinstall--upgradeflake8
flake8srctests6.3. Run Testspythonsetup.pytest7. MetricsMetrics README.md8. Connector ArchitectureArchitecture README.md9. TroubleshootingIn the case you receive the error:OSError: mysql_config not found

or

sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:MySQLSome system libraries or the MySQL driver was not found in the running machine, try installing it by running (On linux machines):sudo apt-get install libmysqlclient-dev python-devIn case the packagelibmysqlclient-devis not available, usedefault-libmysqlclient-dev:sudo apt-get install default-libmysqlclient-dev  python-devIn the case a connector execution hits Data Catalog quota limit, an error will be raised and logged with the following detailement, depending on the performed operation READ/WRITE/SEARCH:status = StatusCode.RESOURCE_EXHAUSTED
details = "Quota exceeded for quota metric 'Read requests' and limit 'Read requests per minute' of service 'datacatalog.googleapis.com' for consumer 'project_number:1111111111111'."
debug_error_string = 
"{"created":"@1587396969.506556000", "description":"Error received from peer ipv4:172.217.29.42:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Quota exceeded for quota metric 'Read requests' and limit 'Read requests per minute' of service 'datacatalog.googleapis.com' for consumer 'project_number:1111111111111'.","grpc_status":8}"For more info about Data Catalog quota, go to:Data Catalog quota docs.