Package: google-datacatalog-vertica-connector
google-datacatalog-vertica-connectorPackage for ingesting Vertica metadata into Google Cloud Data Catalog.Disclaimer: This is not an officially supported Google product.Table of Contents1. Installation1.1. Mac/Linux1.2. Windows1.3. Install from source1.3.1. Get the code1.3.2. Create and activate avirtualenv1.3.3. Install the library2. Environment setup2.1. Auth credentials2.1.1. Create a service account and grant it below roles2.1.2. Download a JSON key and save it as2.2. Set environment variables3. Run entry point3.1. Run Python entry point3.2. Run Docker entry point4. Developer environment4.1. Install and run Yapf formatter4.2. Install and run Flake8 linter4.3. Run Tests5. Troubleshooting1. InstallationInstall this library in avirtualenvusing pip.virtualenvis a tool to
create isolated Python environments. The basic problem it addresses is one of
dependencies and versions, and indirectly permissions.Withvirtualenv, it's possible to install this library without needing system
install permissions, and without clashing with the installed system
dependencies. Make sure you use Python 3.6+.1.1. Mac/Linuxpip3installvirtualenv
virtualenv--pythonpython3.6<your-env>source<your-env>/bin/activate
<your-env>/bin/pipinstallgoogle-datacatalog-vertica-connector1.2. Windowspip3installvirtualenv
virtualenv--pythonpython3.6<your-env>
<your-env>\Scripts\activate
<your-env>\Scripts\pip.exeinstallgoogle-datacatalog-vertica-connector1.3. Install from source1.3.1. Get the codegitclonehttps://github.com/GoogleCloudPlatform/datacatalog-connectors-rdbms/cddatacatalog-connectors-rdbms/google-datacatalog-vertica-connector1.3.2. Create and activate avirtualenvpip3installvirtualenv
virtualenv--pythonpython3.6<your-env>source<your-env>/bin/activate1.3.3. Install the librarypipinstall.2. Environment setup2.1. Auth credentials2.1.1. Create a service account and grant it below rolesData Catalog Admin2.1.2. Download a JSON key and save it as<YOUR-CREDENTIALS_FILES_FOLDER>/vertica2dc-datacatalog-credentials.jsonPlease notice this folder and file will be required in next steps.2.2. Set environment variablesReplace below values according to your environment:exportGOOGLE_APPLICATION_CREDENTIALS=data_catalog_credentials_file3. Run entry point3.1. Run Python entry pointVirtualenvConnect to a server:vertica2datacatalog\--vertica-host<VERTICA-HOST-IP>\--vertica-user<VERTICA-USER>\--vertica-pass<VERTICA-PASSWORD>\--datacatalog-project-id<YOUR-DATACATALOG-PROJECT-ID>\--datacatalog-location-id<YOUR-DATACATALOG-LOCATION-ID>Read metadata from a CSV file:vertica2datacatalog\--raw-metadata-csv<PATH-TO-A-FULL-DUMP-CSV-FILE>\--datacatalog-project-id<YOUR-DATACATALOG-PROJECT-ID>\--datacatalog-location-id<YOUR-DATACATALOG-LOCATION-ID>3.2. Run Docker entry pointdockerbuild--rm--tagvertica2datacatalog.Connect to a server:dockerrun--rm--tty-v<YOUR-CREDENTIALS_FILES_FOLDER>:/data\vertica2datacatalog\--vertica-host<VERTICA-HOST-IP>\--vertica-user<VERTICA-USER>\--vertica-pass<VERTICA-PASSWORD>\--datacatalog-project-id<YOUR-DATACATALOG-PROJECT-ID>\--datacatalog-location-id<YOUR-DATACATALOG-LOCATION-ID>Read metadata from a CSV file:dockerrun--rm--tty-v<YOUR-CREDENTIALS_FILES_FOLDER>:/data\vertica2datacatalog\--raw-metadata-csv/data/<PATH-TO-A-FULL-DUMP-CSV-FILE>\--datacatalog-project-id<YOUR-DATACATALOG-PROJECT-ID>\--datacatalog-location-id<YOUR-DATACATALOG-LOCATION-ID>4. Developer environment4.1. Install and run Yapf formatterpipinstall--upgradeyapf# Auto update filesyapf--in-place--recursivesrctests# Show diffyapf--diff--recursivesrctests# Set up pre-commit hook# From the root of your git project.curl-opre-commit.shhttps://raw.githubusercontent.com/google/yapf/master/plugins/pre-commit.sh
chmoda+xpre-commit.sh
mvpre-commit.sh.git/hooks/pre-commit4.2. Install and run Flake8 linterpipinstall--upgradeflake8
flake8srctests4.3. Run Testspythonsetup.pytest5. TroubleshootingIn the case a connector execution hits Data Catalog quota limit, an error will be raised and logged with the following detailement, depending on the performed operation READ/WRITE/SEARCH:status = StatusCode.RESOURCE_EXHAUSTED
details = "Quota exceeded for quota metric 'Read requests' and limit 'Read requests per minute' of service 'datacatalog.googleapis.com' for consumer 'project_number:1111111111111'."
debug_error_string = 
"{"created":"@1587396969.506556000", "description":"Error received from peer ipv4:172.217.29.42:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Quota exceeded for quota metric 'Read requests' and limit 'Read requests per minute' of service 'datacatalog.googleapis.com' for consumer 'project_number:1111111111111'.","grpc_status":8}"For more info about Data Catalog quota, go to:Data Catalog quota docs.