Package: google-datacatalog-saphana-connector
google-datacatalog-saphana-connectorLibrary for ingesting SAP Hana Database metadata into Google Cloud Data Catalog.Disclaimer: This is not an officially supported Google product.Table of Contents1. Installation1.1. Mac/Linux1.2. Windows1.3. Install from source1.3.1. Get the code1.3.2. Create and activate avirtualenv1.3.3. Install the library2. Environment setup2.1. Auth credentials2.1.1. Create a service account and grant it below roles2.1.2. Download a JSON key and save it as2.2. Set environment variables3. Adapt user configurations4. Run entry point4.1. Run Python entry point4.2. Run Docker entry point5 Scripts inside tools5.1. Run clean up6. Developer environment6.1. Install and run Yapf formatter6.2. Install and run Flake8 linter6.3. Run Tests7. Metrics8. Troubleshooting1. InstallationInstall this library in avirtualenvusing pip.virtualenvis a tool to
create isolated Python environments. The basic problem it addresses is one of
dependencies and versions, and indirectly permissions.Withvirtualenv, it's possible to install this library without needing system
install permissions, and without clashing with the installed system
dependencies. Make sure you use Python 3.6+. This connector is tested in versions
3.6, 3.7, 3.8.1.1. Mac/Linuxpip3installvirtualenv
virtualenv--pythonpython3.7<your-env>source<your-env>/bin/activate
<your-env>/bin/pipinstallgoogle-datacatalog-saphana-connector1.2. Windowspip3installvirtualenv
virtualenv--pythonpython3.7<your-env>
<your-env>\Scripts\activate
<your-env>\Scripts\pip.exeinstallgoogle-datacatalog-saphana-connector1.3. Install from source1.3.1. Get the codegitclonehttps://github.com/GoogleCloudPlatform/datacatalog-connectors-rdbms/cddatacatalog-connectors-rdbms/google-datacatalog-saphana-connector1.3.2. Create and activate avirtualenvpip3installvirtualenv
virtualenv--pythonpython3.7<your-env>source<your-env>/bin/activate1.3.3. Install the librarypipinstall.2. Environment setup2.1. Auth credentials2.1.1. Create a service account and grant it below rolesData Catalog Admin2.1.2. Download a JSON key and save it as<YOUR-CREDENTIALS_FILES_FOLDER>/saphana2dc-credentials.jsonPlease notice this folder and file will be required in next steps.2.2. Set environment variablesReplace below values according to your environment:exportGOOGLE_APPLICATION_CREDENTIALS=data_catalog_credentials_fileexportSAPHANA2DC_DATACATALOG_PROJECT_ID=google_cloud_project_idexportSAPHANA2DC_DATACATALOG_LOCATION_ID=google_cloud_location_idexportSAPHANA2DC_SAPHANA_SERVER=saphana_serverexportSAPHANA2DC_SAPHANA_USERNAME=saphana_usernameexportSAPHANA2DC_SAPHANA_PASSWORD=saphana_passwordexportSAPHANA2DC_SAPHANA_DATABASE=saphana_databaseexportSAPHANA2DC_RAW_METADATA_CSV=saphana_raw_csv(IfsuppliedignorestheSAPHANAservercredentials)3. Adapt user configurationsAlong with default metadata, the connector can enrich metadata with user provided values as well,
such as adding a prefix to each schema and tables name.The table below shows what metadata is scraped by default, and what is configurable.MetadataDescriptionScraped by defaultConfig optionschema_nameName of the SchemaY---schema_ownerOwner of the SchemaY---schema_create_timeSchema creation timeY---table_nameName of a tableY---table_typeType of the table (ROW or COLUMN oriented)Y---has_primary_keyWhether a Table has a primary keyY---table_descriptionTable descriptionY---table_create_timeTable creation timeY---column_nameName of a columnY---column_typeColumn data typeY---column_nullableWhether a column is nullableY---column_maskWhether a column has data maskY---column_mask_expressionColumn mask expressionY---prefixPrefix to be added to schema and tables nameN/Aenrich_metadata.entry_prefixentry_id_pattern_for_prefixEntry ID pattern which the prefix will be appliedN/Aenrich_metadata.entry_id_pattern_for_prefixtable_size_mbSize of a table, in MBNsync_row_countstable_rowsNumber of rows in a tableNsync_row_countsprefixshould comply with Data CatalogentryId:The ID must begin with a letter or underscore, contain only English letters, numbers and underscores, and have at most 64 characters (combined the prefix + the entryId).if theentry_id_pattern_for_prefixis supplied, the prefix will only be applied to this pattern.Sample configuration fileingest_cfg.yamlin the repository root shows what kind of configuration is expected.If you want to enable the user defined config, please add ingest_cfg.yaml to the directory from which you execute the connector and adapt it to your needs.4. Run entry point4.1. Run Python entry pointVirtualenvgoogle-datacatalog-saphana-connector\--datacatalog-project-id=$SAPHANA2DC_DATACATALOG_PROJECT_ID\--datacatalog-location-id=$SAPHANA2DC_DATACATALOG_LOCATION_ID\--saphana-host=$SAPHANA2DC_SAPHANA_SERVER\--saphana-user=$SAPHANA2DC_SAPHANA_USERNAME\--saphana-pass=$SAPHANA2DC_SAPHANA_PASSWORD\--saphana-database=$SAPHANA2DC_SAPHANA_DATABASE\--raw-metadata-csv=$SAPHANA2DC_RAW_METADATA_CSV4.2. Run Docker entry pointdockerbuild-tsaphana2datacatalog.
dockerrun--rm--tty-vYOUR-CREDENTIALS_FILES_FOLDER:/datasaphana2datacatalog\--datacatalog-project-id=$SAPHANA2DC_DATACATALOG_PROJECT_ID\--datacatalog-location-id=$SAPHANA2DC_DATACATALOG_LOCATION_ID\--saphana-host=$SAPHANA2DC_SAPHANA_SERVER\--saphana-user=$SAPHANA2DC_SAPHANA_USERNAME\--saphana-pass=$SAPHANA2DC_SAPHANA_PASSWORD\--saphana-database=$SAPHANA2DC_SAPHANA_DATABASE\--raw-metadata-csv=$SAPHANA2DC_RAW_METADATA_CSV5 Scripts inside tools5.1. Data Catalog clean up# List of projects split by comma. Can be a single value without commaexportSAPHANA2DC_DATACATALOG_PROJECT_IDS=my-project-1,my-project-2# Run the clean uppythontools/cleanup_datacatalog.py--datacatalog-project-ids=$SAPHANA2DC_DATACATALOG_PROJECT_IDS6. Developer environment6.1. Install and run Yapf formatterpipinstall--upgradeyapf# Auto update filesyapf--in-place--recursivesrctests# Show diffyapf--diff--recursivesrctests# Set up pre-commit hook# From the root of your git project.curl-opre-commit.shhttps://raw.githubusercontent.com/google/yapf/master/plugins/pre-commit.sh
chmoda+xpre-commit.sh
mvpre-commit.sh.git/hooks/pre-commit6.2. Install and run Flake8 linterpipinstall--upgradeflake8
flake8srctests6.3. Run Testspythonsetup.pytest7. MetricsMetrics README.md8. TroubleshootingIn the case a connector execution hits Data Catalog quota limit, an error will be raised and logged with the following detailement, depending on the performed operation READ/WRITE/SEARCH:status = StatusCode.RESOURCE_EXHAUSTED
details = "Quota exceeded for quota metric 'Read requests' and limit 'Read requests per minute' of service 'datacatalog.googleapis.com' for consumer 'project_number:1111111111111'."
debug_error_string = 
"{"created":"@1587396969.506556000", "description":"Error received from peer ipv4:172.217.29.42:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Quota exceeded for quota metric 'Read requests' and limit 'Read requests per minute' of service 'datacatalog.googleapis.com' for consumer 'project_number:1111111111111'.","grpc_status":8}"For more info about Data Catalog quota, go to:Data Catalog quota docs.