Package: google-datacatalog-rdbms-connector
google-datacatalog-rdbms-connectorCommon resources for Data Catalog RDBMS connectors.Disclaimer: This is not an officially supported Google product.Table of Contents1. Installation1.1. Mac/Linux1.2. Windows2. Install from source2.1. Get the code2.2. Virtualenv2.2.1. Install Python 3.62.2.2. Create and activate avirtualenv2.2.3. Install3. Developer environment3.1. Install and run YAPF formatter3.2. Install and run Flake8 linter3.3. Install the package in editable mode (i.e. setuptools “develop mode”)3.4. Run the unit tests4. Setting up the RDBMS on a new connector4.1 Add support for optional queries4.2 Add support for SQL Objects1. InstallationInstall this library in avirtualenvusing pip.virtualenvis a tool to
create isolated Python environments. The basic problem it addresses is one of
dependencies and versions, and indirectly permissions.Withvirtualenv, it's possible to install this library without needing system
install permissions, and without clashing with the installed system
dependencies. Make sure you use Python 3.6+.1.1. Mac/Linuxpip3installvirtualenv
virtualenv--pythonpython3.6<your-env>source<your-env>/bin/activate
<your-env>/bin/pipinstallgoogle-datacatalog-rdbms-connector1.2. Windowspip3installvirtualenv
virtualenv--pythonpython3.6<your-env>
<your-env>\Scripts\activate
<your-env>\Scripts\pip.exeinstallgoogle-datacatalog-rdbms-connector2. Install from source2.1. Get the codegitclonehttps://github.com/GoogleCloudPlatform/datacatalog-connectors-rdbms/cddatacatalog-connectors-rdbms/google-datacatalog-rdbms-connector2.2. VirtualenvUsingvirtualenvis optional, but strongly recommended.2.2.1. Install Python 3.62.2.2. Create and activate avirtualenvpip3installvirtualenv
virtualenv--pythonpython3.6<your-env>source<your-env>/bin/activate2.2.3. Installpipinstall.3. Developer environment3.1. Install and run YAPF formatterpipinstall--upgradeyapf# Auto update filesyapf--in-place--recursivesrctests# Show diffyapf--diff--recursivesrctests# Set up pre-commit hook# From the root of your git project.curl-opre-commit.shhttps://raw.githubusercontent.com/google/yapf/master/plugins/pre-commit.sh
chmoda+xpre-commit.sh
mvpre-commit.sh.git/hooks/pre-commit3.2. Install and run Flake8 linterpipinstall--upgradeflake8
flake8srctests3.3. Install the package in editable mode (i.e. setuptools “develop mode”)pipinstall--editable.3.4. Run the unit testspythonsetup.pytest4. Setting up the RDBMS on a new connectorTo set up the RDBMS connector to work with a Relational Database 3 files are needed.metadata_definition.jsonmetadata_query.sqlExtending themetadata_scraperclass and implementing your rdbms connection method:_create_rdbms_connectionfor the metadata_definition file your have fields available for 3 levels:table_container_deftable_defcolumn_defIf you want working examples please take a look at the already implemented connectors for:
Oracle, Teradata, MySQL, PostgreSQL, Greenplum, Redshift and SQLServer.For themetadata_defitiontarget fields you have the following options astarget:LevelTargetDescriptionMandatorytable_container_defcreatorCreator of the Table Container.Ntable_container_defownerOwner of the Table Container.Ntable_container_defupdate_userLast user that updated the Table Container.Ntable_container_defdescTable Container Description.Ntable_defnum_rowsNumber of rows contained in the Table.Ntable_defcreatorCreator of the Table.Ntable_defownerOwner of the Table.Ntable_defupdate_userLast user that updated the Table.Ntable_defdescTable Description.Ntable_deftable_size_MBTable size, in MB.NIf those fields are configured they will be used to create Tags.For columns they are used to create the Data Catalog Entry schema, twotargetfields are used:LevelTargetDescriptionMandatorycolumn_deftypeColumn type.Ycolumn_defdescColumn description.N4.1 Add support for optional queriesYou can use user configuration file and execute optional queries to scrape additional metadata
from a database. You would need to do the following:Create additional SQL queries. At the moment, support for two optional queries are implemented
in this common RDBMS package:Statement for refreshing metadata (e.g. ANALYZE)Query to scrape number of rows in each tableExtend thequery_assemblerclass and implement a methods for getting optional queries:_get_refresh_statementand_get_path_to_num_rows_query.Implement_get_query_assemblerand_execute_refresh_queryon the extension of thematadata_scraperclass.You can see working examples of implementing optional queries in the connectors code for
PostgreSQL and MySQL. Please also refer to these samples to see how user configuration fileingest_cfg.yamlshould look like.4.2 Add support for SQL ObjectsYou can use the SQL Objects mechanism from the SQL connector to ingest SQL objects such as
Functions, Stored Procedures, Views, Materialized Views, and so on. This mechanism uses naming
convention to locate a SQL query and a metadata definition file.You can see working examples of implementing SQL Objects in the test sources.
To enable it you need 3 configuration files:ingest_cfg.yamlsql_objects:-name:'functions'enabled:True-name:'stored_procedures'enabled:TrueTheingest_cfg.yamlfile must be located at the connection execution directory.
You can find a sample in theSAP HANA connector.You can specify a list of SQL Objects with a flag to enable/disable the SQL Objects mechanism.
It will be only considered for objects whoseenabledflag is set toTrue.The relatedqueryandmetadata_definitionfiles are located by thesql_objects.namevalue.
From the above sample, thefunctionSQL Object requires two files:query_functions_sql_object.sqlandmetadata_definition_functions_sql_object.sql.And thestored_proceduresSQL Object requires:query_stored_procedures_sql_object.sqlandmetadata_definition_stored_procedures_sql_object.sql.query_functions_sql_object.sqlImplement a query file that matches the SQL Object name, with the following pattern:query_{name}_sql_object.sqlthis file must be located at the connectorconfigdirectory.
You can find a sample in theSAP HANA connector.metadata_definition_functions_sql_object.jsonImplement a metadata definition file that matches the SQL Object name, with the following
pattern:metadata_definition_{sql_objects.name}_sql_object.sqlthis file must be located at
the connectorconfigdirectory. You can find a sample in theSAP HANA connector.These are the required attributes formetadata_definitionfile:FieldDescriptionkeyKey used to lookup the config filestypeSQL Object typenameName of the SQL ObjectfieldsFields definition scraped from queryThefieldsattributes needs to implement thesourceandtargetdefinitions:FieldDescriptionfields.sourceName of the field scraped by the queryfields.targetObject with target field attributesfields.target.field_nameName of the field when translated to Data Catalog attributesfields.target.modelType of Data Catalog model, can be (tag or entry)fields.target.typeType of the field to be translated to Data Catalog attributesThe SQL Object will be skipped if any of the 3 configuration files are missing.