Package: google-datacatalog-qlik-connector
google-datacatalog-qlik-connectorPackage for ingesting Qlik Sense metadata into Google Cloud Data Catalog,
currently supporting below asset types:Custom Property DefinitionStreamApp (only the published ones)Master Items: DimensionMaster Items: MeasureMaster Items: VisualizationSheet (only the published ones)Disclaimer: This is not an officially supported Google product.Table of Contents1. Installation1.1. Mac/Linux1.2. Windows1.3. Install from source1.3.1. Get the code1.3.2. Create and activate avirtualenv1.3.3. Install the library2. Environment setup2.1. Auth credentials2.1.1. Create a GCP Service Account and grant it below roles2.1.2. Download a JSON key and save it as2.2. Set environment variables3. Running the connector3.1. Python entry point3.2. Docker entry point4. Design decisions4.1. Tag Templates for Custom Property Choice Values5. Developer environment5.1. Install and run Yapf formatter5.2. Install and run Flake8 linter5.3. Run Tests5.4. Additional resources6. Troubleshooting6.1. Qlik APIs compatibility6.2. Data Catalog quota1. InstallationInstall this library in avirtualenvusing pip.virtualenvis a tool
to create isolated Python environments. The basic problem it addresses is one
of dependencies and versions, and indirectly permissions.Withvirtualenv, it's possible to install this library without needing
system install permissions, and without clashing with the installed system
dependencies. Make sure you use Python 3.6+.1.1. Mac/Linuxpip3installvirtualenv
virtualenv--pythonpython3.6<your-env>source<your-env>/bin/activate
<your-env>/bin/pipinstallgoogle-datacatalog-qlik-connector1.2. Windowspip3installvirtualenv
virtualenv--pythonpython3.6<your-env>
<your-env>\Scripts\activate
<your-env>\Scripts\pip.exeinstallgoogle-datacatalog-qlik-connector1.3. Install from source1.3.1. Get the codegitclonehttps://github.com/GoogleCloudPlatform/datacatalog-connectors-bi/cddatacatalog-connectors-bi/google-datacatalog-qlik-connector1.3.2. Create and activate avirtualenvpip3installvirtualenv
virtualenv--pythonpython3.6<your-env>source<your-env>/bin/activate1.3.3. Install the librarypipinstall.2. Environment setup2.1. Auth credentials2.1.1. Create a GCP Service Account and grant it below rolesData Catalog Admin2.1.2. Download a JSON key and save it as<YOUR-CREDENTIALS_FILES_FOLDER>/qlik2dc-datacatalog-credentials.jsonPlease notice this folder and file will be required in next steps.2.2. Set environment variablesThe connector uses Windows-based NTLM authentication, which requires the
username to be provided in the format of<ẃindows-ad-domain>\<username>. When
fulfilling the below environment variables, setQLIK2DC_QLIK_AD_DOMAINwith
the Windows Active Directory domain your user belongs to andQLIK2DC_QLIK_USERNAMEwith the username (no backslash in both).exportGOOGLE_APPLICATION_CREDENTIALS=datacatalog_credentials_fileexportQLIK2DC_QLIK_SERVER=qlik_serverexportQLIK2DC_QLIK_AD_DOMAIN=qlik_ad_domainexportQLIK2DC_QLIK_USERNAME=qlik_usernameexportQLIK2DC_QLIK_PASSWORD=qlik_passwordexportQLIK2DC_DATACATALOG_PROJECT_ID=google_cloud_project_idexportQLIK2DC_DATACATALOG_LOCATION_ID=google_cloud_location_idReplace above values according to your environment. The Data Catalog
credentials file was saved instep
2.1.2.3. Running the connectorThe--qlik-ad-domainargument is optional and defaults to..The--datacatalog-location-idargument is optional and defaults tous.3.1. Python entry pointVirtualenvgoogle-datacatalog-qlik-connector\--qlik-server$QLIK2DC_QLIK_SERVER\[--qlik-ad-domain$QLIK2DC_QLIK_AD_DOMAIN\]--qlik-username$QLIK2DC_QLIK_USERNAME\--qlik-password$QLIK2DC_QLIK_PASSWORD\--datacatalog-project-id$QLIK2DC_DATACATALOG_PROJECT_ID\[--datacatalog-location-id$QLIK2DC_DATACATALOG_LOCATION_ID]3.2. Docker entry pointdockerbuild--rm--tagqlik2datacatalog.
dockerrun--rm--tty-vYOUR-CREDENTIALS_FILES_FOLDER:/data\qlik2datacatalog\--qlik-server$QLIK2DC_QLIK_SERVER\[--qlik-ad-domain$QLIK2DC_QLIK_AD_DOMAIN\]--qlik-username$QLIK2DC_QLIK_USERNAME\--qlik-password$QLIK2DC_QLIK_PASSWORD\--datacatalog-project-id$QLIK2DC_DATACATALOG_PROJECT_ID\[--datacatalog-location-id$QLIK2DC_DATACATALOG_LOCATION_ID]4. Design decisions4.1. Tag Templates for Custom Property Choice ValuesThe current implementation creates a Tag Template for each Custom Property
Choice Value assigned to Streams or Apps in the provided Qlik Sense site. The
rationale behind this decision comprises allowing the connector to synchronize
all metadata scraped from each Custom Property, at the same time it enables
Qlik assets to be easily found by their custom properties — using query strings
such astag:property_name:"<PROPERTY-NAME>"andtag:value:"<SOME-VALUE>".Data Catalog accepts attaching only one Tag per Template to a given Entry, so
there could be metadata loss if the Tag Templates were created in different
ways, e.g. on a Custom Property Definition basis.Lastly, this approach may lead to the creation of several Tag Templates if
there are many Custom Property Values in use in your Qlik Sense site. In case
you would like to suggest a different approach to tackle this problem, pleasefile a feature request. We will be happy to discuss alternative solutions!5. Developer environment5.1. Install and run Yapf formatterpipinstall--upgradeyapf# Auto update filesyapf--in-place--recursivesrctests# Show diffyapf--diff--recursivesrctests# Set up pre-commit hook# From the root of your git project.curl-opre-commit.shhttps://raw.githubusercontent.com/google/yapf/master/plugins/pre-commit.sh
chmoda+xpre-commit.sh
mvpre-commit.sh.git/hooks/pre-commit5.2. Install and run Flake8 linterpipinstall--upgradeflake8
flake8srctests5.3. Run Testspythonsetup.pytest5.4. Additional resourcesPlease refer to theDeveloper Resources
documentation.6. Troubleshooting6.1. Qlik APIs compatibilityThe connector may fail during the scrape stage if the Qlik APIs do not return
metadata in the expected format. As a reference, the below versions were
already validated:Qlik Sense Repository Service APIVERSIONRESULT34.16.0 (September2020)SUCCESSQlik Engine JSON APIVERSIONRESULT12.763.4 (September2020)SUCCESS6.2. Data Catalog quotaIn case a connector execution hits Data Catalog quota limit, an error will be
raised and logged with the following details, depending on the performed
operation (READ/WRITE/SEARCH):status = StatusCode.RESOURCE_EXHAUSTED
details = "Quota exceeded for quota metric 'Read requests' and limit 'Read requests per minute' of service 'datacatalog.googleapis.com' for consumer 'project_number:1111111111111'."
debug_error_string = 
"{"created":"@1587396969.506556000", "description":"Error received from peer ipv4:172.217.29.42:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Quota exceeded for quota metric 'Read requests' and limit 'Read requests per minute' of service 'datacatalog.googleapis.com' for consumer 'project_number:1111111111111'.","grpc_status":8}"For more information on Data Catalog quota, please refer to:Data Catalog
quota docs.