Package: google-datacatalog-sisense-connector
google-datacatalog-sisense-connectorPackage for ingestingSisensemetadata into Google
Cloud Data Catalog, currently supporting the below assets:FolderDashboardWidgetThis sample connector creates Data Catalog tags to enable a data lineage
mechanism that allows users to search the catalog to find where/which
ElastiCube Table fields are used in Widgets or Dashboards. To do so, it
currently processes JAQL query metadata from:Dashboard filtersWidgets fields and filtersNestedformulasNestedfilter.byproperties, used bytop/bottom filtersDisclaimer: This is not an officially supported Google product.This connector is a work in progress!Table of Contents1. Installation1.1. Mac/Linux1.2. Windows1.3. Install from source1.3.1. Get the code1.3.2. Create and activate avirtualenv1.3.3. Install the library2. Environment setup2.1. Auth credentials2.1.1. Create a GCP Service Account and grant it below roles2.1.2. Download a JSON key and save it as2.2. Set environment variables3. Running the connector3.1. sync-catalog3.1.1. Python entry point3.1.2. Docker entry point3.2. find-elasticube-deps3.2.1. Python entry point3.2.2. Docker entry point3.3. list-elasticube-deps3.3.1. Python entry point3.3.2. Docker entry point4. Developer environment4.1. Install and run the YAPF formatter4.2. Install and run the Flake8 linter4.3. Run Tests4.4. Additional resources5. Templates, Tags, and Data Lineage6. Troubleshooting6.1. Sisense APIs compatibility6.2. Data Catalog quota1. InstallationInstall this library in avirtualenvusing pip.Virtualenvis a tool
to create isolated Python environments. The basic problem it addresses is one
of dependencies and versions, and indirectly permissions.Withvirtualenv, it's possible to install this library without needing
system install permissions, and without clashing with the installed system
dependencies. Make sure you use Python 3.6+.1.1. Mac/Linuxpip3installvirtualenv
virtualenv--pythonpython3.6<your-env>source<your-env>/bin/activate
<your-env>/bin/pipinstallgoogle-datacatalog-sisense-connector1.2. Windowspip3installvirtualenv
virtualenv--pythonpython3.6<your-env>
<your-env>\Scripts\activate
<your-env>\Scripts\pip.exeinstallgoogle-datacatalog-sisense-connector1.3. Install from source1.3.1. Get the codegitclonehttps://github.com/GoogleCloudPlatform/datacatalog-connectors-bi/cddatacatalog-connectors-bi/google-datacatalog-sisense-connector1.3.2. Create and activate avirtualenvpip3installvirtualenv
virtualenv--pythonpython3.6<your-env>source<your-env>/bin/activate1.3.3. Install the librarypipinstall.2. Environment setup2.1. Auth credentials2.1.1. Create a GCP Service Account and grant it below rolesData Catalog Admin2.1.2. Download a JSON key and save it as<YOUR-CREDENTIALS_FILES_FOLDER>/sisense2dc-datacatalog-credentials.jsonPlease notice this folder and file will be required in next steps.2.2. Set environment variablesReplace below values according to your environment:exportGOOGLE_APPLICATION_CREDENTIALS=datacatalog_credentials_fileexportSISENSE2DC_SISENSE_SERVER=sisense_serverexportSISENSE2DC_SISENSE_USERNAME=sisense_usernameexportSISENSE2DC_SISENSE_PASSWORD=sisense_passwordexportSISENSE2DC_DATACATALOG_PROJECT_ID=google_cloud_project_idexportSISENSE2DC_DATACATALOG_LOCATION_ID=google_cloud_location_idReplace above values according to your environment. The Data Catalog
credentials file was saved instep
2.1.2.3. Running the connector3.1. sync-catalogSynchronizes Google Data Catalog with a given Sisense server.The--datacatalog-location-idargument is optional and defaults tous.3.1.1. Python entry pointVirtualenvgoogle-datacatalog-sisense-connectorsync-catalog\--sisense-server$SISENSE2DC_SISENSE_SERVER\--sisense-username$SISENSE2DC_SISENSE_USERNAME\--sisense-password$SISENSE2DC_SISENSE_PASSWORD\--datacatalog-project-id$SISENSE2DC_DATACATALOG_PROJECT_ID\[--datacatalog-location-id$SISENSE2DC_DATACATALOG_LOCATION_ID]3.1.2. Docker entry pointdockerbuild--rm--tagsisense2datacatalog.
dockerrun--rm--tty-vYOUR-CREDENTIALS_FILES_FOLDER:/data\sisense2datacatalogsync-catalog\--sisense-server$SISENSE2DC_SISENSE_SERVER\--sisense-username$SISENSE2DC_SISENSE_USERNAME\--sisense-password$SISENSE2DC_SISENSE_PASSWORD\--datacatalog-project-id$SISENSE2DC_DATACATALOG_PROJECT_ID\[--datacatalog-location-id$SISENSE2DC_DATACATALOG_LOCATION_ID]3.2. find-elasticube-depsFinds ElastiCube dependencies through catalog search and prints them in the
console.3.2.1. Python entry pointVirtualenvgoogle-datacatalog-sisense-connectorfind-elasticube-deps\--datasource<datasource>\--table<table>\--column<column>\--datacatalog-project-id$SISENSE2DC_DATACATALOG_PROJECT_ID3.2.2. Docker entry pointdockerbuild--rm--tagsisense2datacatalog.
dockerrun--rm--tty-vYOUR-CREDENTIALS_FILES_FOLDER:/data\sisense2datacatalogfind-elasticube-deps\--datasource<datasource>\--table<table>\--column<column>\--datacatalog-project-id$SISENSE2DC_DATACATALOG_PROJECT_ID3.3. list-elasticube-depsLists ElastiCube dependencies for a given Sisense Dashboard or Widget and
prints them in the console.3.3.1. Python entry pointVirtualenvgoogle-datacatalog-sisense-connectorlist-elasticube-deps\--asset-url<asset-url>\--datacatalog-project-id$SISENSE2DC_DATACATALOG_PROJECT_ID3.3.2. Docker entry pointdockerbuild--rm--tagsisense2datacatalog.
dockerrun--rm--tty-vYOUR-CREDENTIALS_FILES_FOLDER:/data\sisense2datacataloglist-elasticube-deps\--asset-url<asset-url>\--datacatalog-project-id$SISENSE2DC_DATACATALOG_PROJECT_ID4. Developer environment4.1. Install and run the YAPF formatterpipinstall--upgradeyapf# Auto update filesyapf--in-place--recursivesrctests# Show diffyapf--diff--recursivesrctests# Set up pre-commit hook# From the root of your git project.curl-opre-commit.shhttps://raw.githubusercontent.com/google/yapf/master/plugins/pre-commit.sh
chmoda+xpre-commit.sh
mvpre-commit.sh.git/hooks/pre-commit4.2. Install and run the Flake8 linterpipinstall--upgradeflake8
flake8srctests4.3. Run Testspythonsetup.pytest4.4. Additional resourcesPlease refer to theDeveloper Resources
documentation.5. Templates, Tags, and Data LineageThe Data Catalog Tag Templates created by this connector and their usage
scenarios are described below:TAG TEMPLATEFIELDSUSAGEFolder Metadata (sisense_folder_metadata)IdOwner usernameOwner nameId of ParentParent FolderData Catalog Entry for the parent FolderHas childrenChild countHas dashboardsDashboard countSisense Server UrlStore additional metadata for Folder-related Entries.Dashboard Metadata (sisense_dashboard_metadata)IdOwner usernameOwner nameFolder IdFolder NameData Catalog Entry for the FolderData SourceTime it was last publishedTime it was last openedSisense Server UrlStore additional metadata for Dashboard-related Entries.Widget Metadata (sisense_widget_metadata)IdTypeSubtypeOwner usernameOwner nameDashboard IdDashboard TitleData Catalog Entry for the DashboardData SourceSisense Server UrlStore additional metadata for Widget-related Entries.JAQL Metadata (sisense_jaql_metadata)TableColumnDimensionFormulaAggregationSisense Server UrlStore JAQL metadata for ElasticCube-dependent entities such as Dashboard filters, Widget fields and filters.Please notice the connector creates Data Catalog Tags for most of the Dashboard
and Widget properties that depend on JAQL queries, e.g., fields, filters,
nested formulas, and top/bottom filters. Such tags, created from theJAQL
Metadatatemplate, are quite simple: ~4 fields each. The connector uses lots
of them to enable column-level lineage tracking for a given Sisense server.6. Troubleshooting6.1. Sisense APIs compatibilityThe connector may fail during the scrape stage if the Sisense API do not return
metadata in the expected format. As a reference, the below versions were
already validated:Sisense REST API v1.0VERSIONRESULTWindows 8.2.5SUCCESS6.2. Data Catalog quotaIn case a connector execution hits Data Catalog quota limit, an error will be
raised and logged with the following details, depending on the performed
operation (READ/WRITE/SEARCH):status = StatusCode.RESOURCE_EXHAUSTED
details = "Quota exceeded for quota metric 'Read requests' and limit 'Read requests per minute' of service 'datacatalog.googleapis.com' for consumer 'project_number:1111111111111'."
debug_error_string = 
"{"created":"@1587396969.506556000", "description":"Error received from peer ipv4:172.217.29.42:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Quota exceeded for quota metric 'Read requests' and limit 'Read requests per minute' of service 'datacatalog.googleapis.com' for consumer 'project_number:1111111111111'.","grpc_status":8}"For more information on Data Catalog quota, please refer to theproduct
documentation.