Package: google-datacatalog-looker-connector
google-datacatalog-looker-connectorPackage for ingesting Looker metadata into Google Cloud Data Catalog, currently
supporting below asset types:FolderLookDashboardDashboard Element (aka Tile)QueryDisclaimer: This is not an officially supported Google product.Table of Contents1. Installation1.1. Mac/Linux1.2. Windows1.3. Install from source1.3.1. Get the code1.3.2. Create and activate avirtualenv1.3.3. Install the library2. Environment setup2.1. Auth credentials2.1.1. Create a GCP Service Account and grant it below roles2.1.2. Download a JSON key and save it as2.1.3. Create Looker API3 credentials2.1.4. Create a Looker configuration file2.2. Set environment variables3. Running the connector3.1. Python entry point3.2. Docker entry point4. Developer environment4.1. Install and run Yapf formatter4.2. Install and run Flake8 linter4.3. Run Tests4.4. Additional resources5. Troubleshooting5.1. Looker APIs compatibility5.2. Data Catalog quota1. InstallationInstall this library in avirtualenvusing pip.virtualenvis a tool
to create isolated Python environments. The basic problem it addresses is one
of dependencies and versions, and indirectly permissions.Withvirtualenv, it's possible to install this library without needing
system install permissions, and without clashing with the installed system
dependencies. Make sure you use Python3.7.1.1. Mac/Linuxpip3installvirtualenv
virtualenv--pythonpython3.7<your-env>source<your-env>/bin/activate
<your-env>/bin/pipinstallgoogle-datacatalog-looker-connector1.2. Windowspip3installvirtualenv
virtualenv--pythonpython3.7<your-env>
<your-env>\Scripts\activate
<your-env>\Scripts\pip.exeinstallgoogle-datacatalog-looker-connector1.3. Install from source1.3.1. Get the codegitclonehttps://github.com/GoogleCloudPlatform/datacatalog-connectors-bi/cddatacatalog-connectors-bi/google-datacatalog-looker-connector1.3.2. Create and activate avirtualenvpip3installvirtualenv
virtualenv--pythonpython3.7<your-env>source<your-env>/bin/activate1.3.3. Install the librarypipinstall.2. Environment setup2.1. Auth credentials2.1.1. Create a GCP Service Account and grant it below rolesData Catalog Admin2.1.2. Download a JSON key and save it as<YOUR-CREDENTIALS_FILES_FOLDER>/looker2dc-datacatalog-credentials.jsonPlease notice this folder and file will be required in next steps.2.1.3. Create Looker API3 credentialsThe credentials required for API access must be obtained by creating an
API3 key on a user account in the Looker Admin console. The API3 key consists
of a publicclient_idand a privateclient_secret.The shortcut for Looker Admin console is
https:///admin/users/api3_key/2.1.4. Create a Looker configuration fileFile content is described inLooker SDK documentation. Save the file as<YOUR-CREDENTIALS_FILES_FOLDER>/looker2dc-looker-credentials.iniPlease notice this folder and file will be required in next steps.2.2. Set environment variablesexportGOOGLE_APPLICATION_CREDENTIALS=datacatalog_credentials_fileReplace above values according to your environment. The Data Catalog
credentials file was saved instep
2.1.2.3. Running the connector3.1. Python entry pointVirtualenvgoogle-datacatalog-looker-connector\--datacatalog-project-id<YOUR-DATACATALOG-PROJECT-ID>\--looker-credentials-filelooker_credentials_ini_fileReplace above values according to your environment. The Looker credentials
file was saved instep 2.1.4.3.2. Docker entry pointdockerbuild--rm--taglooker2datacatalog.
dockerrun--rm--tty-v<YOUR-CREDENTIALS_FILES_FOLDER>:/data\looker2datacatalog\--datacatalog-project-id<YOUR-DATACATALOG-PROJECT-ID>\--looker-credentials-file/data/looker2dc-looker-credentials.ini4. Developer environment4.1. Install and run Yapf formatterpipinstall--upgradeyapf# Auto update filesyapf--in-place--recursivesrctests# Show diffyapf--diff--recursivesrctests# Set up pre-commit hook# From the root of your git project.curl-opre-commit.shhttps://raw.githubusercontent.com/google/yapf/master/plugins/pre-commit.sh
chmoda+xpre-commit.sh
mvpre-commit.sh.git/hooks/pre-commit4.2. Install and run Flake8 linterpipinstall--upgradeflake8
flake8srctests4.3. Run Testspythonsetup.pytest4.4. Additional resourcesPlease refer to theDeveloper Resources
documentation.5. Troubleshooting5.1. Looker APIs compatibilityThe connector may fail during the scrape stage if the Looker APIs do not return
metadata in the expected format. The code base uses theinit31looker_sdk client.As a reference, the below versions were already validated:VERSIONRESULTLooker API 3.1SUCCESS5.2. Data Catalog quotaIn the case a connector execution hits Data Catalog quota limit, an error will
be raised and logged with the following detailment, depending on the performed
operation READ/WRITE/SEARCH:status = StatusCode.RESOURCE_EXHAUSTED
details = "Quota exceeded for quota metric 'Read requests' and limit 'Read requests per minute' of service 'datacatalog.googleapis.com' for consumer 'project_number:1111111111111'."
debug_error_string = 
"{"created":"@1587396969.506556000", "description":"Error received from peer ipv4:172.217.29.42:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Quota exceeded for quota metric 'Read requests' and limit 'Read requests per minute' of service 'datacatalog.googleapis.com' for consumer 'project_number:1111111111111'.","grpc_status":8}"For more information on Data Catalog quota, please refer to:Data Catalog
quota docs.