Package: informatica-edc-rest-api-samples
Informatica EDC REST-API samples using pythonContains examples for connecting to and generating attribute level lineage in EDC via python.Requirementspython 3.6+Python editors (ide/environments)VS Code - good support for python - free an runs on all platformshttps://code.visualstudio.com/pycharm -https://www.jetbrains.com/pycharm/anaconda - for JupterLab/Notebookshttps://www.anaconda.com/(includes vscode)Eclipse - ide for java/python (python using/installing pydev)Download:http://www.eclipse.org/downloads/eclipse-packages/other useful toolsrest api clients - for testing syntax/api calls + the good ones generate code for many languagespostman -https://www.getpostman.com/insomnia -https://insomnia.rest/Getting Startedverify that python is installed - v3.6+Create a new VSCode/pycharm/Eclipse Project and import/use the files in the python folder (not the java folder)Ensure EDC is running while executing the samples - try/except code will catch & immediately exit
The property suppress_edc_calls can be set in the config.json to bypass any calls to EDCUse a virtual environment with pythonpython3 -m venv venvand thensource venv/bin/activate (for linux/macos)venv/Scripts/activate.ps1 (for windows powershell)
Note:  you may need to executeSet-ExecutionPolicy unrestrictedfor powershell (run powershell as administrator to do this)venv/Scripts/activate.bat (windows cmd)after activatingexecute the following to get the latest version for test purposes
pip3 install --extra-index-urlhttps://test.pypi.org/simple/informatica-edc-rest-api-samplesexecute the following to get a tested version (see coverage overview in htmlcoverage):
pip3 install informatica-edc-rest-api-samplesRun the code (remember to have an activated venv):
python3 run_edc_lineage.pREST API Authenticationthe EDC rest api supports Basic Authentication only - seehttps://yourcatalogserver:port/accessfor detailswe use the python requests module for all http(s) rest calls (very easy to use)when making a rest api call - you can pass either the id/password - or a http header with an encoded passwordfor all examples here, we initially used the id/pwd method - but have switched to use http headersif you are using LDAP authentication - the user must have the security domain and a '' character prefixed to the user ide.g.COMPANY_LDAP\user_ause the encodeUser.py script - to create the basic auth encoding for your user, and store in a variable namedINFA_EDC_AUTHyou can set the variable for each session, so it is not stored anywhereif using docker - you can add this variable to an .env file to pass to docker at runtimeif using VS Code - you can add and "env" setting for individual environment variables used in the debugger (launch.json)e.g. "env" : {"INFA_EDC_AUTH" : "Basic dXNlcjE6YUNvbXBsIWNAdGVkUGEkM3cwcmQ="}, (works but prefer .env file)Note:  any files inside of .vscode (e.g. launch.json) will be excluded from the git repo (each user has their own local version)In this fork setupConnection.py and the functionality to use an .env has been removedHTTPS/TLS/SSL Connections and certificatesassuming your catalog service is https enabled (it should be, if not so your passwords are send in clear text & set verify=False)you will either need to download/copy the certificate (.pem format, not .jks) locallyor set flags to disable certificate authentication (not recommended, but possible)if your ssl certificate is self signed (also not recommended), an additional warning will need to be suppressedmore information about SSL authentication can be foundhttps://3.python-requests.org/user/advanced/#ssl-cert-verificationSample Programs in the ProjectThe following samples may no longer work in this fork of the original project. If you want to use them, please use a clone of the original project, not this fork.encodeUser.py: simple program to prompt for a userid/pwd and optionally a security domain and create a base64 encoded string that can be used for authentication in the http header.  e.g."Basic dXNlcjE6YUNvbXBsIWNAdGVkUGEkM3cwcmQ="use this script before you call use the other scripts, to get the right format for authenticating & not storing passwords in the .py filesan alternate is to prompt for a password within your script & encode the id:passworduseencodeUser27.pyfor legacy pythonEDCQuery_template.py:  a template/skeleton that shows how to connect to the catalog and execute a search using python.  the result-set processing includes handling the paging model.  It also uses theget_fact_valuemethod inedcutils.pyto extract the item name from the facts arrayUtility/Heloer Scriptsedcutils.py:  utility/helper methods for common tasks - like get an attribute valueget_fact_value(item, attrName)edcSessionHelper.py: EDCSession class helps you configure a requests.session object and also provides command-line args for connecting to the catalog (-c/-edcurl EDC URL, -a/--auth auth credentials (see encodeUser.py), -u username (will prompt for pwd - recommend using -a, -s/--sslcert SSLCERT).this class also supports using the following environment vars:INFA_EDC_URL - e.g.http://yourcatalogserver:9085orhttps://yourcatalogserver:9085INFA_EDC_AUTH - e.g. "Basic dXNlcl9hOnJlYWxseXNlY3VyZXBhc3N3b3Jk" - seeencodeUser.pyINFA_EDC_SSL_PEM - certificate to use to connect (or set to None - to disable ssl verfication)for an example of usage - seelistAndCountCustomAttributes.pylistAndCountCustomAttributes.py: find all custom attributes (normal and classification) and count the # of times the attribute is used.  writes results to csv file (output folder can be configured)supports command-line parameters and environment vars for accessing the catalog.uses edcSessionHelper.py to get a session reference to any rest querieslistCustomAttributes.py: simple script to print all custom attributes (name, id, type, sortable, facetable)this script will list both regular custom attributes/2/catalog/models/attributesand reference 'classification' attributes/2/catalog/models/referenceAttributessimilarityReport.py: v10.2.1+ utility to find & export all columns/fields with similar linksnote:  this script will attempt to query all dataelements, even if similarity profiling was not run.  for a better implementation, usesimilarityByResource.pysimilarityByResource.py: utility to find and export column similarity for all resources that similarity profiling was configured.supports command-line parameters and environment vars for accessing the catalog.uses edcSessionHelper.py to get a session reference to any rest queriesdbSchemaReplicationLineage.py: provides the ability to link tables/columns in a database schema that are replicated to other schemas/databases & no scanner exists to automatcially document these relationships.  (e.g. sqoop, scripts/code, goldengate ...)seedbSchemaReplicationLineage.mdfor moreexternalDBLinker.py: script to generate custom lineage for any tables/columns created within an ExternalDatabase/ExternalSchema (often happens with Oracle (dblink) and SQLServer databases (references to databases in views)seeexternalDBLinker.mdfor moredomainSummary.py- queries the catalog to find all instances where data domains are used & counts the # of All, Accepted, Inferred, Rejected for all resources and per resourceoutput is an excel workbook (domain_summary.xlxs) with a worksheet for counts across all resources, and a worksheet per resource with individual counts per resource.  optional output to .csv files (per resource) is also possiblesupports command-line parameters and environment vars for accessing the catalog.uses edcSessionHelper.py to get a session reference to any rest queriesxdocAnalyzer.py- use this script to download xdocs for a resource and analyze the contents (counts # of objects by type, # of attributes) and will analyze all links + connection assignments.  can be useful for troubleshooting (especially for resources that do not yet support reference objects)supports command-line parameters and environment vars for accessing the catalog.uses edcSessionHelper.py to get a session reference to any rest queriessetParentFilterValues.py- use this script to update in bulk relational objects with a custom attribute containing the value of the schema the object belongs to. This will faceting by schema name in search results, as well as creating custom tab pointing to specific database schema within a resource, seesetParentFilterValues.mdfor more info