Package: datacatalog-custom-entries-manager
datacatalog-custom-entries-managerA Python package intended to manage Google CloudData Catalog custom entries, loading metadata
from external sources. Currently supports the CSV and JSON file formats.It is built on top ofGoogleCloudPlatform/datacatalog-connectorsand, differently from the
existing connectors, allows ingesting metadata with no need to connect to other systems than Data
Catalog. Known use cases include validating Custom Entries ingestion workloads before coding their
specific features and loading metadata into development / PoC environments.In case you need not only Entries but also Tags to validate your model/workload, consider givingdatacatalog-custom-model-managera try.Table of Contents1. Environment setup1.1. Python + virtualenv1.1.1. Install Python 3.6+1.1.2. Create a folder1.1.3. Create and activate an isolated Python environment1.1.4. Install the package1.2. Docker1.2.1. Get the source code1.3. Auth credentials1.3.1. Create a service account and grant it below roles1.3.2. Download a JSON key and save it as1.3.3. Set the environment variables2. Manage Custom Entries2.1. Synchronize2.1.1. To a CSV file2.1.2. To a JSON file1. Environment setup1.1. Python + virtualenvUsingvirtualenvis optional, but strongly recommended unless you useDocker.1.1.1. Install Python 3.6+1.1.2. Create a folderThis is recommended so all related stuff will reside at the same place, making it easier to follow
below instructions.mkdir./datacatalog-custom-entries-managercd./datacatalog-custom-entries-managerAll paths starting with./in the next steps are relative to thedatacatalog-custom-entries-managerfolder.1.1.3. Create and activate an isolated Python environmentpipinstall--upgradevirtualenv
python3-mvirtualenv--pythonpython3envsource./env/bin/activate1.1.4. Install the packagepipinstall--upgradedatacatalog-custom-entries-manager1.2. DockerDocker may be used as an alternative to rundatacatalog-custom-entries-manager. In this case,
please disregard theabovevirtualenvsetup instructions.1.2.1. Get the source codegitclonehttps://github.com/ricardolsmendes/datacatalog-custom-entries-managercd./datacatalog-custom-entries-manager1.3. Auth credentials1.3.1. Create a service account and grant it below rolesDataCatalog entryGroup OwnerDataCatalog entry OwnerData Catalog Viewer1.3.2. Download a JSON key and save it as./credentials/datacatalog-custom-entries-manager.json1.3.3. Set the environment variablesThis step can be skipped if you're usingDocker.exportGOOGLE_APPLICATION_CREDENTIALS=./credentials/datacatalog-custom-entries-manager.json2. Manage Custom Entries2.1. Synchronize2.1.1. To a CSV fileSCHEMAThe metadata schema to synchronize Custom Entries is presented below. Use as many lines as needed
to describe all Data Catalog Entries you need.ColumnDescriptionMandatoryuser_specified_systemIndicates the Entry source systemyesgroup_idId of the Entry Group the Entry belongs toyeslinked_resourceThe resource a metadata Entry refers toyesdisplay_nameDisplay information such as title and description; a short name to identify the Entry (theentry_idfield will be generated as a normalized version of the display name)yesdescriptionCan consist of several sentences that describe the Entry contentsnouser_specified_typeA custom value indicating the Entry typeyescreated_atThe creation time of the underlying resource, not of the Data Catalog Entry (format: YYYY-MM-DDTHH:MM:SSZ)noupdated_atThe last-modified time of the underlying resource, not of the Data Catalog Entry (format: YYYY-MM-DDTHH:MM:SSZ)noSAMPLE INPUTsample-input/csvfor reference;Data Catalog Sample Custom Entries(Google Sheets) might help to create/export a CSV file.COMMANDSPython + virtualenvdatacatalog-custom-entriessync\--csv-file<CSV-FILE-PATH>\--project-id<YOUR-PROJECT-ID>--location-id<YOUR-LOCATION-ID>Dockerdockerbuild--rm--tagdatacatalog-custom-entries-manager.
dockerrun--rm--tty\--volume<CREDENTIALS-FILE-FOLDER>:/credentials--volume<CSV-FILE-FOLDER>:/data\datacatalog-custom-entries-managersync\--csv-file/data/<CSV-FILE-PATH>\--project-id<YOUR-PROJECT-ID>--location-id<YOUR-LOCATION-ID>2.1.2. To a JSON fileSTRUCTUREThe metadata structure to synchronize Custom Entries is presented below. Use as many objects as
needed to describe all Data Catalog Entries you need.{"userSpecifiedSystems":[{"name":"STRING","entryGroups":[{"id":"STRING","entries":[{"linkedResource":"STRING","displayName":"STRING","description":"STRING (optional)","type":"STRING","createdAt":"STRING (optional, format: YYYY-MM-DDTHH:MM:SSZ)","updatedAt":"STRING (optional, format: YYYY-MM-DDTHH:MM:SSZ)"}]}]}]}SAMPLE INPUTsample-input/jsonfor reference;COMMANDSPython + virtualenvdatacatalog-custom-entriessync\--json-file<JSON-FILE-PATH>\--project-id<YOUR-PROJECT-ID>--location-id<YOUR-LOCATION-ID>Dockerdockerbuild--rm--tagdatacatalog-custom-entries-manager.
dockerrun--rm--tty\--volume<CREDENTIALS-FILE-FOLDER>:/credentials--volume<CSV-FILE-FOLDER>:/data\datacatalog-custom-entries-managersync\--json-file<JSON-FILE-PATH>\--project-id<YOUR-PROJECT-ID>--location-id<YOUR-LOCATION-ID>