Package: datacatalog-tag-manager
datacatalog-tag-managerA Python package to manage Google Cloud Data Catalog tags, loading metadata from external
sources. Currently supports the CSV file format.Table of Contents1. Environment setup1.1. Python + virtualenv1.1.1. Install Python 3.6+1.1.2. Create a folder1.1.3. Create and activate an isolated Python environment1.1.4. Install the package1.2. Docker1.2.1. Get the source code1.3. Auth credentials1.3.1. Create a service account and grant it below roles1.3.2. Download a JSON key and save it as1.3.3. Set the environment variables2. Manage Tags2.1. Create or Update2.1.1. From a CSV file2.2. Delete2.2.1. From a CSV file1. Environment setup1.1. Python + virtualenvUsingvirtualenvis optional, but strongly recommended unless you useDocker.1.1.1. Install Python 3.6+1.1.2. Create a folderThis is recommended so all related stuff will reside at same place, making it easier to follow
below instructions.mkdir./datacatalog-tag-managercd./datacatalog-tag-managerAll paths starting with./in the next steps are relative to thedatacatalog-tag-managerfolder.1.1.3. Create and activate an isolated Python environmentpipinstall--upgradevirtualenv
python3-mvirtualenv--pythonpython3envsource./env/bin/activate1.1.4. Install the packagepipinstall--upgradedatacatalog-tag-manager1.2. DockerDocker may be used as an alternative to rundatacatalog-tag-manager. In this case, please
disregard theabovevirtualenvsetup instructions.1.2.1. Get the source codegitclonehttps://github.com/ricardolsmendes/datacatalog-tag-managercd./datacatalog-tag-manager1.3. Auth credentials1.3.1. Create a service account and grant it below rolesBigQuery Metadata ViewerData Catalog TagTemplate UserA custom role withbigquery.datasets.updateTagandbigquery.tables.updateTagpermissions1.3.2. Download a JSON key and save it as./credentials/datacatalog-tag-manager.json1.3.3. Set the environment variablesThis step may be skipped if you're usingDocker.exportGOOGLE_APPLICATION_CREDENTIALS=./credentials/datacatalog-tag-manager.json2. Manage Tags2.1. Create or Update2.1.1. From a CSV fileSCHEMAThe metadata schema to create or update Tags is presented below. Use as many lines as needed to
describe all the Tags and Fields you need.ColumnDescriptionMandatorylinked_resource OR entry_nameFull name of the BigQuery or PubSub asset the Entry refers to, or an Entry name if you are working withCustom Entriesyestemplate_nameResource name of the Tag Template for the TagyescolumnAttach Tags to a column belonging to the Entry schemanofield_idId of the Tag fieldyesfield_valueValue of the Tag fieldyesSAMPLE INPUTsample-input/upsert-tagsfor reference;Data Catalog Sample Tags(Google Sheets) might help to create/export a CSV file.COMMANDSPython + virtualenvdatacatalog-tagsupsert--csv-file<CSV-FILE-PATH>Dockerdockerbuild--rm--tagdatacatalog-tag-manager.
dockerrun--rm--tty\--volume<CREDENTIALS-FILE-FOLDER>:/credentials--volume<CSV-FILE-FOLDER>:/data\datacatalog-tag-managerupsert--csv-file/data/<CSV-FILE-PATH>2.2. Delete2.2.1. From a CSV fileSCHEMAThe metadata schema to delete Tags is presented below. Use as many lines as needed to delete all
the Tags you want.ColumnDescriptionMandatorylinked_resource OR entry_nameFull name of the BigQuery or PubSub asset the Entry refers to, or an Entry name if you are working withCustom Entriesyestemplate_nameResource name of the Tag Template of the TagyescolumnDelete Tags from a column belonging to the Entry schemanoSAMPLE INPUTsample-input/delete-tagsfor reference;Data Catalog Sample Tags(Google Sheets) might help to create/export a CSV file.COMMANDSPython + virtualenvdatacatalog-tagsdelete--csv-file<CSV-FILE-PATH>Dockerdockerbuild--rm--tagdatacatalog-tag-manager.
dockerrun--rm--tty\--volume<CREDENTIALS-FILE-FOLDER>:/credentials--volume<CSV-FILE-FOLDER>:/data\datacatalog-tag-managerdelete--csv-file/data/<CSV-FILE-PATH>