Package: datacatalog-util
Datacatalog UtilA Python package to manage Google Cloud Data Catalog helper commands and scripts.Disclaimer: This is not an officially supported Google product.Commands ListGroupCommandDescriptionDocumentation LinkCode RepotagscreateLoad Tags from CSV file.GOGOtagsdeleteDelete Tags from CSV file.GOGOtagsexportExport Tags to CSV file.GOGOtag-templatescreateLoad Templates from CSV file.GOGOtag-templatesdeleteDelete Templates from CSV file.GOGOtag-templatesexportExport Templates to CSV file.GOGOfilesetscreateCreate GCS filesets from CSV file.GOGOfilesetsenrichEnrich GCS filesets with Tags.GOGOfilesetsclean-up-templates-and-tagsCleans up the Fileset Template and their Tags.GOGOfilesetsdeleteDelete GCS filesets from CSV file.GOGOfilesetsexportExport Filesets to CSV file.GOGOobject-storagecreate-entriesCreate Entries for each Object Storage File.GOGOobject-storagedelete-entriesDelete Entries that belong to the Object Storage Files.GOGOExecute Tutorial in Cloud ShellTable of Contents0. Executing in Cloud Shell from PyPi1. Environment setup for local build1.1. Python + virtualenv1.1.1. Install Python 3.6+1.1.2. Get the source code1.1.3. Create and activate an isolated Python environment1.1.4. Install the package1.2. Docker1.3. Auth credentials1.3.1. Create a service account and grant it below roles1.3.2. Download a JSON key and save it as1.3.3. Set the environment variables2. Load Tags from CSV file2.1. Create a CSV file representing the Tags to be created2.1.1 Execute Tutorial in Cloud Shell2.2. Run the datacatalog-util script - Create the Tags2.3. Run the datacatalog-util script - Delete the Tags3. Export Tags to CSV file3.1. A list of CSV files, each representing one Template will be created.3.1.1 Execute Tutorial in Cloud Shell3.2. Run tags export3.3 Run tags export filtering Tag Templates4. Load Templates from CSV file4.1. Create a CSV file representing the Templates to be created4.1.1 Execute Tutorial in Cloud Shell4.2. Run the datacatalog-util script - Create the Tag Templates4.3. Run the datacatalog-util script - Delete the Tag Templates5. Export Templates to CSV file5.1. A CSV file representing the Templates will be created5.1.1 Execute Tutorial in Cloud Shell5.2. Run the datacatalog-util script6. Filesets Commands6.1. Create a CSV file representing the Entry Groups and Entries to be created6.1.1 Execute Tutorial in Cloud Shell6.2. Create the Filesets Entry Groups and Entries6.2.1 Create the Filesets Entry Groups and Entries - with DataFlow SQL types validation6.3. Enrich GCS Filesets with Tags6.3.1 Enrich all fileset entries using Tag Template from a different Project (Good way to reuse the same Template)6.3.2 Execute Fileset Enricher Tutorial in Cloud Shell6.4. clean up template and tags6.5. Delete the Filesets Entry Groups and Entries7. Export Filesets to CSV file7.1. A CSV file representing the Filesets will be created7.1.1 Execute Tutorial in Cloud Shell7.2. Run the datacatalog-util script8. DataCatalog Object Storage commands8.1 Execute Tutorial in Cloud Shell8.2. Create DataCatalog entries based on object storage files8.3. Delete object storage entries on entry group9. Data Catalog Templates Examples0. Executing in Cloud Shell from PyPiIf you want to execute this script directly in cloud shell, download it from PyPi:# Set your SERVICE ACCOUNT, for instructions go to 1.3. Auth credentials# This name is just a suggestion, feel free to name it following your naming conventionsexportGOOGLE_APPLICATION_CREDENTIALS=~/credentials/datacatalog-util-sa.json# Install datacatalog-utilpip3install--upgradedatacatalog-util--user# Add to your PATHexportPATH=~/.local/bin:$PATH# Look for available commandsdatacatalog-util--help1. Environment setup for local build1.1. Python + virtualenvUsingvirtualenvis optional, but strongly recommended unless you useDocker.1.1.1. Install Python 3.6+1.1.2. Get the source codegitclonehttps://github.com/mesmacosta/datacatalog-utilcd./datacatalog-utilAll paths starting with./in the next steps are relative to thedatacatalog-utilfolder.1.1.3. Create and activate an isolated Python environmentpipinstall--upgradevirtualenv
python3-mvirtualenv--pythonpython3envsource./env/bin/activate1.1.4. Install the packagepipinstall--upgrade.1.2. DockerDocker may be used as an alternative to run the script. In this case, please disregard theVirtualenvsetup instructions.1.3. Auth credentials1.3.1. Create a service account and grant it below rolesData Catalog AdminStorage Admin1.3.2. Download a JSON key and save it asThis name is just a suggestion, feel free to name it following your naming conventions./credentials/datacatalog-util-sa.json1.3.3. Set the environment variablesThis step may be skipped if you're usingDocker.exportGOOGLE_APPLICATION_CREDENTIALS=~/credentials/datacatalog-util-sa.json2. Load Tags from CSV file2.1. Create a CSV file representing the Tags to be createdTags are composed of as many lines as required to represent all of their fields. The columns are
described as follows:ColumnDescriptionMandatorylinked_resourceFull name of the asset the Entry refers to.Ytemplate_nameResource name of the Tag Template for the Tag.YcolumnAttach Tags to a column belonging to the Entry schema.Nfield_idId of the Tag field.Yfield_valueValue of the Tag field.YTIPSsample-input/create-tagsfor reference;Data Catalog Sample Tags(Google Sheets) may help to create/export the CSV.2.1.1 Execute Tutorial in Cloud Shell2.2. Run the datacatalog-util script - Create the TagsPython + virtualenvdatacatalog-utiltagscreate--csv-fileCSV_FILE_PATHDockerdockerbuild--rm--tagdatacatalog-util.
dockerrun--rm--tty\--volumeCREDENTIALS_FILE_FOLDER:/credentials--volumeCSV_FILE_FOLDER:/data\datacatalog-utilcreate-tags--csv-file/data/CSV_FILE_NAME2.3. Run the datacatalog-util script - Delete the TagsPython + virtualenvdatacatalog-utiltagsdelete--csv-fileCSV_FILE_PATH3. Export Tags to CSV file3.1. A list of CSV files, each representing one Template will be created.One file with summary with stats about each template, will also be created on the same directory.The columns for the summary file are described as follows:ColumnDescriptiontemplate_nameResource name of the Tag Template for the Tag.tags_countNumber of tags found from the template.tagged_entries_countNumber of tagged entries with the template.tagged_columns_countNumber of tagged columns with the template.tag_string_fields_countNumber of used String fields on tags of the template.tag_bool_fields_countNumber of used Bool fields on tags of the template.tag_double_fields_countNumber of used Double fields on tags of the template.tag_timestamp_fields_countNumber of used Timestamp fields on tags of the template.tag_enum_fields_countNumber of used Enum fields on tags of the template.The columns for each template file are described as follows:ColumnDescriptionrelative_resource_nameFull resource name of the asset the Entry refers to.linked_resourceFull name of the asset the Entry refers to.template_nameResource name of the Tag Template for the Tag.tag_nameResource name of the Tag.columnAttach Tags to a column belonging to the Entry schema.field_idId of the Tag field.field_typeType of the Tag field.field_valueValue of the Tag field.3.1.1 Execute Tutorial in Cloud Shell3.2. Run tags exportPython + virtualenvdatacatalog-utiltagsexport--project-idsmy-project--dir-pathDIR_PATH3.3 Run tags export filtering Tag TemplatesPython + virtualenvdatacatalog-utiltagsexport--project-idsmy-project\--dir-pathDIR_PATH\--tag-templates-namesprojects/my-project/locations/us-central1/tagTemplates/my-template,\projects/my-project/locations/us-central1/tagTemplates/my-template-24. Load Templates from CSV file4.1. Create a CSV file representing the Templates to be createdTemplates are composed of as many lines as required to represent all of their fields. The columns are
described as follows:ColumnDescriptionMandatorytemplate_nameResource name of the Tag Template for the Tag.Ydisplay_nameResource name of the Tag Template for the Tag.Yfield_idId of the Tag Template field.Yfield_display_nameDisplay name of the Tag Template field.Yfield_typeType of the Tag Template field.Yenum_valuesValues for the Enum field.N4.1.1 Execute Tutorial in Cloud Shell4.2. Run the datacatalog-util script - Create the Tag TemplatesPython + virtualenvdatacatalog-utiltag-templatescreate--csv-fileCSV_FILE_PATH4.3. Run the datacatalog-util script - Delete the Tag TemplatesPython + virtualenvdatacatalog-utiltag-templatesdelete--csv-fileCSV_FILE_PATHTIPSsample-input/create-tag-templatesfor reference;5. Export Templates to CSV file5.1. A CSV file representing the Templates will be createdTemplates are composed of as many lines as required to represent all of their fields. The columns are
described as follows:ColumnDescriptiontemplate_nameResource name of the Tag Template for the Tag.display_nameResource name of the Tag Template for the Tag.field_idId of the Tag Template field.field_display_nameDisplay name of the Tag Template field.field_typeType of the Tag Template field.enum_valuesValues for the Enum field.5.1.1 Execute Tutorial in Cloud Shell5.2. Run the datacatalog-util scriptPython + virtualenvdatacatalog-utiltag-templatesexport--project-idsmy-project--file-pathCSV_FILE_PATH6. Filesets Commands6.1. Create a CSV file representing the Entry Groups and Entries to be createdFilesets are composed of as many lines as required to represent all of their fields. The columns are
described as follows:ColumnDescriptionMandatoryentry_group_nameEntry Group Name.Yentry_group_display_nameEntry Group Display Name.Nentry_group_descriptionEntry Group Description.Nentry_idEntry ID.Yentry_display_nameEntry Display Name.Yentry_descriptionEntry Description.Nentry_file_patternsEntry File Patterns.Yschema_column_nameSchema column name.Nschema_column_typeSchema column type.Nschema_column_descriptionSchema column description.Nschema_column_modeSchema column mode.NPlease note that theschema_column_typeis an open string field and accept anything, if you want
to use your fileset with Dataflow SQL, follow the data-types in theofficial docs.6.1.1 Execute Tutorial in Cloud Shell6.2. Create the Filesets Entry Groups and EntriesPython + virtualenvdatacatalog-utilfilesetscreate--csv-fileCSV_FILE_PATHTIPSsample-input/create-filesetsfor reference;If you want to create filesets without schema:sample-input/create-filesets/fileset-entry-opt-1-all-metadata-no-schema.csvfor reference;6.2.1 Create the Filesets Entry Groups and Entries - with DataFlow SQL types validationPython + virtualenvdatacatalog-utilfilesetscreate--csv-fileCSV_FILE_PATH--validate-dataflow-sql-types6.3. Enrich GCS Filesets with TagsUsers are able to choose the Tag fields from the list provided atTagsdatacatalog-utilfilesetsenrich--project-idmy-project6.3.1 Enrich all fileset entries using Tag Template from a different Project (Good way to reuse the same Template)If you are using a different Project, make sure the Service Account has the following permissions on that Project or that Template:Data Catalog TagTemplate CreatorData Catalog TagTemplate Userdatacatalog-utilfilesets\--project-idmy_project\enrich--tag-template-nameprojects/my_different_project/locations/us-central1/tagTemplates/fileset_enricher_findings6.3.2 Execute Fileset Enricher Tutorial in Cloud Shell6.4. clean up template and tagsCleans up the Template and Tags from the Fileset Entries, running the main command will recreate those.datacatalog-utilfilesetsclean-up-templates-and-tags--project-idmy-project6.5. Delete the Filesets Entry Groups and EntriesPython + virtualenvdatacatalog-utilfilesetsdelete--csv-fileCSV_FILE_PATH7. Export Filesets to CSV file7.1. A CSV file representing the Filesets will be createdFilesets are composed of as many lines as required to represent all of their fields. The columns are
described as follows:ColumnDescriptionMandatoryentry_group_nameEntry Group Name.Yentry_group_display_nameEntry Group Display Name.Yentry_group_descriptionEntry Group Description.Yentry_idEntry ID.Yentry_display_nameEntry Display Name.Yentry_descriptionEntry Description.Yentry_file_patternsEntry File Patterns.Yschema_column_nameSchema column name.Nschema_column_typeSchema column type.Nschema_column_descriptionSchema column description.Nschema_column_modeSchema column mode.N7.1.1 Execute Tutorial in Cloud Shell7.2. Run the datacatalog-util scriptPython + virtualenvdatacatalog-utilfilesetsexport--project-idsmy-project--file-pathCSV_FILE_PATH8. DataCatalog Object Storage commands8.1 Execute Tutorial in Cloud Shell8.2. Create DataCatalog entries based on object storage filesdatacatalog-util\object-storagesync-entries--typecloud_storage\--project-idmy_project\--entry-group-nameprojects/my_project/locations/us-central1/entryGroups/my_entry_group\--bucket-prefixmy_bucket8.3. Delete object storage entries on entry groupdatacatalog-util\object-storagedelete-entries--typecloud_storage\--project-idmy_project\--entry-group-nameprojects/my_project/locations/us-central1/entryGroups/my_entry_group9. Data Catalog Templates Examplestemplates_examples.mdHistory0.1.0 (2020-04-08)First release on PyPI.0.1.1 (2020-04-09)Update description.0.2.0 (2020-04-09)Improved groups command line.0.3.0 (2020-04-15)Added datacatalog-tag-template-processor library.0.4.0 (2020-04-15)Added datacatalog-tag-template-exporter library.0.5.0 (2020-04-15)Added datacatalog-tag-exporter library.0.6.0 (2020-04-28)Added datacatalog-fileset-processor library.0.7.0 (2020-04-29)Added new tag delete command.0.8.0 (2020-04-29)Added new fileset export command.0.9.0 (2020-04-29)Added new object storage commands.