Package: datacatalog-fileset-enricher
datacatalog-fileset-enricherA Python package to enrich Google Cloud Data Catalog Fileset Entries with Data Catalog Tags. The goal of this library is to provide useful statistics regarding the GCS files that match the file pattern on the provided Data Catalog Fileset Entry.For instructions on how to create Fileset Entries, please go to the officialGoogle Cloud Docs1. Created TagsTags created by the fileset enricher are composed by the following attributes, and all stats are a snapshot of the
execution time:FieldDescriptionMandatoryexecution_timeExecution time when all stats were collected.YfilesNumber of files found, that matches the prefix.Nmin_file_sizeMinimum file size found in bytes.Nmax_file_sizeMaximum file size found in bytes.Navg_file_sizeAverage file size found in bytes.Ntotal_file_sizeTotal file size found in bytes.Nfirst_created_dateFirst time a file was created in the bucket(s).Nlast_created_dateLast time a file was created in the bucket(s).Nlast_updated_dateLast time a file was updated in the bucket(s).Ncreated_files_by_dayNumber of files created on the same date.Nupdated_files_by_dayNumber of files updated on the same date.NprefixPrefix used to find the files.Nbucket_prefixWhen specified at runtime, buckets without this prefix are ignored.Nbuckets_foundNumber of buckets that matched the prefix.Nfiles_by_bucketNumber of files found on each bucket.Nfiles_by_typeNumber of files found by file type.NIf no fields are specified when running the fileset enricher, all Tag fields will be applied.To generate file statistics and create the Tags this python package, uses the GCSlist_bucketsandlist_blobsAPIs to extract the metadata that matches the file pattern, so their billing policies will apply.2. Environment setup2.1. Get the codegitclonehttps://github.com/mesmacosta/datacatalog-fileset-enrichercddatacatalog-fileset-enricher2.2. Auth credentials2.2.1. Create a service account and grant it below rolesData Catalog Tag EditorData Catalog TagTemplate OwnerData Catalog ViewerStorage Admin or Custom Role with storage.buckets.list acl2.2.2. Download a JSON key and save it as./credentials/datacatalog-fileset-enricher.json2.3. VirtualenvUsingvirtualenvis optional, but strongly recommended unless you useDocker.2.3.1. Install Python 3.6+2.3.2. Create and activate an isolated Python environmentpipinstall--upgradevirtualenv
python3-mvirtualenv--pythonpython3envsource./env/bin/activate2.3.3. Install the dependenciespipinstall--upgrade--editable.2.3.4. Set environment variablesexportGOOGLE_APPLICATION_CREDENTIALS=./credentials/datacatalog-fileset-enricher.json2.4. DockerDocker may be used as an alternative to run all the scripts. In this case, please disregard theVirtualenvinstall instructions.3. Enrich DataCatalog Fileset Entry with Tags3.1. python main.py - Enrich all fileset entriespythonpythonmain.py--project-idmy_project\enrich-gcs-filesetsdockerdockerbuild--rm--tagdatacatalog-fileset-enricher.
dockerrun--rm--tty-vyour_credentials_folder:/datadatacatalog-fileset-enricher\--project-idmy_project\enrich-gcs-filesets3.2. python main.py -- Enrich all fileset entries using template from a different ProjectIf you are using a different project, make sure the Service Account has the following permissions on that project:Data Catalog TagTemplate CreatorData Catalog TagTemplate Userpythonmain.py--project-idmy_project\enrich-gcs-filesets\--tag-template-nameprojects/my_different_project/locations/us-central1/tagTemplates/fileset_enricher_findings3.3. python main.py -- Enrich a single entrypythonmain.py--project-idmy_project\enrich-gcs-filesets\--entry-group-idmy_entry_group\--entry-idmy_entry3.4. python main.py -- Enrich a single entry, specifying desired tag fieldsUsers are able to choose the Tag fields from the list provided atTagspythonmain.py--project-idmy_project\enrich-gcs-filesets\--entry-group-idmy_entry_group\--entry-idmy_entry--tag-fieldsfiles,prefix3.5. python main.py -- Pass a bucket prefix if you want to avoid scanning too many bucketsWhen the bucket_prefix is specified, the list_bucket api calls pass this prefix and avoid scanning buckets
that don't match the prefix. This only applies when there's a wildcard on the bucket_name, otherwise the
get bucket method is called and the bucket_prefix is ignored.pythonmain.py--project-idmy_project\enrich-gcs-filesets\--bucket-prefixmy_bucket3.6. python clean up template and tags (Reversible)Cleans up the Template and Tags from the Fileset Entries, running the main command will recreate those.pythonmain.py--project-idmy_project\clean-up-templates-and-tagsDisclaimersThis is not an officially supported Google product.