Package: datacatalog-fileset-processor
Datacatalog Fileset ProcessorA package to manage Google Cloud Data Catalog Fileset scripts.Disclaimer: This is not an officially supported Google product.Table of ContentsExecuting in Cloud Shell1. Environment setup1.1. Python + virtualenv1.1.1. Install Python 3.6+1.1.2. Get the source code1.1.3. Create and activate an isolated Python environment1.1.4. Install the package1.2. Docker1.3. Auth credentials1.3.1. Create a service account and grant it below roles1.3.2. Download a JSON key and save it as1.3.3. Set the environment variables2. Create Filesets from CSV file2.1. Create a CSV file representing the Entry Groups and Entries to be created2.2. Run the datacatalog-fileset-processor script - Create the Filesets Entry Groups and Entries2.3. Run the datacatalog-fileset-processor script - Delete the Filesets Entry Groups and EntriesExecuting in Cloud Shell# Set your SERVICE ACCOUNT, for instructions go to 1.3. Auth credentials# This name is just a suggestion, feel free to name it following your naming conventionsexportGOOGLE_APPLICATION_CREDENTIALS=~/datacatalog-fileset-processor-sa.json# Install datacatalog-fileset-processorpip3installdatacatalog-fileset-processor--user# Add to your PATHexportPATH=~/.local/bin:$PATH# Look for available commandsdatacatalog-fileset-processor--help1. Environment setup1.1. Python + virtualenvUsingvirtualenvis optional, but strongly recommended unless you useDocker.1.1.1. Install Python 3.6+1.1.2. Get the source codegitclonehttps://github.com/mesmacosta/datacatalog-fileset-processorcd./datacatalog-fileset-processorAll paths starting with./in the next steps are relative to thedatacatalog-fileset-processorfolder.1.1.3. Create and activate an isolated Python environmentpipinstall--upgradevirtualenv
python3-mvirtualenv--pythonpython3envsource./env/bin/activate1.1.4. Install the packagepipinstall--upgrade.1.2. DockerDocker may be used as an alternative to run the script. In this case, please disregard theVirtualenvsetup instructions.1.3. Auth credentials1.3.1. Create a service account and grant it below rolesData Catalog Admin1.3.2. Download a JSON key and save it asThis name is just a suggestion, feel free to name it following your naming conventions./credentials/datacatalog-fileset-processor-sa.json1.3.3. Set the environment variablesThis step may be skipped if you're usingDocker.exportGOOGLE_APPLICATION_CREDENTIALS=~/credentials/datacatalog-fileset-processor-sa.json2. Create Filesets from CSV file2.1. Create a CSV file representing the Entry Groups and Entries to be createdFilesets are composed of as many lines as required to represent all of their fields. The columns are
described as follows:ColumnDescriptionMandatoryentry_group_nameEntry Group Name.Yentry_group_display_nameEntry Group Display Name.Nentry_group_descriptionEntry Group Description.Nentry_idEntry ID.Yentry_display_nameEntry Display Name.Yentry_descriptionEntry Description.Nentry_file_patternsEntry File Patterns.Yschema_column_nameSchema column name.Nschema_column_typeSchema column type.Nschema_column_descriptionSchema column description.Nschema_column_modeSchema column mode.NPlease note that theschema_column_typeis an open string field and accept anything, if you want
to use your fileset with Dataflow SQL, follow the data-types in theofficial docs.2.2. Run the datacatalog-fileset-processor script - Create the Filesets Entry Groups and EntriesPython + virtualenvdatacatalog-fileset-processorfilesetscreate--csv-fileCSV_FILE_PATH2.3. Run the datacatalog-fileset-processor script - Delete the Filesets Entry Groups and EntriesPython + virtualenvdatacatalog-fileset-processorfilesetsdelete--csv-fileCSV_FILE_PATHTIPSsample-input/create-filesetsfor reference;If you want to create filesets without schema:sample-input/create-filesets/fileset-entry-opt-1-all-metadata-no-schema.csvfor reference;History0.1.0 (2020-04-24)First release on PyPI.