Package: podaac-tig
TIG - Tool for Image GenerationTable of contentsOverviewInstall poetryBuildHow to load and use tig moduletig Inputtig OutputCLI CommandsOverviewtig is built to be used within Cumulus ecosystem.  It is depending on cumulus CMA (Cumulus Documentation).
Please refer to theUsagesection for inputs and outputs. TIG itself is a lambda function which runs on top of CMA as its lambda layer.Install poetry (python dependency manager)Install poetry following the directions here:https://python-poetry.org/docs/#installationBe sure to install poetry in an isolated environment from the rest of your system.Be sure to use with a python version less than 3.12BuildJenkins pipeline template is applied to this project.development is based on poetry python environment.poetry run pytest   // to run unit testpoetry install      // to install all dependencies defined in toml filepoetry shell        //to enter the poetry shellpoetry build        // to build the wheelUnit Test can be run using the commandpoetryrunpytestCLI CommandsGenerate ThumbnailsGenerate Config FileRun TigGenerate ThumbnailsUse cli helper script to generate thumbnails for each collection (NOTE: This automatically generates the config file and runs tig)Go to dir `podaac/tig` and look at generate_thumbnails.sh script.  Follow directions at top to setup.Generate Config FileUse cli to create a tig configuration for collectionsgenerate_hitide_config --granule <granule_file> -dataset-id <collection short name> --include-image-variables <csv file image variables> --longitude <lon variable> --latitude <lat variable> --time <time variable> --footprint_strategy <footprint strategy>granule: a sample granule file to generate the configuration for
datset-id: collection short name
include-image-variables: csv file of with image variable names and min max setting for each variable
longitude: longitude variable include the group if they're in group defaults to longitude
latitude: latitude variable include the group if they're in a group defaults to latitude
time: time variable include the group if they're in a group defaults to time
footprint_strategy: strategy to generate footprint will default to None options should be ["periodic", "linestring", "polar", "swot_linestring", "polarsides", "smap"]Run TigUse cli to test thumbnail image generation for a granule with configuration file and palettestig --input_file <granule> --output_dir <output_dir> --config_file <config_file> --palette_dir <palette_dir>Use cli to create a tig configuration for collectionsgenerate_hitide_config --granule <granule_file> -dataset-id <collection short name> --include-image-variables <csv file image variables> --longitude <lon variable> --latitude <lat variable> --time <time variable> --footprint_strategy <footprint strategy>granule: a sample granule file to generate the configuration for
datset-id: collection short name
include-image-variables: csv file of with image variable names and min max setting for each variable
longitude: longitude variable include the group if they're in group defaults to longitude
latitude: latitude variable include the group if they're in a group defaults to latitude
time: time variable include the group if they're in a group defaults to time
footprint_strategy: strategy to generate footprint will default to None options should be ["periodic", "linestring", "polar", "swot_linestring", "polarsides", "smap"]CSV Columnsvariable: name of variable
min: min value for variable
max: max value for variable
palette (optional): the palette to be used for the variable
fill_missing (optional): if the generated images have missing pixel in images most likely resolution is to big, either lower resolution or we can fill in the pixels with surrounding pixel
ppd (optional): resolution of the variable, must be an integerHow to load and use tig moduleProject using tig can include/use the tig as following:module"tig_module"{source="https://cae-artifactory.jpl.nasa.gov/artifactory/general/gov/nasa/podaac/cumulus/tig-terraform/tig-terraform-0.3.0.zip"//Lambdavariablesprefix=var.prefiximage=var.tig_imagerole=module.cumulus.lambda_processing_role_arncmr_environment=var.cmr_environmentsubnet_ids=var.subnet_idssecurity_group_ids=[aws_security_group.no_ingress_all_egress.id]task_logs_retention_in_days=var.task_logs_retention_in_daysconfig_url="https://hitide.podaac.earthdatacloud.nasa.gov/dataset-configs"palette_url="https://hitide.podaac.earthdatacloud.nasa.gov/palettes"memory_size=var.tig_memory_size}and the module input variables explained as below.field nametypedefaultvaluesdescriptionprefixstring(required)A prefix string of lambda function. Ex. prefix = "sample" , created lambda : sample-tigregionstring(required)AWS region where tig lambda is running upon.  Ex. us-west-2cmr_environmentstring(required)dev, sit, opsconfig_bucketstring(required)bucket where dataset config residesconfig_dirstring(required)directory where dataset config file resides. dataset-config file follows the collection_shortname.cfg pattern. Ex. MODIS_A-JPL-L2P-v2019.0.cfgtig_output_bucketstring(required)bucket where tig file is created and writtentig_output_dirstring(required)output directory of created tig(fp) file. file will be created as s3://tig_output_bucket/tig_output_dir/collection_short_name/granule_id.png. ex. s3://my-cumulus-internaldataset-tig/ MODIS_A-JPL-L2P-v2019.0/20200101000000-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.pnglambda_rolestring(required)aws user role to run tig lambdalayerslist(string)(required)list of layers' arn where tig runs upon.security_group_idslist(string)(required)security group idssubnet_idslist(string)(required)subnet ids where tig runs withinconfig_urlstringthe url of where to retrieve configurationspalette_urlstringthe url of where to retrieve palettesECS input variables optionalfield nametypedefaultvaluesdescriptiontig_ecsboolfalseboolean to deploy ecs taskcluster_arnstringcumulus cluster arndesired_countnumber1number of ecs tig task to runlog_destination_arnstringA shared AWS:Log:Destination that receives logs in log_groupsecs_cpunumber700cpu unit to allocate to a tig taskecs_memory_reservationnumber1024memory unit to allocate to tig taskFargate input variables optionalfield nametypedefaultvaluesdescriptiontig_fargateboolfalseboolean to deploy fargate taskfargate_memorynumber2048amount of memory to allocate for a single fargate taskfargate_cpunumber1024amount of cpu to allocate for a single fargate taskfargate_desired_countnumber1desired count of how many fargate taskfargate_min_capacitynumber1minimum number of fargate task when scalingfargate_max_capacitynumber1maximum number of fargate task when scalingscale_dimensionsmap(string)nullcloudwatch dimensions to scale onscale_up_cooldownnumber60seconds before able to scaling up againscale_down_cooldownnumber120seconds before able to scaling down againcomparison_operator_scale_upstringGreaterThanOrEqualToThresholdThe arithmetic operation to use when comparing the specified Statistic and Thresholdevaluation_periods_scale_upnumber1The number of periods over which data is compared to the specified thresholdmetric_name_scale_upstringCPUUtilizationname of the metricnamespace_scale_upstringAWS/ECSnamespace for the alarm's associated metricperiod_scale_upnumber60period in seconds over which the specified statistic is appliedstatistic_scale_upstringAveragestatistic to apply to the metricthreshold_scale_upnumber50threshold for statistic to compare against to trigger stepscale_up_step_adjustmentliststep adjustment to make when scaling up fargatecomparison_operator_scale_downstringevaluation_periods_scale_downnumber1The number of periods over which data is compared to the specified thresholdmetric_name_scale_downstringCPUUtilizationname of the metricnamespace_scale_downstringAWS/ECSnamespace for the alarm's associated metricperiod_scale_downnumber60period in seconds over which the specified statistic is appliedstatistic_scale_downstringAveragestatistic to apply to the metricthreshold_scale_downnumber50threshold for statistic to compare against to trigger stepscale_down_step_adjustmentliststep adjustment to make when scaling down fargatefargate_iam_rolestringiam arn role for fargatemodule output variablesfield nametypedefaultvaluesdescriptiontig_function_namestring(required)The name of deployed tig lambda functiontig_task_arnstring(required)tig lambda aws arntig InputCumulus message with granules payload.  Example below{"granules":[{"files":[{"filename":"s3://bucket/file/with/checksum.dat","checksumType":"md5","checksum":"asdfdsa"},{"filename":"s3://bucket/file/without/checksum.dat",}]}]}tig OutputA tig file will be created under configured tig_output_bucket and tig-output-dir.  filename as granuleId.png. Ex. s3://my-cumulus-internaldataset-tig/ MODIS_A-JPL-L2P-v2019.0/20200101000000-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.pngA file object will be appended to the files[] of processed granule. Example:{"granules":[{"granuleId":"20200101000000-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0","dataType":"MODIS_A-JPL-L2P-v2019.0","sync_granule_duration":2603,"files":[{"bucket":"my-protected","path":"MODIS_A-JPL-L2P-v2019.0/2020/001","filename":"s3://my-protected/MODIS_A-JPL-L2P-v2019.0/20200101000000-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc","size":18232098,"name":"20200101000000-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc","checksumType":"md5","checksum":"aa5204f125ae83847b3b80fa2e571b00","type":"data","url_path":"{cmrMetadata.CollectionReference.ShortName}","filepath":"MODIS_A-JPL-L2P-v2019.0/20200101000000-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc","duplicate_found":true},{"bucket":"my-public","path":"MODIS_A-JPL-L2P-v2019.0/2020/001","filename":"s3://my-public/MODIS_A-JPL-L2P-v2019.0/20200101000000-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc.md5","size":98,"name":"20200101000000-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc.md5","type":"metadata","url_path":"{cmrMetadata.CollectionReference.ShortName}","filepath":"MODIS_A-JPL-L2P-v2019.0/20200101000000-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc.md5","duplicate_found":true},{"bucket":"my-public","filename":"s3://my-public/MODIS_A-JPL-L2P-v2019.0/20200101000000-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.cmr.json","size":1617,"name":"20200101000000-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.cmr.json","type":"metadata","url_path":"{cmrMetadata.CollectionReference.ShortName}","filepath":"MODIS_A-JPL-L2P-v2019.0/20200101000000-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.cmr.json","etag":"\"3e5b9259c5ee7eae5fe71467f151498b\""},{"bucket":"my-internal","filename":"s3://my-internal/dataset-tig/MODIS_A-JPL-L2P-v2019.0/20200101000000-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.png","filepath":"dataset-tig/MODIS_A-JPL-L2P-v2019.0/20200101000000-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.png","size":452,"name":"20200101000000-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.png","type":"metadata"}],}