Repository: linz/topo-workflows
Pull request #983: feat: national hillshade workflows TDE-1426
Author: login: amfage
Created: 2025-03-14T02:17:39Z | Merged: 2025-03-21T03:01:53Z

### Motivation

To create national hillshade datasets derived from the National 1m DEM using the [generate_hillshade.py](https://github.com/linz/topo-imagery/blob/master/scripts/generate_hillshade.py) script, with associated STAC metadata.

### Modifications

Creation of a `national-hillshade` workflowtemplate that uses common sub-templates including `tpl-ti-generate-hillshade`. This will be run using a cron workflow.

### Verification

Argo linting, and workflows run in Argo.


Diff:
diff --git a/templates/topo-imagery/README.md b/templates/topo-imagery/README.md
index e1dc8ae25..f4addd770 100644
--- a/templates/topo-imagery/README.md
+++ b/templates/topo-imagery/README.md
@@ -122,38 +122,36 @@ See [collection_from_items.py](https://github.com/linz/topo-imagery/blob/master/
 
 ## topo-imagery/generate-hillshade - `tpl-create-hillshade`
 
-Template for creating hillshades from TIFFs (DEM).  
+Template for creating hillshades from TIFFs (DEM).
 See [generate_hillshade.py](https://github.com/linz/topo-imagery/pull/1253)
-This `WorkflowTemplate` is also calling other templates in order to create the inputs of the `linz/topo-imagery/generate_hillshade.py` script:
-
-1. `tile-index-validate` to create a list of tiles with their source TIFF files
-2. `group` to split the list created into grouped tiles for parallel processing
-
-It allows generating multiple types of hillshade from the same source TIFF files, as the `preset` parameter expects a list of hillshade presets.
 
 ### Template usage
 
 ```yaml
-- name: create-hillshade
+- name: generate-hillshade
   templateRef:
-    name: tpl-create-hillshade
+    name: tpl-ti-generate-hillshade
     template: main
   arguments:
     parameters:
-      - name: source
-        value: 's3://nz-elevation/new-zealand/new-zealand-contour/dem_8m/2193/'
-      - name: scale
-        value: '50000'
-      - name: group
-        value: '2'
-      - name: version_argo_tasks
-        value: 'v4'
+      - name: group_id
+        value: '{{item}}'
+      - name: hillshade_preset
+        value: '{{=sprig.trim(workflow.parameters.hillshade_preset)}}'
       - name: version_topo_imagery
-        value: 'v7'
-      - name: target
-        value: '{{=sprig.trimSuffix("/", tasks["get-location"].outputs.parameters.location)}}/flat/'
-      - name: preset
-        value: '["greyscale", "igor"]'
+        value: '{{= workflow.parameters.version_topo_imagery}}'
+      - name: target # not using flat/ here, but {{workflow.parameters.hillshade_preset}}/ to keep temporary HS output separate
+        value: '{{=sprig.trimSuffix("/", tasks["get-location"].outputs.parameters.location)}}/{{workflow.parameters.hillshade_preset}}/flat/'
+      - name: collection_id
+        value: '{{tasks.stac-setup-hillshade.outputs.parameters.collection_id}}'
+      - name: gsd
+        value: '{{=sprig.trim(workflow.parameters.gsd)}}'
+      - name: current_datetime
+        value: '{{tasks.stac-setup-hillshade.finishedAt}}'
+    artifacts:
+      - name: group_data
+        from: '{{tasks.group.outputs.artifacts.output}}'
+  withParam: '{{tasks.group.outputs.parameters.output}}'
 ```
 
 The Workflow caller must have the following volume:
@@ -163,8 +161,3 @@ volumes:
   - name: ephemeral
     emptyDir: {}
 ```
-
-### Output
-
-The hillshade TIFF files will be saved in a subdirectory for each of the `preset` within the `target` directory.
-Example: `s3://linz-workflows-scratch/2025-02/03-test-hillshade-94clh/flat/igor/`
diff --git a/templates/topo-imagery/create-hillshade.yaml b/templates/topo-imagery/create-hillshade.yaml
deleted file mode 100644
index 62d464261..000000000
--- a/templates/topo-imagery/create-hillshade.yaml
+++ /dev/null
@@ -1,154 +0,0 @@
-# yaml-language-server: $schema=https://raw.githubusercontent.com/argoproj/argo-workflows/v3.5.5/api/jsonschema/schema.json
-
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  # Template calling linz/topo-imagery for its main task (generate-hillshade)
-  # see https://github.com/linz/topo-imagery/pull/1253
-  name: tpl-create-hillshade
-  labels:
-    linz.govt.nz/category: raster
-spec:
-  templateDefaults:
-    container:
-      imagePullPolicy: Always
-      image: ''
-  entrypoint: main
-  templates:
-    - name: main
-      inputs:
-        parameters:
-          - name: source
-            description: 's3 path to where the TIFF files to generate hillshade from are stored.'
-          - name: scale
-            description: 'Scale of the source data.'
-          - name: group
-            description: 'Number of tiles to group together.'
-          - name: version_argo_tasks
-            description: 'Version of the argo-tasks image to use.'
-            default: 'v4'
-          - name: version_topo_imagery
-            description: 'Version of the topo-imagery image to use.'
-            default: 'v7'
-          - name: target
-            description: 's3 path to where the hillshade tiles will be stored. The different hillshade types will be stored in subdirectories using the preset name.'
-          - name: preset
-            description: 'List of hillshade types to generate. Expect a list of strings.'
-            default: '["greyscale", "igor"]'
-      dag:
-        tasks:
-          - name: tile-index-validate
-            templateRef:
-              name: tpl-at-tile-index-validate
-              template: main
-            arguments:
-              parameters:
-                - name: scale
-                  value: '{{inputs.parameters.scale}}'
-                - name: include
-                  value: '.tiff?$'
-                - name: source
-                  value: '{{=sprig.trim(inputs.parameters.source)}}'
-                - name: source_epsg
-                  value: '2193'
-                - name: validate
-                  value: 'false'
-                - name: retile
-                  value: 'true'
-                - name: preset
-                  value: 'dem_lerc'
-                - name: version
-                  value: '{{= inputs.parameters.version_argo_tasks}}'
-
-          - name: group
-            templateRef:
-              name: tpl-at-group
-              template: main
-            arguments:
-              artifacts:
-                - name: input
-                  from: '{{ tasks.tile-index-validate.outputs.artifacts.files }}'
-              parameters:
-                - name: size
-                  value: '{{inputs.parameters.group}}'
-                - name: version
-                  value: '{{= inputs.parameters.version_argo_tasks}}'
-            depends: 'tile-index-validate'
-
-          - name: hillshade-preset
-            template: hillshade-preset
-            arguments:
-              parameters:
-                - name: group_id
-                  value: '{{item}}'
-                - name: target
-                  value: '{{inputs.parameters.target}}'
-                - name: preset
-                  value: '{{inputs.parameters.preset}}'
-                - name: version_topo_imagery
-                  value: '{{inputs.parameters.version_topo_imagery}}'
-              artifacts:
-                - name: group_data
-                  from: '{{ tasks.group.outputs.artifacts.output }}'
-            depends: 'group'
-            withParam: '{{ tasks.group.outputs.parameters.output }}'
-
-    - name: hillshade-preset
-      inputs:
-        parameters:
-          - name: group_id
-          - name: target
-          - name: preset
-          - name: version_topo_imagery
-        artifacts:
-          - name: group_data
-      steps:
-        - - name: generate-hillshade
-            template: generate-hillshade
-            arguments:
-              parameters:
-                - name: group_id
-                  value: '{{inputs.parameters.group_id}}'
-                - name: target
-                  value: '{{inputs.parameters.target}}'
-                - name: preset
-                  value: '{{item}}'
-                - name: version_topo_imagery
-                  value: '{{inputs.parameters.version_topo_imagery}}'
-              artifacts:
-                - name: group_data
-                  from: '{{inputs.artifacts.group_data}}'
-            withParam: '{{inputs.parameters.preset}}'
-
-    - name: generate-hillshade
-      nodeSelector:
-        karpenter.sh/capacity-type: 'spot'
-      inputs:
-        parameters:
-          - name: group_id
-          - name: target
-          - name: preset
-          - name: version_topo_imagery
-        artifacts:
-          - name: group_data
-            path: /tmp/input/
-      container:
-        image: '019359803926.dkr.ecr.ap-southeast-2.amazonaws.com/topo-imagery:{{=sprig.trim(inputs.parameters.version_topo_imagery)}}'
-        resources:
-          requests:
-            memory: 7.8Gi
-            cpu: 15000m
-            ephemeral-storage: 29.5Gi
-        volumeMounts:
-          # This volume must be defined in the workflow calling this workflowTemplate
-          - name: ephemeral
-            mountPath: '/tmp'
-        args:
-          - python
-          - '/app/scripts/generate_hillshade.py'
-          - '--from-file'
-          - '/tmp/input/{{inputs.parameters.group_id}}.json'
-          - '--target'
-          - '{{=sprig.trimSuffix("/", inputs.parameters.target)}}/{{inputs.parameters.preset}}/'
-          - '--preset'
-          - '{{inputs.parameters.preset}}'
diff --git a/templates/topo-imagery/generate-hillshade.yaml b/templates/topo-imagery/generate-hillshade.yaml
new file mode 100644
index 000000000..0ffa8648a
--- /dev/null
+++ b/templates/topo-imagery/generate-hillshade.yaml
@@ -0,0 +1,78 @@
+# yaml-language-server: $schema=https://raw.githubusercontent.com/argoproj/argo-workflows/v3.5.5/api/jsonschema/schema.json
+
+apiVersion: argoproj.io/v1alpha1
+kind: WorkflowTemplate
+metadata:
+  # Template calling linz/topo-imagery for its main task (generate-hillshade)
+  # see https://github.com/linz/topo-imagery/pull/1253
+  name: tpl-ti-generate-hillshade
+  labels:
+    linz.govt.nz/category: raster
+spec:
+  templateDefaults:
+    container:
+      imagePullPolicy: Always
+      image: ''
+  entrypoint: main
+  templates:
+    - name: main
+      inputs:
+        parameters:
+          - name: version_topo_imagery
+            description: 'Specify a version of the topo-imagery image to use, e.g. "v4.8" or "latest"'
+            default: 'v7'
+
+          - name: group_id
+            description: 'The ID of the group of files to use as the input.'
+            default: ''
+
+          - name: target
+            description: 'Location for output files'
+            default: ''
+
+          - name: hillshade_preset
+            description: 'Hillshade preset to use, must be one of "hillshade" or "hillshade-igor"'
+            default: ''
+
+          - name: gsd
+            description: 'Dataset GSD in metres, e.g., "0.3" for 30 centimetres'
+            default: '1'
+
+          - name: collection_id
+            description: 'Collection ID of existing dataset, or else a generated collection ID'
+            default: ''
+
+          - name: current_datetime
+            description: 'Current date and time of the workflow'
+            default: ''
+
+        artifacts:
+          - name: group_data
+            path: /tmp/input/
+
+      container:
+        image: '019359803926.dkr.ecr.ap-southeast-2.amazonaws.com/topo-imagery:{{=sprig.trim(inputs.parameters.version_topo_imagery)}}'
+        resources:
+          requests:
+            memory: 7.8Gi
+            cpu: 15000m
+            ephemeral-storage: 29.5Gi
+        volumeMounts:
+          # This volume must be defined in the workflow calling this workflowTemplate
+          - name: ephemeral
+            mountPath: '/tmp'
+        args:
+          - python
+          - '/app/scripts/generate_hillshade.py'
+          - '--from-file'
+          - '/tmp/input/{{inputs.parameters.group_id}}.json'
+          - '--target'
+          - '{{=sprig.trimSuffix("/", inputs.parameters.target)}}/'
+          - '--preset'
+          - '{{inputs.parameters.hillshade_preset}}'
+          - '--collection-id'
+          - '{{inputs.parameters.collection_id}}'
+          - '--gsd'
+          - '{{inputs.parameters.gsd}}'
+          - '--current-datetime'
+          - '{{inputs.parameters.current_datetime}}'
diff --git a/workflows/raster/README.md b/workflows/raster/README.md
index 7714bf21f..295e5404c 100644
--- a/workflows/raster/README.md
+++ b/workflows/raster/README.md
@@ -5,6 +5,7 @@
 - [publish-odr](#Publish-odr)
 - [National DEM](#national-dem)
 - [Merge Layers](#merge-layers)
+- [Hillshade](#hillshade)
 - [tests](#Tests)
 
 # Standardising
@@ -295,9 +296,9 @@ See the [copy template](#copy) for more information.
 
 # national-dem
 
-This workflow combines a set of DEMs datasets in order to create a single national dataset composed of 1:50k tiles.
+This workflow combines a set of DEM datasets in order to create a single national dataset composed of 1:50k tiles.
 
-Upon completion all standardised TIFF and STAC files will be located with the ./flat/ directory of the workflow in the artifacts scratch bucket. In addition, a Basemaps link is produced enabling visual QA.
+Upon completion all standardised TIFF and STAC files will be located in the ./{{hillshade_preset}}/flat/ directory of the workflow in the artifacts scratch bucket. In addition, a Basemaps link is produced enabling visual QA.
 
 Publishing to the AWS Registry of Open Data is an optional step [publish-odr](#Publish-odr) that can be run automatically after standardisation.
 
@@ -309,6 +310,7 @@ Publishing to the AWS Registry of Open Data is an optional step [publish-odr](#P
 | config_file    | str  | https://raw.githubusercontent.com/linz/basemaps-config/master/config/tileset/elevation.json | Location of the configuration file listing the source datasets to merge.                                                                                                                                                                                          |
 | odr_url        | str  |                                                                                             | (Optional) If an existing dataset add the S3 path to the dataset here to load existing metadata e.g. "s3://nz-elevation/new-zealand/new-zealand/dem_1m/2193/"                                                                                                     |
 | group          | int  | 2                                                                                           | How many output tiles to process in each standardising task "pod". Change if you have resource or performance issues when standardising a dataset.                                                                                                                |
+| group          | int  | 2                                                                                           | How many output tiles to process in each standardising task "pod". Change if you have resource or performance issues when standardising a dataset.                                                                                                                |
 | publish_to_odr | str  | false                                                                                       | Run [publish-odr](#Publish-odr) after standardising has completed successfully                                                                                                                                                                                    |
 | copy_option    | enum | --force-no-clobber                                                                          | Used only if `publish_to_odr` is true.<dl><dt>`--no-clobber` </dt><dd> Skip overwriting existing files.</dd><dt> `--force` </dt><dd> Overwrite all files. </dd><dt> `--force-no-clobber` </dt><dd> Overwrite only changed files, skip unchanged files. </dd></dl> |
 
@@ -372,6 +374,27 @@ Default values for this workflow should be sufficient for most use cases. Howeve
 | region                  | str     | new-zealand                                                                     | Region of the dataset.                                                                                                                                                                                                                                            |
 | gdal_compression_preset | str     | dem_lerc                                                                        | The GDAL compression preset to use for the output TIFFs.                                                                                                                                                                                                          |
 
+# hillshade
+
+This workflow can use a DEM dataset source (such as a National DEM dataset, for example) to create hillshades.
+
+Upon completion all hillshade TIFF and STAC files will be located in the .`{{workflow.parameters.hillshade_preset}}`/flat/ directory of the workflow in the artifacts scratch bucket. In addition, a Basemaps link is produced enabling visual QA.
+
+Publishing to the AWS Registry of Open Data is an optional step [publish-odr](#Publish-odr) that can be run automatically after hillshade creation.
+
+## Workflow Input Parameters
+
+| Parameter        | Type | Default                                                | Description                                                                                                                                                                                                                                                       |
+| ---------------- | ---- | ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| ticket           | str  |                                                        | Ticket ID e.g. 'TDE-1130'                                                                                                                                                                                                                                         |
+| source           | str  | s3://nz-elevation/new-zealand/new-zealand/dem_1m/2193/ | Location of the input elevation data to create hillshade.                                                                                                                                                                                                         |
+| gsd              | str  | 1                                                      | Dataset GSD in metres, e.g., "1" for 1 metre                                                                                                                                                                                                                      |
+| odr_url          | str  |                                                        | (Optional) If an existing dataset add the S3 path to the dataset here to load existing metadata e.g. "s3://nz-elevation/new-zealand/new-zealand/dem-hillshade_1m/2193/"                                                                                           |
+| hillshade-preset | str  | hillshade                                              | Hillshade preset to use, must be one of "hillshade" or "hillshade-igor"                                                                                                                                                                                           |
+| group            | int  | 4                                                      | How many output tiles to process in each standardising task "pod". Change if you have resource or performance issues when standardising a dataset.                                                                                                                |
+| publish_to_odr   | str  | false                                                  | Run [publish-odr](#Publish-odr) after standardising has completed successfully                                                                                                                                                                                    |
+| copy_option      | enum | --force-no-clobber                                     | Used only if `publish_to_odr` is true.<dl><dt>`--no-clobber` </dt><dd> Skip overwriting existing files.</dd><dt> `--force` </dt><dd> Overwrite all files. </dd><dt> `--force-no-clobber` </dt><dd> Overwrite only changed files, skip unchanged files. </dd></dl> |
+
 # Tests
 
 ## How To Use the Test Workflow
diff --git a/workflows/raster/hillshade.yaml b/workflows/raster/hillshade.yaml
new file mode 100644
index 000000000..7f408003e
--- /dev/null
+++ b/workflows/raster/hillshade.yaml
@@ -0,0 +1,328 @@
+# yaml-language-server: $schema=https://raw.githubusercontent.com/argoproj/argo-workflows/v3.5.5/api/jsonschema/schema.json
+
+apiVersion: argoproj.io/v1alpha1
+kind: WorkflowTemplate
+metadata:
+  name: hillshade
+  labels:
+    linz.govt.nz/category: raster
+    linz.govt.nz/data-type: raster
+spec:
+  parallelism: 50
+  nodeSelector:
+    karpenter.sh/capacity-type: 'spot'
+  entrypoint: main
+  onExit: exit-handler
+  workflowMetadata:
+    labels:
+      linz.govt.nz/region: 'new-zealand'
+    labelsFrom:
+      linz.govt.nz/ticket:
+        expression: workflow.parameters.ticket
+  podMetadata:
+    labels:
+      linz.govt.nz/category: raster
+      linz.govt.nz/data-type: raster
+      linz.govt.nz/region: 'new-zealand'
+  arguments:
+    parameters:
+      - name: version_argo_tasks
+        description: 'Specify a version of the argo-tasks image to use, e.g. "v4.1" or "latest"'
+        value: 'v4'
+      - name: version_basemaps_cli
+        description: 'Specify a version of the basemaps-cli image to use, e.g. "v7.1" or "latest"'
+        value: 'v7'
+      - name: version_topo_imagery
+        description: 'Specify a version of the topo-imagery image to use, e.g. "v4.8" or "latest"'
+        value: 'v7'
+      - name: ticket
+        description: 'Ticket ID, e.g. "TDE-1130"'
+        value: ''
+      - name: source
+        description: 'Location of the input elevation data to create hillshade.'
+        value: 's3://nz-elevation/new-zealand/new-zealand/dem_1m/2193/'
+      - name: gsd
+        description: 'Dataset GSD in metres, e.g., "1" for 1 metre'
+        default: '1'
+      - name: odr_url
+        description: '(Optional) If an existing dataset add the S3 path to the dataset here to load existing metadata e.g. "s3://nz-elevation/new-zealand/new-zealand/dem_1m/2193/"'
+        value: ''
+      - name: hillshade_preset
+        description: 'Hillshade preset to use, must be one of "hillshade" or "hillshade-igor"'
+        value: 'hillshade'
+        enum:
+          - 'hillshade-igor'
+          - 'hillshade'
+      - name: group
+        description: 'How many output tiles to process in each standardise-validate task "pod". Change if you have resource or performance issues when standardising a dataset.'
+        value: '4'
+      - name: publish_to_odr
+        description: 'Create a Pull Request for publishing to imagery or elevation ODR bucket'
+        value: 'false'
+        enum:
+          - 'false'
+          - 'true'
+      - name: copy_option
+        description: 'Do not overwrite existing files with "no-clobber", "force" overwriting files in the target location, or "force-no-clobber" overwriting only changed files, skipping unchanged files'
+        value: '--force-no-clobber'
+        enum:
+          - '--no-clobber'
+          - '--force'
+          - '--force-no-clobber'
+  templateDefaults:
+    container:
+      imagePullPolicy: Always
+      image: ''
+  templates:
+    - name: main
+      retryStrategy:
+        expression: 'false'
+      dag:
+        tasks:
+          - name: get-location
+            templateRef:
+              name: tpl-get-location
+              template: main
+
+          - name: stac-setup-hillshade
+            templateRef:
+              name: tpl-at-stac-setup
+              template: main
+            arguments:
+              parameters:
+                - name: gsd
+                  value: '{{=sprig.trim(workflow.parameters.gsd)}}'
+                - name: region
+                  value: 'new-zealand'
+                - name: geospatial_category
+                  value: 'dem-{{=sprig.trim(workflow.parameters.hillshade_preset)}}'
+                - name: odr_url
+                  value: '{{=sprig.trim(workflow.parameters.odr_url)}}'
+                - name: version
+                  value: '{{= workflow.parameters.version_argo_tasks}}'
+
+          - name: tile-index-validate
+            templateRef:
+              name: tpl-at-tile-index-validate
+              template: main
+            arguments:
+              parameters:
+                - name: scale
+                  value: '50000'
+                - name: include
+                  value: '.tiff?$'
+                - name: source
+                  value: '{{=sprig.trim(workflow.parameters.source)}}'
+                - name: source_epsg
+                  value: '2193'
+                - name: validate
+                  value: 'false'
+                - name: retile
+                  value: 'true'
+                - name: preset
+                  value: 'dem_lerc'
+                - name: version
+                  value: '{{= workflow.parameters.version_argo_tasks}}'
+                - name: includeDerived
+                  value: 'true'
+
+          - name: group
+            templateRef:
+              name: tpl-at-group
+              template: main
+            arguments:
+              artifacts:
+                - name: input
+                  from: '{{ tasks.tile-index-validate.outputs.artifacts.files }}'
+              parameters:
+                - name: size
+                  value: '{{=sprig.trim(workflow.parameters.group)}}'
+                - name: version
+                  value: '{{= workflow.parameters.version_argo_tasks}}'
+            depends: 'tile-index-validate.Succeeded'
+
+          - name: generate-hillshade
+            templateRef:
+              name: tpl-ti-generate-hillshade
+              template: main
+            arguments:
+              parameters:
+                - name: group_id
+                  value: '{{item}}'
+                - name: hillshade_preset
+                  value: '{{=sprig.trim(workflow.parameters.hillshade_preset)}}'
+                - name: version_topo_imagery
+                  value: '{{= workflow.parameters.version_topo_imagery}}'
+                - name: target # not using flat/ here, but {{workflow.parameters.hillshade_preset}}/ to keep temporary HS output separate
+                  value: '{{=sprig.trimSuffix("/", tasks["get-location"].outputs.parameters.location)}}/{{workflow.parameters.hillshade_preset}}/flat/'
+                - name: collection_id
+                  value: '{{tasks.stac-setup-hillshade.outputs.parameters.collection_id}}'
+                - name: gsd
+                  value: '{{=sprig.trim(workflow.parameters.gsd)}}'
+                - name: current_datetime
+                  value: '{{tasks.stac-setup-hillshade.finishedAt}}'
+              artifacts:
+                - name: group_data
+                  from: '{{tasks.group.outputs.artifacts.output}}'
+            depends: 'get-location.Succeeded && stac-setup-hillshade.Succeeded && group.Succeeded'
+            withParam: '{{tasks.group.outputs.parameters.output}}'
+
+          - name: create-collection
+            templateRef:
+              name: tpl-ti-create-collection
+              template: main
+            arguments:
+              parameters:
+                - name: collection_id
+                  value: '{{tasks.stac-setup-hillshade.outputs.parameters.collection_id}}'
+                - name: linz_slug
+                  value: '{{tasks.stac-setup-hillshade.outputs.parameters.linz_slug}}'
+                - name: location
+                  value: '{{=sprig.trimSuffix("/", tasks["get-location"].outputs.parameters.location)}}/{{=sprig.trim(workflow.parameters.hillshade_preset)}}/flat/'
+                - name: current_datetime
+                  value: '{{tasks.stac-setup-hillshade.finishedAt}}' # not stac-setup
+                - name: odr_url
+                  value: '{{=sprig.trim(workflow.parameters.odr_url)}}'
+                - name: category
+                  value: 'dem-{{=sprig.trim(workflow.parameters.hillshade_preset)}}'
+                - name: region
+                  value: 'new-zealand'
+                - name: gsd
+                  value: '{{=sprig.trim(workflow.parameters.gsd)}}'
+                - name: lifecycle
+                  value: 'ongoing'
+                - name: producer
+                  value: 'Toitū Te Whenua Land Information New Zealand'
+                - name: licensor
+                  value: 'Toitū Te Whenua Land Information New Zealand'
+                - name: create_capture_dates
+                  value: 'false'
+                - name: version_topo_imagery
+                  value: '{{= workflow.parameters.version_topo_imagery}}'
+            depends: 'generate-hillshade.Succeeded'
+
+          - name: stac-validate-all
+            templateRef:
+              name: tpl-at-stac-validate
+              template: main
+            arguments:
+              parameters:
+                - name: uri
+                  value: '{{=sprig.trimSuffix("/", tasks["get-location"].outputs.parameters.location)}}/{{=sprig.trim(workflow.parameters.hillshade_preset)}}/flat/collection.json'
+              artifacts:
+                - name: stac-result
+                  raw:
+                    data: '{{tasks.stac-validate-all.outputs.result}}'
+            depends: 'create-collection.Succeeded'
+            when: "'{{workflow.parameters.odr_url}}' == ''"
+
+          - name: stac-validate-only-updated
+            templateRef:
+              name: stac-validate-parallel # Needs to validate a list of STAC documents
+              template: main
+            arguments:
+              parameters:
+                - name: uri
+                  value: '{{=sprig.trimSuffix("/", tasks["get-location"].outputs.parameters.location)}}/{{=sprig.trim(workflow.parameters.hillshade_preset)}}/flat/'
+                - name: recursive
+                  value: 'false' # The Collection might have some Items that are only in the published location
+                - name: checksum_links
+                  value: 'false'
+                - name: checksum_assets
+                  value: 'true'
+                - name: include
+                  value: '\.json$' # FIXME: For now, assuming we have only STAC as JSON files in the scratch location
+                - name: concurrency
+                  value: '50'
+            depends: 'create-collection.Succeeded'
+            when: "'{{workflow.parameters.odr_url}}' != ''"
+
+          - name: create-config
+            arguments:
+              parameters:
+                - name: location
+                  value: '{{=sprig.trimSuffix("/", tasks["get-location"].outputs.parameters.location)}}/{{=sprig.trim(workflow.parameters.hillshade_preset)}}/flat/'
+                - name: bucket
+                  value: '{{tasks.get-location.outputs.parameters.bucket}}'
+                - name: key
+                  value: '{{tasks.get-location.outputs.parameters.key}}/{{=sprig.trim(workflow.parameters.hillshade_preset)}}/flat/'
+            template: create-config
+            depends: 'generate-hillshade.Succeeded'
+
+          - name: publish-odr
+            templateRef:
+              name: publish-odr
+              template: main
+            when: "'{{workflow.parameters.publish_to_odr}}' == 'true'"
+            arguments:
+              parameters:
+                - name: source
+                  value: '{{=sprig.trimSuffix("/", tasks["get-location"].outputs.parameters.location)}}/{{=sprig.trim(workflow.parameters.hillshade_preset)}}/flat/'
+                - name: target_bucket_name
+                  value: 'nz-elevation'
+                - name: copy_option
+                  value: '{{=sprig.trim(workflow.parameters.copy_option)}}'
+                - name: ticket
+                  value: '{{=sprig.trim(workflow.parameters.ticket)}}'
+            depends: '(stac-validate-all.Succeeded || stac-validate-only-updated.Succeeded) && create-config.Succeeded'
+
+      outputs:
+        parameters:
+          - name: target
+            valueFrom:
+              parameter: '{{tasks["get-location"].outputs.parameters.location}}'
+              default: ''
+
+    - name: create-config
+      inputs:
+        parameters:
+          - name: location
+            description: 'Location of the imagery to create config for'
+          - name: bucket
+          - name: key
+      container:
+        image: 'ghcr.io/linz/basemaps/cli:{{=sprig.trim(workflow.parameters.version_basemaps_cli)}}'
+        command: [node, /app/node_modules/@basemaps/cogify/dist/index.cjs]
+        env:
+          - name: AWS_ROLE_CONFIG_PATH
+            value: s3://linz-bucket-config/config.basemaps.json
+        args:
+          - 'config'
+          - '{{=sprig.trimSuffix("/", inputs.parameters.location)}}'
+      outputs:
+        parameters:
+          - name: url
+            description: 'Basemaps URL to view the imagery'
+            valueFrom:
+              path: '/tmp/cogify/config-url'
+          - name: config
+            description: 'Location of the config file'
+            valueFrom:
+              path: '/tmp/cogify/config-path'
+        artifacts:
+          - name: url
+            path: '/tmp/cogify/config-url'
+            s3:
+              bucket: '{{=sprig.trim(inputs.parameters.bucket)}}'
+              key: '{{=sprig.trimSuffix("/", inputs.parameters.key)}}/config-url'
+            archive:
+              none: {}
+
+    - name: exit-handler
+      retryStrategy:
+        limit: '0' # `tpl-exit-handler` retries itself
+      steps:
+        - - name: exit
+            templateRef:
+              name: tpl-exit-handler
+              template: main
+            arguments:
+              parameters:
+                - name: workflow_status
+                  value: '{{workflow.status}}'
+                - name: workflow_parameters
+                  value: '{{workflow.parameters}}'
+
+  volumes:
+    - name: ephemeral
+      emptyDir: {}


Reviews:
user: login: paulfouquet
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-18T22:33:19Z
user: login: paulfouquet
state: DISMISSED
author_association: COLLABORATOR
submitted_at: 2025-03-20T02:46:16Z
user: login: schmidtnz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-20T22:45:54Z
user: login: schmidtnz
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T00:40:26Z
user: login: paulfouquet
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-21T01:16:37Z
user: login: schmidtnz
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T03:01:03Z

Review comments:
diff_hunk: @@ -122,38 +122,36 @@ See [collection_from_items.py](https://github.com/linz/topo-imagery/blob/master/
 
 ## topo-imagery/generate-hillshade - `tpl-create-hillshade`
 
-Template for creating hillshades from TIFFs (DEM).  
+Template for creating hillshades from TIFFs (DEM).
 See [generate_hillshade.py](https://github.com/linz/topo-imagery/pull/1253)
-This `WorkflowTemplate` is also calling other templates in order to create the inputs of the `linz/topo-imagery/generate_hillshade.py` script:
-
-1. `tile-index-validate` to create a list of tiles with their source TIFF files
-2. `group` to split the list created into grouped tiles for parallel processing
-
-It allows generating multiple types of hillshade from the same source TIFF files, as the `preset` parameter expects a list of hillshade presets.
 
 ### Template usage
 
 ```yaml
-- name: create-hillshade
+- name: generate-hillshade
   templateRef:
-    name: tpl-create-hillshade
+    name: tpl-ti-generate-hillshade
     template: main
   arguments:
     parameters:
-      - name: source
-        value: 's3://nz-elevation/new-zealand/new-zealand-contour/dem_8m/2193/'
-      - name: scale
-        value: '50000'
-      - name: group
-        value: '2'
-      - name: version_argo_tasks
-        value: 'v4'
+      - name: group_id
+        value: '{{item}}'
+      - name: hillshade_preset
+        value: '{{=sprig.trim(workflow.parameters.hillshade_preset)}}'
       - name: version_topo_imagery
-        value: 'v7'
-      - name: target
-        value: '{{=sprig.trimSuffix("/", tasks["get-location"].outputs.parameters.location)}}/flat/'
-      - name: preset
-        value: '["greyscale", "igor"]'
+        value: '{{= workflow.parameters.version_topo_imagery}}'
+      - name: target # not using flat/ here, but {{workflow.parameters.hillshade_preset}}/ to keep temporary HS output separate
+        value: '{{=sprig.trimSuffix("/", tasks["get-location"].outputs.parameters.location)}}/{{workflow.parameters.hillshade_preset}}/flat/'
path: templates/topo-imagery/README.md
user: login: paulfouquet
body: The value does not match with the comment above. Is it intended to be `tasks["get-location"].outputs.parameters.location)}}/{{workflow.parameters.hillshade_preset}}/flat/` or `tasks["get-location"].outputs.parameters.location)}}/{{workflow.parameters.hillshade_preset}}/`?
created_at: 2025-03-18T22:33:19Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,322 @@
+# yaml-language-server: $schema=https://raw.githubusercontent.com/argoproj/argo-workflows/v3.5.5/api/jsonschema/schema.json
+
+apiVersion: argoproj.io/v1alpha1
+kind: WorkflowTemplate
+metadata:
+  name: national-hillshade-1m
+  labels:
+    linz.govt.nz/category: raster
+    linz.govt.nz/data-type: raster
+spec:
+  parallelism: 50
+  nodeSelector:
+    karpenter.sh/capacity-type: 'spot'
+  entrypoint: main
+  onExit: exit-handler
+  workflowMetadata:
+    labels:
+      linz.govt.nz/region: 'new-zealand'
+    labelsFrom:
+      linz.govt.nz/ticket:
+        expression: workflow.parameters.ticket
+  podMetadata:
+    labels:
+      linz.govt.nz/category: raster
+      linz.govt.nz/data-type: raster
+      linz.govt.nz/region: 'new-zealand'
+  arguments:
+    parameters:
+      - name: version_argo_tasks
+        description: 'Specify a version of the argo-tasks image to use, e.g. "v4.1" or "latest"'
+        value: 'v4'
+      - name: version_basemaps_cli
+        description: 'Specify a version of the basemaps-cli image to use, e.g. "v7.1" or "latest"'
+        value: 'v7'
+      - name: version_topo_imagery
+        description: 'Specify a version of the topo-imagery image to use, e.g. "v4.8" or "latest"'
+        value: 'v7'
+      - name: ticket
+        description: 'Ticket ID, e.g. "TDE-1130"'
+        value: ''
+      - name: source
+        description: 'Location of the input elevation data to create hillshade.'
+        value: 's3://nz-elevation/new-zealand/new-zealand/dem_1m/2193/'
+      - name: odr_url
+        description: '(Optional) If an existing dataset add the S3 path to the dataset here to load existing metadata e.g. "s3://nz-elevation/new-zealand/new-zealand/dem_1m/2193/"'
+        value: ''
+      - name: hillshade_preset
+        description: 'Hillshade preset to use, must be one of "hillshade" or "hillshade-igor"'
+        value: 'hillshade'
+      - name: group
+        description: 'How many output tiles to process in each standardise-validate task "pod". Change if you have resource or performance issues when standardising a dataset.'
+        value: '4'
+      - name: publish_to_odr
+        description: 'Create a Pull Request for publishing to imagery or elevation ODR bucket'
+        value: 'false'
+        enum:
+          - 'false'
+          - 'true'
+      - name: copy_option
+        description: 'Do not overwrite existing files with "no-clobber", "force" overwriting files in the target location, or "force-no-clobber" overwriting only changed files, skipping unchanged files'
+        value: '--force-no-clobber'
+        enum:
+          - '--no-clobber'
+          - '--force'
+          - '--force-no-clobber'
+  templateDefaults:
+    container:
+      imagePullPolicy: Always
+      image: ''
+  templates:
+    - name: main
+      retryStrategy:
+        expression: 'false'
+      dag:
+        tasks:
+          - name: get-location
+            templateRef:
+              name: tpl-get-location
+              template: main
+
+          - name: stac-setup-hillshade
+            templateRef:
+              name: tpl-at-stac-setup
+              template: main
+            arguments:
+              parameters:
+                - name: gsd
+                  value: '1'
+                - name: region
+                  value: 'new-zealand'
+                - name: geospatial_category
+                  value: 'dem-{{=sprig.trim(workflow.parameters.hillshade_preset)}}'
+                - name: odr_url
+                  value: '{{=sprig.trim(workflow.parameters.odr_url)}}'
+                - name: version
+                  value: '{{= workflow.parameters.version_argo_tasks}}'
+
+          - name: tile-index-validate
+            templateRef:
+              name: tpl-at-tile-index-validate
+              template: main
+            arguments:
+              parameters:
+                - name: scale
+                  value: '50000'
+                - name: include
+                  value: '.tiff?$'
+                - name: source
+                  value: '{{=sprig.trim(workflow.parameters.source)}}'
+                - name: source_epsg
+                  value: '2193'
+                - name: validate
+                  value: 'false'
+                - name: retile
+                  value: 'true'
+                - name: preset
+                  value: 'dem_lerc'
+                - name: version
+                  value: '{{= workflow.parameters.version_argo_tasks}}'
+                - name: includeDerived
+                  value: 'true'
+
+          - name: group
+            templateRef:
+              name: tpl-at-group
+              template: main
+            arguments:
+              artifacts:
+                - name: input
+                  from: '{{ tasks.tile-index-validate.outputs.artifacts.files }}'
+              parameters:
+                - name: size
+                  value: '{{=sprig.trim(workflow.parameters.group)}}'
+                - name: version
+                  value: '{{= workflow.parameters.version_argo_tasks}}'
+            depends: 'tile-index-validate.Succeeded'
+
+          - name: generate-hillshade
+            templateRef:
+              name: tpl-ti-generate-hillshade
+              template: main
+            arguments:
+              parameters:
+                - name: group_id
+                  value: '{{item}}'
+                - name: hillshade_preset
+                  value: '{{=sprig.trim(workflow.parameters.hillshade_preset)}}'
+                - name: version_topo_imagery
+                  value: '{{= workflow.parameters.version_topo_imagery}}'
+                - name: target
+                  value: '{{=sprig.trimSuffix("/", tasks["get-location"].outputs.parameters.location)}}/flat/'
path: workflows/raster/national-hillshade-1m.yaml
user: login: schmidtnz
body: inconsistent with other locations in this file where we (still) use subdirectories
created_at: 2025-03-20T22:45:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -294,9 +295,9 @@ See the [copy template](#copy) for more information.
 
 # national-dem
 
-This workflow combines a set of DEMs datasets in order to create a single national dataset composed of 1:50k tiles.
+This workflow combines a set of DEM datasets in order to create a single national dataset composed of 1:50k tiles.
 
-Upon completion all standardised TIFF and STAC files will be located with the ./flat/ directory of the workflow in the artifacts scratch bucket. In addition, a Basemaps link is produced enabling visual QA.
+Upon completion all standardised TIFF and STAC files will be located in the ./flat/ directory of the workflow in the artifacts scratch bucket. In addition, a Basemaps link is produced enabling visual QA.
path: workflows/raster/README.md
user: login: schmidtnz
body: ```suggestion
Upon completion all standardised TIFF and STAC files will be located in the ./{{hillshade_preset}}/flat/ directory of the workflow in the artifacts scratch bucket. In addition, a Basemaps link is produced enabling visual QA.
```
created_at: 2025-03-21T00:34:40Z
author_association: CONTRIBUTOR

PR comments:
user: login: paulfouquet
created_at: 2025-03-20T22:53:41Z
body: @schmidtnz what are your thoughts about making this template more generic than nation 1m hillshade, but allow to create hillshade from whatever kind of source? 
author_association: COLLABORATOR
user: login: schmidtnz
created_at: 2025-03-20T22:59:17Z
body: > @schmidtnz what are your thoughts about making this template more generic than nation 1m hillshade, but allow to create hillshade from whatever kind of source?

I was thinking about this too, (i.e. changing from `hillshade_preset` to `geospatial_category`) especially as DSM work will be upcoming.

Considering this has been tested as it is now, I'd rather publish and refactor when we look at DSM.
author_association: CONTRIBUTOR