Delivering Fine-grained Environmental Spatial Data in HPC and Cloud Converged Infrastructure PI: Michela Taufer*, Co-PI: Rodrigo Vargas↟ Project Contributors: Paula Olaya*, Camila Roa*, Jakob Luettgau*, Ricardo Llamas↟, Sophia Wen‡, I-Hsin Chung‡, Seetharami Seelam‡, Yoonho Park‡, and Jay Lofstead† *University of Tennessee, Knoxville,↟University of Delaware, ‡IBM, †Sandia National Laboratory Generating Fine-grained Environmental Spatial Data ●Environmental spatial data (i.e., terrain parameters and soil moisture) provide information about phenomena on the surface of the earth ●These data are fundamental in applications such as precision forestry and agriculture, identifying biodiversity loss, hydrology for landscape ecology, understanding vegetation changes, wildfire detection, and spotting at-risk ecosystems1 [1] Jennifer Bernstein, Karen Kemp; The Role of Spatial Science in Environmental Case Studies: A Special Collection from the University of Southern California. Case Studies in the Environment 1 January 2020; 4 (1): 1–5. Generating environmental spatial data at high-resolution is computationally expensive, hindering scientific communities’ ability to access the data Digital Elevation Model Satellite Soil Moisture Data Generation Terrain Parameters Data Transformation Training Data Testing Data ML- Training ML- Inference ML-Model Soil Moisture Predictions Analysis/ Visual Geographic statistics/ maps Terrain Parameter Generation (GEOtiled) Data Transformation ML-Prediction Soil Moisture We develop a workflow called SOMOSPIE (SOil MOisture SPatial Inference Engine) that generates fine-grained terrain parameters and soil moisture Analysis/ Visualization HPC and Cloud Convergence ●Our workflow comprises four stages: a terrain parameter generation, a data transformation, an ML-based prediction, and Analysis/Visualization ●Each of the stages leverages advanced computational techniques to transform data and generate ready-for-use environmental spatial data Our solution scales performance and enables scientific communities to access our end-to-end workflow and data on converged HPC and Cloud infrastructure Soil Moisture Maps @ <1 km Elevation models @ <1 km Terrain P. Generation Data Transformation ML Predict Analysis Visualization Satellite Soil Moisture @ 27 km Precision agriculture Environm. sciences Wildfire Detection SOMOSPIE: SOil MOisture SPatial Inference Engine Train1 Hybrid HPC-Cloud Cluster Train2 Train3 TrainN Inference Inference Inference Inference def train(): … def inference(model): … dsl.pipeline(ml-model): for i in N_years: model=train() inference(model) TektonCompiler(). compile(pipeline, “pipeline.yaml”) Inference Inference Inference Inference Inference Inference Inference Inference Inference Inference … … Predicted Soil Moisture1 Predicted Soil Moisture2 Predicted Soil Moisture3 Predicted Soil MoistureN … Open Data Hub Pipelines We use Open Data Hub Pipelines to orchestrate the execution of our workflow in cloud-native clusters (i.e., Kubernetes and OpenShift) with Cloud Object Storage The authors acknowledge the support of the National Science Foundation through the awards 1841758, 2103845, 2138811, and 1941443 Reproducibility efforts Paper Github Cloud-HPC Integration SOMOSPIE’s workflow Paper Github Trustworthiness and Reproducibility Our solution ensures trustworthiness and reproducibility in the data, software, and environment, wrapped in the scientific discovery loop ●We create a fine-grained containerized environment that automatically provides provenance information and seamlessly attaches it to the workflow components ●The containerization supports easy retrieval for reusability, reproducibility, and portability of the workflow