Repository: bcgov/wps
Pull request #4417: SFMS Calculation datatypes/max value
Author: login: brettedw
Created: 2025-04-07T18:52:08Z | Merged: 2025-04-09T22:45:14Z

- Removes `datatype` arg from `WPSDataset`. Using gdal's `band.DataType` to use same datatype as existing dataset
- Adds logging for weather parameters to check for invalid values as defined by CFFDRS. This is useful because when doing vectorized calculations we don't get much in terms of error messaging
- Adds `max_value` to `warp_to_match`, this ensures we can set a max value when resampling is being done so that we don't get out of range values (ie RH > 100)
# Test Links:
[Landing Page](https://wps-pr-4417-e1e498-dev.apps.silver.devops.gov.bc.ca/)
[MoreCast](https://wps-pr-4417-e1e498-dev.apps.silver.devops.gov.bc.ca/morecast)
[Percentile Calculator](https://wps-pr-4417-e1e498-dev.apps.silver.devops.gov.bc.ca/percentile-calculator)
[C-Haines](https://wps-pr-4417-e1e498-dev.apps.silver.devops.gov.bc.ca/c-haines)
[FireBat](https://wps-pr-4417-e1e498-dev.apps.silver.devops.gov.bc.ca/fire-behaviour-calculator)
[FireBat bookmark](https://wps-pr-4417-e1e498-dev.apps.silver.devops.gov.bc.ca/fire-behaviour-calculator?s=266&f=c5&c=NaN&w=20,s=286&f=c7&c=NaN&w=16,s=1055&f=c7&c=NaN&w=NaN,s=305&f=c7&c=NaN&w=NaN,s=344&f=c5&c=NaN&w=NaN,s=346&f=c7&c=NaN&w=NaN,s=328&f=c7&c=NaN&w=NaN,s=1399&f=c7&c=NaN&w=NaN,s=334&f=c7&c=NaN&w=NaN,s=1082&f=c3&c=NaN&w=NaN,s=388&f=c7&c=NaN&w=NaN,s=309&f=c7&c=NaN&w=16,s=306&f=c7&c=NaN&w=NaN,s=1029&f=c7&c=NaN&w=NaN,s=298&f=c7&c=NaN&w=NaN,s=836&f=c7&c=NaN&w=NaN,s=9999&f=c7&c=NaN&w=NaN)
[Auto Spatial Advisory (ASA)](https://wps-pr-4417-e1e498-dev.apps.silver.devops.gov.bc.ca/auto-spatial-advisory)
[HFI Calculator](https://wps-pr-4417-e1e498-dev.apps.silver.devops.gov.bc.ca/hfi-calculator)
[SFMS Insights](https://wps-pr-4417-e1e498-dev.apps.silver.devops.gov.bc.ca/insights)



Diff:
diff --git a/api/app/sfms/daily_fwi_processor.py b/api/app/sfms/daily_fwi_processor.py
index 8fa113fa5b..4f6cd9be01 100644
--- a/api/app/sfms/daily_fwi_processor.py
+++ b/api/app/sfms/daily_fwi_processor.py
@@ -50,6 +50,7 @@ async def process(
             if not weather_keys_exist:
                 logging.warning(f"Missing weather keys for {model_run_for_hour(self.start_datetime.hour):02} model run")
                 break
+            logger.info(f"Found weather keys: {"\n".join([temp_key, rh_key, wind_speed_key, precip_key])}")
 
             # get and check existence of fwi s3 keys
             dc_key, dmc_key, ffmc_key = self._get_previous_fwi_keys(day, previous_fwi_datetime)
@@ -57,6 +58,7 @@ async def process(
             if not fwi_keys_exist:
                 logging.warning(f"No previous DMC/DC/FFMC keys found for {previous_fwi_datetime.date().isoformat()}")
                 break
+            logger.info(f"Found FWI keys: {"\n".join([dc_key, dmc_key, ffmc_key])}")
 
             temp_key, rh_key, wind_speed_key, precip_key, ffmc_key = self.addresser.gdal_prefix_keys(temp_key, rh_key, wind_speed_key, precip_key, ffmc_key)
             dc_key, dmc_key = self.addresser.gdal_prefix_keys(dc_key, dmc_key)
@@ -67,10 +69,10 @@ async def process(
                     temp_ds, rh_ds, wind_speed_ds, precip_ds, dc_ds, dmc_ds, ffmc_ds = input_datasets
 
                     # Warp weather datasets to match fwi
-                    warped_temp_ds = temp_ds.warp_to_match(dmc_ds, f"{temp_dir}/{os.path.basename(temp_key)}", GDALResamplingMethod.BILINEAR)
-                    warped_rh_ds = rh_ds.warp_to_match(dmc_ds, f"{temp_dir}/{os.path.basename(rh_key)}", GDALResamplingMethod.BILINEAR)
-                    warped_wind_speed_ds = wind_speed_ds.warp_to_match(dmc_ds, f"{temp_dir}/{os.path.basename(wind_speed_key)}", GDALResamplingMethod.BILINEAR)
-                    warped_precip_ds = precip_ds.warp_to_match(dmc_ds, f"{temp_dir}/{os.path.basename(precip_key)}", GDALResamplingMethod.BILINEAR)
+                    warped_temp_ds = temp_ds.warp_to_match(dmc_ds, f"{temp_dir}/{os.path.basename(temp_key)}.tif", GDALResamplingMethod.BILINEAR)
+                    warped_rh_ds = rh_ds.warp_to_match(dmc_ds, f"{temp_dir}/{os.path.basename(rh_key)}.tif", GDALResamplingMethod.BILINEAR, max_value=100)
+                    warped_wind_speed_ds = wind_speed_ds.warp_to_match(dmc_ds, f"{temp_dir}/{os.path.basename(wind_speed_key)}.tif", GDALResamplingMethod.BILINEAR)
+                    warped_precip_ds = precip_ds.warp_to_match(dmc_ds, f"{temp_dir}/{os.path.basename(precip_key)}.tif", GDALResamplingMethod.BILINEAR)
 
                     # close unneeded datasets to reduce memory usage
                     precip_ds.close()
diff --git a/api/app/sfms/fwi_processor.py b/api/app/sfms/fwi_processor.py
index ca6cdf9d93..5f73c59239 100644
--- a/api/app/sfms/fwi_processor.py
+++ b/api/app/sfms/fwi_processor.py
@@ -1,5 +1,6 @@
 import logging
 from time import perf_counter
+from typing import Optional
 
 import numpy as np
 
@@ -9,12 +10,36 @@
 logger = logging.getLogger(__name__)
 
 
+def check_weather_values(rh_array: Optional[np.ndarray] = None, precip_array: Optional[np.ndarray] = None, ws_array: Optional[np.ndarray] = None) -> None:
+    """
+    Logs errors if any weather condition values are out of acceptable ranges.
+
+    :param rh_array: Optional array of relative humidity values (0–100).
+    :param prec_array: Optional array of precipitation values (>= 0).
+    :param ws_array: Optional array of wind speed values (>= 0).
+    """
+    if rh_array is not None and ((rh_array < 0).any() or (rh_array > 100).any()):
+        min_rh = rh_array.min()
+        max_rh = rh_array.max()
+        logger.error(f"Relative humidity values out of bounds (min: {min_rh}, max: {max_rh})")
+
+    if precip_array is not None and (precip_array < 0).any():
+        min_precip = precip_array.min()
+        logger.error(f"Precipitation contains negative values (min: {min_precip})")
+
+    if ws_array is not None and (ws_array < 0).any():
+        min_ws = ws_array.min()
+        logger.error(f"Wind speed contains negative values (min: {min_ws})")
+
+
 def calculate_dc(dc_ds: WPSDataset, temp_ds: WPSDataset, rh_ds: WPSDataset, precip_ds: WPSDataset, latitude: np.ndarray, month: np.ndarray):
     dc_array, _ = dc_ds.replace_nodata_with(0)
     temp_array, _ = temp_ds.replace_nodata_with(0)
     rh_array, _ = rh_ds.replace_nodata_with(0)
     precip_array, _ = precip_ds.replace_nodata_with(0)
 
+    check_weather_values(rh_array, precip_array)
+
     start = perf_counter()
     dc_values = vectorized_dc(dc_array, temp_array, rh_array, precip_array, latitude, month, True)
     logger.info("%f seconds to calculate vectorized dc", perf_counter() - start)
@@ -32,6 +57,8 @@ def calculate_dmc(dmc_ds: WPSDataset, temp_ds: WPSDataset, rh_ds: WPSDataset, pr
     rh_array, _ = rh_ds.replace_nodata_with(0)
     precip_array, _ = precip_ds.replace_nodata_with(0)
 
+    check_weather_values(rh_array, precip_array)
+
     start = perf_counter()
     dmc_values = vectorized_dmc(dmc_array, temp_array, rh_array, precip_array, latitude, month, True)
     logger.info("%f seconds to calculate vectorized dmc", perf_counter() - start)
@@ -65,9 +92,7 @@ def calculate_ffmc(previous_ffmc_ds: WPSDataset, temp_ds: WPSDataset, rh_ds: WPS
     precip_array, _ = precip_ds.replace_nodata_with(0)
     wind_speed_array, _ = wind_speed_ds.replace_nodata_with(0)
 
-    # Due to warping of the rh dataset, rh values can exceed 100 which breaks the ffmc calculation.
-    # Set rh values greater than 100 to the max allowable which is 100.
-    rh_array[rh_array > 100] = 100
+    check_weather_values(rh_array, precip_array, wind_speed_array)
 
     start = perf_counter()
     ffmc_values = vectorized_ffmc(previous_ffmc_array, temp_array, rh_array, wind_speed_array, precip_array)
diff --git a/api/app/sfms/hourly_ffmc_processor.py b/api/app/sfms/hourly_ffmc_processor.py
index 92ffd3d7c2..596a022856 100644
--- a/api/app/sfms/hourly_ffmc_processor.py
+++ b/api/app/sfms/hourly_ffmc_processor.py
@@ -68,10 +68,10 @@ async def process(self, s3_client: S3Client, input_dataset_context: MultiDataset
                     input_datasets = cast(List[WPSDataset], input_datasets)  # Ensure correct type inference
                     temp_ds, rh_ds, wind_speed_ds, precip_ds, hffmc_ds = input_datasets
                     # Warp weather datasets to match hffmc
-                    warped_temp_ds = temp_ds.warp_to_match(hffmc_ds, f"{temp_dir}/{os.path.basename(temp_key)}", GDALResamplingMethod.BILINEAR)
-                    warped_rh_ds = rh_ds.warp_to_match(hffmc_ds, f"{temp_dir}/{os.path.basename(rh_key)}", GDALResamplingMethod.BILINEAR)
-                    warped_wind_speed_ds = wind_speed_ds.warp_to_match(hffmc_ds, f"{temp_dir}/{os.path.basename(wind_speed_key)}", GDALResamplingMethod.BILINEAR)
-                    warped_precip_ds = precip_ds.warp_to_match(hffmc_ds, f"{temp_dir}/{os.path.basename(precip_key)}", GDALResamplingMethod.BILINEAR)
+                    warped_temp_ds = temp_ds.warp_to_match(hffmc_ds, f"{temp_dir}/{os.path.basename(temp_key)}.tif", GDALResamplingMethod.BILINEAR)
+                    warped_rh_ds = rh_ds.warp_to_match(hffmc_ds, f"{temp_dir}/{os.path.basename(rh_key)}.tif", GDALResamplingMethod.BILINEAR, max_value=100)
+                    warped_wind_speed_ds = wind_speed_ds.warp_to_match(hffmc_ds, f"{temp_dir}/{os.path.basename(wind_speed_key)}.tif", GDALResamplingMethod.BILINEAR)
+                    warped_precip_ds = precip_ds.warp_to_match(hffmc_ds, f"{temp_dir}/{os.path.basename(precip_key)}.tif", GDALResamplingMethod.BILINEAR)
 
                     # Create and store new hFFMC dataset
                     hffmc_values, hffmc_no_data_value = calculate_ffmc(hffmc_ds, warped_temp_ds, warped_rh_ds, warped_precip_ds, warped_wind_speed_ds)
diff --git a/api/app/tests/geospatial/test_wps_dataset.py b/api/app/tests/geospatial/test_wps_dataset.py
index 0c717562f7..431bf9b311 100644
--- a/api/app/tests/geospatial/test_wps_dataset.py
+++ b/api/app/tests/geospatial/test_wps_dataset.py
@@ -7,7 +7,7 @@
 from wps_shared.geospatial.wps_dataset import WPSDataset, multi_wps_dataset_context
 from app.tests.dataset_common import create_mock_gdal_dataset, create_test_dataset
 
-hfi_tif = os.path.join(os.path.dirname(__file__), "snow_masked_hfi20240810.tif")
+hfi_tif = os.path.join(os.path.dirname(__file__), "snow_masked_hfi20240810.tif")  # Byte data
 zero_tif = os.path.join(os.path.dirname(__file__), "zero_layer.tif")
 
 
@@ -43,8 +43,11 @@ def test_raster_mul():
     with WPSDataset(hfi_tif) as wps_ds, WPSDataset(zero_tif) as zero_ds:
         output_ds = wps_ds * zero_ds
         raw_ds = output_ds.as_gdal_ds()
-        output_values = raw_ds.GetRasterBand(1).ReadAsArray()
+        output_band = raw_ds.GetRasterBand(1)
+        output_values = output_band.ReadAsArray()
+        output_datatype = output_band.DataType
         assert np.all(output_values == 0)
+        assert output_datatype == gdal.GDT_Byte
 
 
 def test_raster_mul_identity():
@@ -119,6 +122,33 @@ def test_raster_warp():
     mercator_ds = None
 
 
+def test_raster_warp_max_value():
+    # Dataset 1: 100x100 pixels, extent in EPSG:3857
+    extent1 = (-20037508.34, 20037508.34, -20037508.34, 20037508.34)
+    wgs_84_ds = create_test_dataset("test_dataset_1.tif", 100, 100, extent1, 3857, fill_value=90)
+
+    band = wgs_84_ds.GetRasterBand(1)
+    array = band.ReadAsArray()
+    array[0, 0] = 101  # value to be clamped
+    band.WriteArray(array)
+    band.FlushCache()
+
+    # Dataset 2: 100x100 pixels, extent in EPSG:3857
+    extent2 = (-20037508.34, 20037508.34, -20037508.34, 20037508.34)
+    mercator_ds = create_test_dataset("test_dataset_2.tif", 100, 100, extent2, 3857)
+
+    with WPSDataset(ds_path=None, ds=wgs_84_ds) as wps1_ds, WPSDataset(ds_path=None, ds=mercator_ds) as wps2_ds:
+        output_ds: WPSDataset = wps1_ds.warp_to_match(wps2_ds, "/vsimem/test.grib2", max_value=100)  # test that we can update an output path with any extension
+        out_array = output_ds.as_gdal_ds().GetRasterBand(1).ReadAsArray()
+        assert out_array.max() == 100
+
+        # Ensure 90 stayed 90 everywhere since we're doing Nearest Neighbour interp. 100*100 array minus the 1 value we changed
+        assert np.count_nonzero(out_array == 90) == (100 * 100 - 1), "Expected at least one value to remain 99"
+
+    wgs_84_ds = None
+    mercator_ds = None
+
+
 def test_export_to_geotiff():
     extent1 = (-1, 1, -1, 1)  # xmin, xmax, ymin, ymax
     ds_1 = create_test_dataset("test_dataset_1.tif", 3, 3, extent1, 4326, data_type=gdal.GDT_Byte, fill_value=1)
@@ -180,7 +210,7 @@ def test_get_nodata_mask_empty():
 
 def test_from_array():
     extent1 = (-1, 1, -1, 1)  # xmin, xmax, ymin, ymax
-    original_ds = create_test_dataset("test_dataset_1.tif", 100, 100, extent1, 4326)
+    original_ds = create_test_dataset("test_dataset_1.tif", 100, 100, extent1, 4326)  # float32 datatype
     original_ds.GetRasterBand(1).SetNoDataValue(-99)
     og_band = original_ds.GetRasterBand(1)
     og_array = og_band.ReadAsArray()
diff --git a/api/app/tests/sfms/test_daily_fwi_processor.py b/api/app/tests/sfms/test_daily_fwi_processor.py
index da9041aac1..18187230a1 100644
--- a/api/app/tests/sfms/test_daily_fwi_processor.py
+++ b/api/app/tests/sfms/test_daily_fwi_processor.py
@@ -24,7 +24,7 @@ async def test_daily_fwi_processor(mocker: MockerFixture):
     get_weather_data_key_spy = mocker.spy(mock_key_addresser, "get_weather_data_keys")
     gdal_prefix_keys_spy = mocker.spy(mock_key_addresser, "gdal_prefix_keys")
     get_calculated_index_key_spy = mocker.spy(mock_key_addresser, "get_calculated_index_key")
-    
+
     fwi_processor = DailyFWIProcessor(TEST_DATETIME, 2, mock_key_addresser)
 
     # mock weather index, param datasets used for calculations
@@ -127,8 +127,8 @@ async def test_daily_fwi_processor(mocker: MockerFixture):
         ]
 
         assert rh_ds_spy.call_args_list == [
-            mocker.call(mock_dmc_ds, mocker.ANY, GDALResamplingMethod.BILINEAR),
-            mocker.call(mock_dmc_ds, mocker.ANY, GDALResamplingMethod.BILINEAR),
+            mocker.call(mock_dmc_ds, mocker.ANY, GDALResamplingMethod.BILINEAR, max_value=100),
+            mocker.call(mock_dmc_ds, mocker.ANY, GDALResamplingMethod.BILINEAR, max_value=100),
         ]
 
         assert wind_speed_ds_spy.call_args_list == [
@@ -159,7 +159,6 @@ async def test_daily_fwi_processor(mocker: MockerFixture):
             wps_datasets = ffmc_calls[0][1:4]  # Extract dataset arguments
             assert all(isinstance(ds, WPSDataset) for ds in wps_datasets)
 
-
         assert calculate_bui_spy.call_args_list == [
             mocker.call(mock_new_dmc_ds, mock_new_dc_ds),
             mocker.call(mock_new_dmc_ds, mock_new_dc_ds),
diff --git a/api/app/tests/sfms/test_fwi_processor.py b/api/app/tests/sfms/test_fwi_processor.py
index 5015270782..6c79a305f6 100644
--- a/api/app/tests/sfms/test_fwi_processor.py
+++ b/api/app/tests/sfms/test_fwi_processor.py
@@ -1,5 +1,7 @@
+import logging
 import math
 from dataclasses import dataclass
+from unittest.mock import patch
 
 import numpy as np
 import pytest
@@ -7,7 +9,8 @@
 from osgeo import osr
 
 from wps_shared.geospatial.wps_dataset import WPSDataset
-from app.sfms.fwi_processor import calculate_bui, calculate_dc, calculate_dmc, calculate_ffmc, calculate_fwi, calculate_isi
+from app.sfms.fwi_processor import check_weather_values, calculate_bui, calculate_dc, calculate_dmc, calculate_ffmc, calculate_fwi, calculate_isi
+
 
 FWI_ARRAY = np.array([[12, 20], [-999, -999]])
 WEATHER_ARRAY = np.array([[12, 20], [0, 0]])
@@ -243,3 +246,71 @@ def test_calculate_fwi_values(input_datasets):
     static_fwi = fwi(isi_sample, bui_sample)
 
     assert math.isclose(static_fwi, fwi_values[0, 0], abs_tol=0.01)
+
+
+@pytest.mark.parametrize(
+    "rh_array, expected",
+    [
+        (np.array([50, 60, 70]), False),
+        (np.array([-5, 50, 110]), True),
+    ],
+)
+def test_check_rh_logging(rh_array, expected):
+    with patch("app.sfms.fwi_processor.logger") as mock_logger:
+        check_weather_values(rh_array=rh_array)
+        if expected:
+            mock_logger.error.assert_called_once()
+            assert "Relative humidity" in mock_logger.error.call_args[0][0]
+        else:
+            mock_logger.error.assert_not_called()
+
+
+@pytest.mark.parametrize(
+    "precip_array, expected",
+    [
+        (np.array([0.0, 5.0, 10.0]), False),
+        (np.array([-1.0, 0.0, 2.0]), True),
+    ],
+)
+def test_check_prec_logging(precip_array, expected):
+    with patch("app.sfms.fwi_processor.logger") as mock_logger:
+        check_weather_values(precip_array=precip_array)
+        if expected:
+            mock_logger.error.assert_called_once()
+            assert "Precipitation" in mock_logger.error.call_args[0][0]
+        else:
+            mock_logger.error.assert_not_called()
+
+
+@pytest.mark.parametrize(
+    "ws_array, expected",
+    [
+        (np.array([0.0, 3.2, 7.5]), False),
+        (np.array([-0.1, 1.0, 2.0]), True),
+    ],
+)
+def test_check_ws_logging(ws_array, expected):
+    with patch("app.sfms.fwi_processor.logger") as mock_logger:
+        check_weather_values(ws_array=ws_array)
+        if expected:
+            mock_logger.error.assert_called_once()
+            assert "Wind speed" in mock_logger.error.call_args[0][0]
+        else:
+            mock_logger.error.assert_not_called()
+
+
+def test_check_multiple_issues():
+    rh_array = np.array([-10, 105])
+    precip_array = np.array([-2.5])
+    ws_array = np.array([-1.0])
+
+    with patch("app.sfms.fwi_processor.logger") as mock_logger:
+        check_weather_values(rh_array=rh_array, precip_array=precip_array, ws_array=ws_array)
+
+        error_calls = [call.args[0] for call in mock_logger.error.call_args_list]
+
+        assert any("Relative humidity values out of bounds" in msg for msg in error_calls)
+        assert any("Precipitation contains negative values" in msg for msg in error_calls)
+        assert any("Wind speed contains negative values" in msg for msg in error_calls)
+
+        assert mock_logger.error.call_count == 3
diff --git a/api/app/tests/sfms/test_hourly_ffmc_processor.py b/api/app/tests/sfms/test_hourly_ffmc_processor.py
index e9a7f23624..5821a953a4 100644
--- a/api/app/tests/sfms/test_hourly_ffmc_processor.py
+++ b/api/app/tests/sfms/test_hourly_ffmc_processor.py
@@ -128,8 +128,8 @@ async def test_hourly_ffmc_processor(mocker: MockerFixture):
         ]
 
         assert rh_ds_spy.call_args_list == [
-            mocker.call(mock_hffmc_ds, mocker.ANY, GDALResamplingMethod.BILINEAR),
-            mocker.call(mock_hffmc_ds, mocker.ANY, GDALResamplingMethod.BILINEAR),
+            mocker.call(mock_hffmc_ds, mocker.ANY, GDALResamplingMethod.BILINEAR, max_value=100),
+            mocker.call(mock_hffmc_ds, mocker.ANY, GDALResamplingMethod.BILINEAR, max_value=100),
         ]
 
         assert wind_speed_ds_spy.call_args_list == [
@@ -149,4 +149,4 @@ async def test_hourly_ffmc_processor(mocker: MockerFixture):
             assert all(isinstance(ds, WPSDataset) for ds in wps_datasets)
 
         # 1 hffmc per day
-        assert persist_raster_spy.call_count == 2
\ No newline at end of file
+        assert persist_raster_spy.call_count == 2
diff --git a/api/app/weather_models/precip_rdps_model.py b/api/app/weather_models/precip_rdps_model.py
index e45a7fadf6..81c226e0e6 100644
--- a/api/app/weather_models/precip_rdps_model.py
+++ b/api/app/weather_models/precip_rdps_model.py
@@ -4,7 +4,7 @@
 import logging
 import numpy
 import tempfile
-from osgeo import gdal, ogr
+from osgeo import gdal
 from wps_shared import config
 from numba import vectorize
 from wps_shared.utils.s3 import get_client, read_into_memory
diff --git a/wps_shared/wps_shared/geospatial/wps_dataset.py b/wps_shared/wps_shared/geospatial/wps_dataset.py
index e00a3d7c32..32905d986f 100644
--- a/wps_shared/wps_shared/geospatial/wps_dataset.py
+++ b/wps_shared/wps_shared/geospatial/wps_dataset.py
@@ -13,17 +13,17 @@ class WPSDataset:
     A wrapper around gdal datasets for common operations
     """
 
-    def __init__(self, ds_path: Optional[str], ds=None, band=1, chunk_size=256, access=gdal.GA_ReadOnly, datatype=gdal.GDT_Byte):
+    def __init__(self, ds_path: Optional[str], ds=None, band: int = 1, chunk_size: int = 256, access=gdal.GA_ReadOnly):
         self.ds = ds
         self.ds_path = ds_path
         self.band = band
         self.chunk_size = chunk_size
         self.access = access
-        self.datatype = datatype
 
     def __enter__(self):
         if self.ds is None:
             self.ds: gdal.Dataset = gdal.Open(self.ds_path, self.access)
+
         return self
 
     def __exit__(self, *_):
@@ -68,7 +68,7 @@ def from_array(
         # Flush cache to ensure all data is written
         output_band.FlushCache()
 
-        return cls(ds_path=None, ds=output_dataset, datatype=datatype)
+        return cls(ds_path=None, ds=output_dataset)
 
     def __mul__(self, other):
         """
@@ -98,17 +98,19 @@ def __mul__(self, other):
         if geotransform[0] != other.ds.GetGeoTransform()[0] or geotransform[3] != other.ds.GetGeoTransform()[3]:
             raise ValueError("The origins of the two rasters do not match.")
 
+        self_band: gdal.Band = self.ds.GetRasterBand(self.band)
+        other_band: gdal.Band = other.ds.GetRasterBand(self.band)
+
+        datatype = self_band.DataType
+
         # Create the output raster
         driver: gdal.Driver = gdal.GetDriverByName("MEM")
-        out_ds: gdal.Dataset = driver.Create("memory", x_size, y_size, 1, self.datatype)
+        out_ds: gdal.Dataset = driver.Create("memory", x_size, y_size, 1, datatype)
 
         # Set the geotransform and projection
         out_ds.SetGeoTransform(geotransform)
         out_ds.SetProjection(projection)
 
-        self_band: gdal.Band = self.ds.GetRasterBand(self.band)
-        other_band: gdal.Band = other.ds.GetRasterBand(self.band)
-
         # Process in chunks
         for y in range(0, y_size, self.chunk_size):
             y_chunk_size = min(self.chunk_size, y_size - y)
@@ -137,7 +139,13 @@ def __mul__(self, other):
 
         return WPSDataset(ds_path=None, ds=out_ds)
 
-    def warp_to_match(self, other: "WPSDataset", output_path: str, resample_method: GDALResamplingMethod = GDALResamplingMethod.NEAREST_NEIGHBOUR):
+    def warp_to_match(
+        self,
+        other: "WPSDataset",
+        output_path: str,
+        resample_method: GDALResamplingMethod = GDALResamplingMethod.NEAREST_NEIGHBOUR,
+        max_value: float | None = None,
+    ):
         """
         Warp the dataset to match the extent, pixel size, and projection of the other dataset.
 
@@ -155,6 +163,10 @@ def warp_to_match(self, other: "WPSDataset", output_path: str, resample_method:
         miny = maxy + dest_geotransform[5] * other.ds.RasterYSize
         extent = [minx, miny, maxx, maxy]
 
+        # we need the output to be a geotiff, since we cannot update grib files
+        if not output_path.endswith(".tif"):
+            output_path += ".tif"
+
         # Warp to match input option parameters
         warped_ds = gdal.Warp(
             output_path,
@@ -167,6 +179,15 @@ def warp_to_match(self, other: "WPSDataset", output_path: str, resample_method:
                 resampleAlg=resample_method.value,
             ),
         )
+
+        if max_value is not None and warped_ds is not None:
+            band = warped_ds.GetRasterBand(1)
+            array = band.ReadAsArray()
+            if (array > max_value).any():
+                array = np.minimum(array, max_value)  # clamp any value above the max_value to the max_value
+                band.WriteArray(array)
+                band.FlushCache()
+
         return WPSDataset(ds_path=None, ds=warped_ds)
 
     def replace_nodata_with(self, new_no_data_value: int = 0):
@@ -229,11 +250,12 @@ def export_to_geotiff(self, output_path: str):
         projection = self.ds.GetProjection()
 
         band: gdal.Band = self.ds.GetRasterBand(self.band)
+        datatype = band.DataType
         nodata_value = band.GetNoDataValue()
         array = band.ReadAsArray()
 
         rows, cols = array.shape
-        output_dataset: gdal.Dataset = driver.Create(output_path, cols, rows, 1, self.datatype)
+        output_dataset: gdal.Dataset = driver.Create(output_path, cols, rows, 1, datatype)
         output_dataset.SetGeoTransform(geotransform)
         output_dataset.SetProjection(projection)
 


Reviews:
user: login: conbrad
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-07T19:00:15Z
user: login: dgboss
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-07T19:57:23Z
user: login: conbrad
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-07T21:01:41Z
user: login: brettedw
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-07T21:14:35Z
user: login: dgboss
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-07T22:49:07Z
user: login: dgboss
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-07T22:58:37Z
user: login: conbrad
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-07T22:59:35Z
user: login: dgboss
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-07T23:01:55Z
user: login: brettedw
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-07T23:09:21Z
user: login: conbrad
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-07T23:18:55Z
user: login: brettedw
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-07T23:59:15Z
user: login: conbrad
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-08T00:10:03Z
user: login: dgboss
body: Looks great!
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-09T18:11:23Z

Review comments:
diff_hunk: @@ -15,6 +15,10 @@ def calculate_dc(dc_ds: WPSDataset, temp_ds: WPSDataset, rh_ds: WPSDataset, prec
     rh_array, _ = rh_ds.replace_nodata_with(0)
     precip_array, _ = precip_ds.replace_nodata_with(0)
 
+    # Due to warping of the rh dataset, rh values can exceed 100 which breaks the calculation.
path: api/app/sfms/fwi_processor.py
user: login: conbrad
body: I wonder if we should do this correction in `warp_to_match` to keep it in one place. Perhaps it's an optional parameter only applied to rh.
created_at: 2025-04-07T21:01:38Z
author_association: COLLABORATOR
diff_hunk: @@ -15,6 +15,10 @@ def calculate_dc(dc_ds: WPSDataset, temp_ds: WPSDataset, rh_ds: WPSDataset, prec
     rh_array, _ = rh_ds.replace_nodata_with(0)
     precip_array, _ = precip_ds.replace_nodata_with(0)
 
+    # Due to warping of the rh dataset, rh values can exceed 100 which breaks the calculation.
path: api/app/sfms/fwi_processor.py
user: login: brettedw
body: That's a good idea, I'll see if I can try that
created_at: 2025-04-07T21:14:35Z
author_association: COLLABORATOR
diff_hunk: @@ -65,9 +73,9 @@ def calculate_ffmc(previous_ffmc_ds: WPSDataset, temp_ds: WPSDataset, rh_ds: WPS
     precip_array, _ = precip_ds.replace_nodata_with(0)
     wind_speed_array, _ = wind_speed_ds.replace_nodata_with(0)
 
-    # Due to warping of the rh dataset, rh values can exceed 100 which breaks the ffmc calculation.
-    # Set rh values greater than 100 to the max allowable which is 100.
-    rh_array[rh_array > 100] = 100
+    if (rh_array > 100).any():
path: api/app/sfms/fwi_processor.py
user: login: dgboss
body: You could pull this out into a function.
created_at: 2025-04-07T22:49:07Z
author_association: COLLABORATOR
diff_hunk: @@ -137,7 +137,13 @@ def __mul__(self, other):
 
         return WPSDataset(ds_path=None, ds=out_ds)
 
-    def warp_to_match(self, other: "WPSDataset", output_path: str, resample_method: GDALResamplingMethod = GDALResamplingMethod.NEAREST_NEIGHBOUR):
+    def warp_to_match(
+        self,
+        other: "WPSDataset",
+        output_path: str,
+        resample_method: GDALResamplingMethod = GDALResamplingMethod.NEAREST_NEIGHBOUR,
+        max_value: float | None = None,
path: wps_shared/wps_shared/geospatial/wps_dataset.py
user: login: dgboss
body: The `WPSDataset` allows for non-float datatypes (the default is `gdal.GDT_Byte`). It might be safer and keep things more flexible if the application of a max value gets factored out into an explicit function; probably a function that returns a new `WPSDataset` with the updated data. 
created_at: 2025-04-07T22:58:37Z
author_association: COLLABORATOR
diff_hunk: @@ -137,7 +137,13 @@ def __mul__(self, other):
 
         return WPSDataset(ds_path=None, ds=out_ds)
 
-    def warp_to_match(self, other: "WPSDataset", output_path: str, resample_method: GDALResamplingMethod = GDALResamplingMethod.NEAREST_NEIGHBOUR):
+    def warp_to_match(
+        self,
+        other: "WPSDataset",
+        output_path: str,
+        resample_method: GDALResamplingMethod = GDALResamplingMethod.NEAREST_NEIGHBOUR,
+        max_value: float | None = None,
path: wps_shared/wps_shared/geospatial/wps_dataset.py
user: login: dgboss
body: Ha, I just read Conor's comment about creating that logic inside this function! If we're warping a dataset chances are we're dealing with floats so I'm likely being over cautious. Feel free to ignore my comment.
created_at: 2025-04-07T23:01:46Z
author_association: COLLABORATOR
diff_hunk: @@ -137,7 +137,13 @@ def __mul__(self, other):
 
         return WPSDataset(ds_path=None, ds=out_ds)
 
-    def warp_to_match(self, other: "WPSDataset", output_path: str, resample_method: GDALResamplingMethod = GDALResamplingMethod.NEAREST_NEIGHBOUR):
+    def warp_to_match(
+        self,
+        other: "WPSDataset",
+        output_path: str,
+        resample_method: GDALResamplingMethod = GDALResamplingMethod.NEAREST_NEIGHBOUR,
+        max_value: float | None = None,
path: wps_shared/wps_shared/geospatial/wps_dataset.py
user: login: brettedw
body: I could type it as `float | int | None`. I think it should work with whichever number type is provided, but maybe I'm not thinking that through
created_at: 2025-04-07T23:09:21Z
author_association: COLLABORATOR
diff_hunk: @@ -137,7 +137,13 @@ def __mul__(self, other):
 
         return WPSDataset(ds_path=None, ds=out_ds)
 
-    def warp_to_match(self, other: "WPSDataset", output_path: str, resample_method: GDALResamplingMethod = GDALResamplingMethod.NEAREST_NEIGHBOUR):
+    def warp_to_match(
+        self,
+        other: "WPSDataset",
+        output_path: str,
+        resample_method: GDALResamplingMethod = GDALResamplingMethod.NEAREST_NEIGHBOUR,
+        max_value: float | None = None,
path: wps_shared/wps_shared/geospatial/wps_dataset.py
user: login: conbrad
body: You can provide the max value to the `init` function alongside the `datatype`. Better yet, compose the `datatype` and `maxvalue` together in it's own class to make sure they correspond and pass that to `WPSDataset`'s `init`.
created_at: 2025-04-07T23:18:55Z
author_association: COLLABORATOR
diff_hunk: @@ -137,7 +137,13 @@ def __mul__(self, other):
 
         return WPSDataset(ds_path=None, ds=out_ds)
 
-    def warp_to_match(self, other: "WPSDataset", output_path: str, resample_method: GDALResamplingMethod = GDALResamplingMethod.NEAREST_NEIGHBOUR):
+    def warp_to_match(
+        self,
+        other: "WPSDataset",
+        output_path: str,
+        resample_method: GDALResamplingMethod = GDALResamplingMethod.NEAREST_NEIGHBOUR,
+        max_value: float | None = None,
path: wps_shared/wps_shared/geospatial/wps_dataset.py
user: login: brettedw
body: The datatype in the `init` is kind of misleading at the moment in my mind. For example when we are initializing a `WPSDataset`, if we don't explicitly know the datatype of the gdal geotiff ahead of time, it will always be set to `GDT_Byte`. So when we're loading in a temperature geotiff, the `WPSDataset` `datatype` is `GDAL_Byte`, but the gdal dataset `datatype` is `GDT_Float64`
Maybe it's worthwhile to talk about what the intention of providing the `datatype` is? 

created_at: 2025-04-07T23:59:15Z
author_association: COLLABORATOR
diff_hunk: @@ -137,7 +137,13 @@ def __mul__(self, other):
 
         return WPSDataset(ds_path=None, ds=out_ds)
 
-    def warp_to_match(self, other: "WPSDataset", output_path: str, resample_method: GDALResamplingMethod = GDALResamplingMethod.NEAREST_NEIGHBOUR):
+    def warp_to_match(
+        self,
+        other: "WPSDataset",
+        output_path: str,
+        resample_method: GDALResamplingMethod = GDALResamplingMethod.NEAREST_NEIGHBOUR,
+        max_value: float | None = None,
path: wps_shared/wps_shared/geospatial/wps_dataset.py
user: login: conbrad
body: The original intent for `datatype` was to create a memory driver in the `__mul__` operation. The default `GDAL_Byte` was chosen to minimize memory overhead unless the caller explicitly knows they need a larger data width. 

I didn't realize you can grab the data type from a dataset raster band with `band.DataType`. We should use that if we can instead.
created_at: 2025-04-08T00:10:03Z
author_association: COLLABORATOR

PR comments:
user: login: codecov[bot]
created_at: 2025-04-07T19:00:18Z
body: ## [Codecov](https://app.codecov.io/gh/bcgov/wps/pull/4417?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bcgov) Report
Attention: Patch coverage is `59.09091%` with `18 lines` in your changes missing coverage. Please review.
> Project coverage is 70.65%. Comparing base [(`98e72e7`)](https://app.codecov.io/gh/bcgov/wps/commit/98e72e74722bdbb676468747489a57ec4249c60c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bcgov) to head [(`9f421d6`)](https://app.codecov.io/gh/bcgov/wps/commit/9f421d6d93ae937cf52e9ac8ad563104cf99964b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bcgov).
> Report is 1 commits behind head on main.

| [Files with missing lines](https://app.codecov.io/gh/bcgov/wps/pull/4417?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bcgov) | Patch % | Lines |
|---|---|---|
| [wps\_shared/wps\_shared/geospatial/wps\_dataset.py](https://app.codecov.io/gh/bcgov/wps/pull/4417?src=pr&el=tree&filepath=wps_shared%2Fwps_shared%2Fgeospatial%2Fwps_dataset.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bcgov#diff-d3BzX3NoYXJlZC93cHNfc2hhcmVkL2dlb3NwYXRpYWwvd3BzX2RhdGFzZXQucHk=) | 0.00% | [18 Missing :warning: ](https://app.codecov.io/gh/bcgov/wps/pull/4417?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bcgov) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #4417      +/-   ##
==========================================
- Coverage   70.67%   70.65%   -0.03%     
==========================================
  Files         291      291              
  Lines       11154    11180      +26     
  Branches      539      539              
==========================================
+ Hits         7883     7899      +16     
- Misses       3079     3089      +10     
  Partials      192      192              
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/bcgov/wps/pull/4417?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bcgov).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bcgov).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
author_association: NONE
user: login: sonarqubecloud[bot]
created_at: 2025-04-09T22:20:51Z
body: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=bcgov_wps&pullRequest=4417) **Quality Gate passed**  
Issues  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 New issues](https://sonarcloud.io/project/issues?id=bcgov_wps&pullRequest=4417&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=bcgov_wps&pullRequest=4417&issueStatuses=ACCEPTED)

Measures  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=bcgov_wps&pullRequest=4417&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=bcgov_wps&pullRequest=4417&metric=new_coverage&view=list)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=bcgov_wps&pullRequest=4417&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=bcgov_wps&pullRequest=4417)


author_association: NONE