Repository: natcap/invest (python)

Issue:
Habitat Quality handle empty table fields
A user posted an issue on the [forums](https://community.naturalcapitalproject.org/t/habitat-quality-valueerror-could-not-convert-string-to-float/2765) where they were getting the following error:

```python
2022-05-04 16:55:57,649 (natcap.invest.habitat_quality) habitat_quality.execute(259) INFO Starting execute of Habitat Quality model.
2022-05-04 16:55:57,649 (natcap.invest.habitat_quality) habitat_quality.execute(265) INFO Creating workspace
2022-05-04 16:55:57,688 (natcap.invest.habitat_quality) habitat_quality.execute(278) INFO Checking Threat and Sensitivity tables for compliance
2022-05-04 16:55:57,700 (natcap.invest.utils) utils.prepare_workspace(167) ERROR Exception while executing Habitat-Quality
Traceback (most recent call last):
  File "natcap\invest\utils.py", line 165, in prepare_workspace
  File "natcap\invest\ui\model.py", line 1649, in _logged_target
  File "natcap\invest\habitat_quality.py", line 293, in execute
  File "natcap\invest\habitat_quality.py", line 294, in <dictcomp>
ValueError: could not convert string to float: ''
2022-05-04 16:55:57,701 (natcap.invest.utils) utils.prepare_workspace(170) INFO Elapsed time: 0.06s
2022-05-04 16:55:57,701 (natcap.invest.utils) utils.prepare_workspace(173) INFO Execution finished
```

If this is just a case of the threat or sensitivity tables having an empty field, I feel like we could provide a better error message there.

Error reported using InVEST 3.10.2.

Pull request:
Habitat Quality: Improve validation for `MAX_DIST` and `WEIGHT`
## Description

Fixes #1052 

While looking into Issue #1052, I found that, in most cases, our error handling related to empty fields in the Threats Table and Sensitivity Table CSVs has been addressed by improvements to validation since the time the issue was opened. However, in cases where a `NaN` value exists in the `MAX_DIST` or `WEIGHT` columns of the Threats Table, validation passes but the model eventually throws one of the following errors:

- In the case of missing `MAX_DIST`:
```
    if row['max_dist'] <= 0:
       ^^^^^^^^^^^^^^^^^^^^
TypeError: '<=' not supported between instances of 'str' and 'int'
```
- In the case of missing `WEIGHT`:
```
weight_sum = threat_df['weight'].sum()
             ^^^^^^^^^^^^^^^^^^^^^^^^^
[...]
TypeError: can only concatenate str (not "float") to str
```

In order to give the user more helpful feedback, I've added checks to both `execute` (for the benefit of CLI users) and `validate` (for the benefit of Workbench users). 

We _did_ have an existing `if row['max_dist'] <= 0` check in `execute`; in the interest of catching errors as early as possible, I've moved this check up and combined it with the checks for missing `MAX_DIST` and `WEIGHT`. 

## Checklist
- [ ] Updated HISTORY.rst and link to any relevant issue (if these changes are user-facing)
- [ ] Updated the user's guide (if needed)
- [x] Tested the Workbench UI (if relevant)


Patch:
diff --git a/src/natcap/invest/habitat_quality.py b/src/natcap/invest/habitat_quality.py
index 813e94f8c..fdb1865e5 100644
--- a/src/natcap/invest/habitat_quality.py
+++ b/src/natcap/invest/habitat_quality.py
@@ -26,6 +26,12 @@ MISSING_THREAT_RASTER_MSG = gettext(
     "A threat raster for threats: {threat_list} was not found or it "
     "could not be opened by GDAL.")
 DUPLICATE_PATHS_MSG = gettext("Threat paths must be unique. Duplicates: ")
+INVALID_MAX_DIST_MSG = gettext(
+    "The maximum distance value for threats: {threat_list} is less than "
+    "or equal to 0. MAX_DIST must be a positive value.")
+MISSING_MAX_DIST_MSG = gettext(
+    "Maximum distance value is missing for threats: {threat_list}.")
+MISSING_WEIGHT_MSG = gettext("Weight value is missing for threats: {threat_list}.")
 
 MODEL_SPEC = {
     "model_id": "habitat_quality",
@@ -1185,6 +1191,41 @@ def validate(args, limit_to=None):
 
             invalid_keys.add('sensitivity_table_path')
 
+        # check that max_dist and weight values are included in the
+        # threats table and that max_dist >= 0
+        invalid_max_dist = []
+        missing_max_dist = []
+        missing_weight = []
+        for threat, row in threat_df.iterrows():
+            if row['max_dist'] == '':
+                missing_max_dist.append(threat)
+            elif row['max_dist'] <= 0:
+                invalid_max_dist.append(threat)
+
+            if row['weight'] == '':
+                missing_weight.append(threat)
+
+        if invalid_max_dist:
+            validation_warnings.append((
+                ['threats_table_path'],
+                INVALID_MAX_DIST_MSG.format(threat_list=invalid_max_dist)
+            ))
+
+        if missing_max_dist:
+            validation_warnings.append((
+                ['threats_table_path'],
+                MISSING_MAX_DIST_MSG.format(threat_list=missing_max_dist)
+            ))
+
+        if missing_weight:
+            validation_warnings.append((
+                ['threats_table_path'],
+                MISSING_WEIGHT_MSG.format(threat_list=missing_weight)
+            ))
+
+        if invalid_max_dist or missing_max_dist or missing_weight:
+            invalid_keys.add('threats_table_path')
+
         # Validate threat raster paths and their nodata values
         bad_threat_paths = []
         duplicate_paths = []
@@ -1231,8 +1272,6 @@ def validate(args, limit_to=None):
             validation_warnings.append((
                 ['threats_table_path'],
                 DUPLICATE_PATHS_MSG + str(duplicate_paths)))
-
-            if 'threats_table_path' not in invalid_keys:
-                invalid_keys.add('threats_table_path')
+            invalid_keys.add('threats_table_path')
 
     return validation_warnings


Test patch:
diff --git a/tests/test_habitat_quality.py b/tests/test_habitat_quality.py
index 46d8e38a8..d0f7e5d71 100644
--- a/tests/test_habitat_quality.py
+++ b/tests/test_habitat_quality.py
@@ -1373,6 +1373,159 @@ class HabitatQualityTests(unittest.TestCase):
             validate_result[0][1],
             habitat_quality.MISSING_SENSITIVITY_TABLE_THREATS_MSG))
 
+    def test_habitat_quality_validation_invalid_max_dist(self):
+        """Habitat Quality: test validation for max_dist <= 0."""
+        from natcap.invest import habitat_quality
+        from natcap.invest import utils
+
+        args = {
+            'half_saturation_constant': '0.5',
+            'results_suffix': 'regression',
+            'workspace_dir': self.workspace_dir,
+            'n_workers': -1,
+        }
+
+        args['access_vector_path'] = os.path.join(
+            args['workspace_dir'], 'access_samp.shp')
+        make_access_shp(args['access_vector_path'])
+
+        scenarios = ['_bas_', '_cur_', '_fut_']
+        for lulc_val, scenario in enumerate(scenarios, start=1):
+            lulc_array = numpy.ones((100, 100), dtype=numpy.int8)
+            lulc_array[50:, :] = lulc_val
+            args['lulc' + scenario + 'path'] = os.path.join(
+                args['workspace_dir'], 'lc_samp' + scenario + 'b.tif')
+            make_raster_from_array(
+                lulc_array, args['lulc' + scenario + 'path'])
+
+        args['sensitivity_table_path'] = os.path.join(
+            args['workspace_dir'], 'sensitivity_samp.csv')
+        make_sensitivity_samp_csv(args['sensitivity_table_path'])
+
+        make_threats_raster(args['workspace_dir'])
+
+        args['threats_table_path'] = os.path.join(
+            args['workspace_dir'], 'threats_samp.csv')
+
+        # create the threat CSV table
+        with open(args['threats_table_path'], 'w') as open_table:
+            open_table.write(
+                'MAX_DIST,WEIGHT,THREAT,DECAY,BASE_PATH,CUR_PATH,FUT_PATH\n')
+            open_table.write(
+                '0,0.7,threat_1,linear,,threat_1_c.tif,threat_1_f.tif\n')
+            open_table.write(
+                '0.07,1.0,threat_2,exponential,,threat_2_c.tif,'
+                'threat_2_f.tif\n')
+
+        validate_result = habitat_quality.validate(args, limit_to=None)
+        self.assertEqual(len(validate_result), 1)
+        self.assertEqual(validate_result[0][0], ['threats_table_path'])
+        self.assertTrue(utils.matches_format_string(
+            validate_result[0][1],
+            habitat_quality.INVALID_MAX_DIST_MSG))
+
+    def test_habitat_quality_validation_missing_max_dist(self):
+        """Habitat Quality: test validation for missing max_dist."""
+        from natcap.invest import habitat_quality
+        from natcap.invest import utils
+
+        args = {
+            'half_saturation_constant': '0.5',
+            'results_suffix': 'regression',
+            'workspace_dir': self.workspace_dir,
+            'n_workers': -1,
+        }
+
+        args['access_vector_path'] = os.path.join(
+            args['workspace_dir'], 'access_samp.shp')
+        make_access_shp(args['access_vector_path'])
+
+        scenarios = ['_bas_', '_cur_', '_fut_']
+        for lulc_val, scenario in enumerate(scenarios, start=1):
+            lulc_array = numpy.ones((100, 100), dtype=numpy.int8)
+            lulc_array[50:, :] = lulc_val
+            args['lulc' + scenario + 'path'] = os.path.join(
+                args['workspace_dir'], 'lc_samp' + scenario + 'b.tif')
+            make_raster_from_array(
+                lulc_array, args['lulc' + scenario + 'path'])
+
+        args['sensitivity_table_path'] = os.path.join(
+            args['workspace_dir'], 'sensitivity_samp.csv')
+        make_sensitivity_samp_csv(args['sensitivity_table_path'])
+
+        make_threats_raster(args['workspace_dir'])
+
+        args['threats_table_path'] = os.path.join(
+            args['workspace_dir'], 'threats_samp.csv')
+
+        # create the threat CSV table
+        with open(args['threats_table_path'], 'w') as open_table:
+            open_table.write(
+                'MAX_DIST,WEIGHT,THREAT,DECAY,BASE_PATH,CUR_PATH,FUT_PATH\n')
+            open_table.write(
+                ',0.7,threat_1,linear,,threat_1_c.tif,threat_1_f.tif\n')
+            open_table.write(
+                '0.07,1.0,threat_2,exponential,,threat_2_c.tif,'
+                'threat_2_f.tif\n')
+
+        validate_result = habitat_quality.validate(args, limit_to=None)
+        self.assertEqual(len(validate_result), 1)
+        self.assertEqual(validate_result[0][0], ['threats_table_path'])
+        self.assertTrue(utils.matches_format_string(
+            validate_result[0][1],
+            habitat_quality.MISSING_MAX_DIST_MSG))
+
+    def test_habitat_quality_validation_missing_weight(self):
+        """Habitat Quality: test validation for missing weight."""
+        from natcap.invest import habitat_quality
+        from natcap.invest import utils
+
+        args = {
+            'half_saturation_constant': '0.5',
+            'results_suffix': 'regression',
+            'workspace_dir': self.workspace_dir,
+            'n_workers': -1,
+        }
+
+        args['access_vector_path'] = os.path.join(
+            args['workspace_dir'], 'access_samp.shp')
+        make_access_shp(args['access_vector_path'])
+
+        scenarios = ['_bas_', '_cur_', '_fut_']
+        for lulc_val, scenario in enumerate(scenarios, start=1):
+            lulc_array = numpy.ones((100, 100), dtype=numpy.int8)
+            lulc_array[50:, :] = lulc_val
+            args['lulc' + scenario + 'path'] = os.path.join(
+                args['workspace_dir'], 'lc_samp' + scenario + 'b.tif')
+            make_raster_from_array(
+                lulc_array, args['lulc' + scenario + 'path'])
+
+        args['sensitivity_table_path'] = os.path.join(
+            args['workspace_dir'], 'sensitivity_samp.csv')
+        make_sensitivity_samp_csv(args['sensitivity_table_path'])
+
+        make_threats_raster(args['workspace_dir'])
+
+        args['threats_table_path'] = os.path.join(
+            args['workspace_dir'], 'threats_samp.csv')
+
+        # create the threat CSV table
+        with open(args['threats_table_path'], 'w') as open_table:
+            open_table.write(
+                'MAX_DIST,WEIGHT,THREAT,DECAY,BASE_PATH,CUR_PATH,FUT_PATH\n')
+            open_table.write(
+                '0.04,,threat_1,linear,,threat_1_c.tif,threat_1_f.tif\n')
+            open_table.write(
+                '0.07,1.0,threat_2,exponential,,threat_2_c.tif,'
+                'threat_2_f.tif\n')
+
+        validate_result = habitat_quality.validate(args, limit_to=None)
+        self.assertEqual(len(validate_result), 1)
+        self.assertEqual(validate_result[0][0], ['threats_table_path'])
+        self.assertTrue(utils.matches_format_string(
+            validate_result[0][1],
+            habitat_quality.MISSING_WEIGHT_MSG))
+
     def test_habitat_quality_validation_bad_threat_path(self):
         """Habitat Quality: test validation for bad threat paths."""
         from natcap.invest import habitat_quality