Repository: IAMconsortium/nomenclature

Issue:
Add warning-feature to DataValidator
As a next step for scenario validation and vetting, we need to extend the **DataValidator** class to enable showing a warning instead of raising an error.

My proposed implementation is the following:
- Add a **warning-level** attribute to a validation-criteria item, which defaults to None, but can be set to e.g. "red", "yellow"
- This will probably require an intermediate class **DataValidationCriteria(IamcDataFilter)** with the new attribute, and **DataValidationCriteriaValue** and **DataValidationCriteriaBounds** inherit from that.
- If the validation fails and the warning-level is not None, add the value to a warning-list in
    https://github.com/IAMconsortium/nomenclature/blob/3a76cba2f5baf6c7f54be9d45d40c1a698738f6c/nomenclature/processor/data_validator.py#L96

    instead of the error-list.
- If the warning-list is not empty, display the warning-list as a DataFrame (after the error-list, but before raising the error).

Does that make sense, @phackstock?

Discussion:
dc-almeida: After discussion in weekly stand-up, will change the fail message start for the warning case, as it might be confusing to refer to a "failed" validation.
danielhuppmann: Seems like I can't respond to the comment by @phackstock on the warning-level inline, so putting this here.

I think that the suggestion to use integers instead of strings is just confusing to non-expert users, without adding any practical benefit.

In yesterday's ScenarioCompass meeting, we discussed the need for different types of warnings in the sense of "warning for deviation from historical energy data" versus a "biodiversity concern in the future". Here, there is no clear ordering in terms of low/medium/high - instead, this could be clearly differentiated using colors.

Going one step further, it may be useful to define the allowed warning-level-names in the nomenclature.yaml project-config, not the package, so that we don't have to always make a new release when adding new levels.
dc-almeida: As discussed in our meeting, incorporating the following features now:
- warning levels, defaulting to error if not specified (ordering is implicit in the Enum)
- allowing multiple warning levels for same variable (to be refined in future)
- printing warning level in log output df's (this addition can lead to hiding other columns, such as 'variable', to preserve text length; see `test_DataValidator_apply_fails`; the variable is still printed in the criteria header, though, and row numbers ultimately identify error/warning sources)

Follow-up PRs will include:
- a way to refactor all warnings/errors into single df (ordered by warning level) for easy tracing and visualisation
- rewriting warnings for each variable such that only the highest level is printed (when multiple)
- include descriptions of constraints for warnings

Patch:
diff --git a/docs/user_guide/config.rst b/docs/user_guide/config.rst
index f000d7b..57f3db8 100644
--- a/docs/user_guide/config.rst
+++ b/docs/user_guide/config.rst
@@ -126,12 +126,12 @@ By setting *definitions.region.nuts* (optional) in the configuration file:
       nuts:
         nuts-1: [ AT, BE, CZ ]
         nuts-2: [ AT ]
-        nuts-3: [ AT, BE ]
+        nuts-3: true
 
 the nomenclature package will add the selected NUTS regions to the *region* codelist.
 
-In the example above, the package will add all NUTS (1, 2, and 3) for Austria,
-NUTS 1 and 3 for Belgium, and NUTS 1 for Czechia.
+In the example above, the package will add: NUTS 1 regions for Austria, Belgium
+and Czechia, NUTS 2 regions for Austria, NUTS 3 regions for all EU countries.
 
 More details on the list of NUTS regions can be found here: :ref:`nuts`.
 
diff --git a/nomenclature/codelist.py b/nomenclature/codelist.py
index 16adaa2..60b56e4 100644
--- a/nomenclature/codelist.py
+++ b/nomenclature/codelist.py
@@ -733,11 +733,16 @@ class RegionCodeList(CodeList):
         # adding nuts regions
         if config.definitions.region.nuts:
             for level, countries in config.definitions.region.nuts.items():
-                for nuts_region in nuts.get(
-                    level=int(level[-1]), country_code=countries
-                ):
+                if countries is True:
+                    region_list = nuts.get(level=int(level[-1]))
+                else:
+                    region_list = nuts.get(level=int(level[-1]), country_code=countries)
+                for r in region_list:
                     code_list.append(
-                        RegionCode(name=nuts_region.code, hierarchy="NUTS 2021-2024")
+                        RegionCode(
+                            name=r.code,
+                            hierarchy=f"NUTS {level[-1]} regions (2021-2024)",
+                        )
                     )
 
         # importing from an external repository
diff --git a/nomenclature/config.py b/nomenclature/config.py
index ae2c74d..0e6bd5e 100644
--- a/nomenclature/config.py
+++ b/nomenclature/config.py
@@ -107,13 +107,13 @@ class CodeListConfig(BaseModel):
 
 class RegionCodeListConfig(CodeListConfig):
     country: bool = False
-    nuts: dict[str, str | list[str]] | None = None
+    nuts: dict[str, str | list[str] | bool] | None = None
 
     @field_validator("nuts")
     @classmethod
     def check_nuts(
-        cls, v: dict[str, str | list[str]] | None
-    ) -> dict[str, str | list[str]] | None:
+        cls, v: dict[str, str | list[str] | bool] | None
+    ) -> dict[str, str | list[str] | bool] | None:
         if v and not all(k in ["nuts-1", "nuts-2", "nuts-3"] for k in v.keys()):
             raise ValueError(
                 "Invalid fields for `nuts` in configuration. "
diff --git a/nomenclature/processor/data_validator.py b/nomenclature/processor/data_validator.py
index 6bd38df..debf0e1 100644
--- a/nomenclature/processor/data_validator.py
+++ b/nomenclature/processor/data_validator.py
@@ -1,5 +1,6 @@
 import logging
 import textwrap
+from enum import Enum
 from pathlib import Path
 
 import yaml
@@ -16,7 +17,18 @@ from nomenclature.processor.utils import get_relative_path
 logger = logging.getLogger(__name__)
 
 
-class DataValidationCriteriaValue(IamcDataFilter):
+class WarningEnum(str, Enum):
+    high = "high"
+    medium = "medium"
+    low = "low"
+    error = "error"
+
+
+class DataValidationCriteria(IamcDataFilter):
+    warning_level: WarningEnum = WarningEnum.error
+
+
+class DataValidationCriteriaValue(DataValidationCriteria):
     value: float
     rtol: float = 0.0
     atol: float = 0.0
@@ -38,7 +50,7 @@ class DataValidationCriteriaValue(IamcDataFilter):
         return self.model_dump(
             exclude_none=True,
             exclude_unset=True,
-            exclude=["value", "rtol", "atol"],
+            exclude=["warning_level", "value", "rtol", "atol"],
         )
 
     @property
@@ -46,11 +58,11 @@ class DataValidationCriteriaValue(IamcDataFilter):
         return self.model_dump(
             exclude_none=True,
             exclude_unset=True,
-            exclude=["lower_bound", "upper_bound"],
+            exclude=["warning_level", "lower_bound", "upper_bound"],
         )
 
 
-class DataValidationCriteriaBounds(IamcDataFilter):
+class DataValidationCriteriaBounds(DataValidationCriteria):
     upper_bound: float | None = None
     lower_bound: float | None = None
 
@@ -64,6 +76,14 @@ class DataValidationCriteriaBounds(IamcDataFilter):
     def validation_args(self):
         return self.criteria
 
+    @property
+    def criteria(self):
+        return self.model_dump(
+            exclude_none=True,
+            exclude_unset=True,
+            exclude=["warning_level"],
+        )
+
 
 class DataValidator(Processor):
     """Processor for validating IAMC datapoints"""
@@ -89,31 +109,39 @@ class DataValidator(Processor):
         return cls(file=file, criteria_items=content)
 
     def apply(self, df: IamDataFrame) -> IamDataFrame:
-        error_list = []
+        fail_list = []
+        error = False
 
         with adjust_log_level():
             for item in self.criteria_items:
                 failed_validation = df.validate(**item.validation_args)
                 if failed_validation is not None:
-                    error_list.append(
-                        "  Criteria: "
-                        + ", ".join(
-                            [f"{key}: {value}" for key, value in item.criteria.items()]
-                        )
+                    criteria_msg = "  Criteria: " + ", ".join(
+                        [f"{key}: {value}" for key, value in item.criteria.items()]
                     )
-                    error_list.append(
+                    failed_validation["warning_level"] = item.warning_level.value
+                    if item.warning_level == WarningEnum.error:
+                        error = True
+                    fail_list.append(criteria_msg)
+                    fail_list.append(
                         textwrap.indent(str(failed_validation), prefix="    ") + "\n"
                     )
-
-            if error_list:
-                logger.error(
-                    "Failed data validation (file %s):\n%s",
-                    get_relative_path(self.file),
-                    "\n".join(error_list),
+            fail_msg = "(file %s):\n" % get_relative_path(self.file)
+            if error:
+                fail_msg = (
+                    "Data validation with error(s)/warning(s) "
+                    + fail_msg
+                    + "\n".join(fail_list)
                 )
+                logger.error(fail_msg)
                 raise ValueError(
                     "Data validation failed. Please check the log for details."
                 )
+            if fail_list:
+                fail_msg = (
+                    "Data validation with warning(s) " + fail_msg + "\n".join(fail_list)
+                )
+                logger.warning(fail_msg)
         return df
 
     def validate_with_definition(self, dsd: DataStructureDefinition) -> None:


Test patch:
diff --git a/tests/data/validation/validate_data/validate_warning.yaml b/tests/data/validation/validate_data/validate_warning.yaml
new file mode 100644
index 0000000..3482305
--- /dev/null
+++ b/tests/data/validation/validate_data/validate_warning.yaml
@@ -0,0 +1,13 @@
+ - variable: Primary Energy
+   year: 2010
+   upper_bound: 2.5
+   lower_bound: 1
+   warning_level: low
+ - variable: Primary Energy
+   year: 2010
+   upper_bound: 5
+   lower_bound: 1
+ - variable: Primary Energy|Coal
+   year: 2010
+   upper_bound: 5
+   lower_bound: 1
diff --git a/tests/test_definition.py b/tests/test_definition.py
index 6f0ac79..63f9661 100644
--- a/tests/test_definition.py
+++ b/tests/test_definition.py
@@ -70,7 +70,9 @@ def test_definition_general_config_nuts_only():
     obs = DataStructureDefinition(
         TEST_DATA_DIR / "config" / "general-config-only-nuts" / "definitions"
     )
+    # check country codes
     assert all(region[:2] in ("AT", "BE", "CZ") for region in obs.region)
+    # check region import
     assert len([region for region in obs.region if region.startswith("AT")]) == 4
     assert len([region for region in obs.region if region.startswith("BE")]) == 12
     assert len([region for region in obs.region if region.startswith("CZ")]) == 15
diff --git a/tests/test_validate_data.py b/tests/test_validate_data.py
index cf54768..8d0dbf0 100644
--- a/tests/test_validate_data.py
+++ b/tests/test_validate_data.py
@@ -102,23 +102,53 @@ def test_DataValidator_apply_fails(simple_df, file, item_1, item_2, item_3, capl
     data_file = DATA_VALIDATION_TEST_DIR / f"validate_data_fails_{file}.yaml"
     data_validator = DataValidator.from_file(data_file)
 
-    failed_validation_message = f"""Failed data validation (file {data_file.relative_to(Path.cwd())}):
+    failed_validation_message = (
+        "Data validation with error(s)/warning(s) "
+        f"""(file {data_file.relative_to(Path.cwd())}):
   Criteria: variable: ['Primary Energy'], {item_1}
-         model scenario region        variable   unit  year  value
-    0  model_a   scen_a  World  Primary Energy  EJ/yr  2010    6.0
-    1  model_a   scen_b  World  Primary Energy  EJ/yr  2010    7.0
+         model scenario region        variable   unit  year  value warning_level
+    0  model_a   scen_a  World  Primary Energy  EJ/yr  2010    6.0         error
+    1  model_a   scen_b  World  Primary Energy  EJ/yr  2010    7.0         error
 
   Criteria: variable: ['Primary Energy|Coal'], {item_2}
-         model scenario region             variable   unit  year  value
-    0  model_a   scen_a  World  Primary Energy|Coal  EJ/yr  2005    0.5
+         model scenario region  ...  year value  warning_level
+    0  model_a   scen_a  World  ...  2005   0.5          error
+
+    [1 rows x 8 columns]
 
   Criteria: variable: ['Primary Energy'], year: [2005], {item_3}
-         model scenario region        variable   unit  year  value
-    0  model_a   scen_a  World  Primary Energy  EJ/yr  2005    1.0
-    1  model_a   scen_b  World  Primary Energy  EJ/yr  2005    2.0"""
+         model scenario region        variable   unit  year  value warning_level
+    0  model_a   scen_a  World  Primary Energy  EJ/yr  2005    1.0         error
+    1  model_a   scen_b  World  Primary Energy  EJ/yr  2005    2.0         error"""
+    )
 
     with pytest.raises(ValueError, match="Data validation failed"):
         data_validator.apply(simple_df)
 
     # check if the log message contains the correct information
     assert failed_validation_message in caplog.text
+
+
+def test_DataValidator_validate_with_warning(simple_df, caplog):
+    data_validator = DataValidator.from_file(
+        DATA_VALIDATION_TEST_DIR / "validate_warning.yaml"
+    )
+    with pytest.raises(ValueError, match="Data validation failed"):
+        data_validator.apply(simple_df)
+
+    failed_validation_message = (
+        "Data validation with error(s)/warning(s) "
+        f"""(file {(DATA_VALIDATION_TEST_DIR / "validate_warning.yaml").relative_to(Path.cwd())}):
+  Criteria: variable: ['Primary Energy'], year: [2010], upper_bound: 2.5, lower_bound: 1.0
+         model scenario region        variable   unit  year  value warning_level
+    0  model_a   scen_a  World  Primary Energy  EJ/yr  2010    6.0           low
+    1  model_a   scen_b  World  Primary Energy  EJ/yr  2010    7.0           low
+
+  Criteria: variable: ['Primary Energy'], year: [2010], upper_bound: 5.0, lower_bound: 1.0
+         model scenario region        variable   unit  year  value warning_level
+    0  model_a   scen_a  World  Primary Energy  EJ/yr  2010    6.0         error
+    1  model_a   scen_b  World  Primary Energy  EJ/yr  2010    7.0         error"""
+    )
+
+    # only prints two of three criteria in df to be validated
+    assert failed_validation_message in caplog.text