Repository: IAMconsortium/pyam
Pull request #715 (2022-12-07 05:26:21)

Add a `require_data()` method
# Please confirm that this PR has done the following:

- [x] Tests Added
- [x] Documentation Added
- ~Name of contributors Added to AUTHORS.rst~
- [x] Description in RELEASE_NOTES.md Added

# Description of PR

This PR adds a `require_data()` method as a generalized approach to `require_variable()`. The new method returns none if all scenarios have all (combinations) of required values on the dimensions, or it returns a DataFrame of model-scenarios where not all required values are present.

The PR also adds a (data) "coordinates" attribute to quickly access the data columns that are not in the index.

# Question

The existing validation methods like `require_variable()` are very verbose. Here, I have not added any logging output. Any preferences?


codecov[bot]: # [Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/715?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) Report
> Merging [#715](https://codecov.io/gh/IAMconsortium/pyam/pull/715?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) (d5465dd) into [main](https://codecov.io/gh/IAMconsortium/pyam/commit/6c34dc17ddef09aabc29d26379e06fc29fddc12c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) (6c34dc1) will **increase** coverage by `0.0%`.
> The diff coverage is `100.0%`.

```diff
@@          Coverage Diff          @@
##            main    #715   +/-   ##
=====================================
  Coverage   94.9%   94.9%           
=====================================
  Files         58      58           
  Lines       5909    5947   +38     
=====================================
+ Hits        5609    5649   +40     
+ Misses       300     298    -2     
```


| [Impacted Files](https://codecov.io/gh/IAMconsortium/pyam/pull/715?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) | Coverage Δ | |
|---|---|---|
| [pyam/core.py](https://codecov.io/gh/IAMconsortium/pyam/pull/715/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-cHlhbS9jb3JlLnB5) | `95.2% <100.0%> (+0.3%)` | :arrow_up: |
| [tests/test\_feature\_validation.py](https://codecov.io/gh/IAMconsortium/pyam/pull/715/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-dGVzdHMvdGVzdF9mZWF0dXJlX3ZhbGlkYXRpb24ucHk=) | `100.0% <100.0%> (ø)` | |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium)

phackstock: I think the return values should contain **all** identifiers to missing data points, not just _model_ and _scenario_ as it could otherwise lead to ambiguities. 

As an example these data: 

| Model   | Scenario   | Region   | Variable   | Unit   | 2020 | 2025 |
|---------|------------|----------|------------|--------|------|------|
| model_a | scenario_a | region_a | variable_a | unit_a | 1    |      |
| model_a | scenario_a | region_b | variable_a | unit_a | 1    | 2    |

with a required data specification:

```yaml
region: [region_a, region_b]
variable: variable_a
year: [2020, 2025]
```

would currently return _model_a_ and _scenario_a_. That does not tell you the whole story though as for the first line in the data a value for 2025 is missing while for the second line, which is also _model_a_ and _scenario_a_, everything is in order.
danielhuppmann: > I think the return values should contain all identifiers to missing data points, not just model and scenario as it could otherwise lead to ambiguities.

It's not quite straightforward to develop a useful interface (or object) for this "complete" return-object.

Riffing off the example above: Should the returned dataframe contain a column "unit"? If yes, the value should probably be `None` for all columns... If no, the columns of the returned object depend on the arguments, making automated processing difficult.

Second, I see a problem that computing the missing index will be quite a lot of processing, only to be thrown away later anyway.

Suggestion: we leave as is for now (or even simplify just return `bool`) and add an argument "verbose" later?


phackstock: > Suggestion: we leave as is for now (or even simplify just return bool) and add an argument "verbose" later?

My preference would be the latter. Just keeping it as a kind of validator of, "is everything that we require there?". The question of "what is missing?" could almost be a separate one.
phackstock: I had an idea of a different, maybe more simple, approach that might be worth exploring. Would it be an option to simply construct a list of all the combinations of required data input and then loop over them one by one checking if there is data there. During that you would simply save what's missing.
Something like:

```python
def require_data(self, region=None, variable=None, unit=None, year=None, exclude_on_fail=False):
    missing_datapoints = []
    # combine "everything with everything"
    for datapoint in itertools.product([region, variable, unit, year]):
        # obviously this check is a bit more involved but just to illustrate my idea
        if datapoint not in self:
            missing_datapoints.append(datapoint)
    if missing_datapoints:
        return missing_datapoints
````
danielhuppmann: > I had an idea of a different, maybe more simple, approach that might be worth exploring.

Yes, that's the (only) way to go if we want to have a complete list of all missing (combinations of) required items.

But if the use case (for now) is just to know whether all required items exist and (optionally) exclude those scenarios that do meet the requirement from further processing, checking the length of an existing index is much more efficient than creating a second index and comparing the two indices line-by-line.
phackstock: Yes, true. I was already thinking towards providing user feedback to point exactly what data points are missing but that’s probably better handled in a different function if we really need that level of precision.

Patch:
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index b44cacb..49904fd 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -7,7 +7,8 @@ Bump minimum version of **pandas** to v1.2.0 to support automatic engine selecti
 
 ## Individual updates
 
-- [#713](https://github.com/IAMconsortium/pyam/pull/713) Informative error when using lists for filter by level. `Level` now a forbidden column. 
+- [#715](https://github.com/IAMconsortium/pyam/pull/715) Add a `require_data()` method
+- [#713](https://github.com/IAMconsortium/pyam/pull/713) Informative error when using lists for filter by level, `level` now a forbidden column.
 - [#709](https://github.com/IAMconsortium/pyam/pull/709) Hotfix ops to support `fillna=0`
 - [#708](https://github.com/IAMconsortium/pyam/pull/708) Remove 'xls' as by-default-supported file format
 
diff --git a/pyam/core.py b/pyam/core.py
index f2af3ca..2dea809 100755
--- a/pyam/core.py
+++ b/pyam/core.py
@@ -435,6 +435,11 @@ class IamDataFrame(object):
         """Return the list of `data` columns (index names & data coordinates)"""
         return list(self._data.index.names)
 
+    @property
+    def coordinates(self):
+        """Return the list of `data` coordinates (columns not including index names)"""
+        return [i for i in self._data.index.names if i not in self.index.names]
+
     @property
     def time_domain(self):
         """Indicator of the time domain: 'year', 'datetime', or 'mixed'"""
@@ -928,6 +933,78 @@ class IamDataFrame(object):
         if name not in self.meta:
             self.meta[name] = np.nan
 
+    def require_data(
+        self, region=None, variable=None, unit=None, year=None, exclude_on_fail=False
+    ):
+        """Check whether scenarios have values for all combinations of given elements
+
+        Parameters
+        ----------
+        region : str or list of str, optional
+            Required region(s).
+        variable : str or list of str, optional
+            Required variable(s).
+        unit : str or list of str, optional
+            Required unit(s).
+        year : int or list of int, optional
+            Required year(s).
+        exclude_on_fail : bool, optional
+            Set *meta* indicator for scenarios failing validation as `exclude: True`.
+
+        Returns
+        -------
+        pd.DataFrame
+            A dataframe of the *index* of scenarios not satisfying the cr
+        """
+
+        # TODO: option to require values in certain ranges, see `_apply_criteria()`
+
+        # create mapping of required dimensions
+        required = {}
+        n = 1  # expected number of rows per scenario
+        for dim, value in [
+            ("region", region),
+            ("variable", variable),
+            ("unit", unit),
+            ("year", year),
+        ]:
+            if value is not None:
+                required[dim] = value
+                n *= len(to_list(value))
+
+        # fast exit if no arguments values are given
+        if not required:
+            return
+
+        # downselect to relevant rows
+        keep = self._apply_filters(**required)
+        rows = self._data.index[keep]
+
+        # identify scenarios that have none of the required values
+        index_none = self.index.difference(
+            rows.droplevel(level=self.coordinates).drop_duplicates()
+        ).to_frame(index=False)
+
+        # identify scenarios that have some but not all required values
+        columns = [i for i in self.coordinates if i not in required]
+        rows = rows.droplevel(level=columns).drop_duplicates()
+        data = (
+            pd.DataFrame(index=rows)
+            .reset_index(level=list(required))
+            .groupby(self.index.names)
+        )
+
+        index_incomplete = pd.DataFrame(
+            [idx for idx, df in data if len(df) != n], columns=self.index.names
+        )
+
+        # merge all scenarios where not all required data is present
+        index_missing_required = pd.concat([index_none, index_incomplete])
+        if not index_missing_required.empty:
+            if exclude_on_fail:
+                self._exclude_on_fail(index_missing_required)
+            return index_missing_required
+
     def require_variable(self, variable, unit=None, year=None, exclude_on_fail=False):
         """Check whether all scenarios have a required variable
 


Test patch:
diff --git a/tests/test_feature_validation.py b/tests/test_feature_validation.py
index 9001723..99f4911 100644
--- a/tests/test_feature_validation.py
+++ b/tests/test_feature_validation.py
@@ -1,8 +1,46 @@
 import pandas as pd
 import pandas.testing as pdt
+import pytest
+
 from pyam import IamDataFrame, validate, categorize, require_variable, META_IDX
 
 
+@pytest.mark.parametrize(
+    "kwargs",
+    (
+        dict(),
+        dict(variable="Primary Energy"),
+        dict(variable=["Primary Energy"], year=[2005, 2010]),
+    ),
+)
+def test_require_data_pass(test_df_year, kwargs):
+    # check that IamDataFrame with all required data returns None
+    assert test_df_year.require_data(**kwargs) is None
+
+
+@pytest.mark.parametrize(
+    "kwargs",
+    (
+        dict(variable="Primary Energy|Coal"),
+        dict(variable=["Primary Energy"], year=[2005, 2010]),
+    ),
+)
+@pytest.mark.parametrize("exclude_on_fail", (False, True))
+def test_require_data(test_df_year, kwargs, exclude_on_fail):
+    # check different ways of failing when not all required data is present
+
+    test_df_year._data = test_df_year._data[0:5]  # remove value for scen_b & 2010
+
+    obs = test_df_year.require_data(**kwargs, exclude_on_fail=exclude_on_fail)
+    exp = pd.DataFrame([["model_a", "scen_b"]], columns=["model", "scenario"])
+    pdt.assert_frame_equal(obs, exp)
+
+    if exclude_on_fail:
+        list(test_df_year.meta["exclude"]) == [False, True]
+    else:
+        list(test_df_year.meta["exclude"]) == [False, False]
+
+
 def test_require_variable_pass(test_df):
     # checking that the return-type is correct
     obs = test_df.require_variable(variable="Primary Energy", exclude_on_fail=True)