Repository: IAMconsortium/pyam

Issue:
Support logical-and in `require_data()`
# Background

The method `reuqire_data()` applies a logical-or for each dimension, so a scenario is ok if any variable/region from a given list is present.

We should add a keyword argument "method" or "logical" that allow a user to select whether any or all elements should be present.

# Context

In @l-welder & @coroa's **pathways-ensemble-analysis** tool, the following code is used (simplified):

```python
def require_variables(df, list_of_variables):
    for v in list_of_variables:
        df.require_data(variable=v)
```

The following would be much more straightforward:

```python
df.require_data(variable=list_of_variables, how="all")
```

# Default usage

I think that `how="all"` is the more common use case, so we could set the default to `None` (for pyam ^2.0) with a warning that *all* is used by default. In pyam 3.0, we can change the default to *all* and remove the warning.

@phackstock

Discussion:
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/IAMconsortium/pyam/pull/772?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) Report
> Merging [#772](https://app.codecov.io/gh/IAMconsortium/pyam/pull/772?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) (906e0a5) into [main](https://app.codecov.io/gh/IAMconsortium/pyam/commit/9c947a8bb391f3939ac1e9f39f12adf44165eef4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) (9c947a8) will **decrease** coverage by `0.1%`.
> The diff coverage is `100.0%`.

```diff
@@           Coverage Diff           @@
##            main    #772     +/-   ##
=======================================
- Coverage   94.4%   94.4%   -0.1%     
=======================================
  Files         62      62             
  Lines       6130    6132      +2     
=======================================
  Hits        5789    5789             
- Misses       341     343      +2     
```


| [Files Changed](https://app.codecov.io/gh/IAMconsortium/pyam/pull/772?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) | Coverage Δ | |
|---|---|---|
| [pyam/core.py](https://app.codecov.io/gh/IAMconsortium/pyam/pull/772?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-cHlhbS9jb3JlLnB5) | `95.1% <100.0%> (-0.1%)` | :arrow_down: |
| [tests/test\_feature\_validation.py](https://app.codecov.io/gh/IAMconsortium/pyam/pull/772?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-dGVzdHMvdGVzdF9mZWF0dXJlX3ZhbGlkYXRpb24ucHk=) | `100.0% <100.0%> (ø)` | |

... and [1 file with indirect coverage changes](https://app.codecov.io/gh/IAMconsortium/pyam/pull/772/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium)



Patch:
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index 6f7b2a2..768374a 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -19,6 +19,7 @@ instead of `pyam.to_list()`.
 
 ## Individual updates
 
+- [#772](https://github.com/IAMconsortium/pyam/pull/772) Show all missing rows for `require_data()`
 - [#771](https://github.com/IAMconsortium/pyam/pull/771) Refactor to start a separate validation module
 - [#764](https://github.com/IAMconsortium/pyam/pull/764) Clean-up exposing internal methods and attributes
 - [#763](https://github.com/IAMconsortium/pyam/pull/763) Implement a fix against carrying over unused levels when initializing from an indexed pandas object
diff --git a/pyam/core.py b/pyam/core.py
index bdf3fc7..74df64a 100755
--- a/pyam/core.py
+++ b/pyam/core.py
@@ -43,6 +43,7 @@ from pyam.utils import (
     IAMC_IDX,
     SORT_IDX,
     ILLEGAL_COLS,
+    remove_from_list,
 )
 from pyam.filter import (
     datetime_match,
@@ -65,6 +66,7 @@ from pyam.index import (
     get_keep_col,
     verify_index_integrity,
     replace_index_values,
+    append_index_col,
 )
 from pyam.time import swap_time_for_year, swap_year_for_time
 from pyam.logging import raise_data_error, deprecation_warning
@@ -972,7 +974,7 @@ class IamDataFrame(object):
     def require_data(
         self, region=None, variable=None, unit=None, year=None, exclude_on_fail=False
     ):
-        """Check whether scenarios have values for all combinations of given elements
+        """Check whether scenarios have values for all (combinations of) given elements.
 
         Parameters
         ----------
@@ -991,14 +993,13 @@ class IamDataFrame(object):
         Returns
         -------
         :class:`pandas.DataFrame` or None
-            A dataframe of the *index* of scenarios not satisfying the criteria.
+            A dataframe of missing (combinations of) elements for all scenarios.
         """
 
         # TODO: option to require values in certain ranges, see `_apply_criteria()`
 
         # create mapping of required dimensions
         required = {}
-        n = 1  # expected number of rows per scenario
         for dim, value in [
             ("region", region),
             ("variable", variable),
@@ -1006,41 +1007,38 @@ class IamDataFrame(object):
             ("year", year),
         ]:
             if value is not None:
-                required[dim] = value
-                n *= len(to_list(value))
+                required[dim] = to_list(value)
 
-        # fast exit if no arguments values are given
+        # fast exit if no arguments are given
         if not required:
+            logger.warning("No validation criteria provided.")
             return
 
-        # downselect to relevant rows
-        keep = self._apply_filters(**required)
-        rows = self._data.index[keep]
-
-        # identify scenarios that have none of the required values
-        index_none = self.index.difference(
-            rows.droplevel(level=self.coordinates).drop_duplicates()
-        ).to_frame(index=False)
-
-        # identify scenarios that have some but not all required values
-        columns = [i for i in self.coordinates if i not in required]
-        rows = rows.droplevel(level=columns).drop_duplicates()
-        data = (
-            pd.DataFrame(index=rows)
-            .reset_index(level=list(required))
-            .groupby(self.index.names)
+        # create index of required elements
+        index_required = pd.MultiIndex.from_product(
+            required.values(), names=list(required)
         )
 
-        index_incomplete = pd.DataFrame(
-            [idx for idx, df in data if len(df) != n], columns=self.index.names
+        # create scenario index of suitable length, merge required elements as columns
+        n = len(self.index)
+        index = self.index.repeat(len(index_required))
+        for i, name in enumerate(required.keys()):
+            index = append_index_col(
+                index, list(index_required.get_level_values(i)) * n, name=name
+            )
+
+        # identify scenarios that do not have all required elements
+        rows = (
+            self._data.index[self._apply_filters(**required)]
+            .droplevel(level=remove_from_list(self.coordinates, required))
+            .drop_duplicates()
         )
+        missing_required = index.difference(rows)
 
-        # merge all scenarios where not all required data is present
-        index_missing_required = pd.concat([index_none, index_incomplete])
-        if not index_missing_required.empty:
+        if not missing_required.empty:
             if exclude_on_fail:
-                _exclude_on_fail(self, index_missing_required)
-            return index_missing_required
+                _exclude_on_fail(self, missing_required.droplevel(list(required)))
+            return missing_required.to_frame(index=False)
 
     def require_variable(self, variable, unit=None, year=None, exclude_on_fail=False):
         """Check whether all scenarios have a required variable


Test patch:
diff --git a/tests/test_feature_validation.py b/tests/test_feature_validation.py
index 80d165a..48ed8fd 100644
--- a/tests/test_feature_validation.py
+++ b/tests/test_feature_validation.py
@@ -51,13 +51,21 @@ def test_require_data(test_df_year, kwargs, exclude_on_fail):
     df = test_df_year.append(IamDataFrame(DATA_GAS))
 
     obs = df.require_data(**kwargs, exclude_on_fail=exclude_on_fail)
+
     exp = pd.DataFrame([["model_a", "scen_b"]], columns=["model", "scenario"])
+    # add parametrization-dependent columns to expected output
+    if kwargs["variable"] == "Primary Energy|Coal":
+        exp["variable"] = ["Primary Energy|Coal"]
+    else:
+        exp["variable"] = ["Primary Energy"]
+        exp["year"] = [2010]
+
     pdt.assert_frame_equal(obs, exp)
 
     if exclude_on_fail:
-        list(df.exclude) == [False, True]
+        assert list(df.exclude) == [False, True]
     else:
-        list(df.exclude) == [False, False]
+        assert list(df.exclude) == [False, False]
 
 
 def test_require_variable_pass(test_df):