Repository: IAMconsortium/pyam
Pull request #613 (2022-01-30 21:10:04)

Add filter by time_domain
# Please confirm that this PR has done the following:

- [x] Tests Added

# Description of PR

**Note**: this PR is targeted to the `feature/year-and-datetime` branch that collects all developments related to #596.

This PR extends the `filter()` method with an option to filter by the `time_domain`.

codecov[bot]: # [Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/613?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) Report
> Merging [#613](https://codecov.io/gh/IAMconsortium/pyam/pull/613?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) (3d78394) into [feature/year-and-datetime](https://codecov.io/gh/IAMconsortium/pyam/commit/f8580ac43dcbf0c7cb553cb8ecc0c5491008d0c8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) (f8580ac) will **increase** coverage by `0.0%`.
> The diff coverage is `96.5%`.

[![Impacted file tree graph](https://codecov.io/gh/IAMconsortium/pyam/pull/613/graphs/tree.svg?width=650&height=150&src=pr&token=ThxWlNkonJ&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium)](https://codecov.io/gh/IAMconsortium/pyam/pull/613?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium)

```diff
@@                    Coverage Diff                    @@
##           feature/year-and-datetime    #613   +/-   ##
=========================================================
  Coverage                       94.3%   94.3%           
=========================================================
  Files                             54      55    +1     
  Lines                           5478    5501   +23     
=========================================================
+ Hits                            5168    5190   +22     
- Misses                           310     311    +1     
```


| [Impacted Files](https://codecov.io/gh/IAMconsortium/pyam/pull/613?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) | Coverage Δ | |
|---|---|---|
| [pyam/core.py](https://codecov.io/gh/IAMconsortium/pyam/pull/613/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-cHlhbS9jb3JlLnB5) | `94.7% <83.3%> (-0.1%)` | :arrow_down: |
| [pyam/filter.py](https://codecov.io/gh/IAMconsortium/pyam/pull/613/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-cHlhbS9maWx0ZXIucHk=) | `100.0% <100.0%> (ø)` | |
| [tests/conftest.py](https://codecov.io/gh/IAMconsortium/pyam/pull/613/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-dGVzdHMvY29uZnRlc3QucHk=) | `100.0% <100.0%> (ø)` | |
| [tests/test\_core.py](https://codecov.io/gh/IAMconsortium/pyam/pull/613/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-dGVzdHMvdGVzdF9jb3JlLnB5) | `100.0% <100.0%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/613?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/613?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium). Last update [f8580ac...3d78394](https://codecov.io/gh/IAMconsortium/pyam/pull/613?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium).


Patch:
diff --git a/pyam/core.py b/pyam/core.py
index c1dec97..6cb82cb 100755
--- a/pyam/core.py
+++ b/pyam/core.py
@@ -11,6 +11,8 @@ import pandas as pd
 from pathlib import Path
 from tempfile import TemporaryDirectory
 
+from pyam.filter import filter_by_time_domain
+
 try:
     from datapackage import Package
 
@@ -1713,6 +1715,7 @@ class IamDataFrame(object):
              - 'year': takes an integer (int/np.int64), a list of integers or
                a range. Note that the last year of a range is not included,
                so `range(2010, 2015)` is interpreted as `[2010, ..., 2014]`
+             - 'time_domain': can be "year" or "datetime"
              - arguments for filtering by `datetime.datetime` or np.datetime64
                ('month', 'hour', 'time')
              - 'regexp=True' disables pseudo-regexp syntax in `pattern_match()`
@@ -1772,6 +1775,14 @@ class IamDataFrame(object):
                     self._data, cols=self.index.names, unique=False
                 ).isin(cat_idx)
 
+            elif col == "time_domain":
+                # fast-pass if `self` already has selected time-domain
+                if self.time_domain == values:
+                    keep_col = np.ones(len(self), dtype=bool)
+                else:
+                    levels, codes = get_index_levels_codes(self._data, self.time_col)
+                    keep_col = filter_by_time_domain(values, levels, codes)
+
             elif col == "year":
                 levels, codes = get_index_levels_codes(self._data, self.time_col)
                 if self.time_col == "time":
diff --git a/pyam/filter.py b/pyam/filter.py
new file mode 100644
index 0000000..c5e1fa6
--- /dev/null
+++ b/pyam/filter.py
@@ -0,0 +1,14 @@
+import numpy as np
+
+
+def filter_by_time_domain(values, levels, codes):
+    """Internal implementation to filter by time domain"""
+
+    if values == "year":
+        matches = [i for (i, label) in enumerate(levels) if isinstance(label, int)]
+    elif values == "datetime":
+        matches = [i for (i, label) in enumerate(levels) if not isinstance(label, int)]
+    else:
+        raise ValueError(f"Filter by `datetime='{values}'` not supported!")
+
+    return np.isin(codes, matches)


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
index 55eee02..5428c5d 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -176,6 +176,16 @@ def test_df_time():
     yield df
 
 
+# minimal IamDataFrame for specifically testing 'mixed'-time-domain features
+@pytest.fixture(scope="function")
+def test_df_mixed():
+    mapping = dict([(i, j) for i, j in zip(TEST_YEARS, TEST_TIME_MIXED)])
+    df = IamDataFrame(data=TEST_DF.rename(mapping, axis="columns"))
+    for i in META_COLS:
+        df.set_meta(META_DF[i])
+    yield df
+
+
 # minimal test data as pandas.DataFrame (only 'year' time format)
 @pytest.fixture(scope="function")
 def test_pd_df():
diff --git a/tests/test_core.py b/tests/test_core.py
index e5a9816..4dcd4d9 100644
--- a/tests/test_core.py
+++ b/tests/test_core.py
@@ -431,26 +431,38 @@ def test_filter_year(test_df):
         pdt.assert_index_equal(obs.time, EXP_DATETIME_INDEX)
 
 
-# TODO: merge with previous test once TEST_TIME_MIXED is part of `conftest.py:test_df`
-def test_filter_year_mixed_time_domain(test_pd_df):
-    mapping = dict([(i, j) for i, j in zip(TEST_YEARS, TEST_TIME_MIXED)])
-    df = IamDataFrame(data=test_pd_df.rename(mapping, axis="columns"))
+@pytest.mark.parametrize(
+    "arg_year, arg_time",
+    [
+        (dict(year=2005), dict(year=2010)),
+        (dict(time_domain="year"), dict(time_domain="datetime")),
+    ],
+)
+def test_filter_mixed_time_domain(test_df_mixed, arg_year, arg_time):
+    """Assert that reassigning attributes works for filtering from mixed time domain"""
 
-    assert df.time_domain == "mixed"
+    assert test_df_mixed.time_domain == "mixed"
 
     # filtering to datetime-only works as expected
-    obs = df.filter(year=2010)
+    obs = test_df_mixed.filter(**arg_time)
     assert obs.time_domain == "datetime"
     pdt.assert_index_equal(obs.time, pd.DatetimeIndex(["2010-07-21"]))
 
     # filtering to year-only works as expected including changing of time domain
-    obs = df.filter(year=2005)
+    obs = test_df_mixed.filter(**arg_year)
     assert obs.time_col == "year"
     assert obs.time_domain == "year"
     assert obs.year == [2005]
     pdt.assert_index_equal(obs.time, pd.Int64Index([2005]))
 
 
+def test_filter_time_domain_raises(test_df_year):
+    """Assert that error is raised for invalid time_domain filter value"""
+
+    with pytest.raises(ValueError, match="Filter by `datetime='mixed'` not supported!"):
+        test_df_year.filter(time_domain="mixed")
+
+
 @pytest.mark.parametrize("test_month", [6, "June", "Jun", "jun", ["Jun", "jun"]])
 def test_filter_month(test_df, test_month):
     if test_df.time_col == "year":