Repository: IAMconsortium/pyam
Pull request #557 (2021-07-06 09:12:24)

Swap time for year keeping subannual resolution
# Please confirm that this PR has done the following:

- [x] Tests Added
- [x] Documentation Added
- ~Name of contributors Added to AUTHORS.rst~
- [x] Description in RELEASE_NOTES.md Added

# Description of PR

This PR migrates a feature first developed in https://github.com/openENTRANCE/nomenclature to split out the "time" column of an IamDataFrame into a year column (as int) plus keeping the information on the sub-annual resolution as a new extra-column "subannual".

codecov[bot]: # [Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/557?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) Report
> Merging [#557](https://codecov.io/gh/IAMconsortium/pyam/pull/557?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) (fbab4df) into [main](https://codecov.io/gh/IAMconsortium/pyam/commit/6de47ccec5d477a1a35425cea557849df699238c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) (6de47cc) will **increase** coverage by `0.0%`.
> The diff coverage is `100.0%`.

[![Impacted file tree graph](https://codecov.io/gh/IAMconsortium/pyam/pull/557/graphs/tree.svg?width=650&height=150&src=pr&token=ThxWlNkonJ&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium)](https://codecov.io/gh/IAMconsortium/pyam/pull/557?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium)

```diff
@@          Coverage Diff          @@
##            main    #557   +/-   ##
=====================================
  Coverage   93.6%   93.6%           
=====================================
  Files         48      50    +2     
  Lines       5242    5278   +36     
=====================================
+ Hits        4909    4945   +36     
  Misses       333     333           
```


| [Impacted Files](https://codecov.io/gh/IAMconsortium/pyam/pull/557?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) | Coverage Δ | |
|---|---|---|
| [setup.py](https://codecov.io/gh/IAMconsortium/pyam/pull/557/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-c2V0dXAucHk=) | `0.0% <ø> (ø)` | |
| [tests/test\_core.py](https://codecov.io/gh/IAMconsortium/pyam/pull/557/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-dGVzdHMvdGVzdF9jb3JlLnB5) | `100.0% <ø> (ø)` | |
| [pyam/core.py](https://codecov.io/gh/IAMconsortium/pyam/pull/557/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-cHlhbS9jb3JlLnB5) | `94.2% <100.0%> (-0.2%)` | :arrow_down: |
| [pyam/index.py](https://codecov.io/gh/IAMconsortium/pyam/pull/557/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-cHlhbS9pbmRleC5weQ==) | `98.0% <100.0%> (+0.1%)` | :arrow_up: |
| [pyam/time.py](https://codecov.io/gh/IAMconsortium/pyam/pull/557/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-cHlhbS90aW1lLnB5) | `100.0% <100.0%> (ø)` | |
| [tests/test\_time.py](https://codecov.io/gh/IAMconsortium/pyam/pull/557/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-dGVzdHMvdGVzdF90aW1lLnB5) | `100.0% <100.0%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/557?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/557?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium). Last update [6de47cc...fbab4df](https://codecov.io/gh/IAMconsortium/pyam/pull/557?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium).

danielhuppmann: Thanks for the very swift review, @znicholls!

Patch:
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index 4866fa5..898e704 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -1,6 +1,7 @@
 # Next Release
 
- - [#556](https://github.com/IAMconsortium/pyam/pull/556) Set explicit minimum numpy version (1.19.0)
+- [#557](https://github.com/IAMconsortium/pyam/pull/557) Swap time for year keeping subannual resolution
+- [#556](https://github.com/IAMconsortium/pyam/pull/556) Set explicit minimum numpy version (1.19.0)
 
 # Release v1.0.0
 
diff --git a/pyam/core.py b/pyam/core.py
index c39838c..53871df 100755
--- a/pyam/core.py
+++ b/pyam/core.py
@@ -12,6 +12,7 @@ from pathlib import Path
 from tempfile import TemporaryDirectory
 
 from pyam._debiasing import _compute_bias
+from pyam.time import swap_time_for_year
 
 try:
     from datapackage import Package
@@ -601,43 +602,25 @@ class IamDataFrame(object):
         if not inplace:
             return ret
 
-    def swap_time_for_year(self, inplace=False):
+    def swap_time_for_year(self, subannual=False, inplace=False):
         """Convert the `time` column to `year`.
 
         Parameters
         ----------
-        inplace : bool, default False
-            if True, do operation inplace and return None
+        subannual : bool, str or func, optional
+            Merge non-year components of the "time" domain as new column "subannual".
+            Apply `strftime()` on the values of the "time" domain using `subannual`
+            as format (if a string) or using "%m-%d %H:%M%z" (if True).
+            If it is a function, apply the function on the values of the "time" domain.
+        inplace : bool, optional
+            If True, do operation inplace and return None.
 
         Raises
         ------
         ValueError
             "time" is not a column of `self.data`
         """
-        if not self.time_col == "time":
-            raise ValueError("Time domain must be datetime to use this method")
-
-        ret = self.copy() if not inplace else self
-
-        _data = ret.data
-        _data["year"] = _data["time"].apply(lambda x: x.year)
-        _data = _data.drop("time", axis="columns")
-        _index = [v if v != "time" else "year" for v in ret._LONG_IDX]
-
-        rows = _data[_index].duplicated()
-        if any(rows):
-            error_msg = "Swapping time for year causes duplicates in `data`"
-            _raise_data_error(error_msg, _data[_index])
-
-        # assign data and other attributes
-        ret._LONG_IDX = _index
-        ret._data = _data.set_index(ret._LONG_IDX).value
-        ret.time_col = "year"
-        ret._set_attributes()
-        delattr(ret, "time")
-
-        if not inplace:
-            return ret
+        return swap_time_for_year(self, subannual=subannual, inplace=inplace)
 
     def as_pandas(self, meta_cols=True):
         """Return object as a pandas.DataFrame
diff --git a/pyam/index.py b/pyam/index.py
index f5110f4..11d15bd 100644
--- a/pyam/index.py
+++ b/pyam/index.py
@@ -75,6 +75,14 @@ def replace_index_values(df, level, mapping, rows=None):
     return index.set_codes(_codes, n).set_levels(_unique_levels, n)
 
 
+def append_index_col(index, values, name, order=False):
+    """Append a list of `values` as a new column (level) to an `index`"""
+    levels = pd.Index(values).unique()
+    codes = levels.get_indexer(values)
+
+    return append_index_level(index, codes, levels, name, order)
+
+
 def append_index_level(index, codes, level, name, order=False):
     """Append a level to a pd.MultiIndex"""
     if isstr(level):
diff --git a/pyam/time.py b/pyam/time.py
new file mode 100644
index 0000000..79794eb
--- /dev/null
+++ b/pyam/time.py
@@ -0,0 +1,45 @@
+import pandas as pd
+from pyam.index import append_index_col
+from pyam.utils import _raise_data_error
+
+
+def swap_time_for_year(df, inplace, subannual=False):
+    """Internal implementation to swap 'time' domain to 'year' (as int)"""
+    if not df.time_col == "time":
+        raise ValueError("Time domain must be datetime to use this method")
+
+    ret = df.copy() if not inplace else df
+
+    index = ret._data.index
+
+    time = pd.Series(index.get_level_values("time"))
+    order = [v if v != "time" else "year" for v in index.names]
+
+    index = index.droplevel("time")
+    index = append_index_col(index, time.apply(lambda x: x.year), "year", order=order)
+
+    if subannual:
+        # if subannual is True, default to simple datetime format without year
+        if subannual is True:
+            subannual = "%m-%d %H:%M%z"
+        if isinstance(subannual, str):
+            _subannual = time.apply(lambda x: x.strftime(subannual))
+        else:
+            _subannual = time.apply(subannual)
+
+        index = append_index_col(index, _subannual, "subannual")
+
+    rows = index.duplicated()
+    if any(rows):
+        error_msg = "Swapping time for year causes duplicates in `data`"
+        _raise_data_error(error_msg, index[rows].to_frame().reset_index(drop=True))
+
+    # assign data and other attributes
+    ret._LONG_IDX = index.names
+    ret._data.index = index
+    ret.time_col = "year"
+    ret._set_attributes()
+    delattr(ret, "time")
+
+    if not inplace:
+        return ret
diff --git a/setup.py b/setup.py
index 7e1f4fb..60026fd 100755
--- a/setup.py
+++ b/setup.py
@@ -20,7 +20,7 @@ REQUIREMENTS = [
     "iam-units>=2020.4.21",
     "numpy>=1.19.0",
     "requests",
-    "pandas>=1.1.1",
+    "pandas>=1.1.1,<1.3.0",
     "pint",
     "PyYAML",
     "matplotlib>=3.2.0",


Test patch:
diff --git a/tests/test_core.py b/tests/test_core.py
index 5e5dfa8..797eb44 100644
--- a/tests/test_core.py
+++ b/tests/test_core.py
@@ -1059,39 +1059,3 @@ def test_normalize(test_df):
 def test_normalize_not_time(test_df):
     pytest.raises(ValueError, test_df.normalize, variable="foo")
     pytest.raises(ValueError, test_df.normalize, year=2015, variable="foo")
-
-
-@pytest.mark.parametrize("inplace", [True, False])
-def test_swap_time_to_year(test_df, inplace):
-    if "year" in test_df.data:
-        return  # year df not relevant for this test
-
-    exp = test_df.data.copy()
-    exp["year"] = exp["time"].apply(lambda x: x.year)
-    exp = exp.drop("time", axis="columns")
-    exp = IamDataFrame(exp)
-
-    obs = test_df.swap_time_for_year(inplace=inplace)
-
-    if inplace:
-        assert obs is None
-        obs = test_df
-
-    assert compare(obs, exp).empty
-    assert obs.year == [2005, 2010]
-    with pytest.raises(AttributeError):
-        obs.time
-
-
-@pytest.mark.parametrize("inplace", [True, False])
-def test_swap_time_to_year_errors(test_df, inplace):
-    if "year" in test_df.data:
-        with pytest.raises(ValueError):
-            test_df.swap_time_for_year(inplace=inplace)
-        return
-
-    tdf = test_df.data.copy()
-    tdf["time"] = tdf["time"].apply(lambda x: datetime.datetime(2005, x.month, x.day))
-
-    with pytest.raises(ValueError):
-        IamDataFrame(tdf).swap_time_for_year(inplace=inplace)
diff --git a/tests/test_time.py b/tests/test_time.py
new file mode 100644
index 0000000..4d6a56c
--- /dev/null
+++ b/tests/test_time.py
@@ -0,0 +1,93 @@
+import pytest
+import pandas as pd
+from datetime import datetime
+from pyam import IamDataFrame
+from pyam.testing import assert_iamframe_equal
+
+from conftest import TEST_DTS, TEST_TIME_STR, TEST_TIME_STR_HR
+
+
+def get_subannual_df(date1, date2):
+    df = pd.DataFrame(
+        [
+            ["scen_a", "Primary Energy", 2005, date1, 1.0],
+            ["scen_a", "Primary Energy", 2010, date2, 6.0],
+            ["scen_a", "Primary Energy|Coal", 2005, date1, 0.5],
+            ["scen_a", "Primary Energy|Coal", 2010, date2, 3],
+            ["scen_b", "Primary Energy", 2005, date1, 2],
+            ["scen_b", "Primary Energy", 2010, date2, 7],
+        ],
+        columns=["scenario", "variable", "year", "subannual", "value"],
+    )
+    return IamDataFrame(df, model="model_a", region="World", unit="EJ/yr")
+
+
+# this is the subannual column format used in the openENTRANCE project
+OE_DATETIME = ["2005-10-01 23:15+01:00", "2010-10-02 23:15+01:00"]
+OE_SUBANNUAL_FORMAT = lambda x: x.strftime("%m-%d %H:%M%z").replace("+0100", "+01:00")
+
+
+@pytest.mark.parametrize("inplace", [True, False])
+def test_swap_time_to_year(test_df, inplace):
+    """Swap time column for year (int) dropping subannual time resolution (default)"""
+
+    if test_df.time_col == "year":
+        pytest.skip("IamDataFrame with time domain `year` not relevant for this test.")
+
+    exp = test_df.data
+    exp["year"] = exp["time"].apply(lambda x: x.year)
+    exp = exp.drop("time", axis="columns")
+    exp = IamDataFrame(exp, meta=test_df.meta)
+
+    obs = test_df.swap_time_for_year(inplace=inplace)
+
+    if inplace:
+        assert obs is None
+        obs = test_df
+
+    assert_iamframe_equal(obs, exp)
+    match = "'IamDataFrame' object has no attribute 'time'"
+    with pytest.raises(AttributeError, match=match):
+        obs.time
+
+
+@pytest.mark.parametrize(
+    "columns, subannual, dates",
+    [
+        # use the default datetime format without `year`
+        [TEST_DTS, True, ["06-17 00:00", "07-21 00:00"]],
+        [TEST_TIME_STR, True, ["06-17 00:00", "07-21 00:00"]],
+        [TEST_TIME_STR_HR, True, ["06-17 00:00", "07-21 12:00"]],
+        # apply formatting for strftime as str
+        [TEST_DTS, "%m-%d", ["06-17", "07-21"]],
+        # apply openENTRANCE formatting with timezone
+        [OE_DATETIME, OE_SUBANNUAL_FORMAT, ["10-01 23:15+01:00", "10-02 23:15+01:00"]],
+    ],
+)
+@pytest.mark.parametrize("inplace", [True, False])
+def test_swap_time_to_year_subannual(test_pd_df, columns, subannual, dates, inplace):
+    """Swap time column for year (int) keeping subannual resolution as extra-column"""
+
+    test_pd_df.rename({2005: columns[0], 2010: columns[1]}, axis=1, inplace=True)
+    obs = IamDataFrame(test_pd_df).swap_time_for_year(subannual=subannual)
+
+    exp = get_subannual_df(dates[0], dates[1])
+    assert_iamframe_equal(obs, exp)
+
+
+def test_swap_time_to_year_errors(test_df):
+    """Assert that swapping time column for year (int) raises the expected errors"""
+
+    # swapping time to year raises when the IamDataFrame has time domain `year`
+    if test_df.time_col == "year":
+        match = "Time domain must be datetime to use this method"
+        with pytest.raises(ValueError, match=match):
+            test_df.swap_time_for_year()
+
+    else:
+        # set time column to same year so that dropping month/day leads to duplicates
+        tdf = test_df.data
+        tdf["time"] = tdf["time"].apply(lambda x: datetime(2005, x.month, x.day))
+
+        with pytest.raises(ValueError, match="Swapping time for year causes duplicate"):
+            IamDataFrame(tdf).swap_time_for_year()