Repository: IAMconsortium/pyam
Pull request #627 (2022-02-09 22:38:35)

Implement `append` with inconsistent time domain
# Please confirm that this PR has done the following:

- [x] Tests Added

# Description of PR

**Note**: this PR is targeted to the `feature/year-and-datetime` branch that collects all developments related to #596.

This PR extends the `append()` method in case two IamDataFrame instances have inconsistent time domains.

codecov[bot]: # [Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/627?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) Report
> Merging [#627](https://codecov.io/gh/IAMconsortium/pyam/pull/627?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) (b1fa54e) into [feature/year-and-datetime](https://codecov.io/gh/IAMconsortium/pyam/commit/c3c1a5f697c837fa5194481a151d299e501c3097?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) (c3c1a5f) will **decrease** coverage by `0.0%`.
> The diff coverage is `82.8%`.

[![Impacted file tree graph](https://codecov.io/gh/IAMconsortium/pyam/pull/627/graphs/tree.svg?width=650&height=150&src=pr&token=ThxWlNkonJ&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium)](https://codecov.io/gh/IAMconsortium/pyam/pull/627?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium)

```diff
@@                     Coverage Diff                     @@
##           feature/year-and-datetime    #627     +/-   ##
===========================================================
- Coverage                       94.5%   94.4%   -0.1%     
===========================================================
  Files                             57      57             
  Lines                           5590    5612     +22     
===========================================================
+ Hits                            5283    5301     +18     
- Misses                           307     311      +4     
```


| [Impacted Files](https://codecov.io/gh/IAMconsortium/pyam/pull/627?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) | Coverage Δ | |
|---|---|---|
| [tests/test\_feature\_append.py](https://codecov.io/gh/IAMconsortium/pyam/pull/627/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-dGVzdHMvdGVzdF9mZWF0dXJlX2FwcGVuZC5weQ==) | `94.5% <75.0%> (-5.5%)` | :arrow_down: |
| [pyam/core.py](https://codecov.io/gh/IAMconsortium/pyam/pull/627/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-cHlhbS9jb3JlLnB5) | `94.6% <80.0%> (+<0.1%)` | :arrow_up: |
| [pyam/time.py](https://codecov.io/gh/IAMconsortium/pyam/pull/627/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-cHlhbS90aW1lLnB5) | `98.0% <100.0%> (+1.9%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/627?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/627?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium). Last update [c3c1a5f...b1fa54e](https://codecov.io/gh/IAMconsortium/pyam/pull/627?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium).


Patch:
diff --git a/pyam/core.py b/pyam/core.py
index 8175a80..2e5776e 100755
--- a/pyam/core.py
+++ b/pyam/core.py
@@ -541,10 +541,7 @@ class IamDataFrame(object):
             other = IamDataFrame(other, **kwargs)
             ignore_meta_conflict = True
 
-        if self.time_col != other.time_col:
-            raise ValueError("Incompatible time format (`year` vs. `time`)")
-
-        if self.dimensions != other.dimensions:
+        if self.extra_cols != other.extra_cols:
             raise ValueError("Incompatible timeseries data index dimensions")
 
         if other.empty:
@@ -552,6 +549,12 @@ class IamDataFrame(object):
 
         ret = self.copy() if not inplace else self
 
+        if ret.time_col != other.time_col:
+            if ret.time_col == "year":
+                ret.swap_year_for_time(inplace=True)
+            else:
+                other = other.swap_year_for_time(inplace=False)
+
         # merge `meta` tables
         ret.meta = merge_meta(ret.meta, other.meta, ignore_meta_conflict)
 
diff --git a/pyam/time.py b/pyam/time.py
index afc8422..40209c7 100644
--- a/pyam/time.py
+++ b/pyam/time.py
@@ -53,23 +53,25 @@ def swap_year_for_time(df, inplace):
 
     if not df.time_col == "year":
         raise ValueError("Time domain must be 'year' to use this method")
-    if "subannual" not in df.extra_cols:
-        raise ValueError("Data must have a dimension 'subannual' to use this method")
 
     ret = df.copy() if not inplace else df
     index = ret._data.index
-    order = [v if v != "year" else "time" for v in index.names].remove("subannual")
 
-    time_cols = ["year", "subannual"]
-    time_values = zip(*[index.get_level_values(c) for c in time_cols])
-    time = list(map(dateutil.parser.parse, [f"{y}-{s}" for y, s in time_values]))
+    order = [v if v != "year" else "time" for v in index.names]
 
-    index = index.droplevel(["year", "subannual"])
-    index = append_index_col(index, time, "time", order=order)
+    if "subannual" in df.extra_cols:
+        order = order.remove("subannual")
+        time_values = zip(*[index.get_level_values(c) for c in ["year", "subannual"]])
+        time = list(map(dateutil.parser.parse, [f"{y}-{s}" for y, s in time_values]))
+        index = index.droplevel(["year", "subannual"])
+        ret.extra_cols.remove("subannual")
+    else:
+        time = index.get_level_values("year")
+        index = index.droplevel(["year"])
 
-    # assign data and other attributes
+    # add new index column, assign data and other attributes
+    index = append_index_col(index, time, "time", order=order)
     ret._data.index = index
-    ret.extra_cols.remove("subannual")
     ret.time_col = "time"
     ret._set_attributes()
     delattr(ret, "year")


Test patch:
diff --git a/tests/test_feature_append.py b/tests/test_feature_append.py
index 562b9ee..55b7638 100644
--- a/tests/test_feature_append.py
+++ b/tests/test_feature_append.py
@@ -2,8 +2,9 @@ import pytest
 import numpy as np
 import pandas as pd
 from numpy import testing as npt
+from datetime import datetime
 
-from pyam import IamDataFrame
+from pyam import IamDataFrame, IAMC_IDX, assert_iamframe_equal
 
 from .conftest import META_COLS
 
@@ -42,6 +43,39 @@ def test_append_other_scenario(test_df):
     npt.assert_array_equal(ts.iloc[2].values, ts.iloc[3].values)
 
 
+@pytest.mark.parametrize("other", ("time", "year"))
+@pytest.mark.parametrize("time", (datetime(2010, 7, 21), "2010-07-21 00:00:00"))
+@pytest.mark.parametrize("inplace", (True, False))
+def test_append_time_domain(test_pd_df, test_df_mixed, other, time, inplace):
+
+    df_year = IamDataFrame(test_pd_df[IAMC_IDX + [2005]], meta=test_df_mixed.meta)
+    df_time = IamDataFrame(
+        test_pd_df[IAMC_IDX + [2010]].rename({2010: time}, axis="columns")
+    )
+
+    # append `df_time` to `df_year`
+    if other == "time":
+        if inplace:
+            obs = df_year.copy()
+            obs.append(df_time, inplace=True)
+        else:
+            obs = df_year.append(df_time)
+            # assert that original object was not modified
+            assert df_year.year == [2005]
+
+    # append `df_year` to `df_time`
+    else:
+        if inplace:
+            obs = df_time.copy()
+            obs.append(df_year, inplace=True)
+        else:
+            obs = df_time.append(df_year)
+            # assert that original object was not modified
+            assert df_time.time == pd.Index([datetime(2010, 7, 21)])
+
+    assert_iamframe_equal(obs, test_df_mixed)
+
+
 def test_append_reconstructed_time(test_df):
     # check appending dfs with equal time cols created by different methods
     other = test_df.filter(scenario="scen_b").rename({"scenario": {"scen_b": "scen_c"}})