Repository: IAMconsortium/pyam
Pull request #659 (2022-05-13 07:05:46)

add offset method to compute offsets relative to base-year values
# Please confirm that this PR has done the following:

- [x] Tests Added
- [x] Documentation Added
- [x] Name of contributors Added to AUTHORS.rst
- [x] Description in RELEASE_NOTES.md Added

# Description of PR

This PR introduces an `offset()` method which allows a user to compute offsets relative to base-time-periods in the same style as `normalize()`.

        For example, offsetting from `year=2005` will provide data
        *relative* to `year=2005` such that the value in 2005 is 0 and 
        all other values `value[year] - value[2005]`.

        Conceptually this operation performs as:
        ```
        df - df.filter(**kwargs) + padding
        ```

codecov[bot]: # [Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/659?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) Report
> Merging [#659](https://codecov.io/gh/IAMconsortium/pyam/pull/659?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) (2457149) into [main](https://codecov.io/gh/IAMconsortium/pyam/commit/c5e8f9c0ffdf3311b7278e3e4111718f6a7b7134?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) (c5e8f9c) will **increase** coverage by `0.0%`.
> The diff coverage is `100.0%`.

> :exclamation: Current head 2457149 differs from pull request most recent head e5a8ceb. Consider uploading reports for the commit e5a8ceb to get more accurate results

```diff
@@          Coverage Diff          @@
##            main    #659   +/-   ##
=====================================
  Coverage   94.5%   94.6%           
=====================================
  Files         59      59           
  Lines       5742    5769   +27     
=====================================
+ Hits        5431    5458   +27     
  Misses       311     311           
```


| [Impacted Files](https://codecov.io/gh/IAMconsortium/pyam/pull/659?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) | Coverage Δ | |
|---|---|---|
| [pyam/core.py](https://codecov.io/gh/IAMconsortium/pyam/pull/659/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-cHlhbS9jb3JlLnB5) | `94.6% <100.0%> (+<0.1%)` | :arrow_up: |
| [tests/test\_core.py](https://codecov.io/gh/IAMconsortium/pyam/pull/659/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-dGVzdHMvdGVzdF9jb3JlLnB5) | `100.0% <100.0%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/659?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/659?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium). Last update [c5e8f9c...e5a8ceb](https://codecov.io/gh/IAMconsortium/pyam/pull/659?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium).

danielhuppmann: Thanks @coroa for suggesting the high-performance implementation alternative!

I don't mind adding the function directly for now. If discoverability becomes an issue, we could delegate methods operating only on the timeseries data to a submodule, similar to what I did with the [`compute`](https://pyam-iamc.readthedocs.io/en/stable/api/compute.html) module...
gidden: Hi both - have updated based on your suggestions. Should be ready.

Patch:
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index 6654c69..b3b370d 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -1,5 +1,6 @@
 # Next release
 
+- [#659](https://github.com/IAMconsortium/pyam/pull/659) Add an `offset` method
 - [#657](https://github.com/IAMconsortium/pyam/pull/657) Add an `IamSlice` class
 
 # Release v1.4.0
diff --git a/pyam/core.py b/pyam/core.py
index 55a86af..47baca0 100755
--- a/pyam/core.py
+++ b/pyam/core.py
@@ -1269,6 +1269,45 @@ class IamDataFrame(object):
         if not inplace:
             return ret
 
+    def offset(self, padding=0, fill_value=None, inplace=False, **kwargs):
+        """Compute new data which is offset from a specific data point
+
+        For example, offsetting from `year=2005` will provide data
+        *relative* to `year=2005` such that the value in 2005 is 0 and
+        all other values `value[year] - value[2005]`.
+
+        Conceptually this operation performs as:
+        ```
+        df - df.filter(**kwargs) + padding
+        ```
+
+        Note: Currently only supports normalizing to a specific time.
+
+        Parameters
+        ----------
+        padding : float, optional
+            an additional offset padding
+        fill_value : float or None, optional
+            Applied on subtraction. Fills exisiting missing (NaN) values. See
+            https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.subtract.html
+        inplace : bool, optional
+            if :obj:`True`, do operation inplace and return None
+        kwargs
+            the column and value on which to offset (e.g., `year=2005`)
+        """
+        if len(kwargs) > 1 or self.time_col not in kwargs:
+            raise ValueError("Only time(year)-based normalization supported")
+        ret = self.copy() if not inplace else self
+        data = ret._data
+        value = kwargs[self.time_col]
+        base_value = data.loc[data.index.isin([value], level=self.time_col)].droplevel(
+            self.time_col
+        )
+        ret._data = data.subtract(base_value, fill_value=fill_value) + padding
+
+        if not inplace:
+            return ret
+
     def aggregate(
         self,
         variable,
@@ -2793,6 +2832,8 @@ def concat(objs, ignore_meta_conflict=False, **kwargs):
     The *meta* attributes are merged only for those objects of *objs* that are passed
     as :class:`IamDataFrame` instances.
 
+    The :attr:`dimensions` and :attr:`index` names of all elements of *dfs* must be
+    identical. The returned IamDataFrame inherits the dimensions and index names.
     """
     if not islistable(objs) or isinstance(objs, pd.DataFrame):
         raise TypeError(f"'{objs.__class__.__name__}' object is not iterable")
@@ -2844,7 +2885,11 @@ def concat(objs, ignore_meta_conflict=False, **kwargs):
             )
 
     # return as new IamDataFrame, this will verify integrity as part of `__init__()`
-    return IamDataFrame(pd.concat(ret_data, verify_integrity=False), meta=ret_meta)
+    return IamDataFrame(
+        pd.concat(ret_data, verify_integrity=False),
+        meta=ret_meta,
+        index=ret_meta.index.names,
+    )
 
 
 def read_datapackage(path, data="data", meta="meta"):


Test patch:
diff --git a/tests/test_core.py b/tests/test_core.py
index f87d9c1..f13973b 100644
--- a/tests/test_core.py
+++ b/tests/test_core.py
@@ -789,3 +789,24 @@ def test_normalize(test_df):
 def test_normalize_not_time(test_df):
     pytest.raises(ValueError, test_df.normalize, variable="foo")
     pytest.raises(ValueError, test_df.normalize, year=2015, variable="foo")
+
+
+@pytest.mark.parametrize("padding", [0, 2])
+def test_offset(test_df, padding):
+    exp = test_df.data.copy().reset_index(drop=True)
+    exp.loc[1::2, "value"] -= exp["value"][::2].values - padding
+    exp.loc[::2, "value"] -= exp["value"][::2].values - padding
+    # only call with kwarg if padding != 0 (the default)
+    kwargs = {"padding": padding} if padding else {}
+    if "year" in test_df.data:
+        obs = test_df.offset(year=2005, **kwargs).data.reset_index(drop=True)
+    else:
+        obs = test_df.offset(
+            time=datetime.datetime(2005, 6, 17), **kwargs
+        ).data.reset_index(drop=True)
+    pd.testing.assert_frame_equal(obs, exp)
+
+
+def test_offset_not_time(test_df):
+    pytest.raises(ValueError, test_df.offset, variable="foo")
+    pytest.raises(ValueError, test_df.offset, year=2015, variable="foo")
diff --git a/tests/test_feature_append_concat.py b/tests/test_feature_append_concat.py
index 0e030ec..0ea0f90 100644
--- a/tests/test_feature_append_concat.py
+++ b/tests/test_feature_append_concat.py
@@ -104,6 +104,38 @@ def test_concat(test_df, reverse, iterable):
     npt.assert_array_equal(ts.iloc[2].values, ts.iloc[3].values)
 
 
+def test_concat_non_default_index():
+    # Test that merging two IamDataFrames with identical, non-standard index dimensions
+    # preserves the index.
+
+    df1 = IamDataFrame(
+        pd.DataFrame(
+            [["model_a", "scenario_a", "region_a", "variable_a", "unit", 1, 1]],
+            columns=IAMC_IDX + ["version", 2005],
+        ),
+        index=META_IDX + ["version"],
+    )
+    df2 = IamDataFrame(
+        pd.DataFrame(
+            [["model_a", "scenario_a", "region_a", "variable_a", "unit", 2, 2]],
+            columns=IAMC_IDX + ["version", 2005],
+        ),
+        index=META_IDX + ["version"],
+    )
+    exp = IamDataFrame(
+        pd.DataFrame(
+            [
+                ["model_a", "scenario_a", "region_a", "variable_a", "unit", 1, 1],
+                ["model_a", "scenario_a", "region_a", "variable_a", "unit", 2, 2],
+            ],
+            columns=IAMC_IDX + ["version", 2005],
+        ),
+        index=META_IDX + ["version"],
+    )
+
+    assert_iamframe_equal(exp, concat([df1, df2]))
+
+
 @pytest.mark.parametrize("reverse", (False, True))
 def test_concat_with_pd_dataframe(test_df, reverse):
     other = test_df.filter(scenario="scen_b").rename({"scenario": {"scen_b": "scen_c"}})