Repository: IAMconsortium/pyam
Pull request #600 (2022-01-04 13:48:30)

Add a `diff()` method
# Please confirm that this PR has done the following:

- [x] Tests Added
- [x] Documentation Added
- ~Name of contributors Added to AUTHORS.rst~
- [x] Description in RELEASE_NOTES.md Added

# Description of PR

This method implements a `diff()` method similar to the equivalent pandas-method, see https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.diff.html.


codecov[bot]: # [Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/600?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) Report
> Merging [#600](https://codecov.io/gh/IAMconsortium/pyam/pull/600?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) (fc2d366) into [main](https://codecov.io/gh/IAMconsortium/pyam/commit/9f84bcd25f60dce9eb4678fbdb228836b6e5c86b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) (9f84bcd) will **increase** coverage by `0.0%`.
> The diff coverage is `100.0%`.

[![Impacted file tree graph](https://codecov.io/gh/IAMconsortium/pyam/pull/600/graphs/tree.svg?width=650&height=150&src=pr&token=ThxWlNkonJ&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium)](https://codecov.io/gh/IAMconsortium/pyam/pull/600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium)

```diff
@@          Coverage Diff          @@
##            main    #600   +/-   ##
=====================================
  Coverage   94.2%   94.2%           
=====================================
  Files         50      50           
  Lines       5331    5348   +17     
=====================================
+ Hits        5022    5039   +17     
  Misses       309     309           
```


| [Impacted Files](https://codecov.io/gh/IAMconsortium/pyam/pull/600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) | Coverage Δ | |
|---|---|---|
| [pyam/core.py](https://codecov.io/gh/IAMconsortium/pyam/pull/600/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-cHlhbS9jb3JlLnB5) | `94.3% <100.0%> (+<0.1%)` | :arrow_up: |
| [tests/test\_ops.py](https://codecov.io/gh/IAMconsortium/pyam/pull/600/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-dGVzdHMvdGVzdF9vcHMucHk=) | `100.0% <100.0%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/600?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/600?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium). Last update [9f84bcd...fc2d366](https://codecov.io/gh/IAMconsortium/pyam/pull/600?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium).


Patch:
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index 28102df..edfa22c 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -1,5 +1,6 @@
 # Next release
 
+- [#600](https://github.com/IAMconsortium/pyam/pull/600) Add a `diff()` method
 - [#592](https://github.com/IAMconsortium/pyam/pull/592) Fix for running in jupyter-lab notebooks
 - [#590](https://github.com/IAMconsortium/pyam/pull/590) Update expected figures of plotting tests to use matplotlib 3.5
 - [#586](https://github.com/IAMconsortium/pyam/pull/586) Improve error reporting for non-numeric data in any value column
diff --git a/pyam/core.py b/pyam/core.py
index 22dc67e..45b1283 100755
--- a/pyam/core.py
+++ b/pyam/core.py
@@ -1913,6 +1913,7 @@ class IamDataFrame(object):
         See Also
         --------
         add, multiply, divide
+        diff : Compute the difference of timeseries data along the time dimension.
         apply : Apply a custom function on the timeseries data along any axis.
 
         Notes
@@ -2081,6 +2082,10 @@ class IamDataFrame(object):
         :class:`IamDataFrame` or **None**
             Computed timeseries data or None if `append=True`.
 
+        See Also
+        --------
+        add, subtract, multiply, divide, diff
+
         Notes
         -----
         This function uses the :mod:`pint` package and the :mod:`iam-units` registry
@@ -2098,6 +2103,60 @@ class IamDataFrame(object):
         else:
             return IamDataFrame(_value, meta=self.meta)
 
+    def diff(self, mapping, periods=1, append=False):
+        """Compute the difference of timeseries data along the time dimension
+
+        This methods behaves as if applying :meth:`pandas.DataFrame.diff` on the
+        timeseries data in wide format.
+
+        Parameters
+        ----------
+        mapping : dict
+            Mapping of *variable* item(s) to the name(s) of the diff-ed timeseries data,
+            e.g.,
+
+            .. code-block:: python
+
+               {"current variable": "name of the diff-ed variable", ...}
+
+        periods : int, optional
+            Periods to shift for calculating difference, accepts negative values;
+            passed to :meth:`pandas.DataFrame.diff`.
+        append : bool, optional
+            Whether to append aggregated timeseries data to this instance.
+
+        Returns
+        -------
+        :class:`IamDataFrame` or **None**
+            Computed timeseries data or None if `append=True`.
+
+        See Also
+        --------
+        subtract, apply, interpolate
+
+        Notes
+        -----
+        This method behaves as if applying :meth:`pandas.DataFrame.diff` by row in a
+        wide data format, so the difference is computed on the previous existing value.
+        This can lead to unexpected results if the data has inconsistent period lengths.
+
+        Use the following to ensure that no missing values exist prior to computing
+        the difference:
+
+        .. code-block:: python
+
+            df.interpolate(time=df.year)
+
+        """
+        cols = [d for d in self.dimensions if d != self.time_col]
+        _value = self.filter(variable=mapping)._data.groupby(cols).diff(periods=periods)
+        _value.index = replace_index_values(_value.index, "variable", mapping)
+
+        if append:
+            self.append(_value, inplace=True)
+        else:
+            return IamDataFrame(_value, meta=self.meta)
+
     def _to_file_format(self, iamc_index):
         """Return a dataframe suitable for writing to a file"""
         df = self.timeseries(iamc_index=iamc_index).reset_index()


Test patch:
diff --git a/tests/test_ops.py b/tests/test_ops.py
index e7884d0..149a2dd 100644
--- a/tests/test_ops.py
+++ b/tests/test_ops.py
@@ -4,7 +4,7 @@ import operator
 import pint
 from iam_units import registry
 
-from pyam import IamDataFrame
+from pyam import IamDataFrame, IAMC_IDX
 from pyam.testing import assert_iamframe_equal
 from pyam._ops import _op_data
 
@@ -342,3 +342,50 @@ def test_ops_unknown_method(test_df_year):
     """Using an unknown method raises an error"""
     with pytest.raises(ValueError, match="Unknown method: foo"):
         _op_data(test_df_year, "_", "foo", "variable")
+
+
+@pytest.mark.parametrize("periods, year", (({}, 2010), ({"periods": -1}, 2005)))
+@pytest.mark.parametrize("append", (False, True))
+def test_diff(test_df_year, periods, year, append):
+    """Test `diff` method including non-default periods argument"""
+
+    exp = IamDataFrame(
+        pd.DataFrame(
+            [
+                ["model_a", "scen_a", "World", "foo", "EJ/yr", 5],
+                ["model_a", "scen_a", "World", "bar", "EJ/yr", 2.5],
+                ["model_a", "scen_b", "World", "foo", "EJ/yr", 5],
+            ],
+            columns=IAMC_IDX + [year],
+        ),
+        meta=test_df_year.meta,
+    )
+    # values are negative if computing diff in a negative direction
+    if year == 2005:
+        exp._data = -exp._data
+
+    mapping = {"Primary Energy": "foo", "Primary Energy|Coal": "bar"}
+
+    if append:
+        obs = test_df_year.copy()
+        obs.diff(mapping=mapping, append=True, **periods)
+        assert_iamframe_equal(test_df_year.append(exp), obs)
+    else:
+        obs = test_df_year.diff(mapping=mapping, **periods)
+        assert_iamframe_equal(exp, obs)
+
+
+@pytest.mark.parametrize("append", (False, True))
+def test_diff_empty(test_df_year, append):
+    """Assert that `diff` with only one time period returns empty"""
+
+    df = test_df_year.filter(year=2005)
+    mapping = {"Primary Energy": "foo", "Primary Energy|Coal": "bar"}
+
+    if append:
+        obs = df.copy()
+        obs.diff(mapping=mapping, append=True)
+        assert_iamframe_equal(df, obs)  # assert that no data was added
+    else:
+        obs = df.diff(mapping=mapping)
+        assert obs.empty