Repository: IAMconsortium/pyam
Pull request #405 (2020-06-26 07:37:53)

Feature/recursive aggregation
# Please confirm that this PR has done the following:

- [x] Tests Added
- [x] Documentation Added
- [x] Description in RELEASE_NOTES.md Added

# Description of PR

This PR adds the functionality to recursively aggregating variables. Based on #385


danielhuppmann: Note that tests are failing because the IIASA infrastructure is unavailable today...
danielhuppmann: thanks @tburandt! By the way, you could have simply accepted the suggestion directly on this page, no need (any more) to copy-paste it into your editor and then commit...

Can you click "resolve" on all @stickler-ci comments that you cleaned up, please?
tburandt: > thanks @tburandt! By the way, you could have simply accepted the suggestion directly on this page, no need (any more) to copy-paste it into your editor and then commit...
> 
> Can you click "resolve" on all @stickler-ci comments that you cleaned up, please?

Ha, ok, I wasn't actually aware of that (obviously), now I feel a bit ashamed :D

tburandt: Thanks @danielhuppmann for all your support in this PR :)
tburandt: @danielhuppmann 
With the current GENeSYS-MOD dataset, the time for the recursive feature is almost identical to the manual aggregation of the subcategories. After several runs with different scenarios, I couldn't observe any significant difference. However, in my original script, I also used the append-parameter for creating the final dataset for all existing subcategories. 
danielhuppmann: took the liberty to resolve the merge conflict...
codecov[bot]: # [Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/405?src=pr&el=h1) Report
> Merging [#405](https://codecov.io/gh/IAMconsortium/pyam/pull/405?src=pr&el=desc) into [master](https://codecov.io/gh/IAMconsortium/pyam/commit/0d5bc4af5cb426fed11c9a93263c23e041363142&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `95.23%`.

[![Impacted file tree graph](https://codecov.io/gh/IAMconsortium/pyam/pull/405/graphs/tree.svg?width=650&height=150&src=pr&token=ThxWlNkonJ)](https://codecov.io/gh/IAMconsortium/pyam/pull/405?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #405      +/-   ##
==========================================
+ Coverage   93.25%   93.26%   +0.01%     
==========================================
  Files          35       35              
  Lines        4090     4129      +39     
==========================================
+ Hits         3814     3851      +37     
- Misses        276      278       +2     
```


| [Impacted Files](https://codecov.io/gh/IAMconsortium/pyam/pull/405?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pyam/core.py](https://codecov.io/gh/IAMconsortium/pyam/pull/405/diff?src=pr&el=tree#diff-cHlhbS9jb3JlLnB5) | `92.14% <71.42%> (-0.22%)` | :arrow_down: |
| [pyam/\_aggregate.py](https://codecov.io/gh/IAMconsortium/pyam/pull/405/diff?src=pr&el=tree#diff-cHlhbS9fYWdncmVnYXRlLnB5) | `96.52% <100.00%> (+0.68%)` | :arrow_up: |
| [tests/test\_feature\_aggregate.py](https://codecov.io/gh/IAMconsortium/pyam/pull/405/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9mZWF0dXJlX2FnZ3JlZ2F0ZS5weQ==) | `100.00% <100.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/405?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/405?src=pr&el=footer). Last update [0d5bc4a...0ebda62](https://codecov.io/gh/IAMconsortium/pyam/pull/405?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

danielhuppmann: thank you @tburandt! let's keep in mind that this feature could probably be faster - so let's revisit it and refactor the code when aggregating the results takes longer than running the model...

Patch:
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index b34dfb8..aa0250e 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -14,6 +14,7 @@ a utility function `pyam.plotting.mpl_args_to_meta_cols()`.
 - [#409](https://github.com/IAMconsortium/pyam/pull/409) Remove travis and appveyor CI config.
 - [#408](https://github.com/IAMconsortium/pyam/pull/408) Update badges on the docs page and readme.
 - [#407](https://github.com/IAMconsortium/pyam/pull/407) Add Codecov to Github Actions CI.
+- [#405](https://github.com/IAMconsortium/pyam/pull/405) Add ability for recursivley aggregating variables.
 - [#402](https://github.com/IAMconsortium/pyam/pull/402) Refactor `as_pandas()` and docs for more consistent description of `meta`.
 - [#401](https://github.com/IAMconsortium/pyam/pull/401) Read credentials for IIASA-API-Connection by default from known location.
 - [#396](https://github.com/IAMconsortium/pyam/pull/396) Enable casting to `IamDataFrame` multiple times.
diff --git a/pyam/_aggregate.py b/pyam/_aggregate.py
index b5e1b13..e631f31 100644
--- a/pyam/_aggregate.py
+++ b/pyam/_aggregate.py
@@ -6,6 +6,8 @@ from pyam.logging import adjust_log_level
 from pyam.utils import (
     islistable,
     isstr,
+    find_depth,
+    reduce_hierarchy,
     KNOWN_FUNCS,
     META_IDX,
 )
@@ -51,6 +53,41 @@ def _aggregate(df, variable, components=None, method=np.sum):
     return _group_and_agg(_df, [], method)
 
 
+def _aggregate_recursive(df, variable, method=np.sum):
+    """Recursive aggregation along the variable tree"""
+    _df_aggregated = None
+    _df = df.copy()
+
+    # iterate over variables to find all subcategories to be aggregated
+    sub_variables = []
+    for d in reversed(range(1, max(find_depth(df.data.variable)) + 1)):
+        depth = find_depth(df.data.variable)
+        var_list = (
+            df.data.variable[[i == d for i in depth]]
+            .unique()
+        )
+        vars_up = pd.Series(
+            [reduce_hierarchy(i, -1) for i in var_list]).unique()
+
+        if [i for i, entr in enumerate(vars_up) if entr.startswith(variable)]:
+            for v in vars_up:
+                sub_variables.append(v)
+
+    sub_variables = reversed(sorted(set(sub_variables)))
+
+    # iterate over subcategories (bottom-up) and perform aggregation
+    for entry in sub_variables:
+        _df.aggregate(variable=entry, append=True)
+        _df_temp = _df.aggregate(variable=entry, append=False)
+
+        if _df_aggregated is None:
+            _df_aggregated = _df_temp.copy()
+        else:
+            _df_aggregated.append(_df_temp, inplace=True)
+
+    return _df_aggregated.data
+
+
 def _aggregate_region(df, variable, region, subregions=None, components=False,
                       method='sum', weight=None):
     """Internal implementation for aggregating data over subregions"""
diff --git a/pyam/core.py b/pyam/core.py
index b3c659c..78bed07 100755
--- a/pyam/core.py
+++ b/pyam/core.py
@@ -35,6 +35,7 @@ from pyam.utils import (
     sort_data,
     to_int,
     find_depth,
+    reduce_hierarchy,
     pattern_match,
     years_match,
     month_match,
@@ -52,7 +53,7 @@ from pyam.read_ixmp import read_ix
 from pyam.timeseries import fill_series
 from pyam.plotting import mpl_args_to_meta_cols
 from pyam._aggregate import _aggregate, _aggregate_region, _aggregate_time,\
-    _group_and_agg
+    _aggregate_recursive, _group_and_agg
 from pyam.units import convert_unit
 from pyam.logging import deprecation_warning
 
@@ -884,7 +885,8 @@ class IamDataFrame(object):
         if not inplace:
             return ret
 
-    def aggregate(self, variable, components=None, method='sum', append=False):
+    def aggregate(self, variable, components=None, method='sum',
+                  recursive=False, append=False):
         """Aggregate timeseries components or sub-categories within each region
 
         Parameters
@@ -897,11 +899,26 @@ class IamDataFrame(object):
         method : func or str, default 'sum'
             method to use for aggregation,
             e.g. :func:`numpy.mean`, :func:`numpy.sum`, 'min', 'max'
+        recursive : bool, default False
+            iterate recursively over all subcategories of `variable`
         append : bool, default False
             append the aggregate timeseries to `self` and return None,
             else return aggregate timeseries as new :class:`IamDataFrame`
+
+        Notes
+        -----
+        The aggregation function interprets any missing values
+        (:any:`numpy.nan`) for individual components as 0.
         """
-        _df = _aggregate(self, variable, components=components, method=method)
+
+        if recursive is True:
+            if components is not None:
+                msg = 'Recursive aggregation cannot take explicit components'
+                raise ValueError(msg)
+            _df = _aggregate_recursive(self, variable, method=method)
+        else:
+            _df = _aggregate(self, variable, components=components,
+                             method=method)
 
         # return None if there is nothing to aggregate
         if _df is None:


Test patch:
diff --git a/tests/test_feature_aggregate.py b/tests/test_feature_aggregate.py
index 3834c1a..fd9a2d9 100644
--- a/tests/test_feature_aggregate.py
+++ b/tests/test_feature_aggregate.py
@@ -6,7 +6,7 @@ import pandas as pd
 from pyam import check_aggregate, IamDataFrame, IAMC_IDX
 from pyam.testing import assert_iamframe_equal
 
-from conftest import DTS_MAPPING
+from conftest import TEST_YEARS, DTS_MAPPING
 
 LONG_IDX = IAMC_IDX + ['year']
 
@@ -40,6 +40,16 @@ PRICE_MAX_DF = pd.DataFrame([
     columns=LONG_IDX + ['value']
 )
 
+RECURSIVE_DF = pd.DataFrame([
+    ['Secondary Energy|Electricity', 'EJ/yr', 5, 19.],
+    ['Secondary Energy|Electricity|Wind', 'EJ/yr', 5, 17],
+    ['Secondary Energy|Electricity|Wind|Offshore', 'EJ/yr', 1, 5],
+    ['Secondary Energy|Electricity|Wind|Onshore', 'EJ/yr', 4, 12],
+    ['Secondary Energy|Electricity|Solar', 'EJ/yr', np.nan, 2],
+],
+    columns=['variable', 'unit'] + TEST_YEARS
+)
+
 
 @pytest.mark.parametrize("variable,data", (
     ('Primary Energy', PE_MAX_DF),
@@ -121,6 +131,30 @@ def test_aggregate_by_list_with_components_raises(simple_df):
     pytest.raises(ValueError, simple_df.aggregate, v, components=components)
 
 
+@pytest.mark.parametrize("time_col", (('year'), ('time')))
+def test_aggregate_recursive(time_col):
+    # use the feature `recursive=True`
+    data = RECURSIVE_DF if time_col == 'year' \
+        else RECURSIVE_DF.rename(DTS_MAPPING, axis='columns')
+    df = IamDataFrame(data, model='model_a', scenario='scen_a', region='World')
+    df2 = df.rename(scenario={'scen_a': 'scen_b'})
+    df2.data.value *= 2
+    df.append(df2, inplace=True)
+
+    # create object without variables to be aggregated
+    v = 'Secondary Energy|Electricity'
+    agg_vars = [f'{v}{i}' for i in ['', '|Wind']]
+    df_minimal = df.filter(variable=agg_vars, keep=False)
+
+    # return recursively aggregated data as new object
+    obs = df_minimal.aggregate(variable=v, recursive=True)
+    assert_iamframe_equal(obs, df.filter(variable=agg_vars))
+
+    # append to `self`
+    df_minimal.aggregate(variable=v, recursive=True, append=True)
+    assert_iamframe_equal(df_minimal, df)
+
+
 def test_aggregate_empty(simple_df):
     assert simple_df.aggregate('foo') is None