Repository: IAMconsortium/pyam
Pull request #411 (2020-06-29 14:59:38)

Add a feature to pass an explicit weight dataframe to `downscale_region()`
# Please confirm that this PR has done the following:

- [x] Tests Added
- [x] Documentation Added
- [x] Description in RELEASE_NOTES.md Added

# Description of PR

This PR adds an option to pass an explicit weight-dataframe with region (and possibly others) as index dimension and the time domain as columns. Implementation based on many discussions with @maartenbrinkering on his specific use case.


codecov[bot]: # [Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/411?src=pr&el=h1) Report
> Merging [#411](https://codecov.io/gh/IAMconsortium/pyam/pull/411?src=pr&el=desc) into [master](https://codecov.io/gh/IAMconsortium/pyam/commit/824ce31fa9aa8f7fe9326f35ef8461ed9082e173&el=desc) will **increase** coverage by `0.90%`.
> The diff coverage is `92.50%`.

[![Impacted file tree graph](https://codecov.io/gh/IAMconsortium/pyam/pull/411/graphs/tree.svg?width=650&height=150&src=pr&token=ThxWlNkonJ)](https://codecov.io/gh/IAMconsortium/pyam/pull/411?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #411      +/-   ##
==========================================
+ Coverage   92.49%   93.39%   +0.90%     
==========================================
  Files          35       35              
  Lines        4051     4059       +8     
==========================================
+ Hits         3747     3791      +44     
+ Misses        304      268      -36     
```


| [Impacted Files](https://codecov.io/gh/IAMconsortium/pyam/pull/411?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [setup.py](https://codecov.io/gh/IAMconsortium/pyam/pull/411/diff?src=pr&el=tree#diff-c2V0dXAucHk=) | `0.00% <ø> (ø)` | |
| [tests/test\_tutorials.py](https://codecov.io/gh/IAMconsortium/pyam/pull/411/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF90dXRvcmlhbHMucHk=) | `96.42% <0.00%> (+45.81%)` | :arrow_up: |
| [pyam/core.py](https://codecov.io/gh/IAMconsortium/pyam/pull/411/diff?src=pr&el=tree#diff-cHlhbS9jb3JlLnB5) | `92.11% <86.66%> (-0.17%)` | :arrow_down: |
| [tests/test\_feature\_downscale.py](https://codecov.io/gh/IAMconsortium/pyam/pull/411/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9mZWF0dXJlX2Rvd25zY2FsZS5weQ==) | `100.00% <100.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/411?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/411?src=pr&el=footer). Last update [0a25b29...28797b6](https://codecov.io/gh/IAMconsortium/pyam/pull/411?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

danielhuppmann: @fabiosferra, maybe also of interest to you?
l-welder: Tested the new features locally and everything seems to be working as intended. Looking forward to add to this (=
danielhuppmann: Thanks @l-welder for the review and noticing the change in the docs. We actually started out specifying the default in the docstrings, but (if I recall correctly) @khaeru pointed out that the [NumPy-style docstrings](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_numpy.html) convention is to just write "optional", because the default value is anyway obvious from the function signature. So I'm changing this whenever I stumble over the legacy docstrings...

Patch:
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index e4f9a2e..c272156 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -11,6 +11,7 @@ a utility function `pyam.plotting.mpl_args_to_meta_cols()`.
 ## Individual Updates
 
 - [#412](https://github.com/IAMconsortium/pyam/pull/412) Add building the docs to GitHub Actions CI.
+- [#411](https://github.com/IAMconsortium/pyam/pull/411) Add feature to pass an explicit weight dataframe to `downscale_region()`.
 - [#410](https://github.com/IAMconsortium/pyam/pull/410) Activate tutorial tests on GitHub Actions CI (py3.8).
 - [#409](https://github.com/IAMconsortium/pyam/pull/409) Remove travis and appveyor CI config.
 - [#408](https://github.com/IAMconsortium/pyam/pull/408) Update badges on the docs page and readme.
diff --git a/doc/source/tutorials/aggregating_downscaling_consistency.ipynb b/doc/source/tutorials/aggregating_downscaling_consistency.ipynb
index f32a032..31031ce 100644
--- a/doc/source/tutorials/aggregating_downscaling_consistency.ipynb
+++ b/doc/source/tutorials/aggregating_downscaling_consistency.ipynb
@@ -33,7 +33,8 @@
     "1. Aggregate timeseries over sectors (i.e., sub-categories)\n",
     "2. Aggregate timeseries over regions including weighted average\n",
     "3. Downscale timeseries given at a region level to sub-regions using a proxy variable\n",
-    "4. Check the internal consistency of a scenario (ensemble)"
+    "4. Downscale timeseries using an explicit weighting dataframe\n",
+    "5. Check the internal consistency of a scenario (ensemble)"
    ]
   },
   {
@@ -292,13 +293,22 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 3. Downscaling timeseries data to subregions\n",
+    "## 3. Downscaling timeseries data to subregions using a proxy\n",
     "\n",
     "The inverse operation of regional aggregation is \"downscaling\" of timeseries data given at a regional level to a number of subregions, usually using some other data as proxy to divide and allocate the total to the subregions.\n",
     "\n",
     "This section shows an example using the [downscale_region()](https://pyam-iamc.readthedocs.io/en/stable/api/iamdataframe.html#pyam.IamDataFrame.downscale_region) function to divide the total primary energy demand using population as a proxy."
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df.filter(variable='Population').timeseries()"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -334,7 +344,42 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 4. Checking the internal consistency of a scenario (ensemble)\n",
+    "## 4. Downscaling timeseries data to subregions using a weighting dataframe\n",
+    "\n",
+    "In cases where using existing data directly as a proxy (as illustrated in the previous section) is not practical,\n",
+    "a user can also create a weighting dataframe and pass that directly to the `downscale_region()` function.\n",
+    "\n",
+    "The example below uses the weighting factors implied by the population variable for easy comparison to the previous section."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "weight = pd.DataFrame(\n",
+    "    [[0.66, 0.6], [0.33, 0.4]],\n",
+    "    index=pd.Series(['reg_a', 'reg_b'], name='region'),\n",
+    "    columns=pd.Series([2005, 2010], name='year')\n",
+    ")\n",
+    "weight"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df.downscale_region(var_list, weight=weight).timeseries()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 5. Checking the internal consistency of a scenario (ensemble)\n",
     "\n",
     "The previous sections illustrated two functions to validate specific variables across their sectors (sub-categories) or regional disaggregation.\n",
     "These two functions are combined in the [check_internal_consistency()](https://pyam-iamc.readthedocs.io/en/stable/api/iamdataframe.html#pyam.IamDataFrame.check_internal_consistency) feature.\n",
diff --git a/pyam/core.py b/pyam/core.py
index 8b7eb15..53f3be5 100755
--- a/pyam/core.py
+++ b/pyam/core.py
@@ -1124,31 +1124,48 @@ class IamDataFrame(object):
             df.meta = self.meta.loc[_make_index(df.data)]
             return df
 
-    def downscale_region(self, variable, proxy, region='World',
-                         subregions=None, append=False):
+    def downscale_region(self, variable, region='World', subregions=None,
+                         proxy=None, weight=None, append=False):
         """Downscale a timeseries to a number of subregions
 
         Parameters
         ----------
         variable : str or list of str
             variable(s) to be downscaled
-        proxy : str
-            variable to be used as proxy (i.e, weight) for the downscaling
-        region : str, default 'World'
+        region : str, optional
             region from which data will be downscaled
-        subregions : list of str
+        subregions : list of str, optional
             list of subregions, defaults to all regions other than `region`
+            (if using `proxy`) or `region` index (if using `weight`)
+        proxy : str, optional
+            variable (within the :class:`IamDataFrame`) to be used as proxy
+            for regional downscaling
+        weight : class:`pandas.DataFrame`, optional
+            dataframe with time dimension as columns (year or
+            :class:`datetime.datetime`) and regions[, model, scenario] as index
         append : bool, optional
             append the downscaled timeseries to `self` and return None,
             else return downscaled data as new IamDataFrame
         """
-        # get default subregions if not specified
-        subregions = subregions or self._all_other_regions(region)
-
-        # filter relevant data, transform to `pd.Series` with appropriate index
-        _df = self.data[self._apply_filters(variable=proxy, region=subregions)]
-        _proxy = _df.set_index(self._get_cols(['region', self.time_col])).value
-        _total = _df.groupby(self._get_cols([self.time_col])).value.sum()
+        if proxy is not None and weight is not None:
+            raise ValueError(
+                'Using both `proxy` and `weight` arguments is not valid')
+        elif proxy is not None:
+            # get default subregions if not specified and select data from self
+            subregions = subregions or self._all_other_regions(region)
+            rows = self._apply_filters(variable=proxy, region=subregions)
+            cols = self._get_cols(['region', self.time_col])
+            _proxy = self.data[rows].set_index(cols).value
+        elif weight is not None:
+            # downselect weight to subregions or remove `region` from index
+            if subregions is not None:
+                rows = weight.index.isin(subregions, level='region')
+            else:
+                rows = ~weight.index.isin([region], level='region')
+            _proxy = weight[rows].stack()
+        else:
+            raise ValueError(
+                'Either a `proxy` or `weight` argument is required')
 
         _value = (
             self.data[self._apply_filters(variable=variable, region=region)]
@@ -1157,6 +1174,7 @@ class IamDataFrame(object):
         )
 
         # compute downscaled data
+        _total = _proxy.groupby(self.time_col).sum()
         _data = _value * _proxy / _total
 
         if append is True:


Test patch:
diff --git a/tests/test_feature_downscale.py b/tests/test_feature_downscale.py
index 5fdb18f..e48ed44 100644
--- a/tests/test_feature_downscale.py
+++ b/tests/test_feature_downscale.py
@@ -2,11 +2,33 @@ import pytest
 from pyam.testing import assert_iamframe_equal
 
 
+def test_downscale_region_without_method_raises(simple_df):
+    # downscale_region without specifying a method
+    variable = 'Primary Energy'
+    pytest.raises(ValueError, simple_df.downscale_region, variable=variable)
+
+
+def test_downscale_region_with_multiple_methods_raises(simple_df):
+    # downscale_region with specifying both weight and proxy raises
+
+    # create weighting dataframe
+    weight_df = (
+        simple_df.filter(variable='Population').data
+        .pivot_table(index='region', columns=simple_df.time_col,
+                     values='value')
+    )
+
+    # call downscale_region with both proxy and weight
+    variable = 'Primary Energy'
+    pytest.raises(ValueError, simple_df.downscale_region, variable=variable,
+                  proxy='Population', weight=weight_df)
+
+
 @pytest.mark.parametrize("variable", (
     ('Primary Energy'),
     (['Primary Energy', 'Primary Energy|Coal']),
 ))
-def test_downscale_region(simple_df, variable):
+def test_downscale_region_with_proxy(simple_df, variable):
     simple_df.set_meta([1], name='test')
     regions = ['reg_a', 'reg_b']
 
@@ -19,3 +41,59 @@ def test_downscale_region(simple_df, variable):
     inplace = simple_df.filter(variable=variable, region=regions, keep=False)
     inplace.downscale_region(variable, proxy='Population', append=True)
     assert_iamframe_equal(inplace, simple_df)
+
+
+@pytest.mark.parametrize("variable, index", (
+    ('Primary Energy', ['region']),
+    (['Primary Energy', 'Primary Energy|Coal'], ['region']),
+    (['Primary Energy', 'Primary Energy|Coal'], ['model', 'region']),
+))
+def test_downscale_region_with_weight(simple_df, variable, index):
+    simple_df.set_meta([1], name='test')
+    regions = ['reg_a', 'reg_b']
+
+    # create weighting dataframe
+    weight_df = (
+        simple_df.filter(variable='Population', region=regions).data
+        .pivot_table(index=index, columns=simple_df.time_col,
+                     values='value')
+    )
+
+    # return as new IamDataFrame
+    obs = simple_df.downscale_region(variable, weight=weight_df)
+    exp = simple_df.filter(variable=variable, region=regions)
+    assert_iamframe_equal(exp, obs)
+
+    # append to `self` (after removing to-be-downscaled timeseries)
+    inplace = simple_df.filter(variable=variable, region=regions, keep=False)
+    inplace.downscale_region(variable, weight=weight_df, append=True)
+    assert_iamframe_equal(inplace, simple_df)
+
+
+@pytest.mark.parametrize("variable, index", (
+    ('Primary Energy', ['region']),
+    (['Primary Energy', 'Primary Energy|Coal'], ['model', 'region']),
+))
+def test_downscale_region_with_weight_subregions(simple_df, variable, index):
+    simple_df.set_meta([1], name='test')
+    regions = ['reg_a', 'reg_b']
+
+    # create weighting dataframe with an extra "duplicate" region
+    weight_df = (
+        simple_df.filter(variable='Population')
+        .rename(region={'reg_a': 'duplicate'}, append=True)  # add extra region
+        .data
+        .pivot_table(index=index, columns=simple_df.time_col,
+                     values='value')
+    )
+
+    # return as new IamDataFrame
+    ds_args = dict(variable=variable, weight=weight_df, subregions=regions)
+    obs = simple_df.downscale_region(**ds_args)
+    exp = simple_df.filter(variable=variable, region=regions)
+    assert_iamframe_equal(exp, obs)
+
+    # append to `self` (after removing to-be-downscaled timeseries)
+    inplace = simple_df.filter(variable=variable, region=regions, keep=False)
+    inplace.downscale_region(**ds_args, append=True)
+    assert_iamframe_equal(inplace, simple_df)