Repository: IAMconsortium/pyam

Issue:
df.aggregate_region() when inconsistent index (e.g. years)
There are instances when doing weighted aggregation of  regions, when years between the variable and the weight variable may not be aligned or one variable has more years than the other.  One gets error message:
```
  File "c:\github\pyam\pyam\aggregation.py", line 216, in _agg_weight
    raise ValueError("Inconsistent index between variable and weight!")

ValueError: Inconsistent index between variable and weight!
```
which comes from [here](https://github.com/IAMconsortium/pyam/blob/6de47ccec5d477a1a35425cea557849df699238c/pyam/aggregation.py#L215):
```
    if not data.droplevel(["variable", "unit"]).index.equals(weight.index):
        raise ValueError("Inconsistent index between variable and weight!")
```

It would be nice if there's a way to improve the logic here - e.g. for example, the minimum requirement should be not that the indexes are equal, but that the index of the variable A being weighted, is 100% present within the weight variable B. Im rusty on set notation, but basically that index A is a subset of index B?

Any quick  suggestion  how this should be done and/or arguments against?



Discussion:
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/IAMconsortium/pyam/pull/792?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) Report
> Merging [#792](https://app.codecov.io/gh/IAMconsortium/pyam/pull/792?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) (6d11700) into [main](https://app.codecov.io/gh/IAMconsortium/pyam/commit/f023ec3444e0653408dbf95616b31fff745a2ca0?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) (f023ec3) will **increase** coverage by `0.1%`.
> The diff coverage is `100.0%`.

```diff
@@           Coverage Diff           @@
##            main    #792     +/-   ##
=======================================
+ Coverage   94.5%   94.7%   +0.1%     
=======================================
  Files         62      62             
  Lines       5966    5996     +30     
=======================================
+ Hits        5643    5681     +38     
+ Misses       323     315      -8     
```


| [Files](https://app.codecov.io/gh/IAMconsortium/pyam/pull/792?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) | Coverage Δ | |
|---|---|---|
| [pyam/aggregation.py](https://app.codecov.io/gh/IAMconsortium/pyam/pull/792?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-cHlhbS9hZ2dyZWdhdGlvbi5weQ==) | `99.2% <100.0%> (+<0.1%)` | :arrow_up: |
| [tests/test\_feature\_aggregate.py](https://app.codecov.io/gh/IAMconsortium/pyam/pull/792?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-dGVzdHMvdGVzdF9mZWF0dXJlX2FnZ3JlZ2F0ZS5weQ==) | `99.0% <100.0%> (+0.1%)` | :arrow_up: |
| [tests/test\_feature\_growth\_rate.py](https://app.codecov.io/gh/IAMconsortium/pyam/pull/792?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-dGVzdHMvdGVzdF9mZWF0dXJlX2dyb3d0aF9yYXRlLnB5) | `100.0% <100.0%> (ø)` | |
| [tests/test\_io.py](https://app.codecov.io/gh/IAMconsortium/pyam/pull/792?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-dGVzdHMvdGVzdF9pby5weQ==) | `100.0% <100.0%> (ø)` | |

... and [2 files with indirect coverage changes](https://app.codecov.io/gh/IAMconsortium/pyam/pull/792/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium)


danielhuppmann: I vaguely recall that @orichters expressed interest in a more flexible handling of the region-aggregation-with-weights...?
orichters: > I vaguely recall that @orichters expressed interest in a more flexible handling of the region-aggregation-with-weights...?

I do not 😃. Maybe @Renato-Rodrigues, if it was someone from the REMIND team?

Patch:
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index e7d85bc..d373650 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -1,5 +1,7 @@
 # Release v2.0.0
 
+- [#792](https://github.com/IAMconsortium/pyam/pull/792) Support region-aggregation with weights-index >> data-index
+
 ## Highlights
 
 - Use **ixmp4** as dependency for better integration with the IIASA Scenario Explorer database infrastructure 
diff --git a/pyam/aggregation.py b/pyam/aggregation.py
index ee781b6..a02c028 100644
--- a/pyam/aggregation.py
+++ b/pyam/aggregation.py
@@ -4,7 +4,7 @@ import logging
 from itertools import compress
 
 from pyam.index import replace_index_values
-from pyam.logging import adjust_log_level
+from pyam.logging import adjust_log_level, format_log_message
 from pyam.str import find_depth, is_str, reduce_hierarchy
 from pyam.utils import KNOWN_FUNCS, is_list_like, to_list
 from pyam._compare import _compare
@@ -116,7 +116,10 @@ def _aggregate_region(
         raise ValueError("Using weights and components in one operation not supported.")
 
     # default subregions to all regions other than `region`
-    subregions = subregions or df._all_other_regions(region, variable)
+    if weight is None:
+        subregions = subregions or df._all_other_regions(region, variable)
+    else:
+        subregions = subregions or df._all_other_regions(region, [variable, weight])
 
     if not len(subregions):
         logger.info(
@@ -214,10 +217,28 @@ def _agg_weight(data, weight, method, drop_negative_weights):
         raise ValueError("Only method 'np.sum' allowed for weighted average.")
 
     weight = weight.droplevel(["variable", "unit"])
+    data_index = data.droplevel(["variable", "unit"]).index
+
+    # check that weights exist for all data rows
+    missing_weights = data_index.difference(weight.index)
+    if not missing_weights.empty:
+        raise ValueError(
+            format_log_message(
+                "Missing weights for the following data rows", missing_weights
+            )
+        )
 
-    if not data.droplevel(["variable", "unit"]).index.equals(weight.index):
-        raise ValueError("Inconsistent index between variable and weight!")
+    # warn if no data exists for available weights
+    missing_data = weight.index.difference(data_index)
+    if not missing_data.empty:
+        logger.warning(
+            format_log_message(
+                "Ignoring weights for the following missing data rows", missing_data
+            )
+        )
+        weight[missing_data] = np.nan
 
+    # remove (and warn) negative values from weights due to strange behavior
     if drop_negative_weights is True:
         if any(weight < 0):
             logger.warning(


Test patch:
diff --git a/tests/test_feature_aggregate.py b/tests/test_feature_aggregate.py
index 10c27e6..54f75ba 100644
--- a/tests/test_feature_aggregate.py
+++ b/tests/test_feature_aggregate.py
@@ -1,5 +1,5 @@
 import pytest
-import logging
+import re
 
 import numpy as np
 import pandas as pd
@@ -238,7 +238,7 @@ def test_check_aggregate_region_log(simple_df, caplog):
 @pytest.mark.parametrize(
     "variable",
     (
-        ("Primary Energy"),
+        "Primary Energy",
         (["Primary Energy", "Primary Energy|Coal", "Primary Energy|Wind"]),
     ),
 )
@@ -252,7 +252,7 @@ def test_aggregate_region_append(simple_df, variable):
 @pytest.mark.parametrize(
     "variable",
     (
-        ("Primary Energy"),
+        "Primary Energy",
         (["Primary Energy", "Primary Energy|Coal", "Primary Energy|Wind"]),
     ),
 )
@@ -315,7 +315,13 @@ def test_aggregate_region_with_weights(simple_df, caplog):
     exp = simple_df.filter(variable=v, region="World")
     assert_iamframe_equal(simple_df.aggregate_region(v, weight=w), exp)
 
-    # test that dropping negative weights works as expected
+
+def test_aggregate_region_with_negative_weights(simple_df, caplog):
+    # carbon price shouldn't be summed but be weighted by emissions
+    v = "Price|Carbon"
+    w = "Emissions|CO2"
+
+    # dropping negative weights works as expected
     neg_weights_df = simple_df.copy()
     neg_weights_df._data[18] = -6
     exp = simple_df.filter(variable=v, region="World", year=2010)
@@ -329,7 +335,7 @@ def test_aggregate_region_with_weights(simple_df, caplog):
     idx = caplog.messages.index(msg)
     assert caplog.records[idx].levelname == "WARNING"
 
-    # test that not dropping negative weights works as expected
+    # *not* dropping negative weights works as expected
     exp = simple_df.filter(variable=v, region="World")
     exp._data[0] = -8
     assert_iamframe_equal(
@@ -337,16 +343,57 @@ def test_aggregate_region_with_weights(simple_df, caplog):
     )
 
 
-def test_aggregate_region_with_weights_raises(simple_df):
+@pytest.mark.parametrize(
+    "filter_arg,log_message",
+    (
+        (dict(year=2010), ""),
+        (dict(), "model_a   scen_a  reg_b  2005\n1  "),
+    ),
+)
+def test_aggregate_region_with_weights_inconsistent_index(
+    simple_df, caplog, filter_arg, log_message
+):
     # carbon price shouldn't be summed but be weighted by emissions
     v = "Price|Carbon"
     w = "Emissions|CO2"
 
-    # inconsistent index of variable and weight raises an error
-    _df = simple_df.filter(variable=w, region="reg_b", keep=False)
-    with pytest.raises(ValueError, match="Inconsistent index between variable and wei"):
+    log_message = "\n0  " + log_message + "model_a   scen_a  reg_b  2010"
+    if simple_df.time_domain == "datetime":
+        time_col = "     time"
+        log_message = log_message.replace(" 2005", "2005-06-17").replace(
+            " 2010", "2010-07-21"
+        )
+    else:
+        time_col = "year"
+
+    # missing weight row raises an error
+    _df = simple_df.filter(variable=w, region="reg_b", keep=False, **filter_arg)
+    match = r"Missing weights for the following data.*\n.*" + re.escape(log_message)
+    with pytest.raises(ValueError, match=match):
         _df.aggregate_region(v, weight=w)
 
+    # missing data row prints a warning (data-index is a subset of weight-index)
+    exp = simple_df.filter(variable=v, region="World")
+    if not filter_arg:
+        exp._data[0] = 1.0
+    exp._data[1] = 30.0
+    _df = simple_df.filter(variable=v, region="reg_b", keep=False, **filter_arg)
+    assert_iamframe_equal(_df.aggregate_region(v, weight=w), exp)
+
+    msg = (
+        "Ignoring weights for the following missing data rows:\n"
+        f"     model scenario region  {time_col}" + log_message
+    )
+
+    idx = caplog.messages.index(msg)
+    assert caplog.records[idx].levelname == "WARNING"
+
+
+def test_aggregate_region_with_weights_raises(simple_df):
+    # carbon price shouldn't be summed but be weighted by emissions
+    v = "Price|Carbon"
+    w = "Emissions|CO2"
+
     # using weight and method other than 'sum' raises an error
     pytest.raises(ValueError, simple_df.aggregate_region, v, method="max", weight="bar")
 
diff --git a/tests/test_feature_growth_rate.py b/tests/test_feature_growth_rate.py
index fa9ea3b..708e98b 100644
--- a/tests/test_feature_growth_rate.py
+++ b/tests/test_feature_growth_rate.py
@@ -66,5 +66,5 @@ def test_growth_rate_timeseries(x2010, rates):
 def test_growth_rate_timeseries_fails(value):
     """Check that a timeseries reaching/crossing 0 raises"""
 
-    with pytest.raises(ValueError, match="Cannot compute growth rate when*."):
+    with pytest.raises(ValueError, match="Cannot compute growth rate when"):
         growth_rate(pd.Series([1.0, value]))
diff --git a/tests/test_io.py b/tests/test_io.py
index c358e95..fddf907 100644
--- a/tests/test_io.py
+++ b/tests/test_io.py
@@ -43,7 +43,7 @@ def test_not_a_file():
 
 def test_io_list():
     # initializing with a list raises an error
-    match = r"Initializing from list is not supported,*."
+    match = "Initializing from list is not supported,"
     with pytest.raises(ValueError, match=match):
         IamDataFrame([1, 2])