Repository: IAMconsortium/pyam
Pull request #559 (2021-07-06 23:23:44)

Add attribute `dimensions`, fix compatibility with pandas v1.3
# Please confirm that this PR has done the following:

- [x] Tests Added
- [x] Documentation Added
- ~Name of contributors Added to AUTHORS.rst~
- [x] Description in RELEASE_NOTES.md Added

# Description of PR

This PR was triggered by release v1.3 of pandas, which changed the `sort_values()` method causing errors.

In the process, I inserted a new attribute `dimensions` to return the list of "columns" of the IAMC-format, following the principles of the SDMX format detailed by @khaeru at https://github.com/IAMconsortium/nomenclature/issues/10. Then, I refactored all occurrences of `_LONG_IDX` and `IamDataFrame._data.index.names` to that attribute and fixed the output of the `print()` statement to reduce ambiguity (i.e., dimensions is the index names plus the timeseries data coordinates).

The attribute `_LONG_IDX` is added explicitly to ensure compatibility with any existing code, but it is marked as deprecated and will be removed in the next (minor) release.

codecov[bot]: # [Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/559?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) Report
> Merging [#559](https://codecov.io/gh/IAMconsortium/pyam/pull/559?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) (fcf2d04) into [main](https://codecov.io/gh/IAMconsortium/pyam/commit/6de47ccec5d477a1a35425cea557849df699238c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) (6de47cc) will **increase** coverage by `0.0%`.
> The diff coverage is `95.2%`.

[![Impacted file tree graph](https://codecov.io/gh/IAMconsortium/pyam/pull/559/graphs/tree.svg?width=650&height=150&src=pr&token=ThxWlNkonJ&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium)](https://codecov.io/gh/IAMconsortium/pyam/pull/559?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium)

```diff
@@          Coverage Diff          @@
##            main    #559   +/-   ##
=====================================
  Coverage   93.6%   93.6%           
=====================================
  Files         48      48           
  Lines       5242    5248    +6     
=====================================
+ Hits        4909    4915    +6     
  Misses       333     333           
```


| [Impacted Files](https://codecov.io/gh/IAMconsortium/pyam/pull/559?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) | Coverage Δ | |
|---|---|---|
| [pyam/figures.py](https://codecov.io/gh/IAMconsortium/pyam/pull/559/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-cHlhbS9maWd1cmVzLnB5) | `38.0% <0.0%> (ø)` | |
| [setup.py](https://codecov.io/gh/IAMconsortium/pyam/pull/559/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-c2V0dXAucHk=) | `0.0% <ø> (ø)` | |
| [pyam/core.py](https://codecov.io/gh/IAMconsortium/pyam/pull/559/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-cHlhbS9jb3JlLnB5) | `94.3% <96.2%> (+<0.1%)` | :arrow_up: |
| [pyam/\_ops.py](https://codecov.io/gh/IAMconsortium/pyam/pull/559/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-cHlhbS9fb3BzLnB5) | `94.0% <100.0%> (ø)` | |
| [pyam/aggregation.py](https://codecov.io/gh/IAMconsortium/pyam/pull/559/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-cHlhbS9hZ2dyZWdhdGlvbi5weQ==) | `99.1% <100.0%> (ø)` | |
| [tests/test\_core.py](https://codecov.io/gh/IAMconsortium/pyam/pull/559/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-dGVzdHMvdGVzdF9jb3JlLnB5) | `100.0% <100.0%> (ø)` | |
| [tests/test\_feature\_append\_rename.py](https://codecov.io/gh/IAMconsortium/pyam/pull/559/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-dGVzdHMvdGVzdF9mZWF0dXJlX2FwcGVuZF9yZW5hbWUucHk=) | `100.0% <100.0%> (ø)` | |
| [pyam/utils.py](https://codecov.io/gh/IAMconsortium/pyam/pull/559/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-cHlhbS91dGlscy5weQ==) | `91.6% <0.0%> (-0.6%)` | :arrow_down: |
| [pyam/logging.py](https://codecov.io/gh/IAMconsortium/pyam/pull/559/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-cHlhbS9sb2dnaW5nLnB5) | `59.3% <0.0%> (+6.2%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/559?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/559?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium). Last update [6de47cc...fcf2d04](https://codecov.io/gh/IAMconsortium/pyam/pull/559?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium).

danielhuppmann: @gidden @znicholls, any thoughts?
znicholls: Nope, if you're happy I'm happy
danielhuppmann: > Nope, if you're happy I'm happy

Thanks Zeb!

Patch:
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index 898e704..2934474 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -1,5 +1,13 @@
 # Next Release
 
+## API changes
+
+PR [#559](https://github.com/IAMconsortium/pyam/pull/559) marked
+the attribute `_LONG_IDX` as deprecated. Please use `dimensions` instead. 
+
+## Individual updates
+
+- [#559](https://github.com/IAMconsortium/pyam/pull/559) Add attribute dimensions, fix compatibility with pandas v1.3
 - [#557](https://github.com/IAMconsortium/pyam/pull/557) Swap time for year keeping subannual resolution
 - [#556](https://github.com/IAMconsortium/pyam/pull/556) Set explicit minimum numpy version (1.19.0)
 
diff --git a/pyam/_ops.py b/pyam/_ops.py
index 921b3a0..ca1477a 100644
--- a/pyam/_ops.py
+++ b/pyam/_ops.py
@@ -47,7 +47,7 @@ KNOWN_OPS = {
 def _op_data(df, name, method, axis, fillna=None, args=(), ignore_units=False, **kwds):
     """Internal implementation of numerical operations on timeseries"""
 
-    if axis not in df._data.index.names:
+    if axis not in df.dimensions:
         raise ValueError(f"Unknown axis: {axis}")
 
     if method in KNOWN_OPS:
@@ -57,7 +57,7 @@ def _op_data(df, name, method, axis, fillna=None, args=(), ignore_units=False, *
     else:
         raise ValueError(f"Unknown method: {method}")
 
-    cols = df._data.index.names.difference([axis])
+    cols = [d for d in df.dimensions if d != axis]
 
     # replace args and and kwds with values of `df._data` if applicable
     # _data_args and _data_kwds track if an argument was replaced by `df._data` values
diff --git a/pyam/aggregation.py b/pyam/aggregation.py
index 9c88bd4..e5bb871 100644
--- a/pyam/aggregation.py
+++ b/pyam/aggregation.py
@@ -177,7 +177,7 @@ def _aggregate_time(df, variable, column, value, components, method=np.sum):
     # compute aggregate over time
     filter_args = dict(variable=variable)
     filter_args[column] = components
-    index = df._data.index.names.difference([column])
+    index = [d for d in df.dimensions if d != column]
 
     _data = pd.concat(
         [
@@ -191,7 +191,7 @@ def _aggregate_time(df, variable, column, value, components, method=np.sum):
     )
 
     # reset index-level order to original IamDataFrame
-    _data.index = _data.index.reorder_levels(df._LONG_IDX)
+    _data.index = _data.index.reorder_levels(df.dimensions)
 
     return _data
 
diff --git a/pyam/core.py b/pyam/core.py
index 53871df..758a571 100755
--- a/pyam/core.py
+++ b/pyam/core.py
@@ -30,7 +30,6 @@ try:
 except (ImportError, AttributeError):
     has_ix = False
 
-from pyam import plotting
 from pyam.run_control import run_control
 from pyam.utils import (
     write_sheet,
@@ -76,6 +75,7 @@ from pyam.index import (
     verify_index_integrity,
     replace_index_values,
 )
+from pyam.logging import deprecation_warning
 
 logger = logging.getLogger(__name__)
 
@@ -183,7 +183,6 @@ class IamDataFrame(object):
                 raise ValueError(msg)
 
         self._data, index, self.time_col, self.extra_cols = _data
-        self._LONG_IDX = self._data.index.names
 
         # define `meta` dataframe for categorization & quantitative indicators
         self.meta = pd.DataFrame(index=_make_index(self._data, cols=index))
@@ -251,8 +250,8 @@ class IamDataFrame(object):
             The maximum number of meta indicators printed
         """
         # concatenate list of index dimensions and levels
-        info = f"{type(self)}\nIndex dimensions:\n"
-        c1 = max([len(i) for i in self._data.index.names]) + 1
+        info = f"{type(self)}\nIndex:\n"
+        c1 = max([len(i) for i in self.dimensions]) + 1
         c2 = n - c1 - 5
         info += "\n".join(
             [
@@ -261,12 +260,12 @@ class IamDataFrame(object):
             ]
         )
 
-        # concatenate list of index of _data (not in META_IDX)
+        # concatenate list of index of _data (not in index.names)
         info += "\nTimeseries data coordinates:\n"
         info += "\n".join(
             [
                 f"   {i:{c1}}: {print_list(get_index_levels(self._data, i), c2)}"
-                for i in self._data.index.names
+                for i in self.dimensions
                 if i not in self.index.names
             ]
         )
@@ -382,9 +381,21 @@ class IamDataFrame(object):
     def data(self):
         """Return the timeseries data as a long :class:`pandas.DataFrame`"""
         if self.empty:  # reset_index fails on empty with `datetime` column
-            return pd.DataFrame([], columns=self._LONG_IDX + ["value"])
+            return pd.DataFrame([], columns=self.dimensions + ["value"])
         return self._data.reset_index()
 
+    @property
+    def dimensions(self):
+        """Return the list of `data` columns (index names & data coordinates)"""
+        return list(self._data.index.names)
+
+    @property
+    def _LONG_IDX(self):
+        """DEPRECATED - please use `IamDataFrame.dimensions`"""
+        # TODO: deprecated, remove for release >= 1.2
+        deprecation_warning("Use the attribute `dimensions` instead.", "This attribute")
+        return self.dimensions
+
     def copy(self):
         """Make a deepcopy of this object
 
@@ -477,7 +488,7 @@ class IamDataFrame(object):
         if self.time_col != other.time_col:
             raise ValueError("Incompatible time format (`year` vs. `time`)")
 
-        if self._data.index.names != other._data.index.names:
+        if self.dimensions != other.dimensions:
             raise ValueError("Incompatible timeseries data index dimensions")
 
         if other.empty:
@@ -493,9 +504,8 @@ class IamDataFrame(object):
         if verify_integrity:
             verify_index_integrity(_data)
 
-        # merge extra columns in `data` and set `self._LONG_IDX`
+        # merge extra columns in `data`
         ret.extra_cols += [i for i in other.extra_cols if i not in ret.extra_cols]
-        ret._LONG_IDX = IAMC_IDX + [ret.time_col] + ret.extra_cols
         ret._data = _data.sort_index()
         ret._set_attributes()
 
@@ -693,7 +703,7 @@ class IamDataFrame(object):
         if (name or (hasattr(meta, "name") and meta.name)) in [None, False]:
             raise ValueError("Must pass a name or use a named pd.Series")
         name = name or meta.name
-        if name in self._data.index.names:
+        if name in self.dimensions:
             raise ValueError(f"Column {name} already exists in `data`!")
         if name in ILLEGAL_COLS:
             raise ValueError(f"Name {name} is illegal for meta indicators!")
@@ -980,7 +990,7 @@ class IamDataFrame(object):
         mapping.update(kwargs)
 
         # determine columns that are not `model` or `scenario`
-        data_cols = set(self._LONG_IDX) - set(META_IDX)
+        data_cols = set(self.dimensions) - set(META_IDX)
 
         # changing index and data columns can cause model-scenario mismatch
         if any(i in mapping for i in META_IDX) and any(i in mapping for i in data_cols):
@@ -1033,7 +1043,7 @@ class IamDataFrame(object):
 
         # merge using `groupby().sum()` only if duplicates exist
         if has_duplicates:
-            ret._data = ret._data.reset_index().groupby(ret._LONG_IDX).sum().value
+            ret._data = ret._data.reset_index().groupby(ret.dimensions).sum().value
 
         if not inplace:
             return ret
@@ -1134,7 +1144,7 @@ class IamDataFrame(object):
         x = df.set_index(IAMC_IDX)
         x["value"] /= x[x[cols] == value]["value"]
         x = x.reset_index()
-        ret._data = format_time_col(x, self.time_col).set_index(self._LONG_IDX).value
+        ret._data = format_time_col(x, self.time_col).set_index(self.dimensions).value
 
         if not inplace:
             return ret
@@ -1200,7 +1210,7 @@ class IamDataFrame(object):
             self.append(_df, inplace=True)
         else:
             if _df is None or _df.empty:
-                return _empty_iamframe(self._LONG_IDX + ["value"])
+                return _empty_iamframe(self.dimensions + ["value"])
             return IamDataFrame(_df, meta=self.meta)
 
     def check_aggregate(
@@ -1326,7 +1336,7 @@ class IamDataFrame(object):
             self.append(_df, region=region, inplace=True)
         else:
             if _df is None or _df.empty:
-                return _empty_iamframe(self._LONG_IDX + ["value"])
+                return _empty_iamframe(self.dimensions + ["value"])
             return IamDataFrame(_df, region=region, meta=self.meta)
 
     def check_aggregate_region(
@@ -1416,7 +1426,7 @@ class IamDataFrame(object):
                 keys=[region],
                 names=["region"],
             )
-            _df.index = _df.index.reorder_levels(self._LONG_IDX)
+            _df.index = _df.index.reorder_levels(self.dimensions)
             return _df
 
     def aggregate_time(
@@ -1703,7 +1713,7 @@ class IamDataFrame(object):
             elif col == "time" and self.time_col == "time":
                 keep_col = datetime_match(self.data[col], values)
 
-            elif col in self._data.index.names:
+            elif col in self.dimensions:
                 lvl_index, lvl_codes = get_index_levels_codes(self._data, col)
 
                 codes = pattern_match(
@@ -2291,14 +2301,14 @@ class IamDataFrame(object):
 
         # perform aggregations
         if agg == "sum":
-            df = df.groupby(self._LONG_IDX).sum().reset_index()
+            df = df.groupby(self.dimensions).sum().reset_index()
 
         df = (
             df.reindex(columns=columns_orderd)
             .sort_values(SORT_IDX)
             .reset_index(drop=True)
         )
-        ret._data = format_time_col(df, self.time_col).set_index(self._LONG_IDX).value
+        ret._data = format_time_col(df, self.time_col).set_index(self.dimensions).value
 
         if not inplace:
             return ret
@@ -2608,7 +2618,7 @@ def concat(dfs, ignore_meta_conflict=False, **kwargs):
 
     df = as_iamdataframe(dfs[0])
     ret_data, ret_meta = [df._data], df.meta
-    index, time_col = df._data.index.names, df.time_col
+    index, time_col = df.dimensions, df.time_col
 
     for df in dfs[1:]:
         # skip merging meta if element is a pd.DataFrame
@@ -2618,10 +2628,8 @@ def concat(dfs, ignore_meta_conflict=False, **kwargs):
         if df.time_col != time_col:
             raise ValueError("Items have incompatible time format ('year' vs. 'time')!")
 
-        if df._data.index.names != index:
-            raise ValueError(
-                "Items have incompatible timeseries data index dimensions!"
-            )
+        if df.dimensions != index:
+            raise ValueError("Items have incompatible timeseries data dimensions!")
 
         ret_data.append(df._data)
         if _meta_merge:
diff --git a/pyam/figures.py b/pyam/figures.py
index 5c3a212..cd63692 100755
--- a/pyam/figures.py
+++ b/pyam/figures.py
@@ -41,7 +41,7 @@ def sankey(df, mapping):
             "Missing optional dependency `plotly`, use pip or conda to install"
         )
     # Check for duplicates
-    for col in [name for name in df._data.index.names if name != "variable"]:
+    for col in [name for name in df.dimensions if name != "variable"]:
         levels = get_index_levels(df._data, col)
         if len(levels) > 1:
             raise ValueError(f"Non-unique values in column {col}: {levels}")
diff --git a/pyam/time.py b/pyam/time.py
index 79794eb..f5a3dac 100644
--- a/pyam/time.py
+++ b/pyam/time.py
@@ -35,7 +35,6 @@ def swap_time_for_year(df, inplace, subannual=False):
         _raise_data_error(error_msg, index[rows].to_frame().reset_index(drop=True))
 
     # assign data and other attributes
-    ret._LONG_IDX = index.names
     ret._data.index = index
     ret.time_col = "year"
     ret._set_attributes()
diff --git a/setup.py b/setup.py
index 60026fd..920d5c6 100755
--- a/setup.py
+++ b/setup.py
@@ -12,7 +12,7 @@ logo = r"""
 """
 
 # NOTE TO DEVS
-# If you change a minimum version below, please explicitly set that
+# If you change a minimum version below, please explicitly implement the change
 # in our minimum-reqs test in the file ./.github/workflows/pytest-depedency.yml
 # Please also add a section "Dependency changes" to the release notes
 REQUIREMENTS = [
@@ -20,7 +20,7 @@ REQUIREMENTS = [
     "iam-units>=2020.4.21",
     "numpy>=1.19.0",
     "requests",
-    "pandas>=1.1.1,<1.3.0",
+    "pandas>=1.1.1",
     "pint",
     "PyYAML",
     "matplotlib>=3.2.0",


Test patch:
diff --git a/tests/test_core.py b/tests/test_core.py
index 797eb44..fb2d1ae 100644
--- a/tests/test_core.py
+++ b/tests/test_core.py
@@ -192,7 +192,7 @@ def test_print(test_df_year):
     exp = "\n".join(
         [
             "<class 'pyam.core.IamDataFrame'>",
-            "Index dimensions:",
+            "Index:",
             " * model    : model_a (1)",
             " * scenario : scen_a, scen_b (2)",
             "Timeseries data coordinates:",
@@ -215,7 +215,7 @@ def test_print_empty(test_df_year):
     exp = "\n".join(
         [
             "<class 'pyam.core.IamDataFrame'>",
-            "Index dimensions:",
+            "Index:",
             " * model    : (0)",
             " * scenario : (0)",
             "Timeseries data coordinates:",
@@ -324,6 +324,12 @@ def test_unit_mapping(test_pd_df):
     assert obs == {"Primary Energy": ["EJ/yr", "foo"], "Primary Energy|Coal": "EJ/yr"}
 
 
+def test_dimensions(test_df):
+    """Assert that the dimensions attribute works as expected"""
+    assert test_df.dimensions == IAMC_IDX + [test_df.time_col]
+    assert test_df._LONG_IDX == IAMC_IDX + [test_df.time_col]
+
+
 def test_filter_empty_df():
     # test for issue seen in #254
     df = IamDataFrame(data=df_empty)
@@ -818,7 +824,7 @@ def _r5_regions_exp(df):
     df = df.filter(region="World", keep=False)
     data = df.data
     data["region"] = "R5MAF"
-    return sort_data(data, df._LONG_IDX)
+    return sort_data(data, df.dimensions)
 
 
 def test_map_regions_r5(reg_df):
@@ -904,7 +910,7 @@ def test_48b():
         )
     )
     obs = df.map_regions("iso", region_col="r5_region").data
-    obs = sort_data(obs[obs.region.isin(["SSD", "SDN"])], df._LONG_IDX)
+    obs = sort_data(obs[obs.region.isin(["SSD", "SDN"])], df.dimensions)
 
     pd.testing.assert_frame_equal(obs, exp, check_index_type=False)
 
@@ -1021,7 +1027,7 @@ def test_concat_incompatible_cols(test_pd_df):
     test_pd_df["extra_col"] = "foo"
     df2 = IamDataFrame(test_pd_df)
 
-    match = "Items have incompatible timeseries data index dimensions!"
+    match = "Items have incompatible timeseries data dimensions!"
     with pytest.raises(ValueError, match=match):
         concat([df1, df2])
 
diff --git a/tests/test_feature_append_rename.py b/tests/test_feature_append_rename.py
index 63f1f57..ba5c687 100644
--- a/tests/test_feature_append_rename.py
+++ b/tests/test_feature_append_rename.py
@@ -122,14 +122,12 @@ def test_append_extra_col(test_df, shuffle_cols):
     other_data = base_data[base_data["variable"] == "Primary Energy"].copy()
     other_data["variable"] = "Primary Energy|Gas"
     other_df = IamDataFrame(other_data)
-    # cast from FrozenList to list for test
-    other_df._LONG_IDX = list(other_df._LONG_IDX)
 
     if shuffle_cols:
-        c1_idx = other_df._LONG_IDX.index("col_1")
-        c2_idx = other_df._LONG_IDX.index("col_2")
-        other_df._LONG_IDX[c1_idx] = "col_2"
-        other_df._LONG_IDX[c2_idx] = "col_1"
+        c1_idx = other_df.dimensions.index("col_1")
+        c2_idx = other_df.dimensions.index("col_2")
+        other_df.dimensions[c1_idx] = "col_2"
+        other_df.dimensions[c2_idx] = "col_1"
 
     res = base_df.append(other_df)