Repository: IAMconsortium/pyam

Issue:
Helpful accessors confounded by pandas regression
The test below fails and I can't see why. This is the underlying cause of https://github.com/iiasa/climate-assessment/pull/36 I think.

In short: if you subset an `IamDataFrame` and then create new instances within some loop (probably a bad pattern, but let's ignore that for now), the metadata is still based on the original data rather than the subset. In the example, below this means that it looks like a scenario provides a variable when it actually doesn't.

@phackstock cc @danielhuppmann in case it helps with your search for a cause and making sure the bug doesn't come back.

```python
def test_climate_assessment_bug():
    test_df = pd.DataFrame(
        [
            ["model_a", "scen_a", "World", "Primary Energy", "EJ/yr", 1, 6.0],
            ["model_a", "scen_a", "World", "Primary Energy|Coal", "EJ/yr", 0.5, 3],
            ["model_a", "scen_b", "World", "Primary Energy", "EJ/yr", 2, 7],
        ],
        columns=["model", "scenario", "region", "variable", "unit", 2020, 2030],
    )
    test_df = IamDataFrame(test_df)
    for (model, scen), df_scen in test_df.timeseries().groupby(["model", "scenario"]):
        if not (model == "model_a" and scen == "scen_b"):
            continue

        # Primary Energy|Coal isn't part of the dataset from which `df_scen_pyam` is initialised
        assert "Primary Energy|Coal" not in df_scen.index.get_level_values("variable").unique()

        df_scen_pyam = IamDataFrame(df_scen)
        # Yet Primary Energy|Coal appears in the model_a, scen_b subset's `.variable` attribute
        assert "Primary Energy|Coal" not in df_scen_pyam.variable

        # I think this is a Pandas bug
```

Discussion:
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/IAMconsortium/pyam/pull/763?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) Report
> Merging [#763](https://app.codecov.io/gh/IAMconsortium/pyam/pull/763?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) (4ec2855) into [main](https://app.codecov.io/gh/IAMconsortium/pyam/commit/4bfcec0ae0ae9815e67b283e88acf004e989cac2?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) (4bfcec0) will **increase** coverage by `0.0%`.
> The diff coverage is `100.0%`.

```diff
@@          Coverage Diff          @@
##            main    #763   +/-   ##
=====================================
  Coverage   94.4%   94.4%           
=====================================
  Files         59      59           
  Lines       6080    6085    +5     
=====================================
+ Hits        5744    5749    +5     
  Misses       336     336           
```


| [Files Changed](https://app.codecov.io/gh/IAMconsortium/pyam/pull/763?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) | Coverage Δ | |
|---|---|---|
| [pyam/utils.py](https://app.codecov.io/gh/IAMconsortium/pyam/pull/763?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-cHlhbS91dGlscy5weQ==) | `92.7% <100.0%> (+<0.1%)` | :arrow_up: |
| [tests/test\_core.py](https://app.codecov.io/gh/IAMconsortium/pyam/pull/763?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-dGVzdHMvdGVzdF9jb3JlLnB5) | `100.0% <100.0%> (ø)` | |



Patch:
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index 171057e..b5df329 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -14,6 +14,7 @@ When importing an xlsx file created with pyam < 2.0, which has an "exclude" colu
 
 ## Individual updates
 
+- [#763](https://github.com/IAMconsortium/pyam/pull/763) Implement a fix against carrying over unused levels when initializing from an indexed pandas object
 - [#759](https://github.com/IAMconsortium/pyam/pull/759) Excise "exclude" column from meta and add a own attribute
 - [#747](https://github.com/IAMconsortium/pyam/pull/747) Drop support for Python 3.7 #747
 
diff --git a/pyam/utils.py b/pyam/utils.py
index c64b3be..87017aa 100644
--- a/pyam/utils.py
+++ b/pyam/utils.py
@@ -378,6 +378,9 @@ def format_data(df, index, **kwargs):
 
         df = df.reorder_levels(index + REQUIRED_COLS + [time_col] + extra_cols).dropna()
 
+        # remove unused levels to guard against issue #762
+        df.index = df.index.remove_unused_levels()
+
     else:
         if isinstance(df, pd.Series):
             if not df.name:


Test patch:
diff --git a/tests/test_core.py b/tests/test_core.py
index 41ba0da..698f1ba 100644
--- a/tests/test_core.py
+++ b/tests/test_core.py
@@ -128,6 +128,20 @@ def test_init_df_from_timeseries(test_df):
     pd.testing.assert_frame_equal(df.timeseries(), test_df.timeseries())
 
 
+def test_init_df_from_timeseries_unused_levels(test_df):
+    # this test guards against regression for the bug
+    # reported in https://github.com/IAMconsortium/pyam/issues/762
+
+    for (model, scenario), data in test_df.timeseries().groupby(["model", "scenario"]):
+        # we're only interested in the second model-scenario combination
+        if model == "model_a" and scenario == "scen_b":
+            df = IamDataFrame(data)
+
+    # pandas 2.0 does not remove unused levels (here: "Primary Energy|Coal") in groupby
+    # we check that unused levels are removed at initialization of the IamDataFrame
+    assert df.variable == ["Primary Energy"]
+
+
 def test_init_df_with_extra_col(test_pd_df):
     tdf = test_pd_df.copy()