Repository: sktime/sktime

Issue:
[BUG] forecasters broadcast predictions on wrong instances when label as str
**Describe the bug**
<!--
A clear and concise description of what the bug is.
-->

When a panel datatype is given where the instances are labelled as str such that a random name followed by some int, e.g *user1*, *series1* , *T1* etc. the prediction that was made on the instances sometimes gets switched. below is a minimal example to show the problem. 

I suppose this occurs for all forecasters, albeit the test I did wasn't exhaustive, only on `TrendForecaster`, `NaiveForecaster` and `ExponentialSmothing`. I haven't looked at the logic of how forecasters made predictions for hierarchical or panel datatypes but I suspect they were all inherited from the same logic when making predictions for each single series. 

 

**To Reproduce**
<!--
Add a Minimal, Complete, and Verifiable example (for more details, see e.g. https://stackoverflow.com/help/mcve

If the code is too long, feel free to put it in a public gist and link it in the issue: https://gist.github.com
-->

```python
import pandas as pd
from sktime.forecasting.model_selection import ExpandingGreedySplitter
from sktime.forecasting.naive import NaiveForecaster

n_instances=3
n_points=5

temp = pd.DataFrame({
    'y': [i for i in range(n_points*n_instances)],
    'id':["T1"]*n_points + ["T2"]*n_points+ ["T11"]*n_points,  
    # bug still occurs even when timestamp is int-like
    'timestamp': pd.date_range(start='2022-07-01', periods=n_points*n_instances, freq='D') 
}).set_index(['id', 'timestamp'])

fh = [1,2]
train_test_splitter = ExpandingGreedySplitter(test_size=len(fh), folds=1)
forecaster = NaiveForecaster(strategy="last")  # NaiveForecaster() or TrendForecaster()
for train, test in train_test_splitter.split_series(temp): 
    forecaster.fit(train)
    y_pred = forecaster.predict(fh)
    print("=====TEST=====")
    print(test.head(15))
    print("=====PREDICT=====")
    print(y_pred.head(15))
    print("=====TRAIN=====")
    print(train.head(15))
```

**Output** - predictions for T2 and T11 got **switched** as shown in the output below
```
=====TEST=====
                 y
id  timestamp     
T1  2022-07-04   3
    2022-07-05   4
T2  2022-07-09   8
    2022-07-10   9
T11 2022-07-14  13
    2022-07-15  14
=====PREDICT=====
                   y
id  timestamp       
T1  2022-07-04   2.0
    2022-07-05   2.0
T2  2022-07-14  12.0  
    2022-07-15  12.0
T11 2022-07-09   7.0 
    2022-07-10   7.0 
=====TRAIN=====
                 y
id  timestamp     
T1  2022-07-01   0
    2022-07-02   1
    2022-07-03   2
T2  2022-07-06   5
    2022-07-07   6
    2022-07-08   7
T11 2022-07-11  10
    2022-07-12  11
    2022-07-13  12
```


**Expected behavior**
<!--
A clear and concise description of what you expected to happen.
-->
Trivially, the predictions should be broadcasted to the corresponding instance based on the trained data.

**Additional context**
<!--
Add any other context about the problem here.
-->
This bug is spotted when trying to fit naive forecaster in M4 weekly dataset as per in this notebook #5051

**Versions**
<details>

<!--
Please run the following code snippet and paste the output here:

from sktime import show_versions; show_versions()
-->
sktime: 0.21.0
sklearn: 1.3.0
skbase: 0.4.6
numpy: 1.24.3
pandas: 2.0.3
</details>

@fkiraly @benHeid 


Discussion:
This seems related:
https://github.com/sktime/sktime/issues/4683

it was the same for columns, where labels would end up lexicographically ordered.

The issue in #4683 was a combination of `pandas` pivot column ordering switching, and lack of testing.

I would suspect the issue, again, occurs when column labels are not lexicographically ordered.
Confirmed on current `main`, windows, python 3.11.

FYI @hoesler, seems to be the same bug as in #4683 - the index gets lexicographically ordered while the rows do not.
I think the reason is again `pandas` ordering lexicographically in a case where one wouldn't expect.

If you do `list(train.groupby(level=0))`, this orders lexicographically. This happens in `VectorizeDF.items`, in line 264. Same mistake (given unexpected behaviour of `pandas`)

Resolving patch:
diff --git a/sktime/datatypes/_vectorize.py b/sktime/datatypes/_vectorize.py
--- a/sktime/datatypes/_vectorize.py
+++ b/sktime/datatypes/_vectorize.py
@@ -261,7 +261,7 @@ def _iter_cols(inst, group_name=None):
         if is_self_iter:
             yield from _iter_cols(self.X_multiindex)
         else:
-            for name, group in self.X_multiindex.groupby(level=iter_levels):
+            for name, group in self.X_multiindex.groupby(level=iter_levels, sort=False):
                 yield from _iter_cols(group.droplevel(iter_levels), group_name=name)
 
     def _iter_levels(self, iterate_as):
@@ -593,11 +593,14 @@ def _to_iter(e):
         if return_type == "pd.DataFrame":
             df_long = pd.DataFrame(ret)
             cols_right_order = df_long.loc[:, 1].unique()
+            rows_right_order = df_long.loc[:, 0].unique()
 
             df = df_long.pivot(index=0, columns=1, values=2)
-            # DataFrame.pivot sorts the columns (is this a bug? see #4683)
+            # DataFrame.pivot sorts the rows & columns
+            # (is this a bug? see #4683 and #5108)
             # either way, we need to fix this:
             df = df.reindex(cols_right_order, axis=1)
+            df = df.reindex(rows_right_order, axis=0)
 
             # remove "0" and "1" from index/columns name
             df.index.names = [None] * len(df.index.names)


Test patch:
diff --git a/sktime/forecasting/base/tests/test_base.py b/sktime/forecasting/base/tests/test_base.py
--- a/sktime/forecasting/base/tests/test_base.py
+++ b/sktime/forecasting/base/tests/test_base.py
@@ -6,6 +6,7 @@
 from functools import reduce
 from operator import mul
 
+import numpy as np
 import pandas as pd
 import pytest
 from pandas.testing import assert_series_equal
@@ -19,10 +20,7 @@
 from sktime.utils._testing.hierarchical import _make_hierarchical
 from sktime.utils._testing.panel import _make_panel
 from sktime.utils._testing.series import _make_series
-from sktime.utils.validation._dependencies import (
-    _check_estimator_deps,
-    _check_soft_dependencies,
-)
+from sktime.utils.validation._dependencies import _check_estimator_deps
 
 PANEL_MTYPES = ["pd-multiindex", "nested_univ", "numpy3D"]
 HIER_MTYPES = ["pd_multiindex_hier"]
@@ -279,19 +277,15 @@ def test_vectorization_multivariate(mtype, exogeneous):
     assert y_pred_equal_length, msg
 
 
-@pytest.mark.skipif(
-    not _check_soft_dependencies("statsmodels", severity="none"),
-    reason="skip test if required soft dependency not available",
-)
 def test_col_vectorization_correct_col_order():
     """Test that forecaster vectorization preserves column index ordering.
 
     Failure case is as in issue #4683 where the column index is correct,
     but the values are in fact coming from forecasters in jumbled order.
     """
-    from sktime.datasets import load_macroeconomic
-
-    y = load_macroeconomic().iloc[:5]
+    cols = ["realgdp", "realcons", "realinv", "realgovt", "realdpi", "cpi", "m1"]
+    vals = np.random.rand(5, 7)
+    y = pd.DataFrame(vals, columns=cols)
 
     f = NaiveForecaster()
     # force univariate tag to trigger vectorization over columns for sure
@@ -306,6 +300,38 @@ def test_col_vectorization_correct_col_order():
     assert (y_pred == y.iloc[4]).all().all()
 
 
+def test_row_vectorization_correct_row_order():
+    """Test that forecaster vectorization preserves row index ordering.
+
+    Failure case is as in issue #5108 where the row index is correct,
+    but the values are in fact coming from forecasters in jumbled order.
+    """
+    n_instances = 3
+    n_points = 5
+
+    t_ix = pd.date_range(start="2022-07-01", periods=n_points * n_instances, freq="D")
+    y = pd.DataFrame(
+        {
+            "y": [i for i in range(n_points * n_instances)],
+            "id": ["T1"] * n_points + ["T2"] * n_points + ["T11"] * n_points,
+            "timestamp": t_ix,
+        }
+    ).set_index(["id", "timestamp"])
+
+    fh = [1]
+
+    forecaster = NaiveForecaster(strategy="last")
+
+    forecaster.fit(y)
+    y_pred = forecaster.predict(fh)
+
+    last_ix = range(n_points - 1, n_points * n_instances, n_points)
+    y_last = y.iloc[last_ix]
+
+    assert all(y_last.index.get_level_values(0) == y_pred.index.get_level_values(0))
+    assert all(y_last.values == y_pred.values)
+
+
 @pytest.mark.skipif(
     not _check_estimator_deps([ThetaForecaster, VAR], severity="none"),
     reason="skip test if required soft dependency not available",