Repository: sktime/sktime

Issue:
[BUG] check_pdmultiindex_panel incorrect result for metadata is_equally_spaced
**Describe the bug**
<!--
A clear and concise description of what the bug is.
-->
The *check_pdmultiindex_panel* method from sktime.datatypes will return an incorrect result for the "is_equally_spaced" metadata when the time index for each instance has a non-sequential initial time.

**To Reproduce**
<!--
Add a Minimal, Complete, and Verifiable example (for more details, see e.g. https://stackoverflow.com/help/mcve

If the code is too long, feel free to put it in a public gist and link it in the issue: https://gist.github.com
-->

```python
import pandas as pd
from sktime.datatypes import check_is_mtype

example_instance_index = [0, 0, 0, 1, 1, 1, 2, 2, 2]
example_time_index = pd.to_datetime([0, 1, 2, 4, 5, 6, 9, 10, 11], unit='s')
example_multiindex = pd.MultiIndex.from_arrays([example_instance_index, example_time_index])

example_data = pd.DataFrame(index=example_multiindex, data=list(range(0, 9)))
"""example_data
                       0
0 1970-01-01 00:00:00  0
  1970-01-01 00:00:01  1
  1970-01-01 00:00:02  2
1 1970-01-01 00:00:04  3
  1970-01-01 00:00:05  4
  1970-01-01 00:00:06  5
2 1970-01-01 00:00:09  6
  1970-01-01 00:00:10  7
  1970-01-01 00:00:11  8
"""

_, _, metadata = check_is_mtype(example_data, mtype="pd-multiindex", return_metadata=True)

assert metadata['is_equally_spaced'] == True
```

**Expected behavior**
<!--
A clear and concise description of what you expected to happen.
-->
The method should return True for "is_equally_spaced", since for each separate instance in the DataFrame the spacing between time indexes is 1 second.

**Additional context**
<!--
Add any other context about the problem here.
-->
In the check_pdmultiindex_panel method (sktime/datatypes/_panel/_check.py) the is_equally_spaced metadata is computed via
```python
metadata["is_equally_spaced"] = all(
            _index_equally_spaced(group.index.levels[-1]) for _, group in series_groups
        )
```
If the get_level_values() method is used instead of the levels attribute, this bug should be fixed.
```python
metadata["is_equally_spaced"] = all(
            _index_equally_spaced(group.index.get_level_values(-1)) for _, group in series_groups
        )
```

**Versions**
<details>

<!--
Please run the following code snippet and paste the output here:
 
from sktime import show_versions; show_versions()
-->
System:
    python: 3.9.15 | packaged by conda-forge | (main, Nov 22 2022, 15:55:03)  [GCC 10.4.0]
executable: /home/bin/python
   machine: Linux-4.14.305-10ms-x86_64-with-glibc2.27

Python dependencies:
          pip: 23.0.1
   setuptools: 58.1.0
      sklearn: 1.2.1
       sktime: 0.16.1
  statsmodels: 0.13.5
        numpy: 1.23.5
        scipy: 1.10.1
       pandas: 1.5.3
   matplotlib: None
       joblib: 1.2.0
        numba: 0.56.4
     pmdarima: None
      tsfresh: 0.20.0
</details>

<!-- Thanks for contributing! -->



Discussion:
yes, can confirm the bug on windows, python 3.10, current `main`.

I had to think about the "meaning" of `is_equally_spaced` - i.e., whether it should mean "equal index across instances, and also equally spaced". I was not sure whether that is tacitly assumed by estimators, but in the end I feel we should go with the description of the tag.

Also thanks for the proposed fix, this seems to work.

Would appreciate a PR!
If you do that, could you also kindly add your test as a single pytest test at the end of `datatypes.tests.test_check`?
@fkiraly Yes, I can make a PR for this. I've also noticed that a very similar issue exists for the convert method `from_multi_index_to_3d_numpy`, i.e. the bug described in https://github.com/sktime/sktime/issues/3536. Right now, the workaround is to either replace the time index with a `RangeIndex`, which is what the `PaddingTransformer` does, or zero out the time indices for each instance separately. Right now, the method performs the conversion by assuming that each instance will contain the exact same time indices, which causes the numpy shape error. Do you think this method should be updated to allow each instance to contain different time indices, while validating that each instance has equal time series length and spacing?

Resolving patch:
diff --git a/sktime/datatypes/_panel/_check.py b/sktime/datatypes/_panel/_check.py
--- a/sktime/datatypes/_panel/_check.py
+++ b/sktime/datatypes/_panel/_check.py
@@ -241,7 +241,8 @@ def check_pdmultiindex_panel(obj, return_metadata=False, var_name="obj", panel=T
 
         metadata["is_univariate"] = len(obj.columns) < 2
         metadata["is_equally_spaced"] = all(
-            _index_equally_spaced(group.index.levels[-1]) for _, group in series_groups
+            _index_equally_spaced(group.index.get_level_values(-1))
+            for _, group in series_groups
         )
         metadata["is_empty"] = len(index) < 1 or len(obj.columns) < 1
         metadata["n_panels"] = n_panels
diff --git a/sktime/datatypes/_panel/_examples.py b/sktime/datatypes/_panel/_examples.py
--- a/sktime/datatypes/_panel/_examples.py
+++ b/sktime/datatypes/_panel/_examples.py
@@ -98,6 +98,7 @@
     "is_one_panel": True,
     "is_equally_spaced": True,
     "is_equal_length": True,
+    "is_equal_index": True,
     "is_empty": False,
     "has_nans": False,
     "n_instances": 3,
@@ -169,6 +170,7 @@
     "is_one_panel": True,
     "is_equally_spaced": True,
     "is_equal_length": True,
+    "is_equal_index": True,
     "is_empty": False,
     "has_nans": False,
     "n_instances": 3,
@@ -233,7 +235,36 @@
     "is_one_panel": True,
     "is_equally_spaced": True,
     "is_equal_length": True,
+    "is_equal_index": True,
     "is_empty": False,
     "has_nans": False,
     "n_instances": 1,
 }
+
+###
+# example 3: univariate, equally sampled, lossy,
+# targets #4299 pd-multiindex panel incorrect is_equally_spaced
+
+X_instances = [0, 0, 0, 1, 1, 1, 2, 2, 2]
+X_timepoints = pd.to_datetime([0, 1, 2, 4, 5, 6, 9, 10, 11], unit="s")
+X_multiindex = pd.MultiIndex.from_arrays(
+    [X_instances, X_timepoints], names=["instances", "timepoints"]
+)
+
+X = pd.DataFrame(index=X_multiindex, data=list(range(0, 9)), columns=["var_0"])
+
+example_dict[("pd-multiindex", "Panel", 3)] = X
+example_dict_lossy[("pd-multiindex", "Panel", 3)] = False
+
+example_dict_metadata[("Panel", 3)] = {
+    "is_univariate": True,
+    "is_one_series": False,
+    "n_panels": 1,
+    "is_one_panel": True,
+    "is_equally_spaced": True,
+    "is_equal_length": True,
+    "is_equal_index": False,
+    "is_empty": False,
+    "has_nans": False,
+    "n_instances": 3,
+}


Test patch:
diff --git a/sktime/datatypes/tests/test_check.py b/sktime/datatypes/tests/test_check.py
--- a/sktime/datatypes/tests/test_check.py
+++ b/sktime/datatypes/tests/test_check.py
@@ -170,6 +170,11 @@ def test_check_metadata_inference(scitype, mtype, fixture_index):
         if "scitype" in metadata:
             del metadata["scitype"]
 
+        # currently we do not check this field in metadata inference
+        if "is_equal_index" in expected_metadata:
+            expected_metadata = expected_metadata.copy()
+            del expected_metadata["is_equal_index"]
+
         msg = (
             f"check_is_mtype returns wrong metadata on scitype {scitype}, "
             f"mtype {mtype}, fixture {fixture_index}. "
diff --git a/sktime/datatypes/tests/test_utils.py b/sktime/datatypes/tests/test_utils.py
--- a/sktime/datatypes/tests/test_utils.py
+++ b/sktime/datatypes/tests/test_utils.py
@@ -32,7 +32,10 @@
 
 @pytest.mark.parametrize("scitype,mtype", SCITYPE_MTYPE_PAIRS)
 def test_get_time_index(scitype, mtype):
-    """Tests that conversions for scitype agree with from/to example fixtures.
+    """Tests that get_time_index returns the expected output.
+
+    Note: this is tested only for fixtures with equal time index across instances,
+    as get_time_index assumes that.
 
     Parameters
     ----------
@@ -41,7 +44,7 @@ def test_get_time_index(scitype, mtype):
 
     Raises
     ------
-    AssertionError if get_cutoff does not return a length 1 pandas.index
+    AssertionError if get_time_index does not return the expected return
         for any fixture example of given scitype, mtype
     """
     # get_time_index currently does not work for df-list type, skip
@@ -49,11 +52,16 @@ def test_get_time_index(scitype, mtype):
         return None
 
     # retrieve example fixture
-    fixtures = get_examples(mtype=mtype, as_scitype=scitype, return_lossy=False)
+    fixtures = get_examples(mtype=mtype, as_scitype=scitype, return_metadata=True)
+
+    for fixture_tuple in fixtures.values():
+        fixture = fixture_tuple[0]
+        fixture_metadata = fixture_tuple[2]
 
-    for fixture in fixtures.values():
         if fixture is None:
             continue
+        if not fixture_metadata.get("is_equal_index", True):
+            continue
 
         idx = get_time_index(fixture)
 
@@ -96,9 +104,14 @@ def test_get_cutoff(scitype, mtype, return_index, reverse_order, convert_input):
         for any fixture example of given scitype, mtype
     """
     # retrieve example fixture
-    fixtures = get_examples(mtype=mtype, as_scitype=scitype, return_lossy=False)
+    fixtures = get_examples(mtype=mtype, as_scitype=scitype, return_metadata=True)
+
+    for fixture_tuple in fixtures.values():
+        fixture = fixture_tuple[0]
+        fixture_metadata = fixture_tuple[2]
+        fixture_equally_spaced = fixture_metadata.get("is_equally_spaced", True)
+        fixture_equal_index = fixture_metadata.get("is_equal_index", True)
 
-    for fixture in fixtures.values():
         if fixture is None:
             continue
 
@@ -126,7 +139,9 @@ def test_get_cutoff(scitype, mtype, return_index, reverse_order, convert_input):
         if return_index:
             assert len(cutoff) == 1
             if isinstance(cutoff_val, (pd.Period, pd.Timestamp)):
-                assert hasattr(cutoff, "freq") and cutoff.freq is not None
+                assert hasattr(cutoff, "freq")
+                if fixture_equally_spaced and fixture_equal_index:
+                    assert cutoff.freq is not None
 
         if isinstance(fixture, np.ndarray):
             if reverse_order: