Repository: sktime/sktime (python)

Issue:
[BUG] ConformalInterval Memory Usage
**Describe the bug**
Disclaimer: Not sure if really a bug or more a request.

Currently, the ConformalInterval is not very memory efficient, since it constructs internally a dataframe with the size of len(time series) x len(time series) (line 420). However, the matrix represented by this dataframe is sparse for two reasons:
1. It is an upper triangular matrix by definition.
2. Due to the `sample_frac` parameter, multiple rows will stay completely empty. 

Thus, I would propose to use a sparse matrix here instead of a normal dataframe. Thus, the definition of the dataframe would need to be replaced with: 

```python
residuals_matrix = pd.DataFrame.sparse.from_spmatrix(scipy.sparse.csr_array((len(full_y_index), len(full_y_index))), 
                                                             index=full_y_index, columns=full_y_index)
```


Pull request:
[ENN] Improve memory efficiency of `ConformalIntervals` using sparse matrices
#### Reference Issues/PRs
Fixes #5569 

#### What does this implement/fix? Explain your changes.
This PR is using a sparse matrix representation instead of directly a dataframe in `ConformalIntervals` to save the residuals. 


#### Does your contribution introduce a new dependency? If yes, which one?
No
<!--
Only relevant if you changed pyproject.toml.
We try to minimize dependencies in the core dependency set. There
are also further specific instructions to follow for soft dependencies.
See here for handling dependencies in sktime: https://www.sktime.net/en/latest/developer_guide/dependencies.html
-->

#### What should a reviewer concentrate their feedback on?

<!-- This section is particularly useful if you have a pull request that is still in development. You can guide the reviews to focus on the parts that are ready for their comments. We suggest using bullets (indicated by * or -) and filled checkboxes [x] here -->

#### Did you add any tests for the change?
No
<!-- This section is useful if you have added a test in addition to the existing ones. This will ensure that further changes to these files won't introduce the same kind of bug. It is considered good practice to add tests with newly added code to enforce the fact that the code actually works. This will reduce the chance of introducing logical bugs.
-->

#### Any other comments?
<!--
We value all user contributions, no matter how small or complex they are. If you have any questions, feel free to post
in the dev-chat channel on the sktime discord https://discord.com/invite/54ACzaFsn7. If we are slow to review (>3 working days), likewise feel free to ping us on discord. Thank you for your understanding during the review process.
-->

#### PR checklist
<!--
Please go through the checklist below. Please feel free to remove points if they are not applicable.
-->

##### For all contributions
- [ ] I've added myself to the [list of contributors](https://github.com/sktime/sktime/blob/main/CONTRIBUTORS.md) with any new badges I've earned :-)
  How to: add yourself to the [all-contributors file](https://github.com/sktime/sktime/blob/main/.all-contributorsrc) in the `sktime` root directory (not the `CONTRIBUTORS.md`). Common badges: `code` - fixing a bug, or adding code logic. `doc` - writing or improving documentation or docstrings. `bug` - reporting or diagnosing a bug (get this plus `code` if you also fixed the bug in the PR).`maintenance` - CI, test framework, release.
  See here for [full badge reference](https://allcontributors.org/docs/en/emoji-key)
- [ ] Optionally, I've added myself and possibly others to the [CODEOWNERS](https://github.com/sktime/sktime/blob/main/CODEOWNERS) file - do this if you want to become the owner or maintainer of an estimator you added.
  See here for further details on the [algorithm maintainer role](https://www.sktime.net/en/latest/get_involved/governance.html#algorithm-maintainers).
- [ ] The PR title starts with either [ENH], [MNT], [DOC], or [BUG]. [BUG] - bugfix, [MNT] - CI, test framework, [ENH] - adding or improving code, [DOC] - writing or improving documentation or docstrings.

##### For new estimators
- [ ] I've added the estimator to the API reference - in `docs/source/api_reference/taskname.rst`, follow the pattern.
- [ ] I've added one or more illustrative usage examples to the docstring, in a pydocstyle compliant `Examples` section.
- [ ] If the estimator relies on a soft dependency, I've set the `python_dependencies` tag and ensured
  dependency isolation, see the [estimator dependencies guide](https://www.sktime.net/en/latest/developer_guide/dependencies.html#adding-a-soft-dependency).

<!--
Thanks for contributing!
-->


Patch:
diff --git a/sktime/forecasting/conformal.py b/sktime/forecasting/conformal.py
index 175e75437..b63176813 100644
--- a/sktime/forecasting/conformal.py
+++ b/sktime/forecasting/conformal.py
@@ -11,6 +11,7 @@ from math import floor
 
 import numpy as np
 import pandas as pd
+import scipy
 from joblib import Parallel, delayed
 from sklearn.base import clone
 
@@ -417,8 +418,10 @@ class ConformalIntervals(BaseForecaster):
 
         full_y_index = y.iloc[n_initial_window:].index
 
-        residuals_matrix = pd.DataFrame(
-            columns=full_y_index, index=full_y_index, dtype="float"
+        residuals_matrix = pd.DataFrame.sparse.from_spmatrix(
+            scipy.sparse.csr_array((len(full_y_index), len(full_y_index))),
+            index=full_y_index,
+            columns=full_y_index,
         )
 
         if update and hasattr(self, "residuals_matrix_") and not sample_frac:
@@ -427,9 +430,9 @@ class ConformalIntervals(BaseForecaster):
                 overlapping_index = pd.Index(
                     self.residuals_matrix_.index.intersection(full_y_index)
                 ).sort_values()
-                residuals_matrix.loc[
+                residuals_matrix = self.residuals_matrix_.loc[
                     overlapping_index, overlapping_index
-                ] = self.residuals_matrix_.loc[overlapping_index, overlapping_index]
+                ].combine_first(residuals_matrix)
             else:
                 overlapping_index = None
             y_index = remaining_y_index
@@ -466,8 +469,10 @@ class ConformalIntervals(BaseForecaster):
             delayed(_get_residuals_matrix_row)(forecaster.clone(), y, X, id)
             for id in y_index
         )
-        for idx, id in enumerate(y_index):
-            residuals_matrix.loc[id] = all_residuals[idx]
+        all_residuals = pd.concat(all_residuals, axis=1, ignore_index=True).T
+        all_residuals.index = y_index
+        all_residuals = all_residuals.astype(pd.SparseDtype("float", 0))
+        residuals_matrix = all_residuals.combine_first(residuals_matrix)
 
         if overlapping_index is not None:
 
@@ -494,9 +499,10 @@ class ConformalIntervals(BaseForecaster):
                 delayed(_extend_residuals_matrix_row)(y, X, id)
                 for id in overlapping_index
             )
-
-            for idx, id in enumerate(overlapping_index):
-                residuals_matrix.loc[id] = extend_residuals[idx]
+            extend_residuals = pd.concat(extend_residuals, axis=1, ignore_index=True).T
+            extend_residuals.index = overlapping_index
+            extend_residuals = extend_residuals.astype(pd.SparseDtype("float", 0))
+            residuals_matrix = extend_residuals.combine_first(residuals_matrix)
 
         return residuals_matrix
 
diff --git a/sktime/utils/sklearn/__init__.py b/sktime/utils/sklearn/__init__.py
new file mode 100644
index 000000000..5b1546d33
--- /dev/null
+++ b/sktime/utils/sklearn/__init__.py
@@ -0,0 +1,21 @@
+"""Sklearn related utility functionality."""
+
+from sktime.utils.sklearn._adapt_df import prep_skl_df
+from sktime.utils.sklearn._scitype import (
+    is_sklearn_classifier,
+    is_sklearn_clusterer,
+    is_sklearn_estimator,
+    is_sklearn_regressor,
+    is_sklearn_transformer,
+    sklearn_scitype,
+)
+
+__all__ = [
+    "prep_skl_df",
+    "is_sklearn_estimator",
+    "is_sklearn_transformer",
+    "is_sklearn_classifier",
+    "is_sklearn_regressor",
+    "is_sklearn_clusterer",
+    "sklearn_scitype",
+]
diff --git a/sktime/utils/sklearn/_adapt_df.py b/sktime/utils/sklearn/_adapt_df.py
new file mode 100644
index 000000000..300764c1b
--- /dev/null
+++ b/sktime/utils/sklearn/_adapt_df.py
@@ -0,0 +1,31 @@
+"""Utility functions for adapting to sklearn."""
+# copyright: sktime developers, BSD-3-Clause License (see LICENSE file)
+
+import numpy as np
+
+
+def prep_skl_df(df, copy_df=False):
+    """Make df compatible with sklearn input expectations.
+
+    Changes:
+    turns column index into a list of strings
+
+    Parameters
+    ----------
+    df : pd.DataFrame
+        list of indices to sample from
+    copy_df : bool, default=False
+        whether to mutate df or return a copy
+        if False, index of df is mutated
+        if True, original df is not mutated. If index is not a list of strings,
+        a copy is made and the copy is mutated. Otherwise, the original df is returned.
+    """
+    cols = df.columns
+    str_cols = cols.astype(str)
+
+    if not np.all(str_cols == cols):
+        if copy_df:
+            df = df.copy()
+        df.columns = str_cols
+
+    return df


Test patch:
diff --git a/sktime/utils/_testing/deep_equals.py b/sktime/utils/_testing/deep_equals.py
index 84494ad59..0616cc81a 100644
--- a/sktime/utils/_testing/deep_equals.py
+++ b/sktime/utils/_testing/deep_equals.py
@@ -98,11 +98,67 @@ def deep_equals(x, y, return_msg=False):
         else:
             return ret(x.equals(y), ".series_equals, x = {} != y = {}", [x, y])
     elif isinstance(x, pd.DataFrame):
+        # check column names for equality
         if not x.columns.equals(y.columns):
             return ret(
                 False, f".columns, x.columns = {x.columns} != y.columns = {y.columns}"
             )
-        # if columns are equal and at least one is object, recurse over Series
+        # check dtypes for equality
+        if not x.dtypes.equals(y.dtypes):
+            return ret(
+                False, f".dtypes, x.dtypes = {x.dtypes} != y.dtypes = {y.dtypes}"
+            )
+        # check index for equality
+        # we are not recursing due to ambiguity in integer index types
+        # which may differ from pandas version to pandas version
+        # and would upset the type check, e.g., RangeIndex(2) vs Index([0, 1])
+        xix = x.index
+        yix = y.index
+        if hasattr(xix, "dtype") and hasattr(xix, "dtype"):
+            if not xix.dtype == yix.dtype:
+                return ret(
+                    False,
+                    ".index.dtype, x.index.dtype = {} != y.index.dtype = {}",
+                    [xix.dtype, yix.dtype],
+                )
+        if hasattr(xix, "dtypes") and hasattr(yix, "dtypes"):
+            if not x.dtypes.equals(y.dtypes):
+                return ret(
+                    False,
+                    ".index.dtypes, x.dtypes = {} != y.index.dtypes = {}",
+                    [xix.dtypes, yix.dtypes],
+                )
+        ix_eq = xix.equals(yix)
+        if not ix_eq:
+            if not len(xix) == len(yix):
+                return ret(
+                    False,
+                    ".index.len, x.index.len = {} != y.index.len = {}",
+                    [len(xix), len(yix)],
+                )
+            if hasattr(xix, "name") and hasattr(yix, "name"):
+                if not xix.name == yix.name:
+                    return ret(
+                        False,
+                        ".index.name, x.index.name = {} != y.index.name = {}",
+                        [xix.name, yix.name],
+                    )
+            if hasattr(xix, "names") and hasattr(yix, "names"):
+                if not len(xix.names) == len(yix.names):
+                    return ret(
+                        False,
+                        ".index.names, x.index.names = {} != y.index.name = {}",
+                        [xix.names, yix.names],
+                    )
+                if not np.all(xix.names == yix.names):
+                    return ret(
+                        False,
+                        ".index.names, x.index.names = {} != y.index.name = {}",
+                        [xix.names, yix.names],
+                    )
+            elts_eq = np.all(xix == yix)
+            return ret(elts_eq, ".index.equals, x = {} != y = {}", [xix, yix])
+        # if columns, dtypes are equal and at least one is object, recurse over Series
         if sum(x.dtypes == "object") > 0:
             for c in x.columns:
                 is_equal, msg = deep_equals(x[c], y[c], return_msg=True)
@@ -112,6 +168,14 @@ def deep_equals(x, y, return_msg=False):
         else:
             return ret(x.equals(y), ".df_equals, x = {} != y = {}", [x, y])
     elif isinstance(x, pd.Index):
+        if hasattr(x, "dtype") and hasattr(y, "dtype"):
+            if not x.dtype == y.dtype:
+                return ret(False, f".dtype, x.dtype = {x.dtype} != y.dtype = {y.dtype}")
+        if hasattr(x, "dtypes") and hasattr(y, "dtypes"):
+            if not x.dtypes.equals(y.dtypes):
+                return ret(
+                    False, f".dtypes, x.dtypes = {x.dtypes} != y.dtypes = {y.dtypes}"
+                )
         return ret(x.equals(y), ".index_equals, x = {} != y = {}", [x, y])
     elif isinstance(x, np.ndarray):
         if x.dtype != y.dtype:
diff --git a/sktime/utils/sklearn.py b/sktime/utils/sklearn/_scitype.py
similarity index 100%
rename from sktime/utils/sklearn.py
rename to sktime/utils/sklearn/_scitype.py
diff --git a/sktime/utils/sklearn/tests/__init__.py b/sktime/utils/sklearn/tests/__init__.py
new file mode 100644
index 000000000..1d9b760fe
--- /dev/null
+++ b/sktime/utils/sklearn/tests/__init__.py
@@ -0,0 +1,1 @@
+"""Tests for sklearn related utility functionality."""
diff --git a/sktime/utils/sklearn/tests/test_sklearn_df_adapt.py b/sktime/utils/sklearn/tests/test_sklearn_df_adapt.py
new file mode 100644
index 000000000..57b36ebcd
--- /dev/null
+++ b/sktime/utils/sklearn/tests/test_sklearn_df_adapt.py
@@ -0,0 +1,33 @@
+"""Tests for sklearn dataframe coercion."""
+
+__author__ = ["fkiraly"]
+
+import numpy as np
+import pandas as pd
+import pytest
+
+from sktime.utils.sklearn._adapt_df import prep_skl_df
+
+
+@pytest.mark.parametrize("copy_df", [True, False])
+def test_prep_skl_df_coercion(copy_df):
+    """Test that prep_skl_df behaves correctly on the coercion case."""
+    mixed_example = pd.DataFrame({0: [1, 2, 3], "b": [1, 2, 3]})
+
+    res = prep_skl_df(mixed_example, copy_df=copy_df)
+
+    assert np.all(res.columns == ["0", "b"])
+
+    if not copy_df:
+        assert res is mixed_example
+
+
+@pytest.mark.parametrize("copy_df", [True, False])
+def test_prep_skl_df_non_coercion(copy_df):
+    """Test that prep_skl_df behaves correctly on the non-coercion case."""
+    mixed_example = pd.DataFrame({"a": [1, 2, 3], "b": [1, 2, 3]})
+
+    res = prep_skl_df(mixed_example, copy_df=copy_df)
+
+    assert np.all(res.columns == ["a", "b"])
+    assert res is mixed_example