Repository: sktime/sktime
Pull request #5567 (2023-11-21 20:45:54)

[MNT] [Dependabot](deps-dev): Update pytest-xdist requirement from <3.5,>=3.3 to >=3.3,<3.6
Updates the requirements on [pytest-xdist](https://github.com/pytest-dev/pytest-xdist) to permit the latest version.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/pytest-dev/pytest-xdist/blob/master/CHANGELOG.rst">pytest-xdist's changelog</a>.</em></p>
<blockquote>
<h1>pytest-xdist 3.5.0 (2023-11-21)</h1>
<h2>Features</h2>
<ul>
<li><code>[#632](https://github.com/pytest-dev/pytest-xdist/issues/632) &lt;https://github.com/pytest-dev/pytest-xdist/issues/632&gt;</code>_: <code>--dist=loadscope</code> now sorts scopes by number of tests to assign largest scopes early -- in many cases this should improve overall test session running time, as there is less chance of a large scope being left to be processed near the end of the session, leaving other workers idle.</li>
</ul>
<h1>pytest-xdist 3.4.0 (2023-11-11)</h1>
<h2>Features</h2>
<ul>
<li>
<p><code>[#963](https://github.com/pytest-dev/pytest-xdist/issues/963) &lt;https://github.com/pytest-dev/pytest-xdist/issues/963&gt;</code>_: Wait for workers to finish reporting when test run stops early.</p>
<p>This makes sure that the results of in-progress tests are displayed.
Previously these reports were being discarded, losing information about the
test run.</p>
</li>
<li>
<p><code>[#965](https://github.com/pytest-dev/pytest-xdist/issues/965) &lt;https://github.com/pytest-dev/pytest-xdist/issues/965&gt;</code>_: Added support for Python 3.12.</p>
</li>
</ul>
<h1>pytest-xdist 3.3.1 (2023-05-19)</h1>
<h2>Bug Fixes</h2>
<ul>
<li>
<p><code>[#907](https://github.com/pytest-dev/pytest-xdist/issues/907) &lt;https://github.com/pytest-dev/pytest-xdist/issues/907&gt;</code>_: Avoid remote calls during startup as <code>execnet</code> by default does not ensure remote affinity with the
main thread and might accidentally schedule the pytest worker into a non-main thread, which breaks numerous frameworks,
for example <code>asyncio</code>, <code>anyio</code>, <code>PyQt/PySide</code>, etc.</p>
<p>A more safe correction will require thread affinity in <code>execnet</code> (<code>pytest-dev/execnet#96 &lt;https://github.com/pytest-dev/execnet/issues/96&gt;</code>__).</p>
</li>
</ul>
<h1>pytest-xdist 3.3.0 (2023-05-12)</h1>
<h2>Features</h2>
<ul>
<li><code>[#555](https://github.com/pytest-dev/pytest-xdist/issues/555) &lt;https://github.com/pytest-dev/pytest-xdist/issues/555&gt;</code>_: Improved progress output when collecting nodes to be less verbose.</li>
</ul>
<h1>pytest-xdist 3.2.1 (2023-03-12)</h1>
<p>Bug Fixes</p>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/pytest-dev/pytest-xdist/commit/95b309e980796a261045d770f69c016ca741473d"><code>95b309e</code></a> Release 3.5.0</li>
<li><a href="https://github.com/pytest-dev/pytest-xdist/commit/3fe877bc6d7211f31e55d2d561803f76ea95b935"><code>3fe877b</code></a> LoadScope scheduler: Sort scopes by number of tests to assign biggest scopes ...</li>
<li><a href="https://github.com/pytest-dev/pytest-xdist/commit/f36ea25cb53cb960037f4a8510816b81fa313035"><code>f36ea25</code></a> Create GitHub release after deploy (<a href="https://redirect.github.com/pytest-dev/pytest-xdist/issues/973">#973</a>)</li>
<li><a href="https://github.com/pytest-dev/pytest-xdist/commit/d81f57508d064915fc16ace5dea4d43406db2ba0"><code>d81f575</code></a> Merge pull request <a href="https://redirect.github.com/pytest-dev/pytest-xdist/issues/969">#969</a> from pytest-dev/pre-commit-ci-update-config</li>
<li><a href="https://github.com/pytest-dev/pytest-xdist/commit/1d2616418c046d0e6f059b8f76e35a0627409616"><code>1d26164</code></a> [pre-commit.ci] pre-commit autoupdate</li>
<li><a href="https://github.com/pytest-dev/pytest-xdist/commit/5f43575b42d6d4c0078de807880b8b0791b75823"><code>5f43575</code></a> Merge pull request <a href="https://redirect.github.com/pytest-dev/pytest-xdist/issues/966">#966</a> from pytest-dev/release-3.4.0</li>
<li><a href="https://github.com/pytest-dev/pytest-xdist/commit/a6b56112f9b686aed7a354e0d50ecc26ad2d9dfe"><code>a6b5611</code></a> Update CHANGELOG</li>
<li><a href="https://github.com/pytest-dev/pytest-xdist/commit/64f9388f4164e9f64d98a7580933e958ea4a09d5"><code>64f9388</code></a> Add support for Python 3.12</li>
<li><a href="https://github.com/pytest-dev/pytest-xdist/commit/230ba6ad1057574c9f3d42a97f890788cd9ec6c3"><code>230ba6a</code></a> Properly wait for workers when test run terminates early (<a href="https://redirect.github.com/pytest-dev/pytest-xdist/issues/963">#963</a>)</li>
<li><a href="https://github.com/pytest-dev/pytest-xdist/commit/93ca202b400eecc62919bcd9b0bab001e3f3f7ef"><code>93ca202</code></a> fix typo index -&gt; instead</li>
<li>Additional commits viewable in <a href="https://github.com/pytest-dev/pytest-xdist/compare/v3.3.0...v3.5.0">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>


Patch:
diff --git a/pyproject.toml b/pyproject.toml
index 91438fffa..ba5713445 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -116,7 +116,7 @@ all_extras_pandas2 = [
   "arch<6.3.0,>=5.6",
   "cloudpickle",
   "dash!=2.9.0",
-  "dask<2023.7.1",
+  "dask<2023.11.1",
   "dtw-python",
   'esig==0.9.7; python_version < "3.10"',
   'filterpy>=1.4.5; python_version < "3.11"',
@@ -200,7 +200,7 @@ regression = [
 transformations = [
   'esig<0.10,>=0.9.7; python_version < "3.11"',
   "filterpy<1.5,>=1.4.5",
-  "holidays>=0.29,<0.37",
+  "holidays>=0.29,<0.38",
   "mne<1.6,>=1.5",
   'numba<0.59,>=0.53; python_version < "3.12"',
   "pycatch22<0.4.4,>=0.4",
@@ -236,7 +236,7 @@ docs = [
   "Sphinx!=7.2.0,<8.0.0",
   "sphinx-copybutton",
   "sphinx-design<0.6.0",
-  "sphinx-gallery<0.15.0",
+  "sphinx-gallery<0.16.0",
   "sphinx-issues<4.0.0",
   "tabulate",
 ]
@@ -245,7 +245,7 @@ tests = [
   "pytest-cov<4.2,>=4.1",
   "pytest-randomly<3.16,>=3.15",
   "pytest-timeout>=2.1,<2.3",
-  "pytest-xdist>=3.3,<3.5",
+  "pytest-xdist>=3.3,<3.6",
 ]
 
 # CI related soft dependency sets - not for users of sktime, only for developers
diff --git a/sktime/datatypes/_panel/_examples.py b/sktime/datatypes/_panel/_examples.py
index e7c1503cc..d2f4ef0f3 100644
--- a/sktime/datatypes/_panel/_examples.py
+++ b/sktime/datatypes/_panel/_examples.py
@@ -68,7 +68,7 @@ example_dict[("pd-multiindex", "Panel", 0)] = X
 example_dict_lossy[("pd-multiindex", "Panel", 0)] = False
 
 cols = [f"var_{i}" for i in range(2)]
-X = pd.DataFrame(columns=cols, index=[0, 1, 2])
+X = pd.DataFrame(columns=cols, index=pd.RangeIndex(3))
 X["var_0"] = pd.Series(
     [pd.Series([1, 2, 3]), pd.Series([1, 2, 3]), pd.Series([1, 2, 3])]
 )
@@ -143,7 +143,7 @@ example_dict[("pd-multiindex", "Panel", 1)] = X
 example_dict_lossy[("pd-multiindex", "Panel", 1)] = False
 
 cols = [f"var_{i}" for i in range(1)]
-X = pd.DataFrame(columns=cols, index=[0, 1, 2])
+X = pd.DataFrame(columns=cols, index=pd.RangeIndex(3))
 X["var_0"] = pd.Series(
     [pd.Series([4, 5, 6]), pd.Series([4, 55, 6]), pd.Series([42, 5, 6])]
 )
@@ -211,7 +211,7 @@ example_dict[("pd-multiindex", "Panel", 2)] = X
 example_dict_lossy[("pd-multiindex", "Panel", 2)] = False
 
 cols = [f"var_{i}" for i in range(1)]
-X = pd.DataFrame(columns=cols, index=[0])
+X = pd.DataFrame(columns=cols, index=pd.RangeIndex(1))
 X["var_0"] = pd.Series([pd.Series([4, 5, 6])])
 
 example_dict[("nested_univ", "Panel", 2)] = X
diff --git a/sktime/forecasting/conformal.py b/sktime/forecasting/conformal.py
index 175e75437..b63176813 100644
--- a/sktime/forecasting/conformal.py
+++ b/sktime/forecasting/conformal.py
@@ -11,6 +11,7 @@ from math import floor
 
 import numpy as np
 import pandas as pd
+import scipy
 from joblib import Parallel, delayed
 from sklearn.base import clone
 
@@ -417,8 +418,10 @@ class ConformalIntervals(BaseForecaster):
 
         full_y_index = y.iloc[n_initial_window:].index
 
-        residuals_matrix = pd.DataFrame(
-            columns=full_y_index, index=full_y_index, dtype="float"
+        residuals_matrix = pd.DataFrame.sparse.from_spmatrix(
+            scipy.sparse.csr_array((len(full_y_index), len(full_y_index))),
+            index=full_y_index,
+            columns=full_y_index,
         )
 
         if update and hasattr(self, "residuals_matrix_") and not sample_frac:
@@ -427,9 +430,9 @@ class ConformalIntervals(BaseForecaster):
                 overlapping_index = pd.Index(
                     self.residuals_matrix_.index.intersection(full_y_index)
                 ).sort_values()
-                residuals_matrix.loc[
+                residuals_matrix = self.residuals_matrix_.loc[
                     overlapping_index, overlapping_index
-                ] = self.residuals_matrix_.loc[overlapping_index, overlapping_index]
+                ].combine_first(residuals_matrix)
             else:
                 overlapping_index = None
             y_index = remaining_y_index
@@ -466,8 +469,10 @@ class ConformalIntervals(BaseForecaster):
             delayed(_get_residuals_matrix_row)(forecaster.clone(), y, X, id)
             for id in y_index
         )
-        for idx, id in enumerate(y_index):
-            residuals_matrix.loc[id] = all_residuals[idx]
+        all_residuals = pd.concat(all_residuals, axis=1, ignore_index=True).T
+        all_residuals.index = y_index
+        all_residuals = all_residuals.astype(pd.SparseDtype("float", 0))
+        residuals_matrix = all_residuals.combine_first(residuals_matrix)
 
         if overlapping_index is not None:
 
@@ -494,9 +499,10 @@ class ConformalIntervals(BaseForecaster):
                 delayed(_extend_residuals_matrix_row)(y, X, id)
                 for id in overlapping_index
             )
-
-            for idx, id in enumerate(overlapping_index):
-                residuals_matrix.loc[id] = extend_residuals[idx]
+            extend_residuals = pd.concat(extend_residuals, axis=1, ignore_index=True).T
+            extend_residuals.index = overlapping_index
+            extend_residuals = extend_residuals.astype(pd.SparseDtype("float", 0))
+            residuals_matrix = extend_residuals.combine_first(residuals_matrix)
 
         return residuals_matrix
 
diff --git a/sktime/utils/sklearn/__init__.py b/sktime/utils/sklearn/__init__.py
new file mode 100644
index 000000000..5b1546d33
--- /dev/null
+++ b/sktime/utils/sklearn/__init__.py
@@ -0,0 +1,21 @@
+"""Sklearn related utility functionality."""
+
+from sktime.utils.sklearn._adapt_df import prep_skl_df
+from sktime.utils.sklearn._scitype import (
+    is_sklearn_classifier,
+    is_sklearn_clusterer,
+    is_sklearn_estimator,
+    is_sklearn_regressor,
+    is_sklearn_transformer,
+    sklearn_scitype,
+)
+
+__all__ = [
+    "prep_skl_df",
+    "is_sklearn_estimator",
+    "is_sklearn_transformer",
+    "is_sklearn_classifier",
+    "is_sklearn_regressor",
+    "is_sklearn_clusterer",
+    "sklearn_scitype",
+]
diff --git a/sktime/utils/sklearn/_adapt_df.py b/sktime/utils/sklearn/_adapt_df.py
new file mode 100644
index 000000000..300764c1b
--- /dev/null
+++ b/sktime/utils/sklearn/_adapt_df.py
@@ -0,0 +1,31 @@
+"""Utility functions for adapting to sklearn."""
+# copyright: sktime developers, BSD-3-Clause License (see LICENSE file)
+
+import numpy as np
+
+
+def prep_skl_df(df, copy_df=False):
+    """Make df compatible with sklearn input expectations.
+
+    Changes:
+    turns column index into a list of strings
+
+    Parameters
+    ----------
+    df : pd.DataFrame
+        list of indices to sample from
+    copy_df : bool, default=False
+        whether to mutate df or return a copy
+        if False, index of df is mutated
+        if True, original df is not mutated. If index is not a list of strings,
+        a copy is made and the copy is mutated. Otherwise, the original df is returned.
+    """
+    cols = df.columns
+    str_cols = cols.astype(str)
+
+    if not np.all(str_cols == cols):
+        if copy_df:
+            df = df.copy()
+        df.columns = str_cols
+
+    return df


Test patch:
diff --git a/sktime/utils/_testing/deep_equals.py b/sktime/utils/_testing/deep_equals.py
index 84494ad59..0616cc81a 100644
--- a/sktime/utils/_testing/deep_equals.py
+++ b/sktime/utils/_testing/deep_equals.py
@@ -98,11 +98,67 @@ def deep_equals(x, y, return_msg=False):
         else:
             return ret(x.equals(y), ".series_equals, x = {} != y = {}", [x, y])
     elif isinstance(x, pd.DataFrame):
+        # check column names for equality
         if not x.columns.equals(y.columns):
             return ret(
                 False, f".columns, x.columns = {x.columns} != y.columns = {y.columns}"
             )
-        # if columns are equal and at least one is object, recurse over Series
+        # check dtypes for equality
+        if not x.dtypes.equals(y.dtypes):
+            return ret(
+                False, f".dtypes, x.dtypes = {x.dtypes} != y.dtypes = {y.dtypes}"
+            )
+        # check index for equality
+        # we are not recursing due to ambiguity in integer index types
+        # which may differ from pandas version to pandas version
+        # and would upset the type check, e.g., RangeIndex(2) vs Index([0, 1])
+        xix = x.index
+        yix = y.index
+        if hasattr(xix, "dtype") and hasattr(xix, "dtype"):
+            if not xix.dtype == yix.dtype:
+                return ret(
+                    False,
+                    ".index.dtype, x.index.dtype = {} != y.index.dtype = {}",
+                    [xix.dtype, yix.dtype],
+                )
+        if hasattr(xix, "dtypes") and hasattr(yix, "dtypes"):
+            if not x.dtypes.equals(y.dtypes):
+                return ret(
+                    False,
+                    ".index.dtypes, x.dtypes = {} != y.index.dtypes = {}",
+                    [xix.dtypes, yix.dtypes],
+                )
+        ix_eq = xix.equals(yix)
+        if not ix_eq:
+            if not len(xix) == len(yix):
+                return ret(
+                    False,
+                    ".index.len, x.index.len = {} != y.index.len = {}",
+                    [len(xix), len(yix)],
+                )
+            if hasattr(xix, "name") and hasattr(yix, "name"):
+                if not xix.name == yix.name:
+                    return ret(
+                        False,
+                        ".index.name, x.index.name = {} != y.index.name = {}",
+                        [xix.name, yix.name],
+                    )
+            if hasattr(xix, "names") and hasattr(yix, "names"):
+                if not len(xix.names) == len(yix.names):
+                    return ret(
+                        False,
+                        ".index.names, x.index.names = {} != y.index.name = {}",
+                        [xix.names, yix.names],
+                    )
+                if not np.all(xix.names == yix.names):
+                    return ret(
+                        False,
+                        ".index.names, x.index.names = {} != y.index.name = {}",
+                        [xix.names, yix.names],
+                    )
+            elts_eq = np.all(xix == yix)
+            return ret(elts_eq, ".index.equals, x = {} != y = {}", [xix, yix])
+        # if columns, dtypes are equal and at least one is object, recurse over Series
         if sum(x.dtypes == "object") > 0:
             for c in x.columns:
                 is_equal, msg = deep_equals(x[c], y[c], return_msg=True)
@@ -112,6 +168,14 @@ def deep_equals(x, y, return_msg=False):
         else:
             return ret(x.equals(y), ".df_equals, x = {} != y = {}", [x, y])
     elif isinstance(x, pd.Index):
+        if hasattr(x, "dtype") and hasattr(y, "dtype"):
+            if not x.dtype == y.dtype:
+                return ret(False, f".dtype, x.dtype = {x.dtype} != y.dtype = {y.dtype}")
+        if hasattr(x, "dtypes") and hasattr(y, "dtypes"):
+            if not x.dtypes.equals(y.dtypes):
+                return ret(
+                    False, f".dtypes, x.dtypes = {x.dtypes} != y.dtypes = {y.dtypes}"
+                )
         return ret(x.equals(y), ".index_equals, x = {} != y = {}", [x, y])
     elif isinstance(x, np.ndarray):
         if x.dtype != y.dtype:
diff --git a/sktime/utils/sklearn.py b/sktime/utils/sklearn/_scitype.py
similarity index 100%
rename from sktime/utils/sklearn.py
rename to sktime/utils/sklearn/_scitype.py
diff --git a/sktime/utils/sklearn/tests/__init__.py b/sktime/utils/sklearn/tests/__init__.py
new file mode 100644
index 000000000..1d9b760fe
--- /dev/null
+++ b/sktime/utils/sklearn/tests/__init__.py
@@ -0,0 +1,1 @@
+"""Tests for sklearn related utility functionality."""
diff --git a/sktime/utils/sklearn/tests/test_sklearn_df_adapt.py b/sktime/utils/sklearn/tests/test_sklearn_df_adapt.py
new file mode 100644
index 000000000..57b36ebcd
--- /dev/null
+++ b/sktime/utils/sklearn/tests/test_sklearn_df_adapt.py
@@ -0,0 +1,33 @@
+"""Tests for sklearn dataframe coercion."""
+
+__author__ = ["fkiraly"]
+
+import numpy as np
+import pandas as pd
+import pytest
+
+from sktime.utils.sklearn._adapt_df import prep_skl_df
+
+
+@pytest.mark.parametrize("copy_df", [True, False])
+def test_prep_skl_df_coercion(copy_df):
+    """Test that prep_skl_df behaves correctly on the coercion case."""
+    mixed_example = pd.DataFrame({0: [1, 2, 3], "b": [1, 2, 3]})
+
+    res = prep_skl_df(mixed_example, copy_df=copy_df)
+
+    assert np.all(res.columns == ["0", "b"])
+
+    if not copy_df:
+        assert res is mixed_example
+
+
+@pytest.mark.parametrize("copy_df", [True, False])
+def test_prep_skl_df_non_coercion(copy_df):
+    """Test that prep_skl_df behaves correctly on the non-coercion case."""
+    mixed_example = pd.DataFrame({"a": [1, 2, 3], "b": [1, 2, 3]})
+
+    res = prep_skl_df(mixed_example, copy_df=copy_df)
+
+    assert np.all(res.columns == ["a", "b"])
+    assert res is mixed_example