Repository: sktime/sktime
Pull request #6433 (2024-05-16 09:05:59)

[BUG] fix `ForecastX` when `forecaster_X_exogeneous="complement"`
<!--
Welcome to sktime, and thanks for contributing!
Please have a look at our contribution guide:
https://www.sktime.net/en/latest/get_involved/contributing.html
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.

Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests.
If no issue exists, you can open one here: https://github.com/sktime/sktime/issues
-->
fixes https://github.com/sktime/sktime/issues/6405

#### What does this implement/fix? Explain your changes.
<!--
A clear and concise description of what you have implemented.
-->
The issue was the way we were handling the complementation of the columns.

#### PR checklist

##### For all contributions
- [x] The PR title starts with either [ENH], [MNT], [DOC], or [BUG]. [BUG] - bugfix, [MNT] - CI, test framework, [ENH] - adding or improving code, [DOC] - writing or improving documentation or docstrings.


<!--
Thanks for contributing!
-->


fnhirwa: > I think this fixes the bug.
> 
> Best practice is, every bugfix should contain a test or otherwise increase test coverage so the fixed bug would fail that test. That's sometimes called "certifying for the fix".
> 
> In most cases, adding (a simplified or quickened version of) the reproducible code in the bug report does the trick. In cases like estimators, you can work with adding parameter combinations to `get_test_params`. In rare cases, the test system needs to be extended, e.g., by addition of suite tests, if it turns out that an important use case was entirely overlooked - a recent examples is #6428

I agree with this but from my understanding this parameter was tested here https://github.com/sktime/sktime/blob/ed0ee5b8c133aeabf64461a422dfc762db0024cd/sktime/forecasting/compose/_pipeline.py#L1814
yarnabrina: The existing test parameters do not test this argument in presence of explicit mention of `columns`, either as list of strings or as index. I suspect that's how we didn't find it earlier, so adding the test is necessary.

Also, curious what is your point of view about L1578 above? I made a comment/question above.
fnhirwa: > The existing test parameters do not test this argument in presence of explicit mention of `columns`, either as list of strings or as index. I suspect that's how we didn't find it earlier, so adding the test is necessary.
> 
> Also, curious what is your point of view about L1578 above? I made a comment/question above.

This was an overlook I am going to add a test case where we explicitly mention `columns`


Patch:
diff --git a/.all-contributorsrc b/.all-contributorsrc
index 4d039c995..630f97689 100644
--- a/.all-contributorsrc
+++ b/.all-contributorsrc
@@ -2803,6 +2803,15 @@
       "contributions": [
         "maintenance"
       ]
+    },
+    {
+      "login": "helloplayer1",
+      "name": "Julian Haderlein",
+      "avatar_url": "https://avatars.githubusercontent.com/u/32032467?v=4",
+      "profile": "https://github.com/helloplayer1",
+      "contributions": [
+        "doc"
+      ]
     }
   ]
 }
diff --git a/examples/02a_classification_multivariate_cnn.ipynb b/examples/02a_classification_multivariate_cnn.ipynb
deleted file mode 100644
index f4b464240..000000000
--- a/examples/02a_classification_multivariate_cnn.ipynb
+++ /dev/null
@@ -1,136 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Multi-variate time series classification using a simple CNN\n",
-    "\n",
-    "In this notebook, we use sktime to perform for multi-variate time series classification by deep learning."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import numpy as np\n",
-    "import seaborn as sns\n",
-    "from sklearn.model_selection import GridSearchCV\n",
-    "\n",
-    "from sktime.classification.deep_learning.cnn import CNNClassifier\n",
-    "from sktime.datasets import load_basic_motions\n",
-    "\n",
-    "sns.set_style(\"whitegrid\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Load a dataset\n",
-    "The Basic Motions dataset, from [timeseriesclassification.com](http://www.timeseriesclassification.com), has time series in six dimensions."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "X_train, y_train = load_basic_motions(split=\"train\", return_X_y=True)\n",
-    "X_test, y_test = load_basic_motions(split=\"test\", return_X_y=True)\n",
-    "print(X_train.shape)\n",
-    "print(X_test.shape)\n",
-    "print(type(X_train.iloc[1, 1]))\n",
-    "X_train.head()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# The class labels\n",
-    "np.unique(y_train)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Train a deep neural network classifier\n",
-    "Here we choose to use the CNN (convolutional neural network) classifier. Other classifiers provided by sktime-dl include MLP, ResNet, InceptionTime and MCDCNN (Multi Channel Deep Convolutional Neural Network)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "network = CNNClassifier(n_epochs=50, verbose=True)\n",
-    "network.fit(X_train, y_train)\n",
-    "network.score(X_test, y_test)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Grid Search\n",
-    "sktime-dl is compatible with scikit-learn and can use sklearn's GridSearchCV.\n",
-    "\n",
-    "Here we search over two parameters, CNN kernel size and pooling size."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "pycharm": {
-     "is_executing": true
-    }
-   },
-   "outputs": [],
-   "source": [
-    "param_grid = {\"kernel_size\": [7, 9], \"avg_pool_size\": [3, 5]}\n",
-    "grid = GridSearchCV(network, param_grid=param_grid, cv=3)\n",
-    "grid.fit(X_train, y_train)\n",
-    "\n",
-    "print(\"Best cross-validation accuracy: {:.2f}\".format(grid.best_score_))\n",
-    "print(\"Test set score: {:.2f}\".format(grid.score(X_test, y_test)))\n",
-    "print(\"Best parameters: {}\".format(grid.best_params_))"
-   ]
-  }
- ],
- "metadata": {
-  "file_extension": ".py",
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.7.5"
-  },
-  "mimetype": "text/x-python",
-  "name": "python",
-  "npconvert_exporter": "python",
-  "pygments_lexer": "ipython3",
-  "version": 3
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/examples/02b_classification_multivariate_lstmfcn.ipynb b/examples/02b_classification_multivariate_lstmfcn.ipynb
deleted file mode 100644
index f31975c52..000000000
--- a/examples/02b_classification_multivariate_lstmfcn.ipynb
+++ /dev/null
@@ -1,116 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Multi-variate time series classification using a simple CNN\n",
-    "\n",
-    "In this notebook, we use sktime to perform for multi-variate time series classification by deep learning.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import numpy as np\n",
-    "import seaborn as sns\n",
-    "\n",
-    "from sktime.classification.deep_learning.lstmfcn import LSTMFCNClassifier\n",
-    "from sktime.datasets import load_basic_motions\n",
-    "\n",
-    "sns.set_style(\"whitegrid\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Load a dataset\n",
-    "\n",
-    "The Basic Motions dataset, from [timeseriesclassification.com](http://www.timeseriesclassification.com), has time series in six dimensions.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "X_train, y_train = load_basic_motions(split=\"train\", return_X_y=True)\n",
-    "X_test, y_test = load_basic_motions(split=\"test\", return_X_y=True)\n",
-    "print(X_train.shape)\n",
-    "print(X_test.shape)\n",
-    "print(type(X_train.iloc[1, 1]))\n",
-    "X_train.head()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# The class labels\n",
-    "np.unique(y_train)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Train a LSTM-FCN\n",
-    "\n",
-    "In this exampe we use a LSTM-FCN (LongShort Term Memory Fully Convolutional Network) classifier originally implemented in sktime-dl.\n",
-    "Other classifiers provided by sktime-dl include CNN, MLP, ResNet, InceptionTime and MCDCNN.\n",
-    "\n",
-    "The LSTM-FCN estimator is compatible with scikit-learn and can use sklearn's GridSearchCV."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "network = LSTMFCNClassifier(n_epochs=65, verbose=0)\n",
-    "network.fit(X_train, y_train)\n",
-    "network.score(X_test, y_test)"
-   ]
-  }
- ],
- "metadata": {
-  "file_extension": ".py",
-  "kernelspec": {
-   "display_name": "Python 3.9.6 64-bit ('sktime')",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9.6"
-  },
-  "mimetype": "text/x-python",
-  "name": "python",
-  "npconvert_exporter": "python",
-  "pygments_lexer": "ipython3",
-  "version": 3,
-  "vscode": {
-   "interpreter": {
-    "hash": "b1429b21f9b0cdf277ed408d65e56e85a8ed0b1a3006279bd7cbcc8cf36ab395"
-   }
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/examples/02c_classification_multivariate_inceptiontime.ipynb b/examples/02c_classification_multivariate_inceptiontime.ipynb
deleted file mode 100644
index 6094630fd..000000000
--- a/examples/02c_classification_multivariate_inceptiontime.ipynb
+++ /dev/null
@@ -1,153 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Multi-variate time series classification using InceptionTime\n",
-    "\n",
-    "In this notebook, we use InceptionTime to perform for multi-variate time series classification by deep learning."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "pycharm": {
-     "is_executing": true
-    }
-   },
-   "outputs": [],
-   "source": [
-    "\"\"\" Imports \"\"\"\n",
-    "import numpy as np\n",
-    "import seaborn as sns\n",
-    "from sklearn.model_selection import GridSearchCV\n",
-    "\n",
-    "from sktime.classification.deep_learning.inceptiontime import InceptionTimeClassifier\n",
-    "from sktime.datasets import load_basic_motions\n",
-    "\n",
-    "sns.set_style(\"whitegrid\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Load a dataset\n",
-    "The Basic Motions dataset, from [timeseriesclassification.com](http://www.timeseriesclassification.com), has time series in six dimensions."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "pycharm": {
-     "is_executing": true
-    }
-   },
-   "outputs": [],
-   "source": [
-    "X_train, y_train = load_basic_motions(split=\"train\", return_X_y=True)\n",
-    "X_test, y_test = load_basic_motions(split=\"test\", return_X_y=True)\n",
-    "print(X_train.shape)\n",
-    "print(X_test.shape)\n",
-    "X_train.head()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "pycharm": {
-     "is_executing": true
-    }
-   },
-   "outputs": [],
-   "source": [
-    "# The class labels\n",
-    "np.unique(y_train)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Train a deep neural network classifier\n",
-    "Here we choose to use the InceptionTime classifier. Other classifiers provided by sktime-dl include MLP, ResNet, MCDCNN (Multi Channel Deep Convolutional Neural Network)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "pycharm": {
-     "is_executing": true
-    }
-   },
-   "outputs": [],
-   "source": [
-    "network = InceptionTimeClassifier(n_epochs=75, verbose=False)\n",
-    "network.fit(X_train, y_train)\n",
-    "network.score(X_test, y_test)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Grid Search\n",
-    "sktime-dl is compatible with scikit-learn and can use sklearn's GridSearchCV.\n",
-    "\n",
-    "Here we search over two parameters, CNN kernel size and pooling size."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "pycharm": {
-     "is_executing": true
-    }
-   },
-   "outputs": [],
-   "source": [
-    "param_grid = {\"nb_filters\": [16, 32], \"bottleneck_size\": [16, 32]}\n",
-    "grid = GridSearchCV(network, param_grid=param_grid, cv=5)\n",
-    "# commented fit call to pass checks (meet time constraints)\n",
-    "# grid.fit(X_train, y_train)\n",
-    "\n",
-    "# print(\"Best cross-validation accuracy: {:.2f}\".format(grid.best_score_))\n",
-    "# print(\"Test set score: {:.2f}\".format(grid.score(X_test, y_test)))\n",
-    "# print(\"Best parameters: {}\".format(grid.best_params_))"
-   ]
-  }
- ],
- "metadata": {
-  "file_extension": ".py",
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.7.5"
-  },
-  "mimetype": "text/x-python",
-  "name": "python",
-  "npconvert_exporter": "python",
-  "pygments_lexer": "ipython3",
-  "version": 3
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/examples/AA_datatypes_and_datasets.ipynb b/examples/AA_datatypes_and_datasets.ipynb
index acff86913..adb440011 100644
--- a/examples/AA_datatypes_and_datasets.ipynb
+++ b/examples/AA_datatypes_and_datasets.ipynb
@@ -339,7 +339,7 @@
     "* variable names: the `\"np.ndarray\"` mtype cannot represent variable names.\n",
     "* time points: the rows of `obj` correspond to different, distinct time points. \n",
     "* time index: The time index is implicit and by-convention. The `i`-th row (for an integer `i`) is interpreted as an observation at the time point `i`.\n",
-    "* capabilities: cannot represent multivariate series; cannot represent unequally spaced series"
+    "* capabilities: can represent multivariate series; cannot represent unequally spaced series"
    ]
   },
   {
diff --git a/pyproject.toml b/pyproject.toml
index e7a336ab5..ae8a7e8fb 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -95,7 +95,7 @@ all_extras = [
   'numba<0.60,>=0.53',
   'pmdarima!=1.8.1,<3.0.0,>=1.8; python_version < "3.12"',
   'prophet>=1.1; python_version < "3.12"',
-  "pycatch22<0.4.4",
+  "pycatch22<0.4.6",
   "pykalman<0.10,>=0.9.5",
   'pyod>=0.8; python_version < "3.11"',
   "pyts<0.14.0; python_version < '3.12'",
@@ -136,7 +136,7 @@ all_extras_pandas2 = [
   'numba<0.60,>=0.53',
   'pmdarima!=1.8.1,<3.0.0,>=1.8; python_version < "3.12"',
   'prophet>=1.1; python_version < "3.12"',
-  "pycatch22<0.4.4",
+  "pycatch22<0.4.6",
   "pykalman<0.10,>=0.9.5",
   'pyod>=0.8; python_version < "3.11"',
   "scikit_posthocs>=0.6.5",
@@ -207,7 +207,7 @@ regression = [
 transformations = [
   'esig<0.10,>=0.9.7; python_version < "3.11"',
   "filterpy<1.5,>=1.4.5",
-  "holidays>=0.29,<0.49",
+  "holidays>=0.29,<0.50",
   "mne>=1.5,<1.8",
   'numba<0.60,>=0.53',
   "pycatch22>=0.4,<0.4.6",
diff --git a/sktime/classification/base.py b/sktime/classification/base.py
index cca5c0e8e..4f09ca6da 100644
--- a/sktime/classification/base.py
+++ b/sktime/classification/base.py
@@ -727,6 +727,9 @@ class BaseClassifier(BasePanelMixin):
         y_proba = self._predict_proba(X)
         y_pred = y_proba.argmax(axis=1)
 
+        # restore class labels if they were not originally integer
+        y_pred = self.classes_[y_pred]
+
         return y_pred
 
     def _predict_proba(self, X) -> np.ndarray:
diff --git a/sktime/classification/dictionary_based/_boss.py b/sktime/classification/dictionary_based/_boss.py
index 0bcb4def2..0c1b234b1 100644
--- a/sktime/classification/dictionary_based/_boss.py
+++ b/sktime/classification/dictionary_based/_boss.py
@@ -639,9 +639,7 @@ class IndividualBOSS(BaseClassifier):
             Predicted class labels.
         """
         test_bags = self._transformer.transform(X)
-        data_type = type(self._class_vals[0])
-        if data_type == np.str_ or data_type == str:
-            data_type = "object"
+        data_type = self._class_vals.dtype
 
         classes = np.zeros(test_bags.shape[0], dtype=data_type)
 
diff --git a/sktime/classification/distance_based/_proximity_forest.py b/sktime/classification/distance_based/_proximity_forest.py
index 1626b7e42..eb35e836e 100644
--- a/sktime/classification/distance_based/_proximity_forest.py
+++ b/sktime/classification/distance_based/_proximity_forest.py
@@ -900,33 +900,6 @@ class ProximityStump(BaseClassifier):
         self.entropy = gini_gain(self.y, self.y_branches)
         return self
 
-    def _predict(self, X) -> np.ndarray:
-        """Predicts labels for sequences in X.
-
-        Parameters
-        ----------
-        X : array-like or sparse matrix of shape = [n_instances, n_columns]
-            The training input samples.
-            If a Pandas data frame is passed (sktime format)
-            If a Pandas data frame is passed, a check is performed that it
-            only has one column.
-            If not, an exception is thrown, since this classifier does not
-            yet have
-            multivariate capability.
-
-        Returns
-        -------
-        y : array-like, shape =  [n_instances] - predicted class labels
-        """
-        distributions = self._predict_proba(X)
-        predictions = []
-        for instance_index in range(0, X.shape[0]):
-            distribution = distributions[instance_index]
-            prediction = np.argmax(distribution)
-            predictions.append(prediction)
-
-        return np.array(predictions)
-
     def _predict_proba(self, X) -> np.ndarray:
         """Find probability estimates for each class for all cases in X.
 
@@ -1118,33 +1091,6 @@ class ProximityTree(BaseClassifier):
 
         return self
 
-    def _predict(self, X) -> np.ndarray:
-        """Predicts labels for sequences in X.
-
-        Parameters
-        ----------
-        X : array-like or sparse matrix of shape = [n_instances, n_columns]
-            The training input samples.
-            If a Pandas data frame is passed (sktime format)
-            If a Pandas data frame is passed, a check is performed that it
-            only has one column.
-            If not, an exception is thrown, since this classifier does not
-            yet have
-            multivariate capability.
-
-        Returns
-        -------
-        y : array-like, shape =  [n_instances] - predicted class labels
-        """
-        distributions = self._predict_proba(X)
-        predictions = []
-        for instance_index in range(0, X.shape[0]):
-            distribution = distributions[instance_index]
-            prediction = np.argmax(distribution)
-            predictions.append(prediction)
-
-        return np.array(predictions)
-
     def _predict_proba(self, X) -> np.ndarray:
         """Find probability estimates for each class for all cases in X.
 
@@ -1494,33 +1440,6 @@ class ProximityForest(BaseClassifier):
         """
         return tree.predict_proba(X)
 
-    def _predict(self, X) -> np.ndarray:
-        """Predicts labels for sequences in X.
-
-        Parameters
-        ----------
-        X : array-like or sparse matrix of shape = [n_instances, n_columns]
-            The training input samples.
-            If a Pandas data frame is passed (sktime format)
-            If a Pandas data frame is passed, a check is performed that it
-            only has one column.
-            If not, an exception is thrown, since this classifier does not
-            yet have
-            multivariate capability.
-
-        Returns
-        -------
-        y : array-like, shape =  [n_instances] - predicted class labels
-        """
-        distributions = self._predict_proba(X)
-        predictions = []
-        for instance_index in range(0, X.shape[0]):
-            distribution = distributions[instance_index]
-            prediction = np.argmax(distribution)
-            predictions.append(prediction)
-
-        return np.array(predictions)
-
     def _predict_proba(self, X) -> np.ndarray:
         """Find probability estimates for each class for all cases in X.
 
diff --git a/sktime/classification/ensemble/_bagging.py b/sktime/classification/ensemble/_bagging.py
index 17bbcf391..ccb55f30c 100644
--- a/sktime/classification/ensemble/_bagging.py
+++ b/sktime/classification/ensemble/_bagging.py
@@ -163,6 +163,7 @@ class BaggingClassifier(BaseClassifier):
             n_features_ = n_features
 
         self.estimators_ = []
+        self._col_ixis = []
         for _i in range(n_estimators):
             esti = estimator.clone()
             row_iloc = pd.RangeIndex(n)
@@ -183,6 +184,7 @@ class BaggingClassifier(BaseClassifier):
 
             yi = y[row_ss]
             self.estimators_ += [esti.fit(Xi, yi)]
+            self._col_ixis += [col_ix_i]
 
         return self
 
@@ -200,7 +202,14 @@ class BaggingClassifier(BaseClassifier):
             Predicted probabilities using the ordering in classes_.
         """
         classes = pd.Index(self.classes_)
-        y_probas = [est.predict_proba(X) for est in self.estimators_]
+
+        y_probas = []
+        for esti, col_ix_i in zip(self.estimators_, self._col_ixis):
+            Xi = X.loc[:, col_ix_i]
+            if self.bootstrap_features:
+                Xi.columns = pd.RangeIndex(len(col_ix_i))
+
+            y_probas += [esti.predict_proba(Xi)]
 
         est_shape = (len(y_probas[0]), len(classes))
         y_proba_np = np.zeros((len(y_probas), est_shape[0], est_shape[1]))
@@ -238,16 +247,16 @@ class BaggingClassifier(BaseClassifier):
             instance.
             ``create_test_instance`` uses the first (or only) dictionary in ``params``.
         """
-        from sktime.classification.dummy import DummyClassifier
+        from sktime.classification.feature_based import SummaryClassifier
 
-        params1 = {"estimator": DummyClassifier()}
+        params1 = {"estimator": SummaryClassifier()}
         params2 = {
-            "estimator": DummyClassifier(),
+            "estimator": SummaryClassifier(),
             "n_samples": 0.5,
             "n_features": 0.5,
         }
         params3 = {
-            "estimator": DummyClassifier(),
+            "estimator": SummaryClassifier(),
             "n_samples": 7,
             "n_features": 2,
             "bootstrap": False,
@@ -255,7 +264,7 @@ class BaggingClassifier(BaseClassifier):
         }
 
         # force-create a classifier that cannot handle multivariate
-        univariate_dummy = DummyClassifier()
+        univariate_dummy = SummaryClassifier()
         univariate_dummy.set_tags(**{"capability:multivariate": False})
         # this should still result in a multivariate classifier
         params4 = {
diff --git a/sktime/forecasting/compose/_pipeline.py b/sktime/forecasting/compose/_pipeline.py
index 0338e3615..8d372f6dd 100644
--- a/sktime/forecasting/compose/_pipeline.py
+++ b/sktime/forecasting/compose/_pipeline.py
@@ -1376,13 +1376,13 @@ class ForecastX(BaseForecaster):
         self.fh_X = fh_X
         self.behaviour = behaviour
         self.columns = columns
-        self.forecaster_X_exogeneous = forecaster_X_exogeneous
         if isinstance(forecaster_X_exogeneous, str):
             if forecaster_X_exogeneous not in ["None", "complement"]:
                 raise ValueError(
                     'forecaster_X_exogeneous must be one of "None", "complement",'
                     "or a pandas.Index coercible"
                 )
+        self.forecaster_X_exogeneous = forecaster_X_exogeneous
 
         if predict_behaviour not in ["use_forecasts", "use_actuals"]:
             raise ValueError(
@@ -1583,9 +1583,11 @@ class ForecastX(BaseForecaster):
             return None
 
         if ixx == "complement":
-            X_for_fcX = X.drop(columns=self.columns)
+            X_for_fcX = X.drop(columns=self.columns, errors="ignore")
+
             if X_for_fcX.shape[1] < 1:
                 return None
+
             return X_for_fcX
 
         ixx_pd = pd.Index(ixx)
diff --git a/sktime/performance_metrics/forecasting/probabilistic/__init__.py b/sktime/performance_metrics/forecasting/probabilistic/__init__.py
index 5d17073e0..fbab9f8df 100644
--- a/sktime/performance_metrics/forecasting/probabilistic/__init__.py
+++ b/sktime/performance_metrics/forecasting/probabilistic/__init__.py
@@ -17,6 +17,7 @@ __all__ = [
     "EmpiricalCoverage",
     "ConstraintViolation",
     "CRPS",
+    "IntervalWidth",
     "LogLoss",
     "SquaredDistrLoss",
 ]
@@ -25,6 +26,7 @@ from sktime.performance_metrics.forecasting.probabilistic._classes import (
     CRPS,
     ConstraintViolation,
     EmpiricalCoverage,
+    IntervalWidth,
     LogLoss,
     PinballLoss,
     SquaredDistrLoss,
diff --git a/sktime/performance_metrics/forecasting/probabilistic/_classes.py b/sktime/performance_metrics/forecasting/probabilistic/_classes.py
index 42d727677..cfee5b76f 100644
--- a/sktime/performance_metrics/forecasting/probabilistic/_classes.py
+++ b/sktime/performance_metrics/forecasting/probabilistic/_classes.py
@@ -599,6 +599,70 @@ class EmpiricalCoverage(_BaseProbaForecastingErrorMetric):
         return [params1]
 
 
+class IntervalWidth(_BaseProbaForecastingErrorMetric):
+    """Interval width for interval predictions, sometimes also known as sharpness.
+
+    Parameters
+    ----------
+    multioutput : string "uniform_average" or "raw_values" determines how\
+        multioutput results will be treated.
+
+    score_average : bool, optional, default = True
+        specifies whether scores for each quantile should be averaged.
+    """
+
+    _tags = {
+        "scitype:y_pred": "pred_interval",
+        "lower_is_better": True,
+    }
+
+    def __init__(self, multioutput="uniform_average", score_average=True):
+        self.score_average = score_average
+        self.multioutput = multioutput
+        super().__init__(score_average=score_average, multioutput=multioutput)
+
+    def _evaluate_by_index(self, y_true, y_pred, multioutput, **kwargs):
+        """Logic for finding the metric evaluated at each index.
+
+        y_true : pd.Series, pd.DataFrame or np.array of shape (fh,) or \
+            (fh, n_outputs) where fh is the forecasting horizon
+            Ground truth (correct) target values.
+
+        y_pred : pd.Series, pd.DataFrame or np.array of shape (fh,) or  \
+            (fh, n_outputs)  where fh is the forecasting horizon
+            Forecasted values.
+
+        multioutput : string "uniform_average" or "raw_values" determines how \
+            multioutput results will be treated.
+        """
+        lower = y_pred.iloc[:, y_pred.columns.get_level_values(2) == "lower"].to_numpy()
+        upper = y_pred.iloc[:, y_pred.columns.get_level_values(2) == "upper"].to_numpy()
+
+        if not isinstance(y_true, np.ndarray):
+            y_true_np = y_true.to_numpy()
+        else:
+            y_true_np = y_true
+        if y_true_np.ndim == 1:
+            y_true_np = y_true.reshape(-1, 1)
+
+        scores = np.unique(np.round(y_pred.columns.get_level_values(1), 7))
+        vars = np.unique(y_pred.columns.get_level_values(0))
+
+        metric_array = upper - lower
+
+        out_df = pd.DataFrame(
+            metric_array, columns=pd.MultiIndex.from_product([vars, scores])
+        )
+
+        return out_df
+
+    @classmethod
+    def get_test_params(self):
+        """Retrieve test parameters."""
+        params1 = {}
+        return [params1]
+
+
 class ConstraintViolation(_BaseProbaForecastingErrorMetric):
     """Percentage of interval constraint violations for interval predictions.
 
diff --git a/sktime/utils/git_diff.py b/sktime/utils/git_diff.py
index f5cf6ef72..0d53af9ca 100644
--- a/sktime/utils/git_diff.py
+++ b/sktime/utils/git_diff.py
@@ -126,7 +126,6 @@ def get_changed_lines(file_path, only_indented=True):
         return []
 
 
-@lru_cache
 def get_packages_with_changed_specs():
     """Get packages with changed or added specs.
 
@@ -134,6 +133,20 @@ def get_packages_with_changed_specs():
     -------
     list of str : names of packages with changed or added specs
     """
+    return list(_get_packages_with_changed_specs())
+
+
+@lru_cache
+def _get_packages_with_changed_specs():
+    """Get packages with changed or added specs.
+
+    Private version of get_packages_with_changed_specs,
+    to avoid side effects on the list return.
+
+    Returns
+    -------
+    tuple of str : names of packages with changed or added specs
+    """
     from packaging.requirements import Requirement
 
     changed_lines = get_changed_lines("pyproject.toml")
@@ -161,6 +174,6 @@ def get_packages_with_changed_specs():
         packages.append(pkg)
 
     # make unique
-    packages = list(set(packages))
+    packages = tuple(set(packages))
 
     return packages


Test patch:
diff --git a/docs/source/developer_guide/testing_framework.rst b/docs/source/developer_guide/testing_framework.rst
index 40ebc982e..7841b26f5 100644
--- a/docs/source/developer_guide/testing_framework.rst
+++ b/docs/source/developer_guide/testing_framework.rst
@@ -68,7 +68,8 @@ See below for more details, or the `pytest documentation on the topic`_.
 The ``sktime`` plug-in for ``pytest`` generates the tuples of fixture values for this.
 In the above example, we loop over the following fixtures lists:
 
-* ``estimator_instance`` over estimator instances, obtained from all ``sktime`` estimators via ``create_test_instances_and_names``
+* ``estimator_instance`` over estimator instances, obtained from all ``sktime`` estimators via ``create_test_instances_and_names``, which
+  constructs instances from parameter settings in estimator classes' ``get_test_params``.
 
 * ``scenario`` objects, which encodes data inputs and method call sequences to ``estimator_instance`` (explained in further detail below).
 
diff --git a/sktime/classification/dictionary_based/tests/test_boss.py b/sktime/classification/dictionary_based/tests/test_boss.py
index 1bba876ec..f803785b1 100644
--- a/sktime/classification/dictionary_based/tests/test_boss.py
+++ b/sktime/classification/dictionary_based/tests/test_boss.py
@@ -25,7 +25,7 @@ def dataset():
 @pytest.mark.parametrize(
     "new_class,expected_dtype",
     [
-        ({"1": "Class1", "2": "Class2"}, object),
+        ({"1": "Class1", "2": "Class2"}, "same"),
         ({"1": 1, "2": 2}, int),
         ({"1": 1.0, "2": 2.0}, float),
         ({"1": True, "2": False}, bool),
@@ -39,6 +39,9 @@ def test_individual_boss_classes(dataset, new_class, expected_dtype):
     # change class
     y_train = np.array([new_class[y] for y in y_train])
 
+    if expected_dtype == "same":
+        expected_dtype = y_train.dtype
+
     # train iboss and predict X_test
     iboss = IndividualBOSS()
     iboss.fit(X_train, y_train)
diff --git a/sktime/classification/tests/test_all_classifiers.py b/sktime/classification/tests/test_all_classifiers.py
index 173edc08a..cd16cdfb5 100644
--- a/sktime/classification/tests/test_all_classifiers.py
+++ b/sktime/classification/tests/test_all_classifiers.py
@@ -120,20 +120,17 @@ class TestAllClassifiers(ClassifierFixtureGenerator, QuickTester):
         # we only use the first estimator instance for testing
         classname = estimator_class.__name__
 
-        # retrieve expected predict_proba output, and skip test if not available
-        if classname in unit_test_proba.keys():
-            expected_probas = unit_test_proba[classname]
-        else:
-            # skip test if no expected probas are registered
-            return None
-
         # if numba is not installed, some estimators may still try to construct
         # numba dependenct estimators in results_comparison
         # if that is the case, we skip the test
+        if classname in unit_test_proba.keys():
+            parameter_set = "results_comparison"
+        else:
+            parameter_set = "default"
         try:
             # we only use the first estimator instance for testing
             estimator_instance = estimator_class.create_test_instance(
-                parameter_set="results_comparison"
+                parameter_set=parameter_set
             )
         except ModuleNotFoundError as e:
             if not _check_soft_dependencies("numba", severity="none"):
@@ -152,10 +149,18 @@ class TestAllClassifiers(ClassifierFixtureGenerator, QuickTester):
 
         # train classifier and predict probas
         estimator_instance.fit(X_train, y_train)
+
+        y_pred = estimator_instance.predict(X_test.iloc[indices])
+        assert y_pred.dtype == y_train.dtype
+        assert set(y_train).issuperset(set(y_pred))
+
         y_proba = estimator_instance.predict_proba(X_test.iloc[indices])
 
-        # assert probabilities are the same
-        _assert_array_almost_equal(y_proba, expected_probas, decimal=2)
+        # retrieve expected predict_proba output, and skip test if not available
+        if classname in unit_test_proba.keys():
+            expected_probas = unit_test_proba[classname]
+            # assert probabilities are the same
+            _assert_array_almost_equal(y_proba, expected_probas, decimal=2)
 
     def test_classifier_on_basic_motions(self, estimator_class):
         """Test classifier on basic motions data."""
diff --git a/sktime/forecasting/compose/tests/test_forecastx.py b/sktime/forecasting/compose/tests/test_forecastx.py
index 80d308d19..aec4eccbd 100644
--- a/sktime/forecasting/compose/tests/test_forecastx.py
+++ b/sktime/forecasting/compose/tests/test_forecastx.py
@@ -391,3 +391,41 @@ def test_use_of_passed_unknown_X(predict_behaviour_option: str) -> None:
             mock_predict.assert_called_once()
         elif predict_behaviour_option == "use_actuals":
             mock_predict.assert_not_called()
+
+
+@pytest.mark.skipif(
+    not _check_estimator_deps(ARIMA, severity="none"),
+    reason="skip test if required soft dependency is not available",
+)
+@pytest.mark.parametrize("cols_to_forecast", [["GNPDEFL", "GNP"], ["ARMED", "POP"]])
+def test_forecaster_X_exogeneous(cols_to_forecast):
+    """Test that ForecastX forecaster_X uses exogenous data as told by parameter."""
+    from sktime.forecasting.compose import ForecastX
+    from sktime.split import temporal_train_test_split
+
+    y, X = load_longley()
+
+    fh = [1, 2, 3, 4]
+    y_train, _, X_train, X_test = temporal_train_test_split(y, X, test_size=max(fh))
+
+    forecaster = ARIMA()
+    pipeline1 = ForecastX(
+        forecaster.clone(),
+        forecaster_X=forecaster.clone(),
+        columns=cols_to_forecast,
+        forecaster_X_exogeneous="complement",
+    )
+
+    pipeline1.fit(y_train, X=X_train, fh=fh)
+    y_pred1 = pipeline1.predict(X=X_test.drop(columns=cols_to_forecast))
+
+    pipeline2 = ForecastX(
+        forecaster.clone(),
+        forecaster_X=forecaster.clone(),
+        columns=cols_to_forecast,
+        forecaster_X_exogeneous="None",
+    )
+
+    pipeline2.fit(y_train, X=X_train, fh=fh)
+    y_pred2 = pipeline2.predict(X=X_test.drop(columns=cols_to_forecast))
+    np.testing.assert_array_equal(y_pred1.index, y_pred2.index)
diff --git a/sktime/performance_metrics/forecasting/probabilistic/tests/test_probabilistic_metrics.py b/sktime/performance_metrics/forecasting/probabilistic/tests/test_probabilistic_metrics.py
index 2c683a688..b5e450d51 100644
--- a/sktime/performance_metrics/forecasting/probabilistic/tests/test_probabilistic_metrics.py
+++ b/sktime/performance_metrics/forecasting/probabilistic/tests/test_probabilistic_metrics.py
@@ -8,6 +8,7 @@ import pytest
 from sktime.performance_metrics.forecasting.probabilistic import (
     ConstraintViolation,
     EmpiricalCoverage,
+    IntervalWidth,
     PinballLoss,
 )
 from sktime.split import temporal_train_test_split
@@ -20,8 +21,9 @@ quantile_metrics = [
 ]
 
 interval_metrics = [
-    EmpiricalCoverage,
     ConstraintViolation,
+    EmpiricalCoverage,
+    IntervalWidth,
 ]
 
 all_metrics = interval_metrics + quantile_metrics