Repository: sktime/sktime

Issue:
[BUG] Using IndexSubset together with PandasTransformAdaptor(method="dropna") in a pipeline fails
**Describe the bug**
Using `IndexSubset` and `PandasTransformAdaptor(method="dropna")` together in a pipeline fails in some orderings. `PandasTransformAdaptor` appears to be operating on the untransformed/not subset `X`. 

**To Reproduce**
The example below fails. It works with the `PandasTransformAdaptor` commented out.  

The example differences `y`, lags `X`, then sets `y` and `X` to the same index (effectively removing the missing values from `X`). Dropping `nan` values should not have any effect. 

```python
import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sktime.datasets import load_longley
from sktime.forecasting.base import ForecastingHorizon
from sktime.forecasting.compose import (
    ForecastingPipeline,
    TransformedTargetForecaster,
    RecursiveTabularRegressionForecaster,
)
from sklearn.preprocessing import MinMaxScaler
from sktime.transformations.series.adapt import TabularToSeriesAdaptor, PandasTransformAdaptor
from sktime.transformations.series.boxcox import LogTransformer
from sktime.transformations.series.difference import Differencer
from sktime.transformations.series.subset import IndexSubset
from sktime.transformations.series.lag import Lag

y, X = load_longley()
# X.loc["1955", "GNPDEFL"] = np.nan
# X.loc["1947", "POP"] = np.nan

horizon = ForecastingHorizon(np.arange(1, 4), is_relative=True)

pipe = TransformedTargetForecaster(
    [
        LogTransformer(),
        Differencer(lags=1, na_handling="drop_na"),
        ForecastingPipeline(
            [
                Lag(lags=[0, 1], index_out="original"),
                IndexSubset(index_treatment="remove"),
                PandasTransformAdaptor(method="dropna", kwargs={"axis": "columns", "how": "any"}),
                TabularToSeriesAdaptor(MinMaxScaler()),
                RecursiveTabularRegressionForecaster(RandomForestRegressor(), window_length=2),
            ],
        ),
    ]
)
pipe.fit(y, X, fh=horizon)
pipe.predict(fh=horizon, X=X)

pipe._X
pipe.forecaster_.forecaster_._X
```

**Expected behavior**
Drop only features in `X` that are actually missing at this step in the pipeline.

**Versions**
<details>
System:
    python: 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0]
executable: /usr/bin/python3.10
   machine: Linux-5.19.0-50-generic-x86_64-with-glibc2.35
Python dependencies:
          pip: 22.0.2
       sktime: 0.21.0
      sklearn: 1.1.2
       skbase: 0.4.6
        numpy: 1.22.4
        scipy: 1.8.1
       pandas: 1.4.4
   matplotlib: 3.5.3
       joblib: 1.1.0
  statsmodels: 0.13.2
        numba: 0.56.0
     pmdarima: 2.0.1
      tsfresh: 0.19.0
   tensorflow: None
tensorflow_probability: None
</details>



Discussion:
I see - the problem is that the `PandasTransformAdaptor` drops exactly the columns with NAs that it sees at the moment, and these can differ between `fit` and `predict`.

So, in a sense, `PandasTransformAdaptor` does what it's told to do, but the implied behaviour is problematic and does not match the user expectation.

Question, what should the end state be here? Not sure if `PandasTransformAdaptor` can or should be changed.

If it should be changed, how? If not, then what should be the way to get what you want?

Assuming that what you want is dropping exactly the rows that have at least NA in the `fit` data, in both `fit` and `predict`?

Should this be a `DropNA` transformer perhaps? Which can be applied to rows and columns, and for columns remembers the drops from `fit`?
(re status I'm not sure yet whether this is a bug?)
> I see - the problem is that the `PandasTransformAdaptor` drops exactly the columns with NAs that it sees at the moment, and these can differ between `fit` and `predict`.

But in the example, the X in `fit` and `predict` are the same? And the example below (not relying on `IndexSubset`) works.

```python
import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sktime.datasets import load_longley
from sktime.forecasting.base import ForecastingHorizon
from sktime.forecasting.compose import ForecastingPipeline, RecursiveTabularRegressionForecaster
from sklearn.preprocessing import MinMaxScaler
from sktime.transformations.series.adapt import TabularToSeriesAdaptor, PandasTransformAdaptor

y, X = load_longley()
X.loc["1955", "GNPDEFL"] = np.nan
X.loc["1947", "POP"] = np.nan

horizon = ForecastingHorizon(np.arange(1, 4), is_relative=True)

pipe = ForecastingPipeline(
    [
        PandasTransformAdaptor(method="dropna", kwargs={"axis": "columns", "how": "any"}),
        TabularToSeriesAdaptor(MinMaxScaler()),
        RecursiveTabularRegressionForecaster(RandomForestRegressor(), window_length=2),
    ],
)

pipe.fit(y, X, fh=horizon)
pipe.predict(fh=horizon, X=X)

pipe._X
pipe.forecaster_._X
```

> Assuming that what you want is dropping exactly the rows that have at least NA in the `fit` data, in both `fit` and `predict`?

Yes, that's what I had in mind (assuming you meant columns, not rows).

> Should this be a `DropNA` transformer perhaps? Which can be applied to rows and columns, and for columns remembers the drops from `fit`

Perhaps - having this functionality would be nice (not a bug but a feature request then.) 

If it is not overly complex and somewhat beginner friendly, I'd be happy to start working on a `DropNA` transformer. I'd probably need some guidance on where to begin though. 
Ah, thanks! That might be a useful addition!

The guide for adding estimators is here: https://www.sktime.net/en/stable/developer_guide/add_estimators.html

In your case, I would start with the extension template `extension_templates/transformer_supersimple`, since the "drop NA" logic should be a few-liner on top of the simplest template:

* in `_fit`, drop the columns and remember which columns were dropped (write to `self.dropped_cols_` or similar)
* in `_transform`, remember the dropped columns and drop them
* throughout, care should be taken not to mutate the inputs by accidents
* not sure whether we need any parameters for this. What would be useful is to define an axis perhaps? Since some users may like to use it to drop rows, not columns. In that case, nothing needs to be remembered.

Resolving patch:
diff --git a/sktime/transformations/series/dropna.py b/sktime/transformations/series/dropna.py
new file mode 100644
--- /dev/null
+++ b/sktime/transformations/series/dropna.py
@@ -0,0 +1,245 @@
+# copyright: sktime developers, BSD-3-Clause License (see LICENSE file)
+"""Transformer to drop rows or columns containing missing values."""
+
+__author__ = ["hliebert"]
+
+from sktime.transformations.base import BaseTransformer
+
+
+class DropNA(BaseTransformer):
+    """Drop missing values transformation.
+
+    Drops rows or columns with missing values from X. Mostly wraps
+    pandas.DataFrame.dropna, but allows specifying thresh as a fraction of
+    non-missing observations.
+
+    Parameters
+    ----------
+    axis : {0 or 'index', 1 or 'columns'}, default 0
+        Determine if rows or columns which contain missing values are removed.
+        Must be 0 or 'index' for univariate input.
+
+        * 0, or 'index' : Drop rows which contain missing values.
+        * 1, or 'columns' : Drop columns which contain missing value.
+
+    how : {'any', 'all'}, default 'any'
+        Determine if row or column is removed from DataFrame, when we have
+        at least one NA or all NA.
+
+        * 'any' : If any NA values are present, drop that row or column.
+        * 'all' : If all values are NA, drop that row or column.
+
+    thresh : int or float, optional
+         If int, require at least that many non-NA values (as in pandas.dropna).
+         If float, minimum share of non-NA values for rows/columns to be
+         retained. Fraction must be contained within (0,1]. Setting fraction
+         to 1.0 is equivalent to setting how='any'. thresh cannot be combined
+         with how.
+
+    remember : bool, default False if axis==0, True if axis==1
+        If True, drops the same rows/columns in transform as in fit. If false,
+        drops rows/columns according to the NAs seen in transform (equivalent
+        to PandasTransformAdaptor(method="dropna")).
+    """
+
+    _tags = {
+        "univariate-only": False,
+        "scitype:transform-input": "Series",
+        "scitype:transform-output": "Series",
+        "scitype:instancewise": True,
+        "scitype:transform-labels": "None",
+        "X_inner_mtype": ["pd.DataFrame", "pd-multiindex", "pd_multiindex_hier"],
+        "fit_is_empty": False,
+        "capability:inverse_transform": False,
+        "capability:unequal_length": True,
+        "handles-missing-data": False,
+    }
+
+    VALID_AXIS_VALUES = [0, "index", 1, "columns"]
+    VALID_HOW_VALUES = [None, "any", "all"]
+    VALID_THRESH_TYPES = (type(None), int, float)
+    VALID_REMEMBER_TYPES = (type(None), bool)
+
+    def __init__(self, axis=0, how=None, thresh=None, remember=None):
+        self.axis = self._check_axis(axis)
+        self.how = self._check_how(how)
+        self.thresh = self._check_thresh(thresh, how)
+        self.remember = self._check_remember(remember)
+        super().__init__()
+
+        # axis, use numeric axis internally, default rows/index
+        if self.axis == "index":
+            self._axis = 0
+        elif self.axis == "columns":
+            self._axis = 1
+        else:
+            self._axis = self.axis
+
+        # criterion (how/thresh), default to "any" if neither how nor thresh passed
+        if (self.how is None) and (self.thresh is None):
+            self._how = "any"
+        else:
+            self._how = how
+        self._thresh = self.thresh
+
+        # remember, default to remember dropped columns but not rows
+        if self.remember is None:
+            self._remember = bool(self._axis)
+        else:
+            self._remember = self.remember
+
+    def _check_axis(self, axis):
+        """Check axis parameter, should be a valid string as per docstring."""
+        if axis not in self.VALID_AXIS_VALUES:
+            raise ValueError(
+                f'invalid axis parameter value encountered: "{axis}", '
+                f"axis must be one of: {self.VALID_AXIS_VALUES}"
+            )
+
+        return axis
+
+    def _check_how(self, how):
+        """Check how parameter, should be a valid string as per docstring."""
+        if how not in self.VALID_HOW_VALUES:
+            raise ValueError(
+                f'invalid how parameter value encountered: "{how}", '
+                f"how must be one of: {self.VALID_HOW_VALUES}"
+            )
+
+        return how
+
+    def _check_thresh(self, thresh, how):
+        """Check thresh parameter, should be a valid value as per docstring."""
+        if not isinstance(thresh, self.VALID_THRESH_TYPES) or isinstance(thresh, bool):
+            raise TypeError(
+                f'invalid thresh parameter value encountered: "{thresh}", '
+                f"thresh must be of type: {self.VALID_THRESH_TYPES}"
+            )
+        if (isinstance(thresh, int) and not (thresh > 0)) or (
+            isinstance(thresh, float) and not (0 < thresh <= 1)
+        ):
+            raise ValueError(
+                "thresh must be positive integer or a fraction between zero and one"
+            )
+
+        if (how is not None) and (thresh is not None):
+            raise TypeError("thresh cannot be set together with how")
+
+        return thresh
+
+    def _check_remember(self, remember):
+        """Check remember parameter, should be a valid type as per docstring."""
+        if not isinstance(remember, self.VALID_REMEMBER_TYPES):
+            raise TypeError(
+                f'invalid remember parameter value encountered: "{remember}", '
+                f"remember must be of type: {self.VALID_REMEMBER_TYPES}"
+            )
+
+        return remember
+
+    def _fit(self, X, y=None):
+        """Fit transformer to X and y.
+
+        private _fit containing the core logic, called from fit
+
+        Parameters
+        ----------
+        X : pd.DataFrame
+            if self.get_tag("univariate-only")==True:
+                guaranteed to have a single column
+            if self.get_tag("univariate-only")==False: no restrictions apply
+        y : None, present only for interface compatibility
+
+        Returns
+        -------
+        self: reference to self
+        """
+        self.dropped_index_values_ = None
+        self._agg_axis = 1 - self._axis
+
+        mask = None
+        if self._how == "any":
+            mask = X.isna().any(axis=self._agg_axis)
+        elif self._how == "all":
+            mask = X.isna().all(axis=self._agg_axis)
+        elif isinstance(self._thresh, int):
+            mask = X.count(axis=self._agg_axis) < self._thresh
+        elif isinstance(self._thresh, float):
+            mask = X.notna().mean(axis=self._agg_axis) < self._thresh
+
+        if mask is not None:
+            self.dropped_index_values_ = mask.index[mask].to_list()
+        else:
+            self.dropped_index_values_ = None
+
+        return self
+
+    def _transform(self, X, y=None):
+        """Transform X and return a transformed version.
+
+        private _transform containing core logic, called from transform
+
+        Parameters
+        ----------
+        X : pd.DataFrame
+            if self.get_tag("univariate-only")==True:
+                guaranteed to have a single column
+            if self.get_tag("univariate-only")==False: no restrictions apply
+        y : None, present only for interface compatibility
+
+        Returns
+        -------
+        transformed version of X
+        """
+        dropped_index_values = self.dropped_index_values_
+        agg_axis = self._agg_axis
+        axis = self._axis
+        how = self._how
+        thresh = self._thresh
+        remember = self._remember
+
+        if remember:
+            if dropped_index_values is not None:
+                return X.drop(labels=dropped_index_values, axis=axis)
+            else:
+                return X
+        else:
+            if isinstance(thresh, float):
+                mask = X.notna().mean(axis=agg_axis) < thresh
+                index_to_drop = mask.index[mask]
+                return X.drop(labels=index_to_drop, axis=axis)
+            elif isinstance(thresh, int):
+                return X.dropna(axis=axis, thresh=thresh)
+            else:
+                return X.dropna(axis=axis, how=how)
+
+    @classmethod
+    def get_test_params(cls, parameter_set="default"):
+        """Return testing parameter settings for the estimator.
+
+        Parameters
+        ----------
+        parameter_set : str, default="default"
+            Name of the set of test parameters to return, for use in tests. If no
+            special parameters are defined for a value, will return `"default"` set.
+            There are currently no reserved values for transformers.
+
+        Returns
+        -------
+        params : dict or list of dict, default = {}
+            Parameters to create testing instances of the class
+            Each dict are parameters to construct an "interesting" test instance, i.e.,
+            `MyClass(**params)` or `MyClass(**params[i])` creates a valid test instance.
+            `create_test_instance` uses the first (or only) dictionary in `params`
+        """
+        params = [
+            {"axis": 0, "how": "any", "thresh": None},
+            {"axis": 1, "how": "any", "thresh": None},
+            {"axis": 0, "how": "all", "thresh": None},
+            {"axis": 1, "how": "all", "thresh": None},
+            {"axis": 0, "how": None, "thresh": 0.9},
+            {"axis": 1, "how": None, "thresh": 0.9},
+            {"axis": 1, "how": None, "thresh": 3},
+        ]
+
+        return params


Test patch:
diff --git a/sktime/transformations/series/tests/test_dropna.py b/sktime/transformations/series/tests/test_dropna.py
new file mode 100644
--- /dev/null
+++ b/sktime/transformations/series/tests/test_dropna.py
@@ -0,0 +1,133 @@
+#!/usr/bin/env python3 -u
+# copyright: sktime developers, BSD-3-Clause License (see LICENSE file)
+"""Unit tests of DropNA functionality."""
+
+__author__ = ["hliebert"]
+__all__ = []
+
+import numpy as np
+import pytest
+
+from sktime.datasets import load_longley
+from sktime.transformations.series.dropna import DropNA
+from sktime.utils._testing.estimator_checks import _assert_array_almost_equal
+
+y_few_na, X_few_na = load_longley()
+X_few_na.loc["1947", "GNPDEFL"] = np.nan
+X_few_na.loc["1950", "GNPDEFL"] = np.nan
+X_few_na.loc["1950", "GNP"] = np.nan
+
+X_few_na_expected = {
+    "0": {
+        "None": X_few_na.drop(labels=["1947", "1950"], axis=0),
+        "any": X_few_na.drop(labels=["1947", "1950"], axis=0),
+        "all": X_few_na,
+    },
+    "index": {
+        "None": X_few_na.drop(labels=["1947", "1950"], axis=0),
+        "any": X_few_na.drop(labels=["1947", "1950"], axis=0),
+        "all": X_few_na,
+    },
+    "1": {
+        "None": X_few_na.drop(labels=["GNPDEFL", "GNP"], axis=1),
+        "any": X_few_na.drop(labels=["GNPDEFL", "GNP"], axis=1),
+        "all": X_few_na,
+    },
+    "columns": {
+        "None": X_few_na.drop(labels=["GNPDEFL", "GNP"], axis=1),
+        "any": X_few_na.drop(labels=["GNPDEFL", "GNP"], axis=1),
+        "all": X_few_na,
+    },
+}
+
+y_many_na, X_many_na = load_longley()
+X_many_na.loc["1947", "GNPDEFL"] = np.nan
+X_many_na.loc["1950", "GNPDEFL"] = np.nan
+X_many_na.loc["1950", "GNP"] = np.nan
+X_many_na.loc[:, "ARMED"] = np.nan
+X_many_na.loc[:, "POP"] = np.nan
+X_many_na.loc["1960", :] = np.nan
+X_many_na.loc["1962", :] = np.nan
+X_many_na.dropna(axis=1)
+
+X_many_na_expected = {
+    "0": {
+        "None": X_many_na.drop(labels=X_many_na.index, axis=0),
+        "any": X_many_na.drop(labels=X_many_na.index, axis=0),
+        "all": X_many_na.drop(labels=["1960", "1962"]),
+    },
+    "index": {
+        "None": X_many_na.drop(labels=X_many_na.index, axis=0),
+        "any": X_many_na.drop(labels=X_many_na.index, axis=0),
+        "all": X_many_na.drop(labels=["1960", "1962"]),
+    },
+    "1": {
+        "None": X_many_na.drop(labels=X_many_na.columns, axis=1),
+        "any": X_many_na.drop(labels=X_many_na.columns, axis=1),
+        "all": X_many_na.drop(labels=["ARMED", "POP"], axis=1),
+    },
+    "columns": {
+        "None": X_many_na.drop(labels=X_many_na.columns, axis=1),
+        "any": X_many_na.drop(labels=X_many_na.columns, axis=1),
+        "all": X_many_na.drop(labels=["ARMED", "POP"], axis=1),
+    },
+}
+
+
+@pytest.mark.parametrize("axis", DropNA.VALID_AXIS_VALUES)
+@pytest.mark.parametrize("how", DropNA.VALID_HOW_VALUES)
+def test_dropna_few_na(axis, how):
+    """Test expected results on a DataFrame with occasional missings."""
+    transformer = DropNA(axis=axis, how=how, thresh=None)
+    X_transformed = transformer.fit_transform(X_few_na)
+    X_expected = X_few_na_expected[str(axis)][str(how)]
+
+    _assert_array_almost_equal(X_transformed, X_expected)
+
+
+@pytest.mark.parametrize("axis", DropNA.VALID_AXIS_VALUES)
+@pytest.mark.parametrize("how", DropNA.VALID_HOW_VALUES)
+def test_dropna_many_na(axis, how):
+    """Test expected results on a DataFrame with complete columns/rows missing."""
+    transformer = DropNA(axis=axis, how=how, thresh=None)
+    X_transformed = transformer.fit_transform(X_many_na)
+    X_expected = X_many_na_expected[str(axis)][str(how)]
+
+    _assert_array_almost_equal(X_transformed, X_expected)
+
+
+@pytest.mark.parametrize("axis", DropNA.VALID_AXIS_VALUES)
+@pytest.mark.parametrize("how", ["any", "all"])
+@pytest.mark.parametrize("thresh", [10, 0.5])
+@pytest.mark.parametrize("remember", [None, True, False])
+def test_dropna_conflicting_arguments(axis, how, thresh, remember):
+    """Test that how and thresh arguments cannot be both set."""
+    with pytest.raises(TypeError, match=r"thresh cannot be set together with how"):
+        DropNA(axis=axis, how=str(how), thresh=thresh)
+
+
+@pytest.mark.parametrize("axis", DropNA.VALID_AXIS_VALUES)
+@pytest.mark.parametrize("how", [True, False, "invalid_value", [1, 2, 3]])
+@pytest.mark.parametrize("remember", [None, True, False])
+def test_dropna_invalid_arguments_how(axis, how, remember):
+    """Test that invalid arguments for how are not accepted."""
+    with pytest.raises(ValueError):
+        DropNA(axis=axis, how=how, thresh=None, remember=remember)
+
+
+@pytest.mark.parametrize("axis", DropNA.VALID_AXIS_VALUES)
+@pytest.mark.parametrize("thresh", [True, False, "invalid_value", [1, 2, 3]])
+@pytest.mark.parametrize("remember", [None, True, False])
+def test_dropna_invalid_arguments_thresh_type(axis, thresh, remember):
+    """Test that invalid arguments for thresh are not accepted."""
+    with pytest.raises(TypeError):
+        DropNA(axis=axis, how=None, thresh=thresh, remember=remember)
+
+
+@pytest.mark.parametrize("axis", DropNA.VALID_AXIS_VALUES)
+@pytest.mark.parametrize("thresh", [-5, -3.5, 1.5])
+@pytest.mark.parametrize("remember", [None, True, False])
+def test_dropna_invalid_arguments_thresh_value(axis, thresh, remember):
+    """Test that arguments outside sensible range for thresh are not accepted."""
+    with pytest.raises(ValueError):
+        DropNA(axis=axis, how=None, thresh=thresh)