Repository: sktime/sktime

Issue:
ARIMA and AutoARIMA should return pandas.NA when value cannot be predicted, e.g., first index in-sample for start=0, d=1
Update 2021-09-15 - re-opened since a recurring issue.

Not a bug, but an interface convention - currently ARIMA, AutoARIMA and other learners fail when asked for a prediction that is undefined, e.g., difference parameter d=1 and first in-sample index (start=0).

It would be more sensible if `pandas.NA` is returned instead for such indices, perhaps a warning should be raised too that `pandas.NA` are produced.

In first instance, we should update ARIMA and AutoARIMA with this convention.

---

**Describe the bug**

When using `Imputer` with `forecaster = AutoARIMA()`  the error "ValueError: In-sample predictions undefined for start=0 when d=1" is raised.

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-11-2d05f4822fd6> in <module>()
      4 
      5 transformer = Imputer(method="forecaster", forecaster=AutoARIMA()).fit(Z=df.value)
----> 6 y_hat = transformer.transform(Z=df.value)

/usr/local/lib/python3.7/dist-packages/sktime/transformations/series/impute.py in transform(self, Z, X)
    102             z_pred = forecaster.fit(
    103                 z.fillna(method="ffill").fillna(method="backfill")
--> 104             ).predict(fh=fh_ins)
    105             # fill with trend values
    106             z = z.fillna(value=z_pred)

/usr/local/lib/python3.7/dist-packages/sktime/forecasting/base/_sktime.py in predict(self, fh, X, return_pred_int, alpha)
    226         self.check_is_fitted()
    227         self._set_fh(fh)
--> 228         return self._predict(self.fh, X, return_pred_int=return_pred_int, alpha=alpha)
    229 
    230     def compute_pred_int(self, y_pred, alpha=DEFAULT_ALPHA):

/usr/local/lib/python3.7/dist-packages/sktime/forecasting/base/adapters/_pmdarima.py in _predict(self, fh, X, return_pred_int, alpha)
     58         # all values are in-sample
     59         elif fh.is_all_in_sample(self.cutoff):
---> 60             return self._predict_in_sample(fh_ins, **kwargs)
     61 
     62         # both in-sample and out-of-sample values

/usr/local/lib/python3.7/dist-packages/sktime/forecasting/base/adapters/_pmdarima.py in _predict_in_sample(self, fh, X, return_pred_int, alpha)
     80             X=X,
     81             return_conf_int=return_pred_int,
---> 82             alpha=alpha,
     83         )
     84 

/usr/local/lib/python3.7/dist-packages/pmdarima/utils/metaestimators.py in <lambda>(*args, **kwargs)
     51 
     52         # lambda, but not partial, allows help() to work with update_wrapper
---> 53         out = (lambda *args, **kwargs: self.fn(obj, *args, **kwargs))
     54         # update the docstring of the returned function
     55         update_wrapper(out, self.fn)

/usr/local/lib/python3.7/dist-packages/pmdarima/arima/auto.py in predict_in_sample(self, X, start, end, dynamic, return_conf_int, alpha, typ, **kwargs)
    242             return_conf_int=return_conf_int,
    243             alpha=alpha,
--> 244             typ=typ,
    245         )
    246 

/usr/local/lib/python3.7/dist-packages/pmdarima/arima/arima.py in predict_in_sample(self, X, start, end, dynamic, return_conf_int, alpha, **kwargs)
    584         if start is not None and start < d:
    585             raise ValueError("In-sample predictions undefined for start={0} "
--> 586                              "when d={1}".format(start, d))
    587 
    588         # if we fit with exog, make sure one was passed:

ValueError: In-sample predictions undefined for start=0 when d=1
```

**To Reproduce**

Data:
[df.csv](https://github.com/alan-turing-institute/sktime/files/6426660/df.csv) 

```python
import pandas as pd
from sktime.transformations.series.impute import Imputer
from sktime.forecasting.arima import AutoARIMA

df = pd.read_csv('df.csv')  
df['timestamp'] = pd.DatetimeIndex(df['timestamp']).to_period('D')
df = df.set_index('timestamp')

transformer = Imputer(method="forecaster", forecaster=AutoARIMA()).fit(Z=df.value)
y_hat = transformer.transform(Z=df.value)
```

**Expected behavior**

`transformer.transform(Z)` should return `Z` with NAn values replaced by AutoArima predictions.

**Additional context**

I think the problem is in `/usr/local/lib/python3.7/dist-packages/sktime/transformations/series/impute.py` in `transform(self, Z, X)`, line 100, because all the points are being passed to `fh_ins` instead of just the points with NaN.

```python
100             fh_ins = -np.arange(len(z))
101             # fill NaN before fitting with ffill and backfill (heuristic)
102             z_pred = forecaster.fit(
103                 z.fillna(method="ffill").fillna(method="backfill")
104             ).predict(fh=fh_ins)
105             # fill with trend values
106             z = z.fillna(value=z_pred)

```

**Versions**
<details>

```
System:
    python: 3.7.10 (default, Feb 20 2021, 21:17:23)  [GCC 7.5.0]
executable: /usr/bin/python3
   machine: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic

Python dependencies:
          pip: 19.3.1
   setuptools: 56.0.0
      sklearn: 0.24.2
       sktime: 0.6.0
  statsmodels: 0.12.2
        numpy: 1.19.5
        scipy: 1.4.1
       Cython: 0.29.22
       pandas: 1.1.5
   matplotlib: 3.2.2
       joblib: 1.0.1
        numba: 0.51.2
     pmdarima: 1.8.2
      tsfresh: None
```

</details>



Discussion:
Hi @julramos - thanks for raising the issue! 

I think you have correctly identified the problem here. Would you like to open a PR and see if that fixes the issue? 

An alternative quick fix would be to set `d=0` if this is acceptable:
```python
import pandas as pd
import numpy as np
from sktime.transformations.series.impute import Imputer
from sktime.forecasting.arima import AutoARIMA
from sktime.datasets import load_airline

y = load_airline()
y.iloc[np.random.randint(0, len(y), 20)] = np.nan  # add missing values
transformer = Imputer(method="forecaster", forecaster=AutoARIMA(d=0, suppress_warnings=True))
y_hat = transformer.fit_transform(y)
```

Resolving patch:
diff --git a/sktime/transformations/series/impute.py b/sktime/transformations/series/impute.py
--- a/sktime/transformations/series/impute.py
+++ b/sktime/transformations/series/impute.py
@@ -1,6 +1,6 @@
 #!/usr/bin/env python3 -u
 # -*- coding: utf-8 -*-
-
+"""Utilities to impute series with missing values."""
 __author__ = ["Martin Walter"]
 __all__ = ["Imputer"]
 
@@ -8,13 +8,15 @@
 from sktime.utils.validation.series import check_series
 from sktime.forecasting.trend import PolynomialTrendForecaster
 from sklearn.utils import check_random_state
+from sktime.forecasting.base import ForecastingHorizon
+from sklearn.base import clone
 
 import numpy as np
 import pandas as pd
 
 
 class Imputer(_SeriesToSeriesTransformer):
-    """Missing value imputation
+    """Missing value imputation.
 
     Parameters
     ----------
@@ -75,6 +77,7 @@ def __init__(
 
     def transform(self, Z, X=None):
         """Transform data.
+
         Returns a transformed version of Z.
 
         Parameters
@@ -94,7 +97,10 @@ def transform(self, Z, X=None):
         if self.missing_values:
             Z = Z.replace(to_replace=self.missing_values, value=np.nan)
 
-        if self.method == "random":
+        if not _has_missing_values(Z):
+            return Z
+
+        elif self.method == "random":
             if isinstance(Z, pd.DataFrame):
                 for col in Z:
                     Z[col] = Z[col].apply(
@@ -106,26 +112,12 @@ def transform(self, Z, X=None):
             Z = Z.fillna(value=self.value)
         elif self.method in ["backfill", "bfill", "pad", "ffill"]:
             Z = Z.fillna(method=self.method)
-        elif self.method in ["drift", "forecaster"]:
-            if self.method == "forecaster":
-                forecaster = self.forecaster
-            else:
-                forecaster = PolynomialTrendForecaster(degree=1)
-            # in-sample forecasting horizon
-            fh_ins = -np.arange(len(Z))
-            # fill NaN before fitting with ffill and backfill (heuristic)
-            Z = Z.fillna(method="ffill").fillna(method="backfill")
-            # multivariate
-            if isinstance(Z, pd.DataFrame):
-                for col in Z:
-                    forecaster.fit(y=Z[col])
-                    Z_pred = forecaster.predict(fh=fh_ins)
-                    Z[col] = Z[col].fillna(value=Z_pred)
-            # univariate
-            else:
-                forecaster.fit(y=Z)
-                Z_pred = forecaster.predict(fh=fh_ins)
-                Z = Z.fillna(value=Z_pred)
+        elif self.method == "drift":
+            forecaster = PolynomialTrendForecaster(degree=1)
+            Z = _impute_with_forecaster(forecaster, Z)
+        elif self.method == "forecaster":
+            forecaster = clone(self.forecaster)
+            Z = _impute_with_forecaster(forecaster, Z)
         elif self.method == "mean":
             Z = Z.fillna(value=Z.mean())
         elif self.method == "median":
@@ -133,7 +125,7 @@ def transform(self, Z, X=None):
         elif self.method in ["nearest", "linear"]:
             Z = Z.interpolate(method=self.method)
         else:
-            raise ValueError(f"method {self.method} not available")
+            raise ValueError(f"`method`: {self.method} not available.")
         # fill first/last elements of series,
         # as some methods (e.g. "linear") cant impute those
         Z = Z.fillna(method="ffill").fillna(method="backfill")
@@ -164,7 +156,7 @@ def _check_method(self):
             pass
 
     def _get_random(self, Z):
-        """Create a random int or float value
+        """Create a random int or float value.
 
         :param Z: Series
         :type Z: pd.Series
@@ -177,3 +169,40 @@ def _get_random(self, Z):
             return rng.randint(Z.min(), Z.max())
         else:
             return rng.uniform(Z.min(), Z.max())
+
+
+def _impute_with_forecaster(forecaster, Z):
+    """Use a given forecaster for imputation by in-sample predictions.
+
+    Parameters
+    ----------
+    forecaster: Forecaster
+        Forecaster to use for imputation
+    Z : pd.Series or pd.DataFrame
+        Series to impute.
+
+    Returns
+    -------
+    zt : pd.Series or pd.DataFrame
+        Series with imputed values.
+    """
+    if isinstance(Z, pd.Series):
+        series = [Z]
+    elif isinstance(Z, pd.DataFrame):
+        series = [Z[column] for column in Z]
+
+    for z in series:
+        # define fh based on index of missing values
+        na_index = z.index[z.isna()]
+        fh = ForecastingHorizon(values=na_index, is_relative=False)
+
+        # fill NaN before fitting with ffill and backfill (heuristic)
+        forecaster.fit(y=z.fillna(method="ffill").fillna(method="backfill"), fh=fh)
+
+        # replace missing values with predicted values
+        z[na_index] = forecaster.predict()
+    return Z
+
+
+def _has_missing_values(Z):
+    return Z.isnull().to_numpy().any()


Test patch: