Repository: sktime/sktime

Issue:
ARIMA and AutoARIMA should return pandas.NA when value cannot be predicted, e.g., first index in-sample for start=0, d=1
Update 2021-09-15 - re-opened since a recurring issue.

Not a bug, but an interface convention - currently ARIMA, AutoARIMA and other learners fail when asked for a prediction that is undefined, e.g., difference parameter d=1 and first in-sample index (start=0).

It would be more sensible if `pandas.NA` is returned instead for such indices, perhaps a warning should be raised too that `pandas.NA` are produced.

In first instance, we should update ARIMA and AutoARIMA with this convention.

---

**Describe the bug**

When using `Imputer` with `forecaster = AutoARIMA()`  the error "ValueError: In-sample predictions undefined for start=0 when d=1" is raised.

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-11-2d05f4822fd6> in <module>()
      4 
      5 transformer = Imputer(method="forecaster", forecaster=AutoARIMA()).fit(Z=df.value)
----> 6 y_hat = transformer.transform(Z=df.value)

/usr/local/lib/python3.7/dist-packages/sktime/transformations/series/impute.py in transform(self, Z, X)
    102             z_pred = forecaster.fit(
    103                 z.fillna(method="ffill").fillna(method="backfill")
--> 104             ).predict(fh=fh_ins)
    105             # fill with trend values
    106             z = z.fillna(value=z_pred)

/usr/local/lib/python3.7/dist-packages/sktime/forecasting/base/_sktime.py in predict(self, fh, X, return_pred_int, alpha)
    226         self.check_is_fitted()
    227         self._set_fh(fh)
--> 228         return self._predict(self.fh, X, return_pred_int=return_pred_int, alpha=alpha)
    229 
    230     def compute_pred_int(self, y_pred, alpha=DEFAULT_ALPHA):

/usr/local/lib/python3.7/dist-packages/sktime/forecasting/base/adapters/_pmdarima.py in _predict(self, fh, X, return_pred_int, alpha)
     58         # all values are in-sample
     59         elif fh.is_all_in_sample(self.cutoff):
---> 60             return self._predict_in_sample(fh_ins, **kwargs)
     61 
     62         # both in-sample and out-of-sample values

/usr/local/lib/python3.7/dist-packages/sktime/forecasting/base/adapters/_pmdarima.py in _predict_in_sample(self, fh, X, return_pred_int, alpha)
     80             X=X,
     81             return_conf_int=return_pred_int,
---> 82             alpha=alpha,
     83         )
     84 

/usr/local/lib/python3.7/dist-packages/pmdarima/utils/metaestimators.py in <lambda>(*args, **kwargs)
     51 
     52         # lambda, but not partial, allows help() to work with update_wrapper
---> 53         out = (lambda *args, **kwargs: self.fn(obj, *args, **kwargs))
     54         # update the docstring of the returned function
     55         update_wrapper(out, self.fn)

/usr/local/lib/python3.7/dist-packages/pmdarima/arima/auto.py in predict_in_sample(self, X, start, end, dynamic, return_conf_int, alpha, typ, **kwargs)
    242             return_conf_int=return_conf_int,
    243             alpha=alpha,
--> 244             typ=typ,
    245         )
    246 

/usr/local/lib/python3.7/dist-packages/pmdarima/arima/arima.py in predict_in_sample(self, X, start, end, dynamic, return_conf_int, alpha, **kwargs)
    584         if start is not None and start < d:
    585             raise ValueError("In-sample predictions undefined for start={0} "
--> 586                              "when d={1}".format(start, d))
    587 
    588         # if we fit with exog, make sure one was passed:

ValueError: In-sample predictions undefined for start=0 when d=1
```

**To Reproduce**

Data:
[df.csv](https://github.com/alan-turing-institute/sktime/files/6426660/df.csv) 

```python
import pandas as pd
from sktime.transformations.series.impute import Imputer
from sktime.forecasting.arima import AutoARIMA

df = pd.read_csv('df.csv')  
df['timestamp'] = pd.DatetimeIndex(df['timestamp']).to_period('D')
df = df.set_index('timestamp')

transformer = Imputer(method="forecaster", forecaster=AutoARIMA()).fit(Z=df.value)
y_hat = transformer.transform(Z=df.value)
```

**Expected behavior**

`transformer.transform(Z)` should return `Z` with NAn values replaced by AutoArima predictions.

**Additional context**

I think the problem is in `/usr/local/lib/python3.7/dist-packages/sktime/transformations/series/impute.py` in `transform(self, Z, X)`, line 100, because all the points are being passed to `fh_ins` instead of just the points with NaN.

```python
100             fh_ins = -np.arange(len(z))
101             # fill NaN before fitting with ffill and backfill (heuristic)
102             z_pred = forecaster.fit(
103                 z.fillna(method="ffill").fillna(method="backfill")
104             ).predict(fh=fh_ins)
105             # fill with trend values
106             z = z.fillna(value=z_pred)

```

**Versions**
<details>

```
System:
    python: 3.7.10 (default, Feb 20 2021, 21:17:23)  [GCC 7.5.0]
executable: /usr/bin/python3
   machine: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic

Python dependencies:
          pip: 19.3.1
   setuptools: 56.0.0
      sklearn: 0.24.2
       sktime: 0.6.0
  statsmodels: 0.12.2
        numpy: 1.19.5
        scipy: 1.4.1
       Cython: 0.29.22
       pandas: 1.1.5
   matplotlib: 3.2.2
       joblib: 1.0.1
        numba: 0.51.2
     pmdarima: 1.8.2
      tsfresh: None
```

</details>



Discussion:
Hi @julramos - thanks for raising the issue! 

I think you have correctly identified the problem here. Would you like to open a PR and see if that fixes the issue? 

An alternative quick fix would be to set `d=0` if this is acceptable:
```python
import pandas as pd
import numpy as np
from sktime.transformations.series.impute import Imputer
from sktime.forecasting.arima import AutoARIMA
from sktime.datasets import load_airline

y = load_airline()
y.iloc[np.random.randint(0, len(y), 20)] = np.nan  # add missing values
transformer = Imputer(method="forecaster", forecaster=AutoARIMA(d=0, suppress_warnings=True))
y_hat = transformer.fit_transform(y)
```
Reopening with @ilyasmoutawwakil's suggestion that forecasts at indices which the forecaster cannot predict should be NA.

Since we are returning pandas objects, it should be `pandas.NA`, and the column/entry of a nullable `pandas` `dtype`.
I can make a PR for this if no one did yet ^^
no one has, all yours! 😄

Resolving patch:
diff --git a/sktime/forecasting/base/adapters/_pmdarima.py b/sktime/forecasting/base/adapters/_pmdarima.py
--- a/sktime/forecasting/base/adapters/_pmdarima.py
+++ b/sktime/forecasting/base/adapters/_pmdarima.py
@@ -7,7 +7,6 @@
 __all__ = ["_PmdArimaAdapter"]
 
 import pandas as pd
-
 from sktime.forecasting.base._base import DEFAULT_ALPHA
 from sktime.forecasting.base import BaseForecaster
 
@@ -66,9 +65,18 @@ def _predict(self, fh, X=None, return_pred_int=False, alpha=DEFAULT_ALPHA):
 
         # both in-sample and out-of-sample values
         else:
-            y_ins = self._predict_in_sample(fh_ins, **kwargs)
-            y_oos = self._predict_fixed_cutoff(fh_oos, **kwargs)
-            return y_ins.append(y_oos)
+            if return_pred_int:
+                y_ins_pred, y_ins_pred_int = self._predict_in_sample(fh_ins, **kwargs)
+                y_oos_pred, y_oos_pred_int = self._predict_fixed_cutoff(
+                    fh_oos, **kwargs
+                )
+                return y_ins_pred.append(y_oos_pred), y_ins_pred_int.append(
+                    y_oos_pred_int
+                )
+            else:
+                y_ins = self._predict_in_sample(fh_ins, **kwargs)
+                y_oos = self._predict_fixed_cutoff(fh_oos, **kwargs)
+                return y_ins.append(y_oos)
 
     def _predict_in_sample(
         self, fh, X=None, return_pred_int=False, alpha=DEFAULT_ALPHA
@@ -76,9 +84,34 @@ def _predict_in_sample(
         if isinstance(alpha, (list, tuple)):
             raise NotImplementedError("multiple `alpha` values are not yet supported")
 
-        # for in-sample predictions, pmdarima requires zero-based
-        # integer indicies
+        if hasattr(self, "order"):
+            diff_order = self.order[1]
+        else:
+            diff_order = self._forecaster.model_.order[1]
+
+        # Initialize return objects
+        fh_abs = fh.to_absolute(self.cutoff).to_numpy()
+        fh_idx = fh.to_indexer(self.cutoff, from_cutoff=False)
+        y_pred = pd.Series(index=fh_abs)
+        pred_int = pd.DataFrame(index=fh_abs, columns=["lower", "upper"])
+
+        # for in-sample predictions, pmdarima requires zero-based integer indicies
         start, end = fh.to_absolute_int(self._y.index[0], self.cutoff)[[0, -1]]
+        if start < 0:
+            # Can't forecasts earlier to train starting point
+            raise ValueError("Can't make predictions earlier to train starting point")
+        elif start < diff_order:
+            # Can't forecasts earlier to arima's differencing order
+            # But we return NaN for these supposedly forecastable points
+            start = diff_order
+            if end < start:
+                # since we might have forced `start` to surpass `end`
+                end = diff_order
+            # get rid of unforcastable points
+            fh_abs = fh_abs[fh_idx >= diff_order]
+            # reindex accordingly
+            fh_idx = fh_idx[fh_idx >= diff_order] - diff_order
+
         result = self._forecaster.predict_in_sample(
             start=start,
             end=end,
@@ -87,19 +120,16 @@ def _predict_in_sample(
             alpha=alpha,
         )
 
-        fh_abs = fh.to_absolute(self.cutoff)
-        fh_idx = fh.to_indexer(self.cutoff, from_cutoff=False)
         if return_pred_int:
-            # unpack and format results
-            y_pred, pred_int = result
-            y_pred = pd.Series(y_pred[fh_idx], index=fh_abs)
-            pred_int = pd.DataFrame(
-                pred_int[fh_idx, :], index=fh_abs, columns=["lower", "upper"]
+            # unpack results
+            y_pred.loc[fh_abs], pred_int.loc[fh_abs] = (
+                result[0][fh_idx],
+                result[1][fh_idx, :],
             )
             return y_pred, pred_int
-
         else:
-            return pd.Series(result[fh_idx], index=fh_abs)
+            y_pred.loc[fh_abs] = result[fh_idx]
+            return y_pred
 
     def _predict_fixed_cutoff(
         self, fh, X=None, return_pred_int=False, alpha=DEFAULT_ALPHA


Test patch: