Repository: sktime/sktime

Issue:
[ENH] predictive intervals and quantiles for naive forecaster
- [ ] Describe naive predictive intervals (what is the method?)
- [ ] implement for `NaiveForecaster`

Related to the variance forecaster of @IlyasMoutawwakil


Discussion:
Question, what should this be?

I see a number of options:
* detrend in-sample, compute the variance of in-sample residuals. Use this as a predictive variance with normal assumption, get intervals/quantiles from that.
* use rolling residuals, like in @IlyasMoutawwakil´s variance forecaster, then do the same
* use any of the residuals and get empirical quantiles, use the empirical cdf of residuals as a probabilistic prediction

... any other options?
(probably there´s also a few different ways this could look like if we try to write down in formalism)
The naive methods' forecast variance can be expressed analytically under the 2 assumptions below:
- forecast values are normally distributed
- residuals are uncorrelated

https://otexts.com/fpp3/prediction-intervals.html#benchmark-methods
yes, sure, but *which residuals* would be part of the question above?
Can't we use the expression used in #775 and adapt it to the current interface? 
hm, maybe? Looking through it I'm not sure which method/algorithm this proposes to implement.
There's not just one one could use, similar to how there are multiple naive strategies for the point prediction.

Do you have a pointer to the method or algorithm description? Maybe I'm just not seeing it among all the discussion.

Resolving patch:
diff --git a/sktime/forecasting/naive.py b/sktime/forecasting/naive.py
--- a/sktime/forecasting/naive.py
+++ b/sktime/forecasting/naive.py
@@ -48,6 +48,16 @@ class NaiveForecaster(_BaseWindowForecaster):
       - "mean": np.nanmean over rows
     - tile the predictions using the seasonal periodicity
 
+    To compute prediction quantiles, we first estimate the standard error
+    of prediction residuals under the assumption of uncorrelated residuals.
+    The forecast variance is then computed by multiplying the residual
+    variance by a constant. This constant is a small-sample bias adjustment
+    and each method (mean, last, drift) have different formulas for computing
+    the constant. These formulas can be found in the Forecasting:
+    Principles and Practice textbook (Table 5.2) [1]_. Lastly, under the assumption that
+    residuals follow a normal distribution, we use the forecast variance and
+    z-scores of a normal distribution to estimate the prediction quantiles.
+
     Parameters
     ----------
     strategy : {"last", "mean", "drift"}, default="last"
@@ -78,6 +88,12 @@ class NaiveForecaster(_BaseWindowForecaster):
         Window length to use in the `mean` strategy. If None, entire training
             series will be used.
 
+    References
+    ----------
+    .. [1] Hyndman, R.J., & Athanasopoulos, G. (2021) Forecasting:
+        principles and practice, 3rd edition, OTexts: Melbourne, Australia.
+        OTexts.com/fpp3. Accessed on 22 September 2022.
+
     Examples
     --------
     >>> from sktime.datasets import load_airline
@@ -94,6 +110,8 @@ class NaiveForecaster(_BaseWindowForecaster):
         "requires-fh-in-fit": False,
         "handles-missing-data": True,
         "scitype:y": "univariate",
+        "capability:pred_var": True,
+        "capability:pred_int": True,
     }
 
     def __init__(self, strategy="last", window_length=None, sp=1):
@@ -330,6 +348,134 @@ def _predict(self, fh=None, X=None):
 
         return y_pred
 
+    def _predict_quantiles(self, fh, X=None, alpha=0.5):
+        """Compute/return prediction quantiles for a forecast.
+
+        Uses normal distribution as predictive distribution to compute the
+        quantiles.
+
+        Parameters
+        ----------
+        fh : int, list, np.array or ForecastingHorizon
+            Forecasting horizon
+        X : pd.DataFrame, optional (default=None)
+            Exogenous variables are ignored.
+        alpha : float or list of float, optional (default=0.5)
+            A probability or list of, at which quantile forecasts are computed.
+
+        Returns
+        -------
+        pred_quantiles : pd.DataFrame
+            Column has multi-index: first level is variable name from y in fit,
+                second level being the quantile forecasts for each alpha.
+                Quantile forecasts are calculated for each a in alpha.
+            Row index is fh. Entries are quantile forecasts, for var in col index,
+                at quantile probability in second-level col index, for each row index.
+        """
+        y_pred = self.predict(fh)
+        y_pred = convert(y_pred, from_type=self._y_mtype_last_seen, to_type="pd.Series")
+
+        pred_var = self.predict_var(fh)
+        z_scores = norm.ppf(alpha)
+
+        errors = (
+            np.sqrt(pred_var.to_numpy().reshape(len(pred_var), 1)) * z_scores
+        ).reshape(len(y_pred), len(alpha))
+
+        pred_quantiles = pd.DataFrame(
+            errors + y_pred.values.reshape(len(y_pred), 1),
+            columns=pd.MultiIndex.from_product([["Quantiles"], alpha]),
+            index=fh.to_absolute(self.cutoff).to_pandas(),
+        )
+
+        return pred_quantiles
+
+    def _predict_var(self, fh, X=None, cov=False):
+        """Compute/return prediction variance for naive forecasts.
+
+        Variance are computed according to formulas from (Table 5.2)
+        Forecasting: Principles and Practice textbook [1]_.
+
+        Must be run *after* the forecaster has been fitted.
+
+        Parameters
+        ----------
+        fh : int, list, np.array or ForecastingHorizon
+            Forecasting horizon
+        X : pd.DataFrame, optional (default=None)
+            Exogenous variables are ignored.
+        cov : bool, optional (default=False)
+            If True, return the covariance matrix.
+            If False, return the marginal variance.
+
+        Returns
+        -------
+        pred_var :
+            if cov=False, pd.DataFrame with index fh.
+                a vector of same length as fh with predictive marginal variances;
+            if cov=True, pd.DataFrame with index fh and columns fh.
+                a square matrix of size len(fh) with predictive covariance matrix.
+
+        References
+        ----------
+        .. [1] https://otexts.com/fpp3/prediction-intervals.html#benchmark-methods
+        """
+        y = self._y
+        y = convert_to(y, "pd.Series")
+        T = len(y)
+
+        # Compute "past" residuals
+        if self.strategy == "last":
+            y_res = y - y.shift(self.sp)
+        elif self.strategy == "mean":
+            # Since this strategy returns a constant, just predict fh=1 and
+            # transform the constant into a repeated array
+            y_pred = np.repeat(np.squeeze(self.predict(fh=1)), T)
+            y_res = y - y_pred
+        else:
+            # Slope equation from:
+            # https://otexts.com/fpp3/simple-methods.html#drift-method
+            slope = (y.iloc[-1] - y.iloc[0]) / (T - 1)
+
+            # Fitted value = previous value + slope
+            # https://github.com/robjhyndman/forecast/blob/master/R/naive.R#L34
+            y_res = y - (y.shift(self.sp) + slope)
+
+        # Residuals MSE and SE
+        # + 1 degrees of freedom to estimate drift coefficient standard error
+        # https://github.com/robjhyndman/forecast/blob/master/R/naive.R#L79
+        n_nans = np.sum(pd.isna(y_res))
+        mse_res = np.sum(np.square(y_res)) / (T - n_nans - (self.strategy == "drift"))
+        se_res = np.sqrt(mse_res)
+
+        sp = self.sp
+        window_length = self.window_length or T
+        # Formulas from:
+        # https://otexts.com/fpp3/prediction-intervals.html (Table 5.2)
+        partial_se_formulas = {
+            "last": lambda h: np.sqrt(h)
+            if sp == 1
+            else np.sqrt(np.floor((h - 1) / sp) + 1),
+            "mean": lambda h: np.repeat(np.sqrt(1 + (1 / window_length)), len(h)),
+            "drift": lambda h: np.sqrt(h * (1 + (h / (T - 1)))),
+        }
+
+        fh_periods = np.array(fh.to_relative(self.cutoff))
+        marginal_se = se_res * partial_se_formulas[self.strategy](fh_periods)
+        marginal_vars = marginal_se**2
+
+        fh_idx = fh.to_absolute(self.cutoff).to_pandas()
+        if cov:
+            fh_size = len(fh)
+            cov_matrix = np.fill_diagonal(
+                np.zeros(shape=(fh_size, fh_size)), marginal_vars
+            )
+            pred_var = pd.DataFrame(cov_matrix, columns=fh_idx, index=fh_idx)
+        else:
+            pred_var = pd.DataFrame(marginal_vars, index=fh_idx)
+
+        return pred_var
+
     @classmethod
     def get_test_params(cls, parameter_set="default"):
         """Return testing parameter settings for the estimator.


Test patch:
diff --git a/sktime/forecasting/tests/test_naive.py b/sktime/forecasting/tests/test_naive.py
--- a/sktime/forecasting/tests/test_naive.py
+++ b/sktime/forecasting/tests/test_naive.py
@@ -257,3 +257,118 @@ def test_strategy_mean_and_last_seasonal_additional_combinations(
         for i in range(1 + len(test_data) // sp):
             test_data[i * sp : i * sp + sp - window_length] = np.nan
         pd.testing.assert_series_equal(forecast_data, test_data)
+
+
+@pytest.mark.parametrize(
+    "strategy,sp,window_length",
+    [
+        ("last", 1, None),
+        ("last", 24, None),
+        ("mean", 1, None),
+        ("mean", 24, None),
+        ("mean", 1, 24),
+        ("mean", 24, 24),
+        ("drift", 1, None),
+    ],
+)
+def test_naive_predict_var_backwards(strategy, sp, window_length):
+    """Checks naive prediction variance computations.
+
+    Test whether, given h=1 and large T, the forecast standard error
+    is approximately equal to the residual standard errors.
+    This property is noted in the Forecasting: Principles and
+    Practice textbook (FPP3) [1]_.
+
+    More specifically, predict_var computes the forecast standard errors
+    (and hence variance) from the residuals standard errors times
+    some constant. According to FPP3, this operation can be fully inverted.
+    Hence, for this unit test, we redo our computations backwards and
+    check that our results are approximately equal.
+
+    References
+    ----------
+    .. [1] https://otexts.com/fpp3/prediction-intervals.html#benchmark-methods
+    """
+    n_timepoints = 100000
+    mu, sigma = 0.0, 10.0
+    fake_idx = pd.date_range("1980", periods=n_timepoints + 1, freq="H")
+    np.random.seed(42)
+    y = pd.Series(np.random.normal(mu, sigma, size=n_timepoints), index=fake_idx[:-1])
+
+    h = 1
+    forecaster = NaiveForecaster(strategy, sp=sp, window_length=window_length)
+    sigma2 = forecaster.fit(y).predict_var(fh=h)
+    sigma = np.sqrt(sigma2).iloc[0, 0]
+
+    T = len(y.dropna())
+    if strategy == "last":
+        # This is trival because square root of (h) when h=1 is just 1
+        sigma_res = sigma / np.sqrt(h)
+    elif strategy == "mean":
+        sigma_res = sigma / np.sqrt(1 + (1 / T))
+    else:
+        sigma_res = sigma / np.sqrt(1 + (1 / (T - 1)))
+
+    upper_bound, lower_bound = (sigma + 0.001), (sigma - 0.001)
+    assert lower_bound < sigma_res < upper_bound
+
+
+@pytest.mark.parametrize(
+    "strategy,sp,lower,upper",
+    [
+        (
+            "last",
+            1,
+            [-22.07661, -31.71836, -39.11673, -45.35385, -50.84886],
+            [24.47787, 34.11962, 41.51799, 47.75511, 53.25012],
+        ),
+        (
+            "last",
+            24,
+            [-45.45926, -18.718647, -36.369212, -12.929395, -13.960794],
+            [1.025702, 27.766313, 10.115747, 33.555565, 32.524165],
+        ),
+        ("mean", 1, -16.45385, 16.47319),
+        (
+            "drift",
+            1,
+            [-22.07676, -31.71876, -39.11745, -45.35493, -50.85035],
+            [24.47795, 34.11987, 41.51848, 47.75589, 53.25123],
+        ),
+    ],
+)
+def test_naive_predict_interval_against_R_naive(strategy, sp, lower, upper):
+    """Checks naive prediction interval computations.
+
+    Compare prediction interval results with R implementation in [1]_.
+
+    Note:
+    - Seasonality is not applicable to "mean" strategy in R.
+    - Argument "window_length" is not available in R.
+
+    References
+    ----------
+    .. [1] https://github.com/robjhyndman/forecast/blob/master/R/naive.R
+    """
+    n_timepoints = 100000
+    mu, sigma = 0.0, 10.0
+    fake_idx = pd.date_range("1980", periods=n_timepoints + 1, freq="H")
+    np.random.seed(42)
+    y = pd.Series(np.random.normal(mu, sigma, size=n_timepoints), index=fake_idx[:-1])
+
+    h = list(range(1, 6))
+    coverage = 0.90
+    forecaster = NaiveForecaster(strategy, sp=sp)
+    y_pred_ints = forecaster.fit(y).predict_interval(fh=h, coverage=coverage)
+
+    expected = pd.DataFrame(
+        columns=pd.MultiIndex.from_product(
+            [["Coverage"], [coverage], ["lower", "upper"]]
+        ),
+        index=y_pred_ints.index,
+    )
+
+    expected[("Coverage", coverage, "lower")] = lower
+    expected[("Coverage", coverage, "upper")] = upper
+
+    pd.testing.assert_frame_equal(y_pred_ints, expected)