Repository: statsmodels/statsmodels

Issue:
Add another nan-handling method in acf()
#### Is your feature request related to a problem? Please describe

#6641/#6726 revealed that `acf(..., missing="conservative")` did not behave as I would have hoped, and I would thus like to request a new nan-handling policy.  Specifically, it would seem reasonable to have a policy where the n-th lag acf of `t` is computed as something like `np.nanmean(t[:n] * t[-n:])` (with the correct centering and rescaling), i.e. dropping the nan terms in the sum (which "conservative" does, by setting them to zero) but also in the denominator when computing the mean.  The final rescaling should also be done by `np.nanmean(t**2)`, i.e. here too nans should be dropped from the denominator.

#### Describe the solution you'd like
Add a `missing="???"` policy to `acf()` which implements the approach above.  Name up to bikeshedding...

#### Describe alternatives you have considered
None that I can think of.

#### Additional context
N/A


Discussion:
I think this is already done. 

If `x` is the original series, an x0 is the series with NaN replaced by 0, then the mean is estimated as 

```
not_missing = ~np.isnan(x)
mean = x0.sum() / not_missing.sum()
x0[not_mising] -= mean
```

Explainer could probably be improved. 

The acov is then computed from the nan-demeaned x series with 0's in all nan places. 
I haven't checked the math of the implementation, but at least from a documentation point of view, "replaces missing with zero" (the current explanation) is not the same as "dropping nan terms *both* from the numerator and the denominator" I think(?)

Resolving patch:
diff --git a/statsmodels/tsa/stattools.py b/statsmodels/tsa/stattools.py
--- a/statsmodels/tsa/stattools.py
+++ b/statsmodels/tsa/stattools.py
@@ -341,8 +341,11 @@ def acovf(x, unbiased=False, demean=True, fft=None, missing="none", nlag=None):
         the NaNs are to be treated. "none" performs no checks. "raise" raises
         an exception if NaN values are found. "drop" removes the missing
         observations and then estimates the autocovariances treating the
-        non-missing as contiguous. "conservative" replaces missing with
-        zeros.
+        non-missing as contiguous. "conservative" computes the autocovarince
+        using nan-ops so that nans are removed when computing the mean
+        and cross-products that are used to estimate the autocovariance.
+        When using "conservative", n is set to the number of non-missing
+        observations.
     nlag : {int, None}, default None
         Limit the number of autocovariances returned.  Size of returned
         array is nlag + 1.  Setting nlag when fft is False uses a simple,
@@ -527,8 +530,11 @@ def acf(x, unbiased=False, nlags=None, qstat=False, fft=None, alpha=None,
         the NaNs are to be treated. "none" performs no checks. "raise" raises
         an exception if NaN values are found. "drop" removes the missing
         observations and then estimates the autocovariances treating the
-        non-missing as contiguous. "conservative" replaces missing with
-        zeros.
+        non-missing as contiguous. "conservative" computes the autocovarince
+        using nan-ops so that nans are removed when computing the mean
+        and cross-products that are used to estimate the autocovariance.
+        When using "conservative", n is set to the number of non-missing
+        observations.
 
     Returns
     -------


Test patch:
diff --git a/statsmodels/tsa/tests/test_stattools.py b/statsmodels/tsa/tests/test_stattools.py
--- a/statsmodels/tsa/tests/test_stattools.py
+++ b/statsmodels/tsa/tests/test_stattools.py
@@ -1032,3 +1032,19 @@ def test_rand10000_case(self):
         res = zivot_andrews(mdl, regression='c', autolag='t-stat')
         assert_allclose([res[0], res[1], res[3], res[4]],
                         [-3.48223, 0.69111, 25, 7071], rtol=1e-3)
+
+
+def test_acf_conservate_nanops(reset_randomstate):
+    # GH 6729
+    e = np.random.standard_normal(100)
+    for i in range(1, e.shape[0]):
+        e[i] += 0.9 * e[i-1]
+    e[::7] = np.nan
+    result = acf(e, missing="conservative", nlags=10)
+    resid = e - np.nanmean(e)
+    expected = np.ones(11)
+    nobs = e.shape[0]
+    gamma0 = np.nansum(resid * resid)
+    for i in range(1, 10+1):
+        expected[i] = np.nansum(resid[i:] * resid[:nobs-i]) / gamma0
+    assert_allclose(result, expected, rtol=1e-4, atol=1e-4)