Repository: statsmodels/statsmodels

Issue:
add support for NaNs in acovf, acf, pacf
Add some NaN-handling in statsmodels.tsa.stattools
Followed syntax in statsmodels.tsa.ar_model.AR
With reference to this PR: https://github.com/statsmodels/statsmodels/issues/322
And to this discussion: https://groups.google.com/forum/#!msg/pystatsmodels/Nr8E-dPQSsA/vWeAFT5F0_AJ
I had to comment out the import of Bunch in line 9, because it kept giving me an ImportError



Discussion:
IIRC, we had a longer discussion about what exactly to do here in these cases. 

I'd also prefer if we re-used the missing data handler we already have. If it's not possible to re-use it, then we should refactor so we're not duplicating this kind of code.

https://github.com/statsmodels/statsmodels/blob/master/statsmodels/base/data.py#L110

> This will shift the array at every missing value and the lags will not line up anymore.

I'm adding a qualifier to this, dropping might still make sense in some cases

It depends on what the missing values are:

If the time is "real" time, then we cannot shift the time series because it would align all following lags in the wrong way. Pierre who wrote the code for scikits.timeseries what working with wheather/water data in calender time.

If the time is "event" time, then missing values might just mean that no event occured, lags would refer to the last event times. For example if the missing date is a business holiday, then we could think of lags in terms of actual business days.
For consumer data, the events might be shopping trips. lag 1 refers to the last shopping trip, not to an actual calender time. If someone usually shops on Saturday, but skips one Saturday, then the previous lag would refer to the shopping trip two weeks earlier.

`drop` makes sense in the second case, but would be wrong in the first case.

The code for case 1 for acf acovf is not difficult, and I had the function at some point in statsmodels (copied with adjustments from scikits.timeseries)

pacf_ols might be a bit more difficult  (it's not obvious to me how to do it), 
pacf with yule walker just needs the acf with the missing value handling, I guess

I don't find anything in the sandbox, we need to look for the scikits.timeseries source.

@joesnacks   Thanks for looking into this
From what I remember, for the biased version we can just set the missing values to zero. If they are zero, then they don't add to the sum in the autocovariance calculation.
For the denominator if unbiased version, we had something like `np.correlate(mask, mask, 'full')` where `mask = ~np.isnan(maks)`    This counts how many observations enter in each calculation.

It's quite a bit more expensive to calculate, than the no missing case and should have a separate code path. 
One issue: if we overwrite the missing values with zeros, then we have to put the missing values (nans) back in the array or make a copy of the array, so we don't change the data of the user.

@josef-pkt  I made changes just to acovf() this time based on the source code for scikits.timeseries. The test cases seem to give sensible results, but I can't run the scikits.timeseries code to benchmark because I have trouble installing it.

If you're happy with acovf(), I can extend to acf and pacf. 

Seems to me like the user should be able to set the behaviour, like R's na.action=... option.

I agree that users should be able to choose the na action, and to choose not to do any expensive checks.
I might not have been clear before, I just wanted to point out the main problem, (given my short attention span these days).

Reading through the latest change, I think this looks good,
plus adding back in any missing options that we want. 

the changes in acf could just reuse acovf. I think we left some code duplication in there that hasn't been cleaned up.

Thinking about a test case:

If there are only nans at the beginning or at the end of the array, then the results should be the same as for the shortened "drop" array, I guess.

I think that works for the unbiased version, but for the biased version the total number of observations, i.e. the denominator, is different. 
Maybe we should divide by the number of `valid`, non-nan observations in the biased case. We should check what other packages are using.

acovf now works with missing and non-missing data, and provides the 'drop' and 'raise' options. The behaviour from scikits.timeseries is available with missing='conservative'. 

Results from setting 'drop' when the first element of x contains a NaN are the same as results from acovf(x[1:]). Results are also consistent with statsmodels 0.5.0: 

## statsmodels 0.5.0

> > > lags = 3
> > > acovf(x[1:])[:lags]
> > > array([ 10234713.38654895,  10100549.30423607,   9964801.78406805])
> > > acovf(x[1:], unbiased=True)[:lags]
> > > array([ 10234713.38654895,  10150800.79331187,  10064449.80190873])
> > > acovf(x[1:], unbiased=True, demean=False)[:lags]
> > > array([ 62703044.8662048 ,  62572927.49774744,  62443247.12724449])

## this PR:

acovf tests, ignore missing
[ 10234713.38654895  10100549.30423607   9964801.78406805]
[ 10234713.38654895  10150800.79331187  10064449.80190873]
[ nan  nan  nan]
[ nan  nan  nan]
[ nan  nan  nan]
[ nan  nan  nan]
acovf tests with missing, biased:
[ 10442760.76631795  10210103.6372762   10075143.03949556]
[ 10234713.38654895  10100549.30423607   9964801.78406805]
acovf tests with missing, unbiased:
[ 10494457.60179477  10311696.70829387  10226270.185088  ]
[ 10234713.38654895  10150800.79331187  10064449.80190873]
acovf tests with missing, unbiased nodemean:
[ 62703044.8662048   62572927.49774744  62443247.12724449]
[ 62703044.8662048   62572927.49774744  62443247.12724449]
acovf test with missing, unbiased nodemean fft:
[ 62703044.86620478  62572927.49774743  62443247.12724449]
[ 62703044.86620475  62572927.49774739  62443247.12724451]

@jseabold: I agree about not duplicating the existing missing data handler. It's currently implemented as a method of the ModelData class. One could imagine having parent and sister classes of ModelData, and passing the handle_missing() method by inheritance, but I don't see how to do this, or how then to re-use this code in acovf().

In the 'drop' case, one only needs to pre-process the data, and then feed in the pre-processed data into the usual acovf() function. However, for the more conservative method in scikits.timeseries.lib.avf(), simply pre-processing the data is not possible, because the code uses np.correlate(x, x, 'full'), where x is a (demeaned) 1d array, not a matrix of x and its lags. So the only way I could imagine re-using ModelData.handle_missing() in acovf() is if we replace the fast and efficient np.correlate(..., 'full') method taken in scikits.timeseries.lib.avf() with something based on a matrix of x and its lags.

acf() now supports NaNs and relies on acovf(), rather than duplicating the fast fourier transform from acovf().

Test cases agree with statsmodels 0.5.0.

## this commit:

acf tests:
[ nan  nan  nan]
[ 1.          0.97772073  0.96479688]
[ 1.          0.98689127  0.97362783]
[ 1.          0.98689127  0.97362783]

## statsmodels 0.5.0:

> > > x = sm.datasets.macrodata.load().data['realgdp']
> > > acf(x[1:])[:3]
> > > array([ 1.        ,  0.98689127,  0.97362783])

[![Coverage Status](https://coveralls.io/builds/610712/badge)](https://coveralls.io/builds/610712)

Coverage remained the same when pulling **ac742ce3b5fb18fe9d22836d68a9a6a280ba745f on joesnacks:master** into **e1be28edfa7e9a41377dc5e4b5253e558b68658d on statsmodels:master**.

A question or issue while reading through this: Was this supposed to work for a 2d array?  
"drop" will make it more difficult to vectorize. I don't think the drop option can be vectorized.

adding 'missing' cases to the unit tests is still missing

The PACF is not well defined in the `conservative` case since the corresponding correlation matrix may not actually be a correlation matrix - it might not be positive definite.  

This leaves two choices - either refuse to compute, which is probably reasonable, or to check for eigenvalues > 0 and only compute if this is satisfied.  More general estimators for the PACF with missing data are possible, but only using fundamentally different techniques that can explicitly account for missing data.

> "drop" will make it more difficult to vectorize. I don't think the drop option can be vectorized.

Drop would have to be a drop-any for this to work for 2D.

@joesnacks can you add unittests to `\statsmodels\tsa\tests\test_stattools.py`

The problem that Kevin pointed out for the mean calculation should have been caught by unit test, I think.

@bashtage 

> The PACF is not well defined in the conservative case since the corresponding correlation matrix may not actually be a correlation matrix - it might not be positive definite.

I didn't think about it. This is essentially the same problem as in simple correlation with pairwise deletion of missing values, as for example in pandas or R. From the pandas discussion, it sounded like we want to separate the covariance, correlation calculation from "fixing" the covariance or correlation matrix.  We have some correlation_tools to find the nearest positive definite or positive semi-definite matrix.

How does a non-positive definite autocorrelation function look like?  
My guess is that the only case is if one of the covariances is larger than the variance, and autocorrelation coefficient > 1. Can or does this actually happen?

For this PR, I would just add a comment to the docstring, but if you have references to a better way of doing this, we could add that.

It gets sort of arbitrary when finding the closest positive semi definite matrix - plus it really needs to be positive definite, and so then there is another tuning parameter.  

I don't have any particular reference in mind, but I would imagine the correct method would be to estimate things in the frequency domain and then convert back to the time domain, or perhaps some form of EM algorithm if one wanted to stay in the time domain.

http://stat.ethz.ch/R-manual/R-patched/library/stats/html/acf.html :

```
By default, no missing values are allowed. If the na.action function passes through missing values (as na.pass does), the covariances are computed from the complete cases. This means that the estimate computed may well not be a valid autocorrelation sequence, and may contain missing values.
```

I have no idea what that actually means.

```
Following Parzen (1963) we characterize missing observations as being driven by a missing process that is a 0-1 binary variable. In terms of a regression model, the Parzen (1963) approach amounts to plugging in zeros for missing observations. Time series with zeros in place of missing data have been labeled amplitude modulated series by Parzen (1963) which we adopt throughout the paper.
```

Rho, Seung-Hwa, and Timothy J. Vogelsang. "Heteroskedasticity Autocorrelation Robust Inference in Time Series Regressions with Missing Data."  http://www.cireqmontreal.com/wp-content/uploads/2013/11/vogelsang.pdf
I didn't read it, it's just what Google found.

SAS and SPSS also seem to ignore the pairs that contain a nan. SAS has a warning in the documentation, similar
http://stackoverflow.com/questions/7036628/xcorr-for-autocorrelation-with-nan-values

I think we are fine with the current solution, and hope for something better, and eventually full missing value support. I think the main use case is when there is just a small fraction of missing values, where it might not have much effect.

I'm trying to add unit tests to <code>\statsmodels\tsa\tests\test_stattools.py</code>. For the <code>conservative</code> case, I'd like to benchmark against <code>scikits.timeseries.lib.avcf.cvf</code>, but I can't seem to install scikits.timeseries on Windows 64 bit with 64 bit Python (Anaconda distribution). Can anyone recommend how I could amend my installation methods below and/or provide an alternative installer for scikits.timeseries?

I've tried the following methods:
(i) Python 2.6 build from source:
<code>c:\Python27\envs\py26>python c:...\scikits.timeseries-0.91.3\setup.py install</code>
error in scikits.timeseries setup command: Distribution contains no modules or packages for namespace package 'scikits'
<code>c:\Python27\envs\py26>python c:...\scikits.timeseries-0.91.3\setup.py build</code>
error in scikits.timeseries setup command: Distribution contains no modules or packages for namespace package 'scikits'
<code>c:\Python27\envs\py26>scripts\easy_install c:...\scikits.timeseries-0.91.3.tar</code>

```
Processing scikits.timeseries-0.91.3.tar
Writing c:\users\gsher\appdata\local\temp\easy_install-u4pizq\scikits.timeseries-0.91.3\setup.cfg
Running scikits.timeseries-0.91.3\setup.py -q bdist_egg --dist-dir c:\users\...\appdata\local\temp\easy_install-u4pizq\scikits.timeseries-0.91.3\egg-dist-tmp-
gsaqc_
no previously-included directories found matching 'scikits\timeseries\doc\source
\generated'
no previously-included directories found matching 'scikits\timeseries\doc\build'

Looking for python26.dll
error: Setup script exited with error: The system cannot find the file specified
<code>c:\Python27\envs\py26>scripts\pip install c:\...\scikits.time
eries-0.91.3.tar
Unpacking c:\...\scikits.timeseries-0.91.3.tar
  Running setup.py (path:c:\users\...\appdata\local\temp\pip-jl8cyp-build\setup.py) egg_info for package from file:///c:/.../scikits.timeseies-0.91.3.tar
    build_src
    building extension "scikits.timeseries.cseries" sources
    building data_files sources
    build_src: building npy-pkg config files
    no previously-included directories found matching 'scikits\timeseries\doc\s
urce\generated'
    no previously-included directories found matching 'scikits\timeseries\doc\b
ild'
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.3.0 in c:\py
hon27\envs\py26\lib\site-packages (from scikits.timeseries==0.91.3)
Installing collected packages: scikits.timeseries
  Running setup.py install for scikits.timeseries
    unifing config_cc, config, build_clib, build_ext, build commands --compiler
options
    unifing config_fc, config, build_clib, build_ext, build commands --fcompile
 options
    build_src
    building extension "scikits.timeseries.cseries" sources
    building data_files sources
    build_src: building npy-pkg config files
    Looking for python26.dll
    Building msvcr library: "c:\Python27\envs\py26\libs\libmsvcr90d.a" (from C:
Windows\winsxs\amd64_microsoft.vc90.debugcrt_1fc8b3b9a1e18e3b_9.0.21022.8_none_
ec74c6b3093419c\msvcr90d.dll)
    error: The system cannot find the file specified
    Complete output from command c:\Python27\envs\py26\python.exe -c "import setuptools, tokenize;__file__='c:\\users\\gsher\\appdata\\local\\temp\\pip-jl8cyp-uild\\setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().r
place('\r\n', '\n'), __file__, 'exec'))" install --record c:\users\gsher\appdat
\local\temp\pip-noczqm-record\install-record.txt --single-version-externally-ma
aged --compile:
    running install

running build

running config_cc

unifing config_cc, config, build_clib, build_ext, build commands --compiler opt
ons

running config_fc

unifing config_fc, config, build_clib, build_ext, build commands --fcompiler op
ions

running build_src

build_src

building extension "scikits.timeseries.cseries" sources

building data_files sources

build_src: building npy-pkg config files

running build_py

creating build

creating build\lib.win-amd64-2.6

creating build\lib.win-amd64-2.6\scikits

copying .\scikits\__init__.py -> build\lib.win-amd64-2.6\scikits

creating build\lib.win-amd64-2.6\scikits\timeseries

copying scikits\timeseries\const.py -> build\lib.win-amd64-2.6\scikits\timeseri
s

copying scikits\timeseries\extras.py -> build\lib.win-amd64-2.6\scikits\timeser
es

copying scikits\timeseries\parser.py -> build\lib.win-amd64-2.6\scikits\timeser
es

copying scikits\timeseries\setup.py -> build\lib.win-amd64-2.6\scikits\timeseri
s

copying scikits\timeseries\tdates.py -> build\lib.win-amd64-2.6\scikits\timeser
es

copying scikits\timeseries\trecords.py -> build\lib.win-amd64-2.6\scikits\times
ries

copying scikits\timeseries\tseries.py -> build\lib.win-amd64-2.6\scikits\timese
ies

copying scikits\timeseries\version.py -> build\lib.win-amd64-2.6\scikits\timese
ies

copying scikits\timeseries\_preview.py -> build\lib.win-amd64-2.6\scikits\times
ries

copying scikits\timeseries\_tools.py -> build\lib.win-amd64-2.6\scikits\timeser
es

copying scikits\timeseries\__init__.py -> build\lib.win-amd64-2.6\scikits\times
ries

creating build\lib.win-amd64-2.6\scikits\timeseries\lib

copying scikits\timeseries\lib\avcf.py -> build\lib.win-amd64-2.6\scikits\times
ries\lib

copying scikits\timeseries\lib\interpolate.py -> build\lib.win-amd64-2.6\scikit
\timeseries\lib

copying scikits\timeseries\lib\moving_funcs.py -> build\lib.win-amd64-2.6\sciki
s\timeseries\lib

copying scikits\timeseries\lib\plotlib.py -> build\lib.win-amd64-2.6\scikits\ti
eseries\lib

copying scikits\timeseries\lib\reportlib.py -> build\lib.win-amd64-2.6\scikits\
imeseries\lib

copying scikits\timeseries\lib\tstables.py -> build\lib.win-amd64-2.6\scikits\t
meseries\lib

copying scikits\timeseries\lib\__init__.py -> build\lib.win-amd64-2.6\scikits\t
meseries\lib

running build_ext

Looking for python26.dll

Building msvcr library: "c:\Python27\envs\py26\libs\libmsvcr90d.a" (from C:\Win
ows\winsxs\amd64_microsoft.vc90.debugcrt_1fc8b3b9a1e18e3b_9.0.21022.8_none_4ec7
c6b3093419c\msvcr90d.dll)

error: The system cannot find the file specified

----------------------------------------
Cleaning up...
Command c:\Python27\envs\py26\python.exe -c "import setuptools, tokenize;__file
_='c:\\users\\...\\appdata\\local\\temp\\pip-jl8cyp-build\\setup.py';exec(com
ile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __f
le__, 'exec'))" install --record c:\users\...\appdata\local\temp\pip-noczqm-r
cord\install-record.txt --single-version-externally-managed --compile failed wi
h error code 1 in c:\users\...\appdata\local\temp\pip-jl8cyp-build
Storing debug log for failure in C:\Users\...\pip\pip.log
```

I can't figure out why the program can't locate python26.dll, because I can see it in c:\python27\envs\py26\ using Windows Explorer.

(ii) Python 2.7 build from source with easy_install, pip or python ... build and python ... install:
Same errors as above, with 2.6 replaced with 2.7: i.e. cannot find python27.dll, even though I can see this file exists in c:\python27\ using Windows Explorer.

(iii) Python 2.6 with pre-built installer from http://sourceforge.net/projects/pytseries/files/scikits.timeseries/0.91.3:

"Error: Python version 2.6 required, which was not found in the registry."

This might be because I am using Windows 64-bit, while the installer is built on 32-bit.

might not be worth it to figure out how to install scikits\timeseries  if it doesn't work easily
Gohlke still has binaries http://www.lfd.uci.edu/~gohlke/pythonlibs/
pythonxy has it as extra addin which might be the same binaries as Gohlke
https://code.google.com/p/pythonxy/wiki/AdditionalPlugins

I haven't tried to install it in years.

@joesnacks 
For the unit test: I think using only the case where nans are at the beginning and at the end should be enough, The results should be the same for both methods and for the reduced not-nan array without nan handling. Then there is no need to get scikits-timeseries to work.

(I haven't figured out what Stata is doing. The documentation only mentions that nans/missing are not allowed with the fft version, but it doesn't have a missing option. GRETL doesn't seem to be able to handle missing in acf.)

OK @josef-pkt I added some unit tests in <code>test_stattools.py</code> following your above recommendation.

I briefly read through the unit test, I think they are good.

as a check: do the new tests fail without the correction to the demeaning?

BTW: do you know that you can run selective tests on the command line `nosetests  <path to test module>` and similar for directories or individual test function or test classes? 
It's the fastest way to run specific tests during development or debugging.

related: google scholar alert sent me this link http://onlinelibrary.wiley.com/doi/10.1111/jtsa.12072/abstract
but I don't have access to it, and didn't find a ungated copy.

It maybe worth skimming the article, if you have access, to see whether there is anything interesting that would apply also to this case. (not required for this PR)

> BTW: do you know that you can run selective tests on the command line nosetests <path to test module> and similar for directories or individual test function or test classes? 
> It's the fastest way to run specific tests during development or debugging.

You should start a wiki page for new contributions with your list of BTWs.  It is pretty long and useful.

I just checked, and the new tests do fail without the correction to demeaning. We get an acf estimate that is slightly too low:

```
AssertionError:
Arrays are not almost equal to 8 decimals

(mismatch 100.0%)
 x: array([ 0.97803233,  0.96498216,  0.95174931,  0.93761061,  0.92262748,
        0.90735867,  0.89189518,  0.87616742,  0.86023835,  0.84432832,
        0.82854103,  0.81267879,  0.796996  ,  0.78122047,  0.76540652,...
 y: array([ 0.98685781,  0.97371846,  0.96014366,  0.94568545,  0.93054425,
        0.91484806,  0.89901438,  0.8827679 ,  0.86649857,  0.8503736 ,
        0.83435254,  0.81829961,  0.80240987,  0.78635609,  0.77024519,...
```

Did not know about running individual test cases from the command line -- thanks! 

I'm skimming the introduction of the article in the link. The set-up follows a long literature where (i) the binary-state-space <code>nan</code> process R(t) is stationary (but not necessarily iid), and (ii) <code>nan</code>s are uninformative about the (zero-mean) underlying series X(t). Then by treating the <code>nan</code>s as zeros, we observe the 'amplitude-modulated' series R(t)X(t) rather than X(t) itself. The autocovariance of the observed series at lag j can then be written E[R(0)X(0)R(j)X(j)] = [ER(0)R(j)][EX(0)X(j)], and consistent estimation of the autocovariance of series X(t) at lag j can be achieved by estimating the ratio of E[R(0)X(0)R(j)X(j)] to E[R(0)R(j)]. This is my understanding of what we're doing in <code>acovf</code>

I agree, which is why I suggest Parzen as the correct title.  Of course, some description in Notes is helpful.  You might want to see about all 3 if you want something else to do.

Good, 'parzen'  sounds good   (I'm looking again when I'm back to a real computer)


Resolving patch:
diff --git a/statsmodels/tsa/stattools.py b/statsmodels/tsa/stattools.py
--- a/statsmodels/tsa/stattools.py
+++ b/statsmodels/tsa/stattools.py
@@ -1,19 +1,22 @@
 """
 Statistical tools for time series analysis
 """
-from statsmodels.compat.python import (iteritems, range, lrange, string_types, lzip,
-                                zip, map)
+from statsmodels.compat.python import (iteritems, range, lrange, string_types,
+                                       lzip, zip)
+from statsmodels.compat.scipy import _next_regular
+
 import numpy as np
 from numpy.linalg import LinAlgError
 from scipy import stats
+
 from statsmodels.regression.linear_model import OLS, yule_walker
 from statsmodels.tools.tools import add_constant, Bunch
 from statsmodels.tsa.tsatools import lagmat, lagmat2ds, add_trend
 from statsmodels.tsa.adfvalues import mackinnonp, mackinnoncrit
 from statsmodels.tsa._bds import bds
 from statsmodels.tsa.arima_model import ARMA
-from statsmodels.compat.scipy import _next_regular
-from statsmodels.tools.sm_exceptions import InterpolationWarning
+from statsmodels.tools.sm_exceptions import InterpolationWarning, MissingDataError
+
 
 __all__ = ['acovf', 'acf', 'pacf', 'pacf_yw', 'pacf_ols', 'ccovf', 'ccf',
            'periodogram', 'q_stat', 'coint', 'arma_order_select_ic',
@@ -286,8 +289,8 @@ def adfuller(x, maxlag=None, regression="c", autolag='AIC',
             return adfstat, pvalue, usedlag, nobs, critvalues, icbest
 
 
-def acovf(x, unbiased=False, demean=True, fft=False):
-    '''
+def acovf(x, unbiased=False, demean=True, fft=False, missing='none'):
+    """
     Autocovariance for 1D
 
     Parameters
@@ -301,33 +304,78 @@ def acovf(x, unbiased=False, demean=True, fft=False):
     fft : bool
         If True, use FFT convolution.  This method should be preferred
         for long time series.
-
+    missing : str
+        A string in ['none', 'raise', 'conservative', 'drop'] specifying how the NaNs
+        are to be treated.
+        
     Returns
     -------
     acovf : array
         autocovariance function
-    '''
+
+    References
+    -----------
+    .. [1] Parzen, E., 1963. On spectral analysis with missing observations
+           and amplitude modulation. Sankhya: The Indian Journal of
+           Statistics, Series A, pp.383-392.
+    """
     x = np.squeeze(np.asarray(x))
     if x.ndim > 1:
         raise ValueError("x must be 1d. Got %d dims." % x.ndim)
-    n = len(x)
 
-    if demean:
+    missing = missing.lower()
+    if missing not in ['none', 'raise', 'conservative', 'drop']:
+        raise ValueError("missing option %s not understood" % missing)
+    if missing == 'none':
+        deal_with_masked = False
+    else:
+        deal_with_masked = has_missing(x)
+    if deal_with_masked:
+        if missing == 'raise': 
+            raise MissingDataError("NaNs were encountered in the data")
+        notmask_bool = ~np.isnan(x) #bool
+        if missing == 'conservative':
+            x[~notmask_bool] = 0
+        else: #'drop'
+            x = x[notmask_bool] #copies non-missing
+        notmask_int = notmask_bool.astype(int) #int
+
+    if demean and deal_with_masked:
+        # whether 'drop' or 'conservative':
+        xo = x - x.sum()/notmask_int.sum()
+        if missing=='conservative':
+            xo[~notmask_bool] = 0
+    elif demean:
         xo = x - x.mean()
     else:
         xo = x
-    if unbiased:
+
+    n = len(x)
+    if unbiased and deal_with_masked and missing=='conservative':
+        d = np.correlate(notmask_int, notmask_int, 'full')
+    elif unbiased:
         xi = np.arange(1, n + 1)
         d = np.hstack((xi, xi[:-1][::-1]))
-    else:
+    elif deal_with_masked: #biased and NaNs given and ('drop' or 'conservative')
+        d = notmask_int.sum() * np.ones(2*n-1)
+    else: #biased and no NaNs or missing=='none' 
         d = n * np.ones(2 * n - 1)
+
     if fft:
         nobs = len(xo)
-        Frf = np.fft.fft(xo, n=nobs * 2)
-        acov = np.fft.ifft(Frf * np.conjugate(Frf))[:nobs] / d[n - 1:]
-        return acov.real
+        n = _next_regular(2 * nobs + 1)
+        Frf = np.fft.fft(xo, n=n)
+        acov = np.fft.ifft(Frf * np.conjugate(Frf))[:nobs] / d[nobs - 1:]
+        acov = acov.real
     else:
-        return (np.correlate(xo, xo, 'full') / d)[n - 1:]
+        acov = (np.correlate(xo, xo, 'full') / d)[n - 1:]
+
+    if deal_with_masked and missing=='conservative':
+        # restore data for the user
+        x[~notmask_bool] = np.nan
+
+    return acov
+
 
 
 def q_stat(x, nobs, type="ljungbox"):
@@ -362,8 +410,9 @@ def q_stat(x, nobs, type="ljungbox"):
 #NOTE: Changed unbiased to False
 #see for example
 # http://www.itl.nist.gov/div898/handbook/eda/section3/autocopl.htm
-def acf(x, unbiased=False, nlags=40, qstat=False, fft=False, alpha=None):
-    '''
+def acf(x, unbiased=False, nlags=40, qstat=False, fft=False, alpha=None,
+        missing='none'):
+    """
     Autocorrelation function for 1d arrays.
 
     Parameters
@@ -384,6 +433,9 @@ def acf(x, unbiased=False, nlags=40, qstat=False, fft=False, alpha=None):
         returned. For instance if alpha=.05, 95 % confidence intervals are
         returned where the standard deviation is computed according to
         Bartlett\'s formula.
+    missing : str, optional
+        A string in ['none', 'raise', 'conservative', 'drop'] specifying how the NaNs
+        are to be treated.
 
     Returns
     -------
@@ -406,27 +458,17 @@ def acf(x, unbiased=False, nlags=40, qstat=False, fft=False, alpha=None):
 
     If unbiased is true, the denominator for the autocovariance is adjusted
     but the autocorrelation is not an unbiased estimtor.
-    '''
-    nobs = len(x)
-    d = nobs  # changes if unbiased
-    if not fft:
-        avf = acovf(x, unbiased=unbiased, demean=True)
-        #acf = np.take(avf/avf[0], range(1,nlags+1))
-        acf = avf[:nlags + 1] / avf[0]
-    else:
-        x = np.squeeze(np.asarray(x))
-        #JP: move to acovf
-        x0 = x - x.mean()
-        # ensure that we always use a power of 2 or 3 for zero-padding,
-        # this way we'll ensure O(n log n) runtime of the fft.
-        n = _next_regular(2 * nobs + 1)
-        Frf = np.fft.fft(x0, n=n)  # zero-pad for separability
-        if unbiased:
-            d = nobs - np.arange(nobs)
-        acf = np.fft.ifft(Frf * np.conjugate(Frf))[:nobs] / d
-        acf /= acf[0]
-        #acf = np.take(np.real(acf), range(1,nlags+1))
-        acf = np.real(acf[:nlags + 1])   # keep lag 0
+
+    References
+    ----------
+    .. [1] Parzen, E., 1963. On spectral analysis with missing observations
+       and amplitude modulation. Sankhya: The Indian Journal of
+       Statistics, Series A, pp.383-392.
+
+    """
+    nobs = len(x)  # should this shrink for missing='drop' and NaNs in x?
+    avf = acovf(x, unbiased=unbiased, demean=True, fft=fft, missing=missing)
+    acf = avf[:nlags + 1] / avf[0]
     if not (qstat or alpha):
         return acf
     if alpha is not None:
@@ -1056,6 +1098,12 @@ def arma_order_select_ic(y, max_ar=4, max_ma=2, ic='bic', trend='c',
 
     return Bunch(**res)
 
+def has_missing(data):
+    """
+    Returns True if 'data' contains missing entries, otherwise False
+    """
+    return np.isnan(np.sum(data))
+
 
 def kpss(x, regression='c', lags=None, store=False):
     """
@@ -1182,30 +1230,3 @@ def _sigma_est_kpss(resids, nobs, lags):
         resids_prod = np.dot(resids[i:], resids[:nobs - i])
         s_hat += 2 * resids_prod * (1. - (i / (lags + 1.)))
     return s_hat / nobs
-
-
-
-if __name__ == "__main__":
-    import statsmodels.api as sm
-    data = sm.datasets.macrodata.load().data
-    x = data['realgdp']
-# adf is tested now.
-    adf = adfuller(x, 4, autolag=None)
-    adfbic = adfuller(x, autolag="bic")
-    adfaic = adfuller(x, autolag="aic")
-    adftstat = adfuller(x, autolag="t-stat")
-
-# acf is tested now
-#    acf1, ci1, Q, pvalue = acf(x, nlags=40, confint=95, qstat=True)
-#    acf2, ci2, Q2, pvalue2 = acf(x, nlags=40, confint=95, fft=True, qstat=True)
-#    acf3, ci3, Q3, pvalue3 = acf(x, nlags=40, confint=95, qstat=True,
-#                                 unbiased=True)
-#    acf4, ci4, Q4, pvalue4 = acf(x, nlags=40, confint=95, fft=True, qstat=True,
-#                                 unbiased=True)
-
-# pacf is tested now
-#    pacf1 = pacorr(x)
-#    pacfols = pacf_ols(x, nlags=40)
-#    pacfyw = pacf_yw(x, nlags=40, method="mle")
-    y = np.random.normal(size=(100, 2))
-    grangercausalitytests(y, 2)


Test patch:
diff --git a/statsmodels/tsa/tests/test_arima.py b/statsmodels/tsa/tests/test_arima.py
--- a/statsmodels/tsa/tests/test_arima.py
+++ b/statsmodels/tsa/tests/test_arima.py
@@ -1999,7 +1999,7 @@ def test_arima_dates_startatend():
 
 
 def test_arma_missing():
-    from statsmodels.base.data import MissingDataError
+    from statsmodels.tools.sm_exceptions import MissingDataError
     # bug 1343
     y = np.random.random(40)
     y[-1] = np.nan
diff --git a/statsmodels/tsa/tests/test_stattools.py b/statsmodels/tsa/tests/test_stattools.py
--- a/statsmodels/tsa/tests/test_stattools.py
+++ b/statsmodels/tsa/tests/test_stattools.py
@@ -7,11 +7,12 @@
 import numpy as np
 from numpy.testing import (assert_almost_equal, assert_equal, assert_warns,
                            assert_raises, dec, assert_)
-from numpy import genfromtxt#, concatenate
+from numpy import genfromtxt
 from statsmodels.datasets import macrodata, sunspots
 from pandas import Series, Index, DataFrame
 import os
 import warnings
+from statsmodels.tools.sm_exceptions import MissingDataError
 
 DECIMAL_8 = 8
 DECIMAL_6 = 6
@@ -156,9 +157,7 @@ def test_qstat(self):
 
 
 class TestACF_FFT(CheckCorrGram):
-    """
-    Test Autocorrelation Function using FFT
-    """
+    # Test Autocorrelation Function using FFT
     def __init__(self):
         self.acf = self.results['acvarfft']
         self.qstat = self.results['Q1']
@@ -170,7 +169,45 @@ def test_acf(self):
     def test_qstat(self):
         #todo why is res1/qstat 1 short
         assert_almost_equal(self.res1[1], self.qstat, DECIMAL_3)
-
+        
+class TestACFMissing(CheckCorrGram):
+    # Test Autocorrelation Function using Missing
+    def __init__(self):
+        self.x = np.concatenate((np.array([np.nan]),self.x))
+        self.acf = self.results['acvar'] # drop and conservative
+        self.qstat = self.results['Q1']
+        self.res_drop = acf(self.x, nlags=40, qstat=True, alpha=.05, 
+                            missing='drop')
+        self.res_conservative = acf(self.x, nlags=40, qstat=True, alpha=.05, 
+                                    missing='conservative')       
+        self.acf_none = np.empty(40) * np.nan # lags 1 to 40 inclusive
+        self.qstat_none = np.empty(40) * np.nan
+        self.res_none = acf(self.x, nlags=40, qstat=True, alpha=.05,
+                        missing='none')
+    
+    def test_raise(self):
+        assert_raises(MissingDataError, acf, self.x, nlags=40, 
+                      qstat=True, alpha=.05, missing='raise')
+                      
+    def test_acf_none(self):
+        assert_almost_equal(self.res_none[0][1:41], self.acf_none, DECIMAL_8)
+    
+    def test_acf_drop(self):
+        assert_almost_equal(self.res_drop[0][1:41], self.acf, DECIMAL_8)
+    
+    def test_acf_conservative(self):
+        assert_almost_equal(self.res_conservative[0][1:41], self.acf, 
+                            DECIMAL_8)
+
+    def test_qstat_none(self):
+        #todo why is res1/qstat 1 short
+        assert_almost_equal(self.res_none[2], self.qstat_none, DECIMAL_3)
+    
+# how to do this test? the correct q_stat depends on whether nobs=len(x) is 
+# used when x contains NaNs or whether nobs<len(x) when x contains NaNs
+#    def test_qstat_drop(self):
+#        assert_almost_equal(self.res_drop[2][:40], self.qstat, DECIMAL_3)
+        
 
 class TestPACF(CheckCorrGram):
     def __init__(self):