Repository: statsmodels/statsmodels

Issue:
Off-by-one error / loose dtype checking in KPSS statistic
#### Describe the bug

Arguably the KPSS code has a couple of issues. Firstly, there's no check whether the input array has suitable dtype, which can result in some weird error messages.
Second, I _think_ there's an off-by-one error in the internal function `_sigma_est_kpss`, but it only actually causes an exception in a weird edge-case highlighted by the object dtype.

(If you look at the indexing here:
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/stattools.py#L1733
Then you'll see that in the edge case `nobs==lags`, and for the last iteration in the loop we are computing `np.dot` on two empty arrays. The return value in this case is dtype dependent, it defaults to 0 for anything numeric, but if it happens to be an object-array we get None and the line of maths below breaks.)

NB, I don't have access to Kwiatkowski et al. (1992), so can't verify the original equation for myself.

#### Code Sample, a copy-pastable example if possible


```python
import numpy as np
from statsmodels.tsa.stattools import kpss

# Confusing error one:
boring_data = np.full(20, 0).astype(object)
kpss(boring_data, lags="auto")

# Confusing error two, hits possible off-by-one error in _sigma_est_kpss:
weird_data = array([0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0,
       0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0,
       0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0,
       0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0,
       0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0,
       1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1,
       1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1,
       0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0,
       0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0,
       0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0,
       0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0,
       0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0,
       1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1,
       1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0])

kpss(weird_data.astype(object), lags="auto")

```

Both those kpss calls result in an exception.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``



<details>


INSTALLED VERSIONS
------------------
Python: 3.6.5.final.0
OS: Linux 4.15.0-47-generic #50-Ubuntu SMP Wed Mar 13 10:44:52 UTC 2019 x86_64
byteorder: little
LC_ALL: None
LANG: en_GB.UTF-8

Statsmodels
===========

Installed: 0.10.0 (/home/staley/.virtualenvs/snippets/lib/python3.6/site-packages/statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.14.3 (/home/staley/.virtualenvs/snippets/lib/python3.6/site-packages/numpy)
scipy: 1.1.0 (/home/staley/.virtualenvs/snippets/lib/python3.6/site-packages/scipy)
pandas: 0.21.0 (/home/staley/.virtualenvs/snippets/lib/python3.6/site-packages/pandas)
    dateutil: 2.7.2 (/home/staley/.virtualenvs/snippets/lib/python3.6/site-packages/dateutil)
patsy: 0.5.1 (/home/staley/.virtualenvs/snippets/lib/python3.6/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 2.2.2 (/home/staley/.virtualenvs/snippets/lib/python3.6/site-packages/matplotlib)
    backend: module://ipykernel.pylab.backend_inline 
cvxopt: Not installed
joblib: Not installed

Developer Tools
================

IPython: 6.3.1 (/home/staley/.virtualenvs/snippets/lib/python3.6/site-packages/IPython)
    jinja2: 2.10 (/home/staley/.virtualenvs/snippets/lib/python3.6/site-packages/jinja2)
sphinx: Not installed
    pygments: 2.2.0 (/home/staley/.virtualenvs/snippets/lib/python3.6/site-packages/pygments)
pytest: Not installed
virtualenv: 16.0.0 (/home/staley/.virtualenvs/snippets/lib/python3.6/site-packages)




</details>



Discussion:
> (If you look at the indexing here:
> https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/stattools.py#L1733
> Then you'll see that in the edge case `nobs==lags`, and for the last iteration in the loop we are computing `np.dot` on two empty arrays. The return value in this case is dtype dependent, it defaults to 0 for anything numeric, but if it happens to be an object-array we get None and the line of maths below breaks.)

The bug is the missing limit on `lags` that makes sure it is smaller than nobs.  The code is correct for a Newey-West estimator.

Resolving patch:
diff --git a/statsmodels/tsa/stattools.py b/statsmodels/tsa/stattools.py
--- a/statsmodels/tsa/stattools.py
+++ b/statsmodels/tsa/stattools.py
@@ -1682,17 +1682,18 @@ def kpss(x, regression='c', lags=None, store=False):
               'same as lags=\'auto\' which uses an automatic lag length ' \
               'selection method. To silence this warning, either use ' \
               '\'auto\' or \'legacy\''
-        warn(msg, DeprecationWarning)
+        warn(msg, FutureWarning)
     if lags == 'legacy':
         lags = int(np.ceil(12. * np.power(nobs / 100., 1 / 4.)))
+        lags = min(lags, nobs - 1)
     elif lags == 'auto':
         # autolag method of Hobijn et al. (1998)
         lags = _kpss_autolag(resids, nobs)
     else:
         lags = int(lags)
 
-    if lags > nobs:
-        raise ValueError("lags ({}) must be <= number of observations ({})"
+    if lags >= nobs:
+        raise ValueError("lags ({}) must be < number of observations ({})"
                          .format(lags, nobs))
 
     pvals = [0.10, 0.05, 0.025, 0.01]


Test patch:
diff --git a/statsmodels/tsa/tests/test_stattools.py b/statsmodels/tsa/tests/test_stattools.py
--- a/statsmodels/tsa/tests/test_stattools.py
+++ b/statsmodels/tsa/tests/test_stattools.py
@@ -1,8 +1,3 @@
-from statsmodels.compat.numpy import lstsq
-from statsmodels.compat.pandas import assert_index_equal
-from statsmodels.compat.platform import PLATFORM_WIN
-from statsmodels.compat.python import lrange
-
 import os
 import warnings
 
@@ -13,17 +8,22 @@
                            assert_, assert_allclose)
 from pandas import Series, date_range, DataFrame
 
+from statsmodels.compat.numpy import lstsq
+from statsmodels.compat.pandas import assert_index_equal
+from statsmodels.compat.platform import PLATFORM_WIN
+from statsmodels.compat.python import lrange
 from statsmodels.datasets import macrodata, sunspots, nile, randhie, modechoice
 from statsmodels.tools.sm_exceptions import (CollinearityWarning,
-                                             MissingDataError)
+                                             MissingDataError,
+                                             InterpolationWarning)
+from statsmodels.tsa.arima_process import arma_acovf
+from statsmodels.tsa.statespace.sarimax import SARIMAX
 from statsmodels.tsa.stattools import (adfuller, acf, pacf_yw, pacf_ols,
                                        pacf, grangercausalitytests,
                                        coint, acovf, kpss,
                                        arma_order_select_ic, levinson_durbin,
                                        levinson_durbin_pacf, pacf_burg,
                                        innovations_algo, innovations_filter)
-from statsmodels.tsa.arima_process import arma_acovf
-from statsmodels.tsa.statespace.sarimax import SARIMAX
 
 DECIMAL_8 = 8
 DECIMAL_6 = 6
@@ -469,42 +469,47 @@ class TestKPSS(SetupKPSS):
     """
 
     def test_fail_nonvector_input(self, reset_randomstate):
-        with warnings.catch_warnings(record=True):
-            kpss(self.x)  # should be fine
+        # should be fine
+        with pytest.warns(InterpolationWarning):
+            kpss(self.x, lags='legacy')
 
         x = np.random.rand(20, 2)
         assert_raises(ValueError, kpss, x)
 
     def test_fail_unclear_hypothesis(self):
         # these should be fine,
-        with warnings.catch_warnings(record=True) as w:
-            kpss(self.x, 'c')
-            kpss(self.x, 'C')
-            kpss(self.x, 'ct')
-            kpss(self.x, 'CT')
-
-        assert_raises(ValueError, kpss, self.x, "unclear hypothesis")
+        with pytest.warns(InterpolationWarning):
+            kpss(self.x, 'c', lags='legacy')
+        with pytest.warns(InterpolationWarning):
+            kpss(self.x, 'C', lags='legacy')
+        with pytest.warns(InterpolationWarning):
+            kpss(self.x, 'ct', lags='legacy')
+        with pytest.warns(InterpolationWarning):
+            kpss(self.x, 'CT', lags='legacy')
+
+        assert_raises(ValueError, kpss, self.x, "unclear hypothesis",
+                      lags='legacy')
 
     def test_teststat(self):
-        with warnings.catch_warnings(record=True) as w:
+        with pytest.warns(InterpolationWarning):
             kpss_stat, pval, lags, crits = kpss(self.x, 'c', 3)
         assert_almost_equal(kpss_stat, 5.0169, DECIMAL_3)
 
-        with warnings.catch_warnings(record=True) as w:
+        with pytest.warns(InterpolationWarning):
             kpss_stat, pval, lags, crits = kpss(self.x, 'ct', 3)
         assert_almost_equal(kpss_stat, 1.1828, DECIMAL_3)
 
     def test_pval(self):
-        with warnings.catch_warnings(record=True) as w:
+        with pytest.warns(InterpolationWarning):
             kpss_stat, pval, lags, crits = kpss(self.x, 'c', 3)
         assert_equal(pval, 0.01)
 
-        with warnings.catch_warnings(record=True) as w:
+        with pytest.warns(InterpolationWarning):
             kpss_stat, pval, lags, crits = kpss(self.x, 'ct', 3)
         assert_equal(pval, 0.01)
 
     def test_store(self):
-        with warnings.catch_warnings(record=True) as w:
+        with pytest.warns(InterpolationWarning):
             kpss_stat, pval, crit, store = kpss(self.x, 'c', 3, True)
 
         # assert attributes, and make sure they're correct
@@ -514,40 +519,40 @@ def test_store(self):
     # test autolag function _kpss_autolag against SAS 9.3
     def test_lags(self):
         # real GDP from macrodata data set
-        with warnings.catch_warnings(record=True):
-            lags = kpss(self.x, 'c', lags='auto')[2]
-        assert_equal(lags, 9)
+        with pytest.warns(InterpolationWarning):
+            res = kpss(self.x, 'c', lags='auto')
+        assert_equal(res[2], 9)
         # real interest rates from macrodata data set
-        with warnings.catch_warnings(record=True):
-            lags = kpss(sunspots.load().data['SUNACTIVITY'], 'c',
-                        lags='auto')[2]
-        assert_equal(lags, 7)
+        res = kpss(sunspots.load(True).data['SUNACTIVITY'], 'c', lags='auto')
+        assert_equal(res[2], 7)
         # volumes from nile data set
-        with warnings.catch_warnings(record=True):
-            lags = kpss(nile.load().data['volume'], 'c', lags='auto')[2]
-        assert_equal(lags, 5)
+        with pytest.warns(InterpolationWarning):
+            res = kpss(nile.load(True).data['volume'], 'c', lags='auto')
+        assert_equal(res[2], 5)
         # log-coinsurance from randhie data set
-        with warnings.catch_warnings(record=True):
-            lags = kpss(randhie.load().data['lncoins'], 'ct', lags='auto')[2]
-        assert_equal(lags, 75)
+        with pytest.warns(InterpolationWarning):
+            res = kpss(randhie.load(True).data['lncoins'], 'ct', lags='auto')
+        assert_equal(res[2], 75)
         # in-vehicle time from modechoice data set
-        with warnings.catch_warnings(record=True):
-            lags = kpss(modechoice.load().data['invt'], 'ct', lags='auto')[2]
-        assert_equal(lags, 18)
+        with pytest.warns(InterpolationWarning):
+            res = kpss(modechoice.load(True).data['invt'], 'ct', lags='auto')
+        assert_equal(res[2], 18)
 
     def test_kpss_fails_on_nobs_check(self):
-        # Test that if lags exceeds number of observations KPSS raises a clear error
+        # Test that if lags exceeds number of observations KPSS raises a
+        # clear error
+        # GH5925
         nobs = len(self.x)
-        msg = (r"lags \({}\) must be <= number of observations \({}\)"
-               .format(nobs+1, nobs))
+        msg = (r"lags \({}\) must be < number of observations \({}\)"
+               .format(nobs, nobs))
         with pytest.raises(ValueError, match=msg):
-            kpss(self.x, 'c', lags=nobs+1)
+            kpss(self.x, 'c', lags=nobs)
 
     def test_legacy_lags(self):
         # Test legacy lags are the same
-        with warnings.catch_warnings(record=True):
-            lags = kpss(self.x, 'c', lags='legacy')[2]
-        assert_equal(lags, 15)
+        with pytest.warns(InterpolationWarning):
+            res = kpss(self.x, 'c', lags='legacy')
+        assert_equal(res[2], 15)
 
     def test_unknown_lags(self):
         # Test legacy lags are the same
@@ -555,8 +560,8 @@ def test_unknown_lags(self):
             kpss(self.x, 'c', lags='unknown')
 
     def test_deprecation(self):
-        with pytest.deprecated_call():
-            kpss(self.x, 'c', lags=None)
+        with pytest.warns(FutureWarning):
+            kpss(self.x, 'c')
 
 
 def test_pandasacovf():