Repository: statsmodels/statsmodels

Issue:
auto_lag selection in acorr_ljungbox fails when there is no autocorrelation
#### Describe the bug

The `acorr_ljungbox` function has an `auto_lag` flag which determines the optimal lag for performing the Ljung-Box autocorrelations test. When there is no autocorrelation in the data, however, the function fails and returns a "ValueError: zero-size array to reduction operation maximum which has no identity". When there is low autocorrelation, it may be returning an incorrect optimal lag (for theoretical reasons - described under **Details**).

#### Code Sample, a copy-pastable example if possible

```python
import numpy as np
from statsmodels.stats.diagnostic import acorr_ljungbox

data = np.random.rand(1000) # white noise process with no autocorrelation
res = acorr_ljunbox(data, auto_lag=True)

ValueError: zero-size array to reduction operation maximum which has no identity
```
<details>

The `auto_lag` feature attempts to implement the lag selection criterion outlined in Escanciano & Lobato (J. of Econometrics, 2009). However, there are two errors in the implementation, which is performed by the `get_optimal_length` inner function of the` acorr_ljungbox` function. There is also one error in the associated pytest.

The first error is that the optimal_lag is initialized at `0`. When the rest of the routine does not update the optimal_lag (see second error below for why this may happen), the optimal lag length is set at `0`. This causes an empty array to be initialized, and when the function tries to find the `max()` of this array, the `ValueError` related to a zero-size array is returned. Even theoretically, [1] defines the putative optimal lag $\tilde{p}$ as `>=1`.

The second error is that `get_optimal_length` initializes the value of the least penalized test statistic ($L_p$ in eq. (4), pg. 143 in [1]) as `0`. This is incorrect --- there is no theoretical reason why the penalized value of the test statistic should be positive. This is especially true for series that exhibit low degrees of autocorrelation, when the unpenalized Box-Pierce (or Ljung-Box) test statistic can be a very low value. Thus, even if the least penalized test statistic occurs at a large lag, if it has a negative penalized score, `get_optimal_lag` would still return `0` as the optimal lag. 

Finally, the test associated with this function `test_ljungbox_auto_lag_selection()` (in `test_diagnostic.py`) calls `acorr_ljungbox()`, but omits to set `auto_lag = True`. So the test is invalid. 

I've initiated a pull request fixing the above issues and the test. This is my first bug report and pull request, so any guidance is greatly appreciated!

_[1] J. Carlos Escanciano, Ignacio N. Lobato “An automatic Portmanteau test for serial correlation”., J. of Econometrics Volume 151, 2009._

</details>


#### Expected Output

Return the result for the Ljung-Box test with the optimal lag as 1 if autocorrelation is absent, and the correct lag based on the Escanciano & Lobato procedure if autocorrelation is not absent.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

Output of ``import statsmodels.api as sm; sm.show_versions()`` 

INSTALLED VERSIONS
------------------
Python: 3.8.1.final.0

statsmodels
===========

Installed: 0.13.2 (C:\Users\Vasudeva\AppData\Local\Programs\Python\Python38\lib\site-packages\statsmodels)

Required Dependencies
=====================

cython: 0.29.15 (C:\Users\Vasudeva\AppData\Local\Programs\Python\Python38\lib\site-packages\Cython)
numpy: 1.18.1 (C:\Users\Vasudeva\AppData\Local\Programs\Python\Python38\lib\site-packages\numpy)
scipy: 1.4.1 (C:\Users\Vasudeva\AppData\Local\Programs\Python\Python38\lib\site-packages\scipy)
pandas: 1.0.1 (C:\Users\Vasudeva\AppData\Local\Programs\Python\Python38\lib\site-packages\pandas)
    dateutil: 2.8.1 (C:\Users\Vasudeva\AppData\Local\Programs\Python\Python38\lib\site-packages\dateutil)
patsy: 0.5.2 (C:\Users\Vasudeva\AppData\Local\Programs\Python\Python38\lib\site-packages\patsy)

Optional Dependencies
=====================

matplotlib: 3.1.3 (C:\Users\Vasudeva\AppData\Local\Programs\Python\Python38\lib\site-packages\matplotlib)
    backend: TkAgg
cvxopt: Not installed
joblib: 1.1.0 (C:\Users\Vasudeva\AppData\Local\Programs\Python\Python38\lib\site-packages\joblib)

Developer Tools
================

IPython: 7.16.1 (C:\Users\Vasudeva\AppData\Local\Programs\Python\Python38\lib\site-packages\IPython)
    jinja2: 2.11.2 (C:\Users\Vasudeva\AppData\Local\Programs\Python\Python38\lib\site-packages\jinja2)
sphinx: 3.1.2 (C:\Users\Vasudeva\AppData\Local\Programs\Python\Python38\lib\site-packages\sphinx)
    pygments: 2.6.1 (C:\Users\Vasudeva\AppData\Local\Programs\Python\Python38\lib\site-packages\pygments)
pytest: Not installed
virtualenv: Not installed

</details>



Discussion:

Resolving patch:
diff --git a/statsmodels/stats/diagnostic.py b/statsmodels/stats/diagnostic.py
--- a/statsmodels/stats/diagnostic.py
+++ b/statsmodels/stats/diagnostic.py
@@ -399,23 +399,6 @@ def acorr_ljungbox(x, lags=None, boxpierce=False, model_df=0, period=None,
            lb_stat     lb_pvalue
     10  214.106992  1.827374e-40
     """
-    def get_optimal_length(threshold_metric, threshold, maxlag, func):
-        optimal_lag = 0
-        least_penalised = 0
-
-        for lags in range(1, maxlag + 1):
-            if (threshold_metric <= threshold):
-                penalty = lags * np.log(nobs)
-            else:
-                penalty = 2 * lags
-
-            test_statistic = func(lags)
-            penalised = test_statistic - penalty
-            if (penalised > least_penalised):
-                optimal_lag = lags
-                least_penalised = penalised
-
-        return optimal_lag
     # Avoid cyclic import
     from statsmodels.tsa.stattools import acf
     x = array_like(x, "x")
@@ -429,25 +412,30 @@ def get_optimal_length(threshold_metric, threshold, maxlag, func):
     if auto_lag:
         maxlag = nobs - 1
 
-        # Compute threshold metrics
+        # Compute sum of squared autocorrelations
         sacf = acf(x, nlags=maxlag, fft=False)
-        sacf2 = sacf[1:maxlag + 1] ** 2 / (nobs - np.arange(1, maxlag + 1))
+
+        if not boxpierce:
+            q_sacf = (nobs * (nobs + 2) *
+                      np.cumsum(sacf[1:maxlag + 1] ** 2
+                                / (nobs - np.arange(1, maxlag + 1))))
+        else:
+            q_sacf = nobs * np.cumsum(sacf[1:maxlag + 1] ** 2)
+
+        # obtain thresholds
         q = 2.4
         threshold = np.sqrt(q * np.log(nobs))
-        threshold_metric = max(np.abs(sacf)) * np.sqrt(nobs)
+        threshold_metric = np.abs(sacf).max() * np.sqrt(nobs)
 
-        if not boxpierce:
-            lags = get_optimal_length(
-                threshold_metric,
-                threshold, maxlag,
-                lambda p: nobs * (nobs + 2) * np.cumsum(sacf2)[p - 1])
+        # compute penalized sum of squared autocorrelations
+        if (threshold_metric <= threshold):
+            q_sacf = q_sacf - (np.arange(1, nobs) * np.log(nobs))
         else:
-            lags = get_optimal_length(
-                threshold_metric,
-                threshold,
-                maxlag,
-                lambda p: nobs * np.cumsum(sacf[1:maxlag + 1] ** 2)[p - 1])
+            q_sacf = q_sacf - (2 * np.arange(1, nobs))
 
+        # note: np.argmax returns first (i.e., smallest) index of largest value
+        lags = np.argmax(q_sacf)
+        lags = max(1, lags)  # optimal lag has to be at least 1
         lags = int_like(lags, "lags")
         lags = np.arange(1, lags + 1)
     elif period is not None:


Test patch:
diff --git a/statsmodels/stats/tests/test_diagnostic.py b/statsmodels/stats/tests/test_diagnostic.py
--- a/statsmodels/stats/tests/test_diagnostic.py
+++ b/statsmodels/stats/tests/test_diagnostic.py
@@ -1701,17 +1701,29 @@ def test_ljungbox_dof_adj():
     assert np.all(res2.iloc[4:, 1] <= res1.iloc[4:, 1])
 
 
-def test_ljungbox_auto_lag_selection():
+def test_ljungbox_auto_lag_selection(reset_randomstate):
     data = sunspots.load_pandas().data["SUNACTIVITY"]
     res = AutoReg(data, 4, old_names=False).fit()
     resid = res.resid
-    res1 = smsdia.acorr_ljungbox(resid)
-    res2 = smsdia.acorr_ljungbox(resid, model_df=4)
+    res1 = smsdia.acorr_ljungbox(resid, auto_lag=True)
+    res2 = smsdia.acorr_ljungbox(resid, model_df=4, auto_lag=True)
     assert_allclose(res1.iloc[:, 0], res2.iloc[:, 0])
+    # TODO: compare selected lags with Stata/ R to confirm
+    # that corect auto_lag is selected
+    assert res1.shape[0] >= 1
+    assert res2.shape[0] >= 1
     assert np.all(np.isnan(res2.iloc[:4, 1]))
     assert np.all(res2.iloc[4:, 1] <= res1.iloc[4:, 1])
 
 
+def test_ljungbox_auto_lag_whitenoise(reset_randomstate):
+    data = np.random.randn(1000)  # white noise process
+    res = smsdia.acorr_ljungbox(data, auto_lag=True)
+    # TODO: compare selected lags with Stata/ R to confirm
+    # that correct auto_lag is selected
+    assert res.shape[0] >= 1  # auto lag selected must be at least 1
+
+
 def test_ljungbox_errors_warnings():
     data = sunspots.load_pandas().data["SUNACTIVITY"]
     with pytest.raises(ValueError, match="model_df must"):