Repository: statsmodels/statsmodels

Issue:
AIC/BIC inconsistent between AutoReg and ar_select_order
AIC and BIC do not appear to be calculated the same way in `AutoReg` and `ar_select_order`. Here's an example.

```
import numpy as np
from statsmodels.tsa.arima_process import ArmaProcess
from statsmodels.tsa.ar_model import AutoReg, ar_select_order

np.random.seed(99999)

coefs = np.array([0.5, -0.25])
y = ArmaProcess(np.r_[1, -coefs]).generate_sample(250)
```

Then we do model selection.

`modsel = ar_select_order(y, maxlag=5, old_names=False)`

`model.aic` has this:

> {(1, 2): 0.038243172406288724,
>  (1, 2, 3): 0.04552812846926287,
>  (1, 2, 3, 4): 0.0536414426749059,
>  (1, 2, 3, 4, 5): 0.061296136921122346,
>  (1,): 0.0735756692249579,
>  0: 0.26784552200885503}

I believe this means that the AIC for the model with 2 lags is 0.03824...

Estimating the model with 2 lags, I get:

```
res = AutoReg(y, lags=2, old_names=False).fit()
print(res.summary())
```

```
                            AutoReg Model Results                             
==============================================================================
Dep. Variable:                      y   No. Observations:                  250
Model:                     AutoReg(2)   Log Likelihood                -353.692
Method:               Conditional MLE   S.D. of innovations              1.007
Date:                Tue, 03 Nov 2020   AIC                              0.047
Time:                        16:56:07   BIC                              0.103
Sample:                             2   HQIC                             0.070
                                  250                                         
==============================================================================
                 coef    std err          z      P>|z|      [0.025      0.975]
------------------------------------------------------------------------------
const          0.0269      0.064      0.420      0.674      -0.099       0.152
y.L1           0.5051      0.062      8.149      0.000       0.384       0.627
y.L2          -0.2020      0.062     -3.259      0.001      -0.323      -0.080
                                    Roots                                    
```

The AIC for this model doesn't match what we got above. BIC is also different. I tried all the `trend` options, and seem to get the closest with `n` but it's still not quite the same.

Shouldn't these be the same? Or is selection model using different options?


Discussion:
I don't know the code here, so a generic answer.

we want to use the same data when doing lag search. So I guess in ar_select_order the data is truncated to 5 lags even with shorter lags in a model.

you could check `res = AutoReg(y[3:], lags=2, ...`
or something like this
another possible difference in some model is that we want to use a fast approximate method in specification search, and use the best method for the final model.
What @josef-pkt said: Selection requires the LHS variable to be the same, and so the maximum lag length affects the data used to select the model. When you use AutoReg it defaults to the maximum available data. If you want them to be the same you can set `hold_back` in `AutoReg`.
Thanks, that makes sense.

It looks like there's an error in the documentation for BIC. It [says](https://www.statsmodels.org/stable/generated/statsmodels.tsa.ar_model.AutoRegResults.html)

ln(𝜎)+𝑑𝑓𝑚𝑜𝑑𝑒𝑙ln(𝑛𝑜𝑏𝑠)/𝑛𝑜𝑏𝑠

but it is currently calculated as

ln(𝜎^2)+(1+𝑑𝑓𝑚𝑜𝑑𝑒𝑙)ln(𝑛𝑜𝑏𝑠)/𝑛𝑜𝑏𝑠

Also, there's a few places where it says "using Lutkephol’s definition" instead of Lutke**poh**l.

Resolving patch:
diff --git a/statsmodels/tsa/ar_model.py b/statsmodels/tsa/ar_model.py
--- a/statsmodels/tsa/ar_model.py
+++ b/statsmodels/tsa/ar_model.py
@@ -1526,7 +1526,7 @@ def pvalues(self):
     @cache_readonly
     def aic(self):
         """
-        Akaike Information Criterion using Lutkephol's definition.
+        Akaike Information Criterion using Lutkepohl's definition.
 
         :math:`log(sigma) + 2*(1 + k_ar + k_trend)/nobs`
         """
@@ -1541,7 +1541,11 @@ def aic(self):
 
     @cache_readonly
     def hqic(self):
-        """Hannan-Quinn Information Criterion."""
+        r"""
+        Hannan-Quinn Information Criterion using Lutkepohl's definition.
+
+        :math:`\ln(\sigma) + (1 + df_{model}) 2 \ln(\ln(nobs))/nobs`
+        """
         nobs = self.nobs
         # Lutkepohl
         # return np.log(self.sigma2)+ 2 * np.log(np.log(nobs))/nobs * self.k_ar
@@ -1568,10 +1572,10 @@ def fpe(self):
 
     @cache_readonly
     def bic(self):
-        """
-         Bayes Information Criterion
+        r"""
+        Bayes Information Criterion
 
-        :math:`\\log(\\sigma) + (1 + k_ar + k_trend)*\\log(nobs)/nobs`
+        :math:`\ln(\sigma) + (1 + df_{model}) \ln(nobs)/nobs`
         """
         nobs = self.nobs
         # Lutkepohl
@@ -1857,7 +1861,7 @@ def bse(self):  # allow user to specify?
     @cache_readonly
     def aic(self):
         """
-        Akaike Information Criterion using Lutkephol's definition.
+        Akaike Information Criterion using Lutkepohl's definition.
 
         :math:`log(sigma) + 2*(1 + df_model) / nobs`
         """
@@ -1872,7 +1876,11 @@ def aic(self):
 
     @cache_readonly
     def hqic(self):
-        """Hannan-Quinn Information Criterion."""
+        r"""
+        Hannan-Quinn Information Criterion using Lutkepohl's definition.
+
+        :math:`\ln(\sigma) + (1 + df_{model}) 2 \ln(\ln(nobs))/nobs`
+        """
         # Lutkepohl
         # return np.log(self.sigma2)+ 2 * np.log(np.log(nobs))/nobs * self.k_ar
         # R uses all estimated parameters rather than just lags
@@ -1902,7 +1910,7 @@ def bic(self):
         r"""
         Bayes Information Criterion
 
-        :math:`\ln(\sigma) + df_{model} \ln(nobs)/nobs`
+        :math:`\ln(\sigma) + (1 + df_{model}) \ln(nobs)/nobs`
         """
         # Lutkepohl
         # np.log(self.sigma2) + np.log(nobs)/nobs * self.k_ar
diff --git a/statsmodels/tsa/stattools.py b/statsmodels/tsa/stattools.py
--- a/statsmodels/tsa/stattools.py
+++ b/statsmodels/tsa/stattools.py
@@ -186,14 +186,15 @@ def adfuller(
         * "nc" : no constant, no trend.
 
     autolag : {"AIC", "BIC", "t-stat", None}
-        Method to use when automatically determining the lag.
+        Method to use when automatically determining the lag length among the
+        values 0, 1, ..., maxlag.
 
-        * if None, then maxlag lags are used.
-        * if "AIC" (default) or "BIC", then the number of lags is chosen
+        * If "AIC" (default) or "BIC", then the number of lags is chosen
           to minimize the corresponding information criterion.
         * "t-stat" based choice of maxlag.  Starts with maxlag and drops a
           lag until the t-statistic on the last lag length is significant
           using a 5%-sized test.
+        * If None, then the number of included lags is set to maxlag.
     store : bool
         If True, then a result instance is returned additionally to
         the adf statistic. Default is False.


Test patch: