Repository: statsmodels/statsmodels

Issue:
maxlag should be < nobs
I have the sales input data for 2 years say for 2014-2015 from month 04(April)-12(Dec) and need to forecast the value for the year 2016.
In this case,i have only small amount of data say less than 50 rows.
For this i am getting the error "ValueError: maxlag should be < nobs" ,so i browsed and found that i need to set the lesser maxlag value,but have no clue where to set it.
Already tried setting it in fit(maxlag=intValue,ic='bic') but it is not getting assigned with the new value and again the same error.
Where i need to and how i should set it actually?
Please help.
Looking for your response.
This is the code i tried:

```
def test_final_output():
    df = pd.read_csv('../../data/salesoneproduct.csv', index_col=0)
    nobs=len(df)
    df.index.name=None
    df.reset_index(inplace=True)
    start = datetime.datetime.strptime("2014-04-01", "%Y-%m-%d")
    date_list = [start + relativedelta(months=x) for x in range(0,48)]
    df['index'] =date_list
    df.set_index(['index'], inplace=True)
    df.index.name=None
    df.columns= ['riders']
    df['riders'] = df.riders.apply(lambda x: int(x)*100)
    mod = sm.tsa.statespace.SARIMAX(df.riders, trend='n', order=(0,1,0), seasonal_order=(1,1,1,12)
    results = mod.fit(ic='bic',maxlag=10)
    start = datetime.datetime.strptime("2019-04-01", "%Y-%m-%d")
    date_list = [start + relativedelta(months=x) for x in range(0,12)]
    future = pd.DataFrame(index=date_list, columns= df.columns)
    df = pd.concat([df, future])
    df['forecast'] = results.predict(start = 48, end = 59, dynamic= True)
    df[['riders', 'forecast']].ix[-24:].plot(figsize=(12, 8))
    print (df.forecast/100)
```



Discussion:
Thanks for the bug report! In this case, the bug is that we don't give a more informative error message.

The problem here is that starting parameters for the numerical optimization procedure can't be computed for the model you're trying to fit because you have too few observations to perform conditional least squares (which is how we get starting parameters). In this case, there's nothing to be done but specify starting parameters yourself, for example you could do:

```python
start_params = [0, 0, 1.]
results = mod.fit(start_params)
```

I should also note that the `maxlag` and `ic` parameters have no effect for the `SARIMAX` model.

As a final note - the model you're specifying is probably not going to give great results even if you get it to perform the fitting, since you have so little data. I think you have 21 observations, and your seasonal order is 12. That means you don't even have two complete seasons worth of data.
I reformatted the code in the description.
FAQ label because I don't know if there is anything to do. 
Hello Chad,

Why did you state that a model with 1-2 years of observations would not be performing great results?

I am trying to forecast retail sales/demand with SARIMAX using statsmodels. I received also the same error. How do I know which parameters should I be using?

Thank you,
Deniz.
Is exactly two years of data usable for SARIMAX? If I have two complete seasons would I be able to proceed?
> Why did you state that a model with 1-2 years of observations would not be performing great results?

Just because you wouldn't have very many full seasons and you'd risk overfitting the model to a small number of datapoints.  It's similar to why a regression with only a couple of observations may not provide very useful inference.

> Is exactly two years of data usable for SARIMAX? If I have two complete seasons would I be able to proceed?

If that's all the data you have, then that's certainly better than nothing.

However, depending on your model, you may need more than two complete seasons to estimate the starting parameters. See my comments here: https://github.com/statsmodels/statsmodels/issues/4465#issuecomment-381004987
![image](https://user-images.githubusercontent.com/44804259/48552024-84e4cd00-e8d7-11e8-847c-2e2da003e642.png)
Some help to fix up these errors please? 

Thanks in advance.
This is the issue tracker for the Python package “statsmodels”. It seems
like you want to find some help with an R package, so I’m guessing you
would get more help asking on an R mailing list.

I´ll do that then, thanks a lot for your quick response Chad!
Hi Chad
i'm trying the dickey fuller test on yearly data and want to make it stationary but im unable to do it. i have applied that test on monthly and daily but getting an error while applying the ADF on yearly data which i'm going to use for my ARIMA model. your help will be much appreciated.
![capture 3 error](https://user-images.githubusercontent.com/36491007/50190293-a430a900-0320-11e9-9092-e3ee18a3dda3.PNG)
 
Hi @sbnbari can you make a new issue, since your question is about ADF and this issue is about SARIMAX.

Please also copy and paste the error you're receiving (from your screenshot, I can only see the top of the ValueError, but not the whole traceback), and also let us know the output of:

`python -c "import statsmodels; print(statsmodels.__version__)"`

Thanks!
thanks 
check this link please
https://github.com/statsmodels/statsmodels/issues/5432

Resolving patch:
diff --git a/statsmodels/tsa/stattools.py b/statsmodels/tsa/stattools.py
--- a/statsmodels/tsa/stattools.py
+++ b/statsmodels/tsa/stattools.py
@@ -219,13 +219,22 @@ def adfuller(x, maxlag=None, regression="c", autolag='AIC',
     x = np.asarray(x)
     nobs = x.shape[0]
 
+    ntrend = len(regression) if regression != 'nc' else 0
     if maxlag is None:
-        #from Greene referencing Schwert 1989
+        # from Greene referencing Schwert 1989
         maxlag = int(np.ceil(12. * np.power(nobs / 100., 1 / 4.)))
-
+        # -1 for the diff
+        maxlag = min(nobs // 2 - ntrend - 1, maxlag)
+        if maxlag < 0:
+            raise ValueError('sample size is too short to use selected '
+                             'regression component')
+    elif maxlag > nobs // 2 - ntrend - 1:
+        raise ValueError('maxlag must be less than (nobs/2 - 1 - ntrend) '
+                         'where n trend is the number of included '
+                         'deterministic regressors')
     xdiff = np.diff(x)
     xdall = lagmat(xdiff[:, None], maxlag, trim='both', original='in')
-    nobs = xdall.shape[0]  # pylint: disable=E1103
+    nobs = xdall.shape[0]
 
     xdall[:, 0] = x[-nobs - 1:-1]  # replace 0 xdiff with level of x
     xdshort = xdiff[-nobs:]
@@ -237,10 +246,11 @@ def adfuller(x, maxlag=None, regression="c", autolag='AIC',
             fullRHS = add_trend(xdall, regression, prepend=True)
         else:
             fullRHS = xdall
-        startlag = fullRHS.shape[1] - xdall.shape[1] + 1  # 1 for level  # pylint: disable=E1103
-        #search for lag length with smallest information criteria
-        #Note: use the same number of observations to have comparable IC
-        #aic and bic: smaller is better
+        startlag = fullRHS.shape[1] - xdall.shape[1] + 1
+        # 1 for level
+        # search for lag length with smallest information criteria
+        # Note: use the same number of observations to have comparable IC
+        # aic and bic: smaller is better
 
         if not regresults:
             icbest, bestlag = _autolag(OLS, xdshort, fullRHS, startlag,
@@ -253,9 +263,9 @@ def adfuller(x, maxlag=None, regression="c", autolag='AIC',
 
         bestlag -= startlag  # convert to lag not column index
 
-        #rerun ols with best autolag
+        # rerun ols with best autolag
         xdall = lagmat(xdiff[:, None], bestlag, trim='both', original='in')
-        nobs = xdall.shape[0]   # pylint: disable=E1103
+        nobs = xdall.shape[0]
         xdall[:, 0] = x[-nobs - 1:-1]  # replace 0 xdiff with level of x
         xdshort = xdiff[-nobs:]
         usedlag = bestlag


Test patch:
diff --git a/statsmodels/tsa/tests/test_stattools.py b/statsmodels/tsa/tests/test_stattools.py
--- a/statsmodels/tsa/tests/test_stattools.py
+++ b/statsmodels/tsa/tests/test_stattools.py
@@ -876,3 +876,21 @@ def test_innovations_algo_filter_kalman_filter(reset_randomstate):
     assert_allclose(theta[1:, 0], res.filter_results.kalman_gain[0, 0, :-1],
                     atol=atol)
     assert_allclose(llf_obs, res.llf_obs, atol=atol)
+
+
+def test_adfuller_short_series(reset_randomstate):
+    y = np.random.standard_normal(7)
+    res = adfuller(y, store=True)
+    assert res[-1].maxlag == 1
+    y = np.random.standard_normal(2)
+    with pytest.raises(ValueError, match='sample size is too short'):
+        adfuller(y)
+    y = np.random.standard_normal(3)
+    with pytest.raises(ValueError, match='sample size is too short'):
+        adfuller(y, regression='ct')
+
+
+def test_adfuller_maxlag_too_large(reset_randomstate):
+    y = np.random.standard_normal(100)
+    with pytest.raises(ValueError, match='maxlag must be less than'):
+        adfuller(y, maxlag=51)