internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: statsmodels/statsmodels

Issue:
ARIMA: Check if input dataframe has one column
For a while, I was receiving this unhelpful error message when I was initializing an ARIMA model using a 2-D dataframe:

![untitled](https://cloud.githubusercontent.com/assets/1865885/9126379/69137fde-3c61-11e5-83c3-bdbfbd5e20b2.png)

Essentially, the `TypeError` made me think that my input column names were wrong. But what was really happening was I needed to pass in either a 2-D dataframe with one column, or a pandas Series.

It would be nice to throw an error message if a 2-D dataframe is detected with more than one column.



Discussion:
my guess is that this is still open

(issue was lost without labels)
related: #3988

We need to check unit test coverage and verify what happens in different cases of 1-D, 2-D (nobs, 1) and 2-D (nobs, k), k>1
https://stackoverflow.com/questions/46582490/typeerror-ufunc-add-did-not-contain-a-loop-with-signature-for-arima-model
might have a similar problem with the name creation, but too little information to tell
the stack overflow question has the same exception message as
https://github.com/statsmodels/statsmodels/issues/3269#issuecomment-260459140

possibly related to how series.name is handled (or not enough cases are handled)
as pointed out by @jbrockmendel here
https://github.com/statsmodels/statsmodels/issues/3960#issuecomment-334957669
see followup comments

Resolving patch:
diff --git a/statsmodels/tsa/arima_model.py b/statsmodels/tsa/arima_model.py
--- a/statsmodels/tsa/arima_model.py
+++ b/statsmodels/tsa/arima_model.py
@@ -430,6 +430,10 @@ class ARMA(tsbase.TimeSeriesModel):
     def __init__(self, endog, order, exog=None, dates=None, freq=None,
                  missing='none'):
         super(ARMA, self).__init__(endog, exog, dates, freq, missing=missing)
+        # GH 2575
+        _endog = endog if hasattr(endog, 'ndim') else np.asarray(endog)
+        if (_endog.ndim == 2 and _endog.shape[1] != 1) or _endog.ndim > 2:
+            raise ValueError('endog must be 1-d or 2-d with 1 column')
         exog = self.data.exog  # get it after it's gone through processing
         _check_estimable(len(self.endog), sum(order))
         self.k_ar = k_ar = order[0]


Test patch:
diff --git a/statsmodels/tsa/tests/test_arima.py b/statsmodels/tsa/tests/test_arima.py
--- a/statsmodels/tsa/tests/test_arima.py
+++ b/statsmodels/tsa/tests/test_arima.py
@@ -2432,3 +2432,17 @@ def test_endog_int():
     resf = ARIMA(yf.cumsum(), order=(1, 1, 1)).fit(disp=0)
     assert_allclose(res.params, resf.params, rtol=1e-6, atol=1e-5)
     assert_allclose(res.bse, resf.bse, rtol=1e-6, atol=1e-5)
+
+
+def test_multidim_endog(reset_randomstate):
+    y = np.random.randn(1000, 2)
+    with pytest.raises(ValueError):
+        ARMA(y, order=(1, 1))
+
+    y = np.random.randn(1000, 1, 2)
+    with pytest.raises(ValueError):
+        ARMA(y, order=(1, 1))
+
+    y = np.random.randn(1000, 1, 1)
+    with pytest.raises(ValueError):
+        ARMA(y, order=(1, 1))
Repository: statsmodels/statsmodels

Issue:
Missing \\\ in summary2 as_latex function
I found that the .as_latex() function failed when implemented, due to an issue coming from the line including \midline.  It is a simple fix -- just required an extra \\\ before each \midline call, and it compiles.


Discussion:
I can't find midline any where in the code.  Is the issue with hline or midrule?
midrule. Sorry, typo on my part

Resolving patch:
diff --git a/statsmodels/iolib/summary2.py b/statsmodels/iolib/summary2.py
--- a/statsmodels/iolib/summary2.py
+++ b/statsmodels/iolib/summary2.py
@@ -221,7 +221,7 @@ def as_latex(self):
 
         if self._merge_latex:
             # create single tabular object for summary_col
-            tab = re.sub(to_replace,r'\\midrule\n\\midrule\n', tab)
+            tab = re.sub(to_replace,r'\\midrule\n', tab)
 
         out = '\\begin{table}', title, tab, '\\end{table}'
         out = '\n'.join(out)


Test patch:
diff --git a/statsmodels/base/tests/test_penalized.py b/statsmodels/base/tests/test_penalized.py
--- a/statsmodels/base/tests/test_penalized.py
+++ b/statsmodels/base/tests/test_penalized.py
@@ -94,7 +94,10 @@ def test_summary(self):
 
     @pytest.mark.smoke
     def test_summary2(self):
-        self.res1.summary2()
+        summ = self.res1.summary2()
+        assert isinstance(summ.as_latex(), str)
+        assert isinstance(summ.as_html(), str)
+        assert isinstance(summ.as_text(), str)
 
     def test_numdiff(self):
         res1 = self.res1
diff --git a/statsmodels/iolib/tests/test_summary2.py b/statsmodels/iolib/tests/test_summary2.py
--- a/statsmodels/iolib/tests/test_summary2.py
+++ b/statsmodels/iolib/tests/test_summary2.py
@@ -21,7 +21,6 @@ def test_summarycol(self):
 \hline
       &   y I    &   y II    \\
 \midrule
-\midrule
 const & 7.7500   & 12.4231   \\
       & (1.1058) & (3.1872)  \\
 x1    & -0.7500  & -1.5769   \\
Repository: statsmodels/statsmodels

Issue:
BUG: VARMAX start parameters use mean form for trend / exog, when state space formulation expects intercept form
VARMAX start parameters use mean form for trend / exog, when state space formulation expects intercept form. Need to left-multiply by (I - A_1  - ... - A_p).


Discussion:

Resolving patch:
diff --git a/statsmodels/tsa/statespace/varmax.py b/statsmodels/tsa/statespace/varmax.py
--- a/statsmodels/tsa/statespace/varmax.py
+++ b/statsmodels/tsa/statespace/varmax.py
@@ -330,9 +330,9 @@ def start_params(self):
             trendexog_params = np.linalg.pinv(exog).dot(endog)
             endog -= np.dot(exog, trendexog_params)
             if self.k_trend > 0:
-                trend_params = trendexog_params[:self.k_trend].T.ravel()
+                trend_params = trendexog_params[:self.k_trend].T
             if self.k_endog > 0:
-                exog_params = trendexog_params[self.k_trend:].T.ravel()
+                exog_params = trendexog_params[self.k_trend:].T
 
         # B. Run a VAR model on endog to get trend, AR parameters
         ar_params = []
@@ -380,9 +380,24 @@ def start_params(self):
                          ' found. Using zeros as starting parameters.')
                     ma_params *= 0
 
+        # Transform trend / exog params from mean form to intercept form
+        if self.k_ar > 0 and self.k_trend > 0 or self.mle_regression:
+            coefficient_matrices = (
+                ar_params.reshape(
+                    self.k_endog * self.k_ar, self.k_endog
+                ).T
+            ).reshape(self.k_endog, self.k_endog, self.k_ar).T
+
+            tmp = np.eye(self.k_endog) - np.sum(coefficient_matrices, axis=0)
+
+            if self.k_trend > 0:
+                trend_params = np.dot(tmp, trend_params)
+            if self.mle_regression > 0:
+                exog_params = np.dot(tmp, exog_params)
+
         # 1. Intercept terms
         if self.k_trend > 0:
-            params[self._params_trend] = trend_params
+            params[self._params_trend] = trend_params.ravel()
 
         # 2. AR terms
         if self.k_ar > 0:
@@ -394,7 +409,7 @@ def start_params(self):
 
         # 4. Regression terms
         if self.mle_regression:
-            params[self._params_regression] = exog_params
+            params[self._params_regression] = exog_params.ravel()
 
         # 5. State covariance terms
         if self.error_cov_type == 'diagonal':


Test patch:
Repository: statsmodels/statsmodels

Issue:
BUG?  default shift_zeros in contingency tables
see https://github.com/statsmodels/statsmodels/pull/3576#issuecomment-289192389

it looks like it is adding 0.5 everywhere

```
table = np.asarray([[ 1.,  0.,  10.],
                   [ 2.,  1.,  1.],
                   [ 1.,  0.,  0.]])    

>>> sm.stats.Table(table, shift_zeros=False).table
array([[  1.,   0.,  10.],
       [  2.,   1.,   1.],
       [  1.,   0.,   0.]])

>>> sm.stats.Table(table, shift_zeros=True).table
array([[  1.5,   0.5,  10.5],
       [  2.5,   1.5,   1.5],
       [  1.5,   0.5,   0.5]])
```

Also AFAIK, that wouldn't be recommended because it makes the nominal_association test too conservative.




Discussion:

Resolving patch:
diff --git a/statsmodels/stats/contingency_tables.py b/statsmodels/stats/contingency_tables.py
--- a/statsmodels/stats/contingency_tables.py
+++ b/statsmodels/stats/contingency_tables.py
@@ -114,7 +114,7 @@ def __init__(self, table, shift_zeros=True):
         self.table = np.asarray(table, dtype=np.float64)
 
         if shift_zeros and (self.table.min() == 0):
-            self.table = self.table + 0.5
+            self.table[self.table == 0] = 0.5
 
     def __str__(self):
         s = ("A %dx%d contingency table with counts:\n" %


Test patch:
diff --git a/statsmodels/stats/tests/test_contingency_tables.py b/statsmodels/stats/tests/test_contingency_tables.py
--- a/statsmodels/stats/tests/test_contingency_tables.py
+++ b/statsmodels/stats/tests/test_contingency_tables.py
@@ -106,6 +106,23 @@ def test_local_odds():
                     atol=1e-5, rtol=1e-5)
 
 
+def test_shifting():
+
+    t = np.zeros((3, 4), dtype=np.float64)
+    result = np.full((3, 4), 0.5)
+    assert_equal(ctab.Table(t, shift_zeros=False).table, t)
+    assert_equal(ctab.Table(t, shift_zeros=True).table, result)
+
+    t = np.asarray([[0, 1, 2],
+                    [3, 0, 4],
+                    [5, 6, 0]], dtype=np.float64)
+    r = np.asarray([[0.5, 1, 2],
+                    [3, 0.5, 4],
+                    [5, 6, 0.5]], dtype=np.float64)
+    assert_equal(ctab.Table(t).table, r)
+    assert_equal(ctab.Table(t, shift_zeros=True).table, r)
+
+
 def test_stratified_table_cube():
     # Test that we can pass a rank 3 ndarray or a list of rank 2
     # ndarrays to StratifiedTable and get the same results.
Repository: statsmodels/statsmodels

Issue:
BUG: summary2 use float_format
 when creating `_simple_tables` setting float_format is ignored
 see #1964

However, this is incomplete as fix, since there are three dictionaries in `settings`, that have been appended in different methods or functions.

Should there only be one, or is this on purpose?



Discussion:

Resolving patch:
diff --git a/statsmodels/iolib/summary2.py b/statsmodels/iolib/summary2.py
--- a/statsmodels/iolib/summary2.py
+++ b/statsmodels/iolib/summary2.py
@@ -568,7 +568,7 @@ def _df_to_simpletable(df, align='r', float_format="%.4f", header=True,
 
 def _simple_tables(tables, settings, pad_col=None, pad_index=None):
     simple_tables = []
-    float_format = '%.4f'
+    float_format = settings[0]['float_format'] if settings else '%.4f'
     if pad_col is None:
         pad_col = [0] * len(tables)
     if pad_index is None:


Test patch:
diff --git a/statsmodels/iolib/tests/test_summary2.py b/statsmodels/iolib/tests/test_summary2.py
--- a/statsmodels/iolib/tests/test_summary2.py
+++ b/statsmodels/iolib/tests/test_summary2.py
@@ -40,6 +40,30 @@ def test_summarycol(self):
         actual = '\n%s\n' % actual
         assert_equal(desired, actual)
 
+    def test_summarycol_float_format(self):
+        # Test for latex output of summary_col object
+        desired = r"""
+=================
+       y I   y II
+-----------------
+const 7.7   12.4 
+      (1.1) (3.2)
+x1    -0.7  -1.6 
+      (0.2) (0.7)
+=================
+Standard errors
+in parentheses.
+"""
+        x = [1, 5, 7, 3, 5]
+        x = add_constant(x)
+        y1 = [6, 4, 2, 7, 4]
+        y2 = [8, 5, 0, 12, 4]
+        reg1 = OLS(y1, x).fit()
+        reg2 = OLS(y2, x).fit()
+        actual = summary_col([reg1, reg2], float_format='%0.1f').as_text()
+        actual = '%s\n' % actual
+        assert_equal(actual, desired)
+
     def test_summarycol_drop_omitted(self):
         # gh-3702
         x = [1, 5, 7, 3, 5]
Repository: statsmodels/statsmodels

Issue:
ENH: add cov_type to summary in discrete models
cov_type is still missing in the summary top table in Poisson and, I guess, all other discrete models

GLM has it on the right side table, maybe it should be on the left side instead.


Discussion:

Resolving patch:
diff --git a/statsmodels/discrete/discrete_model.py b/statsmodels/discrete/discrete_model.py
--- a/statsmodels/discrete/discrete_model.py
+++ b/statsmodels/discrete/discrete_model.py
@@ -3604,7 +3604,7 @@ def summary(self, yname=None, xname=None, title=None, alpha=.05,
                      ('Method:', ['MLE']),
                      ('Date:', None),
                      ('Time:', None),
-                     ('converged:', ["%s" % self.mle_retvals['converged']])
+                     ('converged:', ["%s" % self.mle_retvals['converged']]),
                     ]
 
         top_right = [('No. Observations:', None),
@@ -3616,6 +3616,9 @@ def summary(self, yname=None, xname=None, title=None, alpha=.05,
                      ('LLR p-value:', ["%#6.4g" % self.llr_pvalue])
                      ]
 
+        if hasattr(self, 'cov_type'):
+            top_left.append(('Covariance Type:', [self.cov_type]))
+
         if title is None:
             title = self.model.__class__.__name__ + ' ' + "Regression Results"
 
diff --git a/statsmodels/genmod/generalized_linear_model.py b/statsmodels/genmod/generalized_linear_model.py
--- a/statsmodels/genmod/generalized_linear_model.py
+++ b/statsmodels/genmod/generalized_linear_model.py
@@ -1869,7 +1869,7 @@ def summary(self, yname=None, xname=None, title=None, alpha=.05):
                      ]
 
         if hasattr(self, 'cov_type'):
-            top_right.append(('Covariance Type:', [self.cov_type]))
+            top_left.append(('Covariance Type:', [self.cov_type]))
 
         if title is None:
             title = "Generalized Linear Model Regression Results"


Test patch:
diff --git a/statsmodels/discrete/tests/test_count_model.py b/statsmodels/discrete/tests/test_count_model.py
--- a/statsmodels/discrete/tests/test_count_model.py
+++ b/statsmodels/discrete/tests/test_count_model.py
@@ -64,7 +64,9 @@ def test_null(self):
 
     @pytest.mark.smoke
     def test_summary(self):
-        self.res1.summary()
+        summ = self.res1.summary()
+        # GH 4581
+        assert 'Covariance Type:' in str(summ)
 
 
 class TestZeroInflatedModel_logit(CheckGeneric):
Repository: statsmodels/statsmodels

Issue:
OLS predict error with list of floats
I ran into an issue using statsmodels in a project I'm working on where passing the predict method of an OLS object a dictionary of lists with a single float in the lists causes an error.  The error occurs in v0.9.0, but this approach worked in v0.8.0.

I'm not sure if this is a minor bug or my misuse and I solved the issue in my project by always passing a DataFrame.

I'm posting a link to a [notebook](https://github.com/bt-/statsmodels_pred_test) where I tested all the combinations of arguments passed to predict for v0.8.0 and v0.9.0 in case this is a bug or someone else runs into a similar issue.

I think this is related to #4552 and #3611


Discussion:
Yes, it looks like #4552, but the underlying problem is that patsy doesn't convert list to ndarray based on the traceback in your notebook.
Converting to a DataFrame is most likely a detour that needs to create an extra copy of the data.

You could try a dict of ndarrays or a dict of pandas Series instead of a dict of list.

Based on the change in #3611 / #4552 this is now by design, AFAIR, i.e. let patsy handle dicts which might make other options possible.

@thequackdaddy Any opinion on this?


I guess looking at the usecase, my humble opinion is that this might be more of a patsy issue. 

Before, a dict like that would be converted to a pandas data frame, then sent to patsy. Now, we just send the unaltered dict. 

My usecase is really that I need some kind of object with a `__getitem__` method that can return just a 1-d array of data from a large, on-disk file. I created custom object that inherits `dict` to do this. So I can be flexible. 

My view of options...

1. We could convert only dicts to pandas data frames. I could add some kind of checking to the `predict` method to tell if its just a plain old dict or something else. Something else, we send to patsy unaltered. Dicts, we first convert to data frames then send. 
2. We could convince patsy to handle lists as the value. 
3. We just don't support dicts with list values. 

I'm happy with any of the 3 options. 

Resolving patch:
diff --git a/statsmodels/base/model.py b/statsmodels/base/model.py
--- a/statsmodels/base/model.py
+++ b/statsmodels/base/model.py
@@ -962,7 +962,7 @@ def predict(self, exog=None, transform=True, *args, **kwargs):
         If a formula was used, then exog is processed in the same way as
         the original data. This transformation needs to have key access to the
         same variable names, and can be a pandas DataFrame or a dict like
-        object.
+        object that contains numpy arrays.
 
         If no formula was used, then the provided exog needs to have the
         same number of columns as the original exog in the model. No


Test patch:
Repository: statsmodels/statsmodels

Issue:
ARIMA: fit(full_output=False) raises TypeError
Version: 0.6.1 (latest pypi). Fitting a model with full output succeeds, but without it fails with

```

/usr/local/lib/python3.4/site-packages/statsmodels/tsa/arima_model.py in fit(self, start_params, trend, method, transparams, solver, maxiter, full_output, disp, callback, **kwargs)
   1102                                            method, transparams, solver,
   1103                                            maxiter, full_output, disp,
-> 1104                                            callback, **kwargs)
   1105         normalized_cov_params = None  # TODO: fix this?
   1106         arima_fit = ARIMAResults(self, arima_fit._results.params,

/usr/local/lib/python3.4/site-packages/statsmodels/tsa/arima_model.py in fit(self, start_params, trend, method, transparams, solver, maxiter, full_output, disp, callback, **kwargs)
    934 
    935         if transparams:  # transform parameters back
--> 936             params = self._transparams(params)
    937 
    938         self.transparams = False  # so methods don't expect transf.

/usr/local/lib/python3.4/site-packages/statsmodels/tsa/arima_model.py in _transparams(self, params)
    599         # just copy exogenous parameters
    600         if k != 0:
--> 601             newparams[:k] = params[:k]
    602 
    603         # AR Coeffs

TypeError: 'NoneType' object is not subscriptable
```

At least you should raise a more friendly error in such cases.



Discussion:

Resolving patch:
diff --git a/statsmodels/base/model.py b/statsmodels/base/model.py
--- a/statsmodels/base/model.py
+++ b/statsmodels/base/model.py
@@ -498,8 +498,8 @@ def hess(params, *args):
         mlefit = LikelihoodModelResults(self, xopt, Hinv, scale=1., **kwds)
 
         # TODO: hardcode scale?
+        mlefit.mle_retvals = retvals
         if isinstance(retvals, dict):
-            mlefit.mle_retvals = retvals
             if warn_convergence and not retvals['converged']:
                 from warnings import warn
                 from statsmodels.tools.sm_exceptions import ConvergenceWarning


Test patch:
diff --git a/statsmodels/tsa/tests/test_arima.py b/statsmodels/tsa/tests/test_arima.py
--- a/statsmodels/tsa/tests/test_arima.py
+++ b/statsmodels/tsa/tests/test_arima.py
@@ -2450,3 +2450,11 @@ def test_multidim_endog(reset_randomstate):
     y = np.random.randn(1000, 1, 1)
     with pytest.raises(ValueError):
         ARMA(y, order=(1, 1))
+
+
+def test_arima_no_full_output():
+    # GH 2752
+    endog = y_arma[:, 6]
+    mod = ARIMA(endog, (1, 0, 1))
+    res = mod.fit(trend="c", disp=-1, full_output=False)
+    assert res.mle_retvals is None
Repository: statsmodels/statsmodels

Issue:
sigma_est_kpss sum(foo) vs foo.sum()
tsa.stattools._sigma_est_kpss starts off with `s_hat = sum(resids**2)`.  I expect that the author didn't realize this is different from `(resids**2).sum()`

```
In [2]: resids = np.random.randn(10**5)

In [3]: sum(resids**2)
Out[3]: 99483.85416165346

In [4]: (resids**2).sum()
Out[4]: 99483.85416165383

In [5]: r2 = resids**2

In [6]: %timeit sum(r2)
100 loops, best of 3: 10.6 ms per loop

In [7]: %timeit r2.sum()
10000 loops, best of 3: 47.2 µs per loop
```


Discussion:

Resolving patch:
diff --git a/statsmodels/tsa/stattools.py b/statsmodels/tsa/stattools.py
--- a/statsmodels/tsa/stattools.py
+++ b/statsmodels/tsa/stattools.py
@@ -1691,11 +1691,12 @@ def kpss(x, regression='c', lags=None, store=False):
         lags = int(lags)
 
     if lags > nobs:
-        raise ValueError("lags ({}) must be <= number of observations ({})".format(lags, nobs))
+        raise ValueError("lags ({}) must be <= number of observations ({})"
+                         .format(lags, nobs))
 
     pvals = [0.10, 0.05, 0.025, 0.01]
 
-    eta = sum(resids.cumsum()**2) / (nobs**2)  # eq. 11, p. 165
+    eta = np.sum(resids.cumsum()**2) / (nobs**2)  # eq. 11, p. 165
     s_hat = _sigma_est_kpss(resids, nobs, lags)
 
     kpss_stat = eta / s_hat
@@ -1727,7 +1728,7 @@ def _sigma_est_kpss(resids, nobs, lags):
     Computes equation 10, p. 164 of Kwiatkowski et al. (1992). This is the
     consistent estimator for the variance.
     """
-    s_hat = sum(resids**2)
+    s_hat = np.sum(resids**2)
     for i in range(1, lags + 1):
         resids_prod = np.dot(resids[i:], resids[:nobs - i])
         s_hat += 2 * resids_prod * (1. - (i / (lags + 1.)))
@@ -1741,7 +1742,7 @@ def _kpss_autolag(resids, nobs):
     (1994), and Schwert (1989). Assumes Bartlett / Newey-West kernel.
     """
     covlags = int(np.power(nobs, 2. / 9.))
-    s0 = sum(resids**2) / nobs
+    s0 = np.sum(resids**2) / nobs
     s1 = 0
     for i in range(1, covlags + 1):
         resids_prod = np.dot(resids[i:], resids[:nobs - i])


Test patch:
Repository: statsmodels/statsmodels

Issue:
BUG: Future pandas deprecation in test_custom_index
#### Describe the bug

A future change in pandas is creating a failure on Pre-testing. This is in master.
see https://travis-ci.org/statsmodels/statsmodels/jobs/542484637

File: `statsmodels/tsa/base/tests/test_tsa_indexes.py`
test: `test_custom_index`


Discussion:

Resolving patch:
diff --git a/statsmodels/tsa/base/tsa_model.py b/statsmodels/tsa/base/tsa_model.py
--- a/statsmodels/tsa/base/tsa_model.py
+++ b/statsmodels/tsa/base/tsa_model.py
@@ -286,10 +286,17 @@ def _get_index_loc(self, key, base_index=None):
                 key = nobs + key
             # Out-of-sample (note that we include key itself in the new index)
             elif key > nobs - 1:
-                stop = base_index._start + (key + 1) * base_index._step
-                index = RangeIndex(start=base_index._start,
+                # See gh5835. Remove the except after pandas 0.25 required.
+                try:
+                    base_index_start = base_index.start
+                    base_index_step = base_index.step
+                except AttributeError:
+                    base_index_start = base_index._start
+                    base_index_step = base_index._step
+                stop = base_index_start + (key + 1) * base_index_step
+                index = RangeIndex(start=base_index_start,
                                    stop=stop,
-                                   step=base_index._step)
+                                   step=base_index_step)
 
         # Special handling for Int64Index
         if (not range_index and int_index and not date_index and


Test patch: