internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: statsmodels/statsmodels

Issue:
BUG/REF: discrete fit_regularized uses print instead of warning
`print(message)` in base.l1solvers_common

in writing a unit test for GeneralizedPoisson fit_regularized #3789, I get the warning "print" if penalization alpha is small. Given that it is a print I don't see a way to catch it in unit test noise.
I'm testing that low penalization has results close to unpenalized fit 

```
>>> tt.res1.model.fit_regularized(alpha=alpha*0.1)
QC check did not pass for 5 out of 10 parameters
Try increasing solver accuracy or number of iterations, decreasing alpha, or switch solvers
Could not trim params automatically due to failed QC check.  Trimming using trim_mode == 'size' will still work.
...
```


Discussion:
Do we need a special warning or do we use `ConvergenceWarning`?

postponing to 0.10,
AFAIR, qc check needs review and documentation, e.g. #2239


Resolving patch:
diff --git a/statsmodels/base/l1_solvers_common.py b/statsmodels/base/l1_solvers_common.py
--- a/statsmodels/base/l1_solvers_common.py
+++ b/statsmodels/base/l1_solvers_common.py
@@ -2,8 +2,11 @@
 Holds common functions for l1 solvers.
 """
 from __future__ import print_function
+
 import numpy as np
+
 from statsmodels.compat.python import range
+from statsmodels.tools.sm_exceptions import ConvergenceWarning
 
 
 def qc_results(params, alpha, score, qc_tol, qc_verbose=False):
@@ -65,7 +68,9 @@ def qc_results(params, alpha, score, qc_tol, qc_verbose=False):
             ', decreasing alpha, or switch solvers'
         if qc_verbose:
             message += _get_verbose_addon(qc_dict)
-        print(message)
+
+        import warnings
+        warnings.warn(message, ConvergenceWarning)
 
     return passed
 
@@ -135,8 +140,10 @@ def do_trim_params(params, k_params, alpha, score, passed, trim_mode,
     if trim_mode == 'off':
         trimmed = np.array([False] * k_params)
     elif trim_mode == 'auto' and not passed:
-        print("Could not trim params automatically due to failed QC "
-              "check.  Trimming using trim_mode == 'size' will still work.")
+        import warnings
+        msg = "Could not trim params automatically due to failed QC check. " \
+              "Trimming using trim_mode == 'size' will still work."
+        warnings.warn(msg, ConvergenceWarning)
         trimmed = np.array([False] * k_params)
     elif trim_mode == 'auto' and passed:
         fprime = score(params)


Test patch:
Repository: statsmodels/statsmodels

Issue:
Autoregressive model fails for trivial input signal
I am trying to start using the AR models in statsmodels. However, it fails for a trivial starting signal consisting only of ones:

```
from statsmodels.tsa.ar_model import AR
import numpy as np

signal = np.ones(20)
ar_mod = AR(signal)
ar_res = ar_mod.fit(4)

ar_res.predict(4, 60)
```

I also opened a related question on Stack Overflow: (http://stackoverflow.com/questions/28094538/autoregressive-model-using-statsmodels-in-python)



Discussion:
@jseabold 

Based on running the example (see stackoverflow question)

failing with 

```
fcast = mu + np.dot(arparams, endog[i:i+p])
ValueError: matrices are not aligned
```

However, I'm pretty sure the model, for example AR(1) with constant, is not identified. Any solution with the same mean `mu / (1-rho(1)` (*), IIRC, should have perfect fit and prediction with error variance 0.

So we don't have identified parameters, but the prediction should always be 1 in the example.

(*) I guess it should be conditional:  `y = mu + rho(1) * y` for y=1

Use `trend='nc'` in fit. It's the same trying to add a trend to a trend bug.

Potentially same fix as #2043.

Yes, looks the same as 2043 to me, although I didn't look at the details of the related changes.

general topic: what to do with singular designs.

moved to 0.8


Resolving patch:
diff --git a/statsmodels/tsa/tsatools.py b/statsmodels/tsa/tsatools.py
--- a/statsmodels/tsa/tsatools.py
+++ b/statsmodels/tsa/tsatools.py
@@ -103,7 +103,9 @@ def safe_is_const(s):
 
         if np.any(col_const):
             if has_constant == 'raise':
-                raise ValueError("x already contains a constant")
+                msg = "x contains a constant. Adding a constant with " \
+                      "trend='{0}' is not allowed.".format(trend)
+                raise ValueError(msg)
             elif has_constant == 'skip':
                 columns = columns[1:]
                 trendarr = trendarr[:, 1:]


Test patch:
diff --git a/statsmodels/tsa/tests/test_ar.py b/statsmodels/tsa/tests/test_ar.py
--- a/statsmodels/tsa/tests/test_ar.py
+++ b/statsmodels/tsa/tests/test_ar.py
@@ -351,7 +351,7 @@ def test_constant_column_trend():
 
     # Fitting with a constant and maxlag=7 raises because of regressor
     #  collinearity.
-    with pytest.raises(ValueError, match="already contains"):
+    with pytest.raises(ValueError, match="trend='c' is not allowed"):
         model.fit(trend="c")
 
     res = model.fit(trend="nc")
Repository: statsmodels/statsmodels

Issue:
Remove Panel from pandas compat
pandas is about to remove Panel altogether, in which case we'll see errors in compat.pandas.


Discussion:

Resolving patch:
diff --git a/statsmodels/compat/pandas.py b/statsmodels/compat/pandas.py
--- a/statsmodels/compat/pandas.py
+++ b/statsmodels/compat/pandas.py
@@ -8,14 +8,18 @@
 version = LooseVersion(pandas.__version__)
 pandas_lte_0_19_2 = version <= LooseVersion('0.19.2')
 pandas_gt_0_19_2 = version > LooseVersion('0.19.2')
-
+pandas_ge_20_0 = version >= LooseVersion('0.20.0')
+pandas_ge_25_0 = version >= LooseVersion('0.25.0')
 
 try:
     from pandas.api.types import is_numeric_dtype  # noqa:F401
 except ImportError:
     from pandas.core.common import is_numeric_dtype  # noqa:F401
 
-if version >= '0.20':
+if pandas_ge_25_0:
+    from pandas.tseries import frequencies  # noqa:F401
+    data_klasses = (pandas.Series, pandas.DataFrame)
+elif pandas_ge_20_0:
     try:
         from pandas.tseries import offsets as frequencies
     except ImportError:


Test patch:
Repository: statsmodels/statsmodels

Issue:
OLS.fit_regularized() keyword arguments are overridden 
It appears kwargs specified by the user in OLS.fit_regularized() get overridden by the defaults here:

https://github.com/statsmodels/statsmodels/blob/ae3c9133de222f200f11ee3e38de3bb7d93772fd/statsmodels/regression/linear_model.py#L994

Thanks!


Discussion:
Yes, it should be defaults.update(kwargs).  I will fix it now.

Resolving patch:
diff --git a/statsmodels/regression/linear_model.py b/statsmodels/regression/linear_model.py
--- a/statsmodels/regression/linear_model.py
+++ b/statsmodels/regression/linear_model.py
@@ -991,13 +991,13 @@ def fit_regularized(self, method="elastic_net", alpha=0.,
         # Set default parameters.
         defaults = {"maxiter":  50, "cnvrg_tol": 1e-10,
                     "zero_tol": 1e-8}
-        kwargs.update(defaults)
+        defaults.update(kwargs)
 
         if method == "sqrt_lasso":
             from statsmodels.base.elastic_net import (
                 RegularizedResults, RegularizedResultsWrapper
             )
-            params = self._sqrt_lasso(alpha, refit, kwargs["zero_tol"])
+            params = self._sqrt_lasso(alpha, refit, defaults["zero_tol"])
             results = RegularizedResults(self, params)
             return RegularizedResultsWrapper(results)
 


Test patch:
Repository: statsmodels/statsmodels

Issue:
Reference for OLSInfluence Cook's distance
Could you please provide a more detailed reference for this formula? I'm assuming the Eubank is Randall L. Eubank, but can't figure out what book/paper this is from.

https://github.com/statsmodels/statsmodels/blob/7634b409c2d65d9f9fcd77d0d3a047136c8d2dfe/statsmodels/stats/outliers_influence.py#L1241-L1243


Discussion:
This is copied from OLSInfluence, i.e. written a long time ago.
This should be in any related reference, e.g. unit tests for OLS were written against SAS.
I was reading Eubank's textbook at the time, which seems to be what this comment refers to.
https://books.google.ca/books?id=q0ZZDwAAQBAJ&lpg=PP1&pg=PA94#v=onepage&q=cooks%20distance&f=false
I currently don't have access to the book to check.

#4732 is the PR for GLMInfluence and has more references.
It's a bit "eclectic", but I think Pregibon 1981 (for GLM Logit) and Cook and Weissberg were some of the main references for GLMInfluence.

Resolving patch:
diff --git a/statsmodels/stats/outliers_influence.py b/statsmodels/stats/outliers_influence.py
--- a/statsmodels/stats/outliers_influence.py
+++ b/statsmodels/stats/outliers_influence.py
@@ -697,10 +697,17 @@ def get_resid_studentized_external(self, sigma=None):
     # same computation as GLMInfluence
     @cache_readonly
     def cooks_distance(self):
-        """Cooks distance
+        """
+        Cooks distance
 
-        uses original results, no nobs loop
+        Uses original results, no nobs loop
 
+        References
+        ----------
+        .. [*] Eubank, R. L. (1999). Nonparametric regression and spline
+            smoothing. CRC press.
+        .. [*] Cook's distance. (n.d.). In Wikipedia. July 2019, from
+            https://en.wikipedia.org/wiki/Cook%27s_distance
         """
         hii = self.hat_matrix_diag
         # Eubank p.93, 94


Test patch:
Repository: statsmodels/statsmodels

Issue:
BUG/TST tsa periodogram 2-sided
Why is the periodogram 2sided, i.e. symmetric around midpoint?
I thought we only keep one side similar to acf. (it sets the value at the zero index to zero, which sounds more like index zero is the base point.)

bug or feature?

Also I don't see any unit tests for it.

Enhancement request also for arima_process.arma_periodogram:  
Specify/return period length for the frequencies to make it easier to interpret. (I don't remember how to "read" frequencies in [0, pi])
(time series analysis instead of signal processing)



Discussion:

Resolving patch:
diff --git a/statsmodels/tsa/stattools.py b/statsmodels/tsa/stattools.py
--- a/statsmodels/tsa/stattools.py
+++ b/statsmodels/tsa/stattools.py
@@ -885,31 +885,37 @@ def ccf(x, y, unbiased=True):
     return cvf / (np.std(x) * np.std(y))
 
 
-def periodogram(X):
+def periodogram(x):
     """
     Returns the periodogram for the natural frequency of X
 
+    .. deprecated::
+       Use scipy.signal.periodogram instead
+
     Parameters
     ----------
-    X : array_like
+    x : array_like
         Array for which the periodogram is desired.
 
     Returns
     -------
     pgram : array
-        1./len(X) * np.abs(np.fft.fft(X))**2
-
+        1./len(x) * np.abs(np.fft.fft(x))**2
 
     References
     ----------
     .. [*] Brockwell, P.J. and Davis, R.A., 2016. Introduction to time series
         and forecasting. Springer.
     """
-    X = np.asarray(X)
+    # TODO: Remove after 0.11
+    import warnings
+    warnings.warn('periodogram is deprecated and will be removed after 0.11. '
+                  'Use scipy.signal.periodogram instead.', FutureWarning)
+    x = np.asarray(x)
     # if kernel == "bartlett":
     #    w = 1 - np.arange(M+1.)/M   #JP removed integer division
 
-    pergr = 1. / len(X) * np.abs(np.fft.fft(X))**2
+    pergr = 1. / len(x) * np.abs(np.fft.fft(x)) ** 2
     pergr[0] = 0.  # what are the implications of this?
     return pergr
 


Test patch:
diff --git a/statsmodels/tsa/tests/test_stattools.py b/statsmodels/tsa/tests/test_stattools.py
--- a/statsmodels/tsa/tests/test_stattools.py
+++ b/statsmodels/tsa/tests/test_stattools.py
@@ -23,7 +23,8 @@
                                        coint, acovf, kpss,
                                        arma_order_select_ic, levinson_durbin,
                                        levinson_durbin_pacf, pacf_burg,
-                                       innovations_algo, innovations_filter)
+                                       innovations_algo, innovations_filter,
+                                       periodogram)
 
 DECIMAL_8 = 8
 DECIMAL_6 = 6
@@ -911,3 +912,8 @@ def test_adfuller_maxlag_too_large(reset_randomstate):
     y = np.random.standard_normal(100)
     with pytest.raises(ValueError, match='maxlag must be less than'):
         adfuller(y, maxlag=51)
+
+
+def test_periodogram_future_warning(reset_randomstate):
+    with pytest.warns(FutureWarning):
+        periodogram(np.random.standard_normal(100))
Repository: statsmodels/statsmodels

Issue:
Mad fun empty
While reviewing the scipy PR 9637 I started poking around at the behavior of `mad` for an empty array. 
PR in scipy is here:
https://github.com/scipy/scipy/pull/9637/files

It looks like the current behavior could throw a runtime warning in this case. 

This PR should make the `statsmodels.robust.mad` function return a `nan` if the input array is empty. 




Discussion:

Resolving patch:
diff --git a/statsmodels/robust/scale.py b/statsmodels/robust/scale.py
--- a/statsmodels/robust/scale.py
+++ b/statsmodels/robust/scale.py
@@ -13,6 +13,7 @@
 from scipy.stats import norm as Gaussian
 from . import norms
 from statsmodels.tools import tools
+from statsmodels.tools.validation import array_like, float_like
 
 
 def mad(a, c=Gaussian.ppf(3/4.), axis=0, center=np.median):
@@ -39,10 +40,14 @@ def mad(a, c=Gaussian.ppf(3/4.), axis=0, center=np.median):
     mad : float
         `mad` = median(abs(`a` - center))/`c`
     """
-    a = np.asarray(a)
-    if callable(center):
+    a = array_like(a, 'a', ndim=None)
+    c = float_like(c, 'c')
+    if callable(center) and a.size:
         center = np.apply_over_axes(center, a, axis)
-    return np.median((np.abs(a-center))/c, axis=axis)
+    else:
+        center = 0.0
+
+    return np.median((np.abs(a-center)) / c, axis=axis)
 
 
 class Huber(object):


Test patch:
diff --git a/statsmodels/robust/tests/test_scale.py b/statsmodels/robust/tests/test_scale.py
--- a/statsmodels/robust/tests/test_scale.py
+++ b/statsmodels/robust/tests/test_scale.py
@@ -62,6 +62,14 @@ def test_mad(self):
         m = scale.mad(self.X)
         assert_equal(m.shape, (10,))
 
+    def test_mad_empty(self):
+        empty = np.empty(0)
+        assert np.isnan(scale.mad(empty))
+        empty = np.empty((10, 100, 0))
+        assert_equal(scale.mad(empty, axis=1), np.empty((10, 0)))
+        empty = np.empty((100, 100, 0, 0))
+        assert_equal(scale.mad(empty, axis=-1), np.empty((100, 100, 0)))
+
     def test_mad_center(self):
         n = scale.mad(self.X, center=0)
         assert_equal(n.shape, (10,))
Repository: statsmodels/statsmodels

Issue:
RegressionResults.__str__ returns None
Also it doesn't get called when accessed on the RegressionResultsWrapper object, since the wrapper has its own `__str__` method.


Discussion:

Resolving patch:
diff --git a/statsmodels/regression/linear_model.py b/statsmodels/regression/linear_model.py
--- a/statsmodels/regression/linear_model.py
+++ b/statsmodels/regression/linear_model.py
@@ -1520,9 +1520,6 @@ def __init__(self, model, params, normalized_cov_params=None, scale=1.,
         for key in kwargs:
             setattr(self, key, kwargs[key])
 
-    def __str__(self):
-        self.summary()
-
     def conf_int(self, alpha=.05, cols=None):
         """
         Returns the confidence interval of the fitted parameters.


Test patch:
Repository: statsmodels/statsmodels

Issue:
DOC: CountModels docstring are missing offset and exposure
Poisson and NegativeBinomial don't have offset and exposure in the class `Parameters` section.

`__doc__` are set in the Poisson and NegativeBinomial classes not in CountModel



Discussion:
NegativeBinomial already has an extra argument so this is easy to add following that pattern.

PR coming up

GLM docstring is also missing offset and exposure.  Needs check with Kerby's open PR

same for CountModel.predict which has incomplete docstring


Resolving patch:
diff --git a/statsmodels/discrete/discrete_model.py b/statsmodels/discrete/discrete_model.py
--- a/statsmodels/discrete/discrete_model.py
+++ b/statsmodels/discrete/discrete_model.py
@@ -766,7 +766,27 @@ def _get_init_kwds(self):
     def predict(self, params, exog=None, exposure=None, offset=None,
                 linear=False):
         """
-        Predict response variable of a count model given exogenous variables.
+        Predict response variable of a count model given exogenous variables
+
+        Parameters
+        ----------
+        params : array_like
+            Model parameters
+        exog : array_like, optional
+            Design / exogenous data. Is exog is None, model exog is used.
+        exposure : array_like, optional
+            Log(exposure) is added to the linear prediction with
+            coefficient equal to 1. If exposure is not provided and exog
+            is None, uses the model's exposure if present.  If not, uses
+            0 as the default value.
+        offset : array_like, optional
+            Offset is added to the linear prediction with coefficient
+            equal to 1. If offset is not provided and exog
+            is None, uses the model's offset if present.  If not, uses
+            0 as the default value.
+        linear : bool
+            If True, returns the linear predicted values.  If False,
+            returns the exponential of the linear predicted value.
 
         Notes
         -----


Test patch:
Repository: statsmodels/statsmodels

Issue:
DISC: Make `arch` usage "official"?
Going through old issues, ARCH/GARCH come up a lot, and in most of these the user is pointed towards the `arch` package. (#437, #1270, #5441)

Should that delegation be made "official" in some sense?  Among other things that would mean replacing sandbox.tsa.garch with a big note pointing users towards `arch`.


Discussion:
I'm +1 on this.

Resolving patch:
diff --git a/statsmodels/sandbox/tsa/garch.py b/statsmodels/sandbox/tsa/garch.py
--- a/statsmodels/sandbox/tsa/garch.py
+++ b/statsmodels/sandbox/tsa/garch.py
@@ -72,6 +72,8 @@
 Created on Feb 6, 2010
 @author: "josef pktd"
 '''
+import warnings
+
 import numpy as np
 from numpy.testing import assert_almost_equal
 
@@ -86,6 +88,13 @@
 from statsmodels.sandbox import tsa
 
 
+FUTURE_WARN = """
+{0} is deprecated and will be removed after 0.11. Please use the `arch`
+package:
+
+https://pypi.org/project/arch/
+https://github.com/bashtage/arch
+"""
 def sumofsq(x, axis=0):
     """Helper function to calculate sum of squares along first axis"""
     return np.sum(x**2, axis=axis)
@@ -310,6 +319,8 @@ class Garch0(TSMLEModel):
 
     '''
     def __init__(self, endog, exog=None):
+        warnings.warn(FUTURE_WARN.format(self.__class__.__name__),
+                      FutureWarning)
         #need to override p,q (nar,nma) correctly
         super(Garch0, self).__init__(endog, exog)
         #set default arma(1,1)
@@ -535,6 +546,8 @@ class Garch(TSMLEModel):
 
     '''
     def __init__(self, endog, exog=None):
+        warnings.warn(FUTURE_WARN.format(self.__class__.__name__),
+                      FutureWarning)
         #need to override p,q (nar,nma) correctly
         super(Garch, self).__init__(endog, exog)
         #set default arma(1,1)
@@ -674,6 +687,10 @@ class AR(LikelihoodModel):
     Fit methods that use super and broyden do not yet work.
     """
     def __init__(self, endog, exog=None, nlags=1):
+        warnings.warn('Deprecated and will be removed after 0.11. Use AR '
+                      'available in statsmodels.tsa.api',
+                      FutureWarning)
+
         if exog is None:    # extend to handle ADL(p,q) model? or subclass?
             exog = endog[:-nlags]
         endog = endog[nlags:]
@@ -833,6 +850,9 @@ class Arma(LikelihoodModel):
     """
 
     def __init__(self, endog, exog=None):
+        warnings.warn('Deprecated and will be removed after 0.11. Use ARMA '
+                      'available in statsmodels.tsa.api',
+                      FutureWarning)
         #need to override p,q (nar,nma) correctly
         super(Arma, self).__init__(endog, exog)
         #set default arma(1,1)


Test patch:
Repository: statsmodels/statsmodels

Issue:
Performance: ProbPlot always fits 
In statsmodels.graphics.gofplots `class ProbPlot`, method `__init__` always fits, even with `fit=False`.


Discussion:
AFAICS from reading the current code.

The fit is always called, but the fit_params are not used, or at least not added to the distribution instance.
This looks like fit is redundant but the distribution still corresponds to the user provided options without fit.

However, the `fit_params` are used in `sample_percentiles` to standardized the data, so we still get the loc-scale adjustment base on the fitted model. The `sample_percentiles` are used in do_plot, but combined with the non-fitted distribution instance `self.dist`.

I'm not sure what this means, i.e. what does the plot really show if fit=False. Is it just standardizing the plot axis, or actually mixing fitted and preset distributions?


Resolving patch:
diff --git a/statsmodels/graphics/gofplots.py b/statsmodels/graphics/gofplots.py
--- a/statsmodels/graphics/gofplots.py
+++ b/statsmodels/graphics/gofplots.py
@@ -173,8 +173,8 @@ def __init__(self, data, dist=stats.norm, fit=False,
         if isinstance(dist, str):
             dist = getattr(stats, dist)
 
-        self.fit_params = dist.fit(data)
         if fit:
+            self.fit_params = dist.fit(data)
             self.loc = self.fit_params[-2]
             self.scale = self.fit_params[-1]
             if len(self.fit_params) > 2:
@@ -195,10 +195,12 @@ def __init__(self, data, dist=stats.norm, fit=False,
                                 'is:\n{cmd}'.format(cmd=cmd))
             self.loc = loc
             self.scale = scale
+            self.fit_params = np.r_[distargs, loc, scale]
         else:
             self.dist = dist
             self.loc = loc
             self.scale = scale
+            self.fit_params = np.r_[loc, scale]
 
         # propertes
         self._cache = {}


Test patch:
diff --git a/statsmodels/graphics/tests/test_gofplots.py b/statsmodels/graphics/tests/test_gofplots.py
--- a/statsmodels/graphics/tests/test_gofplots.py
+++ b/statsmodels/graphics/tests/test_gofplots.py
@@ -102,6 +102,10 @@ def test_probplot_pltkwargs(self, close_figures):
                              markeredgecolor='white',
                              alpha=0.5)
 
+    def test_fit_params(self):
+        assert self.prbplt.fit_params[-2] == self.prbplt.loc
+        assert self.prbplt.fit_params[-1] == self.prbplt.scale
+
 
 class TestProbPlotLongely(BaseProbplotMixin):
     def setup(self):