GitHub repository: statsmodels/statsmodels

Issues:
issue_9165: state: open
title: BUG: Ensure ARIMA simulation is reproducable
body: Ensure rpoducability by creating random_state earlier and using consistently

- [X] closes #9162 
- [X] tests added / passed. 
- [X] code/documentation is well formatted.  
- [X] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

<details>


**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in main and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/main -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>

user: bashtage
issue_9164: state: open
title: statsmodels.tools.tools.add_constant documentation of prepend parameter does not match behavior
body: #### Describe the bug

I discovered this while answering the question posted at https://stackoverflow.com/questions/78051181/cannot-get-a-constant-when-using-statsmodels-add-constant/78051263

The documentation of the prepend parameter for `statmodels.tools.tools.add_contant()` states
    prepend : bool
        If true, the constant is in the first column.  Else the constant is
        appended (last column).

However, the function signature clearly has a default value of True set for the parameter:
`def add_constant(data, prepend=True, has_constant='skip'):`

Looking at Git Blame, it appears that the prepend=True default parameter has always been there (at least 15 years).  
However, the description of the parameter was changed 8 years ago from:

> True and the constant is prepended rather than appended.

At the same time as that change, the function description was changed from:
> This appends a column of ones to an array if prepend==False.
to
> Adds a column of ones to an array


#### Code Sample, a copy-pastable example if possible


```python
import numpy as np
import statsmodels.api as sm

# Assuming future_time_points is an array like this
future_time_points = np.array([2010])

# Stacking future_time_points and its square
stacked_array = np.column_stack((future_time_points, future_time_points ** 2))

# Adding a constant column to the stacked array
array_with_constant = sm.add_constant(stacked_array, has_constant="add")

print(array_with_constant)
```
<details>

I have confirmed the issue still exists in the main branch of the repository.
</details>



#### Expected Output

I expected the new constant to be appended in the last column position:
[[2.0100e+03 4.0401e+06 1.0000e+00]]

It was instead pre-pended in the first column:
[[1.0000e+00 2.0100e+03 4.0401e+06]]

**However,** it appears the code behavior is correct, and the documentation is wrong.
  Recommend changing the function description to:
> Prepends a column of ones to an array

And change the description of the prepend parameter to:
> prepend : bool
        If True (default), the constant is in the first column.  If False. append the constant as the last column.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]
INSTALLED VERSIONS
------------------
Python: 3.11.1.final.0

statsmodels
===========

Installed: 0.14.1 (C:\Users\markm\PythonVenv\scratch\.venv\Lib\site-packages\statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.26.4 (C:\Users\markm\PythonVenv\scratch\.venv\Lib\site-packages\numpy)
scipy: 1.12.0 (C:\Users\markm\PythonVenv\scratch\.venv\Lib\site-packages\scipy)
pandas: 2.2.1 (C:\Users\markm\PythonVenv\scratch\.venv\Lib\site-packages\pandas)
    dateutil: 2.8.2 (C:\Users\markm\PythonVenv\scratch\.venv\Lib\site-packages\dateutil)
patsy: 0.5.6 (C:\Users\markm\PythonVenv\scratch\.venv\Lib\site-packages\patsy)

Optional Dependencies
=====================

matplotlib: Not installed
cvxopt: Not installed
joblib: Not installed

Developer Tools
================

IPython: 8.21.0 (C:\Users\markm\PythonVenv\scratch\.venv\Lib\site-packages\IPython)
    jinja2: 3.1.2 (C:\Users\markm\PythonVenv\scratch\.venv\Lib\site-packages\jinja2)
sphinx: Not installed
    pygments: 2.17.2 (C:\Users\markm\PythonVenv\scratch\.venv\Lib\site-packages\pygments)
pytest: 7.4.3 (C:\Users\markm\PythonVenv\scratch\.venv\Lib\site-packages\pytest)
virtualenv: Not installed

</details>

user: nighanxiety
issue_9163: state: open
title: ENH: distance measures for two covariance, correlation or scatter matrices
body: I'm running into various measure for the closeness of two covariance or scatter matrices.

We should get a function with a collection of those.

current usecase: checking how well a robust cov or scatter estimate estimates the true/dgp covariance or how it compares to standard pearson covariance in the clean gaussian case.

The one I usually use is mse (Frobenious norm of difference cov1 - cov2 divided by nobs).
Others are Kullback-Leibler distance, or some on range of eigenvalues or trace of cov1^{-1} @ cov2.
Note KL and some of the others are not "norms", e.g. symmetry may not hold.

Other options should be to compare shapes only, i.e. scatter matrices with arbitrary scale (e.g. standardize scaling to satisfy trace or determinant constraint).

https://stats.stackexchange.com/questions/14673/measures-of-similarity-or-distance-between-two-covariance-matrices

e.g. based on second answer on stats.stackexchange
from https://www.researchgate.net/publication/4194743_Correlation_Matrix_Distance_a_Meaningful_Measure_for_Evaluation_of_Non-Stationary_MIMO_Channels equation (4)

not subtracting means shows smaller distance than using np.corrcoeff
applied to x1, x2 being raveled covariance estimates.

Note: corrdist is zero if two cov are the same except for a scale factor, i.e. should be fine for scatter and shape matrices.
```
def corrdist(x1, x2):
    """correlation distance between two vectors

    This correspond to one minus the correlation coefficient under the assumption that means are zero.
    """
    if x1.ndim !=1 or x2.ndim !=1:
        raise ValueError("data should be 1-dimensional")
    cross = x1.T @ x2 
    s1 = (x1**2).sum(0)
    s2 = (x1**2).sum(0)
    cmd = 1 - cross / np.sqrt(s1 * s2)
    return cmd
```
user: josef-pkt
issue_9162: state: open
title: `ARIMAResults.simulate` doesn't respect `random_state`
body: #### Describe the bug

The method `ARIMAResults.simulate` doesn't seem to respect the parameter `random_state`. It returns different results for a fixed random state parameter.

#### Code Sample


```python
from statsmodels.tsa.arima.model import ARIMA
import numpy as np

x = np.random.randn(100)
mod = ARIMA(x, order=(1, 0, 0))
res = mod.fit()

for i in range(5):
    print(i, res.simulate(1, random_state=7))
```


#### Expected Output

For me, this prints a different number for each of the iterations. My expectation, however, is that the same number is generated on each invocation of `simulate` because `random_state` is fixed.


#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

INSTALLED VERSIONS
------------------
Python: 3.10.12.final.0
OS: Darwin 23.3.0 Darwin Kernel Version 23.3.0: Wed Dec 20 21:30:44 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T6000 arm64
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8

statsmodels
===========

Installed: 0.14.1 (/Users/adaitche/homebrew/lib/python3.10/site-packages/statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.24.1 (/Users/adaitche/homebrew/lib/python3.10/site-packages/numpy)
scipy: 1.10.0 (/Users/adaitche/homebrew/lib/python3.10/site-packages/scipy)
pandas: 1.5.3 (/Users/adaitche/homebrew/lib/python3.10/site-packages/pandas)
    dateutil: 2.8.2 (/Users/adaitche/homebrew/lib/python3.10/site-packages/dateutil)
patsy: 0.5.6 (/Users/adaitche/homebrew/lib/python3.10/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.6.3 (/Users/adaitche/homebrew/lib/python3.10/site-packages/matplotlib)
    backend: MacOSX
cvxopt: Not installed
joblib: Not installed

Developer Tools
================

IPython: 8.9.0 (/Users/adaitche/homebrew/lib/python3.10/site-packages/IPython)
    jinja2: 3.1.2 (/Users/adaitche/homebrew/lib/python3.10/site-packages/jinja2)
sphinx: Not installed
    pygments: 2.14.0 (/Users/adaitche/homebrew/lib/python3.10/site-packages/pygments)
pytest: 7.4.3 (/Users/adaitche/homebrew/lib/python3.10/site-packages/pytest)
virtualenv: Not installed

</details>

user: adaitche
issue_9161: state: open
title: Zivot Andrews Breakpoint Date
body: Hi, 

I've recently used the zivot andrews unit root test provided in the statsmodels package. However, there is something that I would like to confirm.

Using the Zivot Andrews unit root test across Stata(zandrews), eViews, R([ur.za](http://ur.za/)) and Statsmodels, I found that all four provides the same t-statistics value. However, the breakpoint date varies.
Specifically, the Stata and eViews breakpoint date is always 1 period ahead of the one given by R & Statsmodels. Is this normal?

Thank you for your time
user: KrisKarul
issue_9160: state: open
title: Constraining state-space model parameters between certain bounds 
body: Hi,

I am currently using statsmodels to make state-space models using statsmodels.tsa.statespace.structural.UnobservedComponents and fit these models to my data using maximum likelihood estimation. However, an unconstrained fit leads to overfitting for some of my time series that are very eratic. This occurs because one of the model variance parameters (either sigma2.trend or sigma2.cycle depending on the dataset) becomes very large, allowing for large fluctuations in the corresponding model component, resulting in overfitting. At the same time, the sigma2.irregular parameter, which can be seen as the variance of the noise, is set to almost zero.

I want to constrain my sigma2.irregular parameter to take on values between certain bounds, but when using statsmodels' .fit_constrained functionality, it seems like you can only fix parameters to one given value. Is there any other way to constrain parameters to be somewhere between a minimum and maximum value?
user: matsb0
issue_9159: state: open
title: Is it possible to use quantile regression for produce power law fit?
body: I would like to so a nonlinear quantile regression, using a power law fit.

**This is my current Code, where i can do a u-curved linear quantile regression like (Y = b o + b1X1 + b2X12)...**

df_org = pd.read_csv('Point1.csv',sep=';')
 
df=df_org.dropna()
 
x=df['Tp']
y=df['Hm0']
 
#Plot Scatter
fig = plt.figure()
fig.suptitle('Hm0 versus Tp')
z=density_scatter(x,y,bins=[10,10])
plt.scatter(x,y,c=z,cmap='viridis')
plt.colorbar()
 
 reg_exp = 'Hm0 ~ Tp + I(Tp**2)' #-For linear regression of second degree
 
#Carve out the y and X matrices
y_train, X_train = dmatrices(reg_exp, df, return_type='dataframe')
 
#Plot all regression lines for multiple quantiles
colors = ['orange', 'yellow', 'violet']
i=0
handles = []
quantiles = [0.05, 0.5, 0.95]
for q in quantiles:
    quantile_model = smf.quantreg(formula=reg_exp, data=df)
    quantile_model_results = quantile_model.fit(q=q)
    y_pred_quantile = quantile_model_results.predict(X_train)
 
    label=get_quantileLabel(q,quantile_model_results)
 
    quantile, = plt.plot(X_train['Tp'].sort_values(), y_pred_quantile.sort_values(),lw=1,zorder=2,
        color=colors[i], linestyle='--',  label=label)
    i = i+1
    handles.append(quantile)
 
plt.legend(handles=handles)
plt.show()

![357779ab-3945-42b9-bef4-ad1bd493f245](https://github.com/statsmodels/statsmodels/assets/51142185/20d652bf-1e83-40b9-ad32-759f5b0b09b5)

**I would like to change  the linear regression to a nonlinear regression ( Power (convex): Theta1 * X^Theta2)? Is it possible?**

user: Ana-Bastos
issue_9158: state: open
title: DOC: Fix broken in `linear_regression_diagnostics_plots`
body: - [x] closes #9157 
- [ ] tests added / passed. 
- [ ] code/documentation is well formatted.  
- [ ] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

<details>


**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in main and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/main -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>



<img width="928" alt="image" src="https://github.com/statsmodels/statsmodels/assets/77875500/8109a258-ed9f-4391-b0b1-4e4426d4ee0f">

Previously, the page would break due to an error in the code, specifically because the 'seaborn-paper' style was sometimes missing. To address this, I replaced it with another style and added a fallback to 'default' for added protection.

For now, it works well in my local with `python==3.10` and `matplotlib==3.8.3`
user: luke396
issue_9157: state: open
title: DOC: `linear_regression_diagnostics_plots` in `examples` broken
body: https://www.statsmodels.org/stable/examples/notebooks/generated/linear_regression_diagnostics_plots.html

<img width="939" alt="image" src="https://github.com/statsmodels/statsmodels/assets/77875500/3e9720e3-2b80-45f1-b73f-a954d0599bef">

user: luke396
issue_9156: state: open
title: Zero Inflated Negative Binomial model does not converge
body: hi!

I would like to refer to @jfhawkin [comment](https://github.com/statsmodels/statsmodels/issues/8300#issuecomment-1601325502) and ask if you could look at the convergence of ZINB model because it seems to not be able to converge for zero-inflated count data.

Here is a small example I made:
```
import pandas as pd
import numpy as np
import statsmodels.api as sm
from scipy.stats import nbinom

### Generate random data
size = 100  # size of the data
n = 10  # number of successes for negative binomial
p = 0.5  # probability of success for negative binomial and Poisson
zero_prob = 0.5  # probability of zero for zero-inflated distributions

zinb_data = nbinom.rvs(n, p, size=size)
zinb_data[np.random.random(size) < zero_prob] = 0

df = pd.DataFrame({
            'nb': np.nan,
            'zinb': zinb_data,
        })
        
x = df['zinb']
intercept = sm.add_constant(np.ones(len(x)))

mle_zinb = sm.ZeroInflatedNegativeBinomialP(x, intercept).fit(method='bfgs', maxiter=5000, gtol=1e-12)
```

In my tests, I've noticed that depending on the seed sometimes model is able to converge.
I use this setting of the model because I'm trying to replicate the result from [zeroinfl()](https://arc.net/l/quote/nfqecugg) method in R.

I appreciate your time, please, let me know if the example is not clear, I would be happy to explain it better.
Thank you!

OUTPUT:
> gtol=1e-12)= sm.ZeroInflatedNegativeBinomialP(x, intercept).fit(method='bfgs', maxiter=5000 
> python3.10/site-packages/statsmodels/discrete/discrete_model.py:3935: RuntimeWarning: invalid value encountered in log
>   a1 * np.log(a1) + y * np.log(mu) -
> /python3.10/site-packages/statsmodels/discrete/count_model.py:167: RuntimeWarning: invalid value encountered in log
>   llf[zero_idx] = (np.log(w[zero_idx] +
> python3.10/site-packages/statsmodels/discrete/discrete_model.py:3972: RuntimeWarning: invalid value encountered in log
>   dgterm = dgpart + np.log(a1 / a2) + 1 - a3 / a2
> python3.10/site-packages/statsmodels/discrete/discrete_model.py:4325: RuntimeWarning: overflow encountered in exp
>   return np.exp(linpred)
> python3.10/site-packages/statsmodels/discrete/discrete_model.py:3935: RuntimeWarning: invalid value encountered in multiply
>   a1 * np.log(a1) + y * np.log(mu) -
> python3.10/site-packages/statsmodels/discrete/discrete_model.py:3972: RuntimeWarning: divide by zero encountered in log
>   dgterm = dgpart + np.log(a1 / a2) + 1 - a3 / a2
> python3.10/site-packages/statsmodels/discrete/discrete_model.py:3975: RuntimeWarning: invalid value encountered in multiply
>   dparams = (a4 * dgterm -
> python3.10/site-packages/statsmodels/discrete/discrete_model.py:3935: RuntimeWarning: invalid value encountered in log
>   a1 * np.log(a1) + y * np.log(mu) -
> python3.10/site-packages/statsmodels/discrete/count_model.py:167: RuntimeWarning: invalid value encountered in log
>   llf[zero_idx] = (np.log(w[zero_idx] +
> python3.10/site-packages/statsmodels/discrete/discrete_model.py:3972: RuntimeWarning: invalid value encountered in log
>   dgterm = dgpart + np.log(a1 / a2) + 1 - a3 / a2
> python3.10/site-packages/statsmodels/discrete/discrete_model.py:3972: RuntimeWarning: invalid value encountered in log
>   dgterm = dgpart + np.log(a1 / a2) + 1 - a3 / a2
> python3.10/site-packages/statsmodels/discrete/discrete_model.py:3935: RuntimeWarning: invalid value encountered in log
>   a1 * np.log(a1) + y * np.log(mu) -
> python3.10/site-packages/statsmodels/discrete/count_model.py:167: RuntimeWarning: invalid value encountered in log
>   llf[zero_idx] = (np.log(w[zero_idx] +
> Warning: Desired error not necessarily achieved due to precision loss.
>          Current function value: nan
>          Iterations: 16
>          Function evaluations: 127
>          Gradient evaluations: 127
> python3.10/site-packages/statsmodels/base/model.py:595: HessianInversionWarning: Inverting hessian failed, no bse or cov_params available
>   warnings.warn('Inverting hessian failed, no bse or cov_params '
> python3.10/site-packages/statsmodels/base/model.py:607: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
>   warnings.warn("Maximum Likelihood optimization failed to "
user: Vlasovets
issue_9155: state: open
title: BIC still being computed using deviance despite deprecation warning for version 0.13.0
body: #### Describe the bug

Very straightforward: there is a deprecation warning for version 0.13.0 that is still in the code base. 

According to [this](https://github.com/statsmodels/statsmodels/blob/37bd2e421bad61bc54e8260b86a10a68d17892b3/statsmodels/genmod/generalized_linear_model.py#L1919), default BIC should be computed using log-likelihood, but it's still being calculated using deviance. 

There is a comment explicitly saying **"# Remove after 0.13 when bic changes to bic llf"** in this [line](https://github.com/statsmodels/statsmodels/blob/37bd2e421bad61bc54e8260b86a10a68d17892b3/statsmodels/genmod/generalized_linear_model.py#L63C1-L63C48).

#### Code Sample, a copy-pastable example if possible

It doesn't seem necessary in this case, but let me know if needed.

<details>

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `main`. If your problem has been fixed in an unreleased version, you might be able to use `main` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the main branch of this repository? It helps the limited resources if we know problems exist in the current main branch so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

Warning message should be removed and BIC should use log-likelihood by default.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]

INSTALLED VERSIONS
------------------
Python: 3.11.6.final.0
statsmodels
===========
Installed: 0.14.1 
Required Dependencies
=====================
cython: Not installed
numpy: 1.26.4 
scipy: 1.12.0 
pandas: 2.2.0 
    dateutil: 2.8.2 
patsy: 0.5.6 
Optional Dependencies
=====================
matplotlib: 3.8.2 
    backend: TkAgg 
cvxopt: Not installed
joblib: 1.3.2 
Developer Tools
================
IPython: Not installed
    jinja2: 3.1.3 
sphinx: 7.2.6 
    pygments: 2.17.2 
pytest: 8.0.0 
virtualenv: 20.25.0 

</details>

user: diegodebrito
issue_9152: state: open
title: ENH: outlier robust extreme value statistics and distributions
body: parking some references, I did not look at them

topic: outlier robust estimation of highly skewed distributions.
related issues, e.g. #8690 #9148 #9146 #9128

Most standard theory for outlier robust estimation requires (approximate) symmetry of the distribution.
Extreme value and lifetime models require highly asymmetric, skewed distribution.
So, this looks like a good literature to see what outlier robust approaches are used in the highly skewed cases.
(e.g. minimum distance is used based on some abstracts #7412 #7440)
(I'm more interested in generic approaches, we don't have EVT yet.
Literature search triggered by bias correction for m-estimators that are not fisher consistent in Dupuis and Morgenthaler in #9148)

some disadvantage: literature is largely for estimating distribution (constant) parameters and not for regression.

one possibly interesting topic: distinguishing heavy tailed "observations" and outliers

Alfons, Andreas, Matthias Templ, and Peter Filzmoser. “Robust Estimation of Economic Indicators from Survey Samples Based on Pareto Tail Modelling.” Journal of the Royal Statistical Society Series C: Applied Statistics 62, no. 2 (March 1, 2013): 271–86. https://doi.org/10.1111/j.1467-9876.2012.01063.x.
Andria, Joseph. “A Computational Proposal for a Robust Estimation of the Pareto Tail Index: An Application to Emerging Markets.” Applied Soft Computing 114 (January 1, 2022): 108048. https://doi.org/10.1016/j.asoc.2021.108048.
Bhattacharya, Shrijita, Francois Kamper, and Jan Beirlant. “Outlier Detection Based on Extreme Value Theory and Applications.” Scandinavian Journal of Statistics 50, no. 3 (2023): 1466–1502. https://doi.org/10.1111/sjos.12665.
Dupuis, D. J., and C. A. Field. “Robust Estimation of Extremes.” Canadian Journal of Statistics 26, no. 2 (1998): 199–215. https://doi.org/10.2307/3315505.
Dupuis, Debbie J., and Stephan Morgenthaler. “Robust Weighted Likelihood Estimators with an Application to Bivariate Extreme Value Problems.” Canadian Journal of Statistics 30, no. 1 (2002): 17–36. https://doi.org/10.2307/3315863.
Dupuis, D.J. “Exceedances over High Thresholds: A Guide to Threshold Selection.” Extremes 1, no. 3 (January 1, 1999): 251–61. https://doi.org/10.1023/A:1009914915709.
Fedotenkov, Igor. “A Review of More than One Hundred Pareto-Tail Index Estimators.” Statistica 80, no. 3 (2020): 245–99. https://doi.org/10.6092/issn.1973-2201/9533.
Gomes, M. Ivette, Miranda Cristina, and Manuela Souto de Miranda. “A Note on Robust Estimation of the Extremal Index.” In Nonparametric Statistics, edited by Michele La Rocca, Brunero Liseo, and Luigi Salmaso, 213–25. Springer Proceedings in Mathematics & Statistics. Cham: Springer International Publishing, 2020. https://doi.org/10.1007/978-3-030-57306-5_20.
He, Xuming, and Wing K. Fung. “Method of Medians for Lifetime Data with Weibull Models.” Statistics in Medicine 18, no. 15 (August 15, 1999): 1993–2009. [https://doi.org/10.1002/(SICI)1097-0258(19990815)18:15<1993::AID-SIM165>3.0.CO;2-H](https://doi.org/10.1002/(SICI)1097-0258(19990815)18:15%3C1993::AID-SIM165%3E3.0.CO;2-H).
Juárez, Sergio F., and William R. Schucany. “Robust and Efficient Estimation for the Generalized Pareto Distribution.” Extremes 7, no. 3 (September 1, 2004): 237–51. https://doi.org/10.1007/s10687-005-6475-6.
Lala Bouali, Dalal, Fatah Benatia, Brahim Brahimi, and Christophe Chesneau. “Robust Estimator of Conditional Tail Expectation of Pareto-Type Distribution.” Journal of Statistical Theory and Practice 15, no. 1 (January 3, 2021): 16. https://doi.org/10.1007/s42519-020-00153-0.
Minkah, Richard, Tertius de Wet, and Abhik Ghosh. “Robust Estimation of Pareto-Type Tail Index through an Exponential Regression Model.” Communications in Statistics - Theory and Methods 52, no. 2 (January 17, 2023): 479–98. https://doi.org/10.1080/03610926.2021.1916530.
Peng, Liang, and A.H. Welsh. “Robust Estimation of the Generalized Pareto Distribution.” Extremes 4, no. 1 (March 1, 2001): 53–65. https://doi.org/10.1023/A:1012233423407.
Vandewalle, B., J. Beirlant, A. Christmann, and M. Hubert. “A Robust Estimator for the Tail Index of Pareto-Type Distributions.” Computational Statistics & Data Analysis 51, no. 12 (August 15, 2007): 6252–68. https://doi.org/10.1016/j.csda.2007.01.003.

user: josef-pkt
labels: comp-extremes: extreme value theory
issue_9151: state: open
title: ENH: make sure calling summary after remove data works if cached
body: see https://github.com/statsmodels/statsmodels/issues/9147#issuecomment-1939106367

currently `summary()` does not work for all models after remove_data has been called even when summary was called before remove_data

unit tests for remove_data currently only check that `predict` works after remove_data, but not summary or any other results methods.

- GLM, bug see #9147
- OLS: diagnostics not cached, e.g. jarque_bera


user: josef-pkt
issue_9148: state: open
title: ENH roadmap outlier robust GLM, Poisson, Logit, asymmetric MLE
body: 

- [ ] RobGLM: M-estimator explicit for logit, Poisson and Gamma
   - explicit formulas for expectation exist for those 3 distributions. good summary and formulas in Heritier et al ch.5 and appendix 5
- [ ] generic computation, approximation of fisher correction term, e.g. https://github.com/statsmodels/statsmodels/issues/9146#issuecomment-1937772979
- [ ] bias correction, mentioned in Heritier et al ch. 5 referring to Dupuis and Morgenthaler 2002
- [ ] trimmed likelihood, with or without fisher correction
- [ ] high breakdown, redescending norm functions
- [ ] ...

tbc

Heritier, Stephane, Eva Cantoni, Samuel Copt, and Maria-Pia Victoria-Feser. Robust Methods in Biostatistics. Wiley Series in Probability and Statistics. Chichester, UK: John Wiley & Sons, Ltd, 2009. https://doi.org/10.1002/9780470740538.
 
Dupuis, Debbie J., and Stephan Morgenthaler. “Robust Weighted Likelihood Estimators with an Application to Bivariate Extreme Value Problems.” Canadian Journal of Statistics 30, no. 1 (2002): 17–36. https://doi.org/10.2307/3315863.

user: josef-pkt
issue_9147: state: open
title: BUG/ENH: Saving a model with `remove_data=True` breaks `.summary()`
body: It would be great if a model saved with the data removed would still be able to produce the summary statistics. Potentially it's worthwhile keeping the minimum amount of data in order to reproduce these stats? Especially when using the formula API in which the original data is still saved, see [here](https://github.com/statsmodels/statsmodels/issues/8947) and [here](https://github.com/statsmodels/statsmodels/issues/7494). 


```python
import numpy as np
import statsmodels.api as sm
import statsmodels.formula.api as smf

# load "growth curves pf pigs" dataset
my_data = sm.datasets.get_rdataset("dietox", "geepack").data

# make GLM model
my_formula = "Weight ~ Time"
my_family = sm.families.Gaussian(link=sm.families.links.identity())
model = smf.glm(formula=my_formula, data=my_data, family=my_family)
result = model.fit()

# save model with "remove_data=True" 
result.save('test_model.pickle', remove_data=True)
new_result = sm.load('test_model.pickle')
print(new_result.summary())
```

Output:

![image](https://github.com/statsmodels/statsmodels/assets/12799126/fdfb5b86-f124-4535-9202-1542c9c62640)


user: RoelVerbelen
issue_9146: state: open
title: ENH: right truncated count models for oulier trimming
body: related to robust regression, e.g. #3315 , https://github.com/statsmodels/statsmodels/issues/8690#issuecomment-1913715516

problem is imposing fisher consistency.

one possibility would be to have a full MLE model that is right truncated at the trimming points. Then we should get consistent estimates for the underlying distribution parameter(s)

for example trim at `isf(alpha; theta)` where alpha is e.g. 0.025, 0.01 or similar.
weight function then is indicator function for `(y : y < isf(alpha; theta))`
truncation point will differ across observations

- need a preliminary estimator for parameter theta (mu in poisson case), e.g. robust initial estimate
- iterate weighted/truncated MLE to update theta.
- drop outliers and reestimate truncated MLE

for predict, results we consider it as untruncated model, i.e. predicted mean=mu. This will simplify post-estimation by a large amount. 
(e.g. truncated mean does not have a closed form expression, and we would need to compute it through enumeration.)

First target is for only small counts, i.e. we only need right truncation and not look at left outliers.

possible theoretical problem: 
Trimming/truncation is endogenous, data dependent through parameters and the support of the distribution depends on parameters.
MLE would not apply, but effect should be "negligible" if we only truncate in the far tail.

user: josef-pkt
issue_9145: state: open
title: Passing a Sequence to `trend` argument of ARIMA raises an error.
body: #### Describe the bug

`trend` option in ARIMA should be able to take any sequence, and not an iterable (which is its type hint). Iterables (list, tuple, set or dict) aren't necessarily ordered which can cause issues. Sequence is a better type for that parameter, but it isn't supported. Passing a numpy array for this parameter will raise an error:

```
.venv/lib/python3.10/site-packages/statsmodels/tsa/arima/model.py:158: in __init__
    self._spec_arima = SARIMAXSpecification(
.venv/lib/python3.10/site-packages/statsmodels/tsa/arima/specification.py:380: in __init__
    self.trend_poly, _ = prepare_trend_spec(trend)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

trend = array([0, 0, 2, 1])

    def prepare_trend_spec(trend):
        # Trend
>       if trend is None or trend == 'n':
E       ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
```

#### Code Sample, a copy-pastable example if possible

```python
import numpy as np

from statsmodels.tsa.arima.model import ARIMA

ARIMA(trend=np.array([0, 1, 1])
```

#### Expected Output

We should be able to pass any sequence in `trend`

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]

INSTALLED VERSIONS
------------------
Python: 3.10.11.final.0
OS: Darwin 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:59:33 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T8112 arm64
byteorder: little
LC_ALL: None
LANG: None

statsmodels
===========

Installed: 0.14.1 (/Users/datategy/darts/.venv/lib/python3.10/site-packages/statsmodels)

Required Dependencies
=====================

cython: 3.0.8 (/Users/datategy/darts/.venv/lib/python3.10/site-packages/Cython)
numpy: 1.26.3 (/Users/datategy/darts/.venv/lib/python3.10/site-packages/numpy)
scipy: 1.12.0 (/Users/datategy/darts/.venv/lib/python3.10/site-packages/scipy)
pandas: 2.2.0 (/Users/datategy/darts/.venv/lib/python3.10/site-packages/pandas)
    dateutil: 2.8.2 (/Users/datategy/darts/.venv/lib/python3.10/site-packages/dateutil)
patsy: 0.5.6 (/Users/datategy/darts/.venv/lib/python3.10/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.8.2 (/Users/datategy/darts/.venv/lib/python3.10/site-packages/matplotlib)
    backend: module://matplotlib_inline.backend_inline 
cvxopt: Not installed
joblib: 1.3.2 (/Users/datategy/darts/.venv/lib/python3.10/site-packages/joblib)

Developer Tools
================

IPython: 8.20.0 (/Users/datategy/darts/.venv/lib/python3.10/site-packages/IPython)
    jinja2: 3.1.3 (/Users/datategy/darts/.venv/lib/python3.10/site-packages/jinja2)
sphinx: Not installed
    pygments: 2.17.2 (/Users/datategy/darts/.venv/lib/python3.10/site-packages/pygments)
pytest: 8.0.0 (/Users/datategy/darts/.venv/lib/python3.10/site-packages/pytest)
virtualenv: 20.25.0 (/Users/datategy/darts/.venv/lib/python3.10/site-packages/virtualenv)



</details>

user: MarcBresson
issue_9141: state: open
title: proportion_confint beta fails.
body: #### Describe the bug
statsmodels.stats.proportion.proportion_confint  is failing when 0 successes and very small alph.

#### Code Sample, a copy-pastable example if possible
In the following example one can see that 0/1000 is problematic while 1/1000 is ok. It also shows that for 1000/1000 there is no problem. The last line is the exact computation. 

```python
# Your code here that produces the bug
>>> from statsmodels.stats.proportion import proportion_confint
>>> proportion_confint(0,1000,method='beta', alpha=10**-7)
(0, 0.9999998642895137)
>>> proportion_confint(1,1000,method='beta', alpha=10**-7)
(5.0000001248700306e-11, 0.019662593716583528)
>>> 1-proportion_confint(1000,1000,method='beta', alpha=10**-7)[0]
0.0166707224315088
>>> 1-(10**-7/2)**(1/1000)
0.01667072243152068
```
I suggest to add the following logic to the function which solves the problematic case.

```python
if(count_a == 0):
    return 1-alpha_2**(1/nobs) 
```


#### Expected Output

I'm new to this and I'm willing to implement the posted fix, so I ask for approval if this makes sense.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

>>> import statsmodels.api as sm; sm.show_versions()

INSTALLED VERSIONS
------------------
Python: 3.8.8.final.0
OS: Linux 5.14.0-1054-oem #61-Ubuntu SMP Fri Oct 14 13:05:50 UTC 2022 x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

statsmodels
===========

Installed: 0.12.2 (/home/vasik/anaconda3/lib/python3.8/site-packages/statsmodels)

Required Dependencies
=====================

cython: 0.29.23 (/home/vasik/anaconda3/lib/python3.8/site-packages/Cython)
numpy: 1.22.4 (/home/vasik/anaconda3/lib/python3.8/site-packages/numpy)
scipy: 1.6.2 (/home/vasik/anaconda3/lib/python3.8/site-packages/scipy)
pandas: 1.2.4 (/home/vasik/anaconda3/lib/python3.8/site-packages/pandas)
    dateutil: 2.8.2 (/home/vasik/anaconda3/lib/python3.8/site-packages/dateutil)
patsy: 0.5.1 (/home/vasik/anaconda3/lib/python3.8/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.5.2 (/home/vasik/anaconda3/lib/python3.8/site-packages/matplotlib)
    backend: QtAgg 
cvxopt: Not installed
joblib: 1.0.1 (/home/vasik/anaconda3/lib/python3.8/site-packages/joblib)

Developer Tools
================

IPython: 7.22.0 (/home/vasik/anaconda3/lib/python3.8/site-packages/IPython)
    jinja2: Not installed
sphinx: 4.0.1 (/home/vasik/anaconda3/lib/python3.8/site-packages/sphinx)
    pygments: 2.16.1 (/home/vasik/anaconda3/lib/python3.8/site-packages/pygments)
pytest: 6.2.3 (/home/vasik/anaconda3/lib/python3.8/site-packages/pytest)
virtualenv: Not installed


>>> 


</details>

user: vvoracek
issue_9140: state: open
title: `_fit_minimize` requires update of `no_hess` list: L-BFGS-B and TNC do not use Hessian information (hess)
body: L-BFGS-B and TNC do not use Hessian information (hess). Therefore, these methods should be added to 

https://github.com/statsmodels/statsmodels/blob/23faea30e30ff759c3c9d3f1d57864b2aa68c827/statsmodels/base/optimizer.py#L341
user: pcuestas
issue_9139: state: open
title: `MixedLM.hessian()` returns a tuple instead of just the hessian matrix, causing `MixedLM.fit` to fail when minimizing with methods that require the hessian
body: #### Describe the bug

`MixedLM.hessian()` returns a tuple:

https://github.com/statsmodels/statsmodels/blob/23faea30e30ff759c3c9d3f1d57864b2aa68c827/statsmodels/regression/mixed_linear_model.py#L1855-L1878

https://github.com/statsmodels/statsmodels/blob/23faea30e30ff759c3c9d3f1d57864b2aa68c827/statsmodels/regression/mixed_linear_model.py#L2029

However, it is supposed to return just the hessian matrix, as stated in `LikelyhoodModel.hessian()`:

https://github.com/statsmodels/statsmodels/blob/23faea30e30ff759c3c9d3f1d57864b2aa68c827/statsmodels/base/model.py#L332-L345

The fact that `MixedLM.hessian()` returns a tuple causes optimization to fail when the hessian matrix is used in the minimization algorithm, when the following function is called (within `MixedLM.fit()`):

https://github.com/statsmodels/statsmodels/blob/23faea30e30ff759c3c9d3f1d57864b2aa68c827/statsmodels/base/model.py#L547-L548

(The exception `TypeError: bad operand type for unary -: 'tuple'` is thrown).


user: pcuestas
issue_9138: state: open
title:  singular matrix error in smf.mixedlm
body: #### Describe the bug

I am going to run a generalized linear mixed models as below. 
`       md = smf.mixedlm("y ~ " + covariates_formula, df_final, groups=df_final["subject_label"],
                 re_formula= covariates_formula
                         )
        mdf = md.fit()`
But it reports singular matrix error as shown in the details. 

<details>

`LinAlgError                               Traceback (most recent call last)
[<ipython-input-32-750670676770>](https://localhost:8080/#) in <cell line: 3>()
     42                  re_formula= covariates_formula
     43                          )
---> 44         mdf = md.fit()
     45         print(mdf.summary())

16 frames
[/usr/local/lib/python3.10/dist-packages/statsmodels/regression/mixed_linear_model.py](https://localhost:8080/#) in fit(self, start_params, reml, niter_sa, do_cg, fe_pen, cov_pen, free, full_output, method, **fit_kwargs)
   2190             # Try optimizing one or more times
   2191             for j in range(len(method)):
-> 2192                 rslt = super(MixedLM, self).fit(start_params=packed,
   2193                                                 skip_hessian=True,
   2194                                                 method=method[j],

[/usr/local/lib/python3.10/dist-packages/statsmodels/base/model.py](https://localhost:8080/#) in fit(self, start_params, method, maxiter, full_output, disp, fargs, callback, retall, skip_hessian, **kwargs)
    564 
    565         optimizer = Optimizer()
--> 566         xopt, retvals, optim_settings = optimizer._fit(f, score, start_params,
    567                                                        fargs, kwargs,
    568                                                        hessian=hess,

[/usr/local/lib/python3.10/dist-packages/statsmodels/base/optimizer.py](https://localhost:8080/#) in _fit(self, objective, gradient, start_params, fargs, kwargs, hessian, method, maxiter, full_output, disp, callback, retall)
    240 
    241         func = fit_funcs[method]
--> 242         xopt, retvals = func(objective, gradient, start_params, fargs, kwargs,
    243                              disp=disp, maxiter=maxiter, callback=callback,
    244                              retall=retall, full_output=full_output,

[/usr/local/lib/python3.10/dist-packages/statsmodels/base/optimizer.py](https://localhost:8080/#) in _fit_bfgs(f, score, start_params, fargs, kwargs, disp, maxiter, callback, retall, full_output, hess)
    535     norm = kwargs.setdefault('norm', np.Inf)
    536     epsilon = kwargs.setdefault('epsilon', 1.4901161193847656e-08)
--> 537     retvals = optimize.fmin_bfgs(f, start_params, score, args=fargs,
    538                                  gtol=gtol, norm=norm, epsilon=epsilon,
    539                                  maxiter=maxiter, full_output=full_output,

[/usr/local/lib/python3.10/dist-packages/scipy/optimize/_optimize.py](https://localhost:8080/#) in fmin_bfgs(f, x0, fprime, args, gtol, norm, epsilon, maxiter, full_output, disp, retall, callback, xrtol)
   1358 
   1359     callback = _wrap_callback(callback)
-> 1360     res = _minimize_bfgs(f, x0, args, fprime, callback=callback, **opts)
   1361 
   1362     if full_output:

[/usr/local/lib/python3.10/dist-packages/scipy/optimize/_optimize.py](https://localhost:8080/#) in _minimize_bfgs(fun, x0, args, jac, callback, gtol, norm, eps, maxiter, disp, return_all, finite_diff_rel_step, xrtol, **unknown_options)
   1417         maxiter = len(x0) * 200
   1418 
-> 1419     sf = _prepare_scalar_function(fun, x0, jac, args=args, epsilon=eps,
   1420                                   finite_diff_rel_step=finite_diff_rel_step)
   1421 

[/usr/local/lib/python3.10/dist-packages/scipy/optimize/_optimize.py](https://localhost:8080/#) in _prepare_scalar_function(fun, x0, jac, args, bounds, epsilon, finite_diff_rel_step, hess)
    381     # ScalarFunction caches. Reuse of fun(x) during grad
    382     # calculation reduces overall function evaluations.
--> 383     sf = ScalarFunction(fun, x0, args, grad, hess,
    384                         finite_diff_rel_step, bounds, epsilon=epsilon)
    385 

[/usr/local/lib/python3.10/dist-packages/scipy/optimize/_differentiable_functions.py](https://localhost:8080/#) in __init__(self, fun, x0, args, grad, hess, finite_diff_rel_step, finite_diff_bounds, epsilon)
    156 
    157         self._update_fun_impl = update_fun
--> 158         self._update_fun()
    159 
    160         # Gradient evaluation

[/usr/local/lib/python3.10/dist-packages/scipy/optimize/_differentiable_functions.py](https://localhost:8080/#) in _update_fun(self)
    249     def _update_fun(self):
    250         if not self.f_updated:
--> 251             self._update_fun_impl()
    252             self.f_updated = True
    253 

[/usr/local/lib/python3.10/dist-packages/scipy/optimize/_differentiable_functions.py](https://localhost:8080/#) in update_fun()
    153 
    154         def update_fun():
--> 155             self.f = fun_wrapped(self.x)
    156 
    157         self._update_fun_impl = update_fun

[/usr/local/lib/python3.10/dist-packages/scipy/optimize/_differentiable_functions.py](https://localhost:8080/#) in fun_wrapped(x)
    135             # Overwriting results in undefined behaviour because
    136             # fun(self.x) will change self.x, with the two no longer linked.
--> 137             fx = fun(np.copy(x), *args)
    138             # Make sure the function returns a true scalar
    139             if not np.isscalar(fx):

[/usr/local/lib/python3.10/dist-packages/statsmodels/base/model.py](https://localhost:8080/#) in f(params, *args)
    532 
    533         def f(params, *args):
--> 534             return -self.loglike(params, *args) / nobs
    535 
    536         if method == 'newton':

[/usr/local/lib/python3.10/dist-packages/statsmodels/regression/mixed_linear_model.py](https://localhost:8080/#) in loglike(self, params, profile_fe)
   1497         # Move to the profile set
   1498         if profile_fe:
-> 1499             fe_params, sing = self.get_fe_params(cov_re, vcomp)
   1500             if sing:
   1501                 self._cov_sing += 1

[/usr/local/lib/python3.10/dist-packages/statsmodels/regression/mixed_linear_model.py](https://localhost:8080/#) in get_fe_params(self, cov_re, vcomp, tol)
   1344             fe_params = np.dot(np.linalg.pinv(xtxy[:, 0:-1]), xtxy[:, -1])
   1345         else:
-> 1346             fe_params = np.linalg.solve(xtxy[:, 0:-1], xtxy[:, -1])
   1347 
   1348         return fe_params, sing

/usr/local/lib/python3.10/dist-packages/numpy/core/overrides.py in solve(*args, **kwargs)

[/usr/local/lib/python3.10/dist-packages/numpy/linalg/linalg.py](https://localhost:8080/#) in solve(a, b)
    398     signature = 'DD->D' if isComplexType(t) else 'dd->d'
    399     extobj = get_linalg_error_extobj(_raise_linalgerror_singular)
--> 400     r = gufunc(a, b, signature=signature, extobj=extobj)
    401 
    402     return wrap(r.astype(result_t, copy=False))

[/usr/local/lib/python3.10/dist-packages/numpy/linalg/linalg.py](https://localhost:8080/#) in _raise_linalgerror_singular(err, flag)
     87 
     88 def _raise_linalgerror_singular(err, flag):
---> 89     raise LinAlgError("Singular matrix")
     90 
     91 def _raise_linalgerror_nonposdef(err, flag):

LinAlgError: Singular matrix`

</details>

user: Hanbing1999
issue_9137: state: open
title: BUG: ETS multiplicative error models log-likelihood calculation
body: 
Related to #7331 

**tl;dr:** This PR fixes a small issue in the likelihood calculation of ETS models with multiplicative errors, that caused them to have low ICs and unreasonable forecasts.

***

It appears that the problem raised in #7331 (ETS(MAM) giving unreasonable forecasts) is still there. The fix for the issue had the desired effect for the example time series in the issue but unfortunately the exploding forecasts can still appear for different time series like the one below:
<details><summary>Code to reproduce the plots</summary>

```python
from statsmodels.tsa.exponential_smoothing.ets import ETSModel
import numpy as np
from matplotlib import pyplot as plt

data = np.array([
    214307.1, 331110.3, 324617.4, 230818.8, 275551.5, 232633.5,
    281989.2, 245336.4, 296976. , 207572.7, 252719.4, 210795. ,
    142705.8, 146742.3, 261613.5, 276959.1, 365010. ,  48734.7,
    308919.9, 319401. , 150199.2, 103010.1, 109861.8,  89486.1,
    393645. , 421279.5, 347387.4, 269928. , 171278.7, 158568.9,
    280526.4, 195104.4, 230998.2, 226602.9, 267526.8, 258632.7
])

plt.plot(data)
plt.title("Original time series")
plt.show()

res = ETSModel(
    endog=data,
    seasonal_periods=12,
    error="mul",
    seasonal="mul",
    trend="add",
).fit()
pred = res.forecast(steps=12)
pred_fill = np.zeros_like(data)
pred_fill[:] = np.nan
plt.plot(data, label="endog")
plt.plot(np.concatenate([pred_fill, pred]), label="forecast")
plt.legend()
plt.title("ETS(MAM) forecast - statsmodels (main)")
plt.show()
```

</details>

![image](https://github.com/statsmodels/statsmodels/assets/64217214/5db70668-bb07-4609-bc25-d72c248a11d2)
![image](https://github.com/statsmodels/statsmodels/assets/64217214/cb60deba-733f-404b-b438-25e82fc4ee83)



Moreover, the log-likelihood of the unreasonable forecasts is unreasonably high making the AutoETS implementations (e.g. [sktime](https://github.com/sktime/sktime/blob/main/sktime/forecasting/ets.py), [aeon](https://github.com/aeon-toolkit/aeon/blob/main/aeon/forecasting/ets.py)) that are based on ETS model selection using information criteria (e.g. AIC) select the unreasonable forecasts.
&nbsp;
The problem comes from the Log-Likelihood calculation for ETS models with multiplicative errors, and more specifically the fail-safe mechanism for negative or zero yhat values here:
https://github.com/statsmodels/statsmodels/blob/23faea30e30ff759c3c9d3f1d57864b2aa68c827/statsmodels/tsa/exponential_smoothing/ets.py#L1162-L1169
&nbsp;
This way is not consistent with the literature (see screenshot below):
![image](https://github.com/statsmodels/statsmodels/assets/64217214/5bbbdda3-076f-4ba9-95b1-8885952ba59b)
*screenshot from the [online version](https://www.openforecast.org/adam/ADAMETSEstimationLikelihood.html) of Svetunkov, I. (2023). Forecasting and Analytics with the Augmented Dynamic Adaptive Model (ADAM) (1st ed.). Chapman and Hall/CRC. https://doi.org/10.1201/9781003452652*
&nbsp;

In addition, it's not consistent with any other ETS implementations, where they all seem to follow the formula above for the last term of the log-likelihood, namely log(| yhat |). Below is the relevant code from the other OSS implementations I checked:
* [forecast](https://github.com/robjhyndman/forecast/blob/7d5a856e3ed6579bbd673234807e045c267f4521/src/etscalc.c#L106)
* [smooth](https://github.com/config-i1/smooth/blob/6ead0a2a3115075686b9cfe6240da63ee5db4418/R/adam.R#L2529)
* [nixtla - statsforecast](https://github.com/Nixtla/statsforecast/blob/9d75d22d21bec23063bc1df912cf7d415fd93fd6/statsforecast/ets.py#L96)

All the implementations listed above give similar Log-Likelihood values and AIC as well as reasonable forecasts. The AIC for ETS and this time series is best for the EST(ANN) model hence all the automatic model selection methods from the packages above select this one. Below are the forecasts from these different packages for ETS(MAM) for the same time "problematic" series:

<details><summary>(R) forecast </summary>

```R
library(forecast)
library(ggplot2)

data = ts(
  c(
    214307.1, 331110.3, 324617.4, 230818.8, 275551.5, 232633.5,
    281989.2, 245336.4, 296976. , 207572.7, 252719.4, 210795. ,
    142705.8, 146742.3, 261613.5, 276959.1, 365010. ,  48734.7,
    308919.9, 319401. , 150199.2, 103010.1, 109861.8,  89486.1,
    393645. , 421279.5, 347387.4, 269928. , 171278.7, 158568.9,
    280526.4, 195104.4, 230998.2, 226602.9, 267526.8, 258632.7
  ),
  freq=12
)

fcast_ets_mam = forecast::ets(data, model="MAM", ic="aic") %>% forecast(h=12)
autoplot(fcast_ets_mam)
summary(fcast_ets_mam)
```

</details>

![image](https://github.com/statsmodels/statsmodels/assets/64217214/5de65f19-056c-41c1-a794-74feac46bb85)



<details><summary> (R) smooth </summary>

```R
library(smooth)
library(ggplot2)

data = ts(
  c(
    214307.1, 331110.3, 324617.4, 230818.8, 275551.5, 232633.5,
    281989.2, 245336.4, 296976. , 207572.7, 252719.4, 210795. ,
    142705.8, 146742.3, 261613.5, 276959.1, 365010. ,  48734.7,
    308919.9, 319401. , 150199.2, 103010.1, 109861.8,  89486.1,
    393645. , 421279.5, 347387.4, 269928. , 171278.7, 158568.9,
    280526.4, 195104.4, 230998.2, 226602.9, 267526.8, 258632.7
  ),
  freq=12
)

smooth_es_fit = smooth::es(data, model="MAM")
summary(smooth_es_fit)
smooth_es_fit %>% forecast(h=12) |> plot(main="Smooth ETS(MAM)")
```

</details>

![image](https://github.com/statsmodels/statsmodels/assets/64217214/133df913-6f31-40b4-9d2a-b265c0e1a502)


<details><summary>(Python) nixtla - statsforecast</summary>

```python
from statsforecast.models import AutoETS as StatsforecastETS
import numpy as np
from matplotlib import pyplot as plt

data = np.array([
    214307.1, 331110.3, 324617.4, 230818.8, 275551.5, 232633.5,
    281989.2, 245336.4, 296976. , 207572.7, 252719.4, 210795. ,
    142705.8, 146742.3, 261613.5, 276959.1, 365010. ,  48734.7,
    308919.9, 319401. , 150199.2, 103010.1, 109861.8,  89486.1,
    393645. , 421279.5, 347387.4, 269928. , 171278.7, 158568.9,
    280526.4, 195104.4, 230998.2, 226602.9, 267526.8, 258632.7
])
s_fcaster = StatsforecastETS(model="MAM", season_length=12)
s_fcaster.fit(data)
pred = s_fcaster.predict(h=12)["mean"]
pred_fill = np.empty_like(data)
pred_fill[:] = np.nan
pred = np.concatenate([pred_fill, pred])
plt.plot(data, label="endog")
plt.plot(pred, label="forecast")
plt.title("Nixtla/statsforecast - ETS(MAM)")
plt.show()
print(s_fcaster.__dict__)
```

</details>

![image](https://github.com/statsmodels/statsmodels/assets/64217214/0eb7f320-ffc6-4fc0-8e1b-f2746997db42)


&nbsp;

This PR changes the code for the log-likelihood for multiplicative error models so that it's consistent with the literature and the other ETS implementations. This results in similar forecasts like above:

![image](https://github.com/statsmodels/statsmodels/assets/64217214/fb4d3b80-de25-4e93-8582-a10e83353ab2)


Here are the values for each the log-likelihood and the AIC for ETS(MAM) and that series for each implementation, including the change in statsmodels proposed in this PR.
| ETS Implementation  | Log-Likelihood | AIC |
| ------------- | ------------- | ------------- |
| **statsmodels - main**  | -79.417 | 194.835  |
| **(R) forecast**  | -470.094 |  974.188 |
| **(R) smooth**  | -453.326 |  936.652 |
| **nixtla - statsforecast**  | -467.193  |  968.386 |
| **statsmodels - this PR** | -453.251 |  942.503 |

***

**PS:** Many many many thanks to @config-i1 for helping me to figure this one out! 🙏
user: ltsaprounis
issue_9136: state: open
title: Using PHReg with MICE
body: #### Is your feature request related to a problem? Please describe
Currently it appears that `statsmodels.duration.hazard_regression.PHReg` is not supported for use with `statsmodels.imputation.mice.MICE`. 

Below is a minimal example
```python
import numpy as np
import pandas as pd
import statsmodels.api as sm
import statsmodels.formula.api as smf

# Generic data
d = pd.DataFrame()
d['t'] = [1.1, 2.1, 3.1, 1, 2, 3, 1.2, 2.2, 3.2]
d['d'] = [1, 0, 1, 1, 0, 1, 0, 1, 1]
d['x'] = [1, 1, 0, 0, 1, 1, 0, np.nan, np.nan]
d['z'] = [0, 1, np.nan, 1, 0, 1, 0, 1, 0]

# Complete-case
mod = smf.phreg("t ~ x + z",
                d.dropna(), status='d', ties="efron").fit()
print(mod.summary())

# MICE
imp = sm.MICEData(d)
fml = 't ~ x + z'
mice = sm.MICE(fml, sm.PHReg, imp, init_kwds={"status": "d",
                                              "ties": "efron"})
results = mice.fit(10, 10)
```

At the `mice.fit` call, there is the following error
```
Traceback (most recent call last):
  File "...", line ...., in <module>
    results = mice.fit(5, 10)
  File "...\statsmodels\imputation\mice.py", line 1213, in fit
    return self.combine()
  File "...\statsmodels\imputation\mice.py", line 1232, in combine
    results_uw = results._results
AttributeError: 'PHRegResults' object has no attribute '_results'
```

#### Describe the solution you'd like
It would be nice to have `MICE` behave with `PHReg` as it does with `OLS` and other regression models.

#### Describe alternatives you have considered
I can manually code up MICE but I would rather not in the future. I'm not sure how either of these procedures works in the background, so I'm not sure if `PHRegResults` already has something compatible with `MICE.fit` or if there is a large difference.

#### Additional context
OS: Windows 10
Python: 3.9.4
statsmodels version: 0.14.1
user: pzivich
issue_9135: state: open
title: ExponentialSmoothing's `use_boxcox` parameter can't be set to `log` 
body: #### Describe the bug

The ExponentialSmoothing model's initializer doesn't accept "log" as a value for the `use_boxcox` parameter, even though the documentation says it's accepted, and it's `fit` method allows for it.

Processing of `use_boxcox` https://github.com/statsmodels/statsmodels/blob/23faea30e30ff759c3c9d3f1d57864b2aa68c827/statsmodels/tsa/holtwinters/model.py#L332-L343 

The `fit` method's processing

https://github.com/statsmodels/statsmodels/blob/23faea30e30ff759c3c9d3f1d57864b2aa68c827/statsmodels/tsa/holtwinters/model.py#L1115-L1128

#### Code Sample, a copy-pastable example if possible


```python

import numpy as np
from statsmodels.tsa.holtwinters import ExponentialSmoothing

np.random.seed(0)
n_points = 1500
eps = 10**(-3) # offset so final timeseries is positive everywhere
data = (np.arange(n_points) + 20)\\
      *(np.sin(np.arange(n_points)/n_points) + eps)\\
      *np.random.normal(20, 2, n_points) # multiplicative relationship

model = ExponentialSmoothing(
    endog = data,
    seasonal_periods = n_points,
    trend = "add",
    seasonal = "add",
    use_boxcox = "log"
)
model.fit()

```
<details>

Output of ``import statsmodels as sm; sm.__version__``
'0.14.1'

Error  Trace:

```python

TypeError                                 Traceback (most recent call last)
[<ipython-input-40-d35cba51b053>](https://localhost:8080/#) in <cell line: 9>()
      7 data = (np.arange(n_points) + 20)*(np.sin(np.arange(n_points)/n_points) + eps) * np.random.normal(20, 2, n_points) # multiplicative relationship
      8 
----> 9 model = ExponentialSmoothing(
     10     endog = data,
     11     seasonal_periods = n_points,

2 frames
[/usr/local/lib/python3.10/dist-packages/pandas/util/_decorators.py](https://localhost:8080/#) in wrapper(*args, **kwargs)
    209                 else:
    210                     kwargs[new_arg_name] = new_arg_value
--> 211             return func(*args, **kwargs)
    212 
    213         return cast(F, wrapper)

[/usr/local/lib/python3.10/dist-packages/statsmodels/tsa/holtwinters/model.py](https://localhost:8080/#) in __init__(self, endog, trend, damped_trend, seasonal, seasonal_periods, initialization_method, initial_level, initial_trend, initial_seasonal, use_boxcox, bounds, dates, freq, missing)
    293         self._use_boxcox = use_boxcox
    294         self._lambda = np.nan
--> 295         self._y = self._boxcox()
    296         self._initialize()
    297         self._fixed_parameters = {}

[/usr/local/lib/python3.10/dist-packages/statsmodels/tsa/holtwinters/model.py](https://localhost:8080/#) in _boxcox(self)
    339             y = boxcox(self._y, self._use_boxcox)
    340         else:
--> 341             raise TypeError("use_boxcox must be True, False or a float.")
    342         return y
    343 

TypeError: use_boxcox must be True, False or a float.

```

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

The TypeError is the expected output based on the code

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

INSTALLED VERSIONS
------------------
Python: 3.10.12.final.0
OS: Linux 6.1.58+ #1 SMP PREEMPT_DYNAMIC Sat Nov 18 15:31:17 UTC 2023 x86_64
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8

statsmodels
===========

Installed: 0.14.1 (/usr/local/lib/python3.10/dist-packages/statsmodels)

Required Dependencies
=====================

cython: 3.0.8 (/usr/local/lib/python3.10/dist-packages/Cython)
numpy: 1.23.5 (/usr/local/lib/python3.10/dist-packages/numpy)
scipy: 1.11.4 (/usr/local/lib/python3.10/dist-packages/scipy)
pandas: 1.5.3 (/usr/local/lib/python3.10/dist-packages/pandas)
    dateutil: 2.8.2 (/usr/local/lib/python3.10/dist-packages/dateutil)
patsy: 0.5.6 (/usr/local/lib/python3.10/dist-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.7.1 (/usr/local/lib/python3.10/dist-packages/matplotlib)
    backend: module://matplotlib_inline.backend_inline 
cvxopt: 1.3.2 (/usr/local/lib/python3.10/dist-packages/cvxopt)
joblib: 1.3.2 (/usr/local/lib/python3.10/dist-packages/joblib)

Developer Tools
================

IPython: 7.34.0 (/usr/local/lib/python3.10/dist-packages/IPython)
    jinja2: 3.1.3 (/usr/local/lib/python3.10/dist-packages/jinja2)
sphinx: 5.0.2 (/usr/local/lib/python3.10/dist-packages/sphinx)
    pygments: 2.16.1 (/usr/local/lib/python3.10/dist-packages/pygments)
pytest: 7.4.4 (/usr/local/lib/python3.10/dist-packages/pytest)
virtualenv: Not installed
</details>

user: tcsmaster
issue_9134: state: open
title: ENH design, tools: one-step approximation to params in iterative fixed point M-estimation
body: (random find sorting through articles downloaded for robust in 2016)

Aelst, Stefan Van, and Gert Willems. “Fast and Robust Bootstrap for Multivariate Inference: The R Package FRB.” Journal of Statistical Software 53, no. 3 (2013). https://doi.org/10.18637/jss.v053.i03.
section 3, mainly equ. 16

M, MM, S estimators for outlier robust estimation work with a iterative fixed point algorithm, e.g. IRLS style.
For fast bootstrap, they compute a new parameter based on the fixed point iteration but corrected for a second order Taylor series expansion.

This issue is mainly to park the reference.
It might be useful to applications other than bootstrap. We use one-step `dparams` in related cases like outlier-influence and jackknife.

Many outlier robust estimators are computed through fixed point iteration, and we don't have (yet) score and hessian (or estimating equations and their derivatives, for the case when we don't have one objective function for the M-estimator) to compute a newton-raphson or similar one-step approximations to params_new and dparams.


user: josef-pkt
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_9133: state: open
title: ENH Roadmap outlier robust covariance or scatter matrix
body: see
#3221
#8129

Maronna and Yohai 2017 recommend MM-estimator for p < 15 and Rocke S-estimator for p >= 15.
They only look at affine equivariant estimators.
#8129 does not include those and similar estimators.
Many estimators in there are not affine equivariant or are only approximately affine equivariant (like OGK)

Higher priority target would be to get MM and Rocke-S

- `norm`, weight functions
  - smoothed hard rejection (SHR in MY 2012)
  - Rocke nonmonotonic weight function 
- S-estimator
- MM estimator
- starting cov or samples
  - use `Det` as started in #8129, approximately affine equivariant
  - Pena and Prieto 2007, KSD in MY 2012, "kurtosis plus specific directions", affine equivariant
- ...
user: josef-pkt
issue_9132: state: open
title: ENH: outlier robust regression, OLS, IV2SLS, based on outlier robust covariance/moment matrices
body: once we have robust covariance computation as in #8129, we can compute linear regression models including IV based on those cov.

I don't find right now the reference for using  MCD/MVE and similar for OLS

for IV:

Cohen Freue, Gabriela V., Hernan Ortiz-Molina, and Ruben H. Zamar. “A Natural Robustification of the Ordinary Instrumental Variables Estimator.” Biometrics 69, no. 3 (September 1, 2013): 641–50. https://doi.org/10.1111/biom.12043.


https://github.com/statsmodels/statsmodels/issues/3901#issuecomment-1913180626 OLS based on summary statistics

user: josef-pkt
issue_9131: state: open
title: Warnings can not be supressed in Jupyter
body: #### Describe the bug

Even after setting warnings to "once" in jupyter notebook, the warnings are repeated so many times. Similary, even if I set warnings to "ignore" the warnings are fired. The specific warning I am talking about is PerfectSeparationWarning.
`statsmodels/genmod/generalized_linear_model.py:1257: PerfectSeparationWarning:`

When code is used for multiple calculations together, the jupyter notebook becomes too large (in my case more than 500MB) and could not be opened anymore. 

I have tried every permutation and combination of following codes to suppress multiple warnings: 

```
import warnings
from statsmodels.tools.sm_exceptions import PerfectSeparationWarning

with warnings.catch_warnings():
    warnings.filterwarnings("ignore", category=PerfectSeparationWarning)
    
import warnings
warnings.filterwarnings('ignore')
warnings.simplefilter('ignore')

import warnings
warnings.simplefilter('once', category=UserWarning)
```


#### Code Sample, a copy-pastable example if possible


```python
# Your code here that produces the bug
# This example should be self-contained, and so not rely on external data.
# It should run in a fresh ipython session, and so include all relevant imports.
```
<details>

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `main`. If your problem has been fixed in an unreleased version, you might be able to use `main` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the main branch of this repository? It helps the limited resources if we know problems exist in the current main branch so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

A clear and concise description of what you expected to happen.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]

</details>

user: achinmay17
issue_9130: state: open
title: ENH: GLM models now save the names of input Pandas Series
body: Offset, exposure, freq_weights and var_weights have the name of the series saved on the model object. They can be accessed via the class properties.

Closes #9100

- [x] closes #9100
- [x] tests added / passed. 
- [x] code/documentation is well formatted.  
- [x] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

Getting this as a starting point.  This enables the feature for GLM's. Happy to make any recommended changes.

I didn't edit the release notes, wasn't sure if this would be worth mentioning. Kept the smaller notes in the commit message for now.

I opted for only doing one dimensional arrays in the helper because the caller would need to check the shape if it returned more than one name. Seems like a separate helper for multi dimensional might be needed unless there's a different way we were expecting to use it.

user: jmahlik
issue_9129: state: open
title: Doc/exponentialsmoothing fit method
body: - [x] code/documentation is well formatted.  
- [x] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

<details>


**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in main and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/main -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>

user: MarcBresson
issue_9128: state: open
title: ENH: nonlinear robust estimation, curve fitting, application to distribution parameter estimation
body: 
vague idea

I have robust or non-robust to outliers (squared) mahalanobis distance.
For elliptical distributions this distance, i.e. radial function, has a given distribution.
In the normal case it is chi-square, in the t distribution case it is a scaled F distribution. We don't have a collection of these radial distribution functions for various elliptical distributions.

robust multivariate methods usually use normal reference case with chi2 for calibrating the scale, #8129.
More general we could try to fit possible candidate distributions like chi2, F, and estimate the distribution parameters. Those parameters will reflect scale and kurtosis, tail heaviness, e.g. degrees of freedom in F.
(Similar in qq-plot we can estimate loc and scale by fitting a line in the plot. AFAIR we have the option of using RLM for it, or I used RLM only in examples.)

Estimation could fit some distance measure for quantiles or cdf, but that is in general nonlinear in distribution parameters.
For the outlier application we need an outlier robust estimation procedure, non-linear RLM, truncation, ...

E.g. I want a quick check whether multivariate-t distribution is a good approximation for the radial component of the data and which degrees of freedom would fit best.

related topic goodness-of-fit, minimum distance estimators #7412
 
user: josef-pkt
issue_9127: state: open
title: boolean columns in dataframe fails
body: #### Describe the bug

Not sure if this is a bug or intended. I've tried searching issues but couldn't find a similar issue.

When passing a dataframe as `exog` with `bool` columns an exception occurs because
`pandas` does not cast to `float` with `np.ndarray` when there are `float` and `bool` columns, but defaults to `object` instead. A mix of `float` and `int` columns gets properly cast.





#### Code Sample, a copy-pastable example if possible


```python
import pandas as pd
import statsmodels.api as sm
import numpy as np

rng = np.random.default_rng(0)

# only float and int
X = pd.DataFrame({'A':rng.standard_normal(5),
                  'B':np.array([1,2,3,2,1], int)})
Y = rng.standard_normal(5)
sm.OLS(Y, X).fit() # works

# add a bool column
X['C'] = np.array([True,False,False,True,True],bool)
sm.OLS(Y, X).fit() # does not work

```
<details>

I think changing the `np.asarray` call in https://github.com/statsmodels/statsmodels/blob/main/statsmodels/base/data.py#L507 to `np.asarray(exog, dtype=float)` would simply resolve this.

One could say that it's a bug in pandas (as it had decided not to cast as float in the `np.ndasarray` call), but I presume they have a rationale for that choice. In the statsmodels code, `np.asarray` will already potentially involve casting some columns so casting them explicitly to `float` seems not a huge deal.

Cases where you would not want to do this perhaps are cases where the original `pd.DataFrame` was large and all `int` or `bool` perhaps. Not sure how prevalent such cases are.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

I would expect no exception raised here because it is clear how to cast such dataframes to design matrices for use in `statsmodels`.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

INSTALLED VERSIONS
------------------
Python: 3.11.4.final.0
OS: Darwin 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar  6 20:59:28 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T6000 arm64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

statsmodels
===========

Installed: 0.14.0 (/Users/jonathantaylor/anaconda3/envs/islp_freeze_311/lib/python3.11/site-packages/statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.24.2 (/Users/jonathantaylor/anaconda3/envs/islp_freeze_311/lib/python3.11/site-packages/numpy)
scipy: 1.11.1 (/Users/jonathantaylor/anaconda3/envs/islp_freeze_311/lib/python3.11/site-packages/scipy)
pandas: 1.5.3 (/Users/jonathantaylor/anaconda3/envs/islp_freeze_311/lib/python3.11/site-packages/pandas)
    dateutil: 2.8.2 (/Users/jonathantaylor/anaconda3/envs/islp_freeze_311/lib/python3.11/site-packages/dateutil)
patsy: 0.5.3 (/Users/jonathantaylor/anaconda3/envs/islp_freeze_311/lib/python3.11/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.7.2 (/Users/jonathantaylor/anaconda3/envs/islp_freeze_311/lib/python3.11/site-packages/matplotlib)
Installed osx event loop hook.
    backend: MacOSX 
cvxopt: Not installed
joblib: 1.3.1 (/Users/jonathantaylor/anaconda3/envs/islp_freeze_311/lib/python3.11/site-packages/joblib)

Developer Tools
================

IPython: 8.14.0 (/Users/jonathantaylor/anaconda3/envs/islp_freeze_311/lib/python3.11/site-packages/IPython)
    jinja2: 3.1.2 (/Users/jonathantaylor/anaconda3/envs/islp_freeze_311/lib/python3.11/site-packages/jinja2)
sphinx: 5.3.0 (/Users/jonathantaylor/anaconda3/envs/islp_freeze_311/lib/python3.11/site-packages/sphinx)
    pygments: 2.15.1 (/Users/jonathantaylor/anaconda3/envs/islp_freeze_311/lib/python3.11/site-packages/pygments)
pytest: 7.4.0 (/Users/jonathantaylor/anaconda3/envs/islp_freeze_311/lib/python3.11/site-packages/pytest)
virtualenv: 20.24.3 (/Users/jonathantaylor/anaconda3/envs/islp_freeze_311/lib/python3.11/site-packages/virtualenv)


</details>

user: jonathan-taylor
issue_9126: state: open
title: MAINT: Fixes for pre-release testing
body: Fix dim change from NumPy
Switch freq string

- [ ] closes #xxxx
- [ ] tests added / passed. 
- [ ] code/documentation is well formatted.  
- [ ] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

<details>


**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in main and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/main -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>

user: bashtage
issue_9125: state: open
title: BUG: changed np.round(mean_diff,2) -> mean_diff:.2g to avoid trailing …
body: …digits bug that appears in matplotlib figure output

- [ ] closes #xxxx
- [ ] tests added / passed. 
- [ ] code/documentation is well formatted.  
- [ ] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

<details>


**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in main and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/main -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>

user: jeswcollins
issue_9122: state: open
title: ENH: allow cov_type in specification tests in stats.diagnostic
body: lm-test that are based on auxiliary regression can use cov_type to make the specification test robust to (co)variance and other misspecification.

see https://github.com/statsmodels/statsmodels/issues/9121#issuecomment-1892389971

first type of cases is where we can directly include cov_type as argument and use it directly in the auxiliary regression.
user: josef-pkt
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_9121: state: open
title: linear_reset can't be set with "HAC" and maxlags, KeyErrors
body: #### Describe the bug
Hello, thanks for your time. We've tried several things but it still doesn't work even with the newest version of Statsmodels. We think this might be a bug. Issue is: we were trying to use "statsmodels.stats.diagnostic.linear_reset" with HAC as coviance type and maxlags as the key word args, but we got KeyErrors.

#### Code Sample, a copy-pastable example if possible
```
# Import modules
import numpy as np
import statsmodels.api as sm
from statsmodels.stats.diagnostic import linear_reset

# Generate some random data
np.random.seed(42)
x = np.random.randn(100)
y = 2 * x + np.random.randn(100)

# Fit a linear regression model
model = sm.OLS(y, sm.add_constant(x))
# results = model.fit()

# Perform linear reset test
## opt 1
reset_test = linear_reset(results, power=2, test_type='fitted', cov_type='HAC', cov_kwargs={'maxlags': 2})

## opt 2
# test = linear_reset(results, cov_type='HAC', cov_kwds={'maxlags': 2})

## opt 3
# kwargs = {'maxlags': 2}
# test = linear_reset(results, cov_type='HAC', **kwargs)

## opt 4
# test = linear_reset(results, cov_type='HAC', maxlags=2)

```
```python
# Your code here that produces the bug
# This example should be self-contained, and so not rely on external data.
# It should run in a fresh ipython session, and so include all relevant imports.
```
<details>
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File <command-2970740754718372>:19
     16 model = sm.OLS(y, x).fit(cov_type='HAC', cov_kwds={'maxlags': 2})
     18 # Perform linear reset test
---> 19 reset_test = linear_reset(model, power=2, test_type='fitted', cov_type='HAC', cov_kwargs={'maxlags': 2})
     21 # # Perform the linear reset test with HAC covariance and maxlags=2
     22 # test = linear_reset(results, cov_type='HAC', cov_kwds={'maxlags': 2})
     23 
   (...)
     33 
     34 # Print the test results
     35 print(reset_test.summary())

File /databricks/python/lib/python3.9/site-packages/pandas/util/_decorators.py:207, in deprecate_kwarg.<locals>._deprecate_kwarg.<locals>.wrapper(*args, **kwargs)
    205     else:
    206         kwargs[new_arg_name] = new_arg_value
--> 207 return func(*args, **kwargs)

File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/statsmodels/stats/diagnostic.py:1098, in linear_reset(res, power, test_type, use_f, cov_type, cov_kwargs)
   1096 mod = mod_class(res.model.data.endog, aug_exog)
   1097 cov_kwargs = {} if cov_kwargs is None else cov_kwargs
-> 1098 res = mod.fit(cov_type=cov_type, cov_kwargs=cov_kwargs)
   1099 nrestr = aug_exog.shape[1] - exog.shape[1]
   1100 nparams = aug_exog.shape[1]

File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/statsmodels/regression/linear_model.py:373, in RegressionModel.fit(self, method, cov_type, cov_kwds, use_t, **kwargs)
    370     self.df_resid = self.nobs - self.rank
    372 if isinstance(self, OLS):
--> 373     lfit = OLSResults(
    374         self, beta,
    375         normalized_cov_params=self.normalized_cov_params,
    376         cov_type=cov_type, cov_kwds=cov_kwds, use_t=use_t)
    377 else:
    378     lfit = RegressionResults(
    379         self, beta,
    380         normalized_cov_params=self.normalized_cov_params,
    381         cov_type=cov_type, cov_kwds=cov_kwds, use_t=use_t,
    382         **kwargs)

File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/statsmodels/regression/linear_model.py:1653, in RegressionResults.__init__(self, model, params, normalized_cov_params, scale, cov_type, cov_kwds, use_t, **kwargs)
   1651             use_t = use_t_2
   1652         # TODO: warn or not?
-> 1653     self.get_robustcov_results(cov_type=cov_type, use_self=True,
   1654                                use_t=use_t, **cov_kwds)
   1655 for key in kwargs:
   1656     setattr(self, key, kwargs[key])

File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/statsmodels/regression/linear_model.py:2570, in RegressionResults.get_robustcov_results(self, cov_type, use_t, **kwargs)
   2567     res.cov_params_default = getattr(self, 'cov_' + cov_type.upper())
   2568 elif cov_type.lower() == 'hac':
   2569     # TODO: check if required, default in cov_hac_simple
-> 2570     maxlags = kwargs['maxlags']
   2571     res.cov_kwds['maxlags'] = maxlags
   2572     weights_func = kwargs.get('weights_func', sw.weights_bartlett)
KeyError: 'maxlags'

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `main`. If your problem has been fixed in an unreleased version, you might be able to use `main` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the main branch of this repository? It helps the limited resources if we know problems exist in the current main branch so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output
We expect that the code would produce the p-value from linear RESET test with "HAC" as the covariance estimator.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]
0.14.1
</details>

user: Qingyao-gh
issue_9120: state: open
title: Request: Guidance on Implementing Bounds for Coefficients in Mixed Linear Model
body: #### Is your feature request related to a problem? Please describe
I need to run a mixed linear model with bounds on the coefficients, but I'm uncertain about which classes, methods, or functions I should modify to achieve this.

#### Describe the solution you'd like
I would like guidance on the specific classes, methods, or functions that need modification to implement bounds for coefficients in a mixed linear model.

#### Describe alternatives you have considered
I haven't explored specific alternatives yet and would appreciate advice on different approaches or modifications that achieve the desired outcome.

#### Additional context

Thank you very much for your assistance!
user: tnotenson
issue_9119: state: open
title: Wrong typing for disp in fit functions
body: #### Describe the bug

Hi, In the documentation and code I noticed that in fit functions the function variable 'disp' are typed as boolean, but in examples they contain an integer. These are all in the end passed along to scipy optimizers as int, so I guess all the typings are supposed to be integers instead of booleans?

Place where passed along to scipy: [https://github.com/statsmodels/statsmodels/blob/e7974698fa97a0d55cc842e8f3a6e80d45ec4e79/statsmodels/base/optimizer.py#L246C37-L246C37](https://github.com/statsmodels/statsmodels/blob/e7974698fa97a0d55cc842e8f3a6e80d45ec4e79/statsmodels/base/optimizer.py#L246C37-L246C37)

And example where default value is 5, but typing in docstring is bool:
https://github.com/statsmodels/statsmodels/blob/e7974698fa97a0d55cc842e8f3a6e80d45ec4e79/statsmodels/tsa/statespace/mlemodel.py#L526C23-L526C23

I guess updating the docstring to int at some places will fix this.
Not really a breaking issue, but raised some question marks for me when reading the documentation:)

I tested it, and integer is fully supported, and results in iprint / disp being used for verbose/no logging.
https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fmin_l_bfgs_b.html

Hope this helps out:)
user: lucasfijen
issue_9118: state: open
title: ENH: Add confusion_matrix_statistics function in contingency_tables
body: - [x] tests added / passed. 
- [x] code/documentation is well formatted.  
- [x] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

* This commit generates classification performance metrics using square tables. The metrics include Accuracy, No Information Rate, Kappa, Sensitivity, Specificity, Positive Predictive Value, Negative Predictive Value, Balanced Accuracy, and F1. The code draws inspiration from Caret package in R

* The overall metrics are structured in a dictionary.

* The class-specific metrics output is structured in rows representing different metrics and columns representing different classes (class 1, class 2, class 3, class 4). 

* The test values are verified with confusionMatrix function in Caret package in R. 



user: pasrakan
issue_9114: state: open
title: ENH: Add CFA to factor analysis
body: This PR allows confirmatory factor analyses to be done with the Factor class (which currently only supports exploratory factor analysis).
user: kshedden
issue_9108: state: open
title: RuntimeWarning in zt_ind_solve_power function : invalid value encountered in sqrt
body: #### Describe the bug

Sample size optimization in statsmodels 0.13.5 ends up in RunTimeWarning as nobs and nobs1 become negative values under square root. The optimization algorithm should not allow negative sample sizes nobs or nobs1 either and should correctly convert to positive integers.

#### Code Sample, a copy-pastable example if possible


```python
effect_size = -0.0008095274776159337
power = 0.8
alpha error = 0.1
ratio = 1.0146909910259767,
alternative = 'larger'

sample_size = zt_ind_solve_power(
    effect_size = effect_size,
    power = power,
    alpha = alpha_error,
    sample_size_ration = sample_size_ration,
    alternative = alternative
)

```
Adapting power.py file to print nobs and nobs1 from power function it is clear that optimization end up with negative values for sample sizes: 
```
effect_size: -0.0008095274776159337, power: 0.8, alpha_error: 0.1, ratio: 1.0146909910259767, alternative: larger
nobs: 1.0072919326543943, nobs1: 2.0, ddof: 0
nobs: 25.182298316359862, nobs1: 50.0, ddof: 0
nobs: 25.182298316359862, nobs1: 50.0, ddof: 0
nobs: 25.182298316359862, nobs1: 50.0, ddof: 0
nobs: 1.0072919326543943, nobs1: 2.0, ddof: 0
nobs: [5.03645966], nobs1: [10.], ddof: 0
nobs: [5.03645966], nobs1: [10.], ddof: 0
nobs: [5.03645966], nobs1: [10.], ddof: 0
nobs: [5.03645974], nobs1: [10.00000015], ddof: 0
nobs: [-498.60950666], nobs1: [-990.], ddof: 0
```

#### Expected Output
The solution of sample size should converge to positive int or issue the ConvergenceWarning but not allow negative sample size numbers in optimization.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

```
INSTALLED VERSIONS
------------------
Python: 3.7.3.final.0
OS: Linux 4.18.0-372.16.1.0.1.el8_6.x86_64 #1 SMP Wed Jul 13 11:58:23 PDT 2022 x86_64
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8

statsmodels
===========

Installed: 0.13.5 (/home/alfred_sasko/.virtualenvs/abtest/lib/python3.7/site-packages/statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.21.6 (/home/alfred_sasko/.virtualenvs/abtest/lib/python3.7/site-packages/numpy)
scipy: 1.7.3 (/home/alfred_sasko/.virtualenvs/abtest/lib/python3.7/site-packages/scipy)
pandas: 1.3.5 (/home/alfred_sasko/.virtualenvs/abtest/lib/python3.7/site-packages/pandas)
    dateutil: 2.8.2 (/home/alfred_sasko/.virtualenvs/abtest/lib/python3.7/site-packages/dateutil)
patsy: 0.5.3 (/home/alfred_sasko/.virtualenvs/abtest/lib/python3.7/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.5.3 (/home/alfred_sasko/.virtualenvs/abtest/lib/python3.7/site-packages/matplotlib)
    backend: module://matplotlib_inline.backend_inline 
cvxopt: Not installed
joblib: 1.3.2 (/home/alfred_sasko/.virtualenvs/abtest/lib/python3.7/site-packages/joblib)

Developer Tools
================

IPython: 7.34.0 (/home/alfred_sasko/.virtualenvs/abtest/lib/python3.7/site-packages/IPython)
    jinja2: Not installed
sphinx: Not installed
    pygments: 2.15.1 (/home/alfred_sasko/.virtualenvs/abtest/lib/python3.7/site-packages/pygments)
pytest: 7.4.0 (/home/alfred_sasko/.virtualenvs/abtest/lib/python3.7/site-packages/pytest)
virtualenv: Not installed
```

</details>

user: alfredsasko
issue_9107: state: open
title: Added SVAR option to plot_cum_effects in irf
body: plot_cum_effects function lacked SVAR check: when it was called for SVAR, it returned VAR cumulative IRFs despide them being calculated for SVAR internally. Thus one elif case has beed added to resolve this issue.

Example: 
```
data_d_reer = pd.read_csv(r'data_
[Uploading data_reer.csv…]()
reer.csv', index_col=0)
svar_model = SVAR(data_d_reer, svar_type="A", A=amat, freq='Q').fit()
svar_model.irf().plot_cum_effects();

```
Current behavior: returns VAR cumulative IR's (self.cum_effects).
Expected behavior: returns SVAR cumulative IR's (self.svar_cum_effects)
user: ikuzmin404
issue_9105: state: open
title: Simulate method of ARIMA model missing residuals and disturbances input
body: #### Is your feature request related to a problem? Please describe
I'm trying to replicate Matlab's simulation of ARIMA where exog, residuals and disturbances are used as inputs. I can fit an ARIMA and simulate paths but I can't input residuals and disturbances in the same way as in Matlab, or maybe it is possible and I don't know how to.

#### Describe the solution you'd like
After fitting an ARIMA the simulation method should take residuals and disturbances as input.

#### Describe alternatives you have considered
I have tried using measurement_shocks=smoothed_state and state_shocks=residuals and slicing smoothed_state and state_shocks to only contain the last n elements equal to the number of nsimulation.

#### Additional context
I've also tried to not specify measurement_shocks and state_shocks and compare results with Matlab but I can't verify results have the same distribution.

If these features are already available, I apologise for not understanding the documentation and would appreciate if someone could show me the exact implementation in python/statsmodels.
user: Hemming01
issue_9103: state: open
title: Error estimating mean in rainbow test
body: The function `linear_rainbow` has an option to either order cases by a certain reference case order, or alternatively to calculate Mahalanobis distances to the center of the data and then build the central fit from the `frac` fraction of cases that have the smallest Mahalanobis distances, as was proposed in the original paper by Jessica Utts (1982).  In the latter option, cases in the raw data are not assumed to be sorted. However, the present code will just take the n/2 th case as the center of the data: 

https://github.com/statsmodels/statsmodels/blob/6614975f4468abba15ae6894db276cfe2e29ba63/statsmodels/stats/diagnostic.py#L1229

This is obviously incorrect. The location estimate required in Mahalanobis distances is the mutlivariate mean, not the n/2 th case in the data. I suggest to modify this line to `center_vec = np.mean(exog,axis=0)` and then use `center_vec` instead of `center_obs` going forward.  
user: SvenSerneels
issue_9102: state: open
title: ENH: vertical error bars in tukeyhsd plot_simultaneous
body: - [x] closes #9052 
- [x] tests added / passed
- [x] code/documentation is well formatted.  
- [x] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 
---
<details>
<summary>code</summary>

```py
import matplotlib.pyplot as plt
from statsmodels.stats.multicomp import MultiComparison, pairwise_tukeyhsd

response = [4, 5, 7, 6, 2, 3, 12, 9, 7, 7]
groups = [
    "test1",
    "test1",
    "test2",
    "test2",
    "test3",
    "test3",
    "test4",
    "test4",
    "test5",
    "test5",
]

result = pairwise_tukeyhsd(response, groups, alpha=0.05)

# Plot the results
fig, ax = plt.subplots(figsize=(1, 1))

# Customize the plot appearance
result.plot_simultaneous(
    comparison_name="test3",
    ax=ax,
    figsize=(8, 5),
    xlabel="Response",  # X-axis label
    ylabel="Test Group",  # Y-axis label
    vertical=True,
)

# Reverse the order of labels on the y-axis
# ax.invert_yaxis() # off comments when vertical true

# Add a title to the plot
ax.set_title(f"Tukey's HSD Test for Response")

# Show the plot
plt.show()
```

</details>

Before:
![before](https://github.com/statsmodels/statsmodels/assets/58372366/4fbb08ee-4845-467d-92af-901974dba7a3)
After (with vertical=True):
![after](https://github.com/statsmodels/statsmodels/assets/58372366/2040753b-0940-4080-b705-23d5f72e5fa2)


user: iboraham
issue_9100: state: open
title: Save the offset name in `GLM` and results wrapper
body: #### Is your feature request related to a problem? Please describe
Post model training, it is helpful to know which variable was used as the offset. This aids in post model analysis and deployment.

The offset array is saved and can be accessed after saving the model, but the name of the offset variable is lost when it is a pandas series. The series is [converted to a np.array](https://github.com/statsmodels/statsmodels/blob/ab10165eb897729b50e703b4ea831ae712b53585/statsmodels/genmod/generalized_linear_model.py#L315-L316) which removed the name. Current state, it is difficult to tell which variable may have been used as an offset without tracking it outside the model.

Example use case: Sharing a saved model with a peer. They inspect it to determine what variable was used as the offset in training.

The same may apply to the `var_weights` and `freq_weights` for GLM.

#### Describe the solution you'd like
The model has access on `__init__` to the name of the offset if it is a pandas series. A way to save the offset array's name if it is a series would be wonderful.

Similar to how the endog and exog names can be used in the model summary.

Here's a few ideas I had for how to implement this. Happy to hear if there's a better option.

1. Add an `offset_name` property for GLM
    - Similar to the base models [`endog_names`/`exog_names`](https://github.com/statsmodels/statsmodels/blob/ab10165eb897729b50e703b4ea831ae712b53585/statsmodels/base/model.py#L235-L247)
    - Simple to implement
2. Add it to the `model.data` so it's handled by [`PandasData`](https://github.com/statsmodels/statsmodels/blob/ab10165eb897729b50e703b4ea831ae712b53585/statsmodels/base/data.py#L498)
    - The name could be added back to the offset when making the results wrapper (at least I think that's how it works)
    - I could use some guidance on how to implement this if it is the preferred approach
    - I think it has something to do with the data attrs but it's a bit hard to track down
3. Do not convert to a numpy array if it is a series
    - One could use `model.offset.name` to get at the variable name 
    - Doesn't line up with how the rest of the code works, it expects numpy arrays
    - Likely not a good option
4. User adds `offset_name` attribute to the model class before saving it.
    - Seems like a bad idea, would like support in statsmodels

#### Describe alternatives you have considered
Current workaround is saving the offset name in a separate file, which is not ideal.

#### Additional context
Happy to work on a PR for this.

user: jmahlik
issue_9099: state: open
title: DOC: no docstrings describes diagnostic part of regression `summary`
body: I don't find a docstring that includes description and defintion summary info.

current point: I was not sure whether kurtosis is fisher or pearson, i.e. 0 or 3 in normal case.
It is pearson, normal = 3, after looking at the code in `jarque_bera`.

user: josef-pkt
issue_9097: state: open
title: Big performance difference between statsmodels MixedLM and R lmer
body: Hi,

Using statsmodels version 0.14.0 I notice a big difference in performance when running the same model in R. Using statsmodels' `mixedlm`, the model takes 41 minutes to run while only 1-2 seconds using R's `lmer` package on the same machine. Why is there such a large difference in performance and is there anything I can do speed things up?

I have tried different `method` arguments, but most of them have trouble converging.

My dataset contains 125,066 groups (pairs), has 2 categorical variables and 1 numerical. I am comparing the following code:

```
lmm_model = smf.mixedlm(f'value ~ C(cat_var1) + C(cat_var2) + numerical_var', data=pdf, groups=pdf['unit'])
lmm_results = lmm_model.fit()

print(lmm_results.summary())
```

And in R:
```
summary(lmer('value ~ cat_var1 + cat_var2 + numerical_var + (1|unit)', data=data_r))
```

Thanks in advance!
user: irrationalme
issue_9095: state: open
title: ENH: (outlier) robust multivariate regression
body: (I found several articles that I printed at around 2016)

- robust, RLM is only for univariate endog
- MultivariateLS has been added in #8919
- a first set of robust covariance estimators is in PR #8129, but it mostly assumes mean is known

The target is to add something like a multivariate RLM equivalent, and possibly other robust and resistant estimators (which are also still missing for univariate endog case).

`statsmodels.robust.multivariate`
this could also be the location for constant mean models, i.e. joint estimation of mean/center and covariance.

AFAIR, I did not read much of the robust multivariate regression literature, but more on joint mean-cov estimation.


References (added to zotero between 2016-09-29 and 1016-10-02):

Agulló, Jose, Christophe Croux, and Stefan Van Aelst. “The Multivariate Least-Trimmed Squares Estimator.” Journal of Multivariate Analysis 99, no. 3 (March 1, 2008): 311–38. https://doi.org/10.1016/j.jmva.2006.06.005.

Ben, Marta García, Elena Martínez, and Víctor J. Yohai. “Robust Estimation for the Multivariate Linear Model Based on a τ-Scale.” Journal of Multivariate Analysis 97, no. 7 (August 1, 2006): 1600–1622. https://doi.org/10.1016/j.jmva.2005.08.007.

Gnanadesikan, R., and J. R. Kettenring. “Robust Estimates, Residuals, and Outlier Detection with Multiresponse Data.” Biometrics 28, no. 1 (1972): 81–124. https://doi.org/10.2307/2528963.

Hubert, Mia, Tim Verdonck, and Özlem Yorulmaz. “Fast Robust SUR with Economical and Actuarial Applications.” Statistical Analysis and Data Mining: The ASA Data Science Journal, May 1, 2016, n/a-n/a. https://doi.org/10.1002/sam.11313.

Jung, Kang-Mo. “Multivariate Least-Trimmed Squares Regression Estimator.” Computational Statistics & Data Analysis 48, no. 2 (February 1, 2005): 307–16. https://doi.org/10.1016/j.csda.2004.01.008.

Kent, John T., and David E. Tyler. “Constrained M-Estimation for Multivariate Location and Scatter.” The Annals of Statistics 24, no. 3 (June 1996): 1346–70. https://doi.org/10.1214/aos/1032526973.

Koenker, Roger, and Stephen Portnoy. “M Estimation of Multivariate Regressions.” Journal of the American Statistical Association 85, no. 412 (1990): 1060–68. https://doi.org/10.2307/2289602.

Kudraszow, Nadia L., and Ricardo A. Maronna. “Estimates of MM Type for the Multivariate Linear Model.” Journal of Multivariate Analysis 102, no. 9 (October 2011): 1280–92. https://doi.org/10.1016/j.jmva.2011.04.011.

Lopuhaa, Hendrik P. “Asymptotics of Reweighted Estimators of Multivariate Location and Scatter.” The Annals of Statistics 27, no. 5 (1999): 1638–65.

Lopuhaä, Hendrik P. “Multivariate τ-Estimators for Location and Scatter.” The Canadian Journal of Statistics / La Revue Canadienne de Statistique 19, no. 3 (1991): 307–21. https://doi.org/10.2307/3315396.

Muler, Nora, and V´ictor J. Yohai. “Robust Estimation for Vector Autoregressive Models.” Computational Statistics & Data Analysis, Special issue on Robust Analysis of Complex Data, 65 (September 2013): 68–79. https://doi.org/10.1016/j.csda.2012.02.011.

Rousseeuw, Peter J., Stefan Van Aelst, Katrien Van Driessen, and Jose Agulló. “Robust Multivariate Regression.” Technometrics 46, no. 3 (2004): 293–305.

user: josef-pkt
issue_9094: state: open
title: WIP/ENH: add stats for complex valued random variables
body: see 
#9064
#3528

The main references that I used for this are in first comment of #9064.

currently mainly tools to convert random variable and cov between 3 representations (to park those formulas)
eventual target is OLS, MLE and hypothesis tests (wald) when params are complex valued.


user: josef-pkt
issue_9091: state: open
title: N-Step Predictions in SARIMAXResults.apply
body: #### Is your feature request related to a problem? Please describe
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

I am using SARIMAX to do an n-step prediction forecast. After fitting, I want to evaluate those predictions on my training data using SARIMAXResults.apply(). It only seems possible to evaluate my testing data.

#### Describe the solution you'd like
A clear and concise description of what you want to happen.

The ability to do something similar to [this](https://otexts.com/fpp2/forecasting-on-training-and-test-sets.html) in statsmodels. Where steps can be specified in apply, some new function to retrieve fittedvalues but for a multi-step forecast. 

#### Describe alternatives you have considered
A clear and concise description of any alternative solutions or features you have considered.

Have not found any alternative ways of doing this.

#### Context

N-step predictions are a major use case of SARIMAX modeling. The ability to generate predictions on both training and testing data is important.
user: samuel-gerstein
issue_9090: state: open
title: Breusch-Godfrey test dimensions mismatch
body: I'm trying to use Breusch-Godfrey test for ARDL(0, q) residuals, and I get a "ValueError: all the input array dimensions except for the concatenation axis must match exactly". It is due to the fact that list of residuals is shorter than list of exogenous variables by 2. Is it possible to omit the first q rows of exog_old before calling np.column_stack, or there is some econometric background for them staying there?
![изображение](https://github.com/statsmodels/statsmodels/assets/61021154/b1c979b1-effe-4f86-af2d-36bc2c439f0f)

Here is some example code:
`data = pd.DataFrame(np.random.rand(100, 2), columns=['x', 'y'])`

`model_ardl = ARDL(endog=data.y, lags=0, exog=data[['x']], order=2, causal=False).fit()`

`acorr_breusch_godfrey(model_ardl)`




user: ikuzmin404
issue_9084: state: open
title: Is this a bug for adfuller?
body: #### statsmodels Version

0.14.0

#### Describe the bug

Well, I am not sure if it is a bug since I am not a statistician.  It appears to be one.

Potentially incorrect `k_constant` being set to 1 when running OLS for regression='n'.  This is because by default, if hasconst is set to None (instead of false), OLS would try to detect there is an implicit constant in the input matrix (exog).

#### Code Sample

```python
import numpy as np
from statsmodels.tsa.stattools import adfuller
data = np.asarray ([3, 4, 4, 5, 6, 7, 6, 6, 7, 8, 9, 12, 10, 3, 4, 4, 5, 6, 7, 6, 6, 7, 8, 9, 12, 10, 3, 4, 4, 5, 6, 7, 6, 6, 7, 8, 9, 12, 10, 3, 4, 4, 5, 6, 7, 6, 6, 7, 8, 9, 12, 10, 3, 4, 4, 5, 6, 7, 6, 6, 7, 8, 9, 12, 10, 3, 4, 4, 5, 6, 7, 6, 6, 7, 8, 9, 12, 10, 3, 4, 4, 5, 6, 7, 6, 6, 7, 8, 9, 12, 10, 3, 4, 4, 5, 6, 7, 6, 6, 7, 8, 9, 12, 10, 3, 4, 4, 5, 6, 7, 6, 6, 7, 8, 9, 12, 10, 3, 4, 4, 5, 6, 7, 6, 6, 7, 8, 9, 12, 10])

results = adfuller(data, regression='n', maxlag=20, autolag="aic")
print (results)
```

##### Analysis

If you run the code through pdb and break at line 133 of stattools.py

Then run the following check k_constant flag
```python
for r in results: print ("r: %d, %.15g" % (r, results[r].k_constant))
```
The output is the following:
```
r: 1, 0
r: 2, 0
r: 3, 0
r: 4, 0
r: 5, 0
r: 6, 0
r: 7, 0
r: 8, 0
r: 9, 0
r: 10, 0
r: 11, 0
r: 12, 0
r: 13, 1
r: 14, 1
r: 15, 1
r: 16, 1
r: 17, 1
r: 18, 1
r: 19, 1
r: 20, 1
r: 21, 1
```

Note that I ran adfuller with regression being 'n', that is, there should be no added intercept (constant).  This is the case for the first 12 lags.  Mysteriously, the 13th lag had `k_constant` set to 1, indicating having intercept constant column.

I found out that the issue here is that adfuller runs OLS which automatically detects whether or not there is a constant present in the data even if there is no explicit constant.  In statsmodels/base/data.py line 172 (check_implicit is true and hasconst is None), it uses the approach of adding a const column (all ones) and then checking the rank, vs the rank before to check if there is an implicit constant column.  This resulted in `k_constant` being set to 1.  This is despite the fact that the min / max values for all the columns are different.  While the idea makes sense for OLS, I am not exactly sure it makes sense for adfuller.

Because of `k_constant` is changed from 0 to 1, the degrees of freedom calculations are changed.  In turn, a number of stats values associated with the linear model are changed.

The change in stats led to subsequent changes in stats values computed.  The bestlag picked is affected.  The final calculation for the usedlag is different as well.

Is this a bug or feature for statsmodels' adfuller?
user: coconut2015
issue_9081: state: open
title: VECM calculation calls matrix inverse directly which is not numerically stable; a proposal for a better solution
body: #### Describe the bug

There is vast literature in numerical linear algebra that advise against inverting the matrix directly unless you have a very good reason. Specifically in vecm.py, the _sij function calls matrix multiplication 6 times, and matrix inversion twice. This is not optimal for numerically stability, which is crucial for statistical computing applications. 

My proposed improvement calls matrix multiplication 3 times, and SVD twice, and no matrix inversion needs to be called. Here is the code that produces exact same result.

#### Code Sample, a copy-pastable example if possible
statsmodels/statsmodels/tree/main/statsmodels/tsa/vector_ar/vecm.py
line 416
```python
def _sij(delta_x, delta_y_1_T, y_lag1):
    """Returns matrices and eigenvalues and -vectors used for parameter
    estimation and the calculation of a models loglikelihood.

    Parameters
    ----------
    delta_x : ndarray (k_ar_diff*neqs x nobs)
        (dimensions assuming no deterministic terms are given)
    delta_y_1_T : ndarray (neqs x nobs)
        :math:`(y_1, \\ldots, y_T) - (y_0, \\ldots, y_{T-1})`
    y_lag1 : ndarray (neqs x nobs)
        (dimensions assuming no deterministic terms are given)
        :math:`(y_0, \\ldots, y_{T-1})`

    Returns
    -------
    result : tuple
        A tuple of five ndarrays as well as eigenvalues and -vectors of a
        certain (matrix) product of some of the returned ndarrays.
        (See pp. 294-295 in [1]_ for more information on
        :math:`S_0, S_1, \\lambda_i, \\v_i` for
        :math:`i \\in \\{1, \\dots, K\\}`.)

    References
    ----------
    .. [1] Lütkepohl, H. 2005. *New Introduction to Multiple Time Series Analysis*. Springer.
    """
    nobs = y_lag1.shape[1]
    r0, r1 = _r_matrices(delta_y_1_T, y_lag1, delta_x)
    ``` existing code
    # s00 = np.dot(r0, r0.T) / nobs
    # s01 = np.dot(r0, r1.T) / nobs
    # s10 = s01.T
    # s11 = np.dot(r1, r1.T) / nobs
    # s11_ = inv(_mat_sqrt(s11))
    # # p. 295:
    # s01_s11_ = np.dot(s01, s11_)
    # eig = np.linalg.eig(s01_s11_.T @ inv(s00) @ s01_s11_)
    ```
    # start of proposed improvement
    u0, _, v0h = svd(r0.T, full_matrices=False)
    u1, _, v1h = svd(r1.T, full_matrices=False)
    R = u0.T @ u1 @ v1h
    eig = np.linalg.eig(R.T @ R)
    # end of proposed improvement
    lambd = eig[0]
    v = eig[1]
    # reorder eig_vals to make them decreasing (and order eig_vecs accordingly)
    lambd_order = np.argsort(lambd)[::-1]
    lambd = lambd[lambd_order]
    v = v[:, lambd_order]
    return s00, s01, s10, s11, s11_, lambd, v
```
Why it works:

My proposed solution only requires 4 lines of code. The derivation is as follows:
We are calculating the eigenvalues and eigenvectors of
```
s01_s11_.T @ inv(s00) @ s01_s11_
```
We will replace r0 and r1 by their SVD in this expression. We see that the diagonal matrix of singular values cancel out. The numerical instability of inverse comes from ill-conditioned matrix where we take the inverse of singular values that are very close to 0, and taking the inverse of values very close to zero make the result blow up.


user: hrbzkm98
issue_9080: state: open
title: LU Decomposition error
body: While using SARIMAX, I got an unsual error, Kindly help me out with it.

        ---------------------------------------------------------------------------
        LinAlgError                               Traceback (most recent call last)
        [<ipython-input-14-97b9c4f34c56>](https://localhost:8080/#) in <cell line: 20>()
             22 
             23     model = SARIMAX(training_data, order=(2, 1, 2), seasonal_order=(2, 1, 2, 12))
        ---> 24     model_fit = model.fit()
             25 
             26     output = model_fit.get_forecast(steps=1)

        
        14 frames
        [/usr/local/lib/python3.10/dist-packages/statsmodels/tsa/statespace/representation.py](https://localhost:8080/#) in _initialize_state(self, prefix, complex_step)
           1056             if not self.initialization.initialized:
           1057                 raise RuntimeError('Initialization is incomplete.')
        -> 1058             self._statespaces[prefix].initialize(self.initialization,
           1059                                                  complex_step=complex_step)
           1060         else:

        
        statsmodels/tsa/statespace/_representation.pyx in statsmodels.tsa.statespace._representation.dStatespace.initialize()

        
        statsmodels/tsa/statespace/_representation.pyx in statsmodels.tsa.statespace._representation.dStatespace.initialize()

        
        statsmodels/tsa/statespace/_initialization.pyx in statsmodels.tsa.statespace._initialization.dInitialization.initialize()

        
        statsmodels/tsa/statespace/_initialization.pyx in statsmodels.tsa.statespace._initialization.dInitialization.initialize_stationary_stationary_cov()

        
        statsmodels/tsa/statespace/_tools.pyx in statsmodels.tsa.statespace._tools._dsolve_discrete_lyapunov()

        
        LinAlgError: LU decomposition error.
user: rohan472000
issue_9075: state: open
title: Gibbs sampling for Markov Regression
body: #### Is your feature request related to a problem? Please describe
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
MLE in Markov switching regression  often result in models failing to escaping local maximas, causing errors when running a regression.
#### Describe the solution you'd like
A clear and concise description of what you want to happen.
As suggested in albert and Chib(1993) and  Luca Di Persio and Matteo Frigo (2016), Gibbs sampling  approach can be used to overcome disadvantages of MLE approach used by conventional markov switching models.   #### Describe alternatives you have considered
A clear and concise description of any alternative solutions or features you have considered.

#### Additional context
Add any other context about the feature request here.
user: nj370
issue_9074: state: open
title: ENH: S-estimator and CM-estimator for outlier robust estimation
body: just parking references to read more carefully for implementation
I don't find an issue specifically for S-estimator.

related #8997 MM-estimator

Do we base the estimation on an objective function or on just the estimating equations?
For the former we need the nonlinear version of RLM to optimize w.r.t. beta and sigma. e.g. #3261
For the latter we either need explicit iteration (as in IRLS) alternating between beta and sigma iterations or rootfinding based M-estimators. #5674 #7436


Arslan, O. “A Simple Test to Identify Good Solutions of Redescending M Estimating Equations for Regression.” In Developments in Robust Statistics, edited by Rudolf Dutter, Peter Filzmoser, Ursula Gather, and Peter J. Rousseeuw, 50–61. Heidelberg: Physica-Verlag HD, 2003. https://doi.org/10.1007/978-3-642-57338-5_4.

Arslan, O., O. Edlund, and H. Ekblom. “Algorithms to Compute CM- and S-Estimates for Regression.” Metrika 55, no. 1 (April 1, 2002): 37–51. https://doi.org/10.1007/s001840200185.

———. “Algorithms to Compute CM- and S-Estimates for Regression.” In Developments in Robust Statistics, edited by Rudolf Dutter, Peter Filzmoser, Ursula Gather, and Peter J. Rousseeuw, 62–76. Heidelberg: Physica-Verlag HD, 2003. https://doi.org/10.1007/978-3-642-57338-5_5.

The last two references seem to be the same article.

user: josef-pkt
issue_9073: state: open
title: Holidays with statsmodel X13 Arima
body: Hi, 

I am trying to make Seasonal Adjustment with 
res = sm.tsa.x13_arima_analysis(pandas_series,x12path=r"route\x13as.exe")
And I don't know how to implement the functionality of the holidays, we have the exog variable but I can't find any example to use it properly.

Does someone know how to handle holidays?
user: xcxcxcxcxcxcxcxcxc
issue_9072: state: open
title: Value of 'season' while using yearly data
body: I have total of 40 years of pentad data (5 days for each year). For the purpose of analysis, I have taken pentad average to get 1 value for each year. For STL decomposition of this data what value should be chosen for `season` and `period` qualifiers.
user: ramaaslekar
issue_9071: state: open
title: ENH: make diagnostic and specification tests outlier robust or resistant
body: another general theme

Make diagnostic measures and specification tests outlier robust, which can be especially sensitive if we need higher moments.
The only issue we have, AFAIR, is jarque-bera #8286

(This is a bit of a companion to making specification test misspecification robust)


just parking two references (found by chance, I did not look specifically for this topic)

Alih, Ekele, and Hong Choon Ong. “An Outlier-Resistant Test for Heteroscedasticity in Linear Models.” Journal of Applied Statistics 42, no. 8 (August 3, 2015): 1617–34. https://doi.org/10.1080/02664763.2015.1004623.

Berenguer-Rico, Vanessa, and Ines Wilms. “Heteroscedasticity Testing after Outlier Removal.” Econometric Reviews 40, no. 1 (January 2, 2021): 51–85. https://doi.org/10.1080/07474938.2020.1735749.





user: josef-pkt
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_9070: state: open
title: ENH/DOC  WLS when weights are not var_weights (e.g. survey weighting)
body: Reading Li and Valliant
https://github.com/statsmodels/statsmodels/issues/9067#issuecomment-1823299955

They have a case where the model implied cov_params for WLS params is

cov(beta) = sigma^2 A^{-1} X' W^2 X A^{-1} where A = X' W X

why is W squared in the inner part of the sandwich?

From GLS/WLS:

params = (X' W X)^{-1} X' W y
sandwich cov_params cov(beta) = A^{-1} [X' W V W X] A^{-1}  if V = cov(u) (nobs, nobs)

- If W is correctly specified var_weights, then W = V^{-1}, then sandwich simplifies to cov(beta) = sigma^2 A^{-1} = sigma^2 (X' W X)^{-1} 
   This is default nonrobust cov_params of WLS 
- If W is not correctly specified var_weights, then
  - general case: We need the (HC) sandwich
  - special case: V = I, i.i.d., homoscedastic variance but W is some weighting we use, then
     cov(beta) = sigma^2 A^{-1} X' W^2 X A^{-1}

(see also Green chapter on heteroscedasticity, ch. 11 in 5th edition)


Conclusions:
- WLS cov_params is incorrect if weights are not correctly specified inverse variances.
- generic sandwich form (Huber, White, GMM, HC) provides correct cov_params
- We might want to impose additional assumptions on cov(momend_conditions) and avoid robustness to unspecified misspecification/heteroscedasticity., e.g. the last case above.

The last one is currently not available in WLS or GLM with some weights.

related
#7861
#6794 
#1379 RLM cov_params is not HC
#5327 cov_type for Theil/Ridge, maybe related
#1074 modelled but misspecified heteroscedasticity WLS+HC
and others

It's not yet clear to me what this implies for different models and cases.
IPW, survey weighting: poisson with some weights but assuming correctly specified distribution ?



 
user: josef-pkt
issue_9069: state: open
title: Can statsmodels.MNLogit handle scenarios with only two categories?
body: #### Is your feature request related to a problem? Please describe
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Can statsmodels.MNLogit handle scenarios with only two categories, or does it automatically convert to sm.Logit in cases where there are exactly two categories?
If statsmodels.MNLogit encounters a scenario with only two categories, it will essentially behave like the standard logistic regression, and the model will automatically simplify to sm.Logit.

My dataset contains instances with only two categories and others with more than two categories. I would like to know if statsmodels.MNLogit has the functionality to handle both scenarios. I appreciate your assistance.


#### Describe the solution you'd like
A clear and concise description of what you want to happen.

If statsmodels.MNLogit encounters a scenario with only two categories, it will essentially behave like the standard logistic regression, and the model will automatically simplify to sm.Logit.


#### Describe alternatives you have considered
A clear and concise description of any alternative solutions or features you have considered.

#### Additional context
Add any other context about the feature request here.
user: Ritabear
issue_9068: state: open
title: ENH: combined model checks, diagnostics, outlier influence, multicollinearity
body: support semi-automatic model checking, diagnostics, data problems, ...

#6381, #1908 sanity checks, for model/data (pre-estimation checks), not post-estimation results.
#1126 White's specification test
#7834 diagnostic classes, get_diagnostic

"
Even with a vast arsenal of diagnostics, it is very hard to write down rules that can be used to guide a data analysis. So much is really subjective and subtle.
"
in concluding remarks of
Welsch, Roy E. “[Influential Observations, High Leverage Points, and Outliers in Linear Regression]: Comment.” Statistical Science 1, no. 3 (August 1986): 403–5. https://doi.org/10.1214/ss/1177013625.

He is referring to an MIT software project that tried to have support for automatic "decision making". 
Based on the low number of citation of the referenced paper by Oldford and Peters and of follow-up papers, this never became popular.

One tool we could add is an expanded version of the regression summary table 3.
e.g. a new set of classes ModelChecks that attaches and delegates to the influence, Diagnostic, Multicollinearity, ... classes and provides an overall summary.
For outlier influence we would need explicit thresholds to issue alerts.

I guess we should add an option for "data relationship" type like "sequential", "cross-section", "grouped" to have the appropriate diagnostics,
e.g. ols summary combines sequential autocorrelation tests and cross-section tests which will not apply to all types of data.

For example, we could warn users 
if they use a nonrobust cov_type but have large heteroscedasticity or residual correlation or excess dispersion. IM ratio could choose sandwich according to the data structure.
or if some outlier influence measures like cooks distance are very high for some observations.

This will only become relevant once we have the Diagnostic classes and corresponding specification test for the different models.





user: josef-pkt
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_9067: state: open
title: ENH: WLS outlier influence
body: https://github.com/statsmodels/statsmodels/blob/5598e1034a76a515fa0d71e28bdb8d1c32d9ecd7/statsmodels/stats/outliers_influence.py#L517C36-L517C36

Is it planned to include weighted Studentized residuals to WLS summary table?
(The paper https://doi.org/10.1007/BF03014491 contains some formulas.)
user: lapkin25
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_9064: state: open
title: ENH: statistics and distribution for complex valued random variables
body: related to #3528 but more general or different focus

We could add statistics and models for complex random variables with almost the usual coverage (mean, cov, params, ...).

I did not find much on standard statistics, standard errors, cov_params, asymptotic distribution for hypothesis tests.
However, complex normal distributions are defined through their bivariate representation of real and imaginary parts.
So, we can use all the standard statistics and hypothesis for real and imaginary parts.

If random variable is second order circular (alias proper), then this imposes restrictions on the covariance of the real and imaginary parts. That covariance has to be constructed specifically for those `proper` cases. (This case can also be directly computed with "standard" complex computation.)
If complex random variable is improper, then it corresponds to a bivariate normal distribution without restrictions on cov (except positive semidefinite)

what we need:

- functions that work directly for `proper` complex random variables
- functions, classes for bivariate random variables, e.g. MultivariateOLS
- inference for both single hypothesis (on one real or imaginary part of parameters), e.g. use cov_params for [p.real, p.imag] and joint hypothesis for complex parameter (i.e. using joint distribution of real and complex parts)
- wrapper function/class
  - internally convert to whichever representation is convenient
  - double `results` for both complex parameters and bivariate representation of parameters, methods to specify hypothesis on complex parameter and methods for hypothesis on [real, imag] parts of complex parameters. 
- tools:
  - conversion between different representations (complex, bivariate-real, extended complex)
  - descriptive statistics, e.g. cov, pcov
  - specific functions, e.g. test for second order circularity, Taylor estimate of complex scatter
- specific models, OLSComplexCircular, OLSComplexImproper and MultivariateLS (multivariate complex endog) versions
  - models corresponding to nonlinear LS might be more frequently used than linear models 
- multivariate statistics (maybe eventually)
  - independent component analysis ICA seems to be popular
  - CCA, e.g. used for canonical correlation for x and x.conj()
  - hypothesis tests for patterned cov, (and pcov ?)
  - ... ?

definitions
- circular: distribution is rotation invariant, i.e. assumption on full distribution, all moments (original definition)
- second order circular (proper): assumption on second moments (pseudo-cov is zero), no assumption on higher moments

disadvantage
I guess this will not have a large user base. Signal processing literature does not have much traditional statistics (including inference on mean parameters), and econometrics in signal processing is more time series analysis and forecasting.

main references (not complete)

Adali, Tülay, Peter J. Schreier, and Louis L. Scharf. “Complex-Valued Signal Processing: The Proper Way to Deal With Impropriety.” IEEE Transactions on Signal Processing 59, no. 11 (November 2011): 5101–25. https://doi.org/10.1109/TSP.2011.2162954.

Ollila, E., and V. Koivunen. “Generalized Complex Elliptical Distributions.” In Processing Workshop Proceedings, 2004 Sensor Array and Multichannel Signal, 460–64, 2004. https://doi.org/10.1109/SAM.2004.1502990.

Ollila, Esa. “On the Circularity of a Complex Random Variable.” IEEE Signal Processing Letters 15 (2008): 841–44. https://doi.org/10.1109/LSP.2008.2005050.

Ollila, Esa, Jan Eriksson, and Visa Koivunen. “Complex Elliptically Symmetric Random Variables—Generation, Characterization, and Circularity Tests.” IEEE Transactions on Signal Processing 59, no. 1 (January 2011): 58–69. https://doi.org/10.1109/TSP.2010.2083655.

Ollila, Esa, Visa Koivunen, and H. Vincent Poor. “Complex-Valued Signal Processing — Essential Models, Tools and Statistics.” In 2011 Information Theory and Applications Workshop, 1–10, 2011. https://doi.org/10.1109/ITA.2011.5743596.

Ollila, Esa, David E. Tyler, Visa Koivunen, and H. Vincent Poor. “Complex Elliptically Symmetric Distributions: Survey, New Results and Applications.” IEEE Transactions on Signal Processing 60, no. 11 (November 2012): 5597–5625. https://doi.org/10.1109/TSP.2012.2212433.

Picinbono, B. “On Circularity.” IEEE Transactions on Signal Processing 42, no. 12 (December 1994): 3473–82. https://doi.org/10.1109/78.340781.

———. “Second-Order Complex Random Vectors and Normal Distributions.” IEEE Transactions on Signal Processing 44, no. 10 (October 1996): 2637–40. https://doi.org/10.1109/78.539051.

Picinbono, B., and P. Bondon. “Second-Order Statistics of Complex Signals.” IEEE Transactions on Signal Processing 45, no. 2 (February 1997): 411–20. https://doi.org/10.1109/78.554305.

user: josef-pkt
issue_9063: state: open
title: The predict function error
body: ![image](https://github.com/statsmodels/statsmodels/assets/69052141/288ff943-d1a4-4bd5-bca1-49817c92c50d)
``
![image](https://github.com/statsmodels/statsmodels/assets/69052141/47db5b3b-0edc-46a4-bc4d-498085c5c1a9)

![image](https://github.com/statsmodels/statsmodels/assets/69052141/5527b8db-ed7f-4027-a830-5797e2788cbb)

user: kingtut83
issue_9062: state: open
title: DynamicFactorMQ is not working 
body: I am running [this example](https://www.chadfulton.com/topics/statespace_large_dynamic_factor_models.html) 

In the past I had no problems replicating tis example. Now it seems has something changed and I get the following errors when I  try to create the DFM object: 

```

# Construct the dynamic factor model
model = sm.tsa.DynamicFactorMQ(
    endog_m, endog_quarterly=endog_q,
    factors=factors, factor_orders=factor_orders,
    factor_multiplicities=factor_multiplicities)


```
```

Traceback (most recent call last):
  File "/Users/safishajjouz/opt/anaconda3/envs/myport_management_env/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3378, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-55-40928c00cd95>", line 2, in <module>
    model = sm.tsa.DynamicFactorMQ(
  File "/Users/safishajjouz/opt/anaconda3/envs/myport_management_env/lib/python3.8/site-packages/statsmodels/tsa/statespace/dynamic_factor_mq.py", line 1320, in __init__
    s = self._s = DynamicFactorMQStates(
  File "/Users/safishajjouz/opt/anaconda3/envs/myport_management_env/lib/python3.8/site-packages/statsmodels/tsa/statespace/dynamic_factor_mq.py", line 443, in __init__
    self.factor_block_orders.append(default_block_orders))
  File "/Users/safishajjouz/opt/anaconda3/envs/myport_management_env/lib/python3.8/site-packages/pandas/core/generic.py", line 5989, in __getattr__
    return object.__getattribute__(self, name)
AttributeError: 'Series' object has no attribute 'append'
```
user: msh855
issue_9058: state: open
title: ENH: GLM check for valid endog
body: https://stats.stackexchange.com/questions/630757/glm-invalid-value-encountered-in-log-special-gammaln

uses GLM with Binomial family
endog is a count variable but no n_trials is specified.
Results look wrong, but only a warning in gammaln is issued.

My guess is that somewhere in the computation we don't check that the terms in the computation are in the valid range.

related issue: inappropriate models like log-binomial
Those are for "foreign" links or variance functions that don't impose the correct domain.
But, I think that does not affect checking for valid endog in required range (QMLE can have incorrect endog, e.g. continuous instead of discrete, but I guess it still has to satisfy bound constraints for support of distribution [0, 1], [0, n_trials], R+, R++.

code from the question, with added imports:

```
import numpy as np
import pandas as pd
import statsmodels.api as sm
import statsmodels.formula.api as smf

# creating the Numpy array 
array = np.array([[4., 441, 25, 4, 17], 
                  [10, 444, 49, 8, 9], 
                  [3, 483, 12, 2, 38],
                  [2,447,10,1,43],
                  [4,423,22,3,19],
                  [3,19,19,3,1],
                  [18,445,111,17,4],
                  [7,423,42,7,10],
                  [9,426,53,8,8],
                  [12,378,54,8,7],
                  [36,450,225,36,2],]) 
   
# creating a list of column names 
column_values = ['CONST', 'ACTUAL_PRODTIME', 'TIME_LOGO', 'COST_PPL_LOGO', 'LOGO'] 
  
# creating the dataframe 
df = pd.DataFrame(data = array,   
                  columns = column_values) 

model = smf.glm(
                formula = "CONST ~ ACTUAL_PRODTIME + TIME_LOGO + COST_PPL_LOGO + LOGO",
                data = df, 
                family = sm.families.Binomial())
# Fit the model
result = model.fit()

# Display and interpret results
print(result.summary())
```
user: josef-pkt
issue_9056: state: open
title: How to use X-13ARIMA-SEATS in Ubuntu?
body: Hi,
I would like to use X-13ARIMA-SEATS with Ubuntu.  Do I understand correctly that in:

`statsmodels.tsa.x13.x13_arima_select_order(endog, 
maxorder=(2, 1), 
maxdiff=(2, 1),
 diff=None,
 exog=None, 
log=None, 
outlier=True, 
trading=False, 
forecast_periods=None,
 start=None, freq=None, 
print_stdout=False, 
x12path=None,
 prefer_x13=True, 
tempdir=None)`

I have to use x12path='\x13as_asciisrc-v1-1-b60' with files inside:  

> aaamain.f
abend.f
ac02ae.i
acfar.f
acfast.i
acfdgn.f
acf.f
acfhdr.f
acfptr.prm
acfst.i

I read page: [https://www.census.gov/data/software/x13as.X-13ARIMA-SEATS.html#list-tab-635278563](https://www.census.gov/data/software/x13as.X-13ARIMA-SEATS.html#list-tab-635278563) but I'm still confused.
Should I compile the code in Fortran? 

I will be grateful for any suggestions.


user: PeterPirog
issue_9054: state: open
title: BUG: regression summary() computes diagnostic statistics even if slim=True
body: use case #3528
complex data in OLS raises exception in `summary()`. It fails in omni_normtest.

If I use `.summary(slim=True)`, then that still raises at the same place because the diagnostics are computed even though we don't include them in the summary.


user: josef-pkt
issue_9052: state: open
title: ENH: option to flip plot axes, vertical error bars in tukeyhsd plot_simultaneous
body: plot_simultaneous has horizontal error bars. pairs are on y-axis, diff values are on x axis.

request for flipping x and y axis:
https://stackoverflow.com/questions/77409496/flipping-the-x-y-axis-in-diagram-resulting-from-plot-simultaneous-in-a-tukey

we might have other similar functions that might benefit for the same option
e.g.  the standard plot in meta-analysis also has horizontal error bars.


user: josef-pkt
issue_9051: state: open
title: Panel Vector Autoregression
body: Hi, I saw statsmodel currently has VAR
https://www.statsmodels.org/dev/generated/statsmodels.tsa.vector_ar.var_model.VAR.html

But, I didn't find VAR for panel dataset.
Could you please consider add the panel var?

Thank you.
user: wenxie18
issue_9050: state: open
title: ENH: GMM special cases: exponential mean function (log-link) and quadratic or power variance function
body: looking by chance at Basu, Rathouz 2005 again https://github.com/statsmodels/statsmodels/issues/624#issuecomment-1494917639

and a bit at
Manning, Willard G., Anirban Basu, and John Mullahy. “Generalized Modeling Approaches to Risk Adjustment of Skewed Outcomes Data.” Journal of Health Economics 24, no. 3 (May 1, 2005): 465–88. https://doi.org/10.1016/j.jhealeco.2004.09.011.

Basu Rathouz have 3 types of moment conditions, wrt. mean, link and variance parameters.
It does not necessarily need a quasi-likelihood function, e.g. variance function is `a0 + a1 mu + a2 mu**2` or `a * mu**p)
(we might have some other issues with variance functions that also include generalized Poisson GPP, NBP variance.)
BR use box-cox link for mean function.

 This should be reasonably easy to implement as generic GMM. We need a model with the moment conditions.
We don't need the full multi-link version of models if we use hardcoded parameterized link functions.

For GMM start_params we could take Poisson, exp mean and var = mu


user: josef-pkt
issue_9048: state: open
title: Recursive errors with BetaModel
body: #### Describe the bug

[A clear and concise description of what the bug is. This should explain **why** the current behaviour is a problem and why the expected output is a better solution.]

#### Code Sample, a copy-pastable example if possible


```python
from statsmodels.othermod import betareg

beta_glm = betareg.BetaModel(Y_dum_f1.astype(float), X_dum.astype(float))
beta_result = beta_glm.fit()
beta_result.summary()
```
<details>
With same Y and X, I ran a couple of GLM models in statsmodels.api, but with BetaModel I constantly face Error message that is blank:

The expected output is a summary table of the model result. 
My data has a total of 7776 observations, endog is a 1-d array and exog is 7776*10 (10 variables).
<img width="823" alt="Screenshot 2023-10-30 at 12 17 57 PM" src="https://github.com/statsmodels/statsmodels/assets/102279867/9467e0a8-1af7-4b06-b114-3ccd97928784">

user: sarangerin
issue_9047: state: open
title: BUG: Adjust eval environment for stack depth
body: - [x] closes #9037
- [x] tests added / passed. 
- [x] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

This PR fixes the formula environment for the conditional logit and Poisson, the proportional hazards model, GEE, and QIF.

user: rob-sil
issue_9045: state: open
title: ENH: measurement error with calibration sample, repeat measurements
body: #3187 assumes known measurement error (co)variances

another strand of literature (and an R package) looks at simultaneously estimating the measurement error or it's variances by using more information from either and internal subsample or an external sample.

I have not looked much at the details, one reference for now.
The commonly used term is "calibration".
Some parts are similar to IV, but not with the same full sample (either subsample or separate sample)
(We might run into #8773 when reusing GMM or similar horizontal stacking)

Freedman, Laurence S., Douglas Midthune, Raymond J. Carroll, and Victor Kipnis. “A Comparison of Regression Calibration, Moment Reconstruction and Imputation for Adjusting for Covariate Measurement Error in Regression.” Statistics in Medicine 27, no. 25 (November 10, 2008): 5195–5216. https://doi.org/10.1002/sim.3361.

(This is currently not on my near term roadmap)
user: josef-pkt
issue_9044: state: open
title: ENH: misclassification, measurement error for categorical, binary variables
body: related to #3187 measurement error models

What if respondents to a survey don't answer with or remember the "true" answer to a binary or multiple choice question?

chapters 2 and 3 for simple `stats.proportions` 

Buonaccorsi, John P. Measurement Error: Models, Methods, and Applications. New York: Chapman and Hall/CRC, 2010. https://doi.org/10.1201/9781420066586.

related problem for measurement errors/misclassifiction of regressors 
user: josef-pkt
issue_9043: state: open
title: ENH: control function IV with multinomial treatment
body: related
#8756 categorical, multinomial, multiple treatment with conditional independence `treatment` effect
#7689 general issue for control function approach using IV for endogenous treatment.

The basic model extension that we need here is to use MNLogit or other categorical endog models (OrderedModel ?) as first stage for residual inclusion or control function methods.

I never used MNLogit for something similar (e.g. as moment conditions in GMM). I don't know if MNLogit has all the pieces.

- residuals, pearson, generalized ?
- score_obs and moment conditions

e.g.
Geraci, Andrea, Daniele Fabbri, and Chiara Monfardini. “Testing Exogeneity of Multinomial Regressors in Count Data Models: Does Two-Stage Residual Inclusion Work?” Journal of Econometric Methods 7, no. 1 (January 1, 2018). https://doi.org/10.1515/jem-2014-0019.
(earliest versions in my Zotero, working papers in 2014 and 2015)


related: 
MNLogit as outcome model with endogenous regressors. What's the status of score_test in MNLogit? AFAIR, I never tried.


user: josef-pkt
issue_9039: state: open
title: Leybourne
body: - [ ] closes #xxxx
- [ ] tests added / passed. 
- [ ] code/documentation is well formatted.  
- [ ] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

<details>


**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in main and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/main -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>

user: JimVaranelli
issue_9038: state: open
title: ENH: MNLogit identification with categorical variables
body: https://stackoverflow.com/questions/77352027/mnlogit-summary-return-a-array-with-a-lot-of-nan

My guess would be that if we have categorical variables, then we need observations for each endog/exog-cat cell.
But I'm not sure what identification condition we need.

possible workaround:
add option `hessian_pinv=True` to use pinv for hessian that is only semi-definite but not positive definite.


user: josef-pkt
issue_9037: state: open
title: BUG: Cox Model from formula doesn't grab the correct environment
body: #### Describe the bug

Constructing a proportional hazard model from a formula with `smf.phreg` (aka `PHReg.from_formula`) doesn't use the proper environment.

Specifically, `PHReg` overrides the `from_formula` class method but doesn't adjust or set the `eval_env` keyword. It calls `Model.from_formula` to parse the formula, but `Model.from_formula` doesn't know that it was called indirectly and therefore underestimates its depth by one. In the default case, `Model.from_formula` pulls the environment from where it was called, but this is `hazard_regression.py` rather than the user's environment (where `PHReg.from_formula` was called). User-defined symbols aren't found, though internal ones like `np` are available.

This problem should also show up for all the other subclass implementations of `from_formula`s (e.g., `GEE`, `QIF`, `OrderedModel`).

#### Code Sample, a copy-pastable example if possible


```python
import pandas as pd
import statsmodels.formula.api as smf

data = pd.DataFrame(
	[
		{"y": 2, "x": 2},
		{"y": 2, "x": 4},
		{"y": 4, "x": 6},
		{"y": 4, "x": 8},
	]
)

# Works
smf.phreg("y ~ x", data=data).fit()

def plus_one(x):
	return x + 1

# Works
smf.ols("y ~ plus_one(x)", data=data).fit()

# PatsyError from NameError: name 'plus_one' is not defined
smf.phreg("y ~ plus_one(x)", data=data).fit()
```

#### Expected Output

A clear and concise description of what you expected to happen.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

INSTALLED VERSIONS
------------------
Python: 3.11.5.final.0
OS: Darwin
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

statsmodels
===========

Installed: 0.14.0

Required Dependencies
=====================

cython: Not installed
numpy: 1.26.1
scipy: 1.11.3
pandas: 2.1.1
    dateutil: 2.8.2
patsy: 0.5.3

Optional Dependencies
=====================

matplotlib: Not installed
cvxopt: Not installed
joblib: Not installed

Developer Tools
================

IPython: Not installed
    jinja2: Not installed
sphinx: Not installed
    pygments: Not installed
pytest: Not installed
virtualenv: Not installed

</details>

user: rob-sil
issue_9036: state: open
title: ENH: Improved performance of the ConditionalMNLogit class
body: - [x] tests added / passed. 
- [x] code/documentation is well formatted.  
- [x] properly formatted commit message. See [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

ENH - [ConditionalMNLogit](https://github.com/statsmodels/statsmodels/blob/main/statsmodels/discrete/conditional_models.py) class: in `loglike` method replaces the nested loop to numpy vector operations. In  `score` method, variable `v` is calculated at a time for the selected values (exp_sum). The number 'p' is used to access them.
As a result, `test_conditional_mnlogit_grad `is calculated 2.7 times faster, and `test_conditional_mnlogit_2d`, `test_conditional_mnlogit_3d `1.7 times faster.

Below is a description of how to compare the old and new options.

I separated `test_conditional_`, `conditional_models_` into separate files so that I could run the old and new options locally together for comparison (located them in a common directory). In conditional_models there are two classes `ConditionalMNLogit` and `ConditionalMNLogit_new` - in which I made improvements. In tests, the module is connected directly to use both options for tests. `n = 90` is passed `to gen_mnlogit(n)`

Below is the `test_conditional_` code. The `conditional_models_` code is not posted due to line limitations, I am attaching it as an attachment.

<details>

<summary>test_conditional_  - code</summary>

```
import numpy as np
from conditional_models_ import ConditionalMNLogit, ConditionalMNLogit_new
from statsmodels.tools.numdiff import approx_fprime
from numpy.testing import assert_allclose
import pandas as pd


def gen_mnlogit(n):
    np.random.seed(235)

    g = np.kron(np.ones(5), np.arange(n // 5))
    x1 = np.random.normal(size=n)
    x2 = np.random.normal(size=n)
    xm = np.concatenate((x1[:, None], x2[:, None]), axis=1)
    pa = np.array([[0, 1, -1], [0, 2, -1]])
    lpr = np.dot(xm, pa)
    pr = np.exp(lpr)
    pr /= pr.sum(1)[:, None]
    cpr = pr.cumsum(1)
    y = 2 * np.ones(n)
    u = np.random.uniform(size=n)
    y[u < cpr[:, 2]] = 2
    y[u < cpr[:, 1]] = 1
    y[u < cpr[:, 0]] = 0

    df = pd.DataFrame({"y": y, "x1": x1,
                       "x2": x2, "g": g})
    return df


def test_conditional_mnlogit_grad_new(n):
    df = gen_mnlogit(n)
    model = ConditionalMNLogit_new.from_formula(
        "y ~ 0 + x1 + x2", groups="g", data=df)

    # Compare the gradients to numeric gradients
    for _ in range(5):
        za = np.random.normal(size=4)
        grad = model.score(za)
        ngrad = approx_fprime(za, model.loglike)
        assert_allclose(grad, ngrad, rtol=1e-5, atol=1e-3)


def test_conditional_mnlogit_grad(n):
    df = gen_mnlogit(n)
    model = ConditionalMNLogit.from_formula(
        "y ~ 0 + x1 + x2", groups="g", data=df)

    # Compare the gradients to numeric gradients
    for _ in range(5):
        za = np.random.normal(size=4)
        grad = model.score(za)
        ngrad = approx_fprime(za, model.loglike)
        assert_allclose(grad, ngrad, rtol=1e-5, atol=1e-3)


def test_conditional_mnlogit_2d_new(n):
    df = gen_mnlogit(n)
    model = ConditionalMNLogit_new.from_formula(
        "y ~ 0 + x1 + x2", groups="g", data=df)
    result = model.fit()

    #"""
    # Regression tests
    assert_allclose(
        result.params,
        np.asarray([[0.75592035, -1.58565494],
                    [1.82919869, -1.32594231]]),
        rtol=1e-5, atol=1e-5)
    assert_allclose(
        result.bse,
        np.asarray([[0.68099698, 0.70142727],
                    [0.65190315, 0.59653771]]),
        rtol=1e-5, atol=1e-5)
    #"""


def test_conditional_mnlogit_2d(n):
    df = gen_mnlogit(n)
    model = ConditionalMNLogit.from_formula(
        "y ~ 0 + x1 + x2", groups="g", data=df)
    result = model.fit()

    # Regression tests
    #"""
    assert_allclose(
        result.params,
        np.asarray([[0.75592035, -1.58565494],
                    [1.82919869, -1.32594231]]),
        rtol=1e-5, atol=1e-5)
    assert_allclose(
        result.bse,
        np.asarray([[0.68099698, 0.70142727],
                    [0.65190315, 0.59653771]]),
        rtol=1e-5, atol=1e-5)
    #"""


def test_conditional_mnlogit_3d_new(n):
    df = gen_mnlogit(n)
    df["x3"] = np.random.normal(size=df.shape[0])
    model = ConditionalMNLogit_new.from_formula(
        "y ~ 0 + x1 + x2 + x3", groups="g", data=df)
    result = model.fit()

    #"""
    # Regression tests
    assert_allclose(
        result.params,
        np.asarray([[0.729629, -1.633673],
                    [1.879019, -1.327163],
                    [-0.114124, -0.109378]]),
        atol=1e-5, rtol=1e-5)

    assert_allclose(
        result.bse,
        np.asarray([[0.682965, 0.60472],
                    [0.672947, 0.42401],
                    [0.722631, 0.33663]]),
        atol=1e-5, rtol=1e-5)

    #"""

    # Smoke test
    result.summary()


def test_conditional_mnlogit_3d(n):
    df = gen_mnlogit(n)
    df["x3"] = np.random.normal(size=df.shape[0])
    model = ConditionalMNLogit.from_formula(
        "y ~ 0 + x1 + x2 + x3", groups="g", data=df)
    result = model.fit()

    #"""
    # Regression tests
    assert_allclose(
        result.params,
        np.asarray([[0.729629, -1.633673],
                    [1.879019, -1.327163],
                    [-0.114124, -0.109378]]),
        atol=1e-5, rtol=1e-5)

    assert_allclose(
        result.bse,
        np.asarray([[0.682965, 0.60472],
                    [0.672947, 0.42401],
                    [0.722631, 0.33663]]),
        atol=1e-5, rtol=1e-5)

    #"""

    # Smoke test
    result.summary()

import datetime

n = 90

arr = [[test_conditional_mnlogit_grad_new,
        test_conditional_mnlogit_grad,
        'test_conditional_mnlogit_grad'],
       [test_conditional_mnlogit_2d_new,
        test_conditional_mnlogit_2d,
        'test_conditional_mnlogit_2d'],
       [test_conditional_mnlogit_3d_new,
        test_conditional_mnlogit_3d,
        'test_conditional_mnlogit_3d']]

for i in arr:
    print(i[2]) # name of the called function

    now = datetime.datetime.now()
    i[1](n)
    time_old = datetime.datetime.now() - now

    now = datetime.datetime.now()
    i[0](n)
    time_new = datetime.datetime.now() - now

    print(f'time_old = {time_old}, time_new = {time_new}, '
          f'time_old/time_new = {time_old/time_new}')

    print()
```


</details>

[conditional_models_.py.tar.gz](https://github.com/statsmodels/statsmodels/files/13063756/conditional_models_.py.tar.gz)

Output:
```
n = 90

test_conditional_mnlogit_grad 
time_old = 0:00:00.680232, time_new = 0:00:00.236433, time_old/time_new = 2.8770603088401367
test_conditional_mnlogit_2d
time_old = 0:00:01.908332, time_new = 0:00:01.057139, time_old/time_new = 1.8051855054065737
test_conditional_mnlogit_3d
time_old = 0:00:02.635174, time_new = 0:00:01.468636, time_old/time_new = 1.7943002895203441

n = 2000 # with assert commented 

test_conditional_mnlogit_grad 
time_old = 0:00:16.267467, time_new = 0:00:06.169248, time_old/time_new = 2.636863844669561
test_conditional_mnlogit_2d
time_old = 0:00:45.773023, time_new = 0:00:25.878624, time_old/time_new = 1.768757991151307
test_conditional_mnlogit_3d
time_old = 0:00:57.176253, time_new = 0:00:33.577012, time_old/time_new = 1.7028392222631366
```

<details>

<summary>**Notes**</summary>


* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in main and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/main -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>

user: quant12345
issue_9035: state: open
title: ENH: diagnostic plots with errors in variables, instrumental variables, measurement error
body: problem that I was not aware of as generic issue

In diagnostic plots like residual plots, CPR, ..., we want to have no (or known) pattern of the graph if the null model holds. This requires that there is no correlation between the variables on x and y axis under the null. Deviations can be interpreted as violation of null model.

Problem with errors in variables can be that "residual" are correlated with the plot-x-variable.
This means we need to choose the variables on the axis so that they are (approximately) orthogonal.
(similar to orthogonality conditions in specification testing, CMT, ...)

issue triggered by measurement error, where we should use the "predicted exog" and measurement error corrected residuals.

Carroll, R. J., and C. H. Spiegelman. “Diagnostics for Nonlinearity and Heteroscedasticity in Errors-in-Variables Regression.” Technometrics 34, no. 2 (May 1, 1992): 186–96. https://doi.org/10.1080/00401706.1992.10484907.

Romeo, Giovanni, John P. Buonaccorsi, and Magne Thoresen. “Detecting and Correcting for Heteroscedasticity in the Presence of Measurement Error.” Communications in Statistics - Simulation and Computation 0, no. 0 (2023): 1–17. https://doi.org/10.1080/03610918.2023.2190061.
article only for simple regression

brief search for IV
https://socialsciences.mcmaster.ca/jfox/Courses/SORA-TABA/slides-mixed-models-ivs.pdf
second part has diagnostics for IV2SLS, with references
(using 2nd stage regressors, predicted by instruments, sounds ok)
R ivreg package has vignette with various diagnostics for 2SLS https://cran.r-project.org/web/packages/ivreg/vignettes/Diagnostics-for-2SLS-Regression.html

after residual inclusion, control function:
brief google search finds nothing on specification testing (except exogeneity) or diagnostic plots.
For residual plot with respect to endogenous regression we might need to add the control function/residual, i.e. use predicted regressor.
In GLM we should also be able to use working residuals for "linear predictor residuals" in diagnostic plots. (needs reference search)





user: josef-pkt
issue_9034: state: open
title: ENH: TOST for quantiles, tolerance interval
body: mainly parking a reference
(found looking for Bland-Altman agreement comparisons)

Gwowen Shieh (2020) Assessing Agreement Between Two Methods of
Quantitative Measurements: Exact Test Procedure and Sample Size Calculation, Statistics in
Biopharmaceutical Research, 12:3, 352-359, DOI: 10.1080/19466315.2019.1677495

I didn't try to understand the details.
looks at TOST test for the equivalence range of a normal quantile, one-sample interpreted as the diff of paired 2 samples.
small quantiles indicates agreement between 2 paired measuring methods.

related to tolerance intervals, i.e. get confidence interval for quantiles or for region between 2 quantiles with required coverage/content.

 (AFAIR, Shieh's articles were useful for other stats)
user: josef-pkt
issue_9033: state: open
title: ENH: Berkson measurement error, missing unobserved random effect, WLS
body: (trying to summarize basic setup for Berkson measurement error, not verified with references yet)

x is know, observed or fixed (e.g. a choice variable of agent)
true value of exog that affects endog y has an extra term, x* = x + v
causal structure for endog
y ~ x* b + e

causal effect of choice variable x:
y ~ (x + v) b + e = x b + (v b + e) = x b + u, where u = v b + e

Assume cov(x, v) = 0, cov(x, e) = 0, or independence
then cov(x, u) = 0 and OLS is consistent but not efficient (unless var(u_i) is constant)
This fails if V(v | x) != V(v), does it? We can still have E(v | x) = 0.
(check again what assumptions we need. I'm not sure right now.)

Using weight when non-constant error variances, s2v_i, s2e_i
then weights for WLS should be based on s2e_i + b**2 s2v_i or s2e_i + b' Vv_i b   (Vv = cov of vector v).
Requires iterative WLS to update weights with beta estimates, (similar to HetWLS/FGLS)

For nonlinear models, GLM, discrete:
the independent measurement error in exog creates unobserved heterogeneity and adds a mixing distribution component to the conditional distribution F(y | x). 
We have related issues for GLM/discrete with other reasons for unobserved random effects. e.g. Poisson with normal mixing distribution.


possible to implement:
for linear model we just need the iterative WLS to get the model often used for error-in-variables in some fields like chemometrics.
cov_params: WLS (weights taken as given), fully parametric (weights are function of params) or sandwich?
 

thoughts on equation error:
- y-measurement error, no equation error, `e_i` given, `scale=1` fixed
- split equation error multiplicative: scale is estimated by WLS, implicitly multiplies variance defined by weights.
- y-measurement error, equation error on y: `e_i = ve_i + q` equation error is additive in residual e and u., `scale=1` fixed, ve_i known sigma_q needs to be estimated (mainly guessing here)



user: josef-pkt
issue_9032: state: open
title: linear mixed model with more than one group
body: I am trying to model the data with more than one group in `mixedlm`.  However, I am fully unable to do this despite the fact that it is in the documentation:

```
groups : 1d array-like
A vector of labels determining the groups – data from different groups are independent
``` 


```python
import statsmodels.api as sm
import statsmodels.formula.api as smf
data = sm.datasets.get_rdataset("dietox", "geepack").data
md = smf.mixedlm("Weight ~ Time", data, groups=data[["Pig","Litter"]])
mdf = md.fit()
print(mdf.summary())
```

```
TypeError: unhashable type: 'numpy.ndarray' 
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[77], line 7
      3 import statsmodels.formula.api as smf
      5 data = sm.datasets.get_rdataset("dietox", "geepack").data
----> 7 md = smf.mixedlm("Weight ~ Time", data, groups=data[["Pig","Litter"]])
      9 mdf = md.fit()
     11 print(mdf.summary())

File e:\Users\Hamed\miniconda3\Lib\site-packages\statsmodels\regression\mixed_linear_model.py:1046, in MixedLM.from_formula(cls, formula, data, re_formula, vc_formula, subset, use_sparse, missing, *args, **kwargs)
   1044 kwargs["exog_vc"] = exog_vc
   1045 kwargs["groups"] = groups
-> 1046 mod = super(MixedLM, cls).from_formula(
   1047     formula, data, *args, **kwargs)
   1049 # expand re names to account for pairs of RE
   1050 (param_names,
   1051  exog_re_names,
   1052  exog_re_names_full) = mod._make_param_names(exog_re_names)

File e:\Users\Hamed\miniconda3\Lib\site-packages\statsmodels\base\model.py:229, in Model.from_formula(cls, formula, data, subset, drop_cols, *args, **kwargs)
    223         design_info = design_info.subset(cols)
    225 kwargs.update({'missing_idx': missing_idx,
    226                'missing': missing,
    227                'formula': formula,  # attach formula for unpckling
    228                'design_info': design_info})
--> 229 mod = cls(endog, exog, *args, **kwargs)
    230 mod.formula = formula
    231 # since we got a dataframe, attach the original

File e:\Users\Hamed\miniconda3\Lib\site-packages\statsmodels\regression\mixed_linear_model.py:793, in MixedLM.__init__(self, endog, exog, groups, exog_re, exog_vc, use_sqrt, missing, **kwargs)
    789 self.k_params = self.k_fe + self.k_re2
    791 # Convert the data to the internal representation, which is a
    792 # list of arrays, corresponding to the groups.
--> 793 group_labels = list(set(groups))
    794 group_labels.sort()
    795 row_indices = dict((s, []) for s in group_labels)

TypeError: unhashable type: 'numpy.ndarray'
```
<details>

```
INSTALLED VERSIONS
------------------
Python: 3.11.4.final.0

statsmodels
===========

Installed: 0.14.0 (e:\Users\xxx\miniconda3\Lib\site-packages\statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.24.3 (e:\Users\xxx\miniconda3\Lib\site-packages\numpy)
scipy: 1.11.1 (e:\Users\xxx\miniconda3\Lib\site-packages\scipy)
pandas: 2.0.3 (e:\Users\xxx\miniconda3\Lib\site-packages\pandas)
    dateutil: 2.8.2 (e:\Users\xxx\miniconda3\Lib\site-packages\dateutil)
patsy: 0.5.3 (e:\Users\xxx\miniconda3\Lib\site-packages\patsy)

Optional Dependencies
=====================

matplotlib: 3.7.1 (e:\Users\xxx\miniconda3\Lib\site-packages\matplotlib)
    backend: module://matplotlib_inline.backend_inline 
cvxopt: Not installed
joblib: 1.2.0 (e:\Users\xxx\miniconda3\Lib\site-packages\joblib)

Developer Tools
================

IPython: 8.12.2 (e:\Users\xxx\miniconda3\Lib\site-packages\IPython)
    jinja2: 3.1.2 (e:\Users\xxx\miniconda3\Lib\site-packages\jinja2)
sphinx: Not installed
    pygments: 2.15.1 (e:\Users\xxx\miniconda3\Lib\site-packages\pygments)
pytest: Not installed
virtualenv: Not installed
```
</details>

user: hamedha2
issue_9030: state: open
title: ENH: measurement error: corrected score function in GLM/LEF with canonical link
body: related: linear case #3187

for GLM with canonical link, it is possible to correct the score for covariate measurement errors.
We could add versions of Poisson, Logit, Gaussian, and not for GLM (unless we special case GLM when link is canonical)

This will need optimization based on moment conditions, estimating equations (I have not seen an artificial objective function so far) 
Corrected score function is nonlinear and not of the single index form, e.g. extra quadratic term in params/beta.

good overview chapter 8, Gaussian and Poisson case in section 8,3,2 p. 142
Yi, Grace Y., Aurore Delaigle, and Paul Gustafson, eds. Handbook of Measurement Error Models. New York: Chapman and Hall/CRC, 2021. https://doi.org/10.1201/9781315101279.


Aside:
extra term in corrected score looks a bit like characteristic function approach to endogenous regressor case.
However, we don't have an estimate of the individual error term, instead we have known variances of the covariate (measurement) error.
  
user: josef-pkt
issue_9025: state: open
title: BUG: GLM scale is not updated when computing null model
body: bug or feature ?

GLMResults.null computes null model estimate, but only returns fittedvalues
GLMResults.llnull computes with null fittedvalues, but uses scale from full model

https://stats.stackexchange.com/questions/628309/python-statsmodels-glm-log-likelihood-of-null-model

**update**

llnull does not seem to be unit tested (except Poisson)
main unit tests are for `null_deviance`
It looks like null_deviance does not take scale into account at all (scale is fixed at 1.)

There are some unit tests for pseudo-rsquared/prsquared, but I don't see which models are covered by it.

(I never really looked at this for families that have scale != 1)


user: josef-pkt
issue_9024: state: open
title: ENH: conformalized quantile conditional prediction intervals for count data, poisson
body: this is a special case of #9005 but for discrete data

quick experiments with 2-sample poisson (constant + dummy), assuming correctly specified model:

If means are 5 and 7.5 than discreteness for upper tail ppf(0.9) is large.
If I use ppf(0.9) from get_distribution on new sample for the upper threshold, then the tail prob (relative frequency) is only 0.03.
When I use ppf(0.9) - 1, then the tail rel. frequency is 0.13.
This means that the ppf(0.9) threshold produces valid, but conservative intervals. We could do only better by either randomizing or shifting the threshold only for a subsample.

If I increase poisson means to 50 and 75, then ppf(0.9) has can have incorrect coverage and ppf(0.9) + d or - d will produce better coverage.

My calibration sample is nobs=100, and results vary quite a bit, i.e. threshold correction `d` will be pretty random.
(estimation sample has nobs=200). (I initially set the random.seed and then dropped it)

some thoughts,

- results might change with continuous exog, I expect better coverage on average, but discretens remain conditionally.
- results will change with misspecified model, e.g. excess dispersion where poisson.ppf is not valid and adjustment is important.
- with small means, the discreteness might overwhelm additional uncertainty that is not included in ppf.
- threshold correction `d` only needs to use integer values because observations are discrete (check d in range(-dlow, +dupp) with expanding range if needed)
- definition of threshold: threshold point is considered to be in prediction interval, "outlying observations" are > or <.
  I used `yn > qupp - d, (yn > qupp - d).mean()` where qupp is predicted ppf(0.9).
- alternative: use multiplicative d instead of additive d, then all ceil(ppf(q) + d * m) would be the discretized thresholds (see reference that compares conformalized quantile regression methods using different calibration score functions)

 

user: josef-pkt
issue_9022: state: open
title: DOC: PoissonResults is not in the docs
body: PoissonResults is missing in discretemode.rst
no SeeAlso link from the Poisson class, e.g. fit

strange, PoissonResults existed forever

also
discretemode.rst base class list at end of file is missing some results classes like CountResults.
but CountResults is included in list of specific results classes
NegativeBinomialPResults  seems also to be missing

I guess all count subclasses have their own results class.


user: josef-pkt
issue_9021: state: open
title: ENH/DOC: GMM, 2 linear equations, supply and demand
body: Do we have an example of a simple model with 2 linear simultaneous equations estimated by GMM?
e.g standard market equilibrium with quantity and price, supply and demand functions

(I don't remember the details.)

https://stackoverflow.com/questions/77107115/choices-for-optimization-method-with-generalized-method-of-moments-gmm-in-pyth

https://ocw.mit.edu/courses/14-382-econometrics-spring-2017/resources/mit14_382s17_lec3/

related:
Should we support multi-equation (IV) GMM in a more convenient way?
Computational shortcuts for linear system using linear algebra instead of generic nonlinear optimization?
or leave linear cases to `linearmodels`?
user: josef-pkt
issue_9020: state: open
title: Some questions about SARIMAX 
body: Regarding the SARIMAX model:

1. In the seasonal parameter of `SARIMAX`, we have `(P, D, Q,, s)`. If I have hourly data with daily cycles, should `s` be 365?

2. What is the difference between `SARIMAXResults.append(endog, refit = False)` and `SARIMAXResults.extend(endog)`?
user: lubin-liu
issue_9019: state: open
title: Documentation error in the description of the NegativeBinomial GLM family
body: I'm looking into NegativeBionomial GLM models and got stuck with their description in the documentation.

https://github.com/statsmodels/statsmodels/blame/main/docs/source/glm.rst defines the $b(\theta)$ function for the NegativeBionomial family as:

$-\frac{1}{\alpha}\log(1-\alpha e^\theta)$

I believe the correct form is:

$-\frac{1}{\alpha}\log(1-e^\theta)$

I have not explored further yet. I very much hope this is just a doc typo. 
user: iossifov
issue_9018: state: open
title: Pickle with a model using fit_constrained
body: #### Describe the bug
Loading a model in a pickle if the model was fit using fit_constrained and formula cause an AttributeError: 'numpy.ndarray' object has no attribute 'join'

#### Code Sample, a copy-pastable example if possible

```python
import statsmodels.api as sm
from statsmodels.formula.api import glm
from statsmodels.genmod import families
import pandas as pd
import numpy as np
import scipy.special
rng = np.random.default_rng(seed=42)

# Generate data
N = 100
X = rng.normal(size=N)
Y = rng.normal(size=N)
a = 0.5
p = scipy.special.expit(a*X+(1-a)*Y)
obs = [rng.choice([0,1], p=[q, 1-q]) for q in p]

df = pd.DataFrame({"X":X, "Y":Y, "obs":obs})

# Fit model
mod = glm("obs ~ X + Y - 1", data=df, family=families.Binomial()).fit_constrained("X + Y = 1")

# Save in pickle
fichier = "sm_model.pickle"
mod.save(fichier)

# Load from pickle
mod = sm.load_pickle(fichier)
```
<details>

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

AttributeError: 'numpy.ndarray' object has no attribute 'join'

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]
INSTALLED VERSIONS
------------------
Python: 3.8.16.final.0
OS: Darwin 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul  5 22:17:35 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T8112 arm64
byteorder: little
LC_ALL: None
LANG: fr_FR.UTF-8

statsmodels
===========

Installed: 0.14.0 (/Users/guillaume/miniconda3/lib/python3.8/site-packages/statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.24.4 (/Users/guillaume/miniconda3/lib/python3.8/site-packages/numpy)
scipy: 1.10.1 (/Users/guillaume/miniconda3/lib/python3.8/site-packages/scipy)
pandas: 1.5.3 (/Users/guillaume/miniconda3/lib/python3.8/site-packages/pandas)
    dateutil: 2.8.2 (/Users/guillaume/miniconda3/lib/python3.8/site-packages/dateutil)
patsy: 0.5.3 (/Users/guillaume/miniconda3/lib/python3.8/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.7.1 (/Users/guillaume/miniconda3/lib/python3.8/site-packages/matplotlib)
Installed osx event loop hook.
    backend: MacOSX
cvxopt: Not installed
joblib: 1.2.0 (/Users/guillaume/miniconda3/lib/python3.8/site-packages/joblib)

Developer Tools
================

IPython: 8.12.0 (/Users/guillaume/miniconda3/lib/python3.8/site-packages/IPython)
    jinja2: Not installed
sphinx: Not installed
    pygments: 2.15.1 (/Users/guillaume/miniconda3/lib/python3.8/site-packages/pygments)
pytest: Not installed
virtualenv: Not installed

</details>

user: TennisForecasts
issue_9017: state: open
title: ENH: quantile regression for discrete, count data
body: I'm not sure how difficult it is to implement any of those.
(motivated by conformal quantile regression https://github.com/statsmodels/statsmodels/issues/9005#issuecomment-1735680098 )
I think it would be also useful as diagnostic tool for parametric count regression when we are interested in the entire predictive distributions or some quantiles of it.

Carcaiso, Viviana, and Leonardo Grilli. “Quantile Regression for Count Data: Jittering versus Regression Coefficients Modelling in the Analysis of Credits Earned by University Students after Remote Teaching.” Statistical Methods & Applications, October 12, 2022. https://doi.org/10.1007/s10260-022-00661-2.

Frumento, Paolo, and Matteo Bottai. “Parametric Modeling of Quantile Regression Coefficient Functions.” Biometrics 72, no. 1 (2016): 74–84. https://doi.org/10.1111/biom.12410.

Geraci, Marco, and Alessio Farcomeni. “Mid-Quantile Regression for Discrete Responses.” Statistical Methods in Medical Research 31, no. 5 (May 1, 2022): 821–38. https://doi.org/10.1177/09622802211060525.

Machado, José A. F, and J. M. C. Santos Silva. “Quantiles for Counts.” Journal of the American Statistical Association 100, no. 472 (December 1, 2005): 1226–37. https://doi.org/10.1198/016214505000000330.

user: josef-pkt
issue_9014: state: open
title: Cannot cast ufunc 'subtract' output from dtype('float64') to dtype('int64') with casting rule 'same_kind'
body: please help me guys, I have problems with my coding, this is my first time using python.


---------------------------------------------------------------------------
UFuncTypeError                            Traceback (most recent call last)
Cell In[83], line 2
      1 var_model = VARMAX(train_df, order=(4,0),enforce_stationarity= True)
----> 2 fitted_model = var_model.fit(disp=False)
      3 print(fitted_model.summary())

File ~\anaconda3\Lib\site-packages\statsmodels\tsa\statespace\mlemodel.py:650, in MLEModel.fit(self, start_params, transformed, includes_fixed, cov_type, cov_kwds, method, maxiter, full_output, disp, callback, return_params, optim_score, optim_complex_step, optim_hessian, flags, low_memory, **kwargs)
    530 """
    531 Fits the model by maximum likelihood via Kalman filter.
    532 
   (...)
    647 statsmodels.tsa.statespace.structural.UnobservedComponentsResults
    648 """
    649 if start_params is None:
--> 650     start_params = self.start_params
    651     transformed = True
    652     includes_fixed = True

File ~\anaconda3\Lib\site-packages\statsmodels\tsa\statespace\varmax.py:348, in VARMAX.start_params(self)
    346 if self.k_trend > 0 or self.k_exog > 0:
    347     trendexog_params = np.linalg.pinv(exog).dot(endog)
--> 348     endog -= np.dot(exog, trendexog_params)
    349     if self.k_trend > 0:
    350         trend_params = trendexog_params[:self.k_trend].T

UFuncTypeError: Cannot cast ufunc 'subtract' output from dtype('float64') to dtype('int64') with casting rule 'same_kind'

user: laluna87
issue_9013: state: open
title: ENH/reference: Zero-Inflated Nonnegative Continuous Data
body: mainly parking a reference:

Liu, Lei, Ya-Chen Tina Shih, Robert L. Strawderman, Daowen Zhang, Bankole A. Johnson, and Haitao Chai. “Statistical Analysis of Zero-Inflated Nonnegative Continuous Data: A Review.” Statistical Science 34, no. 2 (May 2019): 253–79. https://doi.org/10.1214/18-STS681.
https://github.com/joyfulstones/zero-inflated-continuous  examples using SAS

compares Tobit with 2-part models

2-part models are similar to count models with a mixture of a mass-point distribution and a distribution for values (weakly) larger than the threshold, in this case the distribution is for non-negative continuous data, e.g. log-normal or gamma.
This is also similar to Tweedie.

The interesting cases are when we cannot estimate the two parts separately (which we do right now in hurdle count models).
Example: if there is correlation  (and tools will be similar to sample selection models with unobserved heterogeneity and/or endogeneity, I guess)










user: josef-pkt
issue_9012: state: open
title: REF/ENH: jackknife - what should we add?
body: I'm reading up a bit on jackknife, mainly for looo residuals and looo changes in params. #9008
(The main reason that I never looked at it more carefully is that we can use bootstrap  as more general version of jackknife when we are willing to use repeated full estimation or cross-validation for other use cases. However, in many cases jackknife can be used without repeated estimation loops.)

some possible uses

- bias correction for parameter estimates
  - I don't know whether it's worth it 
- jackknife cov_params
  - very close to White sandwich cov_type, e.g. Weber 1986  difference in OLS is only small sample correction n / (n - k)
  - #8461 cluster robust cov_type and jackknife with leave one groupcluster out  (I did not look at references for leave one group out)
  - We can stick to White sandwiches and look only at jackknife if it adds anything to that.
- other usages (main target for now)
  - diagnostic (looo)
  - conformal prediction intervals #9005 

Weber, N. C. “The Jackknife and Heteroskedasticity: Consistent Variance Estimation for Regression Models.” Economics Letters 20, no. 2 (January 1, 1986): 161–63. https://doi.org/10.1016/0165-1765(86)90165-5.
Weber has bias corrected cov_params at the end, which IIRC is HC3  (residuals are corrected by diag of hatmatrix)

user: josef-pkt
issue_9010: state: open
title: ENH: Ridge regression, jackknife and looo change in params
body: just parking a note for computational shortcut

Khuran et al 2014 top of page 5252  for standard parameterization of generalized Ridge

Most of the jackknife-Ridge literature uses canonical parameterization, i.e. Ridge penalty is on orthogonalized parameters (eigenvalues/eigenvectors of x'x).
Nyquist section 2.1 refers to Woodbury, (i.e. Sherman-Morrison) formula. But as far as I could check, it does not use the property that the moment and penalization matrices are diagonal matrices. 


Nyquist, Hans. “Applications of the Jackknife Procedure in Ridge Regression.” Computational Statistics & Data Analysis 6, no. 2 (March 1, 1988): 177–83. https://doi.org/10.1016/0167-9473(88)90048-5.

Khurana, Mansi, Yogendra P. Chaubey, and Shalini Chandra. “Jackknifing the Ridge Regression Estimator: A Revisit.” Communications in Statistics - Theory and Methods 43, no. 24 (December 17, 2014): 5249–62. https://doi.org/10.1080/03610926.2012.729640.


related #9008 looo, jackknife residuals
user: josef-pkt
issue_9009: state: open
title: ENH: OLS Influence, more statistics without explicit looo loop
body: question on mailing list

"
I am wondering why the computation times of the dfbeta and dfbetas influence statistics are extremely large. The reason seems to be that the implementation uses results form leave-one-out regression loops, requiring N auxilliary regressions (where N is the number of observations of the data set). In the classical monography by Belsley et al. it is shown that these statistics can be computed extremely fast, even without running OLS regressions.

Why are the algorithms presented in the book not implemented?
"

related: Closed form studentized residuals external #373
 

user: josef-pkt
issue_9008: state: open
title: ENH: outlier, influence  add looo, jackknife residuals
body: motivated by https://github.com/statsmodels/statsmodels/issues/9005#issuecomment-1729932721

We don't have explicit looo, jackknife residuals (resid_response).
I'm not sure whether we already have all the computation for it, for other statistics.

looo residuals look like a useful tool.

In the GLM, discrete cases we could also add other residuals based on looo, e.g. resid_pearson.
other predictive statistics, looo-fittedvalues, looo-implied-var and similar might also be useful.
(model.predict has "which" option and takes params as argument, but likely not a 2dim params vectorized by rows)



user: josef-pkt
issue_9007: state: open
title: Statsmodels ARIMA, reconstructing model prediction with exogenous variable
body: I'm building an ARIMA model using statsmodels.tsa.arima.model, version 0.14.0. I'm having issues manually reproducing the output of the predict method on the training dataset when including exogenous variables, the difference is a million times larger than when manually reproducing a pure AR model. 

Based on this  [link](https://www.statsmodels.org/dev/examples/notebooks/generated/statespace_sarimax_faq.html#Reconstructing-residuals,-fitted-values-and-forecasts-in-SARIMAX-and-ARIMA) I've tried to reproduce the functional form of the ARIMA equation. Assuming I have an ARIMA(1,1,0) model with an exogenous variable, would the predict method use an ARIMA of the form 

$y_{t+1}=y_t+\beta X_{t+1}+\alpha(y_t-y_{t-1}-\beta X_t)$ 

where $y_t$ is my endogenous variable, $X_t$ is the exogenous variable, $\alpha$  and $\beta$ are the autoregressive and exogenous coefficient returned by  `.params`? 
user: ersc96
issue_9005: state: open
title: ENH: conformal, distribution-free, nonparametric prediction intervals
body: This looks like a recent hot topic mainly for machine learning.

Basic idea: use calibration data, separate from estimation/training data, to estimate quantiles and prediction sets or intervals for new observations.

simplest case: In regression with additive residual, we can add/subtract quantile of calibration residuals to the get conformal prediction interval (several alternatives of splitting data, ex-ante split, jacknife/looo, k-fold crossvalidation, ...)
 
https://mapie.readthedocs.io/en/latest/theoretical_description_regression.html    mapie seems to be the main python package for this

Lei, Jing, Max G’Sell, Alessandro Rinaldo, Ryan J. Tibshirani, and Larry Wasserman. “Distribution-Free Predictive Inference for Regression.” Journal of the American Statistical Association 113, no. 523 (July 3, 2018): 1094–1111. https://doi.org/10.1080/01621459.2017.1307116.

large number of references (to much to figure out for now what would be useful for statsmodels)


limitation:
It looks like the coverage "guarantee" is only for average, marginal coverage (over sample of exog, similar to ATE), but there is no distribution-free "guarantee" for the coverage conditional on a specific exog value.

Foygel Barber, Rina, Emmanuel J Candès, Aaditya Ramdas, and Ryan J Tibshirani. “The Limits of Distribution-Free Conditional Predictive Inference.” Information and Inference: A Journal of the IMA 10, no. 2 (June 15, 2021): 455–82. https://doi.org/10.1093/imaiai/iaaa017.

Related: 
I saw some articles (abstracts) that include heteroscedasticity (var(exog)).
More literature looks at time series, i.e. without independence.

Count data:
I did not find any references for conformal prediction interval with count data (based on google scholar search)
This might be related to the above "limits" of conformal prediction intervals that we only have coverage with respect to marginal distribution.
I guess that the main problem to get prediction interval conditional on x would be to find a ranking statistic (prediction "score" function) that is distribution-free, e.g. does not depend on higher moments that depend on exog through e.g. mean.

I guess we can still get conditional (on x) prediction intervals under stronger assumptions, e.g.
Assume regression model with additive i.i.d. residuals. That would still be non-parametric for the residual distribution and better than the current prediction interval based on normal or t distributed residuals.

Aside: There is a literature for adaptive local conformal prediction (or something like that) but I did not even skim those.


Related: diagnostics for this ?
maybe
homogeneity, i.i.d. assumption  (e.g. subsample homogeneity like Hosmer-Lemeshow type of test)
Does calibration set exog reflect population or sample exog?
quantile regression on residual (to see whether quantiles do not depend on exog) 
Compare conformal prediction interval with parametric interval to see whether the latter is "good enough".

clarification:
in (general) regression setting (y, x) is assumed to be i.i.d. and coverage statement is over probability space of (y, x) and not for conditional (y | x)

so far we want the much stronger conditional prediction interval for y | x

user: josef-pkt
issue_9004: state: open
title: Maint: nonempty `statsmodels\__init__.py`
body: `statsmodels\__init__.py` now loads pandas and patsy, and all their non-lazy dependencies 

the reason  is a pandas 2.x compat fix in patsy
`from statsmodels.compat.patsy import monkey_patch_cat_dtype`


(I only looked at it because `import statsmodels` took much longer than expected)

Not a big problem (avoiding pandas and patsy is mainly useful in statsmodels.stats and a few other places)
And I don't see much of an alternative solution. We could put it in `statsmodels.base.model` or `data`, but then we might monkeypatch on each import.

Maybe fix shows up eventually in patsy, or it becomes obsolete if we use formulaic.





user: josef-pkt
issue_9003: state: open
title: BLD: Move to Meson
body: #### Is your feature request related to a problem? Please describe
Python 3.12 no longer supports distutils, so statsmodels cannot be built for 3.12.  The solution is to floolw SciPy/NumPy/pandas and move to meson.

#### Describe the solution you'd like
Switch build to meson.

#### Describe alternatives you have considered
Not aware of any.


user: bashtage
issue_9000: state: open
title: Feat/allow list of lags
body: - [ ] closes #xxxx (not applicable)
- [x] tests added / passed. 
- [x] code/documentation is well formatted.  
- [x] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

<details>


**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in main and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/main -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>

user: camaramm
issue_8998: state: open
title: ENH: fourier polynomial, splines for seasonal pattern with irregular periodicity
body: (semi-random thought seeing comments of users that want annual seasonality with daily data)

years and months don't have all the same number of periods. Assuming fixed periodicity (frequency, number of periods in a cycle) is not really appropriate.
Another example would be working week with unequal number of days if there are holidays/non-working days.

In this case, periodicity is "strictly exogenous".
Missing values would be similar but exogeneity is an assumption.

In GAM and similar splines, we don't need points on a (time) grid. We can convert to continuous, unequal spaced grid, eg. normalize days of the year to (max is) 365, and days can be real numbers, or skip periods if we have holidays, exogenously missing periods.

tsa.deterministic only allows for fixed periodicity `period`, AFAICS.

aside: for constructing forecast periods we could require a `period_base`, e.g. assuming next year has 365 days.

related
"analyzing seasonal time series with GAM"
https://petolau.github.io/Analyzing-double-seasonal-time-series-with-GAM-in-R/

(I might have looked at examples when checking season splines in GAM)

user: josef-pkt
issue_8997: state: open
title: MM estimators
body: In [this](https://github.com/statsmodels/statsmodels/blob/main/examples/notebooks/robust_models_1.ipynb) example notebook, it says that this package currently doesn't have MM estimators, but that they are being worked on. I'm just wondering if there has been any progress on implementing this and if so where I can find it as I'm interested in using an MM-estimator in python.
user: Nin17
issue_8996: state: open
title: Extend `summary2.summary_col` features
body: #### Is your feature request related to a problem? Please describe

I report results of my models via [`statsmodels.iolib.summary2.summary_col`](https://github.com/statsmodels/statsmodels/blob/v0.14.0/statsmodels/iolib/summary2.py#L471).
However, I have to work around the default implementation to fit my own requirements. These include:

1. Report t-stat instead of standard error under the coefficient,
2. Add the adjusted R² and number of observations,
3. Have a different float level for coefficient and t-stats (`'%.4f'` vs. `'%.2f').

#### Describe the solution you'd like

I would like to modify `summary_col` along the lines of something like this:

```diff
  def summary_col(
      results,
+     parentheses: str | None = "std_err",
-     float_format: str ='%.4f',
+     float_format: str | dict[str, str] ='%.4f',
      model_names=(),
      stars=False,
      info_dict=None,
      regressor_order=(),
      drop_omitted=False,
-     include_r2=True,
+     include_r2: bool =True, # adds R^2 only
+     include_adj_r2: bool =True,
+     include_n: bool =False,
  ):
```

Concretely, this means:

1. **`parentheses: str | None = "std_err"`**: Choose whether to add stats below the coefficient (cf. `None`). If so, which one (something along the line of `assert parentheses in ["std_err", "t_stat", "z_stat", "proba"]`).
2. **`float_format: str | dict[str, str] ='%.4f',`**: Pass either one format (current behaviour) that apply to all fields, or something like `{"coefficient": "%.4f", "parentheses": "%.5f", "N": "%d"}`
3. **`include_*`**: Split `include_r2` into two flags (one for R², and another for adjusted R²), and add new ones for N (and other fields?)

The features listed above a independent of each other. I'd be happy to raise a PR for each, but I'd prefer to check beforehand whether these would fit your roadmap — and if you have a better way to tackle my issues.

#### Describe alternatives you have considered

Regarding including adj-R² only and N, I am aware I can use `info_dict` (what I'm currently doing).
I know as well that I can write my own aggregator method, based on `statsmodels.iolib.summary.Summary` attributes.

However, I believe the features outlined above could be beneficial to others than me.

#### Additional context

Currently using `statsmodels==0.14.0`, and using OLS only.
user: EBoisseauSierra
issue_8995: state: open
title: MNLogit doesn't have names attribute
body: #### Describe the bug

The documents says MNLogit object has names (dict) attribute, but it seems it doesn't.
https://www.statsmodels.org/stable/generated/statsmodels.discrete.discrete_model.MNLogit.html

#### Code Sample, a copy-pastable example if possible


```python
import numpy as np
import pandas as pd
import statsmodels.api as sm

roulette=[14, 14, 5, 12, 6, 1, 12, 5, 15, 9, 12, 1, 24, 5, 4, 30, 6, 36, 1, 14,26,3,1,3,31,12,20,8,36,11,17,18,15,32,36,13,5,8,12,36,6,13,24,2,27,1,22,7,33,15,22,6,17,15,30,27,7,3,35,13,2,14,00]
cnt=0 
for i,x in enumerate(roulette):
    if x>0:
        roulette[i]=x%2
    else:
        roulette[i]=-1
data=pd.DataFrame({"val":roulette})
data["val1"]=data["val"].shift(1)
data["val2"]=data["val"].shift(2)
data["val3"]=data["val"].shift(3) 
data["val4"]=data["val"].shift(4)
data["val5"]=data["val"].shift(5)
data.dropna(inplace=True)
model=sm.MNLogit(data["val"],data[["val1","val2","val3","val4","val5"]]).fit()
model.model.names
```
output
```
AttributeError: 'MNLogit' object has no attribute 'names'
```


#### Expected Output

A dict should be returned.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>
INSTALLED VERSIONS
------------------
Python: 3.10.4.final.0

statsmodels
===========

Installed: 0.14.0 (c:\Users\hakos\AppData\Local\Programs\Python\Python310\lib\site-packages\statsmodels)

Required Dependencies
=====================

cython: 3.0.0 (c:\Users\hakos\AppData\Local\Programs\Python\Python310\lib\site-packages\Cython)
numpy: 1.25.1 (c:\Users\hakos\AppData\Local\Programs\Python\Python310\lib\site-packages\numpy)
scipy: 1.10.1 (c:\Users\hakos\AppData\Local\Programs\Python\Python310\lib\site-packages\scipy)
pandas: 1.4.2 (c:\Users\hakos\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas)
    dateutil: 2.8.2 (c:\Users\hakos\AppData\Local\Programs\Python\Python310\lib\site-packages\dateutil)
patsy: 0.5.3 (c:\Users\hakos\AppData\Local\Programs\Python\Python310\lib\site-packages\patsy)

Optional Dependencies
=====================

matplotlib: 3.5.1 (c:\Users\hakos\AppData\Local\Programs\Python\Python310\lib\site-packages\matplotlib)
    backend: module://matplotlib_inline.backend_inline 
cvxopt: Not installed
joblib: 1.1.0 (c:\Users\hakos\AppData\Local\Programs\Python\Python310\lib\site-packages\joblib)

Developer Tools
================

IPython: 8.2.0 (c:\Users\hakos\AppData\Local\Programs\Python\Python310\lib\site-packages\IPython)
    jinja2: Not installed
sphinx: Not installed
    pygments: 2.12.0 (c:\Users\hakos\AppData\Local\Programs\Python\Python310\lib\site-packages\pygments)
pytest: Not installed
virtualenv: Not installed
</details>

user: hakoshie
issue_8991: state: open
title: BUG: SARIMAX Extend with constant exog doesn't work
body: #### Describe the bug

Similar to this issue: https://github.com/statsmodels/statsmodels/issues/7019

It seems the above bug is solved, but when you try to extend a SARIMAX model with constant exog values, it throws an error

#### Code Sample, a copy-pastable example if possible


```python
from statsmodels.tsa.arima.model import ARIMA
import numpy as np

# A normal training demo

# Old data
old_data = np.random.rand(12*10, 1)
old_exo = np.random.rand(10, 1)
old_exo = np.repeat(old_exo, 12).reshape(-1, 1)

# Model fit with the available data
m = ARIMA(old_data, exog=old_exo)
res = m.fit()

# Some new data arrives
new_data = np.random.rand(12, 1)
new_exo = np.repeat(np.random.rand(1, 1), 12).reshape(-1, 1)

# The following line should not fail, but it does
res.extend(new_data, exog=new_exo)

```

We get  `ValueError: A constant trend was included in the model specification, but the `exog` data already contains a column of constants.`

#### Expected Output

It should just extend the model without checking if exog is constant or not, as exog changes every "h" periods. When I try to extend the model after doing "h" periods where exog doesn't change, we get the error above.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>


INSTALLED VERSIONS
------------------
Python: 3.10.11.final.0
OS: Linux 5.14.0-70.30.1.el9_0.x86_64 #1 SMP PREEMPT Thu Nov 3 20:29:04 UTC 2022 x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

statsmodels
===========

Installed: 0.14.0 (/mnt/lustre/ludwig/lqb955/conda_envs/air_forecast/lib/python3.10/site-packages/statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.23.5 (/mnt/lustre/ludwig/lqb955/conda_envs/air_forecast/lib/python3.10/site-packages/numpy)
scipy: 1.11.1 (/mnt/lustre/ludwig/lqb955/conda_envs/air_forecast/lib/python3.10/site-packages/scipy)
pandas: 2.0.3 (/mnt/lustre/ludwig/lqb955/conda_envs/air_forecast/lib/python3.10/site-packages/pandas)
    dateutil: 2.8.2 (/mnt/lustre/ludwig/lqb955/conda_envs/air_forecast/lib/python3.10/site-packages/dateutil)
patsy: 0.5.3 (/mnt/lustre/ludwig/lqb955/conda_envs/air_forecast/lib/python3.10/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.7.1 (/mnt/lustre/ludwig/lqb955/conda_envs/air_forecast/lib/python3.10/site-packages/matplotlib)
    backend: module://matplotlib_inline.backend_inline 
cvxopt: Not installed
joblib: 1.3.1 (/mnt/lustre/ludwig/lqb955/conda_envs/air_forecast/lib/python3.10/site-packages/joblib)

Developer Tools
================

IPython: 8.14.0 (/mnt/lustre/ludwig/lqb955/conda_envs/air_forecast/lib/python3.10/site-packages/IPython)
    jinja2: 3.1.2 (/mnt/lustre/ludwig/lqb955/conda_envs/air_forecast/lib/python3.10/site-packages/jinja2)
sphinx: Not installed
    pygments: 2.15.1 (/mnt/lustre/ludwig/lqb955/conda_envs/air_forecast/lib/python3.10/site-packages/pygments)
pytest: Not installed
virtualenv: Not installed


</details>

user: UgurKap
issue_8987: state: open
title: ENH: Treatment effect for binary endpoint, general outcome models
body: mainly parking a link

https://github.com/pzivich/zEpid/blob/master/zepid/causal/doublyrobust/AIPW.py

zepid: has binary treatment and binary outcome.
our current `treatment_effect` only allows for linear outcome (OLS)


user: josef-pkt
issue_8986: state: open
title: ENH/Design: models with several optional objective/loglike functions
body: partially related: #2949

base LikelihoodModel uses hardcoded method names for loglike, score, ... for nonlinear optimization.

Can we implement a model that has several possible, user chosen objective functions?
example: covariance analysis should have option for mle, ls, wls as objective function. All of those require nonlinear optimization in general.

problems:

- need a switch for selecting appropriate method, loglike, score, ... (e.g. NB assigns method)
- possible incorrect state if results instance computes extras after model has changed (repeated calls to fit)
  - This could be avoided if method choice is in `model.__init__`, but it would in general be different `fit` method.
  - More messy to implement would be to add method selection keyword to relevant model methods (loglike, ...). Results classes would have to transmit keyword. (similarly, we added `scale` keyword to some loglike, ... methods, GLM)

Alternative:
Create separate subclasses for each `fit` estimation method.




user: josef-pkt
issue_8983: state: open
title: fit the SARIMAX via "Conditional Least Square Estimation" similar to SAS PROC ARIMA
body: Thank you for the great work on statsmodels. 

I would like to fit the SARIMAX model via "Conditional Least Square Estimation" not just via "MLE". 

SAS Arima has this functionality but does the StatsModels have it?

When I create this model in SAS Arima, I get completely different results from SAS Arima 

How can I built the following AR-1 model by fitting it via "Conditional Least Square Estimation"?
Link for the "conditional least square" method is as follows. 
https://documentation.sas.com/doc/en/etscdc/14.2/etsug/etsug_arima_details12.htm

Example code that I tried: 
```
import pandas as pd
import numpy as np
from datetime import date
import statsmodels.api as sm

AIR = pd.DataFrame({'air': [112, 118, 132, 129, 121, 135, 148, 148, 136, 119, 104, 118,
                            115, 126, 141, 135, 125, 149, 170, 170, 158, 133, 114, 140,
                            145, 150, 178, 163, 172, 178, 199, 199, 184, 162, 146, 166]})

AIR.index = pd.date_range(start = '1949/1/1', periods = AIR.shape[0], freq = 'M')
AIR.index = [date(x.year, x.month, 1) for x in AIR.index]
AIR.loc[:,'const'] = list(np.ones(AIR.shape[0]))

mod = sm.tsa.statespace.SARIMAX(AIR['air'], order=(1,0,0), 
                                trend = 'c')
results = mod.fit(maxiter=50)   
results.summary()
```

Thank you!

user: unalunsal
issue_8982: state: open
title: Allow `sm.OLS` to work with `Float64` values
body: When working with `Float64` values form `pandas`, `sm.OLS` raises an error

`ValueError: Pandas data cast to numpy dtype of object. Check input data with np.asarray(data).`

Example

```
impor pandas as pd
import statsmodels.api as sm
x = pd.Series([1,2,3,4]).astype('Float64')
y = pd.Series([1,2,3,4]).astype('Float64')
sm.OLS(y,x).fit()
```

It is easy to convert the `Float64` into a `float64` value, but it would be nice if `sm.OLS` can handle that. 
user: marcdelabarrera
issue_8981: state: open
title: multi-variable logistic regression 
body: Does the sm.Logit() function automatically perform multi-variable/multiple logistic regression so that we can incorporate multiple covariates/predictors and the estimated coefficients are adjusted for the other covariates/predictors? 

For example, this function was used to do a logistic regression on dichotomous disease outcome (case vs control) ~ genotype  @ SNP_A + age, however, when we tested this on a different genotype [dichotomous disease outcome (case vs control) ~ genotype  @ SNP_B + age], the coefficient remained exactly the same for age. 
user: klmartinez
issue_8980: state: open
title: Adding leverage statistics for linear regression
body: #### Is your feature request related to a problem?
In applied linear regression it is useful to compute the leverage(influence) of a single observation on the overall model fit.
This can help identify outliers and further assess the model fit.
R provides a cook's distance function amongst other functions to compute the influence of observations.

#### Describe the solution you'd like
I think adding a Cook's distance formula and leverage statistics output for a linear model fit with the statsmodels package would be a great addition to the package. I could help create an initial design for this.

#### Describe alternatives you have considered
A clear and concise description of any alternative solutions or features you have considered.

#### Additional context
Add any other context about the feature request here.
user: sfweller
issue_8979: state: open
title: BUG: fix range_unit_root_test warnings and doc
body: - [ ] closes #8978
- [ ] tests passed. 
- [ ] code/documentation is well formatted.  
- [ ] properly formatted commit message. See [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message).

Corrects `tsa.stattools.range_unit_root_test` so that
1. the docstring correctly states the null hypothesis as _non_-stationarity of `x`;
2. `InterpolationWarning` is only emitted when the actual $p$-value is smaller than 0.01 or larger than 0.95, not whenever the returned $p$-value is 0.01 or 0.95; and
3. `InterpolationWarning` gives correct directions of differences between actual and returned $p$-values.

### Code examples
<details><summary>Documentation</summary>
<p>

The docstring now correctly states the null as a single unit root of `x`.
```
In [1]: import numpy as np
   ...: from statsmodels.tsa.stattools import range_unit_root_test
   ...: ?range_unit_root_test
Signature: range_unit_root_test(x, store=False)
Docstring:
Range unit-root test for stationarity.

Computes the Range Unit-Root (RUR) test for the null
hypothesis that x has a single unit root.
...
```
</p>
</details>

<details><summary>Stationary example</summary>
<p>

`InterpolationWarning` now correctly states that the actual $p$-value is _smaller_ than returned when the RUR statistic is smaller than the $\alpha$ = 0.01 critical value.
```
In [2]: x_constant = np.zeros(250)
   ...: rur_stat, p_value, crit = range_unit_root_test(x_constant)
   ...: print('RUR statistic: {}'.format(rur_stat))
   ...: print('p-value: {}'.format(p_value))
   ...: print('Critical values: {}'.format(crit))
<ipython-input-2-dbbe7f9c027c>:2: InterpolationWarning: The test statistic is outside of the range of p-values available in the
look-up table. The actual p-value is smaller than the p-value returned.

  rur_stat, p_value, crit = range_unit_root_test(x_constant)
RUR statistic: 0.0
p-value: 0.01
Critical values: {'10%': 1.3632, '5%': 1.2242, '2.5%': 1.1137, '1%': 0.9982}
```
</p>
</details>

<details><summary>More non-stationary example</summary>
<p>

`InterpolationWarning` now correctly states that the actual $p$-value is _larger_ than returned when the RUR statistic is larger than the $\alpha$ = 0.95 critical value.
```
In [3]: rng = np.random.default_rng(seed=1234)
   ...: x_morebias = np.cumsum(rng.uniform(-0.4, 0.6, size=250))
   ...: rur_stat, p_value, _ = range_unit_root_test(x_morebias)
   ...: print('RUR statistic: {}'.format(rur_stat))
   ...: print('p-value: {}'.format(p_value))
<ipython-input-3-25d0fd7a3141>:3: InterpolationWarning: The test statistic is outside of the range of p-values available in the
look-up table. The actual p-value is larger than the p-value returned.

  rur_stat, p_value, _ = range_unit_root_test(x_morebias)
RUR statistic: 5.502363128692981
p-value: 0.95
```
</p>
</details>

<details><summary>Less non-stationary example</summary>
<p>

A $p$-value of 0.95 is now returned without warning when the RUR statistic is between the $\alpha$ = 0.9 and $\alpha$ = 0.95 critical values.
```
In [4]: rng = np.random.default_rng(seed=1234)
   ...: x_lessbias = np.cumsum(rng.uniform(-0.45, 0.55, size=250))
   ...: rur_stat, p_value, _ = range_unit_root_test(x_lessbias)
   ...: print('RUR statistic: {}'.format(rur_stat))
   ...: print('p-value: {}'.format(p_value))
RUR statistic: 3.0990321069650117
p-value: 0.95
```
</p>
</details>

### Test output
<details>

```
(.venv) joshualiu@dhcp-10-29-180-140 statsmodels % pytest statsmodels/tsa/tests/test_stattools.py::TestRUR
=========================================== test session starts ===========================================
platform darwin -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0
Using --randomly-seed=728844893
rootdir: /Users/joshualiu/Desktop/statsmodels
configfile: setup.cfg
plugins: randomly-3.15.0, xdist-3.3.1
collected 4 items                                                                                         

statsmodels/tsa/tests/test_stattools.py ....                                                        [100%]

============================================ 4 passed in 0.40s ============================================

```
</details>
user: cjoshliu
issue_8978: state: open
title: tsa.stattools.range_unit_root_test documentation and InterpolationWarning give incorrect null and directions
body: #### Bug description

Per its docstring, the null hypothesis of `tsa.stattools.range_unit_root_test` is that the input time series `x` is stationary. However, the null hypothesis of the RUR test is actually that its input time series has a unit root, that is, that `x` is _non_-stationary. See the abstract and third paragraph of p. 547 in F. Aparicio, A. Escribano, and A.E. Sipols, [J. Time Ser. Anal. **27**, 545 (2006).](https://doi.org/10.1111/j.1467-9892.2006.00474.x)

Relatedly, `InterpolationWarning` gives incorrect directions when $p$-values are out of range of the look-up table and is sometimes emitted unnecessarily.

#### Code samples and expected outputs
<details><summary>Documentation</summary>
<p>

The `range_unit_root_test` docstring states the RUR null hypothesis as stationary `x`. The correct RUR null hypothesis is _non_-stationary `x`. 
```
In [1]: import numpy as np
   ...: from statsmodels.tsa.stattools import range_unit_root_test
   ...: ?range_unit_root_test
Signature: range_unit_root_test(x, store=False)
Docstring:
Range unit-root test for stationarity.

Computes the Range Unit-Root (RUR) test for the null
hypothesis that x is stationary.
...
```
Indeed, the test is actually implemented with the correct null hypothesis of non-stationarity.
```
In [2]: x_trendish = np.cos(np.arange(250))+2.0e-3*np.arange(250)
   ...: rur_stat, p_value, crit, resstore = range_unit_root_test(x_trendish, store=True)
   ...: print('RUR statistic: {}'.format(rur_stat))
   ...: print('p-value: {}'.format(p_value))
   ...: print('Critical values: {}'.format(crit))
   ...: print('Null hypothesis: {}'.format(resstore.H0))
RUR statistic: 1.2649110640673518
p-value: 0.1
Critical values: {'10%': 1.3632, '5%': 1.2242, '2.5%': 1.1137, '1%': 0.9982}
Null hypothesis: The series is not stationary
```
</p>
</details>

<details><summary>Stationary example</summary>
<p>

1. The following constant series `x_constant` is clearly strictly stationary. If the null hypothesis is stationarity, the returned $p$-value should be very high. It is instead very _low_.

2. At a given sample size, lower RUR statistics give lower $p$-values. If the RUR statistic is lower than the critical value of the lowest confidence level, `InterpolationWarning` should warn that the actual $p$-value is smaller than returned. It instead warns that the actual $p$-value is _larger_ than returned.
```
In [3]: x_constant = np.zeros(250)
   ...: rur_stat, p_value, crit = range_unit_root_test(x_constant)
   ...: print('RUR statistic: {}'.format(rur_stat))
   ...: print('p-value: {}'.format(p_value))
   ...: print('Critical values: {}'.format(crit))
<ipython-input-3-dbbe7f9c027c>:2: InterpolationWarning: The test statistic is outside of the range of p-values available in the
look-up table. The actual p-value is larger than the p-value returned.

  rur_stat, p_value, crit = range_unit_root_test(x_constant)
RUR statistic: 0.0
p-value: 0.01
Critical values: {'10%': 1.3632, '5%': 1.2242, '2.5%': 1.1137, '1%': 0.9982}
```
</p>
</details>

<details><summary>More non-stationary example</summary>
<p>

1. The following biased random walk `x_morebias` is clearly non-stationary. If the null hypothesis is stationarity, the returned $p$-value should be very low. It is instead very _high_.

2. According to Table 1 of [Aparicio et al.](https://doi.org/10.1111/j.1467-9892.2006.00474.x), the $\alpha$ = 0.95 critical value at $n$ = 250 is 3.28888. Similarly, the `crit` table in `statsmodels/tsa/stattools.py` has 3.2842. If the RUR statistic is greater than 3.2842, `InterpolationWarning` should warn that the actual $p$-value is larger than returned. It instead warns that the actual $p$-value is _smaller_ than returned.
```
In [4]: rng = np.random.default_rng(seed=1234)
   ...: x_morebias = np.cumsum(rng.uniform(-0.4, 0.6, size=250))
   ...: rur_stat, p_value, _ = range_unit_root_test(x_morebias)
   ...: print('RUR statistic: {}'.format(rur_stat))
   ...: print('p-value: {}'.format(p_value))
<ipython-input-4-25d0fd7a3141>:3: InterpolationWarning: The test statistic is outside of the range of p-values available in the
look-up table. The actual p-value is smaller than the p-value returned.

  rur_stat, p_value, _ = range_unit_root_test(x_morebias)
RUR statistic: 5.502363128692981
p-value: 0.95
```
</p>
</details>

<details><summary>Less non-stationary example</summary>
<p>

1. Issue 1 in the very non-stationary example also occurs here.

2. According to the `crit` table in `statsmodels/tsa/stattools.py`, the $\alpha$ = 0.90 and $\alpha$ = 0.95 critical values are respectively 2.9571 and 3.2842 at $n$ = 250. If the RUR statistic is between these values, the $p$-value should be returned as 0.95 without a warning. Instead, `InterpolationWarning` warns that the actual $p$-value is smaller than returned. This is true, but not correct behavior when the RUR statistic is _between_ critical values: in the "Documentation" example, the actual $p$-value is lower than the returned 0.1, but no warning is emitted. Additionally, the docstring notes that `InterpolationWarning` is only emitted when the RUR statistic is _outside_ the range of critical values.

```
In [5]: rng = np.random.default_rng(seed=1234)
   ...: x_lessbias = np.cumsum(rng.uniform(-0.45, 0.55, size=250))
   ...: rur_stat, p_value, _ = range_unit_root_test(x_lessbias)
   ...: print('RUR statistic: {}'.format(rur_stat))
   ...: print('p-value: {}'.format(p_value))
<ipython-input-5-32751ac38355>:3: InterpolationWarning: The test statistic is outside of the range of p-values available in the
look-up table. The actual p-value is smaller than the p-value returned.

  rur_stat, p_value, _ = range_unit_root_test(x_lessbias)
RUR statistic: 3.0990321069650117
p-value: 0.95
```
</p>
</details>

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

```
INSTALLED VERSIONS
------------------
Python: 3.11.4.final.0
OS: Darwin 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun  8 22:22:19 PDT 2023; root:xnu-8796.121.3~7/RELEASE_ARM64_T8103 arm64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

statsmodels
===========

Installed: 0.14.0 (/Users/joshualiu/Desktop/statsmodels-issue/venv/lib/python3.11/site-packages/statsmodels)

Required Dependencies
=====================

cython: 0.29.36 (/opt/homebrew/lib/python3.11/site-packages/Cython)
numpy: 1.25.2 (/Users/joshualiu/Desktop/statsmodels-issue/venv/lib/python3.11/site-packages/numpy)
scipy: 1.11.1 (/Users/joshualiu/Desktop/statsmodels-issue/venv/lib/python3.11/site-packages/scipy)
pandas: 2.0.3 (/Users/joshualiu/Desktop/statsmodels-issue/venv/lib/python3.11/site-packages/pandas)
    dateutil: 2.8.2 (/Users/joshualiu/Desktop/statsmodels-issue/venv/lib/python3.11/site-packages/dateutil)
patsy: 0.5.3 (/Users/joshualiu/Desktop/statsmodels-issue/venv/lib/python3.11/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.7.2 (/opt/homebrew/lib/python3.11/site-packages/matplotlib)
Installed osx event loop hook.
    backend: MacOSX 
cvxopt: Not installed
joblib: 1.3.2 (/opt/homebrew/lib/python3.11/site-packages/joblib)

Developer Tools
================

IPython: 8.14.0 (/opt/homebrew/Cellar/ipython/8.14.0/libexec/lib/python3.11/site-packages/IPython)
    jinja2: Not installed
sphinx: Not installed
    pygments: 2.16.1 (/opt/homebrew/opt/pygments/lib/python3.11/site-packages/pygments)
pytest: 7.4.0 (/opt/homebrew/lib/python3.11/site-packages/pytest)
virtualenv: Not installed
```
</details>

I have submitted a pull request to fix these issues.
user: cjoshliu
issue_8976: state: open
title:  AttributeError: 'Axes' object has no attribute 'is_first_col'
body:  Describe the bug
the statsmodels version  is 0.14.0,which was updated in Anaconda.
The  scatter_ellipse() generates the massage saying that
AttributeError: 'Axes' object has no attribute 'is_first_col',
however,there's no attribute called  'is_first_col' in   scatter_ellipse() 
and it dosen't generate as least 4 pictures as i expected,just ONE picture.
the massage is as follows:
> File D:\ProgramData\anaconda3\Lib\site-packages\statsmodels\graphics\plot_grids.py:138, in scatter_ellipse(data, level, varnames, ell_kwds, plot_kwds, add_titles, keep_ticks, fig)
    136 if add_titles:
    137     ax.set_title('%s-%s' % (varnames[i], varnames[j]))
--> 138 if not ax.is_first_col():
    139     if not keep_ticks:
    140         ax.set_yticks([])
AttributeError: 'Axes' object has no attribute 'is_first_col'



 Code Sample, 

```import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
from statsmodels.graphics.plot_grids import scatter_ellipse
data_raw = sns.load_dataset('iris')
labels = ['Sepal length','Sepal width',
 'Petal length','Petal width']
fig = plt.figure(figsize=(8,8))
scatter_ellipse(data_raw.iloc[:,:-1],
 varnames=labels, fig=fig)
for s_idx in data_raw.species.unique():
    data= data_raw.loc[data_raw.species == s_idx].iloc[:,:-1]
    fig = plt.figure(figsize=(8,8))
    scatter_ellipse(data, varnames=labels, fig=fig,)
    fig.savefig('scatter ellipse' + s_idx + '.svg', format='svg')
```


 Expected Output 
I just want the IDE not to generate the error message and generate the correct number of pictures.



user: pro-dreamer
issue_8975: state: open
title: DOC: docstrings in robust.norms, improve, reorganize
body: - [x] closes #8807
- [ ] tests added / passed. 
- [x] code/documentation is well formatted.  
- [x] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

<details>


**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in main and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/main -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>

user: Sudo-Guy
issue_8972: state: open
title: MAINT: failing style check: E721 do not compare types
body: example `if type(corr1) == np.ndarray:`
should either use `is` or `isinstance`.
Most likely we want `is` in those cases, i.e. rule out subclasses of ndarray (which might not have compatible behavior)

in statsmodels/tools/data the `is` comparison does not fail the style check
`type(endog) is np.ndarray`


```
statsmodels/genmod/generalized_estimating_equations.py:2408:12: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
statsmodels/genmod/generalized_estimating_equations.py:2415:12: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
statsmodels/regression/mixed_linear_model.py:2951:8: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
statsmodels/stats/correlation_tools.py:638:8: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
statsmodels/stats/correlation_tools.py:650:12: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
statsmodels/stats/correlation_tools.py:660:12: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
statsmodels/tsa/regime_switching/markov_switching.py:412:16: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
statsmodels/tsa/regime_switching/markov_switching.py:414:18: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
statsmodels/tsa/statespace/kalman_filter.py:1270:12: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
statsmodels/tsa/statespace/mlemodel.py:2234:12: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
statsmodels/tsa/statespace/mlemodel.py:5033:20: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
statsmodels/tsa/statespace/sarimax.py:1018:34: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
statsmodels/tsa/statespace/sarimax.py:1020:21: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
statsmodels/tsa/statespace/tests/test_tools.py:396:16: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
statsmodels/tsa/statespace/tests/test_tools.py:449:16: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
statsmodels/tsa/statespace/tests/test_tools.py:458:16: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
statsmodels/tsa/statespace/tests/test_tools.py:396:16: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
statsmodels/tsa/statespace/tests/test_tools.py:449:16: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
statsmodels/tsa/statespace/tests/test_tools.py:458:16: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
statsmodels/tsa/statespace/tools.py:839:16: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
statsmodels/tsa/statespace/tools.py:875:16: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
statsmodels/tsa/statespace/tools.py:1081:8: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
statsmodels/tsa/statespace/tools.py:1360:8: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
statsmodels/tsa/statespace/tools.py:1417:16: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
statsmodels/tsa/vector_ar/tests/JMulTi_results/parse_jmulti_vecm_output.py:83:8: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
Previously passing files failed linting.
```
user: josef-pkt
issue_8971: state: open
title: Dead link for LME docs
body: The docs for LMEs here:
https://github.com/statsmodels/statsmodels/blob/main/docs/source/mixed_linear.rst

Point towards this link as a reference: http://econ.ucsb.edu/~doug/245a/Papers/Mixed%20Effects%20Implement.pdf

But the link is broken - redirects to a professor's webpage instead of towards a PDF.
user: david-cortes
issue_8969: state: open
title: MAINT: Unpin setuptools-scm version
body: - [ ] closes #xxxx
- [ ] tests added / passed. 
- [ ] code/documentation is well formatted.  
- [ ] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

[Version 7.1.0](https://github.com/pypa/setuptools_scm/blob/main/CHANGELOG.rst#v710) was released, and I'd like to package this project using it in [nixpkgs](https://github.com/NixOS/nixpkgs). I have left the upper bound missing since my personal opinions are that upper bounds or version exclusions should only be added for known issues, but I would be happy also to conform to whatever the preference is for this project.


**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in main and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/main -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>

user: tjni
issue_8968: state: open
title: Argument return_df not used in acorr_ljungbox
body: https://github.com/statsmodels/statsmodels/blob/3b61c469ed8d4a6752b5bf01390789512f81f0c6/statsmodels/stats/diagnostic.py#L319
user: cantaro86
issue_8967: state: open
title: ENH: Enable evaluation of BSplines outside their knots
body: ## Draft

This is work-in-progress to enable BSplines to be evaluated outside thier knots, based on their derivatives on their boundary knots. Currently, nothing is plumbed in and there are known errors when dealing with derivatives.

I'd be great to get some feedback on whether this is a reasonable idea before I do more work on it!
user: badge
issue_8966: state: open
title: Structural Equation Modelling (CB-SEM & PLS-SEM)
body: #### Is your feature request related to a problem? Please describe
I've seen past issue (https://github.com/statsmodels/statsmodels/issues/6400#issuecomment-823453344) that mentioned about semopy as package that might be included in statsmodels.

- [semopy](https://gitlab.com/georgy.m/semopy) is a nice CB-SEM package comes with MIT license, but it is stored on GitLab instead of GitHub which I think will be less likely to be contributed.

- There is also a package called [plspm-python](https://github.com/GoogleCloudPlatform/plspm-python) for PLS-SEM but it comes with GPL-3 license.

#### Describe the solution you'd like
Statsmodels is a well-maintained Python package that comes without GPL license, which are advantages for open sourcing data analysis GUI. In management research, especially marketing, SEM is commonly used, but we must utilize commercial software like SmartPLS. I would like to ask whether there is a plan to support SEM inside statsmodels?
user: anggoran
issue_8965: state: open
title: AttributeError: 'ARIMAResults' object has no attribute 'plot_predict'
body: #### Describe the bug

The plot_predict() function generates the following error message
AttributeError: 'ARIMAResults' object has no attribute 'plot_predict'

#### Code Sample, a copy-pastable example if possible
```
from statsmodels.tsa.arima_model import ARIMA
model = ARIMA(df["Returns"], order=(1,1,1))
model_fit = model.fit()
print(model_fit.summary())

from statsmodels.graphics.tsaplots import plot_predict
model_fit.plot_predict()  # Causes AttributeError: 'ARIMAResults' object has no attribute 'plot_predict'
plt.show()
```

I'm using statsmodels version 0.14.0

According the website below, plot_predict should be change as follows.
model_fit.plot_predict()
to 
plot_predict(model_fit)

Please confirm if this is correct as I see no mention of this on the statsmodels website .

https://stackoverflow.com/questions/73112516/arimaresults-object-has-no-attribute-plot-predict-error

user: peterjmayhew
issue_8962: state: open
title: Doc Information Not Precise.
body: The documentation information related to `simple_differencing ` doesn't seem to be precise enough, or at least, given its name, lends itself to confusion.
The available information  which we have is the following:

> If `simple_differencing = True` is used, then the `endog` and `exog` data
>     are differenced prior to putting the model in state-space form. This has
>     the same effect as if the user differenced the data prior to constructing
>     the model, which has implications for using the results:
> 
>     - Forecasts and predictions will be about the *differenced* data, not about
>       the original data. (while if `simple_differencing = False` is used, then
>       forecasts and predictions will be about the original data).
>     - If the original data has an Int64Index, a new RangeIndex will be created
>       for the differenced data that starts from one, and forecasts and
>       predictions will use this new index.

I'm interested in knowing, when `simple_differencing = False`, whether the original data will be differenced or not prior to fitting the model.

I'm asking this, because we could have a case where the user may have already made the data stationary, using other transformations than simple differencing, and more differencing wouldn't be needed.
user: ialvata
issue_8960: state: open
title: REF/Design: deprecate positional use of keyword parameters
body: https://github.com/scipy/scipy/pull/18843

undecided
user: josef-pkt
issue_8958: state: open
title: ENH: Add Gwet's AC1
body: This implements Gwet's AC_1 (for two and three or more raters) and adds the relevant tests.
user: swflint
issue_8957: state: open
title: statsmodels.discrete.conditional_models.ConditionalLogit fails to fit more than 3000 observations
body: #### Describe the bug

statsmodels.discrete.conditional_models.ConditionalLogit suffers from a ton of overflow issues with only 3 features when the sample size gets to around 4000. I have tried with uniformly distributed data between 0 and 1, normally distributed data, and logging the uniform data, with no missing values. Around 1000 samples, the model fits with no warnings. At 2000 and 3000 the model produces many overflow warnings and errors, with 4000 or more samples the kernel crashes almost instantly with no warnings. I have tried using every type of solver provided, but I have not experimented with many combinations of each solver's parameters. Below are the warnings and error message for 2000 samples:

[c:\Work\American\StataDoFiles\.venv\lib\site-packages\statsmodels\discrete\conditional_models.py:283](file:///C:/Work/American/StataDoFiles/.venv/lib/site-packages/statsmodels/discrete/conditional_models.py:283): RuntimeWarning: overflow encountered in scalar add
  v = f(t - 1, k) + f(t - 1, k - 1) * exb[t - 1]
[c:\Work\American\StataDoFiles\.venv\lib\site-packages\statsmodels\discrete\conditional_models.py:319](file:///C:/Work/American/StataDoFiles/.venv/lib/site-packages/statsmodels/discrete/conditional_models.py:319): RuntimeWarning: overflow encountered in add
  u, v = a + c * h, b + d + e * h
[c:\Work\American\StataDoFiles\.venv\lib\site-packages\statsmodels\discrete\conditional_models.py:319](file:///C:/Work/American/StataDoFiles/.venv/lib/site-packages/statsmodels/discrete/conditional_models.py:319): RuntimeWarning: overflow encountered in scalar add
  u, v = a + c * h, b + d + e * h
[c:\Work\American\StataDoFiles\.venv\lib\site-packages\statsmodels\discrete\conditional_models.py:348](file:///C:/Work/American/StataDoFiles/.venv/lib/site-packages/statsmodels/discrete/conditional_models.py:348): RuntimeWarning: invalid value encountered in divide
  return self._xy[grp] - h / d
[c:\Work\American\StataDoFiles\.venv\lib\site-packages\statsmodels\base\model.py:595](file:///C:/Work/American/StataDoFiles/.venv/lib/site-packages/statsmodels/base/model.py:595): HessianInversionWarning: Inverting hessian failed, no bse or cov_params available
  warnings.warn('Inverting hessian failed, no bse or cov_params '
[c:\Work\American\StataDoFiles\.venv\lib\site-packages\statsmodels\base\model.py:607](file:///C:/Work/American/StataDoFiles/.venv/lib/site-packages/statsmodels/base/model.py:607): ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
  warnings.warn("Maximum Likelihood optimization failed to "
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[4], line 11
      8 y = (np.random.uniform(size=n) < pr).astype(int)
     10 m = ConditionalLogit(endog=y, exog=x, groups=g)
---> 11 r = m.fit()

File [c:\Work\American\StataDoFiles\.venv\lib\site-packages\statsmodels\discrete\conditional_models.py:125](file:///C:/Work/American/StataDoFiles/.venv/lib/site-packages/statsmodels/discrete/conditional_models.py:125), in _ConditionalModel.fit(self, start_params, method, maxiter, full_output, disp, fargs, callback, retall, skip_hessian, **kwargs)
    105 def fit(self,
    106         start_params=None,
    107         method='BFGS',
   (...)
    114         skip_hessian=False,
    115         **kwargs):
    117     rslt = super(_ConditionalModel, self).fit(
    118         start_params=start_params,
    119         method=method,
   (...)
    122         disp=disp,
    123         skip_hessian=skip_hessian)
--> 125     crslt = ConditionalResults(self, rslt.params, rslt.cov_params(), 1)
    126     crslt.method = method
    127     crslt.nobs = self.nobs

File [c:\Work\American\StataDoFiles\.venv\lib\site-packages\statsmodels\base\model.py:1515](file:///C:/Work/American/StataDoFiles/.venv/lib/site-packages/statsmodels/base/model.py:1515), in LikelihoodModelResults.cov_params(self, r_matrix, column, scale, cov_p, other)
   1511     dot_fun = np.dot
   1513 if (cov_p is None and self.normalized_cov_params is None and
   1514         not hasattr(self, 'cov_params_default')):
-> 1515     raise ValueError('need covariance of parameters for computing '
   1516                      '(unnormalized) covariances')
   1517 if column is not None and (r_matrix is not None or other is not None):
   1518     raise ValueError('Column should be specified without other '
   1519                      'arguments.')

ValueError: need covariance of parameters for computing (unnormalized) covariances

#### Code Sample, a copy-pastable example if possible

This runs fine:
```python
from statsmodels.discrete.conditional_models import ConditionalLogit
import numpy as np

n=1000
g = np.random.binomial(1, 0.6, size=n)
x = np.column_stack((np.random.normal(size=n), np.random.normal(size=n), np.random.normal(size=n)))
pr = 1 / (1 + np.exp(-x[:,0]))
y = (np.random.uniform(size=n) < pr).astype(int)

m = ConditionalLogit(endog=y, exog=x, groups=g)
r = m.fit()
```

This produces warning and errors out:
```python
from statsmodels.discrete.conditional_models import ConditionalLogit
import numpy as np

n=4000
g = np.random.binomial(1, 0.6, size=n)
x = np.column_stack((np.random.normal(size=n), np.random.normal(size=n), np.random.normal(size=n)))
pr = 1 / (1 + np.exp(-x[:,0]))
y = (np.random.uniform(size=n) < pr).astype(int)

m = ConditionalLogit(endog=y, exog=x, groups=g)
r = m.fit()
```

And this crashes the kernel:
```python
from statsmodels.discrete.conditional_models import ConditionalLogit
import numpy as np

n=1000
g = np.random.binomial(1, 0.6, size=n)
x = np.column_stack((np.random.normal(size=n), np.random.normal(size=n), np.random.normal(size=n)))
pr = 1 / (1 + np.exp(-x[:,0]))
y = (np.random.uniform(size=n) < pr).astype(int)

m = ConditionalLogit(endog=y, exog=x, groups=g)
r = m.fit()
```

#### Expected Output

I would not expect the model to fail with so few observations. Ideally, I will be fitting this to over a million observations at once.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]

INSTALLED VERSIONS
------------------
Python: 3.8.10.final.0

statsmodels
===========

Installed: 0.14.0 (c:\Work\American\StataDoFiles\.venv\lib\site-packages\statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.24.4 (c:\Work\American\StataDoFiles\.venv\lib\site-packages\numpy)
scipy: 1.10.1 (c:\Work\American\StataDoFiles\.venv\lib\site-packages\scipy)
pandas: 2.0.3 (c:\Work\American\StataDoFiles\.venv\lib\site-packages\pandas)
    dateutil: 2.8.2 (c:\Work\American\StataDoFiles\.venv\lib\site-packages\dateutil)
patsy: 0.5.3 (c:\Work\American\StataDoFiles\.venv\lib\site-packages\patsy)

Optional Dependencies
=====================

matplotlib: 3.7.1 (c:\Work\American\StataDoFiles\.venv\lib\site-packages\matplotlib)
    backend: module://matplotlib_inline.backend_inline 
cvxopt: Not installed
joblib: 1.3.1 (c:\Work\American\StataDoFiles\.venv\lib\site-packages\joblib)

Developer Tools
================

IPython: 8.12.2 (c:\Work\American\StataDoFiles\.venv\lib\site-packages\IPython)
    jinja2: Not installed
sphinx: Not installed
    pygments: 2.15.1 (c:\Work\American\StataDoFiles\.venv\lib\site-packages\pygments)
pytest: Not installed
virtualenv: Not installed
</details>

user: Jaage
issue_8955: state: open
title: Make regularization `alpha` parameter more meaningful
body: #### Is your feature request related to a problem? Please describe
In the `GLM` class, I don't understand why, when fitting a regularized model, the loss function is `-(self.loglike(x) / self.nobs - np.sum(alpha * x**2) / 2)` (loss from `_fit_ridge` method). Considering this formula, `alpha` includes both the variance (that I note `sigma2`) of the parameter prior distribution, which is a normal one for Ridge, and the number of the observations (that I note `n_obs`): `alpha = 1/(n_obs*sigma2)`. From the mathematical standpoint, I expect the regularization to have less and less weight when increasing the sample size since the posterior distribution is proportional to the likelihood times the prior distribution (and the likelihood scales with the number of observations) (see [wikipedia](https://en.wikipedia.org/wiki/Posterior_probability#Calculation)). For instance, this is implemented with this logic in the [ridge regression of scikit-learn](https://scikit-learn.org/stable/modules/linear_model.html#regression) (even though there are other models in scikit-learn with the other approach). 

#### Describe the solution you'd like
Considering the above explanation, why didn't you envisage another loss:
- `-(self.loglike(x) - np.sum(alpha * x**2) / 2)`, or 
- `-(self.loglike(x) - np.sum(alpha * x**2) / 2)  / self.nobs` (if the division is required)

`alpha` could then be interpreted as the inverse of the variance of the parameter prior distribution `alpha = 1/sigma2`. For a fixed `alpha`, the regularization would have a large weight if there is few observations and its weight would reduce while increasing the sample size. Nevertheless, I guess there is reason being the current implementation. So the solution I propose is perhaps more a question: why did you consider this loss which makes `alpha = 1/(n_obs*sigma2)`? I personally prefer my proposition which seems more logic (to me) even though I know there are the 2 formulations in statistical packages.

#### Describe alternatives you have considered
If this is done on purpose, don't you think a note may be needed to explain:
- the reason of this trade-off between likelihood and prior, and
- the meaning of `alpha`?

#### Additional context
user: kapytaine
issue_8953: state: open
title: ols() : Row order of result table do not fit the formula
body: #### Is your feature request related to a problem? Please describe
I use `statsmodels.formula.api.ols()` to do linear regression. The ordering of rows in the resulting table is not the same as the variables in the formula.

As an example formula: `'scoreBeeintraechtigung ~ scale(Age) + scoreInformiertheit + scoreNegEinstellung + ZuHeisseTageN + C(AngenehmInWohnung, Treatment(reference="Ja"))'`

The result looks like this
```
                                                      coef     SE      t      p              CI95
Intercept                                           4.2932  0.847  5.070  0.000     (2.63, 5.956)
C(AngenehmInWohnung, Treatment(reference="Ja"))...  1.6363  0.378  4.324  0.000     (0.893, 2.38)
scale(Age)                                         -0.7743  0.176 -4.407  0.000  (-1.119, -0.429)
scoreInformiertheit                                -0.2697  0.117 -2.312  0.021  (-0.499, -0.041)
scoreNegEinstellung                                 0.6791  0.081  8.434  0.000    (0.521, 0.837)
ZuHeisseTageN                                       0.0969  0.019  5.001  0.000    (0.059, 0.135)
```
#### Describe the solution you'd like
The ordering should be the same as in the formula.

user: buhtz
issue_8952: state: open
title: can I manually increase de number of iter, from 500 to 1000? or change the convergence criterion?
body: failing to converge with 500 iters, but coming very close...can I change the maxiter to 1000? 
thanks
user: carloseduardosg
issue_8950: state: open
title: Dynamic Factor Models forecast output 
body: Hi, 

I am using the `DynamicFactorMQ` class to implement a dynamic factor model. In general all good except when I am trying to use the forecast method

Here is for example my data that I feed into the `DynamicFactorMQ` 

```
# Construct the dynamic factor model
returns.index = pd.to_datetime(returns.index)
returns.index = pd.DatetimeIndex(returns.index).to_period('D')


description  3i Group      ASML    Abbvie  ...     VINCI      Visa  Waste Management
date                                       ...                                      
2023-07-03  -0.005387  0.012597  0.002078  ...  0.006463  0.002063         -0.011648
2023-07-04  -0.013784  0.000000  0.000000  ... -0.001738  0.000000          0.000000
2023-07-05  -0.009395 -0.024813  0.019332  ... -0.015511  0.006219         -0.002042
2023-07-06  -0.010870 -0.022860 -0.002689  ... -0.031466 -0.002380          0.001812
2023-07-07   0.004626 -0.003675 -0.012750  ...  0.006873 -0.010173         -0.007878

```

And estimated this model following the same steps as [here](https://www.chadfulton.com/topics/statespace_large_dynamic_factor_models.html#:~:text=This%20notebook%20describes%20working%20with%20these%20models%20in,of%20the%20%E2%80%9Cnews%E2%80%9D%20from%20updated%20datasets%208%20References)
```

model = sm.tsa.DynamicFactorMQ(returns, factors=factors,
                               factor_orders=factor_orders,
                               factor_multiplicities=factor_multiplicities)

results = model.fit(disp=10)

```

As you can see my dataframe ends in '2023-07-07'. Yet, when I use the `forecast` method I get an unexpected outcome. 

For instance using:
```

# Create point forecasts, 3 steps ahead
point_forecasts = results.forecast(steps=3)

# Print the forecasts for the first 5 observed variables
print(point_forecasts.T.head())
```

I get 
```

              2018-03-15  2018-03-16  2018-03-17
3i Group        -0.007512   -0.005220   -0.005549
ASML            -0.003086   -0.003375   -0.003909
Abbvie          -0.002512   -0.002067   -0.002270
Albemarle       -0.004013   -0.005110   -0.004677
Alphabet Inc.   -0.002557   -0.002535   -0.003110
```

That is the forecast is for `2018-03-15` onwards, while I was expecting `2023-07-07` onwards. 

Please note that my Data were daily observations. The outcome was the expected one, only when I resampled to monthly. I see this as a bug. 


user: msh855
issue_8949: state: open
title: ENH: Leybourne-McCabe stationarity test
body: Hi SM - I started looking at stationarity again and forgot that I have a nice implementation of Leybourne-McCabe stationarity test (https://github.com/JimVaranelli/Leybourne-McCabe) which I'm happy to PR if there is any interest. AFAIK, Matlab is the only other implementation which was used for unit test verification. For usability I added an empirical autolag procedure as well as allowing for the possibility of zero lags. MC-derived CVs are also provided. Further details are in the readme.
user: JimVaranelli
issue_8947: state: open
title: How to completely remove training data from model.
body: Dear developers of statsmodels,

First of all, thank you so much for creating such an amazing package.

I've recently used statsmodels (0.14.0) to make a GLM model and I would like to share it with others so that they can input their own data and make predictions.
However, I am prohibited from sharing the data I have used for training.

Therefor, I tried to remove the training data via calling `.save()` with `remove_data=True` , but when I checked the `.model.data.frame` attribute of the fitted model object, it still contained the training data.
To the best of my knowledge the data is also stored in `model.data.orig_exog` and `model.data.orig_endog` as mentioned in [issue7494](https://github.com/statsmodels/statsmodels/issues/7494), but I am not sure where else the training data maybe held.

If you could please enlighten on me how to completely remove training data from the fitted model object that would be great.
Many thanks in advance.

### Reproducible code
```
import numpy as np
import statsmodels.api as sm
import statsmodels.formula.api as smf

# load "growth curves pf pigs" dataset
my_data = sm.datasets.get_rdataset("dietox", "geepack").data

# make GLM model
my_formula = "Weight ~ Time"
my_family = sm.families.Gaussian(link=sm.families.links.identity())
model = smf.glm(formula=my_formula, data=my_data, family=my_family)
result = model.fit()

# save model with "remove_data=True" 
result.save('test_model.pickle', remove_data=True)
new_result = sm.load('test_model.pickle')
print(new_result.model.data.frame)
```
user: hiroki32
issue_8943: state: open
title: Logit.fit(full_output=0) → TypeError: 'NoneType' object is not subscriptable → missing "mle_retvals"
body: Hello,

recently updated to latest sm release (0.14) and newly sm.Logit(Y, X).fit(full_output=0) produces error:
-> 4430         self.converged = mlefit.mle_retvals["converged"]
TypeError: 'NoneType' object is not subscriptable

By removing param (default value 1) or setting it as 1 -- sm.Logit(Y, X).fit(full_output=1)  -- then fit produces standard result as previous versions of sm.

I assume that removing "mle_retvals" by specifing param full_output=0 in the fit is the issue and not desired behavior (didn't produce error in prev. versions).

Best,
Michal
user: MichalRIcar
issue_8938: state: open
title: 0.14.0: pytest is failing in `statsmodels/treatment/tests/test_teffects.py` unit
body: I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.
- `python3 -sBm build -w --no-isolation`
- because I'm calling `build` with `--no-isolation` I'm using during all processes only locally installed modules
- install .whl file in </install/prefix> using 'installer` module
- run pytest with $PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>
- build is performed in env which is *`cut off from access to the public network`* (pytest is executed with `-m "not network"`)

Here is pytest output:
<details>

```console
+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-statsmodels-0.14.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-statsmodels-0.14.0-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -m 'not network' statsmodels --import-mode=importlib
==================================================================================== test session starts ====================================================================================
platform linux -- Python 3.8.17, pytest-7.4.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/statsmodels-0.14.0
configfile: setup.cfg
collected 18241 items / 1 error

========================================================================================== ERRORS ===========================================================================================
_______________________________________________________________ ERROR collecting statsmodels/treatment/tests/test_teffects.py _______________________________________________________________
/usr/lib/python3.8/site-packages/_pytest/runner.py:341: in from_call
    result: Optional[TResult] = func()
/usr/lib/python3.8/site-packages/_pytest/runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
/usr/lib/python3.8/site-packages/_pytest/python.py:536: in collect
    self._inject_setup_module_fixture()
/usr/lib/python3.8/site-packages/_pytest/python.py:550: in _inject_setup_module_fixture
    self.obj, ("setUpModule", "setup_module")
/usr/lib/python3.8/site-packages/_pytest/python.py:315: in obj
    self._obj = obj = self._getobj()
/usr/lib/python3.8/site-packages/_pytest/python.py:533: in _getobj
    return self._importtestmodule()
/usr/lib/python3.8/site-packages/_pytest/python.py:622: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:538: in import_path
    spec.loader.exec_module(mod)  # type: ignore[union-attr]
/usr/lib/python3.8/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
statsmodels/treatment/tests/test_teffects.py:31: in <module>
    res_probit = Probit.from_formula(formula, dta_cat).fit()
../../BUILDROOT/python-statsmodels-0.14.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages/statsmodels/base/model.py:203: in from_formula
    tmp = handle_formula_data(data, None, formula, depth=eval_env,
../../BUILDROOT/python-statsmodels-0.14.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages/statsmodels/formula/formulatools.py:63: in handle_formula_data
    result = dmatrices(formula, Y, depth, return_type='dataframe',
/usr/lib/python3.8/site-packages/patsy/highlevel.py:309: in dmatrices
    (lhs, rhs) = _do_highlevel_design(formula_like, data, eval_env,
/usr/lib/python3.8/site-packages/patsy/highlevel.py:164: in _do_highlevel_design
    design_infos = _try_incr_builders(formula_like, data_iter_maker, eval_env,
/usr/lib/python3.8/site-packages/patsy/highlevel.py:62: in _try_incr_builders
    formula_like = ModelDesc.from_formula(formula_like)
/usr/lib/python3.8/site-packages/patsy/desc.py:164: in from_formula
    tree = parse_formula(tree_or_string)
/usr/lib/python3.8/site-packages/patsy/parse_formula.py:146: in parse_formula
    tree = infix_parse(_tokenize_formula(code, operator_strings),
/usr/lib/python3.8/site-packages/patsy/infix_parser.py:210: in infix_parse
    for token in token_source:
/usr/lib/python3.8/site-packages/patsy/parse_formula.py:89: in _tokenize_formula
    for pytype, token_string, origin in it:
E   TypeError: next expected at least 1 argument, got 0
===================================================================================== warnings summary ======================================================================================
../../../../../usr/lib/python3.8/site-packages/jupyter_client/connect.py:20
  /usr/lib/python3.8/site-packages/jupyter_client/connect.py:20: DeprecationWarning: Jupyter is migrating its paths to use standard platformdirs
  given by the platformdirs library.  To remove this warning and
  see the appropriate new directories, set the environment variable
  `JUPYTER_PLATFORM_DIRS=1` and then run `jupyter --paths`.
  The use of platformdirs will be the default in `jupyter_core` v6
    from jupyter_core.paths import jupyter_data_dir, jupyter_runtime_dir, secure_write

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================================== short test summary info ==================================================================================
ERROR statsmodels/treatment/tests/test_teffects.py - TypeError: next expected at least 1 argument, got 0
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=============================================================================== 1 warning, 1 error in 16.20s ================================================================================
```
</details>

Here is list of installed modules in build env
<details>

```console
Package                       Version
----------------------------- -------
alabaster                     0.7.13
asttokens                     2.2.1
attrs                         23.1.0
Babel                         2.12.1
backcall                      0.2.0
beautifulsoup4                4.12.2
bleach                        6.0.0
build                         0.10.0
charset-normalizer            3.1.0
contourpy                     1.0.7
cycler                        0.11.0
Cython                        0.29.35
decorator                     5.1.1
defusedxml                    0.7.1
distro                        1.8.0
docutils                      0.19
exceptiongroup                1.1.1
executing                     1.2.0
fastjsonschema                2.16.3
fonttools                     4.40.0
gpg                           1.20.0
html5lib                      1.1
idna                          3.4
imagesize                     1.4.1
importlib-metadata            6.7.0
importlib-resources           5.12.0
iniconfig                     2.0.0
installer                     0.7.0
ipython                       8.12.0
jedi                          0.18.2
Jinja2                        3.1.2
jsonschema                    4.17.3
jupyter_client                8.3.0
jupyter_core                  5.3.1
jupyterlab-pygments           0.1.2
kiwisolver                    1.4.4
libcomps                      0.1.19
MarkupSafe                    2.1.2
matplotlib                    3.6.3
matplotlib-inline             0.1.6
mistune                       3.0.1
nbclient                      0.8.0
nbconvert                     7.6.0
nbformat                      5.9.0
nbsphinx                      0.9.2
numpy                         1.24.3
numpydoc                      1.5.0
olefile                       0.46
packaging                     23.1
pandas                        2.0.2
pandocfilters                 1.5.0
parso                         0.8.3
patsy                         0.5.3
pexpect                       4.8.0
pickleshare                   0.7.5
Pillow                        9.5.0
pkgutil_resolve_name          1.3.10
platformdirs                  3.8.0
pluggy                        1.0.0
prompt-toolkit                3.0.38
ptyprocess                    0.7.0
pure-eval                     0.2.2
Pygments                      2.15.1
pyparsing                     3.1.0
pyproject_hooks               1.0.0
pyrsistent                    0.19.3
pytest                        7.4.0
python-dateutil               2.8.2
pytz                          2023.2
PyYAML                        6.0
pyzmq                         24.0.1
requests                      2.31.0
SciPy                         1.8.1
setuptools                    68.0.0
setuptools-scm                7.1.0
six                           1.16.0
snowballstemmer               2.2.0
soupsieve                     2.4.1
Sphinx                        6.2.1
sphinxcontrib-applehelp       1.0.4
sphinxcontrib-devhelp         1.0.2
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.5
stack-data                    0.6.2
tinycss2                      1.2.1
tomli                         2.0.1
tornado                       6.3.2
traitlets                     5.9.0
typing_extensions             4.6.3
urllib3                       1.26.15
wcwidth                       0.2.6
webencodings                  0.5.1
wheel                         0.40.0
zipp                          3.15.0
```
</details>

user: kloczek
issue_8935: state: open
title: statsmodels.tools.sm_exceptions.X13Error: ERROR: Number of user-defined X elements= 166 not equal to a multiple of the number of columns= 4
body: My test data has two columns `M0 `and `exog` (you may download the X13 software and test data from [this link](https://www.dropbox.com/s/v4adf86g2rqzan5/X13_data.zip?dl=0) ).

```
import statsmodels.api as sm
import pandas as pd
import numpy as np
import os

X13PATH = r'D:\data\X13'
os.chdir(X13PATH)
os.environ['X13PATH'] = X13PATH
print(os.environ['X13PATH'])

file_path = '../test_data/X13_test_data.xlsx'
sheet_name = 'Sheet1'

df = pd.read_excel(file_path, sheet_name=sheet_name, index_col='date')
print(df.head(5))

df.index = pd.to_datetime(df.index)
# df = df.rename(columns={'festival_factor': 'exog', 'M0': 'endog'})
res = sm.tsa.x13_arima_analysis(endog=df['M0'], exog=df['exog'], x12path=X13PATH, print_stdout=True)
# res = sm.tsa.x13_arima_analysis(endog=df['M0'], x12path=X13PATH, print_stdout=True)
print(res.seasadj)
print(res.plot)
```
Out:

```
                  M0      exog
date                          
2009-07-31  34239.30  0.000000
2009-08-31  34406.62  0.000000
2009-09-30  36787.89  0.333333
2009-10-31  35730.23  0.516129
2009-11-30  36343.86  0.000000

Traceback (most recent call last):
  File "D:\data\X13_test.py", line 19, in <module>
    res = sm.tsa.x13_arima_analysis(endog=df['M0'], exog=df['exog'], x12path=X13PATH, print_stdout=True)
  File "C:\Users\LSTM\AppData\Roaming\Python\Python310\site-packages\pandas\util\_decorators.py", line 210, in wrapper
    return func(*args, **kwargs)
  File "D:\Program Files\anaconda3\lib\site-packages\statsmodels\tsa\x13.py", line 518, in x13_arima_analysis
    _check_errors(errors)
  File "D:\Program Files\anaconda3\lib\site-packages\statsmodels\tsa\x13.py", line 201, in _check_errors
    raise X13Error(errors)
statsmodels.tools.sm_exceptions.X13Error: ERROR: Number of user-defined X elements= 166
        not equal to a multiple of the number of columns=  4.
```
I use `sm.tsa.x13_arima_analysis(endog=df['M0'], exog=df['exog'], x12path=X13PATH, print_stdout=True)` to perform X13 ARIMA on `M0 `For seasonal adjustment, the `exog `column is used as an exogenous variable, but an error occurs: `statsmodels.tools.sm_exceptions.X13Error: ERROR: Number of user-defined X elements= 166 not equal to a multiple of the number of columns= 4.`

My analysis found that the main cause of the error is caused by `exog=df['exog']`, because `sm.tsa.x13_arima_analysis(endog=df['M0'], x12path=X13PATH, print_stdout=True)` can run normally.
user: ahbon123
issue_8929: state: open
title: ENH: GAM combining different penalizations, splines
body: We don't have an option to combine two different spline types in GAM
https://stackoverflow.com/questions/76541891/use-both-cyclic-and-b-spline-smoothers-in-statsmodels-gam

We had this question once before, soon after I had merged GAM.

Related we don't have a simple way to add different penalties, Penalty classes, either for overlapping or non-overlapping params.
user: josef-pkt
issue_8926: state: open
title: VARProcess.simulate_var Crashes Due to Missing Variable Definition
body: #### Describe the bug

When calling varProcess.simulate_var(num_time_steps), an error is thrown which essentially reads:
```
Traceback (most recent call last):
  File "foo.py", line 191, in <module>
    data1[0,:,:] = my_varProcess.simulate_var(num_time_steps)
  File "/user1/.local/lib/python3.7/site-packages/statsmodels/tsa/vector_ar/var_model.py", line 959, in simulate_var
    offset = self.endog_lagged[:, : self.k_exog].dot(
AttributeError: 'VARProcess' object has no attribute 'endog_lagged'
```

This error seems to suggest that VARProcess.simulate_var() is calling a function for which the VARProcess object has not been properly initialized. It appears that this error would need to be fixed before simulate_var could complete its functional objectives.

For reference, this issue seems loosely related to the following issue from 2018: <BUG/REF: fixing a few VAR problems #4538>

#### Code Sample, a copy-pastable example if possible

```bash
pip install statsmodels
pip install slycot
pip install control
pip install -U numpy
pip install scipy --upgrade
```

```python
import pickle as pkl
import numpy as np
import copy
import control.matlab as ctrl
from statsmodels.tsa.vector_ar.var_model import VARProcess as varp
from scipy.signal import zpk2tf
from scipy.signal.filter_design import bilinear_zpk

data_save_path = "/folder/for/saving/sims"
logs_save_path = "/folder/for/saving/sims/logs"
num_data_samples = 21 
sample_temp_window_size = 1000
max_num_samps_per_preprocessed_file = 5
num_channels = 5
decay1 = 0.98
mu1 = 5
decay2 = 0.9
mu2 = 30
seed = 1337
fs = 1000
LOOP1DELAY = 10
LOOP2DELAY = 3
LOOP1STR = 0.003
LOOP2STR = 0.02
NOISE_COV1 = 1
NOISE_COV2 = 1 
NOISE_COV3 = 1

if __name__=="__main__":
    # create VAR models to simulate data

    # generate intra-region AR coefficient from desired poles
    pole1a = decay1*np.exp(1j*mu1/fs*(2*np.pi))
    pole1b = decay1*np.exp(-1j*mu1/fs*(2*np.pi))
    pole2a = decay2*np.exp(1j*mu2/fs*(2*np.pi))
    pole2b = decay2*np.exp(-1j*mu2/fs*(2*np.pi))

    # For logic behind following code, see zpk2tf docs at https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.zpk2tf.html#scipy.signal.zpk2tf
    # as well as _ at https://python.hotexamples.com/examples/scipy.signal/-/zpk2tf/python-zpk2tf-function-examples.html and https://www.mathworks.com/help/signal/ref/bilinear.html
    
    sys1zz, sys1pz, sys1kz = bilinear_zpk([], [pole1a, pole1b], LOOP1STR, 1/fs) # create continuous analogue of discrete system
    _, sys1Denom = zpk2tf(sys1zz, sys1pz, sys1kz) 
    nCoefs1 = np.prod(list(sys1Denom.shape))-1
    nLags1 = max(LOOP1DELAY, nCoefs1)
    coef1intra = np.zeros((nLags1,1))
    coef1intra[:nCoefs1,0] = -sys1Denom[1:]

    # make inter region AR coefficients simple impulse
    coef1inter = np.zeros((nLags1,1))
    coef1inter[LOOP1DELAY-1,0] = LOOP1STR

    # populate VAR1 parameters
    preCov = np.random.randn(num_channels)/num_channels
    net1Covariance = NOISE_COV1*((preCov + preCov.conjugate())/2 + np.eye(num_channels))
    net1Constant = np.zeros((num_channels,1))
    net1AR = []
    for k in range(nLags1):
        thisLag = np.zeros((1,num_channels, num_channels))
        # the network simulated by this var is 1>2>3>4 with recurrent connections within 1,2,3,&4
        for r in range(4):
            thisLag[0,r,r] = coef1intra[k] 
        thisLag[0,1,0] = coef1inter[k]
        thisLag[0,2,1] = coef1inter[k]
        thisLag[0,3,2] = coef1inter[k]
        net1AR.append(thisLag)
    net1AR = np.concatenate(net1AR, axis=0)
    net1 = varp(net1AR, net1Constant, net1Covariance) # currently assumes exogenous variables () are zeros 
    
    print("__MAIN__: drawing and saving samples")
    samples = []
    subset_counter = 0
    for i in range(num_data_samples):
        print("__MAIN__: \t processing sample i==", i, " of n==", num_data_samples)
        
        # Simulate data from VAR model
        # initialize network data arrays
        data1 = np.zeros((1, sample_temp_window_size, num_channels))

        # generate scores for each network
        scores = np.random.random(1)

        # simulate each window w/ scaled innovation term
        net1.Covariance = copy.deepcopy(net1Covariance) * scores[0]
        
        data1[0,:,:] = net1.simulate_var(sample_temp_window_size) # NOTE: THIS IS WHERE THE ERROR GETS THROWN
        
        # sum network contributions
        curr_samp = data1

        samples.append([curr_samp, scores])
        
        # periodically save sample sets to external files so that full dataset need not be loaded all at once
        if len(samples)+1 == max_num_samps_per_preprocessed_file:
            print("__MAIN__: \t now saving subset_counter==", subset_counter)
            curr_subset_name = "processed_data_subset_"+str(subset_counter)+".pkl"
            with open(os.path.join(data_save_path, curr_subset_name), 'wb') as outfile:
                pkl.dump(samples, outfile)
            samples = []
            subset_counter += 1
        pass

    print("__MAIN__: FINISHED !!!!!!")
    pass

```


#### Expected Output

Rather than an error message being thrown, the above code should save five .pkl files to the given (pre-existing) data_save_path variable, each of which should contain a python list of 1-5 np.array objects with shape (1 x num_channels x sample_temp_window_size ). (Note: the true saved shape of the np.array objects may have axes 1 and 2 transposed, which would also be an acceptable output)

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

INSTALLED VERSIONS
------------------
Python: 3.7.6.final.0
OS: Linux 4.18.0-408.el8.x86_64 #1 SMP Mon Jul 18 17:42:52 UTC 2022 x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

statsmodels
===========

Installed: 0.13.5 (/user1/.local/lib/python3.7/site-packages/statsmodels)

Required Dependencies
=====================

cython: 3.0a1 (/opt/apps/user2/python-3.7.6/lib/python3.7/site-packages/Cython)
numpy: 1.21.6 (/user1/.local/lib/python3.7/site-packages/numpy)
scipy: 1.7.3 (/user1/.local/lib/python3.7/site-packages/scipy)
pandas: 1.0.3 (/opt/apps/user2/python-3.7.6/lib/python3.7/site-packages/pandas)
    dateutil: 2.8.1 (/opt/apps/user2/python-3.7.6/lib/python3.7/site-packages/dateutil)
patsy: 0.5.3 (/user1/.local/lib/python3.7/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.5.3 (/user1/.local/lib/python3.7/site-packages/matplotlib)
    backend: agg
cvxopt: Not installed
joblib: 0.16.0 (/opt/apps/user2/python-3.7.6/lib/python3.7/site-packages/joblib)

Developer Tools
================

IPython: 7.18.1 (/opt/apps/user2/python-3.7.6/lib/python3.7/site-packages/IPython)
    jinja2: 2.11.2 (/opt/apps/user2/python-3.7.6/lib/python3.7/site-packages/jinja2)
sphinx: Not installed
    pygments: 2.7.1 (/opt/apps/user2/python-3.7.6/lib/python3.7/site-packages/pygments)
pytest: Not installed
virtualenv: Not installed

user: zacharycbrown
issue_8925: state: open
title: mnlogit does not have confidence intervals for predictions (available in .get_prediction() for other models in statsmodels)
body: When I tried to call result.get_prediction() after fitting result = smf.mnlogit("dep_var ~ indep_var", data = data).fit() I got an error message that "MultinomialResults object has no attribute 'get_prediction'". Looking at the documentation for MultinomialResults (https://www.statsmodels.org/dev/generated/statsmodels.discrete.discrete_model.MultinomialResults.html), we see that it says that get_prediction() is not implemented for Multinomial. 

If I instead use result.predict(), I get only point predictions and not standard errors and confidence intervals.

While for the regular logit there is workaround in statsmodels whereby we can fit result = smf.glm("dep_var ~ indep_var", data = data, family = sm.families.Binomial()).fit() instead of fitting result = smf.logit(("dep_var ~ indep_var", data = data).fit() and use result.get_prediction() with the results of the glm estimation, multinomial logit is not available via glm in statsmodels. 

 Since including measures of uncertainty when making predictions is important, I think .get_prediction() should be implemented for mnlogit in statsmodels. 

user: svetlanakosterina
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_8924: state: open
title: statsmodels.tsa.x13.x13_arima_analysis cannot open .err file
body: I'm trying to run the x13-arima seasonal adjusment using tsa.

At some step the function `x13_arima_analysis` tries to read a temp file `tmpktsig75f.err` but the code has produced the file  `tmpktsig75f_err.html` so the file can't be opened and it raises an error. 

```
from statsmodels.tsa import x13
import pandas as pd
import numpy as np
x = pd.Series(np.random.uniform(10,2,size=(20,)), index = pd.date_range('2018-01-01', periods = 20, freq = 'Q'))
x13_arima_analysis(endog = x, x12path = r'c:\...\x13as_html-v1-1-b59\x13as')
```


```
File [~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\statsmodels\tsa\x13.py:206](https://file+bbking2-002emit-002eedu.vscode-resource.vscode-cdn.net/mbarrera/git_supply/python_snippets/x13arima/~/AppData/Local/Packages/PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0/LocalCache/local-packages/Python311/site-packages/statsmodels/tsa/x13.py:206), in _open_and_read(fname)
    204 def _open_and_read(fname):
    205     # opens a file, reads it, and make sure it's closed
--> 206     with open(fname, 'r', encoding="utf-8") as fin:
    207         fout = fin.read()
    208     return fout

FileNotFoundError: [Errno 2] No such file or directory: '\\\\...\\tmpktsig75f.err'
```

Is this a known issue? (I'm using Windows)
user: marcdelabarrera
issue_8923: state: open
title: ENH/SUMM: outlier-influence, diagnostic for multi-endog, multivariate models
body: We need to collect what post-estimation results we can add for multi-endog and multivariate endog models, and how we can implement reusable methods.
(I'm working currently without references, just basics for MultivariateLS

main property: 
endog and predict are multivariate, ndim=2, balanced with no missing values in any of the endogs.

**MultivariateLS**

- outlier-influence (mainly multivariate measures, measures for individual endog are as in OLS/WLS)
  - resid_distance: mahalanobis distance for each observation of multivariate resid
  - hat_matrix_diag: common exog, so works analogous to OLS/WLS
  - cook's distance, loglikeobs displacement ?
  - plots ? 
- diagnostics
  - multivariate normality (I have never committed draft versions somewhere (?)) 
- ...

**Multinomial, Ordered**

I should have some issues for this, but currently don't find them.
1-dim resid measure for ordered is "stochastically larger" resid.

**future models**

copula models
other multivariate, multi-endog models
other linear/gaussian models, e.g. SUR
GMM multi-equation models

not clear: should this include 2-stage models like instrumental variables, sample selection or bivariate Probit, ... (full MLE models)


user: josef-pkt
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_8921: state: open
title: TST/Maint: test failures in MICE, pandas compat
body: most likely problem with pandas compatibility

```
FAILED statsmodels/imputation/tests/test_mice.py::TestMICEData::test_default
FAILED statsmodels/imputation/tests/test_mice.py::TestMICEData::test_pertmeth
FAILED statsmodels/imputation/tests/test_mice.py::TestMICEData::test_set_imputer
```

test failure are in cycle_order

```
>       assert_equal(imp_data._cycle_order, ['x5', 'x3', 'x4', 'y', 'x2', 'x1'])
E       AssertionError: 
E       Items are not equal:
E       item=1
E       
E        ACTUAL: 'x4'
E        DESIRED: 'x3'
```
pandas version: pandas-2.0.2
user: josef-pkt
issue_8917: state: open
title: ENH/REF: LikelihoodModelResults, propagate init kwargs to super
body: see https://github.com/statsmodels/statsmodels/issues/4679#issuecomment-877352386

I just saw again that LikelihoodModelResults does not propagate extra kwargs and they are not attached to the results instance.
(current dev case MultivariateLS)

In many models, we have to explicitly attach attributes after creating the results instance in `fit`.
It would be cleaner if we could use them as results `__init__` argument.

One possibility is to use a more explicit `attach_kwds` (or similar name) instead of kwargs. This would make it easier to check for invalid kwargs, currently extra kwargs are ignored and are swallowed by Likel.ihoodModelResults.

Note, I guess there will remain some cases where we change attributes after creating the results instance, in cases where we need to override the LikelihoodModelResults defaults.

(I'm not changing this in my current work on MultivariateLS, because it affects many models and needs more general checking.)


user: josef-pkt
issue_8916: state: open
title: sarimax estimate method question 
body: #### Describe the bug

[This is a quesion about estimate method, when i use sarimax function, why this is fix mle estimate whether i can choose ols/cls/uls，just like sas, thanks.]

#### Code Sample, a copy-pastable example if possible


```python
# Your code here that produces the bug
# This example should be self-contained, and so not rely on external data.
# It should run in a fresh ipython session, and so include all relevant imports.
```
<details>

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `main`. If your problem has been fixed in an unreleased version, you might be able to use `main` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the main branch of this repository? It helps the limited resources if we know problems exist in the current main branch so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

A clear and concise description of what you expected to happen.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]

</details>

user: mitsuha030
issue_8915: state: open
title: Extremely small p-values
body: Hi team, 

I am running OLS and got very small P-values as 0, but I think this is just because it is extremely significant.

> mdf.pvalues
> Intercept                                          0.000000000000000000000000000000000000000000000...
> Age                                                0.000000000000000000000000000000000000000000000...
> Sex                                                0.000000000000000000000000000000000000000000000...
> DLICV                                              0.000002503226143771910958872942143393913738691...
> scanner_lateral_x_brain_position_f25756_2_0        0.075113399418644408167544668231130344793200492...
> scanner_transverse_y_brain_position_f25757_2_0     0.000004369896282589966217174606150086546563215...
> scanner_longitudinal_z_brain_position_f25758_2_0   0.000000000000000000000000000000000000000000000...
> pheno                                              0.000000000000000000000000000000000000000000000...
> dtype: float64

Can you let me know how can I show the results as scientific notation?

I tried to increase the precision of the float in my Python code:

> import pandas as pd
> pd.set_option('display.float_format', lambda x: '%.200f' % x)

This did not work either

Thanks 

user: anbai106
issue_8912: state: open
title: Design/ENH: seaborn as optional dependency ?
body: seaborn has some nice graphics.
The last time that I looked at the seaborn code it looks like it's too nested and structured to copy just individual parts for our use.
(I'm not sure it's worth it. I write only simple statistics plot, e.g. diagnostic plots, without fancy plot elements.)

example pair plot grids
https://stackoverflow.com/questions/48139899/correlation-matrix-plot-with-coefficients-on-one-side-scatterplots-on-another
https://seaborn.pydata.org/tutorial/axis_grids.html#plotting-pairwise-data-relationships


A long time ago I wrote statsmodels.graphics.plot_grids.scatter_ellipse but we don't have much in this area

possible application
acf/ccf plot grids https://github.com/statsmodels/statsmodels/pull/8783#issuecomment-1589685756
VAR acorr_plot is also only very basic
user: josef-pkt
issue_8911: state: open
title: ENH: Add multivariate pdf (dim>2) for Frank and Gumbel copula rebased 8649
body: rebased on main, version of #8649
user: josef-pkt
issue_8910: state: open
title: WIP    ENH: add offset (and exposure) to discrete margeff derivatives
body: #8833

trial version for adding offset and exposure to margeff derivatives.

This is not possible as a quickfix.
`get_margeff` and margeff code will need changes in addition to those in derivatives to support offset and exposure.
I had stopped then partway through refactoring an adding this.

It should be possible to use those derivatives with NonlinearDeltaCov, wald_nonlinear, or (future) get_prediction_nonlinear
Then we would be able to write unit test and examples for them.

(postponed, opening PR so I don't loose track of it)
user: josef-pkt
issue_8909: state: open
title: MAINT: scipy interp2d is deprecated/removed
body: 
https://scipy.github.io/devdocs/notebooks/interp_transition_guide.html

file search only finds one usage of it
`statsmodels.stats.tabledist.TableDist.poly2d`

TableDist is used in `_lilliefors.py`, but poly2d method is not used anywhere AFAICS

user: josef-pkt
issue_8908: state: open
title: MAINT: test_var still uses record arrays and views on it
body: 
```
def get_macrodata():
    data = macrodata.load_pandas().data[["realgdp", "realcons", "realinv"]]
    data = data.to_records(index=False)
    nd = data.view((float, 3), type=np.ndarray)
    nd = np.diff(np.log(nd), axis=0)
    return nd.ravel().view(data.dtype, type=np.ndarray)
```

also: is rpy2 in use or is this stale code (in test_var.py)

```
def generate_var():  # FIXME: make a test?
    import pandas.rpy.common as prp
    from rpy2.robjects import r
    ...
```
user: josef-pkt
issue_8906: state: open
title: ENH: manova type constraint mv_test with L, M, for MNLogit, LikelihoodModel
body: I think we can use patsy's linear constraint functions to create L and M restriction matrices for mv_test applied to MNLogit and similar.
e.g. "y1=y2" to compare parameters of equation 1 and equation 2.

`wald_test_terms` currently does not work for 2-dim params as in MNLogit.
But it would apply to creating the M (or is it L) matrices.

The wald_test_terms constraints would only be for the exog parameters. To get the full constraint matrix for ravelled/stacked params, we need the kronecker product of either eye kron M or something like L cron M. 
(I don't remember right now. I need to check which is which.)

https://github.com/statsmodels/statsmodels/issues/8905#issuecomment-1583081922

user: josef-pkt
issue_8904: state: open
title: FAQ-D, design: multivariate endog, layout, shape overview
body: (I'm trying to get an overview of our multivariate, multi-endog models to see if what shape, layout multi-equation attributes have and whether they are consistent across models)

see also 
- #3137
- #8575
- ...

current models with 2-dim `params`: 
MNLogit, MANOVA/_MultivariateOLS, VAR, VECM, statespace VARMAX, ...?

- 2-dim params, bse
  - MNLogit: `(K, J - 1)`  exog in rows, equations/choices in columns
  - VAR: `(df_model, self.neqs)`
  - MultivariateOLS: (k_exog, k_endgo),   `params = pinv_x.dot(y)`
- ravel, reshape
  - MNLogit
    - to 1-dim: `params = mnfit.params.reshape(self.K, -1, order='F')` (in fit)
    - to 2dim:  `bse.reshape(self.params.shape, order='F')`, `params = params.reshape(self.K, -1, order='F')` (in fit)
  - VAR (also uses 3-dim or list of 2-dim, by lags): 
    - to 1dim: `cov_params = np.kron(np.linalg.inv(z.T @ z), self.sigma_u)`
    - to 2dim:  `stderr.reshape((self.df_model, self.neqs), order="C")`
   - base.LikelihoodModel.t_test and wald_test: `params = self.params.ravel()` (is this consistent with MNLogit?)
- params names
  - base.LikelihoodModel.t_test and wald_test `names = self.model.data.cov_names`
  - see cov_names https://github.com/statsmodels/statsmodels/pull/8907#issuecomment-1584812877
  - cov in MNLogit has order "F", in VAR has order "C" (AFAICS)
- summary()
  - ??? todo 
- predict
- ...
user: josef-pkt
issue_8902: state: open
title: KeyError: (0, 0) for summary() method of MixedLMResultsWrapper
body: #### Describe the bug

I would like to compute a regularized mixed model. After using `.fit_regularized()` the `.summary()` method of the results object returns:

```
Traceback (most recent call last):

  Cell In[36], line 2
    result.summary()

  File ~/.conda/envs/csp_wiesner_johannes/lib/python3.8/site-packages/statsmodels/regression/mixed_linear_model.py:2723 in summary
    sdf[jj, 0] = self.cov_re[i, j]

  File ~/.conda/envs/csp_wiesner_johannes/lib/python3.8/site-packages/pandas/core/frame.py:3761 in __getitem__
    indexer = self.columns.get_loc(key)

  File ~/.conda/envs/csp_wiesner_johannes/lib/python3.8/site-packages/pandas/core/indexes/base.py:3654 in get_loc
    raise KeyError(key) from err

KeyError: (0, 0)
```

#### Code Sample, a copy-pastable example if possible


```python
from statsmodels.regression.mixed_linear_model import MixedLM
import pandas as pd

# recreate dataframe
df = pd.DataFrame.from_dict(
{'Cognition': {0: 1.6918916455399657, 1: -0.8659740639055752, 2: -0.7260401001097657, 
               3: -0.3573630652646015, 4: 3.2169557249585408, 5: -1.351852569895234, 
               6: -1.1175958349091746, 7: 1.4282603161187524, 8: 0.007664156491438558, 
               9: 1.9810784349343789}, 
 'group': {0: 'MDD', 1: 'HC', 2: 'MDD', 3: 'SZ', 4: 'SZ', 5: 'MDD', 6: 'HC', 
           7: 'SZ', 8: 'HC', 9: 'MDD'}, 
 'HIP_rh': {0: 0.6030087333308467, 1: -0.2753717308509293, 2: 0.08376122085063317, 
            3: -1.1422867560432115, 4: -0.11920370317144337, 5: 0.6322097293812017, 
            6: -2.7717229213732795, 7: -1.4557272887144321, 8: 1.094959587039391, 
            9: 1.0869845349546525}}
)

# set up model
model = MixedLM.from_formula(formula="Cognition ~ HIP_rh",
                              data=df,
                              groups=df["group"]
                              )

# this works
result = model.fit()
result.summary()

# this doesn't work
result = model.fit_regularized(method='l1',alpha=0.5)
result.summary()
```
When only using `.fit()`, everything works fine.

<details>

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `main`. If your problem has been fixed in an unreleased version, you might be able to use `main` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the main branch of this repository? It helps the limited resources if we know problems exist in the current main branch so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

Print a results table of the fitted regularized mixed model.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

INSTALLED VERSIONS
------------------
Python: 3.8.16.final.0
OS: Linux 5.15.0-71-generic #78-Ubuntu SMP Tue Apr 18 09:00:29 UTC 2023 x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

statsmodels
===========

Installed: 0.14.0 (/home/johannes.wiesner/.conda/envs/csp_wiesner_johannes/lib/python3.8/site-packages/statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.21.5 (/home/johannes.wiesner/.conda/envs/csp_wiesner_johannes/lib/python3.8/site-packages/numpy)
scipy: 1.10.1 (/home/johannes.wiesner/.conda/envs/csp_wiesner_johannes/lib/python3.8/site-packages/scipy)
pandas: 2.0.1 (/home/johannes.wiesner/.conda/envs/csp_wiesner_johannes/lib/python3.8/site-packages/pandas)
    dateutil: 2.8.2 (/home/johannes.wiesner/.conda/envs/csp_wiesner_johannes/lib/python3.8/site-packages/dateutil)
patsy: 0.5.3 (/home/johannes.wiesner/.conda/envs/csp_wiesner_johannes/lib/python3.8/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.7.1 (/home/johannes.wiesner/.conda/envs/csp_wiesner_johannes/lib/python3.8/site-packages/matplotlib)
    backend: QtAgg 
cvxopt: Not installed
joblib: 1.2.0 (/home/johannes.wiesner/.conda/envs/csp_wiesner_johannes/lib/python3.8/site-packages/joblib)

Developer Tools
================

IPython: 8.12.2 (/home/johannes.wiesner/.conda/envs/csp_wiesner_johannes/lib/python3.8/site-packages/IPython)
    jinja2: 3.1.2 (/home/johannes.wiesner/.conda/envs/csp_wiesner_johannes/lib/python3.8/site-packages/jinja2)
sphinx: 7.0.1 (/home/johannes.wiesner/.conda/envs/csp_wiesner_johannes/lib/python3.8/site-packages/sphinx)
    pygments: 2.15.1 (/home/johannes.wiesner/.conda/envs/csp_wiesner_johannes/lib/python3.8/site-packages/pygments)
pytest: Not installed
virtualenv: Not installed


</details>

user: JohannesWiesner
issue_8901: state: open
title: DOC/ENH: doc examples for multivariate mv_test
body: I don't see any example for how to use mv_test for other hypotheses than default MANOVA.
This needs examples in docstrings and/or notebooks.
e.g. 
row versus column of params (cross-equation or cross-variables)
L versus M constraints on columns versus rows
which is which?


Several examples are in unit tests.
https://stackoverflow.com/questions/76336133/multivariate-linear-hypothesis-testing-using-statsmodels-in-python
user: josef-pkt
issue_8898: state: open
title: ENH: Refactoring or look and feel improvements of STL procedure (_stl.pyx)
body: I would like to address the issues I encountered while trying to understand and follow the code of the STL procedure. The lack of comments and the presence of numerous inexpressive variables made it difficult for me to grasp the logic and flow of the code.

I am unsure whether it would be beneficial to partially rewrite the computational aspects of the procedure or focus only on improving its overall look and feel. In this issue, I propose two code refactoring variants: one involving a partial rewrite of the  implementation and the other focusing on enhancing readability and usability.

Variant 1: Partial Rewriting
- In this variant is review and rewrite some patterns of the STL procedure.
- For example, make some methods static with passing attributes, remove this "work" variable that is really hard to follow (in my opinion), fix all this `work1 = self._work[2, :]` and so on. 

Variant 2: Readability and Usability Enhancements
- In this variant, the primary focus would be on improving the readability and usability of the STL procedure.
- This approach would involve adding comments, providing clear explanations of the code logic, and utilizing expressive variable names.
		
I would like to contribute with PR, but I want to make sure that it is in demand and understand how best to do it.
user: 1pyxa1
issue_8896: state: open
title: ENH: Implement append, apply and extend for ETSResults
body: The current implementation of [`ETSResults`](https://www.statsmodels.org/dev/generated/statsmodels.tsa.exponential_smoothing.ets.ETSResults.html) doesn't have any of the following methods that are available in [`ExponentialSmoothingResults`](https://www.statsmodels.org/dev/generated/statsmodels.tsa.statespace.exponential_smoothing.ExponentialSmoothingResults.html) and the [`ARIMAResults`](https://www.statsmodels.org/dev/generated/statsmodels.tsa.arima.model.ARIMAResults.html#statsmodels.tsa.arima.model.ARIMAResults) models (links to the exponential smoothing equivalents):
- [append](https://www.statsmodels.org/dev/generated/statsmodels.tsa.statespace.exponential_smoothing.ExponentialSmoothingResults.append.html#statsmodels.tsa.statespace.exponential_smoothing.ExponentialSmoothingResults.append)
- [apply](url)
- [extend](https://www.statsmodels.org/dev/generated/statsmodels.tsa.statespace.mlemodel.MLEResults.extend.html#statsmodels.tsa.statespace.mlemodel.MLEResults.extend)

Having these methods available will be very nice, especially for "online" models and also implementations of Bagging ETS with different sources of uncertainty like in the paper below:
- Fotios Petropoulos, Rob J. Hyndman, Christoph Bergmeir, "Exploring the sources of uncertainty: Why does bagging for time series forecasting work?", European Journal of Operational Research, Volume 268, Issue 2, 2018, available at https://www.sciencedirect.com/science/article/abs/pii/S037722171830081X#:~:text=We%20suggest%20that%20another%20reason,forms%20on%20the%20original%20data.

This is possibly related to issue #6881 as the methods for ARIMA and ExponentialSmoothing are inherited from the [`MLEResults`](https://github.com/statsmodels/statsmodels/blob/77cb066320391ffed4196a32491ddca28e8c9122/statsmodels/tsa/statespace/mlemodel.py#L2262) class.

user: ltsaprounis
issue_8895: state: open
title: No out-of-sample forecast for MLEModel.smooth
body: After implementation of Kalman filter with TVPVAR class based on example implementation I realized that there are no out-of-sample forecast option. 
I also didn't find any summary with significance of coefficients (for last observation). Impulse response does not work properly. 
user: Austrell
issue_8894: state: open
title: ExponentialSmoothing Model Produces Different Results on Mac vs Linux
body: ## Description

I am running an `ExponentialSmoothing` model and am seeing differences in the output on Linux vs Mac. I'm trying my best to reconcile these differences to see if I can get them to produce the same results regardless of OS.

## Versions Used

I didn't see anything indicating that these results were sensitive to much to the versions being ran, but for the sake of consistency, I fixed the following versions between environments:

- Python: 3.8.16
- Numpy: 1.24.3
- Pandas: 1.5.3
- Statsmodels: 0.14.0
- Scipy: 1.10.1

## Code to Reproduce

```python
from statsmodels.tsa.holtwinters import ExponentialSmoothing
import pandas as pd

df = pd.read_csv("~/Downloads/AirPassengers.csv")

X = df["#Passengers"].values
ts = df["Month"].values

model = ExponentialSmoothing(
    X,
    trend=None,
    damped_trend=False,
    seasonal=None,
    seasonal_periods=None,
    use_boxcox=False,
    dates=ts,
).fit(
    smoothing_level=None,
    smoothing_trend=None,
    smoothing_seasonal=None,
    use_brute=False,
)

result = model.predict(start=ts[0], end=ts[-1])
print(result.mean())
```

> The dataset being used here can be found on Kaggle [here](https://www.kaggle.com/datasets/rakannimer/air-passengers?resource=download)

This code gives me the following results:

- On my Mac: `275.84320334735634`
- On Linux:     `278.07638822684953`

Further, If I instead check `model.params` I see that the differences arise before even predicting:

On my Mac:

```python
{'smoothing_level': 0.5,
 'smoothing_trend': nan,
 'smoothing_seasonal': nan,
 'damping_trend': nan,
 'initial_level': 118.46666666666658,
 'initial_trend': nan,
 'initial_seasons': array([], dtype=float64),
 'use_boxcox': False,
 'lamda': None,
 'remove_bias': False}
```

On Linux:

```python
{'smoothing_level': 0.9999999850981204,
 'smoothing_trend': nan,
 'smoothing_seasonal': nan,
 'damping_trend': nan,
 'initial_level': 111.99990880905746,
 'initial_trend': nan,
 'initial_seasons': array([], dtype=float64),
 'use_boxcox': False,
 'lamda': None,
 'remove_bias': False}
```

Would someone be able to help me understand why I am getting these differences?
user: justinessert
issue_8893: state: open
title: ValueError: endog must be strictly positive when usingmultiplicative trend or seasonal components.
body: Hi guys so I was trying to apply the exponential smoothing as below 
fitted_model = ExponentialSmoothing(train_data["Sales"],
                                    trend = "mul",
                                    seasonal = "mul",
                                    seasonal_periods = 365).fit()
And I got the error: ValueError: endog must be strictly positive when usingmultiplicative trend or seasonal components.
The data type is float and there are not negative values, also I double checked just to make sure. 
Also, I tried with .abs() but same result.
Any idea what might be the cause ?
Thank you in advance
user: jurges1
issue_8892: state: open
title: ENH/BUG: vectorized NonlinearDeltaCov and get_wald_nonlinear
body: It looks like NonlinearDeltaCov and `_get_wald_nonlinear` work for vectorized (2d) functions, but predicted and se have transposed shapes.

margeff does not have inference `se` if `at="all"`  (used with atexog=exog array with more than one row)

using example from statsmodels.stats.tests.test_deltacov.test_deltacov_margeff
with either Poisson or GLM-Poisson (#8889)


```
def f(p):
    ex = res_poi.model.exog[:5]
    fv = res_poi.model._derivative_exog(p, ex)
    return np.squeeze(fv)

nlp = NonlinearDeltaCov(f, res_poi.params, res_poi.cov_params())

nlpm = res_poi._get_wald_nonlinear(f)

nlp.predicted().shape, nlp.se_vectorized().shape
((5, 10), (10, 5))
nlpm.predicted().shape, nlpm.se_vectorized().shape
((5, 10), (10, 5))
```
user: josef-pkt
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_8891: state: open
title: ENH/REF/BUG: negative df_resid with freq_weights, negative scale
body: first example from
https://stackoverflow.com/questions/76292524/different-weighted-linear-regression-results-for-r-and-python-bug-or-just-diffe

I was trying to use (var) weights as freq_weights but weights are small so that df_resid is negative.
This results in negative scale for GLM gaussian with pearson scale, and nan bse and inference in summary.
The same nan bse happens with `cov_type="HC0"`, but I have not looked at why that happens.

I guess this cannot happen with var_weights and pinv, because df_model is based on rank which cannot be larger than nobs.

What do we do in this case?
raise, warn ?

Related:
AFAIR, we don't explicitly rule out negative weights. Those might raise an exception in some computation, but not intentionally.
 


user: josef-pkt
issue_8890: state: open
title: ENH: hypothesis tests based on cov, hypothesis/contrast class
body: I have commented on this several times, but don't find a specific issue.

Currently wald tests and associated cov_params are integrated into the results classes. They cannot be reused outside of models.

We need a class that can compute hypothesis tests and other inference like conf_int based on mean of effects and their cov.
This will provide t_test and wald_test for, e.g., postestimation results, for which we compute the expected value and a (possibly vectorized) multivariate cov_effect.

AFAIR, there are applications for comparing predictions.

NonlinearDeltaCov is similar already, but it is for a given nonlinear, multivalued or vectorized function, and targeted to the current usecase of nonlinear wald tests and get_prediction.

What we need is a class (maybe Mixin class) that includes the functionality of LikelihoodModelResults t_test and wald_test, and conf_int, i.e. for linear constraints on effects that might be based on a nonlinear, multivalued function.

Not clear yet, vectorization:
AFAICS, NonlinearDeltaCov, has vectorized or joint hypothesis but not both, and only for equal to value hypotheses but not for arbitrary linear constraints (contrasts) like testing that two `effects` are equal.
t_test and wald_test are similar in that the first is vectorized for individual hypothesis (based on `se` and not on joint cov) and the second handles joint hypothesis based on cov.

Full vectorization would have a 2dim array of effects, rows are independent cases, columns represent a multivariate effect.
If cov(_effect) is common to all rows, then vectorization should be relatively straightforward (last axis is multivariate).
If covariance differs across rows, then it gets messy, more memory and joint hypothesis tests require vectorized linear algebra (vectorized quadratic forms).


related example
TreatmentEffectResults subclasses ContrastResults which essentially presents `t_test` results.
(however, "effects" in this case are linear dependent, i.e. we have 3 vectorized hypothesis on 2 underlying effects (ATE and POM0))

possible usecase to check:
get_prediction with multi-valued prediction, e.g. `which="prob"`

GMM with moment conditions for extra effects, and we want an inference class for the extra effects (base on estimated expected value an submatrix of cov_params)

linear constrast of two predictions: current `_wald_nonlinear` can compute effect and hypothesis for one effect, eg. difference between the two predictions, but does not compute the joint cov. 
(This sounds a bit similar to treatment effect case, but for predictions and not for all extra moment conditions.)


Maybe related
vectorization of score test, i.e. we have "joint" versus "separate" option, but it computes the hypothesis test directly (like t_test and wald_test).


Aside: `paired` versus `independent`
Does correlation of "effects" correspond to "paired" version?  (for linear constrasts) (might require more assumptions on specification)
e.g. #8784 evaluating out of sample predictions
Extra in-sample effects in GMM are or may be correlated.
risk ratio: is it average of ratio or ratio of average  #2215  but that also has nonlinear effect E f(x, y) != f(E x, E y)











user: josef-pkt
issue_8888: state: open
title: ENH: NegativeBinomial nbinom link, inverse_deriv2 is missing, check domain restrictions, clipping
body: I was trying out an example with canonical nb link for NegativeBinomial family.

It converges for a simple 2-sample problem.
But fails in convergence or has almost inf cov_params in a more generic case with continuous exog.

inverse_deriv2 is missing and should not be difficult to add

formulas in NegativeBinomial link seem to be correct, manual checking and we have consistency unit tests 

A bit weird: linear predictor needs to be negative
The link function produces negative return.

Worth a try: 
see if we can get better starting values.
Should we clip some more?
#2208 issue for domain violations with some links

motivation:
I was trying to see whether canonical link in negative binomial provides us with interesting features.

aside:
With negativebinomial link, the mean function depends on the dispersion parameter alpha, i.e. we have a parameterized link.
So, the mean parameters cannot be interpreted independently of the dispersion parameter with this link.
 

user: josef-pkt
issue_8884: state: open
title: glm_res.summary() okay but glm_res.params returns nan
body: Everything is in the title. Is this user error? Thanks!

Example current output (using v0.13.5):

```
>>print(glm_res.summary())
                 Generalized Linear Model Regression Results                  
==============================================================================
Dep. Variable:                  count   No. Observations:                    9
Model:                            GLM   Df Residuals:                        7
Model Family:        NegativeBinomial   Df Model:                            1
Link Function:                    Log   Scale:                          1.0000
Method:                          IRLS   Log-Likelihood:                -69.232
Date:                Tue, 16 May 2023   Deviance:                      0.52005
Time:                        22:33:55   Pearson chi2:                    0.474
No. Iterations:                     5   Pseudo R-squ. (CS):             0.4402
Covariance Type:            nonrobust                                         
===============================================================================
                  coef    std err          z      P>|z|      [0.025      0.975]
-------------------------------------------------------------------------------
Intercept      -7.0186      0.483    -14.538      0.000      -7.965      -6.072
timepoint_h    -0.0840      0.034     -2.485      0.013      -0.150      -0.018
===============================================================================

>>glm_res.params
Intercept     NaN
timepoint_h   NaN
dtype: float64


>> glm_res.fit_history["params"]
[inf,
 array([0., 0.]),
 array([-6.99930344, -0.07846853]),
 array([-7.0161946 , -0.08378005]),
 array([-7.01849536, -0.08396935]),
 array([-7.01861196, -0.08395922]),
 array([-7.01860581, -0.08395982])]

>> glm_res._results.params
const   -7.018606
x1      -0.083960
dtype: float64

user: laserson
issue_8882: state: open
title: out of sample prediction of Markov switching models
body: In the problem of multivariate time series prediction，I want to perform out of sample prediction for Markov models( tsa\regime_switching\markov_switching.py). Is there a solution? I have observed that there seems to be a NotImplementedError.


user: huangzhaor
issue_8878: state: open
title: BUG Fix typo in InfeasibleTestError exception string
body: - [ ] closes #xxxx
- [? ] tests added / passed. 
- [x ] code/documentation is well formatted.  
- [x ] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

<details>
I've updated statsmodels/tsa/stattools.py to correct a minor typo (compute -> computed) for the InfeasibleTestError thrown within grangercausalitytests().
<br><br>
I'm not sure how to write a test to check a string but have included a test that will throw the exception in question to verify the string.
</details>

user: emorisse
issue_8877: state: open
title: DOC: check QMLE for GLM with var_weights
body: https://stackoverflow.com/questions/76191135/why-do-i-get-different-results-when-i-want-to-find-the-aic-of-my-model-in-python/76202275#76202275

examples, relationship to check:
I think the dispersion interpretation of `var_weights` should imply that we can estimate dispersed families like NegativeBinomial or BetaBinomial using the undispersed family with var_weights to account for excess dispersion.

just an idea, needs checking, but it would make a good example in the docs.
user: josef-pkt
issue_8871: state: open
title: ENH/Design: too much inheritance, list for ok or for unsupported features
body: Stata has some `ok` flags if some features are not allowed for a model.

Until now I was not convinced that we need it.

However, if/when we add more models as subclasses, then we might want to prohibit some post estimation methods by default.
e.g. when we add subclasses of models based on mixins
PenalizedXxx
MixedXxx
...

The current pattern assumes that some inherited feature either works generically or raises an exceptions somewhere. However, the inherited generic methods might not be generic enough to produce the correct results if they don't raise an exception.

This will be a bit easier once we have "generic" testing of post-estimation features that I started for recent reviews of get_distribution, get_influence, score_test, ....
 

This affects currently mainly discrete, especially count models. That's were we have a larger variety of models and the largest number of submodels.

user: josef-pkt
issue_8868: state: open
title: Compilation failures with Cython 3.0
body: #### Describe the bug

When building against Cython 3.0.0b2, I get the following errors:

<details>
<summary>Error output</summary>

```
Error compiling Cython file:
------------------------------------------------------------
...
    # smooth
    for i in range(n):
        prev = (n+i-1) % n

        # s[t-m] = xhat[prev, 2+m-1]
        yhat[i] = (xhat[prev, 0] * xhat[prev, 1]**phi) + xhat[prev, 2+m-1]
                                                       ^
------------------------------------------------------------

statsmodels/tsa/exponential_smoothing/_ets_smooth.pyx:156:55: Cannot assign type 'npy_float64 complex' to 'float64_t'

Error compiling Cython file:
------------------------------------------------------------
...

        # s[t-m] = xhat[prev, 2+m-1]
        yhat[i] = (xhat[prev, 0] * xhat[prev, 1]**phi) + xhat[prev, 2+m-1]
        # l_t = a * (y_t - s_t-m) + (1-a) * (l_t-1 * b_t-1**phi)
        xhat[i, 0] = (alpha * (y[i] - xhat[prev, 2+m-1])
                      + (1 - alpha) * (xhat[prev, 0] * xhat[prev, 1]**phi))
                      ^
------------------------------------------------------------

statsmodels/tsa/exponential_smoothing/_ets_smooth.pyx:159:22: Cannot assign type 'npy_float64 complex' to 'float64_t'

Error compiling Cython file:
------------------------------------------------------------
...
        # l_t = a * (y_t - s_t-m) + (1-a) * (l_t-1 * b_t-1**phi)
        xhat[i, 0] = (alpha * (y[i] - xhat[prev, 2+m-1])
                      + (1 - alpha) * (xhat[prev, 0] * xhat[prev, 1]**phi))
        # b_t = (b*) * (l_t / l_t-1) + (1 - (b*)) * b_t-1**phi
        xhat[i, 1] = (beta_star * (xhat[i, 0] / xhat[prev, 0])
                      + (1 - beta_star) * xhat[prev, 1]**phi)
                      ^
------------------------------------------------------------

statsmodels/tsa/exponential_smoothing/_ets_smooth.pyx:162:22: Cannot assign type 'npy_float64 complex' to 'float64_t'

Error compiling Cython file:
------------------------------------------------------------
...
    # smooth
    for i in range(n):
        prev = (n+i-1) % n

        # s[t-m] = xhat[prev, 2+m-1]
        yhat[i] = (xhat[prev, 0] * xhat[prev, 1]**phi) * xhat[prev, 2+m-1]
                                                       ^
------------------------------------------------------------

statsmodels/tsa/exponential_smoothing/_ets_smooth.pyx:191:55: Cannot assign type 'npy_float64 complex' to 'float64_t'

Error compiling Cython file:
------------------------------------------------------------
...

        # s[t-m] = xhat[prev, 2+m-1]
        yhat[i] = (xhat[prev, 0] * xhat[prev, 1]**phi) * xhat[prev, 2+m-1]
        # l_t = a * (y_t / s_t-m) + (1-a) * (l_t-1 * b_t-1**phi)
        xhat[i, 0] = (alpha * (y[i] / xhat[prev, 2+m-1])
                      + (1 - alpha) * (xhat[prev, 0] * xhat[prev, 1]**phi))
                      ^
------------------------------------------------------------

statsmodels/tsa/exponential_smoothing/_ets_smooth.pyx:194:22: Cannot assign type 'npy_float64 complex' to 'float64_t'

Error compiling Cython file:
------------------------------------------------------------
...
        # l_t = a * (y_t / s_t-m) + (1-a) * (l_t-1 * b_t-1**phi)
        xhat[i, 0] = (alpha * (y[i] / xhat[prev, 2+m-1])
                      + (1 - alpha) * (xhat[prev, 0] * xhat[prev, 1]**phi))
        # b_t = (b*) * (l_t / l_t-1) + (1 - (b*)) * b_t-1**phi
        xhat[i, 1] = (beta_star * (xhat[i, 0] / xhat[prev, 0])
                      + (1 - beta_star) * xhat[prev, 1]**phi)
                      ^
------------------------------------------------------------

statsmodels/tsa/exponential_smoothing/_ets_smooth.pyx:197:22: Cannot assign type 'npy_float64 complex' to 'float64_t'
```
</details>

While this is still a beta, it means that the release is just around the corner and we're trying to report problems early. Furthermore, some packages *already* decided to require Cython-3 which makes all incompatibilities a true PITA for Linux distributions.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

INSTALLED VERSIONS
------------------
Python: 3.10.11.final.0
OS: Linux 6.2.14-gentoo-dist #1 SMP PREEMPT_DYNAMIC Mon May  1 15:03:02 -00 2023 x86_64
byteorder: little
LC_ALL: None
LANG: C.UTF8

statsmodels
===========

Installed: 0.14.0 (/usr/lib/python3.10/site-packages/statsmodels)

Required Dependencies
=====================

cython: 0.29.34 (/usr/lib/python3.10/site-packages/Cython)
numpy: 1.24.3 (/usr/lib/python3.10/site-packages/numpy)
scipy: 1.10.1 (/usr/lib/python3.10/site-packages/scipy)
pandas: 2.0.1 (/usr/lib/python3.10/site-packages/pandas)
    dateutil: 2.8.2 (/usr/lib/python3.10/site-packages/dateutil)
patsy: 0.5.3 (/usr/lib/python3.10/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.7.1 (/usr/lib/python3.10/site-packages/matplotlib)
    backend: agg 
cvxopt: Not installed
joblib: 1.1.0 (/usr/lib/python3.10/site-packages/joblib)

Developer Tools
================

IPython: 8.13.2 (/usr/lib/python3.10/site-packages/IPython)
    jinja2: 3.1.2 (/usr/lib/python3.10/site-packages/jinja2)
sphinx: 6.2.1 (/usr/lib/python3.10/site-packages/sphinx)
    pygments: 2.15.1 (/usr/lib/python3.10/site-packages/pygments)
pytest: 7.3.1 (/usr/lib/python3.10/site-packages/pytest)
virtualenv: 20.23.0 (/usr/lib/python3.10/site-packages/virtualenv)


</details>

user: mgorny
issue_8859: state: open
title: ENH: variable addition autoregression test using score_tes
body: Just a thought while looking at #8854

We could include past endog as regressors for a score_test that there is no autoregressive, dynamic effect.

This could be done with current score_test, except that we need to truncate initial observations, because past endog (or linpred) will not have the full nobs length. (The first observation(s) will not have past endog values)

If we drop initial observations with missing exog_extra, then the sample changes compared to the estimation sample. The score for the included parameters will not be exactly zero, and score_test (using all moment conditions as we do now) will have an additional term of dropping the score_obs of the dropped initial observation(s).
That will not be relevant in large sample, but could have non-negligible effect in small samples.

Score_test under the no dynamic effect null should be correct, even with endogenous, pre-determined exog_extra, similar to score_test (pseudo-wald) test for endogeneity with null that regressor is not endogenous.

Alternative: 
users can drop initial observations in the model estimate and then use exog_extra with the full length of the estimation sample.

Alternative 2:
impute the missing initial observations.
e.g. impute `exog_extra[:k] = 0` which will have no contribution to the score_obs, (and AFAICS, have zero covariance  of score_obs_extra with included score_obs in contribution to cov(score))
user: josef-pkt
issue_8856: state: open
title: ENH: score_factor, generalized residual for ordered model
body: ordered model has (private) `score_obs_` and score_factor computation w.r.t. to latent function, but is missing the explicit derivatives w.r.t. thresholds.

use cases for generalized residuals, score_factor for latent function

- control function of ordered endogenous regressor
- residual diagnostic
- score_test for variable addition, I think this should also work, when we have score_obs for thresholds (or all params), possibly by numdiff.

One likely implementation issue:
Do we need score_factor for all possible values of the endog variable? For residuals we do.
Currently, loglike and score, and `score_obs_`, compute only the contribution for the observed endog values (saves computation and memory) .

reference for residual diagnostic (also has an explanation of generalized residuals as "score_factor":

Iannario, Maria, and Anna Clara Monti. “Generalized Residuals and Outlier Detection for Ordinal Data with Challenging Data Structures.” Statistical Methods & Applications, February 28, 2023. https://doi.org/10.1007/s10260-023-00686-1.

user: josef-pkt
issue_8854: state: open
title: DOC/ENH/REF:  review score_test
body: docstring for score_test is not clear enough
(Rereading it, I don't understand, remember some of the details)

e.g. cov_type
It should be made clearer that this keyword is only used to override the cov_type defined in `fit`.
AFAIR, I had added this for when we can have different small sample correction in Wald and score_tests for each of the sandwiches.

return is tuple of numbers, returning a HolderTuple would be better

`hypotheses="separate"`: no unit tests AFAICS, 
returns also the score_test for included variables which have statistic (approx) equal to zero.  It's confusing for testing zero constraints with exog_extra, but likely it's needed for other constraints on params (e.g. linear constraints).
Also, it looks like it's a "partial" test, robust to "other" nonzero alternative. (reference?)

Using an exog_extra that is already included raises SingularMatrix exception if hypotheses="joint". 
But if hypothesis is "separate", then the extra condition also shows up with statistic equal to zero.

docstring has no reference
We need at least reference to "sandwich" form of score-test

Notes section: 
The description in "Status" sounds too negative. What, besides "separate" is not verified. `cov_type` is waiting for enhancement.

The description of cov_type HC0 only mentions covariance matrix of the score. It's still the sandwich form, isn't it?

Aside: What's the status with k_extra? Does score_test support hypothesis for mean parameters in models that have extra params?
It looks like discrete test_predict.py has basic unit tests for NBP and GPP.
ZI count model and models in truncated inherit score_test, but I guess it will not work for those. 
BetaModel BetaResults does not have score_test. However it has unit test using the generic score_test function. Those only check that two versions of using score_test produce the same/equivalent results, no outside verification.
user: josef-pkt
issue_8853: state: open
title: ENH: added variable plot, similar to score_test
body: added variable plot in regressionplots is currently limited to GLM, GEE and OLS.

I'm trying to find a more general version that can be used with discrete models.

The version that I came up with is using similar pattern as the score_test.
Given restricted model and (one) exog_extra we use score factor and exog_extra partialled out for included exog

This is base on moment condition

score_obs = score_factor * exog

My current version does not take hessian, variance (cov) of score_obs into account.

plot is partialled exog_extra on horizontal axis and
score_factor (score_residual, generalized residual) on vertical axis

user: josef-pkt
issue_8852: state: open
title: the forecast jump too high when I put exogenous variable for SARIMAX
body: #### Exogenous variable (holiday) make the forecast jump too high 

[Halo Team.
I am currently try to implement SARIMAX for hotel prediction, and put holidays season as exogenous variable. But, the forecast jump too high on that period, compared to real data. ]

#### Code Sample, a copy-pastable example if possible


```python
import numpy as np
import pandas as pd
import statsmodels as sm
from statsmodels.tsa.statespace.sarimax import SARIMAX
import matplotlib.pyplot as plt

holidays_df = pd.DataFrame({'index':list(germany_holidays.keys()),
                           'holiday':list(germany_holidays.values())})
dates = pd.date_range('2023-01-01','2023-12-31', freq='D')
exor_vardf = pd.DataFrame({'value':np.zeros(len(dates))}, index=dates)
for holiday in holidays_df['index']:
    holiday_date = pd.Timestamp(holiday)
    if holiday == 2023-12-26:
        exor_vardf.loc[holidays_df:holidays_df+pd.Timedelta(days=14),'value'] = 1
    else :
        exor_vardf.loc[exor_vardf.index == holiday_date, 'value'] = 1
exor = pd.DataFrame({'value':np.zeros(365)}, index=pd.date_range('2023-01-01', periods=365, freq = 'D'))
exor['holiday'] = exor.index.isin(holidays_df['index'])
christmas, new_year = pd.to_datetime('2023-12-25'), pd.to_datetime('2023-01-01')
exor.loc[christmas:christmas + pd.DateOffset(days=7), 'value'] = 1
exor.loc[new_year:new_year + pd.DateOffset(days=7), 'value'] = 1
exor.loc[exor['holiday'], 'value'] = 1

fitted, confint = SARIMAX_model.predict(n_periods=n_periods, 
                                            return_conf_int=True,
                                            exogenous=forecast_df[['days_index']])

SARIMAX_model = auto_arima(frame[['cnt']], exogenous=exor[['value']],
                           start_p=1, start_q=1,
                           test='adf',
                           max_p=3, max_q=3, m=7,
                           start_P=0, seasonal=True,
                           d=0, D=1, 
                           trace=False,
                           error_action='ignore',  
                           suppress_warnings=True, 
                           stepwise=True)

SARIMAX_model.summary()
```
So, beside the exo variable valued one in every holidays, I also put 1 from 25 Dec until 7 January. thats all. 
The result is, the real data is around 500-1000 at the peak during that days (new year) but the prediction goes to more than 3000.


**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `main`. If your problem has been fixed in an unreleased version, you might be able to use `main` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the main branch of this repository? It helps the limited resources if we know problems exist in the current main branch so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

My expected output would be, when I put the exo variable how to still manage it so it does not predict too far. 
#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]
INSTALLED VERSIONS
------------------
Python: 3.9.13.final.0

statsmodels
===========

Installed: 0.13.2 (C:\Users\uuu\Anaconda3\lib\site-packages\statsmodels)



</details>

user: belajardarinol-hash
issue_8850: state: open
title: ENH: outlier influence xy_outlier plot
body: I don't find a SUMM or roadmap issue for outlier-influence

Just a thought: add an xy-outlier plot with mahalanobis distance of exog on horizontal and some residual, y-outlier on vertical axis.

This works if we don't have the standard horizontal measure, and is more explicit that implicitly redefining attributes.
And it's interesting in it's own.

(in some cases we might have x-outliers that are not influential if they receive very low weight e.g. in IRLS/WLS)

Iannario, Maria, and Anna Clara Monti. “Generalized Residuals and Outlier Detection for Ordinal Data with Challenging Data Structures.” Statistical Methods & Applications, February 28, 2023. https://doi.org/10.1007/s10260-023-00686-1.

user: josef-pkt
issue_8849: state: open
title: BUG/REF  _ConditionalModel.fit ignores kwargs, cov_type
body: I looked by chance again at https://stats.stackexchange.com/questions/526459/how-to-correct-conditional-poisson-standard-errors-for-over-dispersion

In the example the user uses `cov_type` in `fit` but it does not do anything.
kwargs are swallowed, i.e. fit has kwargs but those are not used not transmitted to super().fit

to the topic: 
I don't know whether cov_type, or excess-dispersion scale correction makes sense in conditional models. 
I guess that we need the Poisson or Logit (canonical link) assumption and cannot have QMLE. 
But maybe we only need the  assumption on mean function and moment condition i.e. it might work for exp mean function with multiplicative random effect. (similar to GMMPoisson). (I never looked at that for conditional models.)
Then we need at least HC cov_type for  non-negative continuous endog.
quick check: Stata xtpoisson (FE conditional poisson) allows vce(robust) but not cluster robust.

aside: Currently sandwich cov would not be available because there is no score_obs. The score and loglike are computed based on groups and not based on observations.
There is score_grp, which could be used for "between" robustness correction for cov(score) 

user: josef-pkt
issue_8848: state: open
title: Inverted parameters on statsmodels.tsa.interp.denton
body: #### Describe the bug

The parameter of the function is inverted. First it is suppose to be the benchmark and then the indicator, like the documentation:
```
# def denton(indicator, benchmark, freq="aq", **kwarg):
#    """
#    Denton's method to convert low-frequency to high frequency data.
#
#    Parameters
#    ----------
#    benchmark : array_like
#        The higher frequency benchmark.  A 1d or 2d data series in columns.
#        If 2d, then M series are assumed.
#    indicator
#        A low-frequency indicator series.  It is assumed that there are no
#        pre-sample indicators.  Ie., the first indicators line up with
#        the first benchmark.
#    freq : str {"aq","qm", "other"}
#        "aq" - Benchmarking an annual series to quarterly.
#        "mq" - Benchmarking a quarterly series to monthly.
#        "other" - Custom stride.  A kwarg, k, must be supplied.
```
Instead, it is like this: `def denton(indicator, benchmark, freq="aq", **kwarg)`.

#### Code Sample, a copy-pastable example if possible


```python
# Your code here that produces the bug
# This example should be self-contained, and so not rely on external data.
# It should run in a fresh ipython session, and so include all relevant imports.
```
<details>

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `main`. If your problem has been fixed in an unreleased version, you might be able to use `main` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the main branch of this repository? It helps the limited resources if we know problems exist in the current main branch so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

A clear and concise description of what you expected to happen.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]

</details>

user: brenoimatos
issue_8844: state: open
title: REF: offset, exposure as attributes with None
body: Currently, discrete models (at least some) delete or do not attach attribute if model arguments are None (default).

This requires hasattr and getattr workarounds, e.g.
`offset=getattr(self, "offset", None),`

It would be easier and cleaner to always have the attribute, with None

None might be better than no-op offset=0 and exposure=1, to save on redundant computation. 
I'm not sure we do save the redundant computation and don't just set them to no-op values internally, e.g. we do the latter in predict.
 
#3810 general issue for review and refactor of discrete

user: josef-pkt
issue_8843: state: open
title: TST/BUG/ENH: status discrete post-estimation
body: checking discrete post estimation, including support for offset
I'm writing generic checks that catch any errors to see what the current status is

specific issues:
#8842
user: josef-pkt
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_8842: state: open
title: BUG: offset in truncated model raises valueerror
body: using offset in TruncatedLFPoisson raises
`ValueError: offset is not the same length as endog`

same should be in other truncated models and with exposure

```
        self.model_main = Poisson(self.endog, self.exog,
                                  exposure=exposure,
                                  offset=offset)
```
should use self.exposure and self.offset which are the reduced arrays of nontruncated observations.

The truncated countmodels do not require that we get only nontruncated observations in endog.
user: josef-pkt
issue_8841: state: open
title: FAQ-D/Maint/ENH: sphinx directive versionadded, versionchanged
body: https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#directive-versionadded

This would be very helpful to add.
(I often need to check PRs, blame or release notes to see when parts were added to statsmodels)

```
    Notes
    -----
    .. versionadded:: 0.14.0
       some text if needed

    .. versionchanged:: 2.5
       The *spam* parameter.
```
user: josef-pkt
issue_8833: state: open
title: BUG: discrete CountModel does not support offset, exposure in `_derivative_exog`
body: **update**
I had already opened to issues for that a long time ago
#1818 #2164

discrete CountModel has no offset exposure in `derivative_exog` and use `predict(params, exog)`
```
_derivative_exog(self, params, exog=None, transform="dydx",
                         dummy_idx=None, count_idx=None)
```

**update** `_derivative_predict` same problem missing offset and exposure

Ignoring exposure could be justified by that margins are computed for one unit of exposure. But we might want to see exposure weighted average.
**update** not a bug, it's documented in get_margeff
`When using after Poisson, returns the expected number of events per  period, assuming that the model is loglinear.`
This means we cannot change the default, or it breaks backwards compatibility.

Ignoring nonzero offset is a bug and cannot be justified.
i.e. margeff results will change if the user had specified offset in the model after fixing this

`BinaryModel. _derivative_exog` already includes `offset`
**update** still a bug for "ey" which requires a second call to predict which doesn't have offset

`GenericZeroInflated` method raises NotImplementedError
no method in HurdleCountModel, and truncated models, maybe it's incorrectly inherited from CountModel

Maybe additional bugs: 
HurdleCountResults does not overwrite any `get_xxx` results methods. Are inherited working but incorrectly?



user: josef-pkt
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_8832: state: open
title: ENH: summary for multiple models
body: We need helper class that returns str, repr, to_latex, ... for summaries of several models.
This would essentially just concatenate the formatted output, e.g. strings for printing.
Maybe some subheaders or other adjustments to title to indicate the role of the model. 

Usecase 2-stage models: first stage model(s), second stage outcome model, and GMM model for combined cov_params computation.
In this case all models have the correct cov_params. (outcome model cov_params are corrected for first stage.)
#8812, #7689

alternative: 
results method with option for which summary to return. The user will get only one summary at a time.

With the helper class, the user could just call something like `summary_combined`

This only applies if submodels are "correct" models, not if they are internal auxiliary models that don't have corrected attributes like cov_params.

user: josef-pkt
issue_8827: state: open
title: ENH: tools: computing in blocks for matrix algebra, cov for multivariateLS and multi-equation or multi-stage models
body: Computing a sandwich cov_params for MultivariateLS and multi-equation models can be memory intensive.

One way to avoid larger arrays is to compute in blocks.
`linearmodels` has some utility functions for computing a cov in blocks.
https://github.com/bashtage/linearmodels/blob/main/linearmodels/system/_utility.py

current use case IV and control functions for GMM #7689 #8812
The easiest which I currently use is to horizontally stack all moment conditions, score_obs and use GMM.
This works well if we have only e.g. two equation or two models plus maybe some extra moments as in treatment effect.

However, if there are several endogenous explanatory variables and first stage is MultivariateLS, then using the full, stacked mom_cond will become memory intensive.

For multivariateLS with all identical exog, it will avoid keeping several copies of exog in memory if we loop over the cross equation cov(moments) and build cov in blocks.

related computing the sandwiches after we have cov and hessian blocks
#8803 which assumptions about information matrix equality
#7814 and similar for using inverse of block matrices
#8421 computing hessian in blocks or for extra moment conditions

user: josef-pkt
issue_8824: state: open
title: ENH: control function for endogenous switching regession
body: In endogenous switching regression the parameters change depending on the "treatment", i.e. exog interacts with "treatment".

If switching is "endogenous" we can have correlation of noise terms in first and second stage model.
This can be estimated by two-stage control function method but we have to get the appropriate control function for each regime.

It looks like that we can share most of the code with Two-stage control function, but we need to see what "residual" inclusion we need for these cases.
The advantage of switching regression compared to sample selection is that we observe the same observations in the first and second model.

I didn't check the details yet.
I looked through Chiburis and Lokshin 2007 which have a 2-stage method with ordered Probit. control function is diff in density divided by diff in cdf form normal, (similar to inverse Mills hazard ratio, but defined on diff for ordered segments.)

Murtazashvili, Irina, and Jeffrey M. Wooldridge. “A Control Function Approach to Estimating Switching Regression Models with Endogenous Explanatory Variables and Endogenous Switching.” Journal of Econometrics, Endogeneity Problems in Econometrics, 190, no. 2 (February 1, 2016): 252–66. https://doi.org/10.1016/j.jeconom.2015.06.014.

Chiburis, Richard, and Michael Lokshin. “Maximum Likelihood and Two-Step Estimation of an Ordered-Probit Selection Model.” The Stata Journal 7, no. 2 (June 1, 2007): 167–82. https://doi.org/10.1177/1536867X0700700202.

Lokshin, Michael, and Zurab Sajaia. “Maximum Likelihood Estimation of Endogenous Switching Regression Models.” The Stata Journal 4, no. 3 (August 1, 2004): 282–89. https://doi.org/10.1177/1536867X0400400306.



user: josef-pkt
issue_8818: state: open
title: ENH: tools discrete approximation to empirical distribution
body: Integrating a function at many points with respect to the empirical distribution requires a large amount of computation.

Is there a good way to get a discrete approximation with few points to an empirical distribution?

example usecase
smearing estimates for expectation of nonlinear function with empirical noise inside the function.
case of it is for marginal expected value (average structural function) in IV or control function approach to endogenous explanatory variables #8812.
term "smearing" came from prediction if endog is a nonlinear transformation #2791

We don't need good precision in tails, we only want to compute some expectations like mean of a nonlinear function with full support (no indicator function for tail probabilities).

related
for mixed models with random effects we can use either gauss-hermite quadrature for normal distribution or finite class random effect (where random effect is assumed to be a discrete, multinomial distribution).
Or use halton or other quasi-random numbers for a specified mixture distribution family.

Aquick search found some articles, but I didn't check whether they can be used for this

Barbiero, Alessandro, and Asmerilda Hitaj. “Discrete Approximations of Continuous Probability Distributions Obtained by Minimizing Cramér-von Mises-Type Distances.” Statistical Papers, September 23, 2022. https://doi.org/10.1007/s00362-022-01356-2.

Keefer, Donald L., and Samuel E. Bodily. “Three-Point Approximations for Continuous Random Variables.” Management Science 29, no. 5 (1983): 595–609.

Miller, Allen C., and Thomas R. Rice. “Discrete Approximations of Probability Distributions.” Management Science 29, no. 3 (1983): 352–62.

user: josef-pkt
issue_8817: state: open
title: TST: check and add unit tests
body: - [ ] closes #xxxx
- [ ] tests added / passed. 
- [ ] code/documentation is well formatted.  
- [ ] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

<details>


**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in main and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/main -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>

user: jkriv
issue_8815: state: open
title: Autoreg append expected behaviour when new added samples are less than model lags
body: I am using 

https://www.statsmodels.org/dev/generated/statsmodels.tsa.ar_model.AutoRegResults.append.html#statsmodels.tsa.ar_model.AutoRegResults.append

Based on the documentation:

"Append observations to the ones used to fit the model"

So if I have fitted the model over 200 samples with lag=10, I should be able to add just 1 more sample and make a forecast using the same lag=10 lets say right?

This is what I am doing:

```
model_order = ar_select_order(X, maxlag=lags, ic='aic')
lgs = model_order.ar_lags
model = AutoReg(X, lags = lgs)
model_fit = model.fit()
result = model_fit.forecast(1)
```
Which works.

But if I do this:

`model_fit_new = model_fit.apply(model_fit.forecast(1))
`
or 

`model_fit_new = model_fit.apply(result)
`
I get this error:

`ValueError: ('An exception occured during the creation of the cloned AutoReg instance when applying the existing model specification to the new data. The original traceback appears below.', 'maxlag should be < nobs')`





user: laniakea
issue_8813: state: open
title: ar_model.AutoReg - freq T not understood. Please report if you think this is in error.
body: #### Describe the bug

Setting "seasonal" flag to "True" in AutoReg, creates an error if time series frequency is 15 minutes.

#### Code Sample, a copy-pastable example if possible
```python
model = AutoReg(endog=xtrain['value'], lags=400, trend='c', seasonal=True)
model = model.fit()
```


user: carlosg-m
issue_8812: state: open
title: ENH: add GMMIVCF control function for endogenous explanatory variable
body: getting started with #7689
endogenous regressors in nonlinear models

currently GLM, and only one endogenous regressor

this add `statsmodels.causal` directory
user: josef-pkt
issue_8809: state: open
title: ENH: array api, using computational backend other than numpy
body: Other computational backends provide api similar to numpy, and computation can be dispatched in a compatible way.

This is a possibility to investigate, initially mainly for parts that are outside full models because those contain too much surrounding code that would need to change, e.g. base.data.

maybe that would also work for dask, e.g. #8629

2 sklearn PRs:
https://github.com/scikit-learn/scikit-learn/pull/22554
https://github.com/scikit-learn/scikit-learn/pull/25956

One possibility to strip `base.data` handling is to create a model subclass that does not call `super().__init__`. But it would still need to add non-data initialization like df_xxx.

user: josef-pkt
issue_8808: state: open
title: ENH: more robust norms, and supporting helper functions
body: We could add more robust norms
and helper functions like computing relative efficiency at normal distribution.

Menezes et al has a long list of norms

see also https://github.com/statsmodels/statsmodels/pull/8801#issuecomment-1510076913 and following

Menezes, D. Q. F. de, D. M. Prata, A. R. Secchi, and J. C. Pinto. “A Review on Robust M-Estimators for Regression Analysis.” Computers & Chemical Engineering 147 (April 1, 2021): 107254. https://doi.org/10.1016/j.compchemeng.2021.107254.

I think we don't need many similar norms. I guess they will not make much difference in the estimate.
However, there should be a few that are useful and not too difficult to code.
e.g. R normbase has more, but those require either numerical integration for the rho function or "rational approximation" to it.
Also Menezes et al looks like a good reference for the current norms. (Bughunting would have been easier with having it beforehand.)

aside:
For M-estimation based on moment conditions, we don't actually need the rho function (except that the current default convergence criterion is deviance which uses rho) 

user: josef-pkt
issue_8807: state: open
title: DOC: docstrings in robust.norms, improve, reorganize
body: The docstrings for methods in robust norm have the formulas in the `Returns` section.

Would be better before Parameter section as extended description.
Also converting to `math` will look better if we can get table formatting of equations (align `for z ...` in piecewise functions)


user: josef-pkt
issue_8804: state: open
title: ENH: comparing params or effects across nested nonlinear models, Logit, ...
body: I browsed several articles on this topic but just to get a basic overview

An omitted variable inside a nonlinear mean or inverse link function adds extra noise, so that variance of the true linear predictor increases. The additional noise shifts the mean (Jensen's inequality), similar to issue with control functions.

In binary models like Logit or Probit, the coefficients/params are scaled, i.e. scale dependent. In nested models, extra terms will capture additional random fluctuation, change the underlying scale (variance of linear term), which then implies a different scaling of the parameters.

Several articles show methods to work around this.
The main alternative is to look at "margins", marginal or partial effects instead of looking directly at params.

Two references, but AFAIR I skimmed some more

Williams, Richard, and Abigail Jorgensen. “Comparing Logit & Probit Coefficients between Nested Models.” Social Science Research 109 (January 1, 2023): 102802. https://doi.org/10.1016/j.ssresearch.2022.102802.

Mize, Trenton D., Long Doan, and J. Scott Long. “A General Framework for Comparing Predictions and Marginal Effects across Models.” Sociological Methodology 49, no. 1 (August 1, 2019): 152–89. https://doi.org/10.1177/0081175019852763.

(aside we don't have a generic "post-estimation" label, the main related label is diagnostic.)
user: josef-pkt
issue_8803: state: open
title: ENH: cov_type for nonlinear two-stage models
body: Mainly parking a issue and references. I was just skimming parts.

In treatment effect we use GMM for cov_params which is a robust cov_type.
If we want nonrobust cov_type, then we need to skip some of the robust computations.

Based on my skimming there are different versions of the two-stage cov_params, either nonrobust (correct specification) or robust (sandwiches for some misspecification):

- Murphy-Topel seems to be OPG (*)
- another version uses hessian
- Terza article directly simplifies the computation exploiting the information matrix equality 

I don't know what (our) heckman uses.

Hole, Arne Risa. “Calculating Murphy–Topel Variance Estimates in Stata: A Simplified Procedure.” The Stata Journal 6, no. 4 (November 1, 2006): 521–29. https://doi.org/10.1177/1536867X0600600405.

Palmer, Tom M, Michael V Holmes, Brendan J Keating, and Nuala A Sheehan. “Correcting the Standard Errors of 2-Stage Residual Inclusion Estimators for Mendelian Randomization Studies.” American Journal of Epidemiology 186, no. 9 (November 1, 2017): 1104–14. https://doi.org/10.1093/aje/kwx175.

Terza, Joseph V. “Simpler Standard Errors for Two-Stage Optimization Estimators.” The Stata Journal 16, no. 2 (June 1, 2016): 368–85. https://doi.org/10.1177/1536867X1601600206.

Newey, Whitney K. “A Method of Moments Interpretation of Sequential Estimators.” Economics Letters 14, no. 2 (January 1, 1984): 201–6. https://doi.org/10.1016/0165-1765(84)90083-1.
formula for method of moments, exactly identified GMM, using sandwiches for all parts.

(not clear to me yet what we need)

(*)
**update**
Murphy, Topel, section 5.1 two-step MLE
equ. (29)  assumes and specifies information matrix equality, R_i is name for both.
In the following, they use `R_i`  and so do not specify whether OPG or hessian is used in their final formula equ. (34)
(AFAICS, in equ (33) R still refers to negative hessian, i.e. second derivatives, and omega in equ (30) and (31) is cov(score), i.e. using R for opg.)


 
user: josef-pkt
issue_8802: state: open
title: ENH: specification test, test two non-nested nonlinear models, GLM, discrete 
body: We have specification test for non-nested linear models, but not for nonlinear models like GLM and discrete models.

random read (while taking a break)

Fahrmeir, Ludwig, and Gerhard Tutz. Multivariate Statistical Modelling Based on Generalized Linear Models. Springer Series in Statistics. New York, NY: Springer, 2001. https://doi.org/10.1007/978-1-4757-3454-6.

section 4.3.3. p. 167 Tests for non-nested hypotheses
and especially subsection Generalized Wald and Score Tests.

I need to check references for the details.

maybe
something similar to reset test, adding a linear predictor of an alternative model, either wald or variable addition score test.
(I guess we don't need a two-stage cov_params because under the null the added term is zero.
Theory might be similar to residual inclusion score test for endogeneity)


user: josef-pkt
issue_8796: state: open
title: BUG: Allow string type for groups in NominalGEE
body: The groups argument may be of any type.
user: kshedden
issue_8794: state: open
title: REF/Maint: use pandas to_numpy instead of np.asarray
body: It looks like pandas is by default not dtype compatible with numpy anymore  (if there have been `<nan>` in the dataframe or series.)

asarray by default converts now in many cases to object arrays which are in most cases useless to statsmodels.

current base.data.PandasData uses asarray
```
def _convert_endog_exog(self, endog, exog=None):
        #TODO: remove this when we handle dtype systematically
        endog = np.asarray(endog)
        exog = exog if exog is None else np.asarray(exog)
```

pandas docs are not clear enough to me about what's going on and what to do.
(and I don't have pandas 2.0)

https://hackmd.io/@mntOORP3TCesJJyvg-IdFQ/BJJrTQQMj
sounds like `to_numpy` should work as long as there are no nan in the series, or dataframe

`df.dropna().to_numpy()`  as default ?

dtype conversion (from nullable to regular) will most likely require copying the data, so we don't get views on the underlying pandas data anymore.



user: josef-pkt
issue_8792: state: open
title: TST: check and add unit tests for one parameter models
body: see https://github.com/statsmodels/statsmodels/pull/8780#issuecomment-1507017425

AFAIK, we have almost no unit tests if len(params) is 1.

The smallest exog that I usually use has 2 columns, e.g. for 2 independent sample case.

There might be problems and bugs mainly in generic code, as in the approx_fprime case.

user: josef-pkt
issue_8789: state: open
title: ENH: sandwich cov_type code without symmetry of hessian
body: I never checked whether the generic sandwich code (based on score_obs/momcond and hessian/momcond_deriv is correct for asymmetric hessians.
Do we use the correct transposition of the (inverse) hessian?

I guess there are no unit test for that.

We get asymmetric "hessian" in GMM, conditional moment tests, GEE, RLM , ..., if those are based on moment conditions that do not come from a single objective function (like loglike).
e.g. M-estimator that start with moment conditions and not we a single objective function. (nonlinear robust model with separately specified moment condition for variance) 
Also two-stage models have asymmetric derivatives of score_obs, Murphy/Topel and general versions of it (*)

Current GMM does not yet reuse the generic cov_type code. 
Current conditional moment tests (CMT) also don't use generic cov_type code.
score_test uses cov_params and symmetric hessian based on the unrestricted loglikelihood but the cov of the null hypothesis restrictions is based on the CMT code. 


(*)
Zero-inflated models are not two-stage model, there is one loglikelihood function for both components.
Hurdle model is currently implemented with two separate models, but parameters are independent across submodels, hessian and cov(score) are block-diagonal.

user: josef-pkt
issue_8788: state: open
title: GLM fittedvalues error
body: Hi, i'm having error when using binomial GLM. I fit it with "newton" method. The training is done but returning this warning
`divide by zero encountered in double_scalars`

Then I want to access `model.fittedvalues` however it returns error

ValueError                                Traceback (most recent call last)
Cell In[27], line 1
----> 1 model.fittedvalues

File [~/lib/python3.8/site-packages/statsmodels/base/wrapper.py:40), in ResultsWrapper.__getattribute__(self, attr)
     38     obj = data.wrap_output(obj, how[0], *how[1:])
     39 elif how:
---> 40     obj = data.wrap_output(obj, how=how)
     42 return obj

File [~/lib/python3.8/site-packages/statsmodels/base/data.py:442), in ModelData.wrap_output(self, obj, how, names)
    440     return self.attach_columns(obj)
    441 elif how == 'rows':
--> 442     return self.attach_rows(obj)
    443 elif how == 'cov':
    444     return self.attach_cov(obj)

File [~/lib/python3.8/site-packages/statsmodels/base/data.py:610), in PandasData.attach_rows(self, result)
    608 else:
    609     out = DataFrame(result)
--> 610     out.columns = self.ynames
    611 out.index = self.row_labels[-len(result):]
    612 return out

File [~/lib/python3.8/site-packages/pandas/core/generic.py:5915), in NDFrame.__setattr__(self, name, value)
   5913 try:
   5914     object.__getattribute__(self, name)
-> 5915     return object.__setattr__(self, name, value)
   5916 except AttributeError:
   5917     pass

File [~/lib/python3.8/site-packages/pandas/_libs/properties.pyx:69), in pandas._libs.properties.AxisProperty.__set__()

File [~/lib/python3.8/site-packages/pandas/core/generic.py:823), in NDFrame._set_axis(self, axis, labels)
    821 def _set_axis(self, axis: int, labels: AnyArrayLike | list) -> None:
    822     labels = ensure_index(labels)
--> 823     self._mgr.set_axis(axis, labels)
    824     self._clear_item_cache()

File [~/lib/python3.8/site-packages/pandas/core/internals/managers.py:230), in BaseBlockManager.set_axis(self, axis, new_labels)
    228 def set_axis(self, axis: int, new_labels: Index) -> None:
    229     # Caller is responsible for ensuring we have an Index object.
--> 230     self._validate_set_axis(axis, new_labels)
    231     self.axes[axis] = new_labels

File [~/lib/python3.8/site-packages/pandas/core/internals/base.py:70), in DataManager._validate_set_axis(self, axis, new_labels)
     67     pass
     69 elif new_len != old_len:
---> 70     raise ValueError(
     71         f"Length mismatch: Expected axis has {old_len} elements, new "
     72         f"values have {new_len} elements"
     73     )

ValueError: Length mismatch: Expected axis has 1 elements, new values have 2 elements

Is this error expected? Kindly need your help, Thank you!
user: ardiqbal
issue_8787: state: open
title: get_hat_matrix_diag() returns NaN when weights contain negative values due to np.sqrt on GLM/GAM
body: Thanks for your contribution for this great library.

Due to my knowledge limitation, I need some clarification from experts about below situation that I'm facing. If this is intentional behavior, it would be thankful if you close this issue.

It seems that GLM/GAM both are using `get_hat_matrix_diag` to calculate DoF, etc.(many other values in summary return). And I see `np.sqrt(weights)` applied to `self.model.exog` in it. The thing is, if `weights` have negative values, `hd`(hat matrix diag) become `NaN` and many other values like DoF getting the same. Do you think if I can use `np.sqrt(np.abs(weights))` or something like that to avoid `NaN` and getting appropriate hat matrix diagonal?

Thank you in advance.

**GLM**(https://www.statsmodels.org/dev/_modules/statsmodels/genmod/generalized_linear_model.html#GLMResults)
```
def get_hat_matrix_diag(self, observed=True):
    weights = self.model.hessian_factor(self.params, observed=observed)
    print(weights)
    wexog = np.sqrt(weights)[:, None] * self.model.exog

    hd = (wexog * np.linalg.pinv(wexog).T).sum(1)
    return hd
```

**GAM**(https://www.statsmodels.org/dev/_modules/statsmodels/gam/generalized_additive_model.html#GLMGamResults)
```
def get_hat_matrix_diag(self, observed=True, _axis=1):
    weights = self.model.hessian_factor(self.params, scale=self.scale,
                                        observed=observed)
    print(weights)
    wexog = np.sqrt(weights)[:, None] * self.model.exog

    # we can use inverse hessian directly instead of computing it from
    # WLS/IRLS as in GLM

    # TODO: does `normalized_cov_params * scale` work in all cases?
    # this avoids recomputing hessian, check when used for other models.
    hess_inv = self.normalized_cov_params * self.scale
    # this is in GLM equivalent to the more generic and direct
    # hess_inv = np.linalg.inv(-self.model.hessian(self.params))
    hd = (wexog * hess_inv.dot(wexog.T).T).sum(axis=_axis)
    return hd
```

**Summary example when `weights` have negative values.**
```
                 Generalized Linear Model Regression Results
==============================================================================
Dep. Variable:             Q('*1a78')   No. Observations:                 1000
Model:                         GLMGam   Df Residuals:                      NaN
Model Family:                Gaussian   Df Model:                          NaN
Link Function:                    Log   Scale:                             nan
Method:                         PIRLS   Log-Likelihood:                    nan
Date:                Tue, 11 Apr 2023   Deviance:                   3.3155e+07
Time:                        15:12:33   Pearson chi2:                 3.32e+07
No. Iterations:                    27   Pseudo R-squ. (CS):                nan
Covariance Type:            nonrobust
=====================================================================================
                        coef    std err          z      P>|z|      [0.025      0.975]
-------------------------------------------------------------------------------------
Intercept             3.7317        nan        nan        nan         nan         nan
scale(Q('*ed2e'))     0.3209        nan        nan        nan         nan         nan
scale(Q('*9811'))    -0.1276        nan        nan        nan         nan         nan
*9811_s0              4.2754        nan        nan        nan         nan         nan
*9811_s1             -0.2659        nan        nan        nan         nan         nan
=====================================================================================
```
user: su79eu7k
issue_8786: state: open
title: ENH: Alternative Kernels in HAC estimation
body: Actually, RegressionResults 'kernel' only allows for Bartlett and Uniform kernel weighing functions in the context of robust linear regression, which are taken from "statsmodels.stats.sandwich_covariance.cov_hac". 

It's pretty clear from the literature (authors such as Kiefer, Voegelsang , Sun, Philips, etc.) that kernel selection has an important impact on results. 

Would it be complicated to include additional kernel functions, such as the ones that are present in MATLAB (https://www.mathworks.com/help/econ/hac.html) in statsmodels?

I am actually working on the subject so I wouldn't mind doing it.
user: itsNH98
issue_8785: state: open
title: ENH: generic helper method for in-sample average effect with moment correlation
body: main related issue #7168
this issue is for an implementation for parts of it

similar to margeff and get_prediction

If we want to compute an in-sample average effect, then we should include the correlation of the individual effect with the moment conditions, score_obs,, of the model.

We do this in treatment effect through GMM, but we can add this more generically for individual models.
Another case where we already have this is test_chisquare_prob in count diagnostic. Currently, that uses only OPG instead of derivatives of moment conditions.

Proposal is a new helper function similar to get_margeff and `get_prediction(..., average=True, agg_weights=...)`, but instead of just using the delta method, we take correlation of effect and score_obs into account.
We could add it as an option to the currrent get_margeff, get_prediction, ... methods, but even then we still need the generic helper function to support it.

This is only for in-sample or a subsample of it (sample with weights in {0, 1}), so the out of sample part of get_prediction would not apply.

As a complication: cov_type
AFAIU, we would have to go through the sandwich robust cov_type computation for the joined moment conditions (effect, score_obs)
based on what cov_type was specified in `fit`.
This will be easy to add, if we have the concatenated hessian, current cov_type allow generic (score_obs, hessian) as argument.
Current test_chisquare_prob does not use the cov_type, only OPG, i.e. a nonrobust cov_type, is available.
Delta method as in current get_margeff and get_prediction always uses the cov_params which is based on the selected cov_type.

several related issues for pieces of this and for specific cases for which we could reuse this
#8767
#8745 
#8421
...

I think we will need to support more than one average effect, a 1-dim multivariate effect for each observation, so we get the joint covariance of those effects and can run hypotheses test on them. (similar to ATE, POM1, POM2)

Aside: we still need more generic joint effects, joint covariance for wald tests and for get_prediction.
We still don't have separate helper methods for testing parameter restrictions given only params and cov_params (and use_t, df_resid), It's build into LikelihoodModelResults (cov_params and wald test methods). ContrastResults is separate and can be reused.


user: josef-pkt
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_8784: state: open
title: ENH: out-of-sample prediction, forecast comparison, multiplicity correction (paired t-test)
body: **edit** I found an earlier issue #2291 closing that as duplicate with less information)

In a twitter message, a user of statsmodels mentioned using tukey-hsd to compare models on test data (out-of-sample)

tukey-hsd is inappropriate for this because it assumes samples are independent.


related #8216 suest, when we take into account that models are estimated on the same data, and compare in-sample statistics.

AFAICS
The Diebold-Mariano forecast comparison test, tests the difference of prediction errors. This would be similar to a "paired" t-test (for two correlated samples), while tukey-hsd is based on standard independent sample t-test.

So one possibilty might be to use a paired t-test and correct for multiple testing using one of the p-value correction methods.

I never looked closely at Diebold-Mariano or similar forecast comparison tests. 

An updated perspective on the DM test:

Diebold, Francis X. “Comparing Predictive Accuracy, Twenty Years Later: A Personal Perspective on the Use and Abuse of Diebold–Mariano Tests.” Journal of Business & Economic Statistics 33, no. 1 (January 2, 2015): 1–1. https://doi.org/10.1080/07350015.2014.983236.


pairwise paired t-test sound like a useful function for statsmodels. 
It does not specify and estimate the correlation, so parametric p-value correction as in tukey-hsd will not be available. The p-value correction (for I guess positive correlated samples) should still be valid.


One difference between time series and cross-section data should be that in the latter case we do not have to worry about serial correlation, it's assumed away. We don't need HAC or similar robust standard errors for serially correlated predictions or prediction errors.

aside: quick look at articles citing Diebold 2015
intersection-union test for multiple forecast comparisons
Spreng, Lars, and Giovanni Urga. “Combining P-Values for Multivariate Predictive Ability Testing.” Journal of Business & Economic Statistics 0, no. 0 (April 19, 2022): 1–13. https://doi.org/10.1080/07350015.2022.2067545.


user: josef-pkt
issue_8781: state: open
title: BUG: linear RegressionModel.get_distribution is only for OLS, no weights/heteroscedasticity
body: AFAICS, looking at the code.

for WLS we should interpret weights as var_weights and use heteroscedastic scale in the distribution.
I have not yet thought about the GLS case, maybe NotImplementedError. We don't have "anti-whiten" or coloring for generic GLS

Not tried out yet.
user: josef-pkt
issue_8773: state: open
title: ENH: implementation multi-equation GMM with missing parts, sample selection
body: currently GMM requires that we have moment conditions for all observations.

How can we handle the case when one equation uses only a subsample of a second equation, e.g. as in sample selection models?

specific case: what's the GMM equivalent of heckman sample selection model and how can we fit into GMM implementation?

We could aggregate the individual moment equations over different subsamples, but we also need the cov(mom_cond) for which we don't have a complete sample.

random thought in context of #8745 

more general,
How can we use GMM if we want to model missing not at random or not completely at random, i.e. missingness is endogenous.?


user: josef-pkt
issue_8772: state: open
title: ENH: GAM Mixin, penalized splines for other models than GLM
body: I guess it would not be too difficult to split out penalized splines from GAM so that it can be used with other models that are not in the GLM families.

GAM already uses the penalization mixin, so we mainly need the spline and penalization parts for the model and the extra post-estimation methods for the results classes.

#7128


application could be where we want to add a spline as extra term
e.g. control functions in #8745 to make the statistics more semi-parametric

but initial candidates would be the count models in discrete. That needs adjustment to GAM for the extra params (NBP, GPP or multipart models like hurdle or zero-inflated.)
user: josef-pkt
issue_8771: state: open
title: ENH: recursive bivariate binary choice model, bivariate Probit and Copulas
body: similar to #8765 but when "treatment" and outcome are binary.

recursive, triangular
y1 = 1(α y2 + x1 β + ε1 ≥ 0)
y2 = 1(x2 β2 + ε2 ≥ 0)
ε1, ε2 are correlated

full, parametric MLE but with copulas, so we have flexibility in correlation and CDF-link

bivariate Probit is in Greene

two articles for using copulas

Hasebe, Takuya. “Marginal Effects of a Bivariate Binary Choice Model.” Economics Letters 121, no. 2 (November 1, 2013): 298–301. https://doi.org/10.1016/j.econlet.2013.08.028.

Winkelmann, Rainer. “Copula Bivariate Probit Models: With an Application to Medical Expenditures.” Health Economics 21, no. 12 (2012): 1444–55. https://doi.org/10.1002/hec.1801.


This looks like the simplest case to get started with copulas for discrete random variables, especially in the regression model context.


**update**

an article that shows identification in this model

Han, Sukjin, and Edward J. Vytlacil. “Identification in a Generalization of Bivariate Probit Models with Dummy Endogenous Regressors.” Journal of Econometrics 199, no. 1 (July 1, 2017): 63–73. https://doi.org/10.1016/j.jeconom.2017.04.001.


user: josef-pkt
issue_8770: state: open
title: ARIMA forecasting backtest
body: Hello everyone,

I'm new to time series analysis. I'm currently trying to analyze a price series with ARMA and ARIMA (trying to reproduce the result of "Machine Trading" from Ernest Chan). The goal is to extract the parameters from fitted models and use those parameters in C++ to predict future mid points. In the example below, I tried to use the parameters from the fitted ARMA/ARIMA model to do some predictions on the test set, but I don't know how to compute the prediction manually. I trained a ARIMA(2, 0, 5) model and extracted the parameters from it. However, I cannot reproduce the values came from `forecast()` and `predict()`.

![image](https://user-images.githubusercontent.com/4936182/229750322-478be33f-9622-4277-8987-64743eb97c82.png)


Here's my code
```
# Split the data into training and testing data 
# mid_points is the array containing historical mid points
train_data: npt.NDArray = mid_points[len(mid_points)-252*(24*60-15) * 2:len(mid_points)-252*(24*60-15)] 
test_data: npt.NDArray = mid_points[len(mid_points)-252*(24*60-15):]

# Train models
model = ARIMA(train_data, order=(2, 0, 5), trend="c")
fit_result = model.fit()

# One-step prediction (forecast, predict)
print(fit_result.predict(start=len(train_data), end=len(train_data), dynamic=True))
print(fit_result.forecast(1))
# [0.92627827]
# [0.92627827]

# One-step prediction (apply)
new_res = fit_result.apply(mid_points[len(train_data) - best_q:len(train_data)])
new_res.forecast(1)
# array([0.97994715])

# One-step prediction (manual)
intercept = fit_result.params[0] * (1 - np.sum(fit_result.arparams))
ar_part = np.dot(fit_result.arparams, mid_points[len(train_data) - 2 : len(train_data)][::-1])
ma_part = np.dot(fit_result.maparams, fit_result.forecasts_error[0][-len(fit_result.maparams) :][::-1])
result = intercept + ar_part + ma_part
# result = 0.9800231921389104
```

As you can see in the above code, the value of manual prediction is quite different from `forecast()` and `predict()`. I have checked [link1](https://machinelearningmastery.com/make-manual-predictions-arima-models-python/), [link2](https://stackoverflow.com/questions/63383393/why-statsmodels-arima1-0-0-is-not-equivalent-to-autoreg1) , #8535 and traced old `_get_predict_out_of_sample()`, but I still failed to reproduce the result. Is there any code file that I can trace to reproduce the result from ARMA and ARIMA?
user: jinbo-huang
issue_8769: state: open
title: ENH: (cluster) robust standard errors in treatment effects ATE
body: (random thought. I didn't look at any details.)

Can we add cov_type choices to TreatmentEffect estimation?
I would be available through the GMM options. 

e.g. cluster robust for cluster randomized sampling
in analogy to clustered survey sampling

quick search:

Abadie, Alberto, Susan Athey, Guido W Imbens, and Jeffrey M Wooldridge. “When Should You Adjust Standard Errors for Clustering?*.” The Quarterly Journal of Economics 138, no. 1 (February 1, 2023): 1–35. https://doi.org/10.1093/qje/qjac038.

(surprise: The supporting code is in Python using sklearn)

and google scholar finds articles on cluster randomized trials.

user: josef-pkt
issue_8768: state: open
title: DOC: the two notebook examples for Linear Mixed Models have been deleted
body: #### Describe the bug

The following documentation page https://github.com/statsmodels/statsmodels/wiki/Examples#linear-mixed-models mentions two notebooks as examples for using Linear Mixed Models in statsmodels:


> -  Two examples using both R (LME4) and Statsmodels [HTML View] (http://nbviewer.ipython.org/urls/umich.box.com/shared/static/6tfc1e0q6jincsv5pgfa.ipynb)
> - Regression analysis of healthcare spending in Vietnam, using mixed models and GEE. Stata results provided for comparison. [HTML View] (http://nbviewer.ipython.org/urls/umich.box.com/shared/static/lc6uf6dmabmitjbup3yt.ipynb)

However, the two links point to 403 error pages. I tested it in Firefox and Chrome. Here's a screenshot of one of these error messages:

![image](https://user-images.githubusercontent.com/5843240/229570909-73ed7835-3a1a-4905-83a4-ac930d78291b.png)

After investigating a bit, and following the URLs wrapped in these nbviewer.org links, it seems that the notebooks have simply been deleted, see https://umich.app.box.com/public/static/lc6uf6dmabmitjbup3yt.ipynb and https://umich.app.box.com/public/static/6tfc1e0q6jincsv5pgfa.ipynb.

#### Expected Solution

Either updating the links if they have been moved to somewhere else (I don't know if it's the case), or making the links available again, or simply deleting this section of the documentation.

Thanks,


user: jeanbaptisteb
issue_8767: state: open
title: ENH/design: variance of average effect with extra term, random or fixed exog
body: 
When we have an average nonlinear effect  g_bar = average(g_i)  where g_i = g_i(params, exog) is a nonlinear function, then we compute the variance of g_bar using the delta method. 

it is possible to add the sample variance of g_i as an extra variance term to the variance of g_bar.

I just saw this in Liu et al https://github.com/statsmodels/statsmodels/issues/8762#issuecomment-1488877320 #7193
and I was puzzled about their variance for average effects\.

some articles on background

Greene, Dowd, Norton use only the delta method term, working conditional on the exog in the sample (similar to fixed exog bootstrap)
Terza argues in favor of including the extra sample variance of the effect with the interpretation that it comes from random sampling of exog from a population distribution.

Greene, William H., Bryan E. Dowd, and Edward C. Norton. “Response to ‘Inference Using Sample Means of Parametric Nonlinear Data Transformations.’” Health Services Research 51, no. 3 (2016): 1114–16. https://doi.org/10.1111/1475-6773.12493.

Terza, Joseph V. “Inference Using Sample Means of Parametric Nonlinear Data Transformations.” Health Services Research 51, no. 3 (June 2016): 1109–13. https://doi.org/10.1111/1475-6773.12494.

Dowd, Bryan E., William H. Greene, and Edward C. Norton. “Computation of Standard Errors.” Health Services Research 49, no. 2 (2014): 731–50. https://doi.org/10.1111/1475-6773.12122.

I think so far we only include the delta method term (following Stata and other packages)

I'm not sure how GMM in treatment effect for ATE, POM is treating this. We still use a sandwich covariance that includes the variance of the moment conditions (including definition of ate and pom as moment condition).
(This might be some explanation for what puzzled me in treatment effect when some things were different in my and Stata's GMM implementation of it that did not match up with what I had expected when I checked other ways to compute it. I didn't investigate, I just wanted to match Stata results.)


maybe partially related:
meta-analysis: fixed scale versus WLS scale, usually dressed up as fixed effects versus random effects.

Similar issue: 
Fixed design matrix or population distribution of exog in power and sample size computation.
So far I have mainly looked at fixed sample design (except for nobs or nobs-ratios). In the regression setting I have now seen also the case where randomness of sample exog based on a population distribution of exog is taken into account when computing (unconditional) power.

main question: Do we need to add an option for including the second term?
e.g. 
get_prediction has an aggregate option but only includes delta method term. same for margins, AFAIU.
extra average effects that we will want to include as post-estimation result, e.g. risk ratio, risk difference in binary models

and document appropriately which version is implicitly or explicitly used.
e.g. what does (nonparametrically identified) causal effect literature use?


 


user: josef-pkt
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_8765: state: open
title: ENH: sample selection with copulas
body: This sounds like another interesting and useful application to get started with copula modelling
related #8032 sample selection without gaussian assumption

Smith, Murray D. “Modelling Sample Selection Using Archimedean Copulas.” The Econometrics Journal 6, no. 1 (June 1, 2003): 99–123. https://doi.org/10.1111/1368-423X.00101.

Liu, Ruixuan, and Zhengfei Yu. “Sample Selection Models with Monotone Control Functions.” Journal of Econometrics 226, no. 2 (February 1, 2022): 321–42. https://doi.org/10.1016/j.jeconom.2021.01.010.

Wojtyś, Małgorzata, Giampiero Marra, and Rosalba Radice. “Copula Based Generalized Additive Models for Location, Scale and Shape with Non-Random Sample Selection.” Computational Statistics & Data Analysis 127 (November 1, 2018): 1–14. https://doi.org/10.1016/j.csda.2018.05.001.


aside:
Sample selection models (heckman and similar) have correlated unobserved heterogeneity in the linear predictor, latent index function.
The marginal distribution is then not e.g. Poisson anymore, it's a poisson mixture with overdispersion.

The standard bivariate copula model specifies the marginal distribution and correlation directly, e.g. the marginal distribution would still be Poisson. 
However, we are interested in the conditional and marginal mean which would still need to depend on and vary with the correlation.

(It's not clear to me whether the second version can be used to model endogenous explanatory variables, i.e. distribution parameter of the second conditional distribution depends directly on the first.
Sounds more like as if we want to model the reduced form, i.e. copula/correlation determines distribution conditional on the first random variable, and regression model includes only exogenous variables.
)

tools: need to finish code for conditional distribution in copulas and copula distributions, pdf(y1 | y0) and cdf(y1 | y0)

user: josef-pkt
issue_8764: state: open
title: I did this code and this error occurs after running it
body: ```
import pandas as pd
import numpy as np
import statsmodels.api as sm
from statsmodels.tsa.arima.model import ARIMA
from sklearn.neural_network import MLPRegressor
from sklearn.model_selection import TimeSeriesSplit
from sklearn.base import BaseEstimator, RegressorMixin
import optuna

# Load your dataset from an Excel file
data = pd.read_excel(r"C:\Users\Keller\Desktop\captura dados python\dados.xlsx", engine='openpyxl')

# Keep only the first x rows of the dataset
data = data.head(1000)

# Define the objective function for Optuna to minimize
def objective(trial):
    order = (trial.suggest_int('p', 1, 4), 1, 0)
    seasonal_order = ((trial.suggest_int('P', 1, 4), 0, 0, 12))
    model = ARIMA(endog=data['Fechamento'], order=order, seasonal_order=seasonal_order)
    tscv = TimeSeriesSplit(n_splits=10)
    score = np.mean([sm.tools.eval_measures.rmse(actual, model.fit(train).forecast(len(actual))) for train, actual in tscv.split(data['Fechamento'])])
    return score

# Set the number of trials for Optuna to run
n_trials = 10

# Run the Optuna search to find the best hyperparameters
study = optuna.create_study(direction='minimize')
study.optimize(objective, n_trials=n_trials)

# Get the best model from the Optuna search and fit it to the data
params = study.best_params
order = (params['p'], 1, 0)
seasonal_order = ((params['P'], 0, 0, 12))
model = ARIMA(endog=data['Fechamento'], order=order, seasonal_order=seasonal_order)
model.fit(data['Fechamento'])

# Specify the number of days you want to forecast into the future
future_days = 5

# Create a future dataframe with the specified number of days
future_dates = pd.date_range(start=data.index[-1], periods=future_days + 1, freq='D')[1:]
future = pd.DataFrame(index=future_dates)

# Define a function to predict the next n values using MLP regression
def predict_next_n(data, n):
    X = np.arange(len(data)).reshape(-1, 1)
    y = data.values
    model = MLPRegressor(hidden_layer_sizes=(100,), max_iter=1000)
    model.fit(X[-10:], y[-10:])
    next_n = np.arange(len(data), len(data) + n).reshape(-1, 1)
    return model.predict(next_n)

# Generate predictions for the future dates
future['Fechamento'] = model.forecast(future_days)
future['Amplitude'] = pd.Series(predict_next_n(data['Amplitude'], future_days), index=future.index)
future['Media200D'] = pd.Series(predict_next_n(data['Media200D'], future_days), index=future.index)
future['Media200S'] = pd.Series(predict_next_n(data['Media200S'], future_days), index=future.index)

# Save the forecast to an Excel file
future.to_excel(r"C:\Users\Keller\Desktop\captura dados python\futuro.xlsx", engine='openpyxl')

print(f"Forecast for {future_days} days saved to 'futuro.xlsx'")
```

the error below

[32m[I 2023-04-01 21:59:50,473][0m A new study created in memory with name: no-name-a965a999-b5ef-4c4f-9d19-5ac2167ef3e0[0m

Warning (from warnings module):
  File "C:\Program Files\Python311\Lib\site-packages\statsmodels\tsa\statespace\tools.py", line 536
    y[k-1, i] = (y[k, i] - y[k, k]*y[k, k-i-1]) / (1 - y[k, k]**2)
RuntimeWarning: invalid value encountered in scalar divide

Warning (from warnings module):
  File "C:\Program Files\Python311\Lib\site-packages\statsmodels\tsa\statespace\tools.py", line 538
    x = r / ((1 - r**2)**0.5)
RuntimeWarning: divide by zero encountered in divide

Warning (from warnings module):
  File "C:\Program Files\Python311\Lib\site-packages\statsmodels\tsa\statespace\tools.py", line 538
    x = r / ((1 - r**2)**0.5)
RuntimeWarning: invalid value encountered in sqrt

Warning (from warnings module):
  File "C:\Program Files\Python311\Lib\site-packages\statsmodels\tsa\statespace\tools.py", line 497
    r = unconstrained/((1 + unconstrained**2)**0.5)
RuntimeWarning: invalid value encountered in divide
[33m[W 2023-04-01 21:59:50,699][0m Trial 0 failed with parameters: {'p': 2, 'P': 2} because of the following error: LinAlgError('Schur decomposition solver error.').[0m
Traceback (most recent call last):
  File "C:\Program Files\Python311\Lib\site-packages\optuna\study\_optimize.py", line 200, in _run_trial
    value_or_values = func(trial)
                      ^^^^^^^^^^^
  File "C:/Users/Keller/Desktop/captura dados python/Calculo_Preditivo.py", line 22, in objective
    score = np.mean([sm.tools.eval_measures.rmse(actual, model.fit(train).forecast(len(actual))) for train, actual in tscv.split(data['Fechamento'])])
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:/Users/Keller/Desktop/captura dados python/Calculo_Preditivo.py", line 22, in <listcomp>
    score = np.mean([sm.tools.eval_measures.rmse(actual, model.fit(train).forecast(len(actual))) for train, actual in tscv.split(data['Fechamento'])])
                                                         ^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\statsmodels\tsa\arima\model.py", line 390, in fit
    res = super().fit(
          ^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\statsmodels\tsa\statespace\mlemodel.py", line 704, in fit
    mlefit = super(MLEModel, self).fit(start_params, method=method,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\statsmodels\base\model.py", line 563, in fit
    xopt, retvals, optim_settings = optimizer._fit(f, score, start_params,
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\statsmodels\base\optimizer.py", line 241, in _fit
    xopt, retvals = func(objective, gradient, start_params, fargs, kwargs,
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\statsmodels\base\optimizer.py", line 651, in _fit_lbfgs
    retvals = optimize.fmin_l_bfgs_b(func, start_params, maxiter=maxiter,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\scipy\optimize\_lbfgsb_py.py", line 197, in fmin_l_bfgs_b
    res = _minimize_lbfgsb(fun, x0, args=args, jac=jac, bounds=bounds,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\scipy\optimize\_lbfgsb_py.py", line 305, in _minimize_lbfgsb
    sf = _prepare_scalar_function(fun, x0, jac=jac, args=args, epsilon=eps,
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\scipy\optimize\_optimize.py", line 332, in _prepare_scalar_function
    sf = ScalarFunction(fun, x0, args, grad, hess,
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\scipy\optimize\_differentiable_functions.py", line 158, in __init__
    self._update_fun()
  File "C:\Program Files\Python311\Lib\site-packages\scipy\optimize\_differentiable_functions.py", line 251, in _update_fun
    self._update_fun_impl()
  File "C:\Program Files\Python311\Lib\site-packages\scipy\optimize\_differentiable_functions.py", line 155, in update_fun
    self.f = fun_wrapped(self.x)
             ^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\scipy\optimize\_differentiable_functions.py", line 137, in fun_wrapped
    fx = fun(np.copy(x), *args)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\statsmodels\base\model.py", line 531, in f
    return -self.loglike(params, *args) / nobs
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\statsmodels\tsa\statespace\mlemodel.py", line 939, in loglike
    loglike = self.ssm.loglike(complex_step=complex_step, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\statsmodels\tsa\statespace\kalman_filter.py", line 983, in loglike
    kfilter = self._filter(**kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\site-packages\statsmodels\tsa\statespace\kalman_filter.py", line 903, in _filter
    self._initialize_state(prefix=prefix, complex_step=complex_step)
  File "C:\Program Files\Python311\Lib\site-packages\statsmodels\tsa\statespace\representation.py", line 983, in _initialize_state
    self._statespaces[prefix].initialize(self.initialization,
  File "statsmodels\tsa\statespace\_representation.pyx", line 1373, in statsmodels.tsa.statespace._representation.dStatespace.initialize
  File "statsmodels\tsa\statespace\_representation.pyx", line 1362, in statsmodels.tsa.statespace._representation.dStatespace.initialize
  File "statsmodels\tsa\statespace\_initialization.pyx", line 288, in statsmodels.tsa.statespace._initialization.dInitialization.initialize
  File "statsmodels\tsa\statespace\_initialization.pyx", line 406, in statsmodels.tsa.statespace._initialization.dInitialization.initialize_stationary_stationary_cov
  File "statsmodels\tsa\statespace\_tools.pyx", line 1284, in statsmodels.tsa.statespace._tools._dsolve_discrete_lyapunov
numpy.linalg.LinAlgError: Schur decomposition solver error.
[33m[W 2023-04-01 21:59:50,721][0m Trial 0 failed with value None.[0m
Traceback (most recent call last):
  File "C:/Users/Keller/Desktop/captura dados python/Calculo_Preditivo.py", line 30, in <module>
    study.optimize(objective, n_trials=n_trials)
  File "C:\Program Files\Python311\Lib\site-packages\optuna\study\study.py", line 425, in optimize
    _optimize(
  File "C:\Program Files\Python311\Lib\site-packages\optuna\study\_optimize.py", line 66, in _optimize
    _optimize_sequential(
  File "C:\Program Files\Python311\Lib\site-packages\optuna\study\_optimize.py", line 163, in _optimize_sequential
    frozen_trial = _run_trial(study, func, catch)
  File "C:\Program Files\Python311\Lib\site-packages\optuna\study\_optimize.py", line 251, in _run_trial
    raise func_err
  File "C:\Program Files\Python311\Lib\site-packages\optuna\study\_optimize.py", line 200, in _run_trial
    value_or_values = func(trial)
  File "C:/Users/Keller/Desktop/captura dados python/Calculo_Preditivo.py", line 22, in objective
    score = np.mean([sm.tools.eval_measures.rmse(actual, model.fit(train).forecast(len(actual))) for train, actual in tscv.split(data['Fechamento'])])
  File "C:/Users/Keller/Desktop/captura dados python/Calculo_Preditivo.py", line 22, in <listcomp>
    score = np.mean([sm.tools.eval_measures.rmse(actual, model.fit(train).forecast(len(actual))) for train, actual in tscv.split(data['Fechamento'])])
  File "C:\Program Files\Python311\Lib\site-packages\statsmodels\tsa\arima\model.py", line 390, in fit
    res = super().fit(
  File "C:\Program Files\Python311\Lib\site-packages\statsmodels\tsa\statespace\mlemodel.py", line 704, in fit
    mlefit = super(MLEModel, self).fit(start_params, method=method,
  File "C:\Program Files\Python311\Lib\site-packages\statsmodels\base\model.py", line 563, in fit
    xopt, retvals, optim_settings = optimizer._fit(f, score, start_params,
  File "C:\Program Files\Python311\Lib\site-packages\statsmodels\base\optimizer.py", line 241, in _fit
    xopt, retvals = func(objective, gradient, start_params, fargs, kwargs,
  File "C:\Program Files\Python311\Lib\site-packages\statsmodels\base\optimizer.py", line 651, in _fit_lbfgs
    retvals = optimize.fmin_l_bfgs_b(func, start_params, maxiter=maxiter,
  File "C:\Program Files\Python311\Lib\site-packages\scipy\optimize\_lbfgsb_py.py", line 197, in fmin_l_bfgs_b
    res = _minimize_lbfgsb(fun, x0, args=args, jac=jac, bounds=bounds,
  File "C:\Program Files\Python311\Lib\site-packages\scipy\optimize\_lbfgsb_py.py", line 305, in _minimize_lbfgsb
    sf = _prepare_scalar_function(fun, x0, jac=jac, args=args, epsilon=eps,
  File "C:\Program Files\Python311\Lib\site-packages\scipy\optimize\_optimize.py", line 332, in _prepare_scalar_function
    sf = ScalarFunction(fun, x0, args, grad, hess,
  File "C:\Program Files\Python311\Lib\site-packages\scipy\optimize\_differentiable_functions.py", line 158, in __init__
    self._update_fun()
  File "C:\Program Files\Python311\Lib\site-packages\scipy\optimize\_differentiable_functions.py", line 251, in _update_fun
    self._update_fun_impl()
  File "C:\Program Files\Python311\Lib\site-packages\scipy\optimize\_differentiable_functions.py", line 155, in update_fun
    self.f = fun_wrapped(self.x)
  File "C:\Program Files\Python311\Lib\site-packages\scipy\optimize\_differentiable_functions.py", line 137, in fun_wrapped
    fx = fun(np.copy(x), *args)
  File "C:\Program Files\Python311\Lib\site-packages\statsmodels\base\model.py", line 531, in f
    return -self.loglike(params, *args) / nobs
  File "C:\Program Files\Python311\Lib\site-packages\statsmodels\tsa\statespace\mlemodel.py", line 939, in loglike
    loglike = self.ssm.loglike(complex_step=complex_step, **kwargs)
  File "C:\Program Files\Python311\Lib\site-packages\statsmodels\tsa\statespace\kalman_filter.py", line 983, in loglike
    kfilter = self._filter(**kwargs)
  File "C:\Program Files\Python311\Lib\site-packages\statsmodels\tsa\statespace\kalman_filter.py", line 903, in _filter
    self._initialize_state(prefix=prefix, complex_step=complex_step)
  File "C:\Program Files\Python311\Lib\site-packages\statsmodels\tsa\statespace\representation.py", line 983, in _initialize_state
    self._statespaces[prefix].initialize(self.initialization,
  File "statsmodels\tsa\statespace\_representation.pyx", line 1373, in statsmodels.tsa.statespace._representation.dStatespace.initialize
  File "statsmodels\tsa\statespace\_representation.pyx", line 1362, in statsmodels.tsa.statespace._representation.dStatespace.initialize
  File "statsmodels\tsa\statespace\_initialization.pyx", line 288, in statsmodels.tsa.statespace._initialization.dInitialization.initialize
  File "statsmodels\tsa\statespace\_initialization.pyx", line 406, in statsmodels.tsa.statespace._initialization.dInitialization.initialize_stationary_stationary_cov
  File "statsmodels\tsa\statespace\_tools.pyx", line 1284, in statsmodels.tsa.statespace._tools._dsolve_discrete_lyapunov
numpy.linalg.LinAlgError: Schur decomposition solver error.


user: akeller1992
issue_8763: state: open
title: Question: ADF test regression parameter
body: I want to perform ADF test on my time-series data and it provides 4 options for the regression parameter:
1) c: constant only (default)
2) ct: constant and trend
3) ctt: constant, and linear and quadratic trend
4) n: no constant, no trend

Which option I should choose if the trend of my data looks like this?
![IMG20230331120832](https://user-images.githubusercontent.com/72982917/229020505-2dca4f42-f6ee-4ca9-87eb-23ecf160f664.jpg)




user: chinshanhong
issue_8762: state: open
title: ENH: predicted mean, conditional expectation in nonlinear models with random term in linpred
body: e.g Logit( x b + e) where e is an unobserved random term
Jensen's inequality bites again.

E(y | x) is not Logit(x b)  (or expit ?)

Wooldridge on control function explains average structural function, ASF, that marginalizes/integrates out the `e` random variable, implemented by averaging over the sample `e`.
This looks similar to the "smearing" mean adjustment when endog is a nonlinearly, e.g. log, transformed y. #6611

Wooldridge, Jeffrey M. “Control Function Methods in Applied Econometrics.” The Journal of Human Resources 50, no. 2 (2015): 420–45.

AFAIU, the same applies to random/mixed effects nonlinear models, GLM, discrete.
AFAIR, I ran into this before in the random effect context, or as cause of overdispersion.

What should predict, get_prediction return by default?
What other statistics do we need as option for predict?

In the simple case of nonlinear transformation, the interpretation is in terms of median, quantiles only need monotonicity but no curvature adjustment. That's the interpretation that I prefer for nonlinear models, asymmetric distributions that don't have a mean parameterization, e.g. AFT accelerated failure time models for survival.

In tsa with log transformed endog, ignoring the mean correction often produces better forecasts than the smearing estimate, AFAIR some literature


Related:
In models like Probit, the additional randomness also changes the scale of the distribution of the latent random variable.
The estimated parameters are scaled parameters. b / s
In those cases it is important to look at marginal, partial effects. d E(y | x) / d x
(similar applies to heteroscedastic Logit, Probit. However, there we are fully parametric.)


see also https://github.com/statsmodels/statsmodels/issues/7719#issuecomment-1476843407

user: josef-pkt
issue_8761: state: open
title: Question: In MSTL, where does the constraint on the length of the ts in comparison to the periodicity come from?
body: I have a question concerning the following condition in the function **_process_periods_and_windows** used in MSTL come from:

`# Remove long periods from decomposition`
` 
        if any(period >= self.nobs / 2 for period in periods):
`
` 
warnings.warn(
"A period(s) is larger than half the length of time series." 
" Removing these period(s)."
            )
`

It makes intuitively sense to me, but I would be interested where exactly this is coming from. Thank you very much
user: pabuta
issue_8759: state: open
title: ENH: Are there any options to allow something like a GLS or WLS in the Ordered Models?
body: It seems that remedies for heteroscedasticity, such as modelling for the variances of the error using WLS or GLS, are not supported in the ordinal model module. Are there any solutions like `oglmx` package in R described by the following paper here?
https://cran.r-project.org/web/packages/oglmx/vignettes/oglmxVignette.pdf
It would be helpful if this could be incorporated into Python since this technique is also used by some studies like Hausman, Lo and MacKinlay (1992) and section 3.3 of *The Econometrics of Financial Markets*, Campbell, Lo and MacKinlay (1997). I would be more than grateful if someone could propose a solution to something like this in Python. I am also willing to submit a PR if necessary.
user: Silkdust
issue_8758: state: open
title: BUG/REF ? kwargs in RegressionModel.fit OLS, ...
body: It looks like we can still have misspelled, incorrect keywords in `fit` of linear regression models.
https://stackoverflow.com/questions/75842157/results-summary-reporting-z-instead-of-t-value-for-multiple-ols-regression-using

Any `kwargs` are just added as attributes to the results instance.

I don't know or don't remember what this is used for, maybe some subclasses use it.

If subclasses use it, then we have to make up a list of allowed keywords similar to what I did for `__init__` in models.
user: josef-pkt
issue_8757: state: open
title: ENH: penalization (inverse) proportional to t-values
body: (A semi-random thought, no reference)

Ridge and Lasso penalize all parameter values in the same way (L1 or L2)
With SCAD, we don't penalize large parameters.

However, I think it would be better not to penalize "significant" parameters.
I don't remember what the optimal parameter specific penalization factor is in (generalized) Ridge, but there should be something like this dependence.

With fixed "prior", the prior will be already less important for parameters or directions for which the data contains a large amount of information. However, we might want to reduce penalization for "significant" parameters even more.

example
high multicollinearity, but not singular.
We could use the initial estimate, compute t-values and then reduce the penalization `weight` for those parameters with large t-values.

maybe:
use SVD of initial cov_params and penalize in direction of eigenvectors with small eigenvalues.
in Ridge with 2 almost identical regressors: don't penalize the total effect (sum of the two), but penalize the split.
(this might be the same as reparameterizing (x1, x2) to (x1 + x2, x2 - x1) and penalizing individual parameters in the latter case.)
Related: this might make Ridge closer to invariant w.r.t. linear transformation (B X) (I don't remember what that invariance is called.)

choice of pen_weight:

In general we could have different penalization weights for each parameter and then choose some optimal criterion, GCV, IC, cross-validation. However, that is high dimensional hyper-parameter search.
Instead, what we have is fixed `weights` that are multiplied by a scalar `pen_weight`. In that case chooses good `weights` will improve performance.

computational aside

In OLS cov_params is just proportional to inv(x.T x), so eigenvalues will be related to singular values of x directly.
AFAIR, plain Ridge penalizes eigenvalues/eigenvector with something like 1 / (lambda + eigenvalue). Here we would have different lambda for each eigenvector depending on the size of the eigenvalue, i.e. just a different function of eigenvalues.






user: josef-pkt
issue_8756: state: open
title: ENH: treatment effect with multiple treatments, multinomial propensity score
body: (mainly a reminder to look at this. I downloaded several articles in statistics, medical statistics, ...)
SUMM issue #2443 with reference to Cattaneo and Stata

3 references, but I don't have an overview, and did not read much.

we need 
- a multinomial or similar model for propensity scores, and then 
- potential outcomes for pairwise comparisons or comparison to a reference treatment.

Imai, van Dyk is cited for general definition including multinomail/ordinal and continuous treatment
an older article of Guido Imbens is also often cited in this literature

Imai, Kosuke, and David A van Dyk. “Causal Inference With General Treatment Regimes.” Journal of the American Statistical Association 99, no. 467 (September 1, 2004): 854–66. https://doi.org/10.1198/016214504000001187.

Li, Fan. “Propensity Score Weighting for Causal Inference with Multiple Treatments.” The Annals of Applied Statistics 13, no. 4 (December 2019): 2389–2415. https://doi.org/10.1214/19-AOAS1282.

Lopez, Michael J., and Roee Gutman. “Estimation of Causal Effects with Multiple Treatments: A Review and New Ideas.” Statistical Science 32, no. 3 (2017): 432–54.


user: josef-pkt
issue_8755: state: open
title: ENH: power_approximate: collection of simple formulas for large number of tests
body: a possible idea for power and sample size computation

"breadth instead of depth"

Instead of getting good power computation for individual tests, we could add a large collection of simplified power and sample size computation for a large number of hypothesis tests.

I looked several times at books like Chow et al, but did not like them much.
Those power computation are simplified, e.g. proportion with same var or std under null and alternative. Those are simple asymptotic approximations.
However, they would be easy to implement, and power and sample size computation could be added for a large number of hypothesis tests that we don't have yet. PASS/NCSS referenced those in several cases.


Chow, Shein-Chung, Jun Shao, Hansheng Wang, and Yuliya Lokhnygina. Sample Size Calculations in Clinical Research. 3rd ed. New York: Chapman and Hall/CRC, 2017. https://doi.org/10.1201/9781315183084.

The plan would be to add the collection but separately from the "more serious" individual power and sample size functions, e.g. put them in a new module `power_approximate`.

The advantage is that we would quickly get things for different designs and cases
one sample, 2 sample parallel, cross-over (paired)
cluster randomized trials, group sequential trials
NCSS/PASS has a collection of function similarly named to the list of tests in Chow et al.

Also,
To compute the power or sample size we need almost all computation that are used in the hypothesis test.
So we might get those almost for free. We would have to split out intermediate computations either in helper functions or put them in classes.


user: josef-pkt
issue_8754: state: open
title: ENH: characteristic function inversion, pdf, cdf
body: I'm not sure I will ever get back to this.
I gave up on most of this kind of work because it's difficult to get it to work for regression model outside of additive error, loc-scale distributions and models.
Same applied to orthogonal polynomial approximation to distribution functions. 

parking a MIT licensed matlab package, found via a partial translation to julia
https://github.com/witkovsky/CharFunTool
one interesting detail: choice of bounds for integration interval. AFAIR, I never had an automatic choice for this.
`integration_interval` https://gitlab.com/tom.plaa/CharacteristicInvFourier.jl/-/blob/main/src/CharacteristicInvFourier.jl#L30 

related: fft used for discrete distributions #3556

I don't find "fft" in sandbox.distributions. It looks like I never committed my code when I was working on this.

I find 2 related threads in the mailing list:
https://groups.google.com/g/pystatsmodels/c/jObbnAxykf0/m/ESe7uG-GQy0J
https://groups.google.com/g/pystatsmodels/c/5aKGPCaQzAw/m/CjFXcgm7_LsJ

user: josef-pkt
issue_8753: state: open
title: Calculating the least-squares means from a mixedlm model
body: Hi,

I am trying to convert an R script where I fit a Linear Mixed-Effects Model and extract the  least-squares means (BLUEs):

`basic.lm <- lmer(class ~ Material + (1|Inoc_Dens), data = raw.data)
blues <- lsmeans(basic.lm, "Material")`

So far I implemented this in Python statsmodel but I have trouble to extract the BLUEs values:


```
raw_data["Material"] = pd.Categorical(raw_data["Material"], ordered=False)
basic_lmm = smf.mixedlm("class1 ~ Material", data=raw_data, groups=raw_data["Inoc_Dens"])
result = basic_lmm.fit()
```

How to receive the least-squares means of the model for each "Material"?

Any help would be very much appreciated,
thanks



user: snowformatics
issue_8752: state: open
title: I wrote a customized state space class but could not figure out the error. Please help.
body: Statsmodels - 0.14.0.dev535
@ChadFulton Could you please help me with this error? Thanks!

```
ValueError                                Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_46580/847275192.py in <module>
----> 1 two_factor_fit = two_factor_mod.fit()

~\Anaconda3\envs\new_base\lib\site-packages\statsmodels\tsa\statespace\mlemodel.py in fit(self, start_params, transformed, includes_fixed, cov_type, cov_kwds, method, maxiter, full_output, disp, callback, return_params, optim_score, optim_complex_step, optim_hessian, flags, low_memory, **kwargs)
    726             else:
    727                 func = self.smooth
--> 728             res = func(mlefit.params, transformed=False, includes_fixed=False,
    729                        cov_type=cov_type, cov_kwds=cov_kwds)
    730 

~\Anaconda3\envs\new_base\lib\site-packages\statsmodels\tsa\statespace\mlemodel.py in smooth(self, params, transformed, includes_fixed, complex_step, cov_type, cov_kwds, return_ssm, results_class, results_wrapper_class, **kwargs)
    887 
    888         # Wrap in a results object
--> 889         return self._wrap_results(params, result, return_ssm, cov_type,
    890                                   cov_kwds, results_class,
    891                                   results_wrapper_class)

~\Anaconda3\envs\new_base\lib\site-packages\statsmodels\tsa\statespace\mlemodel.py in _wrap_results(self, params, result, return_raw, cov_type, cov_kwds, results_class, wrapper_class)
    786                 wrapper_class = self._res_classes['fit'][1]
    787 
--> 788             res = results_class(self, params, result, **result_kwargs)
    789             result = wrapper_class(res)
    790         return result

~\Anaconda3\envs\new_base\lib\site-packages\statsmodels\tsa\statespace\mlemodel.py in __init__(self, model, params, results, cov_type, cov_kwds, **kwargs)
   2316         self.param_names = [
   2317             '%s (fixed)' % name if name in self.fixed_params else name
-> 2318             for name in (self.data.param_names or [])]
   2319 
   2320         # Save the state space representation output

~\Anaconda3\envs\new_base\lib\site-packages\statsmodels\base\data.py in param_names(self)
    354     def param_names(self):
    355         # for handling names of 'extra' parameters in summary, etc.
--> 356         return self._param_names or self.xnames
    357 
    358     @param_names.setter

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() 
```
user: YueHaiming
issue_8751: state: open
title: "update" function in sm.tsa.statespace.MLEModel
body: Hi @ChadFulton,

I have a quick question for the "update" function - def update(self, params, **kwargs):

There is a term in the observation intercept vector ($d_t$) called 't' in my code and it changes for each row of the endog data.
You can think of 't' is the time index which starts from 0 (for the first row of the dataframe), 't' =  0 + $\delta_t$ for the second row, 't' = 0 + 2 $\delta_2$ for the third row, ..., etc. $\delta_t$ is a fixed number (one week which is 1/52). Could you please advise me how to accomplish this? That would be much appreciated! Thank you!
user: YueHaiming
issue_8749: state: open
title: ENH/Review: oaxaca-blinder, decompose 2 sample mean in diff exog, diff params effect
body: I never looked at oaxaca-blinger before

uses print statement in Results instead of returning string
https://stackoverflow.com/questions/71490289/explained-and-unexplained-effects-dont-add-up-to-gap-in-oaxaca-blinder-decompos

extension to nonlinear models, Logit, Probit, ...
Sinning, Mathias, Markus Hahn, and Thomas K. Bauer. “The Blinder–Oaxaca Decomposition for Nonlinear Regression Models.” The Stata Journal 8, no. 4 (December 1, 2008): 480–92. https://doi.org/10.1177/1536867X0800800402.

I ran into this while looking at references for mediation effect and interaction effects for Logit, Probit, nonlinear models

oaxaca-blinder sounds a bit like regression adjustment in treatment effect: 2 groups/treatments and separate OLS regression by group.

brief google scholar search for recent articles, around 6700 since 2019

Guo, Kevin, and Guillaume Basse. “The Generalized Oaxaca-Blinder Estimator.” Journal of the American Statistical Association 0, no. 0 (June 16, 2021): 1–13. https://doi.org/10.1080/01621459.2021.1941053.

Rahimi, Ebrahim, and Seyed Saeed Hashemi Nazari. “A Detailed Explanation and Graphical Representation of the Blinder-Oaxaca Decomposition Method with Its Application in Health Inequalities.” Emerging Themes in Epidemiology 18, no. 1 (August 6, 2021): 12. https://doi.org/10.1186/s12982-021-00100-9.

I guess we get some (code) overlap with other methods, decomposition of ATE in regression adjustment?
partially related #8737 #8216


user: josef-pkt
issue_8748: state: open
title: Docstring / documentation for period parameter
body: This request is based on my [question](https://stackoverflow.com/questions/75792509/understanding-period-parameter-in-statsmodel-tsa-seasonal) on stackoverflow.

I tried using both [`STL`](https://www.statsmodels.org/stable/generated/statsmodels.tsa.seasonal.STL.html) and [`seasonal_decompose`](https://www.statsmodels.org/dev/generated/statsmodels.tsa.seasonal.seasonal_decompose.html) from `statsmodels.tsa.seasonal` module.
My question is regarding an input parameter, in both cases called `period` and described in the documentation as follows:

> Periodicity of the sequence

for `STL` case and

> Period of the series

for `seasonal_decompose`.

I believe further description should be added for this input parameter. The docs as it is has little to no use. I can understand that period parameter is the "period of the series", it did not give me any new information really. Furthermore, I still do not fully understand this parameters, but I believe they are different, regardless of the fact that is was probably bad practice to use a parameter with the same name with different meanings, the difference should be at least well documented (provided I am right that they are different, see the stackoverlow [question](https://stackoverflow.com/questions/75792509/understanding-period-parameter-in-statsmodel-tsa-seasonal) for what I concluded they are).

I think better work was done with the description in [MSTL](https://www.statsmodels.org/dev/examples/notebooks/generated/mstl_decomposition.html). Although I have a question, hourly data means that it will use the `freq` of the datetime as reference, so in the example is `freq='H'`? or is it each 24 samples regardless the `freq`?

user: NEGU93
issue_8747: state: open
title: Bug fix freq wasn't passed in SARIMAX class
body: ValueWarning: No frequency information was provided, so inferred frequency D will be used. Even though freq was passed in.

``` python
import pandas as pd
import numpy as np
from statsmodels.tsa.arima.model import ARIMA
from statsmodels.tools.eval_measures import rmse

def predict_future_amounts(data, future_days=5):
    # Convert the data to a pandas DataFrame
    df = pd.DataFrame(data)
    df["dates"] = pd.to_datetime(df["dates"])

    # Prepare the data for the model
    df = df.set_index("dates")
    y = df["amounts"]

    # Train an ARIMA model
    model = ARIMA(y, order=(1, 1, 1), freq='D')
    model_fit = model.fit()

    # Predict future amounts
    y_future = model_fit.forecast(steps=future_days)
    future_dates = pd.date_range(df.index.max() + pd.Timedelta(days=1), periods=future_days)

    # Add the future dates and amounts to the original data
    future_data = {
        "dates": future_dates.strftime('%Y-%m-%d').tolist(),
        "amounts": y_future.tolist()
    }

    return future_data

data = {
    "dates": ["2023-03-01", "2023-03-02", "2023-03-03", "2023-03-04", "2023-03-05"],
    "amounts": [100, 110, 105, 120, 130],
}

future = predict_future_amounts(data)
print(future)
```
Error shown before fix
```cmd output
ValueWarning: No frequency information was provided, so inferred frequency D will be used. 
```


user: sammeboy635
issue_8746: state: open
title: SUMM/ENH: (roadmap) margins and get_prediction with generated exog
body: We are still far away supporting `margins` as in Stata or margins/lsmeans/emmeans in R.

The main problem is getting the information to be able to construct multi-column exog and compute it's effects, e.g. derivatives w.r.t. original variable combined over several columns.
(I briefly looked at formulaic, but it's likely even worse than patsy for getting the required information out of it.)

what's possible

- DOC, notebook: "Do it yourself marginal effects." I have most of it using `_wald_nonlinear`.
- helper functions for constructing exog by user
- add public versions of predict_nonlinear, wald_nonlinear based ib delta method.

maybe worth a try

- black box derivatives for dy/dx, try numerical derivatives with predict "transform" (as generic as using python `eval`)
- similar, can we support derivatives if we have user provided function

possible

- support some cases that we can figure out, e.g. categorical without interaction similar to t_test_pairwise

difficult (not my kind of work)

- can we use our own stateful transforms, e.g. polynomials, similar to what we do in GAM?
- improve formulaic, e.g. 
  - it has formula derivative w.r.t. x using sympy, but it does not work for categorical or categorical-continuous interaction.
  - more stateful transforms with extra methods like derivatives that we need.

many open issues
user: josef-pkt
labels: roadmap: roadmaps, list of todos, overall and by topic
issue_8745: state: open
title: ENH (roadmap) "causal"
body: I'm starting to like `statsmodels.causal` as umbrella folder for anything related to some kind of endogeneity of regressors, both traditional methods and Rubin tradition of ATE.

current

- `treatment` treatment effect
- mediation currently in `stats`
- sandbox or unit tests for IV2SLS, IVGMM, PoissonGMM
- `heckman` in PR (hybrid model class combines MLE and 2-stage)

Todo (likely not to difficult)

- control function (2 stage)
  - continuous endogenous regressor, ivpoisson
  - generic control function, I guess we can just use any two models
- `treatment` for non-linear outcome models, GLM, discrete
- `etreatment` (Stata) similar to treatment but with IV, 
- endogenous missing values, dropout (not completely at random) ???
- IVGMM ??? other models than Poisson ? LEF ???
- endogenous selection models, heckman, ...
  - parametric: probit + correlated unobserved random component in outcome model, 
    linear and nonlinear, e.g. count outcome
  - semi- or nonparametric, e.g. 2stage with series expansion of 1st residual/control functions (Newey), not a structural model.
- ...

TBC

I guess dominant implementation will be 2-stage where main model results returned is outcome/final model. like 2-stage heckman.
Joint fully specified MLE would be separate, e.g. heckman MLE ???
We get a separate category for that, e.g. fully parametric 2-part models (2 submodels) with MLE, possibly with copula for dependence modelling?

user: josef-pkt
labels: roadmap: roadmaps, list of todos, overall and by topic
issue_8741: state: open
title: DOC: unstructured covariance is missing in GEE docs
body: merged in #7059
requested in #7005

I was trying to see how difficult it would be to replicate multivariate model with unstructered within covariance. (for #8722)
`Unstructured` is already implemented but not in docs. 

also not in https://www.statsmodels.org/stable/gee.html#dependence-structures :

Stationary
Equivalence
NominalIndependence

correction to usage:
GEE is more like panel data, common parameters across groups and group specific exog.
Multivariate linear model like MNLogit is group specific params and common exog.


user: josef-pkt
issue_8740: state: open
title: ENH: mv_test, manova tests for MNLogit
body: We can use the restriction matrices of mv_test used in multivariate manova, _MultivariateOLS for MNLogit.

However, the test statistics like Wilks' lambda are derived under normality assumptions. 
Wilks' is "exact" LR-test, AFAIU, with distribution based on ratio of Wishart matrices.

Instead we can use standard wald test in MNLogit and use mv_test mainly to create the appropriate constraint matrices for `L B M = C` hypothesis.
One needed tool is converting L, M and C to `vec` constraints, some kronecker product in MNLogit, 2-dim params without parameter restrictions imposed.



 
user: josef-pkt
issue_8739: state: open
title: ENH: path analysis and directed acyclic graphs DAG
body: "DAGs and Control Variables - How to select control variables for causal inference using Directed Acyclic Graphs"
https://towardsdatascience.com/controls-b63dc69e3d8c

maybe we can support something here.
We don't want to include or become a path package, but some support to eg. generate restrictions in system of equations, recursive systems, multi endog models could be useful.

`SVAR` without the `VAR` ?
or mediator, moderator effects #8733

(related to "causality")


user: josef-pkt
issue_8738: state: open
title: ENH: Instrumental Variable Estimation of Causal Risk Ratios
body: parking a reference for binary outcome 
mainly as motivation and overview what we might want to add.

compares different estimators including GMM (ivpoisson)

Palmer, Tom M., Jonathan A. C. Sterne, Roger M. Harbord, Debbie A. Lawlor, Nuala A. Sheehan, Sha Meng, Raquel Granell, George Davey Smith, and Vanessa Didelez. “Instrumental Variable Estimation of Causal Risk Ratios and Causal Odds Ratios in Mendelian Randomization Analyses.” American Journal of Epidemiology 173, no. 12 (June 15, 2011): 1392–1403. https://doi.org/10.1093/aje/kwr026.

found by chance looking at talk slides by Palmer https://remlapmot.github.io/talk/2010_iscb/
He has several talk slides using GMM or comparing with it https://remlapmot.github.io/talk/2011_stata/

user: josef-pkt
issue_8737: state: open
title: ENH: out-of-sample predict for regression adjustment (treatment, ATE) 
body: comparing predictions (potential outcomes and differences) across different models, similar to regression adjustment in `treatment` but out-of-sample.

This is similar to `margins` for a binary exog, but there we only have one model and need to compare predicted means for two different exog arrays.
In regression adjustment type models, we have two different models, one model per predicted potential outcome.

I guess to get the standard errors, we need the joint cov_params of the two models.

unclear: If the models where estimated on two different subsamples, then the cov_params would just be block-diagonal, assuming independence of observations or at least of subsamples.
Does this mean we don't need the full GMM version for joint inference?
Is this equivalent to full interaction of treatment dummy with exog.

IPW version will be more difficult, I don't have a guess (except using the full GMM cov_params as used in `treatment`)
I guess probability of treatment would not be relevant for prediction, because we want to compute potential outcomes for each treatment choice. Similar to computing expected benefits for choice by expected utility maximization.
However, if the decision maker is not the one making the choice, then we would need also the IPW adjustment and need to predict also the probability that treatment is choses or accepted. E.g. expected benefit or cost unconditional on treatment choice.

Predict for individual observations still has strong parametric assumptions, regression adjustment is not doubly robust, and we are not averaging over unobserved heterogeneity.

(no reference, Just a thought while browsing various treatment type literature)

related issues: predict, get_prediction and margins
user: josef-pkt
issue_8736: state: open
title: Different results in SARIMAX using numerical solver lbfgs
body: #### Describe the bug

I tested on two different machines and I got different results using SARIMAX. Is there an issue with using the solver 'lbfgs'? I would always get some results are off either by a few decimal points or by a few whole numbers. Is there any to check and diagnose this issue?

Also, I tried adding a seed from random, but it didn't have any changes. Can we even use a seed with SARIMAX?

#### Code Sample, a copy-pastable example if possible


```
SARIMAX([1242, 1382, 3010, 2043, 1380, 1212, 3377, 2392, 2997], \
                order=(2,2,2), seasonal_order=(0,0,0,0)).fit(method='lbfgs').forecast(12)
```

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `main`. If your problem has been fixed in an unreleased version, you might be able to use `main` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the main branch of this repository? It helps the limited resources if we know problems exist in the current main branch so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

A clear and concise description of what you expected to happen.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

Machine 1: array([2588.87029964, 3503.78937025, 3116.27753653, 4032.3775198 ,
       3645.12236969, 4561.06050117, 4173.97343396, 5089.74391647,
       4702.82450385, 5618.42735514, 5231.67555229, 6147.11081538])

Machine 2: array([2589.68386629, 3505.80559276, 3120.07125628, 4047.37477868,
       3644.90924254, 4586.03823864, 4196.74420887, 5103.70212923,
       48100.5791796 , 5620.36604405, 5245.41412799, 6203.02998133])

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]

Machine 1:
INSTALLED VERSIONS
------------------
Python: 3.8.5.final.0
OS: Darwin 22.3.0 Darwin Kernel Version 22.3.0: Mon Jan 30 20:42:11 PST 2023; root:xnu-8792.81.3~2/RELEASE_X86_64 x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

statsmodels
===========

Installed: 0.13.2 

Required Dependencies
=====================

cython: 0.29.21 
numpy: 1.22.3 
scipy: 1.5.2 
pandas: 1.1.3 
    dateutil: 2.8.1 
patsy: 0.5.2 

Optional Dependencies
=====================

matplotlib: 3.3.2 
    backend: module://ipykernel.pylab.backend_inline 
cvxopt: Not installed
joblib: 0.17.0 

Developer Tools
================

IPython: 7.19.0 
    jinja2: 2.11.2 )
sphinx: 3.2.1 
    pygments: 2.7.2 
pytest: 6.1.1 
virtualenv: Not installed

Machine 2:
INSTALLED VERSIONS
------------------
Python: 3.9.16.final.0
OS: Linux 5.10.147+ #1 SMP Sat Dec 10 16:00:40 UTC 2022 x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

statsmodels
===========

Installed: 0.13.5 (/usr/local/lib/python3.9/dist-packages/statsmodels)

Required Dependencies
=====================

cython: 0.29.33 (/usr/local/lib/python3.9/dist-packages/Cython)
numpy: 1.22.4 (/usr/local/lib/python3.9/dist-packages/numpy)
scipy: 1.10.1 (/usr/local/lib/python3.9/dist-packages/scipy)
pandas: 1.4.4 (/usr/local/lib/python3.9/dist-packages/pandas)
    dateutil: 2.8.2 (/usr/local/lib/python3.9/dist-packages/dateutil)
patsy: 0.5.3 (/usr/local/lib/python3.9/dist-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.7.1 (/usr/local/lib/python3.9/dist-packages/matplotlib)
    backend: module://ipykernel.pylab.backend_inline 
cvxopt: 1.3.0 (/usr/local/lib/python3.9/dist-packages/cvxopt)
joblib: 1.1.1 (/usr/local/lib/python3.9/dist-packages/joblib)

Developer Tools
================

IPython: 7.9.0 (/usr/local/lib/python3.9/dist-packages/IPython)
    jinja2: 3.1.2 (/usr/local/lib/python3.9/dist-packages/jinja2)
sphinx: 3.5.4 (/usr/local/lib/python3.9/dist-packages/sphinx)
    pygments: 2.6.1 (/usr/local/lib/python3.9/dist-packages/pygments)
pytest: 3.6.4 (/usr/local/lib/python3.9/dist-packages)
virtualenv: Not installed

</details>

user: ckwan48
issue_8734: state: open
title: wendog in statsmodels.discrete.discrete_model.MNLogit can't represent count weights
body: I am using statsmodels 0.12.2

Reading through the documentation of the MNLogit class I initially thought that the attribute wendog could contain weights to represent counts (for each component j) in observation m. 

Reading better, as element in a particular row of wendog has to be either 0 or 1 (depending on the component) I would still be required to have n_obs rows for wendog. Is this correct? Otherwise, what does the n represent in the documentation?


user: princi-pol
issue_8733: state: open
title: ENH: added intervening_variable.py and tests. Updated docs accordingly
body: added intervening variable analysis- classical sobel method and bootstrapping

<details>


**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in main and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/main -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>

user: codydance
issue_8730: state: open
title: BUG: cov_params is positive definite even though exog is singular
body: Mainly an observation for now. I don't know yet what we could do.

I'm trying out an OrderedModel with singular exog (one column duplicated)
cov_params is positive definite, no error, now warning for hessian invertibility
but exog is (near) singular

In this case the user does not get any warning about the collinearity problem, although `bse` are large for the affected params.

My guess is that the finite diff approximation used for hessian in OrderedModel makes hessian positive definite in this example.
It might be not (or less of) a problem in models that have hessian_factor that multiplies (outer product of) exog_i.
(I'm working in a messy notebook with several things based on https://stackoverflow.com/questions/75559325/llr-pvalue-extract-from-orderedresults-statsmodels  with extra random columns in exog, ex_orderedmodel.ipynb)

```
np.linalg.eigvals(res_prob.cov_params())
array([0.2441063 , 0.02922286, 0.02428714, 0.00033394, 0.00546712])

np.linalg.eigvals(mod.exog.T @ mod.exog)
array([3.68102909e+03, 7.74216134e+02, 5.31613486e+01, 4.62111681e+01,
       2.91358409e-14])
```
user: josef-pkt
issue_8729: state: open
title: ENH/BUG: allow fit keyword ridge_factor in newton
body: `model.fit(method="newton", ridge_factor= 1e-5)`

issues a warning about unused kwarg, and ridge_factor is not transmitted to the newton optimizer.
I wanted to try out fit with a large ridge factor to force convergence (to a possibly regularized solution)

I'm not sure whether this ever worked or was a refactoring victim.

AFAICS, it's because ridge_factor is explicit keyword and not in `kwargs` in signature, base.optimizer
```
        def _fit_newton(f, score, start_params, fargs, kwargs, disp=True,
                maxiter=100, callback=None, retall=False,
                full_output=True, hess=None, ridge_factor=1e-10)
```
user: josef-pkt
issue_8727: state: open
title: ENH: df_resid for exog with structural zeros
body: What is the appropriate df_resid for models where exog has structural zeros?

example SUR and MANOVA where exog in "long" version is a diagonal block matrix, e.g. eye kronecker exog.
https://github.com/statsmodels/statsmodels/issues/8722#issuecomment-1464226913
A similar case would be if we vertically stack equations or models, e.g. combine several OLS into one big OLS.

Even if we don't support automatic choices of df_resid, we would need a user option to set or adjust df_resid.
I had already added `df_resid_inference` for similar cases.

I guess this is related to within and between (repeated measures) anova which I never understood, or looked in details.

Related discussion: df or small sample correction in cluster robust cov_type.

likely related #6564
Issue for general degrees of freedom computation (effective df of exog, not because of penalization)
(It was too difficult to go through without very strong motivation)

user: josef-pkt
issue_8725: state: open
title: BUG: SARIMAX with time-varying coefficients works when model is (1,1,0) or (0,1,1), but raises an error when model set to (0,1,0) 
body: Call used:

```
SARIMAX(endog=endog, exog=exog, order=order
    seasonal_order= (0,0,0,4),
    trend= 'n',
enforce_stationarity=True,
enforce_invertibility=True,
time_varying_regression=True,
concentrate_scale = False,
measurement_error = False,
mle_regression = False)
```
Endog has 6 covariates. 

Works when order = (1,1,0) (however state.1 is all zero); also works for order = (0,1,1).  However when using order=(1,1,0) I get the following error:

```
    524     self.ssm._time_invariant = False
    526 # Initialize the fixed components of the statespace model
--> 527 self.ssm['design'] = self.initial_design
    528 self.ssm['state_intercept'] = self.initial_state_intercept
    529 self.ssm['transition'] = self.initial_transition

File C:\Developer\venv\lab\lib\site-packages\statsmodels\tsa\statespace\representation.py:420, in Representation.__setitem__(self, key, value)
    417     if key not in self.shapes:
    418         raise IndexError('"%s" is an invalid state space matrix name'
    419                          % key)
--> 420     setattr(self, key, value)
    421 # If it's a tuple (with a string as the first element) then we must be
    422 # setting a slice of a matrix
    423 elif _type is tuple:

File C:\Developer\venv\lab\lib\site-packages\statsmodels\tsa\statespace\representation.py:54, in MatrixWrapper.__set__(self, obj, value)
     51 shape = obj.shapes[self.attribute]
     53 if len(shape) == 3:
---> 54     value = self._set_matrix(obj, value, shape)
     55 else:
     56     value = self._set_vector(obj, value, shape)

File C:\Developer\venv\lab\lib\site-packages\statsmodels\tsa\statespace\representation.py:68, in MatrixWrapper._set_matrix(self, obj, value, shape)
     65     value = value[None, :]
     67 # Enforce that the matrix is appropriate size
---> 68 validate_matrix_shape(
     69     self.name, value.shape, shape[0], shape[1], obj.nobs
     70 )
     72 # Expand time-invariant matrix
     73 if value.ndim == 2:

File C:\Developer\venv\lab\lib\site-packages\statsmodels\tsa\statespace\tools.py:1477, in validate_matrix_shape(name, shape, nrows, ncols, nobs)
   1474     raise ValueError('Invalid dimensions for %s matrix: requires %d'
   1475                      ' rows, got %d' % (name, nrows, shape[0]))
   1476 if not shape[1] == ncols:
-> 1477     raise ValueError('Invalid dimensions for %s matrix: requires %d'
   1478                      ' columns, got %d' % (name, ncols, shape[1]))
   1480 # If we do not yet know `nobs`, do not allow time-varying arrays
   1481 if nobs is None and not (ndim == 2 or shape[-1] == 1):

ValueError: Invalid dimensions for design matrix: requires 7 columns, got 6

```
user: fcela
issue_8724: state: open
title: Seasonal decompose seasonality data issue
body: The seasonal_decompose function doesn't handle summer-winter time changes when it comes to seasonality. 
The seasonal pattern shifts +- an hour, exactly form the hour when the the clock changes.

user: kntrdzs
issue_8722: state: open
title: ENH: enhance _MultivariateOLS, MANOVA, code duplication, 
body: I thought MANOVA is using `_MultivariateOLS`.
However, it looks like they share code, helper functions, but manova doesn't reuse the _MultivariateOLS class.
There is also quite a bit of code duplication.

I was looking for access to the _MultivariateOLS instance in the MANOVA and it's test result instances, but it's not available.

_MultivariateOLS does not have a summary implemented, which makes it difficult to get a quick overview of results.

context #8713 trying to figure out usage and problems with multi-way manova.


based on an example: _MultivariateOLS runs an identical test to MANOVA

```
formula = 'PC1 + PC2 + PC3 + PC4 ~ C(Genotype, Helmert) * C(Temp, Helmert) * C(Time, Helmert)'
mod = _MultivariateOLS.from_formula(formula, data=p_df)
res = mod.fit()
tt = res.mv_test()
```
but `res` does not have any of the usual results attributes and methods, not even `params`

```
[i for i in dir(res) if not i.startswith("__")]
['_fittedmod',
 'design_info',
 'endog_names',
 'exog_names',
 'mv_test',
 'summary']
```

user: josef-pkt
issue_8721: state: open
title: SUMM: singular exog (again)
body: mainly for linear models, but some might also affect other models.

I'm looking at singular exog again in context of pivoting QR.
I don't have an overview of open issues and possible design.

related
- positive definite hessian in optimization #3099
  -  AFAIR, no check if hessian is computed in fit and skip_hessian is in optimization 
- discussion on not raising if singular #3824
- a FAQ issue for ill-conditioning #6381
- asking for dropping instead of pinv #8670
- PR for pivoting QR #6935
- a similar issue to this one #7695
- old issue for Logit, this should now raise #3367
- MixedLM singular, I'm not familiar with this #6140

to do, to check:

- cutoffs, thresholds: OLS, ... need an option in fit for `rcond` in pinv #2628
  - it looks like matrix rank of cov_param can be much smaller than rank of exog. I guess because of squaring
  - if we use different linalg computations, then we need to check that matrix rank definitions, i.e. thresholds, are consistent, e.g. pinv rcond versus np.linalg.matrix_rank. Our rank attribute in OLS might not be the same as the  pinv rank and not the same as the rank of cov_params. #1662 #3994
  - need to report/warn rank of cov_params if not full rank. But constrained estimation, `fit_constrained, has intentionally reduced rank cov_params, variance is zero of constraints  #1717
  - pinv threshold in OLS is too small, many collinear cases slip through.
- Where is the function that I wrote to drop variables sequentially based on QR? I cannot find it right now.
  https://github.com/statsmodels/statsmodels/pull/2380/files#diff-6c9338f4faac471ff45da570aa82c563661ada8aff75afe24e70f60ae9b0ea85R586 there is also an iterative version somewhere (not mine) 
- scipy.linalg pivoting QR looks pretty "random", that is in repeated runs with different random samples, which variable is dropped varies.
- empty cells, columns with only zeros (this is more urgent) patsy does not check whether cells are empty when creating the design matrix for categorical interactions. AFAIR pinv works to set those params to zero. but df for anova_lm and wald tests can be wrong. #4137 #5398 #8506  #8512
- postestimation, hypothesis tests with singular cov_params. Not clear, not systematically checked, (maybe relevant #4798) 
- still no "estimable" checks and helper functions. #6271
- `fit_collinear` #8580
- ...


user: josef-pkt
issue_8718: state: open
title: CombineResult.plot_forest(ax=ax) does not pass ax to dot_plot()
body: https://github.com/statsmodels/statsmodels/blob/6a9ce0a291fe7b42797c8f29a0da501017831a56/statsmodels/stats/meta_analysis.py#L322

`CombineResults.plot_forest(ax=ax)` does not actually pass the mpl axis when invoking `dot_plot()`. So a new axis is created by dot_plot() whether or not an axis is passed to plot_forest(). I came across this with version 0.13.2 but appears still to be the case.
Removing ax as an explicit argument from plot_forest() would even solve the problem, as optional *kwds _are_ passed to dot_plot() anyway. 
user: mjrosen
issue_8717: state: open
title: Same code+Data+Statsmodels Version -> Different Results
body: Hello guys!

I was running a SARIMA (2,0,1)x(1,0,2)x24 model using my old laptop. It took around 20 to 30 seconds to finish and did not throw any warning. Python version is 3.9.7 and statsmodels version is 0.14.0.dev535.

I got a new laptop a couple of days ago. I installed Anaconda with default Python = 3.9.13. I used night wheels to install  0.14.0.dev535 statsmodels as well. I tried to install the same Numpy and Scipy, but I was not able to do that because of the Anaconda distribution issue (Numpy and Scipy are newer on the new laptop).  I was expecting that the code would run faster because the new laptop has much powerful CPU and more cores. However, it took around 4-5 minutes to finish and throw warning "ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals".  I tried different older versions of Scipy, some did converge but still took 4-5 minutes.

For both, I set maxiter at 10,000.

Can anyone help me to clarify the issue? Much appreciated!
user: YueHaiming
issue_8715: state: open
title: Additional metric added based on the quasi-likekilhood loss function (qlike).
body: - [ ] closes #xxxx
- [ ] tests added / passed. 
- [ ] code/documentation is well formatted.  
- [ ] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

<details>


**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in main and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/main -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>

user: edjanga
issue_8713: state: open
title: Results of n-way MANOVA with interactions differ significantly from results from R's stats::manova
body: (I'm not sure if this is a bug or a feature request - sorry for false alarm if this was expected!)

I get different results when performing n-way MANOVA with interactions when using this implementation and R.

Let's use data from the file I attached as an example. 
[metals.csv](https://github.com/statsmodels/statsmodels/files/10882245/metals.csv)

In R I would use the following formula:

```R
m_data = read.csv('https://github.com/statsmodels/statsmodels/files/10882245/metals.csv')
metal_manova = manova(cbind(C1, C2) ~ extractants * metals, data=m_data)
summary(metal_manova, test='Wilks')
```
This yields the following result:

| |Df |Wilks|approx F|num Df|den Df |Pr(>F)|
|---|---|---|---|---|--- |---|
extractants|9|0.0039340|164.38|18|198|< 2.2e-16 ***|
metals|4|0.0019440|536.58|8|198|< 2.2e-16 ***|
extractants:metals|36|0.0005648|112.97|72|198|< 2.2e-16 ***|
Residuals|100|


Using the following steps:
```python
from statsmodels.multivariate.manova import MANOVA
import pandas as pd
m_df = pd.read_csv('https://github.com/statsmodels/statsmodels/files/10882245/metals.csv')
m_man = MANOVA.from_formula('C1 + C2 ~ extractants * metals', data=m_df)
m_res = m_man.mv_test().summary_frame
m_res[m_res.index.get_level_values(1) == 'Wilks\' lambda']
```

I get the following result:
|Effect|Statistic|Value|Num DF|Den DF|F Value|Pr > F|
|---|---|---|---|---|---|---|                                 
|Intercept|Wilks' lambda|0.999777|2|99.0|0.011051|0.989011|
|extractants|Wilks' lambda|0.808417|18|198.0|1.23418|0.236848|
|metals|Wilks' lambda|0.942882|8|198.0|0.738631|0.657287|
|extractants:metals|Wilks' lambda|0.000565|72|198.0|112.965548|0.0|

i.e. the F values are completely different in these two implementations (and thus the Pr > F is also different). Why is it the case? The results are the same for statsmodels'and R's implementations when there is no interaction between the independent variables - only the analysis with interactions yield different results. Can I specify some parameters that would make the results the same as in R? 

I'm working with version 0.13.5 in WSL2. Python version 3.10.6.

user: mikrzol
issue_8712: state: open
title: STY: change nobs2 to nobs0 for consistency/style
body: - [ ] closes #xxxx (no issue)
- [ ] tests added / passed. 
- [x] code/documentation is well formatted.  
- [x] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

This is a very simple PR which **does not** change functionality of the code whatsoever, just fixes something I found confusing while reading the code. 

For consistency with the paper the code is based on, the [function switches](https://github.com/statsmodels/statsmodels/blob/140f8341da33c064ff6f2e51dedc9428ba249fa4/statsmodels/stats/proportion.py#L1526) `counts2, nobs2` to `counts0, nobs0`. It looks like there was one place this change was missed later in the function, which I just found confusing while reading the code. 
user: bdpedigo
issue_8711: state: open
title: how to fit linear mixed model with a fixed variance component
body: #### Describe the bug
I want to build a linear mixed model with fixed var components. 
variance components = 5 and 10 (experimental error and sampling error, respectively)

But looks like the var components can not be passed to the model. How to fix it? thank you.

UserWarning: Argument params not used by MixedLM.fit
  warnings.warn("Argument %s not used by MixedLM.fit" % x)

#### Code Sample, a copy-pastable example if possible


```# Define the model formula
formula = 'y ~ trt'

# Fit the model
model = smf.mixedlm(formula, data, groups=data['rep'])

# Fix the variance components
vcov = {'rep: Intercept': 10, 'Residual': 5}

# Create a MixedLMParams object with fixed variance components
params = sm.regression.mixed_linear_model.MixedLMParams.from_components(vcomp=vcov)

# Fit the model with fixed variance components
model_fix = model.fit(params=params, method='lbfgs')

# Print a summary of the model results
print(model_fix.summary())
```
<details>


user: JJ-Zhang-DS
issue_8710: state: open
title: Multivariate Granger causality
body: According to the documentation: 

`The data for testing whether the time series in the second column Granger causes the time series in the first column.`

So it seems to visit possible to compute the GC using more time series. However: the following code: 

```
import numpy as np
import statsmodels.api as sm

# Generate example data
nobs = 100
X1 = np.random.randn(nobs)
X2 = np.random.randn(nobs)
X3 = np.random.randn(nobs)
X4 = np.random.randn(nobs)
X5 = np.random.randn(nobs)

# Create numpy array with shape (nobs, 5)
data = np.column_stack((X1, X2, X3, X4, X5))

# Compute Granger causality tests for all pairs of variables and lags up to 2
maxlag = 2
results = sm.tsa.stattools.grangercausalitytests(data, maxlag, verbose=False)

# Print the p-values for the Granger causality tests
for lag in range(1, maxlag+1):
    print(f"Lag {lag}:")
    for i in range(data.shape[1]):
        for j in range(data.shape[1]):
            if i != j:
                print(f"X{i+1} -> X{j+1}: p-value = {results[lag][i+1][j]['ssr_ftest'][1]}")
```

raise the following error: 

`ValueError: wrong shape for coefs`



user: MatteoSerafino
issue_8709: state: open
title: Understanding p value calculation for coefficients of Logit()
body: Could you please help me: I have fitted Logistic Regression model, so I have my data with probabilities and fixed predictors coefficients. How can I use functionality Lodit().summary() for calculation pvalues for my case? May be there is another possibility to calculate pvalues, more appropriate?

Thank you very much
user: loveis98
issue_8706: state: open
title: Only train data in granger causality test?
body: https://github.com/statsmodels/statsmodels/blob/a948e8dee9e99a08f249f67ca2ac5df68fea835b/statsmodels/tsa/stattools.py#L1405

Is the granger causality test only ran on training data? If yes, is that correct? Isn't it possible then to face issues with overfitting?
user: tim-roethig-db
issue_8705: state: open
title: ENH/SUMM (roadmap) Power and sample size computation
body: several open issues and missing extensions
generic issues
- #8162
- #8206

81 open issues for `is:issue is:open power sample size `

### open issues and todos

- better docs: #8647 #4085 
- power classes #8652
- option for "exact" power in discrete rates, proportions, ..., actual power (when different from requested minimum power)
  - sawtooth #2731,
- add (excess) dispersion option to power computation for proportion, rates
- proportion, one and two sample, one sample requires #6721
  - non-zero null #8251
  - match power to test method #8675
  - other tests: equivalence 
- rates, more general #8138
  - #8303 
- power_approximate #8755 
- regression
  - linear, FTestRegressionPower #8646 #8403
  - GLM ?
- generic, based on test statistic, noncentrality, ...
- power for confidence interval
- gof
- other tests, hypothesis test we have or have not, ???
  - paired, correlated 2 sample,  ... e.g. #2743
  - tests for variance/dispersion
- ...
user: josef-pkt
labels: roadmap: roadmaps, list of todos, overall and by topic
issue_8701: state: open
title: Unable to get default parameters of STL function
body: #### Describe the bug

I am trying to get default argument of the SYL function but unable to get the saem. Also, it doesnt seem there is anything called  "__text__signature__" which can be used by `inspect` library to get a signature of the init.

#### Code Sample, a copy-pastable example if possible


```python
import statsmodels.tsa.seasonal.STL
import inspect 
inspect.signature(STL.__init__)
# <Signature (self, /, *args, **kwargs)>
```
<details>

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `main`. If your problem has been fixed in an unreleased version, you might be able to use `main` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the main branch of this repository? It helps the limited resources if we know problems exist in the current main branch so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

expect output:
```
<Signature (self, endog, period=None, seasonal=7, trend=None, low_pass=None,
        seasonal_deg=1, trend_deg=1, low_pass_deg=1, robust=False,
        seasonal_jump=1, trend_jump=1, low_pass_jump=1)
```

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]

INSTALLED VERSIONS
------------------
Python: 3.10.8.final.0
OS: Linux 5.10.102.1-microsoft-standard-WSL2 #1 SMP Wed Mar 2 00:30:59 UTC 2022 x86_64
byteorder: little
LC_ALL: None
LANG: C.UTF-8

statsmodels
===========

Installed: 0.13.5 (/home/user1/miniconda3/envs/ADT/lib/python3.10/site-packages/statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.23.5 (/home/user1/miniconda3/envs/ADT/lib/python3.10/site-packages/numpy)
scipy: 1.10.0 (/home/user1/miniconda3/envs/ADT/lib/python3.10/site-packages/scipy)
pandas: 1.5.2 (/home/user1/miniconda3/envs/ADT/lib/python3.10/site-packages/pandas)
    dateutil: 2.8.2 (/home/user1/miniconda3/envs/ADT/lib/python3.10/site-packages/dateutil)
patsy: 0.5.3 (/home/user1/miniconda3/envs/ADT/lib/python3.10/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.6.2 (/home/user1/miniconda3/envs/ADT/lib/python3.10/site-packages/matplotlib)
    backend: module://matplotlib_inline.backend_inline 
cvxopt: Not installed
joblib: 1.2.0 (/home/user1/miniconda3/envs/ADT/lib/python3.10/site-packages/joblib)

Developer Tools
================

IPython: 8.8.0 (/home/user1/miniconda3/envs/ADT/lib/python3.10/site-packages/IPython)
    jinja2: 3.1.2 (/home/user1/miniconda3/envs/ADT/lib/python3.10/site-packages/jinja2)
sphinx: Not installed
    pygments: 2.14.0 (/home/user1/miniconda3/envs/ADT/lib/python3.10/site-packages/pygments)
pytest: 7.1.2 (/home/user1/miniconda3/envs/ADT/lib/python3.10/site-packages/pytest)
virtualenv: 20.17.1 (/home/user1/miniconda3/envs/ADT/lib/python3.10/site-packages/virtualenv)
</details>

user: Ashish-Tiger
issue_8700: state: open
title: ARDL fitted params in summary() are not real
body: **Problem short description:**

I use ARDL on a sandbox task  to study all the parameters and how they work.
I fitted the model and i get perfect result (task is designed for model to return perfect results).
But when i look at m.params or m.summary(), values of fitted perameters are not the same as model uses to predic 100% preciese values.

<details>
**Problem detailed description:**

```
### EXAMPLE START
# import libs:
import pandas as pd
from statsmodels.tsa.api import ARDL

#create test data from preciese math formula
sq = [x**2 + x + 11 for x in range(0,101)]
tdf = pd.DataFrame({'sq':sq})

# fit ARDL model and print model information
mt1 = ARDL(tdf['sq'], seasonal = False, lags=1, trend = 'ct' ).fit()
print(mt1.params)
print(mt1.summary())
print(mt1.forecast(5))

### EXAMPLE END
```

As you see i'm working with f(t) = t^2 + t + 11 and it colud be represented as recurrent sequence in form of 
f(t) = 1 \* f(t-1) + 2 \* t which is in perfect agreement with what ARDL model is expected to predict: lag=1 dependence and linear time trend. And it does perfect prediction as you can see in output of print(mt1.forecast(5)) line

Expected fitted parameters are: **const = 0**, trend = 2, L1 = 1
But both mt1.summary() and mt1.params return: **const= 2**, trend = 2, L1 =1
Why the constant equals to 2 and how model was able to predict perfect results with this value of const? What is the final equation model uses to predict correct values with this value of const?

General form of equation can be found [here](https://www.statsmodels.org/stable/generated/statsmodels.tsa.ardl.ARDL.html) in "Notes" section.

</details>


#### Expected Output

I expected to get mt1.aparms as
```
  const    0
  trend    2.0
  sq.L1    1.0
  dtype: float64
```
for data of the form f(t) = 1\*f(t-1) + 2\*t  + 0

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]
```
INSTALLED VERSIONS
------------------
Python: 3.9.12.final.0

statsmodels
===========

Installed: 0.13.5 (C:\Users\my_user\Anaconda3\lib\site-packages\statsmodels)

Required Dependencies
=====================

cython: 0.29.28 (C:\Users\my_user\Anaconda3\lib\site-packages\Cython)
numpy: 1.21.5 (C:\Users\my_user\Anaconda3\lib\site-packages\numpy)
scipy: 1.7.3 (C:\Users\my_user\Anaconda3\lib\site-packages\scipy)
pandas: 1.4.2 (C:\Users\my_user\Anaconda3\lib\site-packages\pandas)
    dateutil: 2.8.2 (C:\Users\my_user\Anaconda3\lib\site-packages\dateutil)
patsy: 0.5.2 (C:\Users\my_user\Anaconda3\lib\site-packages\patsy)

Optional Dependencies
=====================

matplotlib: 3.5.1 (C:\Users\my_user\Anaconda3\lib\site-packages\matplotlib)
    backend: module://matplotlib_inline.backend_inline 
cvxopt: Not installed
joblib: 1.1.0 (C:\Users\my_user\Anaconda3\lib\site-packages\joblib)

Developer Tools
================

IPython: 8.2.0 (C:\Users\my_user\Anaconda3\lib\site-packages\IPython)
    jinja2: 2.11.3 (C:\Users\my_user\Anaconda3\lib\site-packages\jinja2)
sphinx: 4.4.0 (C:\Users\my_user\Anaconda3\lib\site-packages\sphinx)
    pygments: 2.11.2 (C:\Users\my_user\Anaconda3\lib\site-packages\pygments)
pytest: 7.1.1 (C:\Users\my_user\Anaconda3\lib\site-packages\pytest)
virtualenv: Not installed
```
</details>

user: satyrmipt
issue_8699: state: open
title: Confusion with se_mean and standard deviation
body: I would like to clarify that I understood conception of se_mean and standard deviation in statsmodels correctly. Could you help me with this? 

In documentation **statsmodels.tsa.base.prediction.PredictionResults.se_mean** we have description that se_mean is the standard deviation of the predicted mean. At the same time in **Release 0.8.0** there is a passage that get_forecast provides standard errors. As far as I know standard deviation and standard errors of mean are not the same things. 

As a rookie in statistics I found in wiki that std is a variation in measurements, while the standard error of the mean is a probabilistic statement about how the sample size will provide a better bound on estimates of the population mean, in light of the central limit theorem. However, standard error can be described as an estimation of that standard deviation. Does it mean that in **statespace.sarimax** we estimate possible future values of standard deviations and the model outputs std which depends on the number of time series point (more additional time points, better prediction of std)? 

I build SARIMAX model and want to construct сonfidence interval as a variation in measurements for forecast. Is it possible to use mean_se for this or I need to convert these values to std by multiplying SE by sqrt(n)? And does n equal the number of data points in time series before forecasting? 

Thank you for yor reply in advance!
user: Elisei-Kungurov
issue_8691: state: open
title: ENH/REF: update stata export file
body: supersedes #1716

I'm using the ado file with Stata 14

trying to write a docstring

```
syntax [namelist(min=1)], [RESName(str)] SAVing(str) [ Format(str) APPend REPlace NOEst]

namelist : list of matrices to be included in output
RESName(str) : name used as prefix for results and matrix names
SAVing(str) : output filename, name of python module
Format(str) : format for numbers in output
options :
    APPend : append results to existing output file
    REPlace : replace content of existing output file or create new output file
    NOEst : do not include the estimation results dictionary `ereturn`
```
user: josef-pkt
issue_8690: state: open
title: ENH: outlier detection, skewed, zero-inflated data
body: 
The Templ et al article uses several methods for outlier detection.
A bit ad-hoc. uses Box-Cox transformation of data. Methods are not distribution specific.
Might be good to get an overview of what we might want to have, e.g. ogk does well for multivariate x-outlier identification.
It might also be useful to compare with distribution specific outliers, e.g. outlier-influence post-estimation for Poisson, .... 

Templ, M., J. Gussenbauer, and P. Filzmoser. “Evaluation of Robust Outlier Detection Methods for Zero-Inflated Complex Data.” Journal of Applied Statistics 47, no. 7 (May 18, 2020): 1144–67. https://doi.org/10.1080/02664763.2019.1671961.

Yang, Jun, Min Xie, and Thong Ngee Goh. “Outlier Identification and Robust Parameter Estimation in a Zero-Inflated Poisson Model.” Journal of Applied Statistics, October 27, 2010. https://doi.org/10.1080/02664760903456426.

user: josef-pkt
issue_8689: state: open
title: TST: Integrating with Google/OSS-fuzz
body: Hi,

I would like to help keep statsmodels running smoothly by adding fuzz testing and integrating it with google/oss-fuzz. In case you are not familiar with fuzz testing, briefly, it is an additional layer of testing (in addition to unit testing) that will help uncover any hidden or hard-to-reach bugs that might have been missed during unit testing. [Wikipedia ](https://en.wikipedia.org/wiki/Fuzzing)explains it nicely:

>In programming and software development, fuzzing or fuzz testing is an automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program. The program is then monitored for exceptions such as crashes, failing built-in code assertions, or potential memory leaks.

Google offers a free, continuous fuzzing service called OSS-fuzz. If statsmodels is integrated into oss-fuzz, the fuzz tests under statsmodels will be built and then run once a day, to search for bugs and vulnerabilities in statsmodels. This service can be integrated with the CI for statsmodels, so that the fuzz tests are run for 10min or so for every pull request, preventing buggy code from being merged.

I've opened up a pull request to add a basic fuzz-testing harness here https://github.com/statsmodels/statsmodels/pull/8687. If you are keen on adding statsmodels to oss-fuzz I'd be happy to champion the integration :)
user: jesslatimer
issue_8688: state: open
title: ENH: model equivalence testing
body: reversing the null hypothesis in model comparisons and gof tests.

not a very popular topic, but should be more used
Main practical problem is how to specify insignificant "effect size" margin for model comparisons and gof.

example where it would work is FTestRegressionPower, i.e. Cohen's f2 effect size that can be related to (partial) R-square. In these cases we have a interpretable scaled noncentrality nc/nobs.

Independently of how to define equivalence margin, we could just add the functions.
The target would be to extend equivalence testing from one and two sample functions to models, model comparisons and diagnostic and specification tests.

These equivalence tests would be targeted to specific statistics, e.g. gof in general, directional misspecification, params, predictive test (?), ... depending on the test statistic that is used.

e.g.
- show that interaction effect is close to zero (equivalent models with and without interaction effect)
- show that some statistics are equivalent for models with different link functions.
- ...




user: josef-pkt
issue_8687: state: open
title: TST: Adds fuzz harness
body: - [ ] closes #xxxx (N/A)
- [x] tests added / passed. 
- [x] code/documentation is well formatted.  
- [x] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

<details>


**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in main and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/main -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>

user: jesslatimer
issue_8686: state: open
title: ENH: out-of-sample moment conditions, score, GMM, MLE
body: looks interesting for diagnostics

out of sample or forecast moment conditions, and it's asymptotic distribution

Hoffman, Dennis, and Adrian Pagan. “PRACTITIONERS CORNER: Post-Sample Prediction Tests for Generalized Method of Moments Estimators.” Oxford Bulletin of Economics and Statistics 51, no. 3 (1989): 333–43. https://doi.org/10.1111/j.1468-0084.1989.mp51003007.x.


We still have the problem that loglike and score can only be evaluated at the model data, endog, exog, i.e. no predictive, out-of-sample statistics. Similarly for GMM, momcond are only in-sample.
In GLM we could use freq_weights to only estimate on a subsample. 

user: josef-pkt
issue_8685: state: open
title: ENH: Firth penalized Poisson
body: This article has explicit expression for `score` of Firth penalized poisson, based on IRLS

motivation is separation in Poisson #1512

Mondol, Momenul Haque, M. Shafiqur Rahman, and Wasimul Bari. “A Penalized Likelihood Approach for Dealing with Separation in Count Data Regression Model.” Communications in Statistics - Simulation and Computation 0, no. 0 (March 30, 2022): 1–15. https://doi.org/10.1080/03610918.2022.2057541.

user: josef-pkt
issue_8683: state: open
title: ENH: recentering cluster robust cov_type in GMM, fixed number of cluster asymptotics
body: mainly parking a reference


Hwang, Jungbin. “Simple and Trustworthy Cluster-Robust GMM Inference.” Journal of Econometrics 222, no. 2 (June 1, 2021): 993–1023. https://doi.org/10.1016/j.jeconom.2020.07.048.

cov_params based on centered and uncentered moment conditions are not asymptotically equivalent if number of clusters is fixed.
in overidentified GMM where moment conditions are not zero at optimum.

Article is too heavy to quickly figure out. I did not find how the centering "constant" is defined. 

(I recently saw other articles for centering issue in GMM. ???)


user: josef-pkt
issue_8680: state: open
title: holt-winters got unexpected results
body: Dear developer,

Thank you for developing this useful package! I literally use it every day!

Today, I discovered a very unexpecting result from using the holt-winters model: when I try to make predictions for stocks, it gave me about 10^6 larger result that the average price series I input. But all inputs before or after are normal. I double-checked a few times but got the same super large results. Below is my code:
`def predHW(M):
    ticker=M[0]
    season=M[1]
    data=M[2]
    trend = M[3]
    seasonal = M[4]
    log = M[5]
    
    if trend == 'mul' or seasonal == 'mul' or log == False:
        data = data.dropna()
    else:
        data = data.pct_change().dropna()
    
    if season <=105:
        window = 52*5
    else:
        window = season*2+1
    
    x = data[-window:]
    fit_add = ExponentialSmoothing(x, seasonal_periods= season, trend=trend, seasonal=seasonal, damped_trend=True).fit() # Holt-Winters additive smoothing
    # Forecast 1 more data point
    y_pred = fit_add.forecast(1)
    
    return ticker, y_pred.iloc[0], data.index[-1]

predHW(['300003.SZ',5,df['close'][:'2019-10-31'],'mul','mul',False])`

My data is a pandas series with string indices:
date
2018-10-31    28.66
2018-11-30    31.30
2018-12-28    20.81
2019-01-31    20.34
2019-02-28    25.83
2019-03-29    26.50
2019-04-30    26.64
2019-05-31    22.98
2019-06-28    23.02
2019-07-31    24.78
2019-08-30    26.92
2019-09-30    25.06
Name: close, dtype: float64
user: RainCamel
issue_8679: state: open
title: ENH/MAINT add typing, mypy or similar
body: I don't like typing annotation in the code, it's ugly and makes the code difficult to read.

However, numpy has stub files `.pyi` for all modules. Those would be appropriate also for statsmodels.
I tried to create stub files with mypy's stubgen, but those are using `Any` for all parameters.

I started with statsmodels.stats because that has relatively simple functions.
The model modules will need classes or class hierarchies as types.

pandas:
https://github.com/pandas-dev/pandas/blob/main/pandas/_typing.py

numpy
has only a few public types in numpy.typing
more details in private https://github.com/numpy/numpy/tree/main/numpy/_typing and individual pyi files

sklearn discussion
https://github.com/scikit-learn/scikit-learn/issues/16705#issuecomment-859378575

user: josef-pkt
issue_8678: state: open
title: SUMM/ENH: (roadmap) penalization
body: (penalization label has currently 75 open issues) 

### models with penalized objective functions

- penalization classes
...
...

- Firth and similar
  - data dependent penalty
  - double penalization Firth + L2
  - data augmentation 
  - alternative penalization for logit/binomial (F2 ? references ?)
- ...

### stats

- penalized cov
- ...
TBC
user: josef-pkt
labels: roadmap: roadmaps, list of todos, overall and by topic
issue_8677: state: open
title: ENH: logit, binomial double penalization Firth and (generalized) Ridge (L2)
body: (mainly to park references)

Gao, Sujuan, and Jianzhao Shen. “Asymptotic Properties of a Double Penalized Maximum Likelihood Estimator in Logistic Regression.” Statistics & Probability Letters 77, no. 9 (May 1, 2007): 925–30. https://doi.org/10.1016/j.spl.2007.01.004.

Shen, Jianzhao, and Sujuan Gao. “A Solution to Separation and Multicollinearity in Multiple Logistic Regression.” Journal of Data Science 6, no. 4 (August 4, 2022): 515–31. https://doi.org/10.6339/JDS.2008.06(4).395.

user: josef-pkt
issue_8676: state: open
title: WIP/ENH: add option null_prop_var in samplesize_proportions_2indep_onetail
body: closes #8675

This will not be in 0.14

No real theoretical justification, and might need other options in the future.

for example
Wang Chow use the same var or std under null and alternative in sample size formula.
My guess is that this corresponds to a wald test for proportions, which does not have good small sample properties.

Wang, Hansheng, and Shein-Chung Chow. “Sample Size Calculation for Comparing Proportions.” In Wiley Encyclopedia of Clinical Trials. John Wiley & Sons, Ltd, 2007. https://doi.org/10.1002/9780471462422.eoct005.

opposite direction
some articles add continuity correction to match more closely Fisher's exact test, which is conservative in small samples.
e.g.
last paragraph in note section with two references
https://search.r-project.org/CRAN/refmans/EnvStats/html/propTestPower.html

another article, Ury and Fleiss use the same var under null and alternative but add continuity correction
Ury, Hans K., and Joseph L. Fleiss. “On Approximate Sample Sizes for Comparing Two Independent Proportions with the Use of Yates’ Correction.” Biometrics 36, no. 2 (1980): 347–51. https://doi.org/10.2307/2529991.

user: josef-pkt
issue_8675: state: open
title: ENH: add option for null_var or null_prop to samplesize_proportions_2indep_onetail 
body: https://stats.stackexchange.com/questions/605466/how-to-get-python-statsmodels-to-match-evan-millers-famous-ab-test-sample-size/605540#605540

I'm not sure what the name of the options should be.
user: josef-pkt
issue_8670: state: open
title: Singular design matrix produce very extreme p-values
body: I found when the design matrix is singular, statsmodels can give me very extreme and wrong p values. In this case, I have x2 + x3 = 1, and statsmodels produced wrong results. 

In R, when x2 + x3 = 1, it will drop x3, fit the model with x2 only, and give NA to x3. I would imagine R implementation is less misleading for most of the users. 

Is there a way that I can get similar singularity handling as R using statsmodels?

<img width="378" alt="Screen Shot 2023-02-13 at 5 28 43 PM" src="https://user-images.githubusercontent.com/58790000/218597527-48da4d49-23f7-4e59-842b-23e9c8df2524.png">
<img width="407" alt="Screen Shot 2023-02-13 at 5 39 33 PM" src="https://user-images.githubusercontent.com/58790000/218598985-f69804dc-d23e-45f6-8b8e-1e03068be69d.png">

user: JasonTan-code
issue_8668: state: open
title: OLS P value not consistent with t value 
body: After running a simple univariate linear regression using OLS I obtain the following model summary

![image](https://user-images.githubusercontent.com/62406542/218344585-07ad7c81-b68d-4505-a1dd-c7c48030b909.png)

The t value is consistent with the t value I calculate manually when I use the R-squared value given. However a |t| of 2.76 in no way corresponds to a p value of 0.010. I am not able to replicate the exact value obtained from model.pvalues using any number of degrees of freedom.  

The code I use to create this model is:

`model = sm.OLS(list(DF[ind]),sm.add_constant(list(DF[de]))).fit()`

where 

`list(DF[ind])` = [1, 0, 0, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1]

and 

`list(DF[de])` = [7087.9, 5820.3, 5726.5, 5092.200000000001, 6391.8, 8094.1, 6491.1, 6852.700000000001, 7096.9, 7336.2, 6315.3, 5800.7, 6961.6, 6389.700000000001, 7977.0, 7199.799999999999, 7672.700000000001, 5165.8, 5243.7, 6628.2, 6309.6, 8531.7, 5336.9, 6744.6, 6522.7, 7255.200000000001, 6494.5, 4757.799999999999, 3726.5, 7140.3, 5000.799999999999]

The exact p value it gives is 0.009916977768826793 where according to that t-statistic it should be 0.11065255490834112.

The closest I can get to the given p value is using a DF of 10 which gives 0.010067636348152975 even though it seems as though the DF should just be 1, and given the coefficient of -0.002 it should certainly not be significantly different than 0.


#### Output of ``import statsmodels.api as sm; sm.show_versions()``

INSTALLED VERSIONS
------------------
Python: 3.11.1.final.0
statsmodels
===========
Installed: 0.13.5 (C:\Users\gavin\PycharmProjects\FTDNS\venv\Lib\site-packages\statsmodels)
Required Dependencies
=====================
cython: Not installed
numpy: 1.24.1 (C:\Users\gavin\PycharmProjects\FTDNS\venv\Lib\site-packages\numpy)
scipy: 1.10.0 (C:\Users\gavin\PycharmProjects\FTDNS\venv\Lib\site-packages\scipy)
pandas: 1.5.2 (C:\Users\gavin\PycharmProjects\FTDNS\venv\Lib\site-packages\pandas)
    dateutil: 2.8.2 (C:\Users\gavin\PycharmProjects\FTDNS\venv\Lib\site-packages\dateutil)
patsy: 0.5.3 (C:\Users\gavin\PycharmProjects\FTDNS\venv\Lib\site-packages\patsy)
Optional Dependencies
=====================
matplotlib: 3.6.3 (C:\Users\gavin\PycharmProjects\FTDNS\venv\Lib\site-packages\matplotlib)
    backend: module://backend_interagg 
cvxopt: Not installed
joblib: 1.2.0 (C:\Users\gavin\PycharmProjects\FTDNS\venv\Lib\site-packages\joblib)
Developer Tools
================
IPython: Not installed
    jinja2: 3.1.2 (C:\Users\gavin\PycharmProjects\FTDNS\venv\Lib\site-packages\jinja2)
sphinx: Not installed
    pygments: Not installed
pytest: Not installed
virtualenv: Not installed


user: gavinkress
labels: type-invalid: invalid bug reports, not a bug or issue
issue_8667: state: open
title: DescrStatsW.quantile() function produce inconsistent results 
body: #### Describe the bug

The quantile() function returns different arrays depending on whether the `probs ` are specified by a manually typed array or an array generated by `np.linspace()`

#### Code Sample, a copy-pastable example if possible


```python
from statsmodels.stats.weightstats import DescrStatsW
import numpy as np

wq = DescrStatsW(np.linspace(1,25,25), weights=[10]*25)
print(wq.quantile(probs = np.linspace(0, 1.0, 6)))
print(wq.quantile(probs = np.array([0, 0.2, 0.4, 0.6, 0.8, 1.0])))
```
<details>

When using `np.linspace(0, 1.0, 6)` to specify the probs, the function returns unexpected results: the 60% quantile is returned as 16.0 rather than 15.5. 

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

p
0.0     1.0
0.2     5.5
0.4    10.5
0.6    15.5
0.8    20.5
1.0    25.0

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

p
0.0     1.0
0.2     5.5
0.4    10.5
0.6    16.0
0.8    20.5
1.0    25.0
dtype: float64
p
0.0     1.0
0.2     5.5
0.4    10.5
0.6    15.5
0.8    20.5
1.0    25.0
dtype: float64

</details>

user: godoodim
issue_8660: state: open
title: ENH: test_proportions_2indep, method = 'logit-smoothed' is not vectorized
body: see #8332

currently raises ValueError in this case.
user: josef-pkt
issue_8655: state: open
title: REF/TST: check rootfinder in power classes
body: I wrote the rootfinder for solve_power in the power classes a long time ago.
scipy behavior has changed and new solvers became available.

The rootfinder would need a check for scipy compatibility and possible improvements.

There is currently no specific problem or bug with it.

However, I just saw a warning running the unit tests

```
statsmodels/stats/tests/test_power.py::test_power_solver_warn
  ...\statsmodels\stats\power.py:108: RuntimeWarning: invalid value encountered in sqrt
    pow_ = stats.norm.sf(crit - d*np.sqrt(nobs)/sigma)
```

and the unit test has the following comments
```
# TODO: can something useful be made from this?
@pytest.mark.xfail(reason='Known failure on modern SciPy >= 0.10', strict=True)
def test_power_solver_warn():
    # messing up the solver to trigger warning
    # I wrote this with scipy 0.9,
    # convergence behavior of scipy 0.11 is different,
    # fails at a different case, but is successful where it failed before
```
user: josef-pkt
issue_8654: state: open
title: SUMM/ENH/REF: statsmodels 1.0
body: Statsmodels 1.0 is still far in the future

Some thoughts on wishlist and requirements

- consistent returns of function, e.g. HolderTuple
- consistent methods and attributes across models
  - for and with consistent post-estimation
  - we might still need categories of models, eg. MLE versus moment estimators, OLS/WLS, GLM and GMM 
- flake and pep8 (but no "black"ening of code that I maintain.
  - related: code patterns. LGTM/CodeQL has a large number of false positives. Do we want to simplify some code to make the "bot" happy?  
- stable frameworky classes, 
  - GenericLikelihoodModel (or alternative new class)
  - GMM
  - Multilink and multipart models (generic class still missing)
- missing core methods and models, especially those that need to be based on new pattern and structure
  - generic multilink models
  - generic multipart models (not clear what we need), as in ZI, hurdle models, Copula models, generic truncated/censored or inflated models. (several submodels)
  - multi-endog and multivariate models, copula regression models would be multi-endog and multi-part
  - endogeneity and two step models, IV 
  - there is some overlap in the latter 3 model categories in either data handling, model structure or results structure.
- code cleanup
  - public, and commonly used methods (models, functions): those should be ok after refactoring for consistency, unit test coverage is ok
  - public methods with unused (?) extensions: need more unit test and verify methods. This affects mainly inherited methods in new classes and methods that come from combining different options. e.g. var_weights and freq_weights in GLM are not supported for all cov_types and post-estimation results.
  - public models or functions not advertised. Some parts look public but are not in docs and might have insufficient unit tests.
  - private methods. In some case we don't clearly distinguish between private and public methods. Either privatize some or make them stable (stable api, sufficient unit test coverage). This applies to many helper and tool functions that were created for internal support but have a public name.
  - unused code: remove it unless there is a strong reason to keep it. e.g. sandbox. 
    (This will be a lot of work if we want to publish sandbox code that is 70% or 80% finished. e.g. I don't want to throw away sandbox.distributions unless superseded by scipy or newer statsmodels code.)
- tsa
  - no idea 
- ...

I think we can do most over several pre-1.0 releases so that 1.0 would not have a large backwards compatibility impact.

one example for CodeQL problem and inconsistent returns: 
We have several functions where the number of returns depends on an option. CodeQL cannot figure out in all cases what the appropriate tuple length is, and then reports false positives. If we always return a HolderTuple or similar, then that would not be the case. (This is in functions that were written a long time ago.)

user: josef-pkt
issue_8653: state: open
title: BUG/REF: OLS, WLS hessian_factor unused keyword arguments
body: (found in random browsing of docs,code)

hessian_factor in linear model has scale and observed keywords, both are unused.
Docstring of method looks like it was copied from GLM.

The scaled hessian_factor would need an estimate of scale, which we don't have.
i.e. this is least squares and not MLE.

Not sure what we need to change besides docstring.
we might need the keywords for consistency with GLM. We need to check where it's used.


user: josef-pkt
issue_8652: state: open
title: ENH/REF/SUMM: enhance, refactor power classes
body: triggered by #8646 and #8651
see also #8159 for power classes without effect size
related issues: ...

The semi-generic power classes were written initially based on effect sizes and packages GPower and R pwr.
Design decisions were based on those packages with the generic structure it is often not "obvious" how to use those.

Target is to make them more directly usable and extend them to new cases, with possibly test specific power classes.

This should be based more on NCSS/PASS and new Stata pss, than on the previous packages.
The more recent power function, especially for cases where var/std differs between null and alternative as in rates and proportions, where heavily based on the NCSS/PASS docs.

Additionally, I want 
- power classes or/and effect sizes based on test statistics, tstat, fstat, ..., with normalized noncentrality, nc / nobs.
  - main difference is e.g. using std of test statistic instead of std of population as in Cohen's d family. 
- ...

not clear yet
- options not yet included, e.g. unequal var in t_test
- more general: power computation that are specification robust, 
   e.g. robust cov_type, excess dispersion in poisson, kurtosis in variance hypothesis tests
- ...

specific todos

- review existing power classes for hidden assumptions especially what special cases they are designed for
  - FTestPower, see comments in #8646 
  - TTestIndPower: assumes equal var, and cohen's d effect size
- maybe distinguish more clearly between keyword we can solve for and keywords that define setting or hypothesis test. For example we will likely need a `method` argument if we make classes for recently added power functions like those for rates and proportions.
- power classes for recently added power functions, rates, proportions, variances, ...
- power classes for TOST and other hypothesis tests that are not point hypothesis
- ...
- ...

I guess (not checked again): The basic power classes for one sample TTestPower can be used for generic case if std in effect size is the std of the (unstandardized) test statistic.
Why is there currently no NormalPower class? We only have NormalIndPower with same equal var assumption as TTestIndPower.
**update**
NormalIndPower can be used for one sample test if ratio=0
```
            ``ratio`` can be set to zero in order to get the power for a
            one sample test.
```
It wouldn't cost much to add a specific NormalPower class.
Aside: 
NormalIndPower has an option in the `__init__` `self.ddof = ddof` instead of as method keyword.
It's the only power class with an `__init__` method
user: josef-pkt
issue_8651: state: open
title: BUG/REF/SUMM: problem I mix up df_num and df_denom very often
body: AFAICS, FTestPower has df_num and df_denom reversed

Note:
ratio = numerator / denominator

scipy F distribution  has dfn and dfd as shape parameters

in testing constraints
df_num = df_constraints = number of constraints
df_denom = df_resid of full model, nobs - k

TBC
user: josef-pkt
issue_8650: state: open
title: SUMM/DOC: functions and classes not in docs
body: We don't have a central summary issue for things, functions and classes, that are left out of the docs, or those are spread out over issues and comments.
(I don't have an overview.)
Some might be left out intentionally, many might have been left out by accident.
We should make a note of those, and might require checking of the api and unit test coverage of those functions and classes.

Note, private methods and functions are mostly left out intentionally.

- [ ] impute_ros #7961
- [ ] ...

Aside: I had written scripts/notebooks to crawl the modules for content. (mostly notebooks for specific cases and not reusable and runnable scripts)
I had used it to compare with `__all__` in modules and with `api` modules.
We need something similar for comparing module content with doc toc content.




user: josef-pkt
issue_8649: state: open
title: ENH: Add multivariate pdf (dim>2) for Frank and Gumbel copula
body: - [x] closes #8631
- [x] tests added / passed. 
- [x] code/documentation is well formatted.  
- [ ] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

I implemented multivariate copula density for Frank and Gumbel (archimedian) using https://arxiv.org/abs/1207.1708

I did some optimizations for the _special module for sterling numbers.
Regarding the polylogarithm: I hope it is ok, that I added an import from package mpmath because this implementation is much more stable numerically.

Tests were added for higher dimensions. However I am not totally sure if I did this the intended way.
user: Tobi0
issue_8647: state: open
title: DOC: improve docs for power classes
body: power functions and classes are difficult to use because there is not clear description of which effect size is required.

todo:

- add notebook for standard cases, several issues for missing notebooks (1 PR)
- add to docstrings for effect size which effect size is required for the power function https://github.com/statsmodels/statsmodels/issues/8646#issuecomment-1416132114
- examples in docstrings, or related functions (e.g. oneway has power example in effectsize function)

#8162 incomplete overview issue, what's available

related
a stats.stackexchange answer for effectsize in FTestAnovaPower
https://stats.stackexchange.com/questions/544027/different-results-for-anova-power-calculation-by-r-and-statsmodels/544144#544144



user: josef-pkt
issue_8645: state: open
title: ENH: add force_uniform to BernsteinDistribution from_data
body: If I want an empirical Berstein copula, then it should have uniform margins.

BernsteinDistribution from_data currently does not have and option to impose uniform margins
`force_uniform` is option in distribution tools `approx_copula_pdf`

example: using rvs from a copula to generate data
rvs is sample drawn from gaussian copula k_dim=3


```
from statsmodels.distributions.bernstein import BernsteinDistribution
import statsmodels.distributions.tools as smdt

bern = BernsteinDistribution.from_data(rvs, 4)
pdfb = smdt.cdf2prob_grid(bern.cdf_grid, prepend=None)
freq = smdt.frequencies_fromdata(rvs, 4, use_ranks=True) / nobs

pdfb.sum(0).sum(1), freq.sum(0).sum(1)
(array([0.2492, 0.2544, 0.2538, 0.2426]), array([0.25, 0.25, 0.25, 0.25]))

bern_m2 = bern.get_marginal(2)
np.diff(bern_m2.cdf_grid)
array([0.2598, 0.2506, 0.2468, 0.2428])
```

user: josef-pkt
issue_8644: state: open
title: ENH: correlation, concordance for discrete distributions, pearson, spearman, kendall
body: I want to know what the 3 correlation coefficients are for a bivariate multinomial distribution given by probabilities.
Need some helper functions for this.

usecase: checking correlation, rho, tau of discretized copula in PR #8642
actually I have histogram copula, but continuous case might be difficult, too much numerical integration (for stepwise constant pdf)

some references, I did not look at them

Bouezmarni, Taoufik, Mhamed Mesfioui, and Abdelouahid Tajar. “On Concordance Measures for Discrete Data and Dependence Properties of Poisson Model.” Journal of Probability and Statistics 2009 (January 27, 2010): e895742. https://doi.org/10.1155/2009/895742.

Mesfioui, Mhamed, and Jean-François Quessy. “Concordance Measures for Multivariate Non-Continuous Random Vectors.” Journal of Multivariate Analysis 101, no. 10 (November 1, 2010): 2398–2410. https://doi.org/10.1016/j.jmva.2010.06.011.

Perrone, Elisa, Edwin R. van den Heuvel, and Zhuozhao Zhan. “Kendall’s Tau Estimator for Bivariate Zero-Inflated Count Data.” arXiv, August 5, 2022. https://doi.org/10.48550/arXiv.2208.03155.

user: josef-pkt
issue_8643: state: open
title: Are out of sample predictions supported in UECM?
body: Currently, it doesn't seem UECM is capable of out of sample prediction

```
from statsmodels.tsa.api import UECM
from statsmodels.datasets.danish_data import load
from statsmodels.tsa.api import ARDL
from statsmodels.tsa.ardl import ardl_select_order

data = load().data
data = data[["lrm", "lry", "ibo", "ide"]]

import datetime
cutoff_dt = datetime.datetime(1986,9,1)

test = data.loc[data.index > cutoff_dt]
data = data[data.index <= cutoff_dt]

sel_res = ardl_select_order(
    data.lrm, 3, data[["lry", "ibo", "ide"]], 3, ic="aic", trend="c"
)

ecm = UECM.from_ardl(sel_res.model)
ecm_res = ecm.fit()
ecm_res.summary()
ecm_res.forecast(steps=3, exog=test[["lry", "ibo", "ide"]])   # NotImplementedError: Out-of-sample forecasts are not supported
ecm_res.predict(start=data.shape[0], end=data.shape[0]+test.shape[0], exog_oos=test[["lry", "ibo", "ide"]]) # NotImplementedError: Out-of-sample forecasts are not supported
```

Is it truly not supported, or am I doing something wrong?

user: fcela
issue_8640: state: open
title: Adding scipy.sparse.linalg.svd as an option for OLS
body: #### Is your feature request related to a problem? Please describe
While doing regressions with large matrices, using standard svd from numpy uses too much RAM. From this [post](https://fa.bianp.net/blog/2012/singular-value-decomposition-in-scipy/) using scipy.sparse.linalg.svds can also be a solution that uses less RAM. 
 
#### Describe the solution you'd like
Add an option to choose the svd algorithm as an optional parameter when using OLS.

#### Describe alternatives you have considered
I am not aware of any other optimizations. Feel free to suggest other ways to improve RAM usage.

#### Additional context
I can't do OLS with a 120k datapoints with 8GB RAM.
user: drdebmath
issue_8639: state: open
title: ENH: meta-analysis for risk difference
body: (parking a reference)

This article looks like it has a good overview of methods for meta-analysis of risk difference, based on a very brief look at the article.

Guo, Juanru, Mengli Xiao, Haitao Chu, and Lifeng Lin. “Meta-Analysis Methods for Risk Difference: A Comparison of Different Models.” Statistical Methods in Medical Research 32, no. 1 (January 1, 2023): 3–21. https://doi.org/10.1177/09622802221125913.

related issues
#7184 2x2xk stratified proportions
#6585 basic meta-analysis issues

user: josef-pkt
issue_8638: state: open
title: ENH: rvs simulate random variables uniform on simplex
body: used, e.g., for simulating multivariate archimedean copulas
found in some copula article (???)

need to use exponential

```
def runif_in_simplex(n):
  ''' Return uniformly random vector in the n-simplex '''

  k = np.random.exponential(scale=1.0, size=n)
  return k / sum(k)
```

https://stackoverflow.com/a/67202070/333700

maybe for some `statsmodels.distribution._tools` module with misc functions (for internal reuse or as recipes if inlined)

also available with dirichlet all params = 1
https://math.stackexchange.com/questions/32618/uniform-distribution-on-a-simplex-via-i-i-d-random-variables
numpy has a dirichlet in np.random



user: josef-pkt
issue_8637: state: open
title: DOC:  section for "generic" topics, fit, post-estimation, ...
body: Stata has description for generic, model independent topics.

We should have something similar for features that are common to many models.

The docs have a background section with just general comments and description of basic structure
https://www.statsmodels.org/dev/user-guide.html#background

We could add a separate section, or we add it to background section.

possible candidates are `fit`, `cov_type` and various post-estimation topics


Aside:
It would also be good to add a section "Watch out. Frequent beginner mistakes"
e.g. add_constant, GLM is one-parameter family (e.g. negbin does not estimate dispersion parameter)
exog is not transformed/changed, e.g. bad scaling, (near) singularity

user: josef-pkt
issue_8636: state: open
title: ENH: `fit` docstrings, generic cov_type docstring template
body: generic fit methods are still missing description for `cov_type` options

define it as a separate string that can be used in templated docstrings

we also need a pattern to add model specific cov_types to the fit docstrings

user: josef-pkt
issue_8635: state: open
title: ENH/SUMM: extensions to copulas, missing parts
body: (need a list of parts missing or that we or I want to have)


- extend archimedean to k_dim > 2, see #8631, 
  - [x] pdf k_dim <= 4 #8633
  - [ ] pdf #8649
  - [x] rvs, #8642
  - [ ] subset, marginal conditional
  - [ ] ...
- [ ] extreme value
   - [ ] rvs
   - [ ] conditional u2 | u1 
   - [ ] ...
- [ ] Bernstein, advertise, examples, usage
- [ ] diagnostics for multivariate copulas e.g. (*1)
- [ ] corner cases, methods not defined at boundaries, needs checking, unit tests, raising exceptions(?)
  - [x] #8662 
  - [ ] cdf at a zero in archimedean
  - [ ] limit of copula args towards independence  (e.g. R switches to independence copula by default at independence arg)
  - [ ] negative correlation if k_dim=2, check, enforce valid range of copula args.
- [ ] ...

general formulas, generic implementations
e.g. 
`sf` survival copula
numerical derivatives as fallback
conditional, and marginal subset formulas

*1 
Marius Hofert & Martin Mächler (2014) A Graphical Goodness-of-Fit Test for
Dependence Models in Higher Dimensions, Journal of Computational and Graphical Statistics,
23:3, 700-716, DOI: 10.1080/10618600.2013.812518
https://doi.org/10.1080/10618600.2013.812518

and section in their book for copulas in R

user: josef-pkt
labels: roadmap: roadmaps, list of todos, overall and by topic
issue_8631: state: open
title: Copula density not correct for d>2
body: It seems that the pdf of some Archimedian Copulas are incorrect for d>2. They do not throw an error or warning in this case. The formulas look generic for any dimension at first but are restricted to d=2. At least for Gumbel and Clayton.
I compared with this: https://arxiv.org/pdf/1207.1708.pdf
For Frank there is a dim check ("pdf is currently only available for bivariate copula").

```
from statsmodels.distributions.copula.archimedean import GumbelCopula
c=GumbelCopula()
c.pdf([.7,.7,.7],20)
> array([3.97063356e-08])
```
For example in this case, a high value for the density would be expected.

Also it would be good to know the source where the formulas have been taken from.

user: Tobi0
labels: type-bug-wrong: serious bugs that silently return incorrect numbers
issue_8629: state: open
title: Statsmodels to also support Dask dataframes.
body: I am trying statsmodels to fit my data to a Logistic Regression model (Logit) but the dataframe I have is not a pandas dataframe but a Dask dataframe.

This is my sample dataset: `smarket_1`
**Response Variable:** `Direction`
```
+-------+------------+--------+--------+--------+--------+--------+--------+--------+-----------+
| const | Year       | Lag1   | Lag2   | Lag3   | Lag4   | Lag5   | Volume | Today  | Direction |
+-------+------------+--------+--------+--------+--------+--------+--------+--------+-----------+
| 0     | 1.0 2001.0 | 0.381  | -0.192 | -2.624 | -1.055 | 5.010  | 1.1913 | 0.959  | 1.0       |
+-------+------------+--------+--------+--------+--------+--------+--------+--------+-----------+
| 1     | 1.0 2001.0 | 0.959  | 0.381  | -0.192 | -2.624 | -1.055 | 1.2965 | 1.032  | 1.0       |
+-------+------------+--------+--------+--------+--------+--------+--------+--------+-----------+
| 2     | 1.0 2001.0 | 1.032  | 0.959  | 0.381  | -0.192 | -2.624 | 1.4112 | -0.623 | 0.0       |
+-------+------------+--------+--------+--------+--------+--------+--------+--------+-----------+
| 3     | 1.0 2001.0 | -0.623 | 1.032  | 0.959  | 0.381  | -0.192 | 1.2760 | 0.614  | 1.0       |
+-------+------------+--------+--------+--------+--------+--------+--------+--------+-----------+
| 4     | 1.0 2001.0 | 0.614  | -0.623 | 1.032  | 0.959  | 0.381  | 1.2057 | 0.213  | 1.0       |
+-------+------------+--------+--------+--------+--------+--------+--------+--------+-----------+
| 5     | 1.0 2001.0 | 0.213  | 0.614  | -0.623 | 1.032  | 0.959  | 1.3491 | 1.392  | 1.0       |
+-------+------------+--------+--------+--------+--------+--------+--------+--------+-----------+
| 6     | 1.0 2001.0 | 1.392  | 0.213  | 0.614  | -0.623 | 1.032  | 1.4450 | -0.403 | 0.0       |
+-------+------------+--------+--------+--------+--------+--------+--------+--------+-----------+
| 7     | 1.0 2001.0 | -0.403 | 1.392  | 0.213  | 0.614  | -0.623 | 1.4078 | 0.027  | 1.0       |
+-------+------------+--------+--------+--------+--------+--------+--------+--------+-----------+
| 8     | 1.0 2001.0 | 0.027  | -0.403 | 1.392  | 0.213  | 0.614  | 1.1640 | 1.303  | 1.0       |
+-------+------------+--------+--------+--------+--------+--------+--------+--------+-----------+
```

If I use the `Logit` class from statsmodels and fit my data:
```
from statsmodels.api import Logit

logistict_reg = Logit(endog = smarket_1['Direction'], exog = smarket_1.drop(labels= 'Direction', axis = 1)).fit()
logistic_reg.summary()
```
I am getting the below error saying:

```ValueError: unrecognized data structures: <class 'dask.dataframe.core.DataFrame'> / <class 'dask.dataframe.core.DataFrame'>```

Next, when I tried converting the dask dataframe to a pandas one using ```.compute( )``` as follows:
```
from statsmodels.api import Logit

logistict_reg = Logit(endog = smarket_1['Direction'], exog = smarket_1.drop(labels= 'Direction', axis = 1).compute()).fit()
```
I am getting the error saying:

```AttributeError: 'Index' object has no attribute 'equals'```

Now, since the `.compute( )` method here converts the dask dataframe to a pandas one, statsmodels still complains about it (probably its the Dask ecosystem which is causing this issue).

I was wondering if statsmodels can also accept and work with the Dask dataframes too, so that the dataframes exceeding the size of the memory can also be worked with,.
user: AdarshNamdev
issue_8627: state: open
title: ENH/REF: refactor multitest to focus on pvalue correction
body: related issue #8619

We should update `multitest` to focus on the pvalue correction and not on rejection decisions as in original articles.

The interface, options and returns will change, so it's better to create a new function.
We cannot change returns in a backwards compatible way.

two tasks

- return HolderTuple with corrected pvalues as main return and extra info, e.g. for fdr, or bonferroni and sidak
   two stage fdr whould return more history besides alphastar, e.g. ri and ntest0 
- streamline code to remove code for rejection decision
- Enhancement: 
  - add "prior" null proportion, true fraction to (one-stage) fdr, and return prior-adjusted corrected pvalues.


to streamlining code:
fdr twostage needs to call fdrcorrection function only once, iteration only needs to update ri and ntest0 by adjusting alpha instead of adjusting corrected p-values



user: josef-pkt
issue_8625: state: open
title: Unable to add two random effects terms in mixedlm
body: #### Describe the bug

Hi,

I am trying to add two random effects terms in a linear mixed effects model using `mixedlm` function but I am running into errors.

Using the example data from tutorial https://www.statsmodels.org/dev/mixed_linear.html

```
import statsmodels.api as sm
import statsmodels.formula.api as smf

data = sm.datasets.get_rdataset("dietox", "geepack").data

md = smf.mixedlm("Weight ~ Time", data, groups=data[["Pig", "Litter"]])

mdf = md.fit()
```

I run into the following error:

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Input In [20], in <cell line: 1>()
----> 1 md = smf.mixedlm("Weight ~ Time", data, groups=data[["Pig", "Litter"]])
      3 mdf = md.fit()

File ~/anaconda3/lib/python3.9/site-packages/statsmodels/regression/mixed_linear_model.py:1046, in MixedLM.from_formula(cls, formula, data, re_formula, vc_formula, subset, use_sparse, missing, *args, **kwargs)
   1044 kwargs["exog_vc"] = exog_vc
   1045 kwargs["groups"] = groups
-> 1046 mod = super(MixedLM, cls).from_formula(
   1047     formula, data, *args, **kwargs)
   1049 # expand re names to account for pairs of RE
   1050 (param_names,
   1051  exog_re_names,
   1052  exog_re_names_full) = mod._make_param_names(exog_re_names)

File ~/anaconda3/lib/python3.9/site-packages/statsmodels/base/model.py:226, in Model.from_formula(cls, formula, data, subset, drop_cols, *args, **kwargs)
    220         design_info = design_info.subset(cols)
    222 kwargs.update({'missing_idx': missing_idx,
    223                'missing': missing,
    224                'formula': formula,  # attach formula for unpckling
    225                'design_info': design_info})
--> 226 mod = cls(endog, exog, *args, **kwargs)
    227 mod.formula = formula
    228 # since we got a dataframe, attach the original

File ~/anaconda3/lib/python3.9/site-packages/statsmodels/regression/mixed_linear_model.py:793, in MixedLM.__init__(self, endog, exog, groups, exog_re, exog_vc, use_sqrt, missing, **kwargs)
    789 self.k_params = self.k_fe + self.k_re2
    791 # Convert the data to the internal representation, which is a
    792 # list of arrays, corresponding to the groups.
--> 793 group_labels = list(set(groups))
    794 group_labels.sort()
    795 row_indices = dict((s, []) for s in group_labels)

TypeError: unhashable type: 'numpy.ndarray'
```

It works fine with one random term but doesn't work with two. Any suggestions?

Thank you!

user: diffusionstats
issue_8622: state: open
title: Double requirement given: scipy<1.10,>=1.3 (already in scipy<1.8,>=1.3, name='scipy')
body: 
Describe the bug
Collecting pmdarima
Downloading https://files.pythonhosted.org/packages/08/e8/7817b5359bbdee06b7902fb4629180c6d8bbe1a35e65f28c6fe67f76c2c3/pmdarima-2.0.2.tar.gz (630kB)
Collecting beniget~=0.4.0 (from pythran)
Downloading https://files.pythonhosted.org/packages/cc/4a/af3f1b3d00efd47309b7a0e28351e06453727fa55d9b3a45fd4b91031a63/beniget-0.4.1-py3-none-any.whl
Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from pythran) (1.17.3)
Collecting gast~=0.5.0 (from pythran)
Downloading https://files.pythonhosted.org/packages/5f/1c/b59500a88c5c3d9d601c5ca62b9df5e0964764472faed82a182958a922c5/gast-0.5.3-py3-none-any.whl
Collecting ply>=3.4 (from pythran)
Downloading https://files.pythonhosted.org/packages/a3/58/35da89ee790598a0700ea49b2a66594140f44dec458c07e8e3d4979137fc/ply-3.11-py2.py3-none-any.whl (49kB)
Collecting joblib>=0.11 (from pmdarima)
Downloading https://files.pythonhosted.org/packages/91/d4/3b4c8e5a30604df4c7518c562d4bf0502f2fa29221459226e140cf846512/joblib-1.2.0-py3-none-any.whl (297kB)
Collecting pandas>=0.19 (from pmdarima)
Downloading https://files.pythonhosted.org/packages/99/f0/f99700ef327e51d291efdf4a6de29e685c4d198cbf8531541fc84d169e0e/pandas-1.3.5.tar.gz (4.7MB)
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Collecting scikit-learn>=0.22 (from pmdarima)
Downloading https://files.pythonhosted.org/packages/75/44/074b780d8ac0b0899937e9b8ba6d5d8873a71b99aa915219251ef85a8890/scikit-learn-1.0.2.tar.gz (6.7MB)
Installing build dependencies: started
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: finished with status 'done'
Collecting scipy>=1.3.2 (from pmdarima)
Requirement already satisfied: setuptools!=50.0.0,>=38.6.0 in /usr/lib/python3/dist-packages (from pmdarima) (40.8.0)
Collecting statsmodels>=0.13.2 (from pmdarima)
Downloading https://files.pythonhosted.org/packages/a4/7d/a919dbad04ec31741eee6a7f7df4a9a9e57e3b863900e48b079a6b832aab/statsmodels-0.13.5.tar.gz (18.4MB)
Installing build dependencies: started
Installing build dependencies: finished with status 'error'
Complete output from command /usr/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-lsrupns_ --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools>=59.2.0 wheel>=0.37.1 cython>=0.29.32,<3 "oldest-supported-numpy; python_version!='3.10' or platform_system!='Windows' or platform_python_implementation=='PyPy'" "numpy==1.22.3; python_version=='3.10' and platform_system=='Windows' and platform_python_implementation != 'PyPy'" "numpy; python_version>='3.11'" "scipy>=1.3,<1.8; python_version=='3.7'" "scipy>=1.3,<1.9; python_version=='3.8' and platform_system=='Windows' and platform_machine=='x86'" "scipy>=1.3,<1.9; python_version=='3.9' and platform_system=='Windows' and platform_machine=='x86'" "scipy>=1.3,<1.10; (python_version>'3.9' or platform_system!='Windows' or platform_machine!='x86') and python_version<'3.12'" "scipy>=1.9; python_version>='3.12'" setuptools_scm[toml]>=7.0,<8:
Ignoring numpy: markers 'python_version == "3.10" and platform_system == "Windows" and platform_python_implementation != "PyPy"' don't match your environment
Ignoring numpy: markers 'python_version >= "3.11"' don't match your environment
Ignoring scipy: markers 'python_version == "3.8" and platform_system == "Windows" and platform_machine == "x86"' don't match your environment
Ignoring scipy: markers 'python_version == "3.9" and platform_system == "Windows" and platform_machine == "x86"' don't match your environment
Double requirement given: scipy<1.10,>=1.3 (already in scipy<1.8,>=1.3, name='scipy')

Command "/usr/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-lsrupns_ --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools>=59.2.0 wheel>=0.37.1 cython>=0.29.32,<3 "oldest-supported-numpy; python_version!='3.10' or platform_system!='Windows' or platform_python_implementation=='PyPy'" "numpy==1.22.3; python_version=='3.10' and platform_system=='Windows' and platform_python_implementation != 'PyPy'" "numpy; python_version>='3.11'" "scipy>=1.3,<1.8; python_version=='3.7'" "scipy>=1.3,<1.9; python_version=='3.8' and platform_system=='Windows' and platform_machine=='x86'" "scipy>=1.3,<1.9; python_version=='3.9' and platform_system=='Windows' and platform_machine=='x86'" "scipy>=1.3,<1.10; (python_version>'3.9' or platform_system!='Windows' or platform_machine!='x86') and python_version<'3.12'" "scipy>=1.9; python_version>='3.12'" setuptools_scm[toml]>=7.0,<8" failed with error code 1 in None

To Reproduce
RUN pip3 install pmdarima in Dockercompose or debian system

Versions
versions dating back to February-22-2022
Expected Behavior
The package to be installed successfully

Actual Behavior
exit with non-zero code: 1

Additional Context
I think the issue is with the the following command where scipy requirements are given twice:

Command "/usr/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-lsrupns_ --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools>=59.2.0 wheel>=0.37.1 cython>=0.29.32,<3 "oldest-supported-numpy; python_version!='3.10' or platform_system!='Windows' or platform_python_implementation=='PyPy'" "numpy==1.22.3; python_version=='3.10' and platform_system=='Windows' and platform_python_implementation != 'PyPy'" "numpy; python_version>='3.11'" "scipy>=1.3,<1.8; python_version=='3.7'" "scipy>=1.3,<1.9; python_version=='3.8' and platform_system=='Windows' and platform_machine=='x86'" "scipy>=1.3,<1.9; python_version=='3.9' and platform_system=='Windows' and platform_machine=='x86'" "scipy>=1.3,<1.10; (python_version>'3.9' or platform_system!='Windows' or platform_machine!='x86') and python_version<'3.12'" "scipy>=1.9; python_version>='3.12'" setuptools_scm[toml]>=7.0,<8" failed with error code 1 in None


user: kailasprasannakumar
issue_8617: state: open
title: DOC: review notebook weights in GLM
body: notebook for weights in GLM seems to have an outdated section

https://www.statsmodels.org/dev/examples/notebooks/generated/glm_weights.html#aggregated-data:-exposure-and-var_weights

`Note: LR test agrees with original observations, pearson_chi2 differs and has the wrong sign.`
plus investigation afterwards

based on a quick look pearson_chi2 is correct and agrees across equivalent representation with or without weights.
(need to go through notebook to check)


user: josef-pkt
issue_8610: state: open
title: ENH: GEE add use_t option for cov_params
body: GEE does not have use_t option, use_t = False is hardcoded in summary, no fit option

Monte Carlo comparison. they use both t and z/normal based tests and pvalues
t-test can be conservative with some of the bias corrections and small number of clusters

Also we might add more options for small sample adjusted, bias reduced cov_types
related to variants of cluster correlation robust sandwiches, CR3, ...

Li, Peng, and David T. Redden. “Small Sample Performance of Bias-Corrected Sandwich Estimators for Cluster-Randomized Trials with Binary Outcomes.” Statistics in Medicine 34, no. 2 (2015): 281–96. https://doi.org/10.1002/sim.6344

reference found in 
Huang, Shuang, Mallorie H Fiero, and Melanie L Bell. “Generalized Estimating Equations in Cluster Randomized Trials with a Small Number of Clusters: Review of Practice and Simulation Study.” Clinical Trials 13, no. 4 (August 1, 2016): 445–49. https://doi.org/10.1177/1740774516643498.

also compares several small sample corrections

user: josef-pkt
issue_8609: state: open
title: ENH: hc3 cov_type for GMM, GMMIVNonlinear
body: this article has pseudo-hatmatrix for small sample corrections for HC2, HC3, ...

moment condition `z (y - g(x, beta))` i.e. nonlinear IV, but not for generic nonlinear GMM

Lin, Eric S., and Ta-Sheng Chou. “Finite-Sample Refinement of GMM Approach to Nonlinear Models under Heteroskedasticity of Unknown Form.” Econometric Reviews 37, no. 1 (January 2, 2018): 1–28. https://doi.org/10.1080/07474938.2014.999499.

I did not find any HC3 equivalent for generic GMM, and nothing for more general non-gaussian MLE.
(Poisson, GLM use the IRLS, WLS hatmatrix)

user: josef-pkt
issue_8606: state: open
title: ENH: cluster bootstrap using weights
body: parking a reference for a useful approach to cluster bootstrap, "resampling" clusters

Instead of actually resampling clusters which would result in different nobs in replication samples, we can use weights by cluster.

Without computational shortcut, we can just use freq_weights in GLM or cluster weights in GEE and compute full model for each replication sample.
I don't know how much we gain in computational efficiency if we just adjust weights in a single model instance. We might only safe on model `__init__` setup cost.

Cheng, Guang, Zhuqing Yu, and Jianhua Z. Huang. “The Cluster Bootstrap Consistency in Generalized Estimating Equations.” Journal of Multivariate Analysis 115 (March 1, 2013): 33–47. https://doi.org/10.1016/j.jmva.2012.09.003.



user: josef-pkt
issue_8605: state: open
title: ENH: GEE for distributions that are not LEF/GLM-families
body: parking references

we already have several issues for variation on GEE/GLM including variance modeling #2898, #5674

Here the extension is to use "standard" GEE but with underlying score functions that are not coming from a LEF family, e.g. zero-inflated count models.

How do we reuse the pattern for GEE estimation with working correlation for score functions or moment conditions that do not come from GLM (mean, variance functions)?
I don't know what the theoretical properties are. Non-LEF, non-GLM do not have the consistency properties robust to misspecification. 

Hall, Daniel B, and Zhengang Zhang. “Marginal Models for Zero Inflated Clustered Data.” Statistical Modelling 4, no. 3 (October 1, 2004): 161–80. https://doi.org/10.1191/1471082X04st076oa.

Kong, Maiying, Sheng Xu, Steven M. Levy, and Somnath Datta. “GEE Type Inference for Clustered Zero-Inflated Negative Binomial Regression with Application to Dental Caries.” Computational Statistics & Data Analysis 85 (May 1, 2015): 54–66. https://doi.org/10.1016/j.csda.2014.11.014.

user: josef-pkt
issue_8601: state: open
title: Improvements to seasonal_decompose: specifying frequency and filling NaN 
body: #### Is your feature request related to a problem? Please describe
The `statsmodels.tsa.seasonal.seasonal_decompose` method throws errors for not specifying a frequency and having NaN values, which can be conveniently addressed within the method by inferring frequencies and `dropna()` respectively.

#### Describe the solution you'd like
Quality of life improvements: include the relevant pandas transformations within the `seasonal_decompose` method. Include detailed error messages when DateTimeIndex is broken, etc.

#### Describe alternatives you have considered
NIL

#### Additional context
I can work on this, but I would like to understand reasons behind the current implementation first.
user: statistactics
issue_8600: state: open
title: Feature Request: Addition of non-parametric test based on Chebyshev's Inequality
body: #### Is your feature request related to a problem? Please describe
Whenever I need to perform a non-parametric test using Chebyshev's Inequality, I have to write the equation myself. It would be good to have it as a part of statsmodel

#### Describe the solution you'd like
A function which takes in a distribution(array), a scalar value to be used for the test, the type of test(one tailed/two tailed), confidence and returns whether the scalar value is within the limits

#### Describe alternatives you have considered
Currently I have to write the code myself so I don't have any other alternative, would be better to have it as a part of statsmodel

#### Additional context
If anyone thinks this will be helpful, do let me know, I can raise a pull request.
user: oshin94
issue_8596: state: open
title: [ENH] Add CRV3 Inference via the Cluster Jackknife to OLS and WLS #8461
body: Hi all! 

This PR closes #8461. 

It adds support for CRV3 cluster robust inference for OLS and WLS via a cluster jackknife. 

In short, the PR adds a new `cov_type`, `cluster-jackknife`, which can be run in the following way: 

```python
crv3 = model.fit(cov_type='cluster-jackknife', cov_kwds={'groups': id})
```
I have updated the docstring in `linear_model.py`.

I have **not yet** added tests, as I'd first like to get feedback on where and how to test the new `cov_type` (and I also have to get up to speed with `pytest`) . There are multiple strategies how to test the new feature: 

- test against implementations in other languages (e.g. Stata's summclust, R's summclust, or R's [sandwich::vcovJK](https://sandwich.r-forge.r-project.org/reference/vcovJK.html)). 
- internal tests: under appropriate small sample corrections and when clusters are singletons, the cluster jackknife is equivalent to the HC3 estimator. 

Below is a small example of the ladder test, using the new `cluster-jackknife` covtype and `HC3` inference: 

```python
from statsmodels.regression.linear_model import OLS
from statsmodels.datasets import grunfeld
from statsmodels.tools.tools import add_constant
import pandas as pd
import numpy as np

dtapa = grunfeld.data.load_pandas()
dtapa_endog = dtapa.endog[:200]
dtapa_exog = dtapa.exog[:200]
exog = add_constant(dtapa_exog[["value", "capital"]], prepend=False)
N = exog.shape[0]
id = pd.Series(range(0, N))

res_hc3 = OLS(dtapa_endog, exog).fit(cov_type = "HC3")
res_crv3 = OLS(dtapa_endog, exog).fit(cov_type = "cluster-jackknife", cov_kwds={'groups': id})

res = pd.DataFrame({'tstat hc3:': res_hc3.tvalues * np.sqrt((N-1) / (N)), 'tstat crv3:' : res_crv3.tvalues})
res
#          tstat hc3:  tstat crv3:
# value     16.093573    16.093573
# capital    3.932688     3.932688
# const     -3.040458    -3.040458
```

Let me know which types of tests you'd like to see =) 
user: s3alfisc
issue_8595: state: open
title: statsmodels.tsa.x13.x13_arima_analysis & Change Specifications
body: #### Is your feature request related to a problem? Please describe
I want to use "X-13ARIMA-SEATS Seasonal Adjustment Program" via Python. I know statsmodels.tsa.x13.x13_arima_analysis is the Python wrapper.

Using statsmodels, is it possible to provide the inputs (specification) files into the Seasonal adjustment procedure? 
Eg, I want to add/drop certain outliers or regressors. How do I go about doing it in  statsmodels?

Thank  you

#### Describe the solution you'd like
A clear and concise description of what you want to happen.

#### Describe alternatives you have considered
A clear and concise description of any alternative solutions or features you have considered.

#### Additional context
Add any other context about the feature request here.
user: PL450
issue_8594: state: open
title: ENH: penalized mixin: custom penalization with model properties
body: (semi-random idea while browsing SAS docs for logistic and firth penalization)

related to Firth penalization #3561

reference
Heinze, G., and Schemper, M. (2002). “A Solution to the Problem of Separation in Logistic Regression.” Statistics in Medicine 21:2409–2419.

penalized loglikelihood is
log L*(beta)  = log L*(beta) + 1/2 log | I(beta)|
where I(beta) is the information matrix evaluated at beta

That means we need the penalization class to have access to the model so we can reuse the hessian (OIM or EIM)
This might currently be tricky or impossible because we define the penalization instance before creating a penalized model instance.

One possibility may be to assign a `_model` attribute to the penalty instance after creating the model. This makes it a bit circulate
The penalized model instance holds the penalty instance which holds the penalized model instance as attributes.
I guess the circularity cannot be avoided unless we use extra args in each method of the penalty class.

Note we need the hessian/EIM of the original model and not the penalized hessian of the penalized model.


 



user: josef-pkt
issue_8592: state: open
title: ENH: model averaging for GLM, discrete, nonlinear models, MLE
body: (parking some references)

previously I only looked at model averaging in linear models. There is also a literature for Poisson, discrete and other nonlinear models.

Zhang, Xinyu, and Chu-An Liu 2022 target prediction, models can be from different families, e.g. Logit and Poisson

Zhang, Xinyu, Dalei Yu, Guohua Zou, and Hua Liang. 2016 require one family, averaging is over different models for linear predictor (including or excluding some regressors). Aggregation weights minimize penalised likelihood similar to information criteria (AIC, BIC) using average k_params (weighted sum over included models) as effective number of parameters. 

I didn't read more than a tiny bit of skimming.

Feng, Yang, Qingfeng Liu, Qingsong Yao, and Guoqing Zhao. “Model Averaging for Nonlinear Regression Models.” Journal of Business & Economic Statistics 40, no. 2 (April 3, 2022): 785–98. https://doi.org/10.1080/07350015.2020.1870477.

Sun, Yuying, Yongmiao Hong, Shouyang Wang, and Xinyu Zhang. “Penalized Time-Varying Model Averaging.” Journal of Econometrics, December 2, 2022. https://doi.org/10.1016/j.jeconom.2022.09.007.

Zhang, Xinyu, and Chu-An Liu. “Model Averaging Prediction by K-Fold Cross-Validation.” Journal of Econometrics, May 16, 2022. https://doi.org/10.1016/j.jeconom.2022.04.007.

Zhang, Xinyu, Dalei Yu, Guohua Zou, and Hua Liang. “Optimal Model Averaging Estimation for Generalized Linear Models and Generalized Linear Mixed-Effects Models.” Journal of the American Statistical Association 111, no. 516 (October 1, 2016): 1775–90. https://doi.org/10.1080/01621459.2015.1115762.

Zhou, Jianhong, Alan T. K. Wan, and Dalei Yu. “Frequentist Model Averaging for Zero-Inflated Poisson Regression Models.” Statistical Analysis and Data Mining: The ASA Data Science Journal 15, no. 6 (2022): 679–91. https://doi.org/10.1002/sam.11598.

Zou, Jiahui, Wendun Wang, Xinyu Zhang, and Guohua Zou. “Optimal Model Averaging for Divergent-Dimensional Poisson Regressions.” Econometric Reviews 41, no. 7 (August 9, 2022): 775–805. https://doi.org/10.1080/07474938.2022.2047508.

user: josef-pkt
issue_8590: state: open
title: ccf only return half cross-correlation coefficient and thus sensitive to input order
body: #### Describe the bug

I use ccf to calculate cross-correlation coefficient between two time-series arrays, but found that only half correlation was given (if I'm using it correctly). Then the maximum coefficient is sensitive to the order of the two input arrays, as the peak correlation coefficient may not be capture in one order.

#### Code Sample, a copy-pastable example if possible
I have a matrix called "all wav", and I want to calculate the cross-correlation coefficient for two of them. Here are what I got in different order:
<img width="621" alt="image" src="https://user-images.githubusercontent.com/103055636/209016840-6fbcec24-9072-451e-aff2-8603cd094f2d.png">

#### Expected Output
Here is what I got directly with numpy.correlate:
<img width="929" alt="image" src="https://user-images.githubusercontent.com/103055636/209017179-978c1141-5314-45e7-9352-57c71d39871c.png">

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

statsmodels
===========

Installed: 0.13.5 

Required Dependencies
=====================

cython: Not installed
numpy: 1.22.4 
scipy: 1.9.1 
pandas: 1.4.4 
    dateutil: 2.8.2
patsy: 0.5.3

user: langchen2022
issue_8589: state: open
title: ENH: confint for probplots, qqplot
body: two related SUMM issues #2183 and #3981
(I didn't find a specific issue for confint, although I looked for those several times. AFAIR I never found reference numbers for unit tests)

https://stackoverflow.com/questions/74874235/confidence-interval-in-normal-q-q-plot-using-statsmodels/74874450
pinguin has confints for qqplot

I don't know how they define those or what method they use. It looks a bit like prediction intervals.





user: josef-pkt
issue_8587: state: open
title: X13ARIMA & statsmodels: X13NotFoundError: x12a and x13as not found on path
body: #### Describe the bug

0


I am on Window OS. I downloaded the Windows version of X13 software from https://www.census.gov/data/software/x13as.X-13ARIMA-SEATS.html#list-tab-635278563.

I would like to use it with my Python code as below.
But I get error:
X13NotFoundError: x12a and x13as not found on path. Give the path, put them on PATH, or set the X12PATH or X13PATH environmental variable.


#### Code Sample, a copy-pastable example if possible
import pandas as pd
from pandas import Timestamp

s = pd.Series(
    {Timestamp('2013-03-01 00:00:00'): 838.2,
 Timestamp('2013-04-01 00:00:00'): 865.17,
 Timestamp('2013-05-01 00:00:00'): 763.0})
     
     
import os
os.chdir(r'C:\Users\user-name\Downloads\x13as_ascii-v1-1-b59\x13as')

import statsmodels.api as sm
sm.tsa.x13_arima_analysis(s)


```python
# Your code here that produces the bug
# This example should be self-contained, and so not rely on external data.
# It should run in a fresh ipython session, and so include all relevant imports.
```
<details>

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `main`. If your problem has been fixed in an unreleased version, you might be able to use `main` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the main branch of this repository? It helps the limited resources if we know problems exist in the current main branch so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

A clear and concise description of what you expected to happen.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]

</details>

user: PL450
issue_8586: state: open
title: Update model with new data
body: I am trying to use seasonal_decompose to stream data is required to use a sliding window. Is there a way to update the model to take advantage of information outside the current sliding window? 

Thans 
user: miguelc3
issue_8583: state: open
title: How come the Poisson model and Tweedie model (with var.power =1 ) are producing different coefficients and results?
body: According to the definition of a Tweedie distribution, when the power parameter is 1, it is equivalent to a Poisson.  
However, when I run a Tweedie model with the var.power parameter set to 1, I'm getting different results than when I run a Poisson. Below is the code I'm using:

For Poisson: 
GLM_model = sm.GLM(df_train_y, sm.add_constant(df_train_X),
             family = sm.families.Poisson(link=sm.genmod.families.links.log()),
             offset = np.log(data_train[exposure_lst])).fit()

For Tweedie:
GLM_model = sm.GLM(df_train_y, #y
             sm.add_constant(df_train_X), #X
             family = sm.families.Tweedie(link=sm.genmod.families.links.log(),  var.power = 1),
             offset = np.log(data_train[exposure_lst])
             ).fit()

This is the output I get for Tweedie:

![image](https://user-images.githubusercontent.com/18639130/208212876-a234ac37-3c3c-4aa1-8fdb-15c4d4840cdc.png)



This is the output I get for Poisson:

![image](https://user-images.githubusercontent.com/18639130/208212925-237f9d78-a6e8-425a-90a6-f7b631ca19d3.png)


Can someone explain the reason for this?  Is this a mistake in the way I'm calling the models or are they implemented different? Thanks. 
user: vivektreddy
issue_8582: state: open
title: Q-Q plot file size scales linearly with data size
body: #### Describe the bug

It appears that the file size of Q-Q plots scales $\mathcal{O}(N)$ for data of size $N$.

#### Code Sample

```python
import matplotlib.pyplot as plt
import numpy as np
import statsmodels.api as sm
from statsmodels.graphics.gofplots import qqplot_2samples

for size in [1e2, 1e3, 1e4, 1e5, 1e6]:
    size = int(size)
    x = np.random.normal(loc=8.5, scale=2.5, size=size)
    y = np.random.normal(loc=8.0, scale=3.0, size=size)
    pp_x = sm.ProbPlot(x)
    pp_y = sm.ProbPlot(y)
    qqplot_2samples(pp_x, pp_y)
    plt.savefig(f"test_{size}.svg")
```
```console
$ ls -lh test_*.svg
-rw-r-----  1 ajstewart  staff    33K Dec 16 14:19 test_100.svg
-rw-r-----  1 ajstewart  staff   128K Dec 16 14:19 test_1000.svg
-rw-r-----  1 ajstewart  staff   1.0M Dec 16 14:19 test_10000.svg
-rw-r-----  1 ajstewart  staff    10M Dec 16 14:19 test_100000.svg
-rw-r-----  1 ajstewart  staff   102M Dec 16 14:20 test_1000000.svg
```

Note: the same behavior can be reproduced with `qqplot` or `qqplot_2samples`.

#### Expected Output

I would expect each plot to contain 100 points (1 for each quantile), but it seems to contain every single point in the dataset. This results in files that are too large to even render, making the plot rather useless for large datasets. Is this the intended behavior? I think it would be more useful to plot 100 points, or at least have an option to do so.

#### Versions

<details>

INSTALLED VERSIONS
------------------
Python: 3.10.8.final.0
OS: Darwin 21.6.0 Darwin Kernel Version 21.6.0: Thu Sep 29 20:13:56 PDT 2022; root:xnu-8020.240.7~1/RELEASE_ARM64_T6000 arm64
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8

statsmodels
===========

Installed: 0.13.5 (/Users/ajstewart/spack/opt/spack/darwin-monterey-m1/apple-clang-14.0.0/py-statsmodels-0.13.5-adxlc7oy3ktl4kh5vovnox6p4xfo42mu/lib/python3.10/site-packages/statsmodels)

Required Dependencies
=====================

cython: 0.29.32 (/Users/ajstewart/spack/opt/spack/darwin-monterey-m1/apple-clang-14.0.0/py-cython-0.29.32-ivsv2b4ous7evv2ubjgvg55kfhmipfha/lib/python3.10/site-packages/Cython)
numpy: 1.23.4 (/Users/ajstewart/spack/opt/spack/darwin-monterey-m1/apple-clang-14.0.0/py-numpy-1.23.4-lrwjrgphd7votelcqhtiofnwjc66qeu7/lib/python3.10/site-packages/numpy)
scipy: 1.9.3 (/Users/ajstewart/spack/opt/spack/darwin-monterey-m1/apple-clang-14.0.0/py-scipy-1.9.3-awasw63634ccwei2d4yzqawpemumjce3/lib/python3.10/site-packages/scipy)
pandas: 1.5.1 (/Users/ajstewart/spack/opt/spack/darwin-monterey-m1/apple-clang-14.0.0/py-pandas-1.5.1-ig3wajvrqdbhneo6hrxjjtqblcxjbm7b/lib/python3.10/site-packages/pandas)
    dateutil: 2.8.2 (/Users/ajstewart/spack/opt/spack/darwin-monterey-m1/apple-clang-14.0.0/py-python-dateutil-2.8.2-6jntibflip4pxdx4zcobobgq6uujtrth/lib/python3.10/site-packages/dateutil)
patsy: 0.5.2 (/Users/ajstewart/spack/opt/spack/darwin-monterey-m1/apple-clang-14.0.0/py-patsy-0.5.2-abnyxte3so75czuftdj4aqc2c34hri2f/lib/python3.10/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.6.2 (/Users/ajstewart/.spack/.spack-env/view/lib/python3.10/site-packages/matplotlib)
    backend: MacOSX 
cvxopt: Not installed
joblib: 1.2.0 (/Users/ajstewart/.spack/.spack-env/view/lib/python3.10/site-packages/joblib)

Developer Tools
================

IPython: 8.5.0 (/Users/ajstewart/.spack/.spack-env/view/lib/python3.10/site-packages/IPython)
    jinja2: 3.1.2 (/Users/ajstewart/.spack/.spack-env/view/lib/python3.10/site-packages/jinja2)
sphinx: 5.3.0 (/Users/ajstewart/.spack/.spack-env/view/lib/python3.10/site-packages/sphinx)
    pygments: 2.13.0 (/Users/ajstewart/.spack/.spack-env/view/lib/python3.10/site-packages/pygments)
pytest: 7.1.3 (/Users/ajstewart/.spack/.spack-env/view/lib/python3.10/site-packages/pytest)
virtualenv: Not installed

</details>

user: adamjstewart
issue_8581: state: open
title: ENH/TST: postestimation after fit_constrained, fit_zeros, fit_collinear
body: I never checked which postestimation results work after imposing constraints, i.e. cov_params is rank reduced.

- wald test should be fine, under the assumption that constrained params have bse=0
- wald test with robust cov_type, I don't know
- score_test: uses score_obs of full, unconstrained model, i.e. tests whether imposed constraints hold in the data
- get_prediction should be fine, it should not be affected by singular cov_params
- get_distribution: not affected, predict is not affected
- get_influence ???
- get diagnostic ???


user: josef-pkt
issue_8580: state: open
title: ENH: fit_collinear, unit tests and make public, advertise
body:  see also #4705 and #6935

`_fit_collinear` is a private method

We need something public that drops collinear columns.
Main work will be to add unit tests for extra keywords in `model.__init__` for `_fit_zeros` which is also private and not in docs.
possible solution: add public `fit_collinear` only to models that we have checked, unit tested, but not generically in `base`.

current fit_collinear uses a single QR decomposition.
In a multicollinearity PR, I also have sequential dropping, new QR after dropping a column (AFAIR)

I was trying out scipy pivoting QR `q, r, p = linalg.qr(a, mode='economic', pivoting=True)`
but again it looks kind of arbitrary we columns need to be dropped


user: josef-pkt
issue_8579: state: open
title: Method `predict` doesn't exist for class `VAR`
body: First of all, thank you guys for your excellent work with `statsmodels`. 

The title says it all: after fitting a VAR model, I cannot find the `predict` method, just the `forecast` and related methods. Is this expected behaviour? For instance, the `ARDL` class has a `predict` method after fitting, which allows for an easy check on in-sample prediction.

I'm running version 0.14.0.dev0+331.g581119c64. Thanks.
user: anthonybritto
issue_8578: state: open
title: [ENH] Add Wild Cluster Bootstrapping to `OLS`
body: This PR adds wild cluster bootstrapping as a `cov_type` into OLS based on the package [`wildboottest`](https://github.com/s3alfisc/wildboottest). 

This PR includes all functionality built into OLS with `cov_type` and `cov_params`. 

I added a test in `statsmodels/regression/tests/test_robustcov.py` and updated the docstring in `linear_model.py`.

A few caveats:

1. Estimating standard errors and confidence intervals  is not trivial in certain instances, so `summary` only includes t-stats and p-values for now.
2. The tests inherit from `CheckOLSRobustCluster` and `CheckOLSRobustNewMixin` and so tests fail for those reasons as well.
3. Not sure where to include this in the docs, as from what I can see, the robust covariance docs are usually generated from docstrings?

user: amichuda
issue_8576: state: open
title: SUMM: finite number of parameters of interest, large number of counfounders
body: Consider that we are only interested in a few parameters in a (linear or non-gaussian) regression model, but have a large number of possible extra explanatory variables (confounders/controls).

What's the implication for cov_params and inference for the parameter of interest if we use regularization or penalization on the extra counfounders or controls?

Both GAM and sure independence screening allow for unpenalized exog (parameters of interest).

I ran into this a few times, but never looked at the theory.

recent article with references

Galbraith, John W., and Victoria Zinde-Walsh. “Simple and Reliable Estimators of Coefficients of Interest in a Model with High-Dimensional Confounding Effects.” Journal of Econometrics 218, no. 2 (October 1, 2020): 609–32. https://doi.org/10.1016/j.jeconom.2020.04.031.
uses PCA on controls

an older article I saw some time ago
Belloni, Alexandre, Victor Chernozhukov, and Christian Hansen. "Inference on treatment effects after selection among high-dimensional controls." The Review of Economic Studies 81, no. 2 (2014): 608-650.

this looks also relatedd

Cattaneo, Matias D., Michael Jansson, and Whitney K. Newey. “Inference in Linear Regression Models with Many Covariates and Heteroscedasticity.” Journal of the American Statistical Association 113, no. 523 (July 3, 2018): 1350–61. https://doi.org/10.1080/01621459.2017.1328360.
without regularization of confounders

we need some explicit penalized models #7336 #4590 #7128 ... 
user: josef-pkt
issue_8575: state: open
title: SUMM/ENH: multivariate endog model and infrastructure
body: related

- #3137
- #4543
- #3153

case to get started is exog are exogenous, i.e. no structural equation modelling
However, we want to impose zero (or linear) restriction on B in y = B x

dual representation B versus params
e.g. 
multinomial/MNLogit and MANOVA use full 2-dim params
OrderedModel uses 1-dim params but predicts 2-dim probabilities.
SVAR has zero restrictions on "A_endog" and converts between 1-dim params and 2-dim parameter matrix "A" (or B ?)
IIRC, SVAR has already many of the pieces


alternative model 
flattened endog with cross-observation relationship
e.g. flattened panel or clustered data and stacked multi-equation models.

GMM: we can horizontally stack "equations" (as in treatment effect model) but we have no single prediction variable, i.e. predict might not be defined.
But we could have GMM models that define either univariate or multivariate predictions. Then we need to support those.


motivation:
I ran by chance into the SEM, covariance pattern analysis literature, recent articles with regularization/penalization, e.g. regularized GLS

starting points:
multivariateLS in support of ANOVA
extension to MLE with zero or linear restriction on individual parameters.

Manova has row or column parameter restriction for wald tests, but no wald tests for individual or raveled parameter.
MNLogit has ???

user: josef-pkt
issue_8574: state: open
title: ENH: simultaneous confint in get_prediction
body: (just a random thought)

we still have simultaneous confint in the sandbox.

simplest would be Bonferroni if we predict just a few observations. That would be the same as using alpha / n.

related: TBFound

aside:
for GLM, one parameter families, can we use endpoint transformation on the simultaneous confint of linpred?
(I don't see why not.)


user: josef-pkt
issue_8573: state: open
title: ENH: OLS get_prediction `which`, endog transform
body: Do we need more `which`, options for predicting other statistics in OLS/WLS?

example
nonlinear transformation of endog if user used that in the model, e.g. log(y) = xb + u

related
Do we need to refactor OLS/WLS/linear model prediction results class to make it more similar or consistent with newer generic prediction results classes?

What do we need to change to use linear prediction results class for other models, e.g. for RLM #8304
user: josef-pkt
issue_8572: state: open
title: DOC: raises and warns sections in docstrings
body: We have now several informative warnings for, e.g., problems with data.
Those would be worth adding to the docstring with some explanations providing hints for common problems.
https://numpydoc.readthedocs.io/en/latest/format.html#raises

This could provide a complement to error or warning message with more information, and alert users in advance.

example
dfuller raises if series is constant, same problem would apply if rmse=0 in more general cases. For those we don't have a specific exception.

ConvergenceWarning in generic fit. might be worth adding. but warning name and message are pretty obvious, generic docstring would not provide much more information.

main candidates might be ValueWarnings (at least of those I recently looked at)
user: josef-pkt
issue_8570: state: open
title: MAINT: Release 0.14
body: What's left to do?

release notes
user: josef-pkt
issue_8563: state: open
title: REF: GLMInfluence class uses old get_prediction, switch linear to which
body: GLMInfluence uses attributes of old get_prediction class that is returned if linear keyword is used.

need to switch or adjust to "which" keyword in get_prediction

see https://github.com/statsmodels/statsmodels/pull/8559#issuecomment-1339755447

user: josef-pkt
issue_8562: state: open
title: REF: BayesMixedGLM predict still uses linear=True
body: should switch to `predict(which="xxx")`

but I never looked at BayesMixedGLM predict before
user: josef-pkt
issue_8561: state: open
title: ENH: detectable proportion for given power, proportion_2indep
body: ```
Let's say I want to compare two proportions with a z-test and I use NormalIndPower.solve_power from the 
statsmodels library to solve for (standardized) effect size. 
How do I convert the standardized result back into a proportion? 
The standardization seems to be cohen's d?
```
https://twitter.com/ryanstravis

needs convenience function and/or more options for power_proportion_2indep.
We don't have Power classes yet for binomial proportions and poisson rates.

I guess it's just rootfinding on a nonlinear equation. But what parameters are given, and what are the assumptions?

related: #8159 and others (I guess)





user: josef-pkt
issue_8556: state: open
title: SimpleExpSmoothing Forecast Returns Same Values
body: Hello,

I ran the example code for simple exponential smoothing from the following documentation:
https://www.statsmodels.org/dev/examples/notebooks/generated/exponential_smoothing.html

The output forecast gives the same value within each forecast type.

```
In [9]: fcast1
Out[9]: 
2008-12-31    483.068515
2009-12-31    483.068515
2010-12-31    483.068515
Freq: A-DEC, Name: $\alpha=0.2$, dtype: float64

In [10]: fcast2
Out[10]: 
2008-12-31    501.837031
2009-12-31    501.837031
2010-12-31    501.837031
Freq: A-DEC, Name: $\alpha=0.6$, dtype: float64


In [11]: fcast3
Out[11]: 
2008-12-31    496.493489
2009-12-31    496.493489
2010-12-31    496.493489
Freq: A-DEC, Name: $\alpha=0.8920003192188526$, dtype: float64
```


I'm using version 0.13.5

**Why am I unable to duplicate your results?**


user: GratePic
issue_8553: state: open
title: ENH: add var_weights or weights to RLM
body: (just a random idea)

If we have prior information about heteroscedasticity, then we need to include var_weights in RLM.

e.g. observations are averages of subsamples with different nobs
or inherent heteroscedasticity as in GLM, discrete.

also we might want to downweigh x-outliers. (as in literature for GLM)
Then we need more general weights, like importance weights. How do those affect inference?

**update**

old issue for weights including RLM #505

I found issue searching for carroll ruppert because I looked at the reference again by chance

Carroll, Raymond J., and David Ruppert. "Robust estimation in heteroscedastic linear models." The annals of statistics (1982): 429-441.


user: josef-pkt
issue_8551: state: open
title: Explosive forecast when using MAdM (ETSModel)
body: #### Describe the bug

Hello, this may not be a bug but I need help trying to understand the behaviour of this specific model on this specific time series.
I currently try several ETS models and choose the one with the minimal AICc. For this time series, the MAdM method is chosen.

#### Code Sample, a copy-pastable example if possible


```python
import numpy as np
    import pandas as pd
    import matplotlib.pyplot as plt
    from statsmodels.tsa.exponential_smoothing.ets import ETSModel

    endog = np.array(
        [
            52.0,
            250.43,
            261.01,
            265.0,
            56.77,
            16.0,
            50.09,
            65.0,
            21.0,
            89.5,
            20.0,
            1.0,
            123.0,
            84.0,
            91.0,
            163.0,
            16.0,
            27.0,
            78.5,
            38.0,
            61.0,
            5.0,
            82.0,
            15.0,
        ]
    )

    fit = ETSModel(
        endog,
        error='mul',
        trend='add',
        seasonal='mul',
        damped_trend=True,
        seasonal_periods=12,
        initialization_method='heuristic',
    ).fit(full_output=False, disp=False)

    plt.plot(endog, label='Demand history')
    plt.plot(fit.fittedvalues, label='Fitted values')
    plt.plot(
        pd.Series(fit.forecast(12), index=range(24, 36)), label='Forecasted values'
    )
    plt.legend(loc='upper left')
```

I've added a simple plot to showcase the forecast results.
![Explosive_Forecast](https://user-images.githubusercontent.com/108351774/205330985-9cadf577-0d24-4fa3-8b04-27c8a1adf562.png)

Let me know if you need anything else to be able to explain why this behaviour occurs.
<details>

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `main`. If your problem has been fixed in an unreleased version, you might be able to use `main` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the main branch of this repository? It helps the limited resources if we know problems exist in the current main branch so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

A clear and concise description of what you expected to happen.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]
INSTALLED VERSIONS
------------------
Python: 3.9.12.final.0

statsmodels
===========

Installed: 0.13.2 (C:\Users\user\miniconda3\envs\development_FE\lib\site-packages\statsmodels)

Required Dependencies
=====================

cython: 0.29.32 (C:\Users\user\miniconda3\envs\development_FE\lib\site-packages\Cython)
numpy: 1.22.3 (C:\Users\user\miniconda3\envs\development_FE\lib\site-packages\numpy) 
scipy: 1.7.3 (C:\Users\user\miniconda3\envs\development_FE\lib\site-packages\scipy)  
pandas: 1.4.2 (C:\Users\user\miniconda3\envs\development_FE\lib\site-packages\pandas)

    dateutil: 2.8.2 (C:\Users\user\miniconda3\envs\development_FE\lib\site-packages\dateutil)
patsy: 0.5.2 (C:\Users\user\miniconda3\envs\development_FE\lib\site-packages\patsy)  

Optional Dependencies
=====================

matplotlib: 3.5.1 (C:\Users\user\miniconda3\envs\development_FE\lib\site-packages\matplotlib)
    backend: QtAgg 
cvxopt: Not installed
joblib: 1.1.0 (C:\Users\user\miniconda3\envs\development_FE\lib\site-packages\joblib)


Developer Tools
================

IPython: 8.4.0 (C:\Users\user\miniconda3\envs\development_FE\lib\site-packages\IPython)
    jinja2: 3.0.3 (C:\Users\user\miniconda3\envs\development_FE\lib\site-packages\jinja2)
sphinx: Not installed
    pygments: 2.11.2 (C:\Users\user\miniconda3\envs\development_FE\lib\site-packages\pygments)
pytest: 7.1.2 (C:\Users\user\miniconda3\envs\development_FE\lib\site-packages\pytest)

virtualenv: Not installed

</details>

user: kimolssonOPT
issue_8549: state: open
title: Non-normally distributed innovation/noise for ARIMA models
body: #### Is your feature request related to a problem? Please describe
As far as I can tell, ARIMA models are missing an option to specify the distribution of the innovation/noise. Specifically want to use a t distribution. If not possible, is there a Python or R package that allows it?

#### Describe the solution you'd like
`AutoReg` function should have an option to specify commonly used non-normal distributions for the innovation/noise.

#### Describe alternatives you have considered
None

#### Additional context

user: cryptoguy4561
issue_8548: state: open
title: Python does not seem to correctly report fitted values in statmodels ARIMA when differencing is involved
body: When fitting an ARIMA model using the statsmodels python implementation I see the following behaviour, python does not seem to correctly provide the values for the differenced lags. I am comparing the results with the ones obtained using the R ARIMA implementation.

```python
import pandas as pd

df=pd.read_csv('ffp2\datasets\euretail.csv')
df.index=pd.date_range(start=df['index'].str[:4].min(),freq='1Q',periods=df.shape[0])
df=df['value']

from statsmodels.tsa.arima.model import ARIMA
model=ARIMA(df, order=(0,1,1), seasonal_order=(0,1,1,4)).fit()
pd.concat((df,model.fittedvalues, model.resid), axis=1).rename(columns={0:'py_fit',1:'py_resid'}).head(10)
```
<details>

Scenario:
- Seasonal non-stationary data which requires
- Single differencing
- Seasonal first differencing of period 4
- We end up with a ARIMA(0,1,1)(0,1,1,4) model

Please note the issue occurs when using the python ARIMA implementation when importing from `statsmodels.tsa.arima.model import ARIMA` I've seen webtutorials where it seems to function correctly but they seem to be using the previous implementation `from statsmodels.tsa.arima_model import ARIMA`, this implementation seems to currently be deprecated https://github.com/statsmodels/statsmodels/issues/3884

Python fitted model below (for reference)
```
                                    SARIMAX Results                                    
=======================================================================================
Dep. Variable:                           value   No. Observations:                   64
Model:             ARIMA(0, 1, 1)x(0, 1, 1, 4)   Log Likelihood                 -34.642
Date:                         Thu, 01 Dec 2022   AIC                             75.285
Time:                                 14:09:56   BIC                             81.517
Sample:                             03-31-1996   HQIC                            77.718
                                  - 12-31-2011                                         
Covariance Type:                           opg                                         
==============================================================================
                 coef    std err          z      P>|z|      [0.025      0.975]
------------------------------------------------------------------------------
ma.L1          0.2903      0.155      1.872      0.061      -0.014       0.594
ma.S.L4       -0.6912      0.132     -5.250      0.000      -0.949      -0.433
sigma2         0.1810      0.034      5.316      0.000       0.114       0.248
===================================================================================
Ljung-Box (L1) (Q):                   0.25   Jarque-Bera (JB):                 1.91
Prob(Q):                              0.62   Prob(JB):                         0.38
Heteroskedasticity (H):               0.76   Skew:                            -0.22
Prob(H) (two-sided):                  0.54   Kurtosis:                         3.77
===================================================================================

Warnings:
[1] Covariance matrix calculated using the outer product of gradients (complex-step).
```

R model code for reference below
```r
chooseCRANmirror(graphics=FALSE, ind=70)
# # Install if needed
# install.packages("fpp2")
# install.packages("readr")
# install.packages("tsibble")
library("fpp2")
library("readr")
library("tsibble")

fit <- Arima(euretail, order=c(0,1,1), seasonal=c(0,1,1))
fitted(fit)
resid(fit)
```


</details>

#### Expected Output

I would expect R and Python implementations to provide much closer results, it seems as if the Python implementation might have a bug

```
            value      py_fit   py_resid      r_fit   r_resid
1996-03-31  89.13    0.000000  89.130000  89.078541  0.051459 <---- (Issue Here)
1996-06-30  89.52   89.130003   0.389997  89.496685  0.023315
1996-09-30  89.88   89.520000   0.360000  89.864432  0.015568
1996-12-31  90.12   89.879998   0.240002  90.143352 -0.023352
1997-03-31  89.19  134.685000 -45.495000  89.380354 -0.190354 <---- (Issue Here)
1997-06-30  89.78   89.579993   0.200007  89.621984  0.158016
1997-09-30  90.03   90.193547  -0.163547  90.164354 -0.134354
1997-12-31  90.38   90.222837   0.157163  90.251028  0.128972
1998-03-31  90.27   89.463010   0.806990  89.602346  0.667654
1998-06-30  90.77   90.951737  -0.181737  90.937687 -0.167687
1998-09-30  91.85   91.019609   0.830391  91.077857  0.772143
1998-12-31  92.51   92.389250   0.120750  92.397782  0.112218
1999-03-31  92.21   92.044928   0.165072  92.056141  0.153859
1999-06-30  92.52   92.752252  -0.232252  92.744481 -0.224481
1999-09-30  93.62   93.066856   0.553144  93.083940  0.536060
```
Dataset
[euretail.csv](https://github.com/statsmodels/statsmodels/files/10134041/euretail.csv)

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>



INSTALLED VERSIONS
------------------
Python: 3.9.6.final.0

statsmodels
===========

Installed: 0.13.5 (c:\Users\alber\Documents\forecasting-principles-and-practice\venv\lib\site-packages\statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.21.4 (c:\Users\alber\Documents\forecasting-principles-and-practice\venv\lib\site-packages\numpy)
scipy: 1.7.3 (c:\Users\alber\Documents\forecasting-principles-and-practice\venv\lib\site-packages\scipy)
pandas: 1.3.5 (c:\Users\alber\Documents\forecasting-principles-and-practice\venv\lib\site-packages\pandas)
    dateutil: 2.8.2 (c:\Users\alber\Documents\forecasting-principles-and-practice\venv\lib\site-packages\dateutil)
patsy: 0.5.2 (c:\Users\alber\Documents\forecasting-principles-and-practice\venv\lib\site-packages\patsy)

Optional Dependencies
=====================

matplotlib: 3.5.1 (c:\Users\alber\Documents\forecasting-principles-and-practice\venv\lib\site-packages\matplotlib)
    backend: module://matplotlib_inline.backend_inline 
cvxopt: Not installed
joblib: 1.1.0 (c:\Users\alber\Documents\forecasting-principles-and-practice\venv\lib\site-packages\joblib)

Developer Tools
================

IPython: 7.30.1 (c:\Users\alber\Documents\forecasting-principles-and-practice\venv\lib\site-packages\IPython)
    jinja2: Not installed
sphinx: Not installed
    pygments: 2.10.0 (c:\Users\alber\Documents\forecasting-principles-and-practice\venv\lib\site-packages\pygments)
pytest: Not installed
virtualenv: Not installed

</details>

user: agr5
issue_8541: state: open
title: BUG/ENH: add random state option to mice and mediation
body: It looks like in "pre"-testing, some random numbers are screwed up, not backwards compatible. I'm still not sure where the test failures come from. Maybe using a given user (unit test) defined random state will fix the issue. (but problem in those unit tests could also be somewhere else)
https://github.com/statsmodels/statsmodels/pull/8540#issuecomment-1329323850

Both mediation and mice use simulation in the estimation, but use only the global np random state.
We need option for users to provide random state, e.g. in `Mediation.fit`

oldest issue #306
discussion on defaults and new numpy random generators #7650

user: josef-pkt
issue_8536: state: open
title: ENH: helper functions loo, influence, residuals
body: (random thought)

related to outlier measures in outlier_influence
#7897 score residuals

Another idea for residuals is to look at loglikeobs. However we need to standardize it to have interpretable magnitude.

one possibility is use loo standard deviation, and maybe loo mean, for standardizing or zscoring.
(We might also want to use outlier robust variance measures if we have possibly many outliers)

Currently, we have explicit loo statistics in outlier_influence or Influence classes that do not require the loo loop.
Those are hardcoded for the predefined statistics.

Instead or as complement we could add standalone helper functions.
e.g.
var_loo, std_loo, mean_loo

resid_xxx_loo_zscored = (resid_xxx - mean_loo) / std_loo
where resid_xxx could, for example, be loglike_obs

"empirically standardized".
(I guess it's a bit similar to Vuong test for non-nested models)

simple case: assume data is given, e.g
mean_loo = (resid_xxx.sum() - resid_xxx) / (nobs-1)

case: taking `dparams` into account
???
We might have to compute all resid_xxx, e.g. loglikeobs or score_obs, at each params + dparams. That might not work without a nobs loop or (nobs, nobs) arrays.
for each i:
llo_i = loglikeobs(params + dparams[i]).sum() 
mean_loo[i] = llo_i.sum() - llo_i[i]) / (nobs - 1)


extra: out-of-sample computation (for control charts or cross-validation)
Standardizing needs to be a stateful transform, i.e. use sample mean and std to standardize out-of-sample observations.
However, that doesn't need a LOO because obs are out of sample already.

(maybe `_stdz` as postfix for names, for standardized, (empirically) zscored )

e.g.
should we provide standardized resid_pearson, but it already has restriction that mean is 1 if correctly specified.


aside, not clear to me:
Does inherent heteroscedasticity as in GLM/LEF like poisson induce heteroscedasticity in the resid_xxx, because higher moments of the distribution are also functions of mean.

maybe look at PIT residuals which should be uniform (under the null that model is correct specified for all obs)
pit_loo, i.e. ppf at params + dparams ?
  
user: josef-pkt
issue_8534: state: open
title: ENH: Likelihood ratio test for logistic regeression
body: #### Is your feature request related to a problem? Please describe
It is useful to compare the value of new predictors and their contribution to a model fit.

#### Describe the solution you'd like
Implementation of the likelihood ratio test for `BinaryResultsWrapper` object.

user: valentynbez
issue_8533: state: open
title: SUMM/ENH/Design overview control charts
body: trying again for an overall design or design requirements for control chart
main original issue #4191 plus 2 PRs for earlier implementation

newer topics
- control charts based on beta distribution or regression and other distributions on (0, 1) or [0, 1] 
- tolerance interval, currently only for poisson stats

**traditional control charts** 
one letter or short names, x, u, p, np, c, u, xbar, ...., based on k * sigma limits
we can add it mainly for familiarity by users 
almost one liners, but it would be good to have a collection
all that is needed is mean or expected value and variance of mean

more general and more interesting components

**quantiles**
use quantile estimate for control limits instead of k*sigma limits
either parametric or nonparametric/flexible distribution, misspecified model (eg. over dispersion)
with or without parameter uncertainty
simple example: one sample tolerance interval for Poisson
We could compare with QuantReg or M-quantiles using less restrictive distributional assumptions.

**parameter estimation using clean data**
`fit` assuming data does not have outliers or out of control observations
This can be either simple estimators like mean, or we connect to models, e.g. beta regression, or regression control charts.
Advantage of using models is that we have built-in inference, e.g. standard errors of parameter estimates, including misspecification robust inference, and get_distribution method.

**parameter estimation and outlier detection using dirty data**
`fit` assuming historical data might have outlier, out-of-control periods
- robust estimation, currently we only have RLM, but nothing for discrete, nonlinear or link models.
- outlier identification
   We have get_influence for some outlier candidate identification, but no robust estimation besides dropping outliers.
  GLM has freq_weights, that we could use to adaptively downweigh outlier candidates.
  We also need something for grouped outlier, e.g. several outlier periods in a row, need to look at outlier windows.  
  We can also reuse control chart itself iteratively to identify outliers, e.g. if we have tolerance intervals, ppf, isf, quantiles


I did not look at implementation issues again.

We could start with Poisson, tolerance interval versus k-sigma limits versus quantreg.
With overdispersion and count distributions that are alternatives to Poisson.
Example: u-chart versus Laney u' Chart (u Prime) versus NBP or GPP, versus zero-inflated, ...
u prime chart adds heterogeneity in mean across batches, (a bit similar to meta-analysis correction for random effects), e.g. empirical standard deviation versus standard deviation implied by the distribution.
https://www.qimacros.com/control-chart-formulas/u-prime-chart-formula/

more difficult multi-link or multi distribution parameter models
e.g. equivalence to tolerance interval or quantiles under parameter uncertainty for BetaModel.

user: josef-pkt
issue_8532: state: open
title: ENH: robust cov, HAC with uniform kernel for moving average, rollingols
body: What is the appropriate cov_params for moving average or rolling regression?

I'm looking at this in context of control charts with moving mean or moving aggregated counts.

If we assume that the individual observations are uncorrelated, then the moving averaging filter induces autocorrelation.
My guess is that we should use HAC with a uniform filter, or newey west filter truncated at window length.

AFAICS, RollingOLS only allows for HC robust cov_type, but does not include correlation ?

aside:
For control charts we can use either
- nonoverlapping windows, i.e. aggregate/average observations into batches, or
- overlapping, moving windows

The question is what the appropriate cov_params and inference is for the second case.

Also related, in peak over threshold in extreme value theory we use nonoverlapping windows with `max` aggregation.

technical aside:
linear filter changes autocorrelation, functions for this are in tsa_tools, AFAIR but I have not looked at those in a very long time.

user: josef-pkt
issue_8528: state: open
title: Setuptools update breaks install of statsmodels [urgent]
body: #### Describe the bug

Setuptools upgraded to 65.6.0 and removed distutils.log. So, we cannot install statsmodels. My code continuously fails.

A larger thread is here: https://github.com/pypa/setuptools/issues/3693

I recommend to quickly patch it by restricting setuptools to be 65.5.1.
user: tulbureandreit
issue_8526: state: open
title: STL implementation cannot handle missing values
body: #### Describe the bug

The current STL season-trend decomposition in statsmodels does not handle missing values from the input. This can be seen in the use of `ffil()` in the provided example, see [here](https://www.statsmodels.org/dev/generated/statsmodels.tsa.seasonal.STL.html#statsmodels.tsa.seasonal.STL). 

However, one of the properties of the original STL algorithm is that it can handle the missing values. See design goal 4 in Sect. 1.1 (Design goal) of the original article
> R. B. Cleveland, W. S. Cleveland, J.E. McRae, and I. Terpenning (1990) STL: A Seasonal-Trend Decomposition Procedure Based on LOESS. Journal of Official Statistics, 6, 3-73.

So I was wondering if this was intentional or a bug. Will it be implemented? If not, should we update the documentation to say that the algorithm has not been fully implemented? 

My understanding is that the algorithm is still widely used, so it would be beneficial to implement this part as well.

#### Code Sample, a copy-pastable example if possible


```python
from statsmodels.datasets import co2
import matplotlib.pyplot as plt
from pandas.plotting import register_matplotlib_converters
from statsmodels.tsa.seasonal import STL

register_matplotlib_converters()
data = co2.load().data
data = data.resample('M').mean()
res = STL(data).fit()
res.plot()
plt.show()
# As can be seen from the plot above, fitting fails without a warning for an input that is within the parameters of the original article.
```

#### Expected Output

Implement the nan fitting from the original article or at least clearly state in the documentation that it is not currently treated.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]

```
INSTALLED VERSIONS
------------------
Python: 3.8.13.final.0
OS: Darwin 22.1.0 Darwin Kernel Version 22.1.0: Sun Oct  9 20:14:30 PDT 2022; root:xnu-8792.41.9~2/RELEASE_ARM64_T8103 arm64
byteorder: little
LC_ALL: None
LANG: None
statsmodels
===========
Installed: 0.13.2 (/Users/user/miniforge3/envs/repo/lib/python3.8/site-packages/statsmodels)
Required Dependencies
=====================
cython: Not installed
numpy: 1.23.3 (/Users/user/miniforge3/envs/repo/lib/python3.8/site-packages/numpy)
scipy: 1.9.2 (/Users/user/miniforge3/envs/repo/lib/python3.8/site-packages/scipy)
pandas: 1.5.0 (/Users/user/miniforge3/envs/repo/lib/python3.8/site-packages/pandas)
    dateutil: 2.8.2 (/Users/user/miniforge3/envs/repo/lib/python3.8/site-packages/dateutil)
patsy: 0.5.3 (/Users/user/miniforge3/envs/repo/lib/python3.8/site-packages/patsy)
Optional Dependencies
=====================
matplotlib: 3.6.1 (/Users/user/miniforge3/envs/repo/lib/python3.8/site-packages/matplotlib)
    backend: module://backend_interagg 
cvxopt: Not installed
joblib: 1.2.0 (/Users/user/miniforge3/envs/repo/lib/python3.8/site-packages/joblib)
Developer Tools
================
IPython: 8.6.0 (/Users/user/miniforge3/envs/repo/lib/python3.8/site-packages/IPython)
    jinja2: 3.1.2 (/Users/user/miniforge3/envs/repo/lib/python3.8/site-packages/jinja2)
sphinx: Not installed
    pygments: 2.13.0 (/Users/user/miniforge3/envs/repo/lib/python3.8/site-packages/pygments)
pytest: 7.1.3 (/Users/user/miniforge3/envs/repo/lib/python3.8/site-packages/pytest)
virtualenv: 20.16.5 (/Users/user/miniforge3/envs/repo/lib/python3.8/site-packages/virtualenv)
INSTALLED VERSIONS
------------------
Python: 3.8.13.final.0
OS: Darwin 22.1.0 Darwin Kernel Version 22.1.0: Sun Oct  9 20:14:30 PDT 2022; root:xnu-8792.41.9~2/RELEASE_ARM64_T8103 arm64
byteorder: little
LC_ALL: None
LANG: None
statsmodels
===========
Installed: 0.13.2 (/Users/user/miniforge3/envs/repo/lib/python3.8/site-packages/statsmodels)
Required Dependencies
=====================
cython: Not installed
numpy: 1.23.3 (/Users/user/miniforge3/envs/repo/lib/python3.8/site-packages/numpy)
scipy: 1.9.2 (/Users/user/miniforge3/envs/repo/lib/python3.8/site-packages/scipy)
pandas: 1.5.0 (/Users/user/miniforge3/envs/repo/lib/python3.8/site-packages/pandas)
    dateutil: 2.8.2 (/Users/user/miniforge3/envs/repo/lib/python3.8/site-packages/dateutil)
patsy: 0.5.3 (/Users/user/miniforge3/envs/repo/lib/python3.8/site-packages/patsy)
Optional Dependencies
=====================
matplotlib: 3.6.1 (/Users/user/miniforge3/envs/repo/lib/python3.8/site-packages/matplotlib)
    backend: module://backend_interagg 
cvxopt: Not installed
joblib: 1.2.0 (/Users/user/miniforge3/envs/repo/lib/python3.8/site-packages/joblib)
Developer Tools
================
IPython: 8.6.0 (/Users/user/miniforge3/envs/repo/lib/python3.8/site-packages/IPython)
    jinja2: 3.1.2 (/Users/user/miniforge3/envs/repo/lib/python3.8/site-packages/jinja2)
sphinx: Not installed
    pygments: 2.13.0 (/Users/user/miniforge3/envs/repo/lib/python3.8/site-packages/pygments)
pytest: 7.1.3 (/Users/user/miniforge3/envs/repo/lib/python3.8/site-packages/pytest)
virtualenv: 20.16.5 (/Users/user/miniforge3/envs/repo/lib/python3.8/site-packages/virtualenv)
```
</details>

user: Alexander-Serov
issue_8525: state: open
title: import error
body: cannot import name '_maybe_get_pandas_wrapper_freq' from 'statsmodels.tsa.filters._utils' (C:\Users\ADMIN\anaconda3\lib\site-packages\statsmodels\tsa\filters\_utils.py). Could anyone assist me with this.

user: JaganS1985
issue_8524: state: open
title: Question: Converge issue of GenericLikelihoodModel
body: I have a GenericLikelihoodModel model that models this:

$$
P(Y=1|X) = \frac{1}{1 + e^{- \Sigma_{i=1}^n \beta_i X_i}} \cdot \left(p \cdot e^{–\gamma T} + (1 - p)\right)
$$

I have:
* Created data generated by this distribution
* Defined a GenericLikelihoodModel such that its likelihood is the one for the above data generation process.
* Fitted via likelihood estimation twice:
    * One with a set of parameters close to the real ones
    * Another with a worse initial guess of the parameters
Both models are terribly bad, none of them gets even close to the actual value of the parameters.

```python

import numpy as np
import pandas as pd
import statsmodels.api as sm
from statsmodels.base.model import GenericLikelihoodModel


def generate_probability_logistic(gamma, beta_0, beta_1, p, n_data):
    x = np.random.normal(size = n_data)
    treatment = np.random.uniform(size = n_data)
    simgoid_x = 1 / (1 + np.exp(-beta_0 - beta_1 * x))
    treatment_factor = (1 - p) + p * np.exp(-gamma * treatment)
    probabilities = simgoid_x * treatment_factor
    outcome = np.random.binomial(n = 1, p = probabilities)
    return pd.DataFrame(
        {
            "x": x,
            "treatment": treatment,
            "y": outcome
        }
    )


class ProbabilityLogistic(GenericLikelihoodModel):
    def loglike(self, params):
        p = params[-1]
        gamma = params[0]
        beta = params[1:-1]
        x = self.exog[:, 1:]
        treatment = self.exog[:, 0]
        y = self.endog
        non_treatment = (1 / (1 + np.exp(- np.sum(beta * x, axis=1))))
        treatment_factor = (1 - p) + p * np.exp(-gamma * treatment)
        p_y = non_treatment * treatment_factor
        y_hat = np.log(p_y) * y + np.log(1 - p_y) * (1 - y)
        return np.sum(y_hat)
    
    def fit(self, start_params = None, maxiter = 10000, maxfun = 5000, **kwds):
        if start_params is None:
            start_params = np.array([0, 0, 0, 0.5])
        return super().fit(start_params = start_params, maxiter = maxiter, maxfun = maxfun, **kwds)
    
real_params = {
    "gamma": 5,
    "beta_0": -1,
    "beta_1": 0,
    "p": 0.1
}

df = generate_probability_logistic(**real_params, n_data=100_000)


exog = df.loc[:, ["treatment", "x"]]
exog = sm.add_constant(exog, prepend=False)
endog = df.y

sm_model = ProbabilityLogistic(endog, exog).fit(
    start_params=np.array([
        real_params["gamma"] + 0.001, 
        real_params["beta_0"] - 0.001,
        real_params["beta_1"] + 0.001,
        real_params["p"] - 0.001
    ]),
    method='nm'
)
print("Starting with very close parameters to the true ones")
print(sm_model.summary())
sm_model_2 = ProbabilityLogistic(endog, exog).fit(
    start_params=np.array([0, 0.0, 0.0, 0.5]),
)

print("Not so close parameters to the true ones")
print(sm_model_2.summary())

```

Do you have any idea to make the fitting more stable? Am I doing something wrong? Thanks in advance!
user: david26694
issue_8520: state: open
title: ENH/TST/SUMM check all get_prediction methods outside tsa
body: (more general than #8519 specific for checking get_prediction across models)

based on the doc index the following classes outside tsa have a `get_prediction` method

so far I mainly checked discrete models, GLMResults and new models BetaModel and OrderedModel
OLS as original implementation
I have not recently checked any of the other ones, which might have an inherited method or inconsistent api

several models are also still missing the method, e.g. Mixed, RLM, .... (which ?)
what about WLS, GLS and their subclasses, they are not listed but might inherit 

base.model.GenericLikelihoodModelResult

discrete.count_model.ZeroInflatedGeneralizedPoissonResult
discrete.count_model.ZeroInflatedNegativeBinomialResult
discrete.count_model.ZeroInflatedPoissonResult
discrete.discrete_model.BinaryResult
discrete.discrete_model.CountResult
discrete.discrete_model.DiscreteResult
discrete.discrete_model.GeneralizedPoissonResult
discrete.discrete_model.LogitResult
discrete.discrete_model.MultinomialResult
discrete.discrete_model.NegativeBinomialResult
discrete.discrete_model.ProbitResult
discrete.truncated_model.HurdleCountResult
discrete.truncated_model.TruncatedLFPoissonResult
discrete.truncated_model.TruncatedNegativeBinomialResult

gam.generalized_additive_model.GLMGamResult
genmod.generalized_estimating_equations.GEEResult
genmod.generalized_linear_model.GLMResult

miscmodels.ordinal_model.OrderedResult
othermod.betareg.BetaResult

regression.linear_model.OLSResult
regression.linear_model.RegressionResult
regression.process_regression.ProcessMLEResult
regression.quantile_regression.QuantRegResult
regression.recursive_ls.RecursiveLSResult

sandbox.regression.gmm.IVRegressionResult

user: josef-pkt
issue_8519: state: open
title: ENH/TST: inherited post-estimation features for GLMGAM
body: from https://github.com/statsmodels/statsmodels/pull/8505#issuecomment-1312040074

I just saw that GAM inherits from GLM `GLMGamResults(GLMResults)`
get_prediction calls the super method
extra keywords are in kwargs, so `which` support and behavior should be inherited from GLM

needs specific unit tests
get_distribution and other post-estimation features will also be inherited


I guess most postestimation should work without problems.
But I never checked any of the new inherited generic features for gam

This will be also another consistency check and check for how generic our generic features are.

penalization should have no effect beyond, params, cov_params and df_xxx (AFAIR)


user: josef-pkt
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_8518: state: open
title: ENH: cox proportional hazard, PHReg, survival function
body: https://stackoverflow.com/questions/74403023/how-to-plot-survival-curve-for-cox-survival-from-statsmodels-api

guessing, I didn't look at it
We need to stitch it together, cox ph maximizes only partial likelihood

Hazard function should be just multiplication by proportionality factor.
But we want also survival function, How do we recover it from the hazard function?

I guess inference like conf_int for it will be very difficult combining baseline uncertainty with PH parameter uncertainty.



user: josef-pkt
issue_8517: state: open
title: BUG/Design GLM predict, get_prediction does not include weights
body: GLM predict only uses the linpred

- we don't have freq_weights in predict
- I guess var_weights will only affect inference for predict
- binomial count, predict is for 1 trial (i.e. prob), no option for n_trials, i.e. predicting counts. `resid_xxx` include n_trials separately

It looks like current behavior is by design, but we might want to have more options.

just an observation while reading the code for #8505 and thinking about which other "which" options to add
e.g. currently my `which="var"` returns only the family var function without extras (no var_weights, no n_trials


user: josef-pkt
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_8516: state: open
title: BUG/TST:  df, k_extra in GenericLikelihoodModel  subclasses, copula models
body: cases that were missed in #8476

I just saw more warnings in unit test suite

CopulaModel in unit tests in statsmodels.distributions.copula.tests.test_model

```
statsmodels/distributions/copula/tests/test_model.py::TestGumbel::test
  /home/vsts/work/1/s/statsmodels/base/model.py:2741: UserWarning: df_model + k_constant + k_extra differs from k_params
    warnings.warn("df_model + k_constant + k_extra "
```
user: josef-pkt
issue_8515: state: open
title: TST: noop unit tests in test_glm_weights
body: While working on DeprecationWarnings for linear keyword in GLM predict and get_prediction (#8505), I saw this

The first test seems to use hasattr on a non-existing attribute which would skip the test
The second compares a result with itself

I didn't investigate the details, or how to fix those

note DeprecationWarning assert is being added in my PR 8505


```
    def test_pearson_chi2(self):
        if hasattr(self.res2, 'chi2'):
            assert_allclose(self.res1.pearson_chi2, self.res2.deviance_p,
                            atol=1e-6, rtol=1e-6)

    def test_getprediction(self):
        with pytest.warns(DeprecationWarning):
            # deprecation warning for linear keyword
            pred = self.res1.get_prediction()
        assert_allclose(pred.linpred.se_mean, pred.linpred.se_mean, rtol=1e-10)
```
user: josef-pkt
issue_8512: state: open
title: BUG/ENH: constrained/collinear fit and degrees of freedom for terms, df_terms,  anova
body: followup question to #8506 and #8336 

If we (semi-) automatically drop exog columns (e.g. fit_colinear), estimated (pinv) regularized params or fit under constraints, then this can effect hypothesis tests for terms. 

question is what is the relevant number of constraints for a joint hypothesis that affect terms.
Hypothesis tests on individual columns or on all params will effectively not be affected, I think. Those hypotheses we already have now.

With categorical exog and two way effects, especially nested effects, we might have zero cells, i.e. a column of zeros.
If we drop those, then the df of the term will be reduced.

AFAIR, we don't have any support for keeping track of df, number of effective parameters for terms (except, AFAIR we added it to GAM)

That is we need effective number of parameters for each term.

im_ratio might apply for pinv OLS
When we only drop columns, then we would just need to keep track of "nonzero" or unrestricted columns.

I don't have any idea (for now) what would be useful for general affine transformation of params as in fit_constrained.

In general, maybe just looking at the matrix rank of the sub-exog or terms would be enough.
I think, I do this already in some specification, lm tests, using rank to determine df_constrained for chi2 test.
  
user: josef-pkt
issue_8506: state: open
title: BUG: Nested ANOVA incorrect results -- wrong matrix definition
body: #### Description of the bug

statsmodels.formula.api.ols run on data with nested factors (i.e. not complete crossing) gives incorrect results.

This problem has been reported at least twice independently

1) https://stackoverflow.com/questions/72950135/nested-anova-in-statsmodels
2) https://stats.stackexchange.com/questions/594889/nested-anova-in-python-and-singular-matrices


#### Code Sample

I'll replicate the code from example 2) above:
```python

import pandas as pd
import statsmodels.formula.api as smf

nobs = 5
nB = 4
nA = 3
A = [f"A{i}" for i in range(1,nA+1) for h in range(nobs*nB)]
B = [f"B{i}{j}" for i in range(1,nA+1) for j in range(1,nB+1) for h in range(nobs)]
obs = list(range(5))*nA*nB

data = (30,-19,-31,-14,-14,0,20,32,11,13,7,5,3,-5,8,28,15,20,20,-48,
       24,16,-18,11,-10,14,11,27,11,8,20,18,8,32,-25,20,-12,0,-5,44,
       14,-18,33,-9,-7,14,8,-19,6,-38,18,16,7,-4,21,-25,13,36,18,-7)

df = pd.DataFrame(data ={
    "A": A,
    "B": B,
    "Observation_number": obs,
},dtype="category")
df = df.join(pd.Series(data=data,name="y"))

model = smf.ols("y ~ A + A:B",data=df).fit()
anova_df = sm.stats.anova_lm(model)
```

The crossing factor A:B has 33 DOF, which is only right if Statsmodels crossed factors nested in A1 with A2.
```
           df        sum_sq     mean_sq         F    PR(>F)
A          3.0   2506.300000  835.433333  2.180719  0.102465
A:B       33.0  10338.241297  313.280039  0.817750  0.726097
Residual  48.0  18388.800000  383.100000       NaN       NaN
```

Running model.summary() shows the condition number is very low:
> [2] The smallest eigenvalue is 4.14e-33. This might indicate that there are strong multicollinearity problems or that the design > matrix is singular.

which you would expect when all-zero columns are left in the model matrix.

#### Requested change

**Add automatic detection and ignore of all-zero columns (inappropriately crossed factors) or give user capability to manually do so.** As Statsmodel works currently, any nested model fitted with OLS will give completely erroneous results.



user: mfissore
issue_8503: state: open
title: DOC: sm.datasets attribute table
body: #### Is your feature request related to a problem? Please describe
I was hoping there would be a table with the sm.datasets attributes in the [documentation](https://www.statsmodels.org/dev/datasets/index.html), but it doesn't exist. This info wasn't too much trouble to figure out, but I think a nice table there would be a good addition.

#### Describe the solution you'd like
I would recommend adding something like the following table (below) to [docs/source/datasets/index.rst](https://github.com/statsmodels/statsmodels/blob/main/docs/source/datasets/index.rst)

I tried to figure out how to change this myself and do a PR, but I'm not at all farmiliar with sphynx and don't want to break anything :)

Thanks for considering!

#### Additional context

|Available Datasets                                                                                                               |Attribute Names    |
|---------------------------------------------------------------------------------------------------------------------------------|------------------|
|[American National Election Survey 1996](https://www.statsmodels.org/dev/datasets/generated/anes96.html)                         |anes96            |
|[Breast Cancer Data](https://www.statsmodels.org/dev/datasets/generated/cancer.html)                                             |cancer            |
|[Bill Greene’s credit scoring data.](https://www.statsmodels.org/dev/datasets/generated/ccard.html)                              |ccard             |
|[Smoking and lung cancer in eight cities in China.](https://www.statsmodels.org/dev/datasets/generated/china_smoking.html)       |china_smoking     |
|[Mauna Loa Weekly Atmospheric CO2 Data](https://www.statsmodels.org/dev/datasets/generated/co2.html)                             |co2               |
|[First 100 days of the US House of Representatives 1995](https://www.statsmodels.org/dev/datasets/generated/committee.html)      |committee         |
|[World Copper Market 1951-1975 Dataset](https://www.statsmodels.org/dev/datasets/generated/copper.html)                          |copper            |
|[US Capital Punishment dataset.](https://www.statsmodels.org/dev/datasets/generated/cpunish.html)                                |cpunish           |
|[Danish Money Demand Data](https://www.statsmodels.org/dev/datasets/generated/danish_data.html)                                  |danish_data       |
|[El Nino - Sea Surface Temperatures](https://www.statsmodels.org/dev/datasets/generated/elnino.html)                             |elnino            |
|[Engel (1857) food expenditure data](https://www.statsmodels.org/dev/datasets/generated/engel.html)                              |engel             |
|[Affairs dataset](https://www.statsmodels.org/dev/datasets/generated/fair.html)                                                  |fair              |
|[World Bank Fertility Data](https://www.statsmodels.org/dev/datasets/generated/fertility.html)                                   |fertility         |
|[Grunfeld (1950) Investment Data](https://www.statsmodels.org/dev/datasets/generated/grunfeld.html)                              |grunfeld          |
|[Transplant Survival Data](https://www.statsmodels.org/dev/datasets/generated/heart.html)                                        |heart             |
|[(West) German interest and inflation rate 1972-1998](https://www.statsmodels.org/dev/datasets/generated/interest_inflation.html)|interest_inflation|
|[Longley dataset](https://www.statsmodels.org/dev/datasets/generated/longley.html)                                               |longley           |
|[United States Macroeconomic data](https://www.statsmodels.org/dev/datasets/generated/macrodata.html)                            |macrodata         |
|[Travel Mode Choice](https://www.statsmodels.org/dev/datasets/generated/modechoice.html)                                         |modechoice        |
|[Nile River flows at Ashwan 1871-1970](https://www.statsmodels.org/dev/datasets/generated/nile.html)                             |nile              |
|[RAND Health Insurance Experiment Data](https://www.statsmodels.org/dev/datasets/generated/randhie.html)                         |randhie           |
|[Taxation Powers Vote for the Scottish Parliament 1997](https://www.statsmodels.org/dev/datasets/generated/scotland.html)        |scotland          |
|[Spector and Mazzeo (1980) - Program Effectiveness Data](https://www.statsmodels.org/dev/datasets/generated/spector.html)        |spector           |
|[Stack loss data](https://www.statsmodels.org/dev/datasets/generated/stackloss.html)                                             |stackloss         |
|[Star98 Educational Dataset](https://www.statsmodels.org/dev/datasets/generated/star98.html)                                     |star98            |
|[Statewide Crime Data 2009](https://www.statsmodels.org/dev/datasets/generated/statecrime.html)                                  |statecrime        |
|[U.S. Strike Duration Data](https://www.statsmodels.org/dev/datasets/generated/strikes.html)                                     |strikes           |
|[Yearly sunspots data 1700-2008](https://www.statsmodels.org/dev/datasets/generated/sunspots.html)                               |sunspots          |

user: coup321
issue_8488: state: open
title: ENH: support some integer indexes in tsa models
body: - [X] closes #8487
- [x] tests added / passed. 
- [X] code/documentation is well formatted.  
- [X] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

<details>

**Notes**:

The PR is fixing the issue and not breaking any tests in statsmodels/tsa/statespace/tests/test_simulate.py. 
I will also add a test for this edge case before making this ready for review.

</details>

user: ltsaprounis
issue_8487: state: open
title: BUG: time index from simulate method not correct for integer index not starting from zero
body: #### Describe the bug

The prediction index in the forecasting models simulation output is not correct for the special case where you have a range or integer index that doesn't start from zero. The  resulting index is as if the endog index started at 0. 

For example, if the index of the endog is [3, 4, 5, 6, 7, 8] the expected index for nsimulations=3 is [9, 10, 11] but what we get instead is [7, 8, 9].

I discovered this while writing an adapter for the statsmodels simulate method in sktime https://github.com/sktime/sktime/pull/3462 cause some of the automated sktime output index checks were failing.

#### Code Sample, a copy-pastable example if possible


```python
from statsmodels.tsa.statespace.dynamic_factor import DynamicFactor # same with SARIMAX and ETS
import pandas as pd
import numpy as np

index = pd.Int64Index(np.arange(12) + 3)

y = pd.DataFrame(
    data= {
        0: np.arange(12),
        1: np.arange(12)/2 + 5,
    },
    index=index
)

forecaster = DynamicFactor(endog=y, k_factors=1, factor_order=1)
forecaster = forecaster.fit()

simulations = forecaster.simulate(
    nsimulations=5,
    repetitions=3,
    anchor="end"
)

expected_simulation_index = pd.Int64Index(np.arange(np.max(index)+1, np.max(index)+6))

assert np.all(simulations.index ==  expected_simulation_index)
```
<details>
This issue is persistent across all the forecasting algorithms I've tried. 
</details>


#### Expected Output
The index when `anchor = "end"` starts from one time step after the endog max index.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

Using the current version from the main branch.
Originally spotted it on version 0.12.1. 

</details>

user: ltsaprounis
issue_8486: state: open
title: ENH: Allow to choose other optimization methods in ETS
body: - [x] closes #8440 
- [ ] tests added / passed. 
- [x] code/documentation is well formatted.  
- [x] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

**Details:**
Added parameter `method` to allow selection of other supported optimization methods from base class `LikelihoodModel`.

**Issues:**
Tested the supported optimization methods but only "nm", "bfgs", and "powell" are working. See screenshots below for the exception returned from the failed methods:

1. 'newton' for Newton-Raphson
![image](https://user-images.githubusercontent.com/93530957/198978927-f7d4f565-0bcb-4f51-a192-f32fb2748be5.png)

2. 'bfgs' for Broyden-Fletcher-Goldfarb-Shanno (BFGS)
![image](https://user-images.githubusercontent.com/93530957/198979021-9be5a6dc-6c5c-4d70-af4a-56d608cd2d45.png)

3. 'cg' for conjugate gradient
![image](https://user-images.githubusercontent.com/93530957/198979194-50dd4874-2a2c-4c75-915c-3178ea337b20.png)

4. 'ncg' for Newton-conjugate gradient
![image](https://user-images.githubusercontent.com/93530957/198979285-3f1348c8-e48c-40f5-88b6-b638640e619f.png)

5. 'basinhopping' for global basin-hopping solver
![image](https://user-images.githubusercontent.com/93530957/198979379-bcfe3844-0d8a-41f1-b07c-1fdbb534caea.png)

6. 'minimize' for generic wrapper of scipy minimize (BFGS by default)
![image](https://user-images.githubusercontent.com/93530957/198979443-6e87a9f2-52e0-42f1-91c1-e6c326530587.png)



user: jasmineliaw
issue_8482: state: open
title: BUG/REF: OrderedModel, df_model is k_params, not extra over null model
body: OrderedModel uses
`self.df_model = self.k_vars + self.k_extra`

The definition of df_model should be k_params - k_null

see #8481 and #8475

I'm working on changing and fixing df_resid, df_model in GenericLikelihoodModel in #8476
With my latest changes df_model definition raises a Warning

The constants only model should have `k_null = number of thresholds`  (for empty exog which cannot include a constant)
so df_model should be k_vars, exog.shape[1]

aside: the null model Results.llnull uses parameter estimates from `OrderedModel.start_params` 


user: josef-pkt
issue_8481: state: open
title: REF/ENH: change df_resid to include extra parameters in MLE, add k_null
body: many related issues, and old problem, which finally needs a solution
main design FAQ-D issue:  #1723

bug in NB #1624
consistent definitions in GenericLikelihoodModel #8476

This is mainly to have a consistent definition of df_resid, df_model and df_null/k_null, and k_params, especially for more models with extra parameters and multi-link models.
Existing models with extra params like NB, GP, ... should already have correct results statistics like aic, llr_pvalue, so those will return the same values as before if df_resid is redefined, the change for those is internal.

for discrete, e.g. count models, include k_extra in df_resid so we don't need to add it specifically in aic, bic and other 

My current thinking is that this should not affect the definitions in GLM (at least not the default). We treat GLM/1-parameter-LEF as QMLE or moment estimator like OLS.





user: josef-pkt
issue_8477: state: open
title: BUG: perfect prediction check in DiscreteModel only applies to binary? no offset
body: while browsing and skimming some code

The perfect prediction check in DiscreteModel ignores offset (new enhancement) and code uses properties that apply only for binary models, Logit and Probit.
If we would check for perfect prediction in countmodels like Poisson, then we would also need to include exposure.


```
    def _check_perfect_pred(self, params, *args):
        endog = self.endog
        fittedvalues = self.cdf(np.dot(self.exog, params[:self.exog.shape[1]]))
        if (self.raise_on_perfect_prediction and
                np.allclose(fittedvalues - endog, 0)):
            msg = "Perfect separation detected, results not available"
            raise PerfectSeparationError(msg)
```

I guess currently this method is only called by binary models.

user: josef-pkt
issue_8475: state: open
title: BUG: default df_resid, df_model in GenericLikelihoodModel
body: It looks like the current GenericLikelihoodModel does not set df_resid, df_model correctly.
see also #7767 and #7759 for df and #4679 for more refactoring

I converted an old gist from a blog article to work with current version.
It uses GenericLikelihoodModel for ZIP (without regressors)
https://www.austinrochford.com/posts/2015-03-03-mle-python-statsmodels.html

my notebook for checking this is `mle_custom_zip.ipynb`

fitting the model emits warnings
```
...statsmodels\base\model.py:2733: UserWarning: df_model + k_constant differs from nparams
  warnings.warn("df_model + k_constant differs from nparams")
...statsmodels\statsmodels\base\model.py:2735: UserWarning: df_resid differs from nobs - nparams
  warnings.warn("df_resid differs from nobs - nparams")
```

and df_resid, df_model are wrong

```
N, model.df_resid, model.df_model, results.df_resid, results.df_model
(1000, 1000.0, -1.0, 1000.0, -1.0)
```

The model code does not set anything related to df_xxx, 

```
class ZeroInflatedPoisson(GenericLikelihoodModel):
    def __init__(self, endog, exog=None, **kwds):
        if exog is None:
            exog = np.zeros_like(endog)
            
        super(ZeroInflatedPoisson, self).__init__(endog, exog, **kwds)
    
    def nloglikeobs(self, params):
        pi = params[0]
        lambda_ = params[1]

        return -np.log(zip_pmf(self.endog, pi=pi, lambda_=lambda_))
    
    def fit(self, start_params=None, maxiter=10000, maxfun=5000, **kwds):
        if start_params is None:
            lambda_start = self.endog.mean()
            excess_zeros = (self.endog == 0).mean() - stats.poisson.pmf(0, lambda_start)
            
            start_params = np.array([excess_zeros, lambda_start])
            
        return super(ZeroInflatedPoisson, self).fit(start_params=start_params,
                                                    maxiter=maxiter, maxfun=maxfun, **kwds)
```

~~One guess for the bug is that the model code specifies neither n_params/k_params, nor a `start_params` with correct `len`.~~
using explicit start_params does not help. The `__init__` does not seem to have an option to set k_params.

I guess there is some problem also if exog=None.

`fit` sets the `_set_extra_params_names` correctly, but does not change df_xxx.

user: josef-pkt
issue_8472: state: open
title: Fail with res.get_forecast()
body: #### Describe the bug
AttributeError                            Traceback (most recent call last)
----> 1 res.forecast(1,x_t = df["A"].values , w_t = df["B"].values)

File c:\Users\XXX\anaconda3\lib\site-packages\statsmodels\base\wrapper.py:113, in make_wrapper.<locals>.wrapper(self, *args, **kwargs)
    111     obj = data.wrap_output(func(results, *args, **kwargs), how[0], how[1:])
    112 elif how:
--> 113     obj = data.wrap_output(func(results, *args, **kwargs), how)
    114 return obj

File c:\Users\XXX\anaconda3\lib\site-packages\statsmodels\tsa\statespace\mlemodel.py:3442, in MLEResults.forecast(self, steps, **kwargs)
   3440 else:
   3441     end = steps
-> 3442 return self.predict(start=self.nobs, end=end, **kwargs)

File c:\Users\XXX\anaconda3\lib\site-packages\statsmodels\tsa\statespace\mlemodel.py:3403, in MLEResults.predict(self, start, end, dynamic, **kwargs)
   3357 """
   3358 In-sample prediction and out-of-sample forecasting
   3359 
   (...)
   3400     including confidence intervals.
   3401 """
   3402 # Perform the prediction
-> 3403 prediction_results = self.get_prediction(start, end, dynamic, **kwargs)
...
   1769                              ' with a regression component require'
   1770                              ' additional exogenous values via the'
   1771                              ' `exog` argument.')

AttributeError: 'CommercialSaleRegression' object has no attribute 'k_exog'
#### Code Sample

```python
class CommercialSaleRegression(sm.tsa.statespace.MLEModel):
    def __init__(self, y_t, x_t, w_t):
        exog = np.c_[x_t, w_t]  # shaped nobs x 2

        super(CommercialSaleRegression, self).__init__(
            endog=y_t, exog=exog, k_states=2, initialization="diffuse"
        )

        # Since the design matrix is time-varying, it must be
        # shaped k_endog x k_states x nobs
        # Notice that exog.T is shaped k_states x nobs, so we
        # just need to add a new first axis with shape 1
        self.ssm["design"] = exog.T[np.newaxis, :, :]  # shaped 1 x 2 x nobs
        self.ssm["selection"] = np.eye(self.k_states)
        self.ssm["transition"] = np.eye(self.k_states)

        # Which parameters need to be positive?
        self.positive_parameters = slice(1, 4)

    @property
    def param_names(self):
        return ["intercept", "var.e", "var.x.coeff", "var.w.coeff"]

    @property
    def start_params(self):
        """
        Defines the starting values for the parameters
        The linear regression gives us reasonable starting values for the constant
        d and the variance of the epsilon error
        """
        exog = sm.add_constant(self.exog)
        res = sm.OLS(self.endog, exog).fit()
        params = np.r_[res.params[0], res.scale, 0.001, 0.001]
        return params

    def transform_params(self, unconstrained):
        """
        We constraint the last three parameters
        ('var.e', 'var.x.coeff', 'var.w.coeff') to be positive,
        because they are variances
        """
        constrained = unconstrained.copy()
        constrained[self.positive_parameters] = (
            constrained[self.positive_parameters] ** 2
        )
        return constrained

    def untransform_params(self, constrained):
        """
        Need to unstransform all the parameters you transformed
        in the `transform_params` function
        """
        unconstrained = constrained.copy()
        unconstrained[self.positive_parameters] = (
            unconstrained[self.positive_parameters] ** 0.5
        )
        return unconstrained

    def update(self, params, **kwargs):
        params = super(CommercialSaleRegression, self).update(params, **kwargs)

        self["obs_intercept", 0, 0] = params[0]
        self["obs_cov", 0, 0] = params[1]
        self["state_cov"] = np.diag(params[2:4])

mod = CommercialSaleRegression(df["target"][:27], df["A"][:27],df["B"][:27])
res = mod.fit()
res.forecast(1,x_t = df["seven A"].values , w_t = df["B"].values)
```

#### Expected Output
out of forecast

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]
INSTALLED VERSIONS
------------------
Python: 3.9.12.final.0

statsmodels
===========

Installed: 0.13.2 (c:\Users\XXX\anaconda3\lib\site-packages\statsmodels)

Required Dependencies
=====================

cython: 0.29.28 (c:\Users\XXX\anaconda3\lib\site-packages\Cython)
numpy: 1.21.5 (c:\Users\XXX\anaconda3\lib\site-packages\numpy)
scipy: 1.7.3 (c:\Users\XXX\anaconda3\lib\site-packages\scipy)
pandas: 1.3.4 (c:\Users\XXXX\anaconda3\lib\site-packages\pandas)
    dateutil: 2.8.2 (c:\Users\XXX\anaconda3\lib\site-packages\dateutil)
patsy: 0.5.2 (c:\Users\XXX\anaconda3\lib\site-packages\patsy)

Optional Dependencies
=====================

matplotlib: 3.5.1 (c:\Users\XXX\anaconda3\lib\site-packages\matplotlib)
    backend: module://matplotlib_inline.backend_inline 
...
pytest: 7.1.1 (c:\Users\XXX\anaconda3\lib\site-packages\pytest)
virtualenv: Not installed
</details>

user: hiiragit123
issue_8468: state: open
title: tests/test_grouputils.py: FutureWarning: Dropping invalid columns in `DataFrameGroupBy.mean` is deprecated
body: [Current error:](https://dev.azure.com/statsmodels/statsmodels-testing/_build/results?buildId=4721&view=logs&j=aabc21e9-9439-578e-0393-ea6ef34c273c&t=3c5b382f-2ffe-594f-225e-664c15c6a882&l=1822)

> statsmodels/tools/tests/test_grouputils.py:127: FutureWarning: Dropping invalid columns in DataFrameGroupBy.mean is deprecated. In a future version, a TypeError will be raised. Before calling .mean, select only columns which should be valid for the function.
>     ).mean(numeric_only=False)[

https://github.com/statsmodels/statsmodels/blob/250617b76e2f77f3d7a190053a945a6f19d15047/statsmodels/tools/tests/test_grouputils.py#L126-L128
user: EwoutH
issue_8465: state: open
title: getting nans from mlefit.conf_int
body: @ChadFulton sorry to bother again.
this is a relevent question to #8460 when i working on a workaround to set bounds on `sarimax.fit`, i could use the confidence intervals from fit results by mlefit, but all i got is `nan`s, including `mlefit.bse`.
is this normal? what should i do to get resonable results?
user: holder-fo
issue_8463: state: open
title: include fittedvalues and resid properties in ThetaModelResults
body: Problem:
I'm trying to get the fitted values and the residuals for the theta model. This is possible for HoltWintersResults, UnobservedComponentsResults, SarimaxResults, and others. Since I'm trying all different approaches, I also need to get the fittedvalues and residuals for the ThetaModelResults

Solution:
ThetaModelResults would have a property of fittedvalues and resid, similar to the models mentioned above. 
user: zach-clem
issue_8461: state: open
title: Adding a Jackknife CRV3 Covariance Matrix Estimator
body: Hi all, 

While implementing [support for wild cluster bootstrapping ](https://github.com/statsmodels/statsmodels/issues/8437)for the OLS method, @amichuda and I also plan to implement a fast jackknife based CRV3 variance covariance estimator based on recent work by MacKinnon, Nielsen & Webb (here is a [brief summary](https://s3alfisc.github.io/summclust/articles/Cluster-Robust-Variance-Estimators-CRV-1-3.html) I've written at some point). The major innovation by MNW is that their Jackknife interpretation makes it feasible to compute a CRV3 estimator with little computational effort. Would you be interested in such a feature not only for use with the wild cluster bootstrap, but in general for inference for `OLS` models? Again, the idea would be to add a new option to the `cov_type` argument in `.fit()`. If yes, we'll go ahead and prepare a second PR for it. 

Best, Alex 
user: s3alfisc
issue_8459: state: open
title: SARIMAX prediction DECAYS to ZERO for future data and does not print correspinding futures dates!  
body: #### Howdy folks, I'm new to python and trying to forecast a time series of total sales from a store using SARIMAX. The data spans  over 5-years (daily) from '2013-01-01' to '2017-08-16' . I wish to predict sales for the next 30 days ( i.e. from '2017-08-17' to '2017-09-15').
Here is the sample data,  read usingthis piece of code 
`import pandas as pd`
`import numpy as np`
`from statsmodels.tsa.statespace.sarimax import SARIMAX`
`import pmdarima  as pmd`

`DF_store = pd.read_csv('Per_store_DF', index_col = "date", parse_dates = True)`
`DF_store`
![image](https://user-images.githubusercontent.com/116379468/197254428-de8017ab-54e8-4757-97bf-d9eaed608d9d.png)

I decided to split the data into **Train** and **Test** sets - 80/20, Then use the **pmdarima** function to guess the optimal ARIMA model order, then train the model with SARIMAX on my endogenous data. 
`N = round((80/100)*len(DF_store))`
`DF_train = DF_store.iloc[:N]`
`DF_test  = DF_store.iloc[N:]`
`DF_train`
# Model Function
`def arima_generator(ts):`
                          `auto_model = pmd.auto_arima(ts, 
                                start_p = 1, 
                                start_q = 1,
                                max_p   = 3,
                                max_q   = 3,
                                m       = 30,
                               test = "adf",
                                trace = True)`
    `return auto_model`

# Apply the model function

`ARIMA_model  = arima_generator(DF_train)`
`ARIMA_model.summary()`

Porblems arise when I train the resulting model on my Training set and try to predict the Test data.  
`model = SARIMAX(DF_train, order =(1,0,0), seas_order = (0,0,0, 30))`
`result = model.fit()`
`result.summary()`

`start = len(DF_train)`
`end   = len(DF_store)`
# Using  
`predict = result.predict(start, end, dynamic = False)'   (same with True)
`DF_train.plot(legend = True, figsize=(15,10))` 
`predict.plot(legend = True)`
`predict`
1) The outpust has no dates 
2) It decays to zero as shown in these pictures. 
![image](https://user-images.githubusercontent.com/116379468/197259413-cb0b827d-834e-4154-9201-d561f6a04668.png)
# Using "start = len(DF_test)" Only
`start = len(DF_test)`
`predict = result.predict(start, dynamic = False)'   (same with True)`
![image](https://user-images.githubusercontent.com/116379468/197260405-a6593ab4-cf9d-4eee-919c-d7ee78f50ac0.png)
# Starting Somewhere along the Train dataset
It predicts to the end of the set but then decays to zero for future values as shown
`start = len(DF_test)`
`end   = len(DF_train)+ len(DF_test)-1`
`predict = result.predict(start, end, dynamic = False)`
![image](https://user-images.githubusercontent.com/116379468/197284243-0ef6ab6b-a0b9-49a1-ad6e-f109294f36b8.png)

#Questions. 
Summary is mymodel will NOT  work for any future dates. 
- why is this happening?
- Is there anything i'm missing? 
I'll be gald if someone could help. 
Thanksin advance. 
This is the dataset: [Per_store_DF.csv](https://github.com/statsmodels/statsmodels/files/9841505/Per_store_DF.csv)

user: ZEVtheNukeGuy
issue_8458: state: open
title: `pairwise_tukeyhsd` output table and `meandiff` confusing convention
body: The meandiff column of the result's table reports the difference of means for group 2 vs group 1. However, the table construction suggests otherwise. 

Why is that?

Let's consider the following code:

```
import numpy as np

np.random.seed(0)
x = np.random.choice(['A','B','C'], 50)
y = np.random.rand(50)

print(pairwise_tukeyhsd(y,x))


Multiple Comparison of Means - Tukey HSD, FWER=0.05
===================================================
group1 group2 meandiff p-adj   lower  upper  reject
---------------------------------------------------
     A      B   0.1506 0.2347   -0.07 0.3712  False
     A      C   0.1105 0.5064 -0.1278 0.3487  False
     B      C  -0.0401    0.9 -0.2865 0.2063  False
---------------------------------------------------

```

by looking at the above table, one would say that the difference of **groupA** and **group2** means is **0.15.** However, this is the difference of group2 - group1. 

```
y[x=='A'].mean() -  y[x=='B'].mean()
-0.1505750775332967
```
user: seralouk
issue_8453: state: open
title: REF: inconsistencies in GLM, discrete
body: for now just to park some issues that need additional checking, review and possibly refactoring

- GLM scale, df_resid, score_test, fit_constrained #7840  
user: josef-pkt
issue_8450: state: open
title: adfuller gives me an incorrect result. Is the numerical stability low ?
body: #### Describe the bug

adfuller gives me very different results if I just slighlty change (add 1e-8 to) my series [1,3,5,7,...,195,197,199].

**edit** see also #8456  results differ across environments


#### Code Sample, a copy-pastable example if possible


```
from statsmodels.tsa.stattools import adfuller

print(adfuller([2*i+1 for i in range(99)]))
# It is stationary: (-8.042419324497985, 1.8319240045049658e-12, 7, 91, {'1%': -3.50434289821397, '5%': -2.8938659630479413, '10%': -2.5840147047458037}, -5852.425844993292)
print(adfuller([2*i+1+1e-8 for i in range(99)]))
# It is not stationary: (0.9120132371109988, 0.9932451165937419, 2, 96, {'1%': -3.5003788874873405, '5%': -2.8921519665075235, '10%': -2.5830997960069446}, -5522.232946110268)
print(adfuller([2*i+1 for i in range(98)]))
# It is not stationary: (2.5578687056001446, 0.9990661662898507, 8, 89, {'1%': -3.506057133647011, '5%': -2.8946066061911946, '10%': -2.5844100201994697}, -5852.890084170324)
print(adfuller([2*i+1 for i in range(100)]))
# It is not stationary: (4.298488441509754, 1.0, 4, 95, {'1%': -3.5011373281819504, '5%': -2.8924800524857854, '10%': -2.5832749307479226}, -5904.713330313655)
```

#### Expected Output

The results of the different `print` should be similar. The series [1,3,5,7,...,195,197,199] is definitely not stationary.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

INSTALLED VERSIONS
------------------
Python: 3.10.4.final.0
OS: Linux 5.15.0-50-generic #56-Ubuntu SMP Tue Sep 20 13:23:26 UTC 2022 x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

statsmodels
===========

Installed: 0.13.2 (/home/victor/.local/lib/python3.10/site-packages/statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.21.5 (/usr/lib/python3/dist-packages/numpy)
scipy: 1.8.0 (/usr/lib/python3/dist-packages/scipy)
pandas: 1.4.3 (/home/victor/.local/lib/python3.10/site-packages/pandas)
    dateutil: 2.8.1 (/usr/lib/python3/dist-packages/dateutil)
patsy: 0.5.3 (/home/victor/.local/lib/python3.10/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.5.1 (/usr/lib/python3/dist-packages/matplotlib)
    backend: TkAgg 
cvxopt: Not installed
joblib: 1.1.0 (/home/victor/.local/lib/python3.10/site-packages/joblib)

Developer Tools
================

IPython: 7.31.1 (/usr/lib/python3/dist-packages/IPython)
    jinja2: 3.0.3 (/usr/lib/python3/dist-packages/jinja2)
sphinx: Not installed
    pygments: 2.11.2 (/usr/lib/python3/dist-packages/pygments)
pytest: Not installed
virtualenv: 20.13.0+ds (/usr/lib/python3/dist-packages/virtualenv)

</details>

user: VictorInpher
issue_8449: state: open
title: ENH: OrderedModel - make params_table for linear thresholds available
body: two statsexchange question why threshold parameters in OrderedModel differ from SPSS and R.

https://stats.stackexchange.com/questions/592680/why-do-statsmodels-python-and-mass-r-output-different-intercept-values-for-m

we already have helper methods for converting threshold params

We could also make standard errors available so that we can create a params_table with linear coefficients instead of the current parameterization.
This will require NonlinearDeltaCov because of multiple correlated parameters and nonlinear functions.
user: josef-pkt
issue_8446: state: open
title: BUG: don't raise if initial x is singular in recursive_olsresiduals, linear_harvey_collier
body: linear_harvey_collier raises exception if initial ols exog is singular.
harvey_collier uses recursive_olsresiduals which raises the exception

https://stackoverflow.com/questions/74084265/statsmodels-linear-harveu-collier-test-valueerror/74090601#74090601

This is not necessary, we already allow for using Ridge, but could use pinv also.
Ridge is only used for the initial estimate but not for updating in recursive residuals.

https://github.com/statsmodels/statsmodels/blob/main/statsmodels/stats/diagnostic.py#L1461

Essentially, the requirement of nonsingular initial exog is too strong.
Because initial nobs = k_vars, we loose rank as soon as one row is duplicate which could happen if we have discrete or discretized continuous variables.

also:
harvey_collier does not have option for ridge coefficient lamda to be used in recursive_olsresiduals

 
user: josef-pkt
issue_8444: state: open
title: ENH: new ordered models, add generalized ordered logit and generalized ordered link models
body: see comment https://github.com/statsmodels/statsmodels/issues/8442#issuecomment-1279662530 and following

It's also a cumulative link model, where params differ by choice P(y < k) = F(x beta_k)

My guess the only or main difference to OrderedModel is to split up params in `_bounds` and `linpred` to compute thresholds with category level specific params_{k} and params_{k-1} 

loglike_obs just uses `_bounds` method
score and hessian are currently computed by numdiff

not checked yet:
I guess there is a computational problem when integration `_bounds` are not increasing,
I guess we need to directly impose nonnegativity of probabilities, prob = max(F(upp) - F(low), 0).
Do we still get sum(probs) = 1 if non-negativity constraint is binding?

Williams 2016 has section 5 on p. 18 about negative probabilities, sounds like they are not clipped to zero

Williams, Richard. “Understanding and Interpreting Generalized Ordered Logit Models.” The Journal of Mathematical Sociology 40, no. 1 (January 2, 2016): 7–20. https://doi.org/10.1080/0022250X.2015.1112384.

  




user: josef-pkt
issue_8443: state: open
title: ENH: post-estimation, margins for OrderedModel
body: (just some thought while looking at nonproportional ordered Logit models)

How far can we already support post-estimation features for OrderedModel?

example:

margins: d Prob(y = k | x) / d x
it's nonlinear, so we need to use e.g. `_get_wald_nonlinear` for delta method
note: target might be a vector, e.g. derivative of prob for all choices k,  similarly to predicted probs of choices.
context #5387 I'm looking at different ways to compute margins for more cases

similarly: what `which` should we add to predict/get_prediction

possible use
in analogy to binary models: average risk ratio (population average margin versus log-binomial)
In Ordered model case, we might want to have other statistics of interest related to competing models, e.g. continuation-ratio prob.
(separate choice of model class from target statistic, e.g. we want to estimate average risk ratio even if the underlying model does not specify constant risk ratio conditional on exog)

user: josef-pkt
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_8442: state: open
title: A Class for Brant test for OrderedModel (using Python)
body: #### Is your feature request related to a problem? Please describe
Absence of brant test for PRA in ordered models 

#### Solution
I have written a class that works for OrderedModels.from_formula by getting inspiration from the brant test in R

I am wondering whether it might be useful as an addendum to OrderedModels in statstmodels.

I lack any experience with submitting patches to existing packages such as statsmodels. 

user: fbarfi
issue_8441: state: open
title: ENH: summary params table, header option for first column
body: 
It would be useful to have an option to add a header for the first column in params_table

example: from #5387
using model.t_test for margin in continuous-discrete interaction  d E(y | x2="a") / d x1
first column is the value of the categorical variable
d E(y | x2="a") / d x1
last row using uniform distribution over categorical (e.g. fictional average using predefined population probabilities)

```
print(tt.summary(xname=["a", "b", "c", "uniform"], title="Marginal Effect of weight"))
                          Marginal Effect of weight                           
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
a              1.0502      0.024     43.048      0.000       1.002       1.099
b              1.0059      0.028     35.436      0.000       0.950       1.062
c              1.0138      0.029     34.734      0.000       0.956       1.072
uniform        1.0233      0.023     44.871      0.000       0.978       1.069
==============================================================================
```

aside: still calling the t_test summary option `xname` is not pretty
for t_test `contrast_name` would be appropriate, but in the current case using a contrast is just a technical detail and not subject relevant.
maybe `row_names`

user: josef-pkt
issue_8440: state: open
title: FAQ: Hardcoded method in `ETSModel.fit()`
body: In `ETSModel.fit()`, the optimization method is hardcoded as "lbfgs": https://github.com/statsmodels/statsmodels/blob/main/statsmodels/tsa/exponential_smoothing/ets.py#L1068

Could it be parameterized like how it does in Holt-Winters? 
https://github.com/statsmodels/statsmodels/blob/main/statsmodels/tsa/holtwinters/model.py#L941

Refers to this post [here](https://github.com/sktime/sktime/issues/3162#issuecomment-1203150643), suspects that ETS model is not robust compared to R and `StatsForecast` due to the current optimization method. Would suggest having the option to apply `Nelder-Mead` method, which is the implementation from both `StatsForecast` and R. Refers to this [post](https://community.rstudio.com/t/parameter-estimation-in-ets-function-forecasting/92508/2) that stated R has implemented `Nelder-Mead` method as well:
> It is a nonlinear constrained optimization using the Nelder-Mead algorithm

user: jasmineliaw
issue_8439: state: open
title: ENH: predict tools, helper function using pandas for exog grid
body: helper function for lsmeans, emmeans, predict exog
related: #5387 ...

Currently we don't have any functions to create a grid of exog values for predict that affect more than a single column.
Relevant for predict and marginal/derivative effects.

lsmeans, emmeans and other margin packages in R (and SAS, ...) have functions that create grids of exog values.

For us, the models do not have enough information about the original data, trying to build the grid there is too late.

So, what we need are helper functions for the user to process the original dataframe assuming there are no irrelevant columns.
This would be all using pandas dataframes and the corresponding methods for categorical variables and quantiles.
Using formulas and formula `transform` will convert this to a design matrix for predict and other methods consistent with the model specification.
(Do we have `_get_predict_exog` in general?  AFAIR, I only added it in a few models.)

bonus:
for purely categorical exog, we might also want to have freq or prob weights for cell frequencies or probabilities in the original sample.
(new get_prediction in discrete allows for aggregation weights.)




user: josef-pkt
issue_8437: state: open
title: Adding a wild cluster bootstrap `cov_type` to models
body: Hi! @s3alfisc and I are working on implementing [wild cluster bootstrapping](https://github.com/s3alfisc/wildboottest) into python and we wanted to add it as a pull request to `statsmodels`. It's still a work in progress, but is there a straightforward way of adding a new `cov_type`?

Would you be able to tell me which files to look at and which classes to inherit to implement?

I can see that there's a `cov_type` file in `base` but not sure if adding to that is enough

Thank you!
user: amichuda
issue_8435: state: open
title: DOC/REF follow-up to merge of notebooks
body: some random observations while skimming new notebooks

- rank_compare https://www.statsmodels.org/dev/examples/notebooks/generated/stats_rankcompare.html
  - test_prob_superior return does not include `value`
  - summary: no indication that t-statistic is for value 0.5 and not 0, if it's easy to change use header `t (p=0.5)` instead of `t`
- stats_poisson
  - code lines too long for tolerance interval, confint quantile and others
- ...
- ...
user: josef-pkt
issue_8426: state: open
title: ENH/REF: rank_compare_2ordinal has None attributes, missing
body: The rankcompareresult instance of rank_compare_2ordinal does not have the statistics from the hypothesis test, those are None
Values and explicitly calling the test agrees with rank_compare_2indep. see below

`rank_compare_2indep` function has some code duplication with methods of the results holder class.
It looks like I didn't DRY this after adding the methods and extended to rank_compare_2ordinal.

I think we can refactor this to have all the test code only in the results/holder class. i.e. don't duplicate the part that `_tstat_generic` or similar are doing.
If we keep the hypothesis test as default statistic, pvalue, ..., then we can create or override None in `RankCompareResult.__init__` so that users don't need to add an extra method call to get the test results.

A quickfix would be to do this based on `statistic is None`, so it adds the attributes for the ordinal version

```
res_o = rank_compare_2ordinal(new, active)

res_o
<class 'statsmodels.stats.nonparametric.RankCompareResult'>
statistic = None
pvalue = None
s1 = None
s2 = None
var1 = 0.0919524144954732
var2 = 0.06075972346751607
var = 0.3098544504968023
var_prob = 0.0014148605045516088
nobs1 = 107
nobs2 = 112
nobs = 219
mean1 = None
mean2 = None
prob1 = 0.4557743658210948
prob2 = 0.5442256341789052
somersd1 = -0.08845126835781036
somersd2 = 0.08845126835781048
df = 204.29842398679557
use_t = True
tuple = (None, None)
```
user: josef-pkt
issue_8423: state: open
title: SUM/ENH: extension to and missing in truncated and hurdle count models
body: overview issues for missing parts and extensions to hurdle, truncated, and 0-1 censored count models added in #7973

missing 

- cov_type: only nonrobust is available, others explicitly raise
- post-estimation, the following currently raise exception #8422
  - [ ] score_test
  - [ ] get_distribution
  - [ ] get_influence
  - [ ] get_diagnostic test_chisquare_prob raises score_obs not available
  - [ ] margins #7193
- distributions currently only pmf of truncatedpoisson and truncatednegbin
  - need rvs to simulate data 
- ...
- ... 

bigger enhancements, more flexibility in model

- [ ] add `exog_zero` as option, see comment below
- [ ] more model choices for zero model, e.g. Logit, Probit, GLM #8021
- [ ] joint estimation of zero and truncated params #8020
   I think this is also a requirement to add penalized (Mixin) estimation, 
  e.g. NB-NB hurdle looks like dispersion parameter is not identified, needs restriction or penalization (to equal dispersion coefficients)
- [ ] GPP hurdle (maybe)
- [ ] ...
- [ ] (later) more flexible truncation, hurdles, modified counts, maybe mainly as generic "modifier" class
- [ ] ...
user: josef-pkt
issue_8422: state: open
title: ENH: missing post-estimation for truncated and hurdle count models
body: 
`score_test`, `get_distribution` and `get_influence` are not supported by truncated and hurdle models.
They raise exceptions a various places mainly indicating a missing model methods, 
e.g. score_factor, hessian (in hurdle model), `_deriv_score_obs_dendog` for influence

found while trying to work on a notebook for the docs
https://github.com/statsmodels/statsmodels/issues/8359#issuecomment-1262627543
user: josef-pkt
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_8421: state: open
title: ENH: tools numerical derivatives for only part of parameters
body: related #7891

We could add a helper function that computes first and second derivatives for only one element of a partition of the parameter space.
We have several applications where we can compute analytical derivatives for part of the parameters, but need numerical derivatives with respect to parameters where we do not have analytical derivatives.

Currently, we either use only numerical derivatives, or hand code it for the special case (example ?).

Main applications

- models with extra params, e.g. negative binomial, generalized poisson, more to come
- score or conditional moment tests, GMM: derivative of extra moment conditions, or cross-derivative in stacked moment conditions as in treatment effect GMM. (??? *1)

In those cases we have partition params = [known, unknown]
First derivative is easy, we just need to write a wrapper function to keep some parameters fixed.
Second derivatives: more difficult because we also need cross-derivatives, off-diagonal block of Hessian.

In the score_test/cmt/GMM application, we have first derivative of moment conditions instead of second derivative. This requires more vectorization because return of target function may be 2-dim. But in most cases we only need the derivative of the sum or mean of moment conditions for that (like in Hessian case).

*1: This might be different. 
We need cross derivatives of extra moment conditions w.r.t. original parameters and derivative of extra moment condition w.r.t. new parameters. We also have a partition of the moment conditions and not just of params.

example
from https://github.com/HDembinski/jacobi/blob/main/src/jacobi/_jacobi.py (under discussion to be added to scipy)
```
    mask : array or None, optional
        If `x` is an array and `mask` is not None, compute the Jacobi matrix only for the
        part of the array selected by the mask.
```
user: josef-pkt
issue_8419: state: open
title: DynamicFactorMQ quarterly forecast does not update when data in quarter-end month is added
body: #### Describe the bug

Quarterly forecast for current quarter using DynamicFactorMQ does not update when new data for the month **ending** that quarter is added. I would expect it to do so based on the 'news' output, which suggests there is a surprise in that data (and where the absolute impacts do not sum to zero).

#### Code Sample: 

m_data and q_data are dicts containing a number of data vintages. Dates contains the dates of each vintage, which are also the keys of the dicts.
factors, factor_orders and factor_multiplicities are specified earlier.

_Creating the model on the baseline data_

```python
model = sm.tsa.DynamicFactorMQ(m_data[dates[0]], 
                               endog_quarterly=q_data[dates[0]], 
                               factors=factors, factor_orders=factor_orders, 
                               factor_multiplicities=factor_multiplicities)
base_results=model.fit(disp=10)
all_results = {dates[0]: base_results}
all_forecasts = {}
all_nowcasts = {}
```

_Getting updated results based on the newer vintages; new dates contains the dates on which I pulled updated monthly data. 
I am using .apply() in case of data revisions_

```python
for vintage in new_dates:
    all_results[vintage] = base_results.apply(
        m_data[vintage], endog_quarterly=q_data[vintage])
```

_Calculating forecasts and nowcasts for each vintage. I am using 'smoothed' to try and overcome the issue explained above.
However it does not seem to work._

```python
for vintage, results in all_results.items():
    all_nowcasts[vintage] = results.predict(
        start=end_curr_q, end=end_curr_q, 
        information_set='smoothed')[fc_var_name].resample('Q').last()
    all_forecasts[vintage] = results.predict(
        start=end_curr_q, end=fc_end_date, 
        information_set='smoothed')[fc_var_name].resample('Q').last()
```

_Computing the effect of the new data on the GDP nowcast (getting the 'news')
When I do this, there does appear to be an impact from adding datapoints in the quarter-end month
But this impact does not match the nowcast (which does not update)_

```python
for i in range(1, len(dates)):
    vintage = dates[i]
    prev_vintage = dates[i-1]
    news_results[vintage] = all_results[vintage].news(
        all_results[prev_vintage], impact_date = end_curr_q, 
        impacted_variable=fc_var_name, comparison_type='previous')
    print(news_results[vintage].summary)
    print(news_results[vintage].summary_details)
    print(news_results[vintage].summary_impacts)
```

#### Expected Output

I expected the nowcast for `end_curr_q` to update. However, it does not when I add new datapoints referring to September (i.e., the month at the end of Q3). I am unsure why this is. I tried to resolve by using `information_set='smoothed'`.

Interestingly, when I calculate the news after adding new datapoints for Sept, the absolute impact column does not sum to zero, suggesting the news may be incorporating the Sept data even if the forecast is not? 

Am i not using `information_set=...` correctly? Why does `.predict()` not update the current-quarter prediction when I add end-quarter month data, but `.news()` seems to?


user: natashamay99
issue_8417: state: open
title: Is `RollingOLS()` as vectorized as possible?
body: I noticed an extremely slow performance of that class on very large dataframes at the point of running `fit()`(it may take more than 1 second per series), compared to the performance of the regular vectorized functions of pandas and numpy so I wonder if it has already been optimized as much as possible or if it can take improvements and if any other alternatives exist to achieve higher performance.
user: epigramx
issue_8416: state: open
title: MarkovRegression gives SVD did not converge
body: #### Describe the bug

Hi, I am using the Markov Regime Switching algorithms implemented in statsmodels to make inference about the regimes in my data. I have learnt from the examples given by the MarkovRegression and the MarkovAutoregression ones, and I have also read the paper by Hamilton 1989 and 1990. I have now 2 time series, and I have taken the log form of both, each series have 100 numbers without nan of inf, and I can run the algorithm with the differenced data, but not the original log data, as the log data without differencing always give me the SVD does not converge error. 

#### Code Sample, a copy-pastable example if possible
my data frame that is already in the log form outputs to this, and I also attached a data file with the same numbers just for convenience.
                  VariableA       VariableB
07/02/1999  7.248433  7.773384
07/09/1999  7.253966  7.790489
07/16/1999  7.263680  7.813592
07/23/1999  7.217297  7.744938
07/30/1999  7.194287  7.730394
08/06/1999  7.175490  7.706163
08/13/1999  7.195112  7.748460
08/20/1999  7.200574  7.753409
08/27/1999  7.209118  7.785721
09/03/1999  7.185576  7.783224
09/10/1999  7.222274  7.849909
09/17/1999  7.208970  7.852828
09/24/1999  7.161234  7.793587
10/01/1999  7.165107  7.797497
10/08/1999  7.205264  7.855738
10/15/1999  7.136722  7.791006
10/22/1999  7.175643  7.822645
10/29/1999  7.227118  7.883823
11/05/1999  7.233094  7.931285
11/12/1999  7.245298  7.975393
11/19/1999  7.263680  8.021749
11/26/1999  7.254531  8.045909
12/03/1999  7.271356  8.062905
12/10/1999  7.268920  8.087948
12/17/1999  7.275519  8.135640
12/23/1999  7.284478  8.194091
12/31/1999  7.302665  8.230577
01/07/2000  7.286534  8.176813
01/14/2000  7.298445  8.229911
01/21/2000  7.281902  8.262688
01/28/2000  7.220008  8.153350
02/04/2000  7.266478  8.270397
02/11/2000  7.241903  8.292173
02/18/2000  7.210264  8.286017
02/25/2000  7.198931  8.341410
03/03/2000  7.251877  8.401670
03/10/2000  7.256826  8.448272
03/17/2000  7.305860  8.415493
03/24/2000  7.349552  8.477204
03/31/2000  7.323336  8.402343
04/07/2000  7.332369  8.374938
04/14/2000  7.220740  8.076360
04/20/2000  7.277420  8.170469
04/28/2000  7.286192  8.245384
05/05/2000  7.271704  8.214736
05/12/2000  7.265919  8.136665
05/19/2000  7.254885  8.095599
05/26/2000  7.230744  8.048788
06/02/2000  7.297091  8.229911
06/09/2000  7.302833  8.246434
06/16/2000  7.305188  8.249967
06/23/2000  7.288586  8.223627
06/30/2000  7.291656  8.247220
07/07/2000  7.309714  8.262946
07/14/2000  7.329094  8.313362
07/21/2000  7.306196  8.278682
07/28/2000  7.266827  8.162231
08/04/2000  7.294207  8.197539
08/11/2000  7.298783  8.206174
08/18/2000  7.313720  8.252055
08/25/2000  7.322180  8.283747
09/01/2000  7.328601  8.323123
09/08/2000  7.325808  8.261139
09/15/2000  7.302159  8.221076
09/22/2000  7.291997  8.236421
09/29/2000  7.281902  8.194506
10/06/2000  7.263154  8.118952
10/13/2000  7.233816  8.094073
10/20/2000  7.252054  8.160375
10/27/2000  7.244942  8.077913
11/03/2000  7.269791  8.117611
11/10/2000  7.224389  7.970395
11/17/2000  7.222931  7.985995
11/24/2000  7.205264  7.947679
12/01/2000  7.185766  7.852633
12/08/2000  7.212663  7.909306
12/15/2000  7.189922  7.864228
12/22/2000  7.190488  7.811568
12/29/2000  7.196687  7.772542
01/05/2001  7.173575  7.737834
01/12/2001  7.193122  7.839919
01/19/2001  7.206748  7.895995
01/26/2001  7.217627  7.880615
02/02/2001  7.210264  7.819435
02/09/2001  7.186523  7.730614
02/16/2001  7.174341  7.708635
02/23/2001  7.128897  7.629247
03/02/2001  7.119029  7.537164
03/09/2001  7.126087  7.516161
03/16/2001  7.057683  7.422374
03/23/2001  7.050556  7.458186
03/30/2001  7.064118  7.371489
04/06/2001  7.039003  7.293018
04/12/2001  7.083598  7.455877
04/20/2001  7.131499  7.577378
04/27/2001  7.137676  7.508239
05/04/2001  7.149328  7.570701
05/11/2001  7.140057  7.518879
05/18/2001  7.166652  7.567346
05/25/2001  7.153834  7.585027
06/01/2001  7.142827  7.523751

And the code I am using to run this:

import pandas as pd
import statsmodels.api as sm

# the above data frame is named as logdata
model = sm.tsa.MarkovRegression(endog=logdata['VariableA'], k_regimes=2, exog=logdata['VariableB'], switching_exog=True, switching_variance=True)
res = model.fit(maxiter=100000)

#### Expected Output

Assuming that I use the logreturns = (logdata - logdata.shift(1)).dropna(), and then use logreturns['VariableA'] as endogenous and use logreturns['VariableB'] as exogenous variables, I have tested many more data like this in my full data set, and most of the time I can get the fitted model and all the properties. However, when using logdata, I always get these error messages:

File "C:\..\markovSwitching.py", line 14, in MSDR
    res = model.fit(maxiter=maxiter)
  File "C:\Users\..\AppData\Local\Programs\Python\Python310\lib\site-packages\statsmodels\tsa\regime_switching\markov_switching.py", line 1113, in fit
    start_params = self._fit_em(start_params, transformed=transformed,
  File "C:\Users\..\AppData\Local\Programs\Python\Python310\lib\site-packages\statsmodels\tsa\regime_switching\markov_switching.py", line 1205, in _fit_em
    out = self._em_iteration(params[-1])
  File "C:\Users\..\AppData\Local\Programs\Python\Python310\lib\site-packages\statsmodels\tsa\regime_switching\markov_regression.py", line 214, in _em_iteration
    coeffs = self._em_exog(result, self.endog, self.exog,
  File "C:\Users\..\AppData\Local\Programs\Python\Python310\lib\site-packages\statsmodels\tsa\regime_switching\markov_regression.py", line 250, in _em_exog
    np.dot(np.linalg.pinv(tmp_exog), tmp_endog))
  File "<__array_function__ internals>", line 180, in pinv
  File "C:\Users\..\AppData\Local\Programs\Python\Python310\lib\site-packages\numpy\linalg\linalg.py", line 1998, in pinv
    u, s, vt = svd(a, full_matrices=False, hermitian=hermitian)
  File "<__array_function__ internals>", line 180, in svd
  File "C:\Users\..\AppData\Local\Programs\Python\Python310\lib\site-packages\numpy\linalg\linalg.py", line 1657, in svd
    u, s, vh = gufunc(a, signature=signature, extobj=extobj)
  File "C:\Users\..\AppData\Local\Programs\Python\Python310\lib\site-packages\numpy\linalg\linalg.py", line 98, in _raise_linalgerror_svd_nonconvergence
    raise LinAlgError("SVD did not converge")
numpy.linalg.LinAlgError: SVD did not converge

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

INSTALLED VERSIONS
------------------
Python: 3.10.5.final.0

statsmodels
===========

Installed: 0.13.2 (C:\Users\..\AppData\Local\Programs\Python\Python310\lib\site-packages\statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.23.0 (C:\Users\..\AppData\Local\Programs\Python\Python310\lib\site-packages\numpy)
scipy: 1.8.1 (C:\Users\..\AppData\Local\Programs\Python\Python310\lib\site-packages\scipy)
pandas: 1.4.3 (C:\Users\..\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas)
    dateutil: 2.8.2 (C:\Users\..\AppData\Local\Programs\Python\Python310\lib\site-packages\dateutil)
patsy: 0.5.2 (C:\Users\..\AppData\Local\Programs\Python\Python310\lib\site-packages\patsy)

Optional Dependencies
=====================

matplotlib: 3.5.2 (C:\Users\..\AppData\Local\Programs\Python\Python310\lib\site-packages\matplotlib)
    backend: TkAgg 
cvxopt: Not installed
joblib: Not installed
[Log.xlsx](https://github.com/statsmodels/statsmodels/files/9635216/Log.xlsx)

Developer Tools
================

IPython: Not installed
    jinja2: 3.1.2 (C:\Users\..\AppData\Local\Programs\Python\Python310\lib\site-packages\jinja2)
sphinx: Not installed
    pygments: Not installed
pytest: 7.1.3 (C:\Users\..\AppData\Local\Programs\Python\Python310\lib\site-packages\pytest)
virtualenv: Not installed


Thank you very much in advance for all of your help!!
user: ARBQuant
issue_8415: state: open
title: freq_weights not used correctly?
body: https://github.com/statsmodels/statsmodels/blob/83032dc493641ebc84a88055137826143102a55b/statsmodels/genmod/families/family.py#L931

Why is this comment here, and not applied? My thinking is that frequency weights are now an array of 1, while the user has passed sufficient information to the program for calculating freq_weights. Instead, the program calculated n_trials, and freq_weights will be defaulted to an array of 1s.

In later steps in def _fit_irls() (in generalized_linear_model.py) the program makes the computation self.iweights * self.n_trials, the self.iweights variable is initially constructed as self.freq_weights * self.var_weights. If the user then specified freq_weights to the program in the same fashion as n_trials is computed, it looks like these are squared, which I am not sure why it would be.

Maybe someone who understands this better could clear this out.
user: majest1c
issue_8413: state: open
title: fit_regularised() is not working for GAM model
body: #### Describe the bug

I tried elastic net fitting of the GAM model. It returns an Attribution Error that the smoother is None. However, standard fitting works.

```python
import statsmodels.api as sm
from statsmodels.gam.api import GLMGam, BSplines

#define smoother
q = np.linspace(0.01,0.90,10)
knots = np.array([train_data[['tmp']].quantile(x).item() for x in q])
bs = BSplines(train_data[['tmp']],
              df = n_knots+degree+1 ,   #n_knots+degree, #number of basis functions-1=degrees of freedom; df >= degree+1
              degree  =   degree, 

              #,
              #knot_kwds=[{'spacing':'quantile','lower_bound':-17,'upper_bound':38}] # deafult is quantile
              knot_kwds=[{'knots':knots}]
              )

#instantiate model
gam_bs = GLMGam(train_data['load'],
                exog = train_data[['trend']], 
                           #data=train_data,
                           smoother=bs,
                           alpha=[630.9573]
                           
                           )
# fit
res_bs = gam_bs.fit() # this works

res_bs = gam_bs.fit_regularized(alpha = 0.1) # this returns error
```


```
in GLMGam.__init__(self, endog, exog, smoother, alpha, family, offset, exposure, missing, **kwargs)
    540 self.exog_linear = exog_linear
    542 self.smoother = smoother
--> 543 self.k_smooths = smoother.k_variables
    544 self.alpha = self._check_alpha(alpha)
    545 penal = MultivariateGamPenalty(smoother, alpha=self.alpha,
    546                                start_idx=k_exog_linear)

AttributeError: 'NoneType' object has no attribute 'k_variables'
```
user: Simply-Adi
issue_8410: state: open
title: FAQ/DOC: econometrics examples R and Python 
body: Great example collection with "equivalent" code in R and Python (statsmodels, linearmodels, arch)
Some examples are not available in packages and coded directly in example, e.g. Tobit

https://web.pdx.edu/~crkl/ceR/

This is a bit similar to UCLA stats examples for statistics but for econometrics.
(UCLA stats does not have python examples)


user: josef-pkt
issue_8409: state: open
title: ENH: warn if (near) singular design in discrete, GLM
body: Optimization in discrete models and GLM might not fail with singular or near-singular matrix.

We should add a warning text to summary similarly as in linear models OLS, ...

example blog post that has collinear dummies
dummy for each weekday except reference and a weekend dummy which is the same as Saturday plus Sunday. 

My guess is that in this large sample case (nobs= 1.6 million), the small ridge penalty in the newton optimizers works as regularized estimator similar to `pinv`  (I haven't looked at this in a while, #953 is still open)

However, standard errors for the collinear variables are huge, so that should point out to users that something is strange or wrong.

from https://medium.com/@lucydickinson/how-to-interpret-the-odds-ratio-with-categorical-variables-in-logistic-regression-5bb38e3fc6a8

![image](https://user-images.githubusercontent.com/440735/190906272-1389e7fa-07e5-4105-b8a4-2b8d3819fdca.png)
user: josef-pkt
issue_8408: state: open
title: Broken links: https://github.com/statsmodels/statsmodels/wiki/Examples#generalized-estimating-equations-gee
body: Start from the bottom of the page I receive 403 error
user: DexterGlass
issue_8407: state: open
title: ENH: t-test, oneway, variance under null, score_test
body: (just a semi-random thought)

AFAIR, our t-test, oneway tests and similar compute variance using estimated means and not means estimated under the null hypothesis.

e.g Welsh anova allows for heteroscedasticity but does not impose equal means hypothesis
`vars_ = np.array([x.var(ddof=1) for x in args])`
The heteroscedasticity assumption is independent of which means we use to compute variances, i.e. we can allow for heteroscedasticity even if we impose the null hypothesis on the means.

Under score_test assumptions, we estimate auxiliary parameters under the null hypothesis.
We have this now for proportions and poisson rates, but the same applies to variance as auxiliary parameter for loc-scale distributions.

i.e. we might want to add an option to t-tests, z-tests and oneway to whether variances are computed under the null restricted model or under the unrestricted (wald) model.

Using equal means will increase the estimated variance(s) and make score_test more conservative. If wald version overrejects, then score test can underreject similar to the cases in count and binary models.
(Similar to score versus wald tests in models, score_tests are explicitly for null hypothesis testing, while wald statistics also work if the null hypothesis does not hold, e.g. standard errors for non-null parameters.)

I have not seen a direct reference for this.
However, there is a literature for LR-like or F tests that recommend keeping the variance estimate, `scale`,  the same across models that are compared. (I don't remember whether the recommendation was to use the variance of the largest model.)

Related: The bootstrap literature for heteroscedasticity or cluster robust standard errors recommend bootstrapping based on the model that imposes the Null (e.g. wild bootstrap and other residual bootstraps)
(not clear to me: permutation bootstrap using heteroscedasticity robust t-test still permutes under the null assumption on means. So the permutation DGP assumes the null on means, even if the test statistic is wald. ???)

user: josef-pkt
issue_8406: state: open
title: alternative definition of negative binomial 
body: #### Is your feature request related to a problem? Please describe
I wanted to use negative binomial to model RNAseq counts data with the sm.NegativeBinomial.fit() function. However, the present function is based on the definition of how many failed trials are needed to achieve a given number of successes with the success probability of p. I would prefer the function to be changed to the alternative definition of the total number of trials needed to achieve a given number of successes with the successful probability of p.

#### Describe the solution you'd like
I would like to have an alternative sm.NegativeBinomial.fit() function based on the alternative negative binomial function.

#### Describe alternatives you have considered
Alternatively, is it possible for me to clone this git, modify the fit function, and install it for my own purpose?

#### Additional context

user: zjgt
issue_8405: state: open
title: FAQ: Robust Linear Model ZeroDivisionError better error message
body: Hi, I am having issue with fitting robust linear model. It seems that whenever the number of predictors > the number of samples, it outputs `ZeroDivisionError: float division by zero` without any helpful message. So, I would like to confirm if this is related to degree of freedom (which is equal to number of sample)? If so, is there any solution to perform this, or this just does not makes sense?

Also, I think it would be great if the error message is propagated properly, i.e. output something like:

```
ValueError: We do not have enough degree of freedom to estimate, i.e. the number of sample is less than the number of predictors.
```
user: darrencl
issue_8404: state: open
title: FAQ: Cochran-Armitage trend test
body: #### Describe the bug

I am getting different statistical results using the statsmodels test_ordinal_association method compared to the CochranArmitageTest method from DescTools in R.

#### Code Sample, a copy-pastable example if possible


```python
import statsmodels.api as sm

table_tumors = pd.DataFrame(
    data = {
        0: [0,50],
        1: [1,49],
        2: [3,47],
        3: [6,44]
    },
    index = [1,0],
)

table = sm.stats.Table(table_tumors)

print(table.test_ordinal_association(
    row_scores=np.array([1,0]),
    col_scores=np.array([0,1,2,3]),
))
```

#### Expected Output

Using R, we obtain the following results:

```R
library(DescTools)

dose <- matrix(c(0,1,3,6,50,49,47,44), byrow=TRUE, nrow=2, dimnames=list(resp=c(1,0), dose=0:3))

Desc(dose)
#CochranArmitageTest(dose)
#
#        Cochran-Armitage test for trend
#
#data:  dose
#Z = 2.9019, dim = 4, p-value = 0.003709
#alternative hypothesis: two.sided
```

Using statsmodels, we obtain:

```
null_mean   15.71072319201995
null_sd     3.5390390971784127
pvalue      0.008669937476153359
statistic   25.0
zscore      2.624801973899118
```

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

```
INSTALLED VERSIONS
------------------
Python: 3.10.7.final.0

statsmodels
===========

Installed: 0.13.2 (C:\Users\Sébastien\AppData\Local\Programs\Python\Python310\lib\site-packages\statsmodels)
Required Dependencies
=====================

cython: Not installed
numpy: 1.23.2 (C:\Users\Sébastien\AppData\Local\Programs\Python\Python310\lib\site-packages\numpy)
scipy: 1.9.1 (C:\Users\Sébastien\AppData\Local\Programs\Python\Python310\lib\site-packages\scipy)
pandas: 1.4.4 (C:\Users\Sébastien\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas)
    dateutil: 2.8.2 (C:\Users\Sébastien\AppData\Local\Programs\Python\Python310\lib\site-packages\dateutil)
patsy: 0.5.2 (C:\Users\Sébastien\AppData\Local\Programs\Python\Python310\lib\site-packages\patsy)
```
user: sbwiecko
issue_8403: state: open
title: ENH/REF: Signature of power() between parent class Power and child class FTestPower does not fit.
body: #### Describe the bug

When using `plot_power()` or `solve_power` with a FTestPower object, one gets an error that positional argument alpha is not specified. This is because, we call the `plot_power()` function of parent class which, expects arguments for `power()` in all inheriting classes in the form of `effect_size, nobs, alpha`. However, inside FTestPower the signature is different.

#### Code Sample, a copy-pastable example if possible


```python
test = FTestPower()
test.plot_power(dep_var='nobs',
                    nobs=np.range(100),
                    effect_size=0.8,
                    alpha=0.05,
```
<details>

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `main`. If your problem has been fixed in an unreleased version, you might be able to use `main` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the main branch of this repository? It helps the limited resources if we know problems exist in the current main branch so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

We would expect to see the power analysis plot. 

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]

INSTALLED VERSIONS
------------------
Python: 3.9.12.final.0
OS: Darwin 21.5.0 Darwin Kernel Version 21.5.0: Tue Apr 26 21:08:29 PDT 2022; root:xnu-8020.121.3~4/RELEASE_ARM64_T8101 x86_64
byteorder: little
LC_ALL: None
LANG: None
statsmodels
===========
Installed: 0.13.2 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/statsmodels)
Required Dependencies
=====================
cython: Not installed
numpy: 1.22.4 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/numpy)
scipy: 1.9.0 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/scipy)
pandas: 1.4.3 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/pandas)
    dateutil: 2.8.2 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/dateutil)
patsy: 0.5.2 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/patsy)
Optional Dependencies
=====================
matplotlib: 3.5.3 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/matplotlib)
Backend MacOSX is interactive backend. Turning interactive mode on.
    backend: MacOSX 
cvxopt: Not installed
joblib: 1.1.0 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/joblib)
Developer Tools
================
IPython: 8.4.0 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/IPython)
    jinja2: 3.0.3 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/jinja2)
sphinx: 3.5.4 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/sphinx)
    pygments: 2.13.0 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/pygments)
pytest: 7.1.2 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/pytest)
virtualenv: 20.16.3 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/virtualenv)
INSTALLED VERSIONS
------------------
Python: 3.9.12.final.0
OS: Darwin 21.5.0 Darwin Kernel Version 21.5.0: Tue Apr 26 21:08:29 PDT 2022; root:xnu-8020.121.3~4/RELEASE_ARM64_T8101 x86_64
byteorder: little
LC_ALL: None
LANG: None
statsmodels
===========
Installed: 0.13.2 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/statsmodels)
Required Dependencies
=====================
cython: Not installed
numpy: 1.22.4 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/numpy)
scipy: 1.9.0 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/scipy)
pandas: 1.4.3 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/pandas)
    dateutil: 2.8.2 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/dateutil)
patsy: 0.5.2 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/patsy)
Optional Dependencies
=====================
matplotlib: 3.5.3 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/matplotlib)
    backend: MacOSX 
cvxopt: Not installed
joblib: 1.1.0 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/joblib)
Developer Tools
================
IPython: 8.4.0 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/IPython)
    jinja2: 3.0.3 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/jinja2)
sphinx: 3.5.4 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/sphinx)
    pygments: 2.13.0 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/pygments)
pytest: 7.1.2 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/pytest)
virtualenv: 20.16.3 (/Users/pengfei_zhao/miniconda3/envs/witte/lib/python3.9/site-packages/virtualenv)
</details>

user: pzdkn
issue_8399: state: open
title: STR decomposition
body: The R package stR has recently appeared on my radar. The seasonal decomposition scheme that they use is quite robust, and it allows for multiple types of seasonality along with the inclusion of exogenous regressor's and complex seasonal smoothing schemes. I've decomposed using stats models GLM, UC, the classical and STL model and believe this is one of the best methods I've used. Wondering if it's on the road map at all to implement this in stats models as there is no implementation on the python side. 

https://www.monash.edu/business/ebs/research/publications/ebs/wp13-15.pdf

Thanks,
J
user: Forecastlife
issue_8397: state: open
title: ENH: use shrinkage variance (semi-pooled) in oneway anova, multiple tests
body: (mainly parking a reference for an old idea)

variance and covariance estimates are not very good in small or very small samples.
One idea is to use penalized or shrinkage (co)variance to get better small sample properties. In oneway and similar cases with heteroscedasticity we can shrink to a pooled estimate.

related
#3197 cov shrinkage, penalized (also related outlier robust cov methods)
#2942 applied to GMM weight/cov matrix
#2882 similar issue for mean params shrinkage/penalization to a target


The following looks like a good starting point for oneway anova and multiple, e.g. pairwise, comparisons (context in #8396 )
Adding this to individual hypothesis test functions in `stats` is much easier than integrating it into `cov_type` in models (which I have not yet tried out).

Cui, Xiangqin, J. T. Gene Hwang, Jing Qiu, Natalie J. Blades, and Gary A. Churchill. 2005. “Improved Statistical Tests for Differential Gene Expression by Shrinking Variance Components Estimates.” Biostatistics 6 (1): 59–75. https://doi.org/10.1093/biostatistics/kxh018.

they include references to previous literature for semi-pooled variances in tests
608 citations, so there might be a lot more in this direction.

extension idea: 
The pooled estimate would have enough observations to use also other estimators than simple variance, e.g. outlier robust estimators, i.e. we could shrink to a robust, pooled variance estimate. (possible problem if robust "variance" is just a dispersion measure possibly calibrated to normal distribution.)


The main objective here is to improve heteroscedasticity robust hypothesis tests in small samples. If some group sizes are small, then variance estimate using only within group information is too noisy.
e.g. Welch anova and multiple comparisons with unequal variance.
(related AFAIR
BF mean anova does not use weights in computing average, but uses group variances for inference.
In meta-analysis we also have the option to use variance weights or not for the weighted average, IIRC
)

The main decision is how much too shrink, e.g. choosing weights between sample and pooled variances. Those weights should depend on sample sizes. If group is large, then we do not need to shrink the group variance.

user: josef-pkt
issue_8396: state: open
title: SUMM/ENH/REF  multiple comparison - overview, open, roadmap
body: I don't have a good overview what the current status is and the roadmap for it should be.
We have several open issues and prs, enhancement requests, code that needs refactoring and more testing and likely bug fixes.
I put it on the 0.15 roadmap #8217

some of it also applies to statsmodels.base, e.g. extension of t_test_pairwise

an older generic issue **Multiple Comparisons** #852 includes discussion on rank based Nemenyi and dunnet's comparison to a control


- oneway, continuous, e.g. tukeyhsd
  - cleanup, open issues (check)
  - #4379
  - #4570
  - dunnet
  - unequal var, #7332
- oneway, ranks
  - #799
  - #6994
  - some discussion in #852
  - 
- two-way ???
  - #4323
- noncontinuous data, binary, count ???
- model post-estimation like t_test_pairwise
  - #3297
- other contrasts than all pairs
  - #4361
  - all versus mean ?
  - sequential difference for ordinal, ordered groups
- helper functions, reusable pieces
  - t-max max-range, p-values using mutlivariate normal or t cdf
    - #3297
  - functions for contrasts and categorical codings (reference ???, I just had one but lost it again)
  - grouping letters #3674
- ...

(still incomplete, there are likely open issues and prs missing from this list)


user: josef-pkt
issue_8395: state: open
title: ENH: closed testing procedures for multiple comparisons, pairwise
body: (post-hoc) multiple comparisons like all pairs comparisons need p-value corrections.

We don't have any closed testing procedure that 

However, sandbox.stats.multicomp has several functions or classes for subsets or partitions of comparisons.
This is unfinished, I never spent enough time on this to get a proper version, e.g. R book for multicomp has chapter(s) for it.

current context
non-parametric comparisons for oneway data: several references describe the closed procedure (but it's a bit messy and I didn't look at the details.
essentially all subset or intersections need to be checked sequentially/recursively (tree form ?) for rejection.

based on comments in the nonparametric comparison literature:
All against one comparisons have proper ordering/monotonicity/positive correlation to use e.g. Holm.
All pairwise comparisons can/does have negative and positively correlated pvalues and generic step up/down procedure are not valid, 
Closed procedures for it need to check all intersection hypothesis.
I'm not sure: AFAIR, the literature for closed procedures use reject decisions for each hypothesis, corrected pvalues would need to have a max or cumulative max along the intersection hypothesis paths.

(That's not a target for me anytime soon, but an idea for using or finishing code in sandbox multicomp.)

related:
Is there a connection between `set_partition` function and grouping/letter codes in #3674 ?




user: josef-pkt
issue_8394: state: open
title: ENH: Critical values for no trend parameter in Coint function
body: I am trying to run an Engle-Granger cointegration test using `statsmodels.tsa.stattools`'s `coint` function with no constant and no trend (using parameter `trend="n"`). For some reason it appears that there is no critical values available for this test. A critical value array is simple returned with three NaN values. See below for a minimum reproducible example. I believe this has something to do with MacKinnon (1991) not including critical values for the no trend, no constant version of the test. The `coint` function is calculating its critical values based on this paper and 2010 update to this paper, I believe. Unfortunately, I have not been able to dig in enough to determine whether this is true and if so what appropriate values would be. 

Minimum reproducible examples:
```
import yfinance as yf
from statsmodels.tsa.stattools import coint

df = yf.download("AAPL, MSFT", start="2020-01-01").get("Adj Close")

res = coint(df.iloc[:,1], df.iloc[:,0], trend="n", maxlag=1)

print(res)
```

Returns: 
`>>> (-1.475451313104536, 0.4798252707083076, [nan, nan, nan])`

Here is a (potentially) useful researchgate thread regarding MacKinnon (1991) no trend, no constant critical values:
https://www.researchgate.net/post/Critical_values_Engle-Granger_cointegration_test
user: jaythequant
issue_8393: state: open
title: ENH: Improve detection of number of CPUs 
body: Hello

`tools/parallel.py` uses `multiprocessing.cpu_count()` to ge the number of available cpus which retruns the number of cpu in the machine. But this is not the same as the number of cpu available to the process. For example, you can run in a taskset context or a batch scheduler like slurm.

see:
```
$ nproc
96
$ taskset -c 1 nproc
1
$ taskset -c 1 python3 -c "import multiprocessing; print(multiprocessing.cpu_count())"
96
```
I would suggest to use len(os.sched_getaffinity(0)) instead of multiprocessing.cpu_count()
```
$ python3 -c "import os; print(len(os.sched_getaffinity(0)))"
96
$ taskset -c 1 python3 -c "import os; print(len(os.sched_getaffinity(0)))"
1
```
regards

Eric
user: EricDeveaud
issue_8392: state: open
title: X13 temp file error
body: #### Describe the bug

Trying to run x13 on Windows 10 with statsmodels 0.12.2 and python 3.8.8  Ran the code below


import statsmodels.api as sm

 ###### Set path to folder where the x13 package is downloaded and extracted from census.gov
XPATH = r'C:\Users\fall-\Documents\Job Python\Alpha\TAA\winx13html_v3-1\WinX13\x13as'
os.chdir(XPATH)
os.environ['X13PATH'] = XPATH

data = sm.datasets.macrodata.load_pandas()
df = data.data.set_index(pd.period_range('1/1/1959', '9/1/2009', freq='Q'))
df = df.dropna()
res = sm.tsa.x13_arima_select_order(df['realgdp'], x12path=XPATH, print_stdout=False)



the output is shown below

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-28-346f73075be7> in <module>
----> 1 res = sm.tsa.x13_arima_select_order(df['realgdp'], x12path=XPATH, print_stdout=False)
      2 res.plot()
      3 plt.show()

~\anaconda3\envs\AlgoAddict_Datacamp\lib\site-packages\pandas\util\_decorators.py in wrapper(*args, **kwargs)
    205                 else:
    206                     kwargs[new_arg_name] = new_arg_value
--> 207             return func(*args, **kwargs)
    208 
    209         return cast(F, wrapper)

~\anaconda3\envs\AlgoAddict_Datacamp\lib\site-packages\statsmodels\tsa\x13.py in x13_arima_select_order(endog, maxorder, maxdiff, diff, exog, log, outlier, trading, forecast_periods, start, freq, print_stdout, x12path, prefer_x13)
    572     in.
    573     """
--> 574     results = x13_arima_analysis(endog, x12path=x12path, exog=exog, log=log,
    575                                  outlier=outlier, trading=trading,
    576                                  forecast_periods=forecast_periods,

~\anaconda3\envs\AlgoAddict_Datacamp\lib\site-packages\pandas\util\_decorators.py in wrapper(*args, **kwargs)
    205                 else:
    206                     kwargs[new_arg_name] = new_arg_value
--> 207             return func(*args, **kwargs)
    208 
    209         return cast(F, wrapper)

~\anaconda3\envs\AlgoAddict_Datacamp\lib\site-packages\statsmodels\tsa\x13.py in x13_arima_analysis(endog, maxorder, maxdiff, diff, exog, log, outlier, trading, forecast_periods, retspec, speconly, start, freq, print_stdout, x12path, prefer_x13)
    449             print(p.stdout.read())
    450         # check for errors
--> 451         errors = _open_and_read(ftempout.name + '.err')
    452         _check_errors(errors)
    453 

~\anaconda3\envs\AlgoAddict_Datacamp\lib\site-packages\statsmodels\tsa\x13.py in _open_and_read(fname)
    204 def _open_and_read(fname):
    205     # opens a file, reads it, and make sure it's closed
--> 206     with open(fname, 'r', encoding="utf-8") as fin:
    207         fout = fin.read()
    208     return fout

FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\fall-\\AppData\\Local\\Temp\\tmp_iu7e69r.err'





user: indilyone
issue_8391: state: open
title: ENH: residual diagnostic test for cluster correlation, is something available?
body: I would like to have a diagnostic test, hopefully based on residuals like LM/score tests, for cluster correlation, i.e. specification test equivalent of cluster robust standard errors.

There is a quite a large literature for lm tests for correlation or dependence in panel data #2756 
But I don't find anything for arbitrary cluster correlation, as for example in GEE, in a brief google scholar search.

There is some literature in mixed effects models for testing variance components.

We might be able to get something based on intraclass correlation

As a measure of cluster correlation, we could compute the information matrix ratio, im_ratio, estimated under the no cluster correlation model, (similarly to looking at ratio between cluster robust and nonrobust cov_params or bse, possibly using GEE)

maybe related: hausman test for fixed versus random effect.
but here I want random/mixed effects (that capture correlation) versus no effect.

current motivation: looking at cluster randomized trials.
In analogy to misspecified heteroscedasticity versus misspecified (serial or cross-sectional) correlation in panel data.
In count data we could also have excess dispersion.

(open question)
user: josef-pkt
issue_8390: state: open
title: ENH: stats for cluster randomized trials - count, binomial
body: PASS has power and sample size computation for cluster randomized trials, 
I never looked at the details

Song and Ahn looks like a good starting point, although it has only very few citation. It describes and compares several methods, including GEE.
The variance correction versions take intracluster correlation into account and looks very similar to excess dispersion in beta-binomial, it's the same formula as far as I remember for dispersion coefficient. So we could use those also for over-dispersion.

Song, James X., and Chul Ahn. 2002. “Cluster Randomization Trials: A Simulation Study.” Biometrical Journal 44 (3): 375–90. [https://doi.org/10.1002/1521-4036(200204)44:3<375::AID-BIMJ375>3.0.CO;2-S](https://doi.org/10.1002/1521-4036(200204)44:3<375::AID-BIMJ375>3.0.CO;2-S)

Song and Ahn is for odds-ratio in 2x2xk stratified two sample proportion Cochran-Mantel-Haenszel Test

2 cases among many others that PASS has
Tests_for_Two_Proportions_in_a_Stratified_Cluster-Randomized_Design-Cochran-Mantel-Haenszel_Test.pdf
Tests_for_the_Difference_Between_Two_Poisson_Rates_in_a_Cluster-Randomized_Design.pdf

main reference for sample size and power in cluster CMH in PASS is 
Xu, X., Zhu, H., and Ahn, C. 2019. 'Sample size considerations for stratified cluster randomization design with
binary outcomes and varying cluster size', Statistics in Medicine, Volume 38, Number 18, pages 3395-3404.


related
correlated, paired samples (cluster size = 2)  (mostly missing except for differenced endog in linear models)
excess dispersion in Binomial and Poisson #2890
2x2xk samples #7184
several of new stats functions for poisson rates allow for dispersion coefficient different from 1.

I guess, the same as for odds-ratio in CMH will apply to other proportions_2indep comparisons #4828
e.g. different effect sizes in meta-analysis






user: josef-pkt
issue_8389: state: open
title: ENH: weighted average treatment effect, subgroup analysis, target population
body: (just parking some references, I only read parts of abstracts, list collected by citations and related articles)

This sounds like the weighted ATE equivalent of get_prediction with weights and average is True.
i.e. instead of just ATT and ATNT we also want AT for a target population or subgroup.


Choi, Byeong Yeob. 2021. “Subclassification Estimation of the Weighted Average Treatment Effect.” Biometrical Journal 63 (8): 1706–28. https://doi.org/10.1002/bimj.202000310.

Li, Fan, Ashley L. Buchanan, and Stephen R. Cole. 2022. “Generalizing Trial Evidence to Target Populations in Non-Nested Designs: Applications to AIDS Clinical Trials.” Journal of the Royal Statistical Society: Series C (Applied Statistics) 71 (3): 669–97. https://doi.org/10.1111/rssc.12550.

Tao, Yebin, and Haoda Fu. 2019. “Doubly Robust Estimation of the Weighted Average Treatment Effect for a Target Population.” Statistics in Medicine 38 (3): 315–25. https://doi.org/10.1002/sim.7980.

Yang, Siyun, Fan Li, Laine E Thomas, and Fan Li. 2021. “Covariate Adjustment in Subgroup Analyses of Randomized Clinical Trials: A Propensity Score Approach.” Clinical Trials 18 (5): 570–81. https://doi.org/10.1177/17407745211028588.

Yang, Siyun, Elizabeth Lorenzi, Georgia Papadogeorgou, Daniel M. Wojdyla, Fan Li, and Laine E. Thomas. 2021. “Propensity Score Weighting for Causal Subgroup Analysis.” Statistics in Medicine 40 (19): 4294–4309. https://doi.org/10.1002/sim.9029.

Zeng, Shuxi, Fan Li, Rui Wang, and Fan Li. 2021. “Propensity Score Weighting for Covariate Adjustment in Randomized Clinical Trials.” Statistics in Medicine 40 (4): 842–58. https://doi.org/10.1002/sim.8805.


One article linked to R package PSweight in abstract    https://github.com/thuizhou/PSweight

user: josef-pkt
issue_8388: state: open
title: ENH: multivariate count models, including count MNLogit
body: based on discussion #8380

- multinomial Logit can be extended to counts similar to GLM-Binomial versus Logit (we can reuse or copy-paste large parts of code for MNLogit)
- other multivariate distributions, I have not looked much at it. I had looked at some literature using copulas for multivariate counts.
- maybe: check how much we can reuse a Poisson loglinear model.

just parking some references:
(I didn't read much more than abstracts)

a good starting point is Zhang, Yiwen, Hua Zhou, Jin Zhou, and Wei Sun. 2017. Their online supplement includes score and hessian formulas.
Bhat 2022 seems to have a good introduction with overview of different approaches

Bhat, Chandra R. 2022. “A Closed-Form Multiple Discrete-Count Extreme Value (MDCNTEV) Model.” Transportation Research Part B: Methodological 164 (October): 65–86. https://doi.org/10.1016/j.trb.2022.08.001.

Corsini, Noemi, and Cinzia Viroli. 2022. “Dealing with Overdispersion in Multivariate Count Data.” Computational Statistics & Data Analysis 170 (June): 107447. https://doi.org/10.1016/j.csda.2022.107447.

Das, Ishapathik, Sumen Sen, N. Rao Chaganty, and Pooja Sengupta. 2019. “Regression for Doubly Inflated Multivariate Poisson Distributions.” Journal of Statistical Computation and Simulation 89 (13): 2549–61. https://doi.org/10.1080/00949655.2019.1625051.

Inouye, David I., Eunho Yang, Genevera I. Allen, and Pradeep Ravikumar. 2017. “A Review of Multivariate Distributions for Count Data Derived from the Poisson Distribution.” WIREs Computational Statistics 9 (3): e1398. https://doi.org/10.1002/wics.1398.

Kim, Juhyun, Yiwen Zhang, Joshua Day, and Hua Zhou. 2018. “MGLM: An R Package for Multivariate Categorical Data Analysis.” The R Journal 10 (1): 73–90.

Koochemeshkian, Pantea, Nuha Zamzami, and Nizar Bouguila. 2020. “Flexible Distribution-Based Regression Models for Count Data: Application to Medical Diagnosis.” Cybernetics and Systems 51 (4): 442–66. https://doi.org/10.1080/01969722.2020.1758464.

Ma, Zichen, Timothy E. Hanson, and Yen-Yi Ho. 2020. “Flexible Bivariate Correlated Count Data Regression.” Statistics in Medicine 39 (25): 3476–90. https://doi.org/10.1002/sim.8676.

Peyhardi, Jean, Pierre Fernique, and Jean-Baptiste Durand. 2021. “Splitting Models for Multivariate Count Data.” Journal of Multivariate Analysis 181 (January): 104677. https://doi.org/10.1016/j.jmva.2020.104677.

Terza, Joseph V., and Paul W. Wilson. 1990. “Analyzing Frequencies of Several Types of Events: A Mixed Multinomial- Poisson Approach.” The Review of Economics and Statistics 72 (1): 108–15. https://doi.org/10.2307/2109745.

Zhang, Yiwen, Hua Zhou, Jin Zhou, and Wei Sun. 2017. “Regression Models for Multivariate Count Data.” Journal of Computational and Graphical Statistics 26 (1): 1–13. https://doi.org/10.1080/10618600.2016.1154063.


user: josef-pkt
issue_8387: state: open
title: ENH: MNLogit for fractional data, allow and verify
body: based on discussion in #8380

loglike score and hessian in MNLogit seem to work for fractional data.
This needs to be verified with some examples and results need to be checked.

As it seems to be ok, we can allow 2-dim endog in MultinomialModel._handle_data, i.e. don't convert endog to wendog from categorical labels to dummy representation and use provided endog as wendog. with I guess some checks like rowsum = 1.

This should work as QMLE with cov_type="HC0"


user: josef-pkt
issue_8385: state: open
title: ENH: add Diebold-Mariano forecast comparison test
body: - [ ] closes #xxxx
- [x] tests added / passed. 
- [x] code/documentation is well formatted.  
- [x] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

<details>
Related to issue #8374 

I found an implementation on this github https://github.com/johntwk/Diebold-Mariano-Test which I modified and adapted to the code base. It produces the same results has the R package. Tests are implemented, but I am not very familiar with unit tests and therefore they may not be well done. 

I am currently working on the multivariate version of the test (more then 2 forecasts).
**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in main and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/main -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>

user: alekracicot
issue_8384: state: open
title: [BUG] Cannot predict after SARIMAXResults `remove_data` method called
body: #### Describe the bug

Cannot predict after SARIMAXResults `remove_data` method called. As an example, for my project the statsmodel results object is ~2 Gb and `remove_data` reduced this to ~200 Mb (which I was super excited about!), but then unable to reuse results object to make predictions and so rather useless.

#### Code Sample, a copy-pastable example if possible
Code example taken from statsmodels sarimax documentation here: https://github.com/statsmodels/statsmodels/issues/new?assignees=&labels=&template=bug_report.md&title=

```python
import numpy as np
import pandas as pd
from scipy.stats import norm
import statsmodels.api as sm
import matplotlib.pyplot as plt
from datetime import datetime
import requests
from io import BytesIO

# Dataset
wpi1 = requests.get('https://www.stata-press.com/data/r12/wpi1.dta').content
data = pd.read_stata(BytesIO(wpi1))
data.index = data.t
# Set the frequency
data.index.freq="QS-OCT"

# Fit the model
mod = sm.tsa.statespace.SARIMAX(data['wpi'], trend='c', order=(1,1,1))
res = mod.fit(disp=False)
print(res.summary())

# Prediction
res.forecast(5)  # this works, but file size is very big

# Remove data & re-run prediction
res.remove_data()  # reduces file size substantially
res.forecast(5)  # does not work! (error msg below)
```
<details>

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `main`. If your problem has been fixed in an unreleased version, you might be able to use `main` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the main branch of this repository? It helps the limited resources if we know problems exist in the current main branch so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

```
1991-01-01    118.358862
1991-04-01    120.340500
1991-07-01    122.167204
1991-10-01    123.858458
1992-01-01    125.431301
Freq: QS-OCT, Name: predicted_mean, dtype: float64
```

#### Error message

```
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/workspace/scripts/../smartshift_load_forecasting/model/arima_fourier_nem/model.py", line 37, in get_forecast_results
    new_forecast = fitted_model.forecast(steps=n_steps, exog=exog)
  File "/virtualenvs/smartshift-load-forecasting-9TtSrW0h-py3.9/lib/python3.9/site-packages/statsmodels/base/wrapper.py", line 113, in wrapper
    obj = data.wrap_output(func(results, *args, **kwargs), how)
  File "/virtualenvs/smartshift-load-forecasting-9TtSrW0h-py3.9/lib/python3.9/site-packages/statsmodels/tsa/statespace/mlemodel.py", line 3442, in forecast
    return self.predict(start=self.nobs, end=end, **kwargs)
  File "/virtualenvs/smartshift-load-forecasting-9TtSrW0h-py3.9/lib/python3.9/site-packages/statsmodels/tsa/statespace/mlemodel.py", line 3403, in predict
    prediction_results = self.get_prediction(start, end, dynamic, **kwargs)
  File "/virtualenvs/smartshift-load-forecasting-9TtSrW0h-py3.9/lib/python3.9/site-packages/statsmodels/tsa/statespace/mlemodel.py", line 3287, in get_prediction
    self.model._get_prediction_index(start, end, index))
  File "/virtualenvs/smartshift-load-forecasting-9TtSrW0h-py3.9/lib/python3.9/site-packages/statsmodels/tsa/base/tsa_model.py", line 833, in _get_prediction_index
    nobs = len(self.endog)
TypeError: object of type 'NoneType' has no len()
```

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]
INSTALLED VERSIONS
------------------
Python: 3.10.4.final.0
OS: Linux 5.14.0-1048-oem #55-Ubuntu SMP Mon Aug 8 14:58:10 UTC 2022 x86_64
byteorder: little
LC_ALL: None
LANG: en_AU.UTF-8

statsmodels
===========

Installed: 0.13.0 (/home/julian/anaconda3/envs/smartshift/lib/python3.10/site-packages/statsmodels)

Required Dependencies
=====================

cython: 0.29.30 (/home/julian/anaconda3/envs/smartshift/lib/python3.10/site-packages/Cython)
numpy: 1.23.1 (/home/julian/anaconda3/envs/smartshift/lib/python3.10/site-packages/numpy)
scipy: 1.7.3 (/home/julian/anaconda3/envs/smartshift/lib/python3.10/site-packages/scipy)
pandas: 1.4.0 (/home/julian/anaconda3/envs/smartshift/lib/python3.10/site-packages/pandas)
    dateutil: 2.8.2 (/home/julian/anaconda3/envs/smartshift/lib/python3.10/site-packages/dateutil)
patsy: 0.5.2 (/home/julian/anaconda3/envs/smartshift/lib/python3.10/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.5.2 (/home/julian/anaconda3/envs/smartshift/lib/python3.10/site-packages/matplotlib)
    backend: module://matplotlib_inline.backend_inline 
cvxopt: Not installed
joblib: 1.1.0 (/home/julian/anaconda3/envs/smartshift/lib/python3.10/site-packages/joblib)

Developer Tools
================

IPython: 8.4.0 (/home/julian/anaconda3/envs/smartshift/lib/python3.10/site-packages/IPython)
    jinja2: Not installed
sphinx: Not installed
    pygments: 2.11.2 (/home/julian/anaconda3/envs/smartshift/lib/python3.10/site-packages/pygments)
pytest: Not installed
virtualenv: Not installed

</details>

user: jelc53
issue_8382: state: open
title: ENH: power, sample size computation for rank_compare, brunner-munzel, Wilcoxon-Mann-Whitney
body: Currently we don't have power and sample size functions for statsmodels.stats.nonparametric.RankCompareResult

I just found this reference, but didn't specifically search for this topic

Happ, Martin, Arne C. Bathke, and Edgar Brunner. 2019. “Optimal Sample Size Planning for the Wilcoxon-Mann-Whitney Test.” Statistics in Medicine 38 (3): 363–75. https://doi.org/10.1002/sim.7983.

related, but we don't have anything yet on group-sequential trials
Nowak, Claus P, Tobias Mütze, and Frank Konietschke. 2022. “Group Sequential Methods for the Mann-Whitney Parameter.” Statistical Methods in Medical Research, June, 09622802221107103. https://doi.org/10.1177/09622802221107103.


user: josef-pkt
issue_8381: state: open
title: mixed effects absence: No way to model heteroskedasticity, choose covariance format of residuals, compute normalized residuals
body: Definition of "normalized residuals": the normalized residuals=resid(x,type='normalized')= (standardized residuals pre-multiplied by the inverse square-root factor of the estimated error/working correlation matrix) as is available from the nlme package.
user: DexterGlass
issue_8380: state: open
title: ENH: MNLogit for counts or frequencies, similar to Binomial
body: Maybe I have missed it in the library, but is there a way to use frequency weights for multinomial logistic regression?  If not, I would like to request it.

**edit** question is for fractional or count endog in MNLogit and not for `freq_weights`
see discussion below
user: brett1479
issue_8379: state: open
title: ENH: add one-sided option to test_ordinal_association, cochran armitage trend test
body: Currently test_ordinal_association only allows for two-sided trend test.
related issue #1130 k-sample trend tests

instead we often want to test also for one-sided alternatives, e.g. positive trend in dose-response models

reference, e.g.

Neuhäuser, Markus, and Ludwig A. Hothorn. 1999. “An Exact Cochran–Armitage Test for Trend When Dose–Response Shapes Are a Priori Unknown.” Computational Statistics & Data Analysis 30 (4): 403–12. https://doi.org/10.1016/S0167-9473(98)00091-7.


this looks easy because the implementation already uses the normal distribution.

I haven't checked yet: Can we add non-zero hypothesis, e.g. linear trend > low(er bound)?
(similar to non-inferiority tests, minimum non-negligible trend)
I guess it should be fine, it's just a wald-test on the trend slope coefficient.


user: josef-pkt
issue_8376: state: open
title: LinAlgError: SVD did not converge error when fitted with SARIMAX model
body: When I fitted the SARIMAX model with 10 data points, it failed to converge when the model tried to fit. I found this when I tried to look at the summary.


```python
import numpy as np
import pandas as pd
from statsmodels.tsa.statespace.sarimax import SARIMAX
import random

y = []
# Set a length of the list to 10
for i in range(0, 10):
    x = round(random.uniform(50, 500), 2)
    y.append(x)

model = SARIMAX(y, order=(0,2,2), seasonal_order=(0,0,0,0))
fitted_model = model.fit()
forecast_results = fitted_model.forecast(12)

```
<details>

ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
  warnings.warn("Maximum Likelihood optimization failed to "

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

A clear and concise description of what you expected to happen.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]
0.13.2

</details>

user: ckwan48
issue_8374: state: open
title: Adding DM & HLN tests to statsmodels
body: #### Is your feature request related to a problem? 
Not related to a problem, but I often want to test if a forecast is statistically different from another forecast. I don't see any implementation of the DM or HLN test in statsmodels.  

#### Describe the solution you'd like
A function taking three arrays of equal lengths representing the forecast 1, forecast 2 & the observed values as input and outputs the test statistic and p-value.

#### Describe alternatives you have considered
Otherwise, maybe a function taking fitted models, a regressor set and a dependent variable vector as input  and outputs the test statistic and p-value.

#### Additional context
Let me know if this is something that belongs in statsmodels. I already have "bad" code for the two tests. I would be interested to implement it myself in statsmodels. I would need directions regarding where such a test should be implemented.
user: alekracicot
issue_8362: state: open
title: plot_acf and acf present incoherent results between then
body: #### Describe the bug

For the following data, plot_acf is presenting the confidence interval on plot different then the acf pure function.



#### Code Sample, a copy-pastable example if possible
OBS.: I tried to create a example with numpy and random data, but the bug did not show.

```python
my_list = [2916.0, 3126.0, 2900.0, 2972.0, 2958.0, 2980.0, 3016.0, 3056.0, 3012.0, 3024.0, 2932.0, 2832.0, 2554.0, 2768.0, 2888.0, 2886.0, 2964.0, 2916.0, 2886.0, 2664.0, 2580.0, 2836.0, 2716.0, 2708.0, 2740.0, 2860.0, 2812.0, 2832.0, 2800.0, 2844.0, 2892.0, 2896.0, 2856.0, 2716.0, 2908.0, 2840.0, 2820.0, 2808.0, 2792.0, 2756.0, 2734.0, 2700.0, 2782.0, 2764.0, 2784.0, 2816.0, 2832.0, 2808.0, 2820.0, 2776.0, 2666.0, 2780.0, 2744.0, 2856.0, 2424.0, 2756.0]

# plot_acf
from statsmodels.graphics import tsaplots
fig = tsaplots.plot_acf(df["rad_max"], missing='conservative', zero=False, auto_ylims=True, alpha=0.05)
plt.show()

# acf
import statsmodels.api as sm
sm.tsa.acf(my_list, missing='conservative', alpha=0.05, bartlett_confint=True)

```
<details>

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `main`. If your problem has been fixed in an unreleased version, you might be able to use `main` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the main branch of this repository? It helps the limited resources if we know problems exist in the current main branch so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

I expected to see the confidence interval in the result of acf function. I read the source code and get that besides my alpha parameter, confidence is begin calculate by x_acf - confidence[0], and x_acf - confidence[1].

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

INSTALLED VERSIONS
------------------
Python: 3.7.12.final.0
OS: Linux 5.10.133+ #1 SMP Fri Jul 29 14:49:43 UTC 2022 x86_64
byteorder: little
LC_ALL: C.UTF-8
LANG: C.UTF-8

statsmodels
===========

Installed: 0.13.2 (/opt/conda/lib/python3.7/site-packages/statsmodels)

Required Dependencies
=====================

cython: 0.29.30 (/opt/conda/lib/python3.7/site-packages/Cython)
numpy: 1.21.6 (/opt/conda/lib/python3.7/site-packages/numpy)
scipy: 1.7.3 (/opt/conda/lib/python3.7/site-packages/scipy)
pandas: 1.3.5 (/opt/conda/lib/python3.7/site-packages/pandas)
    dateutil: 2.8.2 (/opt/conda/lib/python3.7/site-packages/dateutil)
patsy: 0.5.2 (/opt/conda/lib/python3.7/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.5.2 (/opt/conda/lib/python3.7/site-packages/matplotlib)
    backend: module://matplotlib_inline.backend_inline 
cvxopt: Not installed
joblib: 1.0.1 (/opt/conda/lib/python3.7/site-packages/joblib)

Developer Tools
================

IPython: 7.33.0 (/opt/conda/lib/python3.7/site-packages/IPython)
    jinja2: 3.1.2 (/opt/conda/lib/python3.7/site-packages/jinja2)
sphinx: Not installed
    pygments: 2.12.0 (/opt/conda/lib/python3.7/site-packages/pygments)
pytest: 7.1.2 (/opt/conda/lib/python3.7/site-packages/pytest)
virtualenv: 20.15.1 (/opt/conda/lib/python3.7/site-packages/virtualenv)

</details>

user: gregomelo
issue_8361: state: open
title: ENH: discrete (count) models, implement cov_type excess dispersion
body: We could add excess dispersion `scale` option to cov_type in discrete models similar to GLM.

This is mainly relevant for count models, Poisson, NB, GP. 
I don't remember the details how excess dispersion affects binary models. (there is no extra variance parameter in binary models, it just rescales the "true" underlying params.) 

The motivatin usecase is estimating NB2 with dispersion parameter alpha (available in discrete, but not in GLM) but with simple excess dispersion, scale factor for variance function, as in GLM, as alternative to using HC0.
user: josef-pkt
issue_8359: state: open
title: DOC/SUMM  missing notebooks (outside tsa)
body: What notebooks do we need to add, especially for recently added features?

I don't have an overview and most of my private notebooks are dev notebooks with extra experiments or using outdated pre-merge api.
my 0.14 roadmap #7720
overview postestimation #7707

- generic, general
  - cov_type, AFAIR we don't have a notebook in examples nor in rst docs 
- for old code
  - still no comprehensive doc, notebook for cov_type options
  - no GAM notebook
  - asymmetric kernel pdf, cdf, bernstein
- post-estimation diagnostics
  - influence: docs include influence for logit/glm https://www.statsmodels.org/dev/examples/notebooks/generated/influence_glm_logit.html , more recent version in my folder includes additional trys/experiments
  - diagnostic: a few other notebooks when I was working on diagnostic (2021-12-25)  try_betareg_prediction_scoretest.ipynb try_influence_zip.ipynb try_truncatedpoisson_new.ipynb try_discrete_get_distribution.ipynb Poisson_diagnostic_.ipynb (I didn't look at those)
  - get_prediction need some examples for new features similar to margins and predicted means
  - nonlinear prediction wald_nonlinear, not yet public, notebook for use for margins, see issue #5387
  - wald tests, I think we don't have any good summary, quick overview, see e.g. https://stackoverflow.com/a/34232975/333700
    good would be full example including wald_test_terms and pairwise 
  - get_prediction: A good example would be for how to use it to simulate a model given exog. Monte Carlo.
  -  merged
     - [x] overview with poisson merged #8420
- stats
  has very few notebooks in current docs
  - oneway
    several notebooks with examples, dev version and experiments, not systematic, 
    try_welsh_anova.ipynb (includes oneway for scale/dispersion),  try_oneway_equivalence_power_newnames.ipynb, ...
  - nonparametric brunner-munzel, rank_compare
    - `ex_rank_compate.ipynb` mostly dev notebook, 6000 lines, uses merged functions, Munzel and Hauschke example
    - notebook `try_brunner_munzel.ipynb` is original dev notebook, does not use merged functions, has Example from Neuhaeuser and Poulin 2004 and Example from Munzel and Hauschke 2003
    - [x]  #8427
  - poisson: no good notebook, outdated api, not much text, 
    too many functions and methods, need to write short overview version
    - most recent is `check_poisson_inference_pr.ipynb`
    - [x] #8412
  - proportion (can wait until after refactoring)
    - `ex_proportion_2indep_short.ipynb` basic example with all compare and methods in folder notebook_old
    - `try_anovapower_proportionztest.ipynb` basic example for how to use FTestAnovaPower with effect size
  - ...
- newer models
  - count models, only poisson and negbin in https://www.statsmodels.org/dev/examples/notebooks/generated/discrete_choice_overview.html#Poisson
    missing: zero-inflated, new hurdle, NBP, GPP, ... 
    - [x] hurdle: no approximately clean notebook, `try_truncated_hurdle_count.ipynb` includes racd10 example, other notebooks use simulated data
       hurdle notebook merged #8424 
    - truncated not included in hurdle notebook
  - beta
    - notebook in a gist, might be the cleanest https://gist.github.com/josef-pkt/4d0fd829c8fbc1c4237d989ba3dbb088
  - ...
- treatment
  - `ex_treatment_effect_after_merge.ipynb` example ok, needs some cleanup and more text  
  - [x] #8418
- multivariate
  only has PCA
- ... 
user: josef-pkt
issue_8357: state: open
title: MixedLM Predict issue
body: Hi,

I am doing some research on Mixed effects models and using Stats Mixedlm for same. I have Units to be predicted from Price and size at different UPC levels.

So below is the code I used - 

md = smf.mixedlm("Units ~ Avg_Units_Price + C_CUSTOM_BASE_SIZE", file3, groups=file3["UPC_10_digi"], re_formula="~Avg_Units_Price + C_CUSTOM_BASE_SIZE")
mdf = md.fit()
print(mdf.summary())

## Output

![image](https://user-images.githubusercontent.com/82749367/182370799-9ce53179-3bfa-4258-8ce2-423b734de531.png)

mdf.random_effects

![image](https://user-images.githubusercontent.com/82749367/182370934-63aaeddb-15cb-4171-ab1a-0eac9c8ba0a5.png)

I am getting moderate avp fit on predicted values however I have below needs - 

1) I want to predict Units using both fixed and random effects as in below case - 

![image](https://user-images.githubusercontent.com/82749367/182371361-663d832c-7e55-49ed-990c-551ed7285ded.png)


- so I want to predict units at size 20 and Price 11.1. I know for predicting these values both fixed effects and random effects would be used, however I am not able to use predict function as I couldnt get clear example/idea for this.
I have used below code for prediction but its showing multiple arguments error -

![image](https://user-images.githubusercontent.com/82749367/182371874-69db7c35-fede-41c0-bb18-9f01456ae654.png)


Can you please help me in this single case prediction using Mixedlm?

Thanks a lot in advance.  
user: prashanthin
issue_8337: state: open
title: [Refactorings]Few code optimizations 
body: - [ ] closes #xxxx **not a reported issue**
- [ ] tests added / passed. 
- [ ] code/documentation is well formatted.  
- [ ] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

The Numpy [API documentation](https://numpy.org/doc/stable/reference/generated/numpy.linalg.multi_dot.html) recommends using `np.linalg.multi_dot` instead of multiple `np.dot` calls since it is more succinct and effective. This PR is to make the required changes. 

user: maldil
issue_8336: state: open
title: Nested ANOVA
body: #### The issue

When trying to compute two-way nested ANOVA, the results do not equal the appropriate results from R (formulas and data are the same). 


####  Sample

We use "atherosclerosis" dataset from here: https://stepik.org/media/attachments/lesson/9250/atherosclerosis.csv.
To get nested data we replace dose values for age == 2:
```python
df['dose'] = np.where((df['age']==2) & (df['dose']=='D1'),'D3', df.dose)
df['dose'] = np.where((df['age']==2) & (df['dose']=='D2'),'D4', df.dose)
```

So we have dose factor nested into age: values D1 and D2 are in first age and values D3 and D4 are only in the 2nd age.
After getting ANOVA table we have the results below:
```python
mod = ols('expr~age/C(dose)', data=df).fit()
anova_table = sm.stats.anova_lm(mod, typ=1); anova_table
```
![Снимок экрана от 2022-07-01 15-13-30](https://user-images.githubusercontent.com/108528316/176892226-2a412afa-3733-460e-8721-ce7b5f09872a.png)

The total sum of the 'sum_sg' = 1590.257424 + 47.039636 + 197.452754 = 1834.7498139999998 that is NOT equal the right total sum  (computed below) = 1805.5494956433238
```python
grand_mean = df['expr'].mean()
ssq_t = sum((df.expr - grand_mean)**2)
```

#### Expected Output

Let's try to get ANOVA table in R:

```python
df <- read.csv(file = "/mnt/storage/users/kseniya/platform-adpkd-mrwda-aim-imaging/mrwda_training/data_samples/athero_new.csv") 
nest <- aov(df$expr ~ df$age / factor(df$dose))
print(summary(nest))
```
The results:
![Снимок экрана от 2022-07-01 15-22-44](https://user-images.githubusercontent.com/108528316/176893579-af796a0f-1c3a-4c49-93a7-91850da342a3.png)

Why they are not equal? The formulas are the same. Are there any mistakes in computing ANOVA through statsmodels?

The results from R seem to be right, because the total sum 197.5 + 17.8 + 1590.3 = 1805.6 is equal to the total sum computed manually.

user: kseniawa
issue_8331: state: open
title: FAQ/SUMM: Overview of hypothesis tests and related inference (stats)
body: (I don't have an overview and I didn't find an overview issue)

What functions do we have for hypothesis tests and related inference outside of models? 
What functionality should we still add?
There is some overlap with diagnostic and specification tests in post-estimation. #7894
Some overlap with inference in models, that are equivalent or close to standalone inference functions #8145 (issue still empty)

Currently the best overview is the rst doc for `stats`.
We have several large overview issues for proportion and rates, but not much that provides an overview of all different cases.
We also have some overview issues for specific kind of analysis, for example equivalence testing #6497 and power functions #8162.


user: josef-pkt
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_8328: state: open
title: ENH: Functions for summarizing multiple model fitting results.
body: - [ ] closes (None)
- [ ] tests (not tested)
- [x] code/documentation is well formatted.  
- [x] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 


Although there is a question about how to handle multiple summaries with `summary_col`. 
[Python: Do not show dummies in statsmodels summary](https://stackoverflow.com/questions/49471014/python-do-not-show-dummies-in-statsmodels-summary)

However, I thought more featured functions to achieve easy functions of handling with multiple fitting results were needed, and I tried to implement simple one with just horizontal concatenations, and one version which lets a mosaic variable to control layout of outputs. 

See the link below for practical usage. 
https://github.com/toshiakiasakura/statsmodels/blob/summary_multi/examples/notebooks/summary_multi_usage.ipynb

If there is a further suggestion, improvement or need to open a issue, please tell me. 
user: toshiakiasakura
issue_8327: state: open
title: DOC/REF/ENH: follow-up to merging treatment effect
body: I merged treatment effect #8034 essentially in the version where I stopped 4 months ago

Docs still need a lot of work, and we need a notebook (unit test has full example).
Simple example in class docstring would also be helpful.
The current docs are not very informative, no explanations and details (users would have to read the Stata docs)
`TreatmentEffectResults` class has no docstring.

The docs don't indicate that only OLS is supported as outcome model.
Need to warn or raise for current limitations.
current unit tests are OLS + Probit
I guess GLM-Binomial is not supported for treatment model, only Logit and Probit. ???

from_data method is not implemented


**update**

signature of treatment class might not be good
`__init__(self, model, treatment, results_select=None, _cov_type="HC0",
                 **kwds):`

why is results select_optional?
why is results_select after treatment?
design/signature if only one model is used,  ipw and ra?

we always need the outcome and the treatment variable, 
outcome variable is only in outcome `model`, 
treatment indicator is `treatment` argument but also in results_select if that is provided

signature might be a left over of the original implementation for ra




user: josef-pkt
issue_8326: state: open
title: statsmodels.distributions.edgeworth.ExpandedNormal - 4th cumulant higher than 4
body: In line 166 of edgeworth.py  is checked if the imag-part is zero and if abs(r) is smaller than 4. If I choose a fourth cumulant higher than 4 I get this warning and yes in this case the imag-parts are zero and abs(r) is smaller 4. But: where does this limit of 4 comes from? 
Is it based on a paper? 
By the way: I assume that the first cumulant is zero and the second is 1. (scaling, centering in line 163,164) I use statsmodels 0.13.2
user: cube2022
issue_8323: state: open
title: REF: deprecate lower case link classes, duplicates that differ only in capitalization
body: We have many links with lower case names that are duplicate, aliases of link classes with capitalized class names.

We should deprecate them with longer deprecation cycle given that both types of names are commonly used.
Removing names that only differ in capitalization, will remove many doc build warnings.

Problem is that user provides a link instance, so we cannot just switch link names internally in GLM, GEE.
We need to figure out where to put the deprecation warning. (I haven't checked yet.

user: josef-pkt
issue_8322: state: open
title: FAQ-D: doc build, sphinx warnings
body: (notes on doc build, sphinx warnings, put here to make it easier to find)

### section title in properties of classes

example reference section in a cached property
`.../statsmodels/miscmodels/ordinal_model.py:docstring of statsmodels.miscmodels.ordinal_model.OrderedResults:81: CRITICAL: Unexpected section title.`

can be ignored, we have many of those
sections in docstrings of properties are removed in the class docstring, but rendered correctly in docstring of property

### duplicate label, names differ only in capitalization

example: links have many duplicates with capitalized and lower case names, e.g.
`/statsmodels/docs/source/generated/statsmodels.genmod.families.links.cloglog.rst:2: WARNING: duplicate label generated/statsmodels.genmod.families.links.cloglog:statsmodels.genmod.families.links.cloglog, other instance in /home/runner/work/statsmodels/statsmodels/docs/source/generated/statsmodels.genmod.families.links.CLogLog.rst`

can be ignored in docbuild, 
but try to avoid public, documented names of objects that only differ in capitalization


user: josef-pkt
issue_8319: state: open
title: DOC/REF followup to 8166, refactoring and enhancing poisson stats
body: rates.rst wrong function names

https://www.statsmodels.org/dev/generated/statsmodels.stats.rates.test_poisson_2indep.html
- [x] `alternative` description is only for `ratio`

https://www.statsmodels.org/dev/generated/statsmodels.stats.rates.tost_poisson_2indep.html
- [x] docstring not updated for "compare" option, i.e. support for "diff"
- [x] same for nonequivalence https://www.statsmodels.org/dev/generated/statsmodels.stats.rates.nonequivalence_poisson_2indep.html

https://www.statsmodels.org/dev/generated/statsmodels.stats.rates.power_poisson_diff_2indep.html
rendering of References uses extra lines

method = "auto"  check defaults and their description again

user: josef-pkt
issue_8310: state: open
title: Run VARMAX with conditinal MLE
body: Hi everyone, 

is it possible to **run the statsmodels VARMAX function with a conditinal MLE** like in the statsmodels VAR function? The approximate solutions of the VAR are sufficient for my case and the VARMAX has a very long runtime. 
Afterwards, I need to get the impulse response function of the VAR with impulses different from the identity matrix which is the reason why I cannot simply use the statsmodels VAR function. 

Thank you a lot in advance!!
user: redzebras
issue_8306: state: open
title: Bad links of user examples?
body: Hello! It seems that on the page [https://github.com/statsmodels/statsmodels/wiki/Examples](https://github.com/statsmodels/statsmodels/wiki/Examples), the User-contributed examples lead to "400 : Bad Request" in nbviewer (I tested all the GEE links in the end, and some in other sections too). It would be grateful to have these nice examples available again! Thanks!
user: chemman26
issue_8304: state: open
title: ENH: add get_prediction for RLM
body: there is no get_prediction for RLM yet
https://stats.stackexchange.com/questions/578398/getting-confidence-interval-for-prediction-from-statsmodel-robust-linear-model

given that prediction is a linear function we can use the standard method for the confidence interval for predicted mean, conditional expectation.

Not clear whether we want to assume normality to get a prediction interval (for new endog) as in OLS.


user: josef-pkt
issue_8303: state: open
title: ENH: power for poisson 2indep, more options, e.g. diff with nonzero value, Gu and others
body: #8251 similar issue for proportion
followup to #8166 
#8138 general issue for rates, poisson

The power functions in PR #8166 do not cover `compare` and methods systematically.
It currently includes Zhu and Lakkis for ratio, and basic equality test for diff.
Also there is no support yet for different exposures in the two samples. Zhu and Lakkis use average exposure for power and sample size computation.
(more important for NegBin than Poisson)

Basic extension as in #8251 is to add value to power for diff.
e.g. Stucke and Kieser 2013
which just looks like an extension of the current power-diff function.

one-tail sample size function equivalent of samplesize_proportions_2indep_onetail are also missing
e.g. Stucke, Kieser equ (5) and (6)

Others:
PASS also has power function for tests by Gu et al for ratio which would match test_poisson_2indep more closely.

related:
#8166 also does not include power function for 1-sample case


user: josef-pkt
issue_8302: state: open
title: How can I use version 0.14 ??
body: The docs show in version 0.14, acf() and plot_acf() functions have the bartlett_confint() parameter that allows to turn off the use of bartlett's formula. However this version is not available through pip or conda, and installing from source through git is also failing. Any tips on how I can use this??
user: ananda1996ai
issue_8300: state: open
title: FAQ: NegativeBinomial regression's default uses quassi method? Reason for differences with R's glm.nb.
body: I recently tried to use negative binomial regression, but I found the result was not matched with R' mass glm.nb, and the same report was raised also in the StackOverflow. In my case, standard deviation was largely differed.
- [Python Negative Binomial Regression - Results Don't Match those from R](https://stackoverflow.com/questions/42277532/python-negative-binomial-regression-results-dont-match-those-from-r)

After some investigations, I found the results can be matched if I set cov_type="HC0". 
and so, my question is that which is corresponding to quasi-likelihood method? or right implementation? or what causes this difference?  
Since in the case of poisson regression, if over-dispersion is taken into consideration, cov_type ="HC0" is specified. 
However, in the case of negative binomial regression, the estimated results are matched when cov_type is specified. 

Here is the code to reproduce my results. Point estimates are similar but standard deviations are different.

```python
import statsmodels.api as sm
import statsmodels.formula.api as smf
data = sm.datasets.get_rdataset("warpbreaks", "datasets").data
model = smf.glm("breaks ~ wool + C(tension, Treatment('L'))", data=data, family=sm.families.NegativeBinomial() ).fit()
display(model.summary())
model = smf.glm("breaks ~ wool + C(tension, Treatment('L'))", data=data, family=sm.families.NegativeBinomial() ).fit(cov_type='HC0')
display(model.summary())
```

```
  Generalized Linear Model Regression Results                  
==============================================================================
Dep. Variable:                 breaks   No. Observations:                   54
Model:                            GLM   Df Residuals:                       50
Model Family:        NegativeBinomial   Df Model:                            3
Link Function:                    Log   Scale:                          1.0000
Method:                          IRLS   Log-Likelihood:                -233.75
Date:                Tue, 07 Jun 2022   Deviance:                       7.1640
Time:                        02:25:38   Pearson chi2:                     7.05
No. Iterations:                     6   Pseudo R-squ. (CS):            0.05123
Covariance Type:            nonrobust                                         
===================================================================================================
                                      coef    std err          z      P>|z|      [0.025      0.975]
---------------------------------------------------------------------------------------------------
Intercept                           3.6693      0.276     13.290      0.000       3.128       4.210
wool[T.B]                          -0.1823      0.277     -0.658      0.511      -0.726       0.361
C(tension, Treatment('L'))[T.H]    -0.5102      0.340     -1.503      0.133      -1.176       0.155
C(tension, Treatment('L'))[T.M]    -0.2934      0.339     -0.866      0.386      -0.957       0.371
===================================================================================================
                 Generalized Linear Model Regression Results                  
==============================================================================
Dep. Variable:                 breaks   No. Observations:                   54
Model:                            GLM   Df Residuals:                       50
Model Family:        NegativeBinomial   Df Model:                            3
Link Function:                    Log   Scale:                          1.0000
Method:                          IRLS   Log-Likelihood:                -233.75
Date:                Tue, 07 Jun 2022   Deviance:                       7.1640
Time:                        02:25:38   Pearson chi2:                     7.05
No. Iterations:                     6   Pseudo R-squ. (CS):            0.05123
Covariance Type:                  HC0                                         
===================================================================================================
                                      coef    std err          z      P>|z|      [0.025      0.975]
---------------------------------------------------------------------------------------------------
Intercept                           3.6693      0.107     34.217      0.000       3.459       3.879
wool[T.B]                          -0.1823      0.099     -1.840      0.066      -0.377       0.012
C(tension, Treatment('L'))[T.H]    -0.5102      0.122     -4.168      0.000      -0.750      -0.270
C(tension, Treatment('L'))[T.M]    -0.2934      0.126     -2.320      0.020      -0.541      -0.046
===================================================================================================

```

And here is the R output.
```R
library(MASS)
library(datasets)
wb.negbin <- glm.nb(breaks~wool+tension, data=warpbreaks)
summary(wb.negbin)
```
```
Call:
glm.nb(formula = breaks ~ wool + tension, data = warpbreaks, 
    init.theta = 9.944385436, link = log)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-2.0144  -0.9319  -0.2240   0.5828   1.8220  

Coefficients:
            Estimate Std. Error z value Pr(>|z|)    
(Intercept)   3.6734     0.0979  37.520  < 2e-16 ***
woolB        -0.1862     0.1010  -1.844   0.0651 .  
tensionM     -0.2992     0.1217  -2.458   0.0140 *  
tensionH     -0.5114     0.1237  -4.133 3.58e-05 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for Negative Binomial(9.9444) family taken to be 1)

    Null deviance: 75.464  on 53  degrees of freedom
Residual deviance: 53.723  on 50  degrees of freedom
AIC: 408.76

Number of Fisher Scoring iterations: 1


              Theta:  9.94 
          Std. Err.:  2.56 

 2 x log-likelihood:  -398.764 
```


user: toshiakiasakura
issue_8298: state: open
title: Conda install causes important packages to DOWNGRADE
body: #### Describe the bug
On Windows 10, when I try to do "conda install statsmodels" it warns "The following packages with be DOWNGRADED:". This occurs even if I run "conda install --freeze-installed statsmodels"

This is very bad since it is downgrading important packages (see details below). 

<details>
C:\Users\David>conda install -d --freeze-installed statsmodels
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: C:\Users\David\miniconda3\envs\idp_new

  added / updated specs:
    - statsmodels


The following NEW packages will be INSTALLED:

  blas               pkgs/main/win-64::blas-1.0-mkl
  openmp             intel/win-64::openmp-2018.0.3-intel_0
  patsy              pkgs/main/win-64::patsy-0.5.2-py39haa95532_1
  statsmodels        pkgs/main/win-64::statsmodels-0.13.2-py39h2bbff1b_0

The following packages will be REMOVED:

  dpcpp-cpp-rt-2022.1.0-intel_3787
  dpcpp_cpp_rt-2022.1.0-intel_3787
  fortran_rt-2022.1.0-intel_3787
  intel-fortran-rt-2022.1.0-intel_3787
  mkl_umath-0.1.1-py39h7fc9350_26

The following packages will be UPDATED:

  numpy                 intel::numpy-1.21.4-py39ha3f613e_16 --> pkgs/main::numpy-1.22.3-py39h7a0a035_0
  numpy-base         intel::numpy-base-1.21.4-py39h8b6d59b~ --> pkgs/main::numpy-base-1.22.3-py39hca35cd5_0

The following packages will be SUPERSEDED by a higher-priority channel:

  ca-certificates    conda-forge::ca-certificates-2022.5.1~ --> intel::ca-certificates-2022.3.18-haa95532_0
  mkl-service        intel::mkl-service-2.4.0-py39hc0b2338~ --> pkgs/main::mkl-service-2.4.0-py39h2bbff1b_0
  mkl_fft              intel::mkl_fft-1.3.1-py39h9419558_16 --> pkgs/main::mkl_fft-1.3.1-py39h277e83a_0
  mkl_random         intel::mkl_random-1.2.2-py39h206d7d7_~ --> pkgs/main::mkl_random-1.2.2-py39hf11a4ad_0
  numexpr               intel::numexpr-2.8.1-py39hc7a5b94_1 --> pkgs/main::numexpr-2.8.1-py39hb80d3ca_0
  scipy                   intel::scipy-1.7.3-py39h4f83d3f_1 --> pkgs/main::scipy-1.7.3-py39h0a974cb_0

The following packages will be DOWNGRADED:

  icc_rt                                2022.1.0-intel_3787 --> 2020.3-intel_311
  intel-openmp                          2022.1.0-intel_3787 --> 2021.4.0-intel_3556
  intelpython3_core                         2022.1.0-py39_0 --> 0.0.0.custom-0
  mkl                                    2022.1.0-intel_192 --> 2021.4.0-intel_640



DryRunExit: Dry run. Exiting.
</details>

For reference, I have attached the output of "conda list --explicit"
<details>
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: win-64
@EXPLICIT
https://conda.anaconda.org/intel/win-64/ca-certificates-2022.3.18-haa95532_0.tar.bz2
https://conda.anaconda.org/intel/win-64/impi_rt-2021.6.0-intel_546.tar.bz2
https://conda.anaconda.org/intel/win-64/intel-cmplr-lib-rt-2022.1.0-intel_3787.tar.bz2
https://conda.anaconda.org/intel/win-64/intel-cmplr-lic-rt-2022.1.0-intel_3787.tar.bz2
https://conda.anaconda.org/intel/win-64/intel-openmp-2022.1.0-intel_3787.tar.bz2
https://conda.anaconda.org/intel/win-64/intelpython-2022.1.0-0.tar.bz2
https://conda.anaconda.org/intel/win-64/vs2015_runtime-14.27.29016-h5e58377_2.tar.bz2
https://conda.anaconda.org/intel/win-64/intel-fortran-rt-2022.1.0-intel_3787.tar.bz2
https://conda.anaconda.org/intel/win-64/vc-14.2-h21ff451_1.tar.bz2
https://conda.anaconda.org/intel/win-64/bzip2-1.0.8-vc14h53ad9d4_9.tar.bz2
https://conda.anaconda.org/intel/win-64/fortran_rt-2022.1.0-intel_3787.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/jpeg-9e-h8ffe710_1.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/libbrotlicommon-1.0.9-h8ffe710_7.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/libwebp-base-1.2.2-h8ffe710_1.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/lz4-c-1.9.3-h8ffe710_1.tar.bz2
https://conda.anaconda.org/intel/win-64/openssl-1.1.1n-h2bbff1b_0.tar.bz2
https://conda.anaconda.org/intel/win-64/sqlite-3.38.0-h2bbff1b_0.tar.bz2
https://conda.anaconda.org/intel/win-64/tbb-2021.6.0-vc14_intel_834.tar.bz2
https://conda.anaconda.org/intel/win-64/tcl-8.6.10-vc14_2.tar.bz2
https://conda.anaconda.org/intel/win-64/tk-8.6.10-he774522_0.tar.bz2
https://conda.anaconda.org/intel/win-64/xz-5.2.5-h62dcd97_0.tar.bz2
https://conda.anaconda.org/intel/win-64/yaml-0.2.5-he774522_0.tar.bz2
https://conda.anaconda.org/intel/win-64/zlib-1.2.11.1-vc14hc1c5f5b_5.tar.bz2
https://conda.anaconda.org/intel/win-64/dal-2021.6.0-intel_874.tar.bz2
https://conda.anaconda.org/intel/win-64/intel-opencl-rt-2022.1.0-intel_3787.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/libbrotlidec-1.0.9-h8ffe710_7.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/libbrotlienc-1.0.9-h8ffe710_7.tar.bz2
https://conda.anaconda.org/intel/win-64/libpng-1.6.37-vc14h53ad9d4_8.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/libwebp-1.2.2-h57928b3_0.tar.bz2
https://conda.anaconda.org/intel/win-64/mkl-2022.1.0-intel_192.tar.bz2
https://conda.anaconda.org/intel/win-64/python-3.9.10-h35db34a_3.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/zstd-1.4.9-h6255e5f_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/brotli-bin-1.0.9-h8ffe710_7.tar.bz2
https://conda.anaconda.org/intel/win-64/certifi-2021.10.8-py39haa95532_2.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/cycler-0.11.0-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/freetype-2.10.4-h546665d_1.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/libtiff-4.0.10-h21b02b4_1005.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/munkres-1.1.4-pyh9f0ad1d_0.tar.bz2
https://conda.anaconda.org/intel/noarch/pyparsing-3.0.4-pyhd3eb1b0_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/python_abi-3.9-2_cp39.tar.bz2
https://conda.anaconda.org/intel/noarch/pytz-2021.3-pyhd3eb1b0_0.tar.bz2
https://conda.anaconda.org/intel/win-64/pyyaml-6.0-py39h2bbff1b_1.tar.bz2
https://conda.anaconda.org/intel/noarch/six-1.16.0-pyhd3eb1b0_1.tar.bz2
https://conda.anaconda.org/intel/win-64/tbb4py-2021.6.0-py39_intel_834.tar.bz2
https://conda.anaconda.org/intel/noarch/threadpoolctl-2.2.0-pyh0d69192_0.tar.bz2
https://conda.anaconda.org/intel/noarch/wheel-0.37.1-pyhd3eb1b0_0.tar.bz2
https://conda.anaconda.org/intel/win-64/wincertstore-0.2-py39haa95532_2.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/brotli-1.0.9-h8ffe710_7.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/kiwisolver-1.4.2-py39h2e07f2f_1.tar.bz2
https://conda.anaconda.org/intel/win-64/mkl-service-2.4.0-py39hc0b2338_11.tar.bz2
https://conda.anaconda.org/intel/noarch/packaging-21.3-pyhd3eb1b0_0.tar.bz2
https://repo.anaconda.com/pkgs/main/win-64/pillow-9.0.1-py39hdc2b20a_0.conda
https://conda.anaconda.org/conda-forge/noarch/pyaml-21.10.1-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/intel/win-64/python-dateutil-2.8.2-py39_1.tar.bz2
https://conda.anaconda.org/intel/win-64/setuptools-58.0.4-py39haa95532_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/unicodedata2-14.0.0-py39hb82d6ee_1.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/fonttools-4.33.3-py39hb82d6ee_0.tar.bz2
https://conda.anaconda.org/intel/noarch/joblib-1.0.1-pyhd3eb1b0_0.tar.bz2
https://conda.anaconda.org/intel/win-64/pip-21.2.4-py39haa95532_0.tar.bz2
https://conda.anaconda.org/conda-forge/win-64/matplotlib-base-3.5.2-py39h581301d_0.tar.bz2
https://conda.anaconda.org/intel/win-64/daal4py-2021.6.0-py39_intel_874.tar.bz2
https://conda.anaconda.org/intel/win-64/dpcpp-cpp-rt-2022.1.0-intel_3787.tar.bz2
https://conda.anaconda.org/intel/win-64/dpcpp_cpp_rt-2022.1.0-intel_3787.tar.bz2
https://conda.anaconda.org/intel/win-64/icc_rt-2022.1.0-intel_3787.tar.bz2
https://conda.anaconda.org/intel/win-64/numpy-base-1.21.4-py39h8b6d59b_16.tar.bz2
https://conda.anaconda.org/intel/win-64/mkl_fft-1.3.1-py39h9419558_16.tar.bz2
https://conda.anaconda.org/intel/win-64/mkl_random-1.2.2-py39h206d7d7_16.tar.bz2
https://conda.anaconda.org/intel/win-64/mkl_umath-0.1.1-py39h7fc9350_26.tar.bz2
https://conda.anaconda.org/intel/win-64/numpy-1.21.4-py39ha3f613e_16.tar.bz2
https://conda.anaconda.org/intel/win-64/numexpr-2.8.1-py39hc7a5b94_1.tar.bz2
https://conda.anaconda.org/intel/win-64/scipy-1.7.3-py39h4f83d3f_1.tar.bz2
https://conda.anaconda.org/intel/win-64/intelpython3_core-2022.1.0-py39_0.tar.bz2
https://conda.anaconda.org/intel/win-64/pandas-1.3.5-py39h6e2a61c_1.tar.bz2
https://conda.anaconda.org/intel/win-64/scikit-learn-0.24.2-py39hf11a4ad_2.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/scikit-optimize-0.9.0-pyhd8ed1ab_1.tar.bz2
</details>
user: dklein0
issue_8295: state: open
title: It is wrong in the implementation of "Standard Error of the mean" in   statsmodels.stats.descriptivestats
body: #### Describe the bug

in statsmodels.stats.descriptivestats
“std_err” - Standard Error of the mean assuming no correlation
may be wrong ..

#### Code Sample, a copy-pastable example if possible


```python
import numpy as np
import statsmodels.stats.descriptivestats as descriptivestats

np.random.seed(0)
X=   np.random.normal(size=100)
des= descriptivestats.Description(X)

std_err_calculated_in_descriptivestats = des.frame.loc['std_err']

std_err_from_wiki = X.std(ddof=1) / (X.size**.5)
# ref: https://en.wikipedia.org/wiki/Standard_error#Exact_Value

print(f'std_err_calculated_in_descriptivestats= {std_err_calculated_in_descriptivestats}')
print(f'std_err_from_wiki= {std_err_from_wiki}')
```
```python
# the output is as follows
# std_err_calculated_in_descriptivestats= 0    0.01013
# Name: std_err, dtype: float64
# std_err_from_wiki= 0.10129597692685018
```
<details>
# my debug ...


# the source code:
- https://www.statsmodels.org/stable/_modules/statsmodels/stats/descriptivestats.html

```python
std_err.loc[count > 0] /= count.loc[count > 0]
```

# my debug..
``` python
std_err.loc[count > 0] /= count.loc[count > 0]**.5
```


</details>




user: renyuanL
issue_8293: state: open
title: ENH: add inference for variance without normality assumptions
body: first version, mainly bonett for variance confidence interval using kurtosis
written while reading references in 8261

issues:
#8261 inference for variance
#8286 kurtosis and jarque-bera
#8289 inference for correlation (just references without details)

current plan

I just want to get the basic versions in mainly bonett, plain kurtosi-adjusted and traditional inference under normality assumptions.
There are too many options for additional variations of methods, e.g. for kurtosis small sample and bias corrections, and there is no obvious winner for those.

cases:

- one-sample
- variances_2indep 
  - ratio: ftest and exp (as in bonnet)
  - diff not a large literature, confint based on MOVER, but doesn't easily translate to hypothesis test (would need inversion of confint)
  - tost: not implemented yet, a few references are available
- power: just the basics, mainly based on NCSS/PASS (but that only includes methods based on normality assumption of data)

I would like to get some "score" versions (variance of test statistic base on null assumption), but there isn't much literature on it, confint requires derivation and needs to be verified by Monte Carlo.
Current versions and almost all the literature uses estimated variance in variance of test statistic, i.e. is wald-type.

However, exp transformation of variance ratio removes the variance from the variance of the test statistic. variance only enters through the kurtosis estimate.







user: josef-pkt
issue_8291: state: open
title: ENH: tools reusable local (kernel) density estimation function - variance of quantiles
body: It would be useful to have standalone functions for local kde or similar density estimates as used in QuantReg.

my current case:
quantile based skew and kurtosis estimates have variance that depends on the unknow local pdf value similar to how cov_params is computed in QuantReg.
no reference right now, I saw it in something by Gastwirth (IIRC) in context of #8286 and #8261
(however for those cases nonparametric standard errors seem to be very large and not very useful for getting good confidence intervals of quantile based kurtosis)

currently we don't have standalone functions that could be easily reused for other cases  in stats or robust. 
(usecase in robust would be for mquantile link function as approximation to L1-type "check" function)

Also there might be other approaches that don't user kernels for estimating the density at just one point.
e.g. higher order polynomials extension to gaussian or similar, or maybe the "differencing" approximation could be used.







user: josef-pkt
issue_8290: state: open
title: use_boxcox: Inverse transform after generating samples?
body: I am working with (S)ARIMA(X) and Exponential Smoothing. I want to improve the performance of my models with transforming the data before passing it to the model, either with a log tranform oder power transform. For that I use the use_boxcox parameter. 
After generating forecasts with the model, I compare the forecasts with the real test dataset, that is therefore not transformed.

#### Describe the solution you'd like
 I was now wondering, if I should retransform the forecasts out of the model, or if the statsmodel itself can retransform the predictions or forecasts to better compare them to the real test data.

#### Describe alternatives you have considered
Alternative I would retransform the data myself, either using the parameters of the boxcox transform, or writing the transformers myself.
user: jeiglsperger
issue_8289: state: open
title: SUMM/ENH: one correlation coefficient, test, confint  robust to nonnormality
body: related to #8261  inference for variance robust to nonormality using skew and kurtosis
related #6696 for multivariate case which already includes options for using 4th moments

a similar restriction of standard tests applies to inference for correlation coefficient

AFAIU from skimming parts of motivation for confint for pearson correlation coefficient:
Under null of no-correlation the standard test statistic is correct, but does not hold under nonzero correlation, i.e. alternatives or confidence intervals away from zero need different distribution/standard errors.


currently just parking some references
I haven't looked at those and selection mainly based on abstracts

Bishara, Anthony J., and James B. Hittner. 2012. “Testing the Significance of a Correlation with Nonnormal Data: Comparison of Pearson, Spearman, Transformation, and Resampling Approaches.” Psychological Methods 17 (3): 399–417. https://doi.org/10.1037/a0028087.

———. 2015. “Reducing Bias and Error in the Correlation Coefficient Due to Nonnormality.” Educational and Psychological Measurement 75 (5): 785–804. https://doi.org/10.1177/0013164414557639.

Bishara, Anthony J., Jiexiang Li, and Thomas Nash. 2018. “Asymptotic Confidence Intervals for the Pearson Correlation via Skewness and Kurtosis.” British Journal of Mathematical and Statistical Psychology 71 (1): 167–85. https://doi.org/10.1111/bmsp.12113.

Bonett, Douglas G., and Thomas A. Wright. 2000. “Sample Size Requirements for Estimating Pearson, Kendall and Spearman Correlations.” Psychometrika 65 (1): 23–28. https://doi.org/10.1007/BF02294183.

DUNCAN, G. T., and M. W. J. LAYARD. 1973. “A Monte-Carlo Study of Asymptotically Robust Tests for Correlation Coefficients.” Biometrika 60 (3): 551–58. https://doi.org/10.1093/biomet/60.3.551.

Hawkins, D. L. 1989. “Using U Statistics to Derive the Asymptotic Distribution of Fisher’s Z Statistic.” The American Statistician 43 (4): 235–37. https://doi.org/10.2307/2685369.

Hu, Xinjie, Aekyung Jung, and Gengsheng Qin. 2020. “Interval Estimation for the Correlation Coefficient.” The American Statistician 74 (1): 29–36. https://doi.org/10.1080/00031305.2018.1437077.

Puth, Marie-Therese, Markus Neuhäuser, and Graeme D. Ruxton. 2014. “Effective Use of Pearson’s Product–Moment Correlation Coefficient.” Animal Behaviour 93 (July): 183–89. https://doi.org/10.1016/j.anbehav.2014.05.003.

———. 2015. “Effective Use of Spearman’s and Kendall’s Correlation Coefficients for Association between Two Measured Traits.” Animal Behaviour 102 (April): 77–84. https://doi.org/10.1016/j.anbehav.2015.01.010.

Ruscio, John. 2008. “Constructing Confidence Intervals for Spearman’s Rank Correlation with Ordinal Data: A Simulation Study Comparing Analytic and Bootstrap Methods.” Journal of Modern Applied Statistical Methods 7 (2). https://doi.org/10.22237/jmasm/1225512360.




user: josef-pkt
issue_8288: state: open
title: ENH: score- LM-test under misspecified alternative
body: related to #2041
I need to look at the "misspecified alternative" case again.
I'm not sure the current score test works this way for the "vectorized" version (not joint test of variable addition)

main old reference is Bera and Yoon

recent article, there are a few more but I have not looked at them

extension to nonlinear hypothesis tests in GMM

Bera, Anil, Gabriel Montes-Rojas, Walter Sosa-Escudero, and Javier Alejo. 2021. “Tests for Nonlinear Restrictions under Misspecified Alternatives with an Application to Testing Rational Expectation Hypotheses.” The Econometrics Journal 24 (1): 41–57. https://doi.org/10.1093/ectj/utaa010.


my current context #8286
Jarque-Bera joint normality (skew plus kurtosis) test versus separate skew and kurtosis tests

Bera, Anil K., and Gamini Premaratne. 2001. “Adjusting the Tests for Skewness and Kurtosis for Distributional Misspecifications.” SSRN Scholarly Paper 304465. Rochester, NY: Social Science Research Network. https://doi.org/10.2139/ssrn.304465.

It's only a working paper. I didn't see it referenced in the kurtosis literature, a only partially looked at literature for skewness/symmetry tests.
**update** I found the published version
Premaratne, Gamini, and Anil K. Bera. 2017. “Adjusting the Tests for Skewness and Kurtosis for Distributional Misspecifications.” Communications in Statistics - Simulation and Computation 46 (5): 3599–3613. https://doi.org/10.1080/03610918.2014.988254.
and related, an older article for skewness using higher moments for nonnormality
Godfrey, L. G., and C. D. Orme. 1991. “Testing for Skewness of Regression Disturbances.” Economics Letters 37 (1): 31–34. https://doi.org/10.1016/0165-1765(91)90238-G.

There a many more traditional examples for Bera/Yoon robust testing, e.g. i.i.d. versus correlation and/or heteroscedasticity


The point here: 
Check and implement the generic version in score_test and the cm-tests (conditional moment tests).

One implementation problem is that the references partial out the extra effects, while I work mostly with the joint matrix versions.
i.e. the question is which matrices are inverted to get the cov_params of the moment conditions.
I'm not sure the current version of the score_test does this because it looked like it tests the test separately for each variable addition.
(Maybe variable addition in OLS doesn't have the misspecified alternative problem. ???)

also related:
In (almost sure independence) screening, we add several variables at once to capture multicolinearities.

**update**

partially reading Godfrey-Orme 2001 again
AFAIU based on the quick read, we need a one-step estimate to capture a possible parameter change from other restrictions that might also not hold (misspecified alternative) when evaluating the test for the parameter under consideration.
(I didn't look at Godfrey Orme 1996 again, both first added to Zotero in 2013)

It seems to be essentially the same as C(alpha) tests #2052, 2052 has link to #2050 (one-step estimators)

Godfrey, Leslie G., and Chris D. Orme. 1996. “On the Behavior of Conditional Moment Tests in the Presence of Unconsidered Local Alternatives.” International Economic Review 37 (2): 263–81. https://doi.org/10.2307/2527323.

Godfrey, L.G., and C.D. Orme. 2001. “On Improving the Robustness and Reliability of Rao’s Score Test.” Journal of Statistical Planning and Inference 97 (1): 153–76. https://doi.org/10.1016/S0378-3758(00)00351-7.


Godfrey-Orme 2001 p.159
"The use of <theta_restricted> as the first-stage estimator to derive robust tests is acceptable when, as
in this paper, only local departures from the null are considered in asymptotic analyses."

They refer to other articles for case when the other parameter restriction might deviate by a large, O(1), amount. Then sqrt(n) local alternative is not appropriate. (This seems to be allowed in C(alpha) tests.)

Also, they include the m-estimator sandwich form of the robust score test e.g. equ. (3.11)
user: josef-pkt
issue_8286: state: open
title: ENH: robust jarque-bera normality (skew and kurtosis) tests, including multivariate
body: related
#8274
#8269
#8261

I don't actually want normality tests, I want information about skew and kurtosis. (e.g. inference for variance depends on 4th moment but not on other features of a distribution, joint mean-variance inference depends on 3rd and 4th moments).
However the test statistics are still derived under normality assumptions (relevant for higher than 4th moments)
The likely target is `variance_moments` module from #8261 for univariate case.
The multivariate versions (like Doornik-Hansen #382) can go into some some multivariate diagnostic.

some references (just what I saw right now)

Stehlík et al. 2012 has mean, trimmed mean and median as location parameter, has a large class of robustified jarque-bera type tests.
includes proof that they all converge to the corresponding k-th moment, with asymptotic chi2 for JB type statistics.
So far I have kurtosis based on mean and trimmed mean in my draft version for #8261 

Bonett for variance of variance only changes the center in the computation of the 4th moment. This is not outlier robustness.
Outlier robust estimates and tests as in Stehlik also trim the moment computation itself.

Kim, Namhyun. 2015. “Tests Based on Skewness and Kurtosis for Multivariate Normality.” Communications for Statistical Applications and Methods 22 (4): 361–75. https://doi.org/10.5351/CSAM.2015.22.4.361.

———. 2016. “A Robustified Jarque–Bera Test for Multivariate Normality.” Economics Letters 140 (March): 48–52. https://doi.org/10.1016/j.econlet.2016.01.007.

Stehlík, Milan, Zdeněk Fabián, and Luboš Střelec. 2012. “Small Sample Robust Testing for Normality against Pareto Tails.” Communications in Statistics - Simulation and Computation 41 (7): 1167–94. https://doi.org/10.1080/03610918.2012.625849.



user: josef-pkt
issue_8284: state: open
title: QUESTION: Should the design for aic/bic reported by OLS be changed
body: AIC is a relative measure used for model selection - and for feature selection in particular. It's confusing that AIC can be reported by a single model - it is useless and misleading for feature selection task.
Right approach for AIC feature selection algorithm implies to compute variance on all features included and then (for each subset of features) to compute model log likelihood using that variance (computed on all features included). If user does not know that, then he can solve feature selection task by computing AIC on individual models, which is absolutely wrong for feature selection (but still looks intuitively correct from API point of view).

#### Example of code that reproduces wrong solution for feature selection task:

```python
import pandas as pd
import statsmodels.api as sm
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from itertools import combinations

X, y = make_classification(n_samples=2000, n_features=10, n_informative=5, n_redundant=5,
                           flip_y=0.05, random_state=0)

X = pd.DataFrame(X, columns=list(range(10)))
y = pd.Series(y)

train_X, test_X, train_y, test_y = train_test_split(X, y, train_size=1500, random_state=1)

best_feature_set = []
best_aic = 0

for i in range(train_X.shape[1]):
    for subset in combinations(train_X.columns.tolist(), i+1):

        model = sm.OLS(train_y, train_X[list(subset)])
        
        this_aic = model.fit().aic
        
        if this_aic > best_aic:
            best_feature_set = subset
            best_aic = this_aic
            
print("best feature set is:", best_feature_set, " with AIC ", best_aic)
```
user: wrapper228
issue_8283: state: open
title: FAQ: stats inference with method="auto"
body: (I don't find a previous issue, but I mentioned versions of this several times.)

Using a specific default method in hypothesis tests and confidence intervals commits us to this method and we cannot easily change it for backwards compatibility.
In several cases the current default is a "wald" type method which is one of the worst choices in cases like proportions.

choosing the default method has several limitations:

- We might not have implemented the "best" or better methods
- The literature is not clear about what the "best" methods is, or it depends on the settings
- The person writing the code, like I, has not enough overview about what the bets methods are

We have two possibilities to avoid choosing and committing to a default method

- don't provide default, `method=None` where user is required to choose a method (I do this now during development in curent PRs)
- choose default method but don't commit to it, `method="auto" means that we use as default the method that currently looks the best available choice, but we change it if we have and know a different method that is better.

my current example

variance confint: 
method="bonett" looks good but the kurtosis estimate is strongly downward biased for distributions with large kurtosis, and consequently the coverage of the confidence interval is too small if the sample size is not very large. We could improve if we use the kurtosis estimate to choose the method.

Warning: `method="auto" is not reproducible across versions, Choose a specific method to avoid changes in default method  in future versions.




user: josef-pkt
issue_8280: state: open
title: Reference found for runs test length distribution
body: #### Describe the bug

https://github.com/statsmodels/statsmodels/blob/main/statsmodels/sandbox/stats/runs.py includes the following comment:

"This should be extended once I figure out what the distribution of runs of any length k is."

Here's a reference that describes exactly this, with expected value, variance and distribution.

https://www.eg.bucknell.edu/~xmeng/Course/CS6337/Note/master/node44.html

Also describes the distribution of up/down runs, so you could in theory combine all three types of runs tests to give an overall probability that the data is random.

user: shusheer
issue_8277: state: open
title: Stop statsmodels.tsa.x13.x13_arima_analysis from creating temporary files
body: Im using package statsmodels in python. To be precise I am using the function statsmodels.tsa.x13.x13_arima_analysis(). When I use it, it creates a lot if temporary files in folder C:\Users\username\AppData\Local\Temp. I dont want it to create these files, how can i prevent this? Or, if it would be possible to choose where these out files appear that would be great.

I know that the function has a log parameter and i have tried setting this equal to false without any success. 

Thanks in advance,
Felix Kuritzén


user: kuritzen
issue_8276: state: open
title: ENH: add UMPT equivalence test to for some cases
body: I have ignored so far (uniformly most powerful) UMP tests for equivalence and only implemented TOST.

I finally think that we should add them for some easier cases.
Add new function `equivalence_xxx` that selects between UMPT and TOST where TOST is default and delegates to the `tost_xxx` functions.

related, we might also want UMPT version of nonequivvalence, minimum effect tests.

Wellek's book has large parts for UMP equivalence tests but I did not look at those.

current motivation #8261
umpt and tost for comparing variances
Jan and Shieh 2022 use it based of F-test which is nonrobust to nonnormality of the data.
But we should be able to get it for the robust versions like Benett.


Jan, Show-Li, and Gwowen Shieh. 2022. “A Comparative Study of TOST and UMPT Procedures for Evaluating Dispersion Equivalence.” Statistics in Biopharmaceutical Research 14 (2): 162–67. https://doi.org/10.1080/19466315.2020.1821762.

AFAIU UMPT are much less popular in applied work than TOST, and can have some nonintuitive non-monotonicities

user: josef-pkt
issue_8274: state: open
title: ENH: inference for skew and kurtosis without normality assumption
body: I finally found inference for skew and kurtosis for nonnormal distributions

Bao 2013 looks good
Pewsey 2005 seems to be the first to derive the distribution for kurtosis
Mahmoudi et al don't cite Pewsey nor Bao

Bao, Yong. 2013. “On Sample Skewness and Kurtosis.” Econometric Reviews 32 (4): 415–48. https://doi.org/10.1080/07474938.2012.690665.

Mahmoudi, Mohammad Reza, Bui Anh Tuan, and Kim-Hung Pho. 2021. “On Kurtoses of Two Symmetric or Asymmetric Populations.” Journal of Computational and Applied Mathematics 391 (August): 113370. https://doi.org/10.1016/j.cam.2020.113370.

Pewsey, Arthur. 2005. “The Large-Sample Distribution of the Most Fundamental of Statistical Summaries.” Journal of Statistical Planning and Inference 134 (2): 434–44. https://doi.org/10.1016/j.jspi.2004.04.014.


variance for kurtosis needs 8th moment
So, it will only be useful in large samples.
But, if we get a rough confidence interval for kurtosis, then we can guess on the reliability of inference for variance #8261 

user: josef-pkt
issue_8270: state: open
title: ENH: Export `Summary` to MS Word
body: #### Is your feature request related to a problem? Please describe
Not sure if there is an indirect/direct way to solve my current issue. I have downloaded the OLS summary as txt to a txt file,
and it looks like what I need with right format for my thesis. As I need it in MS Word, I tried to transfer the tables into MS word, without luck however. The columns aren't formatted correctly anymore.

#### Describe the solution you'd like
A way to contain the formatting using in the txt version of summary, but than for MS word.

#### Describe alternatives you have considered
The best working alternative for now is to write the csv version of the summary to a txt.file. After this I copy the text in word and convert it to a table. This generates a regular table without the formatting that I get when I download the summary as txt. 

user: KonradProgrammer
issue_8269: state: open
title: ENH: Inference for quantile measures of kurtosis, peakedness, and tail weight
body: (mainly to park a reference for now)

We have quantile based measures for skew and kurtosis (relative to normal distribution)

Here is an article that includes confidence intervals and references to other inference.
(I only did very partial skimming)

Robert G. Staudte (2017) Inference for quantile measures of kurtosis,
peakedness, and tail weight, Communications in Statistics - Theory and Methods, 46:7, 3148-3163,
https://doi.org/10.1080/03610926.2015.1056366

related: including kurtosis to make variance tests more robust to distributional assumptions

#8261
#7235
#6566

one good feature to have would be a 3-valued classification by different measures
e.g. negative, around zero and large positive excess kurtosis,  (I never remember the latin names like leptokurtic, ....)
and same for skew

For benett test in #8261, minitab uses tail heaviness/kurtosis classification to warn about minimum sample size requirements

for extreme value theory we have tail order as a more specific criterion, e.g. #7472

user: josef-pkt
issue_8261: state: open
title: SUMM/ENH: inference for variance, variance estimation
body: I don't know where to put hypothesis tests and confidence intervals for variance and standard deviation.
It should have the usual test_, confint_, tost_, power functions for one and two sample cases as in rates and proportions.

**update**: preliminary decision on module name `variance_moments`  I didn't really like any of the shorter names

oneway case is handled in `oneway` with standard tests. This is mainly equality of scale and dispersion measure, not necessarily of variance.

I want something specific to variance (and not other dispersion measures), that can be for example used in functions like zscore standardized mean or coefficient of variation. one usage MOVER confint for those.

related issue: where do we put skew and kurtosis functions. (e.g. extensions to skew and kurtosis tests, we need one-sided alternatives)
can we have a module for variance and higher moments.

related:
#2765 organizing cov and corr, which is now in stats.multivariate
(outlier) robust scale is in `robust` and some open PRs, note, those can only be converted to variance measure under specific distributionall assumptions) related robust definitions of skew and kurtosis #6790 and in stattools
weightstats has descriptive statistics in the class, but only inference on one and two sample means.

inferential statistics for a single correlation coefficient
I'm not sure where that goes

specific to variance

I would like the kurtosis corrected version of Bonett, and similar variation on it.
supporting code will need kurtosis estimators  
(aside: standard kurtosis estimate is downward (!) biased for heavy tailed distributions)

(I didn't see any reference that uses score confidence intervals or score test for variance. Score test for one variance should be easy. )

- #667
- #7235
  Note, Bonnet is still liberal for heavy-tailed skewed distributions.
- #4144 
- ...


Stata also has bonnet confint, minitab has a good working paper for bonnet, and a kurtosis test


other ideas:
Can we use transformation to normality #3224 of underlying data to get a better variance inference?, e.g. for data that are closer to log-nromal (I have not looked for references for this)

Can we use a "working 4th moment or kurtosis function" with a 4th moment robust variance estimate to improve inference?. e.g. based on GLM Gamma (corresponds to quadratic 4th moment if endog is variance/squared residuals

Right now, I mainly want Bonnet test which looks easy to implement but needs a location.


related multivariate version, covariance, correlation matrix PR #6696
e.g. 
https://github.com/statsmodels/statsmodels/pull/6696/files#diff-fc0daeb7f971edb0e8a8866ed966f86ced2c81767eb394f3a81a6a4761184595R180
has option for given  kurt(osist) and general, the latter uses 4th moment estimate for cov_cov
I doubt those functions work for the univariate case, i.e. cov is a single variance.

user: josef-pkt
issue_8254: state: open
title: Bad understanding in statsmodels.stats.weightstats.DescrStatsW
body: #### Describe the bug

Trying to understand the calculations in DescrStatsW.tconfint_mean() I got confused with some method calls.

By the comments, why the number of observations should be equal to the to sum of weights?.

![image](https://user-images.githubusercontent.com/68517641/165620656-d9568e22-85e4-4671-ab16-cb11797227a1.png)

Also, in the method tconfint_mean, the dof is equal to (sum_weights - 1), but it should be equal to the number of observations -1.

![image](https://user-images.githubusercontent.com/68517641/165621613-37a771c4-436b-4412-8da9-dfcbc65df29e.png)

Also, in the std_mean(), the divisions use sum_weights, but it should use the number of observations.

![image](https://user-images.githubusercontent.com/68517641/165623829-8f81b4e5-043a-48fa-95f7-77f6558e124d.png)




user: alandema
issue_8252: state: open
title: FAQ/ENH/SUMM: modification if zeros are observed - poisson, binomial
body: What to do if an observed count is zero?

related
#7011 zero correction in contingency tables
meta-analysis, AFAIR, I added several options

In those cases above, a fixed number (e.g. 0.5 count) is added to avoid zeros, either conditional or unconditional.

For binomial proportion and poisson rates in tests or confidence intervals we have the option to switch to a different `method`.

- McCracken and Looney compare several modifications and argue in favor of exact interval if count is zero
- Barker also uses exact confint for zero counts in poisson rate
- Brown, et al for confint of one proportion have a section with different methods. They also switch for small counts 1 to 3 to remove undercoverage spikes for some parameter combinations if prop is close to boundary

McCracken, CE, and SW Looney. 2017. “On Finding the Upper Confidence Limit for a Binomial Proportion When Zero Successes Are Observed.” J Biom Biostat 8 (338): 2.

**edit** adding two more references that I have not read, There are a few more related especially related to Bayesian priors

Tuyl, Frank, Richard Gerlach, and Kerrie Mengersen. 2009. “The Rule of Three, Its Variants and Extensions.” International Statistical Review 77 (2): 266–75. https://doi.org/10.1111/j.1751-5823.2009.00078.x.

Winkler, Robert L, James E Smith, and Dennis G Fryback. 2002. “The Role of Informative Priors in Zero-Numerator Problems.” The American Statistician 56 (1): 1–4. https://doi.org/10.1198/000313002753631295.



I have not yet added modification at zero, and have not decided yet which options to add and what default should be.
e.g. current Wald confidence interval collapses to 0 singular point.
https://github.com/statsmodels/statsmodels/issues/6721#issuecomment-1110114402 for proportion
and current PR #8166 for Poisson rates

Currently we add in some cases an epsilon in the computation or clip values inside boundaries, but that is mainly for numerical reasons, e.g. avoid division by zero.

needs decision



user: josef-pkt
issue_8251: state: open
title: ENH: power_proportions_2indep with value different of zero and different `compare`
body: power_proportions_2indep does not support `value` different from equality.
Also, not `compare` option, which will become only relevant if non-zero null is supported.

see comments at around here https://github.com/statsmodels/statsmodels/issues/8049#issuecomment-1025209120

I partially read Farrington and Manning 1990 again.
The main task is to add constrained parameter estimates as in score test. Using this as default extends the pooled estimate under the null and is backwards compatible.

Farrington and Manning also have the option of using the wald variance, which I guess is based on the proportions under the alternative.

It looks like this follows the same pattern as we have for poisson and negbin in PR #8166 
F/M also have an estimate under fixed margins, but they mention that the samplesize or power estimates are not very good.

Aside: proportion has very few references
I cannot find anymore what I used to implement the power function.
The generic part is general in many references, but I don't know what the specific Binomial reference is (using the pooled estimate instead of (wald) props under alternative).


I guess the power for the equality test is also old and common.
e.g. Lachine 1981 Introduction to sample size determination and power analysis for clinical trials
eq. (12) but he uses fraction of total nobs instead of nobs_ratio, and sample size formula is for total nobs instead of nobs1.
(I had read it in the past, so it was one of the references, but not directly the one I implemented)

related
#6721 refactoring, reorganizing proportions

user: josef-pkt
issue_8250: state: open
title: statsmodels-0.13.2: tests fail on gentoo-x86 (i686)
body: #### Describe the bug

When running tests for statsmodels-0.13.2 in a gentoo-x86 system, they end up failing. On the same host with gentoo-x86_64 arch they pass.

#### Code Sample, a copy-pastable example if possible

The tests are ran by a package manager with:
```bash
python_test() {
	local -x MKL_NUM_THREADS=1
	local -x OMP_NUM_THREADS=1
	local jobs=$(makeopts_jobs "${MAKEOPTS}" "$(get_nproc)")

	pushd "${BUILD_DIR}" >/dev/null || die
	"${EPYTHON}" -c "
import statsmodels
statsmodels.test(extra_args=['-vv', '-n', '${jobs}'], exit=True)" \
		|| die "tests fail with ${EPYTHON}"
	rm -rf statsmodels/.pytest_cache || die
	popd >/dev/null || die
}
```
And as said, this works on x86_64. The full build log, including test phase and test failures, is available here:
https://bugs.gentoo.org/attachment.cgi?id=774500 (compressed as .xz)


I tried searching for duplicates before filing this.

> If you are using a released version, have you verified that the bug exists in the main branch of this repository? It helps the limited resources if we know problems exist in the current main branch so that they do not need to check whether the code sample produces a bug in the next release.

Going back few previous versions, they also fail tests on i686. So this doesn't seem like something that happened between 0.13.1 -> 0.13.2. (and today when ran, 0.12.2 procudes over 80 test failures, while 0.13 is at 43).


#### Expected Output

Tests to pass, like they do on x86_64.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

```
>>> import statsmodels.api as sm; sm.show_versions()

INSTALLED VERSIONS
------------------
Python: 3.9.11.final.0
OS: Linux 5.15.32-gentoo-dist #1 SMP Sun Apr 3 12:21:28 EEST 2022 i686
byteorder: little
LC_ALL: None
LANG: C.UTF8

statsmodels
===========

Installed: 0.13.2 (/usr/lib/python3.9/site-packages/statsmodels)

Required Dependencies
=====================

cython: 0.29.28 (/usr/lib/python3.9/site-packages/Cython)
numpy: 1.22.2 (/usr/lib/python3.9/site-packages/numpy)
scipy: 1.7.3 (/usr/lib/python3.9/site-packages/scipy)
pandas: 1.4.1 (/usr/lib/python3.9/site-packages/pandas)
    dateutil: 2.8.2 (/usr/lib/python3.9/site-packages/dateutil)
patsy: 0.5.2 (/usr/lib/python3.9/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: Not installed
cvxopt: Not installed
joblib: Not installed

Developer Tools
================

IPython: Not installed
    jinja2: Not installed
sphinx: Not installed
    pygments: Not installed
pytest: 7.0.1 (/usr/lib/python3.9/site-packages/pytest)
virtualenv: Not installed
```

user: juippis
issue_8242: state: open
title: ENH/REF: revisit t-tost, equivalence for normal samples with estimated variance
body: (just a note as reminder and parking a reference, I don't know anything specific)

recent work on equivalence testing was for oneway anova and non-normal distribution.

I have not looked at the gaussian/t case in a long time. My initial implementation of TOST was pretty mechanical, and I was more interested in the meaning of reversing null and alternative, than statistical complications that might show up in equivalence testing.
I also skipped those chapters in Wellek's book.

looks like a good overview article:

Ocaña, Jordi, M. P. Sánchez, A. Sánchez, and J. L. Carrasco. 2008. “On Equivalence and Bioequivalence Testing.” SORT-Statistics and Operations Research Transactions, 151–76.

The main difference to binomial and poisson case is that we have the variance as a nuisance parameter in normal distribution case.






user: josef-pkt
issue_8240: state: open
title: SUMM: statistical analysis for stratified samples,  oneway strata for 2-sample case
body: (I don't have an overview and we don't have much)

What are the questions and target statistics?

currently:
meta-analysis based on k 2-sample comparisons
stratified contingency table MH test

possible cases:
extending tests and confint for 2indep binomial proportions and poisson rates to stratified case

#7184 proportions in 2x2xk case
#6498 mover confint, which can be extended to stratified case.


user: josef-pkt
issue_8237: state: open
title: FAQ/ENH  two-sided poisson etest and similar are not equal-tailed, central
body: I'm looking at poisson e-test again because I cannot match up an example in a reference (Ng et el for etest-score for diff)
(I don't find a bug in my code and don't rule out typo in article, I can match all other tests)

etest uses the following

```
    if alternative in ['two-sided', '2-sided', '2s']:
        mask = np.abs(stat_space) >= (np.abs(stat_sample) - eps)
    elif alternative in ['larger', 'l']:
        mask = stat_space >= (stat_sample - eps)
    elif alternative in ['smaller', 's']:
        mask = stat_space <= (stat_sample + eps)
```

equal-tailed would use instead `pvalue = 2 * min(pvalue_lower, pvalue_upper)`

i.e. we are similar to minlike, but two-sided order is defined by test statistic (currently either wald or score)
"minstat" instead of minlike?


aside:
we might want to add etest for ratio with methods wald-log and score-log

user: josef-pkt
issue_8236: state: open
title: BUG: Change doc build to not require http
body: #### Describe the bug

Doc building requires network access which makes it fragine.  Examples that need to read remote sources should be removed from the core docs and only be allowed to remain in the notebooks.
user: bashtage
issue_8235: state: open
title: How to interpret MixedLMResults.random_effects results ?
body: Hi team,

I am using statsmodels mixedlm for mixed effect models for one of my projects, and I wanted to understand the random effects coefficients/slopes and intercept. When I ran the random slope and random intercept model I got below results - 

  Mixed Linear Model Regression Results

Model:             MixedLM  Dependent Variable:  Y        
No. Observations:  5000     Method:              REML     
No. Groups:        50       Scale:               0.0793   
Min. group size:   100      Log-Likelihood:      -947.9619
Max. group size:   100      Converged:           No       
Mean group size:   100.0                                  

               Coef.  Std.Err.    z    P>|z| [0.025 0.975]

Intercept      -0.399    0.143  -2.785 0.005 -0.679 -0.118
X1              3.748    0.016 233.561 0.000  3.717  3.780
X2             -0.016    0.014  -1.116 0.264 -0.045  0.012
Group Var       1.017    0.839                            
Group x X1 Cov -0.002    0.109                            
X1 Var          0.001    0.061                            
Group x X2 Cov  0.008    0.070                            
X1 x X2 Cov     0.000    0.011                            
X2 Var          0.001    0.011                            

Above results shows the fixed effect estimates with group variable var and cov. However what I needed is the analysis of random coeffs and for that I used mdf.random_effects and got below results - 


<html xmlns:v="urn:schemas-microsoft-com:vml"
xmlns:o="urn:schemas-microsoft-com:office:office"
xmlns:x="urn:schemas-microsoft-com:office:excel"
xmlns="http://www.w3.org/TR/REC-html40">

<head>

<meta name=ProgId content=Excel.Sheet>
<meta name=Generator content="Microsoft Excel 15">
<link id=Main-File rel=Main-File
href="file:///C:/Users/703305~1/AppData/Local/Temp/msohtmlclip1/01/clip.htm">
<link rel=File-List
href="file:///C:/Users/703305~1/AppData/Local/Temp/msohtmlclip1/01/clip_filelist.xml">
<style>
<!--table
	{mso-displayed-decimal-separator:"\.";
	mso-displayed-thousand-separator:"\,";}
@page
	{margin:.75in .7in .75in .7in;
	mso-header-margin:.3in;
	mso-footer-margin:.3in;}
tr
	{mso-height-source:auto;}
col
	{mso-width-source:auto;}
br
	{mso-data-placement:same-cell;}
td
	{padding-top:1px;
	padding-right:1px;
	padding-left:1px;
	mso-ignore:padding;
	color:black;
	font-size:11.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:Calibri, sans-serif;
	mso-font-charset:0;
	mso-number-format:General;
	text-align:general;
	vertical-align:bottom;
	border:none;
	mso-background-source:auto;
	mso-pattern:auto;
	mso-protection:locked visible;
	white-space:nowrap;
	mso-rotate:0;}
.xl63
	{mso-number-format:Scientific;}
.xl64
	{background:red;
	mso-pattern:black none;}
-->
</style>
</head>

<body link="#0563C1" vlink="#954F72">



  | 0 | 1 | 2 | 3 | 4
-- | -- | -- | -- | -- | --
Group | 1.706555 | -0.59806 | -0.48837 | -1.08287 | 0.90788
X1 | -0.00784 | 4.93E-05 | 0.004484 | 0.011812 | -0.01029
X2 | 0.004487 | -0.00221 | -0.00664 | -0.01063 | 0.005566
  |   |   |   |   |  
Total | 3.740157 | 3.748049 | 3.752484 | 3.759812 | 3.737709
  |   |   |   |   |  
X1 | 3.748 |   |   |   |  



</body>

</html>
 
So random effects returns Group level values for Group, and variables X1 and X2. Does that means for X1 my final estimate for group 0 would be sum of fixed and random estimate i.e. 3.748 + -0.00784 = 3.740. from the definition of random_effects it says - "A dictionary mapping the distinct group values to the conditional means of the random effects for the group given the data." 

So what actually random effect returns as a value and how to interpret these ? I am a bit confused , but will appreciate your help and guidance.

Thanks,
Prashant
























































user: prashanthin
issue_8233: state: open
title: ENH: binom test, conditional poisson rates ratio test, exact, exact-c, midp - minlike versus central
body: ~~maybe a bug,~~ needs checking, and adding equal-tailed, central version

**update** NOT a bug, when I wrote this I had added unit tests for cond-exact and cond-midp against R exactci for "minlike"
unit test test_twosample_poisson_r with results in results.rates.py
That is, current version is correct and we can add the central exact and central midp as separate methods.
`<`end update`>`

test_poisson_2indep has the following code to compute conditional midp for ratio of two poisson rates
```
            pvalue = proportion.binom_test(y1, y_total, prop=bp,
                                           alternative=alternative)
            if method in ['cond-midp']:
                # not inplace in case we still want binom pvalue
                pvalue = pvalue - 0.5 * stats.binom.pmf(y1, y_total, bp)
```

When I wrote this I didn't realize or had forgotten that two-sided `binom_test` (based on scipy) is minlike and not central.
The unit tests use Gu examples which are all one-sided only. In one-sided case, both minlike and central are the same.
No unit tests for two-sided tests of exact-cond and cond-midp

Aside: change name exact-cond to cond-exact to match cond-midp?

AFAIK, the midp correction is usually for the `central` version. I'm not sure the correction is correct also for minlike.

- Does minlike midp makes sense and is it correct?
- If yes, should we add both midp-minlike and midp-central?
- Need to add "binomial_test_central" version when using a proportion test, 
- We still need to clean up and add a `test_proportion` for the one-sample case. (similar to confint_proportion/proportion_confint

also need to clarify docs, docstrings are not explicit enough for minlike versus equal-tailed/central

user: josef-pkt
issue_8231: state: open
title: ENH: Parallel search for auto-regressive models
body: #### Is your feature request related to a problem? Please describe
I've recently come across a situation when I needed to do a lot of AR model searches for a considerable number of samples. There are built-in tools for that in statsmodels, such as `statsmodels.tsa.ar_model.ar_select_order`, but the problem is that they are not parallel.

I kind of have a prototype implementation for parallel model search which I could put a bit more effort into and submit a PR. Would this be a welcome addition for the repo or it is basically a design decision to not have a parallel model search implementation at that level?

#### Describe the solution you'd like
I can implement a parallel model search using `joblib`. doing a search in parallel would be optional and could be implemented as an additional flag for the aforementioned routine. 

#### Describe alternatives you have considered
Parallelization on the data level - doing multiple searches of orders for multiple data slices. 

#### Additional context
This is not a feature request so to say, more like a question about parallel model search, which we can just close if this is of no interest to the project.
user: sofeikov
issue_8229: state: open
title: ENH: Add Friedman's supersmoother for use in MSTL
body: #### Is your feature request related to a problem? Please describe
Currently, the version of MSTL implemented in Statsmodels does not replicate the same behaviour as the original algorithm when a user specifies no seasonal components. When there is no seasonal component, the original algorithm uses [Friedman's super smoother](http://fmwww.bc.edu/RePEc/bocode/s/supsmooth_doc.pdf) to just extract a trend. To reflect the method as described in the [paper](https://arxiv.org/abs/2107.13462) it would be good to add this behaviour to the version of MSTL in Statsmodels. One issue is that Friedman's super smoother is not implemented in Statsmodels. 

#### Describe the solution you'd like
Friedman's supersmoother has an implementation in Python [here](https://github.com/jakevdp/supersmoother). One solution is to use this in the MSTL implementation. However, I'm not sure whether it's actively maintained and whether it would be acceptable to introduce additional dependencies to Statsmodels. 

#### Describe alternatives you have considered
An alternative would be to re-write a version of Friedman's supersmoother in Statsmodels. Another solution would be to use LOWESS as an alternative method to extract the trend in the short term until a version of Friedman's supersmoother becomes available in Statsmodels. 

#### Additional context
An implementation in [R also exists](https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/supsmu).

I'm happy to take this on, but I'd like any guidance from the maintainers about what path to take here. 
user: KishManani
issue_8224: state: open
title: Performing KPSS test, constant array, value error
body: Hello!

I was using statsmodels to perform some stationarity analysis on some robotics data, however at some point I encountered a problem. Namely, when I passed a part of the data, which I later found was constant over the interval, I received an error saying  [**ValueError: cannot convert float NaN to integer]**, and I cannot figure where does it come from

It definitely might be that I don't have a good understanding of the way KPSS works in these, but definitely passing a constant to the test function results in this value error. I have, however, for testing purposes created a linearly spaced array from 1.0 to 1.000001 (and varied this), and I do get useful results in such a case.

I hope you could help me with my concerns.


user: erminkev1
issue_8220: state: open
title: ENH: permutation tests: transformation versus robust statistics
body: #6421 nonzero null
several other issues for permutation

I need to collect references

permutation are easy under common distribution assumption, or simple mean-shift assumption, but they can be nonrobust in general unless we want to change our null hypothesis to e.g. brunner-munzel stochastic dominance, instead of comparisons of mean or mean parameters.

e.g. heteroscedasticity

- transform, zscore data before permutation  (zscore data and then add mean back in)
- use studentized t-test (Welch) with permutation of original data

searching "studetized permutation" finds the second approach using test statistic designed for or robust to heteroscedasticity

I don't remember where I saw the first approach

Implementation should be easy, but I have no overview which permutation under which assumptions, 
I need references that look like good candidates to implement.

user: josef-pkt
issue_8219: state: open
title: ENH: permutation testing intersection-union, union-intersection
body: parking some references.
I'm not sure they are practically really important but the theoretical structure looks interesting.
I have not tried to understand the details yet.
Interesting point is that data is transformed before permutation to be at the indifference (equivalence or nonequivalence) margin

covers
- equivalence tests, tost, intersection-union test
- nonequivalence, minimum effect tests, union-intersection test

Arboretti, R., E. Carrozzo, F. Pesarin, and L. Salmaso. 2018. “Testing for Equivalence: An Intersection-Union Permutation Solution.” Statistics in Biopharmaceutical Research 10 (2): 130–38. https://doi.org/10.1080/19466315.2018.1447993.

Arboretti, Rosa, Fortunato Pesarin, and Luigi Salmaso. 2020. “A Unified Approach to Permutation Testing for Equivalence.” Statistical Methods & Applications, November. https://doi.org/10.1007/s10260-020-00548-0.

Pesarin, Fortunato, Luigi Salmaso, Eleonora Carrozzo, and Rosa Arboretti. 2016. “Union–Intersection Permutation Solution for Two-Sample Equivalence Testing.” Statistics and Computing 26 (3): 693–701. https://doi.org/10.1007/s11222-015-9552-y.
this is for nonequivalence, minimum effect testing with interval null, reverse tost

related 
#6421 permutation with non-zero null

user: josef-pkt
issue_8217: state: open
title: SUMM: roadmap for 0.15 josef
body: (continuation of #7720 roadmap for 0.14)


- gmm for combining models, equations
   -  #8216
- look at polynomial expansion for distributions again #8326
  no specific tasks, just to pick it up again after 8 or 10 years
- profile confint #1791, 
   score confint #2722 and checking constrained optimization again, maybe #5386
- plot and confint for ECDF #8193 #8196
- add support for formulaic
- nonparametric: 
  - #3380 
  - #3492
- predict function still in sandbox #6735
- ...
- ...
- 

new models

- Vectorized OLS with variance shrink options
- ...

robust ( have 4 larger PRs, getting all of them will take 2 to 3 months)
- [ ] #1341
  higher priority, bugfix and results enhancements, rsquared, ...
- [ ] #3261
- [ ] #8125
- [ ] #8129
  maybe combine with other covariance/correlation PRs, e.g. shrinkage
- [ ] other open ENH requests, e.g. cov_type sandwiches, HC, HAC
- [ ] ...

stats

- another round of power enhancements
  - #8251 
  - #8652 
- look at (unconditional) exact tests for binomial and poisson again, 
  - issue #2607 
  - PR for proportion #2608 
  - no code for poisson yet, but etest is implemented
- tests for multinomial data, check what we need or have
- k-sample rank tests, anova, multiple comparisons #6994
  - check, can we use heteroscedasticity robust oneway (BF) with ranks/quasi-ranks directly
  - main problem for implementation: cov for heteroscedasticity robust brunner-munzel type oneway is very complicated and
    description of it is very difficult to understand.
- multiple comparisons move out of sandbox and clean up
  - verify and enhance functions, e.g. max-t pvalues using numerical integration, contrast functions 
  - allow other contrasts, e.g. all versus control (dunnet), all versus mean
  - reuse methods that call 2-indep test functions.
  - maybe subclasses that reuse parts of setup like pairs loops e.g. for rank based multiple comparisons
- control charts, #5506
- ...
user: josef-pkt
labels: roadmap: roadmaps, list of todos, overall and by topic
issue_8216: state: open
title: ENH: SUEST, combining models, predictions, margins (maybe with GMM)
body: (mainly parking idea and references and links)

Stata has suest command to provide inference for combining tw or more models
Wooldridge mentioned in a tweet that suest cannot be used for post-estimation commands like margins or predictions.

suest = seemingly unrelated estimation

 suest works by stacking the model (block diagonal exog) and using cluster robust standard errors to compute cov_params taking into account that each observational unit is in several rows.
I didn't look at the details how cov_params are exactly defined and calculated.

However, it looks to me that it might be the same structure as using GMM for treatment effects, i.e. stack moment conditions horizontally (along axis=1) and let GMM compute the joint cov_params.

related
- #1137 hausman test if one model is efficient
- #7436 exactly identified GMM with further links
- #8034 PR for treatment effect using GMM for cov_params and inference

related issue for inference for comparing several predictions is for the case of a single model, not for combining predictions from several models.
However, it might be related if we use post-estimation commands based on the combined (stacked) model to, for example, compare predictions or margins across models.

Mize, Trenton D., Long Doan, and J. Scott Long. 2019. “A General Framework for Comparing Predictions and Marginal Effects across Models.” Sociological Methodology 49 (1): 152–89. https://doi.org/10.1177/0081175019852763.

code and replication files at
https://www.trentonmize.com/research#h.eg6fmo38wesy

https://www.statalist.org/forums/forum/general-stata-discussion/general/1656859-using-suest-after-margins-with-a-tobit-model

user: josef-pkt
issue_8214: state: open
title: ENH question  score test for mean, wald or score variance estimate
body: something to look at, but I don't know any details yet

standard ztest and t-test use the "wald" variance as variance estimate

under score test assumptions we should use the variance estimated under the null, e.g. null mean instead of observed mean to compute variance.

This is what the model score_test currently does because the full model is re-estimated under the null restriction.

There are some issues in the literature about choosing the variance, but I never looked closely at that.

e.g. "exact" test would treat variance as nuisance parameter and would need to use some approach to accommodate that 
(I guess sup test will not work because domain of variance is unbound)

The opposite of making var depend on null versus alternative
Part of the literature on LR tests, F-tests recommends using the same variance estimate between null and alternative hypothesis to avoid numerical problems.

Partially related: incorporating distributional assumptions as maintained assumption
test of variance are more accurate if normal distribution is assumed for the 4th moment (if normality can be assumed) than tests that are more robust and estimate 4th moments.

parametric distributional assumptions:
We are using poisson variance in poisson test instead of using estimated variance as in a t-test. The latter, or a robust cov_type, is more appropriate if we cannot rely on the poisson assumption for the variance.

also related
local alternatives:
Under the assumption of local alternatives for deriving the test statistic and it's distribution, the variance estimate is always the same asymptotic value for null and sequence of alternative. Asymptotically it doesn't matter in this case.




user: josef-pkt
issue_8212: state: open
title: FAQ-D/TST: consistency checks for hypothesis test, confint and power
body: Some definitional identities that I use for unit tests to check consistency of hypothesis test, confidence intervals and power functions.

Some of the following only applies to central, equal tail confint.

simplified case y observation, effect, v value, alpha, pv pvalue

_, pv = test(y, value) 
low, upp = confint(y, value, alpha, alternative)
pow = power(value_alt, alpha, nobs)

pvalue at confint

test(y, low).pvalue == alpha (two-sided) or == alpha / 2 (one-sided)
test(y, upp).pvalue == alpha (two-sided) or == alpha / 2 (one-sided)

equal tail:

confint(y, value, alpha / 2, alternative=two-sided) == confint(y, value, alpha, alternative=one-sided) for low or upp or both

power:

power(value_null, alpha, nobs) == alpha

power(value_alt, alpha, nobs, two-sided) approximately equal to either 
power(value_alt, 2 * alpha, nobs, lower) or power(value_alt, 2 * alpha, nobs, upper)
if value_alt - value_null is large.


There might be more that I don't remember right now.
e.g. 
for equivalence tests TOST
for equality testing where value is "nil" value, 0 for diff, 1 for ratio








user: josef-pkt
issue_8209: state: open
title: ENH/DOC: Use math formulaes in latex inside docstrings for methods of `RegressionModel`, `LikelihoodModel` and functions from tools and base folders
body: #### Is your feature request related to a problem? Please describe
My feature request is not related to a problem or an existing github issue or any code, but about documentation and improving readability.

#### Describe the solution you'd like
Many functions inside [`statsmodels.discrete.discrete_model.py`](https://github.com/statsmodels/statsmodels/blob/main/statsmodels/discrete/discrete_model.py), if not all, has formulaes in latex in docstrings of functions/methods; the class `discrete.discrete_model.DiscreteModel` inherits from, that is, `statsmodels.base.model.LikelihoodModel`'s `.fit()` method doesn't have any math in latex, but the classes that use it, for ex, `discrete.discrete_model.DiscreteModel` has math at relevant places. But, very few methods in [`statsmodels.regression.linear_model`](https://github.com/statsmodels/statsmodels/blob/main/statsmodels/regression/linear_model.py) has math in docstrings of functions/methods inside classes.

Not just these, [`tools`](https://github.com/statsmodels/statsmodels/tree/main/statsmodels/tools) and [`base`](https://github.com/statsmodels/statsmodels/tree/main/statsmodels/base) folders has lot of math-heavy functions that are imported and used in other modules, and do not include math in docstrings at all. please do consider adding math to the docstrings for such functions, since it is not possible for people like me to buy all the textbooks that you mention in the beginnning of each module(.py) and would help folks like me.

#### Describe alternatives you have considered
including math in latex(`:math:`) inside docstrings of functions or classes where math operations are being done. 

#### Additional context
Add any other context about the feature request here.
user: naveen-marthala
issue_8208: state: open
title: ENH/DOC Check margins at exog for subset or new full exog
body: https://stackoverflow.com/questions/71700418/is-there-a-way-to-subset-data-for-python-statsmodels-get-margeff-like-in-r-ma

get margins for subset of sample exog

Need to check this, I never tried

I thought we could use just the exog[mask], but the docstring for `atexog` sounds different

```
atexog array_like, optional
Optionally, you can provide the exogenous variables over which to get the marginal effects. This should be a dictionary with the key as the zero-indexed column number and the value of the dictionary. Default is None for all independent variables less the constant.
```
fist line says array_like, description says it should be a dict
user: josef-pkt
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_8206: state: open
title: FAQ/SUM: hypothesis tests, power - signatures and conventions
body: we should get a more consistent pattern for the signature of functions, including helper functions
When I write new code I don't always remember the pattern, if there is one, and make inconsistent signatures

TBD

basic convention for 2 indep (at least recently):

sample 1 is treatment, sample 2 is control
H0: `s1 - s2 - value = 0`  and under the alternative it is unequal !=, smaller <, or larger >

in power sample size computation we search for nobs1, given nobs_ratio
nobs2 = nobs1 * nobs_ratio 
(If users want to find nobs_ratio for fixed value of nobs2, then they need to flip indices between sample 1 and 2)

in function that can be either for one-sample or two-sample problems, 
`nobs` in helper function can be either (total) nobs or nobs1

std or var in signature
the recent power functions all have `std` and and so do generic functions in `weightstats`, e.g. `_zstat_generic2` 


user: josef-pkt
issue_8205: state: open
title: ENH: Add other forecasting methods similar to STLForecast for seasonal_decompose and/or HP filter
body: I am using STL to forecast like this:

```python
import matplotlib.pyplot as plt
from statsmodels.datasets import elec_equip as ds
from statsmodels.tsa.arima.model import ARIMA
from statsmodels.tsa.forecasting.stl import STLForecast

elec_equip = ds.load().data
elec_equip.index.freq = 'MS'
stlf = STLForecast(elec_equip, ARIMA, model_kwargs=dict(order=(1, 1, 0), trend="t"))
stlf_res = stlf.fit()

forecast = stlf_res.forecast(24)
plt.plot(elec_equip)
plt.plot(forecast)
plt.show()
```

Is there something similar to `STLForecast()`, but for seasonal_decompose?
Same question for Hodrick-Prescott filter.

I checked https://www.statsmodels.org/devel/tsa.html#module-statsmodels.tsa.forecasting, and I can't see any - to be exact I can only see a Theta model and an STLForecast, so I should assume that these two wrappers only exist?
user: gsamaras
issue_8198: state: open
title: SUMM/ENH: pairwise comparisons, methods, multiplicity correction
body: #7332 PR for games howell
#8168 issue for snk, dunnet-t

currently we only have tukey-hsd as supported method
we have generic allpairs test but without specific p-values like studentized range, requires use of pvalue mutliplicity correction

This article has an overview with formulas in the online supplement
formulas are in and from spss manual IBM_SPSS_Statistics_Algorithms_28.0.0.pdf appendix G

Sauder, Derek C., and Christine E. DeMars. 2019. “An Updated Recommendation for Multiple Comparisons.” Advances in Methods and Practices in Psychological Science 2 (1): 26–44. https://doi.org/10.1177/2515245918808784.


tukey-hsd uses pooled variance (variance of residual from group-demeaning), i.e. assumes homoscedasticity
games-howell and others allow for heteroscedasticity

Sauder, DeMars list 18 methods available in SPSS, 4 of which are for heteroscedasticity

The second distinction is in sequential versus simultaneous methods.

The tukeyhsd function in the sandbox already a switch whether variances are assumed to be equal. However, the method in the MultiConparison class only uses the pooled variance.
Aside: I'm not sure whether the path for unequal variance in the tukeyhsd function is correct. I remember having some bugs in parts that were not used yet at the time.

So I guess we can make the pairwise testing function more generic and use keyword option for the variance assumptions similar to what I did in `oneway`. We can keep alias functions for tukey-hsd and games-howell for name recognition.
Optionally, we can use pvalue multiple testing correction instead of using studentized range distribution.

(I will have to work my way through the details of those methods.)

---

supporting code
#7332 also has plot and letter functions, I have letter assingment in another PR

---

pairwise comparison do not produce a partition of groups. 
(I still have some old code in the sandbox, where I looked at getting partitions)

other articles that I just saw use AIC or model selection criteria to estimate models that partition the set of groups.
might be worth a look
If we use OLS (or other models) and information criteria to select a model, then all we need to to is to create the design matrix for each partition, (or sequence of partitions?)

Cribbie, Robert A., and H. J. Keselman. 2003. “Pairwise Multiple Comparisons: A Model Comparison Approach versus Stepwise Procedures.” British Journal of Mathematical and Statistical Psychology 56 (1): 167–82. https://doi.org/10.1348/000711003321645412.

Dayton, C.   Mitchell. 1998. “Information Criteria for the Paired-Comparisons Problem.” The American Statistician 52 (2): 144–51. https://doi.org/10.1080/00031305.1998.10480554.

 







user: josef-pkt
issue_8196: state: open
title: ENH add plot method to ecdf
body: companion to #8193 add confint

some links and discussion in #6480
with links to matplotlib issue https://github.com/matplotlib/matplotlib/issues/16561

and seaborn which has now ecdf https://github.com/mwaskom/seaborn/pull/2141
https://seaborn.pydata.org/generated/seaborn.ecdfplot.html
we might be able to optionally reuse seaborn, especially for ECDFDiscrete (I haven't tried yet)

user: josef-pkt
issue_8194: state: open
title: ENH: Add quantile location options to qqplot
body: #### Describe the bug

The Q-Q plot dose not match the probplot in scipy.stats perfectly

#### Code Sample, a copy-pastable example if possible


```python
def example3():
    import numpy as np
    import statsmodels.api as sm
    import matplotlib.pyplot as plt
    import scipy.stats
    # Random data points generated
    X , y= data[['gold']].diff(axis=0)[1:], data['silver'].diff()[1:]
    lr.fit(X, y)
    e = y - lr.predict(X)
    e /= np.sqrt(np.mean(e**2))

    s = np.sort(e)
    N = len(s)
    # Filliben's estimate
    q = (np.arange(1, N+1) - 0.3175) / (N + 0.365)
    q[-1] = 0.5**(1/N)
    q[0] = 1 - 0.5**(1/N)
    n = scipy.stats.norm()
    x = n.ppf(q)

    fig = plt.figure()
    ax = fig.add_subplot(111)

    scipy.stats.probplot(e, plot=plt)
    # sm.qqplot(e, line='45', ax=ax) # there are biases between two q-q plots if draw them in one axes.
    ax.set_title('Q-Q plot')

    ax.plot(x, s, 'xr')

    plt.show()


example3()

```
<details>

**Note**: It is not a serious problem. It seems that the QQ plot of statsmodels is not drawn in the right place of the given axes

</details>

<details>

#### Expected Output

two QQ-plots should cover each other, no gaps here.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

Latest version!

</details>

user: Freakwill
issue_8193: state: open
title: ENH: add confint method to ECDF
body: this has very good overview notes, including Dvoretzky-Kiefer-Wolfowitz simultaneous confidence band and pointwise confint_proportion confidence intervals
it looks like both should apply to both discrete and continuous distributions

http://dutiosc.twi.tudelft.nl/~cai/AS2015/lecture1_notes.pdf

I never found reference numbers to unit test, but this is clear enough that we can just add it without "verification"

related #8192  adding weights to ECDF, ecdf discrete


user: josef-pkt
issue_8190: state: open
title: BUG: OLS, WLS, fixed scale  cov_type does not change scale attribute, wrong resid_pearson
body: not verified in an example yet

looking at PR  #2137 again to answer a stackoverflow question https://stackoverflow.com/questions/71604868/how-to-get-the-unscaled-regression-coefficients-errors-using-statsmodels

PR top comment has
"
Do we want to overwrite the scale attribute?
It's not necessary for cov_params and a bit of a pain to change - issues for unrelated refactoring.
skip this for now. this depends on other uses of the scale attribute
"

AFAICS, scale attribute is not changed in current master.

We have results statistic other than wald inference that also use `scale`, 
`resid_pearson` is the only one I see right now, but there might be others in diagnostics, outlier-influence


GLM has the scale_type option to fix the scale. I guess extra results statistic should be correct there because we needed to do more work in GLM to handle different cases for scale.

unit test with comparison of OLS/WLS and GLM will be useful to make this consistent

I'm not sure if this is a bug-wrong or a question of how to define those statistics.
e.g. we need pearson chi2 statistic evaluated at the fixed scale

user: josef-pkt
labels: type-bug-wrong: serious bugs that silently return incorrect numbers
issue_8188: state: open
title: QuantReg no longer supports fit_regularized 
body: #### Is your feature request related to a problem? Please describe
The 0.6.1 version of statsmodels has support for fit_regularized() for QuantReg as mentioned in the docs: https://www.statsmodels.org/0.6.1/generated/statsmodels.regression.quantile_regression.QuantReg.fit_regularized.html

But the more recent versions don't seem to support it anymore. 

#### Describe the solution you'd like
Bringing back regularization for Quantile Regression. 

#### Additional context
Could not find any document or context as to why this was removed from statsmodels. 
user: bharatkrishna
issue_8187: state: open
title: DOC: OLS improve get_prediction PredictionResults
body: prediction results docs are not very informative

e.g. confint is missing description for `obs` keyword, no clear explanation about confint for conditional expectation versus prediction interval confint obs.
https://www.statsmodels.org/dev/generated/statsmodels.regression.linear_model.PredictionResults.conf_int.html


user: josef-pkt
issue_8186: state: open
title: ENH/TST  check inherited get_prediction in QuantReg
body: #6979 general issue for adding get_prediction

I never checked what the inherited get_prediction does for quantile regression.

It inherits from linear regression model.
I think, at least confint_obs does not make sense in quantreg, and I'm not sure whether the other methods are appropriate.

also terminology
"mean" would be the predicted quantile and not the mean of the underlying sample or distribution.

maybe add specific predict and use generic get_prediction instead of inherited regression PredictionResult


Aside:
we could use quantreg to estimate a prediction or tolerance interval for the linear model (OLS), or use it for a related diagnostic on the predictive distribution.
#6979


user: josef-pkt
issue_8185: state: open
title: p-adj values in pairwise_tukeyhsd and MultiComparison bounded by 0.001 and 0.9
body: #### Describe the bug

Dear developer, 

when I'm running pairwise_tukeyhsd or MultiComparison.tukeyhsd I see that the values in p-adj are bounded from below by 0.001 and from about by 0.9.  I see that when I compare to R test TukeyHSD the values of the p-adj can be higher than 0.9 and smaller then 0.001. 

This is a problem for me since I have many other of the column M# (i.e, M1, M2, ..., M900) 
and I'll want to adjust with FDR (False discovery rate )for these values afterward. To do so, values that are much smaller the 0.001 are rounded to 0.001 and will not be significant after FDR.

#### Code Sample python:


```python

# the data 
dat = {'Class_0': ['EKVX', 'EKVX', 'EKVX', 'EKVX', 'EKVX', 'EKVX', 'HOP62', 'HOP62',
                   'HOP62', 'HOP62', 'HOP62', 'HOP62', 'HOP92', 'HOP92', 'HOP92',
                   'HOP92', 'HOP92', 'HOP92', 'MPLT4', 'MPLT4', 'MPLT4', 'MPLT4',
                   'MPLT4', 'MPLT4', 'RPMI8226', 'RPMI8226', 'RPMI8226', 'RPMI8226',
                   'RPMI8226', 'RPMI8226'],
        'M2': [29.0660572, 29.4701607, 29.92649292, 29.62876917, 31.09229301,
              30.73131457, 29.54510531, 30.37553099, 29.22060366, 30.00387611,
              30.2893471, 30.42107896, 30.36357138, 30.04577991, 30.41139568,
              30.18903135, 30.23977308, 30.38604485, 28.45853761, 28.34003309,
              28.49059352, 28.65866869, 28.73640364, 28.76752952, 26.45224629,
              26.17478715, 26.77912804, 26.55138747, 26.69202968, 26.58586463]}

dat = pd.DataFrame(data=dat)

tueky_out = pairwise_tukeyhsd(endog=dat['M2'] , groups=dat['Class_0'], alpha=0.05)
print(tueky_out)
 Multiple Comparison of Means - Tukey HSD, FWER=0.05  
======================================================
group1  group2  meandiff p-adj   lower   upper  reject
------------------------------------------------------
  EKVX    HOP62  -0.0099    0.9 -0.7456  0.7257  False
  EKVX    HOP92   0.2868 0.7575 -0.4489  1.0224  False
  EKVX    MPLT4  -1.4106  0.001 -2.1462 -0.6749   True
  EKVX RPMI8226  -3.4466  0.001 -4.1823 -2.7109   True
 HOP62    HOP92   0.2967 0.7359  -0.439  1.0323  False
 HOP62    MPLT4  -1.4006  0.001 -2.1363  -0.665   True
 HOP62 RPMI8226  -3.4367  0.001 -4.1723  -2.701   True
 HOP92    MPLT4  -1.6973  0.001  -2.433 -0.9616   True
 HOP92 RPMI8226  -3.7334  0.001  -4.469 -2.9977   True
 MPLT4 RPMI8226  -2.0361  0.001 -2.7717 -1.3004   True
------------------------------------------------------

```
#### Code Sample R for the same data:
```
aov.out <-lm(M2 ~ Class_0 ,data =dat )
TukeyHSD.out <-TukeyHSD(aov(aov.out))
> TukeyHSD.out
  Tukey multiple comparisons of means
    95% family-wise confidence level

Fit: aov(formula = aov.out)

$Class_0
                       diff        lwr        upr     p adj
HOP62-EKVX     -0.009924239 -0.7455454  0.7256969 0.9999994
HOP92-EKVX      0.286751449 -0.4488697  1.0223726 0.7814832
MPLT4-EKVX     -1.410553581 -2.1461747 -0.6749324 0.0000670
RPMI8226-EKVX  -3.446607382 -4.1822285 -2.7109862 0.0000000
HOP92-HOP62     0.296675687 -0.4389455  1.0322968 0.7599709
MPLT4-HOP62    -1.400629342 -2.1362505 -0.6650082 0.0000741
RPMI8226-HOP62 -3.436683143 -4.1723043 -2.7010620 0.0000000
MPLT4-HOP92    -1.697305029 -2.4329262 -0.9616839 0.0000039
RPMI8226-HOP92 -3.733358831 -4.4689800 -2.9977377 0.0000000
RPMI8226-MPLT4 -2.036053801 -2.7716750 -1.3004326 0.0000002
```
I have run these for different values of the Columns 'M#' (i.e, M1, ...,M900) 

There is a difference in the values in the adj-pvalue but they are somewhat close between R  and python, however, in statsmodels they are always bounded by 0.001 and 0.9. 

is it possible to change it? 


Kindly help, 

Tal
user: TalWac
issue_8183: state: open
title: Documentation inconsistency in VAR result df_model property
body: #### Describe the bug
The `df_model` property of the VAR models does not seem to reflect what is written in the documentation. In the documentation it is called the number of parameter estimates, but it seems to be the number of parameter estimates **per variable**. The property is given as `self.neqs * self.k_ar + self.k_exog`, which is should be `self.neqs(self.neqs * self.k_ar + self.k_exog)` to match the documentation or the documentation should be changed in my opinion. 


#### Code Sample, a copy-pastable example if possible


```python
from statsmodels.tsa.api import VAR
import numpy as np
data = np.random.random((100, 5)) # 5 variables, 100 observations
fit_result = VAR(data).fit(maxlags=2)
print(fit_result.df_model) # 11
print(fit_result.coefs.size) # 50
print(fit_result.coefs_exog.size) # 5
```



#### Expected Output

I think either the `df_model` property of a VAR fit result or its documentation should be altered. 

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]
INSTALLED VERSIONS
------------------
Python: 3.10.2.final.0
OS: Linux 5.13.0-35-generic #40~20.04.1-Ubuntu SMP Mon Mar 7 09:18:32 UTC 2022 x86_64
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8

statsmodels
===========

Installed: 0.13.0 (/home/rutger/anaconda3/envs/Graduation/lib/python3.10/site-packages/statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.21.2 (/home/rutger/anaconda3/envs/Graduation/lib/python3.10/site-packages/numpy)
scipy: 1.7.3 (/home/rutger/anaconda3/envs/Graduation/lib/python3.10/site-packages/scipy)
pandas: 1.4.1 (/home/rutger/anaconda3/envs/Graduation/lib/python3.10/site-packages/pandas)
    dateutil: 2.8.2 (/home/rutger/anaconda3/envs/Graduation/lib/python3.10/site-packages/dateutil)
patsy: 0.5.2 (/home/rutger/anaconda3/envs/Graduation/lib/python3.10/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.5.1 (/home/rutger/anaconda3/envs/Graduation/lib/python3.10/site-packages/matplotlib)
    backend: module://matplotlib_inline.backend_inline 
cvxopt: Not installed
joblib: 1.1.0 (/home/rutger/anaconda3/envs/Graduation/lib/python3.10/site-packages/joblib)

Developer Tools
================

IPython: 7.29.0 (/home/rutger/anaconda3/envs/Graduation/lib/python3.10/site-packages/IPython)
    jinja2: 3.0.2 (/home/rutger/anaconda3/envs/Graduation/lib/python3.10/site-packages/jinja2)
sphinx: Not installed
    pygments: 2.11.2 (/home/rutger/anaconda3/envs/Graduation/lib/python3.10/site-packages/pygments)
pytest: Not installed
virtualenv: Not installed


</details>

user: RutgerVanBeek
issue_8182: state: open
title: FAQ/ENH: power, sample size computation, find nobs_ratio for fixed total sample size 
body: https://stackoverflow.com/questions/71563792/negative-sample-size-by-statsmodels-ttestindpower-solve-power/71564340

It would be possible to add an option to do this, but it doesn't fit with the current keywords for solve_power.

Below is copy of my answer

Solving for nobs1 in this setting is not possible with `solve_power` because both nobs1 and nobs_ratio need to change in this case. 
`solve_power` can only search for the value of one keyword given the others.

The function used for the rootfinding needs to be changed so that it solves for nobs1 for a given total size.
There might not always exist a root (*) in this case or there could be multiple roots.

In the following I check that power is monotonic over the relevant range of nobs1 and that it includes the desired power of 0.8 in this range of nobs1.
Then we can use a scipy rootfinder to solve for nobs1.


    nobs1 = np.arange(100, 500, 10)
    pow_ = TTestIndPower().power(effect_size=effect_size, nobs1=nobs1, alpha=alpha,
                                 ratio=(total_size - nobs1) / nobs1, alternative='two-sided')
    array([0.50469817, 0.54182567, 0.57681571, 0.60967519, 0.64043692,
           0.6691538 , 0.69589402, 0.72073695, 0.74376976, 0.76508462,
           0.78477648, 0.80294111, 0.81967375, 0.83506793, 0.84921453,
           0.86220121, 0.87411189, 0.88502644, 0.89502052, 0.90416541,
           0.91252805, 0.92017113, 0.92715306, 0.93352823, 0.93934712,
           0.94465644, 0.9494994 , 0.95391587, 0.95794252, 0.96161315,
           0.96495878, 0.96800784, 0.97078643, 0.97331846, 0.97562573,
           0.97772825, 0.97964426, 0.98139039, 0.98298186, 0.9844325 ])
    
    from scipy import optimize
    def power_func(nobs1):
        pow_ = TTestIndPower().power(effect_size=effect_size, nobs1=nobs1, alpha=alpha,
                                     ratio=(total_size - nobs1) / nobs1, alternative='two-sided')
        return pow_ - 0.8
    
    optimize.brentq(power_func, 100, 500)
    208.32446507391262

(*)
As a quick check whether the total sample size is large enough for the desired power, we can compute the required sample size at nobs_ratio equals to 1. Using equal sample sizes will have the largest power for fixed total size.

In the example we need at least a total sample size of 787 to achieve the desired power of 0.8.


    nobs1 = TTestIndPower().solve_power(effect_size=effect_size, nobs1=None, alpha=alpha, power=power, 
                                       ratio=1, alternative='two-sided')
    nobs1, 2 * nobs1
    (393.4056989990322, 786.8113979980644)








user: josef-pkt
issue_8180: state: open
title: ENH: observation specific estimate for one parameter models like GLM
body: (strange idea while looking at various confidence, prediction, tolerance intervals for Poisson.)

for poisson and binomial count we only need one observation to estimate the "model", i.e. the single mean parameter.
in PR #8166, I'm adding confint for one sample poisson.

There might be a possible diagnostic measure comparing the one sample estimate and confint for each observation with the estimate and confint implied by a Poisson regression model.
Maybe get some outlier identification from those.

I'm not sure because I never looked at the background of the deviance definition.
single sample estimate might be similar to saturated model for GLM deviance




user: josef-pkt
issue_8177: state: open
title: ANOVA with frequency weights
body: Hi team,
Will you add ANOVA with frequency weights?  
By the way, remember I asked will you add LSD, SNK, and Dunnett test at #8168? You said almost no. And I just finish LSD and SNK tests by myself, I copy your MultiComparison class down to local and imitate tukeyhsd() to code the snk-q test. I have uploaded it at my stats repository, I'm glad if you give me some suggestion though the notes wrote by Chinese.
Thanks for your time,
Ben
user: shixian77
issue_8174: state: open
title: ENH: better confidence intervals for weighted average or sum of count (non-gaussian) variables
body: related to #8138 and PR #8166

standardized incidence rates are computed as population average based on subgroup rates and weights of group in population.
In the poisson case, this is the weighted sum of heterogeneous poisson rates.

The literature in this area build confidence intervals that are more accurate than simply assuming normal distribution of the weighted sum or average.
AFAICS, those method can be extended to other cases than Poisson (but I have only looked at references for poisson case)

Ng et al have a Monte Carlo with 20 methods, with formulas and references.
Dobson is one of the early articles, and they use a affine transformed univariate poisson confidence interval, where transformation is to match mean and variance.

further references for poisson are in articles, especially Ng. 

Krishnamoorthy use MLS/MOVER confidence interval for weighted average, and for other poisson cases.
That's a good starting point to more general applications. 

Dobson, Annette J., Kari Kuulasmaa, Ernst Eberle, and Josef Scherer. 1991. “Confidence Intervals for Weighted Sums of Poisson Parameters.” Statistics in Medicine 10 (3): 457–62. https://doi.org/10.1002/sim.4780100317.

Krishnamoorthy, K., Jie Peng, and Dan Zhang. 2016. “Modified Large Sample Confidence Intervals for Poisson Distributions: Ratio, Weighted Average, and Product of Means.” Communications in Statistics - Theory and Methods 45 (1): 83–97. https://doi.org/10.1080/03610926.2013.821486.

Ng, Hon Keung Tony, Giovanni Filardo, and Gang Zheng. 2008. “Confidence Interval Estimating Procedures for Standardized Incidence Rates.” Computational Statistics & Data Analysis 52 (7): 3501–16. https://doi.org/10.1016/j.csda.2007.11.004.

The Ng et al article has one worked example that could be used for unit tests of all methods.
another article Swift 2010 mentions that Ng et al have typographical errors in formulas, so they might not all be reliable.

Swift, Michael Bruce. 2010. “A Simulation Study Comparing Methods for Calculating Confidence Intervals for Directly Standardized Rates.” Computational Statistics & Data Analysis 54 (4): 1103–8. https://doi.org/10.1016/j.csda.2009.10.021.

compares a bootstrap method with selected methods from Ng 2008 article



user: josef-pkt
issue_8173: state: open
title: FAQ-D/Design:  API for composite "method" keyword arguments
body: "there should be one obvious way, and one way to get all options"

for statistical or algorithmic "methods" we often have the choice of using modifiers to get different variants of a method

example for hypothesis tests
wald and wald with continuity correction

two possible approaches
- use two keywords such as `method="wald", cc=True`
- use method keyword that combines name, e.g. `method="wald"` and `method="waldcc`

the second is nicer for loops, e.g. in unit tests or for comparison of tests, no need to carry around multiple keywords

```
methods = ["wald", "waldcc"]
for method in methods:
    <do something>
```

However, there might be too many combinations, e.g. if we have many possible variations of a methods.

example
There are several confidence interval methods that reuse confidence intervals from a simpler method, e.g. 
MOVER confint, or 
conditional confint that reduce to univariate confint possibly of a different distribution. (e.g. binomial for conditional poisson ratio comparison of 2 indep)

api design:

make both ways available, 
- shortcut names for common methods, or if there are only two options as in wald and waldcc
- semi-generic method keyword plus extra keyword that gives access to full options, e.g.
  - `"method="mover", sub_method="exact-c"`, `method="cond", sub_method="exact-c"`
  - generic name for second keyword will be better if we have different qualifiers for several main method

We have something similar already with `fit` optimization  methods, with specific optimizers and generic scipy `"minimize"`

current context:
- PR #8166  more hypthesis tests and confint for poisson rates
- need to retrofit existing binomial and poisson 2indep with more "sub-methods" for mover and conditional methods for confint and tests




user: josef-pkt
issue_8171: state: open
title: Maint: flake-8 linting of recently added modules
body: my recently added modules (last year) are not included in strict lint, flake-8 checks, AFAICS

I guess they need to be added to "statsmodels\lint.sh"

old directories like stats, discrete and genmod has individual modules under flake8 strict rules, but not the directories.




user: josef-pkt
issue_8170: state: open
title: TST: unclear test failure
body: copying a test failure (because azure does not keep the logs for long) from an unrelated PR run

looks like the test failed because of time change in two consecutive calls
but I don't see what the purpose of the test assert is

https://dev.azure.com/statsmodels/statsmodels-testing/_build/results?buildId=4405&view=logs&j=ff004b28-f232-5e0f-4df3-a5874d912eca&t=217a4bc8-b148-565c-30de-e7917895fc32

```
reset_randomstate = None

    def test_slim_summary(reset_randomstate):
        y = np.random.standard_normal(100)
        x = np.random.standard_normal((100, 1))
        x = x + np.random.standard_normal((100, 5))
        res = OLS(y, x).fit()
        summ = res.summary()
        summ2 = res.summary()
        slim_summ = res.summary(slim=True)
>       assert str(summ) == str(summ2)
E       AssertionError: assert '            ...ly specified.' == '            ...ly specified.'
E         Skipping 554 identical leading characters in diff, use -v to show
E         Skipping 1694 identical trailing characters in diff, use -v to show
E         -    22:07:09   Log-Li
E         ?           ^
E         +    22:07:08   Log-Li
E         ?           ^

statsmodels/regression/tests/test_regression.py:1621: AssertionError
```
user: josef-pkt
issue_8168: state: open
title: can you pls add LSD-t, SNK-q and Dunnett-t test functions in MultiComparison class?
body: or just give me some suggestions to realize it. thanks a lot
user: shixian77
issue_8167: state: open
title: Cannot import name '_centered' from 'scipy.signal.signaltools'
body: See the following thread on stackoverflow for a description: https://stackoverflow.com/questions/71106940/cannot-import-name-centered-from-scipy-signal-signaltools
user: cgarciga
issue_8163: state: open
title: ENH: maybe add one-sided confidence intervals (for cases with unequal tail probabilities)
body: I'm not sure, if we need this.

Almost all our confidence intervals (AFAIR) are central, with prob of each tail <= alpha / 2
In those cases the one-sided confint is just the corresponding limit of the two sided confint evaluated at alpha * 2

For minlike methods like scipy binomial test and fisher's exact test, this will not be the case. tail probs don't have a restriction, only the sum is < alpha.

The same would be true for profile confidence intervals in regression models.

Note:
actual coverage of confint can be unequal in tails, even by design it uses an equal tail method (e.g. based on normal approximation to the distribution of a test statistic)
(I recently looked at several articles that checked tail asymmetries in coverage for discrete random variables. 
references ???
maybe
Tony Cai, T. 2005. “One-Sided Confidence Intervals in Discrete Distributions.” Journal of Statistical Planning and Inference 131 (1): 63–88. https://doi.org/10.1016/j.jspi.2004.01.005.
)


user: josef-pkt
issue_8162: state: open
title: FAQ/SUMM: Power and sample size computation - what is available.
body: (I lost an overview)

by method:
- power based on effectsize
- power based on explicit distribution (inherent heteroscedasticity)

by response variable:
- mean, continuous
- proportion, value, riskdiff, riskratio, oddsratio
- rates, value, diff, ratio
  - poisson 
  - negbin NB2
- variance ?
- ...
- ...

by sampling:
- 1 sample
- 2 indep
- 2 paired/correlated
- k sample, anova-style

effectsize based works generically given effect size definition for specific problem that has a standard distribution, normal, t, F, chisquare

### list of available effect size functions

- proportion_effectsize 2indep, equality test, based on arcsine transformation
- oneway.effectsize_oneway k-sample, oneway comparison of means
- ???
- ???

### explicit, specific power computation

AFAICS, only a few in `stats.proportions`
no Power classes for rootfinding for those #8159

- power_binom_tost, power_ztost_prop: 1sample
- power_proportions_2indep: 2indep, equality test, var depends on alternative
- oneway power_equivalence_oneway
- ...
- ...


doc issue:
collect recipes, helper functions for effectsize based sample size and power computation

WIP/ENH:
more explicit power functions and classes
my current context: 2indep for proportion, poisson and negbin, for tests with a margin, nonzero-null-value, and for equivalence tests.
#8138

user: josef-pkt
issue_8159: state: open
title: ENH: Power classes for sample size computation without effect size, i.e. explicit variances
body: part of #1197
cases where variance can differ between null and alternative or alternatives (in TOST equivalenc)
current context #8138

My original implementation of the Power classes followed the effect size tradition of incorporating standard deviation in the effect size instead of handling it explicitly in the methods of the power classes.

I have added power function for two sample proportion using std_null and std_alternative.
AFAIR, I didn't change power classes to accommodate for it.

example case
power for equivalence test for comparing ratio of 2 poisson or 2 negative binomial random variables
Zhu 2016 and PASS documentation
needs 3 standard deviation null under alternative at lower and upper equivalence margin, and distribution under alternative.

Poisson, NegBin and similar cases follow the same pattern.

I guess I will need classes for power of equivalence tests, separate from those for standard test of equality (or equality with a  margin value)

user: josef-pkt
issue_8150: state: open
title: Dependencies in requirements.txt are not fixed with "=="
body: #### Describe the bug

Requirements are not fixed with "==" but ">=". When new versions of upstream dependencies are released, older versions of statsmodels occasionally throw errors when pip installing them. As I use statsmodels as a requirement for my own package, I also have to publish a new release of my package with an updated statsmodels requirement in many cases when dependencies of statsmodels are being updated. Also, you guys have to be in a hurry to fix inconsistencies with your upstream packages. I think fixing statsmodels requirements with "==" for each version would prevent all this.

#### Current situation

```
numpy>=1.17
scipy>=1.3
pandas>=0.25
patsy>=0.5.2
packaging>=21.3
```

#### Proposed Solution

```
numpy==1.22
scipy==1.8
pandas==1.4
patsy==0.5
packaging==21.3
```

user: martinbaerwolff
issue_8149: state: open
title: ENH/SUMM: adaptive and sequential design, experiments, testing
body: I'm running into this every once in a while, and never went through the relevant theory.

some topics
decision maker is either designer of sequential experiments, or individuals as in selection models

- early stopping rules
  - example: drop ineffective treatments from a clinical trial
- interim sample size adjustments in sequential experiments
  - estimate nuisance parameters in early stages and redo sample size and power computation (if blinded, then based only on partial information)

I guess related to 

- "informative" drop-outs in panel data (dropour, missing observations not completely at random)
  - conditional independence: similar to missing data problem in TreatmentEffect estimation
  - endogenous ???

currently the references that I'm looking at are mostly for pharamceutical and clinical trials.
IIRC, I have seen adaptive designs for industrial application, e.g. adaptive stress testing

Marschner, Ian C. 2021. “A General Framework for the Analysis of Adaptive Experiments.” Statistical Science 36 (3): 465–92. https://doi.org/10.1214/20-STS803.

It's mostly and overview article, many references

a related article for consequences for meta-analysis
Schou, I. Manjula, and Ian C. Marschner. 2013. “Meta-Analysis of Clinical Trials with Early Stopping: An Investigation of Potential Bias.” Statistics in Medicine 32 (28): 4859–74. https://doi.org/10.1002/sim.5893.

PASS/NCSS has many tests for group-sequential designs.
I did not look closely about whether they are adaptive or not
e.g.
https://www.ncss.com/wp-content/themes/ncss/pdf/Procedures/NCSS/Group-Sequential_Analysis_for_One_Poisson_Rate.pdf

related in R, e.g. in
https://cran.r-project.org/web/views/ClinicalTrials.html
user: josef-pkt
issue_8148: state: open
title: ENH: add power for simple (linear or nonlinear regression models
body: narrow topic: get power for models with simple regressors, e.g. with two or k-sample comparison 
We should be able to compare those with  the standalone functions, e.g. proportions, rates, t-test or oneway

#6759 more general, 
and other issues

issues for specific designs, e.g. cross-over, .... ?

right now I would be glad for recipes and unit tests.

user: josef-pkt
issue_8147: state: open
title: ENH: GLM-GMM (GEE) without strict exogeneity
body: (random find, we had related discussion for GEE)

parking two articles

Arreola, Elsa Vazquez, and Jeffrey R. Wilson. 2020. “Partitioned GMM Marginal Model for Time Dependent Covariates: Applications to Survey Data.” In Computational and Methodological Statistics and Biostatistics: Contemporary Essays in Advancement, edited by Andriëtte Bekker, (Din) Ding-Geng Chen, and Johannes T. Ferreira, 511–28. Emerging Topics in Statistics and Biostatistics. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-030-42196-0_22.

Lalonde, Trent L., Jeffrey R. Wilson, and Jianqiong Yin. 2014. “GMM Logistic Regression Models for Longitudinal Data with Time-Dependent Covariates and Extended Classifications.” Statistics in Medicine 33 (27): 4756–69. https://doi.org/10.1002/sim.6273.

and a recent book
Wilson, Jeffrey R, Elsa Vazquez-Arreola, and (Din) Ding-Geng Chen. 2020. Marginal Models in Analysis of Correlated Binary Data with Time Dependent Covariates. https://link.springer.com/10.1007/978-3-030-48904-5.

I didn't look closely,
my impression is that it is similar to system GMM for dynamic panel, but for nonlinear, nongaussian cases as in GEE.

I think this is violation of strict exogeneity in econometrics, i.e. errors and "exog" are correlation over time

related
#1790 GLM-GMM
#1100 panel Probit
and there should be others

#7901 system GMM


user: josef-pkt
issue_8145: state: open
title: FAQ/SUMM:  hypothesis tests and models - equivalent and similar versions
body: (I don't find an existing issue)

Many standalone hypothesis tests and confidence intervals have an equivalent method using models.

The extension in standalone inference function is that they often have additional methods for better small sample behavior that are not available in the models.  (eg. we don't have score confidence intervals in models yet)
The extension in models compared to inference function is that we have robust cov_types available, and that it is easy to extend to more general settings.

I looked at this in several cases in recent PRs, but I don't see an overview page.

eg. IIRC
Welch-Satterthwaite t-test is OLS + HC3, but with additional degrees of freedom correction that is not available for OLS.

user: josef-pkt
issue_8144: state: open
title: ENH: discrete models with identity link
body: implementation will have to wait until we have multilink models #7793
We can use identity link in GLM, but that does not directly allow for extra params as in discrete NegativeBinomial(P)

for now parking reference and R package

Donoghoe, Mark W., and Ian C. Marschner. 2016. “Estimation of Adjusted Rate Differences Using Additive Negative Binomial Regression.” Statistics in Medicine 35 (18): 3166–78. https://doi.org/10.1002/sim.6960.

https://cran.r-project.org/web/packages/addreg/index.html

related: https://github.com/statsmodels/statsmodels/issues/8138#issuecomment-1047010771

user: josef-pkt
issue_8143: state: open
title: ENH: e-test for other count distributions, similar to poisson etest
body: (just an idea, no literature search)

"exact" or quasi-exact hypothesis tests and similar inference, assume that the distribution is correctly specified, and are not robust to misspecification.

For example, if we have the equivalent tests for negative binomial, or beta-binomial, then we can have tests robust to overdispersion or other variance function deviations to Poisson.

It still would assume that the distribution is correctly specified but it would be a less restrictive distribution than Poisson and Binomial.
At the minimum it would be good for checking the sensitivity of the inference to different variance assumptions.
user: josef-pkt
issue_8142: state: open
title: ENH: Control Charts for Monitoring the Ratio of Two Poisson Rates
body: Yamauchi, Tsubasa, and Linda Lee Ho. 2020. “Control Charts for Monitoring the Ratio of Two Poisson Rates.” Quality and Reliability Engineering International 36 (1): 214–30. https://doi.org/10.1002/qre.2567.



related issues:
#8138 more stats for poisson rates
#4191
#7137

user: josef-pkt
issue_8140: state: open
title: SUMM: aggregation, exposure, heterogeneity
body: mainly to park a reference that I ran into via a not directly related search

Cummings, Peter. Analysis of incidence rates. CRC Press, 2019.
chapter 5 on Criticism of Incidence Rates especially the later sections pointing out Negative Binomial, GEE, .. as alternatives.
(overall the book is not technical enough to write code from it)

The case : hospital number of incidences
what is the normalization and what can be aggregated.
e.g. 
- aggregating to person-years: aggregate over different individuals and different time periods
- looking at cases per individual, or e.g. per hospital admission

aggregation to pearson-years requires poisson process across individuals with identical rates.

Design assumption for us that would be useful:
- each observation has constant rate, so exposure for an observation is aggregating i.i.d. poisson observations
- any modification, heterogeneity is interpreted as constant within observations but can vary across observations, e.g. gamma mixing to get negative binomial
- if we have the same individual with repeated observations, then we need to include it, e.g. as in GEE.

consequence exposure would have a strict poisson aggregation interpretation, time for an observation additive in number of incidents

related:
- #2175
- tweedie aggregation #2915
- zero inflation #7158

main discussion that we had was when Tweedie and var_weights, freq_weights were added to GLM.

another consequence is in interpreting robust `cov_type`

What is the statistic of interest?

e.g. 
- for a hospital the overall incidence rate will be relevant (similar to a general populated ATE)
  - if exp mean function is appropriate to estimate the average effect, then we can as well use Poisson as long as we adjust cov_type
- for an individual prediction, we might be interested in more than a general average
  - e.g. zero-inflated, we have several predict `which` that provide more information


user: josef-pkt
issue_8139: state: open
title: Classification
body: #### Is your feature request related to a problem? Please describe
ı want to classify ordinal logistic reggression 
#### Describe the solution you'd like
ı want to confusion matrix classfication performance more than 2 level ordinal regression


user: ucmazsinan
issue_8138: state: open
title: SUMM/ENH: rates, poisson means - missing parts, one sample, confint, power
body: statsmodels.stats.rates has tests and tost for 2 independent samples.

- no confidence intervals
- no statistics for one sample case, test, tost, confint
- no power

We should be able to get the collection of functions similar to the ones for proportion

related:
- #3954
- #2718
- merged PR with comments and references #6677 

I don't see a related open PR or issue with code. Most likely I never worked on these.


user: josef-pkt
issue_8136: state: open
title: Inf value outputted for likelihood when fitting Tweedie GLM
body: #### Describe the bug

When fitting a generalized linear model with the Tweedie distribution, the current extended quasi-likelihood calculation produces likelihood values of `Inf`. An example where this is happening is provided below. Related to this, when running a summary on the fitted model in the example, it showed a warning message on taking the log of 0 when calculating the likelihood:

```python
RuntimeWarning: divide by zero encountered in log
  llf = np.log(2 * np.pi * scale) + p * np.log(endog)
RuntimeWarning: invalid value encountered in double_scalars
  prsq = 1 - np.exp((self.llnull - self.llf) * (2 / self.nobs))
```

In the source code, it looks like the extended quasi-likelihood is calculated by the `loglike` function and this calls ` loglike_obs` in `statsmodels/genmod/families/family.py`. Under the Tweedie class, `loglike_obs` has this line as part of the likelihood calculation: 
`llf = np.log(2 * np.pi * scale) + p * np.log(endog)`. 

I believe the `Inf` values are coming from the `np.log(endog)` part due to some of the values in the endogenous variable/response variable are 0, so it will result in values of `Inf` when taking the log. The `Inf` values will then result in a final `Inf` value for the extended quasi log-likelihood when the values are aggregated in `loglike` in the line `np.sum(ll_obs * freq_weights)`. However, considering the nature of the Tweedie distribution, we’ll probably always have a lot of values of 0 in the endogenous/response variable and so this will likely lead to the extended quasi-likelihood resulting in an `Inf` for most, if not all, Tweedie models being fitted. 

#### Code Sample, a copy-pastable example if possible

```python
import numpy as np
import statsmodels.api as sm
from statsmodels.genmod.generalized_linear_model import GLMResultsWrapper
import tweedie as tweedie
# generating a response variable and variables for fitting a Tweedie model
np.random.seed(123)
x1 = np.random.uniform(0,10,10000)
x2 = np.random.uniform(0,10,10000)
constant = np.ones(10000)
X = np.hstack((constant.reshape(10000, 1),x1.reshape(10000, 1),x2.reshape(10000, 1)))
b = [1,1,2]
mu = np.exp(np.dot(X, b))
Y = tweedie.tweedie(mu=mu, p=1.5, phi=10).rvs(10000).reshape(10000, 1)
# fitting the model
model = sm.GLM(
        endog=Y,
        exog=X,
        family=sm.genmod.families.family.Tweedie(var_power=1.5, eql = True, link = sm.families.links.log())
    )
fitted_model = model.fit()
# output 
fitted_model.summary()

```
<details>

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `main`. If your problem has been fixed in an unreleased version, you might be able to use `main` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the main branch of this repository? It helps the limited resources if we know problems exist in the current main branch so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

The expected outcome would be a non-Inf log-likelihood value. I’m trying to better understand the current extended quasi-log likelihood calculation conceptually. Is there a way to resolve this since based on the calculation, it will most likely always return an Inf value for Tweedie models?

#### Output of ``import statsmodels.api as sm; sm.show_versions()``
<details>

INSTALLED VERSIONS
------------------

Python: 3.9.10.final.0
statsmodels
===========
Installed: 0.13.1 (C:\DEV\git_repository\statsmodels-23\statsmodels\.venv\lib\site-packages\statsmodels)
Required Dependencies
=====================
cython: Not installed
numpy: 1.22.2 (C:\DEV\git_repository\statsmodels-23\statsmodels\.venv\lib\site-packages\numpy)
scipy: 1.8.0 (C:\DEV\git_repository\statsmodels-23\statsmodels\.venv\lib\site-packages\scipy)
pandas: 1.3.5 (C:\DEV\git_repository\statsmodels-23\statsmodels\.venv\lib\site-packages\pandas)
    dateutil: 2.8.2 (C:\DEV\git_repository\statsmodels-23\statsmodels\.venv\lib\site-packages\dateutil)
patsy: 0.5.2 (C:\DEV\git_repository\statsmodels-23\statsmodels\.venv\lib\site-packages\patsy)
Optional Dependencies
=====================
matplotlib: 3.5.1 (C:\DEV\git_repository\statsmodels-23\statsmodels\.venv\lib\site-packages\matplotlib)
Backend TkAgg is interactive backend. Turning interactive mode on.
    backend: TkAgg 
cvxopt: Not installed
joblib: 1.1.0 (C:\DEV\git_repository\statsmodels-23\statsmodels\.venv\lib\site-packages\joblib)
Developer Tools
================
IPython: 7.31.1 (C:\DEV\git_repository\statsmodels-23\statsmodels\.venv\lib\site-packages\IPython)
    jinja2: 3.0.3 (C:\DEV\git_repository\statsmodels-23\statsmodels\.venv\lib\site-packages\jinja2)
sphinx: 3.5.4 (C:\DEV\git_repository\statsmodels-23\statsmodels\.venv\lib\site-packages\sphinx)
    pygments: 2.11.2 (C:\DEV\git_repository\statsmodels-23\statsmodels\.venv\lib\site-packages\pygments)
pytest: 6.2.5 (C:\DEV\git_repository\statsmodels-23\statsmodels\.venv\lib\site-packages\pytest)
virtualenv: 20.13.1 (C:\DEV\git_repository\statsmodels-23\statsmodels\.venv\lib\site-packages\virtualenv)

</details>

user: ajiang1234
issue_8131: state: open
title: Data sets -- how to get the downloadable names of each
body: How can I get the actual names of all the datasets that can be downloaded from statsmodels for Python? Do you have a repository for them in *.txt or *.csv format? Why not put all of them in GitHub and make them easily available to the pubic?. Seaborn has all their data sets in GitHub and are quite easy to work with...
user: virsto
issue_8129: state: open
title: ENH: Outlier robust covariance - rebased
body: rebased version of #3230

(merge conflicts mainly in imports, not too bad)


overview

### in `robust.covariance`:

`cov_gk` GK  (no reference in docstring)
`cov_ogk` OGK based on Maronna and Zamar 2002   (docstring mentions difference to R)
`cov_tyler` standard Tyler's scatter iteration loop, mean assumed to be zero
`cov_tyler_regularized` shrinkage towards identity in iteration loop, Chen and Wiesel 2011
`cov_weighted` weighted mean and cov, no iteration loop
`cov_tyler_pairs_regularized`  experimental, not all options working
`_cov_iter` generic M-estimator for cov using weight function, available weights: `weights_mvt`, `weights_quantile`

`_cov_starting` incomplete, this is just collecting a list of different robust cov, a bit ad-hoc, 
    for use in high breakdown cov estimators,
    not useful as standalone (unless we make return more informative)

`cov_gk1` gk for just one pair of variables

other helper functions:
`mahalanobis`
`scale_tau`
`_naive_ledoit_wolf_shrinkage`
`_outlier_gy`
`_reweight`
`_weight_mean`
`_winsor`


### in `stats.covariance`:
`corr_normal_scores`  corr based on transforming ranks to normal scores
`corr_quadrant` based on signs, unverified
`transform_corr_normal` rescaling to be consistent at normal distribution, for 'pearson', 'gauss_rank', 'kendal', 'spearman' and
        'quadrant'
`_term_integrate` currently not used 


### status, todos

- consistency with reference distribution, gaussian cov
  - which estimators are consistent, or have option for returning consistent cov
  - recipe for rescaling, or document when rescaling is not available 
- unit test, Monte Carlo checks
  - ogk works very well in normal sample with a few outliers in MC 
- check returns, e.g. cov_ogk and cov_weighted do not return a Holder instance
  possibly additional returns in future (backwards compatible if Holder instance is returned), e.g. support for outlier detection 
- check options, e.g. string options instead of boolean  
- references in docstring are missing or incomplete
- check scale_tau function. try, support qn_scale which is now in robust
- high priority: ogk, tyler, M-estimator with e.g. t-weights 
  (add convenience function for t scatter with option for consistent cov)
- ...
- missing: 
  - joint estimation robust mean - robust cov: Tyler M-estimation, Hettmansperger-Randles (*)
  - corr to cov for stats.covariance, using a robust univariate scale estimate, e.g. Öllerer and  Croux 2015
  - outlier detection, the try_robust_cov_more_updated.ipynb notebook has experiments for outlier detection with references to several articles
- ...


(*) very recent article including joint estimation mean, cov
Taskinen, Sara, Gabriel Frahm, Klaus Nordhausen, and Hannu Oja. “A Review of Tyler’s Shape Matrix and Its Extensions.” In Robust and Multivariate Statistical Methods: Festschrift in Honor of David E. Tyler, edited by Mengxi Yi and Klaus Nordhausen, 23–41. Cham: Springer International Publishing, 2023. https://doi.org/10.1007/978-3-031-22687-8_2.

has also good list of references, some that I used, some more recent than when I worked on it initially



user: josef-pkt
issue_8128: state: open
title: ENH: let approx_fprime accept list
body: it looks like adding type promotion `np.promote_types(float, x.dtype))` broke  numdiff for lists 

(no explicit example because in an example a class called fprime with list params (only one value)


```
\statsmodels\statsmodels_py37_main\statsmodels\statsmodels\tools\numdiff.py in approx_fprime(x, f, epsilon, args, kwargs, centered)
    143     f0 = f(*((x,)+args), **kwargs)
    144     dim = np.atleast_1d(f0).shape  # it could be a scalar
--> 145     grad = np.zeros((n,) + dim, np.promote_types(float, x.dtype))
    146     ei = np.zeros((n,), float)
    147     if not centered:

AttributeError: 'list' object has no attribute 'dtype'
```
user: josef-pkt
issue_8126: state: open
title: ENH: sandbox: refactor GOF class and add exponential -  rebased
body: rebased version of  #308  

written in 2012 for python 2
user: josef-pkt
issue_8125: state: open
title: ENH: LTS, MTL least trimmed squares and ETS and MM-E - rebased
body: rebased version of #452  
(actually rebased from a branch that doesn't have a PR https://github.com/josef-pkt/statsmodels/commits/lts_rebased )

merge conflicts in RLM.fit start_params, weights, initial scale

originally written in 2012, rebased in 2014. 
has old pattern or code compatible with older python (print in example script files are python 2 statements)
user: josef-pkt
issue_8124: state: open
title: ENH: Segmented knots rebased
body: rebased version of #2677

removed last commit which was compat fixes for numpy < 1.9 (in percentile)

status not clear
see old comments https://github.com/statsmodels/statsmodels/pull/2677#issuecomment-226978367
predict and results info incomplete or missing

old notebook
https://github.com/josef-pkt/misc/blob/segmented/notebooks/ex_segmented_regression_sm.ipynb

segmented is in `base`
`get_objective` needs llf as alternative to ssr
then we could try a poisson or similar example

user: josef-pkt
issue_8123: state: open
title: Nested Random Effects Model using MixedLM?
body: #### Is your feature request related to a problem? Please describe
> A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Wondering if there's a way using the `smf.mixedlm` function to specify different random effect formula for 2 variables of interest. 

Desired model: 
`Y ~ time * protein + time + [multiple FE covariates here...] + (1 | site) + (1 + time | subjectID)`

In the current implementation, I believe only this version of a model is possible to define (no random slopes, just random intercepts): 
`Y ~ time * protein + time + [multiple FE covariates here...] + (1 | site) + (1 | subjectID)`

by defining the dictionary passed through `vc_formula` as the following 
```
vcf = {"site": "0 + C(site)", "IID": "0 + C(subjectID)"}
```

and `re_formula='0'`

#### Describe the solution you'd like
> A clear and concise description of what you want to happen.

Looking for a way to have different random effect formulas depending on the variable, so the model can be defined as desired above.

#### Describe alternatives you have considered
> A clear and concise description of any alternative solutions or features you have considered.

From my understanding, statsmodel implements patsy (and therefore must be maintaining some version of patsy for the mixedlm function). This issue was opened on patsy here, https://github.com/pydata/patsy/issues/130 though looks as though patsy is no longer maintained and a PR has not been opened to tackle this. Outside of Python, will be considering doing this in R if not possible.

#### Additional context
> Add any other context about the feature request here.

If this is already possible, I'd love to know! Feel free to point me in the right direction and close this FR - just running out of luck on Google :)
user: m-makarious
issue_8122: state: open
title: Fix error in AR(1) model bracket search
body: #### Is your feature request related to a problem? Please describe
Current bracket finding implementation often fails to find right bracket.
Just using the standard bracket finder in scipy.optimize seems like a good idea.

#### Describe the solution you'd like
Replace:
https://github.com/statsmodels/statsmodels/blob/main/statsmodels/genmod/cov_struct.py#L896-L917

With:
```python
from scipy.optimize import brent, bracket
b_lft,b_ctr,b_rgt, f_lft,f_ctr,f_rgt, _ = bracket(fitfunc,0,1)
self.dep_params = brent(fitfunc, brack=[b_lft,b_ctr,b_rgt])
```
#### Describe alternatives you have considered
-
#### Additional context
-
user: Parskatt
issue_8121: state: open
title: ENH: endogenous treatment effect
body: after #8034 adding equivalent to Stata's eteffects looks pretty straightforward.

Reading the formulas in Stata docs, it looks like it's close to regression adjustment `RA` plus adding the treatment model residuals as control function in the outcome equations.
(Stata shows only the linear control function, I guess we could use some polynomial basis functions as optional alternative)

It should still be under statsmodels.treatment because the focus is on estimating ATE and POM, or their effect on treated  versions.

control function with outcome regression equations as focus (analogue to IV2SLS) would be in `statsmodels.simultaneous`
user: josef-pkt
issue_8120: state: open
title: ENH: Dcm logit rebased2
body: rebased version of #1605
no additional changes

no merge conflicts, but likely outdated in inherited parts, no changes in PR since 2014

(I'm not planning to work on this soon, but want to have it closer to current code and wanted to see if there are serious merge conflicts)
user: josef-pkt
issue_8119: state: open
title: Using new distributions with GLM fit
body: #### Is your feature request related to a problem? Please describe
I would like to use ZeroInflatedPoission distribution for Poisson regression task.
Currently i do not see a possibility where 
`sm.GLM(data.endog, data.exog, family=sm.families.Poisson()),fit()` 
could be replaced with 
`sm.GLM(data.endog, data.exog, family=sm.families.ZeroInflatedPoisson()).fit()`. 

I havent seen any documentation on how to make this possible or if this actually is possible. Is that correct or am I overlooking something?

#### Describe the solution you'd like
I would like to use 
`sm.GLM(data.endog, data.exog, family=sm.families.ZeroInflatedPoisson()).fit()`. 


user: gsujan
issue_8117: state: open
title: STY: code cleanup discrete
body: One problem with code cleanup is that it can cause messy merge conflicts.
However, if there are no large pull requests, then we can clean up additional modules, expecially for whitespace and line length violations.

I think discrete is a candidate. I recently merged most of the larger PRs affecting existing code in discrete

currently we have 10 open PRs with label discrete
https://github.com/statsmodels/statsmodels/pulls?q=is%3Aopen+is%3Apr+label%3Acomp-discrete

one possible problem is that we also need cleanup refactoring to streamline duplicated code, which is not just cosmetic.
user: josef-pkt
issue_8114: state: open
title: ENH: treatment effect with covariates in randomized trials
body: followup to semi-parametric treatment effect under ignorability or conditional independence
#2443 SUMM issue
#8034 PR for teffects

Main difference: selection probability is given and orthogonal to outcome explanatory variables X, e.g. random sampling
maybe similar to survey weights.

parking a reference, there are many related articles that I have not looked at

method looks a bit similar to AIPW, correction term to predicted mean (POM) of a baseline regression model
(no ipw weighting in regression)

Zhang, Min, Anastasios A. Tsiatis, and Marie Davidian. "Improving efficiency of inferences in randomized clinical trials using auxiliary covariates." Biometrics 64, no. 3 (2008): 707-715.

I guess it will follow mostly the same pattern as in new `TreatmentEffect` but without momcond for selection/treatment model, probs takes as given and exogenous.



user: josef-pkt
issue_8106: state: open
title: Change the default value of the `method` argument in the PACF 
body: #### Is your feature request related to a problem? Please describe
The default value for the `method` argument of PACF is `ywunbiased` which is the same as `ywadjusted`. In the [documentation notes](https://www.statsmodels.org/dev/generated/statsmodels.tsa.stattools.pacf.html) itself it states that: 

> Yule-Walker (adjusted) and Levinson-Durbin (adjusted) performed consistently worse than the other options.

#### Describe the solution you'd like
Change the default value of the `method` argument to one of the better performing solutions (e.g., `ywmle`) as described in the documentation.

#### Describe alternatives you have considered
None.

#### Additional context
The PACF with `ywunbiased` often results in partial autocorrelations greater than 1 for larger lags. This also occurs for `ldadjusted`.
user: KishManani
issue_8105: state: open
title: Predictions are going into the direction of 0
body: Dear Chad, 

Thank you very much that you have found time to answer my question regarding the UC model and the start values. 
You are right about the level component. It is the same as y. 
My follow up question is about the exogenous variables. 

When I put zeros as dummy variables the predictions are going to the direction of 0 as well ..... 
Can you tell me what I need to do here exactly? I am really not the best coder around. I have no idea what could be a workaround for this .... Because of course I do not have the moving averages for the future available. 
```
# it is possible to add zeros as exogenous variables but the prediction moves to 0 like this
# it is visible in the plt.plot(outlook[20:])

d = pd.DataFrame(0, index=np.arange(21), columns=df.columns)
f = d[['EMA_5','EMA_15','EMA_60','EMA_90', 'EMA_150', 'EMA_200']]

df2 = pd.concat([df,f])

start = 0
end = len(df) +20
outlook = (res.predict(start = start, end = end, exog = f))

plt.plot(outlook[-20:])
```
![grafik](https://user-images.githubusercontent.com/42875245/152891150-0e337eb1-4e07-4a00-aa60-e4f17f8d80bd.png)
```
plt.plot(outlook[20:])
```
![grafik](https://user-images.githubusercontent.com/42875245/152891120-33cf11b5-18b0-44a7-8c82-bf9045639eeb.png)

Thank you for your response in advance! statsmodels is a great program!
user: waudinio27
issue_8098: state: open
title: ENH: add option for continuity correction in power and sample size for proportions_2indep 
body: see
https://github.com/statsmodels/statsmodels/issues/8049#issuecomment-1028127490
user: josef-pkt
issue_8091: state: open
title: BUG/EHN TypeError when fitting ARIMA with 'innovations_mle' and specifying start_params
body: #### Describe the bug

An error is raised when the method is set to "innovations_mle" and ``start_params`` is not set to ``None`` in the``.fit`` method of ARIMA objects. Thus, we cannot estimate an ARIMA with from arbitrary initialization when demanding fit to use innovations_mle procedure.

```python
import numpy as np
from statsmodels.tsa.arima_process import arma_generate_sample
from statsmodels.tsa.arima.model import ARIMA

y = arma_generate_sample(ar=np.array([0.75]), ma=np.array([0.2]), nsample=100)

ar_1 = ARIMA(endog=y, order=(1,0,0), trend='n')
ar_1.fit(method='innovations_mle', start_params=np.array([0.5]))

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_2368\2835648636.py in <module>
      8         # enforce_invertibility=True)
      9 
---> 10 ar_2.fit(method='innovations_mle', start_params=np.array([0.80]))
     11 

~\miniconda3\envs\ts_forecast\lib\site-packages\statsmodels\tsa\arima\model.py in fit(self, start_params, transformed, includes_fixed, method, method_kwargs, gls, gls_kwargs, cov_type, cov_kwds, return_params, low_memory)
    390                     endog, order=order,
    391                     seasonal_order=seasonal_order,
--> 392                     demean=False, **method_kwargs)
    393 
    394         # In all cases except method='statespace', we now need to extract the

TypeError: innovations_mle() got an unexpected keyword argument 'transformed'
```

I-m using the stable version but I did not see any difference between develop et stable in tsa/arim/estimators/innovations.py nor tsa/arim/model.py files, and so the issue is still likely on develop.

#### Expected Output

The parameters ``transformed`` should not be in the ``**method_kwargs`` dict since such a dict is then passed to a scipy.optimize.minimize function. I had a look in the test files and did not see any test on innovations_mle with ``start_params`` set to its non-default value.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

INSTALLED VERSIONS
------------------
Python: 3.7.12.final.0

statsmodels
===========

Installed: 0.13.1 (C:\Users\Etudiant\miniconda3\envs\statmod\lib\site-packages\statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.21.5 (C:\Users\Etudiant\miniconda3\envs\statmod\lib\site-packages\numpy)
scipy: 1.7.3 (C:\Users\Etudiant\miniconda3\envs\statmod\lib\site-packages\scipy)
pandas: 1.3.5 (C:\Users\Etudiant\miniconda3\envs\statmod\lib\site-packages\pandas)
    dateutil: 2.8.2 (C:\Users\Etudiant\miniconda3\envs\statmod\lib\site-packages\dateutil)
patsy: 0.5.2 (C:\Users\Etudiant\miniconda3\envs\statmod\lib\site-packages\patsy)

Optional Dependencies
=====================

matplotlib: 3.5.1 (C:\Users\Etudiant\miniconda3\envs\statmod\lib\site-packages\matplotlib)
    backend: QtAgg
cvxopt: Not installed
joblib: Not installed

Developer Tools
================

IPython: Not installed
    jinja2: Not installed
sphinx: Not installed
    pygments: Not installed
pytest: Not installed
virtualenv: Not installed
</details>

user: Clej
issue_8051: state: open
title: BUG: `MixedLMResult.summary` does not honor the `title` parameter
body: The argument to the parameter `title` of `MixedLMResult.summary()` is completely ignored.

```python
import statsmodels.api as sm
import statsmodels.formula.api as smf

df = sm.datasets.get_rdataset("dietox", "geepack").data
md = md = smf.mixedlm("Weight ~ Time", df, groups='Pig')
mdf = md.fit()
print(mdf.summary(title='ABCDEF'))
```

Output:

```
         Mixed Linear Model Regression Results
========================================================
Model:            MixedLM Dependent Variable: Weight    
No. Observations: 861     Method:             REML      
No. Groups:       72      Scale:              11.3669   
Min. group size:  11      Log-Likelihood:     -2404.7753
Max. group size:  12      Converged:          Yes       
Mean group size:  12.0                                  
--------------------------------------------------------
             Coef.  Std.Err.    z    P>|z| [0.025 0.975]
--------------------------------------------------------
Intercept    15.724    0.788  19.952 0.000 14.179 17.268
Time          6.943    0.033 207.939 0.000  6.877  7.008
Pig Var      40.394    2.149                            
========================================================
```

Should be instead:

```
                          ABCDEF
========================================================
Model:            MixedLM Dependent Variable: Weight    
No. Observations: 861     Method:             REML      
No. Groups:       72      Scale:              11.3669   
Min. group size:  11      Log-Likelihood:     -2404.7753
Max. group size:  12      Converged:          Yes       
Mean group size:  12.0                                  
--------------------------------------------------------
             Coef.  Std.Err.    z    P>|z| [0.025 0.975]
--------------------------------------------------------
Intercept    15.724    0.788  19.952 0.000 14.179 17.268
Time          6.943    0.033 207.939 0.000  6.877  7.008
Pig Var      40.394    2.149                            
========================================================
```

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

INSTALLED VERSIONS
------------------
Python: 3.10.2.final.0
OS: Darwin 21.3.0 Darwin Kernel Version 21.3.0: Wed Jan  5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_ARM64_T6000 arm64
byteorder: little
LC_ALL: None
LANG: it_IT.UTF-8

statsmodels
===========

Installed: 0.13.1 (/Users/giacomo/Documents/Programmi/python3102venv/lib/python3.10/site-packages/statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.21.5 (/Users/giacomo/Documents/Programmi/python3102venv/lib/python3.10/site-packages/numpy)
scipy: 1.7.3 (/Users/giacomo/Documents/Programmi/python3102venv/lib/python3.10/site-packages/scipy)
pandas: 1.4.0 (/Users/giacomo/Documents/Programmi/python3102venv/lib/python3.10/site-packages/pandas)
    dateutil: 2.8.2 (/Users/giacomo/Documents/Programmi/python3102venv/lib/python3.10/site-packages/dateutil)
patsy: 0.5.2 (/Users/giacomo/Documents/Programmi/python3102venv/lib/python3.10/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.5.1 (/Users/giacomo/Documents/Programmi/python3102venv/lib/python3.10/site-packages/matplotlib)
    backend: MacOSX 
cvxopt: Not installed
joblib: 1.1.0 (/Users/giacomo/Documents/Programmi/python3102venv/lib/python3.10/site-packages/joblib)

Developer Tools
================

IPython: 8.0.1 (/Users/giacomo/Documents/Programmi/python3102venv/lib/python3.10/site-packages/IPython)
    jinja2: 3.0.3 (/Users/giacomo/Documents/Programmi/python3102venv/lib/python3.10/site-packages/jinja2)
sphinx: Not installed
    pygments: 2.11.2 (/Users/giacomo/Documents/Programmi/python3102venv/lib/python3.10/site-packages/pygments)
pytest: Not installed
virtualenv: Not installed

</details>

user: Gattocrucco
issue_8040: state: open
title: OrderedResults.save: pickling not supported
body: #### Describe the bug

I'm running OrderedModel on data from the olympics, where the medal categories is the ordinal dependant variable and categories such as age, weight, and country are the independent variables. When I put the model and fit results into an OrderedResult class, the OrderedResults will not save and states that pickling isn't supported.

The exact error that occurs is "NotImplementedError: Sorry, pickling not yet supported. See https://github.com/pydata/patsy/issues/26 if you want to help."

So is pickling actually supported for the ordinal model or not? Statsmodels is really the only package that can do this for python.

#### Code Sample, a copy-pastable example if possible

import numpy as np
import pandas as pd
import scipy.stats as stats
from pandas.api.types import CategoricalDtype
from statsmodels.miscmodels.ordinal_model import OrderedModel, OrderedResults

summer_model = OrderedModel.from_formula("medals ~ age + height + weight + C(event_category) + C(sex) + C(region)", summer_df,
                                      distr='logit')
summer_fit = summer_model.fit(method='bfgs', maxiter=1000)

my_result = OrderedResults(summer_model, summer_fit._results)

my_result.save("summer_fit")


```python
# Your code here that produces the bug
# This example should be self-contained, and so not rely on external data.
# It should run in a fresh ipython session, and so include all relevant imports.
```
<details>

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `main`. If your problem has been fixed in an unreleased version, you might be able to use `main` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the main branch of this repository? It helps the limited resources if we know problems exist in the current main branch so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

The expected output is the model is pickled and the model persists and can be loaded elsewhere for use.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]


INSTALLED VERSIONS
------------------
Python: 3.7.11.final.0
OS: Darwin 19.6.0 Darwin Kernel Version 19.6.0: Tue Oct 12 18:34:05 PDT 2021; root:xnu-6153.141.43~1/RELEASE_X86_64 x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

statsmodels
===========

Installed: 0.13.1 (/Users/alissa/opt/anaconda3/envs/mlenv/lib/python3.7/site-packages/statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.21.2 (/Users/alissa/opt/anaconda3/envs/mlenv/lib/python3.7/site-packages/numpy)
scipy: 1.7.3 (/Users/alissa/opt/anaconda3/envs/mlenv/lib/python3.7/site-packages/scipy)
pandas: 1.3.5 (/Users/alissa/opt/anaconda3/envs/mlenv/lib/python3.7/site-packages/pandas)
    dateutil: 2.8.2 (/Users/alissa/opt/anaconda3/envs/mlenv/lib/python3.7/site-packages/dateutil)
patsy: 0.5.2 (/Users/alissa/opt/anaconda3/envs/mlenv/lib/python3.7/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.5.0 (/Users/alissa/opt/anaconda3/envs/mlenv/lib/python3.7/site-packages/matplotlib)
    backend: module://matplotlib_inline.backend_inline 
cvxopt: Not installed
joblib: 1.1.0 (/Users/alissa/opt/anaconda3/envs/mlenv/lib/python3.7/site-packages/joblib)

Developer Tools
================

IPython: 7.29.0 (/Users/alissa/opt/anaconda3/envs/mlenv/lib/python3.7/site-packages/IPython)
    jinja2: 3.0.2 (/Users/alissa/opt/anaconda3/envs/mlenv/lib/python3.7/site-packages/jinja2)
sphinx: Not installed
    pygments: 2.10.0 (/Users/alissa/opt/anaconda3/envs/mlenv/lib/python3.7/site-packages/pygments)
pytest: Not installed
virtualenv: Not installed

</details>

user: roeggealissa
issue_8039: state: open
title: ENH: get_prediction inference for in-sample statistics, correlation to score_obs, ATE
body: I mentioned this in some issue, but don't find it. 


get_prediction inference takes uncertainty from estimated parameters into account, but does not take into account that the predicted statistic can be in-sample. 
With in-sample prediction the predicted values might be correlated with the parameter estimates, i.e. correlated with score_obs.
I'm not sure yet in which cases we have problems.

ATE, and chisquare gof-test for zero-inflation or predicted probabilities takes correlation of the predicted or tested target statistic with score_obs into account.

We have this explicitly in the count diagnostic chisquare test using OPG.
ATE, treatment effects inference uses GMM directly to combine and concatenate moment conditions for different models and target statistics. #8034

My guess is that we can use something generic like GMM to include "in-sample" predicted statistics.
i.e. make something like the pattern in treatment effects #8034 into a generic tool for diagnostic and prediction.
e.g. get_prediction standard errors for excess zeros will ignore the in-sample effect, but we could add inference and predicted effects based on GMM and conditional moment tests.

(That's still vague)




user: josef-pkt
issue_8032: state: open
title: ENH: heckman endogenous sample selection with non-gaussian errors
body: Heckman model is not robust to misspecification of the distribution.
One alternative are more non- or semi-parametric methods that I never looked at.
Another alternative is to provide an collection of heckman type models for other distributions that might fit the data better.
elliptical distributions and copula distributions should be good candidates.

“A Heckman Selection-t Model.” might be a good starting point

list of references copied from https://github.com/fsbmat-ufv/ssmodels (no CRAN package for it)

Bastos, Fernando de Souza, and Wagner Barreto-Souza. 2020. “Birnbaum–Saunders Sample Selection Model.” Journal of Applied Statistics.

Bastos, Fernando de Souza, Wagner Barreto-Souza, and Marc G Genton. 2021. “A Generalized Heckman Model with Varying Sample Selection Bias and Dispersion Parameters.” Statistica Sinica.

Heckman, James J. 1976. “The Common Structure of Statistical Models of Truncation, Sample Selection and Limited Dependent Variables and a Simple Estimator for Such Models.” In Annals of Economic and Social Measurement, Volume 5, Number 4, 475–92. NBER.

———. 1979. “Sample Selection Bias as a Specification Error.” Econometrica: Journal of the Econometric Society, 153–61.

Marchenko, Yulia V, and Marc G Genton. 2012. “A Heckman Selection-t Model.” Journal of the American Statistical Association 107 (497): 304–17.

Ogundimu, Emmanuel O, and Jane L Hutton. 2016. “A Sample Selection Model with Skew-Normal Distribution.” Scandinavian Journal of Statistics 43 (1): 172–90.

user: josef-pkt
issue_8029: state: open
title: GAM: help matching to R's mgcv package (interactions, random effects, categorical variables, etc)
body: Hello

I'm trying to replicate a GAM model created in `R` using the `mgcv` package. I am new to GAMs, and started experimenting with `statsmodels` package only yesterday. **I have some questions further down.**

But first, some context.

I have tried, but still have questions if I'm matching everything correctly. For one `s()` and `te()` terms are used, as well as `bs='cr'` and `bs='re'`: bs is the penalized smoothing basis, `'cr'` means cubic regression spline, penalized by conventional integrated square second derivative cubic spline penalty and `'re'` means random effects, penalized by a ridge penalty.

Some terms did not have the number of splines defined, were computed them internally. Bu they are all related to the categorical variable which has 32 unique categories.

The train data looks somewhat like:

<html>
<body>
<!--StartFragment--><b style="font-weight:normal;" id="docs-internal-guid-150a2446-7fff-3e60-8951-5b17f69c539b"><div dir="ltr" style="margin-left:0pt;" align="left">

Y | A | B | C | D
-- | -- | -- | -- | --
1 | 5 | 0.2536 | 0 | “ABCD”
0 | 32 | -1.1789 | 4.1298 | “EFGH”
0 | 55 | 9.8792 | 0 | “IJKL”
1 | 13 | -7.6432 | -0.8477 | “ABCD”

</div><br /></b><!--EndFragment-->
</body>
</html>

So, the code in `R` looks somewhat like:

```
model <- bam(Y ~
           s(A, k=5, bs='cr') +
           s(D, bs='re') +
           ti(A, D, bs='re') +
           ti(B, D, bs='re') +
           s(B, k=4, bs='cr') +
           s(C, k=3, bs='cr'),
           data=train_set
           family=binomial(link="logit"),
           method="REML",
           na.action= na.omit,
           control = gam.control(mgcv.tol = 1e-2,
                                 maxit = 50,
                                 irls.reg = 0,
                                 trace = TRUE))
```

And these are the resulting coefficients:

<html>
<body>
<!--StartFragment--><b style="font-weight:normal;" id="docs-internal-guid-d68e9b00-7fff-5adf-e23c-830c05fb703b"><div dir="ltr" style="margin-left:0pt;" align="left">

Intercept |
-- |
s(A).1 |
s(A).2 |
s(A).3 |
s(A).4 |
s(D).1 |
… |
s(D).32 |
ti(A, D).1 |
… |
ti(A, D).31 |
ti(B, D).1 |
… |
ti(B, D).31 |
s(B).1 |
s(B).2 |
s(B).3 |
s(C).1 |
s(C).2 |

</div></b><!--EndFragment-->
</body>
</html>

So, 4 A coefficients, 32 D coefficients, 31 coefficients for the interaction of A and D, 31 coefficients for the interaction of B and D, 3 B coefficients and 2 C coefficients.

Now, my first draft using `statsmodel` looks like:

```
x_spline = train_set[['A', 'B', 'C']]
bs = BSplines(x_spline, df=[5, 4, 3], degree=[3, 3, 2])

alpha=np.array([0.1, 0.1, 0.1])
gam_bs = GLMGAM.from_formula('Y ~ C(D) + A:C(D) + B:C(D)',
                 data= train_set,
                 smoother=bs, alpha=alpha,
                 family=sm.families.Binomial())

res_bs = gam_bs.fit(tol=1e-2, maxiter=50)
print(res_bs.summary())
```

I followed this [doc ](https://www.statsmodels.org/stable/gam.html) and got some hints for other things from issues here on github.

**1. Regarding the parameters besides the formula, how can I set the method of smoothing parameter estimation to `REML` like in `R`? And how can I set `irls.reg = 0` in `statsmodel`?**

I set the family to `sm.families.Binomial` that by default also takes the logit link, to match the code in `R`. The same also for `tol` and `maxiter`.

**2. Do my Bsplines make sense?** 

Since all `s()` terms, except `s(D)`, have `bs='cr'`, I decided to used BSplines for them. I think it makes sense since 

> B-spline basis is a commonly used spline basis that is based on a special parametrization of a cubic spline

**3. Do I need to set the `degree `to 3 for the BSpline variables, to ensure they are a cubic spline? If not, what value should `degree `have for each variable?**

**4. If I run  `select_penweight()` I will get the same alphas for my model variables as the ones mgcv does?**

I set all alphas to 0.1, because I saw in the doc that after the model fit you ran either `select_penweight()` or `select_penweight_kfold()`, to discover the best alphas. I noticed, in the example, the returned values from `select_penweight()` matched the ones you defined in the alpha variable and I read in a notebook by @josef-pkt that the values specified there were values from the unit tests obtained by the mgcv package.

**5. In my code we have `s(D, bs='re')`, does it make sense in `statsmodel` to match it by adding to the formula `C(D)`? Where can I find the documentation on this `C()`, and what other options are available? If it is penalized by random effects in the original code, `bs='re'` does it still make sense to define it here? If I define it in `x_spline`, how can I set the penalized smoothing basis to be random effects?**

I created my model by using the `from_formula()` GLMGam class method, like in the example in the doc. I read the formula interface only supports the linear part. So I added my variable D, which is categorical, and I saw in some examples here on github the use of `C()` and I thought it made sense using it. Can you confirm?

**6. I have two interactions, in the original code they were `ti(A, D, bs='re')` and `ti(B, D, bs='re')`. I matched in `statsmodel` by adding to the formula A:C(D) + B:C(D). Does it make sense?** 

I think it is not because I get one extra coefficient than I did in R for both interactions (other variables are matching in terms of number of coefficients).

See below for coefficients in `statsmodel`:

<html>
<body>
<!--StartFragment--><b style="font-weight:normal;" id="docs-internal-guid-d68e9b00-7fff-5adf-e23c-830c05fb703b"><div dir="ltr" style="margin-left:0pt;" align="left">

Intercept |
-- |
C(D).[T.1.0] |
... |
C(D).[T.31.0] |
A:C(D)[0.0] |
... |
A:C(D)[0.31] |
B:C(D)[0.0] |
... |
B:C(D)[0.31] |
A_s0 |
A_s1 |
A_s2 |
A_s3 |
B_s0 |
B_s1 |
B_s2 |
C_s0 |
C_s1 |

</div></b><!--EndFragment-->
</body>
</html>

So, 4 A coefficients, 32 D coefficients, **32** coefficients for the interaction of A and D, **32** coefficients for the interaction of B and D, 3 B coefficients and 2 C coefficients.

So I must be doing something wrong to have an extra coefficient for both interactions.

**7. How do I make predictions using the result `statsmodel` GAM model for new data?**

I tried following the example in the issue #7830 and did (doing predict for the train_set just to test the functionality):

`predictions= res_bs.predict(train_set, exog_smooth=train_set[['age', 'norm_gt', 'norm_incident_freq']]`






user: CatarinaPC
issue_8025: state: open
title: BUG: error in `summary` of SVAR results
body: #### Describe the bug

[Hi, I am estimating the structural VAR model, I can fit the model, but there is an error when I want to check the summary of the SVAR model. The error message is: AttributeError: 'SVARResults' object has no attribute 'k_exog_user']

#### Code (python)
Bm = np.array([[1, 'E', 'E'], [0, 1, 'E'],[0, 0, 1]])
model_svar = SVAR(test_data, svar_type='B', B=Bm)
svar_results = model_svar.fit(maxlags=6, maxiter=1000, maxfun=1000, solver='newton')

All above code works, but the following summary reports the error

```python
svar_results.summary()
AttributeError: 'SVARResults' object has no attribute 'k_exog_user'
```
<details>
AttributeError: 'SVARResults' object has no attribute 'k_exog_user'

/.../python/lib/python3.8/site-packages/statsmodels/tsa/vector_ar/var_model.py in summary(self)
   1707         summary : VARSummary
   1708         """
-> 1709         return VARSummary(self)
   1710 
   1711     def irf(self, periods=10, var_decomp=None, var_order=None):

/.../python/lib/python3.8/site-packages/statsmodels/tsa/vector_ar/output.py in __init__(self, estimator)
     69     def __init__(self, estimator):
     70         self.model = estimator
---> 71         self.summary = self.make()
     72 
     73     def __repr__(self):

/.../python/lib/python3.8/site-packages/statsmodels/tsa/vector_ar/output.py in make(self, endog_names, exog_names)
     82         buf.write(self._header_table() + '\n')
     83         buf.write(self._stats_table() + '\n')
---> 84         buf.write(self._coef_table() + '\n')
     85         buf.write(self._resid_info() + '\n')
     86 

/.../python/lib/python3.8/site-packages/statsmodels/tsa/vector_ar/output.py in _coef_table(self)
    153 
    154         buf = StringIO()
--> 155         dim = k * model.k_ar + model.k_trend + model.k_exog_user
    156         for i in range(k):
    157             section = "Results for equation %s" % model.names[i]

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

A clear and concise description of what you expected to happen.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]

</details>

user: StephenZhangAB
issue_8022: state: open
title: Theta forecasting model  gives forecasts  around 100 even if test data is in range of -5% to 5 %
user: AspirnQnt
issue_8021: state: open
title: ENH: enhancements to hurdle count model, GLM or binary models as zero model
body: We are missing Logit, Probit or GLM Binomial as option to use as zero model in the hurdle model.

Main task will be to get the compatible interface, i.e. extra methods or information that are needed by the hurdle model.

In both Hurdle and zero-inflated model it would be useful to have a more generic option than just hardcoded Logit and Probit.
GLM-Binomial has more link options, but is still limited in not allowing parameterized links, where link parameters are also estimated.
e.g. #624


user: josef-pkt
issue_8020: state: open
title: ENH: hurdle count models - joint estimation and common parameters
body: current version of hurdle model in #7973 use separate estimation.

R pscl has option for joint estimation

One use case for joint estimation will be when we want to impose common parameters across submodels (when we use the same family in both).
example in negbin-negbin, we could impose common dispersion

related usecase would be penalized estimation where we penalize towards identical parameters in both models.

We also still need the pieces for joint estimation like score_obs, hessian for robust cov_types. Those are also not available in the PR version.

user: josef-pkt
issue_8019: state: open
title: ENH: add zero-modified distribution
body: We don't have a hurdle distribution yet.

Instead we can add a zero-modified distribution with dargs (prob-zero, count-dargs), 
for given count distribution like poisson, negbin or gpp.

pmf for y>0 is the probability corrected pmf of the count distribution
(in pscl this is the prob-zero ratio returned by predict)

The simplification to hurdle model is that the zero-model is just Bernoulli. 

rvs that I'm currently using https://github.com/statsmodels/statsmodels/issues/7162#issuecomment-1013758309

I guess zero-inflated could be converted to this, but zero inflated distribution is a bit simpler, because it does not require truncation of the count distribution

I guess the same pattern should work for continuous distribution with mass point at zero, except the continuous distribution has Prob(y=0) = 0 (no mass point)


user: josef-pkt
issue_8018: state: open
title: ENH/Design: diagnostics based on linear predictor or tranformed distribution
body: Just an idea: For some models with might want to have diagnostics based on linear predictor or transformed distribution.

example:
accelerated failure time model and log-transformed distribution
Compute diagnostic for the normal model instead the log-normal model.
The log-normal model does not have a mean parameterization and it will be easier to use the diagnostic of the log transformed model, e.g. the normal model instead directly in the log-normal model.

Idea triggered by Inheritance of generically implemented features is great.
e.g. resid_pearson requires resid_response and predict "var". https://github.com/statsmodels/statsmodels/pull/7973#issuecomment-1013325446
We will get this only with difficulty in some models like log-normal and other variance/precision parameterized models like AFT.

Using diagnostics for linear predictor requires that we have an observable statistic to compare it with, eg. working residuals in GLM. (We don't do much with those currently,)
But traditional GLM influence-outlier analysis and diagnostic is mostly based on WLS/IRLS 

Even if we use the transformed model for diagnostics, we still want the main model for the application specific parts, e.g. survival probabilities and quantiles.
And we can still do diagnostics other than residual diagnostics for those, e.g. quantile, gof based diagnostics.

unlikely for 0.14, unless we have diagnostics that we can just wire into those models. eg. GaussianHet diagnostics for log-normal.



user: josef-pkt
issue_8017: state: open
title: ENH: More control over X13 specs
body: #### Is your feature request related to a problem? Please describe

Currently the x13 module only exports 2 functions. While both functions support some basic options with sensible defaults, a lot of the functionality of the x13 module is inaccessible through the python interface. For instance x13_arima_analysis allows you to turn outlier detection on and off with a boolean but it relies exclusively on x13's default options for outlier detection with no ability pass in custom options to x13.

#### Describe the solution you'd like

Currently x13 has a class SeriesSpec but it is partially documented and not exposed via __all__. SeriesSpec's documentation should be fleshed out and included in __all__. Additionally classes to represent the other specs in the x13 documentation should also be developed, documented, and included in __all__.

Additionally, some functions could be refactored and included in __all__ to facilitate running x13 from a group of spec objects and marshalling the results back into pandas series.

#### Describe alternatives you have considered

Developers could handle creating custom spec files and invoking x13 without using statsmodels.

#### Additional context

[The reference manual for X-13ARIMA-SEATS](https://www2.census.gov/software/x-13arima-seats/x-13-data/documentation/docx13as.pdf)
user: CCHiggins
issue_8016: state: open
title: ENH/REF/TST  get_prediction summary_frame labels in pandas index
body: check pandas `index` in get_prediction and improve it if necessary

(separate issue because I'm spending most of my time verifying numbers but not pandas integration, and there are no unit test for index and column names of returned dataframes.)

Similarly, dataframe index for constraints can likely be improved with user options, currently it's just c0, c1, ... or similar default.
I'm not sure what options we already have there.

possible: add a column with information about rows that the user could convert to index

partially related:
#6633  make pandas dataframe returns more informative about metainformation used in the construction of the returned `frame`
user: josef-pkt
issue_7987: state: open
title: ENH: positive normal distribution, normal distribution left truncated at zero
body: (just a semi-random idea)

for large counts normal distribution might be a good approximation

I have DiscretizedCount distribution that largely hardcodes for distributions with support on R+ or R++.
We could use normal distribution left truncated at zero as base distribution.

Main issue is choosing parameterization for regression case,
scipy truncnorm is loc-scale distribution (and handles now too many extreme cases)
Can we have a mean parameterization?
https://en.wikipedia.org/wiki/Truncated_normal_distribution#One_sided_truncation_(of_lower_tail)[4]

Or is it better to use mu (mean of underlying normal distribution)?

google finds many references, but I didn't look closely
user: josef-pkt
issue_7984: state: open
title: R-squarred for MixedLMResults or MixedLM
body: Hello all,

For a project of mine, I am using the statsmodel MixedLM function to compute linear mixed model. In order to present the results, my supervisors are very attached to report r² values of the model. In R, this is implemented with the [MuMIn ](https://cran.r-project.org/web/packages/MuMIn/MuMIn.pdf) function r.squaredGLMM. 

I was wondering whether it would be possible to implement a method to do so?

I would have been happy to implement it myself in my code by extracting the relevant variance measures from the models, but I am not proficient enough in linear mixed to confidently compute r² from the different aspects of the model. If anyone has a quick and easy way to do so without having to modify the package itself, that would be awesome!

Kind regards,

Alex
user: AlexLepauvre
issue_7982: state: open
title: ENH : with calibrated tsa markov regression, get an out-of-sample predict() 
body: #### Is your feature request related to a problem? Please describe
after a model.fit() , I'd like to have a out of sample predict function, that would use the current smoothed_marginal_probabilities, wat we can consider as "calibrated" model 


#### Describe the solution you'd like

As with the current res.predict() that is in sample, I'd like to have it woking in a given future time serie 

#### Describe alternatives you have considered
I tried to fit the model on dummy variables, but it implies giving false transition probabilities to the model.. 


#### Additional context

other libs are done in a fork from scikit learn , but they are not relevant to time series ,nor with such precision as the statsmodel version 

user: CNiemans
issue_7980: state: open
title: MixedLM.fit(do_cg=False) always hits NameError
body: #### Describe the bug

In the **fit** method for **MixedLM** class (part of _mixed_linear_model.py_), there is an option to skip optimization by setting `do_cg=False`. In the method definition there is a variable `rslt` that is first defined in `if do_cg:` block. This means that `rslt` will never be defined if `do_cg=False` leading to the python `NameError`. Given that `do_cg` is an option it seems that this is not the intended behavior. At least there should be better error handling about the issue.

_mixed_linear_model.py_:
```
...
2082        def fit(self, ..., do_cg=True, ...)
...
2180            if do_cg:
...
2191                rslt = super(MixedLM, self).fit(...)
...
2213            converged = rslt.mle_retvals['converged']
...
```




user: mosqueteiro
issue_7978: state: open
title: MixedLM cannot handle `missing`
body: #### Describe the bug

Attempting to use `missing='drop'` with MixedLM results in an IndexError because the class does not properly drop missing rows from every input. Oddly, `smf.mixedlm` does handle missing data properly thanks to #3591.

#### Code Sample, a copy-pastable example if possible

```python
# y is a pandas Series with some missing values
m = sm.MixedLM(y, X, groups=groups, missing='drop') # IndexError
m = smf.mixedlm('y ~ x', X.join(y), groups=groups, missing='drop') # this works
```

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

INSTALLED VERSIONS
------------------
Python: 3.9.9.final.0
OS: Linux 5.10.85-1-lts #1 SMP Tue, 14 Dec 2021 18:50:52 +0000 x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

statsmodels
===========

Installed: 0.13.1 (/opt/mambaforge/envs/GeneStruct/lib/python3.9/site-packages/statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.21.4 (/opt/mambaforge/envs/GeneStruct/lib/python3.9/site-packages/numpy)
scipy: 1.7.3 (/opt/mambaforge/envs/GeneStruct/lib/python3.9/site-packages/scipy)
pandas: 1.3.5 (/opt/mambaforge/envs/GeneStruct/lib/python3.9/site-packages/pandas)
    dateutil: 2.8.2 (/opt/mambaforge/envs/GeneStruct/lib/python3.9/site-packages/dateutil)
patsy: 0.5.2 (/opt/mambaforge/envs/GeneStruct/lib/python3.9/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.5.1 (/opt/mambaforge/envs/GeneStruct/lib/python3.9/site-packages/matplotlib)
    backend: module://matplotlib_inline.backend_inline 
cvxopt: Not installed
joblib: 1.1.0 (/opt/mambaforge/envs/GeneStruct/lib/python3.9/site-packages/joblib)

Developer Tools
================

IPython: 7.30.1 (/opt/mambaforge/envs/GeneStruct/lib/python3.9/site-packages/IPython)
    jinja2: 3.0.3 (/opt/mambaforge/envs/GeneStruct/lib/python3.9/site-packages/jinja2)
sphinx: Not installed
    pygments: 2.10.0 (/opt/mambaforge/envs/GeneStruct/lib/python3.9/site-packages/pygments)
pytest: Not installed
virtualenv: Not installed

</details>

user: kimsin98
issue_7977: state: open
title: ENH: generalized poisson hurdle model estimated with GMM
body: (mainly parking an article)


Zuo, Guoxin, Kang Fu, Xianhua Dai, and Liwei Zhang. 2021. “Generalized Poisson Hurdle Model for Count Data and Its Application in Ear Disease.” Entropy 23 (9): 1206. https://doi.org/10.3390/e23091206.

uses E(y) and E(y**2) as moment conditions

looks interesting for (based on brief partial, skimming of article)

- hurdle model for GPP which is not included in #7973
- application of GMM to count models based on (literal) moment conditions and not on using score_obs as moment condition.

looks a bit similar to additive GMMPoisson when instruments = exog  (based on what I remember of that) #1742

(It's not a prime target to implement as GMM model, but useful as reference. I don't know if IVGPP makes sense as additive error model for endogeneity or unobserved heteroscedasticity.)



user: josef-pkt
issue_7976: state: open
title: ENH: add GLM as option to zero-inflated, also hurdle when available
body: #7132 no inflation with linear/identity link

The main advantage of using GLM for the inflation model is that we have a larger choice of link functions, instead of just logit and probit.

specific issue here:
If we add GLM as submodel in zero-inflated or similar, then we need consistent post-estimation interface in GLM.

I have not tried yet, but at least `predict(which=...)`, still needs to be changed in GLM to be compatible with discrete pattern.

derivatives might not have consistent api either
hessian_factor in GLM still has the "wrong" sign.



user: josef-pkt
issue_7975: state: open
title: FAQ-D/ENH: formulas, tools - moments for truncated distribution
body: We need conditional moments like `E(y | y >= a)`  and `V(y | y >= a) `.

I don't have an overview of how to compute them (except by brute force) and which distributions have simple, analytical formulas for it.
References for it are all over the place. 

Discrete case: we need them for count models #7973
More general usecase is also mean residual life in duration, lifetime models, and similar in extreme value models and risk analysis.

special case zero truncation is partially available in ZeroInflated model.

Brute force: 
scipy distributions have an `expect` method with `conditional` option (I had added the method for these kind of usecases) but it requires explicit integration or (numerically truncated) sums, and cannot be vectorized.

TODO: find and add references






user: josef-pkt
issue_7974: state: open
title: ENH/Design  truncated sample versus truncated distribution
body: Just quickly parking a thought, upcoming design decision.
while looking at truncated countmodels #7973

example: What should `predict(which="mean")` return ?
mean of truncated distribution, e.g E(y | y >=k), or
mean of underlying distribution E(y)

If we just have a truncated sample (e.g. minimum detection limit), then we are still interested in the underlying distribution.
(e.g. in survival models)

If we want to model a truncated distribution, then we just use properties of that distribution (truncation is a detail).

This does not affect estimation which has to take truncation, censoring, ... into account.

But it affects, predict which, and get_distribution and other results reporting





user: josef-pkt
issue_7971: state: open
title: ENH generic category: predict analogue that takes also endog
body: I'm thinking of a more general method to compute statistics, either in or out of sample, that depend on endog also.
Sinilar to predict but with additional endog as argument.

Insample this could be a general method with "which" similar to the attributes, like resid that we have.
Out of sample it would be evaluation measures, e.g. evaluate new observations for outlyingness.

related, special cases
#7947 out of sample loglike, eval_measures
#7919 get_resid

The case I'm currently thinking of is `which="cdf"` but evaluated for given endog, while `which="prob"` in predict computes for a set of endog independent of observation.
use cases for in sample diagnostic, and out of sample performance evaluation

This would be like an observation specific "pvalue" for an outlier test for distribution that don't have a residual representation like gaussian/normal model.

The advantage of a general method is that we can add additional "which" over time without needed new model specific results attributes, similar to the new pattern in predict/get_prediction.

user: josef-pkt
issue_7966: state: open
title: Need an insight on nested logit with statsmodel
body: Please how can we use statsmodels to get the marginal effect of a nested logit, estimated from another package like pylogit since statsmodel cannot estimate nested logit.
user: hamid004
issue_7964: state: open
title: FAQ: GLM hat_matrix_diag versus generalized leverage with non-canonical link
body: It looks like the GLM hat_matrix definition does not agree with the definition of generalized leverage (hat_matrix_diag in MLEInfluence) for Probit.

I'm comparing Probit with GLM-Binomial-probit similar to the existing unit tests comparing Logit and Poisson with their GLM counterparts.
All the pieces agree between Probit and GLM, but the `hat_matrix_diag` differs by a no-trivial amount.
(The difference in terms of ratio is only around 7% on average, but up to 60% for individual observation)

Currently my only guess is that the GLM definition based on IRLS only agrees with the generalized leverage definition in the case of canonical links. Although, GLMInfluence uses the observed hessian (OIM) in the computation of the hat_matrix_diag.
(OIM does not equal EIM with non-canonical link, but we use observed=True consistently.)

AFAIR, the literature for GLM Influence is all for canonical links like Logit and Poisson.
According to the unit tests GLM-gaussian and OLS also agree (using canonical linear link)

to confirm I changed the unit test to compare Probit MLEInfluence with MLEInfluence of GLM-Binomial-Probit.
The results agree (at relatively high tolerance). So the difference is not between Probit and GLM, it's between GLMInfluence and MLEInfluence definitions of hat_matrix

My intuitive guess: 
- GLMInfluence keeps weights as fixed when defining the projection matrix. (based on IRLS similar to WLS)
- MLEInfluence uses the one step Newton-Raphson approximation without keeping any parts fixed, computing the full derivative.



user: josef-pkt
issue_7963: state: open
title: ENH: Allow Vector Autoregression Coefficient Selection
body: #### Describe the solution you'd like
Be able to build a VAR model with only specific coefficients. This would be helpful so that we can identify coefficients with a pvalue indicating significance and creating a model only with these significant coefficients.

#### Describe alternatives you have considered
[At minute 34:00 of this video the guy selects the significant coefficients and manually calculates a forecasted value
](https://www.youtube.com/watch?v=TpQtD7ONfxQ&list=PLh8md8UCjx2ZHgbBzgT_6ya3L3Z_BaeuD&index=1)

user: rcyost
issue_7962: state: open
title: FAQ: How to extract the Standard Deviation of Noise (σε) and autocorrelation coefficient (ϕ) after GLSAR fit
body: I am using stasmodel GLSAR fit on my 216 monthly data at every grid point (total=1736) over a region to see the trend of the desired variable. I am doing the same in following steps:
```
list1=[]
p_list1=[]
for i in range(0,1736):
	glsar_result=sm.GLSAR(y, X, 1,missing='drop').iterative_fit(1)
	list1.append(glsar_result.params[1]), p_list1.append(glsar_result.pvalues[1])
```
Till now things are perfect,
Ahead to this I need **Standard Deviation of Noise (σε) and autocorrelation coefficient (ϕ)** parameters from the fitted result.
How can I assess or calculate same using the model results.
Any remark or help from the community......................Thanks🙏
user: HARSHBARDHANK94
issue_7961: state: open
title: DOC: impute_ros in imputation is missing in docs
body: ros = regression on order statistics

related

I just saw by chance that NCSS has a section on Nondetects-Data Regression with support for several distributions
exponential, extreme value, logistic, log-logistic, lognormal, lognormal10, normal, or Weibull distribution
https://www.ncss.com/wp-content/themes/ncss/pdf/Procedures/NCSS/Nondetects-Data_Regression.pdf

I guess this is mainly a censored regression model. and should be compared with what's in ROS.
We should be able to compare full MLE with ROS version, and maybe support similar usecase as impute_ros.

some recent issues for censored or truncated data
#7638
#7922
#7797



user: josef-pkt
issue_7960: state: open
title: ENH: tools linalg - Mahalanobis distance for singular array, duplicated columns
body: use case: x-outliers, distance measure for models with multiple exog. #7951

If we have two or more exog arrays and we column_stack them, then we can have duplicated columns, and stacked array x_all is singular.

Can we compute the mahalanobis distance using svd from the singular x_all so that it is equivalent to dropping duplicate columns and compute distance from the reduced array?

Alternative would be to concatenate only the orthogonal complement.
But just using SVD or pinv should be more direct. (I'm not sure about the scaling.)


user: josef-pkt
issue_7955: state: open
title: ENH: oulier-influence for misspecified models ?
body: For now just a question: How does model misspecification affect outlier-influence measures?

possible case, scale misspecification
based on https://github.com/statsmodels/statsmodels/pull/7951#issuecomment-994987416

Under full MLE, scale=1, e.g. Poisson or HetModels. Any dispersion/variance parameters are part of the estimated parameters.
However, the model could be misspecified, either in scale, e.g. with over dispersion, or in any of the robust cov_types, HC, ....

GLM, OLS have scale as auxiliary parameter in continuous endog case, but it's not part of params and needs to be treated separately.

GLM allows excess dispersion, scale != 1, also in the full MLE case like Poisson, Binomial.
This affects the studentized residuals in the Influence classes.

more general:
How should we treat misspecified models, models that use robust cov_type or excess scale for wald inference?

Currently. there is no systematic treatment of sandwich cov_params or other misspecification in the Influence classes.

Related: Should we add a `cov_type="excess-scale"` generically. It applies only to models that does not estimate scale directly.


 
user: josef-pkt
issue_7949: state: open
title: TST/ENH: make sure and test that `fit(maxiter=0)` works with unchanged params
body: I think we will get some use cases where we want to get a results instance for fixed parameters.
Inference will only be correct under the assumption that the parameters are the MLE.

possible usecases

- outliers: use params estimated with a robust estimator, We can use get_diagnostic, get_influence and similar to check the outlier candidates.
- score_test: fixed, constrained params in results instance might be another possible version for implementing score_test. 
  The current related versions of score_test require that either constrained params and everything related is provided as arguments, or the  constrained estimate is available as result of `fit_constrained`
- ...

AFAIR, we have some checks for maxiter=0, but no general unit tests.


user: josef-pkt
issue_7947: state: open
title: ENH: out of sample loglike
body: In some cases we need to evaluate loglike or pdf/pmf for values of endog and exog other than the estimation sample.
currently, there is no direct way to do it.

I didn't find a specific issue when searching the issue.
some discussion in comments in #7528 and I guess as part of other issues.

One possible way is to go through get_distribution, or to add more families/`dfamilies`.

Design still open. Maybe wait until we have `dfamilies` or similar.
I'm not sure yet whether get_distribution should go through dfamilies, similar to GLM.


user: josef-pkt
issue_7945: state: open
title: TST: systematic, generic unit tests for model based distributions
body: In #7929 I started to add unit tests for internal consistency of model based distributions and get_distributions.
This is similar to what I did for scipy when I started to work on scipy.stats.distributions.

We can expand this to a set of generic unit tests that can be reused for new models, especially for models where it is difficult to get full reference numbers from other packages.
Also, adding generic tests checks and imposes more consistency in interface, API across models.

I'm currently adding parts of this where I need it, but not fully generic and not very reusable.
user: josef-pkt
issue_7941: state: open
title: ENH: Return a `Summary` for `VAR.summary()`
body: There's the functionality in other models like OLS and AR to print the LaTeX format of the summary but can't do this for the VAR.
user: bailez
issue_7937: state: open
title: Is there any plain to implement fit_regularized().summary()?
body: i need this very much!!

Do you have plan?
user: nickhuangxinyu
issue_7934: state: open
title: Documentation and Implementation Differ for Deviance in Logistic Regression
body: Hi, 

For deviance in Logistic Regression, I noticed that the documentation and the implementation are different. That is, `sm_result.resid_dev` is not [calculated](https://github.com/statsmodels/statsmodels/blob/3828c34b52643d0b5e01ebb3d49663ae91a207c2/statsmodels/discrete/discrete_model.py#L4822) the same as it's [documented](https://www.statsmodels.org/dev/generated/statsmodels.discrete.discrete_model.LogitResults.resid_dev.html#statsmodels.discrete.discrete_model.LogitResults.resid_dev). 

<br/>

My experiment: https://colab.research.google.com/drive/1aPR1rzE69Ikg_og3eNHjc_rSVwST4Efz?usp=sharing

If the implementation is correct, is it OK to update the doc? 

Thank you!
user: JiaqiLiu
issue_7928: state: open
title: FAQ-D/TST class diagram for unit tests
body: It is difficult to get an overview for large test modules like `discrete\tests\test_discrete.py`
`pyreverse  -o png -p statsmodels <path to>\statsmodels\statsmodels\discrete\tests\test_discrete.py`

https://stackoverflow.com/questions/36432630/class-diagram-viewer-application-for-python3-source
needs to have graphviz installed to output png files.




user: josef-pkt
issue_7924: state: open
title: ENH: influence - deletion of multiple observations, leave one group out
body: (mainly to park a reference)

- In panel or clustered data, one recommendation is to compute leave one group out influence statistics. membership in group is predefined.
- If there are multiple outliers, then they can shadow or mask each other. Those are cases for robust estimators with high breakdown point.  membership in group of observations to delete is data, estimation dependent.

If we have group specific fixed effects, e.g. constants/intercepts, then the parameter set would change with deletion of group. In that case we could look only at "parameters of interest" and exclude heterogeneity parameters from the influence analysis.

I never tried this, but I think we can follow the same pattern as in MLEInfluence, with d_params based on one step group deletion.


Chien, Li-Chu. 2013. “Multiple Deletion Diagnostics in Beta Regression Models.” Computational Statistics 28 (4): 1639–61. https://doi.org/10.1007/s00180-012-0370-9.

user: josef-pkt
issue_7923: state: open
title: ENH: `predict(which="_dargs")` private which values for internal use.
body: I would like to add other returns (which) to model.predict that are for internal use.
However, they might not follow the public, results.predict api (e.g. multi-column returns, or tuple returns)

One possibility would be to add the leading underscore to signal it's private.

The main advantage of reusing predict is the argument handling, exog, exog_xxx, offset, exposure, ....
Also model.predict is reused by results.get_prediction, which would be nice to get standard errors and confint for other statistics.

we could even add out-of-sample loglike_obs and other measures (not sure loglike_obs needs endog)

user: josef-pkt
issue_7922: state: open
title: ENH/Design:  Mixin, Subclasses for censoring, truncation, ....
body: #7638 general issue for censoring and selection
other issues for truncation

What class and/or mixin structure can we use to build models that are modifications like censoring and truncation in a modular way.

Eg. PenalizedMixin and MixedMixin essentially modify the existing loglike but don't need new distribution specific parts like cdf in truncated and censored models. Those mixins can reuse existing models.

In censored models we need additionally the conditional cdf and the indicators for the "type", i.e. whether and which censoring.
This need to modify loglike, score, ....

The closest we currently have to this are zero-inflated models, with separate sub-models for the two parts, and subclasses for distribution specific parts.

One possibility is to use the GLM pattern, one generic model class and delegation to dfamilies.
PR #7793: multilink models partially delegate to dfamilies, but not in API, i.e. needs subclasses.
dfamilies will have the advantage of being reusable, in contrast to current models like Poisson where everything is hard coded in the model.

Another possibility is to use subclasses of some CensoredBaseModel, TruncatedBaseModel. Subclasses can reuse existing models or dfamilies or scipy distributions depending on what's available.
e.g. models need to implement a `cdf` method, which we already have in a few models.

Mixin class would still need to add the loglike parts for the censored, observations.
Pure truncation (only values in specified interval are observed) would need to add the rescaling by cdf, i.e. doesn't need a mask.

Partially related: control function, instrumental variables for continuous endogenous regressor. his should also work generically for many MLE models.

And then we have combinations of modifiers, e.g. a censored mixed effect (unobserved heterogenity) models.

And count models, discrete random variables, need separate handling for cdf, sf because of masspoints (<= versus <, > versus >=)
(cdf(b) - cdf(a) is half open interval (a, b] )













user: josef-pkt
issue_7919: state: open
title: ENH: API: results method `residuals` as `get_resid` with options
body: We can add many more residuals in different models than the basic response residuals or GLM residuals.

Similar to information criteria: we stopped adding attributes and added method, so we can have options when computing them.

This would define a public API, so users don't have to work with model methods directly, e.g. score_obs or score_factor.
Stata has some of those in predict.

Out of sample might be nice, but doesn't fit in the current structure. We would need endog additional to exog and other predict keywords and the methods or functions to accept out of sample data. 
(e.g. we don't have out of sample loglike)
That would require much larger changes than just adding a `residuals` method

user: josef-pkt
issue_7918: state: open
title: ENH: multinomial/categorical with different exog in choices, example 0-1 inflated beta
body: (just an idea, while thinking about beta regression)

zero and one inflate beta regression (there is some literature that I didn't look at.)
continuous part can be estimated from truncated sample, i.e. only consiter observations with 0 < y < 1

selection model has 3 outcomes, y = 0, y = 1 and y is interior.
suppose exog for y=0 versus y is interior is different for y = 1 versus y is interior

The full model, relevant for e.g. prediction has 3 parts
pro0, prob1, and beta parameters
that makes 5 link functions and parameters
pi_0, pi_1 and mean and precision parameter of beta
The beta part has probability 1 - pi_0 - pi_1, and we need restriction that this is non-negative.

current restricted models
- multinomial MNLogit: assumes same exog (but different params) in each choice
- ordinal OrderedModel assumes a single latent variable for all choices

I guess multinomial Logit with differ exog per choice (but same params) would work




user: josef-pkt
issue_7913: state: open
title: ENH: analogue to cooks distance for linear combinations.
body: motivation:
#7912 implements MLEInfluence for BetaModel. In this case we have two distribution paramters, mean and a precision parameter.

The general definition of cook's distance in MLEInfluence is a quadratic form of all parameters.
For comparisons, I also computed the mean parameter only distance in a notebook example.
This can be generalized to a linear combination `c * d_params`, which if `c` is 0-1 valued vector, it will select a subset of parameters.

It would be easy to add to the current method in MLEInfluence.

Aside: cooks distance looks similar to distance in hausman test. 
Do we have the same basis for inference, p-values?
Also, as a chisquare test statistic, is there a better `effect_size` than cooks distance or is it already normalized.
e.g. scaled non-centrality parameter for chi-square and f-tests (and t_tests ?)

Do we use a more generic name than Cook's distance?
Some articles use GD as generalized (cook's) distance. Also there are different scaled, normed versions.

user: josef-pkt
issue_7911: state: open
title: TST: TestMICE.test_combine, test_corrpsd_threshold[0], test_mixedlm failing on Debian unstable
body: #### Describe the bug

[In Debian unstable](https://ci.debian.net/packages/s/statsmodels/unstable/amd64/), TestMICE.test_combine, test_corrpsd_threshold[0] and test_mixedlm [are failing](https://ci.debian.net/data/autopkgtest/unstable/amd64/s/statsmodels/16528252/log.gz).  (This log is from statsmodels 0.12.2, but 0.13.1 has the same errors; I haven't tried current main.)

As the output of test_corrpsd_threshold is so close to 0, and the results of TestMice.test_combine and test_mixedlm depend substantially on the np.random state, I suspect that this is a rounding issue not a real incorrect-results bug, but I don't have proof of that.

=================================== FAILURES ===================================
____________________________ TestMICE.test_combine _____________________________

self = <statsmodels.imputation.tests.test_mice.TestMICE object at 0x7f946cec9070>

    @pytest.mark.slow
    def test_combine(self):
    
        np.random.seed(3897)
        x1 = np.random.normal(size=300)
        x2 = np.random.normal(size=300)
        y = x1 + x2 + np.random.normal(size=300)
        x1[0:100] = np.nan
        x2[250:] = np.nan
        df = pd.DataFrame({"x1": x1, "x2": x2, "y": y})
        idata = mice.MICEData(df)
        mi = mice.MICE("y ~ x1 + x2", sm.OLS, idata, n_skip=20)
        result = mi.fit(10, 20)
    
        fmi = np.asarray([0.1778143, 0.11057262, 0.29626521])
>       assert_allclose(result.frac_miss_info, fmi, atol=1e-5)
E       AssertionError: 
E       Not equal to tolerance rtol=1e-07, atol=1e-05
E       
E       Mismatched elements: 3 / 3 (100%)
E       Max absolute difference: 0.17686937
E       Max relative difference: 1.59957657
E        x: array([0.230217, 0.287442, 0.322124])
E        y: array([0.177814, 0.110573, 0.296265])

/usr/lib/python3/dist-packages/statsmodels/imputation/tests/test_mice.py:366: AssertionError
__________________________ test_corrpsd_threshold[0] ___________________________

threshold = 0

    @pytest.mark.parametrize('threshold', [0, 1e-15, 1e-10, 1e-6])
    def test_corrpsd_threshold(threshold):
        x = np.array([[1, -0.9, -0.9], [-0.9, 1, -0.9], [-0.9, -0.9, 1]])
    
        y = corr_nearest(x, n_fact=100, threshold=threshold)
        evals = np.linalg.eigvalsh(y)
>       assert_allclose(evals[0], threshold, rtol=1e-6, atol=1e-15)
E       AssertionError: 
E       Not equal to tolerance rtol=1e-06, atol=1e-15
E       
E       Mismatched elements: 1 / 1 (100%)
E       Max absolute difference: 1.05471187e-15
E       Max relative difference: inf
E        x: array(1.054712e-15)
E        y: array(0)

/usr/lib/python3/dist-packages/statsmodels/stats/tests/test_corrpsd.py:196: AssertionError
_________________________________ test_mixedlm _________________________________

    def test_mixedlm():
    
        np.random.seed(3424)
    
        n = 200
    
        # The exposure (not time varying)
        x = np.random.normal(size=n)
        xv = np.outer(x, np.ones(3))
    
        # The mediator (with random intercept)
        mx = np.asarray([4., 4, 1])
        mx /= np.sqrt(np.sum(mx**2))
        med = mx[0] * np.outer(x, np.ones(3))
        med += mx[1] * np.outer(np.random.normal(size=n), np.ones(3))
        med += mx[2] * np.random.normal(size=(n, 3))
    
        # The outcome (exposure and mediator effects)
        ey = np.outer(x, np.r_[0, 0.5, 1]) + med
    
        # Random structure of the outcome (random intercept and slope)
        ex = np.asarray([5., 2, 2])
        ex /= np.sqrt(np.sum(ex**2))
        e = ex[0] * np.outer(np.random.normal(size=n), np.ones(3))
        e += ex[1] * np.outer(np.random.normal(size=n), np.r_[-1, 0, 1])
        e += ex[2] * np.random.normal(size=(n, 3))
        y = ey + e
    
        # Group membership
        idx = np.outer(np.arange(n), np.ones(3))
    
        # Time
        tim = np.outer(np.ones(n), np.r_[-1, 0, 1])
    
        df = pd.DataFrame({"y": y.flatten(), "x": xv.flatten(),
                           "id": idx.flatten(), "time": tim.flatten(),
                           "med": med.flatten()})
    
        mediator_model = sm.MixedLM.from_formula("med ~ x", groups="id", data=df)
        outcome_model = sm.MixedLM.from_formula("y ~ med + x", groups="id", data=df)
        me = Mediation(outcome_model, mediator_model, "x", "med")
        mr = me.fit(n_rep=2)
        st = mr.summary()
        pm = st.loc["Prop. mediated (average)", "Estimate"]
>       assert_allclose(pm, 0.52, rtol=1e-2, atol=1e-2)
E       AssertionError: 
E       Not equal to tolerance rtol=0.01, atol=0.01
E       
E       Mismatched elements: 1 / 1 (100%)
E       Max absolute difference: 0.01958632
E       Max relative difference: 0.03766599
E        x: array(0.539586)
E        y: array(0.52)

/usr/lib/python3/dist-packages/statsmodels/stats/tests/test_mediation.py:214: AssertionError

#### Code Sample, a copy-pastable example if possible

The statsmodels test suite.

#### Expected Output

The tests should pass.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

The problem started when Debian upgraded from libblas3/liblapack3 3.9 to 3.10.

Python 3.9, numpy 1.19, scipy 1.7, matplotlib 3.3, pandas 1.1.

user: rebecca-palmer
issue_7909: state: open
title: DOC/TST/ENH: score_test for affine constraints - status ?
body: I don't remember whether and how score_test works with affine constraints `r @ params = c`, including fixing one parameter.

Does it work? Is it correct? Do we have unit tests?

two cases

- fit_constrained, allows affine constraints using offset
- fixed parameters set to a nonzero value


I think now whether `c` is zero or not is irrelevant:
All we need is score and hessian of full model evaluated at restricted parameters.
In standard nonrobust score test, we don't need any information about what the constraints are (similar to LR test).
For robust score_test, we need the restriction matrix `r` for the cov, but the effect of value `c` is included in the score.

user: josef-pkt
issue_7908: state: open
title: ENH: specification test for nested models based on score_test version 3
body: score_test version 3 needs the full model and the restricted params
This works even when score_factor and hessian_factor are not available.

A possible case are zero-inflated models where it might not be easy to get score_ and hessian_factors.
To test zero-inflation itself, we would need an interior point, e.g. identity link. Currently ZI models don't use GLM and we only have logit and probid links. But this should work already for variable addition score_tests.

using score_test requires that one model is a parameter restriction of another model, and that computing score and hessian at the restricted parameters is feasible.
example 
Generalized Poisson versus Poisson
NegativeBinomial versus Poisson does NOT work because we cannot evaluate NB at the no excess dispersion parameter.

score_test version 3 is not very userfriendly (but not bad for experts), so some easier to use functions or methods for specific specification and diagnostic tests would be useful.

This score_test version is less attractive than for example the exog_extra version. because we need both models, restricted and unrestricted, and then LR and Wald tests are also available with one extra fit. Score_test might still have better small sample properties than the other two.



user: josef-pkt
issue_7905: state: open
title: ENH: endogeneity test with control function, instruments
body: diagnostic test for e.g. #7689
(I didn't find another specific issue for this right now. It might be somewhere already with a different name.)

When adding the control function to the regressors, then it should be insignificant when the "`exog`" is not endogenous.

We can use a variable addition score_test for this as specification test in the main model.
We can add additional nonlinear basis functions similar to reset tests.


user: josef-pkt
issue_7903: state: open
title: ENH/Design: post-estimation for instrumental variables IV and two-stage estimation
body: Design for `heckman`, ivpoisson, control function approach and similar 2-stage methods.
directly related to #7691 #7689

we have two choices and estimator in the design, current example heckman
- use full MLE or GMM simultaneously for all stages
- use 2-stage with auxiliary and main model

(another example, treatment effects estimators with GMM inference)

My tendency in the discussion of `heckman` PR was towards full model, i.e. bivariate MLE with recursive simultaneity.

However, we might not want the full model for post estimation results if the main focus is on the main model.
The following are some guesses on design issues that will show up
- **wald tests:**
  - full model: main problem is that we need to expand constraint matrices to full params,  We need unique param names. Constraints defined by strings will automatically handle full params. Also allows hypothesis tests for cross-equation constraints. (not sure that is a common usecase in this.)
  - only main model: wald test for main model work generically. No inference for first stage or it needs to be provided separately.
- **predict, get_prediction:**
  - full model: we need many "which" similar to zero-inflated models.  get_prediction possibly needs linear and endpoint versions for more than main exog.
  - only main model: might work generically for some cases. control function has extra generated terms, which, I guess, should be ignored for prediction.
  -  I guess we will have to distinguish between conditional prediction E(y1 | x1, y2) and unconditional prediction  E(y1, y2 | x1, x2). (similar to structural versus reduced form in simultaneous equation models). and/or maybe E(y1 | x1, E(y2 | x2)) in nonlinear case.
 - **diagnostic**
   - most will be model specific
   - Can we reuse e.g. PoissonDiagnostic for ivpoisson?  I guess checking predictive distribution still makes sense.
 - **results attributes and internal attributes** ?
   - aic, (p)rsquared ... full or main?
   - resid main only or bivariate?
   - ...
- **score_test** ? 
 - ...

Another complication: 
Some estimation methods, like GMM and partially OLS, GLM, PHReg, might not estimate and provide all necessary information to have a full model for get_distribution and prediction. In those cases we only get a limited set of post-estimation results.

This sounds painful.

I guess the most useful would be to always provide a model_main, results_main even if we have to construct it separately and independently of the estimation method.



user: josef-pkt
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_7901: state: open
title: ENH/DOC: GMM for panel data. Example/Tuto
body: I am currently trying to use your GMM model, but I can't find any existing project, example or tuto about it. I saw you have done some notebook of example for other models, which is quite nice, thanks. Do you now if tutorial for GMM exist or if it would be possible to create one ?

Maybe there exist some example of GMM statsmodels use that I have not found, if so could you share them with me ? Otherwise, would it be possible to make a notebook for that as you made for other models in examples?

I am currently working on longitudinal (panel) data and I would like to better understand how GMM could apply to my work. Note that I found https://stackoverflow.com/questions/39008793/are-there-any-examples-for-statsmodels-gmm-generalized-method-of-moments , but it only seems to deal with crosssectional data.

Many thanks :)
user: JulietteOrtholand
issue_7899: state: open
title: SUMM/ENH: get_prediction_xxx  special, extended get_prediction methods
body: We have no the underlying infrastructure with NonlinearDeltaCov to add special cases of prediction without much work.

- get_prediction: in my current version this can handle any `model.predict(which=...)` (after one more bugfix). Aggregation weights are still missing.

other possibilities not supported by this

- **get_prediction_partial**: 
  - linear prediction for subset of exog, we only have predict partial in GAM, AFAIR (can be done already using t_test)
  - "eform" can be considered also as partial prediction, i.e. predict for each parameter (of mean function)
  - I have not seen a multi-column, multi-parameter version of `eform` yet. Does it make sense to have link of a subsum of linpred?
- **get_prediction_nonlinear**: 
  - This would be for user specified function (similar to Stata)
  - This would mainly be a predict wrapper for wald_nonlinear
  - It should include agg_weights and average=True option that `_get_wald_nonlinear` does not have. (useful for margeff, predicted ATE)
- **get_prediction_diff** (or compare? e.g. also allow ratio) #2291
  - special case of get_prediction_nonlinear where user provides two sets of exog and we predict ("which"?) the difference.
  - Using margins for count and binary exog would be a special case (e.g. in first exog set everyone is male, in second set everyone is female)
  - possible extension: we use formula `transform`ation for each exog set. This would automatically handle interaction effects and polynomials.
  - do we need paired versus independent (in analogy to t_test) see below for ATE
- **predict_at** (I will not be working on this anytime soon, I guess)
  -  automatic construction of predefined exog sets
  - currently `get_margeff` includes some of this
  - emmeans and lsmeans in other packages
  - The difficult part will be formula handling if we support `terms` (and interaction) and not just columns of exog. It's difficult to get the relevant information about how user/formula constructed the exog.
- ...


Right now I'm mainly interested in whether to add `get_prediction_nonlinear` and/or `get_prediction_diff`

ATE, average treatment effect possible use case for  `get_prediction_diff`
- paired: we predict potential outcome for each observation, i.e. two sets of exog cover the same observations
- unpaired/independent: comparing subset of exog populations, only relevant with `average=True`. Observations in two exog sets need not be the same, i.e. compare average prediction in population 2 with average prediction in population 1.

related nonlinear example: risk ratio in Logit: ratio of average prediction or average of ratio. #2215 #2619




user: josef-pkt
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_7897: state: open
title: ENH: score residuals for diagnostic, influence,
body: just an idea while browsing influence measures for NegativeBinomial and similar

Stata has "score residuals" that are just the score_factor

Instead, we can use normalized score_obs or contribution to score_test of each observation.
`score_obs_i @ inv(hessian) * @ score_obs_i`

This might be similar to score_test with a one observation outlier dummy.
There should be related issues, but I don't find them right now.

score_obs in simple one-parameter models is just `score_factor * x_i`
we need a weighted average (of columns) or quadratic form to get a univariate measure.

One problem: Are score_obs still heteroscedastic?  inherently like in poisson, probit, negative binomial, ...

The advantage of score residuals would be that they apply even in models that don't have a mean parameterization.
We need to generalized influence on mean function to influence on arbitrary distribution `dargs`.


This needs more reading and examples to try out

user: josef-pkt
issue_7896: state: open
title: Feature Request: A test for the statistical information content in IMF components of Empirical Mode Decomposition
body: #### Is your feature request related to a problem? Please describe
The test for whether an IMF from a decomposed time-series using EMD has information or not has is not available in any python library. I could not find a single library anywhere on the internet, python or Matlab where I could perform the test. 

#### Describe the solution you'd like
I want to contribute towards the project by adding the feature to test for the statistical information content in IMF components of Empirical Mode Decomposition.
The methodology was proposed by Zhaohua Wu, & Huang, N. E. (2004) in their paper, "A study of the characteristics of white noise using the empirical mode decomposition method" and has been established as an effective method to analyse trends especially for hydrological time series.
link: [Original Paper](https://www.jstor.org/stable/4143111)
link: [statistical test for trend identification](https://doi.org/10.1016/j.jhydrol.2013.12.039)

#### Describe alternatives you have considered
I tried to find alternatives in MATLAB but its not available hence had to make my own code in python.

#### Additional context
The test will be helpful for people utilizing EMD decompositions and especially in the Hydrological time series analysis. Kindly let me know if the feature is in line with the aim of this project and if I may raise a PR.
user: oshin94
issue_7894: state: open
title: FAQ-D: diagnostic and specification  methods - overview
body: Trying to get an overview of specification and diagnostic tests and plots by model categories.

**general**

- variable addition score_test
- reset type tests
- PIT type test and plots, gof (continuous versus discrete)
- binned endog space,  predictive distribution (chisquare test)
- variance/heteroscedasticity/dispersion tests (most likely model specific)
- nonlinearity, functional form tests (most likely variable addition score or cm tests)
- comparing models, information criteria, likelihood ratio tests, Vuong tests, ...
- ? White test, hausman test, (I did not look at those outside linear models)
- IM ratio ?
- overidentifying restrictions tests (GMM)  (not clear which)
- correlation tests ?
- endogeneity tests ?
- nonparametric tests ?
- bootstrap ?
- cross validation ?
- ...

**discrete - count**

- diagnostic plots (already available)
- excess probability tests (zero-inflation or deflation)
- ...

**discrete - binary**

- ???
- plots ?
- ...

**models without mean parameterization**

examples
- accelerated failure time, R+ models
- models for flexible, multi-parameter distributions
- ...

methods
- predictive distribution, aic, ...
- variable addition tests for each distribution parameter, dargs
- no "moment" tests available, but "probability moments" and "quantile moments" (e.g. for binned endog space)
- ...

**Implementation**

- generic methods (score_test, lm-test, cmt, eventually reusing GMM)
- model specific methods
- this will be a pain to verify because only parts are available in other packages, most likely Monte Carlo verification
- "asymptotically equivalent" methods can be used to cross-check
- generic code will be used by many models and verified in several special cases (which compensate for deficiency of unit tests for generic code)
- get_diagnostic and class hierarchy for diagnostic and specification methods
- some of it will need multi-link models (e.g. `Het` models with score test for heteroscedasticity)
- some methods will be restricted to models that have methods like results.score_test already implemented
- we can reuse and extend predict and get_prediction
- tools: like variables to add in reset type tests, moment restrictions
- difficult work: choosing defaults and recommendations (which "asymptotically equivalent" method do we use)
- ...

**current**

- linear model OLS, diagnostic and specification tests
- Count Diagnostic and Poisson: this is my current area to get this started
- ...

user: josef-pkt
issue_7892: state: open
title: ENH: Ordered model: Is there "parallel test" or "proportional odds" for ordinal regression? 
body: I'm working on building a ordinal logistic regression model  and I'm using the Ordered Model from statsmodels.miscmodels.ordinal_model.

Is there a way I can  get "parallel test" or "proportional odds" ? If not, this issue is indeed a feature request.

Thanks much!
user: amrain-py
issue_7891: state: open
title: ENH: numerical derivatives numdiff for outlier-influence MLEInfluence, NegativeBinomial
body: We currently don't have any numdiff fallback if some analytical derivatives are missing in the model

I'm trying out MLEInlfuence(res) where res is a NegativeBinomialP results instance.
but I get
`AttributeError: 'NegativeBinomialP' object has no attribute '_deriv_score_obs_dendog'`

Currently only Logit and Poisson have `_deriv_score_obs_dendog`, in both cases it's just exog. (I guess because of canonical link)

aside there are also other parts like resid_pearson missing in NegBin for MLEInlfuence to work
we need `predict(which="var")` for that.

(I'm not even sure some of the definitions in MLEInfluence apply if we have extra params or multiple link functions)

quick search, very new article (I didn't look at it)
Luisa Rivas & Manuel Galea (2021) On estimation and influence measures for the Negative Binomial regression model based on Q-function, Communications in Statistics - Theory and Methods, DOI: 10.1080/03610926.2021.1942493


user: josef-pkt
issue_7888: state: open
title: ENH: add get_prediction to MNLogit, NonlinearDeltaCov for multinomial
body: In #7870 I leave MNLogit get_prediction as NotImplementedError
OrderModel also does not have get_prediction and not much for `predict`

The problem in MNLogit is that params is 2-d. We need to ravel to patch params to cov_params.
Similar case is already implemented in discrete `MultinomialModel._derivative_exog` for margins.

see next comment for using NonlinearDeltaCov for average prediction (probabilities of each choice)

related usecase: I started with some diagnostics common to models with categorical/multinomial response, e.g. base on average probabilities (e.g. stochastically-larger residuals) and Hosmer-Lemeshow type binning.


user: josef-pkt
issue_7884: state: open
title: GLS.fit().params contains only 1 value
body: #### Describe the bug
I'm doing a linear fit with `statsmodels.api.GLS`. I stumbled upon a case where `.fit()` produces a result that has only 1 value in `.params`. I'd expect to get 2 - beta and the intercept (`y=beta*x + intercept`). I checked the docs, but sadly I'm no smarter. Am I perhaps using this incorrectly?

#### Code Sample, a copy-pastable example if possible


```python
import statsmodels.api as sm

first_normalized_pc = pd.Series(...)
second_normalized_pc = pd.Series(...)

# get rolling difference
window_len = 100
differences = []
idx_start = 10000
idx_end = 40000

for i in range(idx_start, idx_end):
    window_end = i + window_len
    x = first_normalized_pc[i:window_end]
    S1 = sm.add_constant(x.values)
    y = second_normalized_pc[i:window_end]
    results = sm.GLS(y.values, S1).fit()
    differences.append(results.params[1] * x[-1] - y[-1] + results.params[0])
```
#### Expected Output
I'd naively expect that `results.params` contains 2 items.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``
<details>

INSTALLED VERSIONS
------------------
Python: 3.7.10.final.0
OS: Linux 5.4.0-89-generic #100-Ubuntu SMP Fri Sep 24 14:50:10 UTC 2021 x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

statsmodels
===========

Installed: 0.12.2 (/home/toaster/PROGS/miniconda3/envs/puma-lab/lib/python3.7/site-packages/statsmodels)

Required Dependencies
=====================

cython: 0.29.24 (/home/toaster/PROGS/miniconda3/envs/puma-lab/lib/python3.7/site-packages/Cython)
numpy: 1.20.3 (/home/toaster/PROGS/miniconda3/envs/puma-lab/lib/python3.7/site-packages/numpy)
scipy: 1.5.4 (/home/toaster/PROGS/miniconda3/envs/puma-lab/lib/python3.7/site-packages/scipy)
pandas: 1.3.2 (/home/toaster/PROGS/miniconda3/envs/puma-lab/lib/python3.7/site-packages/pandas)
    dateutil: 2.8.2 (/home/toaster/PROGS/miniconda3/envs/puma-lab/lib/python3.7/site-packages/dateutil)
patsy: 0.5.2 (/home/toaster/PROGS/miniconda3/envs/puma-lab/lib/python3.7/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.4.3 (/home/toaster/PROGS/miniconda3/envs/puma-lab/lib/python3.7/site-packages/matplotlib)
    backend: module://matplotlib_inline.backend_inline 
cvxopt: Not installed
joblib: 1.0.1 (/home/toaster/PROGS/miniconda3/envs/puma-lab/lib/python3.7/site-packages/joblib)

Developer Tools
================

IPython: 7.27.0 (/home/toaster/PROGS/miniconda3/envs/puma-lab/lib/python3.7/site-packages/IPython)
    jinja2: 3.0.1 (/home/toaster/PROGS/miniconda3/envs/puma-lab/lib/python3.7/site-packages/jinja2)
sphinx: Not installed
    pygments: 2.10.0 (/home/toaster/PROGS/miniconda3/envs/puma-lab/lib/python3.7/site-packages/pygments)
pytest: Not installed
virtualenv: Not installed
</details>

user: jmakov
issue_7883: state: open
title: Is it possible to use statsmodels to fit an ornstein-Uhlenbeck process?
body: I read the docs and couldn't find any information regarding this, is it possible to use statsmodels to fit an Ornstein-Uhlenbeck process (in a numpy array) and get the relavant parameters mu, sigma and theta?
user: mghadam
issue_7873: state: open
title: ENH: add predict and get_prediction for which="cdf" 
body: #4745 prediction interval for any distribution, #2015
#7870 PR get_prediction
#7833 get_prediction across models

also: #7793 multilink models dfamilies

I guess the best generic implementation would be to use get_distribution, which in turn needs to transform `params` to predicted distribution (scipy compatible) `dargs`. 
Alternatively, we could have cdf in the family/dfamily e.g. we need to convert params to (d)family dargs and then to scipy.distribution args.
`get_distribution` can directly convert params to scipy, or detour through the family/dfamily. 
(GLM get_distribution goes through family, discrete does not make full use of (d)families yet.)

In order to apply NonlinearDeltaCov, we need it as a function of `params`.
An indirect two-step method would be to get the (multivariate normal) distribution of `dargs` first. That would work well for one parameter families like Poisson.

user: josef-pkt
issue_7869: state: open
title: WIP ENH: PenalizedMixin improve convergence
body: see #7867

This just parks some changes adding start_params as a trial.
Does not work yet. One unit test in test_penalized fails on my computer.

first commit is from #7868, remove here when that PR is merged


user: josef-pkt
issue_7867: state: open
title: ENH: PenalizedMixin convergence problems, find better start_params
body: see https://github.com/statsmodels/statsmodels/issues/7866#issuecomment-963421093

In almost all of our test runs, the unit test passes and the final fit converged.
However, optimization and convergence is fragile and likely fails in many cases without (user) specifying good start_params


user: josef-pkt
issue_7864: state: open
title: BUG/REF: redundant code in genmod get_prediction_glm, ignores link
body: **update** It looks like there are some bugs, some options seem to be ignored, 
e.g. ~~I don't see any offset handling in the code for endpoint transformation confint, it just uses linpred = x b.~~
looks like false alarm
more in later comments

The get_prediction_glm function in genmod has `weights` handling copy pasted from regression linear model version.
`weights` are not used in GLM.
GLM and GLMResults predict do not support weights, no freq_weights or var_weights. Maybe those should be added.

The function needs `exog` everything else we could delegate to results predict instead of using model predict.
The `exog` need to go through `transform` handling.
In GAM we also need creating the basis function for the predict exog.









user: josef-pkt
issue_7863: state: open
title: ENH: approximate t-distribution (tail) quantiles by normal quantiles (prediction intervals)
body: A vague idea

The variance of the t-distribution is not the `scale` of the distribution, it also depends on df.
If we want to use the normal distribution, then we could use scale equal to the implied variance of the t-distribution (i.e. df corrected)

Specific issue: prediction intervals, confint_obs for OLS assuming gaussian residuals.
In that case, we can consider variance as variance of the parameter uncertainty (assumed to be t-distributed) plus the observation noise given by normal distribution.

My guess is that using the df corrected variance for parameters would widen the prediction interval.
I haven't tried it out yet.
Related issue: refactoring and enhancing `get_prediction` #7833

OLS get_prediction has a use_t option, where the observation noise (distribution of residual) is also taken to be the t-distribution. 
(That was heuristic, just so we can use a single distribution for combined parameter uncertainty and residual variance)

A similar idea might apply to other models with (symmetric ?) additive noise. T-Model, ...
e.g. moment matching to approximate a distribution
related #7278 minimum gof-distance approximation to a distribution

The underlying motivation is to get a simpler approximation to prediction interval that takes parameter uncertainty into account than what we need for general distribution classes, eg. poisson, glm #2015.
But this approximation would still be better in small samples than what OLS get_prediction currently does.




user: josef-pkt
issue_7862: state: open
title: ENH: `get_wls_results` as public method in RLM, GLM, IRLS
body: Using the WLS model that replicates the final step would be useful for local inspection and extra results.

I added a not much advertised option in GLM, but no explicit option for RLM (and Quantile Regression although that's not smooth).

Related: Local approximation by Newton-Raphson or Fisher optimization. Davidson and MacKinnon.

Note: AFAIR, RLM and GLM use different definitions of IRLS. GLM uses "working residuals"
RLM is linear model, i.e. linear link only, this will make a difference for response based extra results like prediction.



user: josef-pkt
issue_7861: state: open
title: ENH: handling cluster_weights, freq_weights in models, cov_type, moments, descriptive statistics
body: (semi-random) thought

cluster_weights as special case of freq_weights

e.g. how do we define cluster robust or panel data robust cov_type if there are freq_weights? #2829

If freq_weights are constant within cluster or group, then the freq_weights only affect aggregation across clusters and then are similar to freq_weights for cross-sectional data.

Do we need specific `cluster_weights` option? or
Can we compute weighted freq_weights within cluster statistics, that reduce to cluseter weighted statistics if those freq_weights are constant within cluster?

In the latter case, we wouldn't need cluster_weights, but could write unit tests based on those.

Weights in GEE are or were cluster/group weights.


NOTE: We might also have the opposite, weights within clusters, but unrelated to weighting across clusters.
An example would be IPW-style handling of missing panel or within cluster data, if missingness is not exogenous, random or completely at random.
Some approaches there are like treatment effects literature with Rubin, IPW, .... But I never went through the details for panel or cluster data case for (endogenous) missing treatment.



user: josef-pkt
issue_7838: state: open
title: [BUG] statsmodel.formula.api -- order of terms impacts fit_regularized results
body: #### Describe the bug

Example code shows that the params are different based strictly on the order in which the polynomial terms in the model are declared in the formula. Problem impacts only fit_regularized() and not fit()


#### Code Sample, a copy-pastable example if possible
```
import statsmodels.formula.api as smf
import statsmodels.api as sm
import pandas as pd
import random
#print(sm.show_versions())
data = []
for x1 in range(1, 100) :
    for x2 in range(1, 100) :
        x1 = random.random()
        x2 = random.random()
        data.append((x1, x2, 3*x1*x1 + 2*x1*x2 - 1*x1))
df = pd.DataFrame(data, columns=['X1', 'X2', 'Y'])
res0 = smf.ols(formula = 'Y ~ I(X1*X2) + I(X1**2) + X1 - 1 - X2', data=df).fit_regularized()
print(f"\tfit_regularized#1\n{res0.params}")
res1 = smf.ols(formula = 'Y ~ X1 + I(X1*X2) + I(X1**2) - 1 - X2', data=df).fit()
print(f"\tfit_regularized#2\n{res1.params}")
res2 = smf.ols(formula = 'Y ~ I(X1*X2) + I(X1**2) + X1 - 1 - X2', data=df).fit()
print(f"\tfit#1\n{res2.params}")
res3 = smf.ols(formula = 'Y ~ X1 + I(X1*X2) + I(X1**2) - 1 - X2', data=df).fit()
print(f"\tfit#2\n{res3.params}")
```

### Output of the script
```
	fit_regularized#1
I(X1 * X2)    1.999803
I(X1 ** 2)    2.963199
X1           -0.972459
dtype: float64
	fit_regularized#2
X1           -1.0
I(X1 * X2)    2.0
I(X1 ** 2)    3.0
dtype: float64
	fit#1
I(X1 * X2)    2.0
I(X1 ** 2)    3.0
X1           -1.0
dtype: float64
	fit#2
X1           -1.0
I(X1 * X2)    2.0
I(X1 ** 2)    3.0
dtype: float64
```

#### Output of ``import statsmodels.api as sm; sm.show_versions()``
```
INSTALLED VERSIONS
------------------
Python: 3.9.7.final.0
OS: Darwin 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:21 PDT 2021; root:xnu-7195.141.6~3/RELEASE_X86_64 x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

statsmodels
===========

Installed: 0.13.0 (/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/statsmodels)

Required Dependencies
=====================

cython: 0.29.13 (/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/Cython)
numpy: 1.20.1 (/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/numpy)
scipy: 1.6.1 (/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/scipy)
pandas: 1.2.3 (/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas)
dateutil: 2.8.1 (/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/dateutil)
patsy: 0.5.2 (/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.3.4 (/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/matplotlib)
    backend: MacOSX 
cvxopt: Not installed
joblib: 1.0.1 (/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/joblib)

Developer Tools
================

IPython: 7.21.0 (/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/IPython)
    jinja2: 3.0.1 (/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/jinja2)
sphinx: Not installed
    pygments: 2.8.0 (/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pygments)
pytest: Not installed
virtualenv: Not installed
```

Problem is reproducible under Linux  as well 
```

INSTALLED VERSIONS
------------------
Python: 3.6.8.final.0
OS: Linux 3.10.0-1062.1.2.el7.x86_64 #1 SMP Mon Sep 30 14:19:46 UTC 2019 x86_64
byteorder: little
LC_ALL: en_US.UTF-8
LANG: None

Statsmodels
===========

Installed: 0.10.1 (/org/tools/pypi/3.6.8/1/el-7-x86_64/lib/python3.6/site-packages/statsmodels)

Required Dependencies
=====================

cython: 0.29.13 (/org/tools/pypi/3.6.8/1/el-7-x86_64/lib/python3.6/site-packages/Cython)
numpy: 1.17.3 (/org/tools/pypi/3.6.8/1/el-7-x86_64/lib/python3.6/site-packages/numpy)
scipy: 1.3.1 (/org/tools/pypi/3.6.8/1/el-7-x86_64/lib/python3.6/site-packages/scipy)
pandas: 0.25.2 (/org/tools/pypi/3.6.8/1/el-7-x86_64/lib/python3.6/site-packages/pandas)
dateutil: 2.8.0 (/org/tools/pypi/3.6.8/1/el-7-x86_64/lib/python3.6/site-packages/dateutil)
patsy: 0.5.1 (/org/tools/pypi/3.6.8/1/el-7-x86_64/lib/python3.6/site-packages/patsy)
```
user: snacamavtyn
issue_7836: state: open
title: BUG: SARIMAX is not strict enough about sample size and model lag length
body: I've been working with the following pandas data-frame (named 'ts') for some time now:

```
	ds	y
0	2020-01-31	-10
1	2020-02-29	-15
2	2020-03-31	-293
3	2020-04-30	-75
4	2020-05-31	-187
5	2020-06-30	-199
6	2020-07-31	-359
7	2020-08-31	-136
8	2020-09-30	-344
9	2020-10-31	-56
10	2020-11-30	-206
11	2020-12-31	-20
12	2021-01-31	-91
13	2021-02-28	-108
14	2021-03-31	100
15	2021-04-30	91
16	2021-05-31	17
17	2021-06-30	-117
18	2021-07-31	-60
19	2021-08-31	-87
20	2021-09-30	-20
```
And have realized that for certain subsets of said data-frame, it is not always possible to fit the SARIMAX model:
`(p=2,d=3,q=2)(P=1,D=3,Q=2,m=3)`. What I mean by that is that the code:

```
m=SARIMAX(endog=list(ts.iloc[:nobs,1]), 
                      order=(2,3,2),
                      seasonal_order=(1,3,2,3),
                      trend='c', initialization='approximate_diffuse')

m_fitted=m.fit(method='cg', disp=0)
```

always throws the following error when `nobs=13` :
```
runcell(7, 'C:/Users/username/Desktop/foldername1/foldername2/foldername3/foldername4/foldername5/Plannables_Sarima_Prophet/plannables_Sarima_Prophet_MVP/Sarimax_forecasts_v3.py')
Traceback (most recent call last):

  File "C:\Users\username\Desktop\foldername1\foldername2\foldername3\foldername4\foldername5\Plannables_Sarima_Prophet\plannables_Sarima_Prophet_MVP\Sarimax_forecasts_v3.py", line 225, in <module>
    m_fitted=m.fit(method='cg', disp=0)

  File "C:\Users\username\anaconda3\lib\site-packages\statsmodels\tsa\statespace\mlemodel.py", line 650, in fit
    start_params = self.start_params

  File "C:\Users\username\anaconda3\lib\site-packages\statsmodels\tsa\statespace\sarimax.py", line 953, in start_params
    params_variance) = self._conditional_sum_squares(

  File "C:\Users\username\anaconda3\lib\site-packages\statsmodels\tsa\statespace\sarimax.py", line 836, in _conditional_sum_squares
    Y = endog[k:]

IndexError: too many indices for array: array is 0-dimensional, but 1 were indexed


```
But poses no problems for `nobs<13` nor `nobs>13` (obviously within the interval `[1, 20]`). I figured this might be a bug since it is just a very weird behavior.


user: Mastercol01
issue_7835: state: open
title: ENH: Computational Techniques for Variable Selection in Linear Regression Analysis
body: I would like a way to perform different methods for variable selection including:
- generating all possible regressions
- forward selection
- backward elimination
- stepwise regression

In particular, I have been looking through the documentation to find a way to generate all possible regressions quickly (up to 30 regressors) and to generate a summary of the results. I was unable to find this functionality, but if it does already exist please correct me. 

#### Describe the solution you'd like
I would like a simple method to generate all possible models, and print a summary of results that would look something like this:

![image](https://user-images.githubusercontent.com/22874006/139734779-8b1c87d8-f142-4506-9a6d-d234d633fd57.png)

#### Additional context
I have not contributed to any open source projects yet, however I can implement this if it is recommended by other people in the community. I am very open to advice and suggestions.

The image I used and a description of these different methods for variable selection are from:

Montgomery, Douglas C., et al. “Chapter 10: Variable Selection and Model Building.” Introduction to Linear Regression Analysis, John Wiley &amp; Sons, Inc., Hoboken, 2021. 
user: kmantych
issue_7834: state: open
title: ENH: CountDiagnostic: add plot, get_diagnostic to CountModel
body: I wanted to briefly check some results to compare countmodels.
need quicker access

- add get_diagnostic to result class
- move plot to CountDiagnostic
- limit, add k_max in plot, and maybe to other methods that don't have it yet
  (in my case plot goes to almost k=80, but most of the action is below 10 or 20) 

plotting also works when `res` is an NBP instance
```
dia_nbp = PoissonDiagnostic(res)
dia_nbp.plot_probs();
```

(my example:  average prediction of zeros and confidence interval for it (#7758) in Poisson are very small. Huge excess zeros compared to Poisson. NBP fits very well. I wanted a fast diagnostic to see how Poisson and NBP frequencies/expected counts look like. Results look like a feature and not a bug in my new code.   
example from a unit test case.) 

**update** The plot also works if `res` is a ZeroInflatedPoisson instance (that's for which I initially wrote the plot function for )

user: josef-pkt
issue_7833: state: open
title: SUMM/ENH: get_prediction across models
body: get_prediction outside tsa

current:

- OLS, linear_model: linear prediction with confidence and prediction interval
  - code in `regression._prediction` 
- GLM linear prediction and link.inverse transformed
  - code in `genmod._prediction` 
- discrete.Poisson: follows GLM pattern reuses genmod code
- GAM: delegates to GLM, except of spline handling

no `which` yet: get_prediction so far is only for mean, expectation of endog E(y | x)

algorithm or method cases, all based on cov_params

- linear prediction: simple linear transformation of params
  - prediction interval includes var_resid
- link.inverse function: delta method for se, endpoint transformation for conf_int
  - current code in `genmod._prediction` 
  - this can be extended to multi-link models where mean is orthogonal to other params
  - can be extended to other univariate monotonic functions beside link_mean, e.g. prob in count models
    no generic function yet
- arbitrary function, non-separable multi-link models: need to use delta method for asymptotic results
  - code for delta method in #7758
  - e.g. get_prediction for mean in NBP, GPP
  - `which` in multi-link and extra_params models
- prediction interval for models with non-additive error term (not sure yet)
  - distribution interval without parameter uncertainty. i.e. get_distribution(...).ppf(alpha)
  - distribution interval with parameter uncertainty: ??? messy
- bootstrap, simulated (no specific plans yet)
  - standard bootstrap, e.g. resample obs or parametric residual bootstrap
  - simulate from asymptotic distribution of params (I think I have old experiments for this in Poisson case)
- bias correction: (nothing specific yet)
  - correct mean prediction of nonlinear function for curvature bias (Jensen's inequality), e.g. use second order correction using var. Related: lognormal mean prediction and "smearing" adjustment in nonlinear/log predictions. (aside in TSA the results for correction of mean prediction for log-models is not overwhelmingly in favor.)

For monotonic functions and links, we need reusable get_prediction for linear prediction.
Reusable means both no code duplication and no recomputation. The latter would mean that we can have different monotonic predictions using the same linear get_prediction. However, get_prediction should be relatively cheap.

To check: there might be some overlap with `margins` (after the `at` part, margins and predict_at add construction of exog_predict.

related issues and comments:
TBC

cases to start with
- `which` in Poisson for monotonic, especially `which="prob"`
- mean and `which` in NBP (and GPP) for non-monotonic, multi-parameter models (using NonlinearDeltaCov)
- multi-link models, might be also mostly the same as using NonlinearDeltaCov

user: josef-pkt
issue_7831: state: open
title: OrderedModel: get marginal effects
body: I'm working on building a logistic regression model for an ordinal dependent variable and I'm using the OrderedModel from statsmodels.miscmodels.ordinal_model. 

Maybe I'm just not seeing it right now, but with this model, marginal effects are not computed, only the regression coefficients. Nonetheless, marginal effects are computed for the regular logistic regression model (not ordered) that can be fit with statsmodels.

Is there a way I can build an ordinal logistic regression model and obtain its marginal effects with statsmodels? If not, this issue is indeed a feature request.

Thanks! 
user: lventosa
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_7829: state: open
title: FAQ-D/REF: predict keyword `linear` versus `which`
body: I want to switch to using the `which` keyword instead of the `linear` keyword in predict.
However, we still need the `linear` keyword for consistency across models in generic code.

merged PR #7597 added a deprecation warning for `linear` keyword in CountModel predict, but it triggers in generic code like get_prediction for single link models.

**So, for now:**
"linear=True" dominates the `which` keyword, and we return the linear predictor independent of the value of `which`.
The default for predict that support `which` is switched to `linear=None`.

CountModel.predict applies to Poisson and other count models in `discrete_model`. All of them use the inherited predict, except for the new Poisson method.
 ZeroInflated model only have the `which` keyword and no `linear` keyword. But we don't have generic post-estimation for it yet.
ZI predict currently have `'mean', 'mean-main', 'linear', 'mean-nonzero', 'prob-zero, 'prob', 'prob-main'` as which option.
mean and linear are standard, all others are specific to ZI.

For future
We need a convention for multi-link, multi exog models.

BetaModel has currently "mean"  "precision"  "linpred"  "linpred_precision"
linpred is inconsistent name for linear.

The same will apply to new get_prediction, although we might not need to add a `linear` keyword.
GLMResults.get_prediction has a `linear` keyword but it is essentially ignored.



user: josef-pkt
issue_7828: state: open
title: Forecasting using a Moving Average/Median
body: Maybe I'm just not seeing it in the tsa api, but is there a function that forecasts e.g. observation $T+1$ by taking an average or median of the last $T-p:T$ observations for some positive integer $p$ (and for $p=0$ just take a random walk)?  I know it's trivial, but it would be a nice convenience to avoid having to write a function for this myself :)
user: cgarciga
issue_7827: state: open
title: REF/Design hessian_factor in multi-parameter models, tuple or array
body: We need a consistent convention for how we return score_factor and hessian_factors for multiple parameters

Tuple would be better, because then we don't have to pack and unpack the array each time. If the array is fortran ordered, then we could just use a view on the array and don't need memory copies.
Usage will mostly require that we multiply individual `factors` by the corresponding exog.

betareg.BetaModel uses array, symmetric 2 by 2
draft version for GPP and NBP in #7825 also use arrays
somewhere (?) I use tuples

For generic models, score_factor and hessian_factor will be a combination of distribution derivatives and (inverse) link derivatives (chain rule).

user: josef-pkt
issue_7826: state: open
title: BUG: hessian_factor in GLM has wrong sign as derivative
body: **edit** this is essentially a duplicate of #4714, problem is also in OLS

I'm running into problems with sign convention for hessian_factor and discete/MLE models

hessian_factor should correspond to the second derivative, however GLM reverses the sign to make hessian factor positive.

in PR #7825 I need to reverse the sign for GLM for score_tests
https://github.com/statsmodels/statsmodels/pull/7825/files#diff-f12277aa8b6608078f5075e33fd09d2df74bd8c54bf7c34c7a7c99cc150abc55R221

however MLEInfluence follows the convention of GLM and uses positive hessian_factor

Changing sign GLM.hessian_factor is not backwards compatible, but there might be only minimal use of it by users.

user: josef-pkt
issue_7824: state: open
title: ENH: (theoretical) noncentrality parameter implied by model, data generating process
body: I'm using now Monte Carlo simulations to check how good hypothesis tests are.

AFAIK, we don't have anything to compute the theoretical noncentrality parameter for a parametrically specified model DGP used in simulations.

example

The return of Hosmer-Lemeshow type test for poisson includes several estimates of `nc`.
Test method delegates to statsmodels.stats.diagnostic_gen.test_chisquare_binning

Why is `nc_umvue` negative and much smaller than the other estimates?
I guess the theoretical nc=0 in this simulations which is under the null of correct specification.

```
noncentrality = <class 'statsmodels.tools.testing.Holder'>
    nc = 0.0
    confint = array([1.0000000000000e-100, 8.0294528361405e+000])
    nc_umvue = -1.8074724688857908
    nc_lzd = 0.5320879218523682
    nc_krs = 0.9121507231754883
    nc_median = 1e-100
    name = 'Noncentrality for chisquare-distributed random variable'
```
user: josef-pkt
issue_7823: state: open
title: ENH: Standard Errors for Nonparametric Estimators, lowess
body: It would be wonderful if the nonparametric estimators (esp. lowess ) offered to return confidence intervals instead of just providing fitted values.
user: nickeubank
issue_7822: state: open
title: ENH: batched summing in loglike and score with lower memory
body: A bit similar to out-of-memory computing. #1963
case large nobs, medium k_vars

Compute parts that are used in fit in batches, e.g. use a batched sum to aggregate loglike_obs and score_obs and possibly hessian.
In irls, we would need a batched WLS.

I'm not sure how much we will save for loglike. In some cases it might be mainly computing linpred = x * b in batches.

Currently neither loglike_obs, score_obs nor hessian can be computed for other endog and exog than are in the model attributes. That means loglike cannot just call loglike_obs for a subset of observations.

score_factor and hessian_factor do not include the individual exog/beta derivatives. However, these "factors" depend on the individual observations, e.g. Gaussian/OLS depends on residual, Poisson depends on mean mu.


user: josef-pkt
issue_7821: state: open
title: ENH: score_test for multi-link and multi-parameter (multi distribution args) models
body: add score

My current score_test for models has two versions

- availability of restricted params and full, unrestricted score and hessian
  - this is independent of the composition of params, and would also work for multilink models
  - However, one of the ways to compute restricted params is `fit_constrained` which currently does not work for models with extra params or multi-links.
- variable addition score tests with score and hessian constructed from the score_factor and hessian_factor
  - This currently works only for one-parameter models like GLM, for which score_factor and hessian_factor are 1-dim.
  - For models with several links/exog or with extra_params, score and hessian need multiple derivatives and factors.
    This is currently not implemented.

Models with extra params that are constant, e.g. NegBin and GPP
We need to extend score_test for the mean model. However, 
If the constant extra-params case is the null model, then we could test exog dependent, varying parameter as in the multi-link case.

Multi-link models: Models with exog dependent dispersion or `scale`, BetaModel and #7793
One possible specification test is testing that scale (extra params) is constant, i.e. coefficients of (slope) explanatory variables are zero.
Implementation would need to handle multiple score_factors and hessian_factors.

Multi-part models: e.g. zero-inflation which uses internally two submodels to delegate to.
Possible specification test: constant versus varying, exog-dependent zero-inflation. 
Implementation may be able to delegate to submodels.
#7597 includes specification test for poisson about poisson versus zero-inflated or zero-modified poisson






user: josef-pkt
issue_7819: state: open
title: BUG/REF: Holdertuple `__repr__` prints large arrays, maybe shorten or skip
body: In some cases we add large arrays to the results of hypotheis tests or similar returned in a HolderTuple.
My current example" test_chisquare_binning in #7597 

In notebook, the `HolderTuple.__repr__` prints the full arrays, which can be large.
It's a bit annoying because two thirds of my current notebook length are some of these large arrays.

There should be a way to mainly show the main results, either by skipping long results or using nump's abbreviation of printing large arrays.

user: josef-pkt
issue_7815: state: open
title: ENH/REF: follow-up to merges for 0.14
body: I'm planning to merge some PRs that will need follow-up before the 0.14 release.
This is mainly because I have to bring different parts together and I want to postpone decisions on a final implementation and api. And for now I'm more focused on the overall structure than verifying every detail.

The two big areas will be more post-estimation to existing models, and new categories of models.

General task is API and docstring cleanup and review.

companion to roadmap #7720

## post-estimation

- [x] #7597
  - needs more unit tests, and general review and cleanup
- [ ] #7825
  - high line coverage, but needs unit tests for more cases, e.g. Probit, and against hypothesis tests as in comments
  - (not immediate followup) extension of score_test to models with extra params (NB), and multilink models (BetaModels)
  - (not immediate followup) extending fit_constrained and offset to other models, especially multilink models
- [x] #7758
   - undecided: func_args, implement or drop?
   - eventually allow transformation classes as func
   - undecided: user facing method, test_nl, predict_nl or more generic
   - future: use for get_prediction  DONE
   - needs feedback from internal use cases, possible changes
- [ ] #7870 
- [ ] #7895 
- [ ] #7907
- [ ] #7912
- [ ] #7929
- [ ] #7951
- [ ] ...
- [ ] ...


## new models

- [ ] #7973. 
  - not fully finished, cleanup, raise in unsupported parts, check/test or disallow offset and exposure
  - this includes more unit tests for generic get_prediction/delta-cov, se, confint against Stata margins,
  - get_diagnostic is also inherited, but I only checked and used the plot
- [ ] ...
- [ ] #8034 
  mainly doc improvements and warnings/exceptions #8327
- [ ] ...

## stats

- [x] #8166
  - review docs and api again, maybe add notebook
  - could benefit from a few refactoring
- [ ] ...

Cleanup inconsistent API, deprecations, still to do:

- discrete fittedvalues
- get_prediction 1-dim or 2-dim
- GLM predict `which`, get_prediction Results class


TODO: across the code of most PRs:

- docstring will need review and fixes
- additions to rst docs if not automatic, e.g. new XXXResults classes
- unit test needs more coverage for extras, eg. offset. Maybe refactor and streamline some unit tests. We still get more automatic coverage for new infrastructure like score_test through internal code use like diagnostics, e.g. variable addition, reset tests.


user: josef-pkt
issue_7814: state: open
title: ENH: linalg tools cov_params and partitioned matrix inverse for sub matrix or block
body: (one of the tools that shows up very often in different places. I have no overview what we have)

Suppose we are only interested in the cov_params for a set of parameters of interest. 
We are not interested in the other parameters directly, e.g. if we have a large number of fixed effects similar to absorbed OLS.

An application in the nonlinear setting is when we use sparse matrices. When we have a large number of parameters, then we don't want to construct the full cov_params if we are only interested in some parameters of interest.
(I think we cannot implement sparse matrix support without adding some tools to avoid full cov_params.)

related issues and applications

- #5410 orthogonalizing part of exog
- two-step estimation (like Heckit ?) Murphy Topel 
- asymptotically or expected orthogonality, block-diagonality of cov_params/EIM: eg. some dispersion models.
  - trivial submatrix extraction (I used it in Het MLE models)
- absorb/sparse in GLM: I still don't know how we will do this.
- It shows up in score/LM tests and GMM, even if I often just work with the full matrices in those case.

In contrast to orthogonalizng exog and absorb, here I want something that is directly defined for cov_params or similar cross-product matrices. Although in some cases the implementation might use orthogonalizing exog or score_obs, but that would be just an implementation detail.





user: josef-pkt
issue_7813: state: open
title: ENH: heteroscedasticity in GLSAR, cov_type HC or HAC ?
body: How does the current GLSAR work under heteroscedasticity?

Can we use sandwich cov_type with it?
Estimate of AR polynomial `rho` is consistent but not efficient. Is that true?

Can we add heteroscedasticity to the estimator, i.e. GLSARHet?

#647 status of GLSHet
#7639 MLE for heteroscedasticity and dispersion

partially related: 
There is a literature on score/lm tests for heteroscedasticity and serial correlation that is robust to the other.
(Not implemented AFAIR, but, I guess, cov_type in auxiliary regression does it.)
user: josef-pkt
issue_7808: state: open
title: ENH/DOC: GAM, splines extrapolation and setting outside, exterior knots
body: I don't find anymore how far I got with setting exterior knots in GAM and splines.

We need a doc example for how to set exterior knots and what they imply for extrapolation.
I don't remember much

blog post with extrapolation for splines and similar with mgcv
https://fromthebottomoftheheap.net/2020/06/03/extrapolating-with-gams

stats.stackexchange question:
https://stats.stackexchange.com/questions/548655/cubic-spline-extrapolation

user: josef-pkt
issue_7807: state: open
title: ENH: Het Models with heteroscedasticity depending on mean
body: currently just an idea,

extension to #7793 het models as multilink models
and issue #7639

The multilink/Het models assume that the variance/scale/dispersion depends on exogenous explanatory variables.

In Poisson and GLM except gaussian, the variance depends on the mean. The Het models and GLSHet do not allow directly an iterative fittedvalues as explanatory variable in the var function.

The idea is to allow to replicate something like a variance function that is a power or polynomial in mean, estimated simultaneously with the mean function, as in Poisson or GLM.
(AFAIR, some econometrics literature for GLM started out with nonlinear weighted least squares. Wooldridge?, Cameron/Trivedi? or Davidson/MacKinnon?)

We can do FGLS as a user, but no predefined model that allow this.
2-step WLS is relatively easy, but estimating varfunction by OLS is not efficient.

Inference is not clear: 

- If we use numdiff score and hessian, then derivatives will have a component where mean parameters directly affect variance and, I think, we lose orthogonality in cov_params and mean parameters will get "distorted" to get a better fit for the variance.
- If we use WLS or iterated FGLS, then cov_params will take variance function as fixed. However, one of the regressor terms in the variance function is endogenous and a generated variable. (Maybe that doesn't matter in practice, given the weights, we are still in a LEF, which should be fine if weights don't become "weird", e.g. degenerate.)

One usecase of this would be as diagnostic, e.g. which variance function (in GLM terms, v = a + mu**p, would fit better.

GaussianHet allows for link function for mean, so we can use exp mean function (log-link) to estimate a Poisson/NegBin mean function with different variance assumptions.

The alternative is to use GMM, or an M-estimator directly defined by moment conditions/estimating equations instead of loglike.
GLM allows estimating the mean function based on moment conditions, using variance function as weights, but does not allow estimating variance parameters simultaneously.


update:
In several of my examples for Het, I used the same explanatory variables in mean and variance functions.
This is a more general case. What is missing is to impose cross-component restrictions, so that linpred of variance is a polynomial of linpred in mean function.
In this interpretation, we don't have a generated variable problem in `exog_scale`, but we have the non-zero cross-derivatives from the parameter restrictions, and extra terms in the hessian.
(related: There was something for Mean in Garch that breaks orthogonality.)




user: josef-pkt
issue_7806: state: open
title: Sparse solver for OLS
body: In OLS estimation, when the number of discrete factors in the model is large, the estimation is too slow.
It is better to add sparse solver to speedup the calculation because LSH of the model is sparse for this case.
user: chunkaowang
issue_7805: state: open
title: FAQ: comparing Het models to R gamlss and vgam
body: I'm trying to figure out unit tests for GaussianHet against R.
mnle gls doesn't seem to estimate a variance function depending on general explanatory variables.
I managed to get a heteroscedastic normal model with gamlss and VGAM

The following is based on a simple small sample case
My example has only 9 observations with 2 + 2 parameters. So, we are far from "asymptotic equivalence" and small differences in algorithm or small sample corrections can have a large effect on the results. 
(update: I tried our hidden GLSHet and the estimates are in the same neighborhood, but not very close.)

**params:**
Agreement on parameters is only at a few decimals, closer with vgam if I reduce convergence tolerance. 
Reducing convergence tolerance for gamlss does not much increase the agreement with gamlss.

**bse, cov_params:**
vgam uses IRLS and reports EIM standard errors. In the example they are not very close to OIM, full hessian standard errors.
Standard errors of vgam are close to inverting the submatrix of the hessian that corresponds to mean parameters.

Gamlss reports standard errors close to ours, i.e. OIM, also at similar decimal agreement as params.

**parameterization:**
Both, vgam and gamlss use loglink for the standard deviation, (standardization "scale")
I use, loglink for varianced, i.e. square-scale.
With loglink our reported var parameters are just twice the reported std parameters.
 
**Stata**
Stata's new hetregress uses the same approach as ours for MLE.
e.g. in the simple stata doc example, confint of both mean and var params agree at around 4 decimals, rtol at 1e-4 to 1e-5

user: josef-pkt
issue_7804: state: open
title: ENH: helper function for guessing an interpretable dispersion parameter, Monte Carlo
body: use case: suppose I want to simulate a BetaBinomial or Negative Binomial model. Can we find dispersion parameters based on basic ex-ante information, e.g. average excess dispersion, pearson chi2 scale is 5 or 10.

I think I do implicitly something similar when I compute the start_params for dispersion, eg. in count models.

simplest case:

only categorical variablesin exog:
for Monte Carlo we specify params based on
- conditional mean in cells
- dispersion across some subsets of cells, e.g. high dispersion versus low dispersion counties in the US, dispersion differences in gender (male have poisson dispersion var / mean 50% higher than female)


This goes in the direction of full support for get_distribution for simulated Power analysis and parametric bootstrap or Monte Carlo.


user: josef-pkt
issue_7803: state: open
title: DOC: example oneway or multiway anova, GaussianHet, HC cov_type
body: example for using GaussianHet in PR #7793 

Assume we only have categorical explanatory variables as in anova, and variances differ across some of the categorical variables.

For oneway, we have specific test, Do we have extras, mean confidence interval, ...?
For oneway meta-analysis we have prespecified variance of each sample and we can use WLS.

For estimated fixed effects, we can use HC3 cov_type to correct standard errors and inference.
If we estimate parameters for each cell, e.g. in 2-sample and oneway anova with full dummy set. there is no weighting in individual parameters.
For pooled parameters, weighting might improve efficiency, similar to meta-analysis.

So we can compare oneway Welch, ... results with FGLSHet and GaussianHet, and OLS with HC3.
and misspecified/incorrect case OLS with nonrobust standard errors.
and we could compare it to StudentTHet if there are "outliers" or heavier tails.


I was comparing my results for GaussianHet with gamlss
related question: https://stats.stackexchange.com/questions/297323/what-is-the-correct-way-to-fit-sigma-formula-in-gamlss-to-add-a-per-dependent

We can extend this to overdispersed, heteroscedastic GLM and discrete, e.g. with BetaBinomial or NegBin, GPP.
And BetaModel for fractions. (and GammaHet for positive endog). These two are skewed cases coming from the support of the distribution.









user: josef-pkt
issue_7802: state: open
title: ENH: multicollinearity, identify sequential collinear relationshop
body: R stats has a (weirdly named) `alias` function that seems to identify sequential collinear relationships in exog

https://stackoverflow.com/questions/69585496/is-there-a-python-equivalent-of-r-stats-alias-function
https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/alias

related 
#2380 my large multicollinearity PR, which IIRC also has one-versus-all measures, e.g. eigen vectors
#6268 includes sequential OLS, but I didn't rework/verify that in my last round
#6935 PR to add fit method with sequential dropping, R/Stata style

target here would be a sequential identification of linear relationship, similar to QR, or Gram-Schmidt






user: josef-pkt
issue_7798: state: open
title: BUG: disallow misleading cov_type HCx in models other than linear regression models
body: In GLM and discrete Models, all hcx return the same cov_type, which (IIRC) is HC0 (of HC1 ?), i.e. standard Huber-White sandwich without any corrections for hatmatrix, ...

Based on a twitter thread, I found this blog https://datacolada.org/99
Blog post points out that HC3 is much better than HC1 in OLS.

We might be able to add the hatmatrix correction to GLM and so switch to proper HCx.
(I don't remember having seen that in the literature for GLM.)

First step add warning if HCx, x>0 is used. (Need to verify which one. I usually use HC0 for plain sandwich).

(Related, I recently looked briefly at cluster robust CRx again, similar to bias-reduced CR in GEE)




user: josef-pkt
issue_7797: state: open
title: ENH: censored and truncated models with heteroscedasticity with loc - scale
body: https://search.r-project.org/CRAN/refmans/crch/html/crch.html
tobit model with heteroscedasticity, common censoring thresholds

just a detail

If loglike depends on cdf, then we need derivatives. similar to CDFLink.

In CDFLink, scale is fixed and we only have the mean parameter wrt which we need derivative.
With heteroscedasticity we have loc, scale, and possibly extra parameters of the distribution (df in t-distribution).

If we have a loc-scale family with no extra parameters, then we can standardize the variable, `(y - m) / s`, and then we can use the standard version of the distribution with a single parameter and scale fixed at 1.
I think this should work out that we don't need additional derivatives.

Same applies for OrderedModel, and binary model where heteroscedasticity introduces a nonlinear term with identifying restriction at one point.

In #7793, I keep loc/mean and scale as separate parameters in the distribution




user: josef-pkt
issue_7793: state: open
title: ENH: add multilink models and distribution.dfamilies
body: closes #7778

This will be a continuation of #7778 

now starting with distribution dfamilies
similar to genmod families but focused on MLE with scipy optimizers

I'm planning to add cleaner parts of #7778 based on MultiLinkModel to this PR
user: josef-pkt
issue_7792: state: open
title: Current status of the regularized linear regression?
body: Dear StatsModels developers,

Can you please tell me if it is currently possible to do regularized linear regression in statsmodels? According to [the documentation](https://www.statsmodels.org/0.6.1/generated/statsmodels.regression.linear_model.OLS.fit_regularized.html), it should be possible to do, but my naive attempts do not work. The following code throws the `NotImplemented` error upon printing the results summary. If it is possible to use this functionality, can you please suggest how, or direct me to an example notebook. Briefly scanning through the existing example notebooks I was unable to locate anything suitable
```

import numpy as np
import pandas as pd
import statsmodels.formula.api as smf
import statsmodels.stats.api as sms

url = 'http://vincentarelbundock.github.io/Rdatasets/csv/HistData/Guerry.csv'
dat = pd.read_csv(url)

# Fit regression model (using the natural log of one of the regressaors)
resultsREG = smf.ols('Lottery ~ Literacy + np.log(Pop1831)', data=dat).fit_regularized(L1_wt=0.0, alpha=1000)

# Inspect the results
print(resultsREG.summary())
```
user: aleksejs-fomins
issue_7789: state: open
title: ENH: multipart formulas for multipart models
body: related #6858 using formulaic and formula discussion

For multipart, multi-exog models, #7778, I would like a simple multipart formula, e.g. just splitting on `"|"`
formula 
`BetaBinomial("counts ~ a + b + c | a", data)` 
which defines
endog = counts
exog = matrix(1 + a + b + c)
exog_scale = matrix(1 + a)

last term can be dropped if only constant
`BetaBinomial(`counts ~ a + b + c ", data)` 
implies
exog_scale = None, which get translated by model.__init__ to exog_scale =np.ones

`THetModel(("counts ~ a + b + c | a | 1", data)
T model with heteroscedasticity and constant `df` parameter

what would be nice is constraints
`THetModel(("counts ~ a + b + c | a | = 5", data)
T model with heteroscedasticity and `df` parameter fixed at value 5
(currently in #7778 only implemented for TModel )

extra, maybe:
We could optionally add names for the parameters, parts of multipart formulas

`BetaBinomial("counts ~ a + b + c | scale ~ a", data)` 
name of second parameter is "scale".  There is no endog data related to `scale`
`BetaBinomial("counts ~ a + b + c | precision ~ a", data)` 
name of second parameter is "scale".
user: josef-pkt
issue_7788: state: open
title: printing a `Results` object should print out it's summary 
body: the [`linearmodels`](https://bashtage.github.io/linearmodels/) package has this feature where printing out a model after `.fit()` has been called will automatically printout it's `summary`. wondering if this can be easily added to `statsmodels`.
user: pepelovesvim
issue_7786: state: open
title: ENH: heteroscedastic variance/dispersion function estimation, 4th moment
body: (another semi-random idea)

- scale estimated by pearson chi2 for gamma is often not a good estimate and differs from scale estimat by MLE. #7525
- heteroscedasticity robust estimators of variance parameters has often worse small sample performance than imposing model restrictions on 4th moment.
- if we use a (GLM) gamma model to estimate variance function, then it implicitly uses variance function as square of expected value.
- Wooldridge was arguing in several articles in favor of estimators or tests that are robust to 4th moment assumptions (variance of variance) (but that uses empirical 4th moment)

basic question: 
Can we improve on pearson chi2 scale estimate by using weighted mean of squared pearson residuals?

A bit similar to shrinkage estimation for variance (second moment matrices):
We should use estimators for the variance function that implicitly downweight outlying observations (which are not outliers, but can have a large influence on estimators, especially with heavier tails.)

Basic idea is to improve efficiency (not consistency or bias) of variance estimator which can have better behavior in small samples.
(That's different from inference where we need to correct the variance, cov_params of the parameter estimators.)

similar to WLS with helpful, but misspecified weights. We gain efficiency, but still need to use robust cov_types for inference.

#7639 two part models with estimated heteroscedasticity/dispersion.




user: josef-pkt
issue_7784: state: open
title: ENH: finite support discretized counts, unbalanced ordinal model
body: random idea while thinking about beta-binomial #2632

designer/flexible distributions for counts with finite support.

~~2~~ 3 possible models:

- DiscretizedCount with observation specific upper bound on the (discretized) distribution, latent variable is independent of upper bound.
  maybe with different discretization schemes
  - dropout: same cutoff for all obs, but truncated when individual (exogenously) drops out.
  - cutoffs depend on n_trials: e.g. finite support of latent variable distribution and it is split evenly across opportunities
    (I'm not sure how that would work. Increments for discretization would likely change)
- unbalanced ordinal model (not all observations have the same number of choices, ordinal extension of Binomial with unequal n_trials), 
   - example: questionaire differ in nested coarseness, known splits of categories. some get 5 scale, some get 3 scale, the 3scale is a coarsening of the 5 scale partition of the latent variable space, e.g. (0,1) -> 0, then 1st cutoff of coarses scale corresponds to 2nd cutoff of finer scale.
   - example: only 2 types of questionaires, unrestricted cutoffs. Then, we would have to estimate separate cutoffs for each questionaire type, but a different number of cutoffs for each. (This might be feasible to implement)
- maybe we can just use a truncated version of an existing discretized count or ordinal distribution, (that should corresponds to the "dropout" version)
 
(example: political spectrum, continuous support on [-3, 3], answer is for discretized scale, but coarseness of scales differ)
(example: movie, book,,, ratings: merge ratings based on 3, 4, and 5 star scales, and numeric out of 100)
(example: grades for exams on different scales)

Related: Conditional Logit in unbalanced panel where we condition on the total number of units of a panel unit.


Maybe also related to survey questions where respondents can mark multiple answer. (Of the following 10 answers, mark all that apply to you. Maybe not for the answer, but for the count of how many marks a respondent made.)


and to get more distracted:
endogenous coarsening by topic experts
example rate out of 100.
respondents answer in 10 point intervals if they don't know the topic
respondents answer on a fine scale if they know the topic

(How many words for snow do eskimos/inuit have? https://www.k-international.com/blog/which-language-has-the-most-words-for-snow/ )

user: josef-pkt
issue_7783: state: open
title: ENH: observation-level random effect  models, MixedMixin
body: good generic name OLRE
too long as name for a base class

Harrison, Xavier A. "A comparison of observation-level random effect and Beta-Binomial models for modelling overdispersion in Binomial data in ecology & evolution." PeerJ 3 (2015): e1114.

This would make another good generic class for a new category of models, especially for quick development of prototype models.

Essentially, I have the basics for it in MixedMixin. I initially wrote it for panel/cluster data with unobserved heterogeneity, but one of my examples has Poisson with observation specific random intercept. (I guess with exp link, this would be Poisson with multiplicative log-normal random effect)

We should be able to use NegBin and BetaBinomial as test cases.
Then we can extend to other observation level mixture distribution, PIG, Poisson-normal, ? ... Some of those are pretty commonly used, but messy to compute, because likelihood requires integration.


user: josef-pkt
issue_7781: state: open
title: Remove fully ranked assumption in inter_rater.fleiss_kappa
body: #### Is your feature request related to a problem? Please describe
Fleiss kappa computation does not require a fully ranked data set (Fleiss, 1971, p. 378). The current assumption in the code should not be there.
#### Describe the solution you'd like
Change:
```
...
n_rater = table.sum(1)
n_rat = n_rater.max()
#assume fully ranked
assert n_total == n_sub * n_rat

#marginal frequency  of categories
p_cat = table.sum(0) / n_total

table2 = table * table
p_rat = (table2.sum(1) - n_rat) / (n_rat * (n_rat - 1.))
...
```

to 
```
...
n_rater = table.sum(1)

#marginal frequency  of categories
p_cat = table.sum(0) / n_total

table2 = table * table
p_rat = (table2.sum(1) - n_rater) / (n_rater * (n_rater - 1.))
...
```

user: ultysim
issue_7779: state: open
title: ENH: Add classification performance measures. Closes #1577
body: like #1650 without table flip keyword 
user: sneznaj
issue_7778: state: open
title: ENH: add dispersion models, heteroscedasticity with full MLE
body: **update** this has been generalized to MultiLinkModel, not restricted to mean-variance models anymore.

see #7639

getting started

code based on miscmodels TLinearModel and `__init__` from BetaModel.

class structure not clear yet

- 2-parameter version is largely generic, currently Gaussian and Gamma as subclasses, currently only link_scale, no link for mean
  gamma will need link for mean for endog and mean > 0
- T model 3 parameter model with possibility to constrain 3 parameter to fixed value, only link_scale, mean mean link, and df inoriginal parameterization,, no link to force > 0 (or > eps)

Most likely distribution specific code, loglike, ..., will be delegated to new family classes (or genmod families if available) #7769

user: josef-pkt
labels: superseded: Used for PRs with content that is superseded by a newer PR
issue_7769: state: open
title: ENH: collection of distribution.families, separate from models
body: Our models hardcode loglike, score, score_obs, hessian, .... This makes it difficult to reuse the code across models.
genmod.families are only for one-parameter LEF, with auxiliary support for scale.

The idea is to create new distribution classes in support of models, complementary to scipy, and not subclasses of scipy distributions. Same idea as for genmod families, but covering a larger set of distribution.

- parameterization as used in models, possible several different parameterization e.g. as in GAMLSS
- derivatives for MLE, first and second deriv w.r.t parameters
- other derivatives, e.g. CDFLink needs derivative of cdf, pdf with respect to the rv, e.g. needed for censoring/truncation models.
- get_distribution, code to convert parameters to scipy parameterization
- restrictions on parameters and possible or allowed links.
  I'm not sure whether the families should contain only the info or also the links. (*)
- additional properties of the distribution not available in scipy, e.g. hazard rate for lifetime models.

usecase:

- different models for the same underlying distribution family, e.g. gaussian with heteroscedasticity, truncated models, ...
- reusing loglike, score and similar with different exog, endog than in the model.
  - we needed and need this for prediction "which" in count models. We will need more when we get `get_prediction` inference
- reusable model classes, similar to GLM that works with all supported families.
- computational efficiency/precision: e.g. avoid division by scale, if `precision = 1/scale` parameterization is used as in AFT models.

As alternative, we can write generic models based on scipy distribution classes, e.g. logpdf, cdf, ...
However, those don't include the extras that we want to have if available, e.g. analytical derivatives.
We can delegate to scipy distributions for methods that are available and not in heavy usage, e.g. we usually implement our own logpdf.

(*) As families are currently used in GLM, I would prefer if links where separate and not accessed through the family.
e.g. for nonlinear prediction we just need the link and not the family.

However, if the link is included, then the family could be defined in the final parameterization, e.g. derivatives of the link are included in family derivatives.
GLM does not do that. Adding links and link derivatives is done in GLM class itself, Families take mean `mu` as parameter and not `linpred` (eta in GLM terminology).

I guess those distribution families would have to be private for considerable time, because design will need to change in response to our internal usage.

user: josef-pkt
issue_7765: state: open
title: SUM/REF/ENH: make OLS, Regression model more similar to GLM and MLE models
body: companion to #2107 which is for GLM

OLS, WLS are missing score_obs, and hessian as common methods.

We should add more methods that are consistent with GLM and MLE models so we don't need to write RegressionModel specific code as often.
This does not need to change any of the current methods that use computational simplifications specific to linear model.

One problem with OLS is `scale` handling, similar to GLM.

OLS/... use concentrated likelihood.
The moment conditions, score, take scale as fixed or remove it.

one possibility is to use scale as score_obs parameter, that can also be set to 1 to get moment condition without variance terms.

Similar to GLM
score and score_obs will NOT be the derivative of the concentrated likelihood (I think).

example for using generic code:
sandwich matrices, e.g. information matrix ratio and TIC, GBIC, part of #7597

score/lm test is also model specific

user: josef-pkt
issue_7763: state: open
title: SUMM/REF/ENH:  aic, bic, info_criteria standardize code, consistency across models
body: Can we cover information criteria with generic code?
Which models need special case code?

We have results method `info_criteria` now in GLM and regression.
I'm trying to add it to DiscreteResults with the addition of TIC, GBIC in #7597

So far we have used mostly definition that are model specific (or specific to groups of models) with overrides for cached attributes aic and bic.

One problem is we don't have `k_params`, instead we use `df_model` which has ambiguous definition.
e.g. MNLogit has special code because of the 2-dim params.

add results.k_params as central location ?

Next problem: effective degrees of freedom in penalized estimator.
e.g. GAMResults does not define aic, bic, it inherits from GLM
Are those ic correct? Does new QAIC apply to GAM?
aic, bic should be correct because I had added penweight selection based on it.

constrained estimation: what would be k_params? df_model, df_null, ...
All are reduced by the number of constraints, but len(params) stays unchanged.

GEEResults also inherits from GLM: Does inheriting `info_criteria` make sense or do we need to remove those ic?
GEE has qic method for quasi-likelihood aic.

RLM does not have any ic. AFAIR, they are still waiting in one of my PRs.

2 problems with inheriting `info_criteria`:

- Adding model specific extra ic:
  e.g. GLM: aic, bic are standard, then there are deviance based definitions, and qaic (with common scale)
- general/generic ic that don't apply or that require parts that are not implemented in model.
  e.g. TIC should be pretty general, but it requires the same methods as robust cov_type (e.g. score_obs) (MNLogit does have score_obs)  RLM, MixedLM, GEE don't define score_obs
  The GLM ic might not be appropriate for GEE



 

user: josef-pkt
issue_7761: state: open
title: ENH: subgroup detection, inverse regression and outlier groups
body: I recently saw an article on detection of subgroups in the sample population. I ignored the article.
We have sliced inverse regression in regression.dimred.

The matching literature has tools on analysing the exog structure when matching treated and untreated groups.
In Hosmer-Lemeshow type tests we choose groups in "ex ante" fashion, i.e. without specifically targeting group selection to specific questions.

simple case that leads to binary variable:

Suppose we have a regression problem and we detect that there are many outliers, observations for which the model doesn't fit well. So we get a binary outlier variable.

Question: 
Are outliers random, independent of exog? or
Are there specific groups, or range of exog, that help to identify outliers?

not clear what to do
e.g. throw a huge number of exog in a Logit model with outlier candidate indicator variable as endog.

(I've seen somewhere something that used outlier detection to estimate, essentially mixture models, groups of observations with different behavior.)


user: josef-pkt
issue_7760: state: open
title: ENH: oneway anova, meta-analysis, control charts and outliers/influence
body: making some connections in the hope of reusable code and methods, and maybe getting some better structure.

All 3, oneway anova, meta-analysis, control charts, have in common that in the basic model the expectation of the endog is constant.
The basic regression model is a constant-only model.

anova focuses on testing
meta-analysis on estimation of mean and variance/confint
control charts focus on outliers.
tukey-hsd or similar for multiplicity control

 possibilities:

- combine meta-analysis and oneway methods to get full results
- use equivalent model to get/inherit outlier influence measures
   - we still need outlier, influence measures for hypothesis tests.
 - control chart methods include oneway methods for outlier detection (design uses false alarm rate)
   - we should be able to use predictive distribution directly. (although interest is only on 1 or 2 quantiles) (*) 

example:
multi-site experiments or multi-site monitoring. 

(*) see deltamethod or similar for confidence intervals on non-linear prediction

For symmetric or gaussian distributions we can reuse RLM and robust.
Outlier robust models for GLM and discrete are still missing. (truncated MLE as one generic approach ? outlier dummies?)

user: josef-pkt
issue_7759: state: open
title: DOC review generic_mle notebook
body: https://www.statsmodels.org/stable/examples/notebooks/generated/generic_mle.html

has a warning about df_resid, but summary looks correct

likely related to df_null issue, df_null should be k_constant + 1 = 2 for extra param

also: 
the last section Numerical precision sounds too negative on numerical derivatives
(comment on Twitter saying our numdiff has low precision pointing to the notebook statement)

MASS::glm.nb does not use joint hessian that includes dispersion parameter.
Some difference will come from different algorithm


user: josef-pkt
issue_7756: state: open
title: ENH/TST: llnull for penalized, GAM
body: see also #2580 GAM follow-up

I'm adding a smoke test for llnull in GAM, null, llnull is inherited from GLM. #7751 
It works but there is not unit test to verify correctness.

base.tests.test_penalized has unit tests for `null` model for classes using PenalizedMixin.
I didn't check the status

user: josef-pkt
issue_7754: state: open
title: TST: check xfail in test_discrete
body: changes and refactored to xfails in #4983

It's not clear whether all this xfails are valid.

(I found one that I will remove. Test class used the wrong test in a method instead of using the working test in super/mixin class.

not implemented test for  `normalized_cov_params` is general xfail. 
Do we support `normalized_cov_params`  as cached/stored hessian attribute, or do we kill it?

user: josef-pkt
issue_7752: state: open
title: TST/ENH/REF  review offset in Regression model, OLS
body: We have offset in linear regression model as internal support, introduced for fit_regularized
It adjust likelihood computation with essentially `resid = resid - offset`

needs review and  possibly enhancement
usage would also be for fit_constrained for affine constraints.


I found offset as non-publicly available kwarg in `OLS.__init__` #7741
user: josef-pkt
issue_7746: state: open
title: ENH: GAM penalization splines with shape restrictions
body: I just saw this by chance

penalties to impose shape constraints like monotonicity or convexity
https://github.com/dswah/pyGAM/blob/master/pygam/penalties.py

The penalties are on differences in consecutive coefficients. AFAIK, this only works with knots on a regular, equal-spaced grid.
Restrictions for splines on equal spaced grid are simpler than irregular knots, e.g. based on quantiles.

AFAIR, we support equal spaced knots, but we don't have any convenient support of the penalization for it.
(I guess it would work generically using Penalization classes with user made penalty matrices. #7285)

pygam uses sparse matrices. Pure penalization splines have a large number of knots.

user: josef-pkt
issue_7745: state: open
title: BUG/FAQ-D: inheriting docstring of Model.from_formula does not work
body: I'm currently trying to add the docstring to from_formula classmethod in OrderedModel using
`from_formula.__doc__ = Model.from_formula.__doc__`
This does not do anything. `__doc__` is None.

trying in a notebook I get.

```
OrderedModel.from_formula.__doc__ = Model.from_formula.__doc__
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-68-b4916dbf917f> in <module>
----> 1 OrderedModel.from_formula.__doc__ = Model.from_formula.__doc__

AttributeError: attribute '__doc__' of 'method' objects is not writable
```

There doesn't seem to be a problem for regular methods and the class docstring, but it doesn't work for the classmethod.

Quick browse:
Classes that define their own from_formula but have an explicit docstring, do not have any from_formula docs.
Besides current OrdereModel, the same appears in 
https://www.statsmodels.org/dev/generated/statsmodels.regression.process_regression.ProcessMLE.from_formula.html
https://www.statsmodels.org/dev/generated/statsmodels.discrete.conditional_models.ConditionalPoisson.html
https://www.statsmodels.org/dev/generated/statsmodels.othermod.betareg.BetaModel.html
https://www.statsmodels.org/dev/generated/statsmodels.regression.recursive_ls.RecursiveLS.html

however Appender works

https://www.statsmodels.org/dev/generated/statsmodels.regression.rolling.RollingWLS.from_formula.html

```
    @classmethod
    @Appender(Model.from_formula.__doc__)
    def from_formula(
        cls, formula, data, window, weights=None, subset=None, *args, **kwargs
    ):
```



user: josef-pkt
issue_7741: state: open
title: BUG/ENH: add warning for unknown `kwargs` in `model.__init__`
body: **update** fragility when adding new keywords
One strong reason to add kwarg check is that users might run different versions of statsmodels. If we add a new keyword, then it will work on the new version of statsmodels, but silently ignore it when running the same code with an older version.

---

Just came up on the mailing list

using `GLM(... weights=myweights)` is not doing anything.
kwarg is just swallowed without warning

I think we can check for the keywords that base models support liike missing, ...

Raising an exception is to strict because it's too fragile, e.g. automatic `exog_xxx` handling

```
if kwargs is None:
    self.check_kwargs(kwargs)
```

only works on leaf models to avoid problems with subclassing.

My guess is that it's pretty safe adding it to GLM. discrete is messier because of the larger class hierarchy (and we don't have offset in all those models yet.)


a bit related: I would rather drop the "subset" keyword. I guess there are no unit tests for the case with extra arrays in `__init__`.
(I never even used it.)
user: josef-pkt
issue_7734: state: open
title: MAINT: formula.api is outdated, missing models
body: It's missing most newer models, at least since GPP and zeroinflated.

I'm not sure all of them have verified formula support.
Some/moste two-part models have only partial support, for exog, but not for exog_xxx
user: josef-pkt
issue_7733: state: open
title: BUG: OaxacaResults uses print instead of returning text in summary
body: summary methods should not use print
user: josef-pkt
issue_7728: state: open
title: ENH: 1-dim summary of dfbetas
body: (just a quick idea)

I recently saw a reference to a function using `max(dfbetas, axis=1)`, i.e. largest change of dfbeta in and observation.

The question is how can we make this more invariant to scaling.

possible relative change `dfbetas / params - 1` which is a problem if params is close to zero, i.e. insignificant.

other possibility: use dftvalues which standardizes params by bse 
If we take out nobs in bse we would get something like an dfeffect_size

Related: dfpvalues
There is a literature on the influence of outliers and observations on inference. I never looked at the details for that

Motivation:
We should add more `get_influence` based on generic `MLEInlfuence` in 0.14. 
Should work in many cases automatically, eg. in Beta regression, but needs unit tests.

**update**
related: 
- d linpred using dfbetas effect on linear prediction or on mu, the former is also a 1-parameter linear combination of dfbetas for given exog, e.g. we could use exog.mean(). I guess closely related to dffits for the mu at each observation.
- we can do this also for other statistics, e.g. distribution parameters mu and precision in BetaModel

user: josef-pkt
issue_7722: state: open
title: I want to know the value of "Orthogonalization Disturbance Term" in VARResults.
body: I don't know if the English translation is correct, but guess the term.

I would like to know the value of "Orthogonalization Disturbance Term" in IRF. How can I find it in "statsmodels.tsa.vector_ar.var_model.VARResults"?

The background is that I want to check the response of the IRF to time series data such as price earnings ratio.
I have a question like this.

user: loySoGxj
issue_7720: state: open
title: SUMM roadmap 0.14 josef
body: Just trying to get a rough overview what I'm planning for 0.14

- diagnostics, score tests
  - missing pieces in discrete (avoiding PRs that get too large or widespread).
  -  main difficulty will be writing unit tests. R and Stata don't have comprehensive diagnostics, in R spread out across many 
   packages
  - `score test`, missing pieces for discrete "basic" models, Logit, Probit and Poisson, (other models later)
  - `fit constrained`, should be easy/generic after adding missing pieces in discrete.
  - `get_influence` mostly generic for "basic" discrete after missing pieces have been added
  - `get_diagnostics` basic diagnostic classes, where we can add more tests as they become available.
  - specific groups of diagnostic tests, e.g. reset tests
  - delta method, predict, test #2296, updated in #7758
  - `get_prediction`, 7166 -> #7597 includes it for Logit and Poisson
  - profile confint    #1791
- merged
  - [x] #7597 merged
  - [x] #7825 merged
  - [x] #7758 merged
  - [x] #7870 merged
  - [x] #7895 merged
  - [x] #7900 merged
  - [x] #7907 merged
  - [x] #7912 merged
  - [x] #7929 merged
  - [x] #7951 merged
  - [ ] ...
  - [ ] ...
- almost finished PRs from last year's updating
  - [x] #3183 merged
  - ...
  - ...
- distribution:
  - extend archimedean copulas to k_dim > 3, pdf for k_dim <= 4, unit tests for k_dim=3
    - [x] #8633 
    - [x] #8642
- add new models
  - multilink and dispersion models #7793 
  - heckman
  - Tobit, generic exogenously censored/truncated ?
  - truncated, hurdle count models
    - [x] #7973 merged
    - [ ] follow-up: still needs work, check extra options, cleanup
  - treatment effect 
    - [x] #8034
    - [ ]  looks feasible #3647
  - MVGenExtremeModel ? Copula regression model
  - accelerated failure time ?
  - user friendly models for e.g. Penalized Mixin, sure independence screening #7128
  - mixed mixin maybe #5502
  - ...
- stats ?
  - [x] #8166
  - effect size ?
- Refactoring ?
- GMM ?


for **What's new in 0.14**

- new count models: hurdle, truncated
- new treatment effect
- new features in count models, e.g. offset
- new post-estimation, discrete models, GLM, BetaModel 
  (see overview issue 7707 for coverage, notebook poisson for overview of features)
  - score_test
  - get_prediction 
  - get_distribution
  - get_diagnostic (Poisson diagnostic and base diagnostic for other count models)
  - get_influence: how much of it is new?
  - internal (at least for now): delta method, _wald_test_nonlinear, ...
- stats: 
  - new functions for poisson rates
- ...

smaller additions
- information criteria, gbic, tic, .... (im_ratio advertise or not?)
- ...


user: josef-pkt
issue_7719: state: open
title: ENH/SUMM: diagnostics for Logit/binary/binomial
body: We don't have an issue specific for diagnostics for Logit, binary or binomial. I read Binomial articles mainly to understand GLM diagnostics.
Also, I want to park two R packages below, that look useful based on their docs. I haven't looked much at details yet.

- gof, hosmer-lemeshow type, ...
- residual diagnostics
- plots
- predictive evaluation measures, classification
- influence outlier, generic GLM
- specification tests like RESET test, score tests dispersion, (artificial) variable added, ...
  mostly generic GLM, special link function tests for [0, 1] links, cdflinks

related issues, e.g. https://github.com/statsmodels/statsmodels/issues?q=+is%3Aopen+logit+label%3Atopic-diagnostic+

package `rms`
https://stats.stackexchange.com/a/207512/14187
strange question with negative weights https://stats.stackexchange.com/questions/160638/should-i-distrust-the-g-o-f-for-a-logistic-regression-with-weights-perfomed-with
https://cran.r-project.org/web/packages/rms/rms.pdf
blog post overview examples for rms https://www.nicholas-ollberding.com/post/an-introduction-to-the-harrell-verse-predictive-modeling-using-the-hmisc-and-rms-packages/

package `LogisticDx`
https://cran.r-project.org/web/packages/LogisticDx/LogisticDx.pdf
hasn't been updated since 2015, looks like it has a good list of Binomial diagnostics.
From the docstrings it is not clear what is for Bernoulli/binary and what assumes counts binomial.
Much seems to be for discrete exog, finite number of unique exog.
(2 issues https://github.com/dardisco/LogisticDx 1 bug, 1 explanation of difference in HL to Stata and book)
user: josef-pkt
issue_7718: state: open
title: MarkovAutoRegression Forecasting Out-of-Sample, forecast() and get_forecast()
body: To whom it may concern, if there is not away to forecast future values using the MarkovRegression and MarkovAutoRegression forecast class, can some one edited the tsa.statespace documentation to reflect this reality?
If an out of sample forecast can be produce such as one-step ahead, can an example be added to the documentation to suggest how one would go about doing so?
Thanks so much
user: rericroberts
issue_7716: state: open
title: How To Approach Fit Method Selection for SARIMAX Models
body: Flagging this question from Cross-Validated:

https://stats.stackexchange.com/questions/544155/how-to-approach-fit-method-selection-for-sarimax-models

I am building SARIMAX models using the statsmodels library in Python. Some of these models are single variable models and some are multi variable models.

In general, I am finding that I get very different model fits depending on my selection for the method parameter in the fit method. The documentation lists the following options: (1) Newton-Raphson, (2) Nelder-Mead, (3) Broyden-Fletcher-Goldfarb-Shanno (BFGS), (4) limited-memory BFGS with optional box constraints, (5) modified Powell’s method, (6) conjugate gradient, (7) Newton-conjugate gradient, (8) global basin-hopping solver

Are there any rules of thumb in terms of which method (i.e. solver) should be used in which circumstance?

For example, might one say that one should use the default for single variable models and the conjugate gradient method for multi-variable models?

I can provide an example of the kinds of differences I am seeing in model fits if that would be helpful.
user: dkent287
issue_7715: state: open
title:  glm, poisson, nonlinear: omitted variables that only affect dispersion (and possibly constant)
body: semi-random thought

https://bookdown.org/egarpor/PM-UC3M/glm-diagnostics.html
mentions at the end of the page that we can use PCA in GLM

What's the effect of dropping noise variables? i.e. small PCA factors.
similarly
What is the effect of dropping pure random effects (uncorrelated with fixed effects and constant across observations)?

My guess is that we get unbiased slope coefficients, the constant compensates for the curvature (Jensen's inequality).
Similar result should show up as for log-linear model. conditional expectation has a (constant) variance term added.
Dispersion should be larger, as for example Poisson versus NegativeBinomial (mixture distribution Poisson-Gamma).

For inference on slope coefficients, we would mainly need to include over dispersion, e.g. QMLE or HC cov_type.

I'm not sure there is anything to do.

Main idea would be to check variables that have negligible effect on mean, but affect variance/dispersion.
Similar to linear model diagnostic omitted variables for mean versus heteroscedasticity as function of some extra exog.
(But in nonlinear function noise has an effect on conditional expectation.)

Might be relevant when we get into dispersion modelling.


user: josef-pkt
issue_7712: state: open
title: ENH: GLM compare_lr_test_nonnested, Vuong test for link selection
body: just an idea, no references, should not be difficult

specification test for links in GLM

The plan is to add more links for use with GLM, e.g. parametric links #624
This is now easier in the case of CDFLink which just need a scipy distribution (merged PR #7287 )

one possibility are variable addition test , either Wald or score. We don't have alternative specific artificial regressors yet, RESET type test also have power in that direction.

alternative is to compare several **models with different link functions**.

easiest is just to rank models, e.g. aic (ranks by llf if k_params is unchanged)
or by gof measures, or fit of predictive distribution.

LR-type tests
- nested, e.g. test t versus normal or cauchy
- not nested logit versus probit (shape test), or logit versus (c)loglog (asymmetry)

we still need more flexible link functions, e.g. log-F

Problem with not nested LR test is that distribution is not the same as in the nested case.

as companion to `compare_lr_test` #5372 we could add a not-nested version (or option) Vuong test #3897
(also related #3101 chisquare and deviance tests for variable selection)

also related 

#4264 automatic model search



user: josef-pkt
issue_7711: state: open
title: ENH: Add multicollinearity tools (rebase/improve #2380)
body: - [X] closes #2380 
- [X] tests added / passed. 
- [X] code/documentation is well formatted.  
- [X] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

<details>


**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in main and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/main -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>

user: bashtage
issue_7707: state: open
title: FAQ: overview extra post-estimation results methods
body: (non-TSA models)
#1353 similar issue, more general

extra results methods are available for some models, e.g. `get_influence`, `get_diagnostic`, `get_prediction`, `get_distribution`. `get_margeff`

right part of table are tools, 
`sf, hf` is score_factor and hessian_factor

empty cells are either missing or I have not checked or verified

model | infl | diag | pred | distr | margeff | score/lm | . | sf, hf | offset |
---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
OLS | + | - a | + | + |  | + | | / |
WLS | | | |  |  | + |
GLM | + | - | + | + | / | + | | + | + |
GLMGam ?   | | | |  |  |
RLM   | | | |  |  |
  | | | |  |  |
**discrete** | ++ | | R++ | R++ | R++ | R++ |  |  |  |
*Binary*  | + |  | + | ++ | + | + |  |  |  |
Logit       | ++ |  | + | + | + | + | | + | + |
Probit     | + |  | + | + | + | nt | | + | + |
MNLogit | - |  | - |  | + |
*Count*  | a | R++ | + | + | + | + | | | + |
Poisson  | ++ | ++ | + | ++ | + | + | | + | + |
NB          | - | + | + | ++ | + | / |  | - | + |
NBP        | + | + | + | ++ | ? | + | | + | + |
GPP        | + | + | + | ++ | ? | + | | + | + |
*ZI*        | + | + | + | + |  |  |  |  | / |
ZIP         | + | + | + | ++ | - | / |  | - | / |
ZINBP    | + | + | + | ++ | - | / |  | - | / |
ZIGPP    | + | + | + | ++ | - | / |  | - | / |
*truncated*    | - | + | + |  | - | / |  | - | / |
Hurdle    | - | + | + |  | - | / |  | - | / |
  | | | |  |  |
**othermod** | | | | |  |
Beta | + |  | + | + |  | + | | + | - |
model | infl | diag | pred | distr | margeff | score/lm | . | sf, hf | offset |

footnotes
`++` : implemented directly
`+` :  inherited, no specific method
`R++`: implemented in Results class, not in model class, might not be supported in all subclasses
`a` : separate functions for specification tests and plots
`/` : means partial
`-` : either explicitly missing or raises NontImplementedError (if inherited doesn't work)
`nt` : not unit tested, inherited


Note: GLMGam inherits from GLM, but has not been checked yet #8519 

user: josef-pkt
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_7706: state: open
title: ENH/Design: multivariate results class
body: open question

current MNLogit, VAR, MultivariateOLS (in MANOVA) have 2-dim params.

We will get multipart models with multivariate endog, but 1-dim params.

example copula models, e.g. MVGEV copula genextreme
maybe two stage models under full MLE, e.g. heckman, or bivariate models for endogenous regressors

OrderedModel is in between, 1-dim params and 1-dim categorical/integer endog, but predict is multivariate.

The main part is to handle two or more columns in predict, resid, ...
With 1-dim params, all Wald inference remains unchanged.

Do we want "conditional" prediction, e.g. E(y2 | y1, x)?
Is there something like "marginal" prediction E(y2 | x) ?  (integrated over y1 random part)

predicting functions of y1, y2, e.g. linear combination or nonlinear (drought index in extreme value theory), joint probability events e.g. prob(y1 > t1, y2 > t2)

Is there enough to make a separate base results class?


bivariate copula gev might be a good starting point, also for developing copula regression model

user: josef-pkt
issue_7705: state: open
title: ENH: margins, marginal means for mixedlm
body: https://glennwilliams.me/blog/2021/09/07/estimating-marginal-means-and-pairwise-tests-by-hand-in-python
blog post that explicitly computes marginal means using statsmodels compare to R lme4 + emmeans

I didn't read the blog post, only brief skimming.
I never looked at this for mixedlm, but if it is only for the fixed effects then it should be "generic".

Note: discrete margins don't have specific handling of categorical exog. (just dummy variables)

Main task I guess is to write "predict_at" for categorical variables and categorical interactions.

Aside: t_test_pairwise cannot handle interactions.



user: josef-pkt
issue_7695: state: open
title: ENH/SUMM/Review (multi-) collinearity checks and warnings for cov_params 
body: We should add a warning and identify cases where we have singular cov_params or other problems with positive definiteness

PR #3099 add PD check to hessian inversion in base model fit
#3367 issue with negative diagonal elements in Logit cov_params.

In some cases we allow for or produce singular cov_params, eg. when using pinv in regression, glm irls, or similar. And in cases with constraints.

current case: #7694 quantile regression raises in unindented way when exog is rank deficient. We will fix this, but then we can get a pinv result with singular cov_params (pinv-regularized).
We need some warnings in those cases/

Related: I mentioned this in several comments. Stata has in some cases rank or similar measure for cov_params and not (just) for exog. That can be important in nonlinear models, discrete, glm.

I'm not sure what the general status is. We don't have a consistent policy for this.
user: josef-pkt
issue_7691: state: open
title: ENH/Design  module structure for IV, GMM, endogeneity
body: so far we don't have any models that allow for endogenous regressors (outside of sandbox)

base GMM classes need to be somewhere central to avoid circular imports.

Models that handle endogenous regressors could either go in the directory that corresponds to the exogenous-only case, or we could add a separate folder for "endogeneity".

`linearmodels` package handles the linear case, we have additionally nonlinear and non-normal cases, e.g. with binary or count variables.

Full information MLE can be handled by two- or multi part or multi equation models. 
We might still need separate classes to handle the presence of one endog as regressor of other endog, e.g. predict needs to be recursive or reduced form when we don't take value of auxiliary endog as given (e.g. is given if we predict effect of treatment if treatment==1, ATT)
-> recursive simultaneous versus multivariate models

IV, partial information treat some equations as auxiliary. 
For the binary treatment case, we also have `statsmodels.treatment` as planned location. (case with conditional independence or whatever it is called, IPW instead of IV)

e.g. we have a design conflict in `heckman` class because it is both FIML and IV type.

GMMPoisson additive and multiplicative should be GMMExp because it only assumes exponential mean function. (Wooldridge: "IVPOIS should be called something like IVEXPON")
i.e. it's not specific to count distribution. 
(But I guess we might want to specialize if we add optimal instruments.)

cases where at least one endog is discrete, binary/bernoulli, multinomial, count

- binary or multinomial selection model: outcome only observed if selected is true, or for the selected choice
- binary or multinomial regressor, outcome always observed
- count variable with continuous endogenous regressors, e.g. case for control function, ivpoisson
- bivariate Probit ?
- ...

related: for general cases we will need something like MixedMixin for simulated MLE or quadrature (used in Cameron/Trivedi count book 2nd)

to get started

- heckman in PR  #7636
- GMMExp in sandbox
- count with control function for continuous regressors #7689  (looks relatively easy)
- `treatment` clean up GMM with IPW

I guess a separate directory maybe `simultaneous` would be better so we can more easily share code between different models, e.g. for non-normal, nonlinear mean models in discrete and othermod.
`endogenous` or `endogeneity` is a difficult sell.
`simultaneous` however does not directly characterize `IV` methods, but we don't just have IVs. 

In, the unlikely, case we get system of simultaneous equations, then they could be in `system`.
`multivariate` is for the purely exogenous cases (but multivariate is also linear)

What location for IV2SLS and LinearIVGMM?






user: josef-pkt
issue_7689: state: open
title: ENH: GLM, discrete with endogenous regressors, control functions
body: (back to econometrics)

#1742 ivpoisson, GMMPoisson

control function approach with linear function for endogenous regressor doesn't look so scary.
https://stats.stackexchange.com/questions/281848/control-function-for-iv-poisson-regression
https://www.stata.com/features/overview/poisson-regression-with-endogenous-variables/

Cameron Trivedi count book 2nd edition, p. 402ff has docvis example that compares Poisson-CF, NB2-CF and additive and multiplicative version of GMMPoisson.

main caveat: They recommend bootstrap standard errors, because it is a two step estimation with generated regressor in second stage.
(maybe there is a Murphy/Topel or GMM asymptotic distribution for the two step procedure, see next comment)


related also fully parametric bivariate/multivariate models. 
Cameron Trivedi also include a case with an endogenous categorical regressor modelled through simulated MLE for mixed multinomial.

IIRC, I have seen some time ago somewhere the use of splines for the control functions, to have less restrictive parametric assumptions.

partially related
#7636 heckman procedure with endogenous binary treatment but linear model at final stage.

user: josef-pkt
issue_7688: state: open
title: REF: family.predict family.fitted are misleading names
body: both are just alias for link and link.inverse

`fitted` might be ok, but 
`predict` computes linpred from mu, which is unrelated to any interpretation of `predict`. It is use in `_fit_irls` start_params handling.

searching genmod folder: 
"family.predict" only shows up in that one case of it.
"family.fitted" shows up 4 times in code and once in a docstring.

AFAIR, I never used either of the two, using link and link.inverse is more descriptive and not misleading.






user: josef-pkt
issue_7687: state: open
title: ENH/Design: GLM - add endog type (discrete, continuous, family) specific results statistics
body: Some results statistics in GLM are only available for some types of endog or some families.
We should find a way to add them to the results class as family dependent results.

examples

- pseudo-rsquared #7682 McFadden pseudo R2 is only valid for discrete endog.
- standard prediction table only makes sense in binary, possibly multinomial, case #6252, 
  (Hosmer lemeshow/ binned cases work on predicted probabilities not classification choice)
- `get_diagnostic` diagnostics methods that depend on discrete/continuous or on family, #7597, comments in old PR #7166, e.g.
  - continuous PIT versus discrete plots, 
  - specification tests for specific families,  e.g. Poisson specific overdispersion tests. or binomial specific once we have them

for simple cases like adding McFadden pseudo R2, we need a flag for discrete distributions.
In many cases we fix scale=1 for the same list of cases.

Tweedie is for mixed type, continuous with a mass point. For McFadden pR2 this means it is `not discrete`.

We can have a consistent interface if we add generic methods names like `get_diagnostic` and we return an instance of  family/type specific classes.

This avoids special case methods like pred_table, that apply only in some cases, and would be noise in others. We avoid instance methods as much as possible to avoid pickling problems.
user: josef-pkt
issue_7686: state: open
title: REF/ENH: refactor GLM null, add something like llr_pvalue
body: GLM `null` is not very well written, needs review and refactoring.

some comments in
https://github.com/statsmodels/statsmodels/issues/6953#issuecomment-877833161
https://github.com/statsmodels/statsmodels/pull/7682#issuecomment-912715826

In #7682, I added start_params to the null model fit. This uses endog.mean() and is correct only without weights and without offset exposure. We might to improve this or make it more robust.
(In contrast to discrete models, we didn't have many failures with llnull in GLM.

For llr_pvalue or equivalent:
Current llnull is computed using scale of the full model. This does not correspond to LR test for full MLE.
But there are versions using deviance or sum of squares where keeping scale constant is recommended.

user: josef-pkt
issue_7680: state: open
title: REF/Design: numdiff, clarify and clean up behavior with 2- or multi-dimensional parameters
body: related to vectorized scalar param #1783 which will be added in #7287 as separate functions.

Behavior of numdiff `approx_fprime` and of `approx_fprime_cs` in case of 2-d params are not well defined, but don't raise in all cases.
Trying to make changes to this in #7287 causes test error with approx_fprime_cs in MarkovSwitching unit tests.
approx_fprime breaks and raises a broadcast error for that type of test case (2-d single row params)

approx_fprime creates diagonal matrix in vectorized scalar param case, so it works. #7287 will remove those cases from link functions.

This should be clarified, and maybe raise an exception for cases that we don't want to handle.

related: 
Even when cases work with the implemented numdiff function, it does not necessarily use the correct options. 
e.g. My initial change to handle vectorized scalar params, dropped the usage of epsilon scaled by individual params, using a single epsilon instead.
My guess would be that approx_fprime_cs does something similar in some cases where it works.


user: josef-pkt
issue_7678: state: open
title: RuntimeWarning: overflow encountered in matmul   return err.T @ err
body: #### Describe the bug

I have been running holtwinters exponential smoothing across my dataset - starting with 40 weeks and increasing the the window to the entire dataset (150 weeks). There are some weeks there I get the following error

C:\Users\bmussa\Anaconda3\lib\site-packages\statsmodels\tsa\holtwinters\model.py:80: RuntimeWarning: overflow encountered in matmul
  return err.T @ err

This results in the slope, level and seasonality parameters to behave funny as below:
![image](https://user-images.githubusercontent.com/75366970/131663199-a13b0fcd-bb24-4c81-a318-b41574736104.png)

Normally they look like this
![image](https://user-images.githubusercontent.com/75366970/131663238-031d3234-ed0a-421d-87ff-6e64a0689f70.png)

The issue is only for a handful of weeks in the dataset where I increase the window to include the following week.
Is there something wrong in my data that is causing that to happen? This impact my simulations, forecast and confidence intervals.

you can find my code below

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt 
from statsmodels.tsa.api import ExponentialSmoothing 
from datetime import datetime, timedelta
from numpy import percentile

def add_ci(alpha_val, lower_ci,upper_ci):
    for i in range(-1,-5,-1):
        dataset = simulations[i]
        # calculate 95% confidence intervals (100 - alpha)
        alpha = alpha_val
        # calculate lower percentile (e.g. 2.5)
        lower_p = alpha / 2.0
        # retrieve observation at lower percentile
        lower = percentile(dataset, lower_p)
        print('%.1fth percentile = %.3f' % (lower_p, lower))
        # calculate upper percentile (e.g. 97.5)
        upper_p = (100 - alpha) + (alpha / 2.0)
        # retrieve observation at upper percentile
        upper = percentile(dataset, upper_p)
        print('%.1fth percentile = %.3f' % (upper_p, upper))
        print(i)
        holt_winter[lower_ci].iloc[i]= lower
        holt_winter[upper_ci].iloc[i]= upper

def one_step_ci(simulated_data, alpha_val,selected):
    dataset = simulated_data
    # calculate 95% confidence intervals (100 - alpha)
    alpha = alpha_val
    # calculate lower percentile (e.g. 2.5)
    lower_p = alpha / 2.0
    # retrieve observation at lower percentile
    lower = percentile(dataset, lower_p)
    print('%.1fth percentile = %.3f' % (lower_p, lower))
    # calculate upper percentile (e.g. 97.5)
    upper_p = (100 - alpha) + (alpha / 2.0)
    # retrieve observation at upper percentile
    upper = percentile(dataset, upper_p)
    print('%.1fth percentile = %.3f' % (upper_p, upper))
    if selected =='upper':
        return upper
    elif selected=='lower':
        return lower


data = pd.read_csv('C:\\Users\\bmussa\\SharePoint\\Insight - Documents\\Data Science\\Sales forecasting\\forecasting_categories.csv')

data['date']=pd.to_datetime(data['date'],format='%d/%m/%Y') 
#data.date= data.date.apply(lambda x: x.date()) 
metrics = ['sessions',]#'net_revenue','order_count',


forecast_data =[]
hw_data = []
max_date = "2021-08-30"
for metric in metrics:
    today = '2019-01-07'#'2021-06-28'
    while today <= max_date : #"2021-08-09"
        print(today)
        # holt winters method
        train = pd.concat([pd.DataFrame(data[data['date']<today]['date']),pd.DataFrame(pd.to_numeric(data[data['date']<today][metric]))], axis=1)
        test = pd.concat([pd.DataFrame(data[data['date']>today]['date']),pd.DataFrame(pd.to_numeric(data[data['date']>today][metric]))], axis=1)
        y_hat_avg = test.copy()
    
        fit1 = ExponentialSmoothing(np.asarray(train[metric]) 
                                    ,seasonal_periods=6
                                    ,trend='mul'
                                    , seasonal='add'
                                    , damped_trend=True
                                    ).fit()
        
        for fit in [fit1]:
            pd.DataFrame(np.c_[fit.level,fit.slope, fit.season]).rename(
                columns={0:'level',1:'slope', 2:'season'}).plot(subplots=True)
        plt.show()
        
        four_weeks = fit1.forecast(4)    
        y_hat_avg['Holt_Winter'] = fit1.forecast(len(test))
        
        simulations = fit1.simulate(1
                                    , repetitions=10000
                                    , error="mul"
                                    , random_errors="bootstrap"
                                    )
        
        forecast_dict= {'metric': metric
                , 'date' : (datetime.strptime(today,"%Y-%m-%d")+timedelta(days=6)).strftime('%Y-%m-%d')
                , 'last_value' :  train[metric].iloc[-1]
                , 'first_period' : four_weeks[0]
                , 'second_period' : four_weeks[1]
                , 'third_period' : four_weeks[2]
                , 'fourth_period' : four_weeks[3]
                , 'lower_50': one_step_ci(simulations, 50, 'lower')
                , 'upper_50': one_step_ci(simulations, 50, 'upper')
                , 'lower_80': one_step_ci(simulations, 20, 'lower')
                , 'upper_80': one_step_ci(simulations, 20, 'upper')
                }
        
        forecast_data.append(forecast_dict)
    
        today =(datetime.strptime(today,"%Y-%m-%d")+ timedelta(days=7)).strftime('%Y-%m-%d')
        if today > max_date:
            simulations = fit1.simulate(4
                           , repetitions=10000
                           , error="mul"
                           , random_errors="bootstrap"
                           )
            holt_winter = pd.DataFrame()

            holt_winter = pd.concat([pd.DataFrame(data['date'].dt.strftime('%Y-%m-%d'))
                         ,pd.DataFrame(pd.to_numeric(data[metric]))
                         ,pd.DataFrame(fit1.fittedvalues, columns=['fittedvalues'])
                         ,pd.DataFrame(y_hat_avg['Holt_Winter'])
                         ], axis=1)
            holt_winter['metric'] = metric
            holt_winter['holt_winter_forecast'] =holt_winter['fittedvalues'].fillna(holt_winter['Holt_Winter']) 
            
            holt_winter['lower_ci_80']=0
            holt_winter['upper_ci_80']=0
            
            holt_winter['lower_ci_50']=0
            holt_winter['upper_ci_50']=0
            
            add_ci(20,'lower_ci_80', 'upper_ci_80')
            add_ci(50,'lower_ci_50', 'upper_ci_50')
            hw_data.append(holt_winter)          


forecast_df = pd.DataFrame(forecast_data)
hw_df = pd.concat(hw_data)

forecast_df[['metric', 'date']]= forecast_df[['metric', 'date']].astype(str)
hw_df[['metric', 'date']]= hw_df[['metric', 'date']].astype(str)

forward_estimate_df = pd.merge(hw_df, forecast_df, how="left", on=['metric', 'date'])
forward_estimate_df['lower_50'] =forward_estimate_df['lower_50'].fillna(forward_estimate_df['lower_ci_50']) 
forward_estimate_df['upper_50'] =forward_estimate_df['upper_50'].fillna(forward_estimate_df['upper_ci_50']) 
forward_estimate_df['lower_80'] =forward_estimate_df['lower_80'].fillna(forward_estimate_df['lower_ci_80']) 
forward_estimate_df['upper_80'] =forward_estimate_df['upper_80'].fillna(forward_estimate_df['upper_ci_80']) 

user: bilalmussaukfd
issue_7675: state: open
title: Tidelift
body: Hi statsmodels folks, this is Lisa Martin from the finance department at NumFOCUS. I just wanted to let you know that  we have several sponsored projects that have signed up maintainers with Tidelift and are collecting income from them on a monthly basis. I checked recently on behalf of our affiliated projects and saw that statsmodels has some income available as well. It is a very small amount, but if you are interested, please reach out to me at finance@numfocus.org and I can let you know how to proceed. Have a great week! 
user: lisamartin72
issue_7674: state: open
title: SUMM/ENH effect size measures, collection and api
body: effect size measures need another round

we have code in issues and PRs, but no overview and convenient systematic api for those

e.g. stats.effectsize module has only private functions and is not in docs
https://github.com/statsmodels/statsmodels/pull/7084/files

R package `effectsize` looks good to get collection of functions and semi-generic tie in to models, e.g.
https://rdrr.io/cran/effectsize/man/F_to_eta2.html
Their vignettes might also have a good collections of examples to replicate, e.g.
https://rdrr.io/cran/effectsize/f/vignettes/from_test_statistics.Rmd

for oneway anova we have a good collection of helper functions
https://www.statsmodels.org/dev/stats.html#oneway-anova

Related: adding good summary or summary_frame to results of hypothesis tests, confint and similar, where we can add options for effectsize

motivation, example:
#6598 (possible candidate to still add to 0.13)
adding effectsize based on chi2 to contingency table with unit test using R `effectsize` `chisq_to_phi` function





user: josef-pkt
issue_7666: state: open
title: WIP/ENH: add weibull parametric survival
body: see #4217
based on my old notebook example for weibull, add link function for scale.

I'm not sure yet whether we want to model precision or scale
https://github.com/statsmodels/statsmodels/issues/4217#issuecomment-903787859
or have both versions.

GenericLikelihood needs refactoring with df_null, better handling of extra params in null model
see e.g. https://github.com/statsmodels/statsmodels/issues/1723#issuecomment-903155722

This has currently simple implementation of right censoring.
see #7638 for more censoring options.




user: josef-pkt
issue_7665: state: open
title: Would it be possible to report t stat instead of standard errors in the summary2 function?
body: summary2 provides very nice feature to present results of multiple regressions.  I am wondering it is possible to report t stat instead of standard errors in the summary2 function? 

Adding one more option to summary2 to indicate report se or t will be very helpful(like esttab in stata). If I understand the source code correctly, we just need to add one more argument to the _col_params to allow for the choice of se or t, and simply revise the summary2 accordingly.

user: yszhuang
issue_7664: state: open
title: Mixed Effect Logistic Regression
body: #### Is your feature request related to a problem? Please describe
Mixed-effects logistic regression comes in handy when accessing the binary outcome in groups. It is often used in biomedical sciences.

#### Describe the solution you'd like
Implementation of mixed-effect logistic regression

#### Describe alternatives you have considered
I used libraries `pymer4` and `lmer4` in R to get the results. 

#### Additional context
If the library has an alternative please point it out to me. I wasn't able to find it in the docs. 
user: valentynbez
issue_7660: state: open
title: `SVAR` summary() minor bug
body: `SVAR` throws up an 

`AttributeError: 'SVARResults' object has no attribute 'k_exog_user'`

when I try to print the summary of the fit.

Since `SVAR` doesn't accept any exogenous variables, I assume `k_exog_user`=0? In any case, when I manually assign `k_exog_user`  to the `SVAR.fit()` object, the summary prints just fine.
user: anthonybritto
issue_7658: state: open
title: Exogenous variables behaviour
body: #### Describe the bug

I have been testing manually the behaviour of SARIMAX with exogenous variables.
1) Passing the target as exogenous variable gives perfect predictions, as expected.
2) However, passing the difference with the previous value, in a SARIMAX((1,0,0), (0,0,0,12)) does not generate perfect predictions. I would expect it to learn a perfect model:

Yt = 1*ar.L1 + 1*Exog

Where the exogenous has been defined as Yt - Yt-1 = Yt - ar.L1.

However, the results are not the case, and the coefficient for the exogenous is near 0,5 instead of 1, which was the expected.

Note that this behaviour is similar for different series with similar target.

#### Code Sample, a copy-pastable example if possible

```python
mmm = SARIMAX(df_train["y"], exog=df_train[extra_numerical_features_final],
                  order=(1,0,0), seasonal_order=(0,0,0,12), trend="n")
model = mmm.fit(disp=False)
forecast = model.predict(start=model.nobs, end= model.nobs+n_out-1, exog = df_test[extra_numerical_features_final])
print(model.summary())    
```
<details>

</details>

If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

A model with coefficients 1 for ar.L1 and 1 for the exogenous.

Instead Im getting:

![image](https://user-images.githubusercontent.com/43989119/130064659-54a619bd-ee23-45ee-8e64-81b450d7a53c.png)





user: richardgg93
issue_7657: state: open
title: Fleiss kappa function is broken
body: #### Describe the bug

Hi,

Fleiss kappa measures agreement between raters over categories. Values below zero mean worse-than-random agreement; values up to 1 represent increasingly better-than-random agreement. The range is [-1..1].

For three raters that perform identical, uniform ratings over three categories, we should have kappa around +1. We can represent this with:

`a = np.array([[1,1,1], [1,1,1]])
`

However statsmodels returns a -0.5, i.e. the agreement between raters is worse than random:

`>>> statsmodels.stats.inter_rater.fleiss_kappa(a)
-0.49999999999999994
`

Increasing the sample size brings this implementation's output closer to -0, where it should bring the Fleiss kappa even closer to +1:

`>>> statsmodels.stats.inter_rater.fleiss_kappa(np.array([[2,2,2], [2,2,2]]))
-0.19999999999999993
>>> statsmodels.stats.inter_rater.fleiss_kappa(np.array([[100,100,100], [100,100,100]]))
-0.00334448160535114
`

This is way off the expected behaviour - it might be better for this function to be disabled until it's fixed, if a fix isn't round the corner (note similar issue from 2018, #4387)

user: leondz
issue_7655: state: open
title: IRF for exogenous variables in VARX
body: As far as I understand the documentation, it isn't possible at the moment to study the impact of the exogenous variables on the endogenous in a VARMAX setting.

Do you have any tips as to how one might do this by hand? I am looking for something along the lines of https://stats.stackexchange.com/questions/504276/how-to-assess-the-impact-of-an-exogenous-on-endogenous-variables-in-var

Thanks!
user: anthonybritto
issue_7654: state: open
title: Standard errors for IRF in VARX models
body: Comparing the documentation of the VARMAX and VAR modules, I wonder if it would be possible to include standard errors for the impulse response functions in a VARX setting?

How could these be computed in the meantime?

https://www.statsmodels.org/stable/examples/notebooks/generated/statespace_varmax.html
https://www.statsmodels.org/dev/vector_ar.html

Thanks a lot!
user: anthonybritto
issue_7647: state: open
title: REF/ENH: copula and negative correlation, what to do?
body: Of the currently implemented copulas only elliptical copulas allow for negative correlation. Most copulas require positive or non-nrgative correlation.

If cop_args are set in `__init__` then we can check that args satisfy boundary constraints. We could add this optionally also to other methods. transform function for archimedean and EV copulas have argchecks.

For **fitting** parameters to data:

- negative correlation might break estimation, 
  - for arg_from_tau we can check and raise on negative empirical tau
  - MLE optimization might not converge and fail, e.g. computation assume positive correlation parameters
    sanity check could warn if kendalls tau is negative, even when we use MLE
- do we add checks and warnings by default?
- how well are functions behaved as we approach the boundary of the parameter space?
  - plain mle optimization does not impose parameter restrictions, we would need constraint optimization (if loglike can be evaluated at params boundaries) or reparameterization. 
- problem in regression models: simple bivariate kendalls tau does not take effect of exog into account and can be misleading as indicator for correlation in copula.
 
IndependenceCopula is at the boundary of several copulas. R package has option to switch to IndependenceCopula in that case. Our computation might raise or end with nan (e.g. from zero division) in neighborhood of independence.

ignore for 0.13? except for args_from_tau checking.

What happens if we flip the sign of one marginal distribution to convert negative to positive correlation?

Are there more diagnostics to signal problems?


user: josef-pkt
issue_7646: state: open
title: ENH: multicollinearity and joint significance
body: another semi-random idea.

Can we interpret size of p-values as "importance" of the parameter?

No, in the case of multicollinearity. Individual p-values can be insignificant, but joint p-values can still be large.

One possible method that only works if the number of parameters is not large:
Compute joint hypotheses tests for all subsets of parameters. 
All subset with one element are the standard p-values. Then compute joint test for all pairs, all triples, ...

(There might be something like a Shapley value to measure the overall contribution.)

related to multicollinearity measures (e.g. contribution to significant eigen vectors) #2380

also similar
- all subset regression, trimming the search tree reduces computation
- sure independence screening.
- factor analysis (?) contribution of variables to significant/important factors.
- attempts to do variance decomposition for importance in non-orthogonal designs.

two possible cases of multicollinear variables
- complements: if we have one, then the other helps also. (what's an example for this? *)
- substitutes: we only need one of a pair or set. (e.g. highly positively correlated variables)

*) I guess, e.g. endog depends on difference between two values, but not on values directly.

user: josef-pkt
issue_7642: state: open
title: SUMM/ENH: bivariate binary models, bivariate probit, copulas
body: (some semi-random thoughts, I found an old article on my computer looking for copulas)

- copulas with discrete margins
  Peter Xue-Kun Song: Multivariate Dispersion Models Generated from Gaussian Copula Scandinavian Journal of Statistics , Jun., 2000, Vol. 27, No. 2 (Jun., 2000), pp. 305-320
  I looked only partially at discrete margins, e.g. Logit, Probit or Poisson, no prototype, experimental code yet
- Zelig has blogit, delegates to VGAM, modeled by two marginal functions and a function for odds-ratio 
  http://docs.zeligproject.org/articles/zeligchoice_blogit.html
  https://www.stat.auckland.ac.nz/~yee/VGAM/doc/binomial2.pdf
- related: can we use MNLogit directly, i.e. each choice is a rectangle in joint latent variable space. e.g. bivariate binary has just 4 cells, 
  i.e. multivariate binary is just a multinomial probability space with extra structure, similar to contingency tables and log-linear models
- ...


I already have helper functions for discretizing space of random variables for the multivariate histogram and bernstein grid methods.

Computing bin probabilites requires multivariate cdf.
In the bivariate case, we could also use copulas that have simple, explicit cdf (e.g. bilogistic ?).
If we have the basic structure for the discrete margins, then we might be able to extend it in future to other, computationally relatively simple copulas like nested and vine copulas.

Multivariate Archimedean have common correlation, i.e. are exchangable. Some asymmetries are possible with power transformation.

related possible application is for selection models, #7312 (heckprobit) #7638
 





user: josef-pkt
issue_7639: state: open
title: SUMM/ENH: two-part models with heteroscedasticity, dispersion modelling
body: (I didn't find a general issue for this)

issues
- #1949 MLE for GLS models
- #2889 GLM dispersion modelling

Beta regression has been merged. It includes two link functions, one for mean and one for precision (1/variance_scale)

The same pattern can be used for other distributions, e.g. gaussian, ...
This can include discrete models that have a dispersion parameter, e.g. Negbin and generalized Poisson.
possible alternative/complement is to add a subclass of GLM, but GLM implementation does not model scale as MLE.

We have many issues on diagnostics for heteroscedasticity, but not much on modelling heteroscedasticity.





user: josef-pkt
issue_7638: state: open
title: SUMM/ENH models with censoring and selection
body: possible target for 0.14: censored regression models, and self selection

two PRs

- #7637 Tobit with right and left censoring
- #7636 heckman selection model

censoring will be conceptually easier to get started

problem: parameter estimation is not robust to distributional assumptions, heteroscedasticity, ...
There seems to be a considerable literature on diagnostic tests for Tobit models, and extensions that remove some assumptions.
There is some overlap with duration models, parametric survival/lifetime models.

semi- or nonparametric alternatives
#600 KernelCensoredReg unfinished?  I didn't remember we have something like this, needs checking, status?

other issues:
#1950 MLE for composite models (stub issue, maybe delete in favor of similar issues)
#664 interval censored data 



Green has articles/books for censoring/selection in discrete models.

related: 
- OrderedModel: assumes censoring thresholds are unknown, and are estimated, otherwise similar censoring structure as interval censored.
- DiscretizedCount: distribution and model for interval censoring on regular grid [y, y+1), no regression model yet.
- (zero-)inflated models, eg. mixture of continuous distribution on R+ and mass point at zero. (also tweedie)
- copula to model selection/censoring and main model with dependence. (no support for discrete marginal distributions yet)



I will need to do some background reading for more than cleaning the two open PRs, e.g. diagnostics and other distributions.

user: josef-pkt
issue_7637: state: open
title: WIP ENH Tobit rebased2
body: rebased #2139 which was a rebase of skipper's original PR

conflict with tools decorator, I skipped the commit as conflict resolution
`Patch failed at 0009 ENH: Add transform decorators with tests`

needs another interactive rebase:

- do file still in scikit namespace
- tools/mat2nparray.ado is, I guess, an old, outdated version
- tools/bigmat.ado I have no idea what this is or was used for

I guess this largely predates python 3
PR does not include unit tests results files. I had removed them in 2139. 

user: josef-pkt
issue_7636: state: open
title: Heckman rebased
body: rebased version of #1935

no merge conflicts
extra commit: update pandas usage

no other changes yet.
user: josef-pkt
issue_7633: state: open
title: ENH: AICC as default for linear models?
body: According to a standard reference on model selection, [Burnham and Anderson (2004)](https://doi.org/10.1177%2F0049124104268644), the second-order correction to the AICC should be preferred to the AIC in practice. I quote (p. 270)

" [The AICC] should be used unless n/K > about 40 for the model with the largest value of K. A pervasive mistake in the model selection literature is the use of AIC when AICC really should be used. _Because AIC c converges to AIC, as n gets large, in practice, AICC should be used._ People often conclude that AIC overfits because they failed to use the second-order
criterion, AICC." [My emphasis.]

Since you have already implemented `aicc` in `tools.eval_measures`, I think it would be helpful to simply include it as an attribute of the `Results` class of linear models, perhaps even as the default "AIC measure".

Many thanks for your wonderful work with this package!

**update**

defaults remain unchanged, but we have now `info_criteria` results method with more options (in regression, discrete and GLM)
still open: AICC or equivalent for other models than linear regression models, e.g. GLM

user: anthonybritto
issue_7632: state: open
title: REF: review and refactor for computational efficiency in ZeroInflated models
body: The zeroinflated models have not been reviewed for computational efficiency in loglike and similar.
(My review before merging was mainly functional.)

- hessian uses double loop (but for triangle, and so might not necessarily be slower than the rectangular vectorized solution)
- no, or not enough temp variables, e.g. indexing occurs multiple times within methods, e.g. self.exog_infl[zero_idx, i], np.exp(llf[zero_idx]))
- zero_idx could be attached as attribute instead of computed each time
- score_obs uses numerical derivatives for probit inflation, this should be possible to be computed generically with delegating the binary model specific part to the inflation model.

also the computations are difficult to read, no visible structure (by algebraic priority rules)
user: josef-pkt
issue_7630: state: open
title: ENH/REF: OrderedModel follow-up
body: OrderedModel could be refactored and moved out of miscmodels for 0.13. I'm not sure there is time for this.

I did not find many issues related to current OrderedModel. Enhancements can wait for another release. #7069

Main refactoring and move is to subclass DiscreteModel instead of GenericLikelihoodModel. https://github.com/statsmodels/statsmodels/issues/7069#issuecomment-767307637
I guess in the first round there is not much use in having a joint superclass (subclass of DiscreteModel) with MNLogit because MNLogit uses 2-dim params.

user: josef-pkt
issue_7629: state: open
title: BUG/REF/ENH copula for 0.13 follow-up
body: current copula modules still have parts that need to be cleaned up or finished for 0.13

several items listed in merged PR #7408
following copied from https://github.com/statsmodels/statsmodels/pull/7408#issuecomment-885985621

insuffient unit test coverage in several modules https://codecov.io/gh/statsmodels/statsmodels/tree/copula_7254_rebased/statsmodels/distributions/copula

- [x] `copulas.py` still contains old code, not covered
- [ ] `other_copulas.rvs_kernel` added in last commit has not unit tests
- [x] `archimedean.py` missing test coverage mainly for tau related methods
- [ ] `elliptical.py` tau, rho related methods and tail
- [ ] `depfunc_ev.py` mainly no coverage for `_check_args`
- [ ] `extreme_value` 100% coverage
- [ ] `is_completly_monotonic` only method `is_completly_monotonic` is not covered

`distributions.tools.py` no coverage for `_ecdf_mv` function

`_ecdf_mv` returns counts and not plotting positions.

---

- naming convention, renames
  - random -> rvs ?
  - IndependentCopula -> IndependenceCopula
  - d -> k_dim (or something like that)
  -  ...
- cleanup, enhance theta methods
- empirical copula, beta copula need review, (It is where I stopped development for copula)



user: josef-pkt
issue_7627: state: open
title: ENH/REF/Design: review delegation in models with submodels, e.g. zero-inflated
body: specific part of two- or multi-part models based on full submodel instances

Zero inflated models have two submodels, one for zero inflation and one for the main count model.
Much of the details can be delegated to the submodels.

I never reviewed specifically this delegation.

e.g. why does zeroinflated have a `_hessian_logit` and does not delegate to submodel hessian?

current issue (why I'm looking at it is) formula support

- `GenericZeroInflated.__init__` delegates to super for exog handling and not directly to submodels
- no specific `from_formula` method which could delegate formula handling to submodels.
- predict uses submodel predict methods and would include formula transform if supported.
  However, some of the extra prediction `which` need more than submodel predict and might not work without explicit ndarray exog.

other issues

- what extra methods to we need in submodels to make delegation easier
  - `predict` temporarily replaces endog exog of submodel to compute loglikeobs for new, out-of-sample data
  - What do we need to support variable addition score tests for submodel exog? score_factor, hessian_factor? 
- (for later) fit_constrained transformation could be done on submodel exog. But that is more complicated because we need a new two-part model with replaced submodels with adjusted exogs and offsets.


Related: This is in contrast to two part models that do not use submodels, e.g. Beta regression has only one model but two link functions.






user: josef-pkt
issue_7613: state: open
title: BUG: stochastic trend in tsa.statespace.UnobservedComponents
body: Documentation for this function is here: https://www.statsmodels.org/devel/generated/statsmodels.tsa.statespace.structural.UnobservedComponents.html

**Problem**
The argument combinations (`trend=False`,`stochastic_trend=True`) and (`trend=False`,`stochastic_trend=False`) output different forecast results.

**Expectation**
If `trend=False`, then the boolean passed to `stochastic_trend` should not matter.

**Sample code**
```
mod1 = statsmodels.tsa.UnobservedComponents(y, level=True, trend=False, seasonal=12,
                                    cycle=True, irregular=True, stochastic_level=True,
                                    stochastic_trend=True, stochastic_seasonal=True,
                                    stochastic_cycle=True)

res1 = mod1.fit(disp=False)
forecast1 = res1.forecast(12)

mod2 = statsmodels.tsa.UnobservedComponents(y, level=True, trend=False, seasonal=12,
                                    cycle=True, irregular=True, stochastic_level=True,
                                    stochastic_trend=False, stochastic_seasonal=True,
                                    stochastic_cycle=True)

res2 = mod2.fit(disp=False)
forecast2 = res2.forecast(12)
```
For some reason I can't upload a *.csv file. GitHub is complaining that it's not supported.
user: devindg
issue_7609: state: open
title: GLMGam with patsy's built-in B-Spline Functionality
body: ### Request
Following the provided example for `GLMGam`, it appears that the b-spline terms to have penalized estimation must be input via the `smoother` argument and must be a `BSpline()` object. However, this seems to ignore the built-in specification of splines within patsy. This interrupts how I have a workflow set-up. 

### Describe the solution you'd like
Here is a simple example for a data set with columns 'x1', 'x2', and 'w.' Both 'x1' and 'x2' are binary, while 'w' is continuous (and the variable we want to use b-splines for. Currently, this would have to be input like the following:
```python
gam = sm.GLMGam.from_formula(formula="y ~ x1 + x2", data=d,
                             smoother=BSplines(d[['w', ]], df=[5], degree=[2], include_intercept=True),
                             alpha=[5])
fgam = gam.fit()
```

Instead, the solution would be something along the lines of

```python
gam = sm.GLMGam.from_formula(formula="y ~ x1 + x2 + bs(w, df=3, degree=2)", data=d,
                             alpha=[5])
fgam = gam.fit()
```

### Describe alternatives you have considered
I've written a hack-y solution that is a wrapper around `GLMGam`. Briefly it takes the patsy-specified formula and parses it. For terms not wrapped by bs(...), it passes them individual to patsy, applies the patsy calculation, and stacks them to a numpy array (to pass to `exog`). For the b-spline part, the options are read from the formula and collected in lists.

The b-spline extracted lists are then passed to `BSplines`. The the `BSpline object and extracted exog are passed into `GLMGam`.

There is probably a better solution to this, but this was my first processing attempt to get around this formula issue.  Either way, a feature like this for GLMGam would be nice to have. 
user: pzivich
issue_7606: state: open
title: Obtained Nan-s for ZINB in method summary() 
body: When I pass in zero inflated negative binomial more than 30 features I get nan-s in statistics result (summary). But if give less than 30 features I had normal parameters of statistic. 
Is there a correlation in this case?

scipy.__version__  = '1.2.1'
statsmodels.__version__ = '0.12.2'
550000 samples in data-set 


![Capture](https://user-images.githubusercontent.com/48712018/126979966-0b25b6a2-7011-443d-9bd9-abdea43880cd.PNG)

Found discussion on stackoverflow but this is not my case. Zero-samples are not separable https://stackoverflow.com/questions/62402458/zero-inflated-negative-binomial-distribution-function-nan-warning

user: VladimirovaAlina
issue_7604: state: open
title: ENH: copula model
body: We need a model class to check the arg handling in the copulas. 
An experimental version should be fine for now.

A bivariate model could use the same pattern as the merged Beta regression with two exog and two link functions.
Using extreme value copula with generalized extreme value margins should already have most unit tests for the distribution against R package evd.
Main addition will be the args/params handling and checking that the fit method works reasonably well.

Or maybe we use two submodels as in ZeroInflated count models.
The advantage is that handling multiple univariate distribution parameters is handled in the submodel class, e.g. if we have a Beta regression margin, or something like Negative Binomial with extra params. t-distribution would be a candidate because we would only support continuous marginal distributions in this round.

If we use submodels, then GEV is not yet an option for now because we don't have a univariate GEV model.

One possibility is to create a Model class that just estimates distribution parameters without exog. I did that for Discretized Model.
user: josef-pkt
issue_7602: state: open
title: ENH: outlier-influence leave-one-out statistics for variable selection
body: This is just an idea

We don't directly support generic cross-validation, i.e. pipepline as in sklearn.
We have approximate leave-one-out statistics and related statistics, aic, gcv, computed without cross-val loop.
These statistics like hat_value, df_betas are computed mostly in outlier_influence analysis.
Currently, the main/only focus of outlier-influence code are outlier and influence measures.

Can we use statistics from outlier_influence analysis for variable selection and hyperparameter selection?
e.g. is there a usage for dfbetas?
(relate to that? : #6323 #6346 sensitivity and robustness of estimates for parameter of interest. )

E.g. 
- loop over model candidates computing statistics to compare and evaluate models
- compute relevant outlier influence statistics
- provide summary measures for the models based on approximate LOOO statistics

We also have an (unused?) leave-one-variable-out statistic in some outlier-influence classes.


We do something similar already when we compute gcv for hyper parameter selection in penalized models like GAM.
In those cases we compute hat_matrix_diagonal/influence already in the model and don't need to go through outlier-influence analysis. hat_matrix_diagonal is also already computed as part of some HC cov_types (AFAIR)
I guess we could add `gcv` generically where we have a model or results hat_matrix_diagonal method.

Currently, outlier-influence statistics are computed for the consumption by users. 
We should take advantage of it also in internal use, to provide more than those statistics to users.



user: josef-pkt
issue_7601: state: open
title: ENH: `get_prediction` inferential statistics for predict_prob, delta method, count models
body: The new count models have many `which`, prediction of nonlinear functions of parameters.
But we do not have `get_prediction` for them.

Getting standard errors and confidence intervals requires delta method. In many cases, the prediction functions cannot be reduced to a univariate function and end-point transformation does not apply.

#3415 predict_cdf, predict_prob for discrete/GLM
#2296 PR for delta method and nonlinear wald test
user: josef-pkt
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_7584: state: open
title: ENH/REF Beta Regression follow-up
body: Beta regression was merged in #7543

Beta regression still needs api improvement and cleanup, part of it is to support a second exog.
For some of it we don't have a consistent pattern and similar problems in ZeroInflated models.

user interface:

- formula support for second exog, exog_precision
  - api design
  - not supported in predict, exog transformation is handled by super predict.
- defaults and input checking in predict, related to #4952    
- summary is currently the inherited default, needs to include more info

unit tests:

- no unit test for robust cov_types. has unit test for cov_type="eim"

missing functionality

- support drop data (not all arrays included yet), pickle support not checked
- offset
- rebuilding model e.g. for fit_constrained. bootstrap. Those need extra (generic) support for a second, or more exog.
- post-estimation
  - [x] get_prediction
  - gof, (pit should be easy with get_distribution)
  - predictive evaluation measures ?
  - diagnostics
  - outlier-influence
  - score_test

For much of the latter we have generic versions that work only with one exog and not auxiliary arrays.


user: josef-pkt
issue_7576: state: open
title: ENH/Design handle flexible number of args in multi-part models
body: #3903 multi-part models

We have currently two part models in ZeroInflated models and coming in BetaModel, and in other models like Mixed,

For two part models with a single endog, we have explicit exog, exog_xxx, and we need other keywords, link, link_xxx, offset, offset_xxx and possibly exposure.

For general cases, we might have more parameters of the underlying distribution and need exog and all others for each of them.
e.g. copula for bivariate needs two marginal models, in the multivariate case it can be more marginal models, and it needs to be flexible.

3-parameter distributions, or zero-modified 2 parameter distributions need 3 exog and links.
Although a parameter might not be specified as regression and assumed/hardcoded to be constant across observations.
e.g. df in t-distribution.

Options are to switch to list or tuples (that's what I started in copulas) or add flexible number of `kwargs` in generic case.

One possibility for formulas is to start with multipart formulas separated by `|` as in R (and Kevin's linearmodels), but that wouldn't affect lists of links and offsets.

(for now I stick with two exog models, which is still missing some support)

user: josef-pkt
issue_7573: state: open
title: BUG/REF/Design: non-data kwargs in `Model.__init__`
body: Should we support non-data kwargs in `base.model.Model.__init__`?

see for details: https://github.com/statsmodels/statsmodels/pull/7543#issuecomment-879166101

currently extra kwargs that are not 1-d or 2d arrays, have value replaced by None, in the missing handling.
In #7543 I change this to keeping their values unchanged and attaching them as attributes. Before that change, scalars and kwargs that are not None and without `.ndim` attribute raised an exception, and so where not supported in missing handling.
Note, from_formula always goes through the missing handling but, AFAIK, not the array/pandas interface).

3 options

- don't support non-data kwargs?  
   subclasses have to handle all non-data keywords, this is the case in current models AFAIU
- support all kwargs in base Model, this should be the case with changes in #7543 
- support all kwargs in base Model but issue a warning, (bad design or preferred design is to leave it to subclasses)


The trigger for this case was that I change `GenericLikelihodModel.__init__` to send all kwargs to the super `__init__`






user: josef-pkt
issue_7569: state: open
title: TST: unit test for `_init_keys` recreating models
body: #1732 FAQ-D issue

I almost forgot about adding `_init_keys` to a new model (Beta regression)

We could use python inspect to check whether keywords in the `model__init__` signature are also in `_init_keys` for a generic unit test helper function.

AFAIK, we currently only have unit tests for internal usage, e.g. fit_zeros, fit_constrained, some llnull, and cases where we need to recreate a model like (IIRC) imputation.

(I'm not familiar enough with `inspect` module to do it right away.)
user: josef-pkt
issue_7568: state: open
title: MAINT: Fix incorrect submodule name (statsmodels.family -> sm.families)
body: On the doc, an incorrect submodule name (`sm.family`) is specified. Changed it to the correct submodule name, `sm.families`.


<details>


**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in main and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/main -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>

user: joonro
issue_7567: state: open
title: ENH: more general way for fit constrained (GenericLikelihoodModel) - another round
body: GenericLikelihoodModel added the pattern for fixing params a long time ago, based on parameter transformation or expansion/reduction.

- The only current usecase of the GenericLikelihoodModel version, AFAIK, is in miscmodel.TModel, only for fixing some params
- statespace MLEModel uses a similar pattern for fit_constrained. AFACS also only for fixing some params
- some count models like NB, NBP and GPP use `_transparams` attribute to select whether to switch between linear and exp ilink for extra dispersion parameters
- (without checking current status) stationarity in tsa models like ARIMA was imposed by predefined parameter transformation.
- SVAR uses a mask for identifying restrictions, all are fixed params type constraints.


My generic `fit_constrained` used in GLM also transform exog in addition to params and uses offset.

issue here 
- make constraints based on parameter transformation into a usable, general pattern
- extend it to more general cases than `fix_params`

`fix_params` only uses a mask for parameters fixed at user given values.

can we extend this to

- linear restrictions similar to generic fit_constrained. which constraints can we handle this way.
  - note related issue was in inference for hypothesis tests for linear constraints (somewhere I used the transformation restriction
  - can we get a linear transformation matrix p_constrained = H dot p_unconstrained and inverse function?  linalg tools
- affine restrictions; do we need offset or not?
- nonlinear restriction: p_constrained = H(p_unconstrained) if available this would follow the same pattern with transform_params


fix params: constraint on individual parameters, e.g.  zero constraint, p1 = value = 0
- exog transform:
  if value=0, then drop that exog
  if value not zero, then use x1 * value as offset, (requires offset
- parameter transformation, simple mask to add/drop the fixed param with specific value as in current pattern 

simple case : impose two parameters are the same: p1 = p2 = p
- exog transfromation use `x1 + x2` as regressor, optimize reduced model, 
- parameter transformation: optimize w.r.t. reduced params `p` and compute loglike based on `p1, p2`
  (maybe I have some code for this in orthogonal complement linalg tools) (will in general not be unique)

nonlinear case: e.g. `p1 / p2 = value` or implicit function `g(p1, p2) = value`
parameter transformation, max w.r.t. `p2`, in loglike set `p1 = p2 * value`
But in general case, we need to make implicit function into function that maps constrained to unconstrained parameter space.


specific cases currently not supported to get started:
- extra dispersion params in count models NBP, ...
- models with two exog and cross-parameter restrictions, e.g. Zero-Inflated models or mean-scale in dispersion models like Beta regression.


user: josef-pkt
issue_7564: state: open
title: ENH: missing, unclear docs in newer functions
body: - https://www.statsmodels.org/dev/generated/statsmodels.stats.proportion.test_proportions_2indep.html 
  - doesn't specify what default `method` is, same for `confint_proportions_2indep`
  - use different font style for math `or - value = 0`, initially I didn't see that "or" is a statistic, not just the word
- Does `power_proportions_2indep` correspond to one of the test `method`s ?  I guess not, maybe wald.

- https://www.statsmodels.org/dev/generated/statsmodels.distributions.discrete.DiscretizedModel.fit.html
   - [x] default docstring for fit method doesn't show Parameters, Returns
     **update**  fixed by removing GenericLikelihoodModel docstring, inherits docs from LikelihoodModel.fit
   - `predict` method, missing docstring,, no explanation for "which"
- https://www.statsmodels.org/dev/generated/statsmodels.distributions.discrete.genpoisson_p.html
  - no docstrings in generated docs
  - (refactor, maybe change to capitalized name for class)
- https://www.statsmodels.org/dev/generated/statsmodels.miscmodels.ordinal_model.OrderedModel.fit.html
  - also only non-informative generic docstring
  - OrderedResults, no class docstring, no description line 
  - predict and similar that have results for all choices/cells: Docstrings for methods do not methion if "remainder" choice is included, i.e. (nobs, k) or (nobs, k-1) (I don't know right now without looking at and example)
- https://www.statsmodels.org/dev/distributions.html#copula
  list of copula functions should include depfunc_ev and transforms https://github.com/statsmodels/statsmodels/issues/7629#issuecomment-897175061
- [ ] https://www.statsmodels.org/dev/generated/statsmodels.discrete.discrete_model.GeneralizedPoissonResults.get_distribution.html no docstring
- [x] https://www.statsmodels.org/dev/other_models.html doc problems with Beta regression #7584
  - [x] add to index.rst
  - [x] BetaModel docstring doesn't show in html docs https://www.statsmodels.org/dev/generated/statsmodels.othermod.betareg.BetaModel.html#statsmodels.othermod.betareg.BetaModel why ?
  - [x] no Results class docstring
  - [x] typos, eg. `other` instead of `othermod` in text 
- (ultra) high screening is still missing from docs #7505
not included:
- [ ] https://www.statsmodels.org/dev/generated/statsmodels.stats.moment_helpers.cov2corr.html parameter is under returns
- [ ] ...

- no "bernstein" in online docs 

- old weightstats also needs doc review, e.g. df in ztest in CompareMeans 
user: josef-pkt
issue_7549: state: open
title: MAINT: what is statsmodels.interface
body: I just saw that we have an empty directory
https://github.com/statsmodels/statsmodels/tree/main/statsmodels/interface

no history since dropping scikit from name
I don't remember anything about this.

I guess it's obsolete and can be deleted
user: josef-pkt
issue_7542: state: open
title: ENH: power for GLM, additivity, weights, replication, parametric bootstrap
body: just an idea for power computation based on simulation or parametric bootstrap for GLM/LEF
Assuming likelihood is correctly specified (but simulation model could differ from estimation model, e.g. Poisson-NB2)

related
#6759, #6558 power for GLM, OLS/WLS based on noncentrality
#7103 parametric bootstrap for non-gaussian models
there should also be an issue or issue comments for simulated sample size computation by replication, i.e. have a base design matrix and scale it up by nobs.

#7535 merged GLM get_distribution

The specific idea here is to use additivity in GLM for simulation, this assumes full likelihood model is correctly specified.
The distribution of the sum of random variable that are iid GLM/LEF distributed are in the same family (AFAIR)

- To replicate sample for large obs we can scale up the number of terms in the sum for each given covariate pattern (unique exog)
- We should be able to use the appropriately scaled distribution from `get_distribution` to generate nobs-scaled random sample
- We can estimate the model in terms of covariate patterns using var_weights or freq_weights. (We don't have to increase the number of "rows" in the simulations.)

GLM binomial is simple because we have n_trials as the nobs-replication parameter already.

freq_weights might not work because they also assume unique endog pattern jointly with exog.
var_weights will require taking mean.
exposure corresponds to sum, but only available with count models, similar to binomial n_trials.

(additivity in tweedie is not clear and we had a long discussion on it.)

aside: generalizing `exposure` to an "additivity" or sum parameter didn't take, var_weights are a close substitute





user: josef-pkt
issue_7534: state: open
title: BUG/ENH: type casting, float_like, will break complex step numerical derivatives
body: GLM uses `scale = float_like(scale, "scale")` or including `optional` option. The only use of float_like in genmod is for scale.

AFAICS, this converts complex numbers to float and will break complex step derivatives.

Currently not a bug, 
but this needs to change if we estimate scale by optimization and want derivatives, e.g. #7526 scale estimated by profile-mle.

(Because scipy's gaussian norm.cdf for probit also doesn't handle complex numbers, we currently use mostly standard finite differencing for numerical derivatives. In that case casting to float is not a problem.)

related: python's float and our float_like do not allow vectorized values https://github.com/statsmodels/statsmodels/issues/2889#issuecomment-872569172


user: josef-pkt
issue_7533: state: open
title: ENH: Allow `HoltWintersResults` to `append` to make prediction with a fitted model without refitting it
body: Hi,
First, I want to say I really enjoy the work you have been doing with `statsmodels` and it has helped me tremendously!

#### Is your feature request related to a problem? Please describe
I would like to be able to predict a time series at a future point by just updating the observation horizon without having to refit the model. This is a general point, but in the context of that issue, I'd like it to work with `holtwinters.ExponentialSmoothing`.

#### Describe the solution you'd like
It seems to me the solution should be similar than in the case of `ARIMA`. A fitted `ARIMA` or `ARIMAResults` can just use `append` to update the observation horizon.

#### Describe alternatives you have considered
In some cases ARIMA is equivalent to exponential smoothing and that have been able to do just that with `ARIMAResults`.

Thank you very much! 
user: gtauzin
issue_7531: state: open
title: ENH: Fixed parameters for ARMAX GLS
body: #### Is your feature request related to a problem? Please describe
xref: #6159

#### Describe the solution you'd like
Simple changes built upon #7529 (PR #7530)
user: jackzyliu
issue_7530: state: open
title: ENH: Add support for fixed parameters to SARIMAXSpecification and SARIMAXParams (GH7529)
body: - [x] closes #7529 
- [x] xref #6159
- [x] tests added / passed. 
- [x] code/documentation is well formatted.  
- [x] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

<details>


**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in main and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/main -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>

user: jackzyliu
issue_7529: state: open
title: ENH: Add support for fixed parameters to `SARIMAXSpecification` and `SARIMAXParams`
body: #### Is your feature request related to a problem? Please describe
xref: #6159

#### Describe the solution you'd like
-  `SARIMAXSpecification`:
   - move `hannan_rissanen._validate_fixed_params` to `SARIMAXSpecification.validate_fixed_params` 
   - add `SARIMAXSpecification.split_fixed_params` (similar to the existing `SARIMAXSpecification.split_params`)
- `SARIMAXParams`:
   - add function `SARIMAXParams.set_fixed_params` so that we can set fixed parameters as follows: 
   - implement **getters** such as `.is_exog_param_fixed`, `.is_ar_param_fixed`, ..., `.fixed_exog_params`, `.fixed_ar_params`, ...
   - implement free param **getters/setters** like `.free_exog_params`, `.free_ar_params`, ... 
- `hannan_rissanen`: applying these changes to vastly simplify support for fixed parameters in `hannan_rissanen`

For example, we could set and access fixed parameters as follows:
```
In[32]: from statsmodels.tsa.arima import specification, params
In[33]: spec = specification.SARIMAXSpecification(ar_order=2)
In[34]: fixed_params = {'ar.L1': 1}
In[35]: spec.validate_fixed_params(fixed_params)
In[36]: 
In[36]: p = params.SARIMAXParams(spec=spec)
In[37]: p.set_fixed_params(fixed_params)
In[38]: print(p.ar_names)
['ar.L1', 'ar.L2']
In[39]: print(p.ar_params)
[ 1. nan]
In[40]: print(p.is_ar_param_fixed)
[ True False]
In[41]: print(p.fixed_ar_params)
[1.]
In[42]: p.free_ar_params = [10]
In[43]: print(p.ar_params)
[ 1. 10.]
```


#### Describe alternatives you have considered
- Alternative to `set_fixed_params`: we could instead implement additional **setters** for `.is_exog_param_fixed`, `.fixed_exog_params`, etc., but 1) fixed parameters are usually passed around in the sparse form as a dictionary, and 2) this approach would unnecessarily increase the number of "functions" for `SARIMAXParams`
user: jackzyliu
issue_7528: state: open
title: ENH: add estimated variance of endog to GLMResults
body: 
getting the variance for the estimated GLM model is a bit tedious, e.g. for inverse gaussian example

`results_ig.model.family.variance(results_ig.predict([1])) * results_ig.scale`

I think the best would be to add it as option to predict.

`GLMResults` currently does not define a predict method. It uses the inherited predict method of `base.model.Results`, that delegates to the model `GLM.predict` method.

`base.model.Results.predict` delegates args and kwargs to model predict method, so is flexible to handle additional keywords.

GLM.predict uses boolean `linear=False`. This would need to be generalized to `which` similar to the newer count models.
We will not get `get_prediction` for variance and similar because we don't have cov_params for `scale`.

The current GLM.predict does not use weights for linpred and predicted mean. 
In variance prediction we could have an option to include var_weights, similarly `resid_pearson` (always) includes var_weights

(context: I'm verifying get_distribution with implied mean and variance.)
user: josef-pkt
issue_7526: state: open
title: ENH: add scale="mle" or profile-mle to GLM
body: previous PR and discussion #2353

In #7525 it looks like scale estimated by profile-MLE is better in the gamma case than the default pearson_chi2 based scale estimate.

I think we can add "profile-mle" similar to chi2 and deviance scale estimator.

proposal:

We stick with one parameter LEF, and do not estimate full mle (no joint estimation of scale and mean parameters).
"profile-mle" stays as auxiliary estimation, mean parameters are estimated independently of scale.
This means we are imposing orthogonality in estimation and inference for mean parameters.
(this is different from discussion in #2353 based on quick skimming)

e.g. in my example in #7525, I used

```
from scipy import optimize
om = optimize.minimize(lambda x: -results.model.loglike(results.params, scale=x), 10)
```

This will work for families that have a proper scale/dispersion parameter like gamma and gaussian.
MLE scale for Poisson and Binomial is fixed at 1.

Tweedie is more difficult because we don't have full loglike.
Quasi-loglike might be an option. 

Adding quasi-loglike for Poisson and similar doesn't make much sense, given that we have robust cov_type (HC, excess dispersion) for inference, and there is no proper distribution for quasi-poisson.
My focus in #7525 was to get a better scale estimate to have a better predictive distribution, and not better inference for mean parameters.

Maybe: Do we want something like REML scale estimate, e.g. degrees of freedom correction?
I don't know how this would work for distributions like gamma.

Aside:
Another option would be to add some minimum distance estimator for the scale, to match a gof statistic for the predictive distribution directly. #7412 That might be less sensitive to heavy tails and outliers.








user: josef-pkt
issue_7525: state: open
title: ENH: Bias in GLM scale estimate, scale in gamma larger than MLE
body: I'm experimenting with gamma parameterization using GLM.

It looks like pearson_chi2 scale is upward biased, i.e. larger than in data generating process and larger than profile MLE.
example default GLM scale = 10.222, profile-MLE = 9.652

Pit didn't look very good with default scale

TBC (need to run)
user: josef-pkt
issue_7522: state: open
title: BUG: NegativeBinomialP  transparams in score looks wrong
body: NegativeBinomialP.score as `==` instead of assignment, looks like a BUG

found while browsing code

```
        if self._transparams:
            score[-1] == score[-1] ** 2
            return score
```
user: josef-pkt
issue_7521: state: open
title: ENH: add a function to compute residuals for RollingOLS
body: I'm wondering if it is possible to get residuals from the fit object of a `statsmodels.regression.rolling.RollingOLS` model? Something akin to `statsmodels.regression.linear_model.RegressionResultsWrapper.resid`, `statsmodels.regression.recursive_ls.RecursiveLSResultsWrapper.resid` or `statsmodels.regression.recursive_ls.RecursiveLSResultsWrapper.resid_recursive`?

(As an aside, @ChadFulton is there a DOI associated with Harvey 1989 that you mention [here](https://www.statsmodels.org/stable/generated/statsmodels.regression.recursive_ls.RecursiveLSResults.resid_recursive.html?highlight=Harvey#statsmodels.regression.recursive_ls.RecursiveLSResults.resid_recursive)? As someone unfamiliar with the literature googling Harvey 1989 seems to return a reference to [The condition of postmodernity by David Harvey](https://g.co/kgs/AQdFhZ) which I don't think is the relevant text?)

My impression is that calculating the residuals from the rolling regression would be fairly straightforward? I.e. some generalized version of the following

```python
import numpy
import pandas

from statsmodels.regression.rolling import RollingOLS

n = 1000
x = numpy.random.randn(n, 2)
beta = [2, 1]
y = (beta * x).sum(axis=1) + numpy.random.randn(n)

def rolling_residuals(rolling_parameters, X, y):
    y_hat = X * rolling_parameters
    return y - y_hat

rolling_ols_fit = RollingOLS(endog=y, exog=x, window=250, min_nobs=10).fit()
rolling_residuals(rolling_ols_fit.params, x, y)
```

Is there something I am overlooking for why calculating residuals in this way should be avoided?

user: matthewgilbert
issue_7509: state: open
title: ENH/SUMM: extensions to transformation distributions - bivariate non-central birnbaum-saunders
body: bringing several recent pieces of distributions together for additional extensions to them
I'm mainly parking 3 references.

- Birnbaum-Saunders is a monotonic transformation of normal distribution
- that can be generalized by applying Birnbaum-Saunders transformation to other (elliptical, or real line) distributions, and by adding power transformation instead of square root
- if we add a `loc` parameter to the base (e.g. normal) distribution, then we get a "non-central" version of the distribution
- if we replace the base distribution by a bivariate distribution, then we get a bivariate transformed distribution
- We then can use the distribution as an asymmetric kernel for non-parametric density estimation (although that requires distribution specific kernel parameters)

Kakizawa develops bivariate non-central Birnbaum-Saunders asymmetric kernel. The other two articles are referenced for the non-central and the bivariate extension of standard BS distribution.

Kakizawa, Yoshihide. 2020. “Multivariate Non-Central Birnbaum–Saunders Kernel Density Estimator for Nonnegative Data.” Journal of Statistical Planning and Inference 209 (December): 187–207. https://doi.org/10.1016/j.jspi.2020.03.009.

Guiraud, P., V. Leiva, and R. Fierro. 2009. “A Non-Central Version of the Birnbaum-Saunders Distribution for Reliability Analysis.” IEEE Transactions on Reliability 58 (1): 152–60. https://doi.org/10.1109/TR.2008.2011869.

Kundu, Debasis, N. Balakrishnan, and A. Jamalizadeh. 2010. “Bivariate Birnbaum–Saunders Distribution and Associated Inference.” Journal of Multivariate Analysis 101 (1): 113–25. https://doi.org/10.1016/j.jmva.2009.05.005.

aside: I guess the bivariate extension is equivalent to a gaussian copula with BS margins.
gaussian copula + gaussian margins + BS transformation of margins
= gaussian distribution + BS transformation of margins
= gaussian copula + BS margins

related issues:

transformation for distributions
multivariate distributions and copulas
asymmetric kernel density estimation



user: josef-pkt
issue_7505: state: open
title: DOC: (ultra-high) screening is missing in docs
body: search in our docs for screening only shows the release announcement as related topic

gist from PR #4683 looks okay
https://gist.github.com/josef-pkt/b94cbeb1a0663930bf0aa5381b3a1064

stackoverflow question: picking among 130 regressors
https://stackoverflow.com/questions/67992739/how-to-test-all-possible-iterations-in-a-multiple-linear-regresion-and-return-th

I didn't find a quick way to recommend our screening methods.

related:
#7128 add penalty convenience classes for easier usage, and easier advertising of what we have.

(I haven't looked at this in some time)
user: josef-pkt
issue_7504: state: open
title: FAQ: cov_type HC in models without exog, constant-only models
body: **What is the effect of using `cov_type="hc0"` in models that don't have explanatory variables?**

Observations are assumed to be iid, or we are interested in the unconditional distribution.

case that I'm interested right now is MLE for parameters of a fully specified distribution, e.g. discrete counts #7488 

(mostly guesses below without verifying it, IIUC, IIRC)

- in OLS and GLM with estimated scale, HC0 should have no effect. Covariance of score_obs is just the scale or pearson_chi2 scale, and scale is included.
- in models with fixed scale, using HC0 will correct for deviation of the scale or dispersion from the fixed value, e.g. simple excess dispersion in Poisson
- in fully specified MLE for cases like gaussian. Effect of HC is ?  score_obs includes score with respect to scale parameter. 
  - HC corresponds to 4th moment assumptions for the scale estimate, hessian assumes model, cov(score_obs) could have non-gaussian 4th moment. I'm not sure this has an effect in the no-constant gaussian (GLM) case, because 4th moment estimate is also taken to be constant across observations.
  - misspecified correlation between score_obs components. e.g. scale and mean are uncorrelated in gaussian case, at least asymptotically. Correlation between mean and variance will not be zero for skewed data (3rd moment)
- MLE for general shape, and possibly loc and scale, parameters of a distribution. I'm not sure and never looked at those cases. Heteroscedasticity interpretation doesn't apply (no misspecified variance/covariance of error as in OLS), but we can still have Huber/White misspecified MLE.


my current motivation: 
Discretized count models, distribution fitting #7488 do not have `score_obs` defined yet. So, `cov(score_obs)` and sandwich cov_params are not available. 
Does it matter? 
Can we use information matrix ratio (two thirds of sandwich #7192 ) as a misspecification diagnostic for no exog models?



user: josef-pkt
issue_7500: state: open
title: ENH/Design Should we use autograd, automatic differentiation
body: I just saw that lifelines uses autograd for derivatives like hessian 
https://github.com/CamDavidsonPilon/lifelines/blob/master/lifelines/fitters/weibull_aft_fitter.py

autograd seems to have become popular especially from the machine learning direction.

Is it useful for statsmodels, and is it easy enough to have compatibility with plain numpy and scipy.

old issue for automatic differentiation
user: josef-pkt
issue_7494: state: open
title: remove_data= option does not appear to remove data fully
body: #### Describe the bug

In linear/logistic models fit via the formula interface, calling `.save()` with `remove_data=True` appears to remove some but not all instances of length-nobs structures from the fitted model object.

This seems at odds with the documentation of the remove_data option, which suggests that all such structures would be removed.

#### Code Sample, a copy-pastable example if possible


```python
import pandas as pd
import numpy as np
import statsmodels.api as sm
import statsmodels.formula.api as smf

def invlogit(s):
    return np.exp(s) / (1 + np.exp(s))

n = 1000000 # use large nobs to make size differences obvious
df = pd.DataFrame(
    dict(x1=np.random.normal(size=n),
         x2=np.random.normal(size=n),
         e=np.random.normal(size=n),
        )
).assign(y=lambda f: (invlogit(1 + 2 * f.x1 - f.x2 + f.e) > 0.5).astype(int))

model = smf.logit("y ~ x1 * x2", data=df).fit()

model.save("model.sm", remove_data=False)
model.save("model-nodata.sm", remove_data=True)

model._results.model.data.frame = model._results.model.data.frame.iloc[0:0]
model._results.model.data.orig_exog = model._results.model.data.orig_exog.iloc[0:0]
model._results.model.data.orig_endog = model._results.model.data.orig_endog.iloc[0:0]
model._results.model.wexog = None
model._results.model.wendog = None

model.save("model-nodata-plus.sm", remove_data=True)

!ls -atlr model*.sm
```

Yields:

```
-rw------- 1 root root 120003651 Jun  9 16:03 model.sm
-rw------- 1 root root  80003591 Jun  9 16:03 model-nodata.sm
-rw------- 1 root root      3578 Jun  9 16:03 model-nodata-plus.sm
```

Calling save with `remove_data=True` does result in a reduction of the serialized model size, but it's apparent that some copies of the data/design matrix are still in there. "Manually" removing additional structures as above appears to get all copies of the data out of the model. (I confirm that prediction still works from the deserialized model, at least for the simple formula given above.)

#### Expected Output

Based on the documentation of the `remove_data` option ([ref](https://github.com/statsmodels/statsmodels/blob/77bb1d276c7d11bc8657497b4307aa7575c3e65c/statsmodels/base/model.py#L2173)), I would have expected all instances of these nobs-length arrays/dataframes to be removed from the model.

> Remove data arrays, all nobs arrays from result and model.

Options that occur are (1) if it's not the intent of the `remove_data` option fully to remove all length-nobs structures from the fitted model objects, the documentation might be revised to reflect this, or (2) we might revise `remove_data` so that it fully removes all length-nobs structures from the object.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

```
INSTALLED VERSIONS
------------------
Python: 3.6.12.final.0
OS: Linux 5.10.25-linuxkit #1 SMP Tue Mar 23 09:27:39 UTC 2021 x86_64
byteorder: little
LC_ALL: None
LANG: C.UTF-8
statsmodels
===========
Installed: 0.12.2 (/venv/lib/python3.6/site-packages/statsmodels)
Required Dependencies
=====================
cython: Not installed
numpy: 1.19.0 (/venv/lib/python3.6/site-packages/numpy)
scipy: 1.5.4 (/venv/lib/python3.6/site-packages/scipy)
pandas: 1.1.0 (/venv/lib/python3.6/site-packages/pandas)
    dateutil: 2.8.1 (/venv/lib/python3.6/site-packages/dateutil)
patsy: 0.5.1 (/venv/lib/python3.6/site-packages/patsy)
Optional Dependencies
=====================
matplotlib: 3.3.4 (/venv/lib/python3.6/site-packages/matplotlib)
    backend: module://ipykernel.pylab.backend_inline 
cvxopt: Not installed
joblib: 1.0.1 (/venv/lib/python3.6/site-packages/joblib)
Developer Tools
================
IPython: 7.16.1 (/venv/lib/python3.6/site-packages/IPython)
    jinja2: 2.11.3 (/venv/lib/python3.6/site-packages/jinja2)
sphinx: 4.0.2 (/venv/lib/python3.6/site-packages/sphinx)
    pygments: 2.8.1 (/venv/lib/python3.6/site-packages/pygments)
pytest: 6.2.3 (/venv/lib/python3.6/site-packages/pytest)
virtualenv: Not installed
```

user: mmacpherson
issue_7493: state: open
title: BUG/REF: GenericLikelihoodResults Mixin uses df_model, df_modelwc for aic 
body: aic in ResultsMxin adds 1 for constant by default to df_model, unless attribute `res.hasconst = 0`

this should use or check with `df_resid` which is not a mess like df_model

minimal usage of GenericLikelihoodModel for a new subclass had wrong df_modelwc and consequently wrong aic

It's not really a bug for current models, but very misleading for new or user defined models.

I have not checked if any of the current `miscmodels` rely on this.
user: josef-pkt
issue_7492: state: open
title: ENH: conditional transformation models (Hothorn)
body: separate issue for https://github.com/statsmodels/statsmodels/issues/7479#issuecomment-856852285

Hothorn and co-authors have several articles and R packages for transformation models.
Transformation need not be strictly monotonic (almost Lebesque everywhere). e.g. discretization for count or ordinal models.

Looks interesting based on skimming some parts. Might be useful as a general setup. I'm not sure yet how useful in details.

One interesting part that we don't have yet, is non-parametric links/transformations. In some articles they use Bernstein polynomials which are monotonic if the underlying grid values are monotonic.

slides of talks are at http://user.math.uzh.ch/hothorn/talks.html

related:

- #7142 focus on predictive distributions instead of conditional mean, overall theme in these areas
- #7241, PR #7246  strictly monotonic transformation for continuous variables
- #7479, PR #7488  count models based on discretization
- #7069 ordered/ordinal models miscmodels version is merged
- CDF-links and flexible links


user: josef-pkt
issue_7487: state: open
title: ENH: data fallback to asarray in handle_data_class_factory
body: from twitter:

```
Anyone care to help w/modin+statsmodels.api? Got this from sm.OLS(y,X):
ValueError: unrecognized data structures: <class 'modin.pandas.series.Series'> / <class 'modin.pandas.dataframe.DataFrame'>

Obvi I was calling with y=Series and X=df.
Same code works w/vanilla pandas. Thx!
```

base.data.handle_data_class_factory does not have a fallback option if datatype is not recognized.
All allowed data structures are checked with isinstance or type.

we should have a np.asarray fallback, so it still works with any other data type that can be converted to numpy array.


user: josef-pkt
issue_7486: state: open
title: Design: fittedvalues, predict for models without mean function parameterization
body: When we have a general model that doesn't have a parameterization in terms of a mean function, then what should fittedvalues and default predict correspond to?

Which method or attribute do we use for the estimate of the non mean parameters of a distribution, or parts of it? This is a generalization of what we have in `linpred` and in parameter transformation for `get_distribution`.

examples: 
- flexible count distribution, eg. created by discretization of a continuous distribution #7479, 7480
- flexible continuous distributions #7237
- distributions that focus on quantile estimation without analytical mean functions, e.g. flexible lifetime distributions, accelerated failure time models, distributions that estimate median or another center, eg. lognormal.

The general case is similar to gamlss models with link functions for the distribution parameters.

Current structure of fittedvalues = predict() and resulting residuals `f(y - fittedvalues)` is focused on mean modelling E(Y | X).

example case: models focused on quantiles, i.e. explicit median, quantiles, but no closed form, cheap function for mean E(Y | X)

possibilities:

- (1) keep current definition: brute force computation of mean and use it for fittedvalues and resid
- (2) use another definition, e.g. fittedvalues = predict() estimates median or some other center function if available. That's more difficult if we don't have a center/median function either, e.g. when we want to match the distribution. (Maybe the latter is not relevant, if we have quantiles, then we also get the median.)
- use predict() to return get_distribution parameter. Case just for completeness, I don't like this because we don't have any generic interpretation anymore and no way to define `resid`.
- (3) don't define fittedvalues and no default in predict. We have this currently in GMM where we don't have predict defined, i.e. we only estimate parameters and don't have a fully specified model. In the case here we focus on fully specified model with full predictive distribution. Quantile Regression model also does not define a predict method.

to (3) no fitted values
I think we can use predict more often than now for other things than predicted mean, e.g. in Quantile regression we can return the estimated quantile. The computation for Quantreg uses resid as deviation from quantile.
i.e. we can define things but without reusing current attributes, e.g. discrete models use resid_xxx.
`predict` method can work without default, and we optionally return whatever we have.

Currently the zeroinflated count models have the most `which`, but still default to mean.
Log-normal would also be cheap to get conditional mean

My current thinking is that 
- we want (1) + (3) depending on availability and not (2), i.e. do not use defaults or current attributes with a different interpretation. If we want non-mean analogues to fittedvalues and resid, then we use post-fix `_xxx`.
- we need to cache expensive statistics, e.g. if we compute mean, var or analogues to `linpred`.
- not clear yet: Can we do any generic post-estimation on this? e.g. diagnostics and specification tests. For flexible models with focus on predictive distribution, we will want to look at analogues of predict_prob in count models. Which outlier-influence diagnostics can we get ?















user: josef-pkt
issue_7485: state: open
title: WIP/ENH: probability plot for multivariate normal 
body: see #1307

this is an old branch of mine.  PR so I find it again

parts of it will be outdated and might be obsolete

The prob plot should be extended to elliptical distributions
user: josef-pkt
issue_7484: state: open
title: DOC/TST: how to use knots in gam
body: a question on how to use the `knot_kwds` in gam
https://stackoverflow.com/questions/67838500/how-to-specify-knot-locations-in-bsplines-function-of-statsmodels-gam
and as comment to my gist.

`list of dict` sounds kind of clear, but what if we only want to specify knots for one of several variables?


needs docs and tests

I don't find unit tests for knot_kwds, but I had spent a lot of time on it to match options in R that patsy doesn't have.
The spline basis classes don't have many unit tests that test them directly, most tests are through GAM examples.

user: josef-pkt
issue_7483: state: open
title: ENH: bivariate and multivariate asymmetric kernels
body: continuing  #7346 and merged PR #7364 for asymmetric and #7482 for discrete asymmetric kernels

Currently we only have asymmetric product kernels. In my copula experiments with Beta copula it looks like a diagonal ridge is smoothed and reduces correlation in KDE samples. 
Multivariate kernels that include correlation should do better in these cases.

The first article has bivariate beta kernel, among others.

Kokonendji, Célestin C., and Sobom M. Somé. 2018. “On Multivariate Associated Kernels to Estimate General Density Functions.” Journal of the Korean Statistical Society 47 (1): 112–26. https://doi.org/10.1016/j.jkss.2017.10.002.

Kakizawa, Yoshihide. 2020. “Multivariate Non-Central Birnbaum–Saunders Kernel Density Estimator for Nonnegative Data.” Journal of Statistical Planning and Inference 209 (December): 187–207. https://doi.org/10.1016/j.jspi.2020.03.009.

Bivariate birnbaum-saunders density/kernel has bivariate normal pdf in it. So that should be feasible, and extends to multivariate.
In the univariate case we only have "central" BS. I didn't check why Kakizawa uses non-central version.




user: josef-pkt
issue_7482: state: open
title: ENH: discrete asymmetric kernel for count data
body: discrete analogue to #7346 and merged PR #7364

related to them of fitting predictive distributions, now also for discrete random variables #7480 (designer distributions for counts)

Kiessé, T. Senga. 2017. “On Finite Sample Properties of Nonparametric Discrete Asymmetric Kernel Estimators.” Statistics 51 (5): 1046–60. https://doi.org/10.1080/02331888.2017.1293060.

Kokonendji, Célestin C., and Tristan Senga Kiessé. 2011. “Discrete Associated Kernels Method and Extensions.” Statistical Methodology 8 (6): 497–516. https://doi.org/10.1016/j.stamet.2011.07.002.

My guess is that writing the kernels for Poisson, binomial and negative binomial should be pretty simple in analogy to the continuous asymmetric kernels.
This looks useful for data/frequency smoothing especially for plots and diagnostics.

I only did very brief skimming.
AFAICS. these "standard" discrete kernels cannot fully converge as bandwidth goes to zero.
The articles also have "associated kernels" but I never looked at what those are.


MultivariateKDE has one count kernel, but I think it's a symmetric distance kernel. I didn't look closely at that.


user: josef-pkt
issue_7480: state: open
title: ENH: count models with non-poisson features, under dispersion, heavy tails
body: continuing fitting count distribution instead of (just) estimating mean function

currently we have zero-inflated and over-dispersed compared to Poisson, GPP allows for small under-dispersion.

We should have more count models with special features, e.g. stronger underdispersion and heavy tails

aside: What happened to PIG, poisson-inverse gaussian? It was briefly on the list for the count model GSOC.

I don't have an overview of alternative count models anymore

#7479 count models based on discretized continuous distribution


Note: we might have problems with computing moments for some of the distributions. Similar to the continuous case, we might want to emphasize median and quantiles instead of moments in these cases.

For pure moment estimation QMLE (quasi-poisson) and GMM are enough.

user: josef-pkt
issue_7479: state: open
title: ENH: add discrete/discretized pareto distribution
body: https://github.com/statsmodels/statsmodels/issues/7472#issuecomment-853325436

Buddana, Amrutha, and Tomasz J. Kozubowski. 2014. “Discrete Pareto Distributions.” Economic Quality Control 29 (2): 143–56. https://doi.org/10.1515/eqc-2014-0014.

quick skimming:
looks good

relatively simple explicit forms of pdf, cdf, sf, ppf
mean/moment use special zeta function,
easy rvs from mixture representation.

i.e. good for modelling quantiles, but no mean parameterization.


**update**

after reading or skimming several of the articles:
Articles for discrete/discretize pareto, weibull and similar use a different parameterization than the underlying continuous distribution which changes exp function to a power function (e.g. q**x).  
I only saw one article that provided a reason for the reparameterization and that was because exponential/geometric distribution can be treated as a special case if the parameter goes to zero.




user: josef-pkt
issue_7478: state: open
title: ENH: many fixed effects in GLM, absorb in computation
body: (I don't see a specific GLM issue for this)

#2568 general issue for linear model with some discussion on GLM type models

motivation: #7469 stats.stackexchange question mentioning feglm and feglm.nb in alpaca package

Stammann, 2018 looks good, skimming some parts.
uses alternating iteration with demeaning to update in IRLS

Czarnowske, Daniel, and Amrei Stammann. 2020. “Fixed Effects Binary Choice Models: Estimation and Inference with Long Panels.” ArXiv:1904.04217 [Econ], October. http://arxiv.org/abs/1904.04217.

Stammann, Amrei. 2018. “Fast and Feasible Estimation of Generalized Linear Models with High-Dimensional k-Way Fixed Effects.” ArXiv:1707.01815 [Stat], July. http://arxiv.org/abs/1707.01815.

Stammann, Amrei, Florian Heiß, and Daniel McFadden. 2016. “Estimating Fixed Effects Logit Models with Large Panel Data.” 145837. VfS Annual Conference 2016 (Augsburg): Demographic Change. VfS Annual Conference 2016 (Augsburg): Demographic Change. Verein für Socialpolitik / German Economic Association. https://ideas.repec.org/p/zbw/vfsc16/145837.html.

user: josef-pkt
issue_7476: state: open
title: fit_regularized method is unstable/too sensitive to alpha when using Tweedie distribution
body: #### Describe the bug

The estimated coefficients are extremely sensitive to variations in the alpha parameter when using the Tweedie distribution. Not only it is very hard to find alpha values that generate non-null coefficients, but tiny variations on those values of alpha will drastically change the estimated coefficients.

#### Code Sample, a copy-pastable example if possible


```python
# based on test case from https://github.com/statsmodels/statsmodels/blob/main/statsmodels/genmod/tests/test_glm.py
# line 1998

import statsmodels.api as sm
import numpy as np

np.random.seed(3242)
n = 500
p = 1.5 # Tweedie variance power
x = np.random.normal(size=(n, 4))
lpr = np.dot(x, np.r_[1, -1, 0, 0.5])
mu = np.exp(lpr)
lam = 10 * mu**(2 - p) / (2 - p)
alp = (2 - p) / (p - 1)
bet = 10 * mu**(1 - p) / (p - 1)

y = np.empty(n)
N = np.random.poisson(lam)
for i in range(n):
    y[i] = np.random.gamma(alp, 1 / bet[i], N[i]).sum()
    
fam = sm.families.Tweedie(var_power=p, eql=True)
model2 = sm.GLM(y, x, family=fam)
result2 = model2.fit_regularized(L1_wt=1, alpha=0.07, maxiter=200,
               cnvrg_tol=0.01)
result2.params

# all coefficients will be zero in this case
result3 = model2.fit_regularized(L1_wt=1, alpha=0.08, maxiter=200,
               cnvrg_tol=0.01)
result3.params
```
<details>

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `main`. If your problem has been fixed in an unreleased version, you might be able to use `main` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the main branch of this repository? It helps the limited resources if we know problems exist in the current main branch so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

There isn't a fixed expected output for this, as it will depend on the data, model configurations, etc. In any case, this type of variations seems unreasonable and it doesn't match the behavior when using other packages (like H2O for example).

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]

INSTALLED VERSIONS
------------------
Python: 3.9.5.final.0

statsmodels
===========

Installed: 0.12.2 (C:\Users\alias\repos\feature-selection\.venv\lib\site-packages\statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.20.2 (C:\Users\alias\repos\feature-selection\.venv\lib\site-packages\numpy)
scipy: 1.6.2 (C:\Users\alias\repos\feature-selection\.venv\lib\site-packages\scipy)
pandas: 1.2.3 (C:\Users\alias\repos\feature-selection\.venv\lib\site-packages\pandas)
    dateutil: 2.8.1 (C:\Users\alias\repos\feature-selection\.venv\lib\site-packages\dateutil)
patsy: 0.5.1 (C:\Users\alias\repos\feature-selection\.venv\lib\site-packages\patsy)

Optional Dependencies
=====================

matplotlib: 3.4.2 (C:\Users\alias\repos\feature-selection\.venv\lib\site-packages\matplotlib)
    backend: module://ipykernel.pylab.backend_inline 
cvxopt: Not installed
joblib: 1.0.1 (C:\Users\alias\repos\feature-selection\.venv\lib\site-packages\joblib)

Developer Tools
================

IPython: 7.23.1 (C:\Users\alias\repos\feature-selection\.venv\lib\site-packages\IPython)
    jinja2: 3.0.1 (C:\Users\alias\repos\feature-selection\.venv\lib\site-packages\jinja2)
sphinx: 3.5.4 (C:\Users\alias\repos\feature-selection\.venv\lib\site-packages\sphinx)
    pygments: 2.9.0 (C:\Users\alias\repos\feature-selection\.venv\lib\site-packages\pygments)
pytest: 6.2.4 (C:\Users\alias\repos\feature-selection\.venv\lib\site-packages\pytest)
virtualenv: 20.4.7 (C:\Users\alias\repos\feature-selection\.venv\lib\site-packages\virtualenv)

</details>

user: diegodebrito
issue_7473: state: open
title: add lowess with custom windows
body: Dropping this to ask if there is interest in this extension of the lowess I made.

It allows to define the windows for every datapoint. This makes it possible to be flexible in defining the window size, not being fixed on `frac`.
It e.g. allows to create a flexible window size, e.g. having a time-based window or a growing window. Possibilites are endless :)

If there is interest, I would add the currently missing stuff, like xvals.
user: kaktus42
issue_7472: state: open
title: ENH: pareto tails of a distribution
body: I have wanted this for a long time but no code yet.

How heavy is the tail of a distribution. A experimental notebook https://gist.github.com/josef-pkt/3c66541ceed9447d7a9bce6ed0b8959e

Matlab has for a long time a KDE stitched together with pareto tails. I have seen again articles that use pareto tails to get a better estimate or approximation for the tail distribution of a sample.
Related in extreme value theory we want to compute statistics based on tail behavior. In multivariate case, copulas, we have additional properties on tail dependence.

There is a large literature on power or pareto tails and several python packages for those. What I'm not yet clear about are tails that are lighter than power tails and tails that are lighter than the tails of the exponential distribution, e.g. normal distribution.

Related: 
Nonlinear transformation of distributions affect in many cases the tail behavior. 
Example: Some (?) references mentioned that transformation to bounded interval does not solve the tail problem for kernel density estimation. It mainly replaces it by modelling boundary behavior.
I don't know how good beta kernels and bernstein polynomials are in getting the heaviness of tails of transformed distributions.
(An article of Stephens on transformations mentions tail behavior at [0, 1] boundaries.)

related issues:
#5185 pareto distribution and extreme value theory, including list of python packages
#7278 general issue on approximating distributions
#7412 minimum distance estimators
#7317 adding extreme value theory to statsmodels
#5094 adding pareto regression, not directly related to tail estimation, for survival/lifetime/duration modelling

some possible additions

- as part of EVT: mainly for heavy tailed distributions, POT, genpareto estimation, generalized extreme value distribution for maxima/minima
- piecewise distributions, stitching together central distribution with a tail distribution
- rvs for tail sampling, generic rvs that samples from a truncated distribution in the tail. (check scipy,truncnorm, IIRC I did this already once, but not with focus on tails), see comment below https://github.com/statsmodels/statsmodels/issues/7472#issuecomment-858926467
- light tails I found some literature for lighter tails, but have not read anything there (*)
- hypothesis tests between light/exponential tails and power/pareto tails

Is there a standard distribution for `pdf(x) = c * exp(-x**b)` ?     for `x>=0`

(*) distribution based on `exp(-h(x))` with tail-asymptotic power exponent `exp(-x**b)`   
(exponential distribution b=1, gaussian b=2)

Ferreira, Marta. "A study of exponential-type tails applied to Birnbaum-Saunders models." Chilean Journal of Statistics Vol. 4, No. 1, April 2013, 87–97
http://soche.cl/chjs/volumes/04/01/Ferreira(2013).pdf

Klüppelberg, Claudia. Alexander Lindner. "Extreme value theory for moving average processes with light-tailed innovations." Bernoulli 11 (3) 381 - 410, June 2005. https://doi.org/10.3150/bj/1120591182 




user: josef-pkt
issue_7470: state: open
title: SUM/FAQ: overview elements for nonparametric copulas and multivariate distributions
body: companion to general and specific issues #7454, ...

List of function and classes that we have or still want to get
("currently" means merged and what I include in PR #7254 )

- `distributions.bernstein.BernsteinDistribution`
   requires binned probabilities or cdf on equal spaced grid.
   - methods: cdf, pdf, rvs, get_marginal

- empirical distribution ECDF
  currently only a function for ecdf counts
  - methods ecdf counts, `distributions.tools._ecdf_mv` (preliminary api)
  
- empirical beta copula
  currently only rvs, adds bandwidth and beta parameter options
  multivariate version of beta kernel distribution in `nonparametric.kernels_asymmetric`
  - methods: rvs   `distributions.copula.other_copulas.rvs_kernel`
 
- histogram
  binning is the same as for Bernstein distribution from data (np.histogram)
  no specific methods yet for distribution with piecewise constant pdf 

- product kernel for `nonparametric.kernels_asymmetric`
  nothing yet, can either reuse MultivariateKDE or pattern of bernstein/empirical beta loops
- ...

- tools: mainly `distributions.tools`
  - `_Grid` class to work with regularly spaced grids
  - functions for constructing distribution, probability grid 
     'cdf2prob_grid', 'prob2cdf_grid', 'frequencies_fromdata', 'approx_copula_pdf'
  - '_rankdata_no_ties' vectorized rankdata without tie handling
  - functions to improve approximation: 'average_grid', 'nearest_matrix_margins'
  - kernels in `nonparametric.kernels_asymmetric`
  - ...
 
missing:
- merging low probability cells for use with chisquare test.
  used in `distributions.copula.tests.test_copula.check_cop_random`, 
  intended for t-copula and gaussian copula with missing or difficult to compute cdf
- other gof distance measures including ecdf based
  I had used some to check approximation and rvs for multivariate distribution or copulas.
- plot helper functions
  partially missing, 
  functions for quickly comparing sample with theoretical distributions/copulas or non-parametric or approximation with parametric distributions

user: josef-pkt
issue_7469: state: open
title: ENH: ConditionalModels Logit, Poisson, handle overdispersion, cov_type
body: I don't see an option in the conditional models to adjust cov_params for variance misspecification

https://stats.stackexchange.com/questions/526459/how-to-correct-conditional-poisson-standard-errors-for-over-dispersion

I never looked at any literature for whether and how conditional Logit or Poisson can be adjusted for variance misspecification.
user: josef-pkt
issue_7468: state: open
title: ENH: marginal bivariate copula or distribution from multivariate
body: It would be convenient to have a bivariate representation of a multivariate distribution, e.g. for plotting like in copula plot functions.

BernsteinDistribution has `get_marginal` which AFAICS allows for arbitrary subsets of variables. 

Can we get something similar for multivariate copulas.

2 possibilities:

- get only some statistics, e.g. cdf, pdf
- construct new bivariate/subset copula, if we know the copula family and parameter transformation for marginal distributions that are also multivariate. e.g. we need to know which distributions and copulas are closed under taking subsets.


Once we have the marginal copula, then we can also get the marginal CopulaDistribution with nonuniform univariate margins.

For multivariate distribution case like mv-t we would have the same issue. (There is a reference for the mv-t case, but that one might have been for the conditional distribution.)

Similar issue with conditional distribution, e.g. for multivariate normal, elliptical, copulas, ...

parts like cdf should be easy. We mainly need to bring the theory together and have unit tests.
`cdf_{1,2}(u_1, u_2) = cdf_{1,2,3,4}(u_1, u_2, 1, 1)`

pdf will need partial derivatives w.r.t. bivariate u

user: josef-pkt
issue_7467: state: open
title: ENH: shortcut function to simulate, rvs from Bernstein approximation to copula
body: extreme value copulas are still missing rvs.

It should be possible to get a relatively efficient bernstein copula rvs method that doesn't use too much memory.
Other methods of the bernstein distribution are currently still limited by large memory requirements if number of bins is large.

This is simpler in the rvs case because we only need to compute (beta) random number for one bin per observation. This means we can use a lower bandwidth, i.e. more bins in grid. We can also directly work with the flat grid instead of n-dim grid.

I think it's worth a limited purpose (rvs) function, or specialized class, to do this.

aside: BernsteinDistribution has a `from_data` method, but I don't see any automatic method/function to construct approximation to copula/cdf

related #7454 
user: josef-pkt
issue_7459: state: open
title: select_coint_rank will get a wrong Trace Statistics, comparing with Eviews 10
body: #### Describe the bug

When I am doing Cointegratoin for Y and X, I use Eviews 10 and select_coint_rank in statsmodels.tsa.vector_ar.vecm.

**The bug happens ONLY when k_ar_diff=0.**

in Eviews 10, the Trace Statistics is as below:
```python
Unrestricted Cointegration Rank Test (Trace)				
				
Hypothesized		Trace	0.05	
No. of CE(s)	Eigenvalue	Statistic	Critical Value	Prob.**
				
None *	 0.002157	 24.79537	 15.49471	 0.0015
At most 1	 4.57E-05	 0.514358	 3.841466	 0.4733
				
 Trace test indicates 1 cointegrating eqn(s) at the 0.05 level				
 * denotes rejection of the hypothesis at the 0.05 level				
 **MacKinnon-Haug-Michelis (1999) p-values				
```

but in select_coint_rank I got CointRankResults.summary() and I got Trace statistics as below,
```python
Johansen cointegration test using trace test statistic with 5% significance level
r_0 r_1 test statistic critical value
-------------------------------------
  0   2          33.39          15.49
  1   2          6.617          3.841
-------------------------------------
```

So there are same critical value but different test statistics(in Eviews 24.79537 in Python 33.39).

**The bug will not happen when k_ar_diff=1.**
in Eviews 10, 
```python
Unrestricted Cointegration Rank Test (Trace)				
				
Hypothesized		Trace	0.05	
No. of CE(s)	Eigenvalue	Statistic	Critical Value	Prob.**
				
None *	 0.001840	 21.08898	 15.49471	 0.0064
At most 1	 3.37E-05	 0.379090	 3.841466	 0.5381
				
 Trace test indicates 1 cointegrating eqn(s) at the 0.05 level				
 * denotes rejection of the hypothesis at the 0.05 level				
 **MacKinnon-Haug-Michelis (1999) p-values				
```

in select_coint_rank I got CointRankResults.summary() and I got Trace statistics as below,
```python
Johansen cointegration test using trace test statistic with 5% significance level
r_0 r_1 test statistic critical value
-------------------------------------
  0   2          21.09          15.49
  1   2         0.3791          3.841
-------------------------------------
```

you can see it is correct with Eviews 10.

#### Code Sample, a copy-pastable example if possible

**The bug will happen ONLY when k_ar_diff=0.**
```python
from statsmodels.tsa.vector_ar.vecm import select_coint_rank      
        res = select_coint_rank(arrJohansen,0,0);
        print(res.summary())
```

**The bug will not happen when k_ar_diff=1.**
```python
from statsmodels.tsa.vector_ar.vecm import select_coint_rank      
        res = select_coint_rank(arrJohansen,0,1);
        print(res.summary())
```



If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

i hope it could make as exact as Eviews 10.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

INSTALLED VERSIONS
------------------
Python: 3.8.5.final.0

statsmodels
===========

Installed: 0.12.0 (d:\ProgramData\Anaconda3\lib\site-packages\statsmodels)

Required Dependencies
=====================

cython: 0.29.21 (d:\ProgramData\Anaconda3\lib\site-packages\Cython)
numpy: 1.20.3 (d:\ProgramData\Anaconda3\lib\site-packages\numpy)
scipy: 1.5.2 (d:\ProgramData\Anaconda3\lib\site-packages\scipy)
pandas: 1.1.3 (d:\ProgramData\Anaconda3\lib\site-packages\pandas)
    dateutil: 2.8.1 (d:\ProgramData\Anaconda3\lib\site-packages\dateutil)
patsy: 0.5.1 (d:\ProgramData\Anaconda3\lib\site-packages\patsy)

Optional Dependencies
=====================

matplotlib: 3.3.2 (d:\ProgramData\Anaconda3\lib\site-packages\matplotlib)
    backend: module://ipykernel.pylab.backend_inline 
cvxopt: Not installed
joblib: 0.17.0 (d:\ProgramData\Anaconda3\lib\site-packages\joblib)

Developer Tools
================

IPython: 7.19.0 (d:\ProgramData\Anaconda3\lib\site-packages\IPython)
    jinja2: 2.11.2 (d:\ProgramData\Anaconda3\lib\site-packages\jinja2)
sphinx: 3.2.1 (d:\ProgramData\Anaconda3\lib\site-packages\sphinx)
    pygments: 2.7.2 (d:\ProgramData\Anaconda3\lib\site-packages\pygments)
pytest: 6.1.1 (d:\ProgramData\Anaconda3\lib\site-packages\pytest)
virtualenv: Not installed

</details>

user: GlobalMe2017
issue_7454: state: open
title: ENH/Design empirical copula, multivariate ECDF
body: similar to #7445 histogram copula

What should be the design for an empirical multivariate distribution or empirical copula.

Currently, WIP in #7408, I have written a ECDF function with some speedup in the bivariate case.
In unit tests, I use bin probabilities instead of densities, similarly to what I needed for Bernstein.

Currently, I have Bernstein smoothed empirical copula based on grid probabilites. This could be complemented by empirical Beta copulas, i.e. bi- or multivariate version of Beta KDE (PR #7364 merged).

Looking at docs for R package copBasic (AFAIU, and a few simple examples with it):
EMPIRcop evaluates empirical copula (ECDF) at new point u, v  (returns float/numeric)
EMPIRcopdf (read EMPIRcop_df ) creates data frame for copula cdf of sample points (similar to what I have as ECDF)
EMPIRgrid creates a grid of cdf values, based on empirical copula, I guess, using EMPIRcop, mainly used for plotting

AFAICS, there is no sharing of intermediate results, i.e. all functions compute directly from data.
plotting positions (rank - 0.5) / nobs or rank / (nobs + 1) seems to be only partially included in functions.
focus seems to be on density, not bin probabilities (at least for plotting)

EMPIRcop has an empirical Bernstein copula option, but not sure how that is defined.

Note on usage:
- minimal distance estimation of copula parameters is not as good as MLE. at least without outliers.
- currently I have unit test based on chisquare test with binned probabilities, and contour plot based of bin probabilities (unscaled but same shape as pdf on regular grid with constant grid size delta u.)
- overlapping code with setup of Bernstein distribution ? and histogram copuls/distribution
- simulation: 
   - empirical distribution on support of observations
   - histogram distribution, local linear pdf within bins / grid intervals/ hyper cubes
   - smoothed: Bernstein or Beta KDE (on fixed grid or observations respectively)
 

aside: working on grid does not give a copula and uniform margins if nobs is not a multiple of the number of bins.
We could drop observation to have even bin counts on regular grid, or use new helper function to find a close distribution that has uniform margins.

user: josef-pkt
issue_7452: state: open
title: BUG: append broken in DynamicFactorMQ if updated months do not span updated quarters
body: #### Describe the bug

I have  a `statsmodels.tsa.statespace.mlemodel.MLEResultsWrapper` object  obtained after fitting a `DynamicFactorMQ` model. I would like to apply the `append ` method to the result object with a new step of data (adding one month to the monthly endog and one quarter to the quarterly endog ) but i don't seem to find the right index format for my monthly and quarterly endog DataFrames. I always get the following error `Given "endog" does not have an index that extends the index of the model.`. I should mention that the same happens when using the `extend` method but the `apply` works properly. 

#### Code Sample, a copy-pastable example if possible

here is where i call the method 

![image](https://user-images.githubusercontent.com/23510172/118680114-bf0fec00-b7fe-11eb-895f-0cc032d6a127.png)

here are the monthly and quarterly endog 
![image](https://user-images.githubusercontent.com/23510172/118680293-e6ff4f80-b7fe-11eb-9eb1-99d21a801adb.png)
![image](https://user-images.githubusercontent.com/23510172/118680349-f088b780-b7fe-11eb-9029-4f8eef4e436d.png)

here is the original endog data 

![image](https://user-images.githubusercontent.com/23510172/118680541-1f9f2900-b7ff-11eb-88fb-de4af6572719.png)




#### Expected Output

A new result object containing the filtered.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

INSTALLED VERSIONS
------------------
Python: 3.9.2.final.0

statsmodels
===========
Installed: 0.12.2 

Required Dependencies
=====================

cython: Not installed
numpy: 1.19.2 
scipy: 1.6.2
pandas: 1.2.3
dateutil: 2.8.1 
patsy: 0.5.1 

Optional Dependencies
=====================

matplotlib: 3.3.4 
cvxopt: Not installed
joblib: 1.0.1 

Developer Tools
================

IPython: 7.22.0 
jinja2: 2.11.3 
pygments: 2.8.1 
pytest: Not installed
virtualenv: Not installed


user: Mlaouar
issue_7449: state: open
title: Bayesian VAR
body: If I wrote a basic Bayesian VAR module, could I submit it for inclusion into statsmodels? 
user: cgarciga
issue_7448: state: open
title: ENH: Bootstrapping Confidence Intervals + Statistical Tests
body: #### Is your feature request related to a problem? Please describe
Not a problem per se - new feature due to lack of ability to find this in commonly used python statistical packages

#### Describe the solution you'd like
I'd really like a way to do bootstrapping confidence intervals as well as statistical tests associated with them (e.g. one-sample and two-sample tests) within the package. I could contribute myself although I'm new in terms of open source stuff esp on a package on this level. Especially since R has standard packaging in this. In particularly relevantly for me I'd want to allow for weighted estimates in bootstrapping arbitrary point estimates


#### Describe alternatives you have considered
I found a BSD-licensed package on bootstrapping here - but this package is inactive:

https://github.com/facebookarchive/bootstrapped

In addition I hand-wrote some code to handle bootstrapping myself (mostly for practice - so obviously it's rough, not optimized and POC-like). In my particular use case I'd like to specifically account for weighted point estimates so any method should account for weighting in there optionally.

The R package `bootstrap` is here, but of course this is in R:
https://cran.r-project.org/web/packages/bootstrap/bootstrap.pdf
user: quillan86
issue_7445: state: open
title: ENH: histogram copula
body: I'm not sure what to do here. I need several pieces of a histogram copula but I don't want to publicly support a proper HistogramCopula and coming up with an api for that, at least not now.

Currently the pieces that I need are in recipes and tool functions. I needed it for Bernstein distribution or copula and again in a notebook to check quality of `rvs` (for #7408 ). But using numpy.histogrammdd and the tools function requires work and checking documentation and is not very convenient.
aside: #7444 to convert raw probability grid to have uniform margins.

possible constructor tool methods

- `from_cdf` grid that's pretty straight forward
- `from_pdf` grid tricky if we have pdf on grid points and not bin probabilities, use case StudentTCopula where we don't have cdf (of bins
- `from_data` using np.histogramm_dd we get cell probabilities and the can cumsum for cdf, requires ranks to get uniform margins if nobs is multiple of number of bins.

from_cdf and from_data was used in constructing Bernstein distribution copulas in examples and unit tests.
from_pdf: I wrote distribution.tools.average_grid to compute bin means, but that average grid needs cleaning and transformation to form a proper copula. (e.g. #7408  to get uniform margins). There is still the problem that copula can become unbound at the edges, in examples I stay far enough away from 0, 1 boundaries.

searching for histogram copula finds:
https://rdrr.io/cran/SimCop/man/plot.CopApprox.html
https://rdrr.io/cran/SimCop/man/GetApprox.html
Tajvidi, Nader, and B. A. Turlach. "A general approach to generate random variates for multivariate copulae." Australian & New Zealand Journal of Statistics 60, no. 1 (2018): 140-155.
(only 1 citation)





user: josef-pkt
issue_7444: state: open
title: ENH: tools: closest matrix with given marginals
body: I needed to correct a bivariate binned distribution to have uniform marginals.
(application is from binned copulas for checking results in #7408 )

Iterating works, trial and error

```
pc = ag / ag.sum()
for i in range(20):
    pc = pc / np.sqrt(pc.sum(0) * pc.sum(1)[:,None])
    pc /= pc.sum()
    print()
    print(pc.sum(0), pc.sum(1), pc.sum())
```

iteration idea is from 
Dhillon, Inderjit S., and Joel A. Tropp. "Matrix nearness problems with Bregman divergences." SIAM Journal on Matrix Analysis and Applications 29, no. 4 (2008): 1120-1146.
https://authors.library.caltech.edu/9428/1/DHIsiamjmaa07.pdf

I didn't read it (too much numerical math for me)

p. 1133 section 5.1. Contingency tables with fixed marginals
mentions iterative Bregman projections and sounded similar to what I needed.
(that section was found by Google and the only part I was interested in)

(sqrt is from my earlier non-iterative trials)


user: josef-pkt
issue_7441: state: open
title: ENH: information criteria for robust estimators, RLM
body: related to #6516 IC for M-estimators and #7440 minimum density power divergence

mainly parking two references
I don't remember the status for this for RLM.

Ronchetti also has a robust IC

Kurata, Sumito, and Etsuo Hamada. 2018. “A Robust Generalization and Asymptotic Properties of the Model Selection Criterion Family.” Communications in Statistics - Theory and Methods 47 (3): 532–47. https://doi.org/10.1080/03610926.2017.1307405.
("official" pdf is a preprint)

———. 2020. “On the Consistency and the Robustness in Model Selection Criteria.” Communications in Statistics - Theory and Methods 49 (21): 5175–95. https://doi.org/10.1080/03610926.2019.1615093.

this includes comparison to other IC including GIC/TIC
their proposal uses the same penalization as BIC but based on density power divergence



user: josef-pkt
issue_7440: state: open
title: ENH: minimum density power divergence as robust estimator
body: special case of #7412
some notes for implementation. looks relatively straight forward for cases with simple terms

The main problem is the correction term in objective function and score for fisher consistency.
We can get started with cases that have a simple analytical expression and then add discrete models that need (truncated) summation of support of distribution. 

- simple analytical expressions
  - normal distribution, with or without simultaneous estimation of variance
  - beta regression
  - pareto distribution and regression (Juarez and Schucany 2004 mentions regression case in conclusions)
  - Logit, binary: requires enumeration, but only two points in support of distribution
- discrete models
  - Poisson
  - truncated sum is generic and should work for NegBin and others
- numerical integration over continuous space. This is difficult in regression case because we have one integral per observation.
  - ???

partial overlap with other robust estimator for GLM type models for example Poisson with sum over support for fisher consistency. There exists a shortcut for huber `norm` in Poisson because of the functional form.

needs references
user: josef-pkt
issue_7439: state: open
title: ENH: power compromise error, ratio type 1 - type 2 error
body: Gpower has the option to find compromise power, i.e. fix ratio between type 1 and type 2 error and then find absolute level.

It doesn't fit in the current generic setup for root finding and might need a separate method.
I never looked at it in details.

example
https://stackoverflow.com/questions/67433118/communicate-with-gpower-from-python

(I don't remember if PASS has something similar, I guess not.)

user: josef-pkt
issue_7436: state: open
title: ENH: special case exactly identified GMM, M-estimator estimating equations
body: related
#1790
#2898
#5674

I'm looking at M-estimators that are defined through estimating equations, moment conditions and don't have a specified (or simple) objective function.

computational: We currently don't have a general setup for solving estimating equations (multivariate root finding). Currently we have
- specific optimizers: e.g.  IRLS in GLM and RLM, some in GEE, 
- Our newton only really uses score and hessian but requires objective function

alternative would be to use GMM for the special case of exactly identified model.

One computational issue optimization in GMM
The derivative of GMM objective function is more complicated than just moment condition. It involves already derivative of moment condition. Second derivative of GMM objective function would include second derivatives of moment conditions.
However, cov_params in GMM does not use second derivative (hessian) of objective function. The required matrices are moment condition itself and derivative of moment condition, similarly to MLE.

Current GMM handles exactly identified case, but I guess the computations could be streamlined for the exactly identified case.
Multivariate rootfinding might be computationally more efficient than maximizing GMM objective.

So

- add new GMM class (if it looks a large enough improvement)
- try out scipy.optimize multivariate root finders as alternative to optimization

my current motivation: (outlier) robust estimator for GLM (quasi-GLM) which are often defined in the literature by estimating equations.
older case: RLM with variance functions where mean and variance functions are not from an optimization problem (when I was working on #3261 and similar)

user: josef-pkt
issue_7430: state: open
title: FAQ-D/TST: pytest calls setup_class repeatedly
body: It looks like pytest can call setup_class repeatedly
I saw some cases before but only in local runs, below is unit test timing to "proof" it.

In the first case, TestZeroInflatedModel_probit, both methods just check an assert each, based on precomputed attributes.

```
10.43s setup    statsmodels/discrete/tests/test_count_model.py::TestZeroInflatedModel_probit::test_llf
10.39s setup    statsmodels/discrete/tests/test_count_model.py::TestZeroInflatedModel_probit::test_params

9.47s setup    statsmodels/discrete/tests/test_count_model.py::TestZeroInflatedGeneralizedPoisson_predict::test_mean
9.40s setup    statsmodels/discrete/tests/test_count_model.py::TestZeroInflatedGeneralizedPoisson_predict::test_var
```
from https://dev.azure.com/statsmodels/statsmodels-testing/_build/results?buildId=3313&view=logs&j=5999b18d-e094-51e5-d97a-d1e11af2567f&t=58839285-42ac-525e-b729-37aba1c87bae

at end of https://docs.pytest.org/en/latest/how-to/xunit_setup.html :
```
Remarks:
    It is possible for setup/teardown pairs to be invoked multiple times per testing process.
```

So, that partially defeats the point of using `setup_class` to have expensive computation done only once.

I have no idea if there is a way to work around this. However, we might not want to split assert into a large number of methods in test classes.
But even with only 3 test methods, in the second case above `TestZeroInflatedGeneralizedPoisson_predict` class, pytests uses setup_class twice.

user: josef-pkt
issue_7427: state: open
title: handle_formula_data should sometimes return matrix
body: handle_formula_data() in https://github.com/statsmodels/statsmodels/blob/main/statsmodels/formula/formulatools.py has block 

    if X is not None:
        if data_util._is_using_pandas(Y, X):
            result = dmatrices(formula, (Y, X), depth,
                               return_type='dataframe', NA_action=na_action)
        else:
            result = dmatrices(formula, (Y, X), depth,
                               return_type='dataframe', NA_action=na_action)
    else:
        if data_util._is_using_pandas(Y, None):
            result = dmatrices(formula, Y, depth, return_type='dataframe',
                               NA_action=na_action)
        else:
            result = dmatrices(formula, Y, depth, return_type='dataframe',
                               NA_action=na_action)

The second level condition is not doing anything.  It is setting results identically either way the condition resolves.  I can understand the code testing for a Pandas object, then returning one.  I can't understand the opposite condition, the test for a pandas object failing, then returning a dataframe anyway.  Given https://patsy.readthedocs.io/en/latest/API-reference.html, should this be return_type='matrix' ?  This is probably why my memory usage is blowing up.
user: JTLehman1
issue_7418: state: open
title: ValueError("there should not be a constant in the model")
body: Thank you very much for your help, it worked but this time another error came up.

  Traceback (most recent call last):
  File "logistic.py", line 251, in <module>
    distr='logit')
  File "/home/a/statsmodels/statsmodels/miscmodels/ordinal_model.py", line 144, in __init__
    raise ValueError("there should not be a constant in the model")

I didn't actually understand from this statement. Would you give me suggestions on this problem. I really appreciate for your help.

_Originally posted by @aliyurtsevenn in https://github.com/statsmodels/statsmodels/issues/7417#issuecomment-820575569_
user: aliyurtsevenn
issue_7417: state: open
title: UnboundLocalError: local variable 'warnings' referenced before assignment
body: #### Describe the bug

[A clear and concise description of what the bug is. This should explain **why** the current behaviour is a problem and why the expected output is a better solution.]

#### Code Sample, a copy-pastable example if possible

Hi, I have installed statsmodels v0.13.0.dev0 and tried to perform ordinal logistic regression . I have encountered with the error below. 

Traceback (most recent call last):
  File "logistic.py", line 250, in <module>
    distr='logit')
  File "/home/a/statsmodels/statsmodels/miscmodels/ordinal_model.py", line 124, in __init__
    endog, labels, is_pandas = self._check_inputs(endog, exog)
  File "/home/a/statsmodels/statsmodels/miscmodels/ordinal_model.py", line 196, in _check_inputs
    warnings.warn("the endog has ordered == False, "
UnboundLocalError: local variable 'warnings' referenced before assignment

My code is below;

my_label=reversed_one["Labels"]
Attributes= reversed_one.drop(["Labels"],axis=1)

my_label=my_label.astype("category")
Attributes=Attributes.astype(float)

mod_log = OrderedModel(my_label,
                        Attributes,
                        distr='logit')
res_log = mod_log.fit(method='bfgs', disp=False)
res_log.summary()

Would you inform me about the problem. Thank you in advance! 

Best regards,





```python
# Your code here that produces the bug
# This example should be self-contained, and so not rely on external data.
# It should run in a fresh ipython session, and so include all relevant imports.
```
<details>

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `main`. If your problem has been fixed in an unreleased version, you might be able to use `main` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the main branch of this repository? It helps the limited resources if we know problems exist in the current main branch so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

A clear and concise description of what you expected to happen.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]

</details>

user: aliyurtsevenn
issue_7415: state: open
title: ENH: IVGMM: other results statistics, rsquared 
body: https://stackoverflow.com/questions/67057345/python-output-r-squared-and-rmse

IVGMM NonlinearIVGMM has a predict function given a nonlinear mean function. so it would be possible to compute fit statistics that are based on residuals. IVGMMResults has fittedvalues, resid and ssr.
response residuals are computed in `get_errors` method of the model, IVGMM.

user: josef-pkt
issue_7414: state: open
title: ENH: transformed likelihood mixin
body: just an idea, should be easy to add, and we inherit almost everything

similar to the Mixin classes for modifying models, we can use nonlinear transformation of the loglikelihood, and maximize that as an M-estimator with sandwich cov.
almost a duplicate of #3284

``loglike = self.transform(super().loglike)`

e.g.
https://projecteuclid.org/journals/annals-of-statistics/volume-38/issue-2/Maximum-Lq-likelihood-estimation/10.1214/09-AOS687.full

I ran into this because of https://arxiv.org/abs/1911.11922
I haven't really looked at any of these articles.

One possible problem as in the robust literature might be that we need an additive term for bias correction to get Fisher consistency when we have asymmetric distributions.

#7412 minimum distance estimators
#4266 freq_weights for robust estimation  (e.g. for exog outliers where weights would be constant and independent of optimization)
and other issues related to robust GLM/discrete/nongaussian

#7244 issues for adding transformation classes as reusable tools.



user: josef-pkt
issue_7412: state: open
title: ENH: minimum (gof) distance estimator
body: https://rdrr.io/rforge/distrMod/man/MDEstimator.html
https://stackoverflow.com/questions/67007706/how-to-calculate-efficient-minimum-distance-in-python-regression

**edit** https://rdrr.io/rforge/fitdistrplus/man/mgedist.html
edf based gof criteria for estimation, KS, CvM, AD following Luceno 2006

minimize gof statistic, Hellinger, AD, Cramer von Mises to estimate parametric models or parameters of distribution.

Those are robust to outliers.

I looked at Hellinger a long time ago, but no code in sandbox. 
Only related in sandbox is mutual info, and that was more as correlation measure.

this might be useful if we want to estimate predictive distributions.
#7142

maybe:
For some distributions, MLE for parameter estimation had a bad reputation, e.g. genextreme, and alternative estimators become popular in some fields, e.g. minimum spacings.
(however, even in many of those case MLE works fine with some limitations and good starting values.)

One issue for this types of estimators, tests is whether they extend to conditional distribution, e.g. in regression setting with explanatory parameters.
related to gof testing in regression models:
#7154  gof EDF tests for regression 
#5408
#3904


user: josef-pkt
issue_7411: state: open
title: REF/TST/Maint  cancor: multivariate versus stats version
body: needs checking

We still have my original canonical correlation functions in stats without test coverage.
https://codecov.io/gh/statsmodels/statsmodels/src/master/statsmodels/stats/multivariate_tools.py

in stats version

cc_ranktest looks more like what we use in econometrics (rank for instrument matrix, cointegration rank ?)
cc_stats computes all MANOVA test statistics, but no p-values, `multivariate` `CanCorr.corr_testlooks like only has Wilks' lambda



user: josef-pkt
issue_7403: state: open
title: quantreg in Linear Model raise an Exception
body: I train a quantreg model by using my dataset, there is an exception raised at **line 177**  of quantile_regression.py in package statsmodel.regression.

Exception:
![image](https://user-images.githubusercontent.com/23023313/113833591-70f9d880-97bc-11eb-8ee1-00a076d6d8b8.png)

The detail in source file is below:
![QQ图片20210407162134](https://user-images.githubusercontent.com/23023313/113834393-5b38e300-97bd-11eb-900e-88dadfb0cc32.png)

My code run with:
Python: 3.6.5
statsmodels: 0.12.2


user: MimeZoe0628
issue_7390: state: open
title: AttributeError: type object 'UnobservedComponents' has no attribute 'fix_params'
body: #### Describe the bug

[I get AttributeError: type object 'UnobservedComponents' has no attribute 'fix_params' when I run the code below.]

#### Code Sample, a copy-pastable example if possible
mod=sm.tsa.UnobservedComponents(x, level=parameter[0], seasonal=12, freq_seasonal=None, cycle=parameter[1],damped_cycle=parameter[2],stochastic_cycle=True,stochastic_seasonal=False,autoregressive=parameter[3])

with mod.fix_params({'sigma2.irregular': 0, 'sigma2.level': 0, 'sigma2.trend': 0}):
    res_restricted = model.fit()

```python
# Your code here that produces the bug
# This example should be self-contained, and so not rely on external data.
# It should run in a fresh ipython session, and so include all relevant imports.
```
<details>

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `master`. If your problem has been fixed in an unreleased version, you might be able to use `master` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the master branch of this repository? It helps the limited resources if we know problems exist in the current master so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

A clear and concise description of what you expected to happen.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]

</details>

user: perfectyayra
issue_7388: state: open
title: ENH: extreme value copulas constructed from stable dependence function
body: currently in PR #7296 EV copulas are constructed by Pickand's dependence function

As alternative we can construct them from the stable dependence function 

stable dependence function includes all terms and is usually symmetric in the pattern for components of multivariate random variables
l(v1, v2) = (v1 + v2) * A(v2 / (v1 + v2)) chapter 8, p 268 equ (8,44)

see https://github.com/statsmodels/statsmodels/pull/7296#issuecomment-803630408 and related

One advantage is that it looks easier to extend to higher dimension than bivariate, 
and it might be easier to get derivatives directly from stable dep function.

list of parametric models in terms of stable dependence function, in
section 9.2.2 Some parametric models in "Statistics of Extremes - Theory and Application" 




user: josef-pkt
issue_7387: state: open
title: What does Null value of VIF indicates
body: I'm doing linear regression analysis using python 3.7. I'm using following code to estimate VIF

    from statsmodels.stats.outliers_influence import variance_inflation_factor
    import pandas as pd
    def calc_vif(X):
    
        # Calculating VIF
        vif = pd.DataFrame()
        vif["variables"] = X.columns
        vif["VIF"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]
    
        return(vif)
I'm seeing that for some variables I have VIF as infinity & for some variable it's Null. I checked my data & I don't have any Null in data. 

For me  VIF as infinite means considering that variable as dependent variable & remaining variables as independent variable is giving R square as 1. So VIF=R square/(1-R square) is coming infinite. But I can't able to interpret Null VIF. Can you please help me?
user: niladri2909
issue_7386: state: open
title: Clipping the confidence interval to be in the [0, 1] interval for binomial proportion
body: A note in https://www.statsmodels.org/stable/generated/statsmodels.stats.proportion.proportion_confint.html says that 
**The confidence intervals are clipped to be in the [0, 1] interval in the case of ‘normal’ and ‘agresti_coull’.**

Is it reasonable or a common way to do the clipping as long as the confidence interval is outside [0, 1]?  


Many thanks if any theory or explanation is provided.
user: BCMoo
issue_7384: state: open
title: Function to generate train, test and/or validation set
body: - [ ] closes #1498
- [ ] tests added / passed. 
- [ ] code/documentation is well formatted.  
- [ ] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

<details>
 Function to generate train and test set from data frame was needed.

Added temporary directory statsmodels/statsmodels/datasplit

Added function with a dual feature in file statsmodels/statsmodels/datasplit/datasplitter.py
function: train_test_split()
1. Train test split
2. Train test validation split

Supports Pandas, Lists, and Arrays
Supports shuffle and stratify

Added test cases in file statsmodels/statsmodels/datasplit/testcases.py

</details>

user: NolanMP
issue_7381: state: open
title: Add test for unordered groups
body: Add a test to confirm that the same results are obtained when the data are not sorted by group
user: kshedden
issue_7378: state: open
title: ENH: kernel regression or smoothing with asymmetric kernels, beta, gamma
body: another followup to supporting asymmetric kernels for support (0, 1) or R+

Chen 2000 uses Gasser Muller kernel regression, requires integral of kernel over neighborhood of a point. 
That should be `cdf(ub) - cdf(lb)`, but based on skimming the paper that is not very explicit.

Chen 2002 use local linear (local polynomial smoothing), similar to Fan but we Beta kernel

Natarajan, Song 2020 use the usual Nadaraya–Watson kernel regression (local constant).
That article is recent and has good list of references

References

Chen, Song Xi. 2000. “Beta Kernel Smoothers for Regression Curves.” Statistica Sinica 10 (1): 73–91.

Chen, Song Xi. 2002. “Local Linear Smoothers Using Asymmetric Kernels.” Annals of the Institute of Statistical Mathematics 54 (2): 312–23. https://doi.org/10.1023/A:1022422002138.

Natarajan, Balasubramaniam, and Weixing Song. 2020. “Adaptive Nonparametric Regression on Finite Support.” Communications in Statistics - Theory and Methods 0 (0): 1–21. https://doi.org/10.1080/03610926.2020.1749855.

**update:** good reference for NW kernel regression using gamma kernel:
Shi, Jianhong, and Weixing Song. 2016. “Asymptotic Results in Gamma Kernel Regression.” Communications in Statistics - Theory and Methods 45 (12): 3489–3509. https://doi.org/10.1080/03610926.2014.890225.

user: josef-pkt
issue_7375: state: open
title: ENH: random sampling, rvs, for asymmetric kernel density
body: followup to PR #7364
general issue #7346

see https://github.com/statsmodels/statsmodels/pull/7364#issuecomment-793324570

I have this working for most kernels
~~invgauss and~~ recipinvgauss raise a DomainError in scipy distribution when I try the reparameterization
all others look ok.

Without reference for this, we need a lot more checking. In one example it is difficult to tell what is random noise and what is a possible mistake in the approach.
I haven't tried more than a few random runs with it, generating either 1000 or 2000 random draws.

kde on generated data matches well the original theoretical pdf and the initial kde (`kde_dgp`), see example plot below
What needs more checking is whether the observations are bunched at values.
We need some measure of random noise comparing the noise in initial draw from the theoretical distribution, with the noise in the kde generated random variables.
At least some basic Monte Carlo, to see if sampling properties match.


plot is for nobs=5000 random draws

![image](https://user-images.githubusercontent.com/440735/111001296-87de1400-8351-11eb-95cf-ee2d9a4aaa9b.png)


user: josef-pkt
issue_7369: state: open
title: chisquare and chisquare_power functions are missing in docs
body: #### Describe the bug

Following functions missing in docs:
1. statsmodels.stats.gof.chisquare
2. statsmodels.stats.gof.chisquare_power

#### Expected Output

Both functions exist in code and should appear in the documentation:
https://www.statsmodels.org/devel/stats.html#goodness-of-fit-tests-and-measures

user: dbalabka
issue_7368: state: open
title: summary_col fails with OLS regression after calling remove_data()
body: #### Describe the bug

After fitting an OLS regression model and using remove_data() to reduce the model size, when I try to summarize the results using statsmodels.iolib.summary2.summary_col() I get the following error: 

```python 
TypeError: unsupported operand type(s) for *: 'NoneType' and 'float'
```

#### Code Sample, a copy-pastable example if possible


```python
import statsmodels.formula.api as smf
from statsmodels.iolib.summary2 import summary_col

q = pd.DataFrame({'x': range(1,50000)})
q['y']=2*q.x
q['z']=q.x**2

results = smf.ols('y ~ x + z', data = q).fit()
results.remove_data()

summary_col(results)
```
<details>

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `master`. If your problem has been fixed in an unreleased version, you might be able to use `master` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the master branch of this repository? It helps the limited resources if we know problems exist in the current master so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

Here is the full traceback:

```python

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<command-1299169104690280> in <module>()
     10 results.remove_data()
     11 
---> 12 summary_col(results)

/databricks/python/lib/python3.6/site-packages/statsmodels/iolib/summary2.py in summary_col(results, float_format, model_names, stars, info_dict, regressor_order, drop_omitted)
    471 
    472     cols = [_col_params(x, stars=stars, float_format=float_format) for x in
--> 473             results]
    474 
    475     # Unique column names (pandas has problems merging otherwise)

/databricks/python/lib/python3.6/site-packages/statsmodels/iolib/summary2.py in <listcomp>(.0)
    470         results = [results]
    471 
--> 472     cols = [_col_params(x, stars=stars, float_format=float_format) for x in
    473             results]
    474 

/databricks/python/lib/python3.6/site-packages/statsmodels/iolib/summary2.py in _col_params(result, float_format, stars)
    377 
    378     # Extract parameters
--> 379     res = summary_params(result)
    380     # Format float
    381     for col in res.columns[:2]:

/databricks/python/lib/python3.6/site-packages/statsmodels/iolib/summary2.py in summary_params(results, yname, xname, alpha, use_t, skip_header, float_format)
    344     else:
    345         params = results.params
--> 346         bse = results.bse
    347         tvalues = results.tvalues
    348         pvalues = results.pvalues

/databricks/python/lib/python3.6/site-packages/statsmodels/base/wrapper.py in __getattribute__(self, attr)
     33             pass
     34 
---> 35         obj = getattr(results, attr)
     36         data = results.model.data
     37         how = self._wrap_attrs.get(attr)

/databricks/python/lib/python3.6/site-packages/statsmodels/tools/decorators.py in __get__(self, obj, type)
     95         if _cachedval is None:
     96             # Call the "fget" function
---> 97             _cachedval = self.fget(obj)
     98             # Set the attribute in obj
     99             # print("Setting %s in cache to %s" % (name, _cachedval))

/databricks/python/lib/python3.6/site-packages/statsmodels/regression/linear_model.py in bse(self)
   1600     @cache_readonly
   1601     def bse(self):
-> 1602         return np.sqrt(np.diag(self.cov_params()))
   1603 
   1604     @cache_readonly

/databricks/python/lib/python3.6/site-packages/statsmodels/base/model.py in cov_params(self, r_matrix, column, scale, cov_p, other)
   1178             else:
   1179                 if scale is None:
-> 1180                     scale = self.scale
   1181                 cov_p = self.normalized_cov_params * scale
   1182 

/databricks/python/lib/python3.6/site-packages/statsmodels/tools/decorators.py in __get__(self, obj, type)
     95         if _cachedval is None:
     96             # Call the "fget" function
---> 97             _cachedval = self.fget(obj)
     98             # Set the attribute in obj
     99             # print("Setting %s in cache to %s" % (name, _cachedval))

/databricks/python/lib/python3.6/site-packages/statsmodels/regression/linear_model.py in scale(self)
   1507     @cache_writable()
   1508     def scale(self):
-> 1509         wresid = self.wresid
   1510         return np.dot(wresid, wresid) / self.df_resid
   1511 

/databricks/python/lib/python3.6/site-packages/statsmodels/tools/decorators.py in __get__(self, obj, type)
     95         if _cachedval is None:
     96             # Call the "fget" function
---> 97             _cachedval = self.fget(obj)
     98             # Set the attribute in obj
     99             # print("Setting %s in cache to %s" % (name, _cachedval))

/databricks/python/lib/python3.6/site-packages/statsmodels/regression/linear_model.py in wresid(self)
   1497     def wresid(self):
   1498         return self.model.wendog - self.model.predict(
-> 1499             self.params, self.model.wexog)
   1500 
   1501     @cache_readonly

/databricks/python/lib/python3.6/site-packages/statsmodels/regression/linear_model.py in predict(self, params, exog)
    343             exog = self.exog
    344 
--> 345         return np.dot(exog, params)
    346 
    347     def get_distribution(self, params, scale, exog=None, dist_class=None):

TypeError: unsupported operand type(s) for *: 'NoneType' and 'float'

```

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

INSTALLED VERSIONS
------------------
Python: 3.6.5.final.0
OS: Linux 5.4.0-1037-aws #39~18.04.1-Ubuntu SMP Fri Jan 15 02:48:42 UTC 2021 x86_64
byteorder: little
LC_ALL: None
LANG: C.UTF-8

Statsmodels
===========

Installed: 0.9.0 (/databricks/python/lib/python3.6/site-packages/statsmodels)

Required Dependencies
=====================

cython: 0.28.2 (/databricks/python/lib/python3.6/site-packages/Cython)
numpy: 1.16.2 (/databricks/python/lib/python3.6/site-packages/numpy)
scipy: 1.1.0 (/databricks/python/lib/python3.6/site-packages/scipy)
pandas: 0.25.3 (/databricks/python/lib/python3.6/site-packages/pandas)
    dateutil: 2.8.1 (/databricks/python/lib/python3.6/site-packages/dateutil)
patsy: 0.5.0 (/databricks/python/lib/python3.6/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.2.1 (/databricks/python/lib/python3.6/site-packages/matplotlib)
    backend: agg 
cvxopt: Not installed
joblib: 0.14.0 (/databricks/python/lib/python3.6/site-packages/joblib)

Developer Tools
================

IPython: 6.4.0 (/databricks/python/lib/python3.6/site-packages/IPython)
    jinja2: 2.10 (/databricks/python/lib/python3.6/site-packages/jinja2)
sphinx: Not installed
    pygments: 2.2.0 (/databricks/python/lib/python3.6/site-packages/pygments)
nose: 1.3.7 (/databricks/python/lib/python3.6/site-packages/nose)
pytest: Not installed
virtualenv: 16.0.0 (/databricks/python/lib/python3.6/site-packages)

</details>

I realize the version of statsmodels I am using is old. I am doing the analysis on a remote server, so I have limited ability to upgrade packages.
user: duncanhobbs
issue_7355: state: open
title: Changed endog and exog if given any fixed parameters
body: - [ ] closes #xxxx
- [ ] tests added / passed. 
- [ ] code/documentation is well formatted.  
- [ ] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

<details>


**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in master and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/master -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>

user: madhushree14
issue_7351: state: open
title: ENH: distribution derivatives
body: We have distribution F(x), density f(x) and two derivatives f'(x) and f''(x)
We need those to compute bias, variance, mean squared error and optimal bandwidth in kernel distribution or density estimation
e.g. #7346

Some of this computation should be reusable components,

e.g. 
loc scale family z = (x - m) / s

F(x|m, s) = F(z | 0, 1)
f(x|m, s) = 1 / s f(z | 0, 1)
f'(x| .) = 1 / s**2 f''(z|0, 1)

but also 
d f'(z) / d m = df /dz * dz/dm =  - 1 / s  (f'(z|0, 1)  (reuse f' )

similarly for distribution on R+ with only scale and loc is fixed at zero

shape parameters are extra and cannot be handled like m, s using chain rule in z 

other usage
in MLE we need d logf /d m, d logf / d s as score functions, which can also reuse f' = df / dz
i.e. can be used as part of score factor

Note f' is deriv2_inverse in CDFLink.

(I don't remember if I also needed it for influence measures.)

for asymmetric kernels: Those are themselves distributions on R+. If we have the derivatives of the distributions, then we should also be able to get the derivatives of the kernels and kernel densities.


extension: distribution based on transformation e.g. Bernbaum-Saunders
f' should be composed of derivative of transformation and derivative of base distribution, see #7246

I guess there are not many reusable components in discrete distributions because they are not scale, or loc-scale families
(maybe in exponential family with canonical link)


user: josef-pkt
issue_7349: state: open
title: Ljung-box test using fixed lag value of 1 in ARIMA model
body: I am using an ARIMA model to predict the time series values. I identify the best fit ARIMA model using the AIC value and it turns out that for all the different orders that I tried, the best AIC is returned for the ARIMA order `(4, 0, 1)`.

As I am using `statsmodels.tsa.arima.model.ARIMA` package, I could see the description of the fitted model using `model.summary()` and I could observe the following results for tests:
```
===================================================================================
Ljung-Box (L1) (Q):                   0.00   Jarque-Bera (JB):              4412.50
Prob(Q):                              0.99   Prob(JB):                         0.00
Heteroskedasticity (H):               5.21   Skew:                            -0.01
Prob(H) (two-sided):                  0.00   Kurtosis:                        12.17
===================================================================================
```

I also manually ran the test using the `statsmodels.stats.diagnostic.acorr_ljungbox` and `statsmodels.stats.stattools.jarque_bera` packages respectively. I am using the following code snippet for the same:
```
jbTest = jarque_bera(model.resid)
print("JBTest:", jbTest)
lbTest = acorr_ljungbox(model.resid, model_df=p_+q_, lags=[20], boxpierce=False)
print("LBTest:", lbTest)
```

where, `model` is my best-fit ARIMA model and `p_` and `q_` are the respective orders of the model. The output from the snippet:

```
JBTest: (4412.481863170613, 0.0, -0.008442090306796898, 12.171354626003195)
LBTest: (array([32.85885778]), array([0.00490945]))
```

I can observe that the `Jarque-Bera test` output is close to the approximations from the model. However, it looks like the `Ljunge-box test` output is significantly small and does not match with the output from model. I am not sure if I am mis-interpreting the output from the model description versus the output from the `acorr_ljungbox` function. 

However, when I changed my code to use `lbTest = acorr_ljungbox(model.resid, lags=[1], boxpierce=False)` instead of above snippet, I got values very close to the estimates from the model: `LBTest: (array([0.000103]), array([0.9919025]))`. I did not understand why we are using the `lag=1` as found in the class [here][1].

Also, I am confused about the theoretical understanding of the `nobs_diffuse` value used in the `test_serial_correlation` function in same class. Any pointers would be really helpful!

I am using `statsmodels` version `0.12.2`. 


  [1]: https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/statespace/mlemodel.py#L4511

user: chintanr97
issue_7347: state: open
title: Modeling observed data as AR(1) in DynamicFactor
body: #### Is your feature request related to a problem? Please describe
When specifying the DynamicFactor model, I have an option factor_order to model the unobserved factor has an AR(P) but do not have an option to model the observed data as an AR(p). I tried to add lags of endog in as exog n the model specification, but this did not work since it returned interaction terms between the lagged series in endog. 


#### Describe the solution you'd like
If a solution exist to this problem, i would very much appreciate it  if you pointed me in the right direction. Otherwise, it would be awesome to have a edog_order kwarg. 

#### Additional context
in sum, this is the model we are trying to estimate : 

![Capture d’écran, le 2021-02-24 à 16 31 10](https://user-images.githubusercontent.com/64922990/109068392-cb801f00-76bd-11eb-9418-dd23ef2c79c9.png)

Where the Wt matrix is the matrix of the lag of Yt. Right now, Γt is a full matrix when i try to make it a diagonal matrix

P.S. English isn't my first language, if my explanation isn't clear, let me know and I'll rephrase.

Thanks!
user: philtousi
issue_7346: state: open
title: ENH: kernel density estimation with asymmetric kernels, on R+, unit interval
body: part of #7338

beta, gamma, invgauss and recipinvgauss kernels can be obtained through sccipy's  distributions, with appropriate parameterization.
Birnbaum-Saunders (fatiguelife) should also be possible but I haven't tried yet.

I don't know if MultivariateKDE is setup for asymmetric kernels. In contrast to symmetric kernels, the kernel does not depend on a the distance `t - x`, but it's a lonlinear function K(t, x) where x becomes a shape parameter and bandwidth is a scale parameter.

easy:
- pdf: just uses pdf of distribution averaged over sample points
- cdf: AFAICS, this just uses the `sf` of the distribution averaged over sample points

maybe easy
- add weights so we can use it for binned, histogram data
- rvs: I haven't tried yet, but I guess we can use rvs of distribution in a mixture, (I have something similar for bernstein beta histogram smoothing).

extras
- bandwidth selection
  - IMSE optimal bandwidth, explicit formulas in articles, requires density and first two derivatives of density
  - reference rules of thumb: it's possible to compute optimal bandwidth for a reference distribution on R+ like log-normal or gamma. One reference stated that log-normal reference bw is too small and not recommended in practise. Other articles estimate a gamma distribution and estimate the optimal bandwidth using it as reference distribution
  - cross-validation, .... I didn't look at those, available for MultivariateKDE
- var, mse at a point: also explicit formulas involving density and derivatives

The last part needs kernel specific formulas, i.e. we need kernel classes with extras. Currently, I'm working with just simple functions that compute kernel-pdf or kernel-cdf

other targets
- hazard function
- density derivatives

multivariate
- The univariate kernels can be easily extended to multivariate case using product kernel. as in MultivariateKDE
- There is a literature on bias reduction for multivariate product kernels, and for using non-product multivariate kernel.
   I guess that's not a priority.

performance, speedups, not clear to me yet
- currently I'm vectorizing in sample points, but loop over evaluation/prediction points
- histogram/binning: I guess for large samples we can switch to binning as in fft kde, except we cannot use convolution as speedup

kernels

- unit interval, cube: beta
- R+: gamma and Birnbaum-Saunders seems to be good, others less so
   gamma allows unbound prob at zero, invgauss and recipinvgauss force prob(x=0) = 0
   designer distributions: using power BS or similar transformation we can get different kernels. I don't know if any of them have a large advantage.

tails

I haven't seen any references yet. But the kernels should imply different tail behaviors, e.g. can we get heavy tails?
I guess we might want to choose kernels depending on behavior around x=0 boundary and on the behavior in tails.


status
Currently I have mainly the scipy distribution parameterization of the kernels, which gives pdf and cdf (and maybe rvs)
They work if I choose the bandwidth by visual inspection.

I would like to park those functions and leave the rest for another year.

(list of references coming later)
user: josef-pkt
issue_7344: state: open
title: ENH: integrate the interface to get prediction/forecast results in UnobservedComponents
body: #### Is your feature request related to a problem? Please describe

Sometimes I have difficulty in getting prediction/forecast results of UnobservedComponents.

Because I have to use different methods and properties to get the results depending on the situation,
I often get confused which one to use. Moreover, the code tends to get long and complicated.

In the current implementation, there are two ways to get results:

- UnobservedComponentsResults.get_prediction()
    - one-step-ahead prediction and forecast (sum of all components)
- level, trend, seasonal, cycle properties of UnobservedComponentResults
    - filtered or smoothed state and its covariance for each component

Moreover, in some cases no interface provides the results we'd like to get 
and we have to look for other methods and properties.
For example, 

- out-of-sample forecast for each component
- prediction/forecast results of mixed components (especially variance)

Here is an example of local level + seasonality model (see the details below):
<details>

```python
import statsmodels.api as sm

nile = sm.datasets.get_rdataset("Nile")

model = sm.tsa.UnobservedComponents(nile.data['value'], level='local level', seasonal=10)
results = model.fit()
forecast = results.get_forecast(steps=4)
```

In order to get the out-of-sample forecast for level component, we have to access PredictionResults.prediction_results.results:

```python
# forecast of level component
forecast.prediction_results.results.predicted_state[0]
# forecast variance of level component
forecast.prediction_results.results.predicted_state_cov[0][0]
```

In order to get smoothed variance of level + seasonality (without irregularity component), We have to find covariance matrix.

```python
# smoothed states of level + seasonal components
results.level['smoothed'] + results.seasonal['smoothed']

# smoothed variance of level + seasonal component
results.smoothed_state_cov[:2, :2].sum(0).sum(0)
```

</details>
On the other hand, when you use KFAS of R, you can get the same results only with `predict` function.

#### Describe the solution you'd like

I'd like to make a unified interface by extending `get_prediction` method of UnobservedComponentResults.
It would be simple to get every possible pattern of prediction/forecast only with this method.

My idea is that we need three new option arguments:

- state: str
Select which state you get, `one-step-ahead`, `filtered`, or `smoothed`.
If  the selected state does not exist, use `one-step-ahead` (e.g. out-of-sample forecast).

- component: str or list of str
Select which component you get `level`, `trend`, `seasonal`, `cycle`, `exog`, or `all`.
If the input is list of component names (such as `['level', 'exog']`), return sum of the components.

- observed: bool
If true, add observation noise covariance to the result.
(In KFAS of R, it corresponds to `interval={'confidence' / 'prediction'}` in `predict` function)
user: tatamiya
issue_7341: state: open
title: get_margeff to GLM link=probit() and link=logit()
body: #### Is your feature request related to a problem? Please describe
I see that get_margeff is an available method for probit and logit regression. However, var_weights is not implemented for both logit and probit and therefore I need to use GLM and specify the correct family. The problem is that get_margeff is not available for get_GLMResults. 

#### Describe the solution you'd like
Would it be possible to add the get_margeff method to GLM results? 

#### Describe alternatives you have considered
Adding var_weights to logit and probit would also be awesome but i recall reading that it is a difficult thing to do. 

#### Additional context
I searched the documentation a lot but I might have missed something. If so it would be very appreciated if someone pointed me in the right direction. 

Thank you. 
Alek
user: alekracicot
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_7340: state: open
title: BUG/Design: input/output variable naming: use `{var}_frame`, not `{var}_df` for dataframe
body: based on a complaint in a japanese twitter thread

https://www.statsmodels.org/stable/generated/statsmodels.stats.diagnostic.acorr_ljungbox.html

uses model_df for degrees of freedom, and return_df for dataframe

abbreviation `df` should be reserved for degrees of freedom in code, 
(I also mostly avoid df for dataframe in notebooks)

for summary type results I converged on using `frame`, e.g. `summary_frame`
more general patsy (and maybe we, at least in datasets) use `return_pandas`
user: josef-pkt
issue_7338: state: open
title: ENH: mixing together: histogram smoothing, kde on R+, transformation of distributions
body: (going a bit in circles)

- Bernstein polynomial density estimation is a kind of kernel estimation based on histogram data, or grouped probabilities. domain is interval [0,1]  #7296
- in fast kde we use binned, histogram data but with many points as a speedup, so we can use convolution or fft
- KDEMultivariate has continuous, categorical and discrete, but no restricted continuous domain kernels
- in nonlinear transformation we use some of them to change domain, eg. R to R+ to [0, 1]
- bernstein polynomiald distribution and kde have a mixture distribution interpretation that can be used to generate random numbers


Bouezmarni, Taoufik, and Olivier Scaillet. 2005. “Consistency of Asymmetric Kernel Density Estimators and Smoothed Histograms with Application to Income Data.” Econometric Theory 21 (2): 390–412.

treats both kde and histogram smoothing, see equ. 5 and 6 which are close to what we do for bernstein polynomial distribution on [0, 1]

Chen, Song Xi. 2000. “Probability Density Function Estimation Using Gamma Kernels.” Annals of the Institute of Statistical Mathematics 52 (3): 471–80. https://doi.org/10.1023/A:1004165218295.

example using gamma kernel for R+

Micheaux, Pierre Lafaye de, and Frédéric Ouimet. 2020. “A Study of Seven Asymmetric Kernels for the Estimation of Cumulative Distribution Functions,” November. https://arxiv.org/abs/2011.14893v1.

Kakizawa, Yoshihide. 2018. “Nonparametric Density Estimation for Nonnegative Data, Using Symmetrical-Based Inverse and Reciprocal Inverse Gaussian Kernels through Dual Transformation.” Journal of Statistical Planning and Inference 193 (February): 117–35. https://doi.org/10.1016/j.jspi.2017.08.008.

The article uses dual transformation which is power Birnbaum-Sanders, or power t minus t inverse #7246, refers to next reference, Yang 2006, for it.
Defines classes of kernel distributions based on transformed symmetric/elliptical distributions

Yang, Zhenlin. 2006. “A Modified Family of Power Transformations.” Economics Letters 92 (1): 14–19. https://doi.org/10.1016/j.econlet.2006.01.011.

and related articles, e.g. for other kernel distributions on R+

The above are univariate, for bernstein polynomials we also have multivariate versions.
I have not yet tried transformations for multivariate bernstein polynomial distributions to other domains of the distributions, e.g. use marginal distributions on R+. An advantage of that might be that we get interval lengths increasing in the upper tail, based on transformed equal spaced intervals on [0,1]

(I only skimmed parts of the articles, and haven't looked at details yet.)


Computational:
I guess we cannot use convolution or fft on binned, histogram data on restricted domains like R+ with asymmetric kernels.
This will limit the speed and number of histogram bins that we can handle, similarly to bernstein.
In the univariate case using 500 (default for fft kde IIRC) or 1000 bins should not be a computational problem for vectorized solution.

I think the basic smoothers should be easy to implement. That might be useful for graphics similar to histograms.
Bandwidth selection and standard errors will be more work.






user: josef-pkt
issue_7337: state: open
title: BUG: Lowess extrapolation missing output values.
body: #### Bug description

Suppose input values are defined on [5, 14], Then if you use `lowess` and use `xvals` to get values on a bigger range of Xs (for example [0, 19]), the output may contain undefined values even on the input range of Xs. Additionally, different `frac` values produce different behaviours that are not obvious.
 
In the last example, output is undefined even if you use the same output X values as input X values.

#### Code example

```python
from statsmodels.nonparametric.smoothers_lowess import lowess
import numpy as np
import sys

np.set_printoptions(precision=6, linewidth=sys.maxsize, suppress=True, threshold=sys.maxsize)

x = np.arange(5, 15).astype(np.float)
y = np.arange(5, 15).astype(np.float)
out_x = np.arange(0, 20).astype(np.float)

lowess(y, x, xvals=out_x, frac=0.6)
# array([-0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19.])

lowess(y, x, xvals=out_x, frac=0.5)
# array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., nan, nan, nan, nan, nan, nan, nan, nan])

lowess(y, x, xvals=out_x, frac=0.4)
# array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19.])

lowess(y, x, xvals=out_x, frac=0.3)
# array([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan])

lowess(y, x, xvals=x, frac=0.5)
# array([ 5.,  6.,  7.,  8.,  9., 10., 11., nan, nan, nan])
```

#### Expected output

Ideally, `lowess` output should be defined at least on the range of input Xs.

Also, values do not become NaNs gradually (for example undefined below some `frac` value and defined above).
Because of that it looks like a bug and not like an expected behaviour.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

```

INSTALLED VERSIONS
------------------
Python: 3.6.10.final.0
OS: Linux 5.4.95-1-MANJARO #1 SMP PREEMPT Thu Feb 4 00:38:02 UTC 2021 x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

statsmodels
===========

Installed: 0.12.2 (/home/timofey/.miniconda3/lib/python3.6/site-packages/statsmodels)

Required Dependencies
=====================

cython: 0.29.21 (/home/timofey/.miniconda3/lib/python3.6/site-packages/Cython)
numpy: 1.19.1 (/home/timofey/.miniconda3/lib/python3.6/site-packages/numpy)
scipy: 1.5.2 (/home/timofey/.miniconda3/lib/python3.6/site-packages/scipy)
pandas: 1.1.5 (/home/timofey/.miniconda3/lib/python3.6/site-packages/pandas)
    dateutil: 2.8.1 (/home/timofey/.miniconda3/lib/python3.6/site-packages/dateutil)
patsy: 0.5.1 (/home/timofey/.miniconda3/lib/python3.6/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: Not installed
cvxopt: Not installed
joblib: Not installed

Developer Tools
================

IPython: 7.16.1 (/home/timofey/.miniconda3/lib/python3.6/site-packages/IPython)
    jinja2: 2.11.3 (/home/timofey/.miniconda3/lib/python3.6/site-packages/jinja2)
sphinx: Not installed
    pygments: 2.7.4 (/home/timofey/.miniconda3/lib/python3.6/site-packages/pygments)
pytest: Not installed
virtualenv: 16.7.5 (/home/timofey/.miniconda3/lib/python3.6/site-packages)
```

</details>

user: tprodanov
issue_7336: state: open
title: ENH: more `misc` distributions, penalized, ....
body: We need better advertising for generic features and models, and make it easier to use those.
Currently, we have `miscmodels` as example and test cases for GenericLikelihoodModel, including prototype models like the current Orderedmodel.

We need better illustration for other generic models and distributions.

example

- penalized mixin: it is used only in GAM and in unit tests. No other models to show it's usage or to use it with current model classes, eg. GLMPenalized. 
- copula distributions: current plan is only to provide copulas itself and a generic CopulaDistribution class. 
   I would like two test cases that can be extended later to full models
   - copula with GEV margins for multivariate extreme value applications
   - bivariate beta distribution
- generated/transformed distributions; my current plan for ... is to add the transformation classes and a generic TransformedDistribution class. We need specific cases, e.g. sinh-arcsinh normal and t distributions.

Multivariate discrete models using copulas will need more work, because current WIP uses only continuous distributions.
e.g. bivariate Logit, Probit, Poisson


user: josef-pkt
issue_7335: state: open
title: ENH Design: parameter classes and appropriate link functions
body: lmfit and https://github.com/zfit/zfit/tree/develop/zfit use parameter classes for nonlinear fitting.
We mostly avoided this, except for unpacking and a few other cases, because it wasn't really needed.

Once we get into more multiparameter models, it will become useful to add more generic structure and case specific information.

The main usecase for optimization would be link selection for range constraints.
e.g. copulas can have several parameters with different constraints.

If we have a generic definition of the range of a parameter, we can automatically select a default link,
The parameters are intermediate like mu in GLM, or mu and dispersion in Beta, prob in zero inflated, ...
Those parameters can be connected to a linear predictor based on final parameters, as in GLM.

example would be to support mix and match cases like

- copula distributions with user specified distributions or marginal models
- flexible multiparameter distributions with generators or transformations based on user specified base distribution





user: josef-pkt
issue_7334: state: open
title: Johansen Test Produces Incorrect Values
body: I'm attempting to replicate Ernie Chan's example 2.7 outlined in his seminal book Algorithmic Trading (page 55) in python. There isn't much pertinent material found online but the statsmodel library is very helpful. However the eigenvector my code produces looks incorrect in that the values do not properly correlate to the test data. Here's the code in several steps:

```
import pandas as pd
    import yfinance as yf
    from datetime import datetime
    from dateutil.relativedelta import relativedelta
    
    years = 5
    today = datetime.today().strftime('%Y-%m-%d')
    lastyeartoday = (datetime.today() - relativedelta(years=years)).strftime('%Y-%m-%d')
    symbols = ['BTC-USD', 'BCH-USD','ETH-USD']
    
    df = yf.download(symbols, 
                          start=lastyeartoday, 
                          end=today, 
                          progress=False)
    df = df.dropna()
    data = pd.DataFrame()
    for symbol in symbols:
        data[symbol] = df['Close'][symbol]
    
    data.tail()

This produces the following output:

[![enter image description here][1]][1]

Let's plot the the three series:

    # Plot the prices series
    import matplotlib.pyplot as plt
    %matplotlib inline
    for symbol in symbols:
        data[symbol].plot(figsize=(10,8))
    
    plt.show()
```

Graph:

![2021-02-18_2127](https://user-images.githubusercontent.com/8457499/108504927-f7169a00-72b6-11eb-90fc-b02f56b65f35.png)


Now we run the cointegrated Johansen test on the dataset:

```
import numpy as np
    import pandas as pd
    import statsmodels.api as sm
    
    # data = pd.read_csv("http://web.pdx.edu/~crkl/ceR/data/usyc87.txt",index_col='YEAR',sep='\s+',nrows=66)
    # y = data['Y']
    # c = data['C']
    
    from statsmodels.tsa.vector_ar.vecm import coint_johansen
    
    """
        Johansen cointegration test of the cointegration rank of a VECM
    
        Parameters
        ----------
        endog : array_like (nobs_tot x neqs)
            Data to test
        det_order : int
            * -1 - no deterministic terms - model1
            * 0 - constant term - model3
            * 1 - linear trend
        k_ar_diff : int, nonnegative
            Number of lagged differences in the model.
            
        Returns
        -------
        result: Holder
        An object containing the results which can be accessed using dot-notation. The object’s attributes are
    
        eig: (neqs) - Eigenvalues.
        evec: (neqs x neqs) - Eigenvectors.
        lr1: (neqs) - Trace statistic.
        lr2: (neqs) - Maximum eigenvalue statistic.
        cvt: (neqs x 3) - Critical values (90%, 95%, 99%) for trace statistic.
        cvm: (neqs x 3) - Critical values (90%, 95%, 99%) for maximum eigenvalue statistic. 
        method: str “johansen”
        r0t: (nobs x neqs) - Residuals for Δ𝑌.
        rkt: (nobs x neqs) - Residuals for 𝑌−1.
        ind: (neqs) - Order of eigenvalues.
        """
    
    def joh_output(res):
        output = pd.DataFrame([res.lr2,res.lr1],
                              index=['max_eig_stat',"trace_stat"])
        print(output.T,'\n')
        print("Critical values(90%, 95%, 99%) of max_eig_stat\n",res.cvm,'\n')
        print("Critical values(90%, 95%, 99%) of trace_stat\n",res.cvt,'\n')
    
    
    # model with constant/trend (deterministic) term with lags set to 1
    joh_model = coint_johansen(data,0,1) # k_ar_diff +1 = K
    joh_output(joh_model)
```


![2021-02-18_2129](https://user-images.githubusercontent.com/8457499/108504999-0dbcf100-72b7-11eb-9766-4f44e3092d41.png)


As the test values are far greater than the critical values we can rule out the null hypothesis and declare that there is very high cointegration between the three crypto pairs.

Now let's print the eigenvalues:

array([0.02903038, 0.01993949, 0.00584357])

The first row of our eigenvectors should be considered the strongest in that it has the shortest half-life for mean reversion:

```
    print('Eigenvector in scientific notation:\n{0}\n'.format(joh_model.evec[0]))
    print('Eigenvector in decimal notation:')
    i = 0
    for val in joh_model.evec[0]:
        print('{0}: {1:.10f}'.format(i, val))
        i += 1

```
Result:

Eigenvector in scientific notation:
[ 2.21531848e-04 -1.70103937e-04 -9.40374745e-05]

Eigenvector in decimal notation:
0: 0.0002215318
1: -0.0001701039
2: -0.0000940375

And here's the problem I've mentioned in my introduction. Per Ernie's description these values should correlate with the hedge ratios for each of the crosses. However they are a) way to small b) two of them are negative (obviously incorrect for these three crypto pairs) and c) seem to be completely uncorrelated to the test data (e.g. BTC is obviously trading at a massive premium and should be the smallest value).

Now I'm no math genius and there's a good chance that I messed up somewhere, which is why I provided all the code/steps involved for replication. However the process seems pretty straightforward and given the output there is the possibility that your Johansen test produces incorrect results. If you could run the code on your end and let me know your impressions I'd appreciate it.
user: mmehrle
issue_7332: state: open
title: ENH: Add games howell test
body: - [ ] closes #xxxx
- [X] tests added / passed. 
- [X] code/documentation is well formatted.  
- [X] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

Adding the Games Howell Post Hoc Test.  This is not currently implemented in R or JMP (though there is an add one can get) and AFAIK no other packages.  However, as I note in the tests, you can easily use this [R pub](https://rpubs.com/aaronsc32/games-howell-test#:~:text=Although%20rather%20similar%20to%20Tukey's,equal%20variances%20and%20sample%20sizes.&text=Since%20the%20Games%2DHowell%20test,approaches%20such%20as%20Tukey's%20test) to code up the same test (the pub suggests there is a package one can use, but it does not work, it's better to use the code snippet) and verify the test values.  I added screenshots in the details section below of the results in R using the code linked for the test I included in the PR and the results of using [R version of connecting letters](https://cran.r-project.org/web/packages/multcompView/multcompView.pdf) for the test added for that.

Because this test is an alternative to Tukey HSD but does not assume equal variances and sample sizes, I followed the code for that statistical test (as per a discussion in the google group) as a model for where to put functions/add them to classes, etc. as well as for the tests.  This may not be optimal, but follows existing patterns while making sure all new code is well documented and uses `black` for style.



<details>
R Games Howell results

![Screen Shot 2021-02-19 at 3 21 22 AM](https://user-images.githubusercontent.com/5706027/108499558-53e86500-7263-11eb-9a59-10ca87c21d46.png)



R connecting letters results (the R package calls it `multcompLetters`)
![Screen Shot 2021-02-19 at 3 29 43 AM](https://user-images.githubusercontent.com/5706027/108499380-0a981580-7263-11eb-83e8-5f11a86a3439.png)

**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in master and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/master -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>

user: ameliajtaylor
issue_7328: state: open
title: FAQ: GLM Tweedie Distribution and Identity Link 
body: **update** as FAQ
warnings about links that do not impose domain restrictions for the family
related FAQ issue #2208 which is mostly a discussion when we figured this out the first time.
my comment below is a good summary

---

#### Describe the bug

While exploring Statsmodels GLM functionalities, I noticed that I'm able to fit a GLM with a Tweedie distribution and identity link. However I noticed that on the documentation page identity link is not one of the links listed for Tweedie https://www.statsmodels.org/stable/generated/statsmodels.genmod.families.family.Tweedie.html) and I get the same conclusion if I run sm.families.family.Tweedie.links.

Is the Tweedie distribution and identity link combination currently supported? If so, could this be updated in the documentation? If not, could this be a bug and this combination should have errored out when it was being passed to the training code? 

#### code sample
import numpy as np
import statsmodels
import statsmodels.api as sm 

#create predictor dataset 
np.random.seed(0)
n=100000
p=3
X = np.random.rand(n,p) 

#coefficients
beta = [1,2,1]

#generate target 
mu = np.matmul(X,beta)
Y = np.random.poisson(lam=mu)

#fit model using Tweedie distribution and identity link
model=sm.GLM(endog=Y, exog=X, family=sm.families.Tweedie(link=sm.families.links.identity()))
fitted_model = model.fit()
fitted_model.summary()

#### Expected Output
If Tweedie and identity link combination is not supported, I would have expected this to error out instead of completing the training. If it is, it would be great if this could be added to the documentation so users know this combination is available for use. 

user: ajiang2021
issue_7327: state: open
title: BUG: Fix libsturng issue #6541
body: - [X] closes #6541
- [X] tests added / passed. 
- [X] code/documentation is well formatted.  
- [X] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

I added internal comments to help our future selves.  Based on the literature cited and R code tested, we shifted to returning a useful error message for this unsupported case. 

Also, to reiterate (from the issue), R does not support this case either in their `ptukey` function, it returns a "NAN"s message.  

<details>


**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in master and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/master -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>

user: ameliajtaylor
issue_7326: state: open
title: BUG: Fix libsturng issue #7324
body: - [X] closes #7324
- [X] tests added / passed. 
- [X] code/documentation is well formatted.  
- [X] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

Implements the bug fix proposed in #7324 and adds a test that will cause a failure on master and tests the change we made against a value from R.

<details>


**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in master and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/master -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>

user: ameliajtaylor
issue_7324: state: open
title: libqsturng _psturng fails when degrees of freedom is between 2 and 2.5 and fminbound chooses a p value between 0.9 and 0.9175
body: #### Describe the bug

This bug is related, but different from the bug I first reported [last February](https://github.com/statsmodels/statsmodels/issues/6541). And might be easier to solve.  When the degrees of freedom (v) is >= 2 for and q not in one of the two boundary cases given in `psturng` we get to [line 839](https://github.com/statsmodels/statsmodels/blob/4e77b4b07545f848fe001c24197cf49214a8c2d1/statsmodels/stats/libqsturng/qsturng_.py#L839) and run `fminbound` on `optfunc` which computes `_qsturng`  for p from 0.1 to 0.999.  Depending on `v` and `p` the code needs to interpolate in both dimensions.  The problem is that the way the sets for `_select_vs` and `_select_ps` are lined up, when `2 < v < 2.5` and `0.9 < p < 0.9175`.  In this case, `_select_vs` returns the tuple `1, 2, 3` and `_select_ps` returns `0.850, 0.9, 0.95`.  But the pair `(0.85, 1)` is not in the table --- which it is not expected to be as previous noted, the papers only support `p >= 0.9` for `v < 2`.  

The easiest solution (which also seems correct) seems to be to shift [line 500](https://github.com/statsmodels/statsmodels/blob/4e77b4b07545f848fe001c24197cf49214a8c2d1/statsmodels/stats/libqsturng/qsturng_.py#L839) to be `elif p>= 0.900`.  This would jibe with the previous lines and mathematically, I see no reason why not to do this and then would jibe with the `v's` selected.  

#### Code Sample, a copy-pastable example if possible
Note the statsmodels code has a print value in it that actually helps.  From that we can see we have the case where `p` is just above `0.9` and with `v < 2.5` then as noted above, the code tries to look up `(0.85, 1)`, which as noted, is not in the list.  This is fixed by not having it look that up and keep interpolation of values for `p > 0.9` within p's select all over 0.9. 

```python
from statsmodels.stats.libqsturng import psturng
psturng(12.5872, 24, 2.02)
```
<details>

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `master`. If your problem has been fixed in an unreleased version, you might be able to use `master` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the master branch of this repository? It helps the limited resources if we know problems exist in the current master so that they do not need to check whether the code sample produces a bug in the next release.

As the person who posted the other bug, I know this hasn't come up before and I know it happens both on release and master.  

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

I expected this call to complete with a value that is in between the values we get if we have fewer groups or more (e.g. 14 and 36) which shift the values of `p` tested in the `fminbound` function to fit within the current options without hitting this bug.  

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]
INSTALLED VERSIONS
------------------
Python: 3.7.7.final.0
OS: Darwin 18.7.0 Darwin Kernel Version 18.7.0: Tue Nov 10 00:07:31 PST 2020; root:xnu-4903.278.51~1/RELEASE_X86_64 x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

statsmodels
===========

Installed: 0.12.1 (/Users/ataylor/.pyenv/versions/3.7.7/envs/gameshowell_test/lib/python3.7/site-packages/statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.18.4 (/Users/ataylor/.pyenv/versions/3.7.7/envs/gameshowell_test/lib/python3.7/site-packages/numpy)
scipy: 1.5.4 (/Users/ataylor/.pyenv/versions/3.7.7/envs/gameshowell_test/lib/python3.7/site-packages/scipy)
pandas: 1.1.4 (/Users/ataylor/.pyenv/versions/3.7.7/envs/gameshowell_test/lib/python3.7/site-packages/pandas)
    dateutil: 2.8.1 (/Users/ataylor/.pyenv/versions/3.7.7/envs/gameshowell_test/lib/python3.7/site-packages/dateutil)
patsy: 0.5.1 (/Users/ataylor/.pyenv/versions/3.7.7/envs/gameshowell_test/lib/python3.7/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: Not installed
cvxopt: Not installed
joblib: 1.0.0 (/Users/ataylor/.pyenv/versions/3.7.7/envs/gameshowell_test/lib/python3.7/site-packages/joblib)

Developer Tools
================

IPython: 7.20.0 (/Users/ataylor/.pyenv/versions/3.7.7/envs/gameshowell_test/lib/python3.7/site-packages/IPython)
    jinja2: 2.11.2 (/Users/ataylor/.pyenv/versions/3.7.7/envs/gameshowell_test/lib/python3.7/site-packages/jinja2)
sphinx: Not installed
    pygments: 2.7.4 (/Users/ataylor/.pyenv/versions/3.7.7/envs/gameshowell_test/lib/python3.7/site-packages/pygments)
pytest: 6.1.2 (/Users/ataylor/.pyenv/versions/3.7.7/envs/gameshowell_test/lib/python3.7/site-packages/pytest)
virtualenv: 20.2.1 (/Users/ataylor/.pyenv/versions/3.7.7/envs/gameshowell_test/lib/python3.7/site-packages/virtualenv)

</details>

user: ameliajtaylor
issue_7317: state: open
title: Any plan to integrate Extreme Value Theory
body: Is there any plan for an integration of extreme value theory into statsmodels?

There is [scikit-extremes](https://github.com/kikocorreoso/scikit-extremes) but the project seems abandoned.

I just finished my master's with my thesis in Extreme Value Theory and think I could contribute a thing or two if that is in the scope of statsmodels.
user: pth91
issue_7314: state: open
title: ENH: bernstein polynomials for cdf, ppf and pdf, univariate and multivariate
body: see also https://github.com/statsmodels/statsmodels/pull/7296#issuecomment-770160443

evaluated on a grid, similar to histogram, provides histogram smoothing

scipy.Bpoly works also if we only have one interval [0, 1] and don't use the piecewise feature.
univariate polynomial only

Note, bernstein polynomial basis functions are the same as `binom.pdf` in scipy.stats

e.g. approximation to beta distribution

```
K = 501
k = np.arange(K).astype(float)
n = len(k) - 1.
x = 0.75
a, b = 2, 5
print(x, stats.beta.cdf(x, a, b), (stats.beta.cdf(np.linspace(0,1,K), a, b) * stats.binom.pmf(k, n, x)).sum())
print(x, stats.beta.pdf(x, a, b), (stats.beta.pdf(np.linspace(0,1,K), a, b) * stats.binom.pmf(k, n, x)).sum())
print(x, stats.beta.pdf(x, a, b), (stats.beta.pdf(np.linspace(0,1,K), a, b) * special.binom(n, k) * x**k * (1 - x)**(n - k)).sum())
bpb = BPoly(stats.beta.cdf(np.linspace(0,1,K), a, b)[:, None], [0., 1])
print(x, bpb(x))

0.75 0.995361328125 0.995117628660943
0.75 0.08789062500000004 0.09035257646249084
0.75 0.08789062500000004 0.0903525764624922
0.75 0.9951176286610438
```

Convergence rate of approximation by Bernstein polynomials is slow, but good shape properties, uniform approximation and restrintion to non-negative or positive values.
(in contrast to orthogonal polynomials that might have negative pdf segments)

Advantage compared to just looking at binned values or histogram is that the approximation is a bit better and the approximating functions are smooth. 

I guess when we don't need smoothness, then linear interpolation might do as well or better, e.g. in simulation case using approximation to ppf.

Cheaper than kernel density/distribution methods and extends more easily to multivariate case.
We don't have boundary kernels yet, e.g. for distributions on [0, 1] as we need in copulas
user: josef-pkt
issue_7312: state: open
title: ENH "heckprobit" model and bivariate normal including cdf
body: Do you support the "heckprobit"(probit with sample selection) model?
Stata supports this model. Does statsmodels supporst this model?

Looking forward to your reply, thank you very much!
user: AI-Friend
issue_7310: state: open
title: Add Tweedie distribution: likelihood, pdf and sampling methods
body: #### Is your feature request related to a problem? Please describe
I'd like to have the Tweedie distribution in order to:
- calculate the likelihood
- calculate the pdf
- generate random numbers

Especially the likelihood is interesting for an MLE estimate of the power parameter of a Tweedie GLM, see #2858.

#### Describe the solution you'd like
The next release of scipy, v1.7.0, will have the special function `wright_bessel`, see merged PR https://github.com/scipy/scipy/pull/11313. This function greatly simplifies the computation of the Tweedie pdf.

#### Additional context
In https://github.com/scipy/scipy/issues/11291, it was decided not to have the distribution in scipy itself, only the needed special function.
See also the scipy mailing list https://mail.python.org/pipermail/scipy-dev/2020-March/024074.html.
user: lorentzenchr
issue_7308: state: open
title: incorrect documentation
body: #### Describe the bug

Documentation says trend choices are 'c' or 'nc .
Should be 'c' or 'n'

https://www.statsmodels.org/stable/generated/statsmodels.tsa.arima_model.ARIMA.fit.html


```




user: numeric-lee
issue_7305: state: open
title: tsa.statespace.sarimax.SARIMAXResults.get_prediction().conf_int() returns missing values
body: I have a fit SARIMAX model with 3 binary exogenous variables - in pseudocode:
```
df = pd.DataFrame(
  data={
    'endog': [...],
    'exog_1': [...],
    'exog_n: [...]
  },
  index=dates
)

# ...

model = SARIMAX(
    endog=df[endog],
    exog=df[endog_list],
    order=params[:3],
    seasonal_order=params[3:],
    ...
  ).fit(method='powell', disp=0)

# ...

fcast = model.get_prediction(startdate, forecastend, exog=exog_df)
fcast_means = fcast.predicted_mean
fcast_intervals = fcast.conf_int(alpha=0.05)
lower_bounds = fcast_intervals['lower ' + endog]
upper_bounds = fcast_intervals['upper ' + endog]
```

In some edge cases, I get a point forecast (complete/non-missing `fcast_means`) but intermittent NaN values for `forecast_intervals`.  It appears the cause is that there are negative values in the variance of the predicted mean values `fcast.var_pred_mean` which result in np.nan when we take the square root in `fcast.se_mean`.

Example output:

date | mean | mean_se | mean_ci_lower | mean_ci_upper
-- | -- | -- | -- | --
2020-01-01 | 1513.416667 | 288.675135 | 947.623800 | 2079.209534
2020-02-01 | 1291.583333 | 288.675135 | 725.790466 | 1857.376200
2020-03-01 | 1906.000000 | 0.000002 | 1905.999995 | 1906.000005
2020-04-01 | 1037.703432 | NaN | NaN | NaN
2020-05-01 | 2474.406863 | 0.000002 | 2474.406859 | 2474.406868
2020-06-01 | 1816.406863 | 0.000002 | 1816.406859 | 1816.406868






user: ahgraber
issue_7302: state: open
title: ENH: tools, integrating, probability of hypercubes in multivariate distributions
body: we need tools function with the formulas, because I don't find them when I need them

1-dim : prob is just interval cdf(ub) - cdf(lb)
2-dim: prob of rectangle: cdf(b) - cdf(a) - cdf(d) + cdf(c)  What's the best way to get probs of all cells in a grid? (vectorized) https://github.com/statsmodels/statsmodels/issues/7106#issuecomment-712522978
k-dim: reference joe 2014 copula book p. 27 equ (2.2) https://github.com/statsmodels/statsmodels/issues/7454#issuecomment-1408744501

application 
binning in higher dimensions, e.g. for gof
prob of multivariate discrete with latent variables
#7301 copula with discrete margins

user: josef-pkt
issue_7301: state: open
title: ENH: copula with discrete margins, correlated Logit, Poisson
body: see also #1100

Logit margins is easier to get started than Poisson margins. I only superficially skimmed a few articles
This article looks well written to get started. 

Genest, Christian, Aristidis K. Nikoloulopoulos, Louis-Paul Rivest, and Mathieu Fortin. 2013. “Predicting Dependent Binary Outcomes through Logistic Regressions and Meta-Elliptical Copulas.” Brazilian Journal of Probability and Statistics 27 (3): 265–84. https://doi.org/10.1214/11-BJPS165.

main problem is to compute probabilities on hypercubes

e.g. for binary margin
Genest et al equ. 2.2 and next tables for dim is 2 or 3.

The Copula-Poisson literature might have the general formulas for computing probabilities of hypercubes
https://github.com/statsmodels/statsmodels/issues/7106#issuecomment-712522978



user: josef-pkt
issue_7300: state: open
title: ENH: two sample comparison ordered multinomial 
body: https://stats.stackexchange.com/questions/507430/statistical-test-for-two-categorical-ordinal-variables-with-four-levels

ordered multinomial with small counts, and many ties because of discreteness

Can we do better than Brunner Munzel with asymptotic p-values?

#3575

e.g permutation under no-diff Null.
or bootstrap

I have the ordered multinomial prob_larger statistic. but no special hypothesis test function for it.
user: josef-pkt
issue_7299: state: open
title: ENH transforms: symmetric inverse of u-shaped transforms
body: In sandbox I included hump- and u-shaped transforms like square mapping form R to R+. However, I don't have inverse functions for those transforms.

in Links, we have power transform restricted to R++, which also work for interval [0, 1]

One possibility to invert `square` is to construct a symmetric distribution on R, by allocating sqrt equally to both signs +/- .

example Kotz distribution is sqrt of chi2(df=2 * rho - 1) with equal prob for + and - sign
equ. (2) and (3) in 

Salehi, Mahdi, and Adelchi Azzalini. 2018. “On Application of the Univariate Kotz Distribution and Some of Its Extensions.” METRON 76 (2): 177–201. https://doi.org/10.1007/s40300-018-0137-3.

I guess we can just do compounding/mixing of an R+ distribution with "sign" distribution, prob(-1) = prob(1) = 0.5

possible use: compute cdf/ppf through a "known" or "special" distribution
e.g. mixing with sign distribution, needs to stitch together the parts on positive and negative half line. (I guess using sf and cdf)
scipy uses `special` functions for chi and chi2, cdf, ppf.

user: josef-pkt
issue_7288: state: open
title: ENH: mean, var, moments for finite MixtureDistribution
body: We have mixture distribution function and class in `distributions.mixture_rvs`, mainly for internal use or use in examples (rvs).

We can add moments generically for finite mixtures, non-central moments are additive and simple to compute.
For skew and kurtosis, we could use the helper functions to convert from noncentral moments

https://en.wikipedia.org/wiki/Mixture_distribution#Moments

example, I needed to look at mean and var for zero inflated count models #7161, but those are a special case where one distribution is degenerate at zero (var of component is zero, skew and kurtosis are be nan I guess).
user: josef-pkt
issue_7285: state: open
title: DOC: notebook: do-it-yourself GAM with new distribution
body: see #4256

a recipe for mix and matching existing pieces would be useful

- subclass GenericLikelihoodModel to define a Model for a not supported distribution
- use link classes if needed for parameter restrictions (example BetaRegression)
- add penalized Mixin class to inheritance
- add spline basis with penalization matrices from gam

recipe for last two is in unit tests for gam, penalized mixin, AFAIR

This requires a bit of wiring the pieces together, and there might still be problems.


for generic enhancement in statsmodels:
I think we need a results Mixin for penalized models using some of the code from `gam` as companion to penalized mixin for models.
More gamlss like would be to have a penalized spline Mixin class that combines the penalization Mixin with a GAM mixin.



user: josef-pkt
issue_7284: state: open
title: Inconsistent results from Python Package  sm.tsa.DynamicFactor and SAS ssm
body: Data:  data are simulated in Python and inputs are fixed in both packages.
Simulation Details: One common factor with auto regressive coefficient (0.5), 10 exogenous variables with coefficients either 0.2 or -0.2, and 20 time series with observational error terms following Normal Distribution (0,1). Total number of parameters will be 

State auto regressive coefficient               1
Factor loadings                                                  20
Exogenous variable coefficients                 20*10=200
Observational error terms                            20
Total Parameters                                             241

The total simulated data points are 1000 for each time series. Two sets of parameters are estimated respectively by first 500 and entire set of 1000 data points in both packages. 

Here is a summary.





  | Python | SAS | Observation   Number
-- | -- | -- | --
Profile Log Likelihood | -14894.2 | -14893.4 | 500
Profile Log Likelihood | -29752.8 | -29751.5 | 1000
Computation times | 110.0 | 869.0 | 500
Computation times | 223.0 | 1706.0 | 1000
Average of 241 Relative Difference in Estimates (python   -sas)/sas | -0.5% | 500
Average of 241 Relative Difference in Estimates (python   -sas)/sas | -0.3% | 1000
Average of 241 Relative Difference in Estimates (python std -sas   std)/sas std | 44.0% | 500
Average of 241 Relative Difference in Estimates (python std -sas   std)/sas std | 17.0% | 1000
Average of 241 Ratio of STD( std500/std1000) | 1.777 | 1.467 | 500~100

Comments based on above results
1)	The log likelihood difference is small enough to be ignored. It could be due to convergence criteria, optimization method, etc.
2)	The python package beats SAS package by a factor 8 times in terms of performance.  Amazing improvement!
3)	The overall estimated parameter difference is small. It contains positives or negatives. It is acceptable.
4)	The estimated standard deviations from Python are consistently larger than those from SAS. All are positives
5)	The Average of 241 Ratio of STD from SAS are closer to square-root of 2. This also indicates covariance(variance) calculation is different between Python and SAS.  Most likely the Python one has bugs.

user: shiling2007
issue_7280: state: open
title: ENH: random numbers for generalized poisson
body: We don't have a random number generator function `rvs` for GPP

just 3 links to articles as reminder  (I didn't look at anything)

https://doi.org/10.1080/01966324.1997.10737439
https://doi.org/10.1080/03610918.2014.968725
R package with author of second article
https://cran.r-project.org/web/packages/RNGforGPD/RNGforGPD.pdf

user: josef-pkt
issue_7278: state: open
title: ENH tools, distance measures for distributions, find closest approximation
body: Some distributions have no closed form (or `special`) versions and are not tractable. For some use cases we can use approximation that are easier to work with.

example: moment matching for p-values in hypothesis tests. recent examples that I used was shifted and scaled chisquare approximation to distribution of weighted sum of chisquare random variables, (e.g. LR tests under misspecification or incomplete specification.)

There is also a literature on finding approximations for quantile function, ppf, to make simulating random numbers easier.

In the literature for GLM link functions for binomial I saw it mentioned that t-distribution with around 6 or 7 degrees of freedom is close to logistic distribution.

Tail approximation: extreme value theory looks at distribution of the tail and which distribution provides good approximation to the tail behavior, e.g. pareto tails.
One question is whether a distribution has power tails, exponential tails or some other tails. This is important if we are not just interested in approximating the main part of a distribution.

this article includes a survey of methods to stich a distribution for the tail together with a distribution for the bulk
Scarrott, Carl, and Anna MacDonald. "A review of extreme value threshold estimation and uncertainty quantification." REVSTAT–Statistical Journal 10, no. 1 (2012): 33-60.

Tools:

I'd like something similar to gof test, but for distributions, e.g. match two cdfs, where one is the reference and the other is parameterized to match as closely as possible.
In this case we need to define a distance measure similar to KS test, "sup", integral versions like Cramer, von Mises or Anderson, Darling, or matching just a few quantiles or values of cdf. Then we can minimize the distance to find best matching parameters.

Other alternative, Kullback-Leibler distance. In this case we could also use MLE which minimizes KL distance in misspecified models.

related:




user: josef-pkt
issue_7277: state: open
title: ENH: truncated distribution Model Mixin
body: I think we can support (left) truncated distributions in general, currently we have truncated Poisson (needed fur hurdle model) in a draft stage in a PR.
user: josef-pkt
issue_7275: state: open
title: proportions_ztest() does not return a confidence interval
body: Trying to do a one-sample Z test for a proportion.

In R, this would be:

```
> prop.test(1200, 2500, p=0.44, correct = F)

    1-sample proportions test without continuity correction

data:  1200 out of 2500, null probability 0.44
X-squared = 16.234, df = 1, p-value = 5.599e-05
alternative hypothesis: true p is not equal to 0.44
95 percent confidence interval:
 0.4604617 0.4995996
sample estimates:
   p 
0.48
```

`statsmodels.stats.proportion.proportions_ztest()` also works, but it does not return a confidence interval for the proportion, which limits its usefulness.

Please add a confidence interval to the output of `proportions_ztest()`.
user: FlorinAndrei
issue_7272: state: open
title: ENH: quantile parameterized distributions
body: I found this by chance

https://en.wikipedia.org/wiki/Quantile-parameterized_distribution
assumes bounded support of distribution

It doesn't look popular enough as serious user facing distribution, but might be useful for some cases.

I looked briefly at an article for Johnson Quantile-Parameterized Distributions which seems to have relatively easy to use formulas and good approximation for other distributions, limited to using 3 quantiles.
referenced in last part of section https://en.wikipedia.org/wiki/Quantile-parameterized_distribution#Related_distributions

One usecase seems to be when we need a fast approximation that is easy to use for a distribution that doesn't have some of the properties, ppf, cdf, rvs, in closed form and is expensive to compute or simulate.
user: josef-pkt
issue_7271: state: open
title: ENH/DOC: Add more examples for multiple factors in the dynamic factor model state space model.
body: #### Is your feature request related to a problem? Please describe
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

#### Describe the solution you'd like
A clear and concise description of what you want to happen.

#### Describe alternatives you have considered
A clear and concise description of any alternative solutions or features you have considered.

#### Additional context
Add any other context about the feature request here.
user: shiling2007
issue_7269: state: open
title: wrong matrix dimensions:can't do fit_regularized with multivariate endogenous and exogenous variables
body: #### Describe the bug

I want to do regularized multivariate regression with sqrt_lasso. the endogenous variable dy is of shape 128x64 and exogenous variable x is of shape 128x64 also. When I try running this, I get the following error:

```
Traceback (most recent call last):

  File "/Users/jaemoon/Desktop/python/my_eegnet/pyregress_st_backup", line 357, in <module>
    est=sm.GLS(dy,x).fit_regularized(method='sqrt_lasso')

  File "/Users/jaemoon/opt/anaconda3/envs/phd/lib/python3.7/site-packages/statsmodels/regression/linear_model.py", line 630, in fit_regularized
    refit=refit, **kwargs)

  File "/Users/jaemoon/opt/anaconda3/envs/phd/lib/python3.7/site-packages/statsmodels/regression/linear_model.py", line 1053, in fit_regularized
    params = self._sqrt_lasso(alpha, refit, defaults["zero_tol"])

  File "/Users/jaemoon/opt/anaconda3/envs/phd/lib/python3.7/site-packages/statsmodels/regression/linear_model.py", line 1098, in _sqrt_lasso
    h1[1:, 0] = cvxopt.matrix(self.endog, (n, 1))

TypeError: wrong matrix dimensions
```
#### Code Sample, a copy-pastable example if possible
            `est=sm.GLS(dy,x).fit_regularized(method='sqrt_lasso')`


```python
# Your code here that produces the bug
# This example should be self-contained, and so not rely on external data.
# It should run in a fresh ipython session, and so include all relevant imports.
```
<details>

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `master`. If your problem has been fixed in an unreleased version, you might be able to use `master` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the master branch of this repository? It helps the limited resources if we know problems exist in the current master so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

What I expect to happen is a regularized fit of the exogenous and endogenous variables.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]
INSTALLED VERSIONS
------------------
Python: 3.7.7.final.0
OS: Darwin 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64
byteorder: little
LC_ALL: en_CA.UTF-8
LANG: en_CA.UTF-8

statsmodels
===========

Installed: 0.12.1 (/Users/jaemoon/opt/anaconda3/envs/phd/lib/python3.7/site-packages/statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.19.2 (/Users/jaemoon/opt/anaconda3/envs/phd/lib/python3.7/site-packages/numpy)
scipy: 1.6.0 (/Users/jaemoon/opt/anaconda3/envs/phd/lib/python3.7/site-packages/scipy)
pandas: 1.2.0 (/Users/jaemoon/opt/anaconda3/envs/phd/lib/python3.7/site-packages/pandas)
    dateutil: 2.8.1 (/Users/jaemoon/opt/anaconda3/envs/phd/lib/python3.7/site-packages/dateutil)
patsy: 0.5.1 (/Users/jaemoon/opt/anaconda3/envs/phd/lib/python3.7/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.3.2 (/Users/jaemoon/opt/anaconda3/envs/phd/lib/python3.7/site-packages/matplotlib)
    backend: module://ipykernel.pylab.backend_inline 
cvxopt: 1.2.0 (/Users/jaemoon/opt/anaconda3/envs/phd/lib/python3.7/site-packages/cvxopt)
joblib: 1.0.0 (/Users/jaemoon/opt/anaconda3/envs/phd/lib/python3.7/site-packages/joblib)

Developer Tools
================

IPython: 7.18.1 (/Users/jaemoon/opt/anaconda3/envs/phd/lib/python3.7/site-packages/IPython)
    jinja2: 2.11.2 (/Users/jaemoon/opt/anaconda3/envs/phd/lib/python3.7/site-packages/jinja2)
sphinx: 3.2.1 (/Users/jaemoon/opt/anaconda3/envs/phd/lib/python3.7/site-packages/sphinx)
    pygments: 2.7.4 (/Users/jaemoon/opt/anaconda3/envs/phd/lib/python3.7/site-packages/pygments)
pytest: Not installed
virtualenv: Not installed
</details>

user: moonj94
issue_7266: state: open
title: ENH/SUM: gof test, prob plots with simulated inference, envelope
body: With estimated parameters the asymptotic distribution of a gof test statistic is not pivotal, i.e. it depends on parameter values, outside of pure loc-scale families.
In that case, the commonly recommended approach is to use simulated p-values and confidence bands based on simulation using the null distribution with estimated parameters.

I don't find a specific issue for this, in the general continuous case

#6135 weibull with estimated parameters
#7154 gof tests for regression residuals
#6480  ecdf continuous versus discrete
#7101 and others for binned chisquare tests

last comment in 6135 contains some contributed code
I recently saw one of my old experimental modules with simulated pvalues for KS and AD, but I don't find it again


We need a module to get started with the basic cases, so we have a dedicated location to work on extended versions.

user: josef-pkt
issue_7265: state: open
title: ENH: multivariate gof tests and distance measures
body: briefly mentioned in #4179, no other general issues for it yet

we have issues for multivariate normality

#382 #3238 #6194

nothing for other or general cases

chisquare, KS, AD, ...

use case and special case copulas and multivariate dependence structure.
has larger literature for it e.g. for selecting a copula family for data modelling


For confidence intervals or test we will need bootstrap or simulated inference. 
We still need that also for univariate case with estimated parameters


user: josef-pkt
issue_7264: state: open
title: ENH: local polynomial density estimation
body: Local polynomial density estimation has improved peak and tail behavior, similarly to local polynomial regression compared to basic kernel regression.

One application is improving tail estimation, e.g. by transforming sample to finite interval [0, 1] and then back transforming to original space. example copula

Probit transformation for nonparametric kernel estimation of the copula density_euclid.bj.1489737627
Local likelihood density estimation_euclid.aos.1032298287
Locally parametric nonparametric density estimation_euclid.aos.1032298288

(I guess there should be also fast versions that evaluate on grids similar to kde and kernel regression, but I didn't search for it.)
user: josef-pkt
issue_7263: state: open
title: regression: checking MSE, pvalue, rsquared, etc results in error:shapes (128,64) and (128,64) not aligned: 64 (dim 1) != 128 (dim 0)
body: #### Describe the bug

I wish to look at regression statistics like MSE, rsquared, pvalue etc. but I keep getting the error:
`shapes (128,64) and (128,64) not aligned: 64 (dim 1) != 128 (dim 0)`

How do I get past this error? a bit of context, variables d and x are 128 samples by 64 features.
#### Code Sample, a copy-pastable example if possible

            ```
            x=sm.add_constant(x)
            est=sm.GLS(d,x).fit()
            csr=est.fittedvalues
            # est.mse_total()
            # est.rsquared()

```
```python
# Your code here that produces the bug
# This example should be self-contained, and so not rely on external data.
# It should run in a fresh ipython session, and so include all relevant imports.
```
<details>

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `master`. If your problem has been fixed in an unreleased version, you might be able to use `master` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the master branch of this repository? It helps the limited resources if we know problems exist in the current master so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

I would like to see the actual regression statistics such as mse, rsquared, pvalues etc/

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]

Installed: 0.12.1 (/Users/jaemoon/opt/anaconda3/envs/study2/lib/python3.7/site-packages/statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.19.2 (/Users/jaemoon/opt/anaconda3/envs/study2/lib/python3.7/site-packages/numpy)
scipy: 1.6.0 (/Users/jaemoon/opt/anaconda3/envs/study2/lib/python3.7/site-packages/scipy)
pandas: 1.2.0 (/Users/jaemoon/opt/anaconda3/envs/study2/lib/python3.7/site-packages/pandas)
    dateutil: 2.8.1 (/Users/jaemoon/opt/anaconda3/envs/study2/lib/python3.7/site-packages/dateutil)
patsy: 0.5.1 (/Users/jaemoon/opt/anaconda3/envs/study2/lib/python3.7/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.3.2 (/Users/jaemoon/opt/anaconda3/envs/study2/lib/python3.7/site-packages/matplotlib)
    backend: module://ipykernel.pylab.backend_inline 
cvxopt: 1.2.0 (/Users/jaemoon/opt/anaconda3/envs/study2/lib/python3.7/site-packages/cvxopt)
joblib: 1.0.0 (/Users/jaemoon/opt/anaconda3/envs/study2/lib/python3.7/site-packages/joblib)

Developer Tools
================

IPython: 7.18.1 (/Users/jaemoon/opt/anaconda3/envs/study2/lib/python3.7/site-packages/IPython)
    jinja2: 2.11.2 (/Users/jaemoon/opt/anaconda3/envs/study2/lib/python3.7/site-packages/jinja2)
sphinx: 3.2.1 (/Users/jaemoon/opt/anaconda3/envs/study2/lib/python3.7/site-packages/sphinx)
    pygments: 2.7.4 (/Users/jaemoon/opt/anaconda3/envs/study2/lib/python3.7/site-packages/pygments)
pytest: Not installed
virtualenv: Not installed
</details>

user: moonj94
issue_7258: state: open
title: ENH: extras for copulas, notes
body: see also #4046

random notes because results are all over the literature, and it's often difficult to find a specific item

- multivariate copulas, formulas for Frank, Gumbel and Cook-Johnson
  Savu, Trede 2008, especially appendix, includes derivatives of generator function and inverse
- Book "Extremes in Nature" has list of copula families with properties in appendix C , including tau and rho when available 
  - appendix A simulation has sequential conditional distribution, bivariate and multivariate  
- densities of extreme value copulas, thesis https://ethz.ch/content/dam/ethz/special-interest/math/risklab-dam/documents/walter-saxer-preis/ma-doyon.pdf for bivariate case we also need first and second derivatives of Pickand's dependence function
- densities for multivariate Archimedean: Joe book "Dependence Modelling with Copulas" p. 93 section 3.3 in terms of psi, i.e. phi-inverse. d-dim need d'th derivative of phi-inverse evaluated at sum of phi
- How do we use copulas? What are the statistics of interest? `predict(which=???)`  I don't find much on expected value of a copula model. https://en.wikipedia.org/wiki/Copula_(probability_theory)#Expectation_for_copula_models_and_Monte_Carlo_integration cites McNeil et al 2005 book 


Savu, Cornelia and Trede, Mark(2008) 'Goodness-of-fit tests for parametric families of Archimedean
copulas', Quantitative Finance, 8: 2, 109 — 116
http://dx.doi.org/10.1080/14697680701207639

user: josef-pkt
issue_7256: state: open
title: Training VAR models on batch data
body: Hello,

I am training VAR/state space models as a baseline comparison for a neural network sequence prediction model I've developed. The dataset I'm using is of size [n_ch x n_time x n_trial]. I'd like to train a single VAR model on all of the trials. The data is in batch form to facilitate parallel computations while training the aforementioned machine learning model.

I could reshape the array to concatenate all of the trials into one long system, but stitching together trials would introduce large high-frequency artefacts. Is there an interface to train the model with batched data? If it has not be implemented yet, how would this be done with the current package?
user: m-nolan
issue_7255: state: open
title: ENH: can I do .fit_regularized with multivariate endogenous and exogenous variables (multivariate regression)?
body: #### Is your feature request related to a problem? Please describe
I wish to do multivariate regression but I also want to make the regression model regularized. But it seems that when I pass in 2-d endogenous and exogenous variables, statsmodel does not like it. Is there a way to do this in a multivariate manner?

#### Describe the solution you'd like
A clear and concise description of what you want to happen.
I would like to do regularized multivariate regression.
#### Describe alternatives you have considered
A clear and concise description of any alternative solutions or features you have considered.
Splitting up the multivariate into univariate is one way I considered, but as I am dealing with EEG the channels are not really independent from each other and are potentially related. This is why i'd like to do multivariate regression that is regularized.
#### Additional context
Add any other context about the feature request here.
user: moonj94
issue_7252: state: open
title: How to create generalized weight matrix from multiple linear regression
body: #### Describe the bug

This isn't really a bug, but rather a request for some assistance. ( I didn't know which category to submit an issue). 

So I am dealing with some EEG data in tasks A and B. I wish to do multiple linear regression between them. However, these data are set up as shape (300 trials, 64 channels, 128 samples). I tried concatenating the trials along the time dimension to do MLR, however the resultant accuracy and fit were poor. 

Is there a better way in which to create a 'generalized weight matrix' from MLR that is an accurate fit?



#### Code Sample, a copy-pastable example if possible


```python
        # %% split up training data into separate classes
        # concatenate SP and CS across trials along the time dimension
        # perform statsmodel.api.WLS/OLS/or some other regression type
        # extract regression weights
        # apply to training, validation, and testing data of CS
        spbidx=np.where(label_train==6)
        spoidx=np.where(label_train==7)
        csbidx=np.where(label_train==8)
        csoidx=np.where(label_train==9)        
        rstidx=np.where(label_train==5)
        
        spb=X_train[spbidx[0],:,:]
        spo=X_train[spoidx[0],:,:]
        csb=X_train[csbidx[0],:,:]
        cso=X_train[csoidx[0],:,:]
        sp=np.concatenate((spb,spo),0)
        cs=np.concatenate((csb,cso),0)
        spv=sp.reshape([128*sp.shape[0],64])
        csv=cs.reshape([128*cs.shape[0],64])       
        
        d=spv
        x=csv
        x=sm.add_constant(x)
        est=sm.WLS(d,x).fit()
        csr=np.matmul(x,est.params)
        
        plt.subplot(2,1,1)
        plt.plot(d[0:1000,1])
        plt.subplot(2,1,2)
        plt.plot(csr[0:1000,1])
        plt.show()
```
<details>

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `master`. If your problem has been fixed in an unreleased version, you might be able to use `master` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the master branch of this repository? It helps the limited resources if we know problems exist in the current master so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

A clear and concise description of what you expected to happen.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]

</details>

user: moonj94
issue_7246: state: open
title: ENH: add tools.transforms with several Transformation for distributions
body: see #7241
see #7244 SUMM issues about transformation and function collections

**edit** maybe look also at Box-Cox Cole Green distribution (BCCG) #4256

provides Transformation classes for creating flexible 3 or 4 parameter distributions based on transformation of variables.

Main part missing for generic usage in models are derivatives with respect to the params.
related to derivatives #7245 

For some models we might get case specific derivative, e.g. articles for sinh-arcsinh-t regression with score and hessian.

This has comprehensive unit tests for checking internal consistency, but no outside verification


user: josef-pkt
issue_7245: state: open
title: ENH/Recipe tools chainrule for derivatives of nested parameterized functions
body: we don't use automatic differentiation, but we use chain rule with pieces quite heavily

We might need something reusable when we use parameterized links or transformations.

example  `f(g(x, b), a)`
where `x` is scalar and `a` and `b` are parameters

e.g. 
- likelihood function with parameterized links and x is the linear predictor
- chained transformation f and g which both have parameters, simple example g is standardize transformation with b=(loc, scale)
- transformation of distribution where both transformation and base distribution have parameter, 
  e.g. BirnbaumSaunders-t where f is transform with skew parameter and g is t-distribution with df parameter


Wolfram alpha is helpful
https://www.wolframalpha.com/input/?i=table+d%5En%2Fdb%5En+f%28g%28x%2C+b%29%2C+a%29+for+n+%3D+1+...+2

derivatives wrt. params `b`
table d^n/db^n f(g(x, b), a) for n = 1 ... 2

`g^(0, 1)(x, b) f^(1, 0)(g(x, b), a)`  
`g^(0, 1)(x, b)^2 f^(2, 0)(g(x, b), a) + g^(0, 2)(x, b) f^(1, 0)(g(x, b), a)`

notation
`g^(0, 1)(x, b)`  is 0th derivative of `g` wrt. first argument `x` and 1st derivative wrt. second argument `b`

We can get a dictionary of derivative (i,j) for i,j = 0, 1, 2 with i+j <= 2 to have all first and second partial derivatives


chained case without params is simpler

derivatives of `f(g(x))` w.r.t. `x`

`g'(x) f'(g(x))`  
`g'(x)^2 f''(g(x)) + g''(x) f'(g(x))`

(Just an idea. I don't have code yet that uses parameterized links)

user: josef-pkt
issue_7244: state: open
title: FAQ-D/SUMM transformation functions current code and location
body: We have collections of non-linear functions and links, partially spread out over different parts of statsmodels

### main 

- genmod Links, monotonic
- genmod varfuncs
- robust norms, used as objective functions, mostly symmetric u or v shaped, with redescending tails, not really "transform" classes
- base.transform contains Box-Cox including estimation code
- copulas: Archimedean generator transforms, Pickand dependence function for EV (same pattern but not really a transform)


### other (not in official use)

- statsmodels.sandbox.distributions.transform_functions : old code mostly for u-shaped transformation of variables for distributions, `transformed` module also contains simple examples like `log` transform

### linear/affine

- linalg transformation to impose constraints `base._constraints.TransformRestriction`  linear multivariate transformation
- linalg statsmodels.tools.transform_model only contains StandardizeTransform, with estimation mean, scale, linear transformation
  I guess not in use yet.

we also have other linalg functions that are projections and similar, all linalg versions are linear or affine transformations

### multivariate transforms

- mnlogit, links for GEE
- compositional data transform: Simplex^{k} -> R^{k-1}   where are those?
- ?

### Other in use

- penalization functions, similar to robust norms, are u- or v-shaped
- We also have hardcoded, specific use case transformations, e.g. for hypothesis test of correlation, proportion and similar variables on limited domains or Logit transform in discrete Logit, Exp transform in count models

--- 

### design issues

I think I start `tools.transforms` in parallel to genmod links for new distribution classes #7241, **now** in PR #7246
A collection of functions and derivatives, no data involved. i.e. different from Box-Cox in base.transform 
(`transforms` is plural noun as in `links`)

Another possibility:
Add an `Inverted` class, that just flips transform with inverted transform. With that we can use generic code that uses transforms in both directions, i.e. Link = Inverted(Transform)
e.g. `log = Inverted(Exp)` where `Exp` is a class name and `log` is an instance of a transform class

many related issues and many different internal use cases

---

### list of transformations 
that follow `links`, `transforms` pattern (with derivatives)

links

- Log/Exp
- logit, not a subclass of CDFLink, CDFLink is subclass of Logit
- CLogLog, subclass of logit, What's the equivalence as CDFLink? (extreme value distribution ?)
- power
- NegativeBinomial link, (I never looked closely at that)
- CDFLinks  
   - [x] probit, normal, gaussian, 
   - [x] cauchy
   - [ ] student-t
   - [ ] ..
 
transforms

- [x] SinhArcsinh
- [x] Sinh, Johnson-SU
- [x] BirnbaumSaunders
- [x] TMI, R to R+, name  TMI = "t minus inverse t" = `t - 1/t`.  rename?
- [x] RtoInterval
- [ ] RplustoInterval. odds/logit transformation  (logit without exp)
- [ ] Johnson-SB transformation ?  same as logit transformation, log-odds ?
- [ ] TMI with other powers? generalized BS- or TMI-transformation
- [ ] 

some are combination transformation e.g. johnson-SB is log plus odds/logit transformation
Log-BirnBaumSaunders is a Sinh, BirnbaumSaunders has a square-root compared to TMI


terminology

`Logit` uses log/exp, i.e. combines `Logistic/Odds` + `Log` ?  mapping R -> R+
continued in next comment
user: josef-pkt
issue_7241: state: open
title: ENH: models, distributions based on monotonic transformations
body: see 
#5409
#3224
#624


more flexible distributions and corresponding regression models can be created by monotonic, non-linear transformation of random variables.

My old code for creating distribution classes is in the sandbox. My version doesn't require monotonicity and works also for e.g. square transformation. (At the time I wrote that, I didn't have any real application for it.)

for example tensorflow has sinh-arcsinh as transformation model that can take base distribution as argument.
https://www.tensorflow.org/probability/api_docs/python/tfp/distributions/SinhArcsinh

examples

- sinh-arcsinh
- johnson-SU, i.e. sinh, e.g. based on normal distribution https://github.com/scipy/scipy/blob/master/scipy/stats/_continuous_distns.py#L4032
- Birnbaum-Sanders and variation (R to R++) `sqrt(x) - 1 / sqrt(x)` or `x - 1 / x`
- transformation to unit interval, e.g. Johnson-SB (which might be exp-logit normal)

I'm currently mainly interested in the first 3.

extra
composition of transformation, e.g.
- log birnbaum-sanders is sinh transformation
- birnbaum-sanders is sqrt transformation combined with base `x - 1 / x` transformation. There are other versions and generalizations for it.
- plain logit `1 / (1+x)` versus exp-logit `1 / (1 + exp(x))` where exp transforms R -> R+

implementation

- generic: based on what I have in the sandbox, we create a new distribution that combines transformation and base distribution and takes both as argument
- special cases: 
  - some statistics like moments only have nice or analytical expressions in specific transformation-distribution combinations
    (Distributions have a generic `expect` method based on numerical integration, but that will not be efficient if we have to do it a few thousand or million times)
  - computational efficiency, for regression models we can use simpler expression for loglike and similar in some cases
- tools, requirements
  - transformation and inverse transformation, domain and range of transformation
  - derivatives of the transformation, first and second should be enough
  - derivatives with respect to parameters of the transformation. I don't know yet what we need.

specific issue:

Clipping and enforcing domain for transformation or inverse transformation.
AFAICS, all transformation mapping real line to R+ or interval (0, 1) define open spaces, i.e. we get inf or zero division at boundaries. (similar to some cases with links in GLM or discrete)

implementation convention
Which is transform and which is inverse transform? e.g. Do we define transformation to normality or from normality?
Same question as link versus inverse link classes but more general use cases. Links are currently only used as non-linear mean functions.

**edit** The tradition might be reversed from what I want. 
E,g Log transformation is **to** normality or other distribution, i.e. `z = log(x)` is normal. and `x = exp(z)` is log-normal distributed
(I always find this reversed, I would prefer that transformation is defined from R to restricted space, e.g. R+ or from R+ to (0, 1)
e.g. log-normal is the distribution of an exp transformed normal random variable `exp-normal`.
or Birnbaum-Sanders is the "bs" transformation of a normal distribution, and not: if we "bs" transform a BS random variable, then it is normally distributed.
)
user: josef-pkt
issue_7237: state: open
title: ENH: generated distribution families, flexible distributions
body: Many new distributions can be constructed by various ways of generating new distribution families.

R package with many of them, and a survey for some methods of construction

Nadarajah, Saralees, and Ricardo Rocha. 2016. “Newdistns: An R Package for New Families of Distributions.” Journal of Statistical Software 69 (1): 1–32. https://doi.org/10.18637/jss.v069.i10.

Tahir, Muhammad H., and Gauss M. Cordeiro. 2016. “Compounding of Distributions: A Survey and New Generalized Classes.” Journal of Statistical Distributions and Applications 3 (1): 13. https://doi.org/10.1186/s40488-016-0052-1.

Motivating application are for many of the above are survival models in industry with flexible hazard rates.
There are a huge number of articles describing new distribution, most of the citations are within the same branch of literature on constructing distributions.
Skimming several of them, moment computation are based on infinite series expansion.

I guess most of this "designer" distributions are not really useful in a more general range of applications.

Some construction methods might be useful to model specific features or for specification testing.

---

alternative methods:

https://github.com/statsmodels/statsmodels/issues/624#issuecomment-713253667

one useful is based on transformation of variables, e.g. like log-linear.

sinh-arcsinh looks promising, but for some base distributions there are no closed form expressions for moments

Jones, M. C., and Arthur Pewsey. 2009. “Sinh-Arcsinh Distributions.” Biometrika 96 (4): 761–80. https://doi.org/10.1093/biomet/asp053.

Lemonte, Artur J. 2021. “A Parametric Regression Framework for the Skew Sinh-Arcsinh t Distribution.” Applied Mathematical Modelling 89 (January): 1418–32. https://doi.org/10.1016/j.apm.2020.08.036.

Pewsey, Arthur, and Toshihiro Abe. 2015. “The Sinh-Arcsinhed Logistic Family of Distributions: Properties and Inference.” Annals of the Institute of Statistical Mathematics 67 (3): 573–94. https://doi.org/10.1007/s10463-014-0465-x.












user: josef-pkt
issue_7235: state: open
title: ENH: hypothesis test for heavy tails, deviation from normality
body: We don't have much specifically for checking tail behavior of distributions.

Would be useful to check whether we might run into estimation problems because of observations with large deviations because of outliers or heavy tails/

I found it in minitab technical pater Appendix E as "SJ test"

test statistic is based on ratio standard deviation over mean absolute deviation from median   (like a mean MAD version).
Distribution is normal, and Gel et al use onesided test for heavier than normal tails.
We could also test for lighter than normal tails.

I guess the usage would be similar to the kurtosis test for normality. I don't think we or scipy hase a one-sided test for that.

Gel, Yulia R., Weiwen Miao, and Joseph L. Gastwirth. 2007. “Robust Directed Tests of Normality against Heavy-Tailed Alternatives.” Computational Statistics & Data Analysis 51 (5): 2734–46. https://doi.org/10.1016/j.csda.2006.08.022.

minitab technical paper
https://support.minitab.com/en-us/minitab/18/Assistant_One_Sample_Variance.pdf


Note: 
I don't remember, but I might have seen something similar with IQR, which different quantiles.
Maybe it was a different context with order statistics, quantiles and L-moments.



user: josef-pkt
issue_7234: state: open
title: ENH: 2-sample comparison of variances or scale, confidence intervals
body: part of oneway methods are tests for equality of scale using oneway anova methods, Levene, Brown-Forsythe and variants.

We don't have much explicitly for the 2 sample case. In this case we can get hypothesis tests directional alternatives, confidence intervals for ratio or difference and similar.
I guess we can get all the usual 2 sample statistics, hypothesis tests, ... for variance or scale (e.g. MAD) comparisons.

Bonnet's method and similar use an empirical kurtosis estimate to adjust. 

https://support.minitab.com/en-us/minitab/18/Bonetts_Method_Two_Variances.pdf
I ran into this looking for a minitab technical paper
https://support.minitab.com/en-us/minitab/18/technical-papers/

Main point would be to get methods that are robust to non-normality and some that still have a useful interpretation for which we can e.g. specify an equivalence margin.
This is also an area where bootstrap would help in small samples.


related 

- pairwise methods as followup to variance/scale anova:
https://support.minitab.com/en-us/minitab/18/Multiple_Comparisons_Method_Test_for_Equal_Variances.pdf
- comparing variance of paired samples taking correlation into account


I don't find related issues right now

- covariance and correlation tools refer more to multivariate, matrix versions than k-sample variance, e.g. PR #6696





user: josef-pkt
issue_7227: state: open
title: FAQ: Retrieve the dispersion parameter theta for ZeroInflatedNegativeBinomialP
body: #### Is your feature request related to a problem? Please describe

Is it possible to retrieve the estimated dispersion parameter theta in a manner similar to R?

```
import statsmodels.discrete.count_model as cm
from scipy.stats import nbinom
y = nbinom.rvs(n=1, p=0.5, size=100)
m1 = cm.ZeroInflatedNegativeBinomialP(endog=y, exog=np.ones(len(y))).fit()
```

In R, we do:

```
library(pscl)
m1 <- zeroinfl(y~ 1, dist='negbin')
m1$theta
```

user: afogarty85
issue_7224: state: open
title: TST: don't use statsmodels.api in most unit tests
body: several unit test and test results modules import `statsmodels.api`.

I was importing a dataset from a results module (genmod Lbw) and import takes a long time in the notebook.

I think we shouldn't import the `api` in the unit tests . We have some imports in `base` unit tests where it might be useful to check the api.


user: josef-pkt
issue_7223: state: open
title: ENH/REF  cov_params in GLM oim versus eim, add or clean up options
body: I'm not clear what the options for using oim versus eim in GLM are.
In case of canonical link functions there is no difference between oim and eim (except numerical noise from irrelevant terms)

I'm currently checking noncanonical links for binary Binomial, probit and chauchy. #1878

- fit with method irls always returns the irls cov_params, i.e. EIM based on Fisher scoring
- fit with gradient method returns by default OIM based on observed hessian.
- based on the code fit with gradient method has an option `if cov_type.lower() == 'eim':`
- after irls: I'm not sure (didn't check) what hessian IM version is used in the sandwich. 
   I guess we recompute hessian, or we reuse the normalized_cov_params which is EIM
   **edit** recomputes hessian if score_obs is available (*)

possible changes:

- IRLS should have an option to use OIM.
- It might be useful to have an option to use EIM in the sandwiches. Might be more stable.

My guess is that using OPG for cov_params is not useful, no huge computational advantage and possibly more sampling noise.
But it could be interesting for comparison. 

I was thinking of adding an extra option, either a boolean `oim=True`, or something more flexible that keeps the current default
`infomat="auto"`  with choices "oim", "eim" and possibly "opg"

Putting the option in `cov_type` would not allow us to use EIM in sandwiches.
But we could use oim/eim choice in cov_kwds when using sandwich robust cov.
We will need a flag or an indication whether `normalized_cov_params` is based on oim/hessian or eim/irls

Aside: for unit testing
Searching for some time: I have not found anything in R that allows OIM in glm. So, we can only compare EIM to R.
This might also be the reason that they don't have second derivatives of link functions.

currently I'm cross-checking GLM-probit versus discrete Probit and GLM-cauchy versus OrderedModel with cauchy

(*) GLM and similar currently recompute hessian, AFAICS
when model and results are used and hessian and score_obs are not directly given to sandwich code, then

in `stats.sandwich_covariance`

```
        elif hasattr(results.model, 'score_obs'):
            xu = results.model.score_obs(results.params)
            hessian_inv = np.linalg.inv(results.model.hessian(results.params)
```

If `score_obs` and `jac` are not available, then wresid and normalized_cov_params are used.



user: josef-pkt
issue_7219: state: open
title: [Question] How do exogenous regressors work?
body: #### Is your feature request related to a problem? Please describe

I cannot understand the description of the regression effects ([link](https://www.statsmodels.org/stable/generated/statsmodels.tsa.statespace.structural.UnobservedComponents.html)):

> Regression effects
  \
Exogenous regressors can be pass to the exog argument. The regression coefficients will be estimated by maximum likelihood unless mle_regression=False, in which case the regression coefficients will be included in the state vector where they are essentially estimated via recursive OLS.
  \
If the regression_coefficients are included in the state vector, the recursive estimates are available in the results class in the regression_coefficients attribute.


#### Describe the solution you'd like

Could you describe more details about the behavior of exogenous regressors?

I propose the following two steps:
1. Describe the definition of `state vector` (It would be useful to explain whether  the vector is `\beta_t` or `\mu_t`, and whether time-varying or time-fixed)
2. Describe formulas of the data generating process when using exogenous regressors (e.g. Let `$X_t$` be exogenous regressors, then`$y_t = \mu_t + \beta_tX_t + ...$`)

#### Describe alternatives you have considered
It would be helpful to add the link of the source code where exogenous regressors are used.

#### Additional context
No additional context.
user: fullflu
issue_7216: state: open
title: TST/DOC: count distributions missing, need better unit test coverage
body: the count distributions that are missing in scipy are in `statsmodels.distributions`

- they are not in the docs https://www.statsmodels.org/dev/distributions.html
- unit test coverage mainly compares with special case that reduces to standard distribution, e.g. no inflation, no dispersion -> Poisson
- uncovered lines and bugs #7161
- #6617 improvements to rvs might still be possible, no unit tests for rvs.

related
 #3887 issue for the count Models, follow-up to merge and enhancements

motivation question asking for zip distribution, not zip model
https://stats.stackexchange.com/questions/501383/zero-inflated-poisson-distribution-pzipois-in-python/501387#501387

The count distribution classes were written as part of count PR mainly for supporting the models.
They were never really reviewed as user facing standalone classes, and I'm not sure how much we should advertise them.





user: josef-pkt
issue_7211: state: open
title: SUMM/TST  missing unit tests for GLM Probit
body: continues from https://github.com/statsmodels/statsmodels/issues/7210#issuecomment-746949282

- #300 no unit tests for CDF link
- https://github.com/statsmodels/statsmodels/pull/1870/files#diff-38c484cefbcd7d9aa9c54c405b7b0be83fbf6f721f642df6f9573963e821d661R468 disabled test because I didn't know glm probit
   **edit** this was already fixed in #4620 I'm currently increasing required precision to rto 1e-13 in probit hessian comparison
- #1878 precision loss in GLM probit hessian ?

I was looking at comparison of GLM and Logit, Poisson and Probit for postestimation extras #7166

user: josef-pkt
issue_7210: state: open
title: BUG/ENH: Probit for continuous data 
body: I suspect that Probit only works for binary {0, 1} endog.

If I add some noise to get a continuous endog, then Probit does not produce the same result as the GLM version, but it does for Logit.

I need to go through the math to check, but I think the `q = 2*self.endog - 1` trick in loglike, score and similar might not work for continuous endog. (I haven't looked at this in many years)
My guess is that this makes it similar to MNLogit and OrderedModel, where we only compute the choice that has been selected in that observation.

Or there are some other problems related to a noncanonical link for Binomial.

This web page has loglike and score with and without the q trick
https://www.statlect.com/fundamentals-of-statistics/probit-model-maximum-likelihood
user: josef-pkt
labels: type-bug-wrong: serious bugs that silently return incorrect numbers
issue_7209: state: open
title: ENH: discrete binary models for other, for general distributions
body: I was looking at Probit to see what we need to get MLEInfluence to work.

Probit has the general structure of a distribution based binary Model. There are no shortcuts as with the canonical logit link.
In OrderedModel the distribution is an argument or parameter for the class, i.e. we can use the same class with different distributions.

It should be relatively easy to create a binary model class that take a generic distribution, and we would get the analogue to other links, e.g. cauchy or t.

I don't know whether there would be demand for it, but it shouldn't be much work. It might be useful when writing extension to models that don't fit as well into the GLM setup.
Maybe it even works already with a subclass that just defines cdf and pdf methods.
(I haven't tried)

Binary is a special case of OrderedModel, so the models are already available but with a different parameterization of the constant, and different results structure because ordinal is multivariate.
The equivalence might be useful for unit tests.
user: josef-pkt
issue_7205: state: open
title: Improved Sheather-Jones Bandwidth Selection
body: #### Is your feature request related to a problem? Please describe
Not related to a problem, but an excellent enhancement to the KDE capability offered by statsmodel.  Precisely, introducing Improved Sheather-Jones (ISJ) bandwidth selection method.  ISJ has advantages over the other bandwidths, especially if the data points are known not to be Gaussian.  Thank you.

#### Describe the solution you'd like
When fitting a KDE object, the `bw` field [accepts](https://www.statsmodels.org/stable/generated/statsmodels.nonparametric.kde.KDEUnivariate.fit.html#statsmodels.nonparametric.kde.KDEUnivariate.fit) either str, float, or callable.  The `str` choices are `scott`, `silverman`, and `normal_reference`.  I would like to request a fourth one `isj`:
```
dens = sm.nonparametric.KDEUnivariate(X)
dens.fit(kernel='uni', bw='isj') # this new choice of bandwidth as a new permitted str.
```

#### Describe alternatives you have considered
I tried PISA and KDEpy, and by far statsmodels is my favorite due to its ease to use, versatility, and flexibility.

#### Additional context
Sheather, S. J., and M. C. Jones. “A Reliable Data-Based Bandwidth Selection Method for Kernel Density Estimation.” _Journal of the Royal Statistical Society. Series B (Methodological)_, vol. 53, no. 3, 1991, pp. 683–690. JSTOR, www.jstor.org/stable/2345597. Accessed 14 Dec. 2020.
user: farismismar
issue_7203: state: open
title: ENH/BUG/Precision score_test collinearity problems with exog_extra
body: see https://github.com/statsmodels/statsmodels/pull/7166#issuecomment-743451373 and comments after it

precision problem when testing a redundant collinear exog_extra with score_test function for Poisson, GLM-Poisson and with GLM score_test method

score_test needs precision and corner case review, once we have more unit tests for regular cases.
user: josef-pkt
issue_7198: state: open
title: ENH: testing log-normal or Box-Cox transformed versus normal model
body: https://stackoverflow.com/questions/65117280/how-to-apply-statsmodels-stats-diagnostic-compare-j-test-for-linear-and-log-line

interesting question with not immediate specification test in statsmodels

Vuong test for non-nested models should apply.

But AFAIR, the loglike would have to take the distribution of y into account, not of log(y). 
Using loglike of normal for log(y) is missing thejacobian/derivative of the transformation in the likelihood function.
For log-normal this is just factor `1 / y`
https://en.wikipedia.org/wiki/Log-normal_distribution

related: #4217 and my notebook for parametric survival model for `t` and `log(t)` 

user: josef-pkt
issue_7196: state: open
title: ENH: Do not let ExponentialSmoothing accept arguments for components that are not part of the model
body: #### Describe the bug

I tested ExponentialSmoothing with some known parameters (for example ``smoothing_level``, ``smoothing_seasonal``, leaving ``smoothing_trend`` unknown) on some trivial seasonal data. It does not success at optimizing unknown parameters and obtaining a good prediction. It is a very easy forecasting problem.  I tested the same data in R ``forecast`` package and it works perfectly.

I tested statsmodels 0.12.1 and master branch version 0.13.0.dev0+127.g36b7f5ddf, they both gave bad results.

#### Code Sample, a copy-pastable example if possible


```python
from statsmodels.tsa.api import ExponentialSmoothing
test_data = [1, 2, 3, 4, 5, 6, 6, 5, 4, 3, 2, 1] * 3
predictor = ExponentialSmoothing(test_data, seasonal_periods=12)
pred_results = predictor.fit(smoothing_level=0.1, smoothing_seasonal=0.8)
pred_results.predict(12)
```
<details>

I used both R and an Excel file to find a solution for this forecasting problem. My excel file gave a solution:

```python
smoothing_level: 0.1, initial_level: 3.5
smoothing_trend: 0
smoothing_seasonal: 0.8, 
seasonal coefficients: 

0.29 | 0.57 | 0.86 | 1.14 | 1.43 | 1.71 | 1.71 | 1.43 | 1.14 | 0.86 | 0.57 | 0.29
```
And for the R code:
 
```R
library('forecast')
library('zoo')

data = ts(c(1, 2, 3, 4, 5, 6, 6, 5, 4, 3, 2, 1, 1, 2, 3, 4, 5, 6, 6, 5, 4, 3, 2, 1, 1, 2, 3, 4, 5, 6, 6, 5, 4, 3, 2, 1),
          frequency = 12) 
mod <- hw(
  data,
  h = 12,
  alpha = 0.1,
  gamma = 0.8 
)

summary(mod)

Call:
 hw(y = data, h = 12, alpha = 0.1, gamma = 0.8) 

  Smoothing parameters:
    alpha = 0.1 
    beta  = 0.0101 
    gamma = 0.8 

  Initial states:
    l = 3.5 
    b = 0 
    s = -2.5 -1.5 -0.5 0.5 1.5 2.5
           2.5 1.5 0.5 -0.5 -1.5 -2.5
```
Both give me the good prediction ``1, 2, 3, 4, 5, 6, 6, 5, 4, 3, 2, 1``
</details>


#### Expected Output

The model should predict ``1, 2, 3, 4, 5, 6, 6, 5, 4, 3, 2, 1``. The Python model gives

```python
pred_results.predict(12)

array([3.40661571, 3.16595413, 3.04935872, 3.04442285, 3.13998056,
       3.32598251, 3.59338426, 3.83404583, 3.95064125, 3.95557712,
       3.86001941, 3.67401747, 3.40661572, 3.16595415, 3.04935874,
       3.04442286, 3.13998058, 3.32598252, 3.59338427, 3.83404584,
       3.95064126, 3.95557713, 3.86001942, 3.67401748])

pred_results.params_formatted

smoothing_level: 0.1
initial_level: 3.406616
```
Python deleted all other parameters for trend and seasonal including ``smoothing_seasonal=0.8``.

In general, I set values for at least 2 among 3 parameters ``smoothing_level``, ``smoothing_seasonal``, ``smoothing_trend`` and the ``ExponentialSmoothing`` class omits at least one of known parameters that I set.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]

INSTALLED VERSIONS
------------------
Python: 3.7.6.final.0

statsmodels
===========

Installed: 0.13.0.dev0+127.g36b7f5ddf (C:\Users\NTL.ARGON-CONSULT\Anaconda3\lib\site-packages\statsmodels)

Required Dependencies
=====================

cython: 0.29.15 (C:\Users\NTL.ARGON-CONSULT\Anaconda3\lib\site-packages\Cython)
numpy: 1.18.1 (C:\Users\NTL.ARGON-CONSULT\Anaconda3\lib\site-packages\numpy)
scipy: 1.4.1 (C:\Users\NTL.ARGON-CONSULT\Anaconda3\lib\site-packages\scipy)
pandas: 1.0.1 (C:\Users\NTL.ARGON-CONSULT\Anaconda3\lib\site-packages\pandas)
    dateutil: 2.8.1 (C:\Users\NTL.ARGON-CONSULT\Anaconda3\lib\site-packages\dateutil)
patsy: 0.5.1 (C:\Users\NTL.ARGON-CONSULT\Anaconda3\lib\site-packages\patsy)

Optional Dependencies
=====================

matplotlib: 3.1.3 (C:\Users\NTL.ARGON-CONSULT\Anaconda3\lib\site-packages\matplotlib)
    backend: module://ipykernel.pylab.backend_inline 
cvxopt: Not installed
joblib: 0.14.1 (C:\Users\NTL.ARGON-CONSULT\Anaconda3\lib\site-packages\joblib)

Developer Tools
================

IPython: 7.12.0 (C:\Users\NTL.ARGON-CONSULT\Anaconda3\lib\site-packages\IPython)
    jinja2: 2.11.1 (C:\Users\NTL.ARGON-CONSULT\Anaconda3\lib\site-packages\jinja2)
sphinx: 2.4.0 (C:\Users\NTL.ARGON-CONSULT\Anaconda3\lib\site-packages\sphinx)
    pygments: 2.5.2 (C:\Users\NTL.ARGON-CONSULT\Anaconda3\lib\site-packages\pygments)
pytest: 5.3.5 (C:\Users\NTL.ARGON-CONSULT\Anaconda3\lib\site-packages\pytest)
virtualenv: 20.0.34 (C:\Users\NTL.ARGON-CONSULT\Anaconda3\lib\site-packages\virtualenv)
</details>

user: lnthanh
issue_7195: state: open
title: BUG: Non-Gaussian KDE evaluates the entire pdf to NaN when point is outside domain of support.
body: #### Describe the bug

When the user constructs sm.nonparametric.KDEUnivariate() and passes data to evaluate the PDF at a group of points using a numpy array, if one point in this array returns np.nan (e.g., due to not being in the support of the density), then instead of receiving a numpy array the elements of which are the PDF at their respective points with that one point having np.nan as its respective element, a scalar np.nan is returned for the whole array.

#### Code Sample, a copy-pastable example if possible

```
import numpy as np
import statsmodels.api as sm

X = np.linspace(start=-1, stop=1, num=1000)

dens = sm.nonparametric.KDEUnivariate(X)
dens.fit(kernel='uni', bw='silverman', fft=False) # any other bandwidth might also work

X_test = np.append(X, -2) # note that -2 is not in the support of X.

# This is the error:
print(dens.evaluate(X_test)) # returns nan

# This a workaround, but can be very very slow, depending on the size of X_test
for x in X_test:
    print(dens.evaluate(x))
```
<details>

Here, `X_test` has many elements that belong to the support `X`, while one single element `2` does not.  When attempting to evaluate the density for `X_test` from `print(dens.evaluate(X_test))`, a flatout `nan` scalar is shown.  Using a loop works, but it can be extremely slow if `X_test` since it has a large numpber of elements.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

The expected output should be a numpy vector with the same size as the input numpy vector, but only where the elements do not belong to the support of the PDF that an `np.nan` should be put (as an element in the vector), and not a single scalar `np.nan`.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``
>>> sm.show_versions()

INSTALLED VERSIONS
------------------
Python: 3.8.6.final.0
OS: Darwin 19.6.0 Darwin Kernel Version 19.6.0: Thu Oct 29 22:56:45 PDT 2020; root:xnu-6153.141.2.2~1/RELEASE_X86_64 x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

statsmodels
===========

Installed: 0.12.0 (/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.18.5 (/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy)
scipy: 1.5.3 (/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/scipy)
pandas: 1.1.3 (/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pandas)
    dateutil: 2.8.1 (/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/dateutil)
patsy: 0.5.1 (/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.3.2 (/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/matplotlib)
    backend: MacOSX 
cvxopt: Not installed
joblib: 0.17.0 (/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/joblib)

Developer Tools
================

IPython: 7.18.1 (/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/IPython)
    jinja2: 2.11.2 (/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/jinja2)
sphinx: 3.2.1 (/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/sphinx)
    pygments: 2.7.1 (/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pygments)
pytest: Not installed
virtualenv: Not installed

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]

</details>

user: farismismar
issue_7193: state: open
title: ENH: margins for zero-inflated and hurdle, 2-part models
body: just parking two references that I didn't look at but should be useful

Liu, Xueyan, Wencong Chen, Tian Chen, Hui Zhang, and Bo Zhang. 2020. “Marginal Effects and Incremental Effects in Two-Part Models for Endogenous Healthcare Utilization in Health Services Research.” Health Services and Outcomes Research Methodology 20 (2): 111–39. https://doi.org/10.1007/s10742-020-00211-x.

Liu, Xueyan, Bo Zhang, Li Tang, Zhiwei Zhang, Ning Zhang, Jeroan J. Allison, Deo Kumar Srivastava, and Hui Zhang. 2018. “Are Marginalized Two-Part Models Superior to Non-Marginalized Two-Part Models for Count Data with Excess Zeroes? Estimation of Marginal Effects, Model Misspecification, and Model Selection.” Health Services and Outcomes Research Methodology 18 (3): 175–214. https://doi.org/10.1007/s10742-018-0183-6.

looks a bit similar to log-binomial versus estimating risk ratio as a "margin", ATE

(I read part of "marginalized inflated models" articles until I realized that they change to model to have E(y | x) = exp(x b) based on zero-inflated model. marginalized refers to a single marginal distribution that combines zero-inflation and the main model.
Selecting a misspecified model just to get a simple result for a margin or ate doesn't sound very appropriate to me.)


user: josef-pkt
issue_7192: state: open
title: ENH: results tool  information matrix ratio
body: 
A inv @ B or B @ Ainv shows up in many methods for misspecified models. We need it as a reusable method for results classes
A is second derivative, Hessian, 
B is OPG, or cov(score_obs)

uncorrelated case can just use cov(score_obs)
more general, we should get the cov similar to the inner part of the sandwich cov_type (I guess we shouldn't use the wald based small sample corrections)

some usage

- information criteria: TIC, GIC, GAIC, GBIC
- LR test under misspecification
- IOS test and similar tests for model misspecification, 

possible variation: use EIM for A, instead of OIM, observed Hessian





user: josef-pkt
issue_7188: state: open
title: ENH: NegativeBinomialPHet, GeneralizedPoissonPHet heteroscedasticity in count models with dispersion parameter
body: countmodels where dispersion parameter depends on explanatory variables

#7130 issue for related score_test

This would be a good case to bring several extensions to models together. Most of the distribution specific computations are already available.

estimation of the full model by MLE (with inherited cov_type options)

- 2 exog
- need to separate out score_factor and hessian factor
- use dispersion parameter to extend loglike and derivatives from constant to exog dependent
- add transformation or inverse link function, I guess exp is default for NBP, GPP could have under dispersion, e.g. negative dispersion parameter, linear link or exp link 
- extend predict, predict variance/dispersion factor.
- might need adjustments in computation of current class to have proper broadcasting.

extra
- score_test with 2 exog



user: josef-pkt
issue_7187: state: open
title: ENH/REF/SUMM/Design: new or refactored base classes for models
body: Collecting some problems with and ideas for base Model and Results classes. 
This is currently just some ideas for a roadmap, maybe we need a SMEP before making larger changes to base classes.

The basic structure of our base classes has been unchanged for a very long time. It works well overall but has limitations and warts.

semi random list of what I remember and what's bugging me right now (I might not remember all details correctly)

- optimization shouldn't require `likelihood` to support models that are not based on MLE without misusing names of methods
- signature of results class could also be made more generic, `normalized_cov_params` comes mostly from linear model.
  MLE  results classes in discrete and GenericLikelihoodModel have a signature different from LikelihoodModelResults
- more generic base class, includes optimization and wald inference, e.g. separate generic M-estimator or GMM generic features from MLE
- add more likelihood features that are only appropriate for non-tsa models, We should be able to add methods without having to worry about inappropriate methods inherited by tsa
- related to previous points: DiscreteModel and DiscreteResults have very few "discrete" specific features. They are more like an intermediate MLE class similar to what we might need.
- model and results class for rapid development. Currently there is a gap between LikelihoodModel and GenericLikelihoodModel, e.g. I didn't want to move OrderedModel out of miscmodels because it inherits too much from GenericLikelihoodModel 
Maybe add/move features GenericLikelihoodModel to (non-tsa) LikelihoodModel, similarly for corresponding Results classes

- ... TBC




user: josef-pkt
issue_7186: state: open
title: FAQ/ENH: robust sandwich cov, dispersion in GLM, Binomial Poisson (statistics)
body: some references because I thought the GLM literature in statistics only considers over dispersion factor.

The simplest case of overdispersion in GLM, Binomial and Poisson is just a multiplicative factor which is included in most text book, package manuals.
#1750 overdispersion with just the scale factor

However, the Negbin2 extension to Poisson, and Beta-Binomial extension to Binomial when number of trials vary both have different variance functions that don't just differ in the multiplicative constant.
NegBin1 and Beta-Binomial with constant number of trials have only a multiplicative constant.

As far as I have seen the literature, the sandwich form for cov_params with excess dispersion is not included in most text books for statistics or applied statistics.
(In contrast to econometrics where sandwiches are standard fare in undergraduate textbooks)

Sandwich cov are the standard assumption in GEE, motivated by cluster correlation, and less by misspecified variance function.

two references to get started, (I didn't look much at citations and references)

Moore, D. F., and A. Tsiatis. 1991. “Robust Estimation of the Variance in Moment Methods for Extra-Binomial and Extra-Poisson Variation.” Biometrics 47 (2): 383–401. https://doi.org/10.2307/2532133.

Breslow, N. E. (1990). Tests of hypotheses in overdispersed Poisson regression and other quasilikelihood models. Journal of the American Statistical Association 85, 565-571.

Breslow also briefly discusses the sandwich score test for zero restriction on subset of params.

There are related issues on excess dispersion, score_tests, alternative variance functions

These articles might be useful for documentation and examples. 

user: josef-pkt
issue_7184: state: open
title: ENH: proportions, stratified pairs of independent proportions 2x2xk
body: specific issue for
https://github.com/statsmodels/statsmodels/issues/6380#issuecomment-734445398

I was reading through parts of Tang 2020, and it looks good for implementing this.
test on a  diff measure, RD, OR, RR for 2 independent binomial using k-sample of pairs

main focus in Tang is on score test for non-zero null for RD, OR or RR
difference in proportion for each stratum is estimated under the constraint as in the two-sample case, which we should be able to reuse.

this is a similar setup as we have in meta-analysis https://www.statsmodels.org/dev/generated/statsmodels.stats.meta_analysis.effectsize_2proportions.html
but might differ in that aggregation is more proportion specific

Tang, Yongqiang. 2020. “Score Confidence Intervals and Sample Sizes for Stratified Comparisons of Binomial Proportions.” Statistics in Medicine 39 (24): 3427–57. https://doi.org/10.1002/sim.8674.

related

Lipsitz et al test equal RD across strata with asymptotics number of strata goes to inf, strata size small

Lipsitz, Stuart R., Keith B. G. Dear, Nan M. Laird, and Geert Molenberghs. 1998. “Tests for Homogeneity of the Risk Difference When Data Are Sparse.” Biometrics 54 (1): 148–60. https://doi.org/10.2307/2534003.

Gart discusses homogeneity issues across k strata sample, e.g. what is constant, probs, RD, OR, RR. This is more background for Tang and similar. General issues with aggregating nonlinear statistics affects also other areas (e.g. log-binomial RR )
e.g. aggregating RD does not require that probs are the same across strata. (similar to using difference to remove individual, group specific intercept)

Gart, John J. 1992. “Pooling 2 Times 2 Tables: Asymptotic Moments of Estimators.” Journal of the Royal Statistical Society: Series B (Methodological) 54 (2): 531–39. https://doi.org/10.1111/j.2517-6161.1992.tb01894.x.



Related:
We could also have k samples of two paired proportions, i.e. k-sample stratified collection of McNemar pairs.
I haven't looked at that yet.


user: josef-pkt
issue_7182: state: open
title: ENH: parametric confidence band for (e)cdf, 2-parameter endpoint transformation
body: parking an article with a possible useful method

- provides parametric confidence band for cdf, ecdf using estimated loc-scale uncertainty
- confidence band is constructed by a 2-dimensional analogue to endpoint transformation 

The method for constructing confidence intervals might be applicable to models like NegBin and generalized Poisson where we have 2 parameters for an endpoint transformation of a confidence interval for prediction.
Maybe we can get the analogue to the one-parameter GLM, Poisson, ... case.

I don't understand where the "simultaneous" comes in, My intuition for preserving overall coverage in confidence bands is still confused.

Cheng, R. C. H., and T. C. Iles. 1983. “Confidence Bands for Cumulative Distribution Functions of Continuous Random Variables.” Technometrics 25 (1): 77–86. https://doi.org/10.2307/1267729.

there are more recent articles that cite it and use the same method, e.g.

Escobar, Luis A., Yili Hong, and William Q. Meeker. 2009. “Simultaneous Confidence Bands and Regions for Log-Location-Scale Distributions with Censored Data.” Journal of Statistical Planning and Inference 139 (9): 3231–45. https://doi.org/10.1016/j.jspi.2009.03.005.

Jeng, Shuen-Lin, and William Q. Meeker. 2001. “Parametric Simultaneous Confidence Bands for Cumulative Distributions from Censored Data.” Technometrics 43 (4): 450–61.

Mies, Fabian, and Stefan Bedbur. 2017. “On the Coverage Probabilities of Parametric Confidence Bands for Continuous Distribution and Quantile Functions Constructed via Confidence Regions for a Location-Scale Parameter.” Annals of the Institute of Statistical Mathematics 69 (4): 925–44. https://doi.org/10.1007/s10463-016-0570-0.



more general:
How can we minimize a function over an elliptical constrained space, both cases when function is monotonic and we need to find boundary point and when the function has an interior minimum?

An example for the second, interior case is Berger-Boos exact p-values. Currently we only have it for 1-dimensional, scalar nuisance parameter.

user: josef-pkt
issue_7181: state: open
title: ENH: influence (outlier effect) on hypothesis tests
body: Current influence and outlier diagnostic are focused on parameter estimation. This can be extended to checking the influence of observations and outlier to hypothesis tests.
(I don't find an issue, but I had skimmed some articles in the past)

example for dispersion test in Poisson:
Xiang, Liming, and Andy H. Lee. 2005. “Sensitivity of Test for Overdispersion in Poisson Regression.” Biometrical Journal 47 (2): 167–76. https://doi.org/10.1002/bimj.200310096.

This might be easier for standalone hypothesis tests in stats than for hypothesis tests based on models.
However, if hypothesis tests are based on sufficient statistics, like aggregate counts in tests for proportions and rates, then those are not sufficient to detect influence of individual observations on those sufficient statistics and through them on the hypothesis test results.

for t-tests, I could add them to the weightstats classes as a prototype.

related: can we get the influence of obs on standard error of the mean or parameter estimates,  `bse`, i.e. denominator in t-values?

motivation: Don't publish a "significant effect" that depends on one or a few outlying influential observations.



user: josef-pkt
issue_7180: state: open
title: ENH: IPW weighted counts or relative frequencies as ATE
body: another vague idea, possibly useful for diagnostic or interpretation in count models #1537

In one of the plots I compared observed and predicted counts, relative frequencies and probabilities for subsets defined by binary explanatory variable, e.g. male-female

The aggregated counts depend on the actual exog. If exog systematically differe between the two binary levels, then the comparison of observed versus predicted is not informative about the partial effect of the binary exog.

So we would need something like a potential outcome counts/frequencies that averages over a common population.

If we consider the frequencies or counts as outcome variable in treatment effect estimation, e.g. outcome is binary `y == 0`, then we could use for example IPW to compute a population weighted average.
We use the other explanatory variables as controls, so we get a partial effect of the binary exog.

The idea is to get an observed equivalent to the predicted average probability, similar to what we have for unweighted population average.

I'm trying to get a post-estimation diagnostic and not necessarily estimating a causal effect. Treatment choice estimation has reversed causality in the example, gender does not depend on weight and size. Also we might violate some exclusion or independence restrictions for causality if we use the included exog as "instruments".
(I never looked at the treatment effects literature for the case when the outcome variable is also binary)

Formally, we can just include an outcome variable in the moment conditions, where in this case the outcome variable is an indicator function that computes count frequencies.
(similar to moment conditions for pmf/cdf/bin counts in score tests or test for zero-modification)


user: josef-pkt
issue_7178: state: open
title: ENH specification tests for heterogeneity, constant parameters, especially score/LM tests or CMT
body: still a vague issue, I ran into literature on testing for heterogeneity including information matrix tests.
This is mostly in context of GLM and discrete MLE models (similar to excess dispersion and similar)

possible misspecifications

- random effects and random parameters in fixed effects models
- random parameters depending on explanatory variables (*)
- ...

related
in meta-analysis we have tests for random versus constant mean/effect size

AFAIR, I never looked at score tests with random effect under the alternative
However, several overdispersion tests for count models are motivated by a mixing distribution for random parameters


(*) I guess this is mostly the same as a missing interaction or nonlinearity, e.g. we could interact a spline basis with an explanatory variable and run a variable addition score test



user: josef-pkt
issue_7177: state: open
title: ENH: robust covariance matrix estimation
body: I've been working on a Python implementation of the Orthogonalized Gnanadesikan-Kettenring (OGK) robust covariance estimator of https://www.tandfonline.com/doi/abs/10.1198/004017002188618509. I don't think statsmodels currently offers any robust alternatives for covariance estimators and thought that this might be a nice addition; would there be interest in a PR for this?

I like the OGK estimator because it is a simple idea and unlike many (most?) robust covariance estimators that can deal with "gross error" outliers, is straightforward to compute. 
user: esmucler
issue_7176: state: open
title: ENH: score test for overdispersion in binomial count model
body: very specific issue to add the binomial version of the over dispersion tests
see #7166 for collection specification tests for poisson

Yang et al 2009
score test for some general variance alternatives is `S4` p.3678

the 2010 article includes zero-inflation
It has score equation `T` on bottom of page 2750, but variance needs the partitioned inverse of the full covariance, EIM matrix

Yang, Zhao, James W. Hardin, and Cheryl L. Addy. 2009. “A Note on Dean’s Overdispersion Test.” Journal of Statistical Planning and Inference 139 (10): 3675–78. https://doi.org/10.1016/j.jspi.2009.04.019.

———. 2010. “Some Remarks on Testing Overdispersion in Zero-Inflated Poisson and Binomial Regression Models.” Communications in Statistics - Theory and Methods 39 (15): 2743–52. https://doi.org/10.1080/03610920903177397.

Note, several of these articles don't mention distribution of test statistic
for single hypothesis we can use normal and one-sided tests
using standard chisquare assumes that we are not on the boundary of the parameter space, i.e. we allow for under-dispersion and don't just test against an alternative distribution that has larger dispersion than the null model.

Dean 1992 from which I took excess dispersion tests for Poisson also has 2 tests for Binomial, but Yang et all looks more general and easy to implement.

user: josef-pkt
issue_7172: state: open
title: ENH: score_test for extra parameter, score_factor
body: 
Current score_test for variable addition only looks at adding explanatory variables to the  mean function.
Extra parameters like dispersion in negative binomial or generalized Poisson are assumed to be constant.

However, if we have the score factor for constant extra parameter, then we should be able to add a score test for varying extra parameter depending on some additional explanatory variables.
We can just use chain rule to get derivatives, if there is an inverse link function for the extra. For score test linear is fine because it is only locally, e.g. zero-inflation tests that allow for deflation.

The special case for this is testing for varying dispersion, heteroscedasticity. #7130

example
Xie, Feng-Chang, Jin-Guan Lin, and Bo-Cheng Wei. 2010. “Testing for Varying Zero-Inflation and Dispersion in Generalized Poisson Regression Models.” Journal of Applied Statistics 37 (9): 1509–22. https://doi.org/10.1080/02664760903055442.

user: josef-pkt
issue_7171: state: open
title: ENH: Return MixedLM parameter estimates as series when appropriate 
body: Initial commit
user: kshedden
issue_7169: state: open
title: ENH: ordinal OrderedModel as mixing distribution for finite class mixtures
body: related MixedMixin #5502

If we have a finite class mixture, then the mixing probabilities are multinomial.

However, we can use an ordinal model to get monotonicity in the location of the unobserved classes, simplest case when mixing probabilities are constant and don't depend on other explanatory variables.

It's currently just an idea while thinking about count mixture models. I have never seen a literature for using ordinal model for this.

(I'm not sure this will actually work, but I don't want to try now. I'd like monotonicity in the random intercept of the main equation and might not get it with ordered mixing model.
The main idea was to get parameterization that forces monotonicity in the latent classes.)

user: josef-pkt
issue_7168: state: open
title: ENH/Design target results statistic, ATE, delta method, extra moment condition
body: Just parking a vague idea

In several cases we want to estimate additional statistics to help in the interpretation, these are like parameters that don't show up in the null model
example

- ATE: estimate a regression or doubly robust model and use GMM to get inference for ATE
- relative risk: common is to use log-binomial model, but instead we can just compute average RR for the sample from any standard model
- test for zero-inflation with extra moment condition that estimates excess zero probability
- maybe: smearing estimate of expected value when endog has been transformed, e.g. we estimate log-linear model using `OLS(log(endog), ...)`  (sounds similar to this case but I haven't looked at it in a long time)
- predict maybe for some parts (averaged `"which")
- ...

in general
we can use
- delta method given estimates of params and cov_params
- estimate jointly as an extra moment condition as in GMM

those might be algebraically equivalent in some cases. 

extra moment condition only work for fit sample, i.e. we take into account that parameters are estimated from the same sample as we use for the estimating the additional target statistics

qu: what do we do with out-of sample predictions of the target statistic or when we want to use different population weights in the estimation and the computation of the target statistic?
(Can we use different freq_weights in the moment condition for the target statistic without changing the weights of the estimating equations for the params?)

We already have some of this in GMM and in conditional moment tests, but the latter are designed for specification testing and don't provide estimates of the target parameters.

variable addition score_tests are doing something similar, i.e. estimate score of extra params using correlation and derivatives (score_obs and hessian for augmented model)
I guess that would be similar as a one-step estimator for the augmented model.
Maybe not, variable augmentation will also affect the included parameters, and under null model the extras have no effect.
There might be different cases depending on whether some off-diagonal submatrices of the derivatives, cross-derivatives, are zero. (derivative matrices are not symmetric)
A similar case is mean-variance orthogonality in LEF but not in other models. (scale or dispersion estimate from auxiliary estimation or jointly based on extra moment conditions.)


TBC
I'm not sure yet what we can support generically.


















user: josef-pkt
issue_7167: state: open
title: ENH: Allow weights in RollingWLS to match the window length instead of the data length, and interpret appropriately
body: #### Describe the bug
the weights shall be in the shape of (window,), which matches the number of samples in a given window. However, the source code requires the weights in the shape of (nobs,), which matches the number of total samples.


#### Code Sample, a copy-pastable example if possible


```python
import numpy as np
from statsmodels.regression.rolling import RollingWLS
window = 20
weights = np.arange(1,21)
y = np.random.random(1000)
x = np.random.random(1000)
result = RollingWLS(y, x, window = window,weights = weights).fit()

```
#### Expected Output
instead of normal result, it raises a ValueError
# ValueError: weights is required to have shape (1000,) but has shape (20,)
#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]
INSTALLED VERSIONS
------------------
Python: 3.6.10.final.0

statsmodels
===========

Installed: 0.11.1 (D:\Anaconda3\envs\py36\lib\site-packages\statsmodels)

Required Dependencies
=====================

cython: 0.29.21 (D:\Anaconda3\envs\py36\lib\site-packages\Cython)
numpy: 1.17.0 (D:\Anaconda3\envs\py36\lib\site-packages\numpy)
scipy: 1.5.0 (D:\Anaconda3\envs\py36\lib\site-packages\scipy)
pandas: 0.25.3 (D:\Anaconda3\envs\py36\lib\site-packages\pandas)
    dateutil: 2.8.1 (D:\Anaconda3\envs\py36\lib\site-packages\dateutil)
patsy: 0.5.1 (D:\Anaconda3\envs\py36\lib\site-packages\patsy)

Optional Dependencies
=====================

matplotlib: 3.2.2 (D:\Anaconda3\envs\py36\lib\site-packages\matplotlib)
    backend: module://ipykernel.pylab.backend_inline 
cvxopt: Not installed
joblib: 0.16.0 (D:\Anaconda3\envs\py36\lib\site-packages\joblib)

Developer Tools
================

IPython: 7.16.1 (D:\Anaconda3\envs\py36\lib\site-packages\IPython)
    jinja2: 2.11.2 (D:\Anaconda3\envs\py36\lib\site-packages\jinja2)
sphinx: 3.1.2 (D:\Anaconda3\envs\py36\lib\site-packages\sphinx)
    pygments: 2.6.1 (D:\Anaconda3\envs\py36\lib\site-packages\pygments)
pytest: 5.4.3 (D:\Anaconda3\envs\py36\lib\site-packages\pytest)
virtualenv: Not installed
</details>

user: siaosing
issue_7165: state: open
title: ENH: truncated count models
body: #3802  PR for truncated, censored and hurdle models from GSOC 2017, needs rebase, review and finishing up
PR looks pretty good, but will need quite a bit of work, I guess, especially for checking that all main post estimation results work correctly

this issue is to add some notes and references for truncated count models.

One issue in usage is in supporting points that are truncated, e.g. predict for y=0 so we can use it to check zero inflation or deflation. Hurdle model will have full data without truncation. 

user: josef-pkt
issue_7164: state: open
title: Memory Overflow with ANOVA - can I skip the ols part?
body: The suggested way to calculate the 2-way ANOVA with your software is to first fit the linear model via `ols('rez ~ C(A) + C(B) + C(C)', data).fit()`. In my case of 3D fit with dimensions (400, 200, 48) this breaks down with memory overflow of 18 GiB. However, I don't need a linear model. I only need the ANOVA F-ratios and p-values. I have implemented the calculation of such ratios for 1 and 2-way ANOVAs myself, and it does not require any additional memory, it's just about looping and calculating averages and squares. However, I would like a general implementation, and I struggle a bit with figuring out the degrees of freedom for higher orders. 

Do you already have / plan to implement an ANOVA version that works directly on the data and does not require extensive memory?
user: aleksejs-fomins
issue_7163: state: open
title: ENH: Weighted multivariate KDE
body: #### Is your feature request related to a problem? Please describe

I am trying to implement this R package (https://github.com/powellgenomicslab/Nebulosa) in Python and for this I need a  weighted multivariate KDE implementation in Python with a good bandwidth selector. In the R package they use the ks package (https://cran.r-project.org/web/packages/ks/index.html) which supports unconstrained plugin selectors (e.g. https://www.mvstat.net/tduong/research/publications/chacon-duong-2010-test.pdf).

The scipy implementation is nice and it supports weighted multivariate KDE but it doesn't have a good bandwidth selector, only Silverman and Scott. The ~normal_reference~ CV-based selectors in sm look great, however in `sm.nonparametric.KDEMultivariate`, weighting is not supported. 

#### Describe the solution you'd like

I was wondering if it'd be possible to implement, weighting similar to the one in the KDEUnivariate code (https://www.statsmodels.org/stable/_modules/statsmodels/nonparametric/kde.html#KDEUnivariate).
user: gokceneraslan
issue_7162: state: open
title: ENH: helper function for random numbers from multinomial, right truncated count regression
body: just an internal helper function. I coded it a few times but only in experimental code

Should support simulating a regression model for count data, where we don't have a good rvs, e.g. generalized poisson, or truncated, hurdle versions.

It should be vectorized for nobs arrays with different parameters by nobs, i.e. different multinomial probabilities.

we can either draw a single cell number per obs, or cumulative counts.
user: josef-pkt
issue_7160: state: open
title: ENH: half-normal plot for discrete and glm
body: Moral, Rafael A., John Hinde, and Clarice G. B. Demétrio. 2017. “Half-Normal Plots and Overdispersed Models in R: The Hnp Package.” Journal of Statistical Software 81 (1): 1–23. https://doi.org/10.18637/jss.v081.i10.

The article has a good list of which packages in R provide the pieces for various count models including simulating data for the envelope.


user: josef-pkt
issue_7159: state: open
title: Create empty GLM model and insert parameters
body: Is there any way to create an empty GLM Binomial Model? And then proceed to set the coefficients manually?
user: FranciscoReveriano
issue_7158: state: open
title: FAQ: offset and exposure in zero-inflated count models
body: The effect of exposure in zero-inflated models depends on the interpretation of the inflation process and additivity of the distribution over time.

Aside: a similar issue shows up in the mixing interpretation in overdispersed model like Negative Binomial and Tweedie and the effect of time aggregation on the dispersion.


Feng 2020 recommends using exposure not as an offset variable, but include it as a regular variable with estimated coefficients in both inflation and main model. (I only skimmed parts of the article.)

Feng, Cindy. 2020. “Zero-Inflated Models for Adjusting Varying Exposures: A Cautionary Note on the Pitfalls of Using Offset.” Journal of Applied Statistics 0 (0): 1–23. https://doi.org/10.1080/02664763.2020.1796943.



user: josef-pkt
issue_7155: state: open
title: ENH: GEE with zero-inflated count models, generalized Poisson
body: extensions to zero-inflated count models

Sarvi, Fatemeh, Abbas Moghimbeigi, and Hossein Mahjub. 2019. “GEE-Based Zero-Inflated Generalized Poisson Model for Clustered over or under-Dispersed Count Data.” Journal of Statistical Computation and Simulation 89 (14): 2711–32. https://doi.org/10.1080/00949655.2019.1632857.

Can we extend the GEE approach to general distributions, i.e. outside GLM families or LEF?

Because it's outside the LEF, estimating the mean function will not be orthogonal to other distributional assumptions, e.g. cluster correlation.

Currently we don't have any setup to handle cases like this.


user: josef-pkt
issue_7154: state: open
title: ENH: EDF tests for regression residuals 
body: mainly parking a theoretical result

quote from Stephens' reference entry
"have shown that the asymptotic distributions of EDF statistics are the same as if both 𝜇 and 𝜎 were estimated"

This means we should use the p-values from the tables where loc and scale are estimated, e.g. for gaussian, and not for known distribution.

Pierce, Donald A., and Kenneth J. Kopecky. 1979. “Testing Goodness of Fit for the Distribution of Errors in Regression Models.” Biometrika 66 (1): 1–5. https://doi.org/10.1093/biomet/66.1.1.

Stephens, Michael A. 2016. “Empirical Distribution Function (EDF) Statistics.” In Wiley StatsRef: Statistics Reference Online, 1–13. American Cancer Society. https://doi.org/10.1002/9781118445112.stat02711.pub2.

I didn't read details and don't know if that applies to other distributions, e.g. can we use it for continuous distributions in GLM.
We could get simulated p-values but just simulating a loc-scale model instead of the full regression model.


user: josef-pkt
issue_7153: state: open
title: ENH: quantile residuals, normal scores, uniform scores for GLM and generic
body: quantile residual seems to be a "newer" alias for normal scores, i.e. use ppf to transform to normal distribution

related: uniform scores for count distributions, see #1537 #2027 and maybe others (several articles by Czado et al)

The advantage of normal scores or quantile residuals is that we can use tools for specification test developed for normal distribution.

related
#3224 tools for transforming to normal distribution
#7142 predictive distribution as target

some recent articles but there are also older ones

Feng, Cindy, Longhai Li, and Alireza Sadeghpour. 2020. “A Comparison of Residual Diagnosis Tools for Diagnosing Regression Models for Count Data.” BMC Medical Research Methodology 20 (1): 175. https://doi.org/10.1186/s12874-020-01055-2.

Pereira, Gustavo H. A. 2017. “On Quantile Residuals in Beta Regression.” Communications in Statistics - Simulation and Computation, November. https://www.tandfonline.com/doi/abs/10.1080/03610918.2017.1381740.

Pereira, Gustavo H. A., Juliana Scudilio, Manoel Santos-Neto, Denise A. Botter, and Mônica C. Sandoval. 2020. “A Class of Residuals for Outlier Identification in Zero Adjusted Regression Models.” Journal of Applied Statistics 47 (10): 1833–47. https://doi.org/10.1080/02664763.2019.1696759.

Scudilio, Juliana, and Gustavo H. A. Pereira. 2020. “Adjusted Quantile Residual for Generalized Linear Models.” Computational Statistics 35 (1): 399–421. https://doi.org/10.1007/s00180-019-00896-w.

user: josef-pkt
issue_7152: state: open
title: ENH: GLM-Binomial with beta-binomial variance function
body: related
#2632 over dispersed binomial, beta-binomial
#1777 GLM with foreign variance function

My guess is that most of the things should work out of the box. Maybe we need HC cov_type for inference.
Parameter estimates should be the same as estimating a Beta-Binomial model with fixed dispersion parameter similar to GLM-negative binomial.
Not sure if GLM with fake/quasi binomial provides a way to estimate the dispersion parameter, e.g. in GLM NegBin we can use profile likelihood, otherwise we need auxiliary estimate like overdispersion based on pearson chi2.

The following articles use quasi-likelihood (integral version), but all we need to mean and variance functions as M-estimators

Hinde, John, and Clarice G. B. Demétrio. 1998. “Overdispersion: Models and Estimation.” Computational Statistics & Data Analysis 27 (2): 151–70. https://doi.org/10.1016/S0167-9473(98)00007-3.

Vieira, A. M. C., J. P. Hinde, and C. G. B. Demetrio. 2000. “Zero-Inflated Proportion Data Models Applied to a Biological Control Assay.” Journal of Applied Statistics 27 (3): 373–89. https://doi.org/10.1080/02664760021673.

They mention different versions of beta-binomial similar to negbin versions, but they don't provide details.
Similar to negbin, only one beta-binomial parameterization is in the LEF.

proper and full implementation would need to go into 





user: josef-pkt
issue_7151: state: open
title: ENH: outlier detection through penalized estimation
body: mainly parking some references, I didn't find an existing issue, but I thought there should be at least some comments on this

In (most/all ?) models we can remove the effect of an observation on the parameter estimates by including an observation dummy, observation specific mean shift. If we want to include all, then this adds nobs explanatory variables, and k_vars > nobs.

We can use penalized estimation to shrink the parameters for the outlier dummies, or shrink and threshold them to 0 with SCAD or L1 penalization.

One issue is what we use as a starting point, we cannot start with non-penalized estimate. If we start with fully penalized estimate, then we might still have masking and similar effects.

She, Yiyuan, and Art B. Owen. 2011. “Outlier Detection Using Nonconvex Penalized Regression.” Journal of the American Statistical Association 106 (494): 626–39. https://doi.org/10.1198/jasa.2011.tm10390.

The list of citing articles looks also interesting, but I didn't look closely at any of those.

Aside: I added the article to my Zotero 4 times between 2011 and 2016, but by now we should have all or most of the pieces available. I might have tried something like this during the GAM/penalized PR merges.

We don't have penalized RLM yet.

Also, it would be good to have a prior magnitude of the penalization weight, e.g. penalization weight implied by given effective degrees of freedom.




user: josef-pkt
issue_7150: state: open
title: CLN: common base class for statespace and ets
body: - [x] closes #6881
- [x] tests passed. 

This summarizes all common methods of the statespace methods and the new ETS implementation in a single base class.
user: s-scherrer
issue_7149: state: open
title: ENH: add binned prediction table
body: specific issue for a function that computes prediction table based on binning the space of endog. It can be attached as method to models that don't have pred_table.

Should work for continuous and count models, Maybe a good default would be 5 bins, so it can still be easily printed.

I think we would need predictive distribution, we cannot just compare observations with predicted mean.

related: 
#5388 tools for binning and similar
#7080 add "inconclusive" to binary, multinomial
#7142 predictive distribution
#6252 PR for pred_table in GLM, binary only.

#6430 plots for predicted probabilities, not a cross-table, only comparing margins


maybe confusing
pred table is for classification, i.e. for choice with highest prob.
Maye I want more something with predicted probabilities, i.e. observed versus prob of bins

The only current example is in my count notebook or maybe diagnostic count for new models where we have all predicted probabilities in `predict` method.

Other version:
Instead of binning in endog space, we could groups observations similar to Hosmer-lemeshow test and compare group means and predicted means.
But this would be more a check on nonlinearity, no breaks or homogeneity across groups instead of on predictive distribution

user: josef-pkt
issue_7148: state: open
title: ENH meta-analysis for binary outcomes
body: mainly parking an article to compare with what we already have 

Mathes, Tim, and Oliver Kuss. 2018. “A Comparison of Methods for Meta-Analysis of a Small Number of Studies with Binary Outcomes.” Research Synthesis Methods 9 (3): 366–81. https://doi.org/10.1002/jrsm.1296.
(I only did some minimal skimming)

seems to have many hybrid methods, e.g. one recommended is
PMHK: random effects model with Paule‐Mandel between‐study variance estimator and Hartung‐Knapp‐Sidik‐Jonkman confidence intervals

in conclusion they argue in favor of using t-distribution for inference
includes beta-binomial methods

Bakbergenuly, Ilyas, and Elena Kulinskaya. 2017. “Beta-Binomial Model for Meta-Analysis of Odds Ratios.” Statistics in Medicine 36 (11): 1715–34. https://doi.org/10.1002/sim.7233.
(I didn't look at it)

another related article that I didn't look at

Bakbergenuly, Ilyas, David C. Hoaglin, and Elena Kulinskaya. 2020. “Methods for Estimating Between-Study Variance and Overall Effect in Meta-Analysis of Odds Ratios.” Research Synthesis Methods 11 (3): 426–42. https://doi.org/10.1002/jrsm.1404.



user: josef-pkt
issue_7144: state: open
title: Missing parameter to 'None' is same as 'Drop'
body: Hi, 

cf title, I just checked and it does the same thing, None parameter will just drop rows with NA values. I think it would be nice to put 'drop' as default parameter and delete None, no ? It would be more self-explanatory.

Kind regards, 
Isaac
user: neotis70
issue_7142: state: open
title: SUMM: predictive distribution instead of parameter estimation
body: We have a large emphasis of and support for (mean) parameter estimation and associated inference and interpretation like margins (ATE).

When merging new count models including zero-inflated, the emphasis was on how well they fit the probability distribution.

Some specification and diagnostic tests check how well the distribution fits in some direction, e.g. probability plots for normal, chisquare tests for discrete and GLM.

control charts and outlier detection need to evaluate one or a few observations and decide whether the fit the predictive distribution, often or usually using some ad-hoc or default distributional assumptions.

We have many open issues for predictive performance measures.

So, we need

How well does the predictive distribution fit the data or new data?
What is the model that provides the best distributional predictions?

and what do we need to implement to make this easy to check?

This is to a large extend the opposite of GLM/LEF robust mean function estimation. 
One conclusion in my count notebook was, Poisson works well if we just want to estimate an exponential mean function, but can be pretty bad in predicting probabilities for the count frequencies.

Related:
The emphasis here is choosing the right distribution. This is complementary to variable selection.
Aside: Some article (?) that I recently read was motivating this with propensity score estimation where the predictive probabilities are crucial for the estimate of ATT. The current trend is using machine learning for creating fancier features and feature selection. 
For binary link models the important part is the specification of the link or the distribution of the underlying latent variable. I guess adding enough nonlinear features can compensate for a misspecified link function, if we have enough data to estimate.

related issues:

...


cases

- additive error, residuals
  - linear models like OLS: we have some residual analysis and prob plots and similar for distributional assumptions
  - extensions to elliptical, symmetric models
  - skewed model ???
- continuous distribution with non-additive error, we need conditional distribution of endog F(y | x)
  - Gamma and inverse gaussian in GLM
  - missing distributions for parametric survival, beta regression, ....
- discrete distributions
  - binary
  - count
  - multinomial, ordinal, multivariate counts

- nonparametric, semi-parametric ???
  - quantile regression and similar that matches only some non-mean properties of predictive distribution
  - GMM: we can add distributional moment conditions, e.g. on quantiles or bins.
  - ...

Getting started

???
support in predict
gof specification measures and tests





user: josef-pkt
issue_7137: state: open
title: ENH: control charts for zero-inflated count data
body: There are many articles on control charts for zero-inflated count data, many published in the last two years
(I only skimmed abstracts)

just one reference to an overview article 

Mahmood, Tahir, and Min Xie. 2019. “Models and Monitoring of Zero-Inflated Processes: The Past and Current Trends.” Quality and Reliability Engineering International 35 (8): 2540–57. https://doi.org/10.1002/qre.2547.

citation count for older articles in this area is pretty low.

The main point would be what we should add to models to support this, and see whether we get overlapping code.
E.g. outlier identification #7134 and predictive distributions
`predict` in ZI models already returns optionally all the pieces that describe the distribution of a new observation.
We are still missing confidence intervals, or standard errors for those, e.g. #4745 #2015

It would be nice to have the setup for control charts merged so we could easily try out estimators and methods.

asides: 
- I think diagnostic chisquare test for binned predictive distribution might be useful as a monitoring statistic
- There is a literature on calibration of hypothesis/gof tests for out of sample observations, e.g. for medical diagnostic tests. #6430

user: josef-pkt
issue_7136: state: open
title: ENH: count models with more general inflation or modification at some counts
body: this is a generalization of models like ZIP

I didn't really look at it, but I was browsing abstract of many articles that do some kind of inflation or modification to the probabilities of counts. The article here sounds like a more general version that could encompass several others.
aside: I should resurrect the finite mixture of Poisson code which might have similar properties

Cai, Tianji, Yiwei Xia, and Yisu Zhou. 2018. “Generalized Inflated Discrete Models: A Strategy to Work with Multimodal Discrete Distributions.” Sociological Methods & Research, July, 0049124118782535. https://doi.org/10.1177/0049124118782535.

Xia, Yiwei, Yisu Zhou, and Tianji Cai. 2019. “Gidm: A Command for Generalized Inflated Discrete Models:” The Stata Journal, September. https://doi.org/10.1177/1536867X19874246.

related: zero- or other inflated Ordered Probit #7135 

user: josef-pkt
issue_7135: state: open
title: ENH: add inflated order probit models
body: looks kind of interesting, adding zero inflation or inflation at other points to ordered probit.
(I didn't really look at the articles.)

the first article is still in early view, not yet in an issue

Brown, Sarah, Mark N. Harris, and Christopher Spencer. n.d. “Modelling Category Inflation with Multiple Inflation Processes: Estimation, Specification and Testing.” Oxford Bulletin of Economics and Statistics n/a (n/a). Accessed November 1, 2020. https://doi.org/10.1111/obes.12366.

Harris, Mark N., and Xueyan Zhao. 2007. “A Zero-Inflated Ordered Probit Model, with an Application to Modelling Tobacco Consumption.” Journal of Econometrics 141 (2): 1073–99. https://doi.org/10.1016/j.jeconom.2007.01.002.

my guess: in some cases this might be more like count variables, e.g. interval censored count models with inflation at some points.
related: count models with more general types of inflation or modifications (will be separate issue)

user: josef-pkt
issue_7134: state: open
title: ENH: Influence measures for count models, discrete, special cases of MLE
body: special cases of #4257
AFAIR, we don't have any generic MLE influence classes for specific models, I only have GLMInfluence as special case
maybe most of it is already in the generic class

3 references that look good and use generic MLE definitions of influence measures

Garay, Aldo M., Elizabeth M. Hashimoto, Edwin M. M. Ortega, and Víctor H. Lachos. 2011. “On Estimation and Influence Diagnostics for Zero-Inflated Negative Binomial Regression Models.” Computational Statistics & Data Analysis 55 (3): 1304–18. https://doi.org/10.1016/j.csda.2010.09.019.

Xie, Feng-Chang, and Bo-Cheng Wei. 2007. “Diagnostics Analysis in Censored Generalized Poisson Regression Model.” Journal of Statistical Computation and Simulation 77 (8): 695–708. https://doi.org/10.1080/10629360600581316.

———. 2010. “Influence Analysis for Count Data Based on Generalized Poisson Regression Models.” Statistics 44 (4): 341–60. https://doi.org/10.1080/02331880903138494.

user: josef-pkt
issue_7133: state: open
title: DOC/FAQ: which model does info criteria select with misspecified likelihood
body: A question about interpretation and what effect misspecification has on variable selection

specific subquestion:
In a LEF/GLM model the QMLE estimates the mean function consistently even if variance function is misspecified.

Suppose we want to add more explanatory variables. 

Could they have a significant effect and be included according to some specification tests even if they just capture misspecification in the non-mean part?
Would AIC, BIC, QIC with excess dispersion correction or TIC, choose to include variables because of their higher moment properties?

My guess would be yes.

obvious case: an extra variable is a proxy for unobserved heterogeneity, e.g. NB as mixture of Poisson.

The analogy for specification test is that often we have diagnostic tests that also have power against other misspecifications than the ones they were designed for.


Related to adding LR test and information criteria that are "robust" to some misspecification or don't assume correctly specified likelihood.
Robust for what? 
e.g. we find the model that is closest in terms of KL distance, but it might add things that wouldn't be needed if we had the correct model.

Related: directional specification tests
General GOF and information criteria only care about whether a model fits the data or which model fits best among our candidate models. They don't tell us in which direction we might want to expand our search for candidate models.

QMLE/LEF arguments are based on a correctly specified mean function (emphasized by e.g. Wooldridge)
But we are not sure that we have a correctly specified mean function. 
But, e.g OLS doesn't depend on higher moments so differences in higher moments shouldn't have an effect. However, likelihood base information criteria do (except for gaussian?)

???

I need some examples and simulations
user: josef-pkt
issue_7132: state: open
title: ENH: support zero-inflated models without inflation or with deflation, corner case
body: The ZI models in `discrete.count_model` use either Logit or Probit as inflation model. 
This requires that prob(inflation) > 0 and prob(inflation) -> 0 iff xb -> - inf 

That is we don't reach the no-inflation case and we cannot have negative inflation.

This implies that we cannot test for no inflation with the model.
see #7131 and maybe others for score tests for zero-inflations

Possibility: 
- allow linear probability model, that could have zero or negative probability.
- maybe add model without explanatory variables, i.e. linear prob (with constant or offset only)


if we add GLM as option, then we would get linear link in binomial for free, but I don't remember whether we can evaluate score at negative prob, I guess not.

AFAICS, ZI loglike only uses predicted prob from the inflation model and does not need the loglike of the infl model.



user: josef-pkt
issue_7131: state: open
title: ENH: score test for zero-inflation and over/excess dispersion
body: I wrote separate tests for zero-inflation and over dispersion in Poisson.

more general:
- score test for other count models, distributions, NB, Binomial
- joint test for both, or test for one that is robust the the other

Deng, Dianliang, and Sudhir R. Paul. 2005. “Score Tests for Zero-Inflation and over-Dispersion in Generalized Linear Models.” Statistica Sinica 15 (1): 257–76.

(reference found through penalization article, https://github.com/statsmodels/statsmodels/issues/7128#issuecomment-719935637 )

**update**
I found another article by same authors
Deng, Dianliang, and Sudhir R. Paul. “Score Tests for Zero Inflation in Generalized Linear Models.” Canadian Journal of Statistics 28, no. 3 (2000): 563–70. https://doi.org/10.2307/3315965.

user: josef-pkt
issue_7130: state: open
title: ENH: specification tests for heteroscedasticity in count models, GLM, poisson, binomial
body: just an idea, I didn't look for literature and never saw it

We have tests for over dispersion which is a single scale parameter. Instead variance misspecification could depend on other explanatory variables, not just as function of fitted values.

One model check that is sometimes mentioned is to use offset or exposure variable as explanatory variable. This is more general than fixing the parameter and provides a check on whether the response is "linear", proportional in exposure/offset. 

What's the equivalent of heteroscedasticity specification tests that we have for OLS for GLM and count models?
So far I have mainly looked at misspecification in mean function for GLM.

Note: in GLM/LEF variance misspecification does not affect consistency of mean parameter estimates, but affects efficiency.

Related: tests for predictions, e.g. hosmer-lemeshow or similar chisquare tests will be sensitive to variance misspecification, e.g. Poisson versus GPP versus NBP with identical mean functions.

Related: AFAIR, we don't have anything to estimate the `P` in GPP, NBP. 
(I guess a similar issue to Tweedie power parameter, but we have explicit loglike wrt p and can use profile/concentrated likelihood)

user: josef-pkt
issue_7128: state: open
title: ENH: add penalized classes, cases for penalized mixin
body: I think our PenalizedMixin is underused and not very convenient.

We need specific classes to make it more usable, and so we can start to collect more post-estimation results.
e.g. adding penalization specific information criteria #5404

We might also want to add some with a specific penalty function, e.g. GLMSCAD or similar, e.g. for use with sure independence screening SIS.

extension 
Two part models like zero-inflated, or later models that have mean and variance functions are not yet (fully) supported for variable selection, e.g. SIS, variable addition score_test. Although, PenalizedMixin should be generic enough for those.


related: missing features in GAM #8029
user: josef-pkt
issue_7125: state: open
title: FAQ: No convergence of PoissonBayesMixedGLM: "RuntimeWarning: overflow encountered in exp"
body: Hi! I'm trying to fit a PoissonBayesMixedGLM to my data, but get an "overflow encountered in exp" error.

The model I'm trying to fit might be too complex for the number of data points (28 individuals), but I was wondering if there is a way to figure out what causes the error.

Here's the code I'm using, and the needed data file:


[RPSY1_expression_data.txt](https://github.com/statsmodels/statsmodels/files/5465037/RPSY1_expression_data.txt)


```python
import pandas as pd
from statsmodels.genmod.bayes_mixed_glm import PoissonBayesMixedGLM
path_to_txt = "./RPSY1_expression_data.txt"
data = pd.read_csv(path_to_txt,index_col=0)
vcf = {"dataset": "0 + C(dataset)", "subject": "0 + C(subject_ID)"}
formula = "RPS4Y1 ~ log_size_factor + age  + sex + smoking_status"
model = PoissonBayesMixedGLM.from_formula(
    formula=formula, vc_formulas=vcf, data=data
)
result = model.fit_vb()
```

The data contains nested groups (group level 1: dataset, group level 2: individual), with many measurements per individual. The response variable is RNA counts of a gene (RPS4Y1) that is only expressed in men,  and so the expected output would be that the coefficient for "sex" is large and with low p-value, versus a small/unsignificant coefficient for the other covariates. (The log_size_factor is actually an offset, and should therefore also be significant). I would expect it to be a relatively simple task to fit this, and I am therefore surprised by the error.

When I try to fit the model, I get the following warnings/errors:

```
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:780: RuntimeWarning: overflow encountered in exp
  return -self.vb_elbo(x[:n], np.exp(x[n:]))
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:625: RuntimeWarning: divide by zero encountered in log
  r = (iv + np.sum(np.log(fep_sd)) + np.sum(np.log(vcp_sd)) + np.sum(
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/numpy/core/fromnumeric.py:90: RuntimeWarning: invalid value encountered in reduce
  return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:626: RuntimeWarning: divide by zero encountered in log
  np.log(vc_sd)))
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:784: RuntimeWarning: overflow encountered in exp
  gm, gs = self.vb_elbo_grad(x[:n], np.exp(x[n:]))
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:649: RuntimeWarning: invalid value encountered in true_divide
  r = u / np.sqrt(tv)
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:675: RuntimeWarning: divide by zero encountered in true_divide
  fep_sd_grad += 1 / fep_sd
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:677: RuntimeWarning: divide by zero encountered in true_divide
  vc_sd_grad += 1 / vc_sd
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:785: RuntimeWarning: overflow encountered in exp
  gs *= np.exp(x[n:])
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:814: RuntimeWarning: overflow encountered in exp
  iv -= np.sum((vc_mean**2 + vc_sd**2) * np.exp(2 * (s**2 - m))) / 2
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:832: RuntimeWarning: overflow encountered in exp
  ve = np.exp(2 * (s**2 - m))
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:834: RuntimeWarning: invalid value encountered in multiply
  ds = -2 * u * ve * s
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:839: RuntimeWarning: invalid value encountered in multiply
  vc_sd_grad = -vc_sd.copy() * ve
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:672: RuntimeWarning: invalid value encountered in add
  vc_mean_grad += vc_mean_grad_i
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:676: RuntimeWarning: divide by zero encountered in true_divide
  vcp_sd_grad += 1 / vcp_sd
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:814: RuntimeWarning: overflow encountered in multiply
  iv -= np.sum((vc_mean**2 + vc_sd**2) * np.exp(2 * (s**2 - m))) / 2
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:833: RuntimeWarning: overflow encountered in multiply
  dm = u * ve - 1
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:834: RuntimeWarning: overflow encountered in multiply
  ds = -2 * u * ve * s
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:780: RuntimeWarning: overflow encountered in exp
  return -self.vb_elbo(x[:n], np.exp(x[n:]))
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:625: RuntimeWarning: divide by zero encountered in log
  r = (iv + np.sum(np.log(fep_sd)) + np.sum(np.log(vcp_sd)) + np.sum(
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/numpy/core/fromnumeric.py:90: RuntimeWarning: invalid value encountered in reduce
  return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:626: RuntimeWarning: divide by zero encountered in log
  np.log(vc_sd)))
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:784: RuntimeWarning: overflow encountered in exp
  gm, gs = self.vb_elbo_grad(x[:n], np.exp(x[n:]))
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:649: RuntimeWarning: invalid value encountered in true_divide
  r = u / np.sqrt(tv)
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:675: RuntimeWarning: divide by zero encountered in true_divide
  fep_sd_grad += 1 / fep_sd
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:677: RuntimeWarning: divide by zero encountered in true_divide
  vc_sd_grad += 1 / vc_sd
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:785: RuntimeWarning: overflow encountered in exp
  gs *= np.exp(x[n:])
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:676: RuntimeWarning: divide by zero encountered in true_divide
  vcp_sd_grad += 1 / vcp_sd
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:785: RuntimeWarning: invalid value encountered in multiply
  gs *= np.exp(x[n:])
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:814: RuntimeWarning: overflow encountered in exp
  iv -= np.sum((vc_mean**2 + vc_sd**2) * np.exp(2 * (s**2 - m))) / 2
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:832: RuntimeWarning: overflow encountered in exp
  ve = np.exp(2 * (s**2 - m))
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:834: RuntimeWarning: invalid value encountered in multiply
  ds = -2 * u * ve * s
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:839: RuntimeWarning: invalid value encountered in multiply
  vc_sd_grad = -vc_sd.copy() * ve
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:672: RuntimeWarning: invalid value encountered in add
  vc_mean_grad += vc_mean_grad_i
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:784: RuntimeWarning: overflow encountered in exp
  gm, gs = self.vb_elbo_grad(x[:n], np.exp(x[n:]))
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:649: RuntimeWarning: invalid value encountered in true_divide
  r = u / np.sqrt(tv)
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:832: RuntimeWarning: overflow encountered in exp
  ve = np.exp(2 * (s**2 - m))
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:834: RuntimeWarning: invalid value encountered in multiply
  ds = -2 * u * ve * s
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:839: RuntimeWarning: invalid value encountered in multiply
  vc_sd_grad = -vc_sd.copy() * ve
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:672: RuntimeWarning: invalid value encountered in add
  vc_mean_grad += vc_mean_grad_i
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:675: RuntimeWarning: divide by zero encountered in true_divide
  fep_sd_grad += 1 / fep_sd
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:676: RuntimeWarning: divide by zero encountered in true_divide
  vcp_sd_grad += 1 / vcp_sd
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:677: RuntimeWarning: divide by zero encountered in true_divide
  vc_sd_grad += 1 / vc_sd
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:785: RuntimeWarning: overflow encountered in exp
  gs *= np.exp(x[n:])
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:792: UserWarning: VB fitting did not converge
  warnings.warn("VB fitting did not converge")
/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels/genmod/bayes_mixed_glm.py:796: RuntimeWarning: overflow encountered in exp
  va = np.exp(2 * mm.x[n:])
```

None of the values in my data are extremely large (max is 80 or so), so I don't think that could be the problem. Do you have any idea of why I'm getting these errors, and why the fitting does not converge? Fitting a comparable model but with gaussian noise (sm.MixedLM) does not give any problems. However, I need a Poisson noise model.
Thank you for your help!


<details>

I checked if the same problem exists for the current master (sm.__version__ '0.12.1'), and it does. 

</details>

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

```
INSTALLED VERSIONS
------------------
Python: 3.7.5.final.0
OS: Linux 3.10.0-1062.el7.x86_64 #1 SMP Wed Aug 7 18:08:02 UTC 2019 x86_64
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8

statsmodels
===========

Installed: 0.13.0.dev0+87.g52b142c (/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/statsmodels-0.13.0.dev0+87.g52b142c-py3.7-linux-x86_64.egg/statsmodels)

Required Dependencies
=====================

cython: 0.29.21 (/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/Cython)
numpy: 1.17.4 (/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/numpy)
scipy: 1.4.1 (/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/scipy)
pandas: 1.0.5 (/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/pandas)
    dateutil: 2.8.1 (/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/dateutil)
patsy: 0.5.1 (/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.3.2 (/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/matplotlib)
    backend: module://ipykernel.pylab.backend_inline 
cvxopt: Not installed
joblib: 0.14.0 (/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/joblib)

Developer Tools
================

IPython: 7.9.0 (/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/IPython)
    jinja2: 2.10.3 (/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/jinja2)
sphinx: Not installed
    pygments: 2.4.2 (/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/pygments)
pytest: 5.3.2 (/home/icb/lisa.sikkema/miniconda3/envs/scRNAseq_analysis_backup/lib/python3.7/site-packages/pytest)
virtualenv: Not installed
```

user: LisaSikkema
issue_7119: state: open
title: add offset to PoissonBayesMixedGLM
body: Hi,
I would like to use a PoissonBayesMixedGLM as you guys have implemented, but would like to add an 'extra' offset to the formula. My current formula looks like this:
```
vcf = {"dataset": "0 + C(dataset)", "subject": "0 + C(subject_ID)"}
model = PoissonBayesMixedGLM.from_formula(
    formula="gene_count ~ sex + age + smoking_status + ethnicity",
    vc_formulas=vcf,
    data=mlm_data,
)
```
However, I would like to add a specific offset for each observation I have, and on top of that the model should fit the (general) intercept. I think this could be done either using an offset argument, as is already available for e.g. statsmodels.genmod.generalized_linear_model.GLM. 
```
offset=offset
```
Alternatively, if there is a way to fix the coefficient for one of the model covariates, the formula could look something like this:
```
formula="gene_count ~ sex + age + smoking_status + ethnicity + 1 * offset"
```

Is there already a way to do this and did I miss it? And if not, is there a chance you guys could implement it in the code?

Thanks a lot, you guys are heroes for working on statsmodels.
user: LisaSikkema
issue_7113: state: open
title: ENH: prediction of a weighted average in nonlinear models
body: related #7112 survey support in models.

If we have freq_weights or similar and we want to have a prediction for a weighted average over a subgroup, with weights not necessarily the same as in the estimation sample.

In nonlinear models like GLM and discrete, we want a weighted mean prediction and not a prediction at a weighted mean of exog.

the predicted value itself is easy to compute also by user, just compute components and weighted average, but not standard errors and similar inferential statistics like confidence intervals. 
(It's not obvious to me right now how to do that.)


related: 
For IPW inverse probability or propensity score weighting we compute potential outcome mean POM and ATE, ATT suing GMM (my prototype based on approach in Stata) #2455
But that uses the estimation weights, not separate weights for prediction.

user: josef-pkt
issue_7112: state: open
title: ENH/SUMM: survey sampling support in models, sandwich score_test
body: related:
#3335 issue with my earlier (2017) notes
#3757 issue for GSOC, PRs #3762 #5174
#3375 issue with notes on `weights` for survey sampling

I think I finally found a way to get a clear structure on supporting survey sampling in the models, by working backwards and ignoring sampling details.

Based on comments that I read over time: survey sampling with unit level data is mainly freq_weights/survey_weights + robust standard errors that take sampling correlations into account.

score_test for survey data Rao et al 1998, references Boos for sandwich score_test, My robust score_test is based on that. Center of sandwich is covariance of score_obs estimated through survey methods for a total


Skinner 2019 is a good survey with references to get started on specific issues, includes section on GEE.
includes also a section on aggregated statistics like cell counts, binomial, contingency tables and loglinear model.

Binder 1983 seems to be the canonical reference for sandwich cov in survey methods extended to GLM
Fuller has an article in 1975 for linear model, but not available on internet (Sankhya, maybe nobody ever scanned it. )

Fuller has a book Sampling Statistic that I hadn't looked at before.
Based on table of content it doesn't have anything on discrete endog, GLM, ...
(I like Fuller's books because he includes enough theory and proofs, more technical and econometrics style than most others)

References

Skinner, Chris. 2019. “Analysis of Categorical Data for Complex Surveys.” International Statistical Review 87 (S1): S64–78. https://doi.org/10.1111/insr.12285.

Binder, David A. 1983. “On the Variances of Asymptotically Normal Estimators from Complex Surveys.” International Statistical Review / Revue Internationale de Statistique 51 (3): 279–92. https://doi.org/10.2307/1402588.

Rao, J. N. K., A. J. Scott, and C. J. Skinner. 1998. “Quasi-Score Tests with Survey Data.” Statistica Sinica 8 (4): 1059–70.

Fuller, Wayne A. "Regression analysis for sample survey." Sankhya 37, no. 3 (1975): 117-132. 
Fuller, Wayne A. Sampling statistics. Vol. 560. John Wiley & Sons, 2011.

user: josef-pkt
issue_7110: state: open
title: 'MICEData' object is not iterable
body: I wanted to use MICE to generate imputed datasets beforehand. So I tried to run the simple example provided on the [website ](https://www.statsmodels.org/stable/generated/statsmodels.imputation.mice.MICEData.html)but it gave me an error saying that **'MICEData' object is not iterable**. Not sure if I am doing something wrong or if there is some bug...

#### Code Sample

```python#python 3.6.6

from statsmodels.imputation import mice #version '0.12.0'
import pandas as pd # version '0.25.0'
import numpy as np #version '1.16.3'

df=pd.DataFrame([[1,2,3],[3,4,5],[6,3,np.nan],[np.nan,3,6],[np.nan,np.nan,9]],columns=['col1','col2','col3'])

imp = mice.MICEData(data=df)

j = 0
for data in (imp):
    imp.data.to_csv.to_csv('data%02d.csv' % j)
    j += 1
```


user: Esculab
issue_7108: state: open
title: REF: cleanup and deprecate names for genmod.links,
body: Links still has leftovers from the initial structure as instances, e.g. lower case class names and duplication between lower case and upper case, e.g. log/Log

We should deprecate all lower case names, deleting then duplicates and renaming others to upper case, e.g. probit -> Probit


related probit is an alias subclass of CDFlink, but I think we should be able to add explicit deriv2 instead of using the numdiff version in CDFLink

aside:
Why do we use deriv2 instead of inverse_deriv2 (derivative of mean function) which is easier to compute in gaussian case and others with nice pdf
user: josef-pkt
issue_7106: state: open
title: KDEMultivariate.cdf() > 0 for regions of 0 density
body: #### Describe the bug

I'll start with this: I'm not a statistician, just an engineer with a reasonable understand of how things work. I apologise if I don't explain this problem well.

Currently it appears that the KDEMultivariate.cdf() is not functioning as expected given that for two variables, u and v, and an interval for u with moderate probability density in u and an interval in v with approximately no density in v; that the CDF evaluates this region to be non-zero. In other words, in the KDE space of u, v, p (where p is the probability density), that for a region with effectively no probability density, that the CDF is evaluating this region as though it had non-zero density.

#### Code Sample, a copy-pastable example if possible
I have written an example comparing outputs with scipy gaussian KDE and which produces a simple seaborn plot to sanity check with. Viewing the plot, I expect that on the interval u, v = [ 1±0.3, 20±0.3] that the integral would be ~0 (which I think is the output of the CDF on that interval).

```python
# Third Party
import numpy
import seaborn as sns
from matplotlib import pyplot as plt
from scipy.stats import gaussian_kde
from statsmodels.nonparametric.kernel_density import KDEMultivariate

# Create data set
nobs = 300
numpy.random.seed(1234)  # Seed random generator
u = numpy.random.normal(size=(nobs, 1)).ravel()
v = numpy.random.normal(size=(nobs, 1)).ravel()
uv = numpy.vstack([u.ravel(), v.ravel()])

# Initialise kernels
ku_sm = KDEMultivariate(data=u, var_type="c", bw="normal_reference")
kv_sm = KDEMultivariate(data=v, var_type="c", bw="normal_reference")
kuv_sm = KDEMultivariate(data=uv, var_type="cc", bw="normal_reference")

ku_sci = gaussian_kde(dataset=u)
kv_sci = gaussian_kde(dataset=v)
kuv_sci = gaussian_kde(dataset=uv)


# Choose values to test about
u_o = numpy.array(0).ravel()
v_o = numpy.array(20).ravel()

u_t = numpy.array(u_o)
v_t = numpy.array(v_o)
uv_t = numpy.vstack([u_t, v_t])

# Choose consistent bw for both u and v:
bw = numpy.array(0.3)
bw_uv = numpy.vstack([bw, bw])


# Evaluate cdf on bounds for statsmodels
pa = ku_sm.cdf(u_t - bw / 2)
pb = ku_sm.cdf(u_t + bw / 2)
p_u_t_sm = pb - pa

pa = kv_sm.cdf(v_t - bw / 2)
pb = kv_sm.cdf(v_t + bw / 2)
p_v_t_sm = pb - pa

pa = kuv_sm.cdf(uv_t - bw_uv / 2)
pb = kuv_sm.cdf(uv_t + bw_uv / 2)
p_uv_t_sm = pb - pa

# Evaluate cdf on bounds for scipy
lower = u_t - bw / 2
upper = u_t + bw / 2
p_u_t_sci = ku_sci.integrate_box_1d(lower, upper)

lower = v_t - bw / 2
upper = v_t + bw / 2
p_v_t_sci = kv_sci.integrate_box_1d(lower, upper)

lower = (uv_t - bw_uv / 2).ravel()
upper = (uv_t + bw_uv / 2).ravel()
p_uv_t_sci = kuv_sci.integrate_box(lower, upper)

# Spew results:
print(f"u,v interval lower: {uv_t - bw_uv / 2}")
print(f"u,v interval upper: {uv_t + bw_uv / 2}")
print(f"p_u_t_sm: {p_u_t_sm}")
print(f"p_u_t_sci: {p_u_t_sci}")
print(f"p_v_t_sm: {p_v_t_sm}")
print(f"p_v_t_sci: {p_v_t_sci}")
print(f"p_uv_t_sm: {p_uv_t_sm}")
print(f"p_uv_t_sci: {p_uv_t_sci}")

# Plot for convenience:
g = sns.kdeplot(x=uv[0], y=uv[1], data=uv, thresh=0, cut=80, fill=True, cmap="mako", levels=50)
g.set_xlim(-15, 15)
g.set_ylim(-5, 25)
plt.show()

```

#### Expected Output
What was expected:
p_u_t_sm = 0

What was returned:
p_u_t_sm = 0.0802

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

INSTALLED VERSIONS
------------------
Python: 3.7.3.final.0

statsmodels
===========

Installed: 0.12.0 (C:\Users\*redact*\.virtualenvs\*redact*lib\site-packages\statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.19.2 (C:\Users\*redact*\.virtualenvs\*redact*\lib\site-packages\numpy)
scipy: 1.5.2 (C:\Users\*redact*\virtualenvs\*redact*\lib\site-packages\scipy)
pandas: 1.1.3 (C:\Users\*redact*\virtualenvs\*redact*\lib\site-packages\pandas)
    dateutil: 2.8.1 (C:\Users\*redact*\virtualenvs\*redact*\lib\site-packages\dateutil)
patsy: 0.5.1 (C:\Users\*redact*\virtualenvs\*redact*\lib\site-packages\patsy)

Optional Dependencies
=====================

matplotlib: 3.3.2 (C:\Users\*redact*\virtualenvs\*redact*\lib\site-packages\matplotlib)
    backend: TkAgg
cvxopt: Not installed
joblib: Not installed

Developer Tools
================

IPython: 7.18.1 (C:\Users\*redact*\virtualenvs\*redact*\lib\site-packages\IPython)
    jinja2: Not installed
sphinx: Not installed
    pygments: 2.7.1 (C:\Users\*redact*\virtualenvs\*redact*\lib\site-packages\pygments)
pytest: 6.0.2 (C:\Users\*redact*\virtualenvs\*redact*\lib\site-packages\pytest)
virtualenv: Not installed


</details>

user: Afrozodiacc
issue_7105: state: open
title: The algorithm for banddepth is wrong if there are rapid changes between measured points
body: #### Describe the bug

The fast algorithm used in `statsmodels.graphics.functional.banddepth` is from Sun, Genton and Nychka (2012). However, an exact computation of this depth cannot be made using only the ranks, as they propose, because sudden changes of the function between the grid points can change the depth keeping the ranks of one curve the same, as was illustrated in Ouyang, Xu and Zhurkevich (2018).

I came to this bug as I discovered the same bug in my own package (scikit-fda) and wanted to know how other packages had solved the same problem.

M. Ouyang, H. Xu and A. Zhurkevich, "Parallel Computation of Band Depth," 2018 IEEE International Conference on Computational Science and Engineering (CSE), Bucharest, 2018, pp. 85-90, doi: 10.1109/CSE.2018.00018.

Sun, Y., Genton, M.G. and Nychka, D.W. (2012), Exact fast computation of band depth for large functional datasets: How quickly can one million curves be ranked?. Stat, 1: 68-74. doi:10.1002/sta4.8

user: vnmabus
issue_7104: state: open
title: ENH/TST: example code for other packages as used in unit tests
body: Some of our unit tests don't include the code that replicates the models and post-estimation analysis in the reference package.
This means we have to reverse engineer what was used, or figure out which function/procedure was uses with which options.

e.g. I didn't find Stata code for MNLogit in the repo or in issues/PRs

my current example: 
I have logit, mnlogit and ologit with post-estimation in R, Stata and statsmodels. 
I need a place to keep them and find them again later.
And it would be nice to find those for our models that I haven't figured out yet.

I don't think we have docs or there is a list  of statsmodels equivalent to R, Stata functions and their options.
At least not systematically, there is a lot of information and comparisons in notebooks and unit tests.
user: josef-pkt
issue_7103: state: open
title: ENH: support for parametric bootstrap, simulations for nongaussian models
body: related to #106 
get_distribution in GLM still has problems, e.g. #6650, and doesn't exist for other models (outside tsa)

I'm often doing basic Monte Carlo checks of new functions, but I have to write the code for an example each time again.
We should find a way to make this reusable.

related:
- in stats.oneway I added simulation code to check power computation
- we have open PRs for some simulation based p-values for hypothesis tests, (almost) exact Poisson, contingency tables or 2 independent binomial, ...
  Many of those tests have a close model analogue hypothesis test

side benefit: make it easier to check code that doesn't have an equivalent version in another package 
(I did that for score_test under variance/dispersion misspecification and HC score test.)

 
user: josef-pkt
issue_7102: state: open
title: ENH: support Wald tests for variable addition, specification tests
body: many of the specification tests can be run using different test methods, wald, score, cmt, LR, ...

Using wald tests requires adding new variables that correspond to the specification test and create a new model. This is still less fragile for users than recreating a model internally.

In OLS specification tests cmt and wald test in augmented model are often the same. However, in nonlinear models like GLM and discrete, the Wald version reestimates the IRLS weights, or uses full likelihood optimization, and is not identical to the local test at the original model (LM/score tests or cmt)

two options

- recreate model internally and run specification tests
- provide users with helper functions that provide explanatory variables they can use for model augmentation.
  - we could add nice options like orthogonalize them to the included exog. (needs the orthogonal complement helper function)

To the second: we need those extra variables also for score/LM tests, so the question is mainly whether to make them public.

example implementation: I started functions to created added variables for RESET type tests, bin dummies, splines, ...

application example: gof test for ordinal model that uses bins similar to Hosmer-Lemeshow. HL use wald test in their comparison articles. My prototype code is reset-type using score_test.

user: josef-pkt
issue_7101: state: open
title: ENH: binned conditional gof test for continuous distribution
body: specific application of binned gof test #????

In stats gof applications, I had used chisquare test for continuous distributions by binning the continuous distributions and count observations and probabilities in bins.

We can do something similar with resid or predict in models, but as a conditional moment test taking parameter uncertainty into account.
This is the continuous distribution analogue of tests for discrete distributions and count models.

In additive models like gaussian `y = x b + u, u ~ N(0, scale)` we have normality test for residuals.
For other distributions like gamma or lognormal we would need tests for `F(y | x)`

If we split support of y into finite number of bins, then we can compute binned conditional gof test.
I guess we are supposed to bin based on prior info, and not observation specific.

Observation specific bins could depend on the estimated conditional parameters of the distribution, e.g. mean E(y | x) = m(x b) and implied heteroscedastic variance. But theory might require data independent bin boundaries.


user: josef-pkt
issue_7100: state: open
title: ENH: ordinal improvement, similar to larger prob PR(x > y)
body: 
I found by chance an article criticizing AUC, c-statistic, prob stochastically larger for health risk assessment.
Cook Nancy R. 2007. “Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction.” Circulation 115 (7): 928–35. https://doi.org/10.1161/CIRCULATIONAHA.106.672402.
(I only skimmed the beginning)

assuming ordinal data given by multinomial probabilities

What is the probability that x is at least 2 or 3 categories higher than y?

In PR #7094 I add measures and residuals for P(x > y) + 0.5 P(x == y) and P(x > y) - P(x < y)

I guess based on analogue prediction table, we could compute everything from the 2 multinomial/ordinal distributions.
user: josef-pkt
issue_7096: state: open
title: BUG: MNLogit resid_response raises exception
body: MNLogit needs a model specific resid_response.
same structure as for OrderedModel, which doesn't have fittedvalues and resids yet.

usage: for diagnostics #7094 

```
res1_mnl.resid_response[:10]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-244-4eab6bb5ecd0> in <module>
----> 1 res1_mnl.resid_response[:10]

...\statsmodels\statsmodels\base\wrapper.py in __getattribute__(self, attr)
     32             pass
     33 
---> 34         obj = getattr(results, attr)
     35         data = results.model.data
     36         how = self._wrap_attrs.get(attr)

pandas\_libs\properties.pyx in pandas._libs.properties.CachedProperty.__get__()

...\statsmodels\statsmodels\discrete\discrete_model.py in resid_response(self)
   3602         `endog - fittedvalues`
   3603         """
-> 3604         return self.model.endog - self.predict()
   3605 
   3606     @cache_readonly

ValueError: operands could not be broadcast together with shapes (400,) (400,3) 
```
user: josef-pkt
issue_7095: state: open
title: ENH: add Hosmer-lemeshow gof test
body: a specific issue just for Hosmer-Lemeshow, I opened several more general issues


R package `generalhoslem` has three tests compared in the Hosmer et al articles
https://rdrr.io/cran/generalhoslem/man/logitgof.html
However: "Finally, it has been observed that the results from this implementation of the binary and ordinal Hosmer-Lemeshow tests and the Lipsitz test are slightly different from the Stata implementations. It is not not yet clear why this is but is under investigation." (quote from Details docstring)

for OrderedModel I get similar results to theirs but not the same, eg. my chi2=21.34, theirs is chi2=20.95, both at df=17.
similar to difference in my default to Stata: differences in binsizes in groups, e.g. groups are not exactly the same.
I haven't tried Stata for this yet. (example from unit test for OrderedModel using UCLA data)
WIP for #7094 

**update**: I checked with Stata, chi2=21.504 slightly different from mine

The problem is not the groupsizes because all groups have the same size (nobs is multiple of number of groups)

Problem is in ties: there are 180 out of 400 ties in sort_var (average probability with score np.arange(1, k+1).
The split into groups depends on the sort algorithm.
If I use kind="stable" in argsort, then I get the same chi2 as in R generalhoslem. None of the numpy sorting algorithm replicates the Stata results.

user: josef-pkt
issue_7091: state: open
title: Maximum Likelihood Model Robust Standard Errors
body: When fitting a maximum likelihood model, is there a way to show different standard errors or calculate robust standard errors for the summary table? With OLS, you can change `cov_type` in the `fit` method, but there doesn't seem to be something similar for MLE. It seems like all `kwargs` go to the optimizer. Is that correct?

Thank you for the amazing work on this package! It's essential to my research!
user: amichuda
issue_7085: state: open
title: Support 'factr' keyword (mixed linear model)
body: #### Is your feature request related to a problem? Please describe
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

A warning raise when using 'lbfgs' optimizer and 'factr' keyword argument.

**UserWarning: Argument factr not used by MixedLM.fit
    warnings.warn("Argument %s not used by MixedLM.fit" % x)**

This is because the **_allowed_kwargs** in mixed_linear_model.py script does not have 'factr'. But this argument is needed for some optimizer like 'lbfgs'.

#### Describe the solution you'd like
A clear and concise description of what you want to happen.

Add 'factr' into the **_allowed_kwargs** in  mixed_linear_model.py script.

#### Describe alternatives you have considered
A clear and concise description of any alternative solutions or features you have considered.

#### Additional context
Add any other context about the feature request here.
user: NVTACSI
issue_7081: state: open
title: ENH: diagnostic Hosmer-Lemeshow type binning on left out exog
body: (just a thought)

Hosmer-Lemeshow gof test for binary, multinomial or ordinal bins on predicted probabilities. Others bin on covariate patterns.

Problem: what if we don't have enough covariates included in the model to form groups?
e.g. for developing, I'm currently working with an ordinal model that only has a constant, i.e. all predicted probs are the same, residuals can only take on a few unique values.
The same would be if we only have categorical variables that only have a few cells, eg. two sample case.
Aside: models with canonical links fit frequencies perfectly along categorical, dummy partitions. In those cases partial sum of resid is zero. not true for non-canonical links.
This happened to me also in an example with MNLogit but I didn't pay attention at the time.

In analogy to variable addition score tests, we could add binning based on additional, not included explanatory variables.
This would be like a chisquare, Hosmer-lemeshow analogue of a variable addition score_test.


user: josef-pkt
issue_7080: state: open
title: ENH: prediction table based on 3 decision including "inconclusive" for binary and multinomial
body: I had this idea a long time ago, but didn't find an issue.

Prediction tables for binary models like Logit or Multinomial models like MNLogit, OrderedModel pick the choice with the highest probability.
Instead we could include an inconclusive region around prob = 0.5 (in binary case), and compute the prediction table only for observations with max probabilities large enough.

I haven't tried an example yet, and didn't see in the literature (but I never looked at many of the binary choice prediction accuracy measures)

related 
#1577 classification performance measures
also agreement measures, e.g. https://www.statsmodels.org/dev/stats.html#interrater-reliability-and-agreement

I'm currently looking at some residual definitions and gof for Ordinal and similar models. specifically #7076
OrderedModel currently has a method for the standard `argmax` prediction table.

maybe: 
Some small classes/categories might have no predicted observations. Maybe looking at second choices might identify cases that are "almost" choosing one of those categories.
In multinomial models it is more difficult to come up with asymmetric thresholds for a 0-1 choice or classification.

user: josef-pkt
issue_7079: state: open
title: ENH: heteroscedasticity robust effect size estimate
body: related to #6983

mainly parking a reference

Huber/White sandwich meets effect size measures

Vandekar, Simon, Ran Tao, and Jeffrey Blume. 2020. “A Robust Effect Size Index.” Psychometrika 85 (1): 232–46. https://doi.org/10.1007/s11336-020-09698-2.

I didn't look at it. 
However, the issue shows up already in the simpler cases with Welch t-test and anova.
An open issue is whether standard effect size or heteroscedasticity corrected effect sizes should be *reported*.
We need heteroscedasticity adjusted effect sizes that are appropriate for specific tests for power analysis.


(found while looking for estimation of non-centrality parameter for ncF and nc-chi2)



user: josef-pkt
issue_7078: state: open
title: ENH/SUMM: Multinomial and Ordinal model for counts 
body: OrderedModel and (AFAIR) MNLogit are implemented for binary choices, i.e. counts of a choice are 0 or 1, summing to 1.

If there are only categorical variables, then the data is often provided as a cross-tab or contingency table.
MNLogit and OrderedModel require that the data is expanded to observations.

Similarly for binary choices we have GLM-Binomial as count model and Logit/Probit as Models for a binary endog.

In a multinomial count model we need a 2-dim endog that can have multiple or all nonzero values.
Related: models for multivariate endog, especially compositional data.

aside: for brunner-munzel rank comparison, I wrote a separate `ordinal` function for counts https://www.statsmodels.org/dev/generated/statsmodels.stats.nonparametric.rank_compare_2ordinal.html

**weights:**

in analogy to GLM-Binomial.
If we have `freq_weights`, then we can reduce the data to one observation per cell, i.e. unique (endog, exog) patterns.
If we have `var_weights`, then we could use frequencies for unique exog, but that also requires dense 2-dim endog. This would be close to models for compositional endog.


user: josef-pkt
issue_7076: state: open
title: ENH: diagnostics using stochastic equality p(y > x) - p(y < x)
body: related to brunner munzel, stochastically larger effect size issue #6960  and merged #6971

first article defines sign residuals as diagnostic tool for ordinal regression #7069 #7035 

LI, CHUN, and BRYAN E. SHEPHERD. 2012. “A New Residual for Ordinal Outcomes.” Biometrika 99 (2): 473–80.

Shepherd, Bryan E., Chun Li, and Qi Liu. 2016. “Probability-Scale Residuals for Continuous, Discrete, and Censored Data.” Canadian Journal of Statistics 44 (4): 463–79. https://doi.org/10.1002/cjs.11302.

user: josef-pkt
issue_7074: state: open
title: Getting an error with 2D array exog: ValueError: cannot reshape array of size 180 into shape (90,1)
body: Ultimately, I am trying to compute `P(U, V| Z)`. (I can make computations such as `P(U|Z)` work.)

```python
df.info()
```

````markdown
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 90 entries, 0 to 89
Data columns (total 3 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   amygdala     90 non-null     float64
 1   acc          90 non-null     float64
 2   orientation  90 non-null     int64  
dtypes: float64(2), int64(1)
memory usage: 2.2 KB
````

I'm giving the `exog` parameter a 2D array as per the documentation page and I'm getting an error. 

```python
from statsmodels.nonparametric.kernel_density import KDEMultivariateConditional
kde_con = KDEMultivariateConditional(endog = orientation, exog = df.values[:, 0:2], dep_type = 'o', indep_type = 'c', bw='normal_reference')
```

````markdown
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-153-edc5d606bcc1> in <module>
----> 1 kde_con = KDEMultivariateConditional(endog = orientation, exog = df.values[:,0:2], dep_type = 'o', indep_type = 'c', bw='normal_reference')

D:\dev\envs\py37_jup\lib\site-packages\statsmodels\nonparametric\kernel_density.py in __init__(self, endog, exog, dep_type, indep_type, bw, defaults)
    413         self.k_indep = len(self.indep_type)
    414         self.endog = _adjust_shape(endog, self.k_dep)
--> 415         self.exog = _adjust_shape(exog, self.k_indep)
    416         self.nobs, self.k_dep = np.shape(self.endog)
    417         self.data = np.column_stack((self.endog, self.exog))

D:\dev\envs\py37_jup\lib\site-packages\statsmodels\nonparametric\_kernel_base.py in _adjust_shape(dat, k_vars)
    450         nobs = np.shape(dat)[0]  # ndim >1 so many obs many vars
    451 
--> 452     dat = np.reshape(dat, (nobs, k_vars))
    453     return dat
    454 

<__array_function__ internals> in reshape(*args, **kwargs)

D:\dev\envs\py37_jup\lib\site-packages\numpy\core\fromnumeric.py in reshape(a, newshape, order)
    297            [5, 6]])
    298     """
--> 299     return _wrapfunc(a, 'reshape', newshape, order=order)
    300 
    301 

D:\dev\envs\py37_jup\lib\site-packages\numpy\core\fromnumeric.py in _wrapfunc(obj, method, *args, **kwds)
     56 
     57     try:
---> 58         return bound(*args, **kwds)
     59     except TypeError:
     60         # A TypeError occurs if the object does have such a method in its

ValueError: cannot reshape array of size 180 into shape (90,1)
````
user: rmelikov
issue_7072: state: open
title: Possibly incorrect formula used in IRF code in statsmodels/tsa/vector_ar/irf.py line 576
body: According to Lutkepohl Equation 3.7.8, should not both apiece and bpiece be divided by T  in orth_cov function on statsmodels/tsa/vector_ar/irf.py line 576? 
![image](https://user-images.githubusercontent.com/24209053/94771303-8e488700-03d4-11eb-8cdb-7dff8103835c.png)

As per this image, the covariance of the theta values should be the entire sum divided by T and not only bpiece divided by T.

Code used on the path mentioned:

`  def _orth_cov(self):

        # Lutkepohl 3.7.8
        Ik = np.eye(self.neqs)
        PIk = np.kron(self.P.T, Ik)
        H = self.H

        covs = self._empty_covm(self.periods + 1)
        for i in range(self.periods + 1):
            if i == 0:
                apiece = 0
            else:
                Ci = np.dot(PIk, self.G[i-1])
                apiece = Ci @ self.cov_a @ Ci.T

            Cibar = np.dot(np.kron(Ik, self.irfs[i]), H)
            bpiece = (Cibar @ self.cov_sig @ Cibar.T) / self.T

            # Lutkepohl typo, cov_sig correct
            covs[i] = apiece + bpiece

        return covs`

Link: [https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/vector_ar/irf.py](url)
user: lovishchopra
issue_7071: state: open
title: SUMM/REF: create exog for predict at for margins and predictions
body: (I didn't find an issue directly for automatically creating exog)

We need automatically created exog for prediction as tools to illustrate the effect of explanatory variables.
I'm trying to do this currently for OrderedModel in #7035 

- discrete_margins code has `at` code, but it is mostly for aggregation over sample observations (no prediction on grid, AFAICS)
- `sandbox.predict_functional` has Kerby's code for prediction functions including confint
- GAM has plot_partial to plot only the effect of the spline or smoother
- I wrote a simple helper function in #7035 but only in notebook, endog is 2-d multi-column, mainly in support of plotting

- single column focus variable - oneway
  simplest case, all we need is a grid and fixed values of others, e.g. at mean. We can directly use model exog.

- single column focus variables - two or multi-way
  would be nice for interactions, or splitting continuous prediction across a categorical variable (e.g. effect of age by gender)
  needs double grid

- single column focus variables, but aggregated of other variables
  This is what margins use, but only for average prediction at a point (or two for effect of dummy/count, ATE)  
  For functional prediction, i.e. ATE at many different points of a continuous predictor, would need either large memory or loop over grid points

- multi-column terms and predictors
  - available in GAM for specific term not included in formula, we control the smooth part
  - user provided transformation, we don't have enough info and the user can currently not provide that info
  - using formulas, and going through it for prediction
    This is used in `sandbox.predict_functional` but I never went through the details to see how robust this is.
    Main problem is that we need to automatically create something like the user's original exog.
  - categorical: we can get the relevant info from design_info.
    I used it in `t_test_pairwise` to convert from different encodings, but it is currently not used for prediction. (e.g. in GLM. discrete)

asides:

- location might be statsmodels.base similar to get_predictions
- Results class hierarchy: This does not apply to TSA, at least not directly. We need something separate for non-tsa models. I'm not sure how it would work for proper panel data models, MixedLM and GEE (we only predict fixed effects? averaging over unobserved heterogeneity?).
- extensions to multivariate/multinomial endog: not really directly relevant because this is about creating exog_predict and processing the prediction can be treated separately.






  





user: josef-pkt
issue_7070: state: open
title: ENH: wald test for slope params (wald_chi2) with implicit constant
body: Stata automatically switches in summary from LR chi2 to Wald chi2 for the test against the Null model.

Wald test is easy to do if there is an explicit constant.

What is the restriction matrix for the hypothesis that the prediction is constant if there is only an implicit constant in the model?

This should be just a bit of linalg, that orthogonal complement is zero, but we need it in terms of linear combination of params.

maybe we already have it somewhere.




user: josef-pkt
issue_7069: state: open
title: SUMM/ENH: extension to OrderedModel, ordinal cumulative link models
body: I will merge OrderedModel with core functionality in PR #7035 

R package `ordinal` has a good article, based on very brief skimming
https://cran.r-project.org/web/packages/ordinal/vignettes/clm_article.pdf
e.g. discussion for prediction intervals
https://stats.stackexchange.com/questions/261903/plot-predicted-values-from-a-cumulative-link-model-clm-ordinal

current unit test are written using `MASS:polr` results.

possible extensions

- internal
  - make incremental exp parameterization optional
  - analytical derivatives at least with respect the exog or params_exog, e.g. score_factor, should eventually lead to `score_test`
  - subclass standard classes instead of GenericLikelihood, e.g. DiscreteModel

- interpretation
  - plots, regression, effects plot  (idea: add argmax, i.e. predicting the discrete choice as plot element)
  - margins `margeff`

- diagnostics
  - plots: I started with a kind of RESET plot, i.e. observed versus fitted
  - specification and diagnostic tests: no idea yet. We need some kind of link or RESET test
  - GOF with surrogate variable, I briefly skimmed abstracts, and there is an R package (needs references)

- other extensions
  - #6820  I had opened an issue for penalized models, (*)
  - more "built-in" link/distribution functions, R package ordinal has more (*)
  - related models that weaken assumptions, e.g. parallel, maybe
  - parameterized distributions, cdf with shape parameters analogous to parameterized links for Binomial/binary #624

(Greene and .... have a lot more, but that goes much further than just extensions to OrderedModel)

(*) my guess is that the linearity assumption is pretty restrictive, e.g. middle categories might need more nonlinearities. 
Agresti categorical book has a short section on dispersion.




user: josef-pkt
issue_7068: state: open
title: BUG?  Should OrdinalGEE have no constant?
body: I'm getting an overparameterized model when I use a constant in OrdinalGEE with independence and naive cov_type.

If I remove the constant, I get identical results to Ordered model with Logit in #7035 (except for incremental exp parameterization)
user: josef-pkt
issue_7067: state: open
title: ENH: equivalence testing for multinomial sample
body: found by chance (while looking at diagnostic for multinomial models)
https://en.wikipedia.org/wiki/Multinomial_distribution#Equivalence_tests_for_multinomial_distributions

"The equivalence test for Euclidean distance can be found in text book of Wellek (2010).[2] The equivalence test for the total variation distance is developed in Ostrovski (2017).[3] The exact equivalence test for the specific cumulative distance is proposed in Frey (2009).[4]"
user: josef-pkt
issue_7065: state: open
title: MixedLM Predict Clarification
body: As I was going thru the documentation of MixedLM, I noticed the predict method only returns the fixed effects mean structure of the model. I was curious as to why this is the case. Shouldn't it return the fixed and random affects of the fitted model?

When going over the fittedvalues method it returns both the fixed effects and the predicted random effects. I guess i'm unclear as to why the predict method doesn't include predicted random effects.

I'm fairly new to Mixed models and am teaching myself via the Multilevel modeling in Plain Language book.
user: jkelley7
issue_7061: state: open
title: ENH: Non-gaussian Likelihood in UnobservedComponents
body: #### Is your feature request related to a problem? Please describe
I'm interested in using non-Gaussian, specifically Gamma, likelihood in the `UnobservedComponents` class. I'm dealing with some non-negative right skewed data and I suspect being able to use a Gamma likelihood for the `level` and `trend` components may yield a much better fit. My context is very similar to that of [facebook/prophet #1492](https://github.com/facebook/prophet/pull/1492).

#### Describe the solution you'd like
It would be favorable if the `UnobservedComponents` class could be constructed similarly to the `GLM` class with a `family=sm.families.Gamma()` parameter, as well as other distributions.

#### Describe alternatives you have considered
I've considered using other models in statsmodels, specifically `GLM` which does have this capability, but my use case lends itself really well to the Bayesian Structural Time Series model otherwise. Furthermore I'm using the model with `pycausalimpact` which expects (via explicit type check) that the model passed in be an instance of `UnobservedComponents`.

I did find the `get_distribution` method which seems to affect prediction in the `GLM` but couldn't tell where this was getting called. I added it to `UnobservedComponents` but it doesn't seem to impact the results. Am I on the right tack here at all?

Edit: Removed some erroneous information experimenting with `get_distribution` function found in the `GLM` implementation.

#### Additional context
I'm willing to implement this myself, but will probably need at least a little direction / hints on how to do so.

Thanks!
user: codyrioux
issue_7060: state: open
title: ENH: Add support for inequality constraints in LS estimators
body: i want to use LinearRegression by statsmodel . and i need to creat model than sure the params >0 ,but i can't find any args can do that.
user: luoluoluoluojunxi
issue_7051: state: open
title: Singular matrix error on mixedlm fit
body: Hello,

#### Describe the bug

I get `LinAlgError: Singular matrix` when I try to fit a `mixedlm` model to my data, even though there are no colinear components.

My model is 'Y ~ X', with groups defined as a third 'class' variable.
The singular values of `md.exog` are: [65.90832829,  7.53495367].

Also, by removing a sample I can get the model to fit just fine, but I would rather avoid doing so without undertanding why.

I have included a minimal example that reproduces the same error by fitting on random vectors.

My guess is that the problem is might be due to the fact that groups are in my case meaningless, and have too small a variance and that could cause problems with the estimation of random effects: when I fit the model after removing the mentioned sample, I have the group variance estimated at 0.003, with 0.016 std.err.

I could not find anything relating to this problem, so I am looking for a place to start fixing this issue. I might have also missed something in the documentation.

#### Minimal reproduction example


```python
import statsmodels.formula.api as smf
import numpy as np
import pandas as pd

n = 100

data = np.random.randn(n, 2)
df = pd.DataFrame(data, columns=['Y', 'X'])
df['class'] = pd.Series([i % 3 for i in df.index], index=df.index)

u, s, vt = np.linalg.svd(md.exog, 0)
print(s)

md = smf.mixedlm("Y ~ X", df, groups=df['class'])
mdf = md.fit(method=['lbfgs'])
mdf.summary()
```

##### Output

```
[10.88221833  9.92712667]
---------------------------------------------------------------------------
LinAlgError                               Traceback (most recent call last)
<ipython-input-152-dcd95f907c7c> in <module>
     18 
     19 md = smf.mixedlm("Y ~ X", df, groups=df['class'])
---> 20 mdf = md.fit(method=['lbfgs'])
     21 mdf.summary()

~/anaconda3/envs/mfanalysis/lib/python3.8/site-packages/statsmodels/regression/mixed_linear_model.py in fit(self, start_params, reml, niter_sa, do_cg, fe_pen, cov_pen, free, full_output, method, **fit_kwargs)
   2120             # Try optimizing one or more times
   2121             for j in range(len(method)):
-> 2122                 rslt = super(MixedLM, self).fit(start_params=packed,
   2123                                                 skip_hessian=True,
   2124                                                 method=method[j],

~/anaconda3/envs/mfanalysis/lib/python3.8/site-packages/statsmodels/base/model.py in fit(self, start_params, method, maxiter, full_output, disp, fargs, callback, retall, skip_hessian, **kwargs)
    517         warn_convergence = kwargs.pop('warn_convergence', True)
    518         optimizer = Optimizer()
--> 519         xopt, retvals, optim_settings = optimizer._fit(f, score, start_params,
    520                                                        fargs, kwargs,
    521                                                        hessian=hess,

~/anaconda3/envs/mfanalysis/lib/python3.8/site-packages/statsmodels/base/optimizer.py in _fit(self, objective, gradient, start_params, fargs, kwargs, hessian, method, maxiter, full_output, disp, callback, retall)
    213 
    214         func = fit_funcs[method]
--> 215         xopt, retvals = func(objective, gradient, start_params, fargs, kwargs,
    216                             disp=disp, maxiter=maxiter, callback=callback,
    217                             retall=retall, full_output=full_output,

~/anaconda3/envs/mfanalysis/lib/python3.8/site-packages/statsmodels/base/optimizer.py in _fit_lbfgs(f, score, start_params, fargs, kwargs, disp, maxiter, callback, retall, full_output, hess)
    450         func = f
    451 
--> 452     retvals = optimize.fmin_l_bfgs_b(func, start_params, maxiter=maxiter,
    453                                      callback=callback, args=fargs,
    454                                      bounds=bounds, disp=disp,

~/anaconda3/envs/mfanalysis/lib/python3.8/site-packages/scipy/optimize/lbfgsb.py in fmin_l_bfgs_b(func, x0, fprime, args, approx_grad, bounds, m, factr, pgtol, epsilon, iprint, maxfun, maxiter, disp, callback, maxls)
    195             'maxls': maxls}
    196 
--> 197     res = _minimize_lbfgsb(fun, x0, args=args, jac=jac, bounds=bounds,
    198                            **opts)
    199     d = {'grad': res['jac'],

~/anaconda3/envs/mfanalysis/lib/python3.8/site-packages/scipy/optimize/lbfgsb.py in _minimize_lbfgsb(fun, x0, args, jac, bounds, disp, maxcor, ftol, gtol, eps, maxfun, maxiter, iprint, callback, maxls, finite_diff_rel_step, **unknown_options)
    358             # until the completion of the current minimization iteration.
    359             # Overwrite f and g:
--> 360             f, g = func_and_grad(x)
    361         elif task_str.startswith(b'NEW_X'):
    362             # new iteration

~/anaconda3/envs/mfanalysis/lib/python3.8/site-packages/scipy/optimize/_differentiable_functions.py in fun_and_grad(self, x)
    198         if not np.array_equal(x, self.x):
    199             self._update_x_impl(x)
--> 200         self._update_fun()
    201         self._update_grad()
    202         return self.f, self.g

~/anaconda3/envs/mfanalysis/lib/python3.8/site-packages/scipy/optimize/_differentiable_functions.py in _update_fun(self)
    164     def _update_fun(self):
    165         if not self.f_updated:
--> 166             self._update_fun_impl()
    167             self.f_updated = True
    168 

~/anaconda3/envs/mfanalysis/lib/python3.8/site-packages/scipy/optimize/_differentiable_functions.py in update_fun()
     71 
     72         def update_fun():
---> 73             self.f = fun_wrapped(self.x)
     74 
     75         self._update_fun_impl = update_fun

~/anaconda3/envs/mfanalysis/lib/python3.8/site-packages/scipy/optimize/_differentiable_functions.py in fun_wrapped(x)
     68         def fun_wrapped(x):
     69             self.nfev += 1
---> 70             return fun(x, *args)
     71 
     72         def update_fun():

~/anaconda3/envs/mfanalysis/lib/python3.8/site-packages/statsmodels/base/model.py in f(params, *args)
    499 
    500         def f(params, *args):
--> 501             return -self.loglike(params, *args) / nobs
    502 
    503         if method == 'newton':

~/anaconda3/envs/mfanalysis/lib/python3.8/site-packages/statsmodels/regression/mixed_linear_model.py in loglike(self, params, profile_fe)
   1452         # Move to the profile set
   1453         if profile_fe:
-> 1454             fe_params = self.get_fe_params(cov_re, vcomp)
   1455         else:
   1456             fe_params = params.fe_params

~/anaconda3/envs/mfanalysis/lib/python3.8/site-packages/statsmodels/regression/mixed_linear_model.py in get_fe_params(self, cov_re, vcomp)
   1279             cov_re_inv = np.empty((0, 0))
   1280         else:
-> 1281             cov_re_inv = np.linalg.inv(cov_re)
   1282 
   1283         # Cache these quantities that do not change.

<__array_function__ internals> in inv(*args, **kwargs)

~/anaconda3/envs/mfanalysis/lib/python3.8/site-packages/numpy/linalg/linalg.py in inv(a)
    545     signature = 'D->D' if isComplexType(t) else 'd->d'
    546     extobj = get_linalg_error_extobj(_raise_linalgerror_singular)
--> 547     ainv = _umath_linalg.inv(a, signature=signature, extobj=extobj)
    548     return wrap(ainv.astype(result_t, copy=False))
    549 

~/anaconda3/envs/mfanalysis/lib/python3.8/site-packages/numpy/linalg/linalg.py in _raise_linalgerror_singular(err, flag)
     95 
     96 def _raise_linalgerror_singular(err, flag):
---> 97     raise LinAlgError("Singular matrix")
     98 
     99 def _raise_linalgerror_nonposdef(err, flag):

LinAlgError: Singular matrix
```

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

```
INSTALLED VERSIONS
------------------
Python: 3.8.3.final.0
OS: Linux 5.4.0-47-generic #51~18.04.1-Ubuntu SMP Sat Sep 5 14:35:50 UTC 2020 x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

statsmodels
===========

Installed: 0.12.0 (/home/merlin/anaconda3/envs/mfanalysis/lib/python3.8/site-packages/statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.18.5 (/home/merlin/anaconda3/envs/mfanalysis/lib/python3.8/site-packages/numpy)
scipy: 1.5.0 (/home/merlin/anaconda3/envs/mfanalysis/lib/python3.8/site-packages/scipy)
pandas: 1.1.2 (/home/merlin/anaconda3/envs/mfanalysis/lib/python3.8/site-packages/pandas)
    dateutil: 2.8.1 (/home/merlin/anaconda3/envs/mfanalysis/lib/python3.8/site-packages/dateutil)
patsy: 0.5.1 (/home/merlin/anaconda3/envs/mfanalysis/lib/python3.8/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.2.2 (/home/merlin/anaconda3/envs/mfanalysis/lib/python3.8/site-packages/matplotlib)
    backend: module://ipykernel.pylab.backend_inline 
cvxopt: Not installed
joblib: 0.16.0 (/home/merlin/anaconda3/envs/mfanalysis/lib/python3.8/site-packages/joblib)

Developer Tools
================

IPython: 7.16.1 (/home/merlin/anaconda3/envs/mfanalysis/lib/python3.8/site-packages/IPython)
    jinja2: 2.11.2 (/home/merlin/anaconda3/envs/mfanalysis/lib/python3.8/site-packages/jinja2)
sphinx: Not installed
    pygments: 2.6.1 (/home/merlin/anaconda3/envs/mfanalysis/lib/python3.8/site-packages/pygments)
pytest: Not installed
virtualenv: Not installed
```

</details>

user: MerlinDumeur
issue_7049: state: open
title: BUG GenericLikelihoodModel does not propagate kwargs in `__init__`
body: 
I ran into the bug with `hasconst` that was not propagated
see https://github.com/statsmodels/statsmodels/pull/7035#issuecomment-696121750

There might be other keywords that should be propagated, e.g.
I guess that the data handling treatment for extra arrays like offset does not work, which, I guess, breaks conversion to ndarray and inclusion in missing value handling.

currently `GenericLikelihoodModel.__init__` just attaches everything `self.__dict__.update(kwds)`

However, if I just propagate all `kwds` to the super call, then unit tests for miscmodel `TLinearModel` fail.

This needs review and better separation of which keyword should be propagated to super.

partial fix in #7035: I explicitly propagate `hasconst`, but leave other `kwds` as it is currently done.
user: josef-pkt
issue_7045: state: open
title: ENH: SpecificationWarnings raise in pytest
body: I converted UserWarnings to SpecificationWarning, and unit tests are now failing with the warning

the warning filter is set to "error"

I guess the setting is changed in setup.cfg

```
    error::statsmodels.tools.sm_exceptions.HypothesisTestWarning
    error::statsmodels.tools.sm_exceptions.SpecificationWarning
```



user: josef-pkt
issue_7044: state: open
title: Nanosecond Holt: ValueWarning: No frequency information was provided, so inferred frequency S will be used.
body: I'm getting the above error: ValueWarning: No frequency information was provided, so inferred frequency S will be used. 

```
 fit = Holt(s,damped_trend=False, initialization_method="estimated").fit(optimized=True)
```
Where 
```
print(s) 
2020-08-19 17:54:06.810763    18.8
2020-08-19 17:54:07.810763    18.8
2020-08-19 17:54:08.810763    18.9
2020-08-19 17:54:09.810763    18.9
2020-08-19 17:54:10.810763    18.9
2020-08-19 17:54:11.810763    18.9
2020-08-19 17:54:12.810763    18.9
2020-08-19 17:54:13.810763    18.9
2020-08-19 17:54:14.810763    18.9
2020-08-19 17:54:15.810763    18.9
```
and 
```
print(type(s.index[0]))
<class 'pandas._libs.tslibs.timestamps.Timestamp'>
```
However, s index was created from different unix timestamps in a for loop. 
```
unixtime = 1597859689810763000 
time = pd.to_datetime(unixtime, unit='ns', exact=True)

```
 The datetime objects contain ns data/information as evidence by (and the timestamps are different) 
```
print(str(time.value))
1597859689810763000
```

So I believe that the datetime object does in fact have a frequency of 1s, and I don't expect to see this error. 

Thank you kindly for any help!!!!!


user: Anaisdg
issue_7043: state: open
title: REF: design_info in model or in model.data, which to use
body: https://github.com/statsmodels/statsmodels/pull/7035#issuecomment-694970684

needs to be investigated.

removing intercept attaches the reduced design_info to the model but not `model.data`.
`base.Results.predict` uses `model.data.design_info`.
predict is inconsistent because it checks for `hasattr(self.model, 'formula')`

Can we have cases where model.design_info and model.data.design_info differ?

I will be changing `base.Results.predict` to allow both, but there should be a clear rule
```
design_info = (getattr(self.model, "design_info", None) or
               self.model.data.design_info)
```

#7033 related issue for `formula_info` arguments in `Model.__init__`
user: josef-pkt
issue_7041: state: open
title: FAQ-D: formula handling in models that cannot include a constant
body: Some models cannot include a constant among regressors, e.g. PHReg or OrderedModel in the current parameterization.

Patsy always includes an explicit or implicit constant in exog when categorical variables are included.
We have a workaround to drop an explicit constant and get the reduced design_info.

https://github.com/statsmodels/statsmodels/pull/3095
https://github.com/statsmodels/statsmodels/pull/7035#issuecomment-694452070 starting comment for adding support for dropping explicit intercept in a new model

user: josef-pkt
issue_7036: state: open
title: DOC: State space custom models notebook - state innovation timing convention
body: I am interested in learning how to create custom state space models in statsmodels, and was looking at the examples in `statespace_custom_models.ipynb`. The example in model 1 writes the state equation in terms of the state at time `t` being a function of innovations at time `t`. However, in the general representation (presented in the beginning of the same notebook), it is the state in time `t+1` which is a function of innovations at time `t`. Writing the model in example 1 in the general form will introduce additional random terms in the observation equation. I don't see this being accounted for in the class definition. Not sure if it makes a difference in this example but in general the two forms are not the same and most other examples in the documentation use the representation of the general form. Hope someone can clarify this for me. Thanks in advance.
user: BadWizard
issue_7034: state: open
title: TST: check pickling support for newer models
body: My guess is that we are missing unit tests for pickling of models and results from different fit_xxx, with or without formula.

maybe bugs if `model.data.frame` is None
https://github.com/statsmodels/statsmodels/pull/7021#issuecomment-692750497
user: josef-pkt
issue_7033: state: open
title: ENH: add `formula_info` keyword to `__init__` of models
body: I currently need some extra info in the model in how the formula handled the data (specifically endog) 
see last part in comment https://github.com/statsmodels/statsmodels/pull/7021#issuecomment-692267412

related: We also want additional formula_info for other reasons and cases

#2824
#5342
#5469

A formula_info keyword can replace the current explicit keywords `["missing_idx", "design_info", "formula"]`, and replace attributes like `frame` that are attached in from_formula to the created model instance.

In #7021, I still have problems adding a new `frame` keyword, but correctly implementing `formula_info` would allow that we add additional info without problems to the `formula_info` dict.


user: josef-pkt
issue_7024: state: open
title: ENH: callable start_params method
body: Both LikelihoodModel and GenericLikelihoodModel check for a `start_params`  attribute or property.

We could allow this to be a callable method by adding a check `callable(self.start_params)`

I thought there are some issues for this, but don't find any.


in count models
GeneralizedPoisson and NegativeBinomialP have preliminary start_params models inside fit
ZeroInflated calls `_get_start_params` after some preliminary setup
miscmodel.TModel calls `_set_start_params` from `initialize` to set the `start_params` attribute

user: josef-pkt
issue_7023: state: open
title: ENH: distribution based on link classes
body: ordinal models #7021 require a distribution for the latent variable (cdf, ppf, pdf)
Typical cases are the same as links for Binomial.

We could add a wrapper class that converts link classes to distribution classes with just the main methods. 
Link classes already have relevant derivatives.

We have a similar problem in discrete Logit, Probit that use cdf, pdf but we cannot reuse generic features that require link methods.

Maybe we could just alias all methods to make Link classes multilingual.
user: josef-pkt
issue_7022: state: open
title: ENH: column names in base.Model.predict
body: see https://github.com/statsmodels/statsmodels/pull/6982#issuecomment-686519444

ynames in predict, currently missing

predict is inherited from base.Model, that returns conditionally a pandas Series or DataFrame with appropriate index but no column names. We can add the ynames there.

```
            if predict_results.ndim == 1:
                return pd.Series(predict_results, index=exog_index)
            else:
                return pd.DataFrame(predict_results, index=exog_index)
```

Need to check whether we have anattribute for `yname` that is consistent across models and data format. `yname` is used in summary, do it should be always there.
user: josef-pkt
issue_7021: state: open
title: WIP: `formula_info` as model argument, was: Ordinal model rebased
body: rebased version of #6982 , based on PR #2600 and issue #807
user: josef-pkt
issue_7013: state: open
title: BUG/DOC: stats.api add missing and review
body: I forgot to put the new stats functions into `stats.api`

Those need to be added, but there might be also other missing or outdated parts
user: josef-pkt
issue_7011: state: open
title: Implementation of Haldane correction for odds ratio
body: #### Describe the bug

Haldane correction for 2*2 contingency tables adds 0.5 to all cells if any cell is equal to zero. Note that this is what is described in the documentation: 

https://www.statsmodels.org/stable/generated/statsmodels.stats.contingency_tables.Table2x2.html#statsmodels.stats.contingency_tables.Table2x2

but this is not what is implemented. Only the cells with 0 are added 0.5.

A possible reference: https://besjournals.onlinelibrary.wiley.com/doi/pdf/10.1111/j.2041-210x.2012.00250.x

#### Code Sample, a copy-pastable example if possible

```python
import statsmodels.api as sm

# Case reference
c = 10
d = 20
a = 30
b = 0   
x = np.array([[c, d], [a, b]])
table = sm.stats.Table2x2(x)
odds_ratio = table.oddsratio
confint = table.oddsratio_confint()
odds_ratio, confint

# Test 1 - this is what I would have expected
c = 10+0.5
d = 20+0.5
a = 30+0.5
b = 0+0.5   
x = np.array([[c, d], [a, b]])
table = sm.stats.Table2x2(x, shift_zeros=False)
odds_ratio = table.oddsratio
confint = table.oddsratio_confint()
odds_ratio, confint

# Test 2 - this is what is implemented
c = 10
d = 20
a = 30
b = 0+0.5   
x = np.array([[c, d], [a, b]])
table = sm.stats.Table2x2(x, shift_zeros=False)
odds_ratio = table.oddsratio
confint = table.oddsratio_confint()
odds_ratio, confint
```
#### Expected Output

Expected Test 1 for output instead of Test 2
Ref gives: (0.008333333333333333, (0.0004603352790378829, 0.15085622937608822))
Test 1 gives:
(0.008396641343462616, (0.00046584641453094707, 0.15134512932064634))
Test 2 gives:
(0.008333333333333333, (0.0004603352790378829, 0.15085622937608822))

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

INSTALLED VERSIONS
------------------
Python: 3.8.3.final.0

statsmodels
===========

Installed: 0.11.1 (c:\users\admin\appdata\local\programs\python\python38\lib\site-packages\statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.18.4 (c:\users\admin\appdata\local\programs\python\python38\lib\site-packages\numpy)
scipy: 1.4.1 (c:\users\admin\appdata\local\programs\python\python38\lib\site-packages\scipy)
pandas: 1.0.4 (c:\users\admin\appdata\local\programs\python\python38\lib\site-packages\pandas)
    dateutil: 2.8.1 (c:\users\admin\appdata\local\programs\python\python38\lib\site-packages\dateutil)
patsy: 0.5.1 (c:\users\admin\appdata\local\programs\python\python38\lib\site-packages\patsy)

Optional Dependencies
=====================

matplotlib: 3.2.2 (c:\users\admin\appdata\local\programs\python\python38\lib\site-packages\matplotlib)
    backend: module://ipykernel.pylab.backend_inline 
cvxopt: Not installed
joblib: 0.15.1 (c:\users\admin\appdata\local\programs\python\python38\lib\site-packages\joblib)

Developer Tools
================

IPython: 7.15.0 (c:\users\admin\appdata\local\programs\python\python38\lib\site-packages\IPython)
    jinja2: 2.11.2 (c:\users\admin\appdata\local\programs\python\python38\lib\site-packages\jinja2)
sphinx: Not installed
    pygments: 2.6.1 (c:\users\admin\appdata\local\programs\python\python38\lib\site-packages\pygments)
pytest: Not installed
virtualenv: Not installed

</details>

user: PebetoUofC
issue_7010: state: open
title: DOC/FAQ: explain `alternative` option for one-sided confint, relationship to hypothesis test
body: The docstring explanation is not "obvious", e.g. below

I always have to think about what the one-sided alternatives mean, e.g. (in terms of rejection region or) which null hypothesis are consistent with the data.
Explaining this somewhere at least once would be helpful

```
        alternative : str
            The alternative hypothesis, H1, has to be one of the following

               * 'two-sided' : H1: ``prob - value`` not equal to 0.
               * 'larger' :   H1: ``prob - value > 0``
               * 'smaller' :  H1: ``prob - value < 0``

        Returns
        -------
        lower : float or ndarray
            Lower confidence limit. This is -inf for the one-sided alternative
            "smaller".
        upper : float or ndarray
            Upper confidence limit. This is inf for the one-sided alternative
            "larger".
```
user: josef-pkt
issue_7007: state: open
title: ENH: Add the 4Theta forecasting model
body: Hello - I've been working as a contributor on the [sktime](https://github.com/alan-turing-institute/sktime) project along with @mloning, which aims to provide a common sklearn-type interface to various time series analysis tools.

A while ago we implemented a [basic Theta forecasting model](https://github.com/alan-turing-institute/sktime/blob/master/sktime/forecasting/theta.py) (based on the Hyndman SES + trend approach) and are thinking about implementing the [4Theta](https://github.com/Mcompetitions/M4-methods/blob/master/005%20-%20vangspiliot/Method-Description-4Theta.pdf) variant which amongst other things supports non-linear trends and a multiplicative approach to the modelling.

We saw that you have recently implemented the Optimized Theta Method in statsmodels and wondered if you had any similar plans to implement 4Theta? If so there may be an opportunity to collaborate.
user: big-o
issue_7006: state: open
title: ENH: effect size: which one?
body: effect size again
related to #5896

I'm still trying to figure out which and how to add effect sizes

Problem is that we might want to use different versions of effect sizes for reporting and for use with a specific hypothesis tests.

I briefly browsed the t-test sections of Cohen's power book. The main point seems to have been to create effect sizes and effect size conversions for different sampling cases (2-indep, 2-paired, 1 sample) that can be used with the same power tables.
Also, he doesn't really look at the unequal sample size, heteroscedastic 2-sample case. For the equal sample size case, he argues that the standard computation is fine unless variance differences are very large. 
(However we know that t-test assuming equal variances can have very wrong size under heteroscedasticity when combined with unequal sample sizes.)

For reporting and multistudy comparisons, researchers seem to use "standard" effect size measures, even if underlying model assumptions require a different version of the test.
e.g. report equal-var effect size even for Welch t-test or Welch anova. https://github.com/statsmodels/statsmodels/issues/6980#issuecomment-678722099

normalization (something I often forget):
cohen's d is mean diff standardized by within variance, not by variance of the mean diff. Under equal nobs and equal variance, they differ by a factor of 2 (effect size has sqrt(2)). With unequal nobs but equal var, the difference depends on nobs ratio 
`nobs1 * nobs2 / (nobs1 + nobs2)` 

(IIRC I had a mistake in the usage of the power computation when the sqrt(2) or analogue was missing in the two sample case.)

For power computation, what we really need is a nobs normalized non-centrality parameter, which can be directly computed from the test statistic (F, t, or chi2). (which possibly some adjustments)
That's also what we use for confidence intervals via noncentrality parameter inversion.

My current thinking is of adding two separate groups of effect size measures, one standard and one for specific hypothesis tests.
e.g. in the anova case, we can get "standard" effect size using `use_var="equal"` and the adjusted Welch effect size for power computation using `use_var="unequal"` in the effect size function. Similarly, `fstat` to effect size conversion uses the hypothesis test appropriate definition.
For the first group I would be fine now with those weird names that everyone is using, cohens_d, glass_d, .... And use informative or generic names for power "effect size".

(I need to check details again how GPower and R pwr package handle effect size, or if they even handle heteroscedastic cases. GPower doesn't have Welch anova, AFAIR)





user: josef-pkt
issue_7005: state: open
title: GEE with unstructured covariance
body: I would like to be able to run a GEE analysis using an unstructured covariance. In R this is very easy to do, e.g:

```
library(geepack)

geeglm(
  outcome ~ . -cluster_id,
  id = cluster_id,
  corstr = 'unstructured',
  data = some_data
)
```

Letting `K` be the largest number of observations in any cluster, what this does is set up a `K*K` covariance matrix where all of the off-diagonal elements are free parameters. 

This is an extremely common use-case so I would be quite surprised if it is not currently implemented. But from my reading of the documentation (and digging a bit into the source) I can't figure out how to do this.

If it *is* implemented, I'd appreciate guidance on how to make it work. If it *isn't*, I would be very interested in making a PR for it if someone could take a little time to show me the ropes.

user: alejandroschuler
issue_6998: state: open
title: ENH: lsmeans, margins for mixedlm
body: It looks like lsmeans for mixed models only take fixed effects into account

see answer to
https://stats.stackexchange.com/questions/484493/standard-errors-for-ls-means/484608

I guess this can essentially be computed by t_test including standard errors and confint.

I guess `get_prediction` based on only fixed effects could be added in the same way as it is for OLS. 
(prediction intervals might need random effects, i.e. heteroscedasticity)
(what's the analogue in the simpler case of oneway meta-analysis with random effects ?)


user: josef-pkt
issue_6994: state: open
title: ENH: k-sample rank statistic
body: another followup to PR #6971 which adds brunner munzel and extensions for P(x1 > x2) + 0.5 P(x1 = x2) for the two sample case.
see also #6523 for more general cases of using rank transform.

- meta-analysis and stratified case
  The two sample case should extend immediately to the meta-analysis and stratified case. I guess we can just use the functions in meta-analysis directly with the results for prob as effect size measure, var or std is available for it.

- oneway anova
  - we can use ranks directly with Welch anova, (not rank transformation can introduce heteroscedasticity even if underlying random variables are homoscedastic.
  - compute p_i for comparison of distribution of x_i against the distribution with all samples combined. This avoids problems with transitivity (Condorcet cycles), but has interpretation problems if sample sizes are very unequal, e.g. one sample dominates the combined sample.
- pairwise comparison (I haven't looked at details yet)
  - using only information from the pairs might create cycles again
  - can we directly use p_i from F_i versus combined distribution

main problem is that using ranks in joined or combined sample introduces correlation, i.e. R_i and R_j are not independent even if x_i and x_j are


reference in german

Brunner, Edgar, and Ullrich Munzel. 2013. Nichtparametrische Datenanalyse: unverbundene Stichproben. 2., überarb. Aufl. Statistik und ihre Anwendungen. Berlin: Springer Spektrum.

I read several section, book has a good combination of basic explanations and defintions and the technical parts and proofs, for when we need the latter.




user: josef-pkt
issue_6993: state: open
title: ENH: partial P(x1 > x2) effect size for models, AUC
body: (just an idea, I haven't looked for any literature yet, and haven't seen it mentioned in the two sample case)

PR #6971 adds brunner munzel and extensions for P(x1 > x2) + 0.5 P(x1 = x2)

This could be used to get an effect size to compare two models, e.g. as in LR test after imposing a zero or linear constraint, but can be used for non-nested models. where x1 and x1 are predictions/fitted_values for two different models.
Inference for it won't be accurate, because the fittedvalues x1 and x2 are not independent. I guess, it would be better to use a `paired` version for it.

This is similar to AUC for comparing regression models, maybe, kind of. I never looked at that literature. But the AUC is just a linear transformation of the stochastically larger probability.

In a linear model, params is a mean shift. 
In the wald version, keeping other params fixed, the partial effect would always be a mean shift given by the parameter. So the prob should shift in a similar way as params.
In LR style version, we estimate the restricted models and all params can change.

Motivation for this would be to get an additional effect size as in the literature that advertises P(x1 > x2)  under different names, beyond the two sample case.


user: josef-pkt
issue_6987: state: open
title: FAQ/DOC: overview stats methods based on UCLA stats
body: based on https://stats.idre.ucla.edu/other/mult-pkg/whatstat/

the column alignment works in a notebook but not here


Nature of IV | Nature of DV(s)* | Test(s) | How to SAS | How to Stata | How to SPSS | How to R | Python
-----------  | ---------------- | --------| ---------- | ------------ | ----------- | -------- | ------
0 IVs (1 population) | interval & normal | one-sample t-test | SAS | Stata | SPSS | R | scipy, statsmodels
 | ordinal or interval | one-sample median | SAS | Stata | SPSS | R
 | categorical (2 categories) | binomial test | SAS | Stata | SPSS | R
 | categorical | Chi-square goodness-of-fit | SAS | Stata | SPSS | R
1 IV with 2 levels (independent groups) | interval & normal | 2 independent sample t-test | SAS | Stata | SPSS | R
 | ordinal or interval | Wilcoxon-Mann Whitney test | SAS | Stata | SPSS | R
 | categorical | Chi-square test | SAS | Stata | SPSS | R
 | | Fisher’s exact test | SAS | Stata | SPSS | R
1 IV with 2 or more levels (independent groups) | interval & normal | one-way ANOVA | SAS | Stata | SPSS | R
 | ordinal or interval | Kruskal Wallis | SAS | Stata | SPSS | R
 | categorical | Chi-square test | SAS | Stata | SPSS | R
1 IV with 2 levels (dependent/matched groups) | interval & normal | paired t-test | SAS | Stata | SPSS | R
 | ordinal or interval | Wilcoxon signed ranks test | SAS | Stata | SPSS | R
 | categorical | McNemar | SAS | Stata | SPSS | R
1 IV with 2 or more levels (dependent/matched groups) | interval & normal | one-way repeated measures ANOVA | SAS | Stata | SPSS | R
 | ordinal or interval | Friedman test | SAS | Stata | SPSS | R
 | categorical (2 categories) | repeated measures logistic regression | SAS | Stata | SPSS | R
2 or more IVs (independent groups) | interval & normal | factorial ANOVA | SAS | Stata | SPSS | R
 | ordinal or interval | ordered logistic regression | SAS | Stata | SPSS | R
 | categorical (2 categories) | factorial logistic regression | SAS | Stata | SPSS | R
1 interval IV | interval & normal | correlation | SAS | Stata | SPSS | R
 | interval & normal | simple linear regression | SAS | Stata | SPSS | R
 | ordinal or interval | non-parametric correlation | SAS | Stata | SPSS | R
 | categorical | simple logistic regression | SAS | Stata | SPSS | R
1 or more interval IVs and/or 1 or more categorical IVs | interval & normal | multiple regression | SAS | Stata | SPSS | R
 |  | analysis of covariance | SAS | Stata | SPSS | R
 | categorical | multiple logistic regression | SAS | Stata | SPSS | R
 |  | discriminant analysis | SAS | Stata | SPSS | R
1 IV with 2 or more levels (independent groups) | 2+ interval & normal | one-way MANOVA | SAS | Stata | SPSS | R
 |  2+ interval & normal | multivariate multiple linear regression | SAS | Stata | SPSS | R
0 | interval & normal | factor analysis | SAS | Stata | SPSS | R
 | 2 sets of 2+ interval & normal | canonical correlation | SAS | Stata | SPSS | R

user: josef-pkt
issue_6986: state: open
title: ENH: Add ability to compute residuals from rolling regressions
body: Hi @bavardage @alno @timmie @pag @alexbrc 

Is there anyway to get rolling residuals and the rolling intercept from the rolling regression output.

Best,
Andrew


user: andrewcztrack
issue_6984: state: open
title: ENH: collection of transformation scores for use in nonparametric tests
body: SAS has several other `scores` besides midrank and Van der Werden
http://documentation.sas.com/?docsetId=statug&docsetTarget=statug_npar1way_details08.htm&docsetVersion=15.1&locale=en

in PR #6971 I add a helper function for two sample midranks based on scipy's rankdata

We should get also some of the others in a simple form for reuse with standard statistical tests and models.

Based on brief skimming of the SAS procedure, I cannot tell whether they feed the `scores` just to, e.g., a standard anova.




user: josef-pkt
issue_6983: state: open
title: ENH: effect size and power based on test statistic for other distribution
body: I added power and effect size computation based on F-statistic in oneway PR #6789

We should be able to do the same for other distributions, t-test, chisquare and normal.
Confidence intervals can be obtained through the noncentrality parameter inversion.

For t-test and z-test the effect size will be a variant on cohen's d.
chisquare would also produce something like cohen's f_squared

Under correctly specified MLE assumptions, those might be standard (the usual) effect sizes.
With robust cov_type, Welch, ..., we will get an adjusted version. in analogy to Welch oneway anova based on F-statistic.



user: josef-pkt
issue_6980: state: open
title: FAQ: differences to pingouin
body: I'm trying to compare our results with pingouin's, especially effect sizes that pingouin reports by default

- oneway anova eta2: pingouin uses the same definition of ratio of sum of squares for eta2 in plain anova and in welch anova
  our functions based on F-statistic compute a different eta2 for Welch anova, and another version (nobs normalization) for the function intended for power computation


user: josef-pkt
issue_6979: state: open
title: Tolerance and prediction intervals for regression results
body: Dear All,

have you planned adding tolerance and prediction intervals? 

e.g. https://cran.r-project.org/web/packages/tolerance/tolerance.pdf
If someone else feels there is a need for that, I am willing to share my time to implement such intervals for statsmodels OLS.
user: MoritzImendoerffer
issue_6978: state: open
title: ENH: summary method for hypothesis test and similar
body: Pingouin got ahead of us in terms for nice summary prints/tables for hypothesis tests and similar

Holder and HolderTuple have a plain dict-like default repr that just dumps all attributes, which is convenient but not pretty, and doesn't add results that need a method call like confints.
In some cases, we return now a pandas DataFrame when the results are in table form.

We can use summary `params_table` as in ContrastResults for t_test, example below

extensions that we need
- add optionally more columns, e.g. pingouin has effect size, in some cases we want multiplicity corrected p-values as additional column
- make it easy to combine params table with test notes. params_table is relatively easy to use by itself, eg. t_test results, but I'm not sure how to add text


example: rank_compare results in PR #6971 

We need notes or change in column headers to indicate that `t` and `P>|t|` are for hypothesis `p = 0.5`.
`prob(x1>x2)` is not really correct in discrete case, where it should be the longer `prob(x1>x2) + 0.5 prob(x1=x2)`

```
print(res.summary())
                 Probability sample 1 is stochastically larger                 
===============================================================================
                  coef    std err          t      P>|t|      [0.025      0.975]
-------------------------------------------------------------------------------
prob(x1>x2)     0.4558      0.038     -1.176      0.121       0.382       0.530
===============================================================================
res.
(res.prob1 - 0.5) / np.sqrt(res.var_prob)
-1.1757561456581602
```

user: josef-pkt
issue_6960: state: open
title: ENH: add stochastic superiority effect size
body: a version is also known as CL common language effect size

"
The purpose
of this article is to introduce yet another effect size statistic
but one that is so readily understood by nonstatisticians that we
have chosen to call it the common language effect size indicator.
"

silly name, I advocate adding `bese` "best effect size ever", where the author needs to specify a quasilinear utility function where the linear part is the money that they bet that the effectsize is correct and useful. :)

note, I think this is also Brunner-Munzel statistic or a version or function of it under nonparametric assumptions. #3575

(aside: stochastic dominance and quantitative measures for it are missing)

(~I only skimmed small parts of the following~)

McGraw, Kenneth O., and S. P. Wong. 1992. “A Common Language Effect Size Statistic.” Psychological Bulletin 111 (2): 361–65. https://doi.org/10.1037/0033-2909.111.2.361.

Ruscio, John, and Tara Mullen. 2012. “Confidence Intervals for the Probability of Superiority Effect Size Measure and the Area Under a Receiver Operating Characteristic Curve.” Multivariate Behavioral Research 47 (2): 201–23. https://doi.org/10.1080/00273171.2012.658329.

Vargha, András, and Harold D. Delaney. 2000. “A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong.” Journal of Educational and Behavioral Statistics 25 (2): 101–32. https://doi.org/10.3102/10769986025002101.

python implementation https://github.com/ajschumacher/cles 

related to more general issue
#6523 rank based methods
#3575 Brunner-Munzel test
maybe related #799 rank based multiple comparisons

user: josef-pkt
issue_6947: state: open
title: ENH: One-sided HP filter
body: #### Is your feature request related to a problem? Please describe
the current `statsmodels.tsa.filters.hp_filter.hpfilter` cannot be used realtime, as it is two-sided 

#### Describe the solution you'd like
similar to `statsmodels.tsa.filters.hp_filter.hpfilter` , but one-sided (for realtime use)

#### Describe alternatives you have considered
Considered kalman filters

#### Additional context
Add any other context about the feature request here.
user: vedal
issue_6935: state: open
title: ENH: add functionality Improve the stability for regression models with multicollinearity  to statsmodels.regression (…
body: …WLS, OLS, GLM or any other model that uses WLS) Pseudo Inverse (method=‘pinv’) may product unstable results coefficients (params) values that are extremely larg. The current QR decomposition (mothod=‘qr’) decomposition does not handle the multicollinearity in practice, because of numerical errors in the process of calculating the eigenvalues and the rank of the R matrix.

- [ ] closes #xxxx
- [x] tests added / passed. 
- [ ] code/documentation is well formatted.  
- [ ] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

<details>


**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in master and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/master -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>

user: sergiolevin
issue_6913: state: open
title: FAQ: Helpful bug reporting and quick solutions
body: The quality of a bug report directly affects the time that the bug is expected to remain open. 

An ideal bug report should have two elements:

* **Important:** An initial description of the bug
* **Essential:** a copy-and-pastable demonstration, that is as small as possible, that reproduces the bug. It should:
  * **Not** depend on external data. While it may be possible to include an external data file (e.g., a csv), it is much simpler for developers to copy and paste code to reproduce the bug without first downloading data. 
  * Use simulated data that resembles the original dataset in shape, dtype, and in any other dimensions.

Three examples that range from ideal to difficult to use:

* #6887 is an excellent example of a bug report
* #6778 is a passable example. While the code at the top is enough to reproduce the issue, it cannot be copied, pasted and executed without editing.
* #6203 is a difficult to use bug report.  While it contains a skeleton of an example, it is missing key elements like the definition of the data.

Please follow this guidance when filing a report.


user: bashtage
issue_6887: state: open
title: FAQ: get_prediction fails with regression models after calling remove_data
body: **Edit**
`get_prediction` computes inferential statistics and will only work after `remove_data` if inferential attributes, specifically `scale` in this case, have been cached before data is removed. If cached attributes are accessed, e.g. by summary(), then they will in the cache and still be available after `remove_data`
see comment https://github.com/statsmodels/statsmodels/issues/6887#issuecomment-883592231 below.

---

#### Describe the bug

 the conditional expression at L160 in v0.11 regression._prediction.py will cause an AttributeError with a simple model (exog dim=1) if remove_data has been called on the underlying model. 

#### Code Sample, a copy-pastable example if possible

```python
import numpy as np
import statsmodels.api as sm

# toy data
endog = [i + np.random.normal(scale=0.1) for i in range(100)]
exog = [i for i in range(100)]

# fit 
model = sm.OLS(endog, exog, weights=[1 for _ in range(100)]).fit()
model.summary() # R^2 ~= 1
# works fine
model.get_prediction(1).predicted_mean
model.get_prediction([1]).predicted_mean
model.get_prediction([[1]]).predicted_mean

model.remove_data()
# works fine
model.get_prediction(1).predicted_mean
model.get_prediction([[1]]).predicted_mean

# AttributeError
model.get_prediction([1]).predicted_mean
```
<details>

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `master`. If your problem has been fixed in an unreleased version, you might be able to use `master` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the master branch of this repository? It helps the limited resources if we know problems exist in the current master so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

AttributeError: 'NoneType' object has no attribute 'ndim'

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]
INSTALLED VERSIONS
------------------
Python: 3.6.10.final.0
OS: Linux 3.10.0-862.14.4.el7.YAHOO.20180927.19.x86_64 #1 SMP Thu Sep 27 18:19:45 UTC 2018 x86_64
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8

statsmodels
===========

Installed: 0.11.0 (/grid/6/tmp/yarn-local/usercache/rrymer/appcache/application_1594696455454_937110/container_e24_1594696455454_937110_01_000002/python36-dependencies.zip/statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.16.4 (/grid/6/tmp/yarn-local/usercache/rrymer/appcache/application_1594696455454_937110/container_e24_1594696455454_937110_01_000002/python36-dependencies.zip/numpy)
scipy: 1.5.0 (/opt/python/lib/python3.6/site-packages/scipy)
pandas: 1.0.0 (/grid/6/tmp/yarn-local/usercache/rrymer/appcache/application_1594696455454_937110/container_e24_1594696455454_937110_01_000002/python36-dependencies.zip/pandas)
    dateutil: 2.8.1 (/grid/6/tmp/yarn-local/usercache/rrymer/appcache/application_1594696455454_937110/container_e24_1594696455454_937110_01_000002/jup3.zip/dateutil)
patsy: 0.5.1 (/grid/6/tmp/yarn-local/usercache/rrymer/appcache/application_1594696455454_937110/container_e24_1594696455454_937110_01_000002/python36-dependencies.zip/patsy)

Optional Dependencies
=====================

matplotlib: 3.2.2 (/opt/python/lib/python3.6/site-packages/matplotlib)
    backend: module://ipykernel.pylab.backend_inline 
cvxopt: Not installed
joblib: 0.15.1 (/opt/python/lib/python3.6/site-packages/joblib)

Developer Tools
================

IPython: 6.5.0 (/grid/6/tmp/yarn-local/usercache/rrymer/appcache/application_1594696455454_937110/container_e24_1594696455454_937110_01_000002/jup3.zip/IPython)

</details>

user: rrymer
issue_6881: state: open
title: CLN: remove base class of new ETS implementation
body: The base class of the `ETSModel` class introduced in #6699 is in large parts only a copy of `statespace.mlemodel.MLEModel`, omitting the methods that are unnecessary for `ETSModel`.

It would probably be better to create a common base class for `ETSModel` and `MLEModel`. What would be the best way to go about this?
user: s-scherrer
issue_6878: state: open
title: Finite Population Correction for `proportion_confint`
body: #### Is your feature request related to a problem? Please describe
Confidence intervals are often not precise if population size is finite/small using `proportion_confint`. It would be desirable able to correct for finite population.

#### Describe the solution you'd like
It would be good if `statsmodels.stats.proportion.proportion_confint` had an option to enable FPC (finite population correction)

#### Describe alternatives you have considered
Other similar R packages have this option (https://www.rdocumentation.org/packages/asbio/versions/1.6-3/topics/ci.p)

#### Additional context
E.g.: https://online.stat.psu.edu/stat415_sp20/lesson/6/6.3 for FPC
user: aCampello
issue_6875: state: open
title: errors with mixedlm and multiple groups
body: I'm trying to run a MixedLM and I'd like to use multiple groups, in my datasets are nested data, multiple levels of a hierarchy. I already found that, for a single group, this

`smf.mixedlm("y ~ 0 + level_1", df, groups=['level_2']).fit()`

raises a singluar matrix error, while this

`smf.mixedlm("y ~ 0 + level_1", df, groups='level_2').fit()`

doesn't. Anyhow, when passing two groups (those are levels of a hierarchy), I get

```
mmd = smf.mixedlm("y ~ 0 + level_1", df, groups=['level_2','level_3'])
ValueError                                Traceback (most recent call last)
<ipython-input-292-87d963e12531> in <module>
----> 1 mmd = smf.mixedlm("y ~ 0 + level_1", df, groups=['level_2','level_3'])

~/anaconda3/envs/schist/lib/python3.8/site-packages/statsmodels/regression/mixed_linear_model.py in from_formula(cls, formula, data, re_formula, vc_formula, subset, use_sparse, missing, *args, **kwargs)
   1022         kwargs["exog_vc"] = exog_vc
   1023         kwargs["groups"] = groups
-> 1024         mod = super(MixedLM, cls).from_formula(
   1025             formula, data, *args, **kwargs)
   1026 

~/anaconda3/envs/schist/lib/python3.8/site-packages/statsmodels/base/model.py in from_formula(cls, formula, data, subset, drop_cols, *args, **kwargs)
    192                        'formula': formula,  # attach formula for unpckling
    193                        'design_info': design_info})
--> 194         mod = cls(endog, exog, *args, **kwargs)
    195         mod.formula = formula
    196 

~/anaconda3/envs/schist/lib/python3.8/site-packages/statsmodels/regression/mixed_linear_model.py in __init__(self, endog, exog, groups, exog_re, exog_vc, use_sqrt, missing, **kwargs)
    714         # Calling super creates self.endog, etc. as ndarrays and the
    715         # original exog, endog, etc. are self.data.endog, etc.
--> 716         super(MixedLM, self).__init__(endog, exog, groups=groups,
    717                                       exog_re=exog_re, missing=missing,
    718                                       **kwargs)

~/anaconda3/envs/schist/lib/python3.8/site-packages/statsmodels/base/model.py in __init__(self, endog, exog, **kwargs)
    234 
    235     def __init__(self, endog, exog=None, **kwargs):
--> 236         super(LikelihoodModel, self).__init__(endog, exog, **kwargs)
    237         self.initialize()
    238 

~/anaconda3/envs/schist/lib/python3.8/site-packages/statsmodels/base/model.py in __init__(self, endog, exog, **kwargs)
     74         missing = kwargs.pop('missing', 'none')
     75         hasconst = kwargs.pop('hasconst', None)
---> 76         self.data = self._handle_data(endog, exog, missing, hasconst,
     77                                       **kwargs)
     78         self.k_constant = self.data.k_constant

~/anaconda3/envs/schist/lib/python3.8/site-packages/statsmodels/base/model.py in _handle_data(self, endog, exog, missing, hasconst, **kwargs)
     98 
     99     def _handle_data(self, endog, exog, missing, hasconst, **kwargs):
--> 100         data = handle_data(endog, exog, missing, hasconst, **kwargs)
    101         # kwargs arrays could have changed, easier to just attach here
    102         for key in kwargs:

~/anaconda3/envs/schist/lib/python3.8/site-packages/statsmodels/base/data.py in handle_data(endog, exog, missing, hasconst, **kwargs)
    669 
    670     klass = handle_data_class_factory(endog, exog)
--> 671     return klass(endog, exog=exog, missing=missing, hasconst=hasconst,
    672                  **kwargs)

~/anaconda3/envs/schist/lib/python3.8/site-packages/statsmodels/base/data.py in __init__(self, endog, exog, missing, hasconst, **kwargs)
     69             self.formula = kwargs.pop('formula')
     70         if missing != 'none':
---> 71             arrays, nan_idx = self.handle_missing(endog, exog, missing,
     72                                                   **kwargs)
     73             self.missing_row_idx = nan_idx

~/anaconda3/envs/schist/lib/python3.8/site-packages/statsmodels/base/data.py in handle_missing(cls, endog, exog, missing, **kwargs)
    263 
    264         else:
--> 265             nan_mask = _nan_rows(*combined)
    266             if combined_2d:
    267                 nan_mask = _nan_rows(*(nan_mask[:, None],) + combined_2d)

~/anaconda3/envs/schist/lib/python3.8/site-packages/statsmodels/base/data.py in _nan_rows(*arrs)
     47         return np.logical_or(_asarray_2d_null_rows(x),
     48                              (x_is_boolean_array | _asarray_2d_null_rows(y)))
---> 49     return reduce(_nan_row_maybe_two_inputs, arrs).squeeze()
     50 
     51 

~/anaconda3/envs/schist/lib/python3.8/site-packages/statsmodels/base/data.py in _nan_row_maybe_two_inputs(x, y)
     46         x_is_boolean_array = hasattr(x, 'dtype') and x.dtype == bool and x
     47         return np.logical_or(_asarray_2d_null_rows(x),
---> 48                              (x_is_boolean_array | _asarray_2d_null_rows(y)))
     49     return reduce(_nan_row_maybe_two_inputs, arrs).squeeze()
     50 

ValueError: operands could not be broadcast together with shapes (1523,1) (2,1) 
```

and if I specify the variance:

```
mmd = smf.mixedlm("y ~ 0 + level_1", df, groups=['level_2','level_3'], vc_formula={"level_2": "0+level_2", "level_3": "0+level_4"},)

ValueError                                Traceback (most recent call last)
<ipython-input-293-232c413f8491> in <module>
----> 1 mmd = smf.mixedlm("y ~ 0 + level_1", df, groups=['level_2','level_3'], vc_formula={"level_2": "0+level_2", "level_3": "0+level_4"},)

~/anaconda3/envs/schist/lib/python3.8/site-packages/statsmodels/regression/mixed_linear_model.py in from_formula(cls, formula, data, re_formula, vc_formula, subset, use_sparse, missing, *args, **kwargs)
    993             vc_colnames = []
    994             vc_names = []
--> 995             gb = data.groupby(groups)
    996             kylist = sorted(gb.groups.keys())
    997             vcf = sorted(vc_formula.keys())

~/anaconda3/envs/schist/lib/python3.8/site-packages/pandas/core/frame.py in groupby(self, by, axis, level, as_index, sort, group_keys, squeeze, observed)
   5799         axis = self._get_axis_number(axis)
   5800 
-> 5801         return groupby_generic.DataFrameGroupBy(
   5802             obj=self,
   5803             keys=by,

~/anaconda3/envs/schist/lib/python3.8/site-packages/pandas/core/groupby/groupby.py in __init__(self, obj, keys, axis, level, grouper, exclusions, selection, as_index, sort, group_keys, squeeze, observed, mutated)
    401             from pandas.core.groupby.grouper import get_grouper
    402 
--> 403             grouper, exclusions, obj = get_grouper(
    404                 obj,
    405                 keys,

~/anaconda3/envs/schist/lib/python3.8/site-packages/pandas/core/groupby/grouper.py in get_grouper(obj, key, axis, level, sort, observed, mutated, validate)
    615         # allow us to passing the actual Grouping as the gpr
    616         ping = (
--> 617             Grouping(
    618                 group_axis,
    619                 gpr,

~/anaconda3/envs/schist/lib/python3.8/site-packages/pandas/core/groupby/grouper.py in __init__(self, index, grouper, obj, name, level, sort, observed, in_axis)
    254         self.name = name
    255         self.level = level
--> 256         self.grouper = _convert_grouper(index, grouper)
    257         self.all_grouper = None
    258         self.index = index

~/anaconda3/envs/schist/lib/python3.8/site-packages/pandas/core/groupby/grouper.py in _convert_grouper(axis, grouper)
    655     elif isinstance(grouper, (list, Series, Index, np.ndarray)):
    656         if len(grouper) != len(axis):
--> 657             raise ValueError("Grouper and axis must be same length")
    658         return grouper
    659     else:

ValueError: Grouper and axis must be same length
```



user: dawe
issue_6873: state: open
title: REF/ENH: meta-analysis followup
body: 
- wiring up information about samples : currently we don''t have enough information about individual samples in the results class.

- return type for confidence intervals, tests use now HolderTuple, but confint functions still return tuples.
user: josef-pkt
issue_6861: state: open
title: AR-like model to fit to multiple independent parallel time series ?
body: @ChadFulton 

I have been stuck for days because I cannot find a way to fit a traditional regression model such as ARIMA giving multiple time series as input.

Please let me introduce briefly my problem and what I want to achieve.
I have got thousand trajectories positions of different vehicles (xy coordinates for each position). Let's say each sample (trajectory) is composed of 10 positions (all object trajectories are not necessarily of the same length). It means I have got 10*N different time series (N is the total number of samples). I want to fit a model with all samples for x coordinates and then predict the future position of any new trajectory that I give in input. Then I plan to do the same with another model for y coordinates. I do not say the method will work but I need to implement it to compare it with others (Neural networks, ...)
The hypothesis: a number of time series can be modeled with a single ARIMAX (or other) model (i.e. the same parameters work for all the time series). What is wanted: to fit them all simultaneously.

Can someone help me please? Thank you in advance for your support!
user: noname34
issue_6858: state: open
title: ENH: Replace patsy with formulaic
body: Formulaic is a rewrite of patsy that has been designed for more extensibility.  It already has features that patsy doesn't such as sparse array support. Should consider moving to this when it is a bit more mature.

https://github.com/matthewwardrop/formulaic
user: bashtage
issue_6840: state: open
title: FAQ: Installation for contributing
body: On [this page](https://www.statsmodels.org/dev/dev/git_notes.html), it says to fork, clone and get to hacking away. On [this page](https://www.statsmodels.org/dev/install.html#install) (in "Obtaining the source") it says to clone directly from the unforked repo to be able to run tests. How do run tests on your own fork if the unforked repo is the one that is set up for testing? 

What am I missing here? Also, is this the right way to ask questions?
user: rluedde
issue_6822: state: open
title: ENH: power for TOST using bivariate distribution
body: see also #4212 that mentions PowerTOST package

R package PowerTOST has an "exact" distribution for power (under normal sample) based on bivariate non-central t-distribution.
This goes back to Owen which used it also for two threshold control charts or similar.

In the PowerTOST example, using exact instead of usual univariate nct make only a difference for sample sizes below 15 or so.
So, not really important (Scipy is still missing some of the Owen's XXX special function for this and similar, AFAIR)

But, I haven't seen that before and I don't know whether using bivariate distribution is more important in other cases.
AFAIR, no other package uses this.

```
> power.TOST(CV = 0.25, n = 10, method = "nct")
[1] 0.1496057
> power.TOST(CV = 0.25, n = 10, method = "exact")
[1] 0.2149559

> power.TOST(CV = 0.25, n = 15, method = "nct")
Unbalanced design. n(i)=8/7 assumed.
[1] 0.4561887
> power.TOST(CV = 0.25, n = 15, method = "exact")
Unbalanced design. n(i)=8/7 assumed.
[1] 0.4581458
```
user: josef-pkt
issue_6820: state: open
title: ENH: ordered Logit with penalization
body: see also #807 

just parking two references here (I didn't look closely at them)

Zahid, Faisal M., and Shahla Ramzan. 2012. “Ordinal Ridge Regression with Categorical Predictors.” Journal of Applied Statistics 39 (1): 161–71. https://doi.org/10.1080/02664763.2011.578622.

Zahid, Faisal Maqbool, Shahla Ramzan, and Christian Heumann. 2015. “Regularized Proportional Odds Models.” Journal of Statistical Computation and Simulation 85 (2): 251–68. https://doi.org/10.1080/00949655.2013.814133.

user: josef-pkt
issue_6812: state: open
title: ENH: add Chandrasekhar recursions estimator for SARMA models
body: As noted on the [mailing list](https://groups.google.com/forum/#!topic/pystatsmodels/1Oq1jYbrHgY), recent versions of Gretl estimate SARMA models very quickly, apparently using the methods described in Melard (1984) "Algorithm AS 197: A Fast Algorithm for the Exact Likelihood of Autoregressive-Moving Average Models".

This algorithm is essentially an application of using the Chandrasekhar recursions rather than the Kalman filtering recursions, as we added in #6411, but it is specialized to the ARMA case and apparently is very fast.

I think it should be added along as an estimator to the new `ARIMA` model in the same way as was done for `innovations_mle`. So it would live at e.g. `statsmodels/tsa/arima/estimators/chandrasekhar`.


user: ChadFulton
issue_6803: state: open
title: TST/ENH: support for formulas in zero inflated models
body: question on mailing list

It looks like there are no unit tests and examples for from_formula for ZIP or ZINB.

Usage should be
`ZeroInflatedPoisson.from_formula("count ~ bin + animal", my_dataframe)`

with keyword options for
`exog_infl=None, offset=None, inflation='logit', exposure=None`

In my main notebook I used only numpy directly
https://gist.github.com/josef-pkt/c932904296270d75366a24ee92a4eb2f 

ZeroInflated models are also not in formula.api

user: josef-pkt
issue_6802: state: open
title: How to model a mixed effects model with three (or more) levels of nested random effects
body: Similarly to #3500, I would like to know if it is possible to fit a mixed random effects
model with three random effects when the factors are nested within each other? The extension from the 2 to higher levels of nesting is not obvious to me from the documentation/examples

E.g. in R this would be achieved as follows:
`lmer(y ~ (1|group1/group2/group3), data=df)`
user: hsuominen
issue_6799: state: open
title: Tukey HSD posthoc test for group interactions in two-way ANOVA
body: Hi,

I would like to know how to perform a pairwise comparison for group interactions using Tukey HSD. I tried to use  `pairwise_tukeyhsd` from `statsmodels.stats.multicomp`, but it does not support pairwise comparison for group interactions.

In R it can be done like this (see https://rpubs.com/tmcurley/twowayanova)

`TukeyHSD(len.aov, which = "supp:dose")`

Let me know if there is a solution to achieve this. If not, it will be a good addition to the statsmodels package.


user: reneshbedre
issue_6798: state: open
title: ENH: variance estimation and tests based on pairwise absolute differences
body: I just ran into something related to this (*)

The sum of all pairwise absolute differences can be used as estimate of variance or dispersion.
It's also referred to as Gini mean difference (related to Gini coefficient)

I guess related to #3380 which uses mostly sequential differences for estimating variance/scale

Carina Gerstenberger, Daniel Vogel & Martin Wendler (2019): Tests for Scale
Changes Based on Pairwise Differences, Journal of the American Statistical Association, DOI:
10.1080/01621459.2019.1629938

cited by 2 arxivx papers that might be interesting


(*)
Serfling, Robert Joseph. 2008. Approximation Theorems of Mathematical Statistics. S.l.: John Wiley & Sons.
in chapter 8, p 263 Example A shows a shortcut how to compute sum of pairwise absolute differences
“L-Estimates.” 2008. In Approximation Theorems of Mathematical Statistics, 262–91. John Wiley & Sons, Ltd. https://doi.org/10.1002/9780470316481.ch8.

```
idx = np.triu_indices(nobs, k=1)
np.sum(np.abs(x[idx[0]] - x[idx[1]])) * 2 / (nobs * (nobs - 1))
1.67007648797246

xs = np.sort(x)
ii = np.arange(1, nobs+1)
np.sum((2* ii - nobs - 1) * xs) * 2 / (nobs * (nobs - 1))
1.67007648797246
```

see also computation of L-moments
https://en.wikipedia.org/wiki/L-moment#Sample_L-moments

aside:
Rousseeuw and Croux use median or quantile which need a different computational speedup
https://en.wikipedia.org/wiki/Robust_measures_of_scale#Absolute_pairwise_differences

I don't find an issue or PR for their S_n and Q_n.
I looked at it in the context of robust scale estimates, but it looked too expensive to compute or too much work to implement efficiently for the added benefits.


user: josef-pkt
issue_6794: state: open
title: ENH: interpret importance weights as M-estimator
body: see also #2848 (what kind of weights)

just some thoughts about adding importance weights, and how we define inference/results for it.
at the moment: trimmed anova and t-test #6789 . We could interpret trimming as {0, 1} importance weights.

Do we treat importance weights as fixed?

- RLM: weights are endogenous and implied by the norm/objective function.
  inference is based on M-estimator interpretation
- ELTS robust estimation, dropping observations is endogenous depends on (exog ?) distance from center measure, i.e. based on a test statistic for Mahalanobis distance. Inference ? (I don't remember). AFAIR it's based on a trimmed/truncated/censored distribution.
- Trimmed mean (in anova, t-test): trimming depends on order statistic, but predefined fraction of observations, Inference based on specific variance (var of winsorized sample)
- outlier down weighting in GLM and similar. weights are based on a distance measure for exog.
  I don't remember the names right now.
- user specified, pre-defined names, taken as exogenous, i.e. fixed for inference.
  This should be a standard M-estimator, and AFAIU Stata just switches to a robust HC cov_type.

Importance weights and probability weights are still missing in GLM mainly because it is not clear (to me) what it should imply for inference.
We can compare different cases and check with unit test against e.g. Stata.








user: josef-pkt
issue_6792: state: open
title: BUG/REF: avoid returning scipy named tuples
body: I just saw that linear_harvey_collier returns a named tuple form scipy.stats.

I think we should not return scipy results directly, i.e. we either return a plain tuple (numbers) or we return or own results or holder instance.


user: josef-pkt
issue_6790: state: open
title: WIP/ENH: Stats robust descriptive rebased
body: this has only the first commit of #6526 and originally #6459

This is mainly storing the commit. I haven't checked what we still need here


user: josef-pkt
issue_6784: state: open
title: GLM influence goodness-of-fit sensitivity
body: GLMInfluence does not yet support goodness of fit sensitivity around the deviance and chi-sqaured goodness of fit statistics (as stated in the documentation) 

It should be fairly straightforward to include these, but I cannot see an enhancement story for it (perhaps I'm wrong?) 
user: telferm57
issue_6779: state: open
title: Confidence Interval in Maximum Likelihood Estimation
body: We are considering to use statsmodels for some statistical analysis - but not sure whether we can get confidence interval as part of output from Maximum Likelihood Estimation in statsmodels? It's not described in the documentation - if not possible as part of output - any suggestions how to easily calculate the confidence interval (currently interested in 68% and 95% confidence intervals). Thanks!

https://www.statsmodels.org/stable/examples/notebooks/generated/generic_mle.html
user: geowat
labels: comp-extremes: extreme value theory
issue_6764: state: open
title: BUG/ENH: generalized poisson bad start_params
body: example from #6763 
estimating GeneralizedPoisson without setting start_params has nan loglike and optimization fails.

Optimization is successful using positive or zero dispersion parameter parameter as start_params

e.g.

```
model = sm.GeneralizedPoisson(
    endog=endog, exog=sm.add_constant(df_exog))
results = model.fit(start_params=np.concatenate((results_poi.params, [0.01])), method='bfgs', maxiter=2000, disp=True)
Optimization terminated successfully.
         Current function value: 0.598558
         Iterations: 27
         Function evaluations: 30
         Gradient evaluations: 30
```

or 
```
results = model.fit(start_params=np.zeros(4), method='bfgs', maxiter=2000, disp=True)
Optimization terminated successfully.
         Current function value: 0.598558
         Iterations: 34
         Function evaluations: 37
         Gradient evaluations: 37
```

My guess is that we need to check that the default starting value for the dispersion parameter is valied (non-nan loglike) or clip it to something positive.
user: josef-pkt
issue_6759: state: open
title: ENH: GLM - DOE, power, effect size, noncentrality, sample size
body: (just some thoughts)

continuation of
#6558 noncentrality and power for OLS
My current version uses cov_params and applies to generic wald test in models

The wald test is defined in terms of linear/affine restrictions on parameters.

- For design of prototype dataset, we might want to start with expected proportions under null and alternative for different exog.  
  Example, in oneway or two-way anova, we might want to specify cell means and then compute implied parameters and restriction on those using the link function. We need a helper function for that.

- Noncentrality and power will be defined for chisquare/F or normal/t distribution, i.e. effect size is not directly defined in terms of risk difference, risk ratio or odds ratio. (similar to Cohen's fsqu versus Cohen's standardized mean diff delta). This should be analogous generalization to joint hypothesis test compared to a single parameter restriction in two sample case.

- Variance as a function of mean. I guess a similar issue will show up as in power for a single proportion. The variance differs between null parameters and alternative parameters. 
  The simpler starting point will be local or Wald analysis where we assume variance and weights for irls/cov_params remain unchanged, similar to effectsize power for one proportion.  
  I didn't think yet about how we can handle the case with different cov_params under null and alternative, that wouldn't be local wald or score test anymore.





user: josef-pkt
issue_6755: state: open
title: How to extract the default and set my own tolerance value and max of iterations in statsmodels.sandbox.regression.gmm.fit
body: I wonder how I could check the default value of function tolerance and maximum of iteration when using statsmodels.sandbox.regression.gmm.fit.

For example, I have the following code:

model = gmm_poisson_new(endog=X_W['y'],exog=X_W[covariates], k_moms=num_moment, k_params=num_moment, first=X_W, instrument=None)
res = model.fit(beta0_W, maxiter=5, optim_method='nm', wargs=dict(centered=False))

print(res.summary())

##############
I would like to set my own tolerance value when running the numerical method. How can I do it? Is there any documentation explaining about this? I checked the following one, but seems that no details about this are mentioned:
https://www.statsmodels.org/stable/generated/statsmodels.sandbox.regression.gmm.GMM.fit.html#statsmodels.sandbox.regression.gmm.GMM.fit

Thank you so much!

user: yl-hz
issue_6749: state: open
title: Linear mixed effects model causes console to die when data are too large
body: Whenever I run a linear mixed effect model on a large dataset, the model seems to compute. However, when I query it, I find instead that it's caused the console I am using to lose all its variables and imports. Note that my dataset has approximately 286k lines––big, but still reasonable. Any idea what's happening?
```
In [1]: import statsmodels.api as sm
In [2]: import statsmodels.formula.api as smf
In [3]: import pandas as pd
In [4]: data = pd.read_pickle("data.pkl")
```

This works fine:

```
In [5]: mod = smf.ols(formula='DV ~ IV', data=data).fit()
In [6]: mod.summary()

Out [7]: 

                            OLS Regression Results                            
======================================================================
Dep. Variable:                DV   R-squared:                       0.005
Model:                            OLS   Adj. R-squared:                  0.005
Method:                          Least Squares   F-statistic:                     1020.
Date:                Tue, 26 May 2020   Prob (F-statistic):          3.05e-223
Time:                        17:12:43   Log-Likelihood:             2.5317e+05
No. Observations:              218417   AIC:                        -5.063e+05
Df Residuals:                  218415   BIC:                        -5.063e+05
Df Model:                           1                                         
Covariance Type:            nonrobust                                         
======================================================================
                         coef    std err          t            P>|t|      [0.025      0.975]
--------------------------------------------------------------------------------------
Intercept       0.6118     0.000   1386.513     0.000       0.611       0.613
IV[T.leve_1]    -0.0152  0.000    -31.933      0.000      -0.016      -0.014
======================================================================
Omnibus:                    39541.771   Durbin-Watson:                   1.866
Prob(Omnibus):                  0.000   Jarque-Bera (JB):           217846.285
Skew:                          -0.765   Prob(JB):                         0.00
Kurtosis:                       7.647   Cond. No.                         5.26
======================================================================
```

Now I try this:


```
 In [8]: md = smf.mixedlm("DV ~ IV", data = data,  groups=data["grouping_variable_1"], re_formula = "~grouping_variable_2").fit()

In [9]: md.summary()

Traceback (most recent call last):

  File "<ipython-input-1-c7c36053ef65>", line 1, in <module>
    md.summary()

NameError: name 'md' is not defined
```


user: texturejc
issue_6743: state: open
title: DFM for dimensionality reduction 
body: How can we use DFM for dimensionality reduction as specified in the following paper: https://www.researchgate.net/publication/227702453_Estimating_common_trends_in_multivariate_time_series_using_dynamic_factor_analysis

Is there a PCA analog implementation of DFA to reduce the dimensionality of a multivariate time series?
user: arpnomic
issue_6740: state: open
title: ENH: Estimate SARIMAX using CSS
body: Hey guys, first of all thanks for the wonderful tool. I'm the author of [pmdarima](https://github.com/alkaline-ml/pmdarima) and we are heavy users of your time series estimators. One of the feature requests we've gotten recently is to be able to estimate SARIMAX models using the Conditional Sum of Squares likelihood function, as it tends to be faster. I know that in the old `statsmodels.tsa.arima_model.ARIMA` class, one could pass `method='css'`. However, this is not an option when using the `SARIMAX` class. Is this something that could ever be supported?
user: tgsmith61591
issue_6735: state: open
title: statsmodels.statsmodels.sandbox.predict_functional.py  func: predict_functional  documentation is not available.
body: when I m using func:: predict_functional  then I wanted to see documentation ,it shows **no documentation found** and I concluded that in this function this import
**from statsmodels.compat.pandas import Appender**
is not imported correctly it shows error  **" ImportError: cannot import name 'Appender' from 'statsmodels.compat.pandas'  "**

user: Ritik2604
issue_6721: state: open
title: SUMM/REF/ENH: refactor, reorganize current proportions
body: a reminder for now

Whenever I look for something in proportions.py, I'm getting a bit confused what cases the different functions include. I wrote it a long time ago mostly replicating individual functions in the documentation of other stats packages.

Similar to newer stats like two sample proportions, we should have basic functions with method options for 1-sample case, confint for 1sample proportion follows that pattern.

details:
`stats.proportion.proportion_effectsize` is the arcsine-sqrt version, now also in meta-analysis
it's for the 2 independent sample case, but that's not obvious from the name. It was written for power computation using arcsine and normal distribution.
user: josef-pkt
issue_6720: state: open
title: ENH: actual coverage of binomial proportion confint
body: proportion conf_int cannot have exact coverage because of the discrete sample space
Other packages (including NCSS/PASS) include the actual binomial coverage.

AFAICS, we don't have it available

https://stats.stackexchange.com/questions/466832/how-to-calculate-coverage-probability-from-binomial-confidence-intervals

This should be pretty easy to compute, similar to the power computation using "exact"  binomial distribution (scipy.stats.binom for the confidence region in terms of counts). But I'm not in this neighborhood right now.
user: josef-pkt
issue_6716: state: open
title: ENH/Design: pattern for simulate_power functions
body: For PR #6526 I spent some time simulating power for anova and equivalence-anova to verify the theoretical power computation. This could be dressed up in functions for the users so they can simulate the power for different cases, e.g. trimmed anova, option trim_frac, is not yet supported by the power calculation.

For simple cases a simulate_xxx_power function wouldn't require a complicated setup. We need

- options for the hypothesis test including `value` if nonzero and method options
- part of dgp that is common to null and alternative, distribution of data, variances, ...
- values that differ between alternative and null, either as given values for alternative or specified as diff to the null.
- parts that are invariant for power computation can be unspecified and hard coded , e.g. actual value of common mean in anova under the null is irrelevant for power and can be set to zero.

I never looked closely at what packages in R are doing.


user: josef-pkt
issue_6708: state: open
title: theoretical background: use_t and asymptotic distribution
body: this is just a vague issue to put something on the longterm reading list

In general we use `use_t=False` because the standard asymptotics are normal and chisquare.
However, in many cases using t and F have better small sample properties even if the underlying model is not correctly specified as normal, chisquare, ... This was the motivation for adding `use_t`, `use_f` options.

In statistics properties and distribution of statistics are often derived under the assumption that samples are normal distributed, i.e. that we have a correctly specified likelihood. This gives a theoretical justification for using t and F in many case.

I have seen econometrics articles that justify t and F distribution for test statistics based on asymptotics, but not the usual asymptotics that converges to normal and chisquare.

The related issue is that we need degrees of freedom that take into account how much information that we have. #6564

The objective would be to get better small sample statistics or better justification for them.

This issue needs references when I find them again.

"15 observations ain't no BIG data"




user: josef-pkt
issue_6707: state: open
title: ENH: power and sample size if null distribution is also noncentral
body: see
https://github.com/statsmodels/statsmodels/pull/6526#issuecomment-628328101

This is not really clear to me, but I guess that there is a general pattern that we will eventually need.

In the effect size and meta-analysis literature there are many cases where non-central distributions are important, e.g. standardized mean difference has a non-central t distribution, but meta-analysis works mostly with a normal approximation.

We get a noncentral t distribution if we don't remove the mean of the normal numerator, i.e.
A normally distributed random variable with unit variance and non-zero mean divided by a Chi-squared distributed random variable is noncentral t-distributed, where the noncentrality parameter is equal to the mean of the normal random variable.
https://en.wikipedia.org/wiki/Noncentral_t-distribution#Characterization

power computation would have noncentral distribution both under the null and under the alternative.

In equivalence testing for anova, we have noncentral F under the null and central F under the alternative (defined as all means equal).

The current power classes assume we have central distributions under the null and noncentral under the alternative, except for normal distribution case where both are the same.
user: josef-pkt
issue_6706: state: open
title: ENH: UMP test for equivalence 
body: Wellek's book and the R package has a lot on UMP tests as alternative to tost.

some stats.stackexchange discussions:

https://stats.stackexchange.com/questions/95462/obtaining-p-values-for-ump-t-tests-for-equivalence/
https://stats.stackexchange.com/questions/95667/intuitive-explanation-of-differences-between-tost-and-ump-tests-for-equivalence

I never looked closely at these UMP tests
user: josef-pkt
issue_6702: state: open
title: ENH: GAM or penalized linear spline, piecewise regression
body: (currently just a thought about piecewise regression)

GAM penalization is on variance or curvature defined by second derivatives of splines. The usual is third order splines.

If we use constant or linear splines, then we don't have second derivative curvature, so, I guess, GAM penalization does not work.

We can still use the diff params penalty as in penalization splines.

related (why I was thinking about it)

If we have a piecewise linear model, e.g. interaction between linear function and some binning in large groups by value of exog. then we could also use linear splines to have the linear segments connected.

Penalization splines, i.e. splines with diff penalty, assume equal space knots, however in the linear case there is no reason to restrict to equal spaced knots even with diff penalty.
(Linear truncated power splines would be very similar, but basis functions are highly correlated.)

user: josef-pkt
issue_6700: state: open
title: ENH: classification results like sensitivity with confint
body: Just found by chance

NCSS has a list of classification results for diagnostic tests (as in medical diagnostic)
https://www.ncss.com/software/ncss/ncss-documentation/#DiagnosticTests

All of these are statistics of proportions or comparisons of proportiosn and we can get confint using the corresponding proportion functions. Those are available in statsmodels.

Should we add something like `classification_diagnostics` that creates a dataframe with statistics and confint (and maybe standard errors when available).
user: josef-pkt
issue_6696: state: open
title: ENH: cov and matrix tools to work with covariances and correlations - rebased
body: rebased version of #4143
user: josef-pkt
issue_6692: state: open
title: ENH: more multivariate mean tests
body: continuation of merged PR #4107

see also #3208 #6689

The PR only includes 1-sample test. Need two sample tests and more.
Stata has k-sample test with special methods used in 2-sample case.

Also, in the one sample case we can get tests for linear hypothesis. Currently linear hypothesis are supported by confint.
This is similar to the wald tests in LikelihoodModelResults, if we use (params, cov_params, nobs) as sufficient statistics. (I started with code somewhere for the standalone version of those Results methods.)

I haven't checked vectorization of mvmean functions.

user: josef-pkt
issue_6689: state: open
title: SUMM: organizing basic multivariate stats
body: I'm looking at merging #4107 but I'm not sure of code location

multivariate methods and models PCA, CCA, MANOVA, ... are in `statsmodels.multivariate`

basic one, two and k sample statistics, hypothesis tests, estimators of moments go into `statsmodels.stats`

main areas

- covariance, scatter estimation
  - basic test of covariance/correlation matrices PR #3842, issue #2765 with script file
  - tools PR #4143, rebased PR #6696
  - robust cov/corr PR #3230
  - large dim cov/corr, shrinkage and penalization issue #3197 with script file
  - partial overlap with multicollinearity PR and module ?
- multivariate means: one PR, not much else
  - #4107 hotelling's t and confidence interval, 
  - control charts PRs #5503 #5506
  - outlier tests
- skew, kurtosis, distributions
  - Mardia's skew, kurtosis, where does that go?
  - ?
- multivariate that goes with other topics, e.g. 
   - mv normality with gof?
   - meta_analysis
- multivariate transformation (related)
   - #3763 simplex/compositional data part of statsmodels.multivariate
- related tools
   - stats.multivariate_tools contraints projection and cc_ranktest, cc_stats
   - copulas ?

possible organization

- stats.multivariate: mean estimation and hypothesis tests, outlier, ...
- several private or public modules for cov/corr



user: josef-pkt
issue_6685: state: open
title: FAQ:  equivalence test, TOST and confidence interval
body: (just a note)

equivalence testing with tost is equivalent to using a confidence interval with 2 alpha

However, this is not true for all confidence intervals.
e.g. binomial tests and similar can be "minlike" or "central"

because tost uses two one sided tests at alpha, the corresponding confidence interval is "central". The minlike confidence interval is not consistent with TOST.

example:
2 sample Poisson test with exact conditional method PR #6677 (unit tests for tost didn't match up when using minlike confidence interval from R package `exactci`, but it does when using "central" confidence interval)

user: josef-pkt
issue_6676: state: open
title: ENH: exact test equality of two proportions rebased
body: rebased version of #2608 from 2015

no merge conflict, no extra changes
user: josef-pkt
issue_6673: state: open
title: [Doc] Discrepancy in default scale description
body: [Here](https://github.com/statsmodels/statsmodels/blob/master/statsmodels/genmod/generalized_linear_model.py#L963), it says chi-square divided by `df_resid` which is supported by the [code](https://github.com/statsmodels/statsmodels/blob/master/statsmodels/genmod/generalized_linear_model.py#L772).

On the other hand, [here](https://github.com/statsmodels/statsmodels/blob/master/statsmodels/genmod/generalized_linear_model.py#L741) it says chi-square only.

Happy to submit a PR if I am right.
user: tashrifbillah
issue_6668: state: open
title: Conditional Logit - maximum recursion depth exceeded in comparison
body: #### Describe the bug

Upon running a ConditionalLogit with many observations approx. 1 million (I would actually like to run the code on 10 times as many observations 😅) I encounter an exception, namely a `RecursionError`. It succeeds with fewer observations (handling at least 100000 observations). CPU remains somewhat chill, however, RAM is running at approx. 90-92 pct. when the exception is thrown:


![image](https://user-images.githubusercontent.com/36441764/80353315-785af700-8875-11ea-81fe-e37a85b9692f.png)

![image](https://user-images.githubusercontent.com/36441764/80353242-5d888280-8875-11ea-9526-4b975346e322.png)




#### Code Sample, a copy-pastable example if possible

What I am trying to run:
```python
dependent = df.Elev_fravaerende
exog = df[['value_precip','value_wind', 'value_temperature']]

cm_model_base = cm.ConditionalLogit(dependent, exog, groups = df["id"])
cm_model_result = cm_model_base.fit()
cm_model_result.summary()
```

#### Expected Output

From the exception it is unclear to me what the cause of the problem is. If it is due to memory is there a way to minimize the usage such that the regression is still feasibly? When using 100.000 observations the summary is as follows:



| Dep. Variable:   | Elev_fravaerende   | No. Observations:   |   54183 |
|:-----------------|:-------------------|:--------------------|--------:|
| Model:           | ConditionalLogit   | No. groups:         |   276   |
| Log-Likelihood:  | -13385.            | Min group size:     |    34   |
| Method:          | BFGS               | Max group size:     |   200   |
| Date:            | Mon, 27 Apr 2020   | Mean group size:    |   196.3 |
| Time:            | 18:33:31           | nan                 |   nan   |



<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>coef</th>
      <th>std err</th>
      <th>z</th>
      <th>P&gt;|z|</th>
      <th>[0.025</th>
      <th>0.975]</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>value_precip</th>
      <td>-0.1064</td>
      <td>0.047</td>
      <td>-2.289</td>
      <td>0.022</td>
      <td>-0.198</td>
      <td>-0.015</td>
    </tr>
    <tr>
      <th>value_wind</th>
      <td>0.0446</td>
      <td>0.008</td>
      <td>5.369</td>
      <td>0.000</td>
      <td>0.028</td>
      <td>0.061</td>
    </tr>
    <tr>
      <th>value_temperature</th>
      <td>-0.0585</td>
      <td>0.003</td>
      <td>-19.487</td>
      <td>0.000</td>
      <td>-0.064</td>
      <td>-0.053</td>
    </tr>
  </tbody>
</table>

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>


INSTALLED VERSIONS
------------------
Python: 3.7.3.final.0

Statsmodels
===========

Installed: 0.10.1 (C:\Users\kw7l\AppData\Local\Continuum\miniconda3\envs\listst11\lib\site-packages\statsmodels)

Required Dependencies
=====================

cython: 0.29.16 (C:\Users\kw7l\AppData\Local\Continuum\miniconda3\envs\listst11\lib\site-packages\Cython)
numpy: 1.18.1 (C:\Users\kw7l\AppData\Roaming\Python\Python37\site-packages\numpy)
scipy: 1.3.0 (C:\Users\kw7l\AppData\Local\Continuum\miniconda3\envs\listst11\lib\site-packages\scipy)
pandas: 1.0.0 (C:\Users\kw7l\AppData\Local\Continuum\miniconda3\envs\listst11\lib\site-packages\pandas)
    dateutil: 2.8.0 (C:\Users\kw7l\AppData\Local\Continuum\miniconda3\envs\listst11\lib\site-packages\dateutil)
patsy: 0.5.1 (C:\Users\kw7l\AppData\Local\Continuum\miniconda3\envs\listst11\lib\site-packages\patsy)

Optional Dependencies
=====================

matplotlib: 3.1.0 (C:\Users\kw7l\AppData\Local\Continuum\miniconda3\envs\listst11\lib\site-packages\matplotlib)
    backend: module://ipykernel.pylab.backend_inline 
cvxopt: Not installed
joblib: 0.13.2 (C:\Users\kw7l\AppData\Local\Continuum\miniconda3\envs\listst11\lib\site-packages\joblib)

Developer Tools
================

IPython: 7.7.0 (C:\Users\kw7l\AppData\Local\Continuum\miniconda3\envs\listst11\lib\site-packages\IPython)
    jinja2: 2.10.1 (C:\Users\kw7l\AppData\Local\Continuum\miniconda3\envs\listst11\lib\site-packages\jinja2)
sphinx: Not installed
    pygments: 2.4.2 (C:\Users\kw7l\AppData\Local\Continuum\miniconda3\envs\listst11\lib\site-packages\pygments)
pytest: 5.3.5 (C:\Users\kw7l\AppData\Local\Continuum\miniconda3\envs\listst11\lib\site-packages\pytest)
virtualenv: Not installed

</details>

user: Econ808
issue_6667: state: open
title: SARIMAX model can lead to inaccurate results with tz-aware pandas series
body: #### Describe the bug

SARIMAX is great as it supports pandas series. However, it can be a bit "wrong" with seasonal series in real cases. That's because sometimes the seasonality is based on "time of the day" rather than "periods"; this is particularly true with series with a daily pattern that doesn't change over the DST/solar time change. On that day, you have a "season" of either 23 or 25 hours. Examples can be traffic or electricity consumption. You always go to work at 8, even if 8 today is 23 hours after 8 yesterday (solar time to DST switch). You always turn on your kettle at 7:30, even if today is 25 hours after 7:30 yesterday (DST to solar time).

Now, imagine that you have a pd Series with daily shape like this:

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from statsmodels.tsa.statespace import sarimax

index = pd.date_range(
    "2020", "2021", freq="H", closed="left", tz="Europe/London"
)
values = abs(index.hour - 12)  # time-of-day shape
noise = np.random.rand(len(index))
s = pd.Series(values + noise, index)
```

You want to build a seasonal arima with a daily seasonality:

```python
length = len(s)
train_length = length // 10
remaining = length - train_length
train = s[:train_length]

m = sarimax.SARIMAX(
    train, order=(1, 0, 1), seasonal_order=(1, 0, 1, 24)  # 24 hours in a day
)  # model is probably overfit, but we don't care
r = m.fit()
```

With this model you're saying "I have a 24 hours seasonality", but that's technically wrong twice per year. In fact, let's try to compare the forecast with the actual series


```python
forecast = r.forecast(remaining)
full = train.append(forecast)  # add forecast to train
full["2020-3-26":"2020-4-1"].plot()
s["2020-3-26":"2020-4-1"].plot()
```

![image](https://user-images.githubusercontent.com/34656862/80320976-955dde80-8811-11ea-9966-0fc1f42d23b2.png)
You can see that there's a systematic error that starts on the 29th of March. Thats because that day has 23 hours, so saying that s[t] depends on s[t-24] is wrong.

Now, I don't think how SARIMAX works is wrong: you specify a season of 24 (hours), you get a forecast for a season of 24. But, as I said, I'd expect many real-case scenarios that "pass silently" without being noticed.

So I have two proposals:
1) Raise a warning (unless turned off explicitly) if a series **that is tz-localized and more granular than daily** is passed to the SARIMAX model. Something that explains that seasonality doesn't always makes sense if the seasons are time-of-the-day based.
2) *Maybe* find a way of handling tz-localized series differently **if requested**. For example, we could add an optional kwarg; if the latter is False (default), the fitting will work as it's currently doing (but raising the warning). Otherwise, the series is pre-processed and converted to another that preserves the 24-hours distance between each day. To do that, a fictitious hour must be created on winter-to-summer switch, while an hour must be dropped on summer-to-winter switch. Then the forecast is generated, and post-processed to be localised as the series that was used as input when creating the model. This approach is not perfect as there would be a weird behaviour around those DST/solar switch. But this is obviously better than "having a whole summer wrong".

I wonder if anyone else has been through this too :) This is just a proposal, I thought I could start the discussion.
user: giuliobeseghi
issue_6665: state: open
title: How to estimate alpha in GLMgam and df, degree in BSpline
body: I am confused how to get alpha parameter to input in GLMgam and df, degree in BSpline. is there range of values for each of them ? It would be nice if provided with python example code

thanks
user: ramdhan1989
issue_6661: state: open
title: DOC/ENH: use_t in get_prediction 
body: The docstring for get_prediction does not mention that the distribution used in confint corresponds to the use_t of the results instance.

possible enhancement: add keyword option to override attribute

(context: in meta-analysis packages like metafor also match prediction interval to the distribution used for the parameter estimates. Other packages use different, inconsistent distribution. according to comments in an article.)
user: josef-pkt
issue_6644: state: open
title: QUESTION: sm.Logit fails, while sm.GLM with Binomial link does not
body: #### Describe the bug

This is more of a **question**, and maybe a potential issue (please, correct me where I am wrong)

I was trying to fit a simple `logit` model using `sm.Logit` approach but I was receiving this:
```python
lm.fit(maxiter= 10000)
Warning: Maximum number of iterations has been exceeded.
         Current function value: 0.664288
         Iterations: 10000
Traceback (most recent call last):

  File "<ipython-input-64-b39a0ea8a7d2>", line 1, in <module>
    lm.fit(maxiter= 10000)

  File "C:\Users\Emil\anaconda3\lib\site-packages\statsmodels\discrete\discrete_model.py", line 1913, in fit
    disp=disp, callback=callback, **kwargs)

  File "C:\Users\Emil\anaconda3\lib\site-packages\statsmodels\discrete\discrete_model.py", line 216, in fit
    disp=disp, callback=callback, **kwargs)

  File "C:\Users\Emil\anaconda3\lib\site-packages\statsmodels\base\model.py", line 533, in fit
    Hinv = np.linalg.inv(-retvals['Hessian']) / nobs

  File "<__array_function__ internals>", line 6, in inv

  File "C:\Users\Emil\anaconda3\lib\site-packages\numpy\linalg\linalg.py", line 547, in inv
    ainv = _umath_linalg.inv(a, signature=signature, extobj=extobj)

  File "C:\Users\Emil\anaconda3\lib\site-packages\numpy\linalg\linalg.py", line 97, in _raise_linalgerror_singular
    raise LinAlgError("Singular matrix")
```
Which, AFAIK means that my `endog` has some issues. So, I looked at it:
```python

 y.describe()

count    676.000000
mean       0.573964
std        0.494865
min        0.000000
25%        0.000000
50%        1.000000
75%        1.000000
max        1.000000
Name: benchmark_better, dtype: float64
```
However, when I am trying `sm.GLM(y, X, family = sm.families.Binomial(link = sm.families.links.logit()))` it works **as expected**:
```python

                 Generalized Linear Model Regression Results                  
==============================================================================
Dep. Variable:       benchmark_better   No. Observations:                  676
Model:                            GLM   Df Residuals:                      654
Model Family:                Binomial   Df Model:                           21
Link Function:                  logit   Scale:                          1.0000
Method:                          IRLS   Log-Likelihood:                -449.06
Date:                Tue, 14 Apr 2020   Deviance:                       898.12
Time:                        15:28:45   Pearson chi2:                     677.
No. Iterations:                     4                                         
Covariance Type:            nonrobust                                         
```
AFAIK, the results of `sm.Logit` and `sm.GLM` with `Binomial` link should be `all_close`. If one works, why not the other?
#### Code Sample, a copy-pastable example if possible


Because this is a potential question, no code samples are required I presume
<details>

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `master`. If your problem has been fixed in an unreleased version, you might be able to use `master` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the master branch of this repository? It helps the limited resources if we know problems exist in the current master so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output
Results

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]
```
INSTALLED VERSIONS
------------------
Python: 3.7.7.final.0

statsmodels
===========

Installed: 0.11.0 (C:\Users\Emil\anaconda3\lib\site-packages\statsmodels)

Required Dependencies
=====================

cython: 0.29.15 (C:\Users\Emil\anaconda3\lib\site-packages\Cython)
numpy: 1.18.1 (C:\Users\Emil\anaconda3\lib\site-packages\numpy)
scipy: 1.4.1 (C:\Users\Emil\anaconda3\lib\site-packages\scipy)
pandas: 1.0.3 (C:\Users\Emil\anaconda3\lib\site-packages\pandas)
    dateutil: 2.8.1 (C:\Users\Emil\anaconda3\lib\site-packages\dateutil)
patsy: 0.5.1 (C:\Users\Emil\anaconda3\lib\site-packages\patsy)

Optional Dependencies
=====================

matplotlib: 3.1.3 (C:\Users\Emil\anaconda3\lib\site-packages\matplotlib)
    backend: module://ipykernel.pylab.backend_inline 
cvxopt: Not installed
joblib: 0.14.1 (C:\Users\Emil\anaconda3\lib\site-packages\joblib)

Developer Tools
================

IPython: 7.13.0 (C:\Users\Emil\anaconda3\lib\site-packages\IPython)
    jinja2: 2.11.1 (C:\Users\Emil\anaconda3\lib\site-packages\jinja2)
sphinx: 2.4.4 (C:\Users\Emil\anaconda3\lib\site-packages\sphinx)
    pygments: 2.6.1 (C:\Users\Emil\anaconda3\lib\site-packages\pygments)
pytest: 5.4.1 (C:\Users\Emil\anaconda3\lib\site-packages\pytest)
virtualenv: Not installed
```
</details>

user: emilmirzayev
issue_6640: state: open
title: ENH: meta-analysis meta-regression with cov_type (cluster), including random effect
body: specific issue for adding cov_type to meta-analysis
Hedges et all 2010a looks good (based on partial skimming) and can be used for implemention and we can check how much works out of the box with available cov_type, e.g. in WLS.
The article has one example with data in appendix but only partial results for it in text.

Article includes random effect analysis. Main task is to estimate between random effects variance taking correlation into account. This sounds a bit similar to GEE, but we should be able to also estimate from summary data. In their R function and example the within correlation is user provided.
(GEE has weights, and so can be used with cluster robust standard errors, but we need enough within observations to compute correlation parameters with GEE.)

This is more work going through the details than I want to spend right now.
basic meta-analysis methods in PR #6632

below copies comment from
https://github.com/statsmodels/statsmodels/issues/6585#issuecomment-606271324

using cluster robust standard errors

erratum has a link to northwestern that doesn't exist anymore
Tipton has R and Stata packages

https://www.ipr.northwestern.edu/who-we-are/faculty-experts/tipton.html
https://www.bethtipton.com/meta-analysis

Hedges, Larry V., Elizabeth Tipton, and Matthew C. Johnson. 2010a. “Erratum: Robust Variance Estimation in Meta-Regression with Dependent Effect Size Estimates.” Research Synthesis Methods 1 (2): 164–65. https://doi.org/10.1002/jrsm.17.
———. 2010b. “Robust Variance Estimation in Meta-Regression with Dependent Effect Size Estimates.” Research Synthesis Methods 1 (1): 39–65. https://doi.org/10.1002/jrsm.5.
user: josef-pkt
issue_6639: state: open
title: ENH: proportions for rare events, binomial zeros, meta-analysis and effect size
body: binomial regression on a constant or combining independent samples, where each sample is a 2x2 table.
case for meta-analysis PR #6632

In the case of rare events, it is likely that zeros are observed, i.e. no event in a sample.

The first paper looks like a good summary of several methods, with formula, and an example with data that can be used to unit test several methods. 

I'm not planning to do much for now, but we need a review of options that we can use for zeros in these cases. Current handling of zeros is a bit arbitrary or inconsistent in existing model/functions.
(e.g. contingency tables have an option to add 0.5 to each cell, some functions use 0.5 as continuity correction, PASS recommends adding a "small" number (<0.01) to zeros for power (?) calculations, GLM binomial might work if it "clips" enough. Frith method?)

Piaget‐Rossel, Romain, and Patrick Taffé. 2019. “Meta-Analysis of Rare Events under the Assumption of a Homogeneous Treatment Effect.” Biometrical Journal 61 (6): 1557–74. https://doi.org/10.1002/bimj.201800381.

two more on treatment of zeros
Rücker, Gerta, Guido Schwarzer, James Carpenter, and Ingram Olkin. 2009. “Why Add Anything to Nothing? The Arcsine Difference as a Measure of Treatment Effect in Meta-Analysis with Zero Cells.” Statistics in Medicine 28 (5): 721–38. https://doi.org/10.1002/sim.3511.

Sweeting, Michael J., Alexander J. Sutton, and Paul C. Lambert. 2004. “What to Add to Nothing? Use and Avoidance of Continuity Corrections in Meta-Analysis of Sparse Data.” Statistics in Medicine 23 (9): 1351–75. https://doi.org/10.1002/sim.1761.


user: josef-pkt
issue_6637: state: open
title: SUMM/ENH/REF: Holder and Results classes (apart from Model fit return)
body: (I lost one of my helper functions again in some PR.)

We have several versions and attempts at classes that could be used to return results of hypothesis tests, but still no clear pattern.
i.e. whenever I write a new function or method I have to decide whether I want in that case return a tuple, a specif results class or some generic Holder class.

from lightweight to heavy

- basic Holder class is now in tools.testing, I added `__repr__` to it (maybe in a PR not yet merged)
  #4994 and #5791
- Holder class with tuple access, I don't find my PR anymore, basic Holder class, but allows tuple and indexing access to core results eg. `(statistic, pvalue)`. This is currently my preferred, also for other than hypothesis tests.
   **update** I used it for constraints https://github.com/statsmodels/statsmodels/pull/6521/files#diff-f9e27ad5ee7a0574a3103e73a5a21f8bR35
    more elaborate version now in #6678
- HypothesisTestResults with basic related attributes: old PR #1182
- elaborate Hypothesis test results classes #5001  (looks pretty heavy)

other return types:

- Named tuples are too restrictive and should not be used. Main disadvantage any change is backwards incompatible.
- Plain tuples are very lightweight, traditional, but we cannot change returns, so I would like to reduce their use
- pandas DataFrame: makes sense in some cases where we have many results with similar pattern. Not clear yet what backwards compatibility restrictions are. I guess we can add columns, and rows.
- case specific results class: best option if anything more than trivial returns or results




user: josef-pkt
issue_6633: state: open
title: ENH: add summary/table foot_notes for method options
body: (I had a similar idea a long time ago, when it didn't find a positive response. It's time to add it anyways.)

Problem if we have method options, then currently results outside of models, e.g. hypothesis tests, ... don't keep information about which method was used.
This is important when the same names like statistic, std, p-value, conf_int can be computed based on different methods.

Basic idea is to add `method` attribute to provide information about the method used. We have this to a large extend in the top table of `Results.summary`. And similar, but only for warnings in the trailing text of summary.

More elaborate idea is to actually have a `footnotes` (foot_notes ?) attribute that contains a statement or statements similar to the trailing text in summary. The user can print it, or we can add it as trailing text, if there is a `summary` method.

e.g. 
"Random effects are based on the `<Paule-Mandel>` estimate of the between variance"
"Random effect estimates are based on the `<DL>` estimate of the between variance with HKSJ/WLS scale corrrection to the standard errors. P-values are based on t-distribution."
#6632

This will not be relevant in library use, where we just want to get the core numbers returned


user: josef-pkt
issue_6620: state: open
title: ENH: add rsquared for statespace model #4734
body: - [x] closes #4734
- [ ] tests added / passed. 
- [ ] code/documentation is well formatted.  
- [ ] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

<details>


**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in master and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/master -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>

user: BenjaminLiuPenrose
issue_6615: state: open
title: FAQ/ENH: observed, post-hoc power: don't make invalid conclusions
body: Computing observed power, i.e. power at observed effect sizes, have been strongly criticized in recent literature, but mainly because it is used to argue in favor of null or alternative hypothesis.

However, I'd like to include them together with effect sizes, mainly so we get an additional statistic that complements p-value with a measure of type 2 error. 
Also, we should be able to use the same functions to compute prospective power and sample size computation at different effect sizes and sample sizes.
(currently I have observed power in anova PR)



O’Keefe, Daniel J. 2007. “Brief Report: Post Hoc Power, Observed Power, A Priori Power, Retrospective Power, Prospective Power, Achieved Power: Sorting Out Appropriate Uses of Statistical Power Analyses.” Communication Methods and Measures 1 (4): 291–99. https://doi.org/10.1080/19312450701641375.

Onwuegbuzie, Anthony J., and Nancy L. Leech. 2004. “Post Hoc Power: A Concept Whose Time Has Come.” Understanding Statistics 3 (4): 201–30. https://doi.org/10.1207/s15328031us0304_1.

There are more articles, but I don't find them anymore at the moment.
user: josef-pkt
issue_6612: state: open
title: statsmodels.stats.outliers_influence.variance_inflation_factor(exog, exog_idx) does not allow weights
body: #### Is your feature request related to a problem? Please describe
variance_inflation_factor() function does not support WLS with weights. In economic forecasting by regression models, this feature would be nice to have since OLS does not always fit in the scenario.

#### Describe the solution you'd like
I would like to propose to add the functionality of computing VIF under WLS with weights. For example, currently I use the following function:

def _compute_vif(exog,
                 exog_idx,
                 weights=None,
                 model_config=None):
    """
    Compute variance inflation factor, VIF, for one exogenous variable
        for OLS and WLS that allows weights.
    Parameters
    ----------
    exog: X features [X_1, X_2, ..., X_n]
    exog_idx: ith index for features
    weights: weights
    model_config: {"hasconst": True,
                   "cov_type": "HC3"} by default

    Returns: vif
    -------

    """
    if model_config is None:
        model_config = {"hasconst": True,
                        "cov_type": "HC3"}
    k_vars = exog.shape[1]
    x_i = exog[:, exog_idx]
    mask = np.arange(k_vars) != exog_idx
    x_noti = exog[:, mask]
    if weights is None:
        r_squared_i = OLS(x_i,
                          x_noti,
                          hasconst=model_config["hasconst"]).fit().rsquared
    else:
        r_squared_i = WLS(x_i,
                          x_noti,
                          hasconst=model_config["hasconst"],
                          weights=weights).fit(
            cov_type=model_config["cov_type"]).rsquared
    vif = 1. / (1. - r_squared_i)
    return vif


#### Describe alternatives you have considered
An alternative that I considered is to create a new function for WLS while keeping the old function still under OLS. But from my experience, users prefer calling one function to compute VIF.


user: patrickxli
issue_6611: state: open
title: ENH: tools bias correction for nonlinear transformation
body: there are several issues related to this
#3224 transformation to normality
#2791 backtransform for transformed endog, e.g. log(endog) in time series, smearing 
also related delta method

This issue is specifically for collecting nonlinear transformations with helper methods to estimate the bias correction of nonlinear transformation.
standard case: log-normal <-> normal

current usecase
I would like to get a quick idea about bias when we convert between several measures
e.g. effect size for anova, https://github.com/statsmodels/statsmodels/pull/6526/files#diff-26abd2f0b72810fe723fb299c04911e9R123
(I don't want to bother to have to derive the bias by hand for each case, which is mainly getting the derivatives)

user: josef-pkt
issue_6610: state: open
title: ENH: relationship eta-squared effect size and semi-partial correlation
body: `https://github.com/friendly/heplots/issues/1`

"
I suspect DescTools is reporting the correct results since $\eta^2$ is really just the semi-partial correlation squared, which can be obtained via R^2 differences after removing the IV of interest.
"

We get semi-partial correlation in PR #2380 (at least I wrote the functions in that context)
and get etasq effect size in #6526 for anova case, and open issue #5896

matrix version of semi-partial correlation only works for single columns and not for multi-column categorical variables as in anova.




user: josef-pkt
issue_6598: state: open
title: ENH: added chi^2 effect size in contingency table class, closes #6184
body: - [ ] closes #6184
- [ ] tests added / passed. 
- [ ] code/documentation is well formatted.  
- [ ] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

<details>


**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in master and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/master -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>

user: ZhaoBin-Li
issue_6592: state: open
title: ENH: t-test Welch versus Satterthwaite df in Stata
body: Stata has Welch df for two sample t-test different from Satterthwaite df.
They have the same df option for the effect sizes for the ttest.

I've never seen that before and always thought Welch uses Satterthwaite df.

Do we need to add this?

```
. ttesti 30 13 2.74 30 11 2.24, level(90) unequal welch

Two-sample t test with unequal variances
------------------------------------------------------------------------------
         |     Obs        Mean    Std. Err.   Std. Dev.   [90% Conf. Interval]
---------+--------------------------------------------------------------------
       x |      30          13    .5002533        2.74    12.15001    13.84999
       y |      30          11    .4089662        2.24    10.30511    11.69489
---------+--------------------------------------------------------------------
combined |      60          12    .3457698    2.678321    11.42219    12.57781
---------+--------------------------------------------------------------------
    diff |                   2    .6461476                .9198216    3.080178
------------------------------------------------------------------------------
    diff = mean(x) - mean(y)                                      t =   3.0953
Ho: diff = 0                             Welch's degrees of freedom =  57.6428

    Ha: diff < 0                 Ha: diff != 0                 Ha: diff > 0
 Pr(T < t) = 0.9985         Pr(|T| > |t|) = 0.0030          Pr(T > t) = 0.0015

. ttesti 30 13 2.74 30 11 2.24, level(90) unequal

Two-sample t test with unequal variances
------------------------------------------------------------------------------
         |     Obs        Mean    Std. Err.   Std. Dev.   [90% Conf. Interval]
---------+--------------------------------------------------------------------
       x |      30          13    .5002533        2.74    12.15001    13.84999
       y |      30          11    .4089662        2.24    10.30511    11.69489
---------+--------------------------------------------------------------------
combined |      60          12    .3457698    2.678321    11.42219    12.57781
---------+--------------------------------------------------------------------
    diff |                   2    .6461476                 .919237    3.080763
------------------------------------------------------------------------------
    diff = mean(x) - mean(y)                                      t =   3.0953
Ho: diff = 0                     Satterthwaite's degrees of freedom =  55.7949

    Ha: diff < 0                 Ha: diff != 0                 Ha: diff > 0
 Pr(T < t) = 0.9985         Pr(|T| > |t|) = 0.0031          Pr(T > t) = 0.0015

```

(I'm using Stata 14.2)
user: josef-pkt
issue_6591: state: open
title: SUMM/ENH: equivalence testing and gof, gof effect sizes
body: I have been looking at this for many years, but without usable, interpretable conclusion.
https://github.com/statsmodels/statsmodels/issues/6497#issuecomment-582208933
#1373
https://github.com/rc/dist_mixtures/issues/16#issuecomment-14014716
(I don't have any code that I can find for this)

#6497 overview issue for equivalence testing

I'm coming back to this now through effect size measures for model fit and their confidence intervals. It looks like this area has some thoughts on how to specify a model is a "close enough" fit.
For linear model there are effect sizes and confidence intervals for rsquared or similar.

Steiger 2004 mentions root mean squared error of approximation (p. 166) and mentions MacCallum et al as example. MacCallum et al has around 8500 citations.

Steiger, James H. 2004. “Beyond the F Test: Effect Size Confidence Intervals and Tests of Close Fit in the Analysis of Variance and Contrast Analysis.” Psychological Methods 9 (2): 164–82. https://doi.org/10.1037/1082-989X.9.2.164.

MacCallum, Robert C., Michael W. Browne, and Hazuki M. Sugawara. 1996. “Power Analysis and Determination of Sample Size for Covariance Structure Modeling.” Psychological Methods 1 (2): 130–49. https://doi.org/10.1037/1082-989X.1.2.130.


The same ideas should generalize to other gof statistic, e.g. various chisquare gof tests.

example: Can we reverse hypothesis in Hosmer-Lemeshow type test, to avoid some of the problems with it and get a "non-inferiority" or equivalence test for it?
What are interpretable effect sizes and equivalence margins for binned chisquare gof tests?

vague idea: Vuong test for comparing two non-nested models might be inconclusive, if both models reject (?) or don't (?) reject. Can we turn this into an equivalence test?

maybe another case: IIRC, I recently saw inference or distribution of AIC or similar information criteria.


user: josef-pkt
issue_6589: state: open
title: SUMM/ENH: oneway and extensions
body: I'm trying to get an overview what we can add for oneway anova, and then for generalization to linear models (or generically for all Wald type inference)

cases by generality, including variations liked trimming

- simple balanced oneway with pooled variances
- oneway with unequal sample sizes but pooled variance
- oneway with unequal sample sizes and unequal variances

### hypothesis tests

- omnibus f-test for equal means (maybe allow chisquare use_f=False)
- equivalence, non-inferiority, ... 
- ? follow-up post-hoc, multiple comparisons
- ? combining estimates, meta-analysis

### effect sizes and non-centrality parameter

- standard measures, f, omega, ...
- effect size and non-centrality for specific hypothesis and tests
- confidence intervals for effect sizes and non-centrality

### power and sample size

follows mostly when we have non-centrality parameter

### extension

most of this directly applies to OLS and wald tests

### tools

helper functions e.g. for converting effect sizes, generic pieces

### sample distribution versus population distribution

What's the target population for average effect sizes? 
General issue for prediction and computing average treatment effects, population effects with survey/sampling weights and similar  for which we don't have much code yet.


### Implementation

currently main PR is #6526 but many open issues to parts of this.

- code location: I'm still not sure how far this is going. Currently, I think we can start with a `stats.oneway` but some code will apply or can be extended to other parts and models.

- code structure and duplication: I'm getting lost trying to avoid code duplication and putting everything in accessible and very general functions. So, we duplicate code at least at the beginning and for simple cases.

- multi-purpose functions: There are several different use cases and I'm not sure how to organize it in a reusable way.
One item is to add additional Results statistics to models and hypothesis tests, e.g. LikelihoodModelResults.wald_test has the computation spread out over several functions, but it would be useful to get something directly related to non-centrality.
(I think it would be easy to compute, but not so easy to figure out what and where.)
For power and sample size computation we need non-centrality for planned cases, i.e. we cannot directly use the models, but the models could provide core pieces for the computation. (e.g. average effect size for example sample/population)
specifically for one way anova: We add non-centrality and effect size as result to the hypothesis tests, but we also create standalone function for computing those for planned sample.

user: josef-pkt
issue_6585: state: open
title: ENH: basic meta-analysis methodological issues
body: some methodological details to #1575

Below are two references implying that the basic meta-analysis is not "generic", i.e. independent of what is used as effect measure or parameter to be combined. Referenced and citing articles provide more information (but I didn't look or download those).

The distribution, specifically the expected value, of Q statistic depends on what is used as effect size measure.
This means that this part of the code cannot be completely generic, or needs to get more information about the effect size measure (than what I have right now in my draft code, AFAIR)

Kulinskaya, Elena, and Michael B. Dollinger. 2016. “Commentary on ‘Misunderstandings about Q and “Cochran’s Q Test” in Meta Analysis.’” Statistics in Medicine 35 (4): 501–2. https://doi.org/10.1002/sim.6758.

Hoaglin, David C. 2017. “Practical Challenges of I2 as a Measure of Heterogeneity.” Research Synthesis Methods 8 (3): 254–254. https://doi.org/10.1002/jrsm.1251.

I also haven't checked how much this is relevant for effect size measures or parameters that have been transformed to normality.

user: josef-pkt
issue_6584: state: open
title: ENH: non-inferiority in negative binomial 3-arm trial
body: reference found by chance

Mütze, Tobias, Axel Munk, and Tim Friede. 2016. “Design and Analysis of Three-Arm Trials with Negative Binomially Distributed Endpoints.” Statistics in Medicine 35 (4): 505–21. https://doi.org/10.1002/sim.6738.

includes sample size computation

looks interesting based on very brief skimming

- We don't have anything yet on three-arm trials (but models can be used)
- We don't have any non-inferiority, equivalence testing for count data/models
- We still don't have one-sided test in model Results #1193 #6403

I don't know about computing power and sample size in these cases (I didn't look)

we have some issues for testing rates in two-sample case, e.g. #4882 for NegBin



user: josef-pkt
issue_6583: state: open
title: Chosing the right hypothesis test
body: I have two distributions of volume conservation factors (VCF) Generic and Generic Masked that I want to compare. 
A visualization of the two distributions can be found here: https://imgur.com/a/tgUp7in

The VCF being optimal if equal to 1, I want to show that one distribution is distributed "closer to 1" than the other. Ideally also showing that the contrast variable has no effect on this. 
**Which test is best suited for this purpose?**

I have tried an ANOVA test, but got nonsignificant results, probably because the means are almost equal.

Below is the code for the ANOVA test, which can be found together with the data [here](https://github.com/statsmodels/statsmodels/files/4334016/min_working_example.zip):

```
import pandas as pd
import statsmodels.api as sm
import statsmodels.formula.api as smf

df = pd.read_csv('data.csv')
print('Generic: ')
print(df[['Volume Conservation Factor', 'Processing']].loc[df['Processing'] == 'Generic'].describe(), '\n')
print('Generic Masked: ')
print(df[['Volume Conservation Factor', 'Processing']].loc[df['Processing'] == 'Generic Masked'].describe(), '\n')
dependent_variable='Volume Conservation Factor'
expression='Processing*Contrast'

formula = 'Q("{}") ~ {}'.format(dependent_variable, expression)

ols = smf.ols(formula, df).fit()
print(ols.summary())
print(ols.params)
anova = sm.stats.anova_lm(ols, typ=3)
print(anova)
```

Output:
```
Generic: 
       Volume Conservation Factor
count                   68.000000
mean                     0.963066
std                      0.072723
min                      0.575684
25%                      0.931986
50%                      0.974814
75%                      1.004041
max                      1.083536 

Generic Masked: 
       Volume Conservation Factor
count                   68.000000
mean                     0.966581
std                      0.036193
min                      0.901859
25%                      0.936341
50%                      0.967009
75%                      0.991662
max                      1.053411 

                                   OLS Regression Results                                  
===========================================================================================
Dep. Variable:     Q("Volume Conservation Factor")   R-squared:                       0.063
Model:                                         OLS   Adj. R-squared:                  0.042
Method:                              Least Squares   F-statistic:                     2.977
Date:                             Sun, 15 Mar 2020   Prob (F-statistic):             0.0340
Time:                                     11:16:01   Log-Likelihood:                 200.97
No. Observations:                              136   AIC:                            -393.9
Df Residuals:                                  132   BIC:                            -382.3
Df Model:                                        3                                         
Covariance Type:                         nonrobust                                         
================================================================================================================
                                                   coef    std err          t      P>|t|      [0.025      0.975]
----------------------------------------------------------------------------------------------------------------
Intercept                                        0.9797      0.010    101.947      0.000       0.961       0.999
Processing[T.Generic Masked]                    -0.0018      0.014     -0.130      0.897      -0.029       0.025
Contrast[T.CBV]                                 -0.0333      0.014     -2.450      0.016      -0.060      -0.006
Processing[T.Generic Masked]:Contrast[T.CBV]     0.0106      0.019      0.550      0.583      -0.027       0.049
==============================================================================
Omnibus:                       94.412   Durbin-Watson:                   1.207
Prob(Omnibus):                  0.000   Jarque-Bera (JB):             1112.649
Skew:                          -2.181   Prob(JB):                    2.46e-242
Kurtosis:                      16.317   Cond. No.                         6.85
==============================================================================

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
Intercept                                       0.979713
Processing[T.Generic Masked]                   -0.001771
Contrast[T.CBV]                                -0.033293
Processing[T.Generic Masked]:Contrast[T.CBV]    0.010571
dtype: float64
                        sum_sq     df             F         PR(>F)
Intercept            32.634447    1.0  10393.112525  2.159952e-127
Processing            0.000053    1.0      0.016976   8.965337e-01
Contrast              0.018843    1.0      6.001027   1.560797e-02
Processing:Contrast   0.000950    1.0      0.302506   5.832449e-01
Residual              0.414481  132.0           NaN            NaN

```
user: Jimmy2027
issue_6581: state: open
title: ENH/Design vectorized hypothesis tests
body: #4771 for vectorized OLS
current Results.t_test is vectorized
some tests, e.g. in proportions, are already vectorized

mainly an idea, 
The work will be to go through the hypothesis test and make them work for vectorized cases or check that they do.

context: anova computation. I might use a structure that is not optimal for computational efficiency.

To reduce the overhead in repeated computation, we could vectorize the simple cases.
E.g. in unbalanced anova, we assume that all cases have the same, possibly unbalanced sample sizes, so we have a single groupby, single 1-d group indicator or equivalent, e.g. list of samples is vectorized (list of 2-d arrays instead of list of 1-d arrays)

This would support the case for MonteCarlo or bootstrap resampling, or large experiments with parallel samples with identical sample sizes (biostatistics, ...).

a variation:
For functions based on summary statistics, like `ttest_generic`, `oneway_generic` we might not need identical sample sizes along the vectorization dimension

user: josef-pkt
issue_6576: state: open
title: zenodo entry has only 0.8.0 version as the latest 
body: don't know why, here I see 0.11.1 release and so on. But https://zenodo.org/record/275519#.XmY-T3UpCk8 is only at 0.8.0
user: yarikoptic
issue_6567: state: open
title: New kernel_methods module for new KDE implementation (2)
body: This is a proposal to add a new set of kernel-based methods, for now kernel density estimation.

The main features include:

 - 1D KDE on bounded or unbounded domains, with the possibility to specify the type of bound, and the method to apply at the boundary (cyclic, reflection, log-transformation, ...)
 - nD KDE on bounded or unbounded domains, with both cyclic and reflective boundaries (but not mixing between the two).
 - Non-continuous KDE, for either ordered or non-ordered sets.
 - Multivariate KDE, allowing to mix and match boundary types, including with non-continuous dimensions.
 - FFT and DCT-based calculation for fast grid-based estimations (great for low-dimension problems).
 - Bandwidth estimation by cross-validation, working on all the methods.

This is a new version of https://github.com/statsmodels/statsmodels/pull/2318, as I made a mistake with the branch :/
user: PierreBdR
issue_6566: state: open
title: ENH: kurtosis adjustment for variance tests, Bartlett
body: just a reminder issue (I don't have the references right now)

Bartlett variance test can be improved for non-normal distributions by taking kurtosis into account.

One of the monte carlo articles for #6526  included the kurtosis corrected Bartlett test

There is also a kurtosis correction in multivariate methods that I looked at, another issue (?)
user: josef-pkt
issue_6565: state: open
title: ENH: joint loc and scale hypothesis tests
body: parking a reference

There are similar very recent articles (but I didn't look at any)

sounds interesting based on abstracts, with a "modern" usecase

Staley, James R., Frank Windmeijer, Matthew Suderman, George Davey Smith, and Kate Tilling. 2020. “A Robust Mean and Variance Test with Application to Epigenome-Wide Association Studies.” BioRxiv, February, 2020.02.06.926584. https://doi.org/10.1101/2020.02.06.926584.

related:
#6526 PR for (separate) oneway comparisons for means and variances

user: josef-pkt
issue_6564: state: open
title: ENH: approximate degrees of freedom
body: This looks like an extension of Satterthwaite df or Welch anova df.

Johansen is not a quick read (I don't understand anything which skimming it)

The advantage is that there is a literature that uses general linear restrictions as hypothesis similar to f_test and wald_test in models, instead of the special case formulas in oneway anovas.

Johansen, Soren. 1980. “The Welch-James Approximation to the Distribution of the Residual Sum of Squares in a Weighted Linear Regression.” Biometrika 67 (1): 85–92. https://doi.org/10.2307/2335320.

Lix, Lisa M., and H. J. Keselman. 1995. “Approximate Degrees of Freedom Tests: A Unified Perspective on Testing for Mean Equality.” Psychological Bulletin 117 (3): 547–60. https://doi.org/10.1037/0033-2909.117.3.547.


Eventually it would be good to have a more systematic treatment of df corrections, such as Satterthwaite, Welch and BFMR anova, various corrections in Manova, mixed models (?), ....
(I never tried to figure out the theory behind those)

We don't have any of those for cluster or heteroscedasticity robust cov_type.

An implementation issue:
It would be useful to be able to use models to analyze specific design structures with improved behavior for the special cases/designs.

example: there is also a literature on hypothesis testing for ANCOVA with one or two categorical and one continuous. e.g. comparing growth curves.
That's just a simple OLS with interaction effect, but we might get a better or more robust small sample estimator and inference

user: josef-pkt
issue_6563: state: open
title: ENH: tests comparing outlier robust moment to nonrobust moments
body: 
R package lawstat `sj.test`
"Perform the robust directed test of normality, which is based on the ratio of the classical standard deviation S to the robust standard deviation J (Average Absolute Deviation from the Median, MAAD)
of the sample data. See Gel et al. (2007)."

References
Gel YR, Miao W, Gastwirth JL (2007). “Robust directed tests of normality against heavy-tailed alternatives.”
Computational Statistics&Data Analysis, 51(5), 2734–2746. doi: 10.1016/j.csda.2006.08.022.

This looks interesting. In `describe` (*) I was using mad/var or iqr/var as indicator for the presence of outliers (or heavy tails)

(*) issue ?
I used it for quick summary of data in a notebook with measures to indicate outliers, scaling problems and multicollinearity
user: josef-pkt
issue_6562: state: open
title: ENH : stats for quantiles, median , order statistics
body: I just found a good explanation for confidence intervals for median and percentiles

We don't have any good collection of statistics for order statistics yet.
e.g. confidence intervals for qq-plot, hypothesis test and confidence intervals for median, ...

related model: Quantile Regression, and expectile regression


location for miscellaneous stats functions is `stats.stattools`
for collection of functions that are not many enough to get their own module
Contains currently just a few functions and the robust skew, kurtosis and medcouple

some functions could be moved, for example the normality tests could go to `stats.gof`


user: josef-pkt
issue_6561: state: open
title: ENH: iterator for exog subsets/terms as in wald_test_terms
body: currently just an ides, not sure this works with categorical interactions

Can we create restricted or reduced model by dropping terms or by string matching?

`wald_test_terms` splits up exog into terms or string matched subset to create the restriction matrices for wald tests.

Can we do the same to create a new model?

The advantage would be that we get other measures from the comparison between restricted models and the full model, e.g. change in rsquared for semi-partial correlation

For dropping single columns we can compute semi-partial correlation just from the inverse correlation matrix. see PR multicollinearity
But there is no shortcut for dropping multi-column terms.

other use cases would be likelihood ratio and score tests between reduced and full model, or backward stepwise selection, information criteria for dropping terms 


What I'm not sure about is whether dropping columns is equivalent to dropping the corresponding term from the formula. The change might be in how patsy avoids perfect collinearity when adding terms, e.g. categorical without reference level, or splines without constant.

This might be related to marginality. The meaning changes if we dropp lower order terms.
We could ignore or exclude those cases as I did in wald_test_terms, but I don't know if that rules out interesting cases where the reduced models requires reparameterization.

(maybe a 4-way interaction is in the examples I used for loglinear models (Poisson) might make good test cases to get started)


user: josef-pkt
issue_6559: state: open
title: [DOC] add an example for factor analysis
body: Please consider adding an example for factor analysis along with PCA in the doc.

https://www.princeton.edu/~otorres/Factor.pdf
A general workflow in Stata would be:
- run factor analysis
- rotate
- predict to save factors needed

I tried with `statsmodels.multivariate.factor.Factor` but I was not able to repro my workflow and the `.loadings()` didn't even return anything.

Am I missing something?
user: stevenlis
issue_6558: state: open
title: ENH: power calculation for OLS/WLS
body: (I didn't find an issue. There might be an issue or comments on power under local alternatives.)
**update**
#1758, #2217 issues for local power in likelihood models

I never got started with trying this out.

Here is a relatively simple starting point:

Stata `power onway` uses the OLS analogue to get the noncentrality parameter

reference
O’Brien, R. G., and K. E. Muller. 1993. Unified power analysis for t-tests through multivariate hypotheses. In Applied
Analysis of Variance in Behavioral Science, ed. L. K. Edwards, 297–344. New York: Dekker.
section 8.3 p. 21 looks pretty clear (brief skimming), with several examples using SAS 

I found a copy of that chapter with google search.

getting the anova or t-test analogue for power computation would make an easier case to get started and compare with non-regression results.

**update**
basic function will be in `stats.contrast`, example for OLS oneway anova in unit test.
https://github.com/statsmodels/statsmodels/pull/6526/commits/2e5f7f2191b2d29dbb2fb6d40d65882ffe5de844
user: josef-pkt
issue_6557: state: open
title: ENH: power and sample size for confidence intervals
body: Stata 16 has sample size computation for confidence intervals `ciwidth`, only for 1 or 2 sample mean and 1 sample variance.

The only one we have, is a very simple implementation in `samplesize_confint_proportion` using normal distribution.

No generic support for sample size for confidence intervals.


user: josef-pkt
issue_6555: state: open
title: FAQ: Why are cluster robust p-values so different from those reported by STATA package?
body: I'm running a OLS regression in STATA and the same one in python's Statsmodels. Note that I adjust for clusters (for id and year).

All the outcomes are very similar if not the same. 

But, the Statsmodels *p*-values are off. 

For example, the one for X3 has a *t*-value of 1.951. But Statsmodels assigns a *p*-value of 0.109, while STATA returns 0.052 (as does Excel for 2-tailed tests and df of 573). 

What's going on? 

Below is the output using `import statsmodels.formula.api as sm`, `mod = sm.ols(formula=regression_model, data=data)` and `res = mod.fit(cov_type='cluster', cov_kwds={'groups': np.array(data[[period_id, firm_id]])}, use_t=True)`:


    Statsmodels
    ==============================================================================
    Dep. Variable:                      Y   R-squared:                       0.362
    Model:                            OLS   Adj. R-squared:                  0.355
    Method:                 Least Squares   F-statistic:                     12.90
    Date:                Thu, 27 Feb 2020   Prob (F-statistic):            0.00654
    Time:                        17:16:54   Log-Likelihood:                -1631.8
    No. Observations:                 580   AIC:                             3278.
    Df Residuals:                     573   BIC:                             3308.
    Df Model:                           6                                         
    Covariance Type:              cluster                                         
    ================================================================================
                       coef    std err          t      P>|t|      [0.025      0.975]
    --------------------------------------------------------------------------------
    Intercept       30.4376      4.711      6.461      0.001      18.328      42.548
    X1              -0.3373      0.574     -0.587      0.583      -1.814       1.139
    X2              -1.0456      0.173     -6.043      0.002      -1.490      -0.601
    X3               0.0293      0.015      1.951      0.109      -0.009       0.068
    X4              -0.5298      0.127     -4.186      0.009      -0.855      -0.204
    X5             -14.2937      3.132     -4.564      0.006     -22.344      -6.243
    X6              -1.0144      0.272     -3.733      0.014      -1.713      -0.316
    ==============================================================================
    Omnibus:                      397.145   Durbin-Watson:                   0.485
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):             5643.097
    Skew:                           2.865   Prob(JB):                         0.00
    Kurtosis:                      17.166   Cond. No.                         218.
    ==============================================================================
    
    Warnings:
    [1] Standard Errors are robust to cluster correlation (cluster)
    ==============================================================================
    STATA:
    
    . cluster2 Y X1 X2 X3 X4 X5 X6, fcluster(id) tcluster(year)
     
    Linear regression with 2D clustered SEs                Number of obs =     580
                                                           F(  6,   573) =   32.41
                                                           Prob > F      =  0.0000
    Number of clusters (id) =        99                    R-squared     =  0.3617
    Number of clusters (year) =       6                    Root MSE      =  4.0575
    ------------------------------------------------------------------------------
      man_buffer |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
    -------------+----------------------------------------------------------------
           _cons |   30.43759   4.711003     6.46   0.000     21.18465    39.69053
             X1  |  -.3373264   .5744737    -0.59   0.557    -1.465658    .7910047
             X2  |  -1.045579   .1730091    -6.04   0.000    -1.385388   -.7057694
             X3  |    .029327    .015031     1.95   0.052    -.0001955    .0588496
             X4  |  -.5297968   .1265761    -4.19   0.000    -.7784065    -.281187
             X5  |  -14.29367   3.131845    -4.56   0.000    -20.44497   -8.142374
             X6  |  -1.014446   .2717238    -3.73   0.000    -1.548142   -.4807496
    
    ------------------------------------------------------------------------------
     
    SE clustered by id and year

I run `Statsmodels api:  0.11.0` and `Pandas:  1.0.1`.
user: blucap
issue_6551: state: open
title: ENH: power and sample size function based on smoothed Monte Carlo simulations
body: (Just a thought, I haven't looked for any literature.)

We have the power function for simple anova oneway analysis based on F distribution using effect size. If we move to more robust versions of oneway anova, see #6526, then we might have to use simulations. 

To speed up finding appropriate sample size and power, we might want to smooth the power function on an initially coarse grid.

I think we have the problem that we don't use the same random generated samples across simulations because we need to change the sample size. This adds noise to the simulated power function. If we smooth it, then we don't need to increase the number of replications in the simulation to be large enough to make the "replication noise" negligible.

E.g. we can use splines or penalized splines (GAM). 
Unfortunately, we don't have monotonic spline fitting available yet.

Related: response curve fitting for Monte Carlo p-values or critical values. 
But for power functions we don't tabulate and we don't need a parsimonious polynomial approximation, we have to compute each case as we need it.
user: josef-pkt
issue_6550: state: open
title: TST/REF: pairwise wilcox test with generic multiple testing correction
body: R stats seems to have just a generic pairwise version for wilcox
https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/pairwise.wilcox.test

My guess is that this should then be the same as in sandbox Multicomparison.
AFAIR, I tried it using scipy function on each pair, but didn't have reference package for unit tests.

needs unit tests and cleanup, refactoring, enhancement
user: josef-pkt
issue_6541: state: open
title: libqsturng _psturng fails when degrees of freedom is strictly between 1 and 2.
body: #### Describe the bug
stats.libqsturng._psturng errors out if the degrees of freedom `v` is strictly between 1 and 2.  It handles both `v = 1` and `v >= 2`, but because of these lines: first where it tests [`v==1`](https://github.com/statsmodels/statsmodels/blob/b8094ac80f1fbc2e55f68293b143d29598393521/statsmodels/stats/libqsturng/qsturng_.py#L827), but then the else tries to run [`_qsturng(.1, r, v) `](https://github.com/statsmodels/statsmodels/blob/b8094ac80f1fbc2e55f68293b143d29598393521/statsmodels/stats/libqsturng/qsturng_.py#L835) which if `1 < v < 2` [returns an error.](https://github.com/statsmodels/statsmodels/blob/b8094ac80f1fbc2e55f68293b143d29598393521/statsmodels/stats/libqsturng/qsturng_.py#L680-L681)

I don't think this is the intended behavior since the code handles degrees of freedom 1 and 2 correctly.  I'm still investigating, but I think the correct behavior is to set the previous `v==1` to `1 <= v < 2`.  The code below reproduces the error.  While likely rare as the example uses sample sizes of 2 and 3 with the Satterthwait degrees of freedom, it does happen, and I think the code should handle it.  I'm working on a PR for this bug.  

#### Code Sample, a copy-pastable example if possible
Note this code sample includes a dataframe with more groups than needed to reproduce the error.  This is so that it is easy for other to see that the underlying issue is not just the sample size, but sample size combined with standard deviation that leads to Satterthwait degrees of freedom of ~1.01 which generates the error.  

Also I include a function for computing the Sattherthwait degrees of freedom.  One can also use `sandbox.stats.multicomp.varcorrection_pairs_unequal` where `df_all` is set to be `1/n-1` where `n` is the number of observations in the group for each group considered.  

I ran the following code both on the latest statsmodels released version on the current master.  I also asked about this in the forum and searched the issue tracker and believe this is a new issue. 

```python
import pandas as pd
import numpy as np
from statsmodels.stats.libqsturng import psturng

def degrees_of_freedom(v_1, v_2, n_1, n_2):
    """
    Helper function for computing the Satterthwait degrees of freedom/
    Args:
        v_1 (float): The sample variance for one group.
        v_2 (float): The sample variance for the second group.
        n_1 (float or int): Number of observations for group with standard deviation s_1.
        n_2 (float or int): Number of observations for group with standard deviation s_2.

    Returns:
        (float): Degrees of freedom for use in Games-Howell (also works for Welch's t-test).

    """
    numerator = (v_1 / n_1 + v_2 / n_2) ** 2
    denominator = ((v_1 / n_1) ** 2) / (n_1 - 1) + ((v_2 / n_2) ** 2) / (n_2 - 1)
    return numerator / denominator

test_libqsturng = pd.DataFrame(
    {
        'grp':[1, 1, 1, 1, 2, 2, 3, 3, 4, 4, 4], 
        'obs':[230, 300, 50, 150, 200, 150, 175, 170, 180, 175, 176]
    }
)
summary = test_libqsturng.groupby('grp')['obs'].agg(["mean", "std", "var", "count"])
var_1 = summary.iloc[1]['var']
var_2 = summary.iloc[3]['var']
n_1 = 2
n_2 = 4
difference = summary.iloc[1]['mean'] - summary.iloc[3]['mean']

degrees_freedom = degrees_of_freedom(var_1, var_2, n_1, n_2)
stderr = np.sqrt(0.5 * (var_1/ n_1+ var_2 / n_2))
q_stat = abs(difference) / stderr
p_value = psturng(q_stat, 4, degrees_freedom)  # 4 because there are 4 total groups in the original dataset.
```
### Expected Output
I expected `p_value` to complete, not error out and provide a useful p_value (in this case, likely large because the means within both groups standard deviations.   

#### Output of ``import statsmodels.api as sm; sm.show_versions()``
Below are the results of this line for my locally installed version of master.  I can also upload the same information for the most recent release of statsmodels, if requested.  Seems like a lot of info to load both. 

<details>
INSTALLED VERSIONS
------------------
Python: 3.7.3.final.0
OS: Darwin 18.7.0 Darwin Kernel Version 18.7.0: Tue Aug 20 16:57:14 PDT 2019; root:xnu-4903.271.2~2/RELEASE_X86_64 x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

statsmodels
===========

Installed: v0.12.0.dev0+86.g04d867d9f (/Users/ataylor/z/statsmodels/statsmodels)

Required Dependencies
=====================

cython: 0.29.15 (/Users/ataylor/.pyenv/versions/3.7.3/envs/statsmodels/lib/python3.7/site-packages/Cython)
numpy: 1.18.1 (/Users/ataylor/.pyenv/versions/3.7.3/envs/statsmodels/lib/python3.7/site-packages/numpy)
scipy: 1.4.1 (/Users/ataylor/.pyenv/versions/3.7.3/envs/statsmodels/lib/python3.7/site-packages/scipy)
pandas: 1.0.1 (/Users/ataylor/.pyenv/versions/3.7.3/envs/statsmodels/lib/python3.7/site-packages/pandas)
    dateutil: 2.8.1 (/Users/ataylor/.pyenv/versions/3.7.3/envs/statsmodels/lib/python3.7/site-packages/dateutil)
patsy: 0.5.1 (/Users/ataylor/.pyenv/versions/3.7.3/envs/statsmodels/lib/python3.7/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: Not installed
cvxopt: Not installed
joblib: Not installed

Developer Tools
================

IPython: 7.12.0 (/Users/ataylor/.pyenv/versions/3.7.3/envs/statsmodels/lib/python3.7/site-packages/IPython)
    jinja2: 2.11.1 (/Users/ataylor/.pyenv/versions/3.7.3/envs/statsmodels/lib/python3.7/site-packages/jinja2)
sphinx: Not installed
    pygments: 2.5.2 (/Users/ataylor/.pyenv/versions/3.7.3/envs/statsmodels/lib/python3.7/site-packages/pygments)
pytest: 5.3.5 (/Users/ataylor/.pyenv/versions/3.7.3/envs/statsmodels/lib/python3.7/site-packages/pytest)
virtualenv: Not installed
</details>

user: ameliajtaylor
issue_6537: state: open
title: Error when getting variance decomp from SVAR in statsmodels
body: 0


I have set up an SVAR model with statsmodels.SVARResults. When I attempt to get the variance decomp it throws error or it just returns the memory location. I have tried two ways.

import numpy as np
import numpy as np
import statsmodels.api as sm
from statsmodels.tsa.api import VAR, SVAR
import matplotlib.pyplot as plt

df1 = DF[[various data fields]]



A = np.asarray([[1, 0, 0],[0, 1, 0],['E', 'E', 1]])
B = np.asarray([['E', 0, 0], [0, 'E', 0], [0, 0, 'E']])
A_guess = np.asarray([0.5, 0.25, -0.38])
B_guess = np.asarray([0.5, 0.1, 0.05])
mymodel = SVAR(df1, svar_type='AB', A=A, B=B)
res = mymodel.fit(maxlags=1, maxiter=10000, maxfun=10000, solver='bfgs')



res.fevd()
returns this error:

TypeError: irf() got an unexpected keyword argument 'var_decomp'
When I try it this way:

res.fevd
I simply get the memory location of the instance
user: Havlin
issue_6532: state: open
title: Data shape is chaged after imputation
body: Hello,
I am imputing data using `MiceData()`
Before imputing data, `num_data` has shape: `(30000, 14)`
After imputing `imp_data` has shape: `(29537, 14)`
Here is the code:
`imp_data = mice.MICEData(num_data)`

I dont know why I am missing some rows after imputing, Please help!
user: harrytrinh96
issue_6530: state: open
title: ENH: support use_t in RLM
body: RLM currently has no option to base inference on the t-distribution instead of normal distribution.
It is missing the `use_t` option in `fit` and hard codes normal distribution in Results methods.

related: #1379 and similar for better and more consistent `cov_type` support

usecase:
For PR #6526, I was comparing RLM estimates with two sample trimmed t-test or anova, which use the t or F distribution.
It would be better to use the same distribution also in RLM.

user: josef-pkt
issue_6523: state: open
title: ENH/SUMM ranked data regression, hypothesis test versus nonparametric, rank based test
body: Using ranked data in regression or standard hypothesis tests is close to their rank based counterparts in hypothesis tests.

example in ` statsmodels.sandbox.stats.multicomp.MultiComparison` I had added conversion to ranks.

However, there are some differences in the results.
some links for that

https://eigenfoo.xyz/tests-as-linear/#6.1.4-Python-code:-Kruskal-Wallis (contains many comparisons of models and hypothesis tests)
based on 
https://lindeloev.github.io/tests-as-linear/
which has a good set of links
https://lindeloev.github.io/tests-as-linear/#8_sources_and_further_equivalences

some stats.stackexchange questions and answers and similar
https://stats.stackexchange.com/questions/210529/are-parametric-tests-on-rank-transformed-data-equivalent-to-non-parametric-test
https://stats.stackexchange.com/questions/303269/common-statistical-tests-as-linear-models

https://seriousstats.wordpress.com/2012/02/14/friedman/
with reference to 
Conover, W. J., & Iman, R. L. (1981). Rank transformations as a bridge between parametric and nonparametric statistics. American Statistician, 35, 124-129.


todo:
check kruskal in statsmodels.sandbox.stats.multicomp.MultiComparison again and see how far we can get with using standard methods with rank data.

user: josef-pkt
issue_6516: state: open
title: ENH: AIC for M-estimators, QMLE, over dispersion, effective degrees of freedom
body: Claeskens, Gerda, and Nils Lid Hjort. "Model selection and model averaging." Cambridge Books (2008).

chapter 2 has a good discussion of AIC including for the cases of misspecified likelihood and M-estimators.

The penalization uses `J^{-1} H` for the number of parameters, which is the same or similar to effective degrees of freedom in penalized estimation (IIRC)

Takeuchi information criterion TIC is the misspecification robust version of AIC, similar to sandwich cov_types.
Search for it finds relevant theoretical literature.

GAM uses effective degrees of freedom, `edf` as sum of hat_matrix_diag (i.e. trace of hat matrix), so it doesn't use a generic version using hessian and score_cov.

other relevant area is Influence classes GLM, MLE and generic, where I might have used something similar. Issues? 
user: josef-pkt
issue_6513: state: open
title: CLN/ENH: Give user more control in knockoff
body: Refactor some key steps to separate methods that can be overridden to modify behavior.  Add a hook to allow user-supplied knockoff matrix construction.
user: kshedden
issue_6512: state: open
title: ENH: equivalence testing for regression curves
body: mainly to park a reference to a new topic

Dette, Holger, Kathrin Möllenhoff, Stanislav Volgushev, and Frank Bretz. 2018. “Equivalence of Regression Curves.” Journal of the American Statistical Association 113 (522): 711–29. https://doi.org/10.1080/01621459.2017.1281813.

There is some recent literature on testing and comparing regression curves for equivalence, e.g. comparing equivalence between two growth curves.
There are several articles in this area but I didn't look into any details.
Some articles use explicit distributions for test statistic, but it seems they prefer now bootstrap.

related: simultaneous confidence intervals #2955

related: there is some older literature on testing functional form against non-parametric alternatives, but those use mostly kernel regressors and look pretty difficult and computing intensive. (Dette is one author in that area)
user: josef-pkt
issue_6510: state: open
title: ENH/REF/Design: make smoothers in GAM lazy
body: Just an idea, I don't know if it is worth it.

The smoother classes for GAM compute the main things in the `__init__`. Smoother instances need to be created before creating a GAM instance.
This currently rules out missing value handling because we cannot create a common missing mask for dropping observations before creating instances and do some computation.

With a lazy smoother class, we could do the computation when we are in the `model.__init__` and have already treated `missing` values.

more general case: supporting `exog_extra`. #5469 and related
user: josef-pkt
issue_6503: state: open
title: ENH: effect size standardized mean difference exact distribution
body: standardized mean difference as presented in meta-analysis has a bias correction and a "funny" variance.
AFAICS, those come from the non-central t distribution

Hedges and Olin 1985 
section B3 EXACT CONFIDENCE INTERVALS FOR EFFECT SIZES p.91
Technical commentary p. 104  for exact Expected value and Variance 
standard presentation of effect size p. 78ff

properties of non-central t-distribution
https://en.wikipedia.org/wiki/Noncentral_t-distribution#Moments_of_the_noncentral_t-distribution
which includes the `J` bias correction term which is an approximation to the expression with gamma function.

I'm not sure we want to add anything with noncentral t distribution directly in these cases.
Meta analysis, as far as I have skimmed it, uses normal approximation.
There might be asymmetry and fatter tails in small samples.
I guess we might add at least `use-t` option by the usual small sample approximation argument (even if we use symmetric central instead of non-central t-distribution)

(The references mention that bias correction is only non-negligible if sample size is <=10)

scipy stats has non-central t distribution, but so far I only used the parts needed for power and sample size computation.
user: josef-pkt
issue_6500: state: open
title: ENH: excess dispersion, misspecification, QMLE in meta-analysis
body: (just a thought while skimming Viechtbauer 2016)

When combining separate estimators with WLS, scale should be 1
https://github.com/statsmodels/statsmodels/issues/1575#issuecomment-583196361

When using a random effects model, the random effect captures between heterogeneity that goes beyond within sampling variance (*)

In proportions/count models like binary outcomes, the within study effect size variance is computed under the assumption that there is no excess dispersion, e.g. for risk difference, risk ratio and odds ration.

So, if there is excess dispersion, then fixed effect would ignore it and random effect would attribute the extra variance to the between random effect.

(*)
Viechtbauer, Wolfgang. 2016. “Bias and Efficiency of Meta-Analytic Variance Estimators in the Random-Effects Model:” Journal of Educational and Behavioral Statistics, November. https://doi.org/10.3102/10769986030003261.


I'm not sure what this implies for implementation.
I guess scale=1 should be optional when using WLS style combination, or we need robust standard errors, whatever they are in this case.


user: josef-pkt
issue_6499: state: open
title: Reference: variance stabilizing transform, effect size and meta analysis
body: mainly to park a reference to a book

Kulinskaya, Elena, Stephan Morgenthaler, and Robert G Staudte. 2008. Meta Analysis a Guide to Calibrating and Combining Statistical Evidence. Chichester, West Sussex; Hoboken, NJ: John Wiley & Sons.

uses variance stabilizing transform to create standardized effect sizes
first part has a large collection of standard hypothesis tests for continuous, proportion/Binomial and count data
second part combines statistics and "evidence" in meta-analysis
many examples
only 180 citations in google scholar
(I only skimmed a few parts)

related: 
#1575 meta analysis 
#2802 combining hypothesis tests and estimates 
#3224 transformation to normality

user: josef-pkt
issue_6498: state: open
title: ENH: MOVER confidence interval for two sample comparison
body: MOVER = Method Of Variance Estimates Recovery

An issue search only finds this comment
https://github.com/statsmodels/statsmodels/issues/2955#issuecomment-220075363


Zou, G. Y., and A. Donner. 2008. “Construction of Confidence Limits about Effect Measures: A General Approach.” Statistics in Medicine 27 (10): 1693–1702. https://doi.org/10.1002/sim.3095.

Newcombe, Robert G. 2013. “MOVER-R Confidence Intervals for Ratios and Products of Two Independently Estimated Quantities:” Statistical Methods in Medical Research, October. https://doi.org/10.1177/0962280213502144.

Zou and Donner construct a confidence interval on the difference between parameter estimates from two independent samples based on the two separate confidence intervals of the two individual parameters.
application to risk ratio and lognormal mean (The latter case seems to have many citations)

I'm pretty sure, I implemented something like this already once, but just to try it out, and I don't remember the context.
I guess we can add a generic function based on Zou and Donner. I didn't look much at Newcombe. He adjusts the Zou Donner approach for ratios when there can be a sign change.





user: josef-pkt
issue_6497: state: open
title: FAQ/DOC/SUMM: overview equivalence tests
body: (I wanted to add an ENH issue for one sample TOST, but we already have several)

## One-Sample Equivalence tests

- `statsmodels.stats.weightstats.DescrStatsW.ttost_mean`
- `statsmodels.stats.weightstats.DescrStatsW.ztost_mean`
- `statsmodels.stats.weightstats.ztost` combines one and two sample
- `statsmodels.stats.proportion.proportions_ztost` one and two sample (says only checked for 1 sample)
- `statsmodels.stats.proportion.binom_tost`
- `statsmodels.stats.proportion.proportions_ztost` one and two sample

## Two independent samples

- `statsmodels.stats.weightstats.CompareMeans.ttost_ind`
- `statsmodels.stats.weightstats.CompareMeans.ztost_ind`
- `statsmodels.stats.weightstats.ttost_ind`
- `statsmodels.stats.weightstats.ztost` one and two sample, different signature than `ttost_ind`
- `statsmodels.stats.proportion.proportions_ztost` one and two sample (says only checked for 1 sample)

## Two paired samples

- `statsmodels.stats.weightstats.ttost_paired`

## k independent samples equivalence tests

some functions for oneway merged in #6789 (Welleck equivalence test for anova)

## Power

- `statsmodels.stats.proportion.power_binom_tost`  NO docstring !
- `statsmodels.stats.proportion.power_ztost_prop`


## Noninferiority and superiority with a margin

These are just one-sided tests with `value` as the margin available in the main hypothesis test functions. There are no specific functions for this, e.g. TOST functions call the two one-sided tests.

## missing

- basic: counts, Poisson
- more methods and options for two sample proportions similar to tests and confidence interval options
- anova/k-sample comparison issue #6444, merged in #6789
- equivalence testing for correlation
- equivalence testing for variances https://github.com/statsmodels/statsmodels/issues/667#issuecomment-1108844804
- ...
user: josef-pkt
issue_6496: state: open
title: ENH: power for Welch's ANOVA, unequal variance
body: We have one version of power computation for one-way ANOVA `FTestAnovaPower`.
This can be extended or complemented with power under the assumption that variance is not the same across groups.

Jan, Show-Li, and Gwowen Shieh. 2014. “Sample Size Determinations for Welch’s Test in One-Way Heteroscedastic ANOVA.” British Journal of Mathematical and Statistical Psychology 67 (1): 72–93. https://doi.org/10.1111/bmsp.12006.

looks good, but I didn't check the simulations

---
related aside: 
Docs for `FTestAnovaPower` are not very good, e.g. I have no idea anymore how to use it and how to compute effectsize for anova.
The class has a special method `_solve_effect_size` which is still labeled as "experimental". I don't know if there are problems left with rootfinding using current scipy.

**update**
`FTestAnovaPower` has unit test against a GPower example
http://www.psychologie.hhu.de/fileadmin/redaktion/Fakultaeten/Mathematisch-Naturwissenschaftliche_Fakultaet/Psychologie/AAP/gpower/GPowerManual.pdf p.25
effect size is Cohen's `f` which is `sqrt(variance_between / variance_within)`
assumes homogeneous variance, equal sample size in power computation (or hidden in effect size)
PASS also seems to have only homogeneous variance versions for anova power


user: josef-pkt
issue_6494: state: open
title: ENH: multiple testing p-value correction for discrete sample space
body: The p-values are not uniformly distributed under the Null when the sample space is discrete as in Binomial with small counts.

Bonferroni, Holm and Hochberg can be improved by taking the actual distribution into account.

Zhu, Yalin, and Wenge Guo. 2019. “Family-Wise Error Rate Controlling Procedures for Discrete Data.” Statistics in Biopharmaceutical Research, September. https://www.tandfonline.com/doi/abs/10.1080/19466315.2019.1654912.

and R package https://rdrr.io/cran/MHTdiscrete/man/MHTdiscrete.html

I didn't read the article and don't know if we can gain much power with this, nor how difficult it is to implement the discrete distribution of the p-values.
user: josef-pkt
issue_6488: state: open
title: ENH/DOC: review variance used in proportions_ztest
body: https://stats.stackexchange.com/questions/447462/why-does-the-proportions-ztest-function-in-statsmodels-produce-different-values

default is variance based estimated proportion, i.e. Wald test
SAS uses score test, variance based on null proportion, but only for the one sample test equal null value
SAS also has noninferiority test for one sample, where they use estimated proportion
https://support.sas.com/documentation/cdl/en/statug/63962/HTML/default/viewer.htm#statug_freq_a0000000563.htm

two sample case in SAS is similar to my two proportion PRs, i.e. more options but default seems to be wald test
https://support.sas.com/documentation/cdl/en/statug/63962/HTML/default/viewer.htm#statug_freq_a0000000564.htm


Main change would be to have an easier option for score test (variance under the null) and improved documentation.

I don't like switching automatically between wald and score_tests for different cases in the same function, here one-sample versus two-sample case, even if score test is more popular for one sample and wald test is more popular for two samples.

One possibility would also be to split up the function into separate one sample and two sample functions.

Also, the function uses pooled proportion for the variance computation in two sample case
`p_pooled = np.sum(count) * 1. / np.sum(nobs)` 

The two sample case might need more option, similar to 
https://github.com/statsmodels/statsmodels/pull/4829/files#diff-36a55e208f6d1db357321ee87ef3ab77R1438
or remove the two sample case from proportion_ztest

(I haven't looked at this function in a very long time)
user: josef-pkt
issue_6486: state: open
title: Zero-Inflated Negative Binomial: during the fit, add “prior” on parameters values.
body: Hi all,

I am trying to fit a distribution to some discrete count data that has a lot of zeros. 
ZIP and ZINB seem like a good ideas. ZIP can be used when there is no over-dispersion and ZINB can be used when there is some.

While playing with the ZINB, I noticed that multiple set of parameters could end up in distributions that are very similar. Example:

`
import numpy as np
from statsmodels.distributions import zipoisson, zigenpoisson, zinegbin

n_samples = 1000
d_1 = zinegbin(mu=2,   w=0.7, p=2, alpha=0.8).rvs(n_samples)
d_2 = zinegbin(mu=0.7, w=0.3, p=2, alpha=3  ).rvs(n_samples)


import matplotlib.pyplot as plt

lim = 25

a_heights, a_bins = np.histogram(d_1, bins=[i for i in range(lim)])
b_heights, b_bins = np.histogram(d_2, bins=[i for i in range(lim)])
a_heights, b_heights  = a_heights/n_samples, b_heights/n_samples

fig, ax = plt.subplots(figsize=(10,5))
width = (a_bins[1] - a_bins[0])/3
ax.bar(a_bins[:-1], a_heights, width=width, facecolor='black',label='mu=2,   w=0.7, p=2, alpha=0.8')
ax.bar(b_bins[:-1]+width, b_heights, width=width, facecolor='cornflowerblue', label='mu=0.7, w=0.3, p=2, alpha=3')
plt.legend();
`

I was wondering if there is a easy way of introducing some kind of prior on parameters to guide the model towards one set of parameters rather than another. 

I am multiple fit to do on similar but slightly different datasets and I end up with very different `w` values after the fit.

Thanks !



user: Bolzano-Weierstrass
issue_6485: state: open
title: Seasonal component forecast for X13 analysis
body: 
#### Is your feature request related to a problem? Please describe
Hi Guys,
I'm trying to get the seasonal component forecast from the statsmodels.tsa.x13. I'm using this information to generate forecasts for the seasonally adjusted series.

In statsmodels.tsa.x13.x13_arima_analysis I set the forecast_period=60. As default the code does not return seasonal component forecast as a separate object (or I'm not able to identify it)
I change the code of x13.py to get the d10 (seasonal factor) historical figures, and it's working fine. However I'm not sure how to add seasonal component forecast to this. 
[x13.txt](https://github.com/statsmodels/statsmodels/files/4138738/x13.txt)

#### Describe the solution you'd like
My x13.py code s able to extract d10 historical figures what I need is to somehow extract also the forecast of these figures

#### Describe alternatives you have considered
Looking at the docX13AS.pdf documentation I'm trying to identify the right table and variable that contains the seasonal component forecast but for now with no success
[docX13AS.pdf](https://github.com/statsmodels/statsmodels/files/4138702/docX13AS.pdf)

user: MarcinTom
issue_6479: state: open
title: add methods to get the margins and maginal plot for RegressionResults
body: We have `.get_margeff` for `LogitResults`. Please consider adding the same method for `RegressionResults` so that the users can explore the margins for continuous variables (such as their interactions) in an ols model.

https://www.statalist.org/forums/forum/general-stata-discussion/general/1324886-continuous-interaction-variables-and-margins

https://stats.idre.ucla.edu/stata/faq/how-can-i-explain-a-continuous-by-continuous-interaction-stata-12/

Currently I wonder if there could be a workaround.
user: stevenlis
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_6464: state: open
title: FAQ/ENH: Discrepencies in two-way MANOVA between statsmodels and SPSS
body: I am using Statsmodels (version 0.10.0) to compute a two-way MANOVA.

My model looks like this:

`type_gend= MANOVA.from_formula('type_home_collection + type_kerbside_collection + type_staffed_dop ~  C(PROD_TYPE) * C(GEND) -1', data=df)`
`print(type_gend.mv_test())`

When working on it, I realized some weird things
- Results differ depending on which factor I place first in the model. This is true also when I write the full formula (`type_home_collection + type_kerbside_collection + type_staffed_dop ~  C(PROD_TYPE) + C(GEND) + C(PROD_TYPE):C(GEND) -1`)
- Results are different from what I get in SPSS: the interaction effect is similar, but main effects are not

I checked and my input data is exactly the same in both cases (I actually exported my DataFrame from python).

Unfortunately I'm neither good enough in statistics nor in programming to solve that, but I hope this may help...
user: JMansuy
issue_6444: state: open
title: ENH: equivalence testing for ANOVA, k-sample comparison
body: specific case for #6404 
looks pretty straightforward to implement

Wellek and Wellek-Welch ANOVA with equivalence test.
Equivalence is defined in terms of the test statistic, approximately ratio of between variance and within variance. Distribution in non-central F where non-centrality parameter depends on equivalence margin.

There are also other methods, e.g. pairwise t-test equivalence testing and maybe some variation on tukey-hsd and looking at standard confidence intervals directly.


Cribbie, Robert A., Chantal A. Arpin-Cribbie, and Jamie A. Gruman. 2009. “Tests of Equivalence for One-Way Independent Groups Designs.” The Journal of Experimental Education 78 (1): 1–13. https://doi.org/10.1080/00220970903224552.

Koh, Andy, and Robert Cribbie. 2013. “Robust Tests of Equivalence for k Independent Groups.” British Journal of Mathematical and Statistical Psychology 66 (3): 426–34. https://doi.org/10.1111/j.2044-8317.2012.02056.x.

Shieh, Gwowen. 2016. “A Comparative Appraisal of Two Equivalence Tests for Multiple Standardized Effects.” Computer Methods and Programs in Biomedicine 126 (April): 110–17. https://doi.org/10.1016/j.cmpb.2015.12.004.


aside: where is the code for Welsh's ANOVA?
It looks like the test statistic is the same for the standard equality hypothesis and the equivalence hypothesis.



user: josef-pkt
issue_6437: state: open
title: Plotting Maximum posterior deviates from BinomialBayesMixedGLM?
body: In Bambi which uses PyMC3, it's possible to plot a smoothed histogram (using kernel density estimation) of the marginal posteriors of each stochastic random variable. For example see https://docs.pymc.io/notebooks/getting_started.html#Posterior-analysis.

I'm wondering if some similar post-hoc model evaluation is possible with the random effects one fits using BinomialBayesMixedGLM? Does the algorithm estimate the marginal deviates internally in such a way that they could then be plotted?  The question one is often interested in after fitting such a model is how good are my assumptions of Normality for the random effects i.e. do the marginal distributions of the random effects appear unimodal and symmetric or are they multi-modal or highly skewed? 

FYI, I've been generating ROC curve and AUC statistics as coarse-grained way of assessing my model fits, but I'm wondering if you have other recommendations about how to evaluate different candidate models when the fitting algorithm is BinomialBayesMixedGLM? 

Thanks for your time.
user: mray271
issue_6436: state: open
title: ENH: splines with specific boundary values (predicted at boundaries)
body: I would like to have a spline that starts with predicted value of 0 and ends at predicted value of 1 (outside of x-range) with slope (or asymptotic slope) equal to zero.

example spline fit for reset test, calibration curve #6435

e.g. 
```
res_spline = Logit.from_formula('y ~ bs(linpred, df=4)', {'y': endog, 'linpred': linpred}).fit()
fitted_spline = res_spline.predict()
```

AFAIR from GAM, patsy cannot do this, but our splines in `gam` are more flexible in choosing boundary knots.
But I don't remember how to do this. In GAM, I added a case where the spline is linear at the boundary knots, outside of support of `x` data.

I think patsy has options set boundary slope to zero.

I'm not sure how we can impose specific y-values (especially when we already have linear transformation for constant removal)

user: josef-pkt
issue_6435: state: open
title: ENH: regression plots for binary endog
body: 
a quick prototype for a Logit plot observed versus predicted, using lowess as smoother (local averaging)
see #6430 (calibration) #6434 plot elements like add_lowess

```
fig = plt.Figure(figsize=(8, 6))
ax = fig.add_subplot(1, 1, 1)
ax.plot(fitted[sort_idx], endog[sort_idx], '|')
fig = add_lowess(ax)
#ax.plot(fitted[sort_idx], fitted[sort_idx], color='k', alpha=0.5)
ax.plot([0, 1], [0, 1], color='k', alpha=0.5)
ax.set_xlim(0, 1)
ax.set_ylim(0, 1)
ax.set_title("Logit - observed versus predicted")
ax.set_xlabel('predicted probability')
ax.set_ylabel('response')
fig
```
user: josef-pkt
issue_6434: state: open
title: ENH: add more premade plot elements like `add_lowess`
body: `add_lowess` is very convenient, just one line to add it to a new plot

e.g. smoothed endog for Logit as a function of predicted probabilities
(usecase calibration or reset plots #6430 or regression plot for models like Logit, GLM )

```
from statsmodels.graphics.regressionplots import add_lowess
plt.plot(fitted[sort_idx], endog[sort_idx], '|')
ax = plt.gca()
add_lowess(ax),
```

We also have adding a line in prob plots `qqline` which includes a regression line, but is written specifically for qq plots (includes also 45 degree line)

Other convenient ones would be fitted polynomials or splines for which we would have some confidence intervals.



user: josef-pkt
issue_6432: state: open
title: ENH: ultrahigh screening, add aic, gcv and similar to history
body: Having `aic` or `gcv` available for the models that were estimated during model selection will give as an extra check or evaluation of all or top candidate models.

#6268 for aic/bic based model selection, e.g. all subset models, or stepwise.
also useful for model averaging based on smoothed aic

Penalized SCAD evaluates in-sample. AIC or GCV would be approximations to out of sample behavior.
Proper AIC for penalized needs effective degrees of freedom, but for variable screening the AIC for how many variables are included would also be already useful.

Currently ultrahigh screening does not have selection of the penalty parameter, AFAIR.
In a refinement step, we could find optimal gcv, aic, k-fold cv penalty parameters for the top models with penalized SCAD, (similar to GAM)




user: josef-pkt
issue_6430: state: open
title: ENH: graphic diagnostic binary endog - performance of predicted probabilities
body: Looks interesting (partial skimming)

perfomance of Logit or similar in terms of predicted probabilities and how well they match observered proportions, proportions/frequencies by exog pattern or by some kind of grouping

I think this is mostly a special case of #4262 for binary response models, 
for count data I have the plots for average prediction in a notebook

Van Calster, Ben, Daan Nieboer, Yvonne Vergouwe, Bavo De Cock, Michael J. Pencina, and Ewout W. Steyerberg. 2016. “A Calibration Hierarchy for Risk Models Was Defined: From Utopia to Empirical Data.” Journal of Clinical Epidemiology 74 (June): 167–76. https://doi.org/10.1016/j.jclinepi.2015.12.005.

(There likely is a larger literature, I just ran into this by chance, related articles to something else)



user: josef-pkt
issue_6429: state: open
title: ENH tools: matrix completion method for imputation 
body: https://github.com/iskandr/fancyimpute in minimal maintenance mode
(found via http://www.xavierdupre.fr/app/ensae_teaching_cs/helpsphinx/ml2a/td2a_mlbasic_valeurs_manquantes.html )

I recently saw that Susan Athey has a working paper on arxiv that uses matrix completion methods to impute counterfactual values for treatment effects estimation. (I didn't read it)

Maybe there already exist some methods in python that can be used as tools.

user: josef-pkt
issue_6428: state: open
title: ENH: gof cramer von mises test distribution, skgof
body: https://github.com/wrwrwr/scikit-gof/tree/master/skgof
looks abandoned 
Has approximation for distribution of test statistic (for testing gof with fully specified distribution), KS, AD, CvM
The only one not in scipy or statsmodels might be CvM https://github.com/wrwrwr/scikit-gof/blob/master/skgof/cvmdist.py

user: josef-pkt
issue_6421: state: open
title: ENH: permutation tests with nonzero difference
body: I'm not sure we can do this.

In the two sample case under the null that the two samples have the same mean, `H0: m1 = m0`, we can compute the distribution of the test statistic using permutation.

In general we are allow for a nonzero difference in the Null when possible, i.e. `H0: m1 = m0 + d`, where d is the `value`. This allows us to compute inferiority, superiority and equivalence tests.

Question, can we use permutation test for this null by subtracting the diff from one sample and then permute the transformed observations?

x1 = x1  # unchanged
x0t = x0 + d
E(x1) = m1
E(x0t) = E(x0) + d = m0 + d
H0: E(x1) = E(x0) + d

(sounds fine)

related
older issues
#2216
#3571
#2324
and more recently, simulation methods for contingency tables #3576

#4956 general question about status of resampling methods






user: josef-pkt
issue_6418: state: open
title: TST/DOC/ENH: fit_constrained with weights, WLS
body: https://stackoverflow.com/questions/59747310/weighted-restricted-constrainted-least-square-algebra-in-python

I'm not sure what we have.

Most likely no unit tests for `GLM.fit_constrained` with user specified weights.

Do we need to adjust the reparameterization of exog for weights? 
maybe not for var_weights, but likely for freq_weights.

related: #1749 fit_constrained follow-up (but mainly link to merged PR)

user: josef-pkt
issue_6413: state: open
title: ENH: fitting power/sample size functions from simulations
body: see also #3187 that mentions simulation surface fitting.
However, AFAICS we don't have measurement errors, errors in variables in the case here.

Maruo, Kazushi, Keisuke Tada, Ryota Ishii, and Masahiko Gosho. 2018. “An Efficient Procedure for Calculating Sample Size Through Statistical Simulations.” Statistics in Biopharmaceutical Research 10 (1): 1–8. https://doi.org/10.1080/19466315.2017.1349689.
(I have accepted author manuscript)

Essentially, they use probit to fit the power function (probability of rejection) as function of sqrt(nobs) so they need fewer simulations to compute the sample size for given power.

There are also some articles (which I don't remember now) that use curve fitting to simulations or bootstrap to get better p-values in the tails.

a bit related to design of computer experiments `doe`, except here we fit statistical functions.

splines would be an alternative, however in cases like power and pvalues we have monotonic functions and we don't have yet (semi-) nonparametric methods that impose monotonicity


user: josef-pkt
issue_6409: state: open
title: BUG: exception in qsturng, KeyError
body: no idea what this means, I'm not familiar with the details of the code
qsturng is used for the distribution of tukeyhsd statistic

I was semi-randomly trying out numbers in the functions to find some cases for code paths in #6132 (fuzz testing)

the following case raises an exception that likely a bug

```
import statsmodels.stats.libqsturng.qsturng_ as qs

q_ = 18.666666666666668, 
qs._psturng(np.array([q_]), np.array(5), 1)
0.9111513370468763 5 1
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-100-66d4c2ef23e5> in <module>
      1 q_ = 18.666666666666668,
----> 2 qs._psturng(np.array([q_]), np.array(5), 1)

m:\josef_new\eclipse_ws\statsmodels\statsmodels_py37\statsmodels\statsmodels\stats\libqsturng\qsturng_.py in _psturng(q, r, v)
    832         elif q > _qsturng(.999, r, 1):
    833             return .001
--> 834         return 1. - fminbound(opt_func, .9, .999, args=(r,v))
    835     else:
    836         if q < _qsturng(.1, r, v):

C:\Users\josef\Downloads\WPy64-3740\python-3.7.4.amd64\lib\site-packages\scipy\optimize\optimize.py in fminbound(func, x1, x2, args, xtol, maxfun, full_output, disp)
   1723                'disp': disp}
   1724 
-> 1725     res = _minimize_scalar_bounded(func, (x1, x2), args, **options)
   1726     if full_output:
   1727         return res['x'], res['fun'], res['status'], res['nfev']

C:\Users\josef\Downloads\WPy64-3740\python-3.7.4.amd64\lib\site-packages\scipy\optimize\optimize.py in _minimize_scalar_bounded(func, bounds, args, xatol, maxiter, disp, **unknown_options)
   1824         si = numpy.sign(rat) + (rat == 0)
   1825         x = xf + si * numpy.max([numpy.abs(rat), tol1])
-> 1826         fu = func(x, *args)
   1827         num += 1
   1828         fmin_data = (num, x, fu)

m:\josef_new\eclipse_ws\statsmodels\statsmodels_py37\statsmodels\statsmodels\stats\libqsturng\qsturng_.py in <lambda>(p, r, v)
    825         raise ValueError('q should be >= 0')
    826 
--> 827     opt_func = lambda p, r, v : abs(_qsturng(p, r, v) - q)
    828 
    829     if v == 1:

m:\josef_new\eclipse_ws\statsmodels\statsmodels_py37\statsmodels\statsmodels\stats\libqsturng\qsturng_.py in _qsturng(p, r, v)
    753 
    754     elif p not in p_keys:
--> 755         y = _interpolate_p(p, r, v)
    756 
    757     v = min(v, 1e38)

m:\josef_new\eclipse_ws\statsmodels\statsmodels_py37\statsmodels\statsmodels\stats\libqsturng\qsturng_.py in _interpolate_p(p, r, v)
    528     p0, p1, p2 = _select_ps(p)
    529     try:
--> 530         y0 = _func(A[(p0, v)], p0, r, v) + 1.
    531     except:
    532         print(p,r,v)

KeyError: (0.85, 1)
```
user: josef-pkt
issue_6404: state: open
title: ENH: multiple testing and equivalence
body: If we have multiple equivalence hypothesis tests, then we need to correct for multiple testing.

However, that doesn't look obvious and trivial

parking references, 
Method in original article didn't always control size (FWER), some adjusted methods (e.g. using Holm) described in the reply to a counter example seem to work.

Guilbaud, Olivier J. M. 2013. “Incorrect Error-Rate Control in the Multiplicity-Adjusted Two One-Sided Tests Procedure Proposed by Zheng, Wang, and Zhao (2012): A Counterexample.” Pharmaceutical Statistics 12 (6): 387–89. https://doi.org/10.1002/pst.1596.
Zheng, Cheng, Jixian Wang, and Lihui Zhao. 2012. “Testing Bioequivalence for Multiple Formulations with Power and Sample Size Calculations.” Pharmaceutical Statistics 11 (4): 334–41. https://doi.org/10.1002/pst.1522.
———. 2013. “Authors’ Reply to the Letter to the Editor by Olivier J.M. Guilbaud.” Pharmaceutical Statistics 12 (6): 390–93. https://doi.org/10.1002/pst.1594.

(I didn't do a systematic search for articles related to multiplicity corrected equivalence tests. citation counts are pretty low in this area.)

related: #6403

user: josef-pkt
issue_6403: state: open
title: ENH/DOC/UseCase: using models, OLS, GLM, discrete for equivalence and superiority testing
body: This is a question for how much we already support and what we still need to support this usecase.

I just saw that PASS has equivalence tests for Poisson rates and Binomial proportion
https://ncss-wpengine.netdna-ssl.com/wp-content/themes/ncss/pdf/Procedures/PASS/Equivalence_Tests_for_the_Ratio_of_Two_Poisson_Rates.pdf

Our models have good support for linear hypothesis and we can reuse those also for simpler cases like 2- or k-sample cases. plus we can control for confounders
#2077 regression equivalent of simple, standalone hypothesis test

- TOST is equivalent (at least in linear or r-test case) to checking whether a 2-alpha confidence interval is inside the equivalence margin. The confidence interval for parameters and linear combination is already available.

- Hypothesis test with non-zero value and one sided alternative: This is the basic hypothesis test for score confidence intervals and for equivalence, inferiority and superiority tests.
Currently we don't allow for one-sided alternative in `Results.t_test` #1193

- What is the effect/contrast of interest? 
   - simple case is a linear/affine restriction on parameters, supported by current wald and t tests
   - hypothesis, linear/affine restriction on predictions similar to `margins` e.g. #3744
     There should be an issue about allowing inference with several cases of predict, e.g. comparing prediction at average or average prediction for two cases of exog, e.g. treatment effect.
      Because of the link function, this is not directly a comparison of parameters, 
      e.g. poisson mean for treated versus poisson mean of control.
      I'm not sure what the status here is, predict cannot handle it, but `margins` has some support for discrete changes of a dummy/binary variable. (*)
      

(*) IIRC: margins ends up with a linearized version of the effect of a dummy using delta method to get variance of the difference in two predictions.

I guess we want something like an ATE using a parametric model and standard inference on it, including hypothesis with one-sided alternatives (or equivalence and similar).
Computing this is not difficult for linear models or for `linpred`, but it is in GLM/discrete.






user: josef-pkt
issue_6400: state: open
title: FAQ: related and complementary packages
body: This should be a curated list of related packages, especially to statistics and econometrics topics that statsmodels does not (yet) provide.

(not complete adding links as found, not sorted and organized yet)

https://github.com/scikit-learn/scikit-learn The closest machine learning library. It is easy to find lists of popular machine learning libraries in Python.
Below are packages that are mainly oriented towards more traditional statistics and econometrics topics.

https://github.com/scipy/scipy statsmodels depends on many algorithm in scipy, scipy.stats has distributions and hypothesis tests (and is our older brother)

https://github.com/bashtage/linearmodels  panel data and multivariate linear models
https://github.com/bashtage/arch GARCH models, unit root tests and bootstrap

https://github.com/maximtrp/scikit-posthocs Pairwise multiple comparisons (post hoc) tests

https://github.com/CamDavidsonPilon/lifelines survival analysis

https://github.com/pysal Python Spatial Analysis Library

https://github.com/lmfit/lmfit-py Non-Linear Least Squares






user: josef-pkt
issue_6399: state: open
title: FAQ/DOC overview: proportions hypothesis tests, confint and similar
body: We need a overview table about which hypothesis tests and confidence intervals are available or missing
(see #6380 and #3954

| sample  | method | function(s) |
| --------  | --------  | ----------- |
| 1 sample | |
| | value hyp | binom_test <BR> proportions_ztest |
| | conf int | proportion_confint |
| | equivalence | binom_tost <BR> proportions_ztost |
| | power | NormalIndPower <BR> power_binom_tost <BR> power_ztost_prop |
| | effect size |  |
| 2 sample | |
| | equal hyp | contingency_tables |
| | value hyp | test_proportions_2indep <BR> proportions_ztest |
| | conf int | confint_proportions_2indep |
| | equivalence | tost_proportions_2indep <BR> proportions_ztost |
| | power | NormalIndPower using effect size <BR> power_proportions_2indep |
| | effect size | proportion_effectsize <BR> meta_analysis.effectsize_2proportions |
| k sample | |
| | equal hyp | proportions_chisquare |
| | pairwise hyp | proportions_chisquare_allpairs <BR> proportions_chisquare_pairscontrol |
| | conf int |
| | equivalence | ?
| | power |
| | effect size |
| 2 x 2 x k |
| | est + hyp | StratifiedTable <BR> meta_analysis |
| multinomial | 
| | conf int | multinomial_proportions_confint |
| models |
| | | GLM-Binomial <BR> Logit <BR> Probit |

possible columns: 
- diff, odds-ratio, risk-ratio for 2 and k sample comparisons
- methods: asymptotic, exact, improved, 
   "improved" is all other methods with some small sample improvements compared to (standard) asymptotic.

power, effect size, ...

contingency tables: 2- and k-sample (2 sample stratified)

model based: #3676, different or all sampling cases, but only asymptotic inference

multinomial
case control - all missing ?
...

user: josef-pkt
issue_6388: state: open
title: DOC notebook: use statecrime dataset for multicollinearity and demeaning
body: I was looking at `datasets.statecrime` as an example for multicollinearity measures

This might be a good example for how to treat the constant in these measure, i.e. Belsley versus Snee and Marquardt https://www.jstor.org/stable/2683239

if we demean the data before OLS, then the parameterization is deviations from mean and not the absolute effect (mean + deviation from mean).
Whether to demean depends on the parameterization of interest, but we shouldn't demean to try to solve multicollinearity problems, because it doesn't if we are interested in the original magnitude. (Belsley's argument)

I still need to verify this with OLS for different parameterization, i.e. demean slope parameters or not. The guess is, we can estimate deviation from mean effect pretty accurately, but the estimate for the constant is not precise and also the combined  `constant + diff_from_mean` effect is not precise.
(A bit similar: In GAM we have different options of removing collinearity with mean. However, the (prediction) variance estimate of the spline component is zero for the normalization, e.g. at first observation or at mean. Wood recommended adding the constant in the partial prediction plot, which I also did, to get the combined uncertainty from constant + normalized-differential effect.)

variance decomposition in last singular vector shows close relationship of constant with hs_grad  and a bit less with poverty. coefficient of variation of those two is a bit above 3, which doesn't sound too bad.
This is based on data that is scaled to unit 2-norm, but not demeaned (as recommended by Belsley and also implemented in R mctest and olsrr)
 
```
   cond_index   const  violent  murder  hs_grad  poverty  single   white  \
0       1.000  0.0000   0.0007  0.0006   0.0000   0.0002  0.0001  0.0002   
1       4.280  0.0000   0.0189  0.0507   0.0001   0.0002  0.0000  0.0046   
2       8.392  0.0000   0.0085  0.0204   0.0000   0.0206  0.0000  0.0043   
3      15.109  0.0000   0.7152  0.1131   0.0002   0.0424  0.0038  0.0229   
4      19.118  0.0000   0.2325  0.4967   0.0008   0.2064  0.0048  0.1015   
5      23.970  0.0017   0.0218  0.0150   0.0050   0.1030  0.0944  0.2044   
6      53.421  0.0086   0.0013  0.1468   0.0416   0.1028  0.7119  0.6233   
7     212.623  0.9896   0.0012  0.1567   0.9523   0.5245  0.1850  0.0389   

    urban  
0  0.0009  
1  0.0004  
2  0.3341  
3  0.1794  
4  0.0011  
5  0.3010  
6  0.0364  
7  0.1467
```

However, vif only shows a very large value for the constant.
Note, this uses uncentered rsquared for the constant and centered rsquared for the other variables.

```
[variance_inflation_factor(np.asarray(dframe), i) for i in range(dframe.shape[1])]
[3878.9745506781446,
 4.781873639553377,
 8.48743710672138,
 3.335395880192201,
 4.316838896478471,
 8.91531264838877,
 2.6834382078181136,
 2.0012793619505826]
```

To see which slope variables are collinear with the constant requires that the slope variables are not demeaned and that we use the uncentered rsquared for them.

(I have a notebook, but that's currently a mess with lots of different experiments and different data. other datasets I tried are mpg/autos from GAM unit test and iris data using get_rdatasets.
The iris dataset is a good example for dummy variables, and what might happen with dummy variable trap.  In a quick example, the dummy problem also shows up with demeaned data, but I need to check that again.)
user: josef-pkt
issue_6387: state: open
title: ENH: variance_inflation_factor does not support pandas DataFrame
body: I just tried using a pandas dataframe to get the variable names back.
However, that doesn't work and the user needs to convert to numpy array.

This is related that it was designed for use with `exog`.

we need some nice vector vif that returns vif with column names of the data frame

maybe add loop to `variance_inflation_factor` and check it works in #2380
user: josef-pkt
issue_6386: state: open
title: ENH: GLM, discrete: influence on weights, hessian_factor, drop exog or obs
body: If we use WLS as local model for evaluating diagnostic measures, like multicollinearity and influence measurres in GLM, then we keep weights fixed and don't take into account that it will also change.

As companion to local, fixed weights analysis, we could check whether the weights are stable to, for example, leaving an observation or leaving a variable out.
Especially, adding or dropping an important variable can have a larger effect on weights.

Just an idea, no references, I don't remember having seen this in the literature that I looked at.

context: R package mctest has a corrected variance inflation factor (cvif) that takes into account the effect of multicollinearity on the scale (residual variance) estimate in OLS. I doubt it's very useful as multicollinearity measure, but similar things might be more important in GLM.


For hessian factor we would also get part of the impact on cov_params, which goes in the direction of evaluating influence for inference (which I briefly looked at in the past but not in details yet)

Note in GLM, the weights for IRLS are just a function of the fittedvalues mu, and we need the influence on the prediction mu (via linpred and params).


user: josef-pkt
issue_6385: state: open
title: ENH/Design adding pandas DataFrame styler
body: pandas has nice style options for html with, for example, cell highlighting

I used this in factor analysis for rotation and now also for multicollinearity measures.

In several function like hypothesis test results we have a `summary_frame` method that returns a pandas DataFrame.
One option would be to add a `frame_styled` (or similarly named) method that returns a pandas styler object with DataFrame attached in `data` attribute.

my current example (not sure it's the best way, but it works.) #2380

```
cic, dc = cond_index2(mpg, scaling=True, inverse=True)
exog_names = mpg.columns
df0 = pd.DataFrame(np.round(dc[0] / dc, 3)[:, None], columns=['cond_index'])
dfc = pd.DataFrame(np.round(cic, 4), columns=exog_names)
dfc = pd.concat([df0, dfc], axis=1)
dfc_styled = dfc.style.applymap(lambda x: 'background-color : yellow' if x >= 0.5 else '', 
                                subset=pd.IndexSlice[:, dfc.columns[1:]])
dfc_styled
```

I only used cell highlighting and never checked other options, e.g. display formatting/precision for numbers.

The equivalent for standard `SimpleTable` summary and non-html formats would be to add stars.
(we have supporting code in `iolib` or in a PR for that)
user: josef-pkt
issue_6382: state: open
title: FAQ/DOC/Performance: memory requirements of models
body: related #2113 for cached results properties

https://stackoverflow.com/questions/59568345/how-to-manage-large-a-dataset-for-regression

How much memory do our models need?
How many temporary or cached (nobs, k_vars) arrays are used during estimation?

I have no idea, some guesses

minimal 
- OLS with exog that is fortran contiguous numpy array (asarray doesn't make copy)
we only need pinv(exog), i.e. would need only 2 (nobs, k_vars)-arrays
- WLS and maybe OLS: whiten creates wexog and wendog which are not views on original exog.
- GLM irls: exog, wexog for each irls iteration, pinv(wexog) or similar in WLS or MinimalWLS.
- GLM, discrete with scipy optimizers, exog, score, hessian, but score_obs wouldn't be needed

However:
using pandas and formula interface
- np.asarray(exog_dataframe) might make copy of data
- patsy: in formula exog creation, I have no idea

that is, there might be several additional copies of the data or exog when pandas and formulas are used.

The above is to get through `fit`
Other arrays are created and attached or cached to support results statistics.

Possible actions

- we need more information about actual memory usage
- there might be some redundant cases, e.g. wexog could/should be a reference exog in OLS
- we might compute `score_obs` to compute the `score`. This might not be easy to change.
- some cheaper options exist or could be added, e.g. OPG instead of full hessian in GLM
- docs for how to use pandas and patsy or how not to use them to save on memory

I don't think we gain a lot. I expect that this would not allow exog to increase by more than a third.
This is small compared to sparse dummies and out of core computation.


user: josef-pkt
issue_6381: state: open
title: SUMM/ENH/FAQ avoiding data problems from ill conditioning and tools
body: I didn't realize that my original data is ill conditioned, it's badly scaled and has high multicollinearity? What can I do?

current context: multicollinearity diagnostics for the model.exog which might be badly parameterized.
i.e. we don't want "theoretical" diagnostics that use some transformed data, but we want to know what the problems with our actual `exog` is. #2380

example: vif based on correlation matrix ignores multicollinearity or problems with the constant. OLS cond. number is based on exog and sensitive to scaling.
Belsley 1980 has an example where the constant is almost collinear with a exog with large mean and small variance.
Nist test cases has a similar example with small coefficient of variation. And another example with badly scaled polynomials.

We want to alert users to those problems or make it easy for them to check. e.g. #1908 for adding sanity checks

However, Belsley argues that fundamental ill conditioning cannot be removed by transformation. If the transformation removes ill conditioning, then the transformation itself is ill conditioned, i.e. we just shift the ill conditioning from the model to the transformation.

second: numerical problems in optimization, example Poisson with large values of exog
#1131
#1715
#3925
#1131
#1699  #4577 StandardizeTransform
`fit_transformed` ?

#2062

Note: in nonlinear model like GLM and discrete X could be reasonably well behave, but the nonlinear transform creates numerical problems, e.g. exp under or overflow.
In this case, transforming exog improves convergence of the optimizer and doesn't just shilft ill conditioning.


current plans
functions to identify multicollinearity, condition index, vif and similar
quick summary including bad scaling, e.g. min/max, coefficient of variation of data.
multicollinearity measure both on data/exog and on model and results, score_obs, hessian and cov_params

later: 
look into `fit_transformed` for GLM and discrete again.
alternative estimator, eg. ridge, penalized, Firth

note: Firth includes properties of data including endog
As in perfect separation, (almost) empty cells, there is the additional separate issue whether there is enough (or too much) information about the relationship between exog and endog.

related: perfectly collinear variables or unidentified parameters. What can we still infer? 
e.g. `is_estimable` #6271





user: josef-pkt
issue_6380: state: open
title: FAQ/DOC/ENH k-sample binomial proportion, joint and individual hypothesis tests
body: 

https://stackoverflow.com/questions/59566824/one-sided-proportion-test-for-significantly-high-values

Suppose we have k-sample binomial proportions, i.e. 2 x k contingency table with independent columns.

what can we test if the overall hypothesis is that all proportions are the same?

- joint test: all proportions equal to average proportion
- individual, vectorized test for equality to average maybe with multiple testing correction, or using joint test as gate keeper to preserve overall FWER, or FDR if some nulls might fail
- all pairwise, ...

I don't remember what we have for this case. I guess there are some missing hypothesis tests for the k-sample version.

Proper distribution would have to take into account that the average depends on each proportion. There is a literature for this but I don't remember much.


user: josef-pkt
issue_6378: state: open
title: REf/BUG: scaling in reset test
body: In #5952 reset test was changed to scale also endog and original exog.
In a pure wald-test version, the results are theoretically invariant and can numerically only be more stable.

However, we cannot compare other results, e.g. loglike is based on transformed endog (non-comparable in LR test), 
rsquared ? is rsquared invariant to endog rescaling?

We cannot transform endog outside of linear models, e.g. in GLM and discrete version. 
Do we want a consistent version across models?

(I haven't thought much, just puzzled by #5952 which I just saw.)



user: josef-pkt
issue_6377: state: open
title: SUMM/TST  Monte Carlo verification for score tests
body: see also 
#3905 general Monte Carlo support
several issues on implementing score tests

(I need an overview again, I just found some printouts without enough identifying information)
given that many of these score/lm/cm test versions don't have an analogue in R or Stata

https://github.com/statsmodels/statsmodels/issues/4759#issuecomment-399719909 
    score_test_monte_carlo.ipynb monte carlo and developement code for GLM-Poisson score_test

https://github.com/statsmodels/statsmodels/pull/5711#issuecomment-491962436

related: dispersion tests for Poisson, many versions of the tests but no monte carlo results and norecommendations (I haven't seen a Monte Carlo article that compares those excess dispersion tests)

test of size, distribution of test statistic:

- size by main alphas
- confidence intervals for empirical size
- histogram
- power divergence test on binned/histogram data
- kstest, Do we have anderson-darling when distribution is fully specified?

support for simulating cases ?





user: josef-pkt
issue_6371: state: open
title: BUG/ENH tukey_hsd with only 2 levels, corner case
body: https://stackoverflow.com/questions/57660196/type-error-for-multicomparision-tukeyhsd/59470650#59470650

maybe there is a problem that can be fixed for the degenerate two sample/level case in tukey hsd

I didn't try an example, so this is unconfirmed
user: josef-pkt
issue_6370: state: open
title: ENH: nonparametric specification testing
body: nonparametric Reset test, Breusch-Godfrey test and other specification tests.

Manuel A. Domínguez & Ignacio N. Lobato (2019): Specification testing with
estimated variables, Econometric Reviews, DOI: 10.1080/07474938.2019.1687116

looks good based on skimming the article
I don't understand the details from skimming, e.g. how their wild bootstrap works.

The test statistic has a nonstandard distribution that is not pivotal and so requires bootstrap.
It looks computationally expensive.

But it still might be worth it because it looks pretty general and good performance in their Monte Carlo.
related to other articles with similar approach

Advantage is that it uses empirical process which, AFAICS, does not require a bandwidth selection as kernel or spline methods.

Using splines with not very large number of basis functions looks computationally cheaper but is more semi-nonparametric. Binning and piecewise linear might be a lower dimensional approximation to the empirical process. 

#3690 and related for reset tests
#5408 specification testing with nonparametric alternatives





user: josef-pkt
issue_6366: state: open
title: ENH DOC overview of workflow, common elements of models
body: I don't see a doc page to describe the usual workflow and features or methods common to models.
related #1353 for overview of post-estimation results

e.g.
- model creation: array/DataFrame versus formula api
- fit returns results instance
- post-estimation
  - in results instance: predict and margins, summary, inference and hypothesis tests, ...
  - extras: diagnostic and specification tests, plots, (results methods or standalone functions)
  - tsa: impulse response functions and similar
- model selection: only available for some models. (information criteria)

Doc page might need a split between tsa models and non-tsa models, e.g. predict versus forecast, time index, ...

user: josef-pkt
issue_6364: state: open
title: FAQ-D: links from docs to notebooks
body: I just saw that we have links from docs to notebooks

http://www.statsmodels.org/dev/_sources/glm.rst.txt
```
Detailed examples can be found here:

* `GLM <examples/notebooks/generated/glm.html>`__
* `Formula <examples/notebooks/generated/glm_formula.html>`__
```

Would be good if it also works from docstrings of classes and functions, I haven't checked. 
Then we can more easily refer from docstrings to the examples for more details.

(I had an issue for this in the past, but don't find it.)
user: josef-pkt
issue_6361: state: open
title: Update dynamic_factor.py
body: I'm getting a lot of failing to converge errors in simple DFM setups, so I did a bit of digging. 
It would seem that the default factor normalization option in the PCA code doesn't transform res_pca.loadings along with res_pca.factors, so running with normalization=on results in non-zero mean endog in line 464 and throws off error process estimation. As a result either failing to converge or taking forever.

Turning off normalization in line 452 is one option to fix the issue, another is to replace res_pca.loadings.T in line 464 with res_pca.coeff
(Haven't tested that one).


I don't have a specific test to suggest.
Just put a breakpoint on line 464 and do endog.mean

- [ ] closes #xxxx
- [ ] tests added / passed. 
- [ ] code/documentation is well formatted.  
- [ ] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

<details>


**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in master and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/master -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>

user: dsgerbc
issue_6358: state: open
title: ENH: QR updating for penalized L2 estimation
body: Just an idea while looking at the code for `TheilGLS`

`(x'x + a S) beta = x'y`
can be computed using data augmentation `S_half = S^{-1/2}`, `vstack([x, S_half])`
We use this in PIRLS in GAM

linalg:

we need `(x'x + a S)^{-1} = (x'x + a S_half' S_half)^{-1}

If S_half is low rank, i.e. has few rows, then this is a low rank updating of the inverse matrix, which is cheaper to compute than the full matrix inverse  #6265.
Similar would apply if we use the QR decomposition of x and update the decomposition to `vstack([x, S_half])` (but there we wouldn't have the inverse yet.

The advantage would be that we can do the updating for different pen_weights which are included in S_half

aside: 
we have code for S_half in singular case with reduced number of rows.
tools.linalg.matrix_sqrt   (adding with GAM PR)







user: josef-pkt
issue_6355: state: open
title: ENH tools linalg redundancy in system of linear equations (scipy)
body: https://github.com/scipy/scipy/blob/master/scipy/optimize/_remove_redundancy.py
PR that improves it https://github.com/scipy/scipy/pull/11245

removes redundancies and raises on conflict in system of linear equations A x = b

This might be useful for linear constraints, e.g. currently it is up to the user to specify linear constraints in a feasible way.

might be similar to handling stochastic constraints in penalization, e.g. during GAM PR.

the scipy function looks private
(I only did a quick skimming of the code and haven't tried it out yet)




user: josef-pkt
issue_6354: state: open
title: TST/ENH/Reference: functions in R package asbio
body: found by chance: R package asbio has some interesting functions that we are missing or that can be used for writing unit tests

https://www.rdocumentation.org/packages/asbio/versions/1.5-5

examples

- pairw.oneway all pairwise t_tests with unequal variance
- pairw.anova | Conducts pairwise post hoc and planned comparisons associated with an ANOVA
  not sure about this, multiplicity correction like LSD are outdated and don't preserve family wise error rate, AFAIR
- DH.test   Doornik-Hansen test for multivariate normality.
- BDM.test  Brunner-Dette-Munk test
- partial.R2 Partial correlations of determination in multiple regression
    that's what I was looking for right now, but takes two estimated models (doesn't seem to use computational linalg shortcuts)
- joint.ci.bonf  Calculates joint confidence intervals for parameters in linear models using a Bonferroni procedure.
  maybe useful for unit tests and API design, implementation should be "obvious"

- ci.prat  Confidence intervals for the ratio of binomial and multinomial proportions
  for unit test, do I/we have it
- ci.impt  Confidence interval for the product of two proportions
- ci.p  Confidence interval estimation for the binomial parameter pi using five popular methods.
  same as one we already have? one proportion conf_int
- other pairwise tests like friedman, Kruskal-Wallis
- trim.ranef.test | Robust test for random factors using trimmed means.
- ...





























user: josef-pkt
issue_6351: state: open
title: BUG/Review: internal use of predict with pandas returns
body: https://github.com/statsmodels/statsmodels/pull/5255#issuecomment-426473598

A while ago we changed predict so it returns a pandas Series.
However, internally we had assumed that we just have a plain numpy array.

We want to avoid having suddenly a pandas data structure instead of numpy arrays in the internal computation. However, there are currently no unit test failures that are identified to related to this.
The only source of serious bugs would be if we compute something with `predict()` where pandas behaves differently from numpy.

This needs verification.
Possible workaround would be to add a kwd option to return numpy arrays even if the index for returning pandas is available.

user: josef-pkt
issue_6346: state: open
title: ENH/SUMM: parameters of interest and effect or irrelevance of controls/confounders
body: related #6323, #6223

suppose we have a model `E(y| x, z) = f(x b, z a)` where
x, b are variables and parameter of interest
z are controls/confounders

We want to get a good estimate of `b` or a function of it, but we are not directly interested in `a`.

In this case problems in the estimate of a might not be relevant (have a small impact) for the estimate of `b`.

We should have diagnostic tools to help with evaluating the impact or irrelevance of confounders for the estimation of the parameters of interest.

This is a bit vague and there is no specific model, but we should get more supporting function for the case when we don't want to treat all `exog` in the same way, but instead want to partition them in those `[X, Z]`. This showed up in variable selection. issues, PRs

for multicollinearity diagnostic
Belsley, Kuh, Welsch 1980 p.107ff "Near Collkarity Nullified by Near Orthogonality."
multicollinearity in `Z` is irrelevant for `b` if `Z` are orthogonal to `X`
i.e. we can ignore problems in the confounders if they don't have much effect on the estimate of the parameters of interest.

tools might be multivariate analysis of X and Z, e.g. canonical correlation, which sounds similar to what we want for IV estimation and instruments or weak instruments.

related:
Focused Information criterion (FIC) versus AIC/BIC: We want to choose variables and models to focus on some parameters and not on overall goodness of fit or prediction.


brief thoughts about GLM and nonlinear (single index) models
In GLM and discrete we need the overall prediction to evaluate effects, eg. marginal effects, or the `weights` for the WLS/Hessian/score. However, we only need the partial linear prediction `z a` and don't care whether individual parameters in `a` are well estimated as long as `z a` has a good estimate.

more general:
We should get support for at least some common patterns within exog, instead of treating `exog` as a black box array.
(one case I was thinking of and I don't know how much we already have is 2-way anova with confounders, i.e. ANCOVA where the parameters of interest are the effects or parameters of one or several categorical variables
Fletcher, David, and Peter W. Dillingham. 2011. “Model-Averaged Confidence Intervals for Factorial Experiments.” Computational Statistics & Data Analysis 55 (11): 3041–48. https://doi.org/10.1016/j.csda.2011.05.014.
)



user: josef-pkt
issue_6341: state: open
title: MAINT: Deprecations to handle for v0.13
body: Summary issue for deprecations that we should handle prior to releasing v0.12.  Please feel free to edit / add items.

Timing is probably flexible, but I guess we should consider implementing some of these in master soon after v0.11 is released so that we have the maximum time to check for any problems that arise from them.

- [x] #6332 - TSA models will raise an exception when doing out-of-sample operations with an unsupported index in v0.12+

Things that may or may not need action for v0.12:

- [ ] #6087 - Remove old AR model (deprecation message already exists, in favor of new AutoReg class)
- [ ] #5827 - Add deprecation message for old ARMA/ARIMA models (this was not done in v0.11 so that we could do a little more thorough checking of the new ARIMA class).

user: ChadFulton
issue_6331: state: open
title: ENH: diagnostic, outlier-influence, multicollinearity for penalized estimation
body: (I never looked at this.)

Is there anything special that we can add for penalized estimators or can we apply the standard methods?

current case: multicollinearity measures and vif for ridge regression. Does ridge regression or penalization "cure" multicollinearity?
some references in https://github.com/statsmodels/statsmodels/issues/6330#issuecomment-565079152



user: josef-pkt
issue_6330: state: open
title: ENH: Generalized variance inflation factor, vif for subset of exog or terms
body: vif  can be generalized for subsets of exog or multi-column terms

Fox, John, and Georges Monette. 1992. “Generalized Collinearity Diagnostics.” Journal of the American Statistical Association 87 (417): 178–83. https://doi.org/10.1080/01621459.1992.10475190.

it looks like the car package uses this definition of vif

https://stats.stackexchange.com/questions/136004/vif-for-generalized-linear-model
user: josef-pkt
issue_6328: state: open
title: VARSummary to html
body: The summary of the ols model has the option of outputting it in html (or csv, latex etc) formats.  Is this also possible to do for the VARSummary case?  I searched all available functions in the VARSummary class without success and it wasn't clear to me reading the documentation of 
Input-Output iolib
if
table.SimpleTable
can transform a VARSummary object (or one of its attributes) to html.  Perhaps this is a feature request?
user: Econbrief
issue_6325: state: open
title: REF/Review computational shortcuts for OLS outlier influence regression diagnostics
body: outlier influence and similar regression diagnostics might still have some inefficient computations.
(I improved those as I semi-randomly found computational shortcuts)

article that summarizes several computational shortcuts

Velleman, Paul F., and Roy E. Welsch. 1981. “Efficient Computing of Regression Diagnostics.” The American Statistician 35 (4): 234–42. https://doi.org/10.2307/2683296.

one that I haven't seen before: section 3.1 catcher matrix which is just transposed `pinv(x)`
can be used to construct partial residual plot without explicit regression to partial out other variables.
e.g.  statsmodels.graphics.regressionplots plot_partregress uses OLS to partial out other exog

i.e. we should be able to get partial regression residuals directly from pinv(x), but I haven't tried yet.

related 
PRs #6265, #6268 and #2380 work with moment (`cov(xy)`) or product matrices (`x'x`) and not with `pinv(x)` and are focused on effects/influence of variables and not with effects of observations.



user: josef-pkt
issue_6323: state: open
title: SUMM/ENH: robustness of causal parameters, treatment effect
body: We should have tools specifically to evaluate the robustness of regression coefficients that are interpreted as causal or treatment effect to model specification and variable selection.

This is a companion issue to issues #6237 and #6223 and after preparing subset regression PR #6268 which will have model averaging as follow-up.
see https://github.com/statsmodels/statsmodels/issues/6237#issuecomment-562607519
Young and coauthor have Stata package `mrobust`

another Stata package
Xu, Ran, Kenneth A. Frank, Spiro J. Maroulis, and Joshua M. Rosenberg. 2019. “Konfound: Command to Quantify Robustness of Causal Inferences.” The Stata Journal 19 (3): 523–50. https://doi.org/10.1177/1536867X19874223.


This can be treated as a companion to
- diagnostic and specification testing, which we have so far to test the assumptions of a model and model specification
- (mis)specification robust estimation and inference (GMM/GEE, QMLE and cov_type)
- influence measures (for variables) in `outlier_inference`

(this also goes back to Leamer early 80s)





user: josef-pkt
issue_6308: state: open
title: Maint/Task most recent issues and PRs don't have labels and milestone
body: this will make maintenance and finding things more difficult.
user: josef-pkt
issue_6307: state: open
title: BUG/Review/CLN TheilGLS problems with state in select_pen_weight  
body: TheilGLS has many code comments about attributes not updating correctly during pen_weight search.
Problem with trying to do hyper parameter search inside a model where hyper parameters like pen_weight change the `state` of the model.

I saw this while looking at aic and why hatmatrix_trace is not cached, and used  to define effective df_resid

I don't remember what the status of this is. I partially refactored during GAM merge.

user: josef-pkt
issue_6306: state: open
title: FAQ/FAQ-D sign of aic, bic, ..., min aic is best
body: The computation of information criteria like aic uses negative loglikelihood and penalization is positive, i.e. the information criteria are distance measures or loss functions where smaller is better

`aic =  -2 * self.llf + 2 * (self.df_modelwc)`


Note, some references in statistics and econometrics define aic and similar with the opposite sign of the implementation in statsmodels and other packages, and choose a model that maximizes aic.
( I just got confused again while reading articles with the positive llf definition, and not for the first time
https://github.com/statsmodels/statsmodels/pull/297/commits/4c6bbd260f7470f2d593e4f3f9cbab8a351ecea9
)

user: josef-pkt
issue_6304: state: open
title: Covariance structure
body: can statsmodels give the result of different covariance structures which are realized in the PROC MIXED in SAS?
user: kangsijueshi
issue_6300: state: open
title: LinAlgError: LU decomposition error 
body: I got LinAlgError: LU decomposition error
while training and forecasting in UCM with exog variables
user: manishruti
issue_6287: state: open
title: Q: Logit model with cov_type = "cluster"
body: #### Describe the bug

This is more of a question and a possible bug. When I try to run:

```python
m = sm.Logit(y, X).fit(cov_type = "cluster", groups = groups)
Optimization terminated successfully.
         Current function value: 0.064115
         Iterations 9
Traceback (most recent call last):

  File "<ipython-input-15-2414b74c0cf0>", line 1, in <module>
    m = sm.Logit(y, X).fit(cov_type = "cluster", groups = groups)

  File "C:\Users\Emil Mirzayev\Anaconda3\lib\site-packages\statsmodels\discrete\discrete_model.py", line 1902, in fit
    disp=disp, callback=callback, **kwargs)

  File "C:\Users\Emil Mirzayev\Anaconda3\lib\site-packages\statsmodels\discrete\discrete_model.py", line 217, in fit
    disp=disp, callback=callback, **kwargs)

  File "C:\Users\Emil Mirzayev\Anaconda3\lib\site-packages\statsmodels\base\model.py", line 503, in fit
    mlefit = LikelihoodModelResults(self, xopt, Hinv, scale=1., **kwds)

  File "C:\Users\Emil Mirzayev\Anaconda3\lib\site-packages\statsmodels\base\model.py", line 1247, in __init__
    use_t=use_t, **cov_kwds)

  File "C:\Users\Emil Mirzayev\Anaconda3\lib\site-packages\statsmodels\base\covtype.py", line 256, in get_robustcov_results
    groups = kwds['groups']

KeyError: 'groups'
```
My groups variable is just the column that I want my errors clustered on. in this case, for every unique ID group that I have. Instead, I get this error that there is no group keyword. Am I doing something wrong here?
Even more interesting it for some reason converges and then error pops up
sm version is **0.10.1**
user: emilmirzayev
issue_6271: state: open
title: ENH: add is_estimable check for linear hypothesis, linear constraints
body: see FAQ #1672 

finally a recipe that looks easy (found by chance)

https://go.documentation.sas.com/?cdcId=pgmsascdc&cdcVersion=9.4_3.3&docsetId=statug&docsetTarget=statug_intromod_sect042.htm&locale=en

 `L b` is estimable if and only if $L (X 'X )^{-} (X 'X ) = L $ (see, for example, Searle 1971, p. 185).

e.g. `(X 'X )^{-} = pinv(X'X) = normalized_cov_params` in OLS and different from `X'X` if X is rank deficient
user: josef-pkt
issue_6268: state: open
title: ENH: sweep algorithm and sequential OLS with QR
body: this should add stepwise regression, sweep algorithm and sequential OLS
see #1809

currently it is mainly a script I wrote in 2012 converted to a module and two unit tests

also related to 
PR #2380 multicollinearity properties of inverse covariance, and
PR #6265 for updating and downdating inverse covariance matrices

one submatrix of the sweep matrix contains the inverse product matrix and is updated similarly to #6265

user: josef-pkt
issue_6265: state: open
title: ENH: add functions for up and down-dating inverse matrices
body: see #6253

some tools for working with inverse covariance, precision matrices

user: josef-pkt
issue_6262: state: open
title: z-Test
body: The current behavior of the z-test is problematic:


To my knowledge: If I have a small Gaussian distributed data set and I know the true variance $\sigma$ of the data then the z-test is only appropriate if the standard deviation of the estimator is set to $\sigma/\sqrt{n}$. 
with:    

- $n$ is the sample size (data size).   

In statsmodel the function `sm.stats.ztest` don't has a parameter for the true standard deviation. The function computes the standard deviation of the estimator from the data $\sigma_{\mathcal D}/\sqrt{n}$. This gives another result (in this case a t-test is more appropriate).

I illustrate it by an example (adapted from [Wikipedia](https://en.wikipedia.org/wiki/Z-test): 

Suppose that in a particular geographic region, the mean and standard deviation of scores on a reading test is 100 points, and 12 points, respectively. Our interest is in the scores of 5 students in a particular school who received a mean score of 90.4. We can ask whether this mean score is significantly lower than the regional mean—that is, are the students in this school comparable to a simple random sample of 5 students from the region as a whole, or are their scores surprisingly low?

**Parameter of Interest**: $\mu$, population mean.

**Null Hypothesis:** $\mu$ = 100

**Alternative Hypthosis:** $\mu$ < 100

5 Students with $\mu_e = 90.4$.

```python
loc=mu_0=100
scale=12
n=5

np.random.seed(40) 
data = np.random.normal(loc=96, scale=scale, size=n)
sigma_data = data.std(ddof=1)
mu_data = data.mean()
print(mu_data)

# compute the p-value
std_mu_estimator = scale/np.sqrt(n) # note we know the true std of the data
p_value = scipy.stats.norm.cdf(mu_data, mu_0, sigma_data/std_mu_estimator)
print ("p-value with true std: ", p_value)

# statsmodel is using the data std to calculate the estimator std
# for a z-test is seems not to be the correct way???
print("statsmodel:", sm.stats.ztest(data, value = mu_0, alternative = "smaller"))
# same result as
p_value = scipy.stats.norm.cdf(mu_data, mu_0, sigma_data/np.sqrt(n))
print ("p-value with estiamted std (from data): ", p_value)
```

    90.39845578565013
    p-value with true std:  9.39939613111886e-06
    statsmodel: (-1.7827917200700583, 0.03731010705030108)
    p-value with estiamted std (from data):  0.03731010705030108

Note: Here the difference is quite small, but if the sample size is smaller it becomes more relevant.

Why does `statsmodel` don't have a parameter to set the variance or std of the estimator for a z-test? To my knowledge providing a true std is meaningful for a z-test (in contrast to a t-test).  

Thanks for your answer.


Note: In R the z-Test enforces you to enter the value for the standard deviation, e.g.: 

```R
z.test(data, mu=10)
Error in z.test(data, mu = 10) : You must enter the value for sigma.x
```
user: peroyose
issue_6253: state: open
title: ENH: linalg tools, updating, downdating, local pertubations
body: one more group of linalg helper functions, several experiments and pieces but no code in statsmodels

- up- and down dating inverse product matrix X'X or cov
   - adding and dropping variables
   - adding and dropping columns
   - sweep algorithm
- imposing constraints, one step from unconstrained MLE
   - e.g. can we remove a variable by imposing zero constraint

related: up and down dating of matrix decomposition, especially QR.

current use cases 
- model selection by adding and dropping variables
- diagnostics for adding and removing variables (e.g. wald and score tests)
- outlier diagnostics by adding or removing observations
- out of memory computation, might not use the same tool functions, i.e. optimized for a different case

related
#2050 Gauss-Newton regression, and further cross-links or back links there


user: josef-pkt
issue_6252: state: open
title: ENH: adding pred_table to GLM, closes #1653
body: This closes #1653, adding `pred_table` to `genmod.generalized_linear_model.GLM`. The core functionality is copied from the Discrete case. Here, it only works for Binomial families and otherwise returns `None`. Unit tests, similar to those employed in the discrete case, have also been added.

- [x] closes #1653
- [x] tests added / passed. 
- [x] code/documentation is well formatted.  
- [x] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

<details>


**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in master and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/master -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>

user: tleininger
issue_6251: state: open
title: BUG: numerical precision in vif, don't use rsquared
body: (mainly to have this problem on record)

The current variance_inflation_factor function uses rsquared from the auxiliary regression and converts it to vif:

`vif = 1. / (1. - r_squared_i)`

For small r_squared, floating point precision makes `(1-rsquared)` exactly equal to zero, and vif equal to inf with zero division warning.

If we use  ssr / tss instead of  (1-rsquared), i.e. the unexplained fraction, then the denominator can be very small, floating point around zero instead of floating point around 1

consequence: In many cases we will get very large finite vif instead of vif=Inf.
This might be mainly numerical noise, i.e. vif=1e14 or 1e20 or 1e200 all are essentially collinear. 

The same will happen now in #2380 with a small ridge_factor, which should make even vif=inf (singulat) to something very large but finite.

user: josef-pkt
issue_6249: state: open
title: BUG or unclear definition, variance inflation factor with implicit constant in model
body: There might be an inconsisteny in the usage of `OLS` `rsquared` if there is an implicit constant in exog.
The definition and computation of rsquared changes if not constant is in the model. 

When we drop one of the variables related to implicit constant, then that auxiliary regression does not have a constant, and uses the no-const definition for rsquared.

I'm not sure what the vif for those variables that are part of the implicit constant should be.

Example: one categorical variable with all levels included, no constant, 2 continuous variables.

vif using the current function doesn't match up with how it is computed in #2380


user: josef-pkt
issue_6248: state: open
title: ENH: tools, how do we get all submodels with formulas and interaction terms?
body: #6237
automatic model selection requires that we can create the relevant models. Defining all submodels is difficult If there are multi-column terms like categorical or splines, or interaction terms. 

This shows up at several places, but we don't have any good tools for this.
(I don't remember whether and how I handled some of those cases)
e.g. 
- t_test_pairwise is only for one main effect,
- ultra high screening might allow multicolumn terms or not
- wald_test_terms does string matching or looks at design_info terms
- in loglinear models we only have interaction terms and we can enumerate them including all interactions.
- score_test, variable addition test requires user to provide new exog, no formula support to expand model
- ...

related:
the glmulti vignette seems to spend a lot of time on defining non-redundant models as candidates for model selection.



user: josef-pkt
issue_6247: state: open
title: REF/BUG: improve situation with non-full period indexes
body: Companion issue to #6177.

There are two options with non-full indexes:

1. Check that non-full indexes work for forecasting and add tests / fix any problems.
2. Ignore non-full indexes for forecasting.

The only problem with (2) is that Pandas' `is_full` doesn't work the way we need to to. To do this, we would like to add something like the following to `TimeSeriesModel._init_dates`:

```python
is_full = index.is_full if period_index and is_monotonic else None

if period_index and not is_full:
    warnings.warn('A Period index has been provided, but it is not'
                    ' full and so will be ignored when e.g.'
                    ' forecasting.', ValueWarning)
```

The problem is that `is_full` doesn't work the way I would expect it to with offset multipliers:

```python
index = pd.period_range(start='2000Q1', periods=10, freq='2Q')
print(index.is_full)
```

prints `False`, while for our purposes this actually is a valid (and full) index.
user: ChadFulton
issue_6237: state: open
title: ENH:  multimodel inference, model averaging, specification sensitivity
body: #2883 old issue for model averaging
see also #6223

see question best model
https://stackoverflow.com/questions/58789167/is-there-an-automatic-way-where-python-select-the-variables-that-makes-the-most
with reference to R package glmulti https://www.jstatsoft.org/article/view/v034i12
examples
http://www.metafor-project.org/doku.php/tips:model_selection_with_glmulti_and_mumin

reference (based on glmulti docs)
Burnham, Kenneth P., and David R. Anderson, eds. 2004. Model Selection and Multimodel Inference. New York, NY: Springer New York. https://doi.org/10.1007/b97636.

It looks like there are several recent conference issues in journals in several fields on this.

The part I find now interesting is to get inference and standard errors that takes model uncertainty into account. Before I thought model averaging is only for prediction as in machine learning.

The target for statsmodels could be to provide measures of model uncertainty, and standard errors, for example for treatment effects, that take model uncertainty into account.

user: josef-pkt
issue_6229: state: open
title: ENH: attach Null model params
body: We had question about computing r-squared and loglike for out-of-sample data for cross-validation. (I don't find any related issues right now)

Problem is that r-squared requires assumption on the intercept, or more generally params of the null model.

I think now that we can get a consistent interpretation of R-squared and similar measures if we evaluate the new exog at the null model for comparison. To be able to do that, we need to keep the null model or at least some sufficient statistics like params around.

In the simple OLS rsquared case, this would be using the intercept estimate from the training/estimation data, i.e. `model.endog.mean()`, or zero if no constant has been added.

Some statistics are in `tools.eval_measures`

user: josef-pkt
issue_6223: state: open
title: ENH: specification search summary, plot (researcher degrees of freedom)
body: If we have one parameter of interest, then we can plot how the parameter estimates change with different specifications.
This plot provides nice summary about how specification robust an effect is

https://github.com/joachim-gassen/rdfanalysis   MIT licensed

the plot is useful when there are more possible cases than can easily fit into a `summary_col` type table

(found through some twitter threads)

**edit** https://github.com/aeturrell/specification_curve looks similar with same kind of plot
user: josef-pkt
issue_6221: state: open
title: ENH: hypothesis test that are robust to I(0) and I(1) residuals
body: based on a PR for scipy.stats
https://github.com/scipy/scipy/pull/10985

There are articles for trend and structural break testing that are robust to both short and strong correlation, i.e. I(0) and I(1) residuals.

Vogelsang, but there are additional more recent articles.

main problem is non-standard asymptotic distribution
The test statistic looks like a weighted average between distribution under I(0) assumption and the distribution under the I(1) assumption.

user: josef-pkt
issue_6219: state: open
title: BUG: GAM inherits fit methods that shouldn't apply
body: fit_regularized and fit_constrained in GAM models, GLMGAM and LogitGAM,  are inherited but shouldn't work correctly (I guess)

I just found them in the documentation, e.g.
http://www.statsmodels.org/dev/generated/generated/statsmodels.gam.generalized_additive_model.LogitGam.fit_regularized.html
user: josef-pkt
issue_6211: state: open
title: Generalised Additive Mixed Model
body: #### Is your feature request related to a problem? Please describe

I want to reproduce the GAMM portion of this post:

https://petolau.github.io/Analyzing-double-seasonal-time-series-with-GAM-in-R/

as the R function gamm currently has no python equivalent. Since this is a broad class of models which currently has no equivalent and since you already have GLMGam functionality, this could be a good addition to statsmodels.

#### Describe the solution you'd like
I volunteer to produce this!

We could extend the Linear Mixed Model class like the GLM class has been extended for statsmodels.gam.GLMGam.

#### Describe alternatives you have considered
Since gamm has no python coverage, there is no alternative solution

#### Additional context
I will volunteer to produce this feature, but I will need some guidance.
user: TomKealy
issue_6206: state: open
title: ENH: add cov_type cluster correlation robust but not HC
body: add a partially robust sandwich estimator for clusters or GEE
similar #3495 it is only misspecification robust to part of the covariance

I didn't look at details, my context is regularized, penalized cov estimators #3197 and #2942

I'm reading for regularized estimators
Warton, David I. "Regularized sandwich estimators for analysis of high‐dimensional data using generalized estimating equations." Biometrics 67, no. 1 (2011): 116-123.

which quote Wei 2001 for correlation robust, i.e. identical arbitrary correlation within clusters but otherwise correctly specified.
Pan, Wei. "On the robust variance estimator in generalised estimating equations." Biometrika 88, no. 3 (2001): 901-906.

reference search finds a more recent article that combines Wei 2001 with the Mancl and DeRouen bias-reduced cov estimater (which we already have in GEE)
Wang, Ming, and Qi Long. "Modified robust variance estimator for generalized estimating equations with improved small‐sample performance." Statistics in medicine 30, no. 11 (2011): 1278-1291.

user: josef-pkt
issue_6202: state: open
title: ENH/TST: GAM model with terms with different spline bases
body: this came up in a mailing list discussion
https://groups.google.com/d/msg/pystatsmodels/8exUYkwiky0/Jof0JpJJCwAJ
quoted:

---
BTW the GLMGam(smoother=) takes a single smoother instance rather than an array - is it possible to use say a BSpline for x0 and PeriodicSpline for x1?

Theoretically yes, but I think I never tried that, so there are likely some problems. 
The generic multiple spline class should take a list of individual, univariate splines

possibly GenericSmoothers(AdditiveGamSmoother)
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/gam/smooth_basis.py#L797 

but that would require construction of the univariate smoothers by the user, see the specific BSpline and Cyclic spline smoother classes.

enhancement issue for this would be useful. This needs a simpler API.

The AdditiveSmoother class is just holding the list of all individual, univariate smoothers and delegating to those, AFAIR.

---

There is one basic unit test example with GenericSmoother
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/gam/tests/test_gam.py#L291 

user: josef-pkt
issue_6199: state: open
title: Missing docs for statsmodels.stats.gof.gof_chisquare_discrete
body: #### Describe the bug

It is unclear what we need to provide for `distname`. In case of `poisson` it generates error. Also what is parameter `msg`?

#### Code Sample, a copy-pastable example if possible


```
from statsmodels.stats.gof import gof_chisquare_discrete 
gof_chisquare_discrete('poisson',2, [2,3,4,5], 0.05, 'What is msg?' )
```
Here is error message
```
AttributeError                            Traceback (most recent call last)
<ipython-input-91-c8aecbce82c0> in <module>
----> 1 gof_chisquare_discrete('poisson',2, [2,3,4,5], 0.05, 'What is msg?' )

C:\ProgramData\Anaconda3\envs\env1\lib\site-packages\statsmodels\stats\gof.py in gof_chisquare_discrete(distfn, arg, rvs, alpha, msg)
    209     # construct intervals with minimum mass 1/nsupp
    210     # intervalls are left-half-open as in a cdf difference
--> 211     distsupport = lrange(max(distfn.a, -1000), min(distfn.b, 1000) + 1)
    212     last = 0
    213     distsupp = [max(distfn.a, -1000)]

AttributeError: 'str' object has no attribute 'a'
```

<details>

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `master`. If your problem has been fixed in an unreleased version, you might be able to use `master` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the master branch of this repository? It helps the limited resources if we know problems exist in the current master so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

A clear and concise description of what you expected to happen.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]

</details>

user: BrazilForever11
issue_6197: state: open
title: How 'experimental' is the summary2() method after existing for many years?
body: I am wondering how 'experimental' is summary2 at this point. It has been around for years, and, as far as I can tell, it is stable. Why is it listed as 'experimental' in the docs? Is it still 'experimental' or should the docs be updated? If it is still 'experimental', maybe someone can comment on how it is limited? Hopefully it will not present standard errors in the coefficients column or anything as serious as that!


user: iamstatsman
issue_6195: state: open
title: ENH: add end-of-sample stability or break test
body: (going through some old scripts)
https://groups.google.com/d/msg/pystatsmodels/GleMuKq4ll8/SqxNE8T82-AJ
proposed to include Andrews stability test

(more recent issues looked at similar things in the context of control charts)

user: josef-pkt
issue_6194: state: open
title: ENH/SUMM diagnostic and specification test for multivariate models
body: Currently we only have VAR type models as multivariate models. Those include some diagnostic tests.

We need a central location for multivariate diagnostic tests that are independent of a specific model, e.g. work for any 2-dim residual array.
maybe `stats._diagnostic_multivariate

core tests:

- autocorrelation.
- heteroscedasticity including arch effects, and
- normality, i.e. gaussian distribution

ljung-box autocorrelation and Luetkepohl normality test are included in VAR
(statespace only has univariate ljung-box, ?)
issue #382 for Doornik/Hansen test for multivariate normality

skimming some abstracts, it looks like some tests for VECM have different distribution than the standard VAR or multivariate models (but I didn't read details)

bonus (I haven't looked much for this either)
Tests that are robust to violating auxiliary assumptions, e.g. test that there is no serial correlation that is robust to heteroscedasticity






user: josef-pkt
issue_6191: state: open
title: [DOC] add an exmaple for LASSO
body: Please consider adding an example for [LASSO regression](https://www.statsmodels.org/devel/generated/generated/statsmodels.regression.linear_model.OLS.fit_regularized.html#statsmodels.regression.linear_model.OLS.fit_regularized).

Btw, I came across the following two resources, which might be useful for the implementation of selective inference.
https://www.pnas.org/content/pnas/112/25/7629.full.pdf
https://selective-inference.github.io/Python-software/

user: stevenlis
issue_6190: state: open
title: [FR] add rotation for PCA
body: As discussed here https://github.com/statsmodels/statsmodels/issues/6170, please consider adding rotation for PCA, which is already being experimented in [factor analysis](http://www.statsmodels.org/devel/generated/statsmodels.multivariate.factor.Factor.html#statsmodels.multivariate.factor.Factor).
user: stevenlis
issue_6187: state: open
title: roadmap josef
body: (starting a new roadmap, main tracker for changes in a release are in milestone)
(moved from roadmap 0.11)

- hypothesis testing, diagnostic refactoring and enhancements
  - #5001 base class for test results
  - review OLS diagnostic and move out of sandbox
  - organize GLM and discrete diagnostics

- `results_class` attribute to choose class, #4666, needs consistent naming across models, and inclusion  in appropriate places of the class hierarchy (I'm not sure where that is. It might required checking `__init__` args and our hacky ex-post attaching of attributes in results instances.)
   I used it also in `gam`
user: josef-pkt
issue_6186: state: open
title: ENH/REF: SARIMAX start params
body: Given #5827, we may want to think about reworking `start_params` in SARIMAX, which right now is something like an ad hoc version of Hannan-Rissanen.

However, the Hannan Rissanen of #5827 doesn't support seasonals yet (although it could and this is listed as one of the future improvements in #6159), so maybe that should be done first.
user: ChadFulton
issue_6185: state: open
title: ENH/REF: Follow ups to state space exponential smoothing models
body: This PR collects any follow ups to #6179 that should be addressed prior to a v0.11 release.

- [x] Release notes
- [ ] Example notebook
- [ ] Revisit the `initial_state` attribute in the results object. The potential issue is that the initial state for most state space models is the predicted state, while for the exponential smoothing models it's the filtered state, so that if we added a similar attribute to the base state space class `MLEResults` it would have a different interpretation than the `ExponentialSmoothingResults.initial_state` attribute.
user: ChadFulton
issue_6184: state: open
title: chisquare_effectsize outputs array instead of scalar when probs1 and probs2 are two dimensional arrays
body: #### Describe the bug

GofChisquarePower `chisquare_effectsize` outputs a two element array instead of scalar when probs1 and probs2 are two dimensional arrays.  It should output a numpy float.  The problem seems to be the `.sum(axis)` on line 482.  `axis` needs to be removed.

#### Code Sample, a copy-pastable example if possible


```python
probs0 = np.ones(2, 2)*0.5
probs1 = probs0.copy()

chisquare_effectsize(probs0, probs1, correction=None, cohen=True, axis=0)
```
Outputs `array([0., 0.])`

#### Expected Output

Output should be `0.`

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]
 import statsmodels.api as sm; sm.show_versions()

INSTALLED VERSIONS
------------------
Python: 3.7.4.final.0
OS: Darwin 18.7.0 Darwin Kernel Version 18.7.0: Tue Aug 20 16:57:14 PDT 2019; root:xnu-4903.271.2~2/RELEASE_X86_64 x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

Statsmodels
===========

Installed: 0.10.0 (/Users/user/miniconda/envs/analysis/lib/python3.7/site-packages/statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.17.2 (/Users/user/miniconda/envs/analysis/lib/python3.7/site-packages/numpy)
scipy: 1.2.1 (/Users/user/miniconda/envs/analysis/lib/python3.7/site-packages/scipy)
pandas: 0.24.2 (/Users/user/miniconda/envs/analysis/lib/python3.7/site-packages/pandas)
    dateutil: 2.8.0 (/Users/user/miniconda/envs/analysis/lib/python3.7/site-packages/dateutil)
patsy: 0.5.1 (/Users/user/miniconda/envs/analysis/lib/python3.7/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.1.0 (/Users/user/miniconda/envs/analysis/lib/python3.7/site-packages/matplotlib)
    backend: MacOSX 
cvxopt: Not installed
joblib: 0.13.2 (/Users/user/miniconda/envs/analysis/lib/python3.7/site-packages/joblib)

Developer Tools
================

IPython: 7.8.0 (/Users/user/miniconda/envs/analysis/lib/python3.7/site-packages/IPython)
    jinja2: 2.10.1 (/Users/user/miniconda/envs/analysis/lib/python3.7/site-packages/jinja2)
sphinx: Not installed
    pygments: 2.4.2 (/Users/user/miniconda/envs/analysis/lib/python3.7/site-packages/pygments)
pytest: Not installed
virtualenv: Not installed

</details>

user: AlexGroves
issue_6176: state: open
title: VAR or VARMAX support for lags of exogenous variable
body: To the best of my knowledge, it is not possible to add lags of the exogenous variables in either the VAR model or its state-space equivalent VARMAX.

Is there any known work-around to get a VARX(p,q) model with p lags of the endogenous variables and q lags of the exogenous variables.

If not, could someone please point out the easiest way to implement this? Existing issues/feature suggestions don't cover this issue and suggest that there is generally a lot of work to do on the VAR model.
user: MWSi
issue_6174: state: open
title: statsmodels.stats.power.tt_ind_solve_power able to return non-float types
body: #### Description

When passed a pandas.Series instead of the expected float value for effect_size, the function will return a pandas.Series. While I realize that my input type is incorrect, I feel the function should either return an error or float type, instead, it will return a pandas.Series.

#### Code Sample


```python
from statsmodels.stats.power import tt_ind_solve_power
import pandas as pd

power = tt_ind_solve_power(effect_size = pd.Series([.8], index = ['a'])
                   , nobs1 = 50
                   , alpha = .95
                   , power = None
                   , ratio = 1.0
                   , alternative = 'two-sided'
                  )

type(power)
```

#### Expected Output

```
ValueError(...
```

#### Actual Output
```
pandas.core.series.Series
```
user: vipolsky
issue_6172: state: open
title: ENH: Let statsmodels invoke x11 functions from x13as
body: #### Is your feature request related to a problem?

Hi everyone,

I’m trying to back-test my quant models, but the data were previously adjusted by using the x11 seasonal adjustment methodology.
Sensus.gov’s x13as program allows the user to invoke x11 function and lets the user to designate which mode of the seasonal adjustment decomposition to be performed. (Multiplicative, additive, pseudo-additive, log-additive)
The legacy models we have uses SAS Proc x12 with x11 statement, and I’d like to keep the seasonal adjustment methodology constant even if it’s outdated.
Is there a similar option in statsmodels where I can use the x11 functionalities and designate which mode to use?
Thank you!

#### Describe the solution you'd like

#### Describe alternatives you have considered

#### Additional context
user: mshin254
issue_6169: state: open
title: Interpret MANOVA result
body: I have successfully executed a MANOVA test (using the data from [here](http://chymera.eu/debug/statsmodels/manova.zip)), and with the following script:

```
import numpy as np
import pandas as pd
from statsmodels.multivariate.manova import MANOVA

df = pd.read_csv('data/trajectories.csv')
columns = df.columns
value_columns = [i for i in columns if '/' in i]

# convert exog to dummy
# https://github.com/statsmodels/statsmodels/issues/6155#issuecomment-530637971
exog = (np.asarray(df['session'])[:,None] == np.unique(df['session'])).astype(np.float64)
endog = np.asarray(df[value_columns])

model = MANOVA(endog=endog, exog=exog)
res = model.mv_test()
print(res.summary())
``` 

I get results which look like this:
```
                 Multivariate linear model
============================================================
                                                            
------------------------------------------------------------
           x0           Value  Num DF  Den DF F Value Pr > F
------------------------------------------------------------
          Wilks' lambda 0.6207 6.0000 55.0000  5.6007 0.0001
         Pillai's trace 0.3793 6.0000 55.0000  5.6007 0.0001
 Hotelling-Lawley trace 0.6110 6.0000 55.0000  5.6007 0.0001
    Roy's greatest root 0.6110 6.0000 55.0000  5.6007 0.0001
------------------------------------------------------------
                                                            
------------------------------------------------------------
           x1           Value  Num DF  Den DF F Value Pr > F
------------------------------------------------------------
          Wilks' lambda 0.1771 6.0000 55.0000 42.5804 0.0000
         Pillai's trace 0.8229 6.0000 55.0000 42.5804 0.0000
 Hotelling-Lawley trace 4.6451 6.0000 55.0000 42.5804 0.0000
    Roy's greatest root 4.6451 6.0000 55.0000 42.5804 0.0000
------------------------------------------------------------
                                                            
------------------------------------------------------------
           x2           Value  Num DF  Den DF F Value Pr > F
------------------------------------------------------------
          Wilks' lambda 0.4381 6.0000 55.0000 11.7547 0.0000
         Pillai's trace 0.5619 6.0000 55.0000 11.7547 0.0000
 Hotelling-Lawley trace 1.2823 6.0000 55.0000 11.7547 0.0000
    Roy's greatest root 1.2823 6.0000 55.0000 11.7547 0.0000
------------------------------------------------------------
                                                            
------------------------------------------------------------
           x3           Value  Num DF  Den DF F Value Pr > F
------------------------------------------------------------
          Wilks' lambda 0.2886 6.0000 55.0000 22.5942 0.0000
         Pillai's trace 0.7114 6.0000 55.0000 22.5942 0.0000
 Hotelling-Lawley trace 2.4648 6.0000 55.0000 22.5942 0.0000
    Roy's greatest root 2.4648 6.0000 55.0000 22.5942 0.0000
------------------------------------------------------------
                                                            
------------------------------------------------------------
           x4           Value  Num DF  Den DF F Value Pr > F
------------------------------------------------------------
          Wilks' lambda 0.2402 6.0000 55.0000 28.9928 0.0000
         Pillai's trace 0.7598 6.0000 55.0000 28.9928 0.0000
 Hotelling-Lawley trace 3.1629 6.0000 55.0000 28.9928 0.0000
    Roy's greatest root 3.1629 6.0000 55.0000 28.9928 0.0000
============================================================
```

Since they are five in number, I assume these are results for the levels of the session variable. What I find puzzling is that:

1. All session levels are significant, which is quite strange. Shouldn't at least one be treated as the baseline?
2. I would have expected MANOVA to give me some sort of estimates regarding the multivariate outcome --- i.e. does the variable have a significant effect on all of the outcomes, or some? Any idea how I could address this question?
user: TheChymera
issue_6168: state: open
title: Johansen test wrongly implemented
body: #### Describe the bug
The implementation in the function coint_johansen is simply incorrect when comparing to Johansen's approach. The usage of critical values from his work is therefore incorrect.
For a correct implementation, see for example the package called "Johansen".

1. One of the first lines in the function code is

endog = detrend(endog, det_order)

This procedure is incorrect and yields wrong test statistic. If one assumes a deterministic (polynomial) trend in the levels, one appends corresponding deterministic terms to the z-array. See the package Johansen for reference. The mapping of "p" in coint_johansen to "model" in Johansen is {-1:0, 0:1,1:3}

2. Usage of the variable f. What's the point is de-meaning the variables dx, lx, z?! If one leaves out the  lines

z = detrend(z, f)
dx = detrend(dx, f)
dx = detrend(lx, f)

the resulting variables r0t and rkt will remain unchanged.


#### Code Sample, a copy-pastable example if possible


```python
# Your code here that produces the bug
# This example should be self-contained, and so not rely on external data.
# It should run in a fresh ipython session, and so include all relevant imports.
```
<details>

**Note**: As you can see, there are many issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates.

**Note**: Please be sure you are using the latest released version of `statsmodels`, or a recent build of `master`. If your problem has been fixed in an unreleased version, you might be able to use `master` until a new release occurs. 

**Note**: If you are using a released version, have you verified that the bug exists in the master branch of this repository? It helps the limited resources if we know problems exist in the current master so that they do not need to check whether the code sample produces a bug in the next release.

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output

A clear and concise description of what you expected to happen.

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]

</details>

user: schapotschnikow
issue_6159: state: open
title: ENH: Improvements to new ARIMA-type estimators
body: Collection of follow-ups to #5827. These can/should be broken out into individual PRs. Many are relatively straightforward and would make a good first PR.

### General

- [x] Documentation (none was added in original PR).
- [x] Release notes.
- [ ] Example notebook.
- [x] Double-check how `sm.tsa.arima.ARIMA` works with `fix_params` (it should fail except when the fit method is `statespace`).
- [ ] Estimators that do not support seasonal models per se should support models where the only seasonal part is seasonal differencing.

### GLS

- [ ] Add support for fixed parameters
- [ ] Improve "Returns" documentation for `other_results`.
- [ ] Add documentation for why we have e.g. `include_constant` but not other trend specifications (e.g. that it is to maintain consistency with estimation methods with assumptions that require demeaned series).
- [ ] Fix the following test and put it back into the GLS test suite:

```python
@pytest.mark.low_precision('Test against Example 6.6.3 in Brockwell and Davis'
                           ' (2016)')
# @pytest.mark.xfail(reason="Source appears to find suboptimal parameters")
def test_brockwell_davis_example_663():
    # TODO: the parameters described by BD appear to be suboptimal (based on
    # llf computed from state space form), so that this test fails. Should try
    # to confirm with ITSM2000 (i.e. see if we can get it to find better
    # parameters closer to what we find, or compare llf, or something).
    # TODO: quite a slow test, and xfail anyway due to finding better
    # parameters...
    # Get the data, perform seasonal differencing
    endog = sbl.diff(12).iloc[12:]

    exog = pd.Series((sbl.index > '1983-01-01').astype(int),
                     index=sbl.index).diff(12).iloc[12:]

    res, _ = gls(endog, exog, order=(0, 0, 12), max_iter=3)

    assert_allclose(res.exog_params, -328.45, atol=1e-2)
    assert_allclose(res.ma_params,
                    [.219, .098, .031, .064, .069, .111, .081,
                     .057, .092, -0.28, .183, -.672], atol=1e-3)
    assert_allclose(res.sigma2, 12581, atol=1)
```

### Hannan Rissanen

- [ ] Better warnings / errors when series are short relative to lag length
- [x] Add support for fixed parameters
- [ ] Seems like we could add support for seasonal parameters in this model.
- [ ] Tests for the case with the bias-corrected estimator.

### Innovations MLE

- [ ] Add support for fixed parameters

### Innovations algorithm

- [ ] Add support for ARMA models; see Brockwell and Davis (2016) p.154 and Example 5.1.6.  This estimator should be feasible given the Cython versions of the general innovations algorithm that introduced in #5947.

user: ChadFulton
issue_6158: state: open
title: BinomialBayesMixedGLM predict function returns non-1D array
body: #### Describe the bug

I tried to use BinomialBayesMixedGLM to predict some values so that I can assess the prediction capability of the model. However, when putting in sample data, the output is not 1D predictions but instead the same shape as the input data. This seems to be a bug since the [expected output is a 1-dimensional array of predicted values](https://www.statsmodels.org/stable/generated/statsmodels.genmod.bayes_mixed_glm.BinomialBayesMixedGLM.predict.html#statsmodels.genmod.bayes_mixed_glm.BinomialBayesMixedGLM.predict).

#### Code Sample, a copy-pastable example if possible


```
In[1] : from statsmodels.genmod import bayes_mixed_glm as b
        import pandas as pd
        data = pd.DataFrame(my_private_data, columns=['graduates_next_semester', 'first_course_year', 'cumulative_avg_grade', 'hs_gpa', 'female'])
        formula = 'graduates_next_semester ~ C(first_course_year) + cumulative_avg_grade + hs_gpa + C(female)'
        random = {'a':'0 + C(first_course_year)', 'b':'0 + C(first_course_year)*hs_gpa'}
        model = b.BinomialBayesMixedGLM.from_formula(formula, random, data)
        results = model.fit_vb()

In[2] : data.values
Out: array([[ 0.        ,  0.        , -0.16846258, -3.26481235,  0.        ],
       [ 0.        ,  0.        ,  0.25580621,  0.60181999,  1.        ],
       [ 1.        ,  4.        ,  1.64888846,  1.66018495,  0.        ],
       ...,
       [ 0.        , 20.        ,  0.88437497,  0.15070583,  1.        ],
       [ 0.        , 20.        , -1.72232004, -1.76500254,  0.        ],
       [ 1.        , 20.        ,  0.82043374,  0.27632605,  1.        ]])
In[3] : model.predict(data)
Out : array([[0.5       , 0.5       , 0.60994004, 0.00872048, 0.45798368],
       [0.5       , 0.5       , 0.7986418 , 0.01236763, 0.77830333],
       [0.5       , 0.5       , 0.14909222, 0.00507625, 0.93394245],
       ...,
       [0.73105858, 0.5       , 0.72314575, 0.00128766, 0.86811284],
       [0.73105858, 0.5       , 0.34848906, 0.01597394, 0.15157257],
       [0.73105858, 0.5       , 0.71658333, 0.00144936, 0.86061816]])
```

user: mnky9800n
issue_6154: state: open
title: Why am I getting LinAlgError: Singular matrix error on some datasets when I call "md.fit(reml=False)", but it works fine for other datasets.
body: Please help. I'm confused.

 I am getting 'LinAlgError: Singular matrix' error on some datasets when I call "md.fit(reml=False)", but it works fine for other datasets.

user: damilolah
issue_6153: state: open
title: Do I need to perform cross validation on Mixed Effect Models
body: Hello everyone,

I'm new to mixed effect models and I've followed this tutorial: https://www.statsmodels.org/stable/mixed_linear.html


My code looks like this:

import statsmodels.formula.api as smf

md = smf.mixedlm("Weight ~ Time+color", data, groups=data["Dogs"])

mdf = md.fit()

Question 1: Do I need to do cross validation? If so how do I perform this?

Question 2: mdf.aic returns 'nan'. How can get the AIC on this model created.

Thanks.
user: damilolah
issue_6147: state: open
title: prediction interval with VAR model
body: Some of the models implemented in this wonderful package seem to have a `get_prediction` and `conf_int` methods which can return the prediction or perhaps the confidence interval (for the model parameters).

I am just using the VAR model to model a couple of interdependent time series and I was wondering if it would be possible to get the prediction intervals on the forecast for each of the conditional series?

So, for example, I am using time series for number of customers and the bus transit times in a town and then using the model to predict both the footfall of the customers and the bus arrival times. Would it be possible to get the prediction interval (say for different confidence levels) for these two variables?
user: dagap
issue_6146: state: open
title: VIF & add_constant
body: The VIF function assumes the presence of a constant in the data for OLS. Should we call this out in the documentation?

[VIF Function](https://github.com/statsmodels/statsmodels/blob/fad69bc1138e6b795674329d3ffc9e93f8a46a93/statsmodels/stats/outliers_influence.py#L150)
user: hiramf
issue_6140: state: open
title: Linear Mixed Effects Models: numpy.linalg.LinAlgError: Singular matrix
body: I'm going to find the correlation between the contribution of each type and the 'score'. So I use the proportion of each type as X, the score as Y. I have 150 types and obviously each sample has: x1+x2+...x150=1. Y(score) ∈(0,10) and Y is float. Xi ∈[0,1] and also is float. Each sample has the lable 'region' which shows the area it comes from. I want to use the mixed liner model, where the 150 types(x1,x2...x150) make fixed effect and the 'region' makes the random effect.
I have tried:

    md = smf.mixedlm(formula_md,df,groups=df['region'])
    mdf = md.fit() 

where the formula_md is:

    score ~ class_0 + class_1+ class_2 ... +class_149 - 1

but got the error: numpy.linalg.LinAlgError: Singular matrix
 Does this mean that there is collinearity in my data? I think the answer is yes, I have following questions:
1. Is there any ways for me to fix the collinearity, such as fine the samples pair which has collinearity. After that maybe I can drop some samples.
2. Does this mean that my data situation is not suitable for mixed linear models? But if I really need to take into account the impact of spatial relationships, whatshould I do?
3. I think the statsmodels use Maximum Likelihood Estimation to fix the mixed liner model and use linalg.py( of numpy) to solve a linear matrix equation `ax = y`. But the file (linalg.py) requests `a` must be square and of full-rank, i.e., all rows (or, equivalently,  columns) must be linearly independent; if either is not true, use  `lstsq` for the least-squares best "solution" of the     system/equation.' So probably OLS is suitable for my data?
user: KaninchenM
issue_6138: state: open
title: Linear Mixed Effects Models: can't find data of demo 
body: run Linear Mixed Effects Models as :

    data = sm.datasets.get_rdataset("dietox", "geepack").data
    md = smf.mixedlm("Weight ~ Time", data, groups=data["Pig"])
    mdf = md.fit()
    print(mdf.summary())

got the error: urllib.error.URLError: <urlopen error timed out>. 
When directly accessing the url: https://raw.githubusercontent.com/vincentarelbundock/Rdatasets/master/csv/geepack/, the response is '400: Invalid request'
user: KaninchenM
issue_6135: state: open
title: Goodness of fit test for Weibull with unknown parameters
body: #### Is your feature request related to a problem? Please describe
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

I would like to be able to test data to see whether it comes from a Weibull distribution with unknown parameters

#### Describe the solution you'd like
A clear and concise description of what you want to happen.

According to https://en.m.wikipedia.org/wiki/Anderson%E2%80%93Darling_test 

“Any other family of distributions can be tested but the test for each family is implemented by using a different modification of the basic test statistic and this is referred to critical values specific to that family of distributions. The modifications of the statistic and tables of critical values are given by Stephens (1986)[2] for the exponential, extreme-value, Weibull, gamma, logistic, Cauchy, and von Mises distributions. Tests for the (two-parameter) log-normal distribution can be implemented by transforming the data using a logarithm and using the above test for normality. Details for the required modifications to the test statistic and for the critical values for the normal distribution and the exponential distribution have been published by Pearson & Hartley (1972, Table 54). Details for these distributions, with the addition of the Gumbel distribution, are also given by Shorak & Wellner (1986, p239). Details for the logistic distribution are given by Stephens (1979). A test for the (two parameter) Weibull distribution can be obtained by making use of the fact that the logarithm of a Weibull variate has a Gumbel distribution.”

There is already a test for the normal distribution with unknown parameters https://www.statsmodels.org/stable/generated/statsmodels.stats.diagnostic.normal_ad.html#statsmodels.stats.diagnostic.normal_ad and a separate KS based one for normal and exponential random variables https://www.statsmodels.org/stable/generated/statsmodels.stats.diagnostic.lilliefors.html#statsmodels.stats.diagnostic.lilliefors

It would be great to add Weibull to the list.

#### Describe alternatives you have considered
A clear and concise description of any alternative solutions or features you have considered.

There is an R package that has a KS based test https://cran.r-project.org/web/packages/KScorrect/index.html . An Anderson Darling based one might be preferable however.

#### Additional context
Add any other context about the feature request here.
user: lesshaste
issue_6121: state: open
title: ECDF doesn't handle missing data well.
body: ![image](https://user-images.githubusercontent.com/24930289/63455955-d8aeb480-c41b-11e9-8a57-9d44cb1246c9.png)

user: jiafengkevinchen
issue_6115: state: open
title: DOC - add dataset and examples on conditional models
body: #### Is your feature request related to a problem? Please describe
We need a dataset + examples on usage of `statsmodels.discrete.conditional_models` family - > Conditional logit, conditional mnl, conditional Poisson.

#### Describe the solution you'd like
Wollridge data on crime [here](https://www.cengage.com/cgi-wadsworth/course_products_wp.pl?fid=M20b&product_isbn_issn=9781111531041) fits for this purposes

I would like to do this one myself, however, I could not find any info about adding datasets or examples. Anyone familiar with it can enlighten me on that and I will be grateful
user: emilmirzayev
issue_6086: state: open
title: Initial value calculation for seasonality of ExponentionalSmoothing need a check and better error message
body: In "holtwinters.py" "initial_values" before this line:
`lead, lag = y[m:m + m], y[:m]
`
there should be a check like so:
`if y.shape[0] < m*2: raise ValueError("Some error")`
user: ozeno
issue_6083: state: open
title: Goodness-of-fit test for MNLogit results
body: #### Is your feature request related to a problem? Please describe
I used MNLogit for multinomial logistic regression.
I want to use deviance and pearson_chi2 for goodness of fit test.
But these two values were not found in the MNLogit results.How can I get these two values?
I konw that there are these two attributes in the GLM results.Like this:
![image](https://user-images.githubusercontent.com/1759365/62603653-f0207480-b928-11e9-9e58-0469074dbe8e.png)
Thanks in advance!

code show as below:
```Python
import pandas as pd
import statsmodels.api as sm
subject = ['数学','自然科学','自然科学','数学','数学','自然科学','数学','数学',
              '艺术','自然科学','艺术','数学','自然科学','自然科学','艺术','自然科学',
              '自然科学','自然科学','艺术','数学','数学','艺术','艺术','数学','艺术',
              '艺术','数学','自然科学','数学','艺术']
age = [10,10,10,10,10,10,10,11,11,11,11,11,11,11,11,12,12,12,12,12,12,12,
           13,13,13,13,13,13,13,13]
rsp = pd.Series(subject,name='sub')
pvar_con = pd.Series(age,name='age')
X = sm.add_constant(pvar_con)
model = sm.MNLogit(rsp,X)
rslt = model.fit()
print(rslt.summary())
#deviance = rslt.deviance
#pearson_chi2 = rslt.pearson_chi2
```
user: leavor
issue_6076: state: open
title: KDEMultivariateConditional.cdf inefficient and unnecessary looping
body: #### Is your feature request related to a problem? Please describe
The code https://www.statsmodels.org/stable/_modules/statsmodels/nonparametric/kernel_density.html#KDEMultivariateConditional.cdf has many unnecessary loops which makes the code a lot slower than it should be.

        for i in range(N_data_predict):
            mu_x = gpke(self.bw[self.k_dep:], data=self.exog,
                        data_predict=exog_predict[i, :],
                        var_type=self.indep_type) / self.nobs
            mu_x = np.squeeze(mu_x)
            cdf_endog = gpke(self.bw[0:self.k_dep], data=self.endog,
                             data_predict=endog_predict[i, :],
                             var_type=self.dep_type,
                             ckertype="gaussian_cdf",
                             ukertype="aitchisonaitken_cdf",
                             okertype='wangryzin_cdf', tosum=False)

            cdf_exog = gpke(self.bw[self.k_dep:], data=self.exog,
                            data_predict=exog_predict[i, :],
                            var_type=self.indep_type, tosum=False)
            S = (cdf_endog * cdf_exog).sum(axis=0)
            cdf_est[i] = S / (self.nobs * mu_x)

#### Describe the solution you'd like
Something on the line with

         cdf_exog = gpke(self.bw[self.k_dep:], data=self.exog,
                            data_predict=exog_predict,
                            var_type=self.indep_type, tosum=False)
         mu_x = np.squeeze(cdf_exog.sum())
         cdf_endog = gpke(self.bw[0:self.k_dep], data=self.endog,
                             data_predict=endog_predict,
                             var_type=self.dep_type,
                             ckertype="gaussian_cdf",
                             ukertype="aitchisonaitken_cdf",
                             okertype='wangryzin_cdf', tosum=False)

          S = (cdf_endog * cdf_exog).sum(axis=0)
          cdf_est[i] = S / (mu_x)

To achieve this, the gpke function has to be rewritten by np.einsum

If too hard, at least mu_x  should be calculated directly from cdf_exog.
If too hard, at least loop through all np.unique(cdf_exog) and np.unique(cdf_endog). This would decrease the complex by a square root.

user: cc20002002
issue_6065: state: open
title: UnobservedComponents: time-varying regression coefficients
body: It is impossible to specify state disturbance variances for exogeneous regressors in `UnobservedComponents` (passed by `exog` parameter). Currently, coefficients of exogeneous parameters can only be computed using least squares (`mle_regression=True`) or recursive least squares (`mle_regression=False`).

It would be great to have an option for time-varying regression coefficients. To do this, we need to associate disturbance variables to exogeneous regressors:
<a href="https://www.codecogs.com/eqnedit.php?latex=\begin{cases}&space;y_t&space;=&space;...&space;&plus;&space;b_t&space;X_t&space;&plus;&space;\epsilon_t\\&space;\beta_{t&plus;1}&space;=&space;\beta_t&space;&plus;&space;\eta_t&space;\end{cases}" target="_blank"><img src="https://latex.codecogs.com/gif.latex?\begin{cases}&space;y_t&space;=&space;...&space;&plus;&space;b_t&space;X_t&space;&plus;&space;\epsilon_t\\&space;\beta_{t&plus;1}&space;=&space;\beta_t&space;&plus;&space;\eta_t&space;\end{cases}" title="\begin{cases} y_t = ... + b_t X_t + \epsilon_t\\ \beta_{t+1} = \beta_t + \eta_t \end{cases}" /></a>

It can be done by addining an additional flag (e.g., `time_varying_regression`) to `UnobservedComponents` constructor. If the flag is True, regression coefficients are added to the state vector; at the same time, the same number of state disturbances is added (state disturbances are stored in `self.parameters_state_cov`, if I'm not mistaken). The resulting Kalman filter will be very similar to the one constructed with `mle_regression=False`. This solution should only require a moderate effort.

Time-varying regression coefficients are briefly mentioned in the section 6.2.1. of Time Series Anaylysis by State Space Methods (J. Durbin, S. J. Koopman).  
user: mysterious-ben
issue_6061: state: open
title: FAQ: Contributions Welcome
body: ### Contributing

`statsmodels` welcomes contributions of all sizes, including:

#### docstring fixes, which can be very small
Documentation is hard, and many eyes help fix typos, missing parameters, and many other issues.  We are always happy to accept docstring improvements. The doc builds on [Travis CI](https://travis-ci.org/statsmodels/statsmodels) contains a log of docstrings that aren't up to standards (about 6000 issues at the time of writing).  Improvements to these are especially welcome. 

#### notebooks
Examples are incredibly useful.  If you have a written a notebook that highlights using `sm` in ways that are not already covered in examples, please share these to improve the documentation. 

#### bug fixes
Bug fixes are always welcome.  There are plenty of opportunities on the issue tracker to contribute.

#### enhancements
`statsmodels` is community-driven, and so enhancements are welcome.  Large enhancements, for example, new models should:
* have clean, PEP8 compliant code;
* be tested against other reference implementations (R, SAS, Stata, etc.);
* be added to the appropriate section of the docs
* have a notebook that illustrates how to use the new model. 

Smaller enhancements, for example, a new option on an existing method, are also welcome. More minor enhancements have a lower bar and should:

* have clean, PEP8 compliant code;
* be tested against other reference implementations (R, SAS, Stata, etc.) if possible, or checked for correctness if not (for example, if adding an option to select a subset of output from a function).

#### cleaning
Pure code clean-ups are generally discouraged. (Locally) cleaning up code as part of contributing to one of the other goals is encouraged.
user: bashtage
issue_6032: state: open
title: Getting a static and dynamic forecast from a VAR model
body: Hi,
I would like to get a dynamic forecast (i.e next step forecasts are based on forecasted endogenous values) using a VAR model.
I dont see any way to do this using VAR class. I see that there is a method predict in VARMAX class of statsmodels. I tried using VARMAX, but fitting the same
data gives an error. This is the code below: 

import pandas as pd
from statsmodels.tsa.api import VAR
import statsmodels.api as sm

df_data=pd.read_excel("df_data.xlsx",index_col=0)
[df_data.xlsx](https://github.com/statsmodels/statsmodels/files/3443380/df_data.xlsx)

data=df_data.values
varmodel 	= VAR(data)
results 	= varmodel.fit(12,trend='nc')
static_forecast=results.forecast(df_data[:"1998-01-01"][:-1].values, 60)

for dynamic forecast, I want to use the predict method of VARMAX class and use the same data, but it gives me this error while fitting the model:
mod_dyn= sm.tsa.VARMAX(endog=data,order=(12,0),trend='nc')
res_dyn = mod_var.fit(disp=False)

ValueError: Non-stationary starting autoregressive parameters found with `enforce_stationarity` set to True. 

When I set enforce_stationarity to false, it gives me this error:
LinAlgError: Schur decomposition solver error. 

Also I was looking at DynamicVAR for dynamic forecast but I believe it estimates an expanding or rolling VAR, which I dont need.


user: vikaskumar30
labels: type-bug-wrong: serious bugs that silently return incorrect numbers
issue_6021: state: open
title: WIP: Overflow protected exponential functions
body: - [ ] closes #xxxx
- [ ] tests added / passed. 
- [ ] code/documentation is well formatted.  
- [ ] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy/dev/gitwash/development_workflow.html#writing-the-commit-message). 

<details>


**Notes**:

* It is essential that you add a test when making code changes. Tests are not 
  needed for doc changes.
* When adding a new function, test values should usually be verified in another package (e.g., R/SAS/Stata).
* When fixing a bug, you must add a test that would produce the bug in master and
  then show that it is fixed with the new code.
* New code additions must be well formatted. Changes should pass flake8. If on Linux or OSX, you can
  verify you changes are well formatted by running 
  ```
  git diff upstream/master -u -- "*.py" | flake8 --diff --isolated
  ```
  assuming `flake8` is installed. This command is also available on Windows 
  using the Windows System for Linux once `flake8` is installed in the 
  local Linux environment. While passing this test is not required, it is good practice and it help 
  improve code quality in `statsmodels`.
* Docstring additions must render correctly, including escapes and LaTeX.

</details>

user: bashtage
issue_5978: state: open
title: ENH: Partial least squares (PLS) regression
body: How to apply StatsModels for partial least squares regression?
There is PLSRegression in sklearn, but some statistical results are missing, compared to StatsModels.GLM
user: leavor
issue_5904: state: open
title: STATA Styled Conditional Logit
body: #### Is your feature request related to a problem? Please describe
When fitting conditional logit, if for some features there is no within-group variance it raises an error:
```
ValueError: need covariance of parameters for computing (unnormalized) covariances
```
However, in STATA, it does not raise an error, it rather informs that variable A has no withing group variance, and it will be dropped. Then, it fits the model **without** that variable, mentioning it also in the output. Example (look at the note and also the last variable in results):

```
. clogit choice scaledtrip scaledairlines scaledflights scaledprice containslcc clustersizetomenusize clust
> ercounttomenusize clusterdisptomenudisp , group( individual )
string variables not allowed in varlist;
clusterdisptomenudisp is a string variable
r(109);

. clogit choice scaledtrip scaledairlines scaledflights scaledprice containslcc clustersizetomenusize clust
> ercounttomenusize, group( individual )
note: clustercounttomenusize omitted because of no within-group variance.

Iteration 0:   log likelihood = -20772.193  
Iteration 1:   log likelihood = -19968.164  
Iteration 2:   log likelihood = -19932.252  
Iteration 3:   log likelihood = -19932.117  
Iteration 4:   log likelihood = -19932.117  

Conditional (fixed-effects) logistic regression

                                                Number of obs     =    410,201
                                                LR chi2(6)        =   14317.91
                                                Prob > chi2       =     0.0000
Log likelihood = -19932.117                     Pseudo R2         =     0.2643

----------------------------------------------------------------------------------------
                choice |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-----------------------+----------------------------------------------------------------
            scaledtrip |  -.4886174   .0232772   -20.99   0.000    -.5342399    -.442995
        scaledairlines |  -.7686357   .0365782   -21.01   0.000    -.8403276   -.6969439
         scaledflights |    -1.2399   .0327024   -37.91   0.000    -1.303996   -1.175805
           scaledprice |  -.5370699   .0134553   -39.92   0.000    -.5634418    -.510698
           containslcc |   .6804741   .0404264    16.83   0.000     .6012399    .7597083
 clustersizetomenusize |   -1.27629   .1263185   -10.10   0.000     -1.52387   -1.028711
clustercounttomenusize |          0  (omitted)
----------------------------------------------------------------------------------------

```

I wonder can it be implemented in statsmodels in a way it is done in STATA?
How complex it can be??
user: emilmirzayev
issue_5901: state: open
title: Error in test for sqrt_lasso in OLS.fit_regularized()
body: 
https://github.com/statsmodels/statsmodels/blob/c1e30d9534dbe56346e50517da4dacf023a4aad7/statsmodels/regression/tests/test_regression.py#L1311

Looking at the Belloni et al paper (equation 8), this matches the definition of lambda, but the penalty weight is lambda / n.  So I believe in this test alpha is off by a factor of n.  I would expect:
alpha = 1.1 / np.sqrt(n) * norm.ppf(1 - 0.05 / (2 * p))
It stands to reason that alpha should decay as n grows.  I realize this is just a test, but thought it was worth clarifying.

Thanks!
user: quantvol
issue_5896: state: open
title: Measures of effect size
body: When using statsmodels for hypothesis testing there are no effect sizes calculated/outputted. These are very important when it comes to communicating results because although a p value might be significant, the effect itself can still be tiny and practically meaningless.

It would be really good if measures of effect size could be included in the model summaries. I'm particularly thinking in terms of t-tests and ANOVAs. In the case of t-tests, the most frequently used is Cohen's d, while for ANOVA we have η^2, partial η^2, and ω^2.
user: lewisdunne
issue_5889: state: open
title: quantile_regression should accept optional weight parameter 
body: The quantile regression should accept an optional `weight` parameter. Stata supports this: https://www.stata.com/manuals13/rqreg.pdf. I believe this changes the definition of a[ quantile from count to density](https://github.com/statsmodels/statsmodels/blob/a11c03d4af842dd7e5d3e8abaed621fe976f819a/statsmodels/regression/quantile_regression.py#L175-L176). 
user: cancan101
issue_5872: state: open
title: FAQ: proportions_ztest specify prop_var
body: Hi, 

I am a newbie to statsmodels, and have a question on the proportions_ztest function. 
Its definition suggests that the prop_var can be specified and directly uses for computing the z-value. My understanding is that if the prop_var is the same as the sample var, the function should return the same result, but it seems it doesnt. Did i miss anything? can anyone help me to understand?

Thanks

1. proportions_ztest(30, 100,.25)
result: z=1.09
2. proportions_ztest(30, 100,.25,prop_var=0.21) - 0.21 = p*(1-p) = 0.3*(1-0.3)
result: z=1.23

"prop_var : False or float in (0, 1)
    If prop_var is false, then the variance of the proportion estimate is
    calculated based on the sample proportion. Alternatively, a proportion
    can be specified to calculate this variance. Common use case is to
    use the proportion under the Null hypothesis to specify the variance
    of the proportion estimate"
user: josephzcl
issue_5870: state: open
title: How to compute "mu" and "sigma" of Likelihood by  SAS NLMIXED?
body: I'd like to calculate the values “mu” and “sigma” of likelihood for normal distribution using the following SAS program, NLMIXED. I tried it, but I guess “sigma” wasn’t properly calculated. According to UNIVARIATE, it’s 18.8846. So, I think the “sigma” would be around “18” as well. Where’s the wrong? Would you please advise me about something?
This is the reference site.

> https://blogs.sas.com/content/iml/2017/06/14/maximum-likelihood-estimates-in-sas.html

```
data dt00;
 input x @@;
 cards;
2 2 3 3 3 4 4 15 9 13 10 7 11 5
;
run;

proc nlmixed data = dt00;
parms mu 1 sigma 1;
bounds 0 < sigma;
N = 14;
L = - N/2 * log(sigma) - 1 / 2 * sum((x - mu)**2)/sigma;
model x ~ general(L);
run;

```
Parameter Estimate
mu 6.5000 
sigma 1.2526 

```
proc univariate data = dt00 noprint;
 var x;
 output out = st00 mean = mean var = var;
run;
```

mean 6.5
var 18.8846
user: 51sep
issue_5861: state: open
title: How to compute pseudo R^2 for GLM estimators
body: Hi, How can I compute the psquared for GLM estimators? I don't find the result in the summary as opposed to Logit and other estimators.
Thank you for giving me what values to use in the result to manually compute psquared.

Thank you.
user: jpainam
issue_5800: state: open
title: Results from Statsmodels Vector Auto Regression (VAR)
body: I am trying to use VAR from Statsmodels library of Python to analyse some time series problem. It is basically a multivariate time series, that predicts sales figures of different products at time t, from previous values ( at t-1, t-2 etc.) of the same products.

I have followed the example here:

https://www.statsmodels.org/stable/vector_ar.html

However one requirement to use VAR is given below:

**The VAR class assumes that the passed time series are stationary. Non-stationary or trending data can often be transformed to be stationary by first-differencing or some other method.**

As a result I did make my time series stationary by differencing and then built the model. Just wondering when I obtain the results by using results.forecast, how do I interpret the results to eliminate the effect of differencing? How do I get back the result as original (i.e. without differencing)? Is there any method within the VAR module?

Any help will be much appreciated.

Regards, Anindya
user: aschattopadhyay
issue_5788: state: open
title: Between RMAnova
body: Since #3303, there is a much-appreciated Repeated-measured ANOVA in statsmodels. However, it so far does not allow for between factors.
user: jona-sassenhagen
issue_5786: state: open
title: TST: test_glm.test_tweedie_EQL_upper_limit doesnt use `scale` inside loop
body: It looks like the existence of this loop at all is an artifact of being copy/pasted from test_tweedie_EQL_poisson_limit.

Please advise.
user: jbrockmendel
issue_5777: state: open
title: Fit multiple trials with BinomialBayesMixedGLM 
body: Allow users to pass in a two dimensional array of total successes and total failures as the `endog` parameter as in statsmodels.api.GLM. 

I believe that the lines that need to be rewritten are 

https://github.com/statsmodels/statsmodels/blob/master/statsmodels/genmod/bayes_mixed_glm.py#L329 
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/genmod/bayes_mixed_glm.py#L625-L626
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/genmod/bayes_mixed_glm.py#L582 
user: zscore
issue_5776: state: open
title: Better Documentation for BinomialBayesMixedGLM
body: The documentation for BinomialBayesMixedGLM is borrowed from its parent class and is inaccurate. We should tidy it up so that it's easier to understand how to use this class.
user: zscore
issue_5773: state: open
title: SUMM: Roundup of sandbox exposure, unused files, redundant code, ...
body: Collecting these in one place to avoid Issue proliferation; will update as appropriate.

Sandbox
---------
Parts of sandbox that are imported from non-sandbox (cc @ChadFulton if you were wondering how much of the code implicitly relies on sandbox):

- [ ] `sandbox.stats.multicomp`
  - [ ] `stats.multicomp` imports `tukeyhsd` and `MultiComparison` (there is only one function defined directly in `stats.multicomp`)
  - [ ] `examples.try_tukey_hsd` gets `tukeyhsd` and `MultiComparison` directly from the sandbox module, could get it from the non-sandbox file.
  - [ ] `stats.contrast` imports `contrast_allpairs` (also duplicated in `sandbox.stats.contrast_tools`)
  - [ ] `stats.tests.test_qsturng` uses `get_tukeyQcrit`
- [ ] `sandbox.tsa.fftarma.Armafft`
    - [ ] In several of the places this is imported it is only for the `generate_sample` method, which is defined on the non-sandbox parent class `ArmaProcess`.  Use `ArmaProcess` directly.
   - [ ] The only remaining place where `fftarma` is imported is in `test_arima_process`, where it has two tests for `Armafft`.  These could be moved to a test file in sandbox specific to `fftarma`.
- [ ] `sandbox.stats.diagnostic` almost all of `stats.diagnostic` comes from here.
  - [ ] The _good_ news is that aside from this pass-through, the only direct import from `sandbox.diagnostics` is in examples/ex_arch_canada.py for `acorr_lm`.
- [ ] `statsmodels.sandbox.regression.predstd` is imported in numerous test and example files to get `wls_prediction_std` (looks like 10 imports total, 13 if we count notebook files)
    - Also in `stats.outliers_influence` and `graphics.regressionplots` (i.e. user-facing code)
- [ ] `sandbox.regression.penalized` is imported in several test files for `TheilGLS`.  This includes a `test_theil` file in `regression.tests` that sends mixed messages as to the status of this file.
- [ ] `sandbox.tsa.garch` is imported in statsmodels/examples/tsa/ex_arma.py for its `Arma` class
- [ ] `sandbox.nonparametric.kernels` is imported into 2 files in `nonparametric` and 3 files in `nonparametric.tests`.
   - `sandbox.nonparametric` files are imported into 6 examples/ files
- [ ] `sandbox.distributions.mv_normal` is imported into `distributions.mixture_rvs` and `distributions.tests.test_mixture`
  - [ ] `sandbox.distributions.sppatch` is imported into examples/ex_generic_mle_tdist.py
    - [ ] `sppatch` modified scipy objects in-place.  Can we avoid this?
- [ ] `sandbox.tools.cross_val.LeaveOneOut` is imported into `stats.outlier_influence`.  
This class is also duplicated in `nonparametric._kernel_base`
Slightly different behavior with the same name.
- [ ] `sandbox.stats.runs` provides `Runs`, `runstest_1samp`, and `runstest_2samp` directly to `stats.api`.
  - [ ] The same three are imported in `stats.tests.test_nonparametric`, along with `sandbox.stats.runs.mcnemar`
- [ ] `statsmodels.sandbox.panel.random_panel` is imported for `PanelSample` in `regression.tests.test_theil`

De Facto Sandbox
------------------
Non-Sandbox code that belongs in sandbox or dustbin:
- [ ] `nonparametric.smoothers_lowess_old` (#5142)
- [ ] `tsa.descriptivestats`
- [ ] `tsa.mlemodel` (#5139)
- [ ] `tsa.arma_mle`
- [ ] `tsa.varma_process`
- [ ] `miscmodels.try_mlecov`
- [ ] `iolib.stata_summary_examples` (#5061)
- [ ] `iolib.tests.test_summary_old` (#5064)

Chopping Block
----------------
Partially moved from #5137
Files that appear unlikely to become useful.
- [ ] README_l1.txt (#4957)
- [ ] statsmodels/
  - [x] info.py (#4981)
  - [x] examples/t_est_rlm.py (#5109, #5100)
  - [ ] examples/tsa/lagpolynomial.py
  - [ ] graphics/plot_grids.py (https://github.com/statsmodels/statsmodels/issues/5137#issuecomment-445551612)
  - [ ] statsmodels/satasets/anes96/src/anes96.csv appears to be a duplicate
- [ ] statsmodels/sandbox/
  - [x] tests/datamlw.py (#4941)
  - [x] tests/model_results.py (#5070)
  - [x] examples/ex_kaplan_meier.py (#5071)
  - [ ] mle.py - starts with "What's the origin of this file? It is not ours"
  - [ ] nos4.mtx - only used by mle.py
  - [ ] mlogitpath.lyx - no idea why we would need a lyx file
  - [ ] formula.py - supplanted by patsy (#4940, #5692)
  - [ ] examples/ex_formula.py - based on formula.py (#5692)
  - [ ] examples/ex_formula_factor.py - based on formula.py (#5692)
  - [ ] contrast_old.py - only used by formula (#5692)
  - [ ] tools_old.py - only used by formula (#5692)
  - [ ] dataset_notes.rst - likely redundant with README in `datasets` directory
  - [ ] bspline.py (imports `from models import _hbspline` which does not exist AFAIK)
  - [ ] mcevaluate/mcresuts_arma1.txt
  - [ ] regression/ols_anova_original.py - OK I haven't looked it this closely, but it looks like it was superceded and then never removed.
  - [ ] regression/notes_runml.txt
  - [ ] tests/test_bspline.py.txt - if no one noticed this was a .txt, cant be too important
  - [ ] tools/TODO.txt
  - [ ] tsa/notes_organize.txt - still refers to scikits.statsmodels
  - [ ] stats/notes_fdr.txt
  - [ ] regression/sympy_diff.py
  - [ ] panel/sandwich_covariance.py
- [ ] tools/
  - [ ] generate_formula_api.py (#5118)
  - [ ] code_maintenance.py
  - [ ] gh_api.py
  - [ ] github_statspy
  - [ ] matplotlibrc.qt4
  - [ ] matplotlibrc.qt5
  - [ ] README.rst
  - [ ] update_web.py
  - [ ] binstar/
  - [ ] check_dirs.py
  - [ ] backport_pr.py
- [ ] docs/
  - [ ] GLMNotes.lyx - ? not touched since 2012
  - [ ] GLMNotes.pdf - ? not touched since 2012
  - [ ] fix_longtable.py - ? not touched since 2012, appears unused
  - [ ] source/glm_techn1.rst.TXT - "Just a placeholder", last edit 2013
  - [ ] source/glm_techn2.rst.TXT - "Just a placeholder", last edit 2013
  - [ ] source/regression_techn1.rst.TXT- "Just a placeholder", last edit 2013

Duplication
-----------
Copy/Pasted code that should be imported instead:
- [ ] `sandbox.tsa.garch` copies all of `LikelihoodModel` (#4422)
- [ ] `LeaveOneOut` class in both `sandbox.tools.cross_val` and `nonparametric._kernel_base`
- [ ] `LikelihoodModel._fit_colinear` comment "Copied from PR #2380 remove when merge" (#5023)
- [ ] `gof.gof_binning_discrete` comment "copy/paste, remove code duplication when it works" (#5125)
- [ ] `base._parameter_inference._lm_robust` comment "this is a copy from stats._diagnostic_other" (#5772)

Disabled Code
---------------
Commenting out code is a way of temporarily disabling it that tends to become permanent and make things more difficult for subsequent readers.
- [ ] `tsa.interp.denton.denton`
- [ ] `discrete.discrete_model.Weibull`
- [ ] `sm.open_help` (#5134)

Namespace Issues
-------------------
- [ ] `tools.tools` namespace is counter-intuitive (#4992)
- [ ] `tools._testing` and `tools.testing`: either choose just one or make a clear distinction between what belongs where (#4876)
- [ ] What differentiates ...?:
  - examples/
  - statsmodels/examples/
  - statsmodels/sandbox/examples/
  - statsmodels/sandbox/distribution/examples/
  - docs/source/plots/

Unsorted
---------
- [ ] `miscmodels.tests.test_generic_mle` contains tests for GenericLikelihoodModel, not anything in `miscmodels`  (#5763).  If, as I suspect, `miscmodels` serves primarily as tests/examples for `GenericLikelihoodModel`, that should be made explicit and implement systematically.
- [ ] Many files like e.g. `sandbox.regression.example_kernridge` that may belong in e.g. andbox/regression/examples/?
- [ ] Overlap between datasets/README.txt, tools/dataset_rst.py, sandbox/dataset_notes.rst, possibly others in docs/?
user: jbrockmendel
issue_5766: state: open
title: test_qsturng.test_all_to_tbl skipped with "remove from testsuite" comment
body: ```
    #remove from testsuite, used only for table generation and fails on
    #Debian S390, no idea why
    @pytest.mark.skip
    def test_all_to_tbl(self):
        from statsmodels.stats.libqsturng.make_tbls import T,R
        ps, rs, vs, qs = [], [], [], []
        for p in T:
            for v in T[p]:
                for r in iterkeys(R):
                    ps.append(p)
                    vs.append(v)
                    rs.append(r)
                    qs.append(T[p][v][R[r]])

        qs = np.array(qs)
        errors = np.abs(qs-qsturng(ps,rs,vs))/qs
        assert_equal(np.array([]), np.where(errors > .03)[0])
```

What do we make of this "remove from testsuite" comment?  Should we enable the test and conditionally xfail on Debian S390 like we did for the lme test on OSX?
user: jbrockmendel
issue_5763: state: open
title: miscmodels.tests.test_generic_mle doesnt import miscmodels
body: If the tests are for GenericLikelihoodModel, then this probably belongs in base/tests/.
user: jbrockmendel
issue_5755: state: open
title: SUMM: Non-reproducible file references
body: Most of these are paths to R or Stata scripts that are used to create a results files.  These need to be pointed towards relative paths in the repo in order for these to be reproducible.

- [ ] koul_and_mc.py calls `np.genfromtxt('/home/justin/rverify.csv', ...)`
- [ ] iolib/tests/gen_dates.do `insheet using "/home/skipper/statsmodels/statsmodels-skipper/statsmodels/iolib/tests/stata_dates.csv"`
- [ ] sandbox/examples/example_sysreg.py `data3 = np.genfromtxt('/home/skipper/school/MetricsII/Greene 150  TableF5-1.txt', names=True)`
- [ ] sandbox/tests/GreeneEx15_1.s `dta <- read.table('/home/skipper/school/MetricsII/Greene\ TableF5-1.txt', header = TRUE)`
- [ ] stats/tests/results/anova.R `source('/home/skipper/statsmodels/statsmodels/tools/topy.R')`
- [ ] tsa/tests/arima.do `insheet using "/home/skipper/statsmodels/statsmodels-skipper/statsmodels/datasets/macrodata/macrodata.csv"` (and a couple others)
- [ ] tsa/tests/arima112.do `insheet using "/home/skipper/statsmodels/statsmodels-skipper/statsmodels/datasets/macrodata/macrodata.csv"` (and a couple others)
- [ ] tsa/tests/arima211.do `insheet using "/home/skipper/statsmodels/statsmodels-skipper/statsmodels/datasets/macrodata/macrodata.csv"` (and a couple others)
- [ ] tsa/tests/results/arima.R `dta <- read.csv("/home/skipper/statsmodels/statsmodels-skipper/statsmodels/datasets/macrodata/macrodata.csv")`
- [ ] tsa/tests/results/arima_forecast.inp `open /home/skipper/statsmodels/statsmodels/statsmodels/datasets/macrodata/macrodata.csv` (one other)
- [ ] tsa/tests/results/arma_forecast.inp `open /home/skipper/statsmodels/statsmodels/statsmodels/tsa/tests/results/y_arma_data.csv`
- [ ] tsa/tests/results/corrgram.do `insheet using "/home/skipper/statsmodels/statsmodels-skipper/statsmodels/datasets/macrodata/macrodata.csv", double clear` (one other)
- [ ] tsa/vector_ar/tests/svar.do `insheet using "/home/skipper/statsmodels/statsmodels-skipper/scikits/statsmodels/datasets/macrodata/macrodata.csv", double clear`
- [ ] tsa/vector_ar/tests/svartest.R `data <- read.csv("C:\\statsmodels\\statsmodels-bartbkr\\scikits\\statsmodels\\datasets\\macrodata\\macrodata.csv")` (and 2 more commented-out)
- [ ] tsa/vector_ar/tests/var.R `data <- read.csv('/home/wesm/code/statsmodels/scikits/statsmodels/datasets/macrodata/macrodata.csv')`
- [ ] stats/libqsturng/CH.r `setwd('D:\\USERS\\roger\\programming\\python\\development\\qsturng')`
- [ ] gam/tests/results/results_mpg_bs_poisson.r `source("M:\\josef_new\\eclipse_ws\\statsmodels\\statsmodels_py34_pr\\tools\\R2nparray\\R\\R2nparray.R")`  (@josef-pkt I've seen R2nparray used elsewhere with just `library(R2nparray)`; any chance that would work here?)
- [ ] tsa/statespace/tests/results/test_var.R `dta <- read.dta13('~/projects/statsmodels/statsmodels/tsa/tests/results/lutkepohl2.dta')` (one other)
- [ ] tsa/statespace/tests/results/test_exact_diffuse_filtering_multivariate.R `setwd('~/projects/statsmodels-0.9/statsmodels/')`
- [ ]  tsa/statespace/tests/results/test_exact_diffuse_filtering.R `setwd('~/projects/statsmodels-0.9/statsmodels/')`
- [ ] regression/tests/results/test_rls.R `macrodata <- read.csv('/Users/fulton/projects/statsmodels/statsmodels/datasets/macrodata/macrodata.csv')` (one other)
- [ ] regression/tests/results/test_rls.do `insheet using /Users/fulton/projects/statsmodels/statsmodels/datasets/macrodata/macrodata.csv, clear`
- [ ] compare_arma.py references r'C:\Josef\eclipsegworkspace\statsmodels-josef-experimental-gsoc\scikits\statsmodels\tsa\y_arma22.txt'; that file doesn't appear to exist anymore.
user: jbrockmendel
issue_5754: state: open
title: ENH:  p-value correction when we run several tests
body: p-value correction when we run several tests should take the total number of hypothesis tests into account, 
With bonferroni or sidak we would just need something like a ddof, i.e. user should be able to specify the total number of tests.
This does not work with the other method, because we need all p-values in the p-value correction.

example: combine the results of t_test_pairwise with some explicit t_tests as in #5741.

a possible solution:
create a function that takes a list of instances of ContrastResults or similar Results classes and adds the multiple testing p-value correction with a nice combined summary table.
user: josef-pkt
issue_5753: state: open
title: ENH: t_test_pairwise: allow list of names to specify levels for comparisons
body: Currently t_test_pairwise only uses the formula information.
Instead we can allow users to specify a list of exog/params names for which we should include the comparison.

e.g. #5741 is using pairwise comparison for the interaction effect between binary and categorical
`['Treatment[T.True]:Type[A]', 'Treatment[T.True]:Type[B]', 'Treatment[T.True]:Type[C]']`

The idea is similar to wald_test_terms where I allowed either formula or list of string information.
user: josef-pkt
issue_5752: state: open
title: Q: Ancient dependencies in examples/sandbox?
body: There are a number of places where `numdifftools`, `nipy`, or `nitime` are imported or otherwise mentioned.  Are these going to be reinstated as dependencies at some point?  Or can the imports at least be cleared at and replaced with statsmodels-internal versions?
user: jbrockmendel
issue_5741: state: open
title: Post-hoc t-test (and more general methodology question)
body: #### Is your feature request related to a problem? Please describe
A popular statistical output in psychology and biology (and I am sure many other fields which I am less familiar with) is a series of p-values for the levels of an interaction effect of one categorical and one binary variable. Generally the idea is that a treatment is applied to a wide variety of entity categories, and the researcher is interested whether statements can be made which differentiate the effect of the treatment across categories.

#### Describe the solution you'd like
It would be really cool to have this as a method of the model fit class.

#### Describe alternatives you've considered
I have hacked this together for a number of scientific articles ([1](https://bitbucket.org/TheChymera/mwhp/src/66e12095b793478c28d88f4e3f9ed2b33fd2836e/perfusionANOVA-proximal.py#lines-18),[1.1](https://www.nature.com/articles/s41598-019-42479-1),[2](https://bitbucket.org/TheChymera/rcmp/src/5b40151480fd4b420acfac6e2d7006a55560e53b/chemotaxis_4T1_model.py#lines-36) --- these are self-contained examples, of which at least the first should be reliably reproducible), and while the code is short, afaict efficient, and certainty not spaghetti, it is not very generalizable. If you can walk me through it, I would be happy to try and contribute it.

#### Additional context
I do have a broader question which might also impact how this feature could be implemented. The way I have handled post-hoc t-tests thus far, is:

1. Establish that the interaction term is significant in a model which includes both main effects and the interaction term (`value ~ binary*categorical`).
2. Set up a second model which expresses the entire binary treatment effect per-category (`value ~ categorical + binary:categorical`).
3. Set up an f-contrast of the `binary:categorical` levels, and ascertain that the significance is the same as obtained under `1.` --- I assume it always should be.
4. Set up t-tests (with `fit.t_test()`) for all combinations of the `binary:categorical` levels.
5. Control for multiple comparisons

This is, however, insufficient. While this tells me in how far I can make meaningful statements about whether `binary[T.True]:categorical[T.A]` is different than `binary[T.True]:categorical[T.B]`, it doesn't tell me whether a significant interaction effect might actually lead to the vanishing of the main binary treatment effect for that particular categorical variable. I could include additional t-tests for the individual terms, but then the f-contrast becomes an entirely different one than that of the interaction term in the initial model.

I'm unsure how to proceed here, should the t-tests between the interaction effects be pooled together with the t-tests for the significance of each individual interaction effect? Or should they be performed separately, along with separate multiple comparison corrections?
user: TheChymera
issue_5722: state: open
title: ENH: helper function is_nested, support for nested categorical
body: a helper function in the old formula code
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/sandbox/formula.py#L655

We don't have any explicit support for nested categoricals, 
patsy does something but I never looked closely and it looked like just data handling without handling implications for identifiability

We have one or several issues with an example where nested dummies caused problems. (which issue?)

related 
#5364 score test and similar with collinear columns
e.g. when we are binning to create bin or group dummies we need to check the rank or check for collinearity.






user: josef-pkt
issue_5714: state: open
title: FR: Requesting "weights" keyword argument in Logit and Probit regressions
body: #### Is your feature request related to a problem? Please describe
Weights in regression can be useful, for instance to correct for heterogeneity. The ability to add weights to each observation is a pretty standard feature in most regression packages, for instance the "weights" argument in [R's glm function](https://www.rdocumentation.org/packages/stats/versions/3.6.0/topics/glm)

#### Describe the solution you'd like
I'd like a "weights" keyword argument where a vector with the same dimension as the number of rows may be added.

#### Describe alternatives you've considered
There is a workaround as described [here](https://stackoverflow.com/questions/23339416/is-there-a-way-to-implement-sample-weights-in-python-logistic-regression) which is to duplicate rows to "trick" the procedure into using weights. Of course then the inferential stats will be way off.

#### Additional context
Add any other context about the feature request here.
user: baogorek
issue_5708: state: open
title: Issue Labels
body: Labels to consider implementing:
- [ ] CI
- [ ] EX (Example)
- [ ] SUMM
- [ ] Something for "consider implementing FooBar model"
- [ ] Segfaults (severe, un-recoverable bugs)
- [ ] Internal consistency (#3629, #3689, #4030, #4522, #4742...)
- [ ] Overflows/Numeric
- [ ] Failing Test (#4850)/Inconsistency on Windows (#4737)/...
- [ ] Comment/Question (#4956, #4961)
- [ ] Regression (might want a more specific name for this context; e.g. #5333)

A motivated maintainer can accomplish immediately:
- [ ] #1280: Remove branches from main repo

@bashtage collecting these here instead of pinging you a zillion times.  I've looked through about half the Issues so far, will post below if I identify more

- [ ] #5406: RLS
- [x] #5330: RLS
- [x] #3982: RLS -- not sure about this label
- [x] #3075: RLS
- [ ] #3076: RLS
- [x] #3431: RLS
- [ ] #4507: RLS
- [ ] #4573: RLS
- [x] #5290: RLS
- [x] #5371: BLD
- [x] #5035: BLD
- [x] #4407: BLD
- [x] #4160: BLD
- [x] #4150: BLD
- [x] #3596: BLD
- [x] #3373: BLD
- [x] #4657: BLD
- [ ] #5326: PERF
- [ ] #5199: PERF
- [ ] #4752: PERF
- [ ] #3082: PERF (closeable?)
- [ ] #3237: PERF
- [ ] #3399: PERF
- [ ] #3542: PERF
- [ ] #3609: PERF
- [ ] #4187: PERF
- [ ] #4239: PERF  --> already has tag, but I think duplicate of something else above
- [ ] #4546: PERF
- [ ] #4608: PERF
- [ ] #5344: PERF
- [ ] #89: PERF (likely closable)
- [ ] #296: PERF
- [ ] #2360: PERF
- [ ] #2447: PERF
- [x] #2585: PERF
- [x] #2914: PERF
- [ ] #1663: PERF
- [ ] #1848: PERF
- [ ] #2113: PERF
- [ ] #5688: BUG
- [ ] #4618: BUG
- [ ] #4544: BUG, pandas-compat
- [ ] #4541: BUG, pandas-compat
- [ ] #4332: BUG (not sure about label, CLN doesnt seem right)
- [ ] #5134: CLN
- [ ] #5126: CLN
- [ ] #5125: CLN
- [ ] #5111: CLN
- [ ] #5062: CLN
- [ ] #5023: CLN
- [ ] #5019: CLN
- [ ] #4994: CLN
- [ ] #4422: CLN
- [ ] #4419: CLN
- [ ] #4062: CLN
- [ ] #4110: CLN
- [ ] #5025: CLN
- [ ] #4220: CLN, REF, ENH
- [ ] #5225: EX
- [ ] #5037: EX
- [ ] #2326: EX
- [ ] #4935: TST
- [ ] #3078: TST, ENH
- [ ] #5082: TST
- [ ] #4966: TST
- [ ] #1376: TST
- [ ] #4549: pandas-wrapping
- [ ] #4541: pandas-wrapping
- [ ] #375: pandas-wrapping
- [ ] #814: pandas-wrapping
- [ ] #2270: pandas-wrapping
- [ ] #1342: pandas-categorical
- [ ] #4544: pandas-categorical
- [ ] #4894: pandas-compat
- [ ] #2585: pandas-compat
- [ ] #3306: DOC
- [ ] #4085: DOC
- [ ] #4199: DOC
- [ ] #3433: DEPR
- [ ] #3462: DEPR
- [ ] #5092: DEPR
- [ ] #4964: DEPR/DES
- [ ] #5019: MAINT
- [ ] #5472: py27-specific
- [ ] #583: NumericalPrecision
- [ ] #2262: NumericalPrecision
- [ ] #2484: Naming Conventions/Internal Consistency
- [ ] #2314: Naming Conventions/Internal Consistency
- [ ] #1664: Naming Conventions/Internal Consistency
- [ ] #2207: Naming Conventions/Internal Consistency
- [ ] #1162: Naming Conventions/Internal Consistency
- [ ] #1181: Naming Conventions/Internal Consistency
- [ ] #1335: Naming Conventions/Internal Consistency
- [ ] #487: Naming Conventions/Internal Consistency
- [ ] #73: Naming Conventions/Internal Consistency
- [ ] #157: Naming Conventions/Internal Consistency

Closeable? (see also #5155):
- [x] #3555
- [x] #3214
- [ ] #3780
- [x] #4466
- [x] #4204 Closable
- [x] #475: genfromdta
- [x] #476: genfromdta
- [ ] #469
- [x] #695: pylint
- [ ] #780
- [ ] #829
- [x] #833: closed by #5146
- [x] #903: closed by #5592
- [ ] #886
- [x] #1063
- [ ] #1237
- [ ] #2463
- [x] #2593: closed by #5042
- [x] #1893
- [ ] #1086
- [ ] #1324: str/bytes
- [x] #1290: unicode
- [ ] #1746: unicode
- [ ] #1892
- [ ] #2341
- [x] #1557 discussion of replacing rpy2, should be long-defunct
- [ ] #437 - close as duplicate of #5441, which has the FAQ label
- [ ] #1270 - close as duplicate of #5441, which has the FAQ label
user: jbrockmendel
issue_5689: state: open
title: DEPR: is sandbox.gam supplanted by gam?
body: Seems like the new one should strictly dominate the old one.
user: jbrockmendel
issue_5676: state: open
title: Review/BUG/REF GLM Gaussian scale in normalized_cov_params and cov_params
body: scale handling for GLM with estimated scale needs a check

this is a follow-up to #5316 so I can close that
user: josef-pkt
issue_5674: state: open
title: SUMM/ENH: generic model estimating equations, M-estimator, gmm
body: generic model based on estimating equation for example add a non-MLE variance estimate in addition to score/estimating equation of mean function

related 
#4729 score or CM test based on EE or GMM
#2860 GLM family with just mean and variance function specified

Norman Breslow: "Score tests in overdispersed GLM's"
I think this is just a paper that I printed, not sure about publication (most likely old GLIM conference issue)
Combines GLM score function for mean with a dispersion estimated using Pearson chi2 as moment condition.
Derivatives of scores/estimating equations are not symmetric because variance EE is not given by a derivative of the loglikelihood function.
Derivation of score test does not assume correctly specified variance, i.e. is HC.

We have the same pattern in (outlier) robust estimation when the M-estimator is specified as estimating equations where variance is not derived from the objective function.

This case is essentially just GMM with exactly identified moment conditions.
So we could use that special case of GMM as a generic framework to handle those models.
user: josef-pkt
issue_5632: state: open
title: CI: Azure follow-up/roundup
body: Follow-ups following #5617 

- [x] Trim unnecessary builds from Travis/Appveyor (e.g. #5627)
- [ ] Is the Publish job actually necessary on azure?  That part of the config was retained from the template file.
- [ ] do linting in a standalone azure job (like pandas)
- [ ] configure specific dependency versions on azure
- [x] the py27 linux build installation of pandas takes ~8 minutes instead of a few seconds; it apparently is building from source instead of using a wheel.  Avoid that.

Appending checkboxes from #5154 to combine these issues:

- [ ] GH tag for CI
- [x] Empower a junior maintainer with comparative advantage in CI.
- [x] set codecov settings to allow margin of error (comment from @rgommers) (closed by #5635)
- [ ]  use only one of codecov/coveralls?  both seem to give apparently random failures, might cut down on those.
- [ ] config for codacy to avoid false-positives
- [x] #5144 Use CircleCI to take some of the pressure off Travis
- [x] #5110 TST: randomize order of tests (closed by #5148)
- [ ] #5082 TST: Codecov False-Positives caused by Noise in mixed_linear_model tests?
- [x] #4779 MAINT: travis explicitly specify MKL or OpenBLAS (closed by #4823)
- [x] #2069 run unit tests on Mac OSX

user: jbrockmendel
issue_5564: state: open
title: design: API for `methods` for diagnostic tests
body: I'm looking at a RESET article and they use Davidson/MacKinnon auxiliary regression similar to the LM tests in specification tests for OLS.
I was recently focusing more on score_tests for variable addition.

We have at least 3 ways to compute a variable addition based score test:

- lm test based on auxiliary OLS/WLS regression (which I guess is just a conditional moment test in general)
- score_test: without auxiliary regression
- estimate full model with added artificial variables and use Wald test there.

and maybe
- LR type tests comparing the augmented with the original model

We need a `method` argument to choose among those.
In some cases, the results will be identical (up to computational precision) and not just asymptotically equivalent. But this will not be the case in general.
In nonlinear models like GLM, we will have the differences in at which parameters auxiliary statistics are evaluated and/or whether we use a one-step Fisher or Newton-Raphson approximation.

Separate choice and possible option is which `cov_type` is used for Wald or score tests.

For functions that implement general versions of these tests we need also an option for the `alternative` hypothesis or the `direction` in which we want to check for model misspecification.

for example the mess with Poisson excess or overdispersion tests where I have 6 to 8 versions some of which depend on which variance function is assumed as alternative, all are implemented as CMT. AFAIR.

user: josef-pkt
issue_5563: state: open
title: SARIMAX: incremental Kalman filter (question)
body: Could you please let me know if it is possible to use the current SARIMAX code to apply Kalman filter incrementally (to make it faster)?

Here is what I mean. Imagine, one fits an SARIMAX model (#1) using data with indices in `[iFitBegin, iFitEnd]`, then one wants to use such a model to make predictions for data with indices in `[iFitEnd+1, iDataEnd]`. To do that in the current code, it seems, one has to create a new SARIMAX model (#2) using data with indices in `[iFitBegin, iDataEnd]` and apply Kalman filter on the model #2 with parameters taken from model #1. After that one can request predictions from model #2. However, it is expensive computationally to do (for large models) if predictions are done on streaming data (**where new data arrives one data point at a time**) since every time one has to redo all Kalman filtering from `iFitBegin` to the index `iPred` (the one, at which a prediction is required). Mathematically, Kalman filter is applied incrementally, i.e. if Kalman filtering is done up to index `iPred`, then to do filtering up to index `iPred+1` one just has to take the state at `iPred` and only apply one step of Kalman filter using a single new data point at `iPred+1`, and so it would be great if one could do it in the code incrementally as well.

Could you please let me know if there is a way to do such incremental Kalman filtering (i.e. one state update at a time) in the current SARIMAX implementation?

Thank you for your help!!!
user: vss888
issue_5558: state: open
title: pull_LinearIVGMM.fitgmm
body: trying to improve the gmm.LinearIVGMM.fitgmm() code in order to compute standard errors, j-statistic of Sargan test and its related p-value. After being computed via the function fit.gmm, each of theses results is obtained through one of the functions defined below, at the end of the script. 
user: colindonfack
issue_5554: state: open
title: Request : OLS should have feature to drop np.inf or -np.inf
body: When running OLS with R-style formula, taking natural log on explanatory variables generates 
-np.inf values on some observations. 

Commonly it is dropped in the regression, and I was expecting that (missing = 'drop') would solve the problem. However, it seems statsmodels do not drop np.inf or -np.inf and causes error. (Result summary has all NaN values.)

I found it meticulous to generate dataframe with patsy and manually drop np.inf values before running regression. 

I hope statsmodels had feature to drop np.inf or -np.inf values so that I can run formula regression on the fly. 




user: matthewgson
issue_5549: state: open
title: Generating One Generalized Model for Multiple Datasets?
body: I have multiple sets of multivariate time series which each represent health data for different people. The datasets share the same set of features, but contain varying numbers of observations.  I'm assuming the data is stationary and the same underlying process governs each set of data. I want to make **ONE**  generalized model for all the time series. 

As of now, my approach has been modeled after this [answer ](https://stats.stackexchange.com/a/23123/187505) by Rob Hyndman. Essentially you concatenate all the multivariate time series together with a set of zeros between each dataset. 

I fit a VARMAX model on to this large data set from a previous year and then use the method described by @ChadFulton [here](https://github.com/statsmodels/statsmodels/issues/2788#issuecomment-175209448) to make predictions on each individual dataset for the current year. 

I see a couple of potential issues with the methodology I am currently using. The biggest is that even though the same process describes the interaction of each set of multivariate data. The data for each person is on different scales. I think that this would disrupt the fitting process of the model? Is there a way to fit a model on every dataset individually, essentially averaging the fits so that the variance in scale between datasets isn't an issue? 

One way I have thought about solving this is to perform a standard scale of each individual dataset before concatenating them together. I'm not super well-versed in this field, but I think this would cause the data sets with larger values to lose some of their descriptory power. Is there a way to call loop a fit over many data sets and essentially average the fits based on the number of observations for each fit? 
user: nicholasg97
issue_5547: state: open
title: FAQ/DOC: statsmodels.nonparametric.kde.kdensity: Epanechnikov bandwidth differs
body: **resolution**
Statsmodels uses Epanechnikov on standardized domain [-1, 1] which is `epa2` in Stata.
The bandwidth needs to be multiplied by `sqrt(5)` to get the bandwidth equivalent to the non-standard domain version used in Stata and R.

-----

Hello!
I have some problems with setting the bandwidth in my KDE plots in Python.

In R, I get the following plot using the Epanechnikov Kernel and a bandwidth of 0.1:
`plot(density(y,bw=0.1, kernel ="epanechnikov"), col="red")` 
![Rplot01](https://user-images.githubusercontent.com/48911472/55010088-e3d3f480-4fe3-11e9-8d84-ebd54c5b8bc9.png)

However, if I use the following code in Python with the same kernel and bandwidth:
`from statsmodels.nonparametric.kde import kdensity`
`density = kdensity(y, kernel='epa', bw=0.1)`
`plt.plot(density[1],density[0], color = 'red')`
I get a very different result:
![statsmodels_kdeplot01](https://user-images.githubusercontent.com/48911472/55010186-1382fc80-4fe4-11e9-999f-263ea9bf62c9.png)

In the documentation of the function, it says:

> bw : str, float:  “scott” - 1.059 * A * nobs ** (-1/5.), where A is min(std(X),IQR/1.34) “silverman” - .9 * A * nobs ** (-1/5.), where A is min(std(X),IQR/1.34) If a float is given, it is the bandwidth.


So, it should interpret the parameter of the bandwidth directly. However, as you can see, it doesn't.

Moreover, I get the same output in Python if I use the seaborn.kdeplot function:
`import seaborn as sns`
`sns.kdeplot(y,kernel = 'epa', bw = 0.1)`
![seaborn_kdeplot01](https://user-images.githubusercontent.com/48911472/55010511-986e1600-4fe4-11e9-9680-beb549e9fa59.png)


Any idea what is going on here? Do I have to add some parameters?

user: myosotise
issue_5540: state: open
title: ENH: Testing instruments Weakness in 2SLS regression
body: Hello, I am a new user of statsmodels, currently in first year of Econ PhD at university of Montreal, interested in programming.

Added the _.Effective_Fstat()_ method to _IV2SLS.fit()_ which calculate the effective F statistic
 suggested by Olea and Pflueger (2013) to test for weak instruments in 2SLS regression with 
one regressor instrumented. 

TODO: test against Stata's weakivtest and provide unit test
	add critical values and pvlues calculations
user: RegisKouassi
issue_5531: state: open
title: FAQ: statsmodels equivalent to R's "estimable" from gmodels
body: Thank you for all your work on this,

I looking around trying to find a way to estimate fixed effects and their respective standard errors for all categorical levels of interest. 

For example, if I have two groups (control and treatment) and I using dummy coding to treat control as the reference, The parameter estimate (fixed effect) for control will be the intercept, and the parameter estimate for the  treatment will be equal to the addition of the following parameter estimates `intercept + C(group, Treatment("control"))[T.treatment]`.

How do I get the simple fixed effects of Treatment and Control with appropriate standard errors?

The apparent R function that can do this is called [estimable](https://www.rdocumentation.org/packages/gmodels/versions/2.18.1/topics/estimable) from [gmodels](https://www.rdocumentation.org/packages/gmodels/versions/2.18.1), but I just can't seem to find out how to do the same thing in statsmodels.
user: jdkent
issue_5530: state: open
title: Runtime warnings in cohens_kappa for degenerate input data
body: **System:**
statsmodels 0.9.0, numpy 1.15.4 and python 3.6.7 on MacOS.

**Problem description**
I receive warning messages for the following code:

```
from statsmodels.stats.inter_rater import cohens_kappa
contingency =  [[60.0, 29.0], [0.0, 0.0]]
cohens_kappa(contingency)
```

The warning messages read like this:
```
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/statsmodels/stats/inter_rater.py:502: RuntimeWarning: invalid value encountered in sqrt
  self['std_kappa'] = np.sqrt(self['var_kappa'])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/statsmodels/stats/inter_rater.py:505: RuntimeWarning: invalid value encountered in double_scalars
  self['z_value'] = self['kappa'] / self['std_kappa0']

```

**Expected behavior:**

- In case there are conditions on the input data (for example no line in the contingency table with all zeros), I would expect a more user-friendly warning. For example "Degenerate contingency table encountered, the resulting kappa values should not be trusted.")
- Better: avoid the above warnings by catching the special case and set the dependent coefficients accordingly.


**Details**
Obviously, the test-case is degenerate. But, a valid return value should still be possible, no?

If you look at the source code ([inter_rater.py](https://github.com/statsmodels/statsmodels/blob/7a33ef0ba2900f8e1d27fbd052c7a28d5a626342/statsmodels/stats/inter_rater.py#L502)), the values for the sample are as follows:

```
        ...
        self['kappa'] = 0
        self['var_kappa'] = -1.1749090272489766e-17
        self['var_kappa0'] = 0
        ...
```

The problematic lines are:
```
        self['std_kappa'] = np.sqrt(self['var_kappa'])      # First warning: sqrt of negative number
        self['std_kappa0'] = np.sqrt(self['var_kappa0'])
        self['z_value'] = self['kappa'] / self['std_kappa0'] # Second warning: Division by zero
```


user: normanius
issue_5528: state: open
title: C statistic
body: Hi,
I was using statsmodels and found that there was no operational function to compute the J-difference between two nested GMM models in the linear case. There is a compare_j() function I saw in the module but it is not guaranteed that the difference is always positive.
user: CheikhNokho
issue_5526: state: open
title: DOCS: Dynamic factor notebook specifications
body: @mboldin noted that in the estimation output of the dynamic factor model in this notebook, the coefficients of the error lag polynomial for industrial production are non-stationary.

1. I had thought that `transform_params` would have eliminated this possibility, so need to check into this.
2. @mboldin suggested that an AR(2) probably overparameterizes these, which is a good point.

```
====================================================================================================
                                       coef    std err          z      P>|z|      [0.025      0.975]
----------------------------------------------------------------------------------------------------
...
L1.e(std_indprod).e(std_indprod)    -1.1537      0.013    -85.932      0.000      -1.180      -1.127
L2.e(std_indprod).e(std_indprod)    -0.9867      0.014    -72.925      0.000      -1.013      -0.960
```
user: ChadFulton
issue_5525: state: open
title: [QUESTION] Fixed effect logit (Conditional Logit)
body: Hey there!
Thanks for this awesome package. I have a question regarding **Fixed effect models, especially Logit** in `statsmodels`.
Is there any implementation of this? I looked at the documentation and could not find any mentions.
I have a discrete choice modeling problem for 100 individuals. The dependent variable is `choice` and independent variables are `price, duration, comfort`
I want to run a Logit model with fixed effects on individuals. If it is not possible out of the box in `statsmodels` maybe, there is some workaround?
Is making `individual` a categorical variable may solve this?

Thanks!
user: emilmirzayev
issue_5514: state: open
title: Model_Selection
body: I am currently working on forecasting for different number of categories and using ARIMA. Each category having its own time-series data. Is there any feature in statsmodels which automate the process of selecting the best model for each category. If yes kindly guide me. If not i would like to open a pull request for this feature. and work on it.
user: hassaanseeker
issue_5506: state: open
title: WIP ENH: Add stats.control_charts
body: Taking over this PR (https://github.com/statsmodels/statsmodels/issues/4191) to build control charts: https://github.com/statsmodels/statsmodels/issues/4191




user: shangyian
issue_5505: state: open
title: SUMM: control charts, heteroscedasticity, heterogeneity, predicted distribution/moments
body: (some vague random thoughts)

standard control charts use a estimate of the standard deviation to specify the control limits (3 sigma)

traditional Xbar charts seem to use withing group variation (e.g. based on within group range statistic)
The following article argues in favor of using a between group range statistic to capture between group variation
https://www.smartersolutions.com/orl/30000-ft-level-performance-metric-reporting_6sigmaforum.pdf

minitab handles excess dispersion in Poisson and Binomial charts by estimating the excess dispersion based on the between group variation.

more general: we want to know the spread or quantiles for the distribution of a new observation, i.e. we want to predict a new observation and some of it's quantiles.
(a bit similar to GARCH, or predicting several moments of the prediction distribution)

- unobserved heterogeneity or unconditional distribution: If we don't observe the variables driving heteroscedasticity/heterogeneity, then we observe only the unconditional distribution which is a mixture distribution. The appropriate standard deviation is the total variation, or between variation if we use groups. The within variance will not correctly estimate this.
- conditional distribution or observed heterogeneity: If we can observe the variables affecting the variance or distributional properties, then we can improve on the unconditional prediction using a model for other moments than mean. In this case we use group specific standard deviation and group specific prediction intervals.

We don't have anything GARCH like, that would predict the variance or excess dispersion.

Similar to the discussion on excess dispersion in GLM, we could have different sources that results in overall excess dispersion or heteroscedasticity, e.g. random effects or withing group correlation.

Our current QMLE estimates the mean function being robust to misspecified higher moments (heteroscedasticity or correlation).

To get prediction intervals for new observations, we also need the corresponding models for variance and higher moments, e.g. QMLE for variance that is robust to higher moment misspecification.
(maybe GLM-Gamma with robust cov_type)

We essentially need to apply what we have for mean to variance. 
e.g. do we want population averaged effect, and if yes for which population, or do we want conditional effects under stronger parametric assumptions.







user: josef-pkt
issue_5503: state: open
title: WIP ENH: add stats.control_charts
body: adding by getting started with control charts script from #4191 to stats.control_charts

example converted to notebook is at https://gist.github.com/josef-pkt/f97874ca89237e78a505392803f6b40a


user: josef-pkt
issue_5502: state: open
title: WIP ENH: Mixed mixin2
body: new PR. replaces #3030

MixedMixin for random effects in MLE models, GLM and discrete

This version still uses a local uncommitted copy of the initial Halton sequence. That needs to be replaced by the halton sequence in master.

no unit tests, but examples in a gist notebook





user: josef-pkt
issue_5500: state: open
title: FAQ-D/SUMM: model and results methods, required and usage
body: What methods are required?
What are the methods used for?

We need additional methods in models and results classes that are required by some extra functionality. The extra generic functionality becomes available if the additional methods are available.

This issue is a companion for #1970 which has required attributes.

examples for usage: 

- margins requires derivatives
- sandwiches require score_obs
- score_test requires score_factor, hessian_factor for shortcut for variable addition test
- MixedMixin requires loglike_obs 
- outlier-influence: requires some derivatives, hat_matrix_diag, ...
- two-part models like ZeroInflated count models use loglikeobs
- ...

GenericLikelihoodModel has loglikeobs used to compute numerical derivative for score_obs. I did not see any other usage in a quick look.

loglike is not directly needed for estimating equation optimizers, e.g. ls in OLS/WLS, irls or our newton. (e.g. tweedie does not yet have loglike)




user: josef-pkt
issue_5499: state: open
title: ENH: tools gauss-hermite quadrature
body: mainly to park an old script for how to do gauss-hermite integration with scipy.
code is most likely outdated because scipy deprecated some of the orthogonal polynomial function.

related to numerical integration:
- halton and similar quasi-random sequences are in master, for DOE
- application to one-way mixed models like Poisson is somewhere #2392 and not committed MixedMixin

Stata uses adaptive quadrature for mixed models, which I haven't figured out (and I'm not sure we can do it in a computational efficient/vectorized way).
#2516  PR for GSOC project for MixedGLM, uses Laplace IIRC
Bayesian Mixed GLM is in master


user: josef-pkt
issue_5498: state: open
title: ENH: gam like penalized (orthogonal, fractional) polynomials
body: Now that GAM is in master, we could use polynomial, especially, orthogonal polynomials with a similar structure.
It would also be a good case for extending and generalizing the supporting code, e.g. exog_extra or multi-column terms in predict, partial predict, diagnostic plots

If we use numpy polynomials (instead of scipy as in my early cases), then we should also be able to get second derivatives to penalized IMSE (integrated mean squared error), similar to GAM B-splines.
In my previous penalized polynomial cases, the penalization matrix (prior cov) was "made up", i.e. somewhat arbitrary increase of penality with order/power of polynomial term.

The cubic splines have code for transforming the data to a standard range, e.g. [-1, 1] interval.


user: josef-pkt
issue_5492: state: open
title: FAQ Having trouble getting Exogenous names in model summaries
body: Hi. I am using using statsmodels installed with Anaconda with following versions:
```

>>> statsmodels.__version__
'0.9.0'
>>> exit()

(base) C:\Users\emirzayev>conda --version
conda 4.6.2

```
Now when I fit a model, in summary table, I do not see the names of the variables. Only x1, x2, xN. Is there a way to have the variable names also in the summary? Or this change is permanent.
```
                           Logit Regression Results                           
==============================================================================
Dep. Variable:                 choice   No. Observations:                 1766
Model:                          Logit   Df Residuals:                     1757
Method:                           MLE   Df Model:                            8
Date:                Tue, 12 Feb 2019   Pseudo R-squ.:                  -2.762
Time:                        14:13:09   Log-Likelihood:                -1214.0
converged:                       True   LL-Null:                       -322.66
                                        LLR p-value:                     1.000
==============================================================================
                 coef    std err          z      P>|z|      [0.025      0.975]
------------------------------------------------------------------------------
x1             0.0751      0.070      1.073      0.283      -0.062       0.212
x2            -0.0944      0.109     -0.866      0.386      -0.308       0.119
x3            -0.1273      0.103     -1.233      0.218      -0.330       0.075
x4             0.0641      0.050      1.273      0.203      -0.035       0.163
x5             0.0682      0.054      1.266      0.205      -0.037       0.174
x6             0.1070      0.122      0.880      0.379      -0.131       0.345
x7            -0.0595      0.076     -0.778      0.437      -0.209       0.090
x8             0.1410      0.143      0.987      0.324      -0.139       0.421
x9             0.1769      0.093      1.896      0.058      -0.006       0.360
==============================================================================
```

Thanks beforehand
user: emilmirzayev
issue_5490: state: open
title: ENH: Add penalization summary, closes #5461
body: Added the `add_penal_table` method to the `summary` class which collects the penalization information and returns it as a SimpleTable instance.
user: stevenbw
issue_5488: state: open
title: BUG: get_margeff on Multinomial logit, exception if count=True
body: Good Evening.
I'm a student and I'm using Python on Google Colab for my thesis.
I have a large dataset of 1876261 observations and i've computed the multinomial logit on it.
Now, I would like to compute the Average Partial Effects but i'm experiencing some problems.

Here's my code:
#Multinomial Logit
import pylab as pl
import statsmodels.api as sm
model_eqn = "grade ~ int_rate + log_loan_amnt + log_annual_inc + dti + home_ownership +log_emp_length_num + chargeoff_within_12_mths + application_type"
model = sm.MNLogit.from_formula(model_eqn, random_subset).fit()
st.chisqprob = lambda chisq, df: st.chi2.sf(chisq, df)
print(model1.summary())
#APE
model_margeff=model.get_margeff(at='overall', method='dydx', count=True)
model_margeff.summary()

1 problem) when i run the code on the entire model, the session crashes.
2 problem) I have computed the AME on a subset of the original dataset. So, i've selected 10000 observations from it.

#subset dataframe
random_subset = myLCData1.sample(n=10000)
print(random_subset.head())

Then i've computed again the MLN Logit with the "random_subset" and re-compute the AME.
#multinomial with subset
import pylab as pl
import statsmodels.api as sm
model_eqn = "grade ~ int_rate + log_loan_amnt + log_annual_inc + dti + home_ownership +log_emp_length_num + chargeoff_within_12_mths + application_type"
model1 = sm.MNLogit.from_formula(model_eqn, random_subset).fit()
st.chisqprob = lambda chisq, df: st.chi2.sf(chisq, df)
print(model1.summary())
#APE
model_margeff=model1.get_margeff(at='overall', method='dydx', count=True)
model_margeff.summary()

But it gives me this ERROR:

ValueErrorTraceback (most recent call last)
<ipython-input-49-234df2f32820> in <module>()
----> 1 model_margeff=model1.get_margeff(at='overall', method='dydx', count=True)
      2 model_margeff.summary()

/usr/local/lib/python2.7/dist-packages/statsmodels/discrete/discrete_model.pyc in get_margeff(self, at, method, atexog, dummy, count)
   2502         """
   2503         from statsmodels.discrete.discrete_margins import DiscreteMargins
-> 2504         return DiscreteMargins(self, (at, method, atexog, dummy, count))
   2505 
   2506     def summary(self, yname=None, xname=None, title=None, alpha=.05,

/usr/local/lib/python2.7/dist-packages/statsmodels/discrete/discrete_margins.pyc in __init__(self, results, args, kwargs)
    423         self._cache = resettable_cache()
    424         self.results = results
--> 425         self.get_margeff(*args, **kwargs)
    426 
    427     def _reset(self):

/usr/local/lib/python2.7/dist-packages/statsmodels/discrete/discrete_margins.pyc in get_margeff(self, at, method, atexog, dummy, count)
    704                                                 model._derivative_exog,
    705                                                 dummy_idx, count_idx,
--> 706                                                 method, J)
    707 
    708             # reshape for multi-equation

/usr/local/lib/python2.7/dist-packages/statsmodels/discrete/discrete_margins.pyc in margeff_cov_with_se(model, params, exog, cov_params, at, derivative, dummy_ind, count_ind, method, J)
    342     cov_me = margeff_cov_params(model, params, exog, cov_params, at,
    343                                               derivative, dummy_ind,
--> 344                                               count_ind, method, J)
    345     return cov_me, np.sqrt(np.diag(cov_me))
    346 

/usr/local/lib/python2.7/dist-packages/statsmodels/discrete/discrete_margins.pyc in margeff_cov_params(model, params, exog, cov_params, at, derivative, dummy_ind, count_ind, method, J)
    325         if count_ind is not None:
    326             jacobian_mat = _margeff_cov_params_count(model, jacobian_mat,
--> 327                                 params, exog, count_ind, method, J)
    328     else:
    329         jacobian_mat = derivative

/usr/local/lib/python2.7/dist-packages/statsmodels/discrete/discrete_margins.pyc in _margeff_cov_params_count(model, cov_margins, params, exog, count_ind, method, J)
    242         exog0 = exog.copy()
    243         exog0[:,i] -= 1
--> 244         dfdb0 = model._derivative_predict(params, exog0, method)
    245         exog0[:,i] += 2
    246         dfdb1 = model._derivative_predict(params, exog0, method)

/usr/local/lib/python2.7/dist-packages/statsmodels/discrete/discrete_model.pyc in _derivative_predict(self, params, exog, transform)
    635         X = np.tile(exog, J-1)
    636         # this is the derivative wrt the base level
--> 637         F0 = -repeat_eXB * X / sum_eXB ** 2
    638         # this is the derivative wrt the other levels when
    639         # dF_j / dParams_j (ie., own equation)

ValueError: operands could not be broadcast together with shapes (10000,42) (10000,54) 

Could someone help me?

Thank you for you time.
user: benn510
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_5487: state: open
title: manova - getting canonical variates (i.e. coefficients of discriminant function)
body: From the E(rror) and H(ypotheses) matrices calculated [here](https://github.com/statsmodels/statsmodels/blob/master/statsmodels/multivariate/multivariate_ols.py#L320), it's possible to get the discriminant function coefficients with `W_B = inv(E).dot(H); eig_val, eig_vec = eig(W_B)`. `eig_vec` contains the coefficients of the variables that maximally separate conditions.

Could we add to the result dict one of (E, H), W_B, or (eig_val, eig_vec)?

It's possible that I'm missing another way to calculate these coefficients from the results that are already available but I don't see it.
user: cboulay
issue_5486: state: open
title: DOC/ENH summary_col missing in docs, reuse for descriptive stats
body: A stackoverflow question ask for a similar table as summary_col but for descriptive statistics (mean, std)
https://stackoverflow.com/questions/54466196/descriptive-statistics-in-python-with-std-in-parentheses

search in the online sphinx docs for `summary_col` comes up empty (except for release notes)

code is at https://github.com/statsmodels/statsmodels/blob/master/statsmodels/iolib/summary2.py#L437
user: josef-pkt
issue_5482: state: open
title: FAQ-D: debugging warnings with pytest
body: pytest does not show which unit test issues a warning

To find the warning and debug it, we can convert the warning into an error

e.g.
`>pytest -v --pdb -W error::RuntimeWarning M:...\statsmodels\gam\tests\test_gam.py`

example usage https://github.com/statsmodels/statsmodels/pull/5481#issuecomment-458649621
user: josef-pkt
issue_5479: state: open
title: suggestion for CompareMeans when passing DataFrame
body: I was exposed to using .from_data() classmethod to return a `CompareMeans` object in https://github.com/statsmodels/statsmodels/issues/4774 .
 
The example @adrienpacifico provided: 
> ```python
> import pandas as pd
> import statsmodels.stats
> 
> df1 = pd.DataFrame({"Age": [21,23,54], 'Income': [200, 150,600]})
> df2 = pd.DataFrame({"Age": [30,26,34], 'Income': [400, 250,100]})
> 
> print(statsmodels.stats.weightstats.CompareMeans.from_data(df1,df2).summary())
> ```
> 
> #### Output
> ```
>                           Test for equality of means                          
> ==============================================================================
>                  coef    std err          t      P>|t|      [0.025      0.975]
> ------------------------------------------------------------------------------
> subset #1      2.6667     10.929      0.244      0.819     -27.677      33.011
> subset #2     66.6667    166.667      0.400      0.710    -396.074     529.408
> ==============================================================================
> ```

Instead of the column name issue mentioned by @josef-pkt. I would strongly suggest to rethink of the input `dtype` of this classmethod. By all means, unlike `numpy` array, passing two `DataFrame` could be very confusing due to the existence of column name, which gives meanings to the underlying data.

I could literately change the name of the columns in both dataframe to whatever I want and then get the exact same result.

```python
import pandas as pd
import statsmodels.stats.api as sms

df1 = pd.DataFrame({'a': [21,23,54], 'b': [200, 150,600]})
df2 = pd.DataFrame({'c': [30,26,34], 'd': [400, 250,100]})
print(sms.CompareMeans.from_data(df1, df2).summary())

# Test for equality of means                          
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
subset #1      2.6667     10.929      0.244      0.819     -27.677      33.011
subset #2     66.6667    166.667      0.400      0.710    -396.074     529.408
==============================================================================
```
The output will vary if I change the postion of the columns in the dataframe like the following:
```python
df1 = pd.DataFrame({'a': [21,23,54], 'b': [200, 150,600]})
df2 = pd.DataFrame({'d': [400, 250,100], 'c': [30,26,34]})
print(sms.CompareMeans.from_data(df1, df2).summary())

# Test for equality of means                          
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
subset #1   -217.3333     87.259     -2.491      0.067    -459.603      24.936
subset #2    286.6667    142.419      2.013      0.114    -108.751     682.085
==============================================================================
```
However, before and after the change,  there is no difference of the data stored in `df2`. I would say both of the outputs are meaningless since we don't know what the connection between those two dataframe is. In the real word, we mainly dealing with dataframe in a tidy format. I think `StatsModels` as a statistical analysis package shouldn't worry about this formatting issue like column names but require its users to do any necessary data tidying before using it. The above two dataframe should be merged as `df` like the following with a new column to idenfy group informaion.

![image](https://user-images.githubusercontent.com/33796896/51820002-679e8880-22a2-11e9-9311-5ca8be96f276.png)

And then use `.loc` to index data and pass them to `sms.CompareMeans()`
```python
group1 = df.loc[df.Group == 'group1', 'Age']
group2 = df.loc[df.Group == 'group2', 'Age']
cm = sms.CompareMeans.from_data(group1, group2)
print(cm.summary())

# Test for equality of means                          
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
subset #1      2.6667     10.929      0.244      0.819     -27.677      33.011
==============================================================================
```

I wish there could be a function that could automatically return the summary for both `Age` and ` Income` columns. I can simply pass the name of the dataframe, list of columns storing data, and the column with group label. 

user: stevenlis
issue_5478: state: open
title: DescrStatsW attributes listed as methods
body: The [doc](https://www.statsmodels.org/dev/generated/statsmodels.stats.weightstats.DescrStatsW.html) for `DescrStatsW` listed some attributes of the class such as `.mean`, and `.std` as methods `.mean()` and `.std()`, which will return `TypeError: 'numpy.float64' object is not callable`. 

I would say it would be less confusing if they can be separately listed. 
user: stevenlis
issue_5477: state: open
title: no support for incorporating transfer functions
body: any plans to do this
user: dthboyd
issue_5474: state: open
title: ENH: regression plots for heteroscedasticity
body: 
example
`plt.plot(res_bsf.fittedvalues, res_bsf.resid_response**2, '.')`

I tried a bit in the last part in this gist (using GLMGam with Gaussian)
https://gist.github.com/josef-pkt/2f78444b9e71294e0cd33f48fdc89a5e

looks like there are "variance outliers"
I switched to using RLM for a variance regression
```
from statsmodels.regression.linear_model import OLS
from statsmodels.robust.robust_linear_model import RLM
from numpy.polynomial.polynomial import polyvander
res_rlmhet = RLM(res_bsf.resid_response**2, polyvander(res_bsf.fittedvalues, 1)).fit()
print(res_rlmhet.summary())
```

observation: linear term is significant, but when I add the quadratic term, then all coefficients are insignificant.

one problem is that fittedvalues are not well scaled for higher order polynomials

user: josef-pkt
issue_5473: state: open
title: ENH/REF plot_partial_residuals versus plot_partial in GLM/GAM
body: Based on quick skimming, GLMResults.plot_partial_residual is just a CPR (component plus residual) plot for a single column of exog.
However it uses the weights, weighted residuals in the plot. (those might be one of the available residuals)

GLMGamResults.plot_partial plot partial prediction and optional cpr scatter points but only for the linear prediction, using resid_working.
It also adds confidence intervals for the partial linear prediction.

regression plot for OLS also have a CPR plot for a single exog column.

what can we improve, enhance or make more consistent?

- prediction intervals ? #2621
- dependence on multicolumn exog (e.g. splines, polynomial terms)
- what happens with categorical exog?
- which residuals
- how generic can we make this


(I found plot_partial_residual by tab completion because it was next to the new GAM plot_partial)


user: josef-pkt
issue_5472: state: open
title: MAINT: python 2.7 specific subclasses of ResultsWrappers
body: see https://github.com/statsmodels/statsmodels/pull/5370#issuecomment-455697166

It looks like using a subclass that doesn't do anything extra is not needed anymore in python 3.
We could change this after dropping support for python 2.7

discrete has many cases like the following
```
class CountResultsWrapper(lm.RegressionResultsWrapper):
    pass
wrap.populate_wrapper(CountResultsWrapper, CountResults)
```
or in count_model
```
class ZeroInflatedPoissonResultsWrapper(lm.RegressionResultsWrapper):
    pass
wrap.populate_wrapper(ZeroInflatedPoissonResultsWrapper,
                      ZeroInflatedPoissonResults)
```
user: josef-pkt
issue_5469: state: open
title: ENH/Design: design_info for parts of exog
body: This is a specific part of #5342 https://github.com/statsmodels/statsmodels/issues/5342#issuecomment-454937381

I have difficulty to figure out how design_info, param_names and similar are handled in base model.
Essentially, I don't see where and by what code things get attached to a model or model.data instance.

My example is GLMGAM, where I tried to get formula and data handling exog_linear separate from the spline components in exog. But I don't understand why we have have attributes that I thought shouldn't be there  and why some attributes like exog_names are not what I think they should be.

example
`from_formula` adds kwargs in the call to the class `__init__`. Any extra kwargs are attached to the model, except `'design_info', 'formula'` are explicitly excluded in `base.model.Model._handle_data`.

It would be better to check this on a simple example because GLMGAM has too many other moving parts that affect the behavior.

The basic idea

- We use our standard data and formula handling for the core exog, but not for the `exog_extra`.
- transformation in predict, and and maybe constraint creation in wald tests will need extra code to split between the core exog and exog_extra.

I thought this might also be useful for the case with a large number of nuisance dummy variables, but I'm not sure whether it will help for that case. The "absorb" models have possibly a different exog and inference structure.
In the case of this issue, the split between exog_core (?) and exog_extra is an interface and data handling question, the underlying numpy arrays in the model are not affected by this split, i.e. we have full `exog`, `params` and `cov_params`.

Aside: exog in tsa might be similar if/when we support formulas, i.e. we commonly split between user provided exog and internally added trend and other deterministic terms, even though the composition of the full `exog` is independent of its composition and user interface  in the main computation/estimation.


user: josef-pkt
issue_5466: state: open
title: DESIGN: State space: default standard errors: OPG vs OIM
body: **TL;DR**: we currently use outer-product of gradients (OPG) as the default information matrix approximation, as does Stata and Eviews.  Some limited experiments on AR, MA models (see below) suggest that this may not be the best option. Does anyone know of results / references / reasons why we should use OPG over one of the other methods? Is it just the guaranteed nonsingularity of the resulting matrix?

----------

Currently the default in state space models is to approximate the information matrix via the outer product of gradients (OPG).  This is e.g. also the default in Stata and Eviews for state space models and for arima models.

However, I'm not aware a reason / reference that makes this the preferred option for approximating the information matrix, other than the fact that the OPG itself is guaranteed to be positive (semi)definite?

Both Stata and Eviews have the observed information matrix (OIM) computation available, although it is not the default. I'm wondering if anyone knows why this is, given that it appears (see below) that the OIM may do a better job, and even that the numerical approximation to the Hessian may do a better job.

Since Brockwell and Davis (2002) give some analytic formulas for the covariance of AR(p) and MA(q) 
models for p=1,2 and q=1,2, I thought it might be interesting to compare the three methods for approximating the information matrix, available in state space models:

1. Outer product of gradients (computed using `score_obs`)
2. Numerical approximation of the Hessian (via complex step differentiation)
3. Observed information matrix (computation described in Harvey, 1989)

The results of my experiment are at: https://gist.github.com/ChadFulton/986f7bfbf90e162e8f3526880a833891

It appears (in the small number of examples that I looked at) that the OIM gives the closest answer to the theoretical covariance matrix, and that the numerical approximation is often as good or only slightly worse. In these examples, the OPG is always worst.
user: ChadFulton
issue_5461: state: open
title: ENH: summary specific information for penalized
body: `PenalizedMixin` subclasses use the inherited `summary` which does not contain any information about the penalization besides the model name.

When reading the summary in a notebook it would be useful to see which penalization type or class has been used and what the penalization weights are.
Maybe extra option: allow dropping some parameters from the summary table, e.g. those corresponding to large spline/polynomial basis functions or those thresholded to zero. (There is a github issue for dropping extra categorical fixed effects from the summary, IIRC.)

We should be able to add some extra information to the inherited method, i.e. without copying the super summary method, which would be model dependent.

possibilities

- use extra attributes in top table, e.g. super summary checks for a dict, `_summary_extra` and appends it
- add a 3rd (or 4th) table at bottom
- add to extra string (which contains warnings in OLS/WLS)
- write a separate summary_penalization method that provides an extra table or text with more info about the penalization. (e.g. we should/could also add informative `__str__` and `__repr__` for penalty classes which could be reused).

The problem might be similar with modifier `fit_xxx` methods, e.g. `fit_constrained`.


user: josef-pkt
issue_5459: state: open
title: SARIMAX python np.linalg.linalg.LinAlgError: LU decomposition error
body: Hello,
I have a problem with time series analysis. I have a dataset with 5 features. Following is the subset of my input dataset:

date,price,year,day,totaltx
1/1/2016 0:00,434.46,2016,1,126762
1/2/2016 0:00,433.59,2016,2,147449
1/3/2016 0:00,430.36,2016,3,148661
1/4/2016 0:00,433.49,2016,4,185279
1/5/2016 0:00,432.25,2016,5,178723
1/6/2016 0:00,429.46,2016,6,184207

My endogenous data is price column and exogenous data is totaltx price.
**This is the code I am running and getting an error:**

```
import statsmodels.api as sm
import pandas as pd
import numpy as np
from numpy.linalg import LinAlgError

def arima(filteredData, coinOutput, window, horizon, trainLength):
    start_index = 0
    end_index = 0
    inputNumber = filteredData.shape[0]
    predictions = np.array([], dtype=np.float32)
    prices = np.array([], dtype=np.float32)
    # sliding on time series data with 1 day step
    while ((end_index) < inputNumber - 1):
        end_index = start_index + trainLength
        trainFeatures = filteredData[start_index:end_index]["totaltx"]
        trainOutput = coinOutput[start_index:end_index]["price"]

        arima = sm.tsa.statespace.SARIMAX(endog=trainOutput.values, exog=trainFeatures.values, order=(window, 0, 0))
        arima_fit = arima.fit(disp=0)
        testdata=filteredData[end_index:end_index+1]["totaltx"]
        total_sample = end_index-start_index
        predicted = arima_fit.predict(start=total_sample, end=total_sample, exog=np.array(testdata.values).reshape(-1,1))
        price = coinOutput[end_index:end_index + 1]["price"].values

        predictions = np.append(predictions, predicted)
        prices = np.append(prices, price)

        start_index = start_index + 1
    return predictions, prices

def processCoins(bitcoinPrice, window, horizon):
    output = bitcoinPrice[horizon:][["date", "day", "year", "price"]]
    return output

trainLength=100;
for window in [3,5]:
    for horizon in [1,2,5,7,10]:
        bitcoinPrice = pd.read_csv("..\\prices.csv", sep=",")
        coinOutput = processCoins(bitcoinPrice, window, horizon)
        predictions, prices = arima(bitcoinPrice, coinOutput, window, horizon, trainLength)

```
In this code, I am using rolling window regression technique. I am training arima for start_index:end_index and predicting the test data with end_index:end_index+1

**This the error that is thrown from my code:**
```
Traceback (most recent call last):
  File "C:/PycharmProjects/coinLogPrediction/src/arima.py", line 115, in <module>
    predictions, prices = arima(filteredBitcoinPrice, coinOutput, window, horizon, trainLength, outputFile)
  File "C:/PycharmProjects/coinLogPrediction/src/arima.py", line 64, in arima
    arima_fit = arima.fit(disp=0)
  File "C:\AppData\Local\Continuum\Anaconda3\lib\site-packages\statsmodels\tsa\statespace\mlemodel.py", line 469, in fit
    skip_hessian=True, **kwargs)
  File "C:\AppData\Local\Continuum\Anaconda3\lib\site-packages\statsmodels\base\model.py", line 466, in fit
    full_output=full_output)
  File "C:\AppData\Local\Continuum\Anaconda3\lib\site-packages\statsmodels\base\optimizer.py", line 191, in _fit
    hess=hessian)
  File "C:\AppData\Local\Continuum\Anaconda3\lib\site-packages\statsmodels\base\optimizer.py", line 410, in _fit_lbfgs
    **extra_kwargs)
  File "C:\AppData\Local\Continuum\Anaconda3\lib\site-packages\scipy\optimize\lbfgsb.py", line 193, in fmin_l_bfgs_b
    **opts)
  File "C:\AppData\Local\Continuum\Anaconda3\lib\site-packages\scipy\optimize\lbfgsb.py", line 328, in _minimize_lbfgsb
    f, g = func_and_grad(x)
  File "C:\AppData\Local\Continuum\Anaconda3\lib\site-packages\scipy\optimize\lbfgsb.py", line 273, in func_and_grad
    f = fun(x, *args)
  File "C:\AppData\Local\Continuum\Anaconda3\lib\site-packages\scipy\optimize\optimize.py", line 292, in function_wrapper
    return function(*(wrapper_args + args))
  File "C:\AppData\Local\Continuum\Anaconda3\lib\site-packages\statsmodels\base\model.py", line 440, in f
    return -self.loglike(params, *args) / nobs
  File "C:\AppData\Local\Continuum\Anaconda3\lib\site-packages\statsmodels\tsa\statespace\mlemodel.py", line 646, in loglike
    loglike = self.ssm.loglike(complex_step=complex_step, **kwargs)
  File "C:\AppData\Local\Continuum\Anaconda3\lib\site-packages\statsmodels\tsa\statespace\kalman_filter.py", line 825, in loglike
    kfilter = self._filter(**kwargs)
  File "C:\AppData\Local\Continuum\Anaconda3\lib\site-packages\statsmodels\tsa\statespace\kalman_filter.py", line 747, in _filter
    self._initialize_state(prefix=prefix, complex_step=complex_step)
  File "C:\AppData\Local\Continuum\Anaconda3\lib\site-packages\statsmodels\tsa\statespace\representation.py", line 723, in _initialize_state
    self._statespaces[prefix].initialize_stationary(complex_step)
  File "_representation.pyx", line 1351, in statsmodels.tsa.statespace._representation.dStatespace.initialize_stationary
  File "_tools.pyx", line 1151, in statsmodels.tsa.statespace._tools._dsolve_discrete_lyapunov
numpy.linalg.linalg.LinAlgError: LU decomposition error.
```

I believe that there is a bug in statsmodels if I do not have any error. Can you please help me to solve it?
user: ncabay
issue_5451: state: open
title: why VAR.fit()  return VARResultsWrapper rather than VARResults
body: hello guys, I have a problem, can sombody give me a hint?

I write some code followng the example of:

http://www.statsmodels.org/dev/generated/statsmodels.tsa.vector_ar.var_model.VAR.html

However the fit()function return VARResultsWrapper rather than VARResults, however I can't  use the roots()function and test_causality()function of an object of VARResults, which is a neccesarity of building a VAR model.

What am I suppose to overcome this problem?

Thanks!
user: Kuntakimp
issue_5445: state: open
title: missing='drop' returning "Exception: Data must be 1-dimensional" error for sm.Logit
body: I am trying to perform binary logit and my data has some NaN. It looks like I should be able to use the "missing='drop'" option but I am getting an exception ("Exception: Data must be 1-dimensional"). I am using statsmodel version 0.6.1 (update: tried this with 0.9.0 and received same error).

```
import statsmodels.api as sm

...
logit = sm.Logit(y,x,missing='drop')
```

Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2018.2.4\helpers\pydev\_pydevd_bundle\pydevd_exec2.py", line 3, in Exec
    exec(exp, global_vars, local_vars)
  File "<input>", line 1, in <module>
  File "C:\Anaconda\envs\geo\lib\site-packages\statsmodels\discrete\discrete_model.py", line 418, in __init__
    super(BinaryModel, self).__init__(endog, exog, **kwargs)
  File "C:\Anaconda\envs\geo\lib\site-packages\statsmodels\discrete\discrete_model.py", line 171, in __init__
    super(DiscreteModel, self).__init__(endog, exog, **kwargs)
  File "C:\Anaconda\envs\geo\lib\site-packages\statsmodels\base\model.py", line 212, in __init__
    super(LikelihoodModel, self).__init__(endog, exog, **kwargs)
  File "C:\Anaconda\envs\geo\lib\site-packages\statsmodels\base\model.py", line 64, in __init__
    **kwargs)
  File "C:\Anaconda\envs\geo\lib\site-packages\statsmodels\base\model.py", line 87, in _handle_data
    data = handle_data(endog, exog, missing, hasconst, **kwargs)
  File "C:\Anaconda\envs\geo\lib\site-packages\statsmodels\base\data.py", line 633, in handle_data
    **kwargs)
  File "C:\Anaconda\envs\geo\lib\site-packages\statsmodels\base\data.py", line 65, in __init__
    **kwargs)
  File "C:\Anaconda\envs\geo\lib\site-packages\statsmodels\base\data.py", line 261, in handle_missing
    nan_mask = _nan_rows(*combined)
  File "C:\Anaconda\envs\geo\lib\site-packages\statsmodels\base\data.py", line 47, in _nan_rows
    return reduce(_nan_row_maybe_two_inputs, arrs).squeeze()
  File "C:\Anaconda\envs\geo\lib\site-packages\statsmodels\base\data.py", line 46, in _nan_row_maybe_two_inputs
    (x_is_boolean_array | _asarray_2d_null_rows(y)))
  File "C:\Anaconda\envs\geo\lib\site-packages\pandas\core\ops.py", line 936, in wrapper
    index=self.index)).__finalize__(self)
  File "C:\Anaconda\envs\geo\lib\site-packages\pandas\core\series.py", line 248, in __init__
    raise_cast_failure=True)
  File "C:\Anaconda\envs\geo\lib\site-packages\pandas\core\series.py", line 3027, in _sanitize_array
    raise Exception('Data must be 1-dimensional')
Exception: Data must be 1-dimensional
user: jtamerius
issue_5443: state: open
title: DOC/Design: RegularizedInvCovariance in statsmodels.stats
body: RegularizedInvCovariance is missing in the documentation, not linked to in any rst file.
merged as part of 'distributed' PR #3396

I don't remember anything about this.

related: I have PR #3230 with more robust and shrinkage covariance estimation
a bit related: #4143 matrix tools for covariances, and several related issues (and experimental code)

RegularizedInvCovariance might be too generic as a name, i.e. which regularization, we will have several of them, depending on the penalization or shrinkage or thresholding function.


user: josef-pkt
issue_5442: state: open
title: DOC: outdated or incomplete parts
body: semi-random browsing of docs

- [x] http://www.statsmodels.org/dev/_sources/gmm.rst.txt is outdated, e.g. we now have `LinearIVGMM`
- rlm.rst does not have a intro section (i.e. should mention that we only have M-estimator)
- glm.rst has formulas in Technical section, but no intro and no examples
- stats http://www.statsmodels.org/dev/stats.html#multiple-tests-and-multiple-comparison-procedures 
  still has the list of essentially internal functions
- missing in stats docs: #5443 `RegularizedInvCovariance`,  add for 0.11
- missing in stats docs: `multivariate_tools`, currently mainly internal and my example usage, needs `tools` decision. (several recent issues for collecting and organizing tools https://github.com/statsmodels/statsmodels/issues?q=is%3Aissue+is%3Aopen+tools+label%3Acomp-tools )
- missing stats functions: `correlation_tools.kernel_covariance`
- notes on diagnostics needs updating and checking, e.g. exponential lilliefors is missing in http://www.statsmodels.org/dev/diagnostic.html#normality-and-distribution-tests 
  misnomer: exponential is a distribution option in `kstest_normal (it's not just normal anymore) `https://www.statsmodels.org/dev/generated/statsmodels.stats.diagnostic.kstest_normal.html
- [x] GLMInfluence is missing, method has dead link to it http://www.statsmodels.org/dev/generated/statsmodels.genmod.generalized_linear_model.GLMResults.get_influence.html
- missing: my recent enhancements (partially internal) are missing in api and docs #4917 e.g. ultra high screening
- missing notebooks
  - VECM: mentioned here https://github.com/statsmodels/statsmodels/pull/4692#discussion_r197617712
- incomplete, bare bones notebooks
  - http://www.statsmodels.org/dev/examples/notebooks/generated/predict.html looks old and not informative enough, e.g. get_prediction is missing, stateful transforms, ... (some notes are in replies in issues, mailing list and stackoverflow)



user: josef-pkt
issue_5438: state: open
title: Exponential smoothing with seasonality initial Level value is weird
body: For Exponential Smoothing with seasonality, the initial Level (if not provided by the user) is set as follows:
`y[np.arange(self.nobs) % m == 0].mean()`
https://github.com/statsmodels/statsmodels/blob/da9d7e9431095025d6bd66d25cb87d46220769fe/statsmodels/tsa/holtwinters.py#L769

In other words, take the mean of all the points corresponding to the first season.  

This is an unusual way to initialize, and produces strange results for time series with a strongly non-zero slope: you'll start with a Level close to the mean of the series (which might not be anywhere close to the level at the start of the series).  Then you'll have huge initial seasonality to compensate.  The result is a forecast that looks fine in total but whose components are weird.  Ideally, the mean of the seasonality time series should be zero (for addititive seasonality) or one (for multiplicative), but this method results in values far from that.

Hyndman (whose text is mentioned in the documentation) [provides some suggestions](https://robjhyndman.com/hyndsight/hw-initialization/).  His simplest method is to take the mean of the first period (not the first season of each period, as is currently implemented above).  

[NIST's method](https://web.archive.org/web/20180312182644/http://www.itl.nist.gov:80/div898/handbook/pmc/section4/pmc435.htm) is even simpler: just use y[0], identical to the non-seasonal version.  (They don't explicitly mention it, but the example on the next page shows that result.)

I think we should use Hyndman's 1998 method, and change L769 to be
```
l0 = y[:m].mean() if l0 is None else l0
```

I can put up a PR if you wish.  

Thanks!

user: aelavender
issue_5430: state: open
title: MICEData update produces SettingWithCopyWarning when using a data frame with mixed types
body: When running MICE on a data frame that contains an int column, I get a SettingWithCopyWarning (I'm not imputing the int column; in fact in my original data it's an ID column that is not used in the imputation at all). Here's a short example (note that column `b` is type integer).

```python
import pandas as pd
import numpy as np
from statsmodels.imputation.mice import MICEData

d = pd.DataFrame({'a':[1, 2, np.nan, 4], 'b':[3, 2, 1, 4]})

print(d.dtypes)

miceData = MICEData(d)
miceData.update_all()
```

output:
```
a    float64
b      int64
dtype: object
/Users/matthewc/anaconda/envs/py37/lib/python3.7/site-packages/pandas/core/indexing.py:189: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  self._setitem_with_indexer(indexer, value)
```

If you convert column b to `float`, the warning disappears. I'm guessing this is because Pandas doesn't make a copy behind the scenes when all the columns are of type float.

It appears that the culprit is [this line](https://github.com/statsmodels/statsmodels/blob/master/statsmodels/imputation/mice.py#L398), where a chain of column indexing and location-based indexing is used. 

Also mentioned in passing in #4393.

I'm working on a solution to this in my branch.

sm.show_versions output:
<details>
INSTALLED VERSIONS
------------------
Python: 3.7.0.final.0
OS: Darwin 18.2.0 Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST 2018; root:xnu-4903.231.4~2/RELEASE_X86_64 x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

Statsmodels
===========

Installed: 0.9.0 (/Users/matthewc/anaconda/envs/py37/lib/python3.7/site-packages/statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.14.5 (/Users/matthewc/anaconda/envs/py37/lib/python3.7/site-packages/numpy)
scipy: 1.1.0 (/Users/matthewc/anaconda/envs/py37/lib/python3.7/site-packages/scipy)
pandas: 0.23.4 (/Users/matthewc/anaconda/envs/py37/lib/python3.7/site-packages/pandas)
    dateutil: 2.7.3 (/Users/matthewc/anaconda/envs/py37/lib/python3.7/site-packages/dateutil)
patsy: 0.5.0 (/Users/matthewc/anaconda/envs/py37/lib/python3.7/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 2.2.2 (/Users/matthewc/anaconda/envs/py37/lib/python3.7/site-packages/matplotlib)
    backend: module://ipykernel.pylab.backend_inline 
cvxopt: Not installed
joblib: Not installed

Developer Tools
================

IPython: 6.4.0 (/Users/matthewc/anaconda/envs/py37/lib/python3.7/site-packages/IPython)
    jinja2: 2.10 (/Users/matthewc/anaconda/envs/py37/lib/python3.7/site-packages/jinja2)
sphinx: Not installed
    pygments: 2.2.0 (/Users/matthewc/anaconda/envs/py37/lib/python3.7/site-packages/pygments)
nose: Not installed
pytest: 4.0.2 (/Users/matthewc/anaconda/envs/py37/lib/python3.7/site-packages)
virtualenv: Not installed
</details>
user: mattwigway
issue_5425: state: open
title: Feature request: per-bean color in the box plot.
body: Hello, 

Would it be possible to pass an array of colors to the "bean_color" parameter of the beanplot function so we can have different colors for each point?

Thanks in advance,


user: kvermeul
issue_5416: state: open
title: ENH: missing value handling in basic statistics
body: Currently we don't have much support for missing values outside of the model option for rowwise deletion.

#2630 is for improving descriptive statistics.

For univariate statistics like mean, std including robust statistics like mad or the quantile based skew and kurtosis, it would be better to have case/elementwise instead of rowwise deletion.

One problem is that simple vectorization will not work anymore if columns have nans/missing values in different rows. For simple statistics there are nan-aware functions, but we might need either a mask solution like MaskedArray statistics uses (essentially with 0/1 weights after changing to something `finite`) or looping over columns/series.

user: josef-pkt
issue_5413: state: open
title: ENH/DOC: dispersion, mixture of poisson random variables, panel data
body: (random idea based on looking at a rates and intensities example)

This is similar to the insurance case discussed with @thequackdaddy 

Suppose we observed accidents of drivers over several periods. Each driver can have one or more accidents.
The question is whether accidents are random with same intensity across events, periods and drivers.

If intensities are all the same, then Poisson is appropriate.  
If intensities vary by driver, i.e. unobserved individual heterogeneity, then the overall intensity will be a mixture of poisson distributions and the overall count distribution will have over dispersion.
There could also be a serial correlation effect from events, i.e. previous accidents lower or increase the intensity conditional on all variables including individual random effect.
(i.e. difficulty in distinguishing state dependence from individual heterogeneity (random effects constant 
over time).
In the second case, negative binomial or similar would help describing the overall pattern but cannot distinguish few accidents by many drivers versus clustering of accidents on a few drivers.

What we could do is mainly documentation of this case and show how to use various models and diagnostics to analyse those cases, and figure out which structure the "accidents" intensities have.
As a helper function we could have something that creates random samples with this pattern.

Aside: Is there a computational advantage when analyzing Poisson models with individual specific random effects in panel/longitudinal data of using gamma mixing distribution instead of normal or discrete distribution?
I.e. do we gain anything in the repeated observations case by using the mixing assumption that underlies the negative binomial distribution?  (I never checked this. My guess is that we don't gain anything, and cannot avoid numerical integration, if there are time varying covariates.)

user: josef-pkt
issue_5410: state: open
title: ENH linalg tools weighted orthogonalization
body: (just a quick idea)

We orthogonalize in several cases one set from another set of variables. 
If we have weights, then we need to orthogonalize with respect to the measure defined by the weights.

I think in the WLS case we can just use the weighted variables, e.g. `wexog`.

usecase:
hessian/EIM in GLM when "absorbing" some exog like categorical fixed effects. #5266
Once we have the `hessian_factor`, the hessian is similar to the one in WLS. We should be able to make it block-diagonal by reparameterization and compute the inverse by partitioned inverse or inverting just the block of interest.

(I don't have time right now to think about how and whether that actually works.)

The setup is similar to score_test for added variables in GLM when we use the partitioned inverse for the distribution of the score.
user: josef-pkt
issue_5409: state: open
title: SUMM/ENH: transformation models versus parameterized or nonparametric link functions
body: #624 parameterized link function
#3224 nonlinear transformation to normality

AFAIU:
link function GLM, single index models use
`E(y|x) = f(x b)`
transformation model uses
`z = f(y|x) ~ N(g(x b), sigma)`   or other distribution than normal

`f` can be assumed to be 
- known
- parameterized family of functions (e.g. Box-Cox)
- semi-, non-parametric: splines, kernel or similar

aside:  transformation model with log/exp shows up in AFT where we have both distributions (for `y` and for `z`)

I don't have an overview right now, but just saw this reference
Wang, Xin, Vivekananda Roy, and Zhengyuan Zhu. 2018. “A New Algorithm to Estimate Monotone Nonparametric Link Functions and a Comparison with Parametric Approach.” Statistics and Computing 28 (5): 1083–94. https://doi.org/10.1007/s11222-017-9781-3.
uses P-splines
I guess the article is similar to related articles for spline versions of single index models.










user: josef-pkt
issue_5408: state: open
title: ENH/SUMM: gof test, diagnostics against nonparametric alternative
body: A pretty wide topic


for now just parking a reference
Fan, Jianqing, and Li-Shan Huang. 2001. “Goodness-of-Fit Tests for Parametric Regression Models.” Journal of the American Statistical Association 96 (454): 640–52. https://doi.org/10.1198/016214501753168316.
based on Neyman test
mentions Rayner and Best for Neyman smooth tests

more general
- parametric diagnostic tests like link test, reset test, variable addition tests for non-linearities
- nonparametric kernel alternatives: this showed up when kernel regression (and kde) were initially merged, AFAIR we don't have a test function
- partitioning gof tests, similar to Hosmer-Lemeshow


user: josef-pkt
issue_5407: state: open
title: ENH: penalization for categorical exog
body: penalizing categorical regressors

parking two references

Gertheiss, Jan, and Gerhard Tutz. 2009. “Penalized Regression with Ordinal Predictors.” International Statistical Review 77 (3): 345–65. https://doi.org/10.1111/j.1751-5823.2009.00088.x.
Tutz, Gerhard, and Jan Gertheiss. 2016. “Regularized Regression for Categorical Data:” Statistical Modelling, May. https://doi.org/10.1177/1471082X16642560.

AFAIR, I read the 2009 article carefully but didn't read the 2016 article (not downloaded before, and was after my time working on penalized in 2015)

#2882 penalization, shrinkage to subspace 
#4038 SUMM overview issue
#4215 effect of pinv or ridge regularization for categorical variables

I used penalization of categorical variable, e.g. shrink to common value or common consecutive values, in notebooks. There should be more issues on this, especially also for helper functions for contrast/restriction matrices.
However, I didn't find a specific issue in a brief issue search.

It is easy to implement as a user with the available penalization classes, but we might be able to provide more automatic support, especially with the availability of factor information in patsy's `design_info`.
The idea might be that we can create restriction matrices similar to what I implemented for `wald_test_pairwise` for main effects.

one idea related to and in analogy to GAM:
In GAM with from_formula we could allow users to create the design matrix including the spline basis columns through the standard patsy formula. In that case we need some extra keywords that request the penalization of some terms.
We could do the same for factor variables, i.e. use standard from_formula but then add penalization for the parameters of categorical variables through an extra keyword.

But first step would be to write a standalone helper function that constructs the restriction matrix from a model of from the design_info. (And users could feed that to a Penalized Model class)


user: josef-pkt
issue_5405: state: open
title: ENH: disaggregation, intertemporal and spatial
body: Just adding a link to an interesting question

https://stackoverflow.com/questions/53599203/adjusting-time-series-data-based-on-seasonality-of-another-time-series
disaggregating one monthly time series to a daily time series based on a second time series that is observed daily

I never looked at this literature, but Skipper had added tsa.interp.denton for macro application (monthly, quarterly and annual data)

there is an R package with several methods for intertemporal disaggregation
https://journal.r-project.org/archive/2013-2/sax-steiner.pdf
https://cran.r-project.org/web/packages/tempdisagg/

Monthly to daily disaggregation sounds more tricky because number of days per month is not constant.

Maybe using a cyclic spline with days as continuous variable representing fractions of a month works in this case, if there is a monthly pattern. 
Otherwise a month would be just a collection of days of the week, maybe with an interpolating trend for the monthly data.
Also weeks are don't have a regular seasonal structure with respect to years or months.






user: josef-pkt
issue_5404: state: open
title: ENH: effective degrees of freedom for Penalty classes, e.g. SCADSmoothed
body: Just a diagnostic idea for Sure Independence Screening:

Currently I use a shortcut to the score test to rank candidates for addition. Dropping of variables is driven by the SCAD penalized estimation. The total number of variables included in the model is fixed as a hyper parameter.
However, it is possible that we don't include enough variables, i.e. our limit on the number of variables to add to the model is too small.

In analogy to splines and GAM: It is recommended to check that the effective degrees of freedom is not too close to the count/number of variables. If it is close to the limit, then the number of knots in the GAM case might be too small.
In the screening case the maximum number of variables to include might be too small.

We should have a statistic to quickly check whether the limit on the number of variables might be abinding constraint.

i.e. we need to extend some of the things that are available in GAM to other and more general cases.
user: josef-pkt
issue_5403: state: open
title: ENH: helper function or options for splines, various polynomial basis in nonlinearity tests
body: In one of my GAM examples I used spline basis functions to test for linearity against nonlinearity.
The usual as in `diagnostic` is to add power polynomial. Using an option for splines or other polynomials (from numpy) will provide more flexibility.

It becomes kind of trivial with available score and wald test, and eventually compare_lr_test, but some nice helper function or method would make this much easier, than trying to figure out what the 3 to 5 lines are.

related: e.g. 
#2041 score_test
#3690 reset_test
OLS diagnostic lm test  (can already take a user `func` which I didn't remember)
ww.statsmodels.org/devel/generated/statsmodels.stats.diagnostic.linear_lm.html
user: josef-pkt
issue_5402: state: open
title: ENH: special models: generalized linear array models
body: (just parking a reference that I looked at a while ago)

AFAIU: balanced panel with 2-way (or k-way) crossed fixed effects represented in matrix form to save on memory and computation, instead of using long form and Kronecker products.

https://en.wikipedia.org/wiki/Generalized_linear_array_model
http://www.macs.hw.ac.uk/~iain/research/ISI_IDC_GLAM.pdf

Currie, Iain D., Maria Durban, and Paul HC Eilers. "Generalized linear array models with applications to multidimensional smoothing." Journal of the Royal Statistical Society: Series B (Statistical Methodology) 68, no. 2 (2006): 259-280.

examples use P-splines for contemporaneous and intertemporal smoothing.



user: josef-pkt
issue_5400: state: open
title: plot_components  smoothed observations
body: The plot_components method of statsmodels/tsa/statespace/structural.py has a "which" argument to choose whether to plot the results of the Kalman filter or smoother.

However if `observed==True` the predicted observations and confidence intervals are always calculated with the filter results even if `which=="smoothed"`:

https://github.com/statsmodels/statsmodels/blob/428b49c08bcad07c37f5230d77b78af6487db38a/statsmodels/tsa/statespace/structural.py#L1599-L1600

Should this be replaced with something like:

```
if which == "smoothed":
    predict = self.smoother_results.smoothed_forecasts[0]
    std_errors = np.sqrt(self.filter_results.smoothed_forecasts_error_cov[0, 0])
else:
    predict = self.filter_results.forecasts[0]
    std_errors = np.sqrt(self.filter_results.forecasts_error_cov[0, 0])
```

user: AliHeggie
issue_5398: state: open
title: BUG/ENH: singular exog and anova_lm
body: based on a stackoverflow question
https://stackoverflow.com/questions/53489106/why-am-i-getting-a-inf-or-nan-error-even-though-data-has-none

it looks like anova_lm raises an exception with singular exog, maybe an empty cell case.

This needs to be investigated and maybe find a solution, a better exception message or a justification why we want to ignore this specific case.





user: josef-pkt
issue_5394: state: open
title: ENH/REF safe exp, prevent overflow and underflow
body: mgcv has a safe `exp`, `notExp`, `notExp2` to prevent overflow and underflow.

This will be useful where we use a log-link, GLM, Poisson, NegativeBinomial, ... which have a tendency to overflow if the exog are not well scaled (i.e. have larger values)


user: josef-pkt
issue_5392: state: open
title: Adaptive Kalman filter for state-space models?
body: This is a question.

I was wondering if adaptive Kalman filtering is implemented for state-space models?

If it not implemented, could you, please, let me know what the relevant files are that I need to modify to add it.

Thank you for your help!
user: vss888
issue_5391: state: open
title: double seasonal SARIMAX
body: Hello, 
I want to forecast electricity demand. It typically has some daily and weekly seasonal pattern. Therefore, some people claim, one might use a double seasonal ARIMA approach. 

I wonder, whether can be realized with the current SARIMAX implementation somehow? 

Setting the seasonal_order blindly to 168, which is 24 hours times 7 days or one week, kills my memory ( > 20GB). I would be very happy if someone has any suggestions. 
Best regards
user: philipphanemann
issue_5388: state: open
title: ENH/Design: collect tools for neighborhood indices, binning, coarsening
body: This is similar to collecting linalg tools.

We need every once in a while function for defining neighborhood or converting continuous variables into groups or bins.

two different cases

- neighborhoods of each observation: 
  - for one, possibly multivariate variable: find neighbors e.g. local regression
  - for two variables: find neighbors in second dataset, e.g. matching estimators PSM
- binning or coarsening (has weird name in pandas and R): partition continuous sample space 
   partition dataset into finite discrete subsets
   several use cases already for binning, e.g. kde, in my differencing nonparametric PR, gof tests
  - grid: equal spaced
  - coarsening: approximately equal spaced

aside: setup cost and keeping intermediate information around.

AFAIR (might be incorrect):
- lowess does not store the neighborhood indices, it is directly integrated in the estimation model, This makes it impossible to swap the neighborhood
- full, non-binned kde also loops through the intervals without keeping neighborhood indices
- scikit-learn keeps indices or neighborhood data as attribute for reuse during optimization and similar.

current use cases in statsmodels (AFAIR)

- binning: 
  - kde, my kernel regression, differencing PR (includes 2-D binning IIRC)
  - gof-tests, equal spacing or equal weight bins. chisquare test, Hosmer-Lemeshow (not yet) but similar in discrete._diagnostic_count
  - GAM, splines knot selection, defines intervals based on quantiles (of data) or equal spaced
- neighbor matching
  - imputation
  - propensity score matching PR

related
- discrete points instead of intervals: e.g. `predict` when we want to evaluate a function on percentiles of exog.

We need helper functions for the `predict_at` and `margins` use case to create new selective datasets, e.g. similar to the lsmeans case.
It is a bit cumbersome to create DataFrames when looking at different percentiles of one or two variables keeping the others constant or averaging over the others, or creating it for each level of the categorical variables. It's similar if we want to plot predictions at predefined grid values.







  
user: josef-pkt
issue_5387: state: open
title: SUMM/DOC/ENH interaction between continuous, categorical exog
body: general issue: I have no idea how well this works, and AFAIK we don't have an example

How well does patsy handle this? What is the parameterization? Which collinear columns are dropped?

post-estimation support

so far we have nothing, AFAIK

- wald tests, nothing specific, 
  - `wald_test_terms` should work if names or terms have a identifiable structure
  - premade specific tests for interaction effects, e.g. diff across categorical levels
  - wald test restriction matrix for diff should be just two calls to np.eye or diag with the right indices
     (I'm doing something similar for zero tests in GAM test_significance)
- margins, partial predict: nothing yet. We need information about underlying variables
- graphics, plots: also nothing yet. We still don't have interaction plots that work for models neither for categorical nor for categorical-continuous interaction.

We might have enough information in the design_info to support this when formulas are used.

interaction exog columns created by statsmodels:
- GAM in mgcv has a `by` option to split estimate and interaction between smooth component and categorical variable #5332
  I don't know (yet) how we should handle collinear columns in this case, e.g. main effect plus "left-over" interaction effect, or only full interaction effects. `:` versus `*` in formulas.




user: josef-pkt
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_5386: state: open
title: ENH: helper method for profile/restricted likelihood
body: just an idea for reusable components.

We have several cases where we want to look at changes of the likelihood function if we vary an additional parameter.

This can be a hyper parameter like penalization weight in GAM, penalized models (my current use case) or the dispersion parameter in GLM-NegativeBinomial.
Or this can be a parameter that is assumed to be fixed as in profile confidence interval or score confidence interval computation, issue #2722, PR #1791 

dual usage
Kerby mentioned in the context of profile confidence interval that it would be useful to evaluate the profile likelihood on a grid instead of just looking for the conf_int bounds.
Similarly, I was looking at values for the penalization parameters in GAM on a grid and use it for optimization.

Currently I am using internal nested functions for the optimization or rootfinding. Instead the idea is to make this generally available, i.e. add a method `get_loglike_function` that returns the loglikelihood function or lambda function that is just a function of one, or a list of parameters. The function will have the `state`, internal attributes, for the things that remain fixed during optimization.
We can add keywords like start_params for additional options that depend on the parameter values.
(e.g. for warm start when evaluating on a grid.)

This would allow arbitrary usage, the function can be evaluated on a grid, inside optimization or rootfinding routines. And would allow basic numerical computation like numerical derivatives.

The main advantage is that users have access to the model and parameter specific computational shortcuts. At the moment users need to create new models each time they want to change a fixed `model.__init__` parameter.
see comments in #2722

Note: We need to make sure that we don't change the model itself, i.e. leave or keep model attributes unchanged. This is not a problem when we have to create an auxiliary model for the profile likelihood, but could happen if we change a "hyperparameter" directly.
In the worst case we might need a cleanup stage after usage, which would require a context manager or `yield` to reset attributes when the user/caller is finished. (I'm not sure how this would work.) Or a flag to do cleanup if `fit` is called and the state is "dirty". This sounds fragile in either case.



user: josef-pkt
issue_5385: state: open
title: ENH: design_info for endog
body: patsy also provides a design_info for endog, that we currently don't attach and don't use.

This can be useful to look at possible endog transformation, or endog names in case of 2-D endog.

`y, X = build_design_matrices([y.design_info. X.design_info], newdata)`
https://groups.google.com/d/msg/pystatsmodels/sDED93s5Pk4/Ktf9QpPMAQAJ

related issues
#1449 endog transformation in predict
#2733 endog in Logit and GLM-Binomial treated as categorical by patsy, and a few more similar to this #878, #2181

user: josef-pkt
issue_5384: state: open
title: Covariance matrix is singular or near-singular, with condition number    nan. Standard errors may be unstable.
body: Hi I am trying to use customized LocalLinearTrend. I have one observation variable and 12 unobserved states + 1 intercept vector with dimension as 12*1.  Here is the code and error. 

class LocalLinearTrend(sm.tsa.statespace.MLEModel):
    def __init__(self, endog):
        # Model order
        k_states = k_posdef = 12

        # Initialize the statespace
        super(LocalLinearTrend, self).__init__(
            endog, k_states=k_states, k_posdef=k_posdef,
            initialization='approximate_diffuse',
            loglikelihood_burn=k_states
        )

        # Initialize the matrices
        self.ssm['design'] = np.array([1,1,1,1,1,1,1,1,1,1,1,1])
        self.ssm['transition'] = np.array([[1,0,0,0,0,0,0,0,0,0,0,0],
                                       [0,1,0,0,0,0,0,0,0,0,0,0],
                                       [0,0,1,0,0,0,0,0,0,0,0,0],
                                       [0,0,0,1,0,0,0,0,0,0,0,0],
                                       [0,0,0,0,1,0,0,0,0,0,0,0],
                                       [0,0,0,0,0,1,0,0,0,0,0,0],
                                       [0,0,0,0,0,0,1,0,0,0,0,0],
                                       [0,0,0,0,0,0,0,1,0,0,0,0],
                                       [0,0,0,0,0,0,0,0,1,0,0,0],
                                       [0,0,0,0,0,0,0,0,0,1,0,0],
                                       [0,0,0,0,0,0,0,0,0,0,1,0],
                                       [0,0,0,0,0,0,0,0,0,0,0,1]])
        self.ssm['selection'] = np.eye(k_states)
        self.ssm['state_intercept']=optimum_cost_sale_quarter_201604.values

        # Cache some indices
        self._state_cov_idx = ('state_cov',) + np.diag_indices(k_posdef)

    @property
    def param_names(self):
        return ['sigma2.measurement', 'sigma2.level', 'sigma2.trend']

    @property
    def start_params(self):
        return [np.std(self.endog)]*13

    def transform_params(self, unconstrained):
        return unconstrained**2

    def untransform_params(self, constrained):
        return constrained**0.5

    def update(self, params, *args, **kwargs):
        params = super(LocalLinearTrend, self).update(params, *args, **kwargs)
        
        # Observation covariance
        self.ssm['obs_cov',0,0] = params[0]

        # State covariance
        self.ssm[self._state_cov_idx] = params[1:]
mod=LocalLinearTrend(sales_time_201604)
res1=mod.fit()
print (res1.summary())
user: ZhonghuaGao
issue_5382: state: open
title: ENH: add OLSVectorized, initial version
body: see #2203 adding special_linear_model
see #4771 adding vectorized OLS

this subclasses OLS and RegressionResults but only a subset of results will work

The core attributes are working correctly, 
Among the methods I only checked conf_int and t_test so far.

user: josef-pkt
issue_5381: state: open
title: FAQ/ENH: penalty parameter optimization problems
body: It can be difficult to minimize aic, bic or gcv to find the optimial smoothing or penalty parameter in GAM.

I looked at the behavior of scipy optimizers when only the objective function is provided. (mgcv also has approximate derivative but I have not included any.) 
The example is miles per gallon in city traffic depending on 2 categorical variable and 2 smooth, B-spline terms, weight and horse power. The correlation between weight and horse power is 0.75 and both variables have a similar downward sloping effect on mpg.

- penalization parameters have to be positive or non-negative, so it is best to use a log-link, i.e. use `exp(alpha)` for the optimization
- the penalty parameter estimates depends on starting values and optimization method, nelder-mead, bfgs or basinhopping.
- using basinhopping with nelder-mead seems to converge resonably well to the same values from different start_params. Using default lbfgsb with basinhopping did not look good.
- none of the estimated penalty parameters are close to those of mgcv. But mgcv values are large and there does not seem to be much effect of varying the penalty parameter in that range.
- aic undersmooths for some penalty parameter estimates, often undersmooths only one component
- gcv has very smooth estimate, and likely oversmooths
- using the average between aic and gcv estimate looks good in graph.

This means we/users need to check the results in case the optimizer gets ends in local optima or near-optima.
One possibility is to start gcv minimization at the aic estimate and vice-versa to avoid that those are in different local optima.
Another possibility in case of high correlation would be to change the penalty parameters in opposite direction for new starting values, eg. double one and half the other. This would help in case when a local minimum has one spline function oversmoothed and the other undersmoothed.

**visual inspection**
using the plot_partial method can be used to quickly see the smoothness of the estimated partial prediction
`fig = res_aic.plot_partial(0, cpr=True)`


current computational structure:

- we can use the internal fit_pirls to change the penalization parameter alpha during the optimization.
- The `fit` method currently does not have an alpha argument and we need to create a new model with the chosen alpha. The smoother or spline instance can be reused when creating a new model.
- Currently we cannot have several results instances that are based on different alpha and hold the same model instance because the alpha is a model attribute and not a results parameter.

This structure will likely change in future versions.

simplest usage:

```
from scipy.optimize import fmin, basinhopping
fun = lambda p: gam_bs._fit_pirls(alpha=[np.exp(p[0]), np.exp(p[1])]).aic
a = fmin(fun, x_aic)
print(a, np.exp(a))
gam_bs_aic = GLMGam(df_autos['city_mpg'], exog=exog, smoother=bs,
                alpha=np.exp(a))
res_aic = gam_bs_aic.fit()

fig = plt.figure(figsize=(6, 8))
ax0 = fig.add_subplot(211)
fig = res_aic.plot_partial(0, cpr=True, ax=ax0)
ax1 = fig.add_subplot(212)
fig = res_aic.plot_partial(1, cpr=True, ax=ax1)
```

or using basinhopping
```
x0 = [1, 1]
basinhopping(fun, x0, minimizer_kwargs={'method': 'Nelder-Mead'}, niter=10)
```

user: josef-pkt
issue_5379: state: open
title: ENH: weights for GAM and nonparametrics, especially var_weights
body: The new GAM implementation does not have support var_weights or freq_weights.
Maybe also lowess or similar could have data weights.

Somewhere (?) I saw it mentioned that we can handle repeated observations by combining them to an average and use var_weights.

The advantage for non- or semiparametric estimation would be that we have unique exog, and don't have to worry about replicates. For example lowess currently cannot handle large number of replicates or categorical explanatory variables because of the way neighborhoods are computed.

However, the problem is that var_weights or freq_weights would have to be taken into account when we choose knot location based on quantiles of the observed exog. The current algorithm for quantile knots base on patsy/mgcv does not handle weighted quantiles. On the other hand, there is also an advantage to selecting knots based on quantiles of the unique data, i.e. we avoid setting knots to close to each other, but we also loose large knot density at high exog density regions.

Related model: An old Silverman article on splines uses var_weights for the motorcycle dataset which has clear heteroscedasticity increasing in the explanatory variable (time).
(I have not looked at heteroscedasticity in splines yet, except that we have automatic, inherited support for cov_types like HC0.

One detail: helper functions to aggregate raw data to condensed data either by combining unique observations and using freq_weights or combining unique exog with average endog and var_weights.
AFAIR, there are some examples in the GLM weights notebooks or unit tests.







user: josef-pkt
issue_5378: state: open
title: ENH: L1 penalized trend filtering
body: this is mainly to park some references

trend filtering is similar in behavior to adaptive splines, but similarly to truncated power splines can be used for sparse penalization, i.e. we can penalize parts/coefficients to zero.

From the approach it sounds similar to the differencing estimators in issue #3384 and PR #3380, but no cross citation in the references.

Tibshirani, Ryan J. "Adaptive piecewise polynomial estimation via trend filtering." The Annals of Statistics 42, no. 1 (2014): 285-323.
https://arxiv.org/pdf/1304.2986.pdf

Wang, Yu-Xiang, Alex Smola, and Ryan Tibshirani. "The falling factorial basis and its statistical applications." In International Conference on Machine Learning, pp. 730-738. 2014.
http://proceedings.mlr.press/v32/wange14.pdf


Kim, Seung-Jean, Kwangmoo Koh, Stephen Boyd, and Dimitry Gorinevsky. "L1 Trend Filtering." SIAM review 51, no. 2 (2009): 339-360.
This article has some implementation in various programming languages on github but essentially all are GPL.
https://github.com/bugra/l1  is Apache, but doesn't look like a lot of code.


(I did not read any of the articles, but saw it briefly described in some course lecture notes by Tibshirani and coauthors.)






user: josef-pkt
issue_5375: state: open
title: FAQ/DOC: diagnostic, overdispersion tests and tools for Poisson
body: see #1537 and issues for general score/LM tests

I lost a bit an overview.
What's available as official or experimental or hiding in notebooks?

e.g. https://stats.stackexchange.com/questions/66586/is-there-a-test-to-determine-whether-glm-overdispersion-is-significant
quick answer for statsmodels? and do we have unit tests against those R functions?

user: josef-pkt
issue_5372: state: open
title: ENH: compare_lr_test for GLM
body: general issue #3446 with several other related issues

This is specifically to get started with a quick implementation for GLM for nested models
I would like to have it for GAM to compare with wald and score tests for smooth components.

`compare_lr_test(...., method='loglike')`
doc string: lr type tests for comparing nested models, self is unrestricted, other is restricted model

method can be in 'loglike'/'llf', 'deviance'/'dev' or 'pearson_chi2'

all are distributed chi2
basic setup: assume MLE is correctly specified and we ignore over dispersion or other complications in the first version.

problem: if we use deviance or pearson_chi2 for scale estimate, then there is no gof test based on those two, but we should get the equivalent for a nested F or chi2 test in OLS/linear model.


user: josef-pkt
issue_5368: state: open
title: BUG/ENH: summary fractional/float instead of integer in nobs, df_xxx
body: GLM summary and wald_test summary display integers even if underlying statistic is float.

nobs/wnobs, df_resid, df_denom, df_num can all be fractional numbers in some cases instead of integers.

We need a nice `int_or_float` formatting.
user: josef-pkt
issue_5367: state: open
title: REF/Design: split `XXXResults.__init__`, add `initialize_inference`
body: This is currently just an idea, I haven't looked at the details.

`Results.__init__` currently have two tasks: 
- attach model and params  (available for prediction)
- set up attributes mainly cov_params for inference.

In GAM I get into a mess because I need to override df_resid, and the scale estimate that was provided by `model.fit` before computing cov_params and similar.
https://github.com/statsmodels/statsmodels/pull/5296#issuecomment-435643063

One possibility to get a hook in the correct place would be that we add a new method `_initialize_inference` that can manipulate results attributes before continuing with the results creation.

The alternative design would be to add those preparation work for inference into the `model.fit` itself.

related 
#4664 general issue for nuisance parameters that are not in params
#2393 SUMM: creating and recreating results instances
user: josef-pkt
issue_5366: state: open
title: ENH/Design: do we need a PenalizedResultsMixin
body: I don't know yet.

Currently we don't have a special generic Results class for penalized models. GamResults subclasses only the corresponding model results class, e.g. `GLMGAMResults(GLMResults)` and and so does TheilGLS `TheilRegressionResults(RegressionResults)`

A PenalizedResultsMixin might become useful when we have some generic results methods to support penalized models, and some idea about what should and can be provided consistently across penalized model.

one case where this might be useful is the have a "convention" for how to handle effective degrees of freedom in PenalizedMixin subclasses, and e.g. where to compute the hat_matrix_diag or a generic equivalent of it.



user: josef-pkt
issue_5365: state: open
title: ENH: Wald test with effective degrees of freedom
body: In penalized estimation the effective degrees of freedom, edf, differ by term or parameter.
e.g. in GAM, mgcv in R, the Wald test that the entire smooth term is zero uses the edf for the F-test, i.e. the number of constraints imposed is not an integer but is the edf which is smaller.

To handle this generically, we will need to enhance the Wald test methods to include parameter specific edf.
needed in GAM PR #5296 but I don't think I will implement it there.

We ran into parameter specific df somewhere before (maybe in imputation or survey)

maybe related: In panel data we also have the parameter specific df depending on whether we have within effects or between effects and whether the asymptotics is large nobs or large n_cluster.
Cluster robust standard errors use number of clusters and large n_cluster asymptotics. 
AFAIR we use a minimum in the 2 crossed clusters case, which might provide an upper bound on the df in wald tests.

Similarly, I think we can argue that using the integer/number of restrictions should provide an upper bound and might make the test conservative, ? (just a guess, I didn't go through the math.)

user: josef-pkt
issue_5364: state: open
title: BUG/ENH: LM, score_test variable addition with redundant variables (rank)
body: #2041 general issue

I'm running a variable addition GLM.score_test
`res_glm.model.score_test(res_glm.params, exog_extra=bs.basis_)`
to test the addition of splines when the current model includes the linear term.

The score test uses the number of columns as the number of restriction, while the actual addition is smaller because of the repeated subspace.

example base on GAM PR #5296 

bs.basis_ includes the basis functions for two variables. Those two are included as linear terms in `res_glm.model`. Each basis includes the linear function as a linear combination of basis functions.
The constant has been removed from basis function, so only the two linear/trend terms are "duplicated".

```
res_glm.model.score_test(res_glm.params, exog_extra=bs.basis_)
(array([ 108.91677455]), array([  3.09383926e-14]), 20)

bs.basis_.shape
(203, 20)

from statsmodels.stats.multivariate_tools import partial_project
projection_resid = partial_project(bs.basis_, res_glm.model.exog)
np.linalg.matrix_rank(projection_resid.resid)
18
```

**update**
using the pca helper function knowing already how many factors to keep

test statistic is the same, but p-value is corrected to take account of the reduced number of constraints

```
from statsmodels.sandbox.tools.tools_pca import pca
red = pca(projection_resid.resid, keepdim=18)[1]
res_glm.model.score_test(res_glm.params, exog_extra=red)
(array([ 108.91677455]), array([  5.00598345e-15]), 18)
```
user: josef-pkt
issue_5363: state: open
title: ENH: predict_partial: prediction for subset of exog
body: partial prediction of the linear predictor is used in GAM, #5296, so we can see the nonlinear function implied by a subset of exog (spline basis in the GAM case).

This should be useful in more general, e.g. when we have terms that include several columns.
The idea would be similar to `wald_test_terms`.

The main usecase is only for the linear predictor part, but we can extend it to the `eform`, e.g. `exp(linpred_partial)` for Poisson or Logit.

It will not make sense transforming to the endog space using inverse link function because that needs all exog, i.e. needs to be covered by `predict`, `predict_at` similar to `margins`.

I'm not sure what the API should look like, how does user specify which part to predict.
E.g. when a formula was used, then we could find categoricals and splines, or string/name matching.
The user still needs to provide the "appropriate" data if we want partial predict with new exog.

GAM is a special case because we create and have the information about the design in our model.

user: josef-pkt
issue_5362: state: open
title: Adjusted boxplot for skewed distribution
body: Boxplots are good for outlier analysis, but do not perform as well on skewed distributions. To better handle skewness, an "adjusted boxplot" method has been suggested.

Statsmodels contains [an ingredient of the adjusted boxplot function](https://www.statsmodels.org/dev/generated/statsmodels.stats.stattools.medcouple.html), even referencing the original paper.

## Feature request
Add an `adjusted_boxplot` method to help with outlier analysis on skewed datasets.
user: brylie
issue_5358: state: open
title: FAQ/ENH/Design: points for spline penalization matrix
body: This is another design issue based on GAM PR #5296 

Everything AFAIU, I didn't work my way through all the details

example UnivariateBSplines
```
basis, der_basis, der2_basis = make_bsplines_basis(self.x, self.df, self.degree)
cov_der2 = np.dot(der2_basis.T, der2_basis)
```

cov_der2 is the penalty matrix which is computed on the observed x, and not on the knots are a fixed grid of x points.
AFAIU, this computes the average weighted by the empirical distribution of x.

`integral D_2 f(x)' D_2 f(x)  d F(x)`
instead of
`integral D_2 f(x)' D_2 f(x)  d x`

D_2 f(x) is second derivative of a spline basis function.

I think the implication of this is that we penalize more heavily fluctuations in neighborhoods that have many points, which is just the opposite of what we would usually want. This is compensated in the quantile knot spacing by having also more knots in those high density neighborhoods.

Somewhere I read the case of using weights W = {w(x_i)}_i provided by the user, e.g.
`integral D_2 f(x)' W D_2 f(x)  d F(x)`

To allow for this we would need options for x used in penalty derivative computation x_der2, and for user specified weights.
This would allow for penalization that is unequally weighted by observation or neighborhoods of x, e.g. useful for adaptive splines if user weights are chosen by some adaptive criterion.

Another options would be using simplex or other numerical integration procedures for arrays instead of the simple average.

interesting reference
Wand, M. P., and J. T. Ormerod. 2008. “On Semiparametric Regression with O’sullivan Penalized Splines.” Australian & New Zealand Journal of Statistics 50 (2): 179–98. https://doi.org/10.1111/j.1467-842X.2008.00507.x.

They have the algorithm how to construct the penalty matrix using simplex integration that uses knot and half-way in between knot points, but I think is an estimate of the integral w.r.t. x.

Aside: they use exterior knots that are all replicated to get near-linear extrapolation if exterior knots are outside of the data interval.
They mention that P-splines (Marx-Eilers) just extend knot spacing to the exterior knots, i.e. no replication.

It looks like B-splines like in patsy are O'Sullivan splines. (I haven't seen any other characteristic besides replicated boundary knots compared to "just plain B-splines".)

Documentation:
- show it just works without details and matches mgcv up to "known differences"
- explain implementation details for those interested.

Problem of not having the implementation details, i.e. (math) description of what version is implemented: It is difficult to figure out what differences across methods or packages are as in #5355.

e.g. I don't know what CyclicCubicSpline does, which has the same penalty matrix as mgcv.



user: josef-pkt
issue_5355: state: open
title: Why are the critical values in coint_johansen in statsmodels in Python so different from the ones in ca.jo in urca in R?
body: I just asked [this](https://stats.stackexchange.com/questions/374202/why-are-the-critical-values-in-coint-johansen-in-statsmodels-in-python-so-differ) question on stats.stackexchange.com.

I have attached the dataset here to make it easy to reproduce.

[data.txt](https://github.com/statsmodels/statsmodels/files/2524162/data.txt)


user: c00kiemonster
issue_5350: state: open
title: ENH: penalized RLM and QuantileRegression similar to GAM
body: aside: pygam has penalized Expectile regression according to the examples/documentation

Currently RLM and QuantileRegression only use a IRLS algorithm

For RLM we can add gradient fit with scipy optimizers which we could use with the PenalizedMixin.
some complication is that RLM has it's own cov_type's

QuantileRegression would be more difficult, but we could get the smoothed approximation through RLM estimation, but the cov_type is a special case for kink at zero, i.e. uses a kernel density estimate at zero. The standard errors differ by a large amount in some quick experiments comparing RLM with QuantileRegression (while working on expectiles or M-quantiles)

Alternatively we would have to use penalized IRLS for quantile regression as in the GAM version.

patsy's splines work well with Quantile Regression, but we don't have a penalized version and need to limit the number of knots without it.

a reference (I didn't read it)
Yoshida, Takuma. 2016. “Asymptotics and Smoothing Parameter Selection for Penalized Spline Regression with Various Loss Functions.” Statistica Neerlandica 70 (4): 278–303. https://doi.org/10.1111/stan.12088.

user: josef-pkt
issue_5349: state: open
title: ENH: GAM: support/domain for derivative penalty
body: In the description of the spline bases in the mgcv documentation it sounds like the derivatives uses in computing the derivative penalty can be different from the knots used in evaluating the spline.
e.g. b-spline documentation has a discussion on extrapolation and penalizing only in the interior or also at the outer knots.
(it also mentions that extrapolation uses linear prediction)

The current GAM code in #5296 evaluates the derivatives only at the data points, i.e. we don't penalize outside of the data domain. There is no option and currently no supporting code to evaluate derivatives at other points. 
This is irrelevant in the current B-spline case because all outside knots are replicated knots at the boundary, and we cannot do extrapolation.

I might add in #5296 that we can evaluate the spline derivatives at arbitrary points for prediction, but will not change the points used in the penalization.

I think that one implication of using datapoints is that high `x` density areas have more weight in the penalization, i.e. we integrate with respect to the (empirical) distribution of x, instead of using an unweighted integral over an interval of the real line.

Also, currently gam only allows for second derivative penalties.

(This will not be relevant for P-splines with equal-spaced knots and difference (of params) penalty.)

user: josef-pkt
issue_5347: state: open
title: FAQ/Design: stateful transform for exog across datasets
body: When we use statefull transformation in the creation of the design matrix, `exog`, then the parameterization is data dependent.

This is the case with categorical variables if not all levels are available in all datasets. However, in this case the parameters for the missing levels cannot be estimated.

Another example are splines with data dependent knot location.

A polynomial basis will be statefull if it includes data dependent transformation. In R polynomials in formulas are orthogonalized and the orthogonalizing transform is statefull. When using numpy polynomial it is possible to map the data into a fixed domain for better numerical properties. Because this mapping depends on the data, this transformation is also stateful.
In contrast simple polynomial terms like x, `x**2`, `x**3`, ... are not stateful but have widely varying values if x is not restricted to small numbers.

When we want to have the same parameterization across datasets, for example to compare the estimated parameters across datasets, then we need to fix the parameterization of the stateful transformations before creating individual models.
For example, we can choose the knots based on the entire set of data and use the same knots when creating and estimating individual models.
If we are only interested in the prediction, then the choosing a common parameterization is less important and we can choose an appropriate model separately for each dataset.  
user: josef-pkt
issue_5346: state: open
title: ENH: generated exog, two stage estimation and inference
body: just some thoughts and cross-model connections for generated regressors, data dependent exog transformation, two-stage or nested estimation and implications for inference

see https://github.com/statsmodels/statsmodels/pull/5296#issuecomment-433484696

Note: inference for nested or alternating optimization is more difficult than two-stage because all stages depend on each other, i.e. simultaneous inference instead of sequential inference, and derivative of moment conditions doesn't have recursive structure in general.

another connection also used in treatment effects:
The final statistic of interest might not be the model parameters themselves, eg. ATE or ATT.
a bit related (found because it was mentioned on twitter)
Visually Communicating and Teaching Intuition for Influence Functions
https://arxiv.org/abs/1810.03260
an explanation for using influence functions to derive the distribution of treatment effects.
(I stopped reading after a bit of skimming, it mainly looks at the nonparametric case.) 

user: josef-pkt
issue_5342: state: open
title: ENH/Design: statsmodels equivalent of design_info
body: (I didn't find an issue, but I commented on this already)

We need some `design_info` when we are not directly using patsy's formulas.

current case: GAM creates multi-column `terms` that need special treatment. Currently this is handled directly in the model and results classes.
To make this more generic we need some additional information for terms that are created by statsmodels, or by the user, in which case the user needs to provide the information.

A generic idea was to allow for `exog_extra` in models that are user specified terms, similar to using terms like splines or categoricals in the formulas, but under the control of statsmodels without formula parsing. 
Current example is polynomial trend in tsa models, and seasonal dummies in VECM.

Another usecase would be if we use patsy for individual terms but then want to combine this term into a full `exog` used in the models.
Related would be a `drop` methods for formula terms.
One problem with formula terms is that those are not independent of each other, e.g. dropping a reference category for categoricals depends on the presence of a constant, and it's more complicated with interaction terms.

This is mainly a convenience feature for users, e.g. we can provide additional post-estimation results without the user having to provide the extra information.
Similar to what we can do now with the additional information in patsy's design_info, for example `wald_test_terms`. 
GAM will allow partial prediction, i.e. the contribution of a "term" to the linear predictor.
Another application is `margins` with respect to a non-linear function in an explanatory variable that is a linear combination of basis functions like polynomials or splines, which we don't have yet.

First part of design_info would be just named terms with index to the relevant columns of exog and of params, plus maybe a reference to the underlying instance that handles that term, e.g. a penalized spline instance.



user: josef-pkt
issue_5341: state: open
title: DOC/TST: example and unit test for equivalence of fit_constrained and large penalization
body: AFAIR, I don't have any examples or unit test cases that compares fit_constrained or fit_zeros with penalized estimation, i.e. comparing exact constraints with heavily penalized stochastic constraints.

This would be a bit similar to constrained optimization with a Lagrangian where `pen_weights * weights` are the Lagrange multipliers.
user: josef-pkt
issue_5340: state: open
title: ENH/Design: combination and addition of penalty functions
body: Combination and addition of penalties or penalty functions is currently just implemented as a few special cases. 
We need a consistent pattern and generic classes for it.

There are at least 3 different ways how to combine penalties 
(sounds a bit similar to join options in dataframe merge)

- (1) add additional restrictions to an existing penalty function
- (2) combine two separate penalty functions defined for same `params`, e.g. L1+L2 in elastic net
- (3) concatenate penalty functions for different, non-overlapping terms or params

(1) is possible in the base._penalties class by row concatenating `restrictions`
(2) is missing outside of fit_regularized elastic net. In base._penalties we currently would have to define a new Penalty subclass for the specific combination
(3) is implemented in GAM MultivariateGamPenalty for L2 case

GAM MultivariateGamPenalty sums over individual penalty classes, but assumes in some parts that those penalty classes partition `params`, (with possibly and unpenalized inital part of params).

We need something similar for (2) with summing over individual penalty classes that are all defined over the same parameter space.
Difference between (2) and (3) is mainly the embedding of individual penalties into a large common parameter space. MultivariateGamPenalty currently uses start_idx and mask.

user: josef-pkt
issue_5339: state: open
title: ENH: GAM Penalty classes locally quadratic approximation
body: `MultivariateGamPenalty`

`penalty_matrix`, `penalty_matrices` hold the matrix or prior-covariance for the quadratic penalization in GAM.

Should those depend on `params`?

In L2 penalization second derivatives and penalty matrix are constant.
If we want to use them as local approximation, then they would need to vary as we change the point of local approximation.
But I'm not sure we need dependence on `params` for L2 approximation to L1 or other penalization functions.


user: josef-pkt
issue_5334: state: open
title: Design: options for parameterization/constraints in GAM
body: A follow-up issue to #2744: API design

I am adding identifiability constraints to penalized splines, corresponding to patsy's `constraints='center'` options.

What is the default and how should we let users choose this?

- mgcv always imposes the constraint. It looks like there is no option to turn it of when using `gam`
- patsy doesn't seem to impose it, even if there is already a constant.

We need the constraint when we have multiple GAM components or a separate constant in `exog`.
We don't need the constraint if we have just one simple component and no constant in `exog`.

the default should be something like

```
if exog is None or has_const(exog):
    impose constraint on first component
if component is not the first component:
    always impose constraint
```

regression through origin: users may want to impose constraint even if there is no constant in exog.
(That was my first test case because I hadn't understood that mgcv imposes constrained even if there is no intercept.)
Always imposing constraint as in mgcv would work, but then users have to add the constant to exog in most use cases, which is inconvenient.
Should we add exog=ones if exog is None, and we always impose constraint? I don't like that, looks like a messy API.

what about interactions?

We will also need to get eventually the `from_formula` interface, given that patsy now provides enough information in `design_info`. patsy will add the const by default, but the 'center' default might be wrong.

needs review after we see how well GAM works.


user: josef-pkt
issue_5332: state: open
title: ENH/Design: support for spline-dummy interaction in penalized/GAM
body: R mgcv has a `by` option for the interaction effect of penalized splines and a categorical variables.

I don't know yet how we will support this. patsy can create the interaction term using its splines.

two cases (that I can think of)

- spline has full support for each categorical level
- categorical level partitions the support of the explanatory variable in the spline, e.g. regression continuity design.

I think the former should be easy using standard interaction (like kronecker product for unbalanced dummies).
The latter will be more difficult because the spline cannot be estimated in the unobserved part.
I have a simple example without penalization in the causal impact, synthetic control PR #3647, or in one of the related experimental notebooks.
user: josef-pkt
issue_5331: state: open
title: BUG/REF: Regression Results use_t wrong if cov_type='nonrobust'
body: I'm not sure anymore about the below. 
AFAICS, use_t is still handled through get_robustcov_results if cov_type is not 'nonrobust'

`TheilRegressionResults` subclasses RegressionResults but RegressionResults but setting use_t is/seems broken.

Use_t support in TheilGLS will be added by #5296, but the current  `RegressionResults.__init__` doesn't handle this case. This can cause problems for other subclasses and is not TheilGLS specific.


```
        if cov_type == 'nonrobust':
            self.cov_type = 'nonrobust'
            self.cov_kwds = {
                'description': 'Standard Errors assume that the ' +
                'covariance matrix of the errors is correctly ' +
                'specified.'}
            if use_t is None:
                self.use_t = True  # TODO: class default
        else:
            if cov_kwds is None:
                cov_kwds = {}
            if 'use_t' in cov_kwds:
                # TODO: we want to get rid of 'use_t' in cov_kwds
                use_t_2 = cov_kwds.pop('use_t')
                if use_t is None:
                    use_t = use_t_2
                # TODO: warn or not?
            self.get_robustcov_results(cov_type=cov_type, use_self=True,
                                       use_t=use_t, **cov_kwds)
```

I'm not sure what's going on 
TheilGLS does not use the cov_type in the RegressionResults instantiation which implies `cov_type='nonrobust'` 
```
lfit = TheilRegressionResults(self, params,
                       normalized_cov_params=normalized_cov_params, use_t=use_t)
```
this avoids going through `self.get_robustcov_results` which doesn't support the TheilGLS cov_types.

(maybe some bug still in #5296 that I cannot see yet.)
user: josef-pkt
issue_5328: state: open
title: Determine with what certainty one distribution can be chosen as a better aproximator for a desired value than another distribution.
body: I have the following situation, I am testing two variations of a process against each other. I know that the ideal outcome value is 1. I would like to know which process I can prefer (and with what certainty) based on a sampling of their output values.

(for the same set of inputs, so there's a common random factor, but that bit I think is trivial).

Please see the figure below and find a full minimal working example [here](http://chymera.eu/debug/stats/least_different.zip).

![boxplot](https://user-images.githubusercontent.com/950524/47180696-7380c300-d321-11e8-9df8-408400ad60d7.png)

The way I see it (for a given level of the process variation variable) H0 is that there is a greater or equal effect of the other level.
Since I am unsure how to encode this variable I have opted to instead test the (imho incomplete) H0 that there is no effect for the given level and that it is not significantly different from the other level. This still lacks the constraint that there shouldn't be a greater effect from the other level. I have no idea how that could be encoded.

At any rate I fail long before getting there.

I have tried to subtract one (to make zero the ideal value), and to model the data without an intercept, so I can handle the two levels individually, but given the code:

```python
import pandas as pd
from os import path
import numpy as np
import pandas as pd
import statsmodels.formula.api as smf

volume_path = path.abspath('volumes.csv')
df = pd.read_csv(volume_path)

df = df.loc[df['Processing']!='Unprocessed']
df = df.loc[((df['Processing']=='Legacy') & (df['Template']=='Legacy')) | ((df['Processing']=='Generic') & (df['Template']=='Generic'))]

df['Volume Change Factor'] = df['Volume Change Factor']-1

model=smf.mixedlm('Q("Volume Change Factor") ~ Processing-1', df, groups='uID')

fit = model.fit()
summary = fit.summary()
print(fit)
print(summary)

omnibus_tests = np.eye(len(fit.params))
omnibus_tests = np.delete(omnibus_tests, (2), axis=0)
omnibus_tests[0,1] = -1
print("Generic better than Legacy?")
print(omnibus_tests)
anova = fit.f_test(omnibus_tests)
print(anova)

omnibus_tests = np.eye(len(fit.params))
omnibus_tests = np.delete(omnibus_tests, (2), axis=0)
omnibus_tests[0,1] = -1
omnibus_tests[1,0] = 1
omnibus_tests[1,1] = 0
print("Legacy better than Generic?")
print(omnibus_tests)
anova = fit.f_test(omnibus_tests)
print(anova)
```

I get 

```
/usr/lib64/python3.6/site-packages/statsmodels/regression/mixed_linear_model.py:2092: ConvergenceWarning: The MLE may be on the boundary of the parameter space.
  warnings.warn(msg, ConvergenceWarning)
<statsmodels.regression.mixed_linear_model.MixedLMResultsWrapper object at 0x7f57bc5d5f98>
                 Mixed Linear Model Regression Results
=======================================================================
Model:            MixedLM Dependent Variable: Q("Volume Change Factor")
No. Observations: 136     Method:             REML                     
No. Groups:       68      Scale:              0.0038                   
Min. group size:  2       Likelihood:         160.0737                 
Max. group size:  2       Converged:          Yes                      
Mean group size:  2.0                                                  
------------------------------------------------------------------------
                        Coef.   Std.Err.     z     P>|z|  [0.025  0.975]
------------------------------------------------------------------------
Processing[Generic]      0.036     0.009    4.071  0.000   0.019   0.053
Processing[Legacy]      -0.275     0.009  -31.364  0.000  -0.293  -0.258
uID Var                  0.001     0.013                                
=======================================================================

Generic better than Legacy?
[[ 1. -1.  0.]
 [ 0.  1.  0.]]
<F test: F=array([[578.95292511]]), p=1.1575867403053478e-66, df_denom=134, df_num=2>
Legacy better than Generic?
[[ 1. -1.  0.]
 [ 1.  0.  0.]]
<F test: F=array([[578.95292511]]), p=1.1575867403053893e-66, df_denom=134, df_num=2>
```

Based on this it seems that both levels are almost equal in how well they approximate the value, but that is clearly not the case, as seen in the figure.

Any idea how I can perform this comparison with statsmodels? 
user: TheChymera
issue_5327: state: open
title: ENH: sandwich cov_type of Theil/Ridge versus HC0
body: In a test case for GAM using the motorcycle acceleration data, the difference between TheilGLS sandwich and HC0 in GLMPenalized is large.
TheilGLS agrees with Ve of R mgcv.

One theoretical difference: 
Theil/Ridge sandwich assumes homoscedasticity, the residuals do not enter the sandwich, while 
HC0 has a sandwich that is also robust to heteroscedasticity.

We had once the issue of AC versus HAC, i.e. autocorrelation robustness without heteroscedasticity.
Similarly, I guess we can have a sandwich for likelihood misspecification without being robust to heteroscedasticity, IIRC, that is the assumption in RLM, i.e. M-estimation without likelihood function but with homoscedasticity.

I guess we need an extra cov_type to handle this case.
(simplest "cheating" would be to use resid=1 when computing the HC0 or other sandwich but then add scale to cov_params, i.e. we would have a normalized_cov_params sandwich.)

(GLMPenalized agrees with R mgcv for the "nonrobust" cov_params, except for the missing effective degrees of freedom, edf, correction for scale.)

user: josef-pkt
issue_5315: state: open
title: BUG: PenalizedMixin GLM, hessian scale handling
body: I think there is a bug in scale handling in GLM Gaussian using PenalizedMixin.
related to https://github.com/statsmodels/statsmodels/pull/5296#issuecomment-427966367

GLM fit sets temporarily scale=1, but then hessian and score_obs for the post-estimation results uses the actual scale from results or recomputed based on scale type.

However in the penalized case, we have

`loglike(params, scale=?) + pen_weights * penalty`

when results instance computes cov_params, either hessian or a sandwich, then it uses hessian and score_obs based on actual scale, which causes the relative weights of loglike and penalty to change.

There is currently a problem in the unit test comparing GLM-Gaussian to TheilRidge with a difference in bse for penalized params.
However, when I compute the HC sandwich directly using hessian and score_obs based on scale=1, and the use the scale to multiply this normalized_cov_params, then GLM-Gaussian and TheilRidge cov_params agree.

This would be fine for the Gaussian case where hessian and score are multiplicative in scale, but not for families where scale has an additional effect on score and hessian. (There are one or two of those, but I don't remember which families.)

TheilRidge uses the RegressionResult behavior of multiplying normalized_cov_params by the scale where normalized_cov_params is either the unscaled hessian or the unscaled HC0 sandwich.

possible fix:
multiple penalty by scale, if scale is given in kwds.
In that case we leave the relative weights unchanged for whatever scale is used to evaluate loglike and derivatives.


user: josef-pkt
issue_5311: state: open
title: SUMM: effective degrees of freedom, model complexity
body: This issue is mainly to link to several comments and issues where this shows up.

This shows up in penalized estimation.
But it is related according to Wood on splines in R book also to mixed effects models and Bayesian models.

https://github.com/statsmodels/statsmodels/pull/5296#issuecomment-427673159
seems to be the most general version that I have seen so far
`edf = trace( inv(cov_params ) hessian)`
derivation based on AIC, kullback-leibler approximation, hessian here is the unpenalized observed or expected hessian

I have several comments in penalization issues and PRs (Todo: find them)
`df_resid = nobs - edf` is need for the general case in PenalizedMixin.

It also showed up in the GLM/MLE/generic influence PR with edf based on the generic projection matrix analogue using general influence function `d fittedvalues_i / d y_i`

Generalized degrees of freedom: this is a more general approach based on influence function computed through simulations.
Hauenstein et al also used the asymptotic equivalence of aic and cross-validation to back out the model complexity `p ≈ llf − llf_CV` or using AICc: `p ≈ (llf − llf_cv) * (nobs - 1) / (llf − llf_cv + nobs)`
 Severin Hauenstein, Simon N. Wood & Carsten F. Dormann (2018) Computing AIC for black-box models using generalized degrees of freedom: A comparison with cross-validation, Communications in Statistics - Simulation and Computation, 47:5, 1382-1396, DOI: 10.1080/03610918.2017.1315728 

I'm not sure this is practically very relevant for us, because we would like to use hyperparameter selection based on information criteria so we don't have to do full cross validation.

(just an idea; We could also try to back out edf/df_resid from other statistics, similar to the aic approach of Hauenstein at al.
maybe: use scale estimate to back out ddof or df_resid. e.g. using probability plot to check loc and scale of resid**2. no idea yet whether that makes sense, i.e. whether we get additional information given that we also don't know the scale itself, and we might not want to assume something about third and fourth moments.)







user: josef-pkt
issue_5307: state: open
title: WIP/ENH: New kernel_methods module for new KDE implementation 
body: This PR is based upon #2318, updated to reflect the current master branch.

I've been looking for a package to fit 1D KDEs with support for weighted data and assuming linearity at the boundaries of the data. I came across `ptqt_fit` and then found #2318 which adds this functionality to `statsmodels`.

So far I've done the following so I have a personal branch that I can use:
 - Merge #2318 into the current master branch
 - Refactored the tests to use `pytest` instead of `nose`
 - Removed backwards compatibility for Python/numpy/scipy versions that are older than the current requirements in `setup.py`.
 - Fixed a bug in `statsmodels.kernel_methods.bandwidths.botev`

As implied in #4220, **is there interest in getting this merged in the near(ish) future?** If so, I'm happy to spend some more time improving this. My current todo list is:

- [ ] Use this branch for a few days to get a better idea what the API is like and if it should be modified
- [ ] Fix the flake8 issues
- [ ] Rework the file structure to avoid some (almost) circular dependencies
- [ ] Use `__all__` everywhere to better separate the API from implementation details
- [ ] Add input validation on some properties to avoid large stack traces
- [ ] Make `kde_methods.LogTransform` inherit from `kde_methods.Transform1D` instead  of being a `namedtuple`
- [x] BUG: Allow transforms to switch the definition of upper and lower
- [ ] Investigate the notebooks in the tests directory
user: chrisburr
issue_5302: state: open
title: ENH/REF: fit_constrained, fit_zeros for penalized, PenalizedMixin models
body: It's not clear how to do the exog and parameter transformation to impose constraints in penalized models.

This https://github.com/statsmodels/statsmodels/pull/5296#issuecomment-427074482 and the following comments show the problems we run into with GLMGAM.
It might a "a bit" easier in the plain PenalizedMixin case because it takes exog as given.

For now this is an open question, I'm dropping it for the GAM PR

user: josef-pkt
issue_5299: state: open
title: ENH: State space: dynamic regression class
body: Implements a dynamic regression model of the form

$y_t = x_t \beta_t + \epsilon_t$

x and \beta can be vectors to support multiple exogenous regressors.

Each exogenous regressor in $\beta$ evolves according to a univariate unobserved components model.

At present $\beta$ may have an irregular, local level and linear trend component. The local level and trend can be deterministic or stochastic.

It is intended to extend the modelling options for $\beta$ to include the full suite of terms routinely found in unobserved components models including dummy seasonal terms, harmonic seasonal terms and autoregressive terms.

This is my fist contribution to an open source project so I don't expect the code to be acceptable in it's current state. In particular tests and documentation are required.

I'd appreciate any and all pointers for how to get this to a state where it is a valuable addition to Statsmodels.

Discussion of this feature can be found here:
https://groups.google.com/forum/?hl=en#!topic/pystatsmodels/rEbbZJPxzIo
user: AliHeggie
issue_5294: state: open
title: REF/Review interface to optimizers
body: one refactoring:
allow that methods used as objective functions, score and hessian are optional
e.g. GLM has `gmmobjective` and `gmmobjective_cu`
M estimators like RLM don't have a loglike.

Review:
`base.model.Model.fit` has 
`xopt, retvals, optim_settings = optimizer._fit(f, score, start_params, ..`
how much does this differ now from minimize directly? (our wrapper precedes scipy minimize)
It looks like the main part of our work is after this call when checking Hessian.

One difference to scipy is in our own optimizer methods, e.g. newton and L1 for discrete.

(related: We still have Wald inference In LikelihoodModel instead of a more generic class)
user: josef-pkt
issue_5293: state: open
title: DISC: Generalize hessian/cov_params variants in mlemodel
body: The implementations of _hessian_foo and _cov_params_bar in statespace.mlemodel are especially well done.  Large parts of them, I think, are applicable in more general cases, and could be shared higher up the class hierarchy.  I'm curious if anyone else has thoughts on this.

Some benefits:
 - Sharing code currently implemented in both `mlemodel` and `markov_switching` (e.g `cov_params_opg`)
 - Having `_hessian_complex_step`/`_score_finite_difference` etc readily available makes it very convenient to write tests analytic implementations
 - If cov_params_opg, cov_params_oim, etc go higher up the class hierarchy, then the corresponding `_get_robustcov_results` can be de-duplicated/centralized and have small differences ironed out.

Downsides
 - Work
 - The implementations in mlemodel all have `transformed`, `approx_complex_step`, and `approx_centered` in their signatures.  Making this viable might require making those kwargs ubiquitous (not necessarily a bad thing, just not obiously desirable)

Anyone else have thoughts on this?

@ChadFulton are any of the score/hessian/cov_params variants in mlemodel specific to statespace?
user: jbrockmendel
issue_5288: state: open
title: TST: wald_test_terms, more unit tests for no formula case
body: 
https://github.com/statsmodels/statsmodels/pull/4978#discussion_r222011547

There is only one test case for wald_test_terms which is in base test_generic_methods `TestWaldAnovaOLS.test_noformula`


user: josef-pkt
issue_5284: state: open
title: Informative IV2SLS error message when instrument doesn't contain exog
body: `IV2SLS` currently throws some RuntimeWarnings and produces wrong coefficients and some `nan` standard errors when `instrument` doesn't contain `exog`. This threw me off as I didn't expect to need to add `exog` to `instrument` until I re-read the documentation, so it'd be helpful to have a more informative warning or error message in this case (or to just add `exog` to `instrument` behind the scenes as needed).

Here's my [notebook (with reproducible data)](https://colab.research.google.com/drive/1omFquFq4IYL-Gh9pVDlAHRpoSqjXq6zg) where I hit this.
user: MaxGhenis
issue_5282: state: open
title: REF/Design: separate GLM Binomial Count from Proportion
body: Binomial with n_trials != 1 is a pain because it doesn't follow the standard pattern of the models and results. We have to special case it all over the place.
e.g. #4599 and now in #5255 
and we ran into it several times in weights and scale PRs

two old "design" issues #203 and #2192

We cannot really fundamentally change Binomial because of backwards compatibility, and I don't see a way how we could for example specify model.endog in a backwards and forwards compatible way during a deprecation period.

One possibility would be to replace the current Binomial by two classes one for counts and one for proportion. The problem is mainly in the interface, internally they are all the same and we have already several ways of specifying the model after the addition of var_weights.

BinomialCount

`__init__(endog, exog, n_trials, ...)`
where endog is success count.
This is similar to most stats.proportions functions

BinomialProportion
`__init__(endog, exog, n_trials, ...)`
where endog is success proportion.
this is similar to how Binomial currently behaves after the internal transformation

both would be models that follow standard patterns and convention, where in the first everything is specified in counts and in the second everything is specified in proportions.

The two models are equivalent to (Binomial is shortcut for GLM(... family=Binomial,...))
`Binomial(proportion, exog, var_weights=1/n_trials...)`

additionally `n_trials` would be equivalent to a generalized `exposure` variable (which is currently restricted to log link)

The simpler alternative would be to keep `endog` as counts (*) and use `wendog` for proportion instead of changing endog. (*) or proportions if user uses var_weights instead of n_trials.
For recreating the model, i.e. #4599, it would still be better to have n_trials as a keyword instead of the 2-D endog in the `GLM.__init__`

It might be possible to make this alternative work with deprecation, i.e. condition on whether endog is 2-D and whether n_trials keyword is not None. Then we could deprecate 2-D endog.



user: josef-pkt
issue_5281: state: open
title: REF/BUG?: GLM scaletype is a model attribute (but settable in fit)
body: AFAICS, not a BUG with wrong `results`

llf and llnull use the results.scale and are therefore independent of the model.scaletype (after results.scale has been set in GLM.fit and `Results.__init__`)

summary shows the scale but not the scaletype

search for "scaletype" does not show any usage of it in GLMResults.

So, it looks like the GLM.scaletype attribute is only used to set the default in and during fit.

found while investigating scale in GLM NegativeBinomial for #5277






user: josef-pkt
issue_5266: state: open
title: memory error with Poisson regression on sparse fixed effects dummy variables
body: I need to perform a Poisson regression on data with around 80,000 dummy variables used to capture two different fixed effects parameters in the model, but get a memory error when trying to do so.

The exog dataframe has shape (972000, 81618).

```
---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
<ipython-input-53-6e1d4ee9e698> in <module>()
      1 # model
----> 2 mod = Poisson(regression_df['murder'], regression_without_responses_df)

~/anaconda3/lib/python3.7/site-packages/statsmodels/discrete/discrete_model.py in __init__(self, endog, exog, offset, exposure, missing, **kwargs)
    729         super(CountModel, self).__init__(endog, exog, missing=missing,
    730                                          offset=offset,
--> 731                                          exposure=exposure, **kwargs)
    732         if exposure is not None:
    733             self.exposure = np.log(self.exposure)

~/anaconda3/lib/python3.7/site-packages/statsmodels/discrete/discrete_model.py in __init__(self, endog, exog, **kwargs)
    169     """
    170     def __init__(self, endog, exog, **kwargs):
--> 171         super(DiscreteModel, self).__init__(endog, exog, **kwargs)
    172         self.raise_on_perfect_prediction = True
    173 

~/anaconda3/lib/python3.7/site-packages/statsmodels/base/model.py in __init__(self, endog, exog, **kwargs)
    210 
    211     def __init__(self, endog, exog=None, **kwargs):
--> 212         super(LikelihoodModel, self).__init__(endog, exog, **kwargs)
    213         self.initialize()
    214 

~/anaconda3/lib/python3.7/site-packages/statsmodels/base/model.py in __init__(self, endog, exog, **kwargs)
     62         hasconst = kwargs.pop('hasconst', None)
     63         self.data = self._handle_data(endog, exog, missing, hasconst,
---> 64                                       **kwargs)
     65         self.k_constant = self.data.k_constant
     66         self.exog = self.data.exog

~/anaconda3/lib/python3.7/site-packages/statsmodels/base/model.py in _handle_data(self, endog, exog, missing, hasconst, **kwargs)
     85 
     86     def _handle_data(self, endog, exog, missing, hasconst, **kwargs):
---> 87         data = handle_data(endog, exog, missing, hasconst, **kwargs)
     88         # kwargs arrays could have changed, easier to just attach here
     89         for key in kwargs:

~/anaconda3/lib/python3.7/site-packages/statsmodels/base/data.py in handle_data(endog, exog, missing, hasconst, **kwargs)
    631     klass = handle_data_class_factory(endog, exog)
    632     return klass(endog, exog=exog, missing=missing, hasconst=hasconst,
--> 633                  **kwargs)

~/anaconda3/lib/python3.7/site-packages/statsmodels/base/data.py in __init__(self, endog, exog, missing, hasconst, **kwargs)
     74             self.orig_endog = endog
     75             self.orig_exog = exog
---> 76             self.endog, self.exog = self._convert_endog_exog(endog, exog)
     77 
     78         # this has side-effects, attaches k_constant and const_idx

~/anaconda3/lib/python3.7/site-packages/statsmodels/base/data.py in _convert_endog_exog(self, endog, exog)
    470         #TODO: remove this when we handle dtype systematically
    471         endog = np.asarray(endog)
--> 472         exog = exog if exog is None else np.asarray(exog)
    473         if endog.dtype == object or exog is not None and exog.dtype == object:
    474             raise ValueError("Pandas data cast to numpy dtype of object. "

~/anaconda3/lib/python3.7/site-packages/numpy/core/numeric.py in asarray(a, dtype, order)
    499 
    500     """
--> 501     return array(a, dtype, copy=False, order=order)
    502 
    503 

~/anaconda3/lib/python3.7/site-packages/pandas/core/generic.py in __array__(self, dtype)
   1604 
   1605     def __array__(self, dtype=None):
-> 1606         return com._values_from_object(self)
   1607 
   1608     def __array_wrap__(self, result, context=None):

pandas/_libs/lib.pyx in pandas._libs.lib.values_from_object()

~/anaconda3/lib/python3.7/site-packages/pandas/core/generic.py in get_values(self)
   4686                [nan,  3.]])
   4687         """
-> 4688         return self.values
   4689 
   4690     def get_dtype_counts(self):

~/anaconda3/lib/python3.7/site-packages/pandas/core/generic.py in values(self)
   4631         """
   4632         self._consolidate_inplace()
-> 4633         return self._data.as_array(transpose=self._AXIS_REVERSED)
   4634 
   4635     @property

~/anaconda3/lib/python3.7/site-packages/pandas/core/internals.py in as_array(self, transpose, items)
   3947             arr = mgr.blocks[0].get_values()
   3948         else:
-> 3949             arr = mgr._interleave()
   3950 
   3951         return arr.transpose() if transpose else arr

~/anaconda3/lib/python3.7/site-packages/pandas/core/internals.py in _interleave(self)
   3958         dtype = _interleaved_dtype(self.blocks)
   3959 
-> 3960         result = np.empty(self.shape, dtype=dtype)
   3961 
   3962         if result.shape[0] == 0:

MemoryError: 
```
user: databasedav
issue_5256: state: open
title: REF/Review: holtwinters Results pattern, inconsistent ?
body: I am trying to add cached properties `fittedvalues` and `resid` generically to base.model.Results in 5255

holtwinters uses attributes instead of cached properties
I get one test failure after adding the generic methods in all environments
https://travis-ci.org/statsmodels/statsmodels/jobs/432552696

```
    def test_direct_holt_add():
        mod = SimpleExpSmoothing(housing_data)
        res = mod.fit()
        x = np.squeeze(np.asarray(mod.endog))
        alpha = res.params['smoothing_level']
        l, b, f, err, xhat = _simple_dbl_exp_smoother(x, alpha, beta=0.0,
                                                      l0=res.params['initial_level'], b0=0.0,
                                                      nforecast=5)
    
        assert_allclose(l, res.level)
        assert_allclose(f, res.level.iloc[-1] * np.ones(5))
        assert_allclose(f, res.forecast(5))
    
        mod = ExponentialSmoothing(housing_data, trend='add')
        res = mod.fit()
        x = np.squeeze(np.asarray(mod.endog))
        alpha = res.params['smoothing_level']
        beta = res.params['smoothing_slope']
        l, b, f, err, xhat = _simple_dbl_exp_smoother(x, alpha, beta=beta,
                                                      l0=res.params['initial_level'],
                                                      b0=res.params['initial_slope'], nforecast=5)
    
>       assert_allclose(xhat, res.fittedvalues)
E       AssertionError: 
E       Not equal to tolerance rtol=1e-07, atol=0
E       
E       (shapes (714,), (1,) mismatch)
E        x: array([ 96.199944,  96.221455,  99.021455, 127.721455, 150.821455,
E              152.521455, 147.821455, 148.121455, 138.221455, 136.421455,
E              120.021455, 104.721455,  95.621455,  86.021455,  90.721455,...
E        y: array([111.521455])
```

I have no idea yet what's going on.
user: josef-pkt
issue_5255: state: open
title: ENH: add base Results.fittedvalues and resid
body: 
see #37 especially https://github.com/statsmodels/statsmodels/issues/37#issuecomment-424019626

(unrelated aside: we still have 9 open issues that predate github move)
user: josef-pkt
issue_5245: state: open
title: FAQ-D: pattern for matplotlib plotting unit tests
body: The test module needs to import pytest
`import pytest`

If pyplot, `plt`, is needed in the test module, then it should be imported in a `try ... except` clause because matplotlib is and optional dependency

```
try:
    import matplotlib.pyplot as plt
except ImportError:
    pass
```

A plotting unit test will be automatically skipped and figures will be closed using the following pytest fixtures provided by the `pytest.mark.matplotlib` decorator and the `close_figures argument

```
@pytest.mark.matplotlib
def test_plot_xxx(close_figures):
```

changed in #4913 and #5165 

user: josef-pkt
issue_5216: state: open
title: Priorities: popularity of packages in Stata and R
body: just an informally collected list based on download statistic of user packages (i.e. I skimmed the list and took the ones that look relevant)

most top packages are for data handling and infrastructure, I only list packages with econometrics/statistics

## Stata (ssc downloads in June 2018, top 50 `ssc whatshot, n(50)` in Stata)

- ivreg2
- psmatch2
- reghdfe
- ranktest:  Kleibergen version
- xttest3: wald test for panel data ?
- xtabond2
- diff:  Difference-in-Difference estimators
- margeff: margeff for categorical and limdep data ?
- synth: synthetic control causal influence
- xtivreg2

## R (more difficult to collect because there are almost no statistics packages in top 100)

- lme4
- car
- mvtnorm  (as dependency
- quantreg Koenker
- kernlab
- robustbase/robust
- sandwich
- MatrixModels (using sparse matrices ?)
- lmtest
- survival
- pbkrtest: Methods for Mixed Model Comparison
- pls: partial least squares and PCA
- MASS
- ModelMetrics (eval_measures ?)
- psych
- quantmod: quantitative finance
(that stops at 160 of total, direct plus indirect, downloads

what I didn't see: mgcv, panel data, and proportions and similar for clinical trials and epidemiology






user: josef-pkt
issue_5189: state: open
title: Errors while implementing Toda-Yamamoto Granger Causality Test with Statsmodels
body: Hi,

I'm trying to implementing the Toda-Yamamoto Granger Causality procedure on Python with Statsmodels. The process is boiled down to these steps, which was explained clearly [here:](https://davegiles.blogspot.com/2011/04/testing-for-granger-causality.html)

> Now, here are the basic steps for the T-Y procedure:
Test each of the time-series to determine their order of integration. Ideally, this should involve using a test (such as the ADF test) for which the null hypothesis is non-stationarity; as well as a test (such as the KPSS test) for which the null is stationarity. It's good to have a cross-check.
Let the maximum order of integration for the group of time-series be m. So, if there are two time-series and one is found to be I(1) and the other is I(2), then m = 2. If one is I(0) and the other is I(1), then m = 1, etc.
Set up a VAR model in the levels of the data, regardless of the orders of integration of the various time-series. Most importantly, you must not difference the data, no matter what you found at Step 1.
Determine the appropriate maximum lag length for the variables in the VAR, say p, using the usual methods. Specifically, base the choice of p on the usual information criteria, such as AIC, SIC.
Make sure that the VAR is well-specified. For example, ensure that there is no serial correlation in the residuals. If need be, increase p until any autocorrelation issues are resolved.
If two or more of the time-series have the same order of integration, at Step 1, then test to see if they are cointegrated, preferably using Johansen's methodology (based on your VAR) for a reliable result.
No matter what you conclude about cointegration at Step 6, this is not going to affect what follows. It just provides a possible cross-check on the validity of your results at the very end of the analysis.
Now take the preferred VAR model and add in m additional lags of each of the variables into each of the equations.
Test for Granger non-causality as follows. For expository purposes, suppose that the VAR has two equations, one for X and one for Y. Test the hypothesis that the coefficients of (only) the first p lagged values of X are zero in the Y equation, using a standard Wald test. Then do the same thing for the coefficients of the lagged values of Y in the X equation.
It's essential that you don't include the coefficients for the 'extra' m lags when you perform the Wald tests. They are there just to fix up the asymptotics.
The Wald test statistics will be asymptotically chi-square distributed with p d.o.f., under the null.
Rejection of the null implies a rejection of Granger non-causality. That is, a rejection supports the presence of Granger causality.
Finally, look back at what you concluded in Step 6 about cointegration

I've written these down to the codes bellow to test the Granger causality of different interest rate curves on stock but I am not sure if they follow the procedure correctly. And with some interest rates it fail to fit in VARMAX and throws either on of these errors: 'Singular forecast error covariance matrix encountered at period' or '-th leading minor of the array is not positive definite'. I'm pretty new with statsmodels as well as time series analysis so any advices on how to overcome these would be much appreciated!

```
[`import pandas as pd
import datetime as dt
import matplotlib.pyplot as plt 
import numpy as np
import quandl
import statsmodels.tsa.api as smt
import statsmodels.api as sm
import statsmodels.tsa.stattools as ts
import statsmodels.tsa
from arch.unitroot import KPSS, ADF
import seaborn as sns
from statsmodels.tsa.api import VAR, DynamicVAR

def Stationarity (X, cutoff = 0.05):
    pvalueADF = ADF(X).pvalue
    pvalueKPSS = KPSS(X).pvalue
    cvalueADF = ADF(X).critical_values[str("{0:.0%}".format(cutoff))]
    statADF = ADF(X).stat
    cvalueKPSS = KPSS(X).critical_values[str("{0:.0%}".format(cutoff))]
    statKPSS = KPSS(X).stat
    
    if pvalueADF < cutoff and pvalueKPSS > cutoff and statADF < cvalueADF and statKPSS < cvalueKPSS:
        print ('p-value AdFuller = ' + str(pvalueADF) + ' p-value KPSS = ' + str(pvalueKPSS) + ' Stationary')
        return True
    else:
        print('p-value = ' + str(pvalueADF) + ' p-value KPSS = ' + str(pvalueKPSS) + ' Non-Stationary')
        return False

def order_of_integration(series, cutoff=0.05):
    if Stationarity(series, cutoff=cutoff) == True:
        return 0
    else:
        i = 1
        s = series.diff(periods=i).dropna()
        while Stationarity(s, cutoff=cutoff) == False:
            s = s.diff(periods=i).dropna()
            i += 1
        return i
def Op_Lag(df):
    inn_model = VAR(df)
    aic = inn_model.select_order(maxlags=10).aic
    hqic = inn_model.select_order(maxlags=10).hqic
    inn_lag = max(aic, hqic)
    op_lag = inn_lag
    model = sm.tsa.VARMAX(df, order = (op_lag,0), trend = 'c', enforce_stationarity = False)
    result = model.fit()
    resi = pd.DataFrame(result.test_serial_correlation('ljungbox')[0])
    op_lag_pvalue = resi.iloc[1,inn_lag-1]
    
    while op_lag_pvalue<0.95:
        op_lag += 1
        model = sm.tsa.VARMAX(df, order = (op_lag,0), trend = 'c', enforce_stationarity = False)
        result = model.fit(maxiter = 200, disp = False)
        resi = pd.DataFrame(result.test_serial_correlation('ljungbox')[0])
        op_lag_pvalue = resi.iloc[1,op_lag-1]
        
    return op_lag

def wald_test(interest_rate):
    df = pd.read_excel('data.xlsx', 0, header = 0,  index_col = 0, usecols = [0,1], parse_dates = True, convert_float = True, na_values = 'NA')
    df.columns = ['REITS']
    rate = quandl.get("USTREASURY/YIELD", end_date="2018-08-17", start_date='1990-01-02', collapse='daily')
    rate.sort_index(axis = 0,inplace=True, ascending = False)
    rate.columns = rate.columns.str.replace(' ','')
    df = df.join(rate[interest_rate],how ='inner')
    df.dropna(inplace=True)
    df.index = pd.DatetimeIndex(df.index).date
    start = dt.date(2009,7,1)
    df = df.loc[df.index > start]
    op_lag = Op_Lag(df)
    max_ooi =[]
    for col in df.columns:
        ooi = order_of_integration(df[col])
        max_ooi = np.append(max_ooi, ooi)
    max_ooi = int(np.amax(max_ooi))
    lag = op_lag + max_ooi
    model = sm.tsa.VARMAX(df, order = (lag,0), trend = 'c', enforce_stationarity = False)
    result = model.fit(maxiter = 200, disp = False)
    string = interest_rate + '.REITS'
    exog = [i for i in model.param_names if string in i][:-1]
    hypo = ''.join(['('+ i + '=0),' for i in exog[:-1]]) + '('+ exog[-max_ooi] + ')'
    wald = result.wald_test(hypo)
    return(wald.statistic, wald.pvalue, wald.df_denom)

wald_test('1MO')`
```

Cheers!

Vi
user: vanvivoi
issue_5186: state: open
title: FAQ-D/Release: creating tags for versioneer, create v0.10.0.dev tag
body: Need a dev tag in statsmodels/statsmodels to get correct version numbers now that we switched to versioneer.

Ideally 

```
git tag -a v0.10.0.dev0 ff0d659fdcaf746e0472f2c4b4c348c8a11b92d2
```
which corresponds to the same commit where the original setup was modified to have version 0.10.0dev,

https://github.com/statsmodels/statsmodels/commit/ff0d659fdcaf746e0472f2c4b4c348c8a11b92d2#diff-2eeaed663bd0d25b7e608891384b7298

user: bashtage
issue_5185: state: open
title: SUMM/ENH: extreme value distributions, pareto
body: Another ancient theme with nothing public in statsmodels.

a brief github search with python repos for extreme value
https://github.com/wafo-project/pywafo package by Per A. Brodtkorb but GPL
https://github.com/kikocorreoso/scikit-extremes MIT alpha not released yet, GEV
https://github.com/AlexLSmith/extreme-value-analysis MIT, some interesting parts
https://github.com/kniehaus/EVT (no license, looks abandoned)

there are several python packages on pareto/power distribution

Almost everything in R is GPL
http://www.ral.ucar.edu/~ericg/softextreme.php contains a list of R packages

there are a few user functions in Stata that I never looked at
https://ideas.repec.org/c/boc/bocode/s457953.html extreme
https://davegiles.blogspot.com/2015/01/extreme-value-modelling-in-stata.html (blog)
https://ideas.repec.org/c/boc/bocode/s456832.html paretofit
https://ideas.repec.org/c/boc/bocode/s456892.html gevfit

statsmodels:
statsmodels.miscmodels.tests.test_generic_mle has MyPareto as a test case, no externally verified numbers
sandbox distribution has genpareto and try_pot for some experiments and draft functions (from 2010)
sandbox distribution copula also has extreme value copulas
old times: 
  maximum spacing estimator in old scipy issue,  (Per Brodtkorb)
  discussion about how well MLE works with good choice of start_params (Per Brodtkorb and I )
  lmoments python wrapper of Fortran code in hydroclimpy

this is mostly univariate distribution estimation without explanatory variables
R's fExtremes might provide a useful table of content for useful helper functions

I never looked at regression version of those, i.e. conditional on explanatory variables as in VGAM

(motivation: I was trying variation of miscmodels MyPareto for #5143 )





user: josef-pkt
issue_5174: state: open
title: Survey analysis methods
body: This is the survey methods project from GSOC 2017

GSOC PR is #3762
user: kshedden
issue_5169: state: open
title: ENH/BUG: bootstrap and recreating models
body: This is a specific point of #4679 

GenericLikelihoodModel is the only built-in bootstrap method (besides some confint bootstrap in tsa).
There are currently problems, incomplete and outdated support for recreating a model for the bootstrap iteration. 

specific case: I don't know how to get bootstrap for the MyPareto unit test cases to work 
https://github.com/statsmodels/statsmodels/pull/5143#issuecomment-419987909
**edit** I had printed out from the wrong results instance, everything in bootstrap with constrained model looks fine, although a bit cumbersome to use.

todo:

- need unit tests that check what attributes, keywords are used in the recreated bootstrap models
- refactor `_init_keys`/`cloneattr`
- check how constrainted estimation, `transform_params` is supposed to work with recreating models
- provide a prototype `miscmodel`, so we have it documented and have a unit test case for it.

#5143 fixes an issue when exog=None and a problem with `cloneattr`, however that's incomplete. It also has an example without constraints that seems to work.

user: josef-pkt
issue_5151: state: open
title: ENH: Add automated model selection for TSA models
body: This PR is a follow-up to #4621 (automatic methods GSOC), that so far does two things:

- Rewrites `exponentialsmoothing.py` and `sarimax.py` to standardize the interface, provide more comprehensive results, and add a variety of ways to specify the model search space.
- Adds `structural.py` that adds support for `UnobservedComponents` models.

A notebook showing the basics of the proposed interface is at https://gist.github.com/ChadFulton/eb3f926e8925072196efa3c75e7987f5

Of course this is very early stages. Two big TODOs are:

- Unit tests
- Try to abstract out some of the functionality to avoid the boilerplate in each file.

(I will be out of touch for the next few days so I won't be responding, but any feedback would be greatly appreciated!)
user: ChadFulton
issue_5125: state: open
title: gof.gof_binning_discrete "# copy/paste, remove code duplication when it works"
body: Does it work?
user: jbrockmendel
issue_5113: state: open
title: ENH: general wald tests, granger cauality test for non-stationary endog
body: https://stackoverflow.com/questions/52189915/toda-yamamoto-granger-causality-test-in-python/52191716#52191716

https://davegiles.blogspot.com/2011/04/testing-for-granger-causality.html

https://doi.org/10.1080/07474939608800362

If some variables are non-stationary, then standard granger causality Wald test has not the standard chi2 distribution. If additional lags (corresponding to the order of integration) are added but not included in the Wald test, then the asymptotic distribution of the Wald test is standard chi2.

AFAICS from browsing a bit the VECM code, there is an additional lag added, `p = k_ar + 1` (but I don't see whether it is included in the wald test hypothesis)
user: josef-pkt
issue_5111: state: open
title: README_l1.txt references non-existent files
body: e.g. l1_demo/sklearn_compare.py

If there is anything worth salvaging from this file, it should probably just be rolled into README.rst.
user: jbrockmendel
issue_5098: state: open
title: Use of Breusch-Godfrey test with any residue series and exog as parameter
body: I'm trying to use the Breusch-Godfrey test with a residue series that I generated with another model, but it seems that the function only accepts a Results instance. From the code, it seems that the only part of the Residue instance that is used is to extract the exogenous series. Is it possible to change the test function so it would accept any series, maybe with a parameter for the exogenous part?
user: ispmarin
issue_5094: state: open
title: Inclusion of non-negative continuous probability distributions (e.g. Weibull, Pareto) for regression?
body: Hi,

Apologies if this is not the right place to ask/request this functionality. I have been exploring the code of `statsmodels` to learn whether this (fantastic!) piece of software includes **continuous probability distributions for multivariate regression** (e.g. Weibull, Pareto, exponential), in a similar fashion as the [discrete case](https://github.com/statsmodels/statsmodels/issues/4952#issuecomment-414786975) (e.g Poisson, Neg. Binomial and its zero-inflated versions). I am modelling daily values of precipitation data, and this type of "power law" models (continuous or discrete) are really handy for these tasks, because they handle decently skewed distributions. 

For what I understood, with the current version of `statsmodels`, it is possible to fit a multivariate Gamma distribution as a combination of a GLM and a link function from the gamma family (please, correct me if I use the wrong terms):
```
def fitting_gamma(xtrain, ytrain):
    # Shape of xtr/ytr: (800 rows, 8 columns) / (800 rows, 1 column)
    gam_mod = sm.GLM(ytrain, xtrain, family=sm.families.Gamma(link=sm.genmod.families.links.identity), disp=1).fit(method="nm", maxiter=100)
    gam_pred = gam_mod.predict(xtrain)
    return [gam_mod, gam_pred]

gam_mod, gam_pred  = fitting_gamma(xtrain, ytrain)
prediction = gam_mod.predict(xtest)
# more stuff
```
Thus, I can train my gamma model with part of my multivariate data , and then test its performance with unseen data to calculate few statistical metrics. 

In Python, there are some [great examples](https://stackoverflow.com/questions/6620471/fitting-empirical-distribution-to-theoretical-ones-with-scipy-python) to learn how to fit probability distributions to univariate data, mostly relying on `scipy.stats`. However, I could not find similar software packages doing the same for multivariate data. Actually with `statsmodels`, I think the only continuous distributions that I found is gamma (perhaps I overlooked the rest).

Is there any way of doing the same multivariate fitting I did with Gamma but with a Weibull/Pareto distribution? Are there any plans to include these multivariate regression models in `statsmodels`?

Thanks a lot for your work!
user: Irene-GM
issue_5092: state: open
title: DEPR: Deprecations from prior versions
body: There is a lot of code marked as deprecated or to be removed.  This issue is intended to keep track of them and improve the odds those deprecations are followed up on.

Short-run, ~four things are actionable:

a) things that were marked as to-be-removed in version 0.8 or 0.9 should _actually_ be removed in version 0.10.
b) things that were marked as removed "in the future" should be updated to be more specific.
c) decide which if any of these messages are fine being kept indefinitely, e.g. perhaps the "nobs is deprecated in favor of lags"

----
Related:
- [ ] #3433 REF: Removal of deprecated objects for 0.9
- [ ] #3434 DESIGN: AR, VAR, SVAR signature deprecations in 0.9
- [ ] #3462 REF deprecate lower case link classes
- [x] #1063 Remove version compatibility code

----

- [x] `stats._lilliefors.lillifors` "will be removed in 0.9"
- [x] `stats.diagnostic.acorr_breush_godfrey` "will be removed in 0.9"
- [x] `stats.diagnostic.het_breushpagan` "will be removed in 0.9"
- [x] `stats.diagnostic.StatTestMC` "delete when testing is finished", "this has been copied to tools/mctools.py, with improvements"
- [x] `stats.multicomp.fdrcorrection_bak` "Old version, to be deleted"
- [x] `stats.multicomp.maxzero` "Not used anymore"
- [x] `stats.multicomp.maxzerodown` "Not used anymore"
- [x] `stats.multicomp.rejectionline` "Not used anymore"
- [ ] `stats.runs.symmetry_bowker` "Deprecated, use stats.TableSymmetry instead" <--at least 3 years like this
- [ ] `stats.runs.mcnemar`  "Deprecated, use stats.TableSymmetry instead" <--at least 3 years like this
- [ ] `stats.runs.cochrans_q` "Deprecated, use stats.TableSymmetry instead" <--at least 3 years like this
- [ ] `GLMResults.resid_anscombe` "Anscombe residuals currently unscaled. In a future release they will be scaled"
- [ ] `genmod.families.family.Family` "Calling Family(..) with a link class as argument is deprecated"
- [x] `base._penalties._check_wts` "`wts` method is deprecated. Use `weights` instead"
- [ ] `base._penalties.Penalty.grad` "grad method is deprecated. Use `deriv` instead"
- [ ] `base._penalties.CovariancePenalty.grad` "grad method is deprecated. Use `deriv` instead"
- [x] `datasets.utils.as_numpy_dataset`: "load will return datasets containing pandas DataFrames and Series in the future"
- [x] `statespace.tools.prepare_trend_spec` "Argument option trend='nc' is deprecated.  Please use option trend='n'."
- [x] `arima_process.arma_acf` "nobs is deprecated in favor of lags"
- [x] `arima_process.arma_pacf` "nobs is deprecated in favor of lags"
- [x] `arima_process.arma_impulse_response` "nobs is deprecated in favor of lags"
- [x] `arima_process.arma2ma` "nobs is deprecated in favor of lags"
- [x] `arima_process.arma2ar` "nobs is deprecated in favor of lags"
- [x] `vector_ar.dynamic.DynamicVAR` "is deprecated and will be removed in a future version"
- [x] `vector_ar.dynamic.DynamicPanelVAR` "is deprecated and will be removed in a future version"
- [x] `vector_ar.util.get_logdet` "will be removed in 0.8.0"
- [ ] `discrete_margins.Margins` "Mostly a do nothing class" superceded by `DiscreteMargins` below

user: jbrockmendel
issue_5087: state: open
title: Review sandbox/multilinear.py
body: A forgotten module, no unit tests

I just found it by chance and didn't look carefully enough what should be done with it.
merged in #854 based on #683
user: josef-pkt
issue_5082: state: open
title: TST: Codecov False-Positives Caused by Noise in mixed_linear_model tests?
body: codecov report for #5080 (which touches two lines in the sandbox, should have zero effect on coverage):

https://codecov.io/gh/statsmodels/statsmodels/compare/f324895b1ef9a15006649928e14263bb56cf0a28...d32903a1d75dd5f2a7f63929662f43f9984c79a3/changes

It is registering a decrease in coverage because of some lines not hit in `mixed_linear_model`.  I speculate that this is related to the flaky test described in #4414.  If correct, then this is _good_ news, since it is something we can fix and not a problem in codecov.

cc: @bashtage @kshedden 
user: jbrockmendel
issue_5077: state: open
title: Typesettable output for F-statistics
body: I generally integrate my analysis (increasingly much of it using statsmodels) directly into the code which generates my document (so that results get autoupdated at recompile time, c.f. [RepSeP](https://github.com/TheChymera/RepSeP)).

I have a compatibility function which allows me to get TeX-syntax output based on statsmodels ANOVA F-statistic reults --- or just a representation which looks better in the terminal.
Currently this is [reposited in my general-purpose small animal fMRI anlysis package](https://github.com/IBT-FMI/SAMRI/blob/2857d27e36a6b48e0b737a25ceef9553cb5f2565/samri/typesetting.py#L33), but this is really inapropriate.

It's also too small of a feature to package separately.

I'm curious if I could somehow (even if it necessitates some modification) get this included in statsmodles. If so, where?
user: TheChymera
issue_5063: state: open
title: ENH: Quantile based estimator
body: I just found this mentioned in #1119

I don't remember anything right now, but it might be useful for #3981 #2183

not rebased yet, so awfully outdated (pre-py3)
user: josef-pkt
issue_5062: state: open
title: imports in both iolib.__init__ and iolib.api
body: The usual pattern puts these imports only in `api`.  Why are they also in `__init__`?
user: jbrockmendel
issue_5054: state: open
title: ENH: Add ARFIMA model
body: Is ARFIMA model available in the package or at least somewhere in python?
user: hlbkin
issue_5046: state: open
title: get_tukeyQcrit vs get_tukeyQcrit2?
body: in sandbox.stats.multicomp (which unfortunately is used outside the sandbox despite having abysmal QA), these functions have identical docstrings.  What is the meaningful difference?  Why is the first used in some contexts and the latter in others?
user: jbrockmendel
issue_5037: state: open
title: examples files un-runnable
body: Maybe run these in the CI?  We shouldn't be advertising broken examples.

```
$ python examples/run_all.py
Are you sure you want to run all of the examples?
          This is done mainly to check that they are up to date.
          (y/n) >>> y


Executing example file example_glsar.py
---------------------------------------
**************************************
ERROR in example file example_glsar.py
**************************************
Traceback (most recent call last):
  File "examples/run_all.py", line 43, in <module>
    exec(open(run_all_f).read())
IOError: [Errno 2] No such file or directory: 'example_glsar.py'
```
user: jbrockmendel
issue_5034: state: open
title: Custom Patsy Contrasts do not work in MixedLM
body: Patsy v. 0.5.0
Statsmodels v. 0.9.0

Minimal working example:

```
import pandas as pd
import numpy as np
import statsmodels.formula.api as smf
from patsy.contrasts import ContrastMatrix

# Custom contrasts
class SimpleCoding(object):
    def __init__(self):
        pass
    
    def _simple_contrast(self, levels):
        nlevels = len(levels)
        contr = -1./nlevels * np.ones((nlevels, nlevels-1))
        contr[1:][np.diag_indices(nlevels-1)] = (nlevels-1.)/nlevels
        return contr

    def code_with_intercept(self, levels):
        contrast = np.column_stack((np.ones(len(levels)),
                                    self._simple_contrast(levels)))
        return ContrastMatrix(contrast, _name_levels("Simp.", levels))
    
    def code_without_intercept(self, levels):
        contrast = self._simple_contrast(levels)
        return ContrastMatrix(contrast, _name_levels("Simp.", levels)[1:])
    
# Mock dataframe
y = np.random.normal(0, 1, size=1000)
x = np.tile(["A", "B"], 500)
g = np.tile(np.arange(0, 10), 100)
df = pd.DataFrame({"X":x, "Y":y, "G":g})

smf.ols("Y ~ C(X, SimpleCoding)", data=df).fit().summary() # Will pass
smf.mixedlm("Y ~ C(X, SimpleCoding)", data=df, groups="G").fit().summary() # Will fail: PatsyError: Error evaluating factor: NameError: name 'SimpleCoding' is not defined: Y ~ C(X, SimpleCoding)
```

It seems that MixedLM does not have access to the full namespace, but I'm unsure how to fix it.
user: QuentinAndre
issue_5023: state: open
title: “Copied from PR #2380 remove when merged”
body: In LilkelohoodModel._fit_colinear

I suggest we follow instructions
user: jbrockmendel
issue_5019: state: open
title: _diagnostic_count functions starting with "test_" should renamed
user: jbrockmendel
issue_4992: state: open
title: tools.tools namespace is counter-intuitive
body: `pinv_extended` is in `tools.tools`, but `pinv` and `pinv2` are in `tools.linalg`.  There should be a single obvious place for users/contributors to look for linalg-related functions.
user: jbrockmendel
issue_4969: state: open
title: ENH: Add inf/nan protection to regression models
body: Add an inf/nan check for regression models

closes #3468
closes #1642
user: bashtage
issue_4966: state: open
title: Audit Test Suite
body: There is a lot of heterogeneity in the quality of the tests.  An audit-like process should attempt to identify/address (in no particular order):

- What parts of the code have only smoke tests?
- What parts of the tests are not getting run?
  - Commented-out
  - mangled names
  - miscellany like`def junk` in `discrete.tests.test_constrained`
  - stranded in `__main__` sections
  - incorrectly located in `__main__` sections of non-test files.
- Are there any xfailed tests that have been fixed? Or can be marked with `strict=True`?
- A lot of effort went into creating the results files to compare against (props to our `sm` forebearers).  Are these reproducible?  (and if not, can they be made reproducible?)
- In some cases results were subsequently "hand-edited" for various reasons.  Are these well-documented?
- Some "example" files have snuck into test directories; where should they go?
- Can test runtime be significantly reduced by efficient use of `pytest.fixture`s?
- Are there combinations of parameters that can be tested more thoroughly using `pytest.mark.parametrize`?
- Can the tests be otherwise be made less verbose/clearer?
- Other modernizations that should be made?  e.g. IIUC `assert_almost_equal` is discouraged and `assert_allclose` should be used instead.
- Are there places where `assert_allclose` tolerances can be reduced?
- grep turns up 5 occurrences of "FIXME" and 218 occurrences of "TODO" in test directories
- There are a whole bunch of occurrences of things like:

```
        cls.res1 = mymodel.fit(method="lbfgs", disp=0, maxiter=50000,
                #m=12, pgtol=1e-7, factr=1e3, # 5 failures
                #m=20, pgtol=1e-8, factr=1e2, # 3 failures
                #m=30, pgtol=1e-9, factr=1e1, # 1 failure
                m=40, pgtol=1e-10, factr=5e0,
                loglike_and_score=mymodel.loglike_and_score)
```
```
        get_robustcov_results(cls.res1._results, 'cluster',
                                                  groups=group,
                                                  use_correction=True,
                                                  df_correction=True,  #TODO has no effect
                                                  use_t=False, #True,
                                                  use_self=True)
```
```
        model = sm.Logit(y_bin, x)  #, exposure=np.ones(nobs), offset=np.zeros(nobs)) #bug with default
```
The TODO is reasonably clear and helpful, but the commented-out `True` and the commented-out fit parameters are not helpful in their current form.

-----
This is a pretty huge task.  A few steps in this direction: #4941, #4936, #4932, #4907, #4875, #4863, #4506, #4488, #4305

user: jbrockmendel
issue_4964: state: open
title: Deprecate multiple-return signatures
body: e.g. tsa.stattools._autolag can return 2 values or 3 depending on inputs.  After a deprecation cycle this should be changed to a single return signature.

I'll update this issue with a list of affected functions.

- [ ] `tsa.stattools._autolag`
- [ ] `tsa.stattools.adfuller`
- [ ] `tsa.stattools.acf`
- [ ] `tsa.stattools.pacf`
- [ ] `tsa.stattools.kpss`

user: jbrockmendel
issue_4962: state: open
title: FAQ-D: running doctest on docstring of a function
body: the following is copied from https://github.com/statsmodels/statsmodels/issues/1810#issuecomment-48253113 but not updated
It still works this way, but currently I'm getting errors from my setup because I have missing names (or import problems because I haven't recompiled statsmodels)


run doctest just for a specific function or method:

```
import doctest

from statsmodels.regression.linear_model import OLS, OLSResults

doctest.run_docstring_examples(OLSResults.t_test, globals())
doctest.run_docstring_examples(OLSResults.f_test, globals())
```
doctests fail because of formatting, missing np.squeeze, numeric noise in float numbers
but relatively easy to check correctness by visual inspection.
`
user: josef-pkt
issue_4961: state: open
title: The APIs confused me when I wanted to forecast with ARMA
body: There are three "arma"s, ARMA, ARMAProcess, ARMAResults. I only want to forecast by arma(2,3) with known parameters (phi and theta), need not fitting.

I thought we should use ARMAResults, that requires a model when initialization. (Why dose a result need a model as an argument when the model can be derived from parameters?)

Following is the code. It requests `sigma2` (I set it to be 1, but it should has a default value), `nobs` which should be calculated automatically, then `k_trend` (I just let it be 0), then `transparams`, I have no idea now.

```python
import statsmodels.api as sm
import statsmodels.tsa.arima_model

import numpy as np

np.random.seed(12345)

# ar = np.r_[1, -1, 0.24] # add zero-lag and negate
# ma = np.r_[1, 0.4,0.2,0.1] # add zero-lag
# model = sm.tsa.ArmaProcess(ar, ma)
# model.isstationary == model.isinvertible == True

y=[1.704,0.527,1.041,0.942,0.555,-1.002,-0.585,0.01,-6.38,0.525]   # observations
# predict the 11th
model = sm.tsa.ARMA(y, (2, 3))
model.sigma2=1
model.nobs=10
model.k_trend=0
result = statsmodels.tsa.arima_model.ARMAResults(model, params=(1, -0.24, 0.4,0.2,0.1))
result.forecast(1)

# or use ARMA.predict(params)
```

Maybe I don't understand the module very well, please help me for forecast/predict.
user: Freakwill
issue_4956: state: open
title: Status of resampling methods in statsmodels
body: I was just curious what the current state of affairs was with respect to resampling methods in `statsmodels`, and how much room there is for base functionality that could potentially support other routines (or be available for people to use creatively in their own projects).  For instance, are there plans or is there interest in a collection of basic bootstrapping methods that could be organized into a separate module?  Or perhaps routines for doing simple permutation hypothesis testing (I see @josef-pkt discussed this in #3571, but I am thinking outside of a modeling context for now; just fairly simple testing to start).
user: dsaxton
issue_4942: state: open
title: Suggestion: include quasi-likelihood models
body: I think quasi-likelihood models (https://en.wikipedia.org/wiki/Quasi-likelihood) could make a useful addition to `statsmodels`, especially since they currently do not seem to have an implementation in Python (they are however supported in R, even within the base `glm` function).  I do see that `statsmodels` includes generalized estimating equations, but I don't know enough about these to say how much overlap these have with quasi-likelihood methods.

In any event, one common use case for these models is dealing with overdispersion in count data, where a Poisson distribution may not provide the best fit (see issue #2632).
user: dsaxton
issue_4935: state: open
title: skip_one set correctly in IRLS tests?
body: In test_glm and test_glm_weights a bunch of tests are run in a loop in test_wtd_gradient_irls, test_gradient_irls_eim, test_gradient_irls.  In each of these a variable `skip_one` is set to `False` outside the loop, then in one location in the loop is set to `True`.  Is that intentional?  it seems more likely in context that `skip_one` should be set to `False` in every step of the loop except for that one.
user: jbrockmendel
issue_4930: state: open
title: Where does genmod.tests.gee_*check belong?
body: These aren't tests.  Are they examples?  Results?  These files are in reasonably good shape, should be made useful somehow.
user: jbrockmendel
issue_4919: state: open
title: STY: intentional pep-8 whitespace violation
body: 
I don't like not having whitespace around '=' in long list of keyword arguments. Autogenerated cases are in the Stata results modules.
I also prefer to use it for results (`Holder` or `Bunch`) classes with, e.g.,  5 or more keywords

example results class for a hypothesis test that I'm just writing

```
        res = Holder(statistic = statistic,
                     pvalue = pvalue,
                     compare = compare,
                     method = method,
                     diff = diff,
                     ratio = ratio,
                     odds_ratio = odds_ratio,
                     alternative = alternative,
                     )
```
versus pep-8 compliant:

```
        res = Holder(statistic=statistic,
                     pvalue=pvalue,
                     compare=compare,
                     method=method,
                     diff=diff,
                     ratio=ratio,
                     odds_ratio=odds_ratio,
                     alternative=alternative,
                     )
```
user: josef-pkt
issue_4917: state: open
title: DOC/REL: what to advertise of my recent merges
body: https://github.com/statsmodels/statsmodels/issues/4908#issuecomment-413341722

AFAIR, several of my recent ENH are not (yet) public, e.g. penalized and ultra-high screening.
I should get at least some notebook illustrating "experimental" usage. My notebooks on my computer using penalized mixin are outdated (from before default penal class changed from Scad to no penalization)

similar about score_test work


user: josef-pkt
issue_4916: state: open
title: Pairwise p-values for AnovaRM
body: Statsmodels now has a repeated-measured anova, `AnovaRM`.  However, as far as I can tell there is no way to get the p-value between individual pairs of within-subject parameters.  Would it be possible to add that?
user: toddrjen
issue_4905: state: open
title: base._parameter_inference should not copy code from _diagnostic_other
body: That's what "import" is for.
user: jbrockmendel
issue_4892: state: open
title: improve get_prediction documentation
body: Hi, 


thanks to [this blog post](https://www.digitalocean.com/community/tutorials/a-guide-to-time-series-forecasting-with-arima-in-python-3#step-2-—-importing-packages-and-loading-data) I realized that `get_predection` also provide confidence intervals ... maybe [the doc](https://www.statsmodels.org/dev/generated/statsmodels.tsa.statespace.mlemodel.MLEResults.get_prediction.html#statsmodels-tsa-statespace-mlemodel-mleresults-get-prediction) should be more clear about what it returns ...

Thanks for the support
user: ggrrll
issue_4887: state: open
title: Don't Use Sandbox Code Outside Sandbox
user: jbrockmendel
issue_4882: state: open
title: ENH: power, sample size for 2 sample negative binomial rates
body: (mainly parking a reference, downloaded in 2015, but I don't remember anything)

Zhu, Haiyuan, and Hassan Lakkis. 2014. “Sample Size Calculation for Comparing Two Negative Binomial Rates.” Statistics in Medicine 33 (3): 376–87. https://doi.org/10.1002/sim.5947.

we also need hypothesis test and confidence interval, but maybe only asymptotic
(eg. can we check robustness of inference for binomial and poisson rates to over dispersion, e.g. inflate the variance)

companion to two sample tests for binomial proportion and poisson rates.
A generalization is required to take account of overdispersion, i.e. extra variance parameter
Beta-binomial might be similar for binomial rates.

see also:
#2723, #2718 for poisson
#4828 proportion
and others

user: josef-pkt
issue_4876: state: open
title: tools._testing and tools.testing
body: Are both needed?  What's the relevant distinction?
user: jbrockmendel
issue_4866: state: open
title: ENH: 2 proportion confint with skew correction - Gart Nam
body: add score test and confidence interval with skewness correction

It has more equal tails than Miettinen Nurminen, and is the default in NCSS/PASS for inferiority and similar tests for two proportions.

reference Gart/Name articles
and R package `rateci` and that authors article(s)
Laud, Peter J. "Equal‐tailed confidence intervals for comparison of rates." Pharmaceutical statistics 16, no. 5 (2017): 334-348.
and corrigendum for odds-ratio version. 

I will not include this in #4829 (according to current plans)
Main work will be to figure out what iterative solver/rootfinder to use. Laud 2017 Appendix might be useful for this.

Aside: `rateci` and Laud also include the skewness correction for Poisson rate confint
related to #2723 
user: josef-pkt
issue_4856: state: open
title: ENH add var_weights to weightstats.DescrStatsW
body: (I don't find an open issue for it)

The basic results would correspond to using WLS for one or two sample cases.

I'm not sure how far we get by reusing freq_weights but normalizing freq_weights.sum() to nobs.
According to GLM weights discussion and computation of WLS rsquared, there are some differences where we need to be careful about using sqrt weights versus weights in the aggregation.

user: josef-pkt
issue_4847: state: open
title: mixedlm runs indefinitely without producing output
body: I am fitting a mixed model with about 10,000 groups, eg: 
``
    import statsmodels.formula.api as smf
``
``
    model = smf.mixedlm('Y ~ cov_1 + cov_2 + cov_3', 
        data, 
        groups = data['grouping_variable']).fit()
``

for most substratifications of my data this works fine and takes about 2 minutes to run. But occasionally, it runs indefinitely (for hours) without producing any output. I assume there is some idiosyncrasy of the data which is causing this but I'm not sure what it might be. I'm running version 0.8.0. 
user: epierson9
issue_4839: state: open
title: MNLogit does not raise an error if coefficients become nan
body: I ran

`otu.dummyDS.values` has dtype int64, values of [0, 1, 2], and contains 336 observations.
`normed` is a matrix of observations and variables, float64, very scarce, shape = (336, 3348)
Running a normal MLE logit on this should (and does) fail to converge. However, when running the code below, MNLogit claimed the 'model converged' and did not raise any warnings. 

```
mnlogit = discrete_model.MNLogit(otu.dummyDS.values, normed)

mle_fit = mnlogit.fit()
```

Worryingly, running a sparse regression `mnlogit.fit_regularized(alpha=1)` never seems to converge or exceed max_iters. However, the exact same matrix on using the `sklearn` algorithm does terminate with convergence and no errors within 1-2 minutes. 
user: dangeles
issue_4838: state: open
title: MNLogit.initialize() cannot operate on a 1-d vector
body: I initialize using `otu_vars`, an obs x vars np array, and feed it `dummyDS` as an endogenous obs x 1 vector:
 
`mnlogit = discrete_model.MNLogit(dummyDS, otu_vars)`

Then I call the initialize function:
`mnlogit.initialize()`

I get the following error:

> ---------------------------------------------------------------------------
> AxisError                                 Traceback (most recent call last)
> <ipython-input-39-fa0f9013df49> in <module>()
>       1 normed = StandardScaler().fit_transform(otu[otu_vars].values)
>       2 mnlogit = discrete_model.MNLogit(otu.dummyDS.values, normed)
> ----> 3 mnlogit.initialize()
> 
> /anaconda3/lib/python3.5/site-packages/statsmodels/discrete/discrete_model.py in initialize(self)
>     555         super(MultinomialModel, self).initialize()
>     556         # This is also a "whiten" method in other models (eg regression)
> --> 557         self.endog = self.endog.argmax(1)  # turn it into an array of col idx
>     558         self.J = self.wendog.shape[1]
>     559         self.K = self.exog.shape[1]
> 
> AxisError: axis 1 is out of bounds for array of dimension 1


However, initializing the object by calling it returns no error, and I can even fit the data to a log function....

EDIT: I noticed I mis-named the `otu_vars` variable when I first named it (I referred to it as `normed`) probably causing confusion. This doesn't alter the nature of the problem.
user: dangeles
issue_4828: state: open
title: SUMM/ENH/Overview: two sample proportion, independent
body: **update:** basic non-exact methods merged in #2605, original PR #4829, issue #2605

----

two sample binomial proportion analogue to Poisson rates
#2718, https://github.com/josef-pkt/misc/wiki/Exact-and-Asymptotic-Tests---Rates,-Poisson
and one-sample proportion
https://github.com/josef-pkt/misc/blob/7fb0160bb7f8fd5eef00e276c61ec7fbe2e20238/notebooks/proportion_one_power.ipynb

#3954 more general issue, rates, proportion, paired sample, ...

## results
- hypothesis test, pvalue
- confidence interval
- power and sample size

## comparisons/hypothesis
- difference in proportion, support non-zero `value`
- (risk) ratio
- odds ratio
- equality `value=0` for diff, or `value=1` for ratios

Basic version needs two-sided and one-sided alternative
TOST equivalence can reuse one-sided with value != 0.
Equality tests can take advantage of permutation or similar "exact" methods. E.g. fisher's exact test.

## methods
- wald
- score
- exact and pseudo-/semi-exact

the list of methods differs by references, e.g. PASS/NCSS documentation, 
Newcombe 1998 has 11 methods for `diff` confint
Fagerland at al 2015 have 6-8 methods for confint for each diff, ratio and odds ration, not including berger boos
https://doi.org/10.1080/10543406.2018.1452028 has 97 methods according to title for one-sided diff inference.

I guess we get at least 5 to 10 including options, e.g.

- wald: with and without adjustments, e.g. add 0.5 or 1 observation. fast
- score_test: with or without adjustments, e.g. small sample correction (`nobs / (nobs - 1)`) or skewness/Bartlett correction (Gart and Nam)
- (semi-)exact: based on binomial distribution
  - `e-test`: use MLE for nuisance parameter
  - `sup` sup/max over nuisance parameter
  - `berger-boos`
  - `conditional` only available for odds-ratio cases (analogue to fisher's exact test, both margins fixed)
     most likely I ignore this case for the general functions
  - variants for which test statistic is used to rank cases, e.g. Agresti-Min (one two-sided score test) versus two one-sided score tests)
  - mid-p variants

## What's available in various issues/PRs and what's missing?

confint 
#2605 issue, https://github.com/josef-pkt/statsmodels/pull/5 score_confint code from PropCI
and code in issue comments

#2608 PR hypothesis tests for equality, including exact, berger-boos, 
#2607 issue for exact

model analogue, GLM
score and profile loglike confint
https://github.com/statsmodels/statsmodels/issues/4798#issuecomment-405522059
#1791

## code structure, API

*public*
`confint_proportion_2indep`  draft version (draft version, uncommitted)
`test_proportion_2indep`, ? not written yet,  maybe as analogue to confint function
???

*internal*
- function for constrained estimation in 2 independent sample case: mainly Miettinen and Nurminen 1985
- test statistics
- ztest_generic usage
- confint explicit
- confint numerical inversion and root finding

classes or functions, e.g. #2608 uses class for exact and berger-boos
`confint_proportion_2indep` is a single function with code for wald confint inside, but needs to delegate for score and exact confints
more complex confint will need tests as helper functions, for simple cases we can compute confint directly.

power, sample size: I haven't looked yet. NCSS uses exact computation for asymptotic/approximate tests and confint, at least for small samples


user: josef-pkt
issue_4817: state: open
title: ENH: wald_test, t_test, string specification
body: There are problems with wald_test and t_test for MixedLM that is missing features of the standard versions.

https://github.com/statsmodels/statsmodels/issues/4787#issuecomment-408874312
related issues #2100, #2231


user: josef-pkt
issue_4813: state: open
title: Add files via upload
body: This python file provides the functionality to optimise the dataset for Multiple Regression Models using Backward Elimination. Columns are first dropped using the adjusted R-Squared values and then to further optimise the data, p-values of the columns are also checked.

Pass the features(X) followed by the labels(y) and the significance level you want to consider into the backward_eliminaton function. It will return the optimised dataset.

NOTE: No need to pass the dataset with the constant column already added. It will automatically add the column.
user: anishgoyal24
issue_4812: state: open
title: Update statsmodels.stats.gof.gof_chisquare_discrete docstring
body: http://www.statsmodels.org/stable/generated/statsmodels.stats.gof.gof_chisquare_discrete.html#statsmodels.stats.gof.gof_chisquare_discrete

You can see it takes an argument named `distfn` but the docs (still?) show `distname` as the argument. Their types differ, so it's not just a renaming of the argument.
user: PDiracDelta
issue_4811: state: open
title: statsmodels.nonparametric.kde.KDEUnivariate inconsistency
body: Hi,

I am currently using the `KDEunivariate` function to calculate kernel densities. But each time I run the code with the same parameters on the same variable, I get different results which are completely different altogether. Is there any reason to why this is happening ?

My dataset has ~70K observations of the variable

Thanks
user: rtmlp
issue_4804: state: open
title: .predict() loses categorical response labels
body: This issue may be the same as referenced in #4544 

Example code:
```
# Data

response = np.random.randint(-1, 2, 100)
predictor1 = np.random.uniform(size = 100)
predictor2 = np.random.uniform(size = 100)
predictor3 = np.random.uniform(size = 100)
df = pd.DataFrame({'response': response, 'pred1': predictor1, 'pred2': predictor2, 'pred3': predictor3})

# Model

formula = 'response ~ pred1 + pred2 + pred3'
fit = smf.mnlogit(formula = formula, data = df).fit()

# Predictions 

predict = fit.predict(df[['pred1', 'pred2', 'pred3']])
predict
```
Our `df` looks like this: 
![screen shot 2018-07-19 at 11 48 29 am](https://user-images.githubusercontent.com/39494755/42954034-b42d5b68-8b49-11e8-9530-f124dd206b89.png)
The `response` column contains 3 possible outcomes: -1, 0 and 1. 

However, `predict` looks like this:
![screen shot 2018-07-19 at 11 49 30 am](https://user-images.githubusercontent.com/39494755/42954082-d367b46a-8b49-11e8-9f48-66adcb2ece54.png)
The prediction columns are labelled 0, 1 and 2.

Looking at the [documentation](https://tedboy.github.io/statsmodels_doc/generated/generated/statsmodels.formula.api.MNLogit.predict.html#statsmodels.formula.api.MNLogit.predict), we can see that "Column 0 is the base case," and from looking at fit.summary() we can see which response is the base case (-1, in this case). This still leaves us with no idea as to to correspondence between the other two columns in `predict` (1 and 2) and the other two responses (0 and 1). If my response variable had more categories I would be even more in the dark.

Thanks!

I am using `statsmodels 0.9.0`
user: Abraham-Leventhal
issue_4803: state: open
title: Causality, treatment effect, DAG (Perl)
body: python package
https://github.com/akelleh/causality

related to #2871 which includes links to packages for matching estimators.

(IIRC I have seen other python packages using graphical models like DAG based.) 

user: josef-pkt
issue_4802: state: open
title: reference package python: Bayesian GLM
body: https://github.com/bambinos/bambi on top of pymc3 or Stan.

uses formulas for fixed effects.

(found while browsing a few packages from 
https://github.com/statsmodels/statsmodels/network/dependents?dependent_type=PACKAGE )
user: josef-pkt
issue_4799: state: open
title: ENH: estimation subject to nonlinear constraints, e.g. in GLM
body: just a use case idea:

profile and score confidence intervals for parameters need constrained estimation. In GLM the constraints are linear.

If we want the analogous profile and score confidence interval on the predicted values, e.g. probabilities in Binomial, then we would need a nonlinear constraint that prob takes on a specific value.
(On second thought:) It might be possible to transform this to a constraint on a linear combination of parameters in some cases with single-index or link function, by getting the equivalent constraints on the linear predictor.

The current fit_constrained only allows linear/affine constraints on params that are coefficients for exog.

user: josef-pkt
issue_4798: state: open
title: BUG: GLM score_test singular matrix
body: **edit** this looks more like a problem for score and profile likelihood confidence intervals with a link function that does not satisfy domain constraints.

I got an example where scoretest raises with singular matrix in np.linalg.solve
This is using #4716, and a score confidence interval example.

Maybe I need to use pinv instead of solve.

```
0.23119 -71.3544794856 [ 0.05000118]
Traceback (most recent call last):
  File "E:\Josef_new\scripts\score_confint_naive.py", line 55, in <module>
    low = optimize.brentq(diff_scoretest, wald_ci[0]*2, wald_ci[0]*0.5, xtol=1e-5)
  File "...\WinPython-64bit-3.4.4.5Qt5\python-3.4.4.amd64\lib\site-packages\scipy\optimize\zeros.py", line 442, in brentq
    r = _zeros._brentq(f,a,b,xtol,rtol,maxiter,args,full_output,disp)
  File "E:\Josef_new\scripts\score_confint_naive.py", line 41, in diff_scoretest
    pvalue = resc.score_test()[1]
  File "E...\statsmodels_py34_pr\statsmodels\genmod\generalized_linear_model.py", line 1671, in score_test
    observed=observed)
  File "...\statsmodels_py34_pr\statsmodels\base\_parameter_inference.py", line 249, in score_test
    chi2stat = score.dot(np.linalg.solve(cov_score_test, score[:, None]))
  File "...\WinPython-64bit-3.4.4.5Qt5\python-3.4.4.amd64\lib\site-packages\numpy\linalg\linalg.py", line 384, in solve
    r = gufunc(a, b, signature=signature, extobj=extobj)
  File "...\WinPython-64bit-3.4.4.5Qt5\python-3.4.4.amd64\lib\site-packages\numpy\linalg\linalg.py", line 90, in _raise_linalgerror_singular
    raise LinAlgError("Singular matrix")
numpy.linalg.linalg.LinAlgError: Singular matrix
```
example is during score confidence interval search
2-sample proportion, second exog is indicator with 11 of 100 obs in group 0
```
>>> y.tolist()
[1, 0, 1, 0, 1, 1, 0, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 0, 1, 1, 0, 1, 1, 0, 1, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1, 0, 0, 1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1]

>>> x[:,1].tolist()
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
```
user: josef-pkt
issue_4797: state: open
title: Inconsistent behaviour of statsmodels.stats.weightstats.DescrStatsW
body: The code says all:

    In [1]: DescrStatsW([1,1],weights=[1,1]).mean
    Out[1]: 1.0

    In [2]: DescrStatsW([1],weights=[1]).mean
    Out[2]: array([1.])

Why an array is returned on the second time. It does not seem right.
user: jachymb
issue_4792: state: open
title: Some perhaps significant changes and advices for summary2
body: Recently I want to use python to output statistic results like Stata's command `esttab` . But I only know two methods like  `summary()` and `summary2()`  by reading official examples. In addition, statistical results cannot be output to external files like Excel . So I read the source code of above two methods expect that I  can  do something. Ultimately, I make some code changes  in summary2. Fortunately, the new summary2  can directly output the results of multiple models with stars by it's  `summary_col()`  function. And at the same time, we can use pandas method  `to_excel()`  or  `to_csv`  to export the summary results as .xls or .csv file. Besides, my modifications also support Panel Regression from the package [`linearmodels`](https://pypi.org/project/linearmodels/). Of course, I also found some bugs in summary2.
My main idea is to make all the parts of summary table as DataFrame and then append them into `self.tables`.So pandas output method works. Detailed changes and descriptions and possible bugs in summary2 are shown below, and some examples are given at the end. All changes will be marked with #+ Arabic number like `# +1: `,`#+2: ` , and unchanged parts are replaced by `......`
```
class Summary(object):
     
          ......
     
    # +1: 
    # I added the output method based on pd.DataFrame().to_excel/csv(). 
    # I merged  the results when they are output, mainly in order to make 
    #  the output be distinguishable and more beautiful when printing. 
    #  Maybe there are other better ways.

    def to_excel(self,path=None):
        tables = self.tables
        import  os
        cwd = os.getcwd()
        if path:
            path = path 
        else:
            path = cwd + '\\summary_results.xlsx'
        summ_df = pd.concat(tables,axis=0)
        return summ_df.to_excel(path)
    def to_csv(self,path=None):
        tables = self.tables
        import  os
        cwd = os.getcwd()
        if  path:
            path = path 
        else:
            path = cwd + '\\summary_results.csv'
        summ_df = pd.concat(tables,axis=0)
        return summ_df.to_csv(path)

def _measure_tables(tables, settings):
    '''Compare width of ascii tables in a list and calculate padding values.
    We add space to each col_sep to get us as close as possible to the
    width of the largest table. Then, we add a few spaces to the first
    column to pad the rest.
    '''
    # simple_tables = _simple_tables(tables, settings)
    # tab = [x.as_text() for x in simple_tables]

    # length = [len(x.splitlines()[0]) for x in tab]
    # len_max = max(length)
    # pad_sep = []
    # pad_index = []

    # for i in range(len(tab)):
    #     nsep = tables[i].shape[1] - 1
    #     pad = int((len_max - length[i]) / nsep)
    #     pad_sep.append(pad)
    #     len_new = length[i] + nsep * pad
    #     pad_index.append(len_max - len_new)

    # return pad_sep, pad_index, max(length)

    #+2 : 
    # Above codes may have two bugs:
    #Bug1: If tables or settings is an empty list, 
             # then _simple_tables() will return [].
             # that means length is also empty , 
             # so max() will raise an error. 
   # Bug2: If table[i] just has one column, '/nsep' will raise ZeroDivisionError. 
             # So I added exception capture codes as follows.

    simple_tables = _simple_tables(tables, settings)
   
    if simple_tables == []:
        len_max = 0
        pad_sep = None
        pad_index = None
    else:
        tab = [x.as_text() for x in simple_tables]
        length = [len(x.splitlines()[0]) for x in tab]
        len_max = max(length)
        pad_sep = []
        pad_index = []
        for i in range(len(tab)):
            nsep = tables[i].shape[1] - 1
            try:
                pad = int((len_max - length[i]) / nsep)
            except (ZeroDivisionError):
                pad = int((len_max - length[i]))
            pad_sep.append(pad)
            len_new = length[i] + nsep * pad
            pad_index.append(len_max - len_new)

    return pad_sep, pad_index, len_max

      ......

def summary_model(results):
    '''Create a dict with information about the model
    '''
    def time_now(*args, **kwds):
        now = datetime.datetime.now()
        return now.strftime('%Y-%m-%d %H:%M')
    info = OrderedDict()

 #+3:
 # I added some informations of  Panel regression from the package linearmodels. 
 # Panel regression has some different attribute names, but it doesn't matter here.

    info['Model:'] = lambda x: x.model.__class__.__name__
    info['Model Family:'] = lambda x: x.family.__class.__name__
    info['Link Function:'] = lambda x: x.family.link.__class__.__name__
    info['Dependent Variable:'] = lambda x: x.model.endog_names
    # add1  
    info['Dependent Variable:'] = lambda x: x.model.dependent.vars[0]
    
    info['Date:'] = time_now
    info['No. Observations:'] = lambda x: "%#6d" % x.nobs
    info['Df Model:'] = lambda x: "%#6d" % x.df_model
    info['Df Residuals:'] = lambda x: "%#6d" % x.df_resid
    info['Converged:'] = lambda x: x.mle_retvals['converged']
    info['No. Iterations:'] = lambda x: x.mle_retvals['iterations']
    info['Method:'] = lambda x: x.method
    info['Norm:'] = lambda x: x.fit_options['norm']
    info['Scale Est.:'] = lambda x: x.fit_options['scale_est']
    info['Cov. Type:'] = lambda x: x.fit_options['cov']
    # add2 
    # I added the x.cov_type item because some model 
    # there is no fit_options  attribute like OLS model

    info['Covariance Type:'] = lambda x: x.cov_type
    info['Covariance Type:'] = lambda x: x._cov_type # Panel

    info['R-squared:'] = lambda x: "%#8.3f" % x.rsquared
    info['Adj. R-squared:'] = lambda x: "%#8.3f" % x.rsquared_adj
    info['Pseudo R-squared:'] = lambda x: "%#8.3f" % x.prsquared
    info['AIC:'] = lambda x: "%8.4f" % x.aic
    info['BIC:'] = lambda x: "%8.4f" % x.bic
    info['Log-Likelihood:'] = lambda x: "%#8.5g" % x.llf
    # add 3
    info['Log-Likelihood:'] = lambda x: "%#8.5g" % x.loglike

    info['LL-Null:'] = lambda x: "%#8.5g" % x.llnull
    info['LLR p-value:'] = lambda x: "%#8.5g" % x.llr_pvalue
    info['Deviance:'] = lambda x: "%#8.5g" % x.deviance
    info['Pearson chi2:'] = lambda x: "%#6.3g" % x.pearson_chi2
    info['F-statistic:'] = lambda x: "%#8.4g" % x.fvalue
    # add4
    info['F-statistic:'] = lambda x: "%#8.4g" % x.f_statistic.stat

    info['Prob (F-statistic):'] = lambda x: "%#6.3g" % x.f_pvalue
    # add5
    info['Prob (F-statistic):'] = lambda x: "%#6.3g" % x.f_statistic.pval

    info['Scale:'] = lambda x: "%#8.5g" % x.scale
    # add6
    info['Effects:'] = lambda x: ','.join(['%#8s' % i for i in x.included_effects])
   
    out = OrderedDict()
    for key, func in iteritems(info):
        try:
            out[key] = func(results)
        # NOTE: some models don't have loglike defined (RLM), so that's NIE
        except (AttributeError, KeyError, NotImplementedError):
            pass
    return out

def summary_params(results, yname=None, xname=None, alpha=.05, use_t=True,
                   skip_header=False, float_format="%.4f"):
    '''create a summary table of parameters from results instance

    Parameters
    ----------
    res : results instance
        some required information is directly taken from the result
        instance
    yname : string or None
        optional name for the endogenous variable, default is "y"
    xname : list of strings or None
        optional names for the exogenous variables, default is "var_xx"
    alpha : float
        significance level for the confidence intervals
    use_t : bool
        indicator whether the p-values are based on the Student-t
        distribution (if True) or on the normal distribution (if False)
    skip_headers : bool
        If false (default), then the header row is added. If true, then no
        header row is added.
    float_format : string
        float formatting options (e.g. ".3g")

    Returns
    -------
    params_table : SimpleTable instance
    '''
    from linearmodels.panel.results import PanelEffectsResults
    from linearmodels.panel.results import RandomEffectsResults 
    from linearmodels.panel.results import PanelResults
    res_tuple = (PanelEffectsResults,PanelResults,RandomEffectsResults)

    if isinstance(results, tuple):
        results, params, std_err, tvalues, pvalues, conf_int = results
    # else:
    #     params = results.params
    #     bse = results.bse
    #     tvalues = results.tvalues
    #     pvalues = results.pvalues
    #     conf_int = results.conf_int(alpha)
   
   #+4 : 
   # I added Panel results whose some attributes name are different.
   # So I modified the code as follows.

    elif isinstance(results,res_tuple):
        bse = results.std_errors
        tvalues = results.tstats
        conf_int = results.conf_int(1-alpha)
    else:
        bse = results.bse
        tvalues = results.tvalues
        conf_int = results.conf_int(alpha) 
    params = results.params
    pvalues = results.pvalues

    data = np.array([params, bse, tvalues, pvalues]).T
    data = np.hstack([data, conf_int])
    data = pd.DataFrame(data)

    if use_t:
        data.columns = ['Coef.', 'Std.Err.', 't', 'P>|t|',
                        '[' + str(alpha/2), str(1-alpha/2) + ']']
    else:
        data.columns = ['Coef.', 'Std.Err.', 'z', 'P>|z|',
                        '[' + str(alpha/2), str(1-alpha/2) + ']']

    if not xname:
        # data.index = results.model.exog_names
        try:
            data.index = results.model.exog_names
        except (AttributeError):
            data.index = results.model.exog.vars
    else:
        data.index = xname

    return data

    #+5:
    # The following function just can stack standard errors,but  we
    #  usually use t statistics in reality. I modified the function to 
    # support one of standard errors, t or pvalues by parameter 'show' .
    #+6:
    # Bug: There exists different names for intercept item in different models,
    # for example, an OLS model named it 'Intercept' while 'const' in logit models.
    # So I also added a function to uniform the name to facilitate the data merge.

## Vertical summary instance for multiple models
# def _col_params(result, float_format='%.4f', stars=True):
#     '''Stack coefficients and standard errors in single column
#     '''

#     # Extract parameters
#     res = summary_params(result)
#     # Format float
#     for col in res.columns[:2]:
#         res[col] = res[col].apply(lambda x: float_format % x)
#     # Std.Errors in parentheses
#     res.ix[:, 1] = '(' + res.ix[:, 1] + ')'
#     # Significance stars
#     if stars:
#         idx = res.ix[:, 3] < .1
#         res.ix[idx, 0] = res.ix[idx, 0] + '*'
#         idx = res.ix[:, 3] < .05
#         res.ix[idx, 0] = res.ix[idx, 0] + '*'
#         idx = res.ix[:, 3] < .01
#         res.ix[idx, 0] = res.ix[idx, 0] + '*'
#     # Stack Coefs and Std.Errors
#     res = res.ix[:, :2]
#     res = res.stack()
#     res = pd.DataFrame(res)
#     res.columns = [str(result.model.endog_names)]
#     return res

def _col_params(result, float_format='%.4f', stars=True,show='t'):
    '''Stack coefficients and standard errors in single column
    '''

 # I add the parameter 'show' equals 't' to display tvalues by default,
 # 'p' for pvalues and 'se' for std.err.
    
    # Extract parameters
    res = summary_params(result)
   
    # Format float
    # Note that scientific number will be formatted to  'str' type though '%.4f'

    for col in res.columns[:3]:
        res[col] = res[col].apply(lambda x: float_format % x)
    res.iloc[:,3] = np.around(res.iloc[:,3],4)
    
    # Significance stars
    # .ix method will be deprecated,so .loc has been used.

    if stars:
        idx = res.iloc[:, 3] < .1
        res.loc[res.index[idx], res.columns[0]] += '*'
        idx = res.iloc[:, 3] < .05
        res.loc[res.index[idx], res.columns[0]] += '*'
        idx = res.iloc[:, 3] < .01
        res.loc[res.index[idx], res.columns[0]] += '*'

    # Std.Errors or tvalues or  pvalues in parentheses
    res.iloc[:,3] = res.iloc[:,3].apply(lambda x: float_format % x) # pvalues to str
    res.iloc[:, 1] = '(' + res.iloc[:, 1] + ')'
    res.iloc[:, 2] = '(' + res.iloc[:, 2] + ')'
    res.iloc[:, 3] = '(' + res.iloc[:, 3] + ')'

    # Stack Coefs and Std.Errors or pvalues
    if show is 't':
        res = res.iloc[:,[0,2]]
    elif show is 'se':
        res = res.iloc[:, :2]
    elif show is 'p':
        res = res.iloc[:,[0,3]]
    res = res.stack()
    res = pd.DataFrame(res)
    try:
        res.columns = [str(result.model.endog_names)]
    except (AttributeError):
        res.columns = result.model.dependent.vars #for PanelOLS
    
   # I added the index name transfromation function 
   # to deal with MultiIndex and single level index.

    def _Intercept_2const(df):
        from pandas.core.indexes.multi import MultiIndex
        if df.index.contains('Intercept'):
            if isinstance(df.index,MultiIndex):
                new_index = []
                for i in df.index.values:
                    i = list(i)
                    if 'Intercept' in i:
                        i[i.index('Intercept')] = 'const'
                    new_index.append(i)
                multi_index = lzip(*new_index)
                df.index = MultiIndex.from_arrays(multi_index)
            else:
                index_list = df.index.tolist()
                idx = index_list.index('Intercept')
                index_list[idx] = 'const'
                df.index = index_list
        return df
    return _Intercept_2const(res)

# def _col_info(result, info_dict=None):
#     '''Stack model info in a column
#     '''
#     if info_dict is None:
#         info_dict = {}
#     out = []
#     index = []
#     for i in info_dict:
#         if isinstance(info_dict[i], dict):
#             # this is a specific model info_dict, but not for this result...
#             continue
#         try:
#             out.append(info_dict[i](result))
#         except:
#             out.append('')
#         index.append(i)
#     out = pd.DataFrame({str(result.model.endog_names): out}, index=index)
#     return out

   #+7 :
   # I modified the above function,main work is that 
   # I rename the parameter 'info_dict' to 'more_info',which is a list not a dict.
   # Besides, I build a default dict to contain some model information 
   # from summary_model(), that will be printed by default and 
   # users can append other statistics by more_info parameter.
   
def _col_info(result, more_info=None):
   
    '''Stack model info in a column
    '''
    model_info = summary_model(result)
    default_info_ = OrderedDict()
    default_info_['Model:'] = lambda x: x.get('Model:')
    default_info_['No. Observations:'] = lambda x: x.get('No. Observations:')
    default_info_['R-squared:'] = lambda x: x.get('R-squared:')
    default_info_['Adj. R-squared:'] = lambda x: x.get('Adj. R-squared:')                    
    default_info_['Pseudo R-squared:'] = lambda x: x.get('Pseudo R-squared:')
    default_info_['F-statistic:'] = lambda x: x.get('F-statistic:')
    default_info_['Covariance Type:'] = lambda x: x.get('Covariance Type:')
    default_info_['Eeffects:'] = lambda x: x.get('Effects:')
    default_info_['Covariance Type:'] = lambda x: x.get('Covariance Type:')

    default_info = default_info_.copy()
    for k,v in default_info_.items():
        if v(model_info):
            default_info[k] = v(model_info)
        else:
            default_info.pop(k) # pop the item whose value is none.
            
    if more_info is None:
        more_info = default_info
    else:
        if not isinstance(more_info,list):
            more_info = [more_info]
        for i in more_info:
            try:
                default_info[i] = getattr(result,i)
            except (AttributeError, KeyError, NotImplementedError) as e:
                raise e
        more_info = default_info
    try:
        out = pd.DataFrame(more_info, index=[result.model.endog_names]).T
    except (AttributeError):
        out = pd.DataFrame(more_info, index=result.model.dependent.vars).T
    return out

# def _make_unique(list_of_names):
#     if len(set(list_of_names)) == len(list_of_names):
#         return list_of_names
#     # pandas does not like it if multiple columns have the same names
#     from collections import defaultdict
#     name_counter = defaultdict(str)
#     header = []
#     for _name in list_of_names:
#         name_counter[_name] += "I"
#         header.append(_name+" " + name_counter[_name])
#     return header
   
   #+8:
   # Above function has a flaw that non-duplicated names will be add a suffix.
   # And the time when endog_names duplicate four or more times ,the y 
   # names will be like 'y IIII' or 'y IIIIII...'.So I used the Arabic numerals.

def _make_unique(list_of_names):
    if len(set(list_of_names)) == len(list_of_names):
        return list_of_names
    # pandas does not like it if multiple columns have the same names
    from collections import defaultdict
    dic_of_names = defaultdict(list)
    for i,v in enumerate(list_of_names):
        dic_of_names[v].append(i)
    for v in  dic_of_names.values():
        if len(v)>1:
            c = 0
            for i in v:
                c += 1
                list_of_names[i] += '_%i' % c
    return list_of_names

   #+9:
   # The following function is the most critical to work.
   # In this function  I added the parameters 'show' and 'title',
   # and changed the default value of 'stars' into 'True',
   # Then  I changed the dict parameter 'info_dict' as a list one named 'more_info'.
   # Finally I put 'const'  at the first location by default in regressor_order.
   #10+:
   # Bug: np.unique() will disrupt the original order of list,
   # this can result in index confusion.

# def summary_col(results, float_format='%.4f', model_names=[], stars=False,
#                 info_dict=None, regressor_order=[]): 
    # """
    # Summarize multiple results instances side-by-side (coefs and SEs)

    # Parameters
    # ----------
    # results : statsmodels results instance or list of result instances
    # float_format : string
    #     float format for coefficients and standard errors
    #     Default : '%.4f'
    # model_names : list of strings of length len(results) if the names are not
    #     unique, a roman number will be appended to all model names
    # stars : bool
    #     print significance stars
    # info_dict : dict
    #     dict of lambda functions to be applied to results instances to retrieve
    #     model info. To use specific information for different models, add a
    #     (nested) info_dict with model name as the key.
    #     Example: `info_dict = {"N":..., "R2": ..., "OLS":{"R2":...}}` would
    #     only show `R2` for OLS regression models, but additionally `N` for
    #     all other results.
    #     Default : None (use the info_dict specified in
    #     result.default_model_infos, if this property exists)
    # regressor_order : list of strings
    #     list of names of the regressors in the desired order. All regressors
    #     not specified will be appended to the end of the list.
    # """

def summary_col(results, float_format='%.4f', model_names=[], stars=True,
                more_info=None, regressor_order=[],show='t',title=None): 
    if not isinstance(results, list):
        results = [results]

    cols = [_col_params(x, stars=stars, float_format=float_format,show=show) for x in
            results]

    # Unique column names (pandas has problems merging otherwise)
    if model_names:
        colnames = _make_unique(model_names)
    else:
        colnames = _make_unique([x.columns[0] for x in cols])
    for i in range(len(cols)):
        cols[i].columns = [colnames[i]]

    merg = lambda x, y: x.merge(y, how='outer', right_index=True,
                                left_index=True)
    summ = reduce(merg, cols)

    # if regressor_order:
    if not regressor_order:
        regressor_order = ['const']
    
    varnames = summ.index.get_level_values(0).tolist()
    ordered = [x for x in regressor_order if x in varnames]
    unordered = [x for x in varnames if x not in regressor_order + ['']]

    # Note: np.unique can disrupt the original order  of list 'unordered'.
    # Then pd.Series().unique()  works well.

    # order = ordered + list(np.unique(unordered))
    order = ordered + list(pd.Series(unordered).unique())

    f = lambda idx: sum([[x + 'coef', x + 'stde'] for x in idx], [])
    # summ.index = f(np.unique(varnames))
    summ.index = f(pd.Series(varnames).unique())
    summ = summ.reindex(f(order))
    summ.index = [x[:-4] for x in summ.index]

    idx = pd.Series(lrange(summ.shape[0])) % 2 == 1
    summ.index = np.where(idx, '', summ.index.get_level_values(0))
    summ = summ.fillna('')
    
    # add infos about the models.
#     if info_dict:
#         cols = [_col_info(x, info_dict.get(x.model.__class__.__name__,
#                                            info_dict)) for x in results]
#     else:
#         cols = [_col_info(x, getattr(x, "default_model_infos", None)) for x in
#                 results]
    
       cols = [_col_info(x,more_info=more_info) for x in results]
    
    # use unique column names, otherwise the merge will not succeed
    for df , name in zip(cols, _make_unique([df.columns[0] for df in cols])):
        df.columns = [name]
    merg = lambda x, y: x.merge(y, how='outer', right_index=True,
                                left_index=True)
    info = reduce(merg, cols)
    info.columns = summ.columns
    info = info.fillna('')
#     dat = pd.DataFrame(np.vstack([summ, info]))  # pd.concat better, but error
#     dat.columns = summ.columns
#     dat.index = pd.Index(summ.index.tolist() + info.index.tolist())
#     summ = dat

#     summ = summ.fillna('')

#     smry = Summary()
#     smry.add_df(summ, header=True, align='l')
#     smry.add_text('Standard errors in parentheses.')
#     if stars:
#         smry.add_text('* p<.1, ** p<.05, ***p<.01')*p<.01')
#     return smry

    if show is 't':
        note = ['\t t statistics in parentheses.']
    if show is 'se':
        note = ['\t Std. error in parentheses.']
    if show is 'p':
        note = ['\t pvalues in parentheses.']
    if stars:
        note +=  ['\t * p<.1, ** p<.05, ***p<.01']

#Here  I tried two ways to put extra text in index-location or
# columns-location,finally found the former is better.

#     note_df = pd.DataFrame(note,index=['note']+['']*(len(note)-1),
#                                                        columns=[summ.columns[0]])

    note_df = pd.DataFrame([ ],index=['note:']+note,
                                                columns=summ.columns).fillna('')
#     summ_all = pd.concat([summ,info,note_df],axis=0)
    
    if title is not None:
        title = str(title)
    else:
        title = '\t Results Summary'
    
    # Here I tried to construct a title DataFrame and 
    # adjust the location of title corresponding to the length of columns. 
    # But I failed because of not good printing effect.
    
    # col_len = len(summ.columns)
    # fake_data = ['']*col_len
    # if col_len % 2 == 1:
    #     from math  import ceil
    #     i = ceil(col_len/2)
    # else:
    #     i = int(col_len/2)
    # fake_data[i-1] = title
    # title_df = pd.DataFrame([fake_data],index=[''],columns=summ.columns).fillna('')
    
    title_df = pd.DataFrame([],index=[title],columns=summ.columns).fillna('')
    
    smry = Summary()
    smry.add_df(title_df,header=False,align='l') # title DF
    smry.add_df(summ, header=True, align='l') # params DF
    smry.add_df(info, header=False, align='l') # model information DF
    smry.add_df(note_df, header=False, align='l') # extra text DF
    return smry

   ......

```
Next show some examples including OLS,GLM,GEE,LOGIT and Panel regression results.Other models do not test yet. But what can be determined is that multi-equation models  like VAR model does not work here.
```
# Load the data and fit
import numpy as np
import pandas as pd
import statsmodels.api as sm
import statsmodels.formula.api as smf
# ols
dat = sm.datasets.get_rdataset("Guerry", "HistData").data
res_ols = smf.ols('Lottery ~ Literacy + np.log(Pop1831)', data=dat).fit()
#glm
data = sm.datasets.scotland.load()
data.exog = sm.add_constant(data.exog)
gamma_model = sm.GLM(data.endog, data.exog, family=sm.families.Gamma())
res_glm = gamma_model.fit()
# gee
data = sm.datasets.get_rdataset('epil', package='MASS').data
fam = sm.families.Poisson()
ind = sm.cov_struct.Exchangeable()
mod = smf.gee("y ~ age + trt + base", "subject", data, cov_struct=ind, family=fam)
res_gee = mod.fit()
# logit
spector_data = sm.datasets.spector.load()
spector_data.exog = sm.add_constant(spector_data.exog)
logit_mod = sm.Logit(spector_data.endog, spector_data.exog)
res_logit = logit_mod.fit()

# load panel data and fit the model
from linearmodels.datasets import wage_panel
data = wage_panel.load()
year = pd.Categorical(data.year)
data = data.set_index(['nr', 'year'])
data['year'] = year

from linearmodels.panel import PooledOLS
exog_vars = ['black','hisp','exper','expersq','married', 'educ', 'union', 'year']
exog = sm.add_constant(data[exog_vars])
mod = PooledOLS(data.lwage, exog)
res_pooled = mod.fit()

from linearmodels.panel import PanelOLS
exog_vars = ['expersq','union','married']
exog = sm.add_constant(data[exog_vars])
mod = PanelOLS(data.lwage, exog, entity_effects=True, time_effects=True)
res_fe_re = mod.fit()

from linearmodels.panel import FirstDifferenceOLS
exog_vars = ['exper','expersq', 'union', 'married']
exog = data[exog_vars]
mod = FirstDifferenceOLS(data.lwage, exog)
res_fd = mod.fit()

exog_vars = ['black','hisp','exper','expersq','married', 'educ', 'union']
exog = sm.add_constant(data[exog_vars])
mod = PooledOLS(data.lwage, exog)

res_robust = mod.fit(cov_type='robust')
res_clust_entity = mod.fit(cov_type='clustered', cluster_entity=True)
res_clust_entity_time = mod.fit(cov_type='clustered', cluster_entity=True, cluster_time=True)
```
Then we import the function `summary_col()` from the modified summary2 that I named summary3 as a module.Thus we can  directly output the concatenated results with stars and some default model informations.
```
from summary3 import summary_col 
```
For single regression result,we can directly pass the result object,surely a list is better:
```
# summary_col(res_ols)
summary_col([res_ols]) 
```
This will return the Summary  class instance, in Notebook the output is:
![](https://github.com/YangShuangjie/summary3/blob/master/pictures/Snipaste_2018-07-10_04-03-08.jpg?raw=true)
We can also use print function to output as text. Parameter `more_info` will add new model information to print. For example,
```
print(summary_col([res_ols,res_glm,res_gee,res_logit],more_info=['df_model','scale']))
``` 
The incompete output is
![](https://github.com/YangShuangjie/summary3/blob/master/pictures/Snipaste_2018-07-10_04-52-11.jpg?raw=true)
We can also use `regressor_order` to designate the order of variables,`show` to display the anyone of pvalues,tvalues or std.err you want,`title` to define a custom title for your table. 
```
print(sumary_col([res_fe_re,res_fd,res_robust,res_clust_entity,res_clust_entity_time],
             regressor_order=['black'],show='se',title='Panel Results Summary Table'))
```
The output is
![](https://github.com/YangShuangjie/summary3/blob/master/pictures/Snipaste_2018-07-10_04-54-10.jpg?raw=true)
Finally,if you want to export the summary results to external files,you can do like this
```
summary_col([res_glm,res_logit]).to_excel()
```
Above will obtain a excel file in your working directory named 'summary_results'.
 Of course you can define the filename and path just like  use pandas(actually indeed so) .
```
summary_col([res_clust_entity,res_fd]).to_csv('your path\\filename.csv')
```


user: young2j
issue_4791: state: open
title: varmax with TypeError: not all arguments converted during string formatting
body: Hi, 

I pull the raw data that i want to run VARMAX on by codes below (inserted the snapshot of how the dataframe looks like) 

```python
endog=blp.historicalRequest(
    ['USGG2YR Index','USGG5YR Index','USGG10YR Index','USGG30YR Index'], 'PX_LAST', "19911231",
    date.today().replace(day=calendar.monthrange(date.today().year, date.today().month)[1]).strftime('%Y%m%d'),periodicitySelection='MONTHLY')
endog['Date'] =endog.index
endog['Date']= pd.to_datetime(endog['Date']).apply(lambda x: x.date())
endog.index=endog['Date']
endog.drop(endog.iloc[:,4:],inplace=True,axis=1)
```

![image](https://user-images.githubusercontent.com/40484063/42448591-31c8b910-83b0-11e8-99b3-49654539c8fe.png)

When i run the following codes. 

```python
mod_var1 = statsmodels.tsa.statespace.varmax.VARMAX(
    endog[['USGG2YR Index','USGG5YR Index','USGG10YR Index','USGG30YR Index']], exog=None,
    order=(3, 0), trend='c', error_cov_type='unstructured', measurement_error=False, 
    enforce_stationarity=True, enforce_invertibility=True) 

res_var1 = mod_var1.fit()
```

I got the following error - 
    for i in range(self.k_endog)

TypeError: not all arguments converted during string formatting

Does anyone has any idea what was the issue and how to resolve it? 
Thanks very much 

user: lightnino
issue_4774: state: open
title: CompareMeans summary: Add pandas columns names
body: ## Problem/question

The output of a CompareMeans summary, when pandas DataFrames are inputed, is constituted of rows labels named `"subset #1", ..., "subset #N"` .

Wouldn't it be possible to get pandas columns names instead ?

### Small exemple:
#### Code
```python 
import pandas as pd
import statsmodels.stats

df1 = pd.DataFrame({"Age": [21,23,54], 'Income': [200, 150,600]})
df2 = pd.DataFrame({"Age": [30,26,34], 'Income': [400, 250,100]})

print(statsmodels.stats.weightstats.CompareMeans.from_data(df1,df2).summary())

```
#### Output

```
                          Test for equality of means                          
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
subset #1      2.6667     10.929      0.244      0.819     -27.677      33.011
subset #2     66.6667    166.667      0.400      0.710    -396.074     529.408
==============================================================================
```

#### Expected

```
                          Test for equality of means                          
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
Age         2.6667     10.929      0.244      0.819     -27.677      33.011
Income     66.6667    166.667      0.400      0.710    -396.074     529.408
==============================================================================
```



## Solution ?

Apparently names are generated from that line:

https://github.com/statsmodels/statsmodels/blob/0b5cea54c7db8b836bf01e4301f1a112eec54027/statsmodels/stats/weightstats.py#L810

What would be necessary to include columns names ?


ps: statsmodels version: 0.9.0

user: adrienpacifico
issue_4771: state: open
title: ENH: Vectorized OLS, simplified Multivariate Linear Regression using OLS
body: I'm attempting to do multivariate linear regression using statsmodels. For that, I am using the Ordinary Least Squares model. Just to be precise, this is not multiple linear regression, but multivariate - for the case AX=b, b has multiple dimensions. I am using a set number of components (A, shape (1024, 4)) to fit a number of spectra (b, shape(1024, 50). 

Now, the docstring of `OLS()` describes the first input variable (in this case my y) as:
```
endog : array-like
    1-d endogenous response variable. The dependent variable.
```
which emphasises that this should not be used for my 2D `b`.

At this point, I should first ask:
**Is there a better model for me to use?**

However, the following case gives the correct multivariate fitting using the above model:
```python
>>> import statsmodels.api as sm
>>> import numpy as np

>>> length = 1024
>>> pixels = 50
>>> number_of_components = 4
>>> ymulti = np.random.random((length, pixels))
>>> ysingle = np.random.random((length, ))
>>> A = np.random.random((length, number_of_components))

>>> model = sm.OLS(ymulti, A)
>>> result = model.fit()
>>> result.params.shape
(4, 50)

```
The problem is that all the other functionality of the model, including `.summary()` and `result.HC3_se` is showing the errors, and fails due to shape mismatch. I'm just showing two examples here, but there are many others, all with similar errors.

They all appear to be related to simple fixes, i.e. taking the transpose of the second entry of the numpy dot product.

The `summary()` might be a bit more tricky, since the components now describe a much higher number of pixels, so I'm not sure how that would be handled.
```python
>>> result.summary()
~/miniconda3/envs/py36/lib/python3.6/site-packages/statsmodels/regression/linear_model.py in ssr(self)
   1513     def ssr(self):
   1514         wresid = self.wresid
-> 1515         return np.dot(wresid, wresid)
   1516 
   1517     @cache_readonly

ValueError: shapes (1024,50) and (1024,50) not aligned: 50 (dim 1) != 1024 (dim 0)
```

```python
>>> result.HC3_se
...error traceback removed for clarity...
~/miniconda3/envs/py36/lib/python3.6/site-packages/statsmodels/regression/linear_model.py in cov_HC3(self)
   1680         h = np.diag(chain_dot(
   1681             self.model.wexog, self.normalized_cov_params, self.model.wexog.T))
-> 1682         self.het_scale = (self.wresid / (1 - h))**2
   1683         cov_HC3 = self._HCCM(self.het_scale)
   1684         return cov_HC3

ValueError: operands could not be broadcast together with shapes (1024,50) (1024,) 
```
user: thomasaarholt
issue_4759: state: open
title: SUMM: generic enhancements GLM, discrete
body: this is just to keep track of current additions, because I lost a bit the overview where what is being added

-  #4620 fix GLM hessian factor, clarify role of scale and concentrated loglike, no ENH
   test cases GLM vs discrete
- #4576 penalized Mixin and penalties
   - base Model _fit_zeros and _fit_collinear
   - penalized mixin: classes in base, but no public classes yet for specific Models
   - base/tests/test_penalized.py cases for Logit, Probit, Poisson and GLM counterparts
   - needs review for cov_params
- #4683 screening
  - base._screening, no public functions, needs public Penalized function for #4576
  - base/tests/test_screening.py cases Poisson, Logit, GLM-Poisson GLM-Logit, GLM-Gaussian
  - ENH: add discrete poisson  `resid_pearson`
- #2096 score/lm and cm tests
  - stats/_diagnostic_other.py  base functionality for score and conditional moment tests
    still includes Poisson over/excess dispersion tests
  - base/_parameter_inference.py  score/lm test for attaching to model results classes
  - tests/test_score_test.py   only GLM-Poisson
  - stats/tests/test_diagnostic_other.py   only OLS
  - ENH: discrete Poisson: add `family`, `score_factor`, `hessian_factor`
  - ENH: GPP: add `resid_response`
- #4732  GLMInfluence
  - stats/outliers_influence.py  MLEInfluence, GLMInfluence
  - ENH: GLM: add `_deriv_mean_dparams`, `_deriv_score_obs_dendog` derivatives for MLEInfluence
  - ENH: GLMResults: add `get_hat_matrix_diag` and `get_influence`
  - stats/tests/test_influence.py cases GLM-Logit, GLM-Binomial, GLM-Gaussian


still open:

- #4716 score tests, discrete
  - ENH discrete BinaryModel add `offset` and `fit_constrained`
  - discrete/tests/test_constrained.py case Logit
  - GLMResults: add `score_test` method
  - discrete/tests/test_sandwich_cov.py  unit tests comparing GLM and discrete counterparts
  - ENH: add score_factor and hessian_factor to remaining models (except Negbin) ?
  - ...

targets:

- add missing pieces to discrete classes
- add `score_test` to no-extra params discrete classes Logit, Probit, Poisson
- add `get_influence` with MLEInfluence for Logit, Probit, Poisson
- extend `score_test` and `get_influence` to extra params models NBP and GPP
- ignore NegativeBinomial ?
- future: two part models, ZeroInflated, BetaRegression.
- fit_constrained for extra params models ?
- public, official Penalized classes, GAM ?
- diagnostic tests based on score/lm test





user: josef-pkt
issue_4755: state: open
title: ENH: RecursiveLS: Add recursive t-test
body: Along with some improvements / fixes, adds the ability to compute a t-test for any subset of the data after estimating a model via `RecursiveLS`. For example:

```python
dta = sm.datasets.macrodata.load_pandas().data
dta.index = pd.PeriodIndex(start='1959Q1', end='2009Q3', freq='Q')
endog = dta.infl
exog = sm.add_constant(dta[['m1', 'unemp']])

# Run RecursiveLS over the entire model
mod = sm.RecursiveLS(endog, exog)
res = mod.fit()

# Compute the t-test using the first 100 observations
t = 99
restriction = 'm1 + unemp = 1, const = 0.5'
ttest_rls = res.t_test(restriction, loc=t)

# Now, run OLS over the restricted sample and compute the t-test
mod_ols = sm.OLS(endog[:t], exog[:t])
res_ols = mod_ols.fit()
ttest_ols = res_ols.t_test(restriction)

# Compare the results
print(ttest_rls)
print(ttest_ols)
```

yields

```
                             Test for Constraints                             
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
c0            -1.4091      0.227    -10.628      0.000      -1.859      -0.959
c1             4.2860      0.985      3.845      0.000       2.332       6.240
==============================================================================
                             Test for Constraints                             
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
c0            -1.4091      0.227    -10.628      0.000      -1.859      -0.959
c1             4.2860      0.985      3.845      0.000       2.332       6.240
==============================================================================
```
user: ChadFulton
issue_4754: state: open
title: ENH/Design: warn users if there is no constant ?
body: Should we issue a warning if there is no constant in the exog?

I think it's not really needed, but it would be helpful to new users, and as a reminder even for experienced users, i.e. I forget it sometimes myself and see it only when I look at the summary() or count params.

(after seeing models on stackoverflow without constant whether intentional or not.)
user: josef-pkt
issue_4752: state: open
title: ENH: speeding up model training by moving it to GPU?
body: I am searching for a way to speed up model training since we need to train many of them (i.e. dozens of millions). To profile things up, I took an example model, which takes 190 seconds to train. Out of those 75% are spent in kalman_filter.py:733(_filter) and the next 17% are spent in numpy.core.multiarray.dot.

Looking at statsmodels/tsa/kalmanf/kalman_loglike.pyx, I can see that the code is already heavily optimized for CPU by using Fortran BLAS libraries. So one idea is to move at least the two functions to run on GPUs instead of CPUs.

Please, let me know if you have a feeling of much it could help and which Python CUDA library (CuPy, Numba, MinPy, PyTorch, TensorFlow, CULA, scikit-cuda, PyCUDA, PyOpenCL, pyculib, theano, etc) would be a good choice to proceed?

Thank you for your help!
user: vss888
issue_4749: state: open
title: Strange huber estimation failure
body: Running a huber estimate on the data in the attached script, produces NaN for the new scale estimate on the first huber iteration due to the fact that the SQRT gets a negative result as input.

[bug_huber.txt](https://github.com/statsmodels/statsmodels/files/2110979/bug_huber.txt)

user: arminsch
issue_4747: state: open
title: BUG: GenericKDE, custom bandwidth is assumed to be ndarray-like bool-indexable
body: First of all, let me thank you for the `statsmodels.nonparametric` subpackage, its is wonderful with all its features and regression support (even though it may not be the most efficient computation-wise).

When a GenericKDE is supplied with a custom bandwidth [`GenericKDE._compute_efficient` returns it as is](https://github.com/statsmodels/statsmodels/blob/master/statsmodels/nonparametric/_kernel_base.py#L182). However, some code that runs subsequently [assumes that `bw` supports ndarray-like bool indexing](https://github.com/statsmodels/statsmodels/blob/master/statsmodels/nonparametric/_kernel_base.py#L513). I suggest to simply return `np.atleast_1d(bw)` when it is custom.
I can contribute such a change and a testcase if you agree on this approach.

The docs mention that `bw` should be array_like, but that includes a list AFAIK as well and a list does not support such indexing.

### Version
 `0.8.0`

### Steps to reproduce

```python
import numpy as np
import statsmodels.api as sm

N = 100
x = np.arange(N)
y = x**2 + np.random.normal(size=N)
kr = sm.nonparametric.KernelReg([y], [x], 'c', bw=[0.5], defaults=sm.nonparametric.EstimatorSettings(efficient=True))
Ey = kr.fit(x)
```
throws:
```Python
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-9-f36766e055b6> in <module>()
----> 1 kr.fit(x)

~/anaconda3/lib/python3.6/site-packages/statsmodels/nonparametric/kernel_regression.py in fit(self, data_predict)
    374         for i in range(N_data_predict):
    375             mean_mfx = func(self.bw, self.endog, self.exog,
--> 376                             data_predict=data_predict[i, :])
    377             mean[i] = mean_mfx[0]
    378             mfx_c = np.squeeze(mean_mfx[1])

~/anaconda3/lib/python3.6/site-packages/statsmodels/nonparametric/kernel_regression.py in _est_loc_linear(self, bw, endog, exog, data_predict)
    158                    #ukertype='aitchison_aitken_reg',
    159                    #okertype='wangryzin_reg',
--> 160                    tosum=False) / float(nobs)
    161         # Create the matrix on p.492 in [7], after the multiplication w/ K_h,ij
    162         # See also p. 38 in [2]

~/anaconda3/lib/python3.6/site-packages/statsmodels/nonparametric/_kernel_base.py in gpke(bw, data, data_predict, var_type, ckertype, okertype, ukertype, tosum)
    512 
    513     iscontinuous = np.array([c == 'c' for c in var_type])
--> 514     dens = Kval.prod(axis=1) / np.prod(bw[iscontinuous])
    515     if tosum:
    516         return dens.sum(axis=0)

TypeError: only integer scalar arrays can be converted to a scalar index
```
### Workaround
Supply a real ndarray as `bw=np.array([0.5])`
user: smartass101
issue_4745: state: open
title: ENH: GLM, discrete nonlinear prediction confint for ppf and sf
body: just an idea to get a rough measure on a confidence interval for prediction in GLM or single index models.

get_prediction for GLM estimates a confidence interval for the mean prediction.
ppf and sf are nonlinear functions of the predicted mean, e.g. Poisson

get a upper confidence bound on the sf and a lower confidence bound on ppf, eg. 
```
confint_low, confint_upp = results.get_prediction(alpha=alpha_parameter_uncertainty)
alpha = 0.05
poisson.sf(alpha, confint_upp)
poisson.ppf(alpha, confint_low)
```

This "should" be a bound on the double alpha confidence interval for an observation, taking both parameter uncertainty and randomness of endog into account.
In terms of implementation this is just one side of the nonlinear function endpoint transformation confint.

The application would be to get some indication for outliers in asymmetric distributions that does not rely on some transformed residuals, such as resid_anscombe or resid_deviance.
(I'm currently looking at studentized residuals and thresholds for outlier_influence in Binomial #4732)

(There should be related issues, but I only used MonteCarlo before for predictive distribution with parameter uncertainty.)

user: josef-pkt
issue_4742: state: open
title: REF/Design: should families be stateless
body: specific issue from #4741

families.Binomial stores n_trials `n`

I thought families are stateless, except for link and family/link parameters, i.e. no attributes that refer to a specific dataset.

When we recreate models, then we reuse the family and link instances. So, if that depends on data, then we need to watch out that we have the correct attributes, and don't modify the existing attributes inplace.

The alternative design would be to only keep those attributes in the GLM model instance, but then we need n_trials as additional parameter, that only applies to the Binomial family, which is not nice either.

One problem is that I would like to reuse families and distribution classes for other things, which is more difficult if families are stateful. (e.g. #3915 and related)


user: josef-pkt
issue_4741: state: open
title: REF/ENH: recreating models with different obs or nobs
body: It looks like the current recreating models with `_get_init_kwds` cannot be easily extended to subsampling, e.g. in bootstrap, cross-validation or leave-one-out loops.

The problem is in GenericLikelihoodModelResults for bootstrap cov_params.

Right now I want an explicit LOOO loop for influence and outlier diagnostic for GLM (my current code will be in the next comment). My current solution is a GLM specific hack with binomial count as example case.

- we need to check which init_kwds are full length, so we can add them to the subsampling/indexing
- family.Binomial has `n` attribute, which is created during `initialize`

for GLM-Binomial the recent change was to use n_trials as init_kwd and using the model.endog instead of providing the original count data. #4601
user: josef-pkt
issue_4740: state: open
title: REF/Review OLSInfluence
body: #4732 extension of influence to other models

OLSInfluence hasn't been reviewed and refactored since I added it and Skipper did some adjustments.

main refactoring: 
- streamline for with and without full LOOO loop.
- add more generic solution instead of OLS specific
- add hat_matrix_diag to model/results class
- what variants of measures should be cached and what should be available as properties without caching, e.g. SAS has dfbeta and dfbetas, where the second just divides by bse.
- add more premade plots, attach to OLSInfluence/Influence classes

most of those will not be addressed and fixed by #4732 


user: josef-pkt
issue_4738: state: open
title: Overflow encountred
body: Hello,

I want to get a statestics for multinomial logistic regression, however I got this warnings : 

![image](https://user-images.githubusercontent.com/13202562/41298452-b218385c-6e61-11e8-89c3-b0268ebf5c18.png)
And the results is just Nan values,

here is my train data

![image](https://user-images.githubusercontent.com/13202562/41298567-e896338e-6e61-11e8-97af-5b81e66b917b.png)
 and my test data : 

![image](https://user-images.githubusercontent.com/13202562/41298824-64984260-6e62-11e8-851a-70998c4b7358.png)

user: 7belrhalmia
issue_4735: state: open
title: MixedLM.predict inconsistently fails for df with string
body: Reproducible minimum example:
```python
import numpy as np
import pandas as pd
import statsmodels.api as sm

data = pd.DataFrame({'A': np.arange(10),
                     'B': ['a'] * 10,
                     'G': [0]*5 + [1] * 5,
                     'Y': np.arange(10) + np.random.randn(10)})
print(data.head())

model = sm.MixedLM.from_formula('Y ~ A', groups='G', re_formula='1',
                                data=data.iloc[1:,])
result = model.fit()

# This works
print(result.predict(data.head(1)))

# This fails with AttributeError: 'DataFrame' object has no attribute 'dtype'
print(result.predict(data.iloc[0, :]))
```
**If you run it without column "B", or change it to `'B': [1] * 10,` (in the data = pd.DataFrame step), it works every time.**

_Also, there are times that we get the_

> ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals

_warning as well, but if you just re-run it a few more times, it'll work (through the fitting part) without any warnings, and the final error will still show up, regardless of the warning._

From `show_versions` (running with Anaconda distribution):
```
INSTALLED VERSIONS
------------------
Python: 3.6.5.final.0
OS: Linux 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27 10:45:36 UTC 2018 x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

Statsmodels
===========

Installed: 0.9.0 (/path/to/miniconda3/envs/virtual/lib/python3.6/site-packages/statsmodels)

Required Dependencies
=====================

cython: Not installed
numpy: 1.14.3 (/path/to/miniconda3/envs/virtual/lib/python3.6/site-packages/numpy)
scipy: 1.1.0 (/path/to/miniconda3/envs/virtual/lib/python3.6/site-packages/scipy)
pandas: 0.23.0 (/path/to/miniconda3/envs/virtual/lib/python3.6/site-packages/pandas)
    dateutil: 2.7.3 (/path/to/miniconda3/envs/virtual/lib/python3.6/site-packages/dateutil)
patsy: 0.5.0 (/path/to/miniconda3/envs/virtual/lib/python3.6/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 2.2.2 (/path/to/miniconda3/envs/virtual/lib/python3.6/site-packages/matplotlib)
    backend: module://ipykernel.pylab.backend_inline 
cvxopt: Not installed
joblib: Not installed

Developer Tools
================

IPython: 6.4.0 (/path/to/miniconda3/envs/virtual/lib/python3.6/site-packages/IPython)
    jinja2: 2.10 (/path/to/miniconda3/envs/virtual/lib/python3.6/site-packages/jinja2)
sphinx: 1.7.4 (/path/to/miniconda3/envs/virtual/lib/python3.6/site-packages/sphinx)
    pygments: 2.2.0 (/path/to/miniconda3/envs/virtual/lib/python3.6/site-packages/pygments)
nose: Not installed
pytest: Not installed
virtualenv: Not installed
```
user: kenjioman
issue_4734: state: open
title: ENH: Add rsquared and variants to state space models
body: Currently the state space models don't compute any kind of R^2 type value, but it might be handy / desirable in some cases. "Forecasting, structural time series models and the Kalman filter" (Harvey, 1989) section 5.5.5 describes three variants:

- R^2 comparing fit to a model with just the mean (i.e. usual R^2): 1 - SSE / SSM
- R^2 comparing fit to a random walk + drift model: 1 - SSE  / SSDM (SSDM = sum of squares of first differences around the mean)
- R^2 comparing fit to a random walk + drift + seasonal dummies: 1 - SSE / SSDSM (SSDSM = sum of squares of first differences around the seasonal means)

This should not be too difficult to add (and also not urgent), so it might be good for someone who is interested in contributing a first PR.
user: ChadFulton
issue_4733: state: open
title: REF: scale versus mse_resid in RegressionModel, OLS
body: OLSInfluence (and I guess in many other places) we use results.mse_resid instead of scale.

There is also still the problem that scale is an attribute `RegressionResults.__init__` keyword and a cache_writable attribute.
I thought I had an issue for that, but don't find any.

The point is that I would like to standardize models on scale as resid variance estimate.
Right now the occasion is that I would like to make Influence classes more generic and need more consistency across models. #4732
user: josef-pkt
issue_4731: state: open
title: SUMM: influence function for penalized or constrained models
body: a companion issue to #4257
topic: towards generic local influence functions and measures, more theory and applications
I still don't have a good overview how the different special cases connect. 

Here are a few more articles related to recent extensions in statsmodels

The second article by Zhu and Zhang uses the influence function to derive some influence measures. Their setup is based on misspecified likelihood (White 1982)

I haven't read the other papers (AFAIR)

Avella-Medina looks very interesting, e.g. influence function for penalized M-estimators like lasso or scad based on locally smooth approximation. If I understand correctly based on skimming some parts, then the asymptotic variance/distribution only takes the parameters that were not penalized to zero, i.e. block-diagonal where second block is zero. I guess we have a asymptotic justification for using some standard M-estimator sandwich cov_params for the nonzero params, but whether that has good small/medium sized sample properties is not clear to me, and doubtful. (e.g. M-estimator RLM cov_params for a smoothed centile function are not close to the kink, local kernel estimate version in QuantReg.)
The local smoothing is pretty much what I used in penalized/penalties which is the computational trick used by Fan and others.

Paula and Cysneiros's article is closer to linear model and GLM and just has a bunch of linear algebra similar to the score_test for linear constraints. (based only a 1 minute look)

Venezuela et al I didn't look at it, but it also needs to handle the sandwich form for the covariance and is not based on an optimization problem, i.e. GEE case.

There are more related articles where I didn't look much, e.g. on influence in linear Ridge regression, and of course a very large number for specific distributions (but that's mostly standard MLE)

## References

Avella-Medina, Marco. 2017. “Influence Functions for Penalized M-Estimators.” Bernoulli 23 (4B): 3178–96. https://doi.org/10.3150/16-BEJ841.

Zhu, Hongtu, and Heping Zhang. 2004. “A Diagnostic Procedure Based on Local Influence.” Biometrika 91 (3): 579–89. https://doi.org/10.1093/biomet/91.3.579.


Paula, Gilberto A., and Francisco José A. Cysneiros. 2010. “Local Influence Under Parameter Constraints.” Communications in Statistics - Theory and Methods 39 (7): 1212–28. https://doi.org/10.1080/03610920902871438.

Venezuela, Maria Kelly, Mônica Carneiro Sandoval, and Denise Aparecida Botter. 2011. “Local Influence in Estimating Equations.” Computational Statistics & Data Analysis 55 (4): 1867–83. https://doi.org/10.1016/j.csda.2010.10.020.



user: josef-pkt
issue_4730: state: open
title: ENH/SUMM  MLE based mean-variance models
body: I would like to get fully specified MLE models for mean-variance models, e.g. we don't have HetGaussian yet, just HetGLS, another case Beta regression and I guess we can add a variance function to GeneralizedPoisson, Another possible case is Gamma for which there exist some example scripts as MLE and as GMM.

What I would like are different cases for score/LM or CM test for joint or separate mean and variance hypothesis, see #4729 

This is specific for MLE cases, we have separate issues for QMLE or GMM or sequential/iterative estimators in GLM/LEF (like HetGLS extended to HetGLM).

Another one that I bookmarked but never looked at closely: BetaBinomial as dispersed Binomial.
user: josef-pkt
issue_4729: state: open
title: ENH/TST: score/LM and CM tests based on estimating equations or GMM
body: When we start with estimating or moment equations, then the derivative is not necessarily symmetric, in contrast to when we start from a likelihood maximization problem.

This case is implemented, but no example or test case yet.
I don't have currently a test case case for score_test or CM test that is not likelihood based. In some cases I'm not sure whether I use the right upper or lower off-diagonal block. In the symmetric Hessian case it didn't matter.
e.g. in the case with score_deriv_cu versus score_deriv_uc

reading Breslow again, cited in stats._diagnostic_other

Breslow, Norman. 1989. “Score Tests in Overdispersed GLM’s.” In Statistical
Modelling, edited by Adriano Decarli, Brian J. Francis, Robert Gilchrist, and
Gilg U. H. Seeber, 64–74. Lecture Notes in Statistics 57. Springer New York.
http://link.springer.com/chapter/10.1007/978-1-4612-3680-1_8.

In the joint specification of mean and variance, Breslow uses the estimating equations directly. The moment equation for the variance function is based on pearson_residual function or in a second version an a pseudo-likelihood base moment equation.

The partial derivatives of the moment condition is zero for mean parameter with respect to variance parameter, but nonzero for variance parameter with respect to mean parameters.
That implies that we can separate the mean parameter estimate from the variance assumption but not vice-versa.
It's not clear to me when we can ignore that mean parameters are estimated when testing heteroscedasticity and correlation. Breslow includes the effect. In Gaussian we can ignore it. AFAIR, the Poisson dispersion tests also ignore it.
(I don't remember the details, but it's related to symmetry and third moment. I need to check GMM again when just working with arbitrary moment conditions.)

Breslow has the long linalg formula for variance of the score for the dispersion parameter when it is "model-based", i.e. under the assumption of correctly specified variance, but it is not clear to me what the "empirical" i.e. robust version is. Difference is on whether we use empirical 3rd and 4th moment or model based (e.g. mu**4 in Poisson, similar to sigma**4 for Gaussian.)

I don't have an example yet for MLE based variance function robust score test, e.g. testing dispersion parameter in GeneralizedPoisson under the Null that dispersion is Poisson special case. 
AFAIR, Poisson is in the interior of the parameter space for GPP, in contrast to NegativeBinomial.

Overdispersion test in Poisson are essentially CM tests, and the generic framework with asymmetric score_deriv is not completely finished or tested.

related
#4716 add score-test to models
#2096 added background computation for score/LM and CM tests

user: josef-pkt
issue_4724: state: open
title: OLS.fit_regularized(L1_wt=0) returns params as numpy array instead of pandas Series
body: A pandas Series is returned as soon as L1_wt is different from 0.

user: kdebrab
issue_4718: state: open
title: REF/ENH: fit_constrained generic method
body: #4716 adds offset to the discrete binary models

We can add fit_constrained to more models. However, we might have to generalize the pattern, or make sure results instances are correctly created. Similar, case in fit_zeros and fit_collinear merged in #4576

- fit_constrained currently has 3 times essentially the same code
  GLM.fit_constrained, Poisson.fit_constrained and outsourced to a standalone function base.constraints.fit_constrained_wrap.
  `fit_constrained_wrap` is not yet used in any model, but should be able to handle the details for most models.

- inheritance or where do we attach it: next stage will just add discrete.BinaryModel

- extra params: NegativeBinomial and GeneralizedPoisson have one extra param, This might or might not work, i.e. the constraint needs to be only on exog-params. The extra param might go through without changes. `TransformRestriction` needs to be checked whether it leaves unconstrained params unaffected/untransformed

- multiple exog: Zero-Inflated has offset only for the main model, not the inflation model. params includes, concatenates the params of both models. Similarly, heteroscedastic models like Beta will have two exog. I don't remember whether hurdle models have two offset variables.  
  Restricting the "main model" might work similar to the extra-params case where we ignore non-main-exog params in the constraints.  
  Adding constraints for the second exog will be more work, and we will not be able to handle "cross-equation" constraints through exog transformation.








user: josef-pkt
issue_4715: state: open
title: ENH: families and distributions - add derivative wrt main parameters, e.g. mu, linpred, scale
body: genmod families don't have derivatives, derivatives of loglike are composed in GLM score_factor and hessian_factor.

I am adding score factor d loglike / d linpred and d loglike / d params_extra to discrete models, and hessian_factor for the second derivative.

I think we can standardize this on the families/distributions, so it's easier to use chain rule to create the pieces for more flexible derivatives.
e.g. d loglike / d mu * d mu / d params would also work for nonlinear models. and we can have reusable d loglike / d mu.
(Aside: Where did I put my GLMNonlinear prototype?)

genmod families hold the reference to the link function, in general I would prefer to have the link function directly in the model.
E.g. as experiment for score_test I wanted  to attach the link or inverse link function to Poisson. However, the generic interface for GLM always uses `model.families.link`. To have consistent access to the link function, I needed to attach the family to discrete Poisson in #2096

Related, but maybe too big of a change and break with statistics tradition: I would really like to invert the link classes to mean function classes, so we don't have to work back to front outside of IRLS.

aside: working with 2-D score and hessian factors in NBP and GPP count models doesn't look too bad. It will be a bit more work for two exog models, like zero-inflated and beta.

This entire story goes more towards GAMLSS
see #3915 for general issue
user: josef-pkt
issue_4714: state: open
title: ENH: sign convention for hessian_factor
body: I'm getting confused about the sign of hessian_factor. 

I was adding hessian factor to discrete so that `hessian = sum_i(hessian_factor_i * outer(exog_i))`, in an upcoming PR

However, GLM hessian_factor has the opposite sign
hessian =
```
factor = self.hessian_factor(params, scale=scale, observed=observed)
hess = -np.dot(self.exog.T * factor, self.exog)
``` 

I don't remember why GLM has that sign. The only reason that I can think of is so that hessian_factor is positive. My guess this came from SPSS.

The recently merged score_test assumes that the GLM definition prevails and raises with a opposite sign Poisson.
user: josef-pkt
issue_4706: state: open
title: SUMM/Review diagnostic and specification tests
body: I wrote the diagnostic tests initially mainly base on Green and a few articles, mostly with translating formulas to code.
This was a long time ago, and it needs revisiting to see what those tests are actually doing, and to extend them beyond OLS.

3 kinds of test in terms of using different versions and options

- identical results (up to numerical precision) but different ways of computing them
- asymptotically equivalent results under some given assumptions, that might agree numerically in small samples only in special cases
- results that differ because different assumptions on maintained or auxiliary assumptions, i.e. robust to other specification issues.

Most of the, especially older, econometrics text books and articles are not very specific on this, e.g. 
"We can estimate Sigma by ..."
"An asymptotically equivalent way of estimating Sigma is ..."

issues:


- centered versus uncentered rsquared
  https://github.com/statsmodels/statsmodels/pull/2096#issuecomment-394160965

- projection + rsquared versus wald
  Green describes for projecting on some variables/exog that are unconstrained and then use uncetered rsquared for the OLS with projected variables. (My guess is now that if constant is the only absorbed variable, then this is identical to using centered rsquared, as in het_xxx.

- non-OPG versions
  all the auxiliary regression versions assume OPG. Davidson and MacKinnon argue that those don't have good small sample properties

- mix observed Hessian and orthogonal/blockdiagnonal property of expected Hessian.
  AFAIU in LEF we use the property that asymptotically or in expected terms variance or second moments have zero off diagonal block with respect to mean parameters. We often impose this even when we use the observed Hessian within blocks. E,g, het test don't take into account that mean parameter are estimated , at least not in the moment conditions. mean tests don't take variance second moment assumptions into account. I think in small samples, the off-diagonal block wouldn't be zero, and I think it wouldn't be zero even in expected terms if the LEF is not the true distribution (LEF property of block diagonality is still a maintained assumption).
  Related: Green 5th edition on nonlinear models explains that mean and variance don't have the blockdiagonal structure of the (asymptotic) hessian.
user: josef-pkt
issue_4705: state: open
title: Drop redundant columns from exog
body: Do we have a way to drop redundant columns automatically from exog during model initialization?  This comes up sometimes when working with formulas having complicated interactions and dummy coding.  Sometimes there are linear dependencies in the design matrices that cannot easily be removed by modifying the formula.  I end up splicing the code below into the model `__init__` function. 

```
q, r = np.linalg.qr(exog)                                            
dr = np.abs(np.diag(r))                                              
if np.any(dr < 1e-10):                                               
    ii = np.flatnonzero(dr >= 1e-10)
    exog = exog[:, ii]
    if exog_names != None:
        exog_names = [exog_names[i] for i in ii] 
```
user: kshedden
issue_4701: state: open
title: NotImplementedError when predicting with BinomialBayesMixedGLM.
body: Hi! After I used fit_vb() to fit a Binomial Bayes Mixed GLM instead of fit() -as suggested by Josef in issue #4700- I tried to get the fitter values with predict() but got a NotImplementedError again. Is there any way this could be solved?
Again, I'm using the 0.9.0 version.
Thanks!
user: jpilorget
issue_4697: state: open
title: FAQ-D/REF/ENH: module, package structure for diagnostics
body: e.g. problem with circular imports
https://github.com/statsmodels/statsmodels/pull/2096#issuecomment-393570790

current:

- Wald test are in statsmodels.base, available for almost all models
- anova_lm in stats, standalone function, not imported or used as methods for any model
- current diagnostics and outlier_influence are in stats, supports mainly OLS, few methods attached to results with lazy imports, e.g. get_influence and outlier_test in OLSResults
- experimental discrete._diagnostics_count, private, not yet used outside unit tests (and one of my example notebooks)

to come:

- score_test for parameter restrictions: conceptually close to statsmodels.base like Wald tests, but will not be available for many models

others are for specification and diagnostic tests that are not simple restrictions on (exog) parameters, but some of them will be able to reuse a generic variable addition score test
:

- generic tools for LM/CM type tests: PR #2096 currently in stats._diagnostic_other
- specific specification test applicable to many models. This should be an extension of current `stats.diagnostic` to other models than OLS
- diagnostics specific to some models, e.g. overdispersion in count models, or testing Poisson versus NegativeBinomial, or link test in GLM.

How do we split this up and distribute it across modules and directories without getting into circular import and requiring a messy jumping around in api and source?

graphics has it's own directory
If we put most diagnostics into stats, then we have several modules there, including some that would be model specific.
If we put more than the minimal set of specification tests next to the models, then we jump around a lot, and need to watch out for import cycles.

user: josef-pkt
issue_4694: state: open
title: ENH/REF: GLM.score_test scale handling
body: refactor and test scale handling in GLM.score

- comment in docstring
  `not yet verified for case with scale not equal to 1.`
  check normal, gamma and others work correctly. scale handling in loglike and score have been refactored and fixed in statsmodels 0.9

- allow QMLE excess-dispersion for poisson and families with scale=1
  - compute dispersion scale, #4693 
  - use estimated scale in computing score, score_factor and hessian_factor.
  There might be a direct correction using the scale, but I didn't try to figure that out and might not be generic.

user: josef-pkt
issue_4693: state: open
title: ENH: GLM.estimate_scale   add option to choose scaletype
body: The GLM.estimate_scale method currently uses only the scaletype attribute to select the scale method.

We should add an option so we can override this when we want to compute different scale types from a results instance.
This could be reused for pearson chi2 and deviance scale in results instance,
and to handle excess dispersion for cov_type or LR and score test, even if the underlying scaletype attribute specifies a default. 
i.e. scale type is mostly a `fit` argument, in terms of inference, but a model attribute because we use it to concentrate out scale in loglike and related.

(That part is still the mess because scale is both a model parameter and an auxiliary parameter irrelevant for estimation of mean parameters.)

user: josef-pkt
issue_4687: state: open
title: BUG: follow-up Penalized mle scad, weights in SmoothedSCAD 
body: `weights` have only a unit test case for L2, but not for the others

e.g. SCADSmoothed with weights looks wrong and raises in #4683
https://github.com/statsmodels/statsmodels/pull/4683#issuecomment-392334228

user: josef-pkt
issue_4685: state: open
title: ENH: penalization with conjugate priors and artificial observations
body: Just a thought for a GLM analogue of treating penalties as prior observations.
This is mainly a theoretical issue to see if we get a group of cases where we can have a semi-generic implementation with a consistent theoretical interpretation.

TheilGLS: The original Theil, Goldberger article use the interpretation of the prior as the results from a previous sample. In the normal L2 context this would correspond to a normal-normal conjugate pair (assuming known variance). 
The moment conditions just form a weighted mean of prior and sample second moment matrix.

Jeffreys prior for proportion_confint just corresponds to adding 0.5 initial observations to both successes and failures. (beta-binomial conjugate pair)

Firth penalization uses Jeffreys prior more generally for GLM. #3561
I didn't really go through the computational details yet about how they use data augmentation for the Firth case. AFAIU, they change all endog observations directly.

GAM, PIRLS, uses the local WLS approximation to compute locally something similar or analogous to TheilGLS, either adding artificial observations or combining moment matrices.
(The current implementation in GAM adds artificial observations, AFAIR.)

AFAIR, all GLM-families/LEF with canonical link functions have conjugate prior pairs.

maybe this is mostly the point that Greenland makes in several articles I read, reference in #3561 

about implementation:
- For cases where we have a (data independent) prior interpretation, we can use penalty functions as in base._penalties
- For data dependent penalization (empirical Bayes and Jeffreys priors) we can try data augmentation or using the likelihood function of the data model to add the penalization.

e.g. maybe, a guess
The analogy to adding 0.5 observations in proportion_confint and in contingency tables could be just adding 0.5 (or something) similar observations for each cell, e.g. if we have empty cells, interaction terms where patsy adds a column of zeros.
I'm not sure how this works for continuous exog, i.e. just start with the one proportion or contingency table case.





user: josef-pkt
issue_4684: state: open
title: ENH: add fused L1 and SCAD penalization
body: I'm not sure I remember the literature, and what they call fused.

Current `restriction` in SCAD and L2 works on single constraints.
AFAIR: fused LASSO use hierarchical constraint, one penalization function on individual elements and another penalization function on the entire term. The latter part corresponds to a joint hypothesis for the constraints.
example: IIRC, is to use L2 inside and L1 for the entire term, i.e. we drop or keep all columns corresponding to a term together.

`base._penalties` has currently no class that allows this.

What I have instead is structured penalization e.g. on all-pairs differences, which can be combined with another linear constraint on the overall magnitude.
e.g. what happens if I impose a SCAD constraint that penalized the sum of coefficients belonging to a term, additional to the all pairs or sequential differences?  
(Note sum only enforces going to zero if signs of all params are the same, otherwise we need sum of abs which is not linear in params.)

aside: There was also a power penalization `abs(params)**(power_coefficient)`, for which I don't remember the details either. which could be in between L1 and L2 with power_coefficient between 1 and 2. (I think there was some problem mentioned in the literature for power_coefficient < 1).





user: josef-pkt
issue_4682: state: open
title: ENH tools resampling, bootstrap with subset restrictions
body: If we resample observation from the existing observations, then the sample might not have "nice" properties, for some definition of "nice".

examples:
- non-parametric or semi-nonparametric estimation
  If we resample uniformly, then some areas might not get enough observations, we are not able to estimate some parts well.
- categorical variables
  Resampling might cause empty cells, and singular exog. This also shows up in outlier robust feasible LTS when sampling starting observations (Its even more serious there because we use sub-sampling.)
- panel data or other grouped or clustered data if we don't resample entire groups, but want to have some observations in each group.

This is a bit similar to the actual data sampling, e.g. in treatment effects estimation, when we don't have enough `overlap`.
(And what I still don't know much about is small area estimation, i.e. small samples for some locality.)

This is also similar to survey design, except here it's just computational where the dataset is the "population".

example:
If we have a univariate non-parametric estimator, then we could split the support/observations into intervals and sample proportionally from each interval.
If we want to simulate treatment effects, then we could impose having enough overlap in the tails.

One other related area: If we need inference that depends on sampling, then we would have to look at and correct for biased sampling like importance sampling.
user: josef-pkt
issue_4681: state: open
title: ENH: low memory cache cache_readonly
body: related #2113 memory footprint

To cache or not to cache?

GenericLikelihoodModelResults caches score_obs, but score_obs could be pretty large

```
    @cache_readonly
    def score_obsv(self):
        """cached Jacobian of log-likelihood
        """
        return self.model.score_obs(self.params)
```

In general we have to decide when adding a lazy attribute whether to cache it or not, e.g. GLM caches all residual types. This has the trade-off between speed and memory, which will be different for small and large datasets.

One possibility: make caching of some intermediate results optional.
E,g. add a flag/attribute `memory_low=True` to reduce caching and recompute on repeated use. 
Or use an attribute that allows more levels, `memory_usage='low'` 'medium' 'high' 'ultra-low'
(where ultra-low could be out of memory or working in patches as much as possible)
Current would be approximately medium

This would require that the cache decorator works conditional on this flag.



user: josef-pkt
issue_4680: state: open
title: ENH: combine bootstrap with out-of-sample cross-validation
body: just an idea, I don't know if we can efficiently implement this

Simple bootstrap that samples observations requires a long loop. The idea is to combine this with cross-validation.
Simple bootstrap creates and index for the observations that are included once or several times.
e.g. GenericLikelihoodModelResults bootstrap has

```
rvsind = np.random.randint(self.nobs, size=self.nobs)
self.endog[rvsind]
self.exog[rvsind, :]
```

Is there a cheap way to get the index or mask of left out observations?

quick try:

```
nobs = 10
np.random.seed(123)
rvsind = np.random.randint(nobs, size=nobs)
y = np.arange(nobs)

mask = np.zeros(nobs, np.bool_)
mask[rvsind] = True
y_boots = y[rvsind]
print(y_boots)
print(np.unique(y_boots))
print(np.nonzero(mask)[0])
print(np.nonzero(~mask)[0])
```

```
[2 2 6 1 3 9 6 1 0 1]
[0 1 2 3 6 9]
[0 1 2 3 6 9]
[4 5 7 8]
```

aside: 
What's the expected number of left-out observations or it's distribution?
I guess: Larger nobs will have a large fraction of observations left out. What's the large sample limit?

user: josef-pkt
issue_4679: state: open
title: SUMM/REF/ENH: GenericLikelihoodModel needs updating
body: triggered by #4676 bootstrap breaks if exog is None.

AFAICS, GenericLikelihoodModel hasn't been updated in 5 years. It needs a general update to catch up with changes in the rest of statsmodels, some enhancements (cov_type='bootstrap') and there are several open issues.

two main updates to catch up with change in patter

- `_init_keys`, and `_get_init_kwds` for recreating the model. This only covers keyword for `model.__init__` while cloneattr can pick any attribute.
  - **update** init keys added to bootstrap in https://github.com/statsmodels/statsmodels/pull/7488 
- cov_type: AFAIR the inherited robust cov_types work, if the relevant methods like score_obs are specified in the subclass. We can extend this to allow for bootstrap and remove the "old fashionedly" or wrongly named covjac, covjhj and their bse versions from ResultsMixin.

- bootstrap does not permute extra init arrays, e.g. if we have weights or offset.
- bootstrap should compute cov_params and not just bse
- bootstrap start_params: we could use the `params` that were estimated by fit, 
  and remove any detour through cloneattr mentioned in code comments.

different type of bootstrap sampling schemes, e.g. residual bootstrap could be added as bootstrap/cov_kwds options.

other changes to bootstrap are a separate issue, e.g. bootstrap for hypothesis tests or confidence intervals based on pivotal statistic need separate methods and don't fit in the current model-results structure directly.

Some models as possible test cases or miscmodels cases are

- beta regression, PR still open (but it might also turn into a LikelihoodModel subclass)
- distribution fitting #2527
- simple mixture distributions #4676 

Note, beta regression can have two exog which would need a corrected bootstrap.

about the name
Why did I use GenericLikelihoodModelResults instead of GenericLikelihoodResults?
Should we still rename it? 
There is no way to make it backwards compatible during deprecation, but it will rarely be used by class name.







user: josef-pkt
issue_4678: state: open
title: ENH: add generic cov_type='opg'
body: The `ResultMixin` for GenericLikelihoodModelResults has still `covjac` for the OPG cov_params.

OPG is available in statespace models, but not for all MLE that define score_obs.
In linear regression models, like OLS, OPG is the same as 'nonrobust'.
I'm not sure, but in GLM it is most likely the same as EIM and normalized_cov_params returned by WLS in fit_irls (at least for canonical link).

It should be easy to add this to the generic `_get_robustcov_results`.


user: josef-pkt
issue_4670: state: open
title: BUG/REF: RLM cov_type is stored in model.cov not in results instance
body: RLMResults.bcov_scaled computed cov_params depending on `model.cov` which stores the cov_type.

This is not a bug that produces wrong numbers because bcov_scaled is called in in `RLMResults.__init__`
`self.cov_params_default = self.bcov_scaled`.

`summary` use `results.fit_options` which are set in RLM.fit, and therefore uses the correct results attribute.
`summary2` might not include `cov_type`, it is just the generic summary2

see #1379 for a general issue about cov_type in RLM
user: josef-pkt
issue_4669: state: open
title: ENH/REF: cov_types for penalized - TheilGLS versus PenalizedMixin
body: TheilGLS cov_type sandwich and PenalizedMixin  cov_type=HC don't agree for bse in #4576 unit tests, except for first 1 or 2 parameters.

TheilGLS "sandwich" is not "HC", it is based on
`xpxi_sandwich = xpxi.dot(xx).dot(xpxi)`

The outer part corresponds to the hessian (without scale). The inner part is the same as OLS nonrobust.
This is not heteroscedasticity robust which would need score_obs in the inner part. Without HC, the constant variance separates out and partially cancels with xpxi.
Needs checking again. I don't remember the derivation, but I think is "standard" Ridge definition.

PenalizedMixin uses the standard cov_type implementation using penalized score_obs and penalized Hessian.
Do we need a non-HC sandwich as optional cov_type?
The analogy in GLM would be cov_type dispersion, i.e. no HC but allowing excess dispersion with the sandwich. But I think that doesn't generalize to general QMLE/M-Estimors like those in discrete.
I don't remember the math here either.

RLM still uses special cov_types, but, IIRC or if I remember the derivations correctly that I did the last time I checked that, is not HC either.

We only have partially comparisons to other packages available for penalized cov_types, so large parts of it rely on having the correct derivation of the formulas.

We should get at least consistent definitions and options among our models in regression, discrete, GLM and RLM.

(The GAM PR has unit tests for cov_params against mgcv, but mgcv change bse maybe 10 times over its history including options for effective degrees of freedom, and I don't know which versions we would match.)
 




user: josef-pkt
issue_4667: state: open
title: ENH: distributed screening for ultra-high data
body: This is a follow-up to ultra-high or sure independence screening #2944 and in analogy the `distributed_estimation` but going horizontal instead of vertical in the dataset.

We can do patch processing in parallel or for out of memory data as in distributed, and then combine the patch results into a final result.

The way I have currently screening set up is by a screen_vars method that takes an array of exog. The same screener instance can be used with several arrays of exog to allow for more exog than what we want to fit in memory. Parallel computation would need separate instances or copies of the screener.

What I currently don't allow is loading data during the screening iteration. However, allowing that would not be very efficient because we have to load the data for each iteration.
The two stage process of separate pre-screening and then a final screening among the subset winners will be more efficient.
(One reason that I still might want to patch even if exog is in-memory is to cut down on the size of intermediate arrays. That however also requires a distributed/patched sort or partition, e.g. find best k variables across several patches.)

One task is to check how much of the `distributed` infrastructure we can reuse for patching variables instead of observations.

terminology: 
**ultra-high** means that the number of potential regressors is much larger than the number of observations,


user: josef-pkt
issue_4666: state: open
title: FAQ-D/REF/ENH: use `results_class` attribute in general
body: When we create a model that inherits the `fit` method, then it is convenient to have an attribute that tells the super class which results class to use for creating the results instance.
This is convenient if we have to override or add a few methods to the specific Results instance.

- #2122 was an early issue for GenericLikelihoodModels
- The discrete count models use it for ZeroInflated model, but uses `result_class` instead of `results_class`
- statespace, GMM and distributed use `results_class` based on file search
- RegressionModel.fit does a specific instance check for OLS, `isinstance(self, OLS)`
  There might be more like this with hardcoded special cases that haven't been changed yet.
- Some discrete model like Logit, CountModel have a fit method that essentially only creates the correct results instance and otherwise just delegates to the super(...).fit

We also need the results_wrapper_class, which is also used by statespace models
zero inflated count model uses 4 classes, e.g.

```
        self.result_class = ZeroInflatedPoissonResults
        self.result_class_wrapper = ZeroInflatedPoissonResultsWrapper
        self.result_class_reg = L1ZeroInflatedPoissonResults
        self.result_class_reg_wrapper = L1ZeroInflatedPoissonResultsWrapper
```

Currently I'm working on subclasses of GLM and some discrete models which should also get an option for `results_class`. Penalized #4576 currently uses the default Results class without any specific information about the penalization, which needs to be changed.

There are a few more issues or comments in other issues related to this.

So, I think we should add it as general feature for the model classes.


user: josef-pkt
issue_4665: state: open
title: REF/BUG: GLMResults.llf does not use GLM.loglike
body: `GLMResults.llf` computes loglike directly instead of going through the model loglike.

I don't think the results are currently incorrect for the standard.
It special cases the linear Gaussian case which ignores the attached scale estimate, which could differ based on some scale options.

However, this does not generalize if we use a modified loglike, e.g. in penalized models.
user: josef-pkt
issue_4664: state: open
title: ENH/REF: params_aux keys for auxiliary parameters
body: A design idea to make auxiliary parameters behave similar to params in results instance when calling model methods.

This is partially a follow-up to issues with scale in GLM loglike, score and hessian.
But I ran into a similar problem with `pen_weight` in penalized models, where pen_weight is currently a model attribute, but we want to estimate it and treat it like a parameter, i.e. use the same model with several results instances that have different `pen_weight`s.

The idea is that the results instance calls the model method, specifically loglike, score and hessian, always with the auxiliary parameters as additional keywords
For a more generic solution we would need to add a `params_aux_keys` to know which auxiliary parameters are required.

Limitation: 
This does not work for auxiliary parameters that are defined in instances of some classes attached to the model, for example parameterized 
- families like NegativeBinomial in GLM, 
- norms in RLM or 
- penalty classes in PenalizedMixin.

It most likely also will not work for special fit methods, e.g. alpha in fit_regularized. 
Or it needs to be extended to models where auxiliary params work through other paths than loglike, score and hessian.

related:
#4585 hyper parameters, this would cover some of those, those that enter loglike
#4616 problem with concentrated out scale in GLM
#3359 scale should be results attribute
#1776 fixing nuisance parameter during estimation

user: josef-pkt
issue_4647: state: open
title: ENH: recreating models with dropped or added exog (eg. null model)
body: It will become more difficult to recreate a model if extra attributes also depend on the number of exog.

Current problem: 
In penalized with vector weights or restrictions those weights vectors or restrictions multiply params.
When the null model is computed, e.g. for llnull or null_deviance, then we not only need to adjust exog, but also adjust any other attributes, like weights or restrictions that assume a given set of params.

I have a dirty workaround in my penalized PR #4576 by special casing if params is just one value, but that does not generalize, and doesn't work if we only want to penalize one param.

We can work around this for the null model by explicitly using a `get_null` method in the model.
However, we cannot recreate a model that has some values dropped.
E.g. we need to adjust the penalization and drop the parts that correspond to a dropped exog variable.

Right now I need a workaround for the llnull and null_deviance so summary()/summary2() don't raise exceptions in discrete and GLM.


user: josef-pkt
issue_4646: state: open
title: ENH: reweighting in quantile regression
body: The heart of the reweighting in quantile regression : https://github.com/statsmodels/statsmodels/blob/master/statsmodels/regression/quantile_regression.py#L174
is this block:
```
mask = np.abs(resid) < .000001
resid[mask] = ((resid[mask] >= 0) * 2 - 1) * .000001
resid = np.where(resid < 0, q * resid, (1-q) * resid)
resid = np.abs(resid)
xstar = exog / resid[:, np.newaxis]
```

First, I can't understand the special treatment for values smaller than 1e-6 
Second, can't this be simplified using np.maximum? The pinball loss at x is the maximum of qx and (q-1) x

```
resid = np.maximum(q * resid, (q - 1) * resid)
xstar = exog / resid[:, np.newaxis]
```

Happy to send a PR if this looks ok
user: mathurinm
issue_4642: state: open
title: REF: internal API change in penalties. Penalty classes
body: the Penalty class has currently alpha and wts as argument or attributes.

In my penalized PR I used pen_weight instead of alpha in the model, and full name `weights` instead of wts.

AFAICS based on a file search, the penalty classes are currently only used in `mixed_linear_model` and the corresponding unit tests.

renaming `alpha` in `fit_regularized` would be an external backwards incompatible change.

The separation between alpha/pen_weights and weights is not clear or consistent because alpha, as specified by users in fit_regularized can also be a vector that then corresponds to `alpha * weights` or `pen_weight * weights`.
In general, we need the full weights vector (if needed) and some weight parameter like alpha or pen_weights that are chosen by cross-validation or similar. I.e. `weights` are fixed, but `pen_weights` or `alpha` are part of the model search.
user: josef-pkt
issue_4641: state: open
title: statsmodels.graphics.functional.banddepth uses factorial() for normalisation
body: Factorial() will fail for relatively small sample sizes, resulting in NaNs for the banddepth. A different method would allow larger sample sizes to be used.
user: chummersone
issue_4631: state: open
title: degree of difference to be 2
body: I need to fit ARIMA(12,2,1), but it gives  me error.

```
 #Fit ARIMA(12,2,1) on train data and find parameters

model = ARIMA(train, order=(12,2,1))

model_fit = model.fit(disp=0, start_params=model_fit0.params)

#predict train data

print('Forecasting Training Data') 

pred_train = model_fit.predict(start=1,end=size,typ='levels')

MSE_train = mean_squared_error(train, pred_train)

RMSE_train = sqrt(MSE_train)

MAE_train = mean_absolute_error(train,pred_train)

MAPE_train = MAPE(train,pred_train)

```

the error message is 
ValueError: The start index -1 of the original series has been differenced away
user: 2M-kotb
issue_4625: state: open
title: SUMM/ENH/REF: performance review of GLM, fit_gradient
body: There is still some slack in GLM, especially fit_gradient, to improve performance in terms of speed and memory consumption

related issues:

- #4622. PR #4624 redundant/unused pinv_exog,  will be merged soon
- don't compute scale during irls fit isssue ????
- #4623 better start_params for fit_gradient if irls is not used
- #3772 general idea for a fit_cache

...

other possibilities:

- irls start_params for fit_gradient only needs params:
  - we can copy params and delete `irls_rslt` immediately to free up memory
  - irls_rslt don't need to compute cov_params, especially not a robust cov_type
  - it should be possible to use `skip_hessian` or a return option to only get the params out of the irls fit
- irls: add option to use moment matrices
- memory usage in fit_irls

user: josef-pkt
issue_4623: state: open
title: GLM newton optimization and no start_params
body: One issue that I always had with the `newton` is that I needed to set either `start_params` or set `max_start_irls` > 0 or newton optimization would fail. 

OIM seems to be more robust (with the OIM fix in #4620) now than before, to the point that when you have a model with non-canonical link (as is common with me), OIM feels like its now able to deal with a poorly selected `start_params`.

See this gist that shows OIM successfully fitting, but EIM failing.

https://gist.github.com/thequackdaddy/1875975af44281adb98db9045b52d8da

I believe when `start_params` are not provided and `max_start_irls=0`, the default is to make `start_params=np.zeros(model.exog.shape[1])`. IRLS, on the other hand, generates a starting `mu` equal to the mean of endog.

So here's a question. Is there a way to come up with better start params? 

Perhaps there is a very compuationally cheap algorithm to pick start_params in some way so that `model.family.link(np.dot(model.exog, start_params))=model.endog.mean()`? 

From a quick experiment, if I set my constant column in `exog` so that the first guess of `lin_pred=mu` , it does converge. Is there a good algorithm for this?  
user: thequackdaddy
issue_4621: state: open
title: WIP: Automatic forecasting GSoC-2018-Project
body: @ChadFulton I have only pushed the function definition.  We can discuss the improvements/ additions over here.
user: abhijeetpanda12
issue_4619: state: open
title: REF: Irrelevant state in UnobservedComponents freq. seasonal when s even
body: For the group of frequency domain seasonal components, the transition matrix is block diagonal, where the blocks are:

```
[[cos(L_j),  sin(L_j)],
 [-sin(L_j), cos(L_j)]]
```

where `L_j = np.pi * j * (2 / s)` and `j = 1, 2, ..., [s/2]`. Now, when `s` is even, the last iterate `j = (s / 2)` so that `L_j = np.pi`. Then the blocks are:

```
[[-1,  0],
 [0,  -1]]
```

Now, for each block, the corresponding block of the design matrix is `[1, 0]`.

The upshot is that when `s` is even, the last state is not selected by the design matrix and evolves as an independent random walk - i.e. it is irrelevant and should not be included. For some reason Durbin and Koopman (2012) don't explicitlymention this, but see e.g. Proietti (2000).

(This doesn't cause any problems right now, since we are using approximate diffuse initialization, but when we switch to exact diffuse initialization it will cause the filter to never switch out of the exact diffuse phase, since no information informs that irrelevant state).
user: ChadFulton
issue_4612: state: open
title: TST/REF: collection of artificial datasets for testing
body: We should collect somewhere a list of functions to create artificial, simulated datasets for the unit tests.

We have our datasets that are used in many unit tests, and we use random generated datasets in other unit tests. However, we don't have a central location for these artificial datasets.

Related: for nonparametric and semi-parametric (spline, polynomial) fitting we also have nonlinear functions to generate artificial datasets. (I thought there was an open issue for those from many years ago, but don't find it.)

The advantage of artificial datasets in contrast to real datasets is that artificial datasets are "nice" for specific problems because we can control the data generating process. Our real datasets are often good for detecting problems in "not-nice" cases, but are also often not fast to estimate.

The current pattern is often to copy-paste the setup.py of a test class to create the model, which is extra work but could have the advantage of fine tuning the created datase.
Kerby and thequackdaddy used a collection of GLM models in the unit test for testing equivalence across models or model options.

Right now I need something for handling many models for generic_methods, fit_penalized, fit_regularized, fit_colinear, ...
E.g. GLM-Binomial count model has been neglected in many generic tests. Start98 dataset used in the GLM test is not very good for testing penalized, two many exog, all are significant in unpenalized model.



user: josef-pkt
issue_4611: state: open
title: CompareMeans.summary() still couldn't work well with 1-D array
body: The same problem described in [https://github.com/statsmodels/statsmodels/issues/3783](url) happened again when i was trying to use CompareMeans.summary().
Here's my code:
`import numpy as np
import statsmodels.api as sm
import pandas as pd 
sample1 = np.random.normal(loc=3,scale=0.1,size=10000)
sample2 = np.random.normal(loc=3.3,scale=0.2,size=10000)
sample1_ext = sm.stats.DescrStatsW(sample1)
sample2_ext = sm.stats.DescrStatsW(sample2)
sm.stats.CompareMeans(sample1_ext,sample2_ext).summary(use_t=False,alpha=0.05,value=0)
`
And there comes the error: 
~/anaconda/lib/python3.6/site-packages/statsmodels/iolib/summary.py in <listcomp>(.0)
    468     exog_idx = lrange(len(xname))
    469 
--> 470     params_data = lzip([forg(params[i], prec=4) for i in exog_idx],
    471                        [forg(std_err[i]) for i in exog_idx],
    472                        [forg(tvalues[i]) for i in exog_idx],

IndexError: invalid index to scalar variable.


user: JackZYSun
issue_4610: state: open
title: ENH/Design: support for unpenalized exog in fit_penalized, fit_regularized
body: related to #4608 better start_params for fit_regularized in GLM.

A common (non-machine learning) usecase for penalization/variable selection is to regularize or penalize extra variables that are not the main variables of interest. 
In these cases we have a some exog that are not penalized.

The unpenalized exog can be used to get good starting values using the standard non-penalized model, E,g, in GLM we need the IRLS weights that depend on a good initial estimate on mu.
For ultra-high screening I used the resid_pearson of previous estimates to get a measure of the **additional** contribution of candidate variables to add to the model. (related would be score/LM test base on an initial model.) (IIRC I have one version of ultra-high screening (somewhere?) where users can explicitly specify a set of unpenalized, always to keep exog)

specifically: 
GLM.fit_regularized alpha can be a vector with zeros for unpenalized exog. We can use the zero mask to create an intial unpenalized model for the start_params and set the start_params of the remaining variables to zero.
Similarly, my penalized/SCAD PR #4576 could use the same approach (not implemented yet)
user: josef-pkt
issue_4609: state: open
title: ENH: penalties Penalty function: elastic net and penalty addition
body: Penalty classes in base._penalties do not contain features to add penalty terms, e.g. L1 + L2 as in elastic net.

e.g. I guess we can use smoothed-L1 plus L2-Ridge to get the scipy optimizer version of elastic net for in memory models where k_exog is not too large.

user: josef-pkt
issue_4608: state: open
title: ENH: GLM.fit_regularized needs better default_params
body: default_params in elasticnet is `np.zeros(k_exog)` which has the wrong initial estimate for mu in the first few rounds.

convergence looks very slow based on the time and number of iterations it takes, if the number of non-zero coefficients is large (k_exog=21 in Star98)
But I don't know how large the initial steps are in the convergence history.
Optionally storing the params convergence history would be useful
user: josef-pkt
issue_4606: state: open
title: DOC: GLM.fit_regularized docstring doesn't show default values
body: GLM.fit_regularized  uses `**kwargs` for some penalization keywords but doesn't show what the defaults are.
e.g. it is not visible what L1_wt default is

```
defaults = {"maxiter": 50, "L1_wt": 1, "cnvrg_tol": 1e-10,
                    "zero_tol": 1e-10}
```
user: josef-pkt
issue_4600: state: open
title: BUG/ENH: fit_constrained with GLM-Binomial
body: consequence of #4599 Binomial with counts stores transformed endog.

fit_constrained with GLM-Binomial counts is most likely incorrect.

current unit tests
- genmod/tests/test_constrained Gaussian only (added to fix scale handling for cov_params)
- discrete/tests/test_constrained Poisson and Logit for both discrete models and GLM  
  (uses GLM-Binomial with binary endog.)



user: josef-pkt
labels: type-bug-wrong: serious bugs that silently return incorrect numbers
issue_4599: state: open
title: BUG: recreating Binomial model instance with n_trials != 1
body: https://github.com/statsmodels/statsmodels/issues/4597#issuecomment-386888049

GLM Binomial attaches a transformed endog to the model. This means that we cannot use our standard pattern for recreating a GLM Binomial model (unless it's a Bernoulli case).

There might be related bugs, where we use recreated models, e.g. most likely in fit_constrained and similar.
I don't know, have not checked yet, how MICE or other prefix methods handle this.
The case where I found this is in `GLMResults.null` which looks incorrect to me for Binomial, although there are unit tests for null_deviance.

Attaching a transformed endog is not a good design and it had me confused in several other places.

It would be better to use a different name, like wendog, in OLS/WLS. However, that would be a backwards incompatible refactoring for users that access the underlying model.endog.

Alternatively we can use a specific fix just for recreating a model, i.e. recover the original endog.



user: josef-pkt
issue_4598: state: open
title: DOC/TST/Review: binomial with non-canonical link 
body: https://github.com/statsmodels/statsmodels/issues/4597#issuecomment-386881123


`test_glm.py` has a comment, which I don't know what it means, or what's the reason for the comment
```
#TODO:
#Non-Canonical Links for the Binomial family require the algorithm to be
#slightly changed
```

I never looked at this case, i.e. Binomial count model with n_trials != 1, not Bernoulli.

The link function for the binomial probability can be anything. 
The question is whether the current code uses some computation that assumes canonical link, i.e. logit.
The only source could be in the aggregation for n_trials which is the only difference to other families in the implementation.

e.g. related: log-binomial is an example that we looked at, but only with respect to params and possible problems with support violations. ( #2208, #2215)

There might be nothing wrong in this case, but it needs checking.



user: josef-pkt
issue_4591: state: open
title: DOC/TST/ENH/Design:  using scipy contrained optimizers in regular models
body: I think this is a functionality that we inherit now from scipy through the minimize fit method.
(I might have seen an example where someone uses it.)

We don't have any examples and not verification for it.
E.g. what happens to inference with inequality constraints. 

Linear and nonlinear equality constraints are theoretically fine, but I don't know if we would have correct cov_params in that case.

Needs examples to check, document, test, and warn for things that work automatically, but don't produce correct results.

user: josef-pkt
issue_4590: state: open
title: ENH/Design/FAQ-D: location of penalized functions and classes
body: Do we need a new a new directory `statsmodels.penalized`?

specifically if we have user facing standalone functions

My plan was to spread out penalized versions of existing models and functions so they are next to the current models and functions. But maybe that doesn't work for everything.

This is similar to what to do about robust methods where we already have statsmodels.robust. The main reason to keep penalized and robust close to existing models is to avoid putting them in a separate corner where they will be ignored by many users, e.g. as the current separation of them in text books.

maybe rename statsmodels.gam to statsmodels.penalized

components

- fit_regularized Those are methods of the models with support objects in base
- penalized statistics: e.g. shrinkage covariance
- GAM: currently in separate folder in PR
- generic penalized models, e.g. PoissonPenalized, GLMPenalized as in my PR, with supporting objects in base
- TheilGLS (Ridge) currently in sandbox.regression 
- standalone function? e.g. ultrahigh screening (design not clear yet, currently a function that takes an existing model, and may be a bit similar to MICE as prefix class.)

at some point I started a folder `datarich`  in the sandbox but that name didn't catch on (not even in econometrics)

also related: 
Variable selection where the emphasize is on selection and not on penalized parameters. Those should essentially be combined with penalization because penalization is one major method to select. (L1, SCAD)

user: josef-pkt
issue_4585: state: open
title: SUMM/Design/ENH: API and structure for hyper parameter selection
body: What should be our long term pattern for hyper parameter selection?

examples: lag selection in tsa models, penalty in penalized models.

tradeoff: clean, more "fool-proof" design versus computational efficiency

Our old pattern was to add those hyper parameters to fit, e.g. some tsa models still have lag selection "ic" in fit. This can potentially take advantage of computationally efficient search without creating new models each time.  
Our newer pattern is to put hyper parameters in the `model.__init__` to avoid incorrect state, attributes in models. This requires that a new model is created for each set of values for the hyper parameters. This adds additional overhead and cannot take advantage of unchanging intermediate results.

**lag selection:**
There are some open issues for lag selection in tsa which still needs refactoring, e.g. #4536 for VAR

**example penalization:**  
In penalized or regularized regression, the underlying data, endog and exog, does not change.

In #4576 I avoided the issue of a proper API so far, and make penalization weights as a model attribute, but I change the attribute before running fit. It is essentially a `__init__` keyword that can be manipulated as a fit keyword. but will cause incorrect stale state if several results instances refer to the same model.

TheilGLS, and current `fit_regularized` model methods specify penalization parameters in a `fit` method. However, in those cases there should not be any penalization attributes attached to the model instance. 
However, this requires that there are separate methods that are used by fit_regularized, so un-regularized methods do not need to check model attributes about the penalization.
This does not work if, for example, loglike is used for penalized and unpenalized estimation.
In #4576, I added a keyword for the penalization to loglike and similar, but it is not used because the currently standard results class (not specialized for penalized fit) does not know about those extra keywords.
So if we want to avoid model attributes that change during hyper parameter search, we would have to attach them to the model, and use them as arguments in any model calls.
AFAIR, we do this now (0.9) for scale in GLM to avoid needing scale as model attribute.

**fit_constrained:**
There might be a similar problem in fit_constrained, when constraints are considered as hyper parameters. I don't remember the details right now, or never looked at it specifically in terms of location for attaching attributes.
The current fit_constrained is somewhat different because it works by creating a new auxiliary model.
An alternative that we don't widely use yet is to add constrained optimization. Example is imposing stationarity in ARMA variants, or in miscmodels TModel that uses the generic reparameterization for imposing constraints. 
However, constraints on parameters during optimization are simpler issue, because once we have the params, then we don't need to consider the constraints anymore, i.e. transform_params is internal to `fit` and an attribute that changes only temporarily.
For imposed binding constraints, the constraints need to be taken into account in the model and results.


user: josef-pkt
issue_4582: state: open
title: REF/ENH vif variance inflation factor and feature selection
body: I have fixed `variance_inflation_factor` function and `feature_selection_vif` function.

In `variance_inflation_factor` function,
add `add_constant` argument to choose explicitly.

In `feature_selection_vif` function,
All `print` functions were deleted as you mentioned before.
user: pydemia
issue_4579: state: open
title: SUMM/ENH: tools and helpers for penalized models
body: subtopic of #4038

(I don't have an overview where the different pieces use for penalization are.)

specifically: Where is a quadratic penalty function, i.e. normal prior?

**Penalization classes**
main (intended) location: `base._penalties` but doesn't have much yet.
#4576 adds SCADpenalty

GAM #4575 adds new directory that includes the penalized spline tools
`statsmodels/gam/gam_penalties.py`

**Restriction/contrasts**
- some things are next to sandbox/regression/penalized TheilRidge:   
   coef_restriction_meandiff, coef_restriction_diffbase and coef_restriction_diffseq
- t_test_pairwise function is in stats/contrasts with some helper functions for pairwise contrasts  
   _contrast_pairs, _embed_constraints, _constraints_factor which uses some contrast functions from
   sandbox.stats.multicomp (contrast_allpairs, and others are also there)
- fit_constrained : I don't see helper functions right now, but it includes the string specification for restrictions. fit_constrained and penalized fit should have a similar usage API for equality and stochastic restrictions respectively. see also #4568 (wild card constraints)
- We need an API to make an instance of a Penalization class with desired constraints. Those instances could be defined by users and provided to the penalization model class, similar to family in GLM. That's for cases where users want to have more control. 
(e.g. I think MICE or initial GSOC version specifies default or allows fine tuning for which imputation models should be used. Similar for specifying `norm` in RLM.)


...

TheilGLS uses arguments `r_matrix=None, q_matrix=None, sigma_prior=None, sigma=None`
which corresponds to prior
`R * params ~ N(q, sigma_prior)`, 
and likelihood for normal linear GLS model
`y ~ N(X params, sigma)`
except we don't rely on the normality assumption in GLS and Quasi-MLE, nor on a Bayesian interpretation.
One difference to full Bayesian version is that the prior can be incomplete, i.e. `rank(R) < k_params` or k_rwos < k_columns, and we have unrestricted parameters or unrestricted linear combinations.


user: josef-pkt
issue_4577: state: open
title: DOC document tools StandardizeTransform
body: just found by chance browsing old PRs. I didn't remember I had written this class.
The mainly internal tools and helper function are not well documented and not easily discoverable.

StandardizeTransform in statsmodels/tools/transform_model.py 
(we have transformation functions and classes spread out over different areas for various use cases.)

initially written to fix test failure
general issue #1131

possible usage e.g. in getting better start_params




user: josef-pkt
issue_4570: state: open
title: ENH: tukeyhsd, pairwise comparison from summary statistics
body: https://stackoverflow.com/questions/50070927/anova-table-and-pairwise-test

tukey-hsd based on mean and var/std as sufficient statistics.

I have not looked at it, but this should be made possible and easy to add, for one way anova

related 
#4379 tukey-hsd improvements
#4323 discussion of two way anova

user: josef-pkt
issue_4568: state: open
title: ENH: define constraints by strings with wild cards
body: a general idea

see also related, more general issue #4361 

we need constraints or contrast matrices in several places, 

- original definition is in t_test and similar.
- test_wald_terms uses some string matching to construct constraints
- fit_constrained uses the same pattern, IIRC
- penalized fit needs restriction matrices to define *stochastic constraints*

proposal:

- I think we can create a standalone function following the t_test pattern.
- It would be convenient to allow for wild cards as in Stata, 
  e.g. "region*" for all parameters that encode a categorical variable
- The latter could by also used with sparse matrices, e.g. as in variance components in mixed models, or variation on sparse OLS and sparse penalized models.

`get_constraints(constraints_string, param_names)`
or some similar name

possible extension to automatic constraint patterns
e.g. all pairwise #4365
user: josef-pkt
issue_4567: state: open
title: SUMM/ENH merging penalized mle, GAM
body: see #4038 for more general issue for penalized models

we have long standing open PRs (from 2015, not yet rebased)
general penalized MLE and structured L2 for specialized models

- #2374 my original penalized MLE Mixin
- #2744 GAM, original PR is #2435

related 
- #1665 merged PR TheilRidge in linear model

related mixed models
- MixedLM
- BayesMixedGLM for Poisson and Binomial
- no PR for my draft versions for Poisson using halton sequences ?

unit tests, other stats packages

- ???
- GAM has unit tests against mgcv
- TheilRidge has unit tests against 
- cross-checking using equivalence and overlap across models (subject to different definition or estimation of penalization weight, prior weights or random effects variance)



user: josef-pkt
issue_4564: state: open
title: ENH: Multivariate Markov Switching Models
body: The `statespace` component already has an excellent module for fitting Markov switching models; however, only univariate models are supported. Is there a way to extend it so as to support multivariate models? Thanks!
user: haginile
issue_4558: state: open
title: test_arima.test_arima_dates_startatend comment needs clarification
body: ```
def test_arima_dates_startatend():
    # bug
    np.random.seed(18)
```

`# bug` is not a helpful comment.  This needs clarification, ideally with a GH reference to the relevant bug.
user: jbrockmendel
issue_4554: state: open
title: ENH: options for cholsigmainv in GLS
body: just a quick reminder, I don't remember right now

merged PR #4218 has some discussion on possible improvements for GLS

user: josef-pkt
issue_4553: state: open
title: TestCoint_t ambiguous expected answer
body: https://github.com/statsmodels/statsmodels/blob/1309f1519f27341b72e238b21aa066017ef19abc/statsmodels/tsa/tests/test_stattools.py#L279

```
    @classmethod
    def setup_class(cls):
        #cls.coint_t = coint(cls.y1, cls.y2, trend="c")[0]
        cls.coint_t = coint(cls.y1, cls.y2, trend="c", maxlag=0, autolag=None)[0]
        cls.teststat = -1.8208817
        cls.teststat = -1.830170986148
```

First, the commented-out line should either be documented or deleted (hint: deleted).

Second, there should not be two teststat attributes set here.  I'd assume the -1.83. supplants the -1.82, but have no idea why or if it can be trusted.
user: jbrockmendel
issue_4550: state: open
title: SUMM/ENH/REF: Distributed follow-up
body: AFAICS, we don't have an issue yet, and we haven't done any post merge review.

Distributed estimation was merged in #3396

current problem (for 0.9):

notebook in examples doesn't run, and doesn't explain anything.
I don't know which results we should look at.

I just realized  I'm on an outdated branch 
#4261 fixed the notebook example

accessing params works
`predict` works
I don't see working inference, cov_params in the example results instance




user: josef-pkt
issue_4549: state: open
title: REF: Wrapping "columns"/"columns_eq"/"rows"/"rows_eq"
body: Notes:
   "rows_eq" referenced in the subject does not currently exist
   This discussion assumes we're using PandasData.

`MultinomialResults.params` is currently wrapped incorrectly.  It returns a DataFrame that (correctly) has the exog_names as its Index, but incorrectly has numbered columns instead of named columns.  The following example is based off of test_discrete.test_mnlogit_factor

```
dta = sm.datasets.anes96.load_pandas()
dta['endog'] = dta.endog.replace(dict(zip(range(7), 'ABCDEFG')))
exog = sm.add_constant(dta.exog, prepend=True)
mod = MNLogit(dta.endog, exog)
res = mod.fit(disp=0)
>>> res.params
                 0         1         2         3         4          5
const    -0.373402 -2.250913 -3.665584 -7.613843 -7.060478 -12.105751
logpopul -0.011536 -0.088751 -0.105967 -0.091557 -0.093285  -0.140881
selfLR    0.297714  0.391669  0.573451  1.278772  1.346962   2.070080
age      -0.024945 -0.022898 -0.014851 -0.008681 -0.017904  -0.009433
educ      0.082491  0.181043 -0.007152  0.199828  0.216939   0.321926
income    0.005197  0.047874  0.057575  0.084498  0.080958   0.108894
```

The columns here _should_ be `['B', 'C', 'D, 'E', F', 'G']`.  Ideally in fact, it would be `pd.Index(['B', 'C', 'D', 'E', 'F', 'G'], name='PID')` so the name of the original endog column is retained.

The natural analogue for this is VARResults, which uses `columns_eq` for params (and tvalues, pvalues, bse, which all have the same wrapping issue in `MultinomialResults`).  Fixing this in MNResults isn't entirely trivial because MultinomialModel sets `self.data.ynames` to just the string "PID" and defines other attributes as a workaround for `summary` and `summary2`.

I'll avoid suggesting solutions so they don't get reflexively FUDed to death.
user: jbrockmendel
issue_4548: state: open
title: ENH: VAR IRF and FEVD - bootstrap confidence intervals
body: VAR has Monte Carlo simulation error bands for IRF that do not have unit tests.

According to the documentation of Stata's `irf create` Stata uses a parametric residual bootstrap, i.e. estimated residuals are used in simulating the VAR.
`VARResults.irf_errband_mc` uses draws from the multivariate normal distribution in the MC simulation.

So I guess we cannot use Stata's bootstrap confidence interval for unit testing our mc confidence intervals when using actual data that might not be close enough to normal distribution.
But the bootstrap intervals might still be closer to the mc intervals than mc intervals are to asym intervals.



user: josef-pkt
issue_4547: state: open
title: REF/Design VAR difference in parameterization of trend from exog
body: In unit tests for #4538 
https://github.com/statsmodels/statsmodels/pull/4538/files#diff-0b3676a023d33c575bc092d6820c4269R713
I compare trend='ct' with specifying part of the trend through exog.

The versions that include exog agree at high numerical precision, but specifying trend produces a estimate of the constant/intercept that only agrees at around rtol=5e-3. the linear trend component still agrees at high precision (around 2e-13).

Is there a difference in how constant is specified or estimated?

VAR._estimate_var has a comment that sounds strange for intercept, otherwise, AFAICS, it just compensates for the trimming.
```
        # the following modification of z is necessary to get the same results
        # as JMulTi for the constant-term-parameter...
```

Given that the unit test pass, this might be a question of how intercept is defined and not a bug.
But it's still strange, because simple linear algebra is supposed to agree at better than 1e-12 or 1e-14.
Also, irf and forecast agree across equivalent models and better than 1e-12 (in unit tests), so the difference in the intercept params is somewhere compensated for. 
user: josef-pkt
issue_4546: state: open
title: REF: VAR IRF avoid repeated run of MC error bands
body: When we want to get the numbers for the irf error band and plot it, then we need to run the MC simulation twice, i.e. irf plot and irf errorband_mc both use the MC, but there is no way to share the simulation results.

One possibility is to memoize/cache the last mc error bands for given options.
Another possibility is to let the user hold on to the error bands and provide it to the plot function.


user: josef-pkt
issue_4545: state: open
title: REF: VAR IRF plotting, use asym confidence interval instead of standard error
body: vector_ar IRAnalysis does not have a `errband_asym` method. Instead it computes the standard deviation from `cov`, and hands it all the way into the plotting function.

So after going through several methods and functions, the conversion of se to confint is in function `plot_with_error`
```
    if error is not None:
        if stderr_type == 'asym':
            q = util.norm_signif_level(alpha)
            plot_action(y - q * error, error_fmt)
            plot_action(y + q * error, error_fmt)
        if stderr_type in ('mc','sz1','sz2','sz3'):
            plot_action(error[0], error_fmt)
            plot_action(error[1], error_fmt)
```

proposal:
add `errband_asym` and kick out all the special casing in the plot helper functions.
user: josef-pkt
issue_4544: state: open
title: MNLogit.from_formula loses labels in categorical
body: In test_mnlogit_factor:

```
    dta = sm.datasets.anes96.load_pandas()
    dta['endog'] = dta.endog.replace(dict(zip(range(7), 'ABCDEFG')))
    # [...]
    mod = MNLogit.from_formula('PID ~ ' + ' + '.join(dta.exog.columns),
                               dta.data)
    res2 = mod.fit(disp=0)
```

Looking at e.g. res2.summary() you'll see headers "PID=1", "PID=2", ... "PID=6", but those should be (and are in the non-from_formula variant) "PID=A", "PID=B", ..., "PID=G"

This also shows up in a bunch of other things, e.g. `predict`.  (params gets wrapped incorrectly for both from_formula and non-from_formula cases)
user: jbrockmendel
issue_4543: state: open
title: ENH: multivariate WLS
body: (I never looked at this)

What model can we use instead of MultivariateOLS when we have changing variance, cov_resid?

example: I'm using the growthrate (diff log) of the macro dataset in VAR and the variance looks smaller in the second half of observations, i.e. we would have, for example, a structural break in the cov_resid.

The structure would be a bit similar to multivariate garch, except we have a simpler, non-autoregressive structure for the cov_resid.

Can we just rescale/whiten the endog and exog as in WLS?
Can we use HC cov_type?

Is MultivariateWLS still equivalent to WLS equation by equation?   like SUR with identical exog and OLS.

(My guess is yes to all, at least for changes in variance and not cross-correlation. So it shouldn't be too difficult.
The literature on multivariate GARCH has more discussion about assuming different patterns for changes in cov_resid.
)

user: josef-pkt
issue_4542: state: open
title: BUG/ENH: initvalues in varsim
body: vector_ar.utils.varsim has `initvalues` option but those are not used.
`varsim(coefs, intercept, sig_u, steps=100, initvalues=None, seed=None)`

VARResults uses varsim in irf_errband_mc and irf_resim
VARProcess has a plotsim method, and I am adding a `simulate_var` method.

I think we need an option to condition on the observed initial observations `initvalues = endog[:p]`

initvalues are currently set to zero, i.e. they don't take a non-zero intercept into account
I'm changing this to use offset/intercept also for initial period, 
However, I think we should use long run intercept under stationarity assumption with unconditional mean for initial observation. I'm not sure how we would add an unconditional effect of historical exog, i.e. even before the p initial conditions.
I think if we condition on initial endog, then we can just use the intercept, trend and exog from the VAR autoregression, and we don't need to initialize with a long run, unconditional expectation. Conditioning on observed initial endog is more important in the exog case,, when we are restricted to simulating the same periods as the data because for those we have the exog observations available.

This is currently not really that important because in most cases for simulations we drop periods for warmup/burnin as is done for example in the irf_xxx simulation.
user: josef-pkt
issue_4541: state: open
title: MultinomialResults.predict() doesn't wrap pandas
user: jbrockmendel
issue_4536: state: open
title: REF: VAR outsource var lag selection
body: VAR has too many choice in `fit` that should be in `__init__`
one of them is lag selection `ic=None`

it's partially modularized because it uses a separate select_order method which uses the `_estimate_var` method.

AFAICS, the lag selection uses different number of observations, i.e. trims initial observation by actual lag instead of by maxlag..
AFAICS, lag_selection doesn't reuse any of the datamatrices, i.e. we repeatedly start from scratch
`z = util.get_var_endog(endog, lags, trend=trend, has_constant='raise')`

possible alternative
Create VAR with order `lags = maxlags` and then select the relevant parts of the data matrices, trimmed at common maxlag. Then recreate a VAR with the selected number of lags.


user: josef-pkt
issue_4535: state: open
title: REF/BUG: exog, trend, ... in VAR
body: (some notes, several other open issues) 

exog support in VAR is incomplete
currently intercept or trend is not added to exog.
VARProcess does not directly have information about the coef composition.

coeffs in VARProcess and VARResults is split out and reshaped from params.

forecast needs the full prediction and all components
some other process methods (need to) ignore exog/trend/...

VAR model has predict method that looks similar to other TImeSeriesModels, but doesn't handle exog (bug). 
There is no predict method in VARResults or VARProcess. Those two classes do not subclass any TSA model, i.e. doesn't have direct access to forecast date handling.

Explicit handling of trend and seasonal components has the advantage that we can automatically construct out of sample "exog" values, which we cannot do for user provided exog. But otherwise it would be more convenient to keep trend and other "exog" in `exog`.

TBC

right now I would just like to fix the missing intercept in VARProcess. (but I don't see my way through this yet.)

user: josef-pkt
issue_4528: state: open
title: Special-Casing in discrete_model fit re-calling get_robustcov_results
body: I'd like to clarify the logic around why/when calls to get_robustcov_results are made in discrete_model.

The obvious one is in `DiscreteResults.__init__`.  That should be calling `self._get_robustcov_results instead of `covtype.get_robustcov_results`, but whatever, there should always be as many ways to do something as possible amirite.

The other calls are in NegativeBinomialP.fit, NegativeBinomial.fit, and GeneralizedPoisson.fit.  It looks like the point of these is that if `_transparams` is being applied then the `cov_params` calculated during the `super(...).fit` call will be wrong, so `get_robustcov_results` needs to be re-run.

But the extra calls to get_robustcov_results are being made unconditionally, not just in the transparam cases.  What's the reasoning here?
user: jbrockmendel
issue_4525: state: open
title: SUMM/ENH: generic support for 2D params
body: I don't find a SUMM issue for it, There are issues for individual pieces that are broken.

what we need

- raveled params, e.g. `params_ravel` attribute, ravelling and names need to be consistent
- names: combined names that can be used in t_test and similar to specify constraints as strings and to get through patsy `transform`, e.g. "kron"(ynames, xnames)
- multi-index for pandas wrapping, same content as "names" but as a multi-index
- pandas wrapper code for things with raveled params, e.g. cov_2d (already exists ?), conf_int_2d,
   PandasData already has a `attach_generic_columns_2d` that might be a possible generic candidate (but I haven't looked for it's usage yet)
- base methods that need to call params_ravelled and combined "names", e.g. t_test, f_test, ....

test cases that need fixing or extensions
- MNLogit
- MultivariateOLS (currently as version with limited functionality)
- VAR (later because it needs more changes and doesn't inherit LikelihoodModelResults


user: josef-pkt
issue_4522: state: open
title: REF: consistent naming: results_frame, summary_frame, table
body: There is some inconsistency in the results DataFrame after various t_test, wald_test and other Results methods.

t_test return has `summary_frame` method
t_test_pairwise has `result_frame` attribute
wald_test_terms has `table` attribute and `summary_frame` method without options.
OLSInfluence has `summary_frame` method without options.
get_prediction (OLS, GLM) PredictionResults has `summary_frame` method with options

There should be a few more outside of tsa.   
I didn't check tsa. 
user: josef-pkt
issue_4521: state: open
title: FAQ-D: Why -0.1 in GeneralizedPoisson, 0.5 in NegativeBinomial and NegativeBinomialP?
body: GeneralizedPoisson
```
        if start_params is None:
            offset = getattr(self, "offset", 0) + getattr(self, "exposure", 0)
            if np.size(offset) == 1 and offset == 0:
                offset = None
            optim_kwds_prelim = {'disp': 0, 'skip_hessian': True,
                                 'warn_convergence': False}
            optim_kwds_prelim.update(kwargs.get('optim_kwds_prelim', {}))
            mod_poi = Poisson(self.endog, self.exog, offset=offset)
            res_poi = mod_poi.fit(**optim_kwds_prelim)
            start_params = res_poi.params
            a = self._estimate_dispersion(res_poi.predict(), res_poi.resid,
                                          df_resid=res_poi.df_resid)
            start_params = np.append(start_params, max(-0.1, a))
```

NegativeBinomial
```    
    if start_params is None:
            # Use poisson fit as first guess.
            #TODO, Warning: this assumes exposure is logged
            offset = getattr(self, "offset", 0) + getattr(self, "exposure", 0)
            if np.size(offset) == 1 and offset == 0:
                offset = None
            optim_kwds_prelim = {'disp': 0, 'skip_hessian': True,
                                 'warn_convergence': False}
            optim_kwds_prelim.update(kwargs.get('optim_kwds_prelim', {}))
            mod_poi = Poisson(self.endog, self.exog, offset=offset)
            res_poi = mod_poi.fit(**optim_kwds_prelim)
            start_params = res_poi.params
            if self.loglike_method.startswith('nb'):
                a = self._estimate_dispersion(res_poi.predict(), res_poi.resid,
                                              df_resid=res_poi.df_resid)
                start_params = np.append(start_params, max(0.05, a))
```

NegativeBinomialP
```
        if start_params is None:
            offset = getattr(self, "offset", 0) + getattr(self, "exposure", 0)
            if np.size(offset) == 1 and offset == 0:
                offset = None

            optim_kwds_prelim = {'disp': 0, 'skip_hessian': True,
                                 'warn_convergence': False}
            optim_kwds_prelim.update(kwargs.get('optim_kwds_prelim', {}))
            mod_poi = Poisson(self.endog, self.exog, offset=offset)
            res_poi = mod_poi.fit(**optim_kwds_prelim)
            start_params = res_poi.params
            a = self._estimate_dispersion(res_poi.predict(), res_poi.resid,
                                          df_resid=res_poi.df_resid)
            start_params = np.append(start_params, max(0.05, a))
```

These are identical except for the `if self.loglike_method.startswith('nb')` and the fact that GeneralizedPoisson uses `max(-0.1, a)` where the others use `max(0.05, a)`.  What's the reasoning behind the constants -0.1 and 0.05?  (all three classes use `np.append(start_params 0.1)` in `fit_regularized`, and the 0.1 constant should probably also be documented)
user: jbrockmendel
issue_4517: state: open
title: _get_start_params_null for count_model?
body: Poisson, GeneralizedPoisson, NegativeBinomial, and NegativeBinomialP could all share an implementation of `_get_start_params_null`, but before that implementation can go in CountModel, it needs to be extended to be valid for GenericZeroInflated subclasses (or overridden by GenericZeroInflated).  Is there a reasonable implementation available?
user: jbrockmendel
issue_4514: state: open
title: ENH: helper function ar from roots, arma_process
body: AFAICS, we don't have a helper function to get the ar polynomial or ar coefficients from the roots.

usage: e.g. we could shrink the roots to be stationary and use them as start_params in ARMA `_fit_start_params_hr`

see #4513 for some improvements and fixes for `_fit_start_params_hr`
user: josef-pkt
issue_4509: state: open
title: ENH/Question: heteroscedasticity robust cov_type with other exog
body: I'm not sure what the theory is.

In general HC sandwiches use exog * u or the score_obs, i.e. consider only correlation with the included exog. 
What happens if the residuals/noise depend on other exog? Does it help for anything.

AFAIU, from the theory it shouldn't help. cov_params are consistent with the usual HC.
The question is whether it can help in small samples.

motivation: should we include a exog_other in the cov_type computation?
user: josef-pkt
issue_4508: state: open
title: ENH/BUG: MixedLM no convergence information, mle_retvals
body: MixedLM does not attach the mle_retvals to the results instance.
Except for `converged` boolean there is no information about convergence.

E.g. I wanted to see what's going on with convergence failures, and I wanted to check whether providing starts_params speeds up convergence.
user: josef-pkt
issue_4495: state: open
title: GLM docstring resid_pearson description doesnt match code
body: GLMResults docstring
```
    resid_pearson : array
        Pearson residuals.  The Pearson residuals are defined as
        (`endog` - `mu`)/sqrt(VAR(`mu`)) where VAR is the distribution
        specific variance function.  See statsmodels.families.family and
        statsmodels.families.varfuncs for more information.
```
Actual code
```
    @cache_readonly
    def resid_pearson(self):
        return (np.sqrt(self._n_trials) * (self.model.endog - self.mu) *
                np.sqrt(self._var_weights) /
                np.sqrt(self.family.variance(self.mu)))
```

The docstring is a bit over-the-top in verbosity here, but no big deal.  It looks like it hasn't kept up with more recent changes to the code.

BTW the GLM docstring helpfully lists all the model attributes twice.
user: jbrockmendel
issue_4482: state: open
title: Orthogonal Regression in statsmodels
body: Can statsmodels do orthogonal regression (also known as total least squares). I.e. the red line in this question: https://stats.stackexchange.com/questions/332819/line-of-best-fit-does-not-look-like-a-good-fit-why

Or is this not used often enough to warrant it being part of statsmodels?
user: oliverangelil
issue_4478: state: open
title: GLM PredictionResults has tvalues; linear_model doesn't.  Intentional?
body: The classes are very similar, and have a comment that they were copied from ContrastResults.  Is this particular difference intentional?

Why is the GLM version's `se_obs` raising NotImplementedError?
user: jbrockmendel
issue_4476: state: open
title: ARMAResults.plot_predict --> hard coding intentional?
body: ARMAResults.plot_predict calls `self.model._get_prediction_index(start, end, dynamic=False)`.  Why is dynamic hard-coded here?  `dynamic` is a kwarg to the plot_predict method, so I would imagine it would be passed through here (which is what is done in `ARIMAResults.plot_predict`)
user: jbrockmendel
issue_4474: state: open
title: iolib.summary unused variables
body: @josef-pkt we're down to just a couple of spots in iolib.summary that need clarification.

- [x] `alp` is assigned to but never used.  Please advise.
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/iolib/summary.py#L536

- [ ] `endog_basename` is assigned to but never used.  Please advise.
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/iolib/summary.py#L677

- [ ] Commented-out Method tuple and a comment about RLM that I'm not sure what to make of.  Please advise.
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/iolib/summary.py#L318
user: jbrockmendel
issue_4464: state: open
title: ARMAProcess references non-existent self.lags
user: jbrockmendel
issue_4462: state: open
title: SUMM/BUG/REF: VAR trend/exog in auxiliary results, forecast_cov
body: VECM PR has changed some things in VAR that has broken some auxiliary results or function (like VARProcess.mean) and (maybe) changed behavior for others like forecast_cov.

We might need some quick fixes for 0.9

## forecast_cov, forecast_interval

https://github.com/statsmodels/statsmodels/commit/bdb8a292e90156bdb6388fb8bf5e9e9fdf2d8b72#r28501721 dropped a term in forecast_cov

Stata (I currently have 14) does not compute forecast se if exog is specified. AFAICS, Stata does not have trend as a special option. Trend has to be specified in `exog`.
Stata takes parameter uncertainty in VAR/SVAR without exog into account for the forecast se, but not for VECM.

unit tests: 
forecast_cov has one old test that checks only mse (but not h-step ahead forecast error AFAICS)
forecast_interval:   
   old TestVARResults test has only smoke test, no assert  
   VECM has tests against jmulti, AFAICS

note `mse` is "theoretical forecast error variance matrices" without parameter uncertainty, using impulse reponse function (marep)

VARResults overrides forecast_cov of VARProcess; before VECM it added parameter uncertainty (if that is correctly computed) to the inherited `mse`. After VECM it just returns `mse`.

possible quickfix that needs unit tests ( to check that _omega_forc_cov is correct):
add paramter uncertainty `_omega_forc_cov` but only for the case when we only have a constant besides endog, and warn otherwise.


user: josef-pkt
issue_4459: state: open
title: var_model._forecast_vars docstring is wrong --> code super-redundant
body: `var_model` has a buttload of functions that are also implemented as frikkin identical methods.  This is, to use the technical term, dumb.  And it leads to things like this:

```
def _forecast_vars(steps, ma_coefs, sig_u):
    """_forecast_vars function used by VECMResults. Note that the definition
    of the local variable covs is the same as in VARProcess and as such it
    differs from the one in VARResults!
```

... nope!  There's a module-level `forecast_cov` (and module-level alias `mse`), a `VARProcess.mse` that is identical to the module-level function (and an alias `forecast_cov`), and a method `VARResults.forecast_cov` that just returns `VARResults.mse` (and does some other calculations that it discards, because [...crickets])

Which means that the docstring above is wrong, and `_forecast_vars` and `forecast_interval` module-level functions are _also_ completely redundant.

Allowing this to pile up is making it easier to make new mistakes.  Grade A FootGun.  @ChadFulton moral support for a Butlerian Jihad?
user: jbrockmendel
issue_4454: state: open
title: LikelihoodModel.information docstring accurate?
body: ```    
    def information(self, params):
        """
        Fisher information matrix of model

        Returns -Hessian of loglike evaluated at params.
        """
        raise NotImplementedError
```

If the docstring is accurate, then this should return `-self.hessian(params)` instead of raising.
user: jbrockmendel
issue_4450: state: open
title: `if gen_left is None:` case not hit in summary.summary_top
body: What's the envisioned use case here?  This is never hit in the tests.

https://github.com/statsmodels/statsmodels/blob/master/statsmodels/iolib/summary.py#L324
user: jbrockmendel
issue_4448: state: open
title: tsatools.vech redundant with vector_ar.util.vech?
user: jbrockmendel
issue_4442: state: open
title: VARProcess `exog` vs `intercept`
body: New issue taking place of #4436, where discussion has become unproductive.

Code to reproduce error:
```
from statsmodels.tsa.vector_ar.var_model import *
endog = np.random.randn(100, 3)
exog = np.random.randn(100, 1)
model = VAR(endog=endog , exog=exog)
res = model.fit(maxlags=2)
>>> res.mean()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "statsmodels/tsa/vector_ar/var_model.py", line 820, in mean
    return np.linalg.solve(self._char_mat, self.exog)
  File "/usr/local/lib/python2.7/site-packages/numpy/linalg/linalg.py", line 390, in solve
    r = gufunc(a, b, signature=signature, extobj=extobj)
ValueError: solve: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (m,m),(m,n)->(m,n) (size 100 is different from 3)
```
user: jbrockmendel
issue_4427: state: open
title: ENH: summary_table, select or drop some params
body: The iolib.summary.summary_params has some, currently unused code, to include only selected variable.

`    exog_idx = lrange(len(xname))`

However, this is hard coded to the full list (by xname) and not used as an argument to the function.

This should be easy to turn into an option.

user: josef-pkt
issue_4426: state: open
title: REF: exog_names versus param_names
body: There is currently an incomplete refactoring to switch from `exog_names` to the more generic `param_names`.

- base.model is now using mainly `param_names`
- most of the individual models (that I checked) use and modify `exog_names` (e.g. count models with extra params)
- some supporting code like summary `_getnames` uses `exog_names`
- exog_names is in model, param_names in model.data
- base.data uses xnames and param_names
- results wrapper: those need names for the pandas Series or DataFrames. I'm not sure how the xnames/exog_names/param_names works for that.
  I think, model.exog_names is a reference to data.xnames. When we modify model.exog_names it correctly modifies the names for the pandas wrapper code. (i.e. we need the correct names in `model.data`)

Given the last point, it is unlikely that we can make structural changes.
The main refactoring that we might be able to do is to essentially rename exog_names to param_names, and consistently use param_names in all models.

One problem is that we have almost no test coverage for names. e.g. #4340 and some earlier bugs when summary is wrong because the names don't have the right length.

Another issue for summary will be when summary doesn't print the "params", either by dropping some or adding some. But summary methods of individual models can use the `xname` (singular !?) argument in summary supporting code.

related
#1970 required attributes
user: josef-pkt
issue_4425: state: open
title: REF: update conventions in GMM, API changes
body: #2640 there is already a general GMM round 3 issue 

GMM needs another round also in small things
e.g. non-intuitive or not standard pattern (where I had to read the docstring or code to get it right)

- rename optim_args to optim_kwds: it's a dict for kwargs, not a list or args
- maybe: take `disp` out of optim_kwds into a separate keyword option   
  (but optim_args is used in all fitxxx methods, so we would need extra disp everywhere, unless we only add it to fit as convenient override but keep it in optim_kwds)

missing test coverage for GMM class itself, currently only subclasses have unit tests.


ENH: (I don't know if there is an issue for this already)

IV currently has only one mom/residual that is multiplied by instruments.
example in #4340 has two IV and two mom that are multiplied by each IV, i.e. like kronecker product on rows. Moment conditions are for mean and variance multiplied by the same instruments.
Is this case common enough to support it in the IV-GMM classes? or do we require using GMM for this case?


user: josef-pkt
issue_4423: state: open
title: Which sandbox models are “real”?
body: It would be helpful to be able to distinguish non-abandoned code (eg GMM apparently) from everything else.
user: jbrockmendel
issue_4422: state: open
title: LikelihoodModel "copied from model.py" should be removed from garch
user: jbrockmendel
issue_4417: state: open
title: TypeError when using statsmodels.OLS with NaNs and missing = 'drop'
body: How to reproduce:
```python
from datetime import date
import numpy as np
import pandas as pd
import statsmodels.api as sm

X = pd.Series([np.NaN], pd.DatetimeIndex([date(2017,1,1)], freq='AS-JAN'))
Y = pd.Series([10], pd.DatetimeIndex([date(2017,1,1)], freq='AS-JAN'))
sm.OLS(X, sm.add_constant(Y), missing='drop')
```
Traceback:
```python
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/virtualenvs/test/lib/python2.7/site-packages/statsmodels/regression/linear_model.py", line 631, in __init__
    hasconst=hasconst, **kwargs)
  File "/virtualenvs/test/lib/python2.7/site-packages/statsmodels/regression/linear_model.py", line 526, in __init__
    weights=weights, hasconst=hasconst, **kwargs)
  File "/virtualenvs/test/lib/python2.7/site-packages/statsmodels/regression/linear_model.py", line 95, in __init__
    super(RegressionModel, self).__init__(endog, exog, **kwargs)
  File "/virtualenvs/test/lib/python2.7/site-packages/statsmodels/base/model.py", line 212, in __init__
    super(LikelihoodModel, self).__init__(endog, exog, **kwargs)
  File "/virtualenvs/test/lib/python2.7/site-packages/statsmodels/base/model.py", line 63, in __init__
    **kwargs)
  File "/virtualenvs/test/lib/python2.7/site-packages/statsmodels/base/model.py", line 88, in _handle_data
    data = handle_data(endog, exog, missing, hasconst, **kwargs)
  File "/virtualenvs/test/lib/python2.7/site-packages/statsmodels/base/data.py", line 630, in handle_data
    **kwargs)
  File "/virtualenvs/test/lib/python2.7/site-packages/statsmodels/base/data.py", line 65, in __init__
    **kwargs)
  File "/virtualenvs/test/lib/python2.7/site-packages/statsmodels/base/data.py", line 284, in handle_missing
    combined = dict(zip(combined_names, lmap(drop_nans, combined)))
  File "/virtualenvs/test/lib/python2.7/site-packages/statsmodels/base/data.py", line 282, in <lambda>
    drop_nans = lambda x: cls._drop_nans(x, nan_mask)
  File "/virtualenvs/test/lib/python2.7/site-packages/statsmodels/base/data.py", line 478, in _drop_nans
    return x.ix[nan_mask]
  File "/virtualenvs/test/lib/python2.7/site-packages/pandas/core/indexing.py", line 121, in __getitem__
    return self._getitem_axis(key, axis=0)
  File "/virtualenvs/test/lib/python2.7/site-packages/pandas/core/indexing.py", line 1077, in _getitem_axis
    return self._get_label(key, axis=axis)
  File "/virtualenvs/test/lib/python2.7/site-packages/pandas/core/indexing.py", line 136, in _get_label
    return self.obj._xs(label, axis=axis)
  File "/virtualenvs/test/lib/python2.7/site-packages/pandas/core/generic.py", line 2032, in xs
    loc = self.index.get_loc(key)
  File "/virtualenvs/test/lib/python2.7/site-packages/pandas/core/indexes/datetimes.py", line 1435, in get_loc
    stamp = Timestamp(key, tz=self.tz)
  File "pandas/_libs/tslib.pyx", line 402, in pandas._libs.tslib.Timestamp.__new__ (pandas/_libs/tslib.c:10051)
  File "pandas/_libs/tslib.pyx", line 1528, in pandas._libs.tslib.convert_to_tsobject (pandas/_libs/tslib.c:28851)
TypeError: Cannot convert input [False] of type <type 'numpy.bool_'> to Timestamp
```
In first call to pandas:
```python
  File "/virtualenvs/test/lib/python2.7/site-packages/statsmodels/base/data.py", line 478, in _drop_nans
    return x.ix[nan_mask]
```
value of `nan_mask` is `False`.

----
version:
```python
>>> sm.version.full_version
'0.8.0'
```
Tested on pandas versions 0.20.3 and 0.22.0.

user: dferens
issue_4415: state: open
title: RLM._initialize docstring correct?
body: It looks like the bit about resetting the history is inaccurate.  Please advise.

```    def _initialize(self):
        """
        Initializes the model for the IRLS fit.
        Resets the history and number of iterations.
        """
        self.pinv_wexog = np.linalg.pinv(self.exog)
        self.normalized_cov_params = np.dot(self.pinv_wexog,
                                        np.transpose(self.pinv_wexog))
        self.df_resid = (np.float(self.exog.shape[0] -
                         np_matrix_rank(self.exog)))
        self.df_model = np.float(np_matrix_rank(self.exog)-1)
        self.nobs = float(self.endog.shape[0])
```
user: jbrockmendel
issue_4412: state: open
title: results_arima unloadable
body: @ChadFulton since I'm trying to give jkpt some space, but someone should be aware of this.

```
from statsmodels.tsa.tests.results import results_arima
ra = results_arima.ARIMA112()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "statsmodels/tsa/tests/results/results_arima.py", line 290, in __init__
    self.forecast = forecast_results['fc112c'][-25:]
ValueError: no field of name fc112c
```

and even if that error were fixed, `ARIMA112.__init__` would set a bunch of attributes and then overwrite them all by setting `self.__dict__`.  I don't know how we're expected to trust any of this old code.
user: jbrockmendel
issue_4408: state: open
title: FAQ: multipletests 'bh' benjamini-hochberg correction returns a lot of identical p-values
body: statsmodels.sandbox.stats.multicomp.multipletests 'bh' benjamini-hochberg correction returns a lot of identical p-values, I think the snippet below is self-explanatory. To my (limited) statistical knowledge, this is not supposed to happen.
EDIT: one gets the same results using the R stats library ```library(stats); p.adjust(pvalues, method = "BH", n=length(pvalues)) ``` so I suppose this behaviour *is* correct? I will not delete the post until someone can confirm this. If someone can explain this behaviour, that would also be nice.

```
import numpy as np
np.random.seed(123)
pvalues = np.random.rand(20)
pvalues
```

> array([ 0.69646919,  0.28613933,  0.22685145,  0.55131477,  0.71946897,
>         0.42310646,  0.9807642 ,  0.68482974,  0.4809319 ,  0.39211752,
>         0.34317802,  0.72904971,  0.43857224,  0.0596779 ,  0.39804426,
>         0.73799541,  0.18249173,  0.17545176,  0.53155137,  0.53182759])

```
multipletests(pvals=pvalues, alpha=0.05, method='fdr_bh', is_sorted=False, returnsorted=False)
```

> ... (irrelevant)
> array([ 0.77683727,  0.77683727,  0.77683727,  0.77683727,  0.77683727,
>          0.77683727,  0.9807642 ,  0.77683727,  0.77683727,  0.77683727,
>          0.77683727,  0.77683727,  0.77683727,  0.77683727,  0.77683727,
>          0.77683727,  0.77683727,  0.77683727,  0.77683727,  0.77683727]),
>  0.0025613787765302876,
>  0.0025)
user: PDiracDelta
issue_4406: state: open
title: ENH: information criteria or penalized mean squared error on subsample 
body: just an idea, I'm not sure it will work

Hyndam, e.g. in Hyndman Khandakar (2008) "Automatic Forecasting ..." and in blog post (*) is negative on using information criteria across several model types, e.g. when using different amount of differencing in SARIMA or comparing exponential smoothing with SARIMA, or full MLE versus conditional MLE. One argument is different initial conditions conditions for the process and different sample because some initial observations are dropped.
There is no problem comparing llf based IC for log or box-cox transformation as long as likelihood function is adjusted for the transformation, nor for comparing different distributional assumptions, e.g. normal versus t.)

(*) https://robjhyndman.com/hyndsight/aic/

However, I think we can compute the comparison statistic, ic or penalized mse, just on the last several observations, e.g. a fixed number (appropriate for the frequency and length of the sample data) or a fraction of observations.
This is similar to using a hold-out, test subsample, but it is still in-sample but only on the recent part of the dataset.

One advantage is that it also looks at the subsample that is relevant for forecasting if there are structural breaks, change in trends or seasonal pattern. (related to Chow test or similar)
Compared to out-of-sample forecasting and cross-validation this has still the advantage that it is much faster. (i.e. useful at least for a preliminary search to narrow down which models might be considered for  cross-validation.)

Related: measures to compute weights for forecast combination, e.g. EVIEWS documentation summarizes several of them. There seems to be the same dichotomy, either full training sample measures, or out-of-sample forecast measures. In between these, we could use the recent subsample of the training data.


For cross-sectional data this is not very useful because we don't have a prior order of observations. We would have to define some measure for how close the prediction sample (exog) is to subsamples of the training data.  (It's a bit like matching literature but matching prediction to training samples.)

related issues should be for automatic forecasting and model selection
e.g. #2571
user: josef-pkt
issue_4400: state: open
title: FAQ/ENH/DOC  interaction of two seasonal components in forecasting
body: (It's not clear to me how this can be done or what the best way is to do this in statsmodels.)

related older issue #2402

Suppose we have hourly data with a daily and a weekly seasonal cycle, Tue, We, Th have roughly the same within day pattern, Mo, Fr, Sa, So each has some unique parts in the within day pattern. 

Many human activities follow this seasonal pattern, e.g. electricity load, at half hour intervals. (For long time series we would have additionally the annual seasonal pattern plus holiday calender effects, plus irregular time periods from switching to daylight savings time, ....)

ideas:

Use deterministic weekly seasonal component, e.g. with Fourier series, that also captures within day fluctuation that differs by weekday, i.e. we have a complicated weekly seasonal pattern but no more daily pattern. Short term adjustments over a few days and within days can be handled by standard SARMA.

Similarly, Periodic splines interacted with "day type", i.e. pooled splines for Tue, We, Th, and separate splines for each of the other days. Then the concatenation of all day splines would create the weekly pattern.
(Sounds good, with a day type indicator, we could even have a Sunday=holiday in the middle of the week.) Does patsy have periodic splines? (scipy does but not in the form of basis functions.)

SARIMA with two seasons likely runs into memory problems in the dense statespace formulation.

 









user: josef-pkt
issue_4399: state: open
title: ENH: outlier robust seasonal_decompose
body: related #4044 using non- or semi-parametric methods

(just an idea, no references for now)
Replace the estimators for the components with outlier robust methods.
This is a similar idea as using lowess for the trend, i.e. robust non-parametric smoothing, but also applied to seasonal pattern.

`seasonal_decompose` uses filters that are not robust to outliers, e.g. mean for seasonal component and convolution filter for trend. Outliers, either isolated or an episode or outliers can or will distort the seasonal pattern and the local trend.

I have seen some plots on stackoverflow or blog post where this seems to happen. 

Additionally, the default window length looks often too small, and I haven't seen users overriding the `filt` window. A short window increases the local effect of an outlier, although it doesn't spread out the effect as much as a larger window. Also short term fluctuations over several periods will be reflected in the local trend.

One possibility to make changing and experimenting with the window length easier, would be to accept an integer for `filt` that defines the window length used with the current uniform kernel.








user: josef-pkt
issue_4397: state: open
title: test_glm_weights commented-out/unused
body: https://github.com/statsmodels/statsmodels/blob/master/statsmodels/genmod/tests/test_glm_weights.py#L213
```    @pytest.mark.xfail(reason='Known to fail')
    def test_basic(self):
        super(TestGlmPoissonPwNr, self).test_basic()

    @pytest.mark.xfail(reason='Known to fail')
    def test_compare_optimizers(self):
        super(TestGlmPoissonPwNr, self).test_compare_optimizers()
```

"Known to fail" is not a helpful error message.

https://github.com/statsmodels/statsmodels/blob/master/statsmodels/genmod/tests/test_glm_weights.py#L231
```
        cls.res1 = GLM(cpunish_data.endog, cpunish_data.exog,
                        family=sm.families.Poisson(), freq_weights=fweights
                        ).fit(cov_type='HC0') #, cov_kwds={'use_correction':False})
```

Is the `cov_kwds={'use_correction': False}` needed?  As usual: should be deleted or explained.  This shows up in a couple of places in this file.

https://github.com/statsmodels/statsmodels/blob/master/statsmodels/genmod/tests/test_glm_weights.py#L275
```        
        # no wnobs yet in sandwich covariance calcualtion
        cls.corr_fact = 1 / np.sqrt(n_groups / (n_groups - 1))   #np.sqrt((wsum - 1.) / wsum)
```
Does the "no wnobs..." comment relate to the commented-out sqrt?


https://github.com/statsmodels/statsmodels/blob/master/statsmodels/genmod/tests/test_glm_weights.py#L903
```
    y = y_true + 2 * np.random.randn(nobs)
```
`y` defined but unused.  What is it for?

https://github.com/statsmodels/statsmodels/blob/master/statsmodels/genmod/tests/test_glm_weights.py#L201
`aweights` is defined but unused.  Should it be?  This shows up a couple of times in this file.
user: jbrockmendel
issue_4393: state: open
title: MICE issue
body: hi, I have a dataframe, and I am trying very simplistic imputation with MICE and I keep getting an error:

> RuntimeWarning: divide by zero encountered in double_scalars return np.dot(wresid, wresid) / self.df_resid

Dataset source:
[Food-consumption.csv]( https://www.dropbox.com/s/okrn8zo9xz62vuq/food-consumption.csv?dl=0
)
My code is as follows:
```
import pandas as pd
import statsmodels.api as sm
from statsmodels.imputation.mice import MICEData

food = pd.read_csv('Data/food-consumption.csv', )
food.drop('Country', axis=1, inplace=True)
f_imp = MICEData(food)
f_imp.set_imputer('Sweetener', formula='Tea+Frozen_veggies+Frozen_fish+Tinned_fruit+Crisp_bread+Garlic', k_pmm=5)
f_imp.update_all()
```
IDE: Jupyternotebook
Statsmodels v. 0.80.0
Pandas v. 0.22.0
Numpy: 1.14.1
user: BekzodR
issue_4387: state: open
title: ENH/FAQ: Fleiss Kappa giving nan results, randolph's kappa
body: I'm trying to use Fleiss Kappa to calculate the agreement of several raters.

First I have an array of two subjects and 7 raters per subject. There are two categories, however, the second category is never chosen.

```
ratings = [[0,0,0,0,0,0,0],
           [0,0,0,0,0,0,0]]
```
As we can see, the raters agree well on the two subjects, so we expect `kappa=1`.

however, I receive `nan`:

```
aggr =aggregate_raters(ratings, n_cat=2)[0]
kappa = fleiss_kappa(aggr)
print(kappa) # nan
```

Am I using the function wrong? According to [some online calculator](http://justusrandolph.net/kappa/) it should give 1.0
user: skjerns
issue_4383: state: open
title: defined but not used in test_glm_weights
body: https://github.com/statsmodels/statsmodels/blob/master/statsmodels/genmod/tests/test_glm_weights.py#L141

This kind of thing shows 7 times, of which two are commented-out:
```
        # compare with discrete, start close to save time
        modd = discrete.Poisson(cpunish_data.endog, cpunish_data.exog)
```

This is at the end of setup_class and nothing is done with `modd`... should something be done with it?
user: jbrockmendel
issue_4379: state: open
title: REF/ENH/SUMM refactor and enhance Multicomparison.tukeyhsd
body: Multicomparison.tukeyhsd has been written a long time ago and never been updated, besides some fixes and enhancements.

It's pre-pandas and uses structured dtype to build the summary SimpleTable.

I don't want to rewrite it using pandas, e.g. groupby, given that the current numpy based group statistics never needed maintenance (AFAIR).
However, the code structure is not very useful for enhancements. The code is spread over a standalone function, the Multicomparison.tukeyhsd method and the TukeyHSDResults class. Too much is done on the lower level and can be moved into the results class, making it easier to change and add extra results.

TukeyHSDResults is not just a Holder class anymore, given that plotting method was added a long time ago.

The standalone tukeyhsd function is a full test function returning a tuple with many elements, however does not include p-values. #2346 Arguments are summary statistics `mean_all, nobs_all, var_all`
I'm not sure we should change it much. It is also in the official api through statsmodels.stats.multicomp.
specifically: do we add p-values to the tukeyhsd function and hand it all the way up to TukeyHSDResults or do we directly add the p-values to the results class. 

what's the usage pattern?  (I'm not sure)

general idea for hypothesis test: support both simple return of numbers for large scale hypothesis testing and returning informative results classes for closer, interactive investigation. I doubt there is much large scale hypothesis testing with tukeyhsd. In that case we are free to just focus on the results class.

internal usage: Which parts of the results class can be reused across different multiple comparison hypothesis tests? #4365 added a pairwise comparison method to the models also with just a basic results class which essentially just holds a dataframe. 
Other mutliple comparisons are in various experimental and prototype code, e.g. kruskal-nemenyi. I have no idea yet what we will need there.

issues bugs and enhancements

- #3584 compatibility bug, PR in progress
- #2762 missing group labels and group information
- #2346 missing pvalues

and more general
- tukeyhsd for (balanced) two-way ANOVA

**proposed solution for now:**

- move summary table creation to results class
- add attributes or cached attributes to results class
- maybe add new methods to results class: How do we add numerical integration mvnormal p-values? Those are not the first choice, slower and more fragile than qsturng, but maybe more accurate in some cases and fewer range limitations.
- add summary_frame so users get a pandas compatible summary to work with (similar to t_test and t_test_pairwise) (e.g. save to csv as in #2762)

related:
We have different methods to compute multiple comparison p-values. Make it more reusable, e.g. make it simpler to add qsturng pvalues to other hypothesis test. We should get a standard distribution interface to qsturng (sf/pvalue and ppf/critvalue).






user: josef-pkt
issue_4372: state: open
title: FAQ: One-sample proportion z-test -- Implementation Issue?
body: In the case of one-sample test, the function `proportions_ztest(count, nobs, value=None, alternative='two-sided', prop_var=False)` from `statsmodels.stats.proportion` calculates the variance based on **the sample proportion**, unless `prop_var` is specified, like so:

```
    if prop_var:
        p_pooled = prop_var
    var_ = p_pooled * (1 - p_pooled) * nobs_fact
```

However, explicitly specifying `prop_var` is redundant, since in my opinion it should be picked up by the parameter `value`. I personally do not see a reason why the variance should be calculated by default using the **sample proportion** instead of the **null hypothesis value**.

I am keen to hear your opinion on this.



user: Nolataari
issue_4369: state: open
title: BUG: Mice Error - recursion depth in patsy with many variables
body: **edit**

one bug found here is now #4375 with PR #4377 

unresolved: There is a limitation in the number of variables that MICEData can handle caused by a recursion depth limit in patsy's formula handling (details in comments below)

------------
Hi,

Apologies to bother you
I have written the code attached below and get the following message:
"ValueError: shape mismatch: value array of shape (3,1) could not be broadcast to indexing result of shape (3,)"

I know that this problem has been mentioned before and a quick fix was suggested but I get this error with the latest version of stats models, numpy and pandas

Any suggestions?

Thanks in advance!



import statsmodels
import numpy as np
import pandas as pd

input=pd.read_csv(path)
num_samp, num_input = input.shape 
input.columns=[str(i)for i in range(0,num_input)]
input.columns= "VAR" + input.columns 

from statsmodels.imputation.mice import MICEData as mice
data_imp_tmp=mice(input)
data_imp_tmp.update_all(n_iter=1000)
user: Elesa
issue_4361: state: open
title: ENH: automatically building contrast/restriction matrices
body: We have several open issues for this. e.g. #852, #1668, #2362
The attempt that is currently most integrated with patsy is in an extension of https://github.com/statsmodels/statsmodels/issues/2687#issuecomment-372779808 extension merged in #4365 `t_test_pairwise`

There is also preliminary code all over the place, but I never managed to figure out a general structure and API

two possibilities using patsy formula factor information and specifying "base contrasts or encoding" explicitly

## patsy

patsy attaches the contrast matrix, `cm` below, used to build the encoding, #2687 We don't currently use that information.
AFAICS:
- dot product of cm and params provides base parameterization and corresponding cov_params
- cov_params will be singular, and I guess we need to check the rank (?) for estimable contrasts/restrictions
- based on restriction value (`diff`) and reparameterized cov_params (where we need only submatrix of cov_params), we can work on selected effects/constraints


## arrays and explicit formulas

what do we need?
description of current encoding, e.g. is there a reference level, what is the label for the reference level
labels, column/params indices


target contrast/restriction
mnemonic all pairs, ....

current: ???

- sandbox.regression.penalized contains 3 function like `coef_restriction_meandiff` for structured penalization or stochastic constraints.
- sandbox.stats.multicomp contains 3 functions like `contrast_all_one` (which is one of my early attempts)
- sandbox.stats.contrast_tools is a more general attempt based on multicomp
- stats.contrats has pre-statsmodel `contrastfromcols` to check full rank that I never figured out AFAIR
- tools.tools has pre-statsmodels `isestimable` which I also never figured out
- special case multivariate with MANOVA tests (2-D params, col and row hypothesis), no multiple testing AFAIR


related: contrasts/constructed exog for prediction, LSMEANS, margins, ... #2167 and hypothesis tests, inference on linear restrictions of predicted means (like treatment effects)

TODO/Usage:

partially based on what I started in #2687 

- get term and factor info from design_info
- "normalize", i.e. get a base parameterization for "means", i.e. coefficients
- can be subset cov_params, so we are independent of other params in model?
- construct target restriction/contrast matrix
- t_test
- multiple testing p-value correction, corrected confidence intervals (not available yet, e.g. #2955)
- reporting, summary and plots

**update** in #4365 I combined step 2 "normalize" and step 4 target restriction matrix into a combined restriction matrix that applies to the actual parameterization in the model and can directly be used with t_test and similar.

implementation detail: Where do we put these things?

Aside:
issue ??? suggest using prediction instead of coefficients. This would work for linear prediction (e.g. in GLM) but nonlinear prediction of response would have different effect. Predicted values are likely less close to normal distributed than coefficients, predicted values depend on values of other exog as in margins or predict_at.
prediction and LSMEANS runs into the same problem as here in that we need to construct the auxiliary exog (i.e. artificial sample reflecting the factor levels)


user: josef-pkt
issue_4360: state: open
title: ENH t_test add constrast_names
body: currently t_test in a results class returns a ContrastResults instance, but doesn't allow specifying names or labels for the linear/affine restrictions. Both ContrastResults summary and summary_frame have an xname option.

I think a better API would be to allow specifying the constraint_names in t_test itself, and to rename `xname` to something more accurate.

open: what do we call it?
contrast_names, restriction_names, constraint_names, labels ?



user: josef-pkt
issue_4359: state: open
title: BUG/ENH: hard coded settings in stats.gof.gof_chisquare_discrete
body: https://stackoverflow.com/questions/49314132/statistical-test-for-random-graphs-edges-binomial-distribution/49322957#49322957

"the limits are set at -1000 and +1000"

"It looks like that threshold and the number of bins are hardcoded for the initial usecase and have never been generalized, especially not for large random variables or large samples"

we need options and expanding limits/bounds (or use ppf)

related aside: 
- numpy has histogram binning options, maybe some of those are good defaults.
- #4262 chisquare after estimating a model, What's the rate at which number of bins should increase? Do we have a rule of thumb as for autocorrelation?

user: josef-pkt
issue_4357: state: open
title: commented-out in results_arima and results_arima
body: Good news.  We're almost done with the "what was the intention here?" issues.  A little over half of whats left are in results_arma or results_arima.

results_arima: it looks like maybe older results came from a different package and were commented out (instead of deleted) for [some reason?].  I don't see any indication of why these are still useful, or why they one set of results is preferred over the other.

https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/tests/results/results_arima.py#L430
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/tests/results/results_arima.py#L332
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/tests/results/results_arima.py#L212
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/tests/results/results_arima.py#L91
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/tests/results/results_arima.py#L58
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/tests/results/results_arima.py#L46

results_arma: pretty much the same deal, but with less readable formatting.
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/tests/results/results_arma.py#L731
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/tests/results/results_arma.py#L721
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/tests/results/results_arma.py#L687
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/tests/results/results_arma.py#L678
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/tests/results/results_arma.py#L670
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/tests/results/results_arma.py#L659
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/tests/results/results_arma.py#L610
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/tests/results/results_arma.py#L596
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/tests/results/results_arma.py#L548
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/tests/results/results_arma.py#L478
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/tests/results/results_arma.py#L410
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/tests/results/results_arma.py#L358
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/tests/results/results_arma.py#L315
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/tests/results/results_arma.py#L257
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/tests/results/results_arma.py#L202
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/tests/results/results_arma.py#L138
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/tests/results/results_arma.py#L75

user: jbrockmendel
issue_4355: state: open
title: sm.stats unused/commented-out
body: _lilliefors.get_lilliefors_table defines `alpha_large, size_large, crit_lf_large` but then these are never used.  Whats the intended use?  Are these still needed?

tabledist.TableDist.prob has commented-out code:
```
        critv = self._critvals(n)
        alpha = self.alpha
#        if self.signcrit == 1:
#            if x < critv[0]:  #generalize: ? np.sign(x - critvals[0]) == self.signcrit:
#                return alpha[0]
#            elif x > critv[-1]:
#                return alpha[-1]
#        elif self.signcrit == -1:
#            if x > critv[0]:
#                return alpha[0]
#            elif x < critv[-1]:
#                return alpha[-1]

        if self.signcrit < 1:
            #reverse if critv is decreasing
            critv, alpha = critv[::-1], alpha[::-1]
```

Is any of the commented-out stuff here still relevant?

A little bit further down in the TableDist.prob is
```        
   return interp1d(critv, alpha)(x)[()]
```

What's the `[()]` all about?

contrast.ContrastResults.compute_matrix has a try/except to define self.rank:
```
        try:
            self.rank = self.matrix.shape[1]
        except:
            self.rank = 1
```

AFAICT there is no `matrix` attribute, so this is just always setting rank=1.  Is that intended?

In acorr_lm, a variable `xdiff=np.diff(x)` is defined and then never used.  Should it be?

In test_diagnostic a whole bunch of things are defined and then never used.  Should they be?
user: jbrockmendel
issue_4354: state: open
title: diagn attribute added in RegressionResults.summary
body: Looks unused.  Is there a reason for this?  Its also done in sandbox.regression.gmm and is commented out in quantile_regression.
user: jbrockmendel
issue_4348: state: open
title: Unused and commented-out in iolib.summary
body: A few places where I need input to determine what is commented out or unused temporarily vs "temporarily".

`model_methods` is defined but unused.  Will it be used in the near future?
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/iolib/summary.py#L92

gen_stubs_left and gen_data_left are commented out.  Are they making a comeback?
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/iolib/summary.py#L139

Is "checke" a typo?  I'm not sure what this comment means
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/iolib/summary.py#L196

TODO note for JP; is this going to be addressed in the near future? It may belong in a GH issue instead of a fossilized comment.
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/iolib/summary.py#L229

Are the RLM or Method notes here still relevant?https://github.com/statsmodels/statsmodels/blob/master/statsmodels/iolib/summary.py#L318

Are needed_values making a comeback?https://github.com/statsmodels/statsmodels/blob/master/statsmodels/iolib/summary.py#L345

What does the `gen_fmt` comment mean here?https://github.com/statsmodels/statsmodels/blob/master/statsmodels/iolib/summary.py#L383

Are the extra returns gen_table_left, gen_table right going to be re-implemented?https://github.com/statsmodels/statsmodels/blob/master/statsmodels/iolib/summary.py#L403

Will the skip_headers2 kwarg ever be used?https://github.com/statsmodels/statsmodels/blob/master/statsmodels/iolib/summary.py#L621

Are the comments about VAR and transposing params still relevant?https://github.com/statsmodels/statsmodels/blob/master/statsmodels/iolib/summary.py#L667

endog_basename is defined and never used.  Will it be?  If so, for what? https://github.com/statsmodels/statsmodels/blob/master/statsmodels/iolib/summary.py#L674
user: jbrockmendel
issue_4346: state: open
title: should predict_kwds be used somewhere?
body: https://github.com/statsmodels/statsmodels/blob/master/statsmodels/base/tests/test_generic_methods.py#L36
user: jbrockmendel
issue_4345: state: open
title: test_regression.TestGLS_alt_sigma.res3 unused
body: Should something be done with that?
user: jbrockmendel
issue_4335: state: open
title: commented-out code in _constraints
body: https://github.com/statsmodels/statsmodels/blob/master/statsmodels/base/_constraints.py#L65

```
       # It makes it easier to interpret simple restrictions, e.g. b1 + b2 = 0
        # TODO: make this work, there is something wrong, does not round-trip
        #       need to adjust constant
        #evecs_maxabs = np.max(np.abs(evecs), 0)
        #evecs = evecs / evecs_maxabs
```

Any idea what this is for or if it is still relevant?  Can it just be deleted?
user: jbrockmendel
issue_4333: state: open
title: Singular forecast error covariance matrix encountered at period
body: Hi,

I am a beginner in tsa package with python. I am trying to estimate the following model using statespace methods - 
y(t) = alpha*x(t) + v(t)
c(t) = x(t) + u(t)
x(t) = x(t-1) + e(t)

I first simulated a sample and defined the class to call the MLEModel module but I get an error that says "Singular forecast error covariance matrix encountered at period 0" and I can't figure out why. Please see below the complete code. Any help will be greatly appreciated!

Thanks!

```python
import statsmodels.tsa.statespace as sp
import numpy as np
from statsmodels.tsa.statespace.tools import (constrain_stationary_univariate, unconstrain_stationary_univariate)
```

# simulate sample

```python
alpha, sigmasqv, sigmasqu, sigmasqe = 1, 0.5, 1.5, 1
T = 500
burn =  T/2

einit = sigmasqe**0.5*np.random.randn(T+burn)
uinit = sigmasqu**0.5*np.random.randn(T+burn)
vinit = sigmasqv**0.5*np.random.randn(T+burn)
xinit, cinit, yinit = np.zeros(T+burn), np.zeros(T+burn), np.zeros(T+burn)
xinit[0] = einit[0]

for t in range(1,T+burn):
    xinit[t] = xinit[t-1] + einit[t]

cinit = xinit + uinit
yinit = alpha*xinit + vinit
y, c, x = yinit[burn:], cinit[burn:], xinit[burn:]

simdata = np.concatenate((y,c), axis=0).reshape(2,T)

# Construct the model
class simplemodel(sp.mlemodel.MLEModel):
    start_params = [0, 1, 1, 1.5]
    param_names = ['alpha', 'sigma.v', 'sigma.u', 'sigma.e']

    def __init__(self, endog):
        # Initialize the state space model
        super(simplemodel, self).__init__(endog, k_states=1, k_posdef=1)

        # Setup the fixed components of the state space representation
        self['design', 1, 0] = 1.
        self['transition', 0, 0] = 1.
        self['selection', 0, 0] = 1.
        self.initialize_approximate_diffuse()
        self.loglikelihood_burn = 1

    def transform_params(self, params):
        alpha = constrain_stationary_univariate(params[0:1])
        sigmasq = constrain_stationary_univariate(params[1:4]**2)
        return np.r_[alpha, sigmasq]

    def untransform_params(self, params):
        alpha = unconstrain_stationary_univariate(params[0:1])
        sigma = unconstrain_stationary_univariate(params[1:4]**0.5)
        return np.r_[alpha, sigma]

    # Describe how parameters enter the model
    def update(self, params, **kwargs):
        params = super(simplemodel, self).update(params, **kwargs)

        self['design', 0, 0] = params[0]
        self['obs_cov', 0, 0] = params[1] 
        self['obs_cov', 1, 1] = params[2]
        self['state_cov', 0, 0] = params[3]
        

model = simplemodel(simdata)
res = model.fit()
print res.summary()
```
user: samanvayagarwal
issue_4332: state: open
title: Incorrect error message in tools.tool?
body: https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tools/tools.py#L205

```
    # handle ndarrays and catch array-like for an error
    elif data.__class__ is np.ndarray or not isinstance(data, np.ndarray):
        if not isinstance(data, np.ndarray):
            raise NotImplementedError("Array-like objects are not supported")
```
The message that Array-like objects are not supported doesn't make sense to me.  It looks like it should be the exact opposite: only array-like objects are supported.  Am I missing something?

For that matter, is there a reason why the first condition is _specifically_ ndarray without allowing subclasses (i.e. change `data.__class__ is np.ndarray` to `isinstance(data, np.ndarray)`)?
user: jbrockmendel
issue_4328: state: open
title: Stepwise linear regression with AIC?
body: What is the Python statsmodels equivalent for R step() function of stepwise regression with AIC as criteria?

I found a [stepwise regression with p-value as criteria](https://datascience.stackexchange.com/questions/24405/how-to-do-stepwise-regression-using-sklearn/24447#24447), is there something similar, but with AIC?

I'm also okay with other python packages.
user: KathyGCY
issue_4323: state: open
title: FAQ/ENH: two-way Tukey HSD
body: This is a possible bug that I encountered while running Tukey HSD test on a dataset.  The Python ANOVA table found some significant effects (p-values < 0.005) which are same results as with R and SAS.
Yet, Python's Tukey HSD reports no significant comparisons of the factors levels while R find significant comparisons associated with the factors that were significant in ANOVA table.

[Detergent.xlsx](https://github.com/statsmodels/statsmodels/files/1787781/Detergent.xlsx)

import statsmodels.formula.api as smf
import pandas as pd
from statsmodels.formula.api import ols
from statsmodels.stats.anova import anova_lm
from statsmodels.graphics.factorplots import interaction_plot
import statsmodels.stats.multicomp as multi
model = smf.ols('Cleaness ~ C(Stain) + C(DETERGENT)', data=melted_df).fit()
aov_table = anova_lm(model, typ=2)

# Multiple comparisons : issues are found in the output of this code.
mcTreatment    = multi.MultiComparison(melted_df['Cleaness'], melted_df['DETERGENT'])
results_Treatment  = mcTreatment.tukeyhsd()


user: dcasmr
issue_4297: state: open
title: ENH/REF: Glm allow complex step derivatives
body: currently just parking two commits that I needed for non-linear GLM in
https://github.com/statsmodels/statsmodels/issues/2179#issuecomment-362822388

use of `fabs` prevents the use of complex step derivatives from numdiff

extra comma removal in numdiff already has a PR

user: josef-pkt
issue_4279: state: open
title: ENH/recipe: use WLS for local post-estimation for non-linear least squares 
body: Similar idea to using WLS for GLM diagnostics #3743 #4268, we can also use WLS for other nonlinear models to get local post-estimation analysis.

Note: #2050 I don't remember details right now, but for some results it is relevant which local approximation we use.

Statsmodels still doesn't have standard non-linear estimators like non-linear least squares.
There are several recent questions for it on stackoverflow, e.g.
https://stackoverflow.com/questions/48989708/how-can-i-implement-a-regression-of-a-variable-on-a-function-in-python-nelson-s

My general recommendation is to use lmfit which has good support for starting parameters and constraints. However, using Jacobian of the non-linear mean function as exog in WLS, would make postestimation results available for the local approximation.
In this case cov_params would be based on OPG because we don't have the actual hessian.

This would work if lmfit (or whatever package is used) provides parameter estimate and Jacobian, 
Do we need weights? (I don't remember right now. I guess in the non-gaussian case when we need the score_obs and not just the Jacobian)

If we also get the Hessian, then we could to full local analysis, but that would require special code and won't be covered with WLS.

related: #2179 nonlinear GLM, which shows some potential problems with getting and using hessian.

user: josef-pkt
issue_4278: state: open
title: REF/ENH/Design:  link versus mean function
body: an old design question: 
Do we parameterize GLM or single index models with mean function or link function, i.e. transform linpred (link.inverse) or transform expected value (link)?

The main question: Is it worth it to duplicate the code and add inverse classes for the links?

I don't think we ever want to reparameterize GLM because it would take it too far from the literature and user expectations.

- code organization: Outside GLM most models and transformations are defined by inverse link (some articles even use the term link for the inverse link, i.e. mean function. ) 
   This becomes relevant as we add more models that require one or several links or transformations #3915  
   Currently we use hard coded parameter transformation in several cases (mainly exp for positive)
- numerical efficiency and sensitivity to corner cases: this is the complementary issue to #4269. Would it be numerically more stable to use the mean function and it's derivative, instead of those defined by links? 
   Eg, `link.deriv` is 0 if evaluated at mu = 0, but we use generally often need `1/link.deriv`


(motivation: I'm reading/skimming too many old GLM articles which I guess only apply to canonical cases, but I'm mainly interested in the general cases, and it's work to figure out whether and what applies in the general case. If we have to do a general derivation of the equations or formulas, then using mean function is easier than link function.)



user: josef-pkt
issue_4276: state: open
title: ENH/Example: GLM-QMLE parameterized link and variance functions and margins
body: mainly adding a reference to an application that brings several parts together

- parameterized link function - Box-Cox transformation  #624
- parameterized variance function - either power or quadratic (similar to Tweedie and Neg Bin) #1777
- based on estimating equations, not an objective function, EE joint for mean and variance function
   #1790 (GLMGMM) #2860 (GLM based on EE)
- margins and mean/expectation modelling as main objective

also related in motivation and discussion:

- use transformed endog versus GLM-like index model #2791 and shows up in several other issues

bonus: They describe how to compute start_params

Basu, Anirban, and Paul J. Rathouz. 2005. “Estimating Marginal and Incremental Effects on Health Outcomes Using Flexible Link and Variance Function Models.” Biostatistics 6 (1): 93–109. https://doi.org/10.1093/biostatistics/kxh020.

two comment on their methods:

- computational: They use Box-Cox link directly which does not enforce the domain constraints. They need additional constraints on the estimation, linpred, params, to get a non-negative or numerically valid mean response prediction. (exception in box-cox with negative linpred ?)
- theoretical: Their standard errors for margins have an extra term for sampling variance (going from sample to population effect). AFAIK we and Stata don't use this, AFAIU, and it might be related to survey sampling inference. Our margins evaluate at sample/empirical distribution of exog and only take parameter uncertainty into account.

aside: 
Manning, Basu, and Mullahy. 2005 have an article on using Generalized Gamma in GLM style analysis, I haven't looked at it, but this could be another target when implementing Generalized Gamma (with log-link) for accelerated failure time AFT model.
i.e. more (general) models for continuous, positive or non-negative response variables with or without censoring. This might be a bit like a parametric analogue of the above QMLE.

Manning, Willard G., Anirban Basu, and John Mullahy. 2005. “Generalized Modeling Approaches to Risk Adjustment of Skewed Outcomes Data.” Journal of Health Economics 24 (3): 465–88. https://doi.org/10.1016/j.jhealeco.2004.09.011.

user: josef-pkt
issue_4269: state: open
title: ENH/REF  GLM simplify computation if is_canonical
body: triggered by looking at the computation of IRLS weights in #4267

We currently don't use any special casing and simplified computation in GLM when a canonical link function is used.

This would create some speed improvements, but the more important point is that it will most likely increase numerical stability and precision.

Example Pregibon 1981 on regression diagnostics points out that the weights in IRLS are zero for the corner case p=0 and p=1 in Binomial. However, we divide by the variance which divides by zero in this case, most likely 0/0. I guess we get currently around this with clipping away from the boundaries.

Other maybe more important cases are gamma and negative binomial where canonical link functions don't work so well, and the non-canonical log link works better.
I haven't checked, but my guess is that some of the messy instabilities in corner cases will likely disappear.

(a bit related: thequackdaddy added an EIM option for the hessian in fit to avoid the complicated OIM which is generic for non-canonical cases. However, we don't skip the unnecessary part for OIM in the canonical link case, where OIM==EIM, IIRC)


user: josef-pkt
issue_4268: state: open
title: ENH: GLMInfluence, extend OLSInfluence to WLS
body: this is a specific issue to get the analogue of stats.outliers_influence.OLSInfluence for GLM and WLS, 
It might be possibly to also extend to other models that follow the same pattern and can provide residuals and hat_matrix (maybe RLM and TheilRidge).

the more general issues are
#3336 general diagnostics and specification for other models
#3743 general GLM/WLS diagnostics, the current issue would implement the influence/outlier part
#4257 influence and outliers based on general properties of MLE and M-Estimators. the current issue is a special case where influence function, hessian, score are defined specifically for the model as OLS analogue.

#4267 one specific implementation part: attach weights and hat_matrix_diag so they can be used for the outlieres and influence measures

Implementation

preliminary work in gist and discussion in
https://github.com/statsmodels/statsmodels/issues/4267#issuecomment-368196483

For GLMInfluence we can essentially just translate formulas to code, e.g. Pregibon, Williams and a few other references (my reading was partially random, and I didn't check any GLM text books)

For WLS, I don't have a reference. Giltinan, Carrol and Ruppert 1986 consider the HetGLS case for WLS, i.e. it's a two-parameter model with influence also coming through the estimation of the variance function.
My guess is that for the fixed weights WLS (var_weights) case, we can just use wexog and wresid in all parts of OLSInfluence instead of the unweighted attributes. That sounds correct for influence on parameters and parameter influence. 
However, I guess we want the influence on prediction to be on the original endog.

unit tests, reference functions in other packages
I have no idea yet.
according to https://stackoverflow.com/questions/47686227/poisson-regression-in-statsmodels-and-r R has a glm.diag and glm.diag.plots
https://stat.ethz.ch/R-manual/R-devel/library/boot/html/glm.diag.html
I don't know why it is in `boot`, `boot` has license: "unlimited"
another
https://stats.stackexchange.com/questions/45401/how-to-validate-diagnose-a-gamma-glm-in-r


user: josef-pkt
issue_4267: state: open
title: GLM function not returning weights
body: I have run into a problem using the GLM function with Binomial family. I used the code below to create an instance of the glm:

logit_instance = sm.GLM(default_array, predictors_matrix, family=sm.families.Binomial())
result = logit_instance.fit()

From this result I would like to get the weights of the final iteration. According to documentation, this should be accessible using result.weights, but this does not work. Also the dir(results) does not show 'weights' or anything similar as an attribute of result.

Is there some  way to resolve this issue?
user: pinxterenm
issue_4266: state: open
title: ENH: outlier robust GLM using freq_weights
body: bumping an application using the freq_weights that are now available in GLM
(There should be related issues, but I didn't find any)

#2873 general issue for outlier robust estimation through weights.

references:
...  (don't remember)
downweight by both outliers and influence (y and x outliers)


my guess is that this should now be easy by just iterating over a GLM fit with updating of freq_weights.

(maybe a difficulty will be to get Fisher consistency in the weight function ???)


user: josef-pkt
issue_4264: state: open
title: ENH: Automatic model search - e.g. count models, parametric survival
body: a bit similar to the idea of automatic forecasting: Find best fitting distributional assumption in MLE models.

main advantages compared to users doing it themselves:

- predefined sequence, automatic search paths
- efficient use of start_params and no redundant computation of estimating a start_params model

outcome
- list of models with AIC, BIC, score test, LR/Vuong tests.

one open problem will be how to combine variable selection with distribution or model class selection

about start_params and sequence of models (some models are not available yet)

**count models**
What I'm thinking of is to automate the model search as I did in this gist https://gist.github.com/josef-pkt/c932904296270d75366a24ee92a4eb2f It is without variable selection, and start_params models are internal and invisible.
To improve convergence in some of the more difficult models with extra parameters, we estimate simpler models first to get better start_params, for example NegativeBinomial uses a Poisson initial estimate. Stata in some cases runs the constant-only model before estimating the full model. The auxiliary models for estimating start_params are not stored and attached, so we need to compute them again if we want to do model comparison.

In the count case we have several distribution Poisson, NegativeBinomial (with dispersion power parameter P as given parameter) and GeneralizedPoisson (also with P). Then we have the zero-inflated versions. Still to come are hurdle/zero-modified and truncated versions.

One problem is that many of these models have convergence problems or run into corner cases when the model is not appropriate for the data, similar to many time series models. (e.g. NegativeBinomial has problems if there is underdispersion because NB only works for overdispersion. A zero-inflated model doesn't make much sense if we have zero-deflation.)

The search path selection should have two objectives, both largely based on avoiding convergence problems.

- use simple models to get start_params for the messier models, i.e. bottom up selection
- use diagnostic tests (e.g. for over- or under-dispersion or for zero-inflation or deflation, or use, when available, the truncated model to fit only part of the endog. This should avoid "obviously" misspecified models

extras that will eventually become available:
- outliers and robust estimation, e.g. trim or down-weight observations identified as outlier candidates or bad influential points.

**example: parametric survival and accelerated failure time models.**

- simple model to flexible models: simplest is exponential, Weibul, log-logistic and log-normal are intermediate (one extra parameter), generalized log-gamma has 2 extra parameters, log-F distribution (I haven't looked at it yet) has more extra parameters and nests all others. Refernences mention that it is useful for model checking and selection, but it might be a nightmare to estimate unless we have good start_params.
generalized log-gamma nests Weibul and log-logistic and, at the boundary, log-normal. If we can get those to converge, then we can estimate generalized log-gamma from each of the nested models as start_params (and immediately compare loglike or information and fit criteria).

other specification choices
- AFT versus PH: assumption on basic parameterization
- should extra parameters be constant or depend on explanatory variables. e.g. in the log failure time version this would correspond to a heteroscedastic model where the scale/variance parameter also depends on some exog, which would then not be a standard AFT model anymore.




user: josef-pkt
issue_4263: state: open
title: Possible confusion in the documentation
body: I consider a two-sample one-sided test. Refer to this part of the documentation: https://github.com/statsmodels/statsmodels/blob/cd48743a8d721dc5e53b3b72db659da6fa779a33/statsmodels/stats/proportion.py#L729-L733

In the two-sample case, if I understand correctly, `prop` should be interpret as the proportion in the seconds sample and `value` as the one in the first sample. However, this seems to be the other way around.

Here's an example:

If we want to show that the proportion of the the first sample (`prop1`) is larger (or equal) to the proportion of the seconds sample (`prop2`) then we should take as  the null-hypothesis that `prop_1 < prop_2`. Then the alternative one is `prop1 >= prop2`. In this case, as per the documentation, we should run:

```python
import pandas as pd
from statsmodels.stats import proportion

proportion.proportions_ztest(data.COUNT, data.NOBS, alternative='smaller')
```
The results of an experiment are aggregated below:

```python
data = pd.DataFrame({
    "COUNT": [120, 145],
     "NOBS": [200, 300]
    }, index=['Sample1', 'Sample2'])
data['prop'] = data.COUNT / data.NOBS
```

Yielding:

```
         COUNT  NOBS      prop
Sample1    120   200  0.600000
Sample2    145   300  0.483333
```

In this case, it seems like the null hypothesis should be rejected. However, the p-value obtained is
```python
# p-valued yielded:
# 0.994776201476926
```

This is naturally a p-value which doesn't allow a rejection of the null hypothesis. Note that the other way around yields a low p-value:

```python
proportion.proportions_ztest(data.COUNT, data.NOBS, alternative='larger')
# pval: 0.005223798523074002
```

Am I misunderstanding the docs, the theory or is the docs actually wrong?


user: drorata
issue_4262: state: open
title: ENH/Design: diagnostics based on grouping observations by prediction
body: (another idea for diagnostic checks that should work in a relatively general way)

split observations into subgroups or strata by a predicted value, e.g. linear prediction in GLM or parametric survival models, and then run some less restrictive analysis on the subgroups and check whether the results for subgroups are consistent with the overall result.
This is an extension of analyzing subgroups if there are only a few exog patterns with discrete exog.


related ideas:

- Hosmer-Lemeshow test for gof in logit with some continuous exog works that way
- treatment effect estimation using subgroups, similar to or special case of propensity score matching
- data coarsening: general argument for discretization from authors of zelig in R
- parametric survival model, see below


parametric survival modelling

Collett, D. 2003. Modelling Survival Data in Medical Research. 2nd ed. Chapman & Hall/CRC Texts in Statistical Science Series. Boca Raton, Fla: Chapman & Hall/CRC.

page 241: split observations by `risk score`, i.e. linear predictor, and then run nonparametric analysis (Kaplan-Meier) for each subgroup to check whether parametric assumptions are ok.

possible problem: how many groups to use, e.g. problems of Hosmer-Lemeshow test if number of groups is kept small when sample size increases.
(possible idea: balance power and precision)

related #3904  Andrews has an Econometrica article for chisquare test similar to Hosmer-Lemeshow
The theory allows essentially arbitrary splitting by endog or exog.

user: josef-pkt
issue_4260: state: open
title: FAQ: How to fit a two-factor nested design (no random effects!)
body: How to fit a two-factor nested design in ``statsmodels``? I have two fixed factors in my experimental design: a fixed factor A and a (fixed) factor B which is nested in A. Both of them are fixed. The model equation would be:

   y_ijk = μ + α_i + β_j(i) + ε_ijk

with
  - μ intercept
  - α_i constant subject to Σ_i α_i = 0
  - β_j(i) constant subject to Σ_j β_j(i) = 0 for all i
  - ε_ijk ~ N(0,σ^2)

I want to test the hypothesis:

  H_0:  ∀i. α_i = 0     vs.     H_1:  ∃i. α_i≠0

Thank you very much!
Thomas
user: 00tau
issue_4258: state: open
title: Probability to stars utility
body: Would this be a fitting addition to this library? Equivalent to `R`'s representation of significance stars.

I think `statsmodels/tools/tools.py` would be a good place for it but do let me know if there is a more fitting location.

I can make a PR if it is an appreciated addition.

```python
def pvalue_to_stars(p):
    """
    Represent a p-value's significance with stars.

    Parameters
    ----------
    p : float

    Returns
    -------
    str
        Representation of the p-value's significance.

    """
    if not 0 <= p <= 1:
        raise ValueError('p must be in range [0, 1]')

    if p <= 0.0001:
        return '****'
    elif p <= 0.001:
        return '***'
    elif p <= 0.01:
        return '**'
    elif p <= 0.05:
        return '*'
    else:
        return ''
```

user: clintval
issue_4257: state: open
title: ENH: influence measure - general for MLE and M-estimators
body: **edit** see #4732 for updated information and implementation

(a random find)

Hosmer,Lemeshow, and May in chapter 8 refer to a influence function definition of Cooks Distance and use score residuals. They refer it to a Rand working paper that I don't have access to.
Their application is exponential and weibull survival or failure time model (AFT)

this is essentially
`cov_params dot score_obs_i`
and quadratic form (cooks distance
`score_obs_i dot cov_params dot score_obs_i`

Hosmer, David W., Stanley Lemeshow, and Susanne May. 2008. Applied Survival Analysis: Regression Modeling of Time-to-Event Data. 2nd ed. Wiley Series in Probability and Statistics. Hoboken, N.J: Wiley-Interscience.

Hall, Gaineford J., William H. Rogers, and Daryl Pregibon. "Outliers matter in survival analysis." (1982).
Rand working paper, no open access version is available

I haven't found any directly related article (in a brief search)

of related interest
influence of an observation on prediction (fittedvalues)

aside: AFAICS, most of the GLM influence literature uses the analogue to linear models (related to use of IRLS), and does not directly use a general definition. (related also to generalized projection matrix, ...)



user: josef-pkt
issue_4256: state: open
title: ENH: GAMLSS - Box-Cox Cole Green distribution
body: Would it be possible to include a functionality for fitting a model like in GAMLSS? (already mentioned in issue #3915 and #2636)
 
I am thinking of a simple model with one independent and one dependent variable: 
[GAMLSS example](https://raw.githubusercontent.com/xi2pi/reference-curves/master/docs/figures/ex_1_reference_curves.png)
It should have the option to choose the link functions and distribution. I am interested in fitting the Box-Cox Cole Green distribution (BCCG) published by [Cole](https://www.ncbi.nlm.nih.gov/pubmed/1518992) (also called LMS method). This method has found application in many medical fields for creating reference curves.

Rigby and Stasinopoulos give enough information in their publications for the implementation (e.g. RS() fitting procedure):
- R. A. Rigby and D. M. Stasinopoulos, “Smooth centile curves for skew and kurtotic data modelled using the Box-Cox power exponential distribution,” Stat. Med., vol. 23, no. 19, pp. 3053–3076, 2004.
- R. A. Rigby, D. M. Stasinopoulos, and P. W. Lane, “Generalized additive models for location, scale and shape,” J. R. Stat. Soc. Ser. C Appl. Stat., vol. 54, no. 3, pp. 507–554, 2005.

I tried to implement it but was not so lucky so far.
.



user: xi2pi
issue_4251: state: open
title: docs: running the test suite, sm.discrete.test() not available
body: I've been browsing through the developer docs found an example that does not work.

On http://www.statsmodels.org/stable/dev/test_notes.html#running-the-test-suite page

It says you can run test sub-modules with `sm.discrete.test()` which throws an error `AttributeError: module 'statsmodels.api' has no attribute 'discrete'`

I see that `statsmodels.api` does not import `.discrete` directely but imports classes inside of discrete.
user: jkornblum
issue_4241: state: open
title: WIP (do not merge): add optional numba case in _count_tsa
body: this is just a trial balloon to see how optional numba works on testing ci, travis and appveyor


user: josef-pkt
issue_4239: state: open
title: ENH/Design using numba
body: http://matthewrocklin.com/blog/work/2018/01/30/the-case-for-numba

Install problem  (using WinPython 3.4.4.5 )
import numba in idle resulted in dll load failure for llvmlite
import numba in Spyder mentioned the name of the missing Windows runtime dll
no problems after installing the runtime from the Microsoft website,

Trying out a simple example that builds a distance matrix and has only one part as reduce operation, see script below

Observations:

- numba is slower in single use
  numba seems to start fresh with JIT after each restart. Initially I ran it with restarting each time and 
  numpy was faster than numba except for huge intermediate arrays.
- warmed up numba is faster even in small memory cases
- adding an inner numpy reduce makes numba slow
- even cold numba is faster than numpy if huge intermediate arrays are required



k is the reduce dimension, final array that is created is (nobs, nobs)
numpy intermediate is (nobs, nobs, k)
(1000, 1000, 200) has 200 Million elements and takes 1.6 GB memory, intermediate array is 200 times larger than final result

```
nobs, k =  1000 5
0.2906498908996582  numba loop
0.5433850288391113  numba plus numpy reduce
0.05804109573364258  numpy vectorized

nobs, k =  1000 5
0.005995035171508789
0.20914912223815918
0.0570218563079834

nobs, k =  1000 5
0.007009983062744141
0.2091350555419922
0.05802583694458008

nobs, k =  1000 10
0.010007143020629883
0.21416091918945312
0.10007190704345703

nobs, k =  1000 20
0.018024921417236328
0.24717497825622559
0.1701200008392334

nobs, k =  1000 100
0.08105897903442383
0.3332359790802002
0.7575359344482422

nobs, k =  1000 200
0.18114495277404785
0.4610259532928467
1.5761082172393799
```


```
import numpy as np
import numba
import time


@numba.jit
def pairwise_python(X):
    M = X.shape[0]
    N = X.shape[1]

    D = np.empty((M, M), dtype=np.float)
    for i in range(M):
        for j in range(M):
            d = 0.0
            for k in range(N):
                tmp = X[i, k] - X[j, k]
                d += tmp * tmp
            D[i, j] = np.sqrt(d)
    return D

@numba.jit
def pairwise_python2(X):
    M = X.shape[0]
    N = X.shape[1]

    D = np.empty((M, M), dtype=np.float)
    for i in range(M):
        for j in range(M):
            tmp = X[i, :] - X[j, :]
            d = (tmp**2).sum()
            D[i, j] = np.sqrt(d)
    return D


nobs, k = 1000, 20
for k in [5, 5, 5, 10, 20, 100, 200]:
    x = np.random.randn(nobs, k)
    print('nobs, k = ', nobs, k)

    t0 = time.time()
    dd = pairwise_python(x)
    t1 = time.time()
    print(t1-t0)

    t0 = time.time()
    dd1 = pairwise_python2(x)
    t1 = time.time()
    print(t1-t0)

    t0 = time.time()
    ddn = np.sqrt(((x[:,None,:] - x[None, :, :])**2).sum(-1))
    t1 = time.time()
    print(t1-t0)
```

user: josef-pkt
issue_4236: state: open
title: REF/ENH: statespace, corner case zero variance
body: When I run the statespace test locally with `pytest -s`, then I see a huge number of invalid value in linalg.det. The following is a partial check of one case with pdb.

I guess it would be better to force variance and diag(cov) to be strictly positive, e.g. with a Ridge factor, to avoid numerical problems in the corner case where either variance is zero or a cov is singular.

Note, there are no nans or infs, numpy.linalg.det complains/warns because determinant is zero.


`>pytest -s -v  --pdb "M:\...\statsmodels_py34_pr\statsmodels\tsa\statespace\tests\test_smoothing.py"`

I guess the following refers to 
`cls.results.filter_results.predicted_state_cov[:, :, i]`

```
> c:\...\winpython-64bit-3.4.4.5qt5\python-3.4.4.amd64\lib\sit
e-packages\numpy\linalg\linalg.py(1776)det()
-> r = _umath_linalg.det(a, signature=signature)
(Pdb) a
a = array([[ 0.60579176,  0.        ,  0.15009155],
       [ 0.        ,  0.        ,  0.        ],
       [ 0.15009155,  0.        ,  0.60579176]])

(Pdb) import numpy as np
(Pdb) np.linalg.det(a)
*** FloatingPointError: invalid value encountered in det

(Pdb) np.linalg.det(a+1e-16)
3.4445618483717824e-17

(Pdb) np.linalg.det(a+ 1e-16*np.eye(*a.shape))
3.4445618483717824e-17
```
user: josef-pkt
issue_4234: state: open
title: ENH/REF/TST: prob_weights versus Stata pweights
body: bottom line: I might not agree with Stata's definition of pweights

- Stata seems to use aweights + HC1  
   (according to comments in Stata documentation and reverse engineering the results

- I think prob_weights should be freq_weights (scaled to nobs) + HC1 (or other robust cov_type)

test case TestGlmPoissonPwNr (which uses vcv robust in Stata, i.e. HC0/HC1), currently has xfail (known failures)

If I switch to var_weights + HC1, then the test passes for bse with `corr_fact = np.sqrt((nobs-1)/nobs)`
The version to check this
```
class TestGlmPoissonPwNr(CheckWeight):
    @classmethod
    def setup_class(cls):
        fweights = [1, 1, 1, 2, 2, 2, 3, 3, 3, 1, 1, 1, 2, 2, 2, 3, 3]
        # faking aweights by using normalized freq_weights
        fweights = np.array(fweights)
        wsum = fweights.sum()
        nobs = len(cpunish_data.endog)
        fweights = fweights / wsum * nobs

        cls.res1 = GLM(cpunish_data.endog, cpunish_data.exog,
                    family=sm.families.Poisson(), var_weights=fweights).fit(cov_type='HC1')
        # compare with discrete, start close to save time
        #modd = discrete.Poisson(cpunish_data.endog, cpunish_data.exog)
        cls.res2 = res_stata.results_poisson_pweight_nonrobust
```

for bse:
```
(Pdb) corr_fact = np.sqrt(16/17)
(Pdb) corr_fact
0.97014250014533188
(Pdb) res1.bse /( corr_fact * res2.bse)
array([ 1.00000038,  1.        ,  0.99999991,  0.99999979,  0.99999997,
        1.00000012,  1.00000002])
```

however resid_pearson does not include var_weights in this case in Stata

```
(Pdb) res1.resid_pearson/np.sqrt(res1.model.var_weights) / resid_all['resid_pear
son']
array([ 1.00000012,  1.00000024,  1.00000006,  0.99999758,  1.00000031,
        0.99999996,  0.99999925,  1.00000049,  1.00000003,  1.00000014,
        1.00000029,  0.99999979,  1.00000002,  1.00000092,  0.99999964,
        1.00000018,  0.99999999])
```

pearson chi2 is strange  (33 is sum of original weights before normalizing to nobs=17)
Stata uses the original sum of weights, not the normalized sum of weights.

```
(Pdb) res2.deviance_p / (res1.resid_pearson**2).sum() / (33/17)
0.99999994482509491

(Pdb) res2.deviance_p / res1.pearson_chi2 / (33/17)
0.99999994482509491
```

Conclusion: While we can reproduce Stata's results, I'm not sure we want to use the same definitions.
Alternative or to confirm: check what Stata's `svy` prefix does with simple inverse probability sampling.

Aside: freq_weights + HC0 agrees with Stata according to TestGlmPoissonFwHC

(I don't change TestGlmPoissonPwNr for now and it remains xfails)

user: josef-pkt
issue_4233: state: open
title: ENH: random sampling with kde, KDEMultivariate
body: KDEMultivariate does not have a rvs or sampling method
https://stats.stackexchange.com/questions/325206/sampling-a-statsmodel-kdemultivariate

I have currently no clear idea how to do this and never looked into it.

here is a previous answer for the univariate case using ppf
https://stackoverflow.com/questions/40513711/generating-random-numbers-for-testing-a-kernel-density-estimate

scipy.gaussian kde uses sampling a point plus normal noise corresponding to the kernel.

KDEMultivariate uses a product kernel which would imply that we can add independent noise (AFAIU). So we would need samplers/rvs for the different kernels including ordered discrete. (unordered categorical does not smooth or averages all other observations ?) 


user: josef-pkt
issue_4232: state: open
title: ENH: count time series
body: 
parking a reference to an R package

Liboschik, Tobias, Konstantinos Fokianos, and Roland Fried. 2017. “Tscount : An R Package for Analysis of Count Time Series Following Generalized Linear Models.” Journal of Statistical Software 82 (5). https://doi.org/10.18637/jss.v082.i05.

I was reading on this topic in the past including articles by Fokianos. Having an R package should make the design and unit testing easier.
Using something GLM like should be easier than the INAR models based on `thinning`


related, similar purpose but different model type are GAS models, available in pyflux
http://www.pyflux.com/docs/gas.html


user: josef-pkt
issue_4230: state: open
title: TST: investigate and silence convergence warnings for discrete
body: see #3845, and companion to #4229 

The following are the convergence warnings with discrete models, in base and discrete unit tests.
from https://travis-ci.org/statsmodels/statsmodels/jobs/333445133

- check why we still have so many convergence failures, I thought it should be better after the latest 
improvements to start_params in discrete count models

- silence those that cannot be fixed with catch_warnings. In most cases we should have verified numbers that test whether the results are correct independent of warnings.

The zero-inflated count_model cases might still be waiting for the truncated models to get better start_params

```
statsmodels/base/tests/test_shrink_pickle.py::TestRemoveDataPickleNegativeBinomial::()::test_remove_data_pickle
  /home/travis/build/statsmodels/statsmodels/statsmodels/base/model.py:511: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
    "Check mle_retvals", ConvergenceWarning)
statsmodels/base/tests/test_shrink_pickle.py::TestRemoveDataPickleNegativeBinomial::()::test_remove_data_docstring
  /home/travis/build/statsmodels/statsmodels/statsmodels/base/model.py:511: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
    "Check mle_retvals", ConvergenceWarning)
statsmodels/base/tests/test_shrink_pickle.py::TestRemoveDataPickleNegativeBinomial::()::test_pickle_wrapper
  /home/travis/build/statsmodels/statsmodels/statsmodels/base/model.py:511: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
    "Check mle_retvals", ConvergenceWarning)
statsmodels/discrete/tests/test_count_model.py::TestZeroInflatedModel_logit::()::test_params
  /home/travis/build/statsmodels/statsmodels/statsmodels/base/model.py:511: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
    "Check mle_retvals", ConvergenceWarning)
statsmodels/discrete/tests/test_count_model.py::TestZeroInflatedModel_probit::()::test_params
  /home/travis/build/statsmodels/statsmodels/statsmodels/base/model.py:511: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
    "Check mle_retvals", ConvergenceWarning)
statsmodels/discrete/tests/test_count_model.py::TestZeroInflatedModel_offset::()::test_params
  /home/travis/build/statsmodels/statsmodels/statsmodels/base/model.py:511: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
    "Check mle_retvals", ConvergenceWarning)
statsmodels/discrete/tests/test_count_model.py::TestZeroInflatedPoisson_predict::()::test_mean
  /home/travis/build/statsmodels/statsmodels/statsmodels/base/model.py:511: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
    "Check mle_retvals", ConvergenceWarning)
statsmodels/discrete/tests/test_count_model.py::TestZeroInflatedGeneralizedPoisson::()::test_params
  /home/travis/build/statsmodels/statsmodels/statsmodels/base/model.py:511: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
    "Check mle_retvals", ConvergenceWarning)
statsmodels/discrete/tests/test_count_model.py::TestZeroInflatedGeneralizedPoisson::()::test_minimize
  /home/travis/build/statsmodels/statsmodels/statsmodels/base/model.py:511: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
    "Check mle_retvals", ConvergenceWarning)
statsmodels/discrete/tests/test_count_model.py::TestZeroInflatedGeneralizedPoisson_predict::()::test_mean
  /home/travis/build/statsmodels/statsmodels/statsmodels/base/model.py:511: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
    "Check mle_retvals", ConvergenceWarning)
statsmodels/discrete/tests/test_count_model.py::TestZeroInflatedNegativeBinomialP::()::test_minimize
  /home/travis/build/statsmodels/statsmodels/statsmodels/base/model.py:511: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
    "Check mle_retvals", ConvergenceWarning)
statsmodels/discrete/tests/test_count_model.py::TestZeroInflatedNegativeBinomialP_predict::()::test_mean
  /home/travis/build/statsmodels/statsmodels/statsmodels/base/model.py:511: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
    "Check mle_retvals", ConvergenceWarning)
statsmodels/discrete/tests/test_count_model.py::TestZeroInflatedNegativeBinomialP_predict2::()::test_mean
  /home/travis/build/statsmodels/statsmodels/statsmodels/base/model.py:488: HessianInversionWarning: Inverting hessian failed, no bse or cov_params available
    'available', HessianInversionWarning)
statsmodels/discrete/tests/test_discrete.py::TestLogitL1Compatability::()::test_params
  /home/travis/build/statsmodels/statsmodels/statsmodels/base/model.py:511: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
    "Check mle_retvals", ConvergenceWarning)
statsmodels/discrete/tests/test_discrete.py::TestProbitL1Compatability::()::test_params
  /home/travis/build/statsmodels/statsmodels/statsmodels/base/model.py:511: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
    "Check mle_retvals", ConvergenceWarning)
statsmodels/discrete/tests/test_discrete.py::TestL1AlphaZeroLogit::()::test_converged
  /home/travis/build/statsmodels/statsmodels/statsmodels/base/model.py:511: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
    "Check mle_retvals", ConvergenceWarning)
statsmodels/discrete/tests/test_discrete.py::TestGeneralizedPoisson_transparams::()::test_bse
  /home/travis/build/statsmodels/statsmodels/statsmodels/discrete/discrete_model.py:1369: RuntimeWarning: Paramter "use_transparams" is ignored
    RuntimeWarning)
statsmodels/discrete/tests/test_discrete.py::TestNegativeBinomialPNB1Newton::()::test_llf
  /home/travis/build/statsmodels/statsmodels/statsmodels/discrete/discrete_model.py:3095: RuntimeWarning: Parameter "use_transparams" is ignored
    RuntimeWarning)
statsmodels/discrete/tests/test_discrete.py::TestNegativeBinomialNB1Null::()::test_llnull
  /home/travis/build/statsmodels/statsmodels/statsmodels/base/model.py:511: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
    "Check mle_retvals", ConvergenceWarning)
statsmodels/discrete/tests/test_discrete.py::test_unchanging_degrees_of_freedom
  /home/travis/build/statsmodels/statsmodels/statsmodels/base/model.py:511: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
    "Check mle_retvals", ConvergenceWarning)
```
user: josef-pkt
issue_4222: state: open
title: coint always warns: UserWarning: y0 and y1 are perfectly colinear.  Cointegration test is not reliable in this case.
body: Hello!

I may well have misunderstood the use of the `coint` function, but I cannot use it without getting the following warning: 

`UserWarning: y0 and y1 are perfectly colinear.  Cointegration test is not reliable in this case.`

For example, which I stole from #3190:

```python
from statsmodels.tsa.stattools import coint
zz = np.random.randn(500).cumsum()
coint(zz, zz, maxlag=3, autolag=None)
``` 

Produces this warning, which is odd. Can anyone tell me what is wrong?

I'm using statsmodels v 0.8
user: wjdhamilton
issue_4220: state: open
title: REF: Kernel density estimation cleanup and improvement
body: I would like to help cleaning up the code related to kernel density estimation, and if possible also improve it. Is this of any interest, @josef-pkt ? If so, some help getting started would be great!

## General observations

Some general observations.

- The ``statsmodels/nonparametric`` package has not gotten much attention recently.
- There is a ``statsmodels/sandbox/nonparametric`` package too, it has also received little attention.
- According to [this 2013 blog post](https://jakevdp.github.io/blog/2013/12/01/kernel-density-estimation/#Scaling-with-the-Number-of-Points) by Vanderplas, the ``statsmodels`` KDE's are slower than scipy/sklearn.
- No popular/maintained Python package appears to implement [variable kernel density estimation](https://en.wikipedia.org/wiki/Variable_kernel_density_estimation) yet. This seems like a great opportunity.
- The general quality of the code in ``statsmodels/nonparametric`` is mixed. Some modules appear OK, but some appear unused. There are many [PEP8](https://www.python.org/dev/peps/pep-0008/) violations. There are 22 TODOs in the code which might indicate starting points for improvements (11 in ``statsmodels/sandbox/nonparametric`` too).

## Suggested improvements

I would like to help improve the quality of code associated with KDE. My ideas are, in order of least-to-most ambitious:

- Remove files which are not used (it has been years, if it has not been completed by now, it probably never will be completed).
- Read over the files that are in use. Conform to PEP8, write more comments. Re-factor the code if needed. Re-factor tests if needed.
- Increase speed so that we match the sklearn/scipy implementations.
- Read up on literature, implement more advanced KDE algorithms (e.g. variable bandwidth).

## Questions and advice

- General thoughts and comments on the above is greatly appreciated.
- The main question: How positive are you to these suggested changes in general, @josef-pkt ? (or others!)
- Is deleting unused files OK? (especially in the sandbox)
- If I start implementing some algorithms/code, what is the threshold for uploading my work to the sandbox?


I feel like I should add a few sentences about myself. I've been programming in Python for 3-4 years, have a degree in math, and I am interested in learning more about statistics/numerics/programming through a project of suitable difficulty. Contributing to statsmodels seems like a great place to start. I hope you will be positive to my ideas and suggestions, and would appreciate any feedback on the above.

-Tommy


user: tommyod
issue_4217: state: open
title: ENH: parametric survival/failure time models
body: (I don't see an issue, but I thought it was mentioned before.)

Stata has accelerated failure time and proportional hazard parametric survival models. Implementing those is a GSOC idea that doesn't require a heavy statistics background, similar to last years count models.

parking some references:
the two books seem to have enough math, but I didn't read much of them

Kalbfleisch, John D., and Ross L. Prentice. 2002. The Statistical Analysis of Failure Time Data. Wiley Series in Probability and Statistics. Hoboken, NJ, USA: John Wiley & Sons, Inc. https://doi.org/10.1002/9781118032985.
chapter two has the intro to exponential, weibull and other parametric models, including log-linear AFT.

Lee, Elisa T., and John Wenyu Wang. 2003. Statistical Methods for Survival Data Analysis Wiley Series in Probability and Statistics. Hoboken, NJ, USA: John Wiley & Sons, Inc. https://doi.org/10.1002/0471458546.
the parts are a bit spread out, chapter 6 distributions, chapter 7 parametric models without covariates, chapter 11 with covariates.

my guess

- starting with exponential and weibull in continuous time should be easy
- use those two to get general setup like class structure, main methods, Pattern should follow Poisson and similar with log-link/exponential mean function
- add required extra methods to parametric distribution, hazard, ...
- data handling and supporting exploratory graphical support for surviaval/failure times
- numerical details for derivatives, score, hessian, predict AFAPossible
- (extras for later) diagnostics, goodness-of-fit, gof, comparing nested or non-nested models

One of the extensions asked for in scipy is estimation just of distribution parameters (no covariates) for general censored models.

related parts: discrete time, grouped or interval censored data, see also #1764 #2763 #2764
(semi-parametric survival with Poisson or GLM)






user: josef-pkt
issue_4216: state: open
title: DOC: Added explanation of fdr_bh to docstring of fdrcorrection
body: Added a bit of additional documentation to the fdrcorrection method so that it's easier to follow how the Benjamini-Hochberg algorithm is implemented. Suggestions for improvement most welcome.
user: levlitichev
issue_4215: state: open
title: FAQ/ENH: categorical exog - interpretation of pinv regularization, constraints
body: (random thoughts e.g. age period cohort survival model with Poisson)

comparing pinv/PCA, ridge with tiny penal_weight and constraints if we have several categorical variables and their interaction but we use the full over-parameterized set of dummy variables.

Question: Do pinv and Ridge give the same result as the sum of dummy effects = 0 constraint?

My guess is yes, but it needs to be tried out. And maybe it only holds in the linear model, and maybe only for balanced data.

usage: identification with over-parameterized models, pinv comes for free, ridge is easy, constraints need to know the composition of exog.

related: imposing informative priors on categorical effects, e.g. all effects are close to each other, or are sequentially close if ordered (structured prior covariance in Ridge/Theil), issue ?


user: josef-pkt
issue_4212: state: open
title: TST/ENH/Reference: clinical trials R packages
body: just found this through a Quora question (I didn't look at any details, just for an overview)

There are too many things that we don't have. But it has a good list of R packages that we can use to write unit tests against in the proportion, rates, contingency tables and similar hypothesis tests, confidence interval and power computation.
see also #3954, #2718 (poisson rates), #2605 (2-sample proportion)

with list of R packages:
https://www.quora.com/What-are-the-important-machine-learning-and-statistical-packages-that-you-use-in-R-that-dont-have-an-equivalent-available-in-Python/answer/Adrian-Olszewski-1

with list of R task views
https://www.quora.com/What-domain-specific-statistical-capabilities-exist-in-R-but-do-not-in-Python


some that sounded interesting
- https://cran.r-project.org/web/packages/asypow/asypow.pdf
     similar to post-estimation power computation based on Wald (or Likelihood Ratio ?) test, using asymptotic covariance, parameters and constraints 
     based on skimming the function description, cites only one general reference
     I guess related to #3676, and computing non-centrality for local alternatives

- https://cran.r-project.org/web/packages/binomSamSize/index.html
  based on TOC, it sounds like we have those already at least for 1-sample case, but likely not all

- https://cran.r-project.org/web/packages/PowerTOST/index.html
  I wrote some but don't remember whether I used this, I guess not, since I was using GPower as reference

- https://cran.r-project.org/web/packages/TrialSize/TrialSize.pdf
  this has good coverage of different tests, one sample, two sample, ratio, ....
  based on book, which I most likely looked at and AFAIR it is all based on asymptotics and normal distribution. IIRC, then it's not that interesting because it's mostly "generic" z-test with transformed variables. (breadth but not enough depth ?)


completely missing:

- analysis of two- or multi-stage designs
- design of experiments


just a thought
The TOC in these and packages or references in other issues (?) are a pretty laundry list.
One requirement will be to find a structure that doesn't require "more than 80 functions" that essentially reduce to a just a few generic computations. 
In other cases we do have a lot of `method` options as in proportion confidence interval once we allow for various small sample approximations or improvements.
Also, having explicit function minimizes how much background we have to know or remember to use it.
(e.g. what is the transformation to make a poisson rate closer to normal? IIRC it's sqrt. for binomial it's ..., for correlation it is fisher z or similar.)

user: josef-pkt
issue_4210: state: open
title: TST: comparing models, e.g. GLM GEE - better support with parameterized tests
body: background:
using a line count recipe, about 38% of our lines of code are unit tests, or 50% if we include counting modules in tests/results (but those are at least partially autogenerated, e.g. in Stata).
Those 40% of our code in unit tests are only marginally reviewed.

specific case: we need reusable set of GLM families.
Kerby wrote a list for test_glm.test_gradient_irls, thequackdaddy copied and reused it in test_glm_weights.test_wtd_gradient_irls

I would like something similar to test GEE against GLM, e.g. for #4209 

We have similar unit test for equivalent models, GLM-Gaussian versus OLS/WLS, GLM versus discrete, I needed something similar for fixing fit_constrained in GLM-Gaussian versus reparameterized OLS, and so on. base generic tests are also exploiting some "tautologies" or equivalences to cross-check results.

I think we can streamline unit test and coverage by using more helper functions and parameterized tests (as newer version of yield based tests similar to what I wrote for scipy's distributions)

This is in almost all cases not a substitute for verifying unit test for the core result of models, but it should make it possible to more easily add unit tests for extra results.
user: josef-pkt
issue_4192: state: open
title: ENH: fit_constrained attach also transformation
body: investigating problems in fit_constrained with weights
https://github.com/statsmodels/statsmodels/pull/3856#issuecomment-355788921

currently the transformation is not attached to the results, so we cannot check from the outside whether it is correct.

base._constraints.fit_constrained uses `transf = TransformRestriction(R, q)` but doesn't return it.
Changing the number of returns will not be backwards compatible but it is an internal function.

user: josef-pkt
issue_4191: state: open
title: ENH: Control Charts
body: Getting started with Control Charts, mainly multivariate where statistics is similar to several other issues.
I put a script in the next comment with some experiments done in a few hours.
(Issue so I find this again, I will come back to it another time.)

current reference:
Bersimis, Psarakis, Panaretos (2007): Multivariate Statistical Process Control Charts: An Overview

many open design issues

example Hotelling T2 shewart chart (one observation)

3 distributions for mahalanobis distance (details somewhere in a robust issue or PR)
- chisquare: in large samples or known covariance
- beta: insample estimate of covariance, evaluation of in-sample observation
- F: covariance estimated in a different sample, evaluation of out-of-sample observation

in control charts: beta is used in phase one, F is used in phase two
that is we need to distinguish of an observation is in-sample or out-of-sample

what information do we need and keep track of:

- "static" charts like shewart chart
   - need only mean and standard deviation or control limits
   - we need estimation sample for updating the estimates 
   - we need some history, recent observations if we want to plot
- "dynamic" charts like EWMA or CUSUM
   - need also last "state" to evaluate next observation, others are the same 

Note: sample for updating estimation and sample for plotting with recent history do not use the same observations if the history includes out-of-control/outlier observations. Those are plotted but not included in the estimation.

Sheward, EWMA or CUSUM are the basic charts, but then we also have model based charts, e.g. using regression or time series models with charting of residuals or forecast error, and decomposition, PCA, PLS, based charts.

a different pattern are self-starting and automatic updating charts: those don't distinguish between phase 1 estimation and phase 2 charting.

current version 3 classes with largely copy-paste because I don't know what the common attributes will be.

MEWMA doesn't have explicit distribution for control limits. The usual method seems to be simulation, but bootstrap is also possible.
user: josef-pkt
issue_4190: state: open
title: ValueError in proportion_confint(method = "binom_test")
body: Hello, I am using statsmodels 0.8.0 and when calling proportion_confint with method 'binom_test' I get the next error: "ValueError: f(a) and f(b) must have different signs".  (Full error message copied below)

It only happens with certain values of count and nobs, for example when count=1 and nobs=103. 

Thanks in advanced!

 
The error message is:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-4-26ee07969933> in <module>()
----> 1 proportion_confint(1,103,method='binom_test')

~/.virtualenvs/negative-scoring/lib/python3.6/site-packages/statsmodels/stats/proportion.py in proportion_confint(count, nobs, alpha, method)
     84             ci_low = 0
     85         else:
---> 86             ci_low = optimize.brentq(func, float_info.min, q_)
     87         if count == nobs:
     88             ci_upp = 1

~/.virtualenvs/negative-scoring/lib/python3.6/site-packages/scipy/optimize/zeros.py in brentq(f, a, b, args, xtol, rtol, maxiter, full_output, disp)
    517     if rtol < _rtol:
    518         raise ValueError("rtol too small (%g < %g)" % (rtol, _rtol))
--> 519     r = _zeros._brentq(f,a,b,xtol,rtol,maxiter,args,full_output,disp)
    520     return results_c(full_output, r)
    521 

ValueError: f(a) and f(b) must have different signs
user: ulisesvid
issue_4188: state: open
title: ENH: Add support for on-line filtering in state space models.
body: It would be nice to add support for getting filtered output for new observations in state space models (see e.g. https://github.com/statsmodels/statsmodels/issues/2788#issuecomment-354781682). Presumable this would look something like:

```python
mod = sm.tsa.SARIMAX(endog, order=(1, 1, 0))
res = mod.fit()

while True:
    updated_endog = get_new_observations()
    res.append_observations(updated_endog)
```

Note that this is not really the same thing as the cross-validation type questions we get, since on-line filtering implies adding new observations that are continuous with the existing observations, whereas cross-validation could in principle have any structure.

A couple of issues:

- Should `res.append_observations(updated_endog)` alter the existing results object, or should it return a new object? If it returns a new object, should it duplicate all the data in the original object, or should it only correspond to the updated objects?
- At least for the first version of this, I think we should not try to update the smoothed output. (There are formulas to update at least the smoothed state and smoothed state covariance, but there is other smoothed output that I don't know how to update).
    - If we're just modifying the original results object, then I think that at first if on-line filtering is used, we should probably actually delete the smoothed results arrays and issue a warning that we've done so. This is so that there's no confusion about what's in the result object (e.g. smoothed results with different `nobs` than the filtered results).
- If the state space model has time-varying system matrices, then these would need to be given in the `append_observations` call, similar to the requirements we currently have in the `forecast` calls.

I guess people will try to use this for cross-validation, but a warning in the documentation it shouldn't be used for cross-validation is probably the best we can do.

Other thoughts @jengelman?
user: ChadFulton
issue_4187: state: open
title: KernelReg performance in a for loop
body: Hi, everyone
So sorry to post this under Issues, I can't enter google account and mailing list.

I'm new in python. I have a task to fit many 1D profile(about 10000 profiles) obtained from electron beam to Gaussian. 

Since the raw data is basically very noisy, I had to denoise before fitting, and KernelReg is a good way to do this as far as I know(in fact, I Know little for denoising).

For each profile, I firstly call KernelReg and then lmfit to extract the center, sigma, amp and offset of raw data in a for loop.

However, depending on the test of 100 data, if I use only lmfit the runtime is 2.4 seconds(cprofiler), and if combining KernelReg and lmfit the runtime would be 272 seconds.

The cprofiler displays a bottleneck in the KernelReg call.

So, my question is **how to improve the performance of KernelReg call**, and **is this a good choice for denoising and fit using KernelReg+lmfit**?.

Best Regards!
user: Landau1908
issue_4182: state: open
title: ENH: Add plot_states and Pandas filtered state functions to MLEModel
body: A common part of post-estimation work for statespace models is looking at how the state variable means/variances change over time.  Both UnobservedComponents and RecursiveLS have really nice helper functions for plotting state means.  It'd be great to get something similar for SARIMAX, VARMAX, et al and custom models without having to subclass all of them, by adding this to MLEModel.  Additionally, functions that return the filtered states and state covariances in Pandas dataframes would be really convenient.  

As far as I can tell, this could be done by reusing the existing code from RecursiveLS and a bit of refactoring for the existing two models.  Happy to submit a PR for either or both parts of this if there aren't further complications I'm not aware of.
user: jengelman
issue_4179: state: open
title: SUMM/Overview/Roadmap: multivariate
body: What's the status and roadmap for multivariate Statistics

## main methods

either available or started in statsmodels (or related packages)

* Models
  * _MultivariateOLS
  * SUR
  * ...
  * GMM (can we get more explicit or direct multivariate, multi equation support?)
  * panel data, GEE, system of equations, VAR, VECM, multivariate statespace models,
    two-part or multi-part models (e.g. heckman type, bivariate Probit)
    many related models, not in narrow definition of multivariate statistics
  * large sparse versions of Models

* Principal Components Analysis (PCA)

* Canonical Correlation Analysis (CCA)

* Factor Analysis

## other methods

some are available in scikit-learn, those are not high priority unless there are specific results statistics that are important.

* Partial Least Square (PLS)
* Classical Multidimensional Scaling (MDS)
* Linear Discriminant Analysis (LDA)
* Multiclass LDA
* Independent Component Analysis (ICA), FastICA
* Probabilistic PCA
* Kernel PCA
* Correspondence Analysis
* Cluster Analysis (scipy)
* analogues for ordered variables: ?, e.g. item response models
* Redundancy Analysis

## standalone and supporting hypothesis tests

hypothesis tests and related statistics including confidence intervals

* tests on mean, Hotelling, multivariate control charts
* MANOVA
* tests of covariances and correlation, covariance structure analysis
* gof tests for distributions
* rank test
* test for non-continuous data, e.g. multinomial, multivariate GLM
* non-Pearson correlation and association measures (polychoric, robust, regularized, ...), 
  (kendalltau and spearman in scipy)

## supporting code

* factor rotation, procrustes
* matrix tools
* multivariate transformation, e.g. compositional
* correlation_tools, closest positive definite
* missing values, imputation (what and where does it belong?)

R taskview also lists multivariate distributions including copulas

list of methods partially taken from https://github.com/JuliaStats/MultivariateStats.jl, TOC of Stata mv, and R taskview https://cran.r-project.org/web/views/Multivariate.html

## Todo, Priorities

???
whatever someone is working on and contributes

user: josef-pkt
labels: roadmap: roadmaps, list of todos, overall and by topic
issue_4178: state: open
title: ENH: Include FRED-MD dataset?
body: I wonder if we might think of including the FRED-MD dataset in Statsmodels; see https://research.stlouisfed.org/econ/mccracken/fred-databases/.

**Cons**: it's about 0.5 mb

**Pros**: It provides a large number of curated macro time series could be useful for users, and could be used for examples and unit tests. It's also straightforward to keep updated each release, since we just replace it with the new CSV file from the website.

One motivation is that I often use the `macrodata` dataset for simple examples, but it's getting out of date and doesn't include very many series.
user: ChadFulton
issue_4177: state: open
title: ENH:  factors, PCA, PLS with 3d, functional data and forecasting
body: partially related to https://github.com/statsmodels/statsmodels/issues/4175#issuecomment-353568943
and multivariate control charts and monitoring #3296
and functional data analysis #3836

2 references below (I read the first but not the second) application is process control where process need to run for a fixed time (batch) until we see the outcome, final product.

The interesting part is in prediction and outlier, out-of-control detection based on partial information about the current batch or process run.

switching to within day prediction analogy: Suppose we have, e.g., hourly data observed for many days, treat within day as a functional process, not a time series process.
Deseasonalize, remove average of same hours. After a few hours of observations predict the deviation from the deterministic seasonal component for the remainder of the day and whether it will be a "regular" day.
I haven't seen this prediction problem in functional statistics/econometrics (given also that I never looked much at it).

I'm thinking more about the functional data analysis analogy, but the actual approach is multivariate statistic, i.e. observations are batches x variables x batch-observations    (or days x variables x hours), reshaped into 2-dimensional with batches in rows   (reshape(n_batches, -1))  
(variables x batch-observations) make one large multivariate variable for each batch with assumed independence across batches.

We don't have anything for this forecasting task, and I don't have a recipe for it either. We need some extra code, because it does not come out of the same analysis of historical "complete" data.
(Can we predict early in a year, whether it will be an elnino year? temperature data in datasets, examples in graphics.functional.
Another similar story but largely parametric: a long time ago I was following a webpage that predicted the final movie gross based on the sales in opening weekend and from the first few weeks. One difference there is that the number of observations in a batch or seasonal cycle is not fixed.)

Background approach in the first article is PLS. The prediction or process control based on partial information is handled by treating the remaining observations of the same day/batch as missing values.
Second article is referenced for the missing value handling and partial projection in PCA and PLS.


Nomikos, Paul, and John F. MacGregor. 1995b. “Multi-Way Partial Least Squares in Monitoring Batch Processes.” Chemometrics and Intelligent Laboratory Systems, InCINC ’94 Selected papers from the First International Chemometrics Internet Conference, 30 (1):97–108. https://doi.org/10.1016/0169-7439(95)00043-7.

Nomikos, Paul, and John F. MacGregor. 1995a. “Multivariate SPC Charts for Monitoring Batch Processes.” Technometrics 37 (1):41–59. https://doi.org/10.2307/1269152.


another article to match up terminology in some fields: tensor based for 3-D data structure

It's just a literature survey, no formulas

Fanaee-T, Hadi, and João Gama. 2016. “Tensor-Based Anomaly Detection: An Interdisciplinary Survey.” Knowledge-Based Systems 98 (Supplement C):130–47. https://doi.org/10.1016/j.knosys.2016.01.027.


user: josef-pkt
issue_4175: state: open
title: ENH: canonical correlation as supervised PCA, Factor Analysis
body: (just an idea, I haven't read anything specific about this)

Decomposition methods like PCA or Factor Analysis do not necessarily get the factors that are most important or explanatory in regression, i.e. explaining other outcome variables.

(In the wine example that I used for factor analysis, the factors from the wine characteristics don't seem to be closely related to the regression of quality on those characteristics. PCA regression has usually strong theoretical assumptions on the factor structure which does not always hold if it's not dominated by "global" effects, e.g. global business cycle versus geographic/spatial heterogeneity, or perfect versus monopolistic competition as market size grows.)

CanCorr captures the "supervised" relationship between explanatory and outcome variables. 
The idea is that we can use rotation of the exog factors similar to PCA to get interpretable factors as latent explanatory variables.

Wikipedia on Canonical Correlation has one reference for the regression application
Tofallis, Chris. "Model building with multiple dependent variables and constraints." Journal of the Royal Statistical Society: Series D (The Statistician) 48, no. 3 (1999): 371-378.
(article is too verbal, explains using Excel, and not much formulas/math/derivation)

My guess is that it will be similar to reduced rank regression. #2625 (and maybe #1111 for hypothesis and rank testing), and maybe some tools and methods overlap.
(also I never looked enough at FAVAR)

(see also https://github.com/statsmodels/statsmodels/issues/4159#issuecomment-350517110 for other possible "model or method sharing")

**edit** adding some references for rotation in CCA, mostly 70's

Cliff, Norman, and David J. Krus. 1976. “Interpretation of Canonical Analysis: Rotated vs. Unrotated Solutions.” Psychometrika 41 (1):35–42. https://doi.org/10.1007/BF02291696.

Krus, David J., Thomas J. Reynolds, and Patricia H. Krus. 1976. “Rotation in Canonical Variate Analysis.” Educational and Psychological Measurement 36 (3):725–30. https://doi.org/10.1177/001316447603600320.

Satomura, Hironori, and Kohei Adachi. 2013. “Oblique Rotaton in Canonical Correlation Analysis Reformulated as Maximizing the Generalized Coefficient of Determination.” Psychometrika 78 (3):526–37. https://doi.org/10.1007/s11336-012-9310-4.

There is also a literature and debate on PLS-SEM, that sounds related but I haven't read anything, e.g. as starting point

Bentler, Peter M., and Wenjing Huang. 2014. “On Components, Latent Variables, PLS and Simple Methods: Reactions to Rigdon’s Rethinking of PLS.” Long Range Planning, Rethinking Partial Least Squares Path Modeling: Looking Back and Moving Forward, 47 (3):138–45. https://doi.org/10.1016/j.lrp.2014.02.005.

user: josef-pkt
issue_4172: state: open
title: BUG/ENH: rotation promax is not "standard"
body: likely bug, possibly method or algorithm that doesn't match Stata or R
see https://github.com/statsmodels/statsmodels/issues/4164#issuecomment-352763437

promax rotation does not match Stata nor R
possibly #4171 procrustes method

power parameter: 
my **impression** is that the power parameter should be increased by 1, default k=2
`H = np.abs(V)**k/V`

If k=2, then this is just the sign matrix, AFAICS
Is this correct?

(Note, changing k doesn't bring promax closer to the Stata result for the rotated loadings matrix)

(I never read anything about promax, except checking the Stata manual for a brief description that they have.)
user: josef-pkt
issue_4171: state: open
title: ENH/Review procrustes in factor_rotation
body: Note: this is not ours, it is copied from the factor_rotation package

numerically: this solve the least squares problem, should use a better method linalg.solve or linalg.pinv

algorithm:

Stata has several algorithms, most of them iterative
scipy has https://docs.scipy.org/doc/scipy-0.19.1/reference/generated/scipy.spatial.procrustes.html
and https://docs.scipy.org/doc/scipy-0.19.1/reference/generated/scipy.linalg.orthogonal_procrustes.html

we need oblique

issue: there is most likely a problem with rotations "promax" that doesn't match Stata nor R. One likely possibility is that the procrustes rotation differs. (the rest of the promax code is either verified, e.g. using varimax as prerotation, or looks simple, power target matrix)
user: josef-pkt
issue_4169: state: open
title: ENH: factor analysis Heywood cases
body: I just ran the white wine example in Stata
https://gist.github.com/josef-pkt/1da753c160a054c9f342e735ee9be05e
which shows a Heywood case with one communality > 1

Stata reports, when using ipf
```
Beware: solution is a Heywood case
(i.e., invalid or boundary values of uniqueness)

Variable         |  Factor1   Factor2   Factor3 |   Uniqueness 
         density |   1.1050   -0.1253   -0.4318 |     -0.4231 

```

values are close enough, but not the same between Stata and statsmodels, at around 2nd decimal

Stata defaults to maxiter=1, which has high agreement between Stata and statsmodels, identical at print precision in summaries (4 decimals) for unrotated loadings.
At maxiter=1, there is no communality > 1

For MLE, Stata warns again about Heywood case, but sets uniqueness to zero

```
(tests formally not valid because a Heywood case was encountered)

Variable      |  Factor1   Factor2   Factor3 |   Uniqueness 
fixedacidity |   0.9994   -0.0353    0.0036 |      0.0000 
density       |   0.2992    0.9330   -0.2001 |      0.0000 
```
user: josef-pkt
issue_4168: state: open
title: DOC: location of ANOVA, AnovaRM
body: why is `anova_lm` a top level item in the TOC?

http://www.statsmodels.org/devel/anova.html

`MANOVA` is in `multivariate` (I'm adding it to the rst now)
But I'm not sure where to put `AnovaRM` which is in stats together with `anova_lm`
user: josef-pkt
issue_4165: state: open
title: ENH: polychoric correlation for ordered variables
body: The assumption is that a ordered variable is generated by a underlying latent variable that multivariate normal distributed. The ordered variable is a interval censored observed variable with unknown interval boundaries.
Given that the data is only ordered loc and scale of underlying latent variable is not determined, and is usually taken to be standard multivariate normal (loc=0, diag(cov) = eye)

standard method (relatively fast)

- get interval censoring threshold from marginal frequencies of the data.
- for all pairs of observations solve for the correlation coefficient given the thresholds, correlation coefficient is the only unknown.
- possibly check for positive (semi-) definitness

alternative is solving for thresholds and correlation coefficients simultaneously
problems:
- if only pairs are considered, then pairs will have different censoring thresholds
- if the full model is considered, then it becomes a very large estimation problem

main point in the computation is step 2 above:
- Note, first order condition for estimation does not include a normal cdf, only pdf
- it should be possible to compute the bivariate normal integrals with scipy (ignoring older papers that include computing normal cdf in the algorithm)
- starting values and optimization/root-finding method. Given that there is only a scalar parameter to search for, it should be possible to make this reasonabley efficient (even if slower than specialized Fortran code).

bonus: 
asymptotic standard errors or cov_params for the estimate of the polychoric correlation coefficient.


References

(based on my Zotero, I don't remember what I looked at)

Jöreskog 1994 is the only important one that I remember

Fox, John. 2010. Polycor: Polychoric and Polyserial Correlations (version 0.7-8). http://cran.r-project.org/web/packages/polycor/index.html.

Jin, Shaobo, and Fan Yang-Wallentin. 2017. “Asymptotic Robustness Study of the Polychoric Correlation Estimation.” Psychometrika 82 (1):67–85. https://doi.org/10.1007/s11336-016-9512-2.

Jöreskog, Karl G. 1990. “New Developments in LISREL: Analysis of Ordinal Variables Using Polychoric Correlations and Weighted Least Squares.” Quality and Quantity 24 (4):387–404. https://doi.org/10.1007/BF00152012.

———. 1994. “On the Estimation of Polychoric Correlations and Their Asymptotic Covariance Matrix.” Psychometrika 59 (3):381–89. https://doi.org/10.1007/BF02296131.

Maydeu-Olivares, Alberto, Carlos García-Forero, David Gallardo-Pujol, and Jordi Renom. 2009. “Testing Categorized Bivariate Normality with Two-Stage Polychoric Correlation Estimates.” Methodology: European Journal of Research Methods for the Behavioral and Social Sciences 5 (4):131–36. https://doi.org/10.1027/1614-2241.5.4.131.




user: josef-pkt
issue_4164: state: open
title: ENH: factor analysis follow-up
body: #3294 (merged in #4161) has several comments discussion design ideas. Most likely some refactoring will be necessary with the addition of new features.

PR #4156 has MLE for factor analysis

Stata has several additional results and post estimation results that we are still missing. Some results only apply if FA has been estimated by MLE

a few ENH possibilities

- factor scoring: compute factor arrays, not unique and several methods are possible
- more methods to choose the number of factors
- implementation: use initially a large number of factors and compute some results for each number of factors, as far as those are independent of initial choice of factors. e.g. Stata allows to rotate fewer factors than computed in the raw loadings matrix.
- hypothesis tests, fit statistics: likelihood based statistics require MLE.
- display help for interpreting rotated loadings: cutoff to skip printing small loadings coefficients, sort
- ...


unclear to me: cov versus corr

The basic results are scale invariant or equivariant, but some extra results might not be, and for other results we want to go back to the original scaling, i.e. we need standard deviation or cov.
example: I was getting confused about computing the underlying factors because using correlation or standardized exog changes the scale to normalized/beta coefficients.
(There is a considerable literature on using corr instead of cov in FA and covariance structure models. Not all methods are invariant or equivariant, but I didn't read any details.)


user: josef-pkt
issue_4159: state: open
title: ENH/SUMM: Factor Models - inference for factor analysis
body: 
related PRs

#3294 adds factor analysis
#4156 adds MLE for factor analysis/model based on Bai/Li
#4143 add asymptotic covariance matrices for correlation and covariance estimates which should be extended to structured models for covariance or correlation

quote Schmitt 2011
"""
Lastly, another underutilized component of EFA is statistical hypothesis testing of the factor
pattern loadings (see Cudeck & O’Dell, 1994). Even in reviews of appropriate methods for conducting
EFA, hypothesis testing of factor pattern loadings are rarely mentioned (e.g., Fabrigar
et al., 1999; Henson & Roberts, 2006). Historically, hypothesis testing of factor loadings within
EFA has been given little consideration due to computational complexity of the estimated standard
errors (Jennrich, 2007). Fortunately, the standard errors have recently become more readily
available for EFA in programs such as Mplus.
"""

Bai and Li 2012 have asymptotics under various identification restrictions, but asymptotics are double-infinite, given finite number of factors, the number of observation and the number of observed variables go to infinite. (Usual pattern in this literature, IIUC, factors are identified as number of variables goes to inf, but number of parameter in set of loading matrices increases with number of variables)

There is some old literature in psychometric that has standard errors for loading matrices for different rotation, e.g. several articles by Jennrich


Schmitt, Thomas A. 2011. “Current Methodological Considerations in Exploratory and Confirmatory Factor Analysis.” Journal of Psychoeducational Assessment 29 (4):304–21. https://doi.org/10.1177/0734282911406653.

Bai, Jushan, and Kunpeng Li. 2012. “Statistical Analysis of Factor Models of High Dimension.” The Annals of Statistics 40 (1):436–65. https://doi.org/10.1214/11-AOS966.

Jennrich, Robert I. 1974. “Simplified Formulae for Standard Errors in Maximum-Likelihood Factor Analysis.” British Journal of Mathematical and Statistical Psychology 27 (1):122–31. https://doi.org/10.1111/j.2044-8317.1974.tb00533.x.


I didn't read anything and skimmed just a little bit. Main puzzle was how the econometrics literature Bai/Li, SVAR, FAVAR matches up with the statistics literature covariance models, SEM and FA.

My impression from Jennrich: FA models are initially identified by the estimation algorithm, e.g. some "standard" MLE estimation method, and then rotated. Standard errors can be obtained by delta rule with the rotation Jacobian.
In econometrics imposing structural identifying restrictions on the estimator itself and then obtain inference directly seems to be more common. 


the search found also the next three: 
Not read and I don't know how closely related the articles are, but authors are worth a read.
Fuller article and as author also indicates that there is a connection to measurement error models (where I like his book.

Amemiya, Yasuo, and Wayne A. Fuller. 1984. “Estimation for the Multivariate Errors-in-Variables Model with Estimated Error Covariance Matrix.” The Annals of Statistics 12 (2):497–509.

Amemiya, Yasuo, Wayne A Fuller, and Sastry G Pantula. 1987. “The Asymptotic Distributions of Some Estimators for a Factor Analysis Model.” Journal of Multivariate Analysis 22 (1):51–64. https://doi.org/10.1016/0047-259X(87)90074-1.

Anderson, T. W. 1984. “Estimating Linear Statistical Relationships.” The Annals of Statistics 12 (1):1–45. https://doi.org/10.1214/aos/1176346390.

one book that sounds interesting as overview reading, especially for making the link to nonnormal, noncontinuous data, e.g. includes categorical variables and exponential family, binomial, Poisson:

Bartholomew, David J., M. Knott, and Irini Moustaki. 2011. Latent Variable Models and Factor Analysis: A Unified Approach. 3rd ed. Wiley Series in Probability and Statistics. Chichester, West Sussex: Wiley.









user: josef-pkt
issue_4158: state: open
title: Out of memory on get_margeff() 
body: I have a dataset of about 400 thousand observations. It is not too large, at around 100 MB of space. Version is 0.8.0. Unfortunately, as data is confidential, I cannot include it, but I include the code below.

I estimate the model with:

```
import statsmodels.formula.api as st
import pandas as pd

d = pd.read_feather('mydata.feather')
m = st.mnlogit(
    formula = 'y ~ sex + vismin + special_needs + esl + t_score1 + t_score2 + t_score3', 
    data = d
).fit()
```

So far everything works as expected. The mlogit estimates look fine. However, the following:

```
m.get_margeff()
```

Never completes, and runs out of memory (it takes 30 GB on a 32 GB machine before it fails). There is an object in there that appears to just keep growing.

Getting marginal effects at the means works fine:

```
m.get_margeff(at = 'mean')
```

Any hints would be very helpful as to where the trouble might be. For comparison, Stata does this task with about 1 1/2 GB memory footprint throughout.
user: dpavlic
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_4157: state: open
title: ENH: large sample multinomial, frequencies with excess noise/dispersion
body: "A large sample can damage your p-value"

If we have a gof test like chisquare or contingency table test with very large numbers, then the large sample asymptotics makes hypothesis test or similar inference very strict.
If just rejecting the gof null hypothesis is not useful, then something else needs to be used. Essentially, there is too much noise for the underlying multinomial model to be correctly specified  and we need some extra noise.

current case https://stackoverflow.com/questions/47699132/stats-chisquare-wrongly-returns-p-value-0/47708736
but I ran into the same before comparing one sample against an estimated distribution.

possibilities

- model the noise: eg. #2708 with Poisson noise
- allow for excess dispersion similar to GLM/discrete with cov_type='HCx', We don't have anything yet for standalone one, two, k-sample tests. #2890 and cross-links there
- equivalence testing: I don't find an issue, I thought I wrote an equivalence hypothesis version of chisquare gof test, but I don't see it either. The main practical problem is how to define the equivalence interval.
  Aside: covariance structure and SEM literature has some articles for equivalence testing in terms of model is "close enough" to the data. 

We don't have anything yet that provides an answer in this case, but I would like to have or know of something that "works".


user: josef-pkt
issue_4153: state: open
title: ENH: Item Response Theory/Models - categorical PCA/FA
body: (mainly to get started with an issue and to park two links)

https://stats.stackexchange.com/questions/252317/similarities-and-differences-between-irt-model-and-logistic-regression-model
https://stats.stackexchange.com/questions/215404/is-there-factor-analysis-or-pca-for-ordinal-or-binary-data

I saw it mentioned several times in the PCA and factor analysis literature as alternative when variables are categorical or mixed instead of continuous.

Browsing a bit for the underlying statistical methods (given what little I know):

- basic structure is Logit or Probit, either univariate for binary/binomial outcomes or multivariate/multinomial, multinomial version is usually for ordered categories
- the multiparameter version is nonlinear, not linear plus link as in GLM or Logit
- the unobserved factors, item and individual random effects, require mixed GLM or mixed Logit/Probit/Multinomial
- there might also be a connection to fractional models, i.e. logistic regression or similar for "continuous"  data (many discrete points in bounded interval)

So, we need to add some extensions and new models, but IRT might also provide test cases for the generic, not application specific models.

There are some articles and books that make directly the link to mixed effects GLM. Most of the general IRT literature is more on the application to multiple-choice testing.

user: josef-pkt
issue_4151: state: open
title: ENH  QIC, GEE correlation selection criteria
body: It looks like we never opened an issue for QIC, an information criterion for GEE
https://groups.google.com/d/msg/pystatsmodels/cRjFSWdj68I/Yhi40PT6AgAJ

#3288 the only a bit related issue that I find

one of the articles that I had downloaded, and just saw again:

Westgate, Philip M. 2014. “Improving the Correlation Structure Selection Approach for Generalized Estimating Equations and Balanced Longitudinal Data.” Statistics in Medicine 33 (13):2222–37. https://doi.org/10.1002/sim.6106.

releated to correlation structure selection and testing in general #2765 and recent issues on covariance and correlation strutures for multivariate stats.

GEE covstruct is a more difficult case because it allows for unbalanced panels, i.e. cluster/group sizes are allowed to vary, which does not allow the use of the matrix versions that are used with balanced panels.
user: josef-pkt
issue_4148: state: open
title: ENH: covariance/correlation tests in high dimension
body: related to 
#2765 #4143  traditional/old statistics for covariance and correlation structures and tests
#3197 penalized and shrinkage estimator for covariances or inverse covariances

These are just some random references that I didn't read, there seems to be a large recent and current literature for it. 

(There is also a multivariate textbook by Japanese authors where I don't have the reference right now. Each chapter has sections on the high dimensional case. I didn't understand the book, a bit theory/math heavy when just skimming some sections.)

Cai, T. Tony, and Anru Zhang. 2016. “Inference for High-Dimensional Differential Correlation Matrices.” Journal of Multivariate Analysis 143 (Supplement C):107–26. https://doi.org/10.1016/j.jmva.2015.08.019.

Himeno, Tetsuto, and Takayuki Yamada. 2014. “Estimations for Some Functions of Covariance Matrix in High Dimension under Non-Normality and Its Applications.” Journal of Multivariate Analysis 130 (Supplement C):27–44. https://doi.org/10.1016/j.jmva.2014.04.020.

Hyodo, Masashi, Nobumichi Shutoh, Takahiro Nishiyama, and Tatjana Pavlenko. 2015. “Testing Block-Diagonal Covariance Structure for High-Dimensional Data.” Statistica Neerlandica 69 (4):460–82. https://doi.org/10.1111/stan.12068.

Li, Weiming, and Yingli Qin. 2014. “Hypothesis Testing for High-Dimensional Covariance Matrices.” Journal of Multivariate Analysis 128 (Supplement C):108–19. https://doi.org/10.1016/j.jmva.2014.03.013.

Mao, Guangyu. 2016. “A Note on Tests for High-Dimensional Covariance Matrices.” Statistics & Probability Letters 117 (Supplement C):89–92. https://doi.org/10.1016/j.spl.2016.05.002.

Zhang, Jie, and Meng Pan. 2016. “A High-Dimension Two-Sample Test for the Mean Using Cluster Subspaces.” Computational Statistics & Data Analysis 97 (Supplement C):87–97. https://doi.org/10.1016/j.csda.2015.12.004.

user: josef-pkt
issue_4146: state: open
title: Error When Passing a value for the 'marker' as a kwarg to plot_acf and plot_pacf
body: According to the statsmodels documentation, it should be possible to change the marker type for the smt.graphics.plot_acf and smt.graphics.plot_pacf functions by specifying a 'marker' kwarg. 

If you look at the source code however, if the 'use_vlines' argument is set to True then any kwargs are passed on to the vlines plotting function. This is problematic because the 'marker' keyword is not used in the vlines function and throws an error. 
user: mflara
issue_4145: state: open
title: Fitting an AR model on non-continuous multi-trial data
body: I have a univariate 2D dataset made up of many non-consecutive trials (or slices/windows) of a time series (one trial is not continuous with another trial). I would like to fit an AR model to this collection of trials for forecasting on an exogenous trial.

My first approach is to concatenate all of the trials into a long 1-dimensional array, padding with NaNs to separate the trials, and then use `AR(missing='drop')` to ignore the NaNs. However, I'm not sure how the `missing` parameter works - does "drop" simply remove the NaNs and concatenate the array on both sides? This is not what I would want, since the AR model would then be trained on the data as if it was continuous, which it is not. The points between the trials shouldn't have any correlation, and the model should not use data from one trial to predict the time series of a new trial.

Is there a better way to train a univariate AR model on a collection of discontinuous windows of data? Thanks!
user: gabrielibagon
issue_4144: state: open
title: SUMM/Reference robust cov_type for covariance analysis
body: The following article makes the analogue between linear mean models and covariance structure models and the assumption about correctly specified weight/residual covariance, in this case mainly about 4th moments. (I didn't read it yet but looks good based on superficial skimming.)
It also targets intuition for more general audience, AFAICS.

Savalei, Victoria. 2014. “Understanding Robust Corrections in Structural Equation Modeling.” Structural Equation Modeling: A Multidisciplinary Journal 21 (1):149–60. https://doi.org/10.1080/10705511.2013.824793.


free association:

- this is similar to older articles by Wooldridge for robust 4th moment assumptions in second moment specification tests
- GLM/GEE analogue: The covariance structure often has a linear structure but the variance/covariance depends on the expected values which in this case are second moments. This is similar to using GLM with gamma distribution (var is squared mean) for variance functions. In GEE, we could specify the working correlation based on the asymptotic covariance assumptions for the covariance model.
- The underlying model is often GLS, but with a multivariate or block endog. The debate is on the assumption on the weight matrix (in GLS/GMM sense) and whether it is assumed that we need a sandwich or not (correctly versus incorrectly specified weight matrix/ GEE cov_struct).

aside (again): 
The small sample performance of unrestricted 4th moment based inference is pretty awful. There is a large literature on improving this with correction terms or using F instead of chisquare distribution. Something like that will be necessary to make this useful, that is, an implementation purely based on asymptotics is a lot less useful than the corresponding models for the mean function.

aside: The standard term or acronym in the literature for SEM or covariance structures for unrestricted 4th moment based inference is ADF, asymptotically distribution free inference.
Bentler with co-authors and many follow-up articles.

**edit**

This is one of the original articles with the theory for covariance structure models
Browne, M. W. 1984. “Asymptotically Distribution-Free Methods for the Analysis of Covariance Structures.” British Journal of Mathematical and Statistical Psychology 37 (1):62–83. https://doi.org/10.1111/j.2044-8317.1984.tb00789.x.
and there are similar for correlation structure models

one possible plan:
Add `covariance_models` or `correlation_models` module to multivariate for full models as a more general class than factor models. The basic version would be similar to a multivariate endog GLS model, where endog are vech(y y') i.e. each row is vech of cross product of covariance (assuming zero mean of removed expected mean (1st moment of data). In many cases, not only normal distribution, we still have asymptotic orthogonality of 1st and 2nd moment estimators. (i.e. similar to GARCH on residuals)
(This would provide a setup to collect and organize all things, estimator weight matrices and LR and Wald inference, with a specific purpose. Unorganized pieces are in #4143)

user: josef-pkt
issue_4143: state: open
title: WIP ENH: cov and matrix tools to work with covariances and correlations
body: a semi-random collection of matrix function and estimators for the covariance of sample correlation and covariance matrices.

very unfinished. I didn't even try out all functions. API needs a lot of work. This is partially an "academic" exercise understanding how to work with and compute the asymptotic covariance of covariance and correlation matrices. 

In contrast to #2765, I didn't find much in multivariate textbooks and not in Stata. There is a psych package in R that has many correlation functions or hypothesis tests. This PR is written based on a collection of articles going back to 1970. (references will follow)

I'm not sure yet about how and where this fits in. currently everything is in private module cov_tools. Overall this goes in the direction of modeling second moments, so far statsmodels is mostly focused on first moments.

* include matrix tools #4140 (names not informative nor pep-8 compliant, written based on the first article that I looked at when starting to write code. Neudecker changes the capital letter for various matrices across articles)
* cov_cov and cov_corr: asymptotic covariance of a covariance and correlation estimator targeting various patters/structures or linear restrictions on those. Main assumptions are one of normal, elliptical and distribution free, which in this case affects the assumptions on the 4th moments. 
* eventually the target is structured covariance and correlation estimators with either MLE, iterated GLS and GMM. They are all similar but but use different weighting matrices and the assumptions that underly those.
* hypothesis tests on correlation coefficients: Once we have the cov_corr, the rest is relatively standard.
* hypothesis tests on patterns of cov and corr: Most hypothesis tests in #2765 is base on likelihood ratio tests. The literature underlying this PR goes more in the direction of score/LM and Wald tests.

related and not in here: GMM/unrestricted 4th moments can have bad small sample properties. There are some articles that use F instead of the usual chisquare distribution to improve this.

related, but nothing available yet: kurtosis measures to have better (approximation to) asymptotic distribution of test statistic in elliptically symmetric case, e.g. #3280 (several, many articles for multivariate methods). (I don't know if we need multivariate Mardia kurtosis or some average univariate measure)

list of references (incomplete, but most of what I looked at)

Magnus, Jan R., and H. Neudecker. 1986. “Symmetry, 0-1 Matrices and Jacobians: A Review.” Econometric Theory 2 (2):157–90.
Neudecker, H., and A. M. Wesselman. 1990. “The Asymptotic Variance Matrix of the Sample Correlation Matrix.” Linear Algebra and Its Applications 127 (January):589–99. https://doi.org/10.1016/0024-3795(90)90363-H.
Neudecker, Heinz. 1996. “The Asymptotic Variance Matrices of the Sample Correlation Matrix in Elliptical and Normal Situations and Their Proportionality.” Linear Algebra and Its Applications, Linear Algebra and Statistics: In Celebration of C. R. Rao’s 75th Birthday (September 10, 1995), 237 (April):127–32. https://doi.org/10.1016/0024-3795(95)00351-7.
Satorra, Albert, and Heinz Neudecker. 1997. “Compact Matrix Expressions for Generalized Wald Tests of Equality of Moment Vectors.” Journal of Multivariate Analysis 63 (2):259–76. https://doi.org/10.1006/jmva.1997.1696.
Steiger, James H., and A. Ralph Hakstian. 1982. “The Asymptotic Distribution of Elements of a Correlation Matrix: Theory and Application.” British Journal of Mathematical and Statistical Psychology 35 (2):208–15. https://doi.org/10.1111/j.2044-8317.1982.tb00653.x.

Browne, M. W. 1974. “Generalized Least Squares Estimators in the Analysis of Covariance Structures.” South African Statistical Journal 8 (1):1–24.
———. 1977. “The Analysis of Patterned Correlation Matrices by Generalized Least Squares.” British Journal of Mathematical and Statistical Psychology 30 (1):113–24. https://doi.org/10.1111/j.2044-8317.1977.tb00730.x.
———. 1984. “Asymptotically Distribution-Free Methods for the Analysis of Covariance Structures.” British Journal of Mathematical and Statistical Psychology 37 (1):62–83. https://doi.org/10.1111/j.2044-8317.1984.tb00789.x.
Huang, Yafei, and Peter M. Bentler. 2015. “Behavior of Asymptotically Distribution Free Test Statistics in Covariance Versus Correlation Structure Analysis.” Structural Equation Modeling: A Multidisciplinary Journal 22 (4):489–503. https://doi.org/10.1080/10705511.2014.954078.
Steiger, James H. 1980a. “Tests for Comparing Elements of a Correlation Matrix.” Psychological Bulletin 87 (2):245–51. https://doi.org/10.1037/0033-2909.87.2.245.
———. 1980b. “Testing Pattern Hypotheses On Correlation Matrices: Alternative Statistics And Some Empirical Results.” Multivariate Behavioral Research 15 (3):335–52. https://doi.org/10.1207/s15327906mbr1503_7.

Yuan, K.-H., and P. M. Bentler. 1999. “F Tests for Mean and Covariance Structure Analysis.” Journal of Educational and Behavioral Statistics 24 (3):225–43. https://doi.org/10.3102/10769986024003225.

Hayakawa, Kazuhiko. 2016. “On the Effect of Weighting Matrix in GMM Specification Test.” Journal of Statistical Planning and Inference 178 (November):84–98. https://doi.org/10.1016/j.jspi.2016.06.003.
Prokhorov, Artem. 2009. “On Relative Efficiency of Quasi-MLE and GMM Estimators of Covariance Structure Models.” Economics Letters 102 (1):4–6. https://doi.org/10.1016/j.econlet.2008.08.019.


**edit**
I found LISREL book that has a summary with collection of formulas in the appendix
16.1.5 General Covariance Structures (in appendix F)
Jöreskog, Karl G., Ulf H. Olsson, and Fan Y. Wallentin. 2016. Multivariate Analysis with LISREL. Springer Series in Statistics. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-33153-9.
includes also multivariate Mardia skew and kurtosis, chapter 12 appendix B
user: josef-pkt
labels: superseded: Used for PRs with content that is superseded by a newer PR
issue_4142: state: open
title: ENH: ztest for correlated means or moments
body: We have standalone ztest and ttest for independent and paired samples, but we don't have standalone function for correlated moments. This is similar to t_test or wald_test in models where we have the full covariance matrix to test correlated parameters.

example: testing or comparing 2 correlations with one variable in common, then the two correlation coefficients are correlated.

simple function based on summary statistic that can be extended to one-sided alternatives and to full inference mixin. (Written for a test case on testing correlations.)

```
def ztest_correlated(diff, var0, var1, covar, nobs):
    std = np.sqrt((var0 + var1 - 2 * covar) / nobs)
    statistic = diff / std
    from scipy import stats
    pvalue = stats.norm.sf(np.abs(statistic)) * 2
    return statistic, pvalue
```

user: josef-pkt
issue_4140: state: open
title: ENH: matrix tools for covariances
body: Another set of matrix tools, those based on vec, vech and similar operators. Magnus and Neudecker or Neudecker and other co-authors.
I have a preliminary version used for computing estimates of the covariance of the sample covariance and correlation. The current version is mainly "academic", i.e. not useful for large covariance matrices since the dimensions go up to `k_vars**2 x k_vars**2`, and assume balanced data, i.e. all multivariate observations have same k_vars. That is for example not assumed in GEE cov_struct, where cov are within a panel unit or a group that can have different number of observation per group.

We also need to organize other matrix/linalg tools, and clean them up to what's helpful/used and what are just draft versions.
e.g. 
stats.multivariate_tools has one version of project and canonical correlation
tools.moment_helpers has cov2corr
matrix classes are in ...?  sandbox\archive\linalg_decomp_1.py (not used)
stats.correlation_tools has only the nearest cov or corr functions
tsa.vector_ar.utils has some like vech
more ?




user: josef-pkt
issue_4139: state: open
title: ENH: add callback weight and cov function options to GMM
body: A general idea how to users or special case models affect how the weight and covariance matrix in GMM is computed. This might also extend to cov_type.

This should be an easier option to try out or use different weight and covariance matrices when the standard GMM approach (similar to OPG, unrestricted covariance of moment conditions) is not very accurate in small samples, e.g. many moment conditions or higher order moments (when estimating variance and covariance)
This will also make it easier to replicate parametric models like MLE or QMLE or to add regularization to the weight matrix.

(specific case: Estimating covariance parameters with GMM is not very accurate in small samples because the optimial weight matrix uses forth moments that are difficult to estimate accurately in small samples.)

another benefit:
I think we can then reuse GMM also for the case when moment conditions are given in the aggregate/average and not for individual observations. Example quantile in sandbox which still doesn't fit properly the pattern, and covariance/correlation patterns based on sample covariance matrix.
user: josef-pkt
issue_4138: state: open
title: coint() 
body: Coint is giving me p values > 1% but scores < 1% crit value
user: DanielAristidou
issue_4137: state: open
title: SUMM/FAQ: empty cells, unobserved effects, categorical interactions
body: (random thoughts triggered by stackoverflow question. I didn't find a directly related issue, but some cases for empty cells in exog categorical interactions are in some issues or PRs)

## empty cells in categorical interactions in exog

patsy keeps all (non-reference) interaction effects even if they are a column of zeros. 
https://stackoverflow.com/questions/47444502/how-to-stop-patsy-from-creating-redundant-interactions-of-categorical-variables answer reparameterizes the pair of categorical variables

- It doesn't really matter in the case of OLS with pinv because the unobserved effect gets regularized to zero. possible problem is when regularization does not kick in see #2628. 
- However, that does not work for estimators that require non-singular exog, e.g. in discrete models and to some extend in GLM.
- explicit handling as part of multicollinearity treatment, i.e. automatically drop columns ???


## interactions with discrete or limited support endog

related empty cells in contingency tables

There was recently a stackoverflow question about non-convergence and non-invertible hessian in Logit when exog had full rank. Question was deleted and data was private, so no reproducible example is available. I guess that case is related to partial separation (empty cells in combined endog-exog contingency table) for which we don't have any checks or examples, and the explanations are vague to me. Our check for perfect separation is for perfect prediction, but SAS/SPSS have checks for partial separation. Similar effects should also occur with multinomial and poisson models.

Solution in contingency tables is to add a positive value instead of the zeros, and similar Firth penalization can handle the case in Logit.

## continuous variables

A bit related to this is extrapolation with continuous variables. This was for example discussed in treatment effect (???) literature with restrictions to predict within the support points. The main problem is that prediction and forecasting outside the support "envelope" completely relies on the functional form assumption for extrapolation. Related issue is in extrapolation with splines which patsy does not allow.
Similar issue is in propensity score matching or inverse probability weighting if the sample is unbalanced in the sense that we have very few observation of the "minority" treatment level for some set or range of exog.

In the continuous case we can have some standard measures like influence measure or Mahalanobis distance to measure the outlyingness of specific exog values.


I'm not sure what we can do within statsmodels. We need diagnostics to identify problematic cases and  more penalization or variable selection to work around in some cases.

user: josef-pkt
issue_4136: state: open
title: ENH: more flexibility in wald_test_terms
body: https://stackoverflow.com/questions/47395061/how-to-calculate-effect-tests-in-linear-regression-in-python

How do we get the main effect as separate test?
Is in in a default?  string matching only works for all effects.

example?
user: josef-pkt
issue_4130: state: open
title: REF/Design explicit internal nan policy
body: #3481 added some explicit nan propagation to some functions to remove noise from irrelevant warnings.

Models have options for missing value handling. All other functions ignore nans, i.e. don't check and let them propagate like all floats.

Silent nan propagation is an intentional behavior to get results, e.g. params_table, even if some results are nan, e.g. when bse is not available. However, it causes noise in getting warnings when the nan results is as expected, but it might also hide problems in the code or on the user side. 
If it's on the user side, then explicit warnings or a `nan_policy` would be better than getting warnings somewhere in the computations or no indication for problems at all.

I don't have much of an overview of where we have and allow for nans:

- linear algebra: not finite are not allowed and we currently don't have enough optional/default checks
- discrete L1 explicitly introduced nans and nan dot product for parameters regularized to zero. 
- similar to discrete L1 we have cases where we don't have cov_params/bse for all parameters. (AFAIK the behavior in those models is model specific)
- nans in edge cases and propagation of nans if there are problems. Those need to be allowed and are better than exceptions when we still need and want incomplete results. For example, summary() should always be available and invalid or missing results should be nan. (explicitly done, e.g. in some diagnostics and fvalue)
- standalone functions should eventually get a proper missing/nan handling if they are user facing and if we need options for internal usage. e.g. helper functions and hypothesis tests in stats.

user: josef-pkt
issue_4129: state: open
title: shape mismatch issue when using Mice imputer
body: get the following error
shape mismatch: value array of shape (53,1) could not be broadcast to indexing result of shape (53,).
I'm using numpy 1.13.1 version and statsmodels 0.8.0

My code is:
import statsmodels.api as sm
from statsmodels.imputation import mice
imp = mice.MICEData(df_test)
fml = 'col1~col2+col3' #colnames in dataframe
model = mice.MICE(fml, sm.OLS, imp)
model.fit()


Full log:


---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-132-ffe455c40c1a> in <module>()
----> 1 mice.fit(10, 10)

~/development/datascience/datascience/lib/python3.5/site-packages/statsmodels/imputation/mice.py in fit(self, n_burnin, n_imputations)
   1226 
   1227         # Run without fitting the analysis model
-> 1228         self.data.update_all(n_burnin)
   1229 
   1230         for j in range(n_imputations):

~/development/datascience/datascience/lib/python3.5/site-packages/statsmodels/imputation/mice.py in update_all(self, n_iter)
    413         for k in range(n_iter):
    414             for vname in self._cycle_order:
--> 415                 self.update(vname)
    416 
    417         if self.history_callback is not None:

~/development/datascience/datascience/lib/python3.5/site-packages/statsmodels/imputation/mice.py in update(self, vname)
   1003 
   1004         self.perturb_params(vname)
-> 1005         self.impute(vname)
   1006 
   1007 

~/development/datascience/datascience/lib/python3.5/site-packages/statsmodels/imputation/mice.py in impute(self, vname)
    986         # Wrap this in case we later add additional imputation
    987         # methods.
--> 988         self.impute_pmm(vname)
    989 
    990 

~/development/datascience/datascience/lib/python3.5/site-packages/statsmodels/imputation/mice.py in impute_pmm(self, vname)
   1078 
   1079         imputed_miss = np.array(endog_obs[iz])
-> 1080         self._store_changes(vname, imputed_miss)
   1081 
   1082 

~/development/datascience/datascience/lib/python3.5/site-packages/statsmodels/imputation/mice.py in _store_changes(self, col, vals)
    393         ix = self.ix_miss[col]
    394         if len(ix) > 0:
--> 395             self.data[col].iloc[ix] = vals
    396 
    397 

~/development/datascience/datascience/lib/python3.5/site-packages/pandas/core/indexing.py in __setitem__(self, key, value)
    139             key = com._apply_if_callable(key, self.obj)
    140         indexer = self._get_setitem_indexer(key)
--> 141         self._setitem_with_indexer(indexer, value)
    142 
    143     def _has_valid_type(self, k, axis):

~/development/datascience/datascience/lib/python3.5/site-packages/pandas/core/indexing.py in _setitem_with_indexer(self, indexer, value)
    577             self.obj._consolidate_inplace()
    578             self.obj._data = self.obj._data.setitem(indexer=indexer,
--> 579                                                     value=value)
    580             self.obj._maybe_update_cacher(clear=True)
    581 

~/development/datascience/datascience/lib/python3.5/site-packages/pandas/core/internals.py in setitem(self, **kwargs)
   3166 
   3167     def setitem(self, **kwargs):
-> 3168         return self.apply('setitem', **kwargs)
   3169 
   3170     def putmask(self, **kwargs):

~/development/datascience/datascience/lib/python3.5/site-packages/pandas/core/internals.py in apply(self, f, axes, filter, do_integrity_check, consolidate, **kwargs)
   3054 
   3055             kwargs['mgr'] = self
-> 3056             applied = getattr(b, f)(**kwargs)
   3057             result_blocks = _extend_blocks(applied, result_blocks)
   3058 

~/development/datascience/datascience/lib/python3.5/site-packages/pandas/core/internals.py in setitem(self, indexer, value, mgr)
    738             # set
    739             else:
--> 740                 values[indexer] = value
    741 
    742             # coerce and try to infer the dtypes of the result

ValueError: shape mismatch: value array of shape (53,1) could not be broadcast to indexing result of shape (53,)

user: matanazulay
issue_4128: state: open
title: ENH: raise more informative ValueErrors, e.g. input dimension
body: In many cases we don't use basic checks on the properties of the provided arguments, which is not informative to users if the exception is raised somewhere in the code

e.g. adfuller requires 1-D (and we cannot vectorize it)
https://stackoverflow.com/questions/47376300/adfuller-test-received-valueerror-too-many-values-to-unpack

e.g. inter_rater  uses assert instead of ValueError
https://stackoverflow.com/questions/47370320/assertionerror-when-calculating-kappa

user: josef-pkt
issue_4127: state: open
title: WIP ENH:  SURCompact
body: SUR model where exog differ across equations. Mainly a computational exercise to try out this data structure for the representation of the SUR model.
(In terms of memory requirement there should not be much difference to a fully sparse version with sparse block diagonal exog and cov_resid.)

This is supposed to be an implementation with a compact exog, following in this case Green 7th edition, but AFAIR it is standard formulation in all text books. (maybe Davidson MacKinnon was my previous reference)

exog is column_stacked list of exog for all equations
cov_resid is expanded to stacke exog and multivariate endog
```
covi_ex = np.repeat(np.repeat(covi, ks, axis=1), ks, axis=0)
covi_y = np.repeat(covi, ks, axis=0)
```
where I stopped:
- the simple/general version is implemented, but not unit tests against Stata yet
- currently the code also seems to correctly handle cov_hc0 for a diagonal cov_resid, equivalent to OLS and cov_type = 'cluster'. score_obs is only correct for diagonal cov_resid
- the extension is to get a cholesky weighted score_obs. That has initial code but not tried out yet. The idea is to get the column_stacked version of score_obs so we can use any cov_type with it. see #4121.
  I guess we could then also use pinv on obs, instead of solving the moment conditions directly which I currently do.

also this has two representations of params, standard params is 1-D, second version params_block is sparse block diagonal for the multivariate representation. Y = X B + U, where Y and U are (nobs, k_endog) and X is column_stacked exogs with shape (nobs, sum(k_vars))

special feature: call back function for cov_resid. 
We don't hard code the way cov_resid is estimated, so we can use structured, regularized or robust cov_resid or inverse cov_resid similar to GEE, This makes it closer to GMM/GEE, i.e. structured working cov or GMM weights matrix and sandwich cov_params. e.g. #2942, i.e. GMM weights and cov_score in cov_params don't have to necessarily agree.

This does currently not allow common parameters across equations. sysreg GSOC PR #2500 allows constraints that are added to the moment equation.

user: josef-pkt
issue_4122: state: open
title: Formula splines with missing values
body: There seems to be a problem handling formulas with splines in which the variables used to form the spline basis functions have missing values:

The code below raises an exception deep in the data handling code.  Replacing `bs(x1, 3)` with `x1` works.

```
import statsmodels.api as sm
import pandas as pd
import numpy as np

df = pd.DataFrame({"x1": np.linspace(0, 10, 10)})
df["y"] = df.x1 + np.random.normal(size=10)
df["x1"].iloc[0] = np.nan

model = sm.OLS.from_formula("y ~ bs(x1, 3)", data=df, missing='drop')
result = model.fit()
```
user: kshedden
issue_4121: state: open
title: ENH: SUR and sandwich robust covariance - cov_type
body: unanswered question

http://forums.eviews.com/viewtopic.php?t=9556
https://www.stata.com/statalist/archive/2010-09/msg01077.html

marginally related
Bilodeau, Martin, and Pierre Duchesne. "Robust estimation of the SUR model." Canadian Journal of Statistics 28, no. 2 (2000): 277-288.
uses S-estimator, but can similarly apply to M-estimator, starts out with sandwich covariance but then assumes iid elliptical distribution, and is not HC or HAC (as RLM is not)

user: josef-pkt
issue_4119: state: open
title: MAINT: deprecate sandbox multitest
body: I didn't remember that multitest is out of sandbox. In any case the official path was already before through statsmodels.stats.

However, it looks like the sandbox functions where never deprecated.

https://stackoverflow.com/questions/47217357/attributeerror-module-statsmodels-sandbox-has-no-attribute-stats
user: josef-pkt
issue_4118: state: open
title: Statsmodels prints uninformative message
body: When running a major file of code I kept getting

> not in depth 1

and similar messages. I had no clue how to analyze this (in my larger system), and assumed it was an error message thrown by multiprocessing (something about workers that were inaccessible or thelike). 

After a text search in my whole site-packages, I found the culprit:

[In base/data.py, line 107]( https://github.com/statsmodels/statsmodels/blob/master/statsmodels/base/data.py#L107) , it just says


```
            for depth in [2, 3, 1, 0, 4]:  # sequence is a guess where to likely find it
                try:
                    _, design = dmatrices(d['formula'], data, eval_env=depth,
                                          return_type='dataframe')
                    break
                except (NameError, PatsyError) as e:
                    print('not in depth %d' % depth)
                    exc.append(e)   # why do I need a reference from outside except block
                    pass
```

This error handling is confusing at the least.


user: eXcuvator
issue_4115: state: open
title: ENH: agreement of affine measurements
body: just a random idea

https://stats.stackexchange.com/questions/167151/bland-altman-tukey-mean-difference-plot-for-differing-scales
#4112

suppose we have to measurements using different scales (e.g. one in Celsius and one in Fahrenheit). Both measurements are noisy but should agree under the null assumption.

one possibility would be to regress one measurement on the other, but that allocates all measurement error to one side (i.e. we are not doing anything like measurement error model or orthogonal regression)

symmetric alternative
use PCA to get one common factor and treat this as the "average" standardized measurement. I guess we can use the loading matrix to get the transformation to "standardized" measurements.
(One problem is that we need to standardize by standard deviation to have equal weights in PCA.)
Then we should be able to get something like Bland-Altman or Tukey mean-difference plot and have `agreement` measures for each observation.

BTW: I think know stats `inter_rater` should be renamed to `agreement`. Note, inter_rater is currently only for categorical or ordered discrete variables not for continuous measurements as here.
user: josef-pkt
issue_4102: state: open
title: Review old hypothesis test function
body: statsmodels/sandbox/stats/runs.html#median_test_ksample
uses print to issue warning for small sample size

Also some docstrings contain developer comment, that haven't been reviewed, rewritten or removed.
as follow-up to #4101 unclear docstring

diagnostic test also have mostly not been reviewed since they were added.

tests related to contingency tables got a makeover

and we finally need the `return_results` decision across all functions.

user: josef-pkt
issue_4101: state: open
title: Documentation for `sign_test` parameter `mu0` is questionable
body: For the function `statsmodels.stats.descriptivestats.sign_test(samp, mu0=0)`, the [documentation](http://www.statsmodels.org/devel/generated/statsmodels.stats.descriptivestats.sign_test.html#statsmodels.stats.descriptivestats.sign_test)  for the `mu0` parameter says:
> See Notes for the definition of the sign test. mu0 is 0 by default, but it is common to set it to the median.

Taking a look at the Notes:

> M = (N(+) - N(-))/2
> where N(+) is the number of values above mu0, N(-) is the number of values below

If you set `mu0` to the median, by definition, the number of values above and below `mu0` are equal, so `M = 0`, and the test always returns `M = 0` and `p = 1.0`.  

Setting `mu0` to the median can't be very useful.  I'm not sure what the intent of the `mu0` parameter was, and I don't see any reference to parameters in the [Wikipedia article](https://en.wikipedia.org/wiki/Sign_test) (after all, its primary virtue lies in being nonparametric!).  Perhaps the documentation could be clarified.
user: jkleint
issue_4091: state: open
title: Power documentation and Example
body: Improvements to power documentation. Comments and suggestions welcome!

Working on suggestions noted in https://github.com/statsmodels/statsmodels/issues/4085

Future Work:
* Examples for `plot_power()` function
* Potentially remove references to user-facing `power()` function and point everyone to the `solve_power()` versions
* Figure out how to document the relationship between the classes and the helper functions
user: mikekaminsky
issue_4090: state: open
title: sw.cov_crosssection_0 docstring says func is wrong
body: There's a cleverly hidden secret message in this [docstring](https://github.com/statsmodels/statsmodels/blob/master/statsmodels/stats/sandwich_covariance.py#L513):
```
def cov_crosssection_0(results, group):
    '''this one is still wrong, use cov_cluster instead'''
```

translated it means "this function should be deleted."

While we're in the neighborhood, `stats.sandwich_covariance` functions never called in tests:
`_HCCM`
`cov_hc0`
`cov_hc1`
`cov_hc2`
`cov_hc3`
`_HCCM1`


Cases never reached in tests:
`_get_sandwich_arrays` case where `isinstance(results, tuple)`
`_get_sandwich_arrays` case where `hasattr(results.model, 'jac')`.
`cov_nw_groupsum` case where `elif use_correction in ['c', 'cluster']:`

The "jac" case I think is for a long-since deprecated implementation of jacobian.
user: jbrockmendel
issue_4085: state: open
title: DOC: Confusion in power functions and documentation
body: Opening this as an issue to solicit opinions from the maintainers before I go in and do a bunch of documentation writing

### Pain Point
Recently, I experienced a lot of confusion trying to figure out which functions to use to run a power analysis. After digging under the hood for a while, I now understand the difference stems from the difference between the `TTestIndPower` class and the `TTestPower`  class in the [power module](https://github.com/statsmodels/statsmodels/blob/master/statsmodels/stats/power.py).

The first two results from [the obvious google search](https://www.google.com/search?q=statsmodels+power+analysis&oq=statsmodels+power+analy&aqs=chrome.0.69i59j69i57.4151j0j7&sourceid=chrome&ie=UTF-8)
* [tt_ind_solve_power](http://www.statsmodels.org/dev/generated/statsmodels.stats.power.tt_ind_solve_power.html)
* [TTestIndPower](http://www.statsmodels.org/dev/generated/statsmodels.stats.power.TTestIndPower.html)
aren't particularly clear for novices.

At first, I was trying to compare the results of using the helper functions `ttest_power` and `tt_ind_solve_power` and was flummoxed as to why the resulting power didn't match. I now see why that was incorrect, but I think this package could do a better job helping users get to what they want quicker. **Or** I could just be especially slow on the uptake.

### Suggested Changes
* Adding documentation describing the differences between the power calculation frameworks with example code and interpretation
* Potentially adding some additional explanatory output along the lines of R's `power.prop.test()`

Definitely open to thoughts / suggestions / other ideas on how to add clarity here. Happy to work on a first-pass of doc-writing as well.

user: mikekaminsky
issue_4073: state: open
title: ENH: functional data analysis highest density region
body: PR #3876 was merged as rebased version in #4049 
This implements a plot function similar to the existing functional plots.

However, #3876  contains more discussion and started out as more general class based implementation.

I didn't go through all the algorithmic details, and have only a rough idea. Several things look like they could be used for more general use cases in statsmodels.functional (which currently doesn't exist)

see #3836


user: josef-pkt
issue_4062: state: open
title: omni_normtest TODO ntoe
body: from stats.stattools.omni_normtest
```
    # TODO: change to exception in summary branch and catch in summary()
    # behavior changed between scipy 0.9 and 0.10
```

scipy is up to at least 0.19.1.  Is this actionable?
user: jbrockmendel
issue_4061: state: open
title: REF/BUG: consistent lag choice for default lag acorr_ljungbox
body: https://github.com/statsmodels/statsmodels/pull/3360#issuecomment-274151203

merged PR #4059 changed default lag for acorr_ljungbox
statespace.mlemodel uses large default lag



user: josef-pkt
issue_4045: state: open
title: DOC: example application A/B testing
body: mainly to park a link to https://stackoverflow.blog/2017/10/17/power-calculations-p-values-ab-testing-stack-overflow/

proportion tests, power and a R shiny application, with a use case description that focuses on statistics.

We should have all the tools to replicate it in Python 
user: josef-pkt
issue_4039: state: open
title: TST/FAQ-D/Design: ensure data is in `model.__init__`
body: idea base on #3882 

We should support predicting with a new model and new dataset using existing params, e.g.

```
params = model_class(endog, exog, ...).fit(...).params
prediction_new = model_class(endog_new_or_fake, exog_new).predict(params)
```

same for loglike, score and similar. This is currently used in the new count models as shortcut for predicted probabilities (exp(loglike)).

This only works for methods that are supported by the model:

- no transformation of exog_new because that is in results (same with pandas wrapping)
- no inference because it those are results methods, would need recreating results, e.g. full support for maxiter=0, but would also need pre-existing cov_params.
- possible problem for nuisance or auxiliary parameters that are not in `params`, e.g. scale. But in most or all existing cases those are not needed for prediction.

We can use generic test to check that this is satisfied, which would catch models that specify too much in fit. I think all Models outside of tsa will satisfy this already. Some tsa models like ARIMA have not been fully refactored after we changed our initial design from using model specification in `fit`.

Several related issues (todo: find them)
Outside of tsa we also have the option to use new data as argument in model.predict, but in tsa we also need initial observations for endog, and as history for forecasting.




user: josef-pkt
issue_4038: state: open
title: SUMM/ENH: models with penalization
body: related issue: covariance shrinkage, and possible other non-model penalization

PRs, current code in statsmodels

- discrete fit_regularized
- elastic net fit_regularized, GLM, OLS/WLS, minimal Results class
-  #1665 TheilGLS PR, merged, sandbox, needs follow-up
- #2374 penalized MLE, main PR for Mixin and generic setup, penalization functions
- #2744 (#2435), #2580  GAM, penalized splines, main application
- #4683 (merged), issue #2944 ultra high screening/SCAD
 

issues, some just with some references

- #2293 old issue with design ideas (partially reflected in PRs)
- #3210, #2882 structured penalization,  where are code and examples? partially next to TheilGLS
- #3323, #3250
- #3561 Firth type bias-reducing penalization, no code yet

and I guess some more, and some with specific design questions


**algorithms**

- explicit linalg solution, Theil/Ridge
- penalized IRLS, local quadratic approximation
- artificial observations, adjustments to observations (used in GAM)
- scipy optimizers for smooth problems
- L1-penalized constrained optimization, interior point used in discrete
- coordinate descend used in elastic net
- backfitting with univariate penalization (only in ancient sandbox)
- iterative special algorithm: ultrahigh screening

**the pieces**

- penalization functions and associated functions like shrinkage or thresholding functions
- structures/contrasts as penalization target or prior structure
- optimization, as model method or as standalone classes
- results: unclear inference except of M-estimator with L2 "prior"
  - short term: maybe ignore dropped (shrunk to zero variables) as in discrete fit_regularized

one only partially solved problem is how to generically generate the instance of the results class, 
currently some workarounds and partial support

another problem: verifying unit tests are difficult because parts are not available in other packages. Some parts can be cross-checked with different implementation and models, e.g. GAM has unit tests but cov_params is difficult because there are many possible choices and the R package changed several times it's implementation.
user: josef-pkt
issue_4031: state: open
title: Could you use label "Good as First PR" a bit more ?
body: Hi, 
I'm would like to invest myself in this package. However due to the large number of issues it is difficult to know which issues I should take. I have noticed the "Good as First PR" Label but unfortunately there are only few and old issues.

So  could you tag next issues that are not so difficult to fix in order that I try to do my first push request ?

I'm particularly interested to work on Time Series models.

Many thanks !
 
Malick
user: malickf
issue_4030: state: open
title: SUMM: Topic Week - Internal Consistency
body: - Model/Results attributes consistency
   - `GLMResults._endog` --> `self.endog` or just `self.model.endog` #4028
   - some Results classes attach `endog, exog, nobs,...`, others don't.
   - Results attributes getting pinned to models #4016, #3984
   - Attributes in FooResultsWrapper #3920 
   - Make `**kwargs` explicit where possible, validation #3890, #4019, #2163
   - Avoid creating new attributes outside of `__init__`.  Ideally avoid changing things in-place.  #3787.
    - LikelihoodModelResults object has no attribute mle_retvals #3708
    - "J" in GEEMargins?  #3671
    - cov_struct.Nested AttributeError 'vcomp_coeff'  #3668
    - fit_regularized has no get_prediction, cov_params #3626
    - PoissonZiGMLE: predict not implemented?  #3496
    -  AR/ARMA/... avoid `set_sigma2` pattern; avoid setting model attributes that are really Result attributes during `fit`.
    - VAR, trend needs to be in `__init__` #2273 (this one is tough but would be a huge win)
   - required attributes of models and results #1970

- Class Hierarchy
   - VARResults, SVARREsults, VECMResults don't subclass Results (TODO: double-check).
   - `DiscreteResults.__init__` doesn't call `super(...)`
   - This makes it harder to reason about what attributes can consistently be expected to exist in a Results instance.
   - SVAR should subclass VAR, will raise AttributeError in `fit` if `select_order` is reached.
   - tsa.statespace has some special handling for model.data.  Life would be better if special cases were mainlined.
   - MICEResults subclasses LikelihoodModelResults, but MICE does not subclass Model.  So MICEResults.model does not behave as expected.  #3324

- `get_robustcov_results`
  - Defined in several places; There Can Be Only One #4014
  - Usually called in `FooResults.__init__`, but sometimes called in `FooModel.fit` #4025
  - Small differences between the versions in linear_model vs base.covtype
  - Small differences between the versionsin regime_switching vs statespace

- Naming Consistency
   - Lots of places in sandbox and tsa that don't follow naming conventions #4015
   - `self.y` --> `self.endog` in e.g. VARResults
  - Naming of weights arguments in WLS #3891 
   - orig_endog vs endog_lagged vs... #3689
   - `regressor` vs `exog` for IV, 2SLS? #2248

- Degrees of Freedom
    - mixed_linear_model doesn't define df_model, aic/bic define their own `df` variables #3742
    - when to use k_exog vs rank?  #1928
    - when to use `# assumes constant`?  Make this explicit
    - GEE cases not tested?  #3726
    - aic in AR/ARIMA/... #3574
    - df_resid in ar/arima/var_model  #3568
    - ddof for models #2799
    - arma as `_ic_df_model` that is just df_model+1, deprecate/remove

- Modernization
    - Min versions #3999 
    - Functions that are no longer used because upstream versions are available #4008
    - cythonize may now be handled upstream, need to check.
    - deprecate lower-case link classes #3462
    - replace panel with xarray #2917

- flake8 cleanup and related
   - many `if __name__ == "__main__"` blocks should be removed; some of them can be turned into tests
   - Some base class methods just `pass` instead of raising NotImplementedError or AbstractMethodError
    - Some places print instead of warn #3163
    - lots of commented out print statements
    - 1-letter variable names

- Remove
    - sandbox functions that have been replaced #3442
    - removal of deprecated objects for 0.9 #3443
    - large parts of `statsmodels.tools` and `statsmodels.stats` are never used in the "core", may belong in sandbox.  The idea being to have a clear separation between "actively-maintained, thoroughly tested" and "not that".
    - summary vs summary2: highlander style

- Other
    - MixedLinearModel setting params.fe_params inconsistently #3660
    - Can parts of miscmodels be subsumed into discrete.count_models?  #3738
    - Some types of tests return HypothesisTestResults or something, most dont
    - Functions with multiple return types/shapes (usually tuples of differing lengths)
    - sandbox functions imported by non-sandbox belong in non-sandbox
    - tools._testing vs tools.testing?
    - To make maintenance easier medium-long term, can we make some things explicitly internal so as to avoid backward-compat concerns?
user: jbrockmendel
issue_4025: state: open
title: Inconsistency in get_robustcov usage
body: Some models call some variant of `get_robustcov_results` at the end of `fit` (e.g. `GenericZeroInflated`, `GeneralizedPoisson`, `NegativeBinomialP`, `NegativeBinomial`).  Others do this in `XYZResults.__init__` (e.g. `DiscreteResults`,`MLEResults`, `LikelihoodModelResults`, `GLMResults`).  There aren't many Results subclasses (`VARResults`, `SVARResults`, `VECMResults` come to mind) that don't inherit from `LikelihoodModelResults`, so that covers most everything.

So the question is: why do some models call `get_robustcov_results` in `fit`?
user: jbrockmendel
issue_4024: state: open
title: REF/BUG/Design: GLM versus OLS get_prediction in near-singular corner case
body: partially related to #4001 and #2628 and checking get_prediction

I used a near-singular example, just below the "singular" rcond cutoff for investigating #4001 
see also #3994 changes in default rcond in numpy (for lstsq)

When I use the same example for prediction, then I get numbers in GLM, but not when I run the same examle as OLS.
In OLS get_prediction the mean var is negative and mean_se is nan.

This is just another edge case as in #2628 with almost singular design but where regularization does not kick in yet.
However, I don't see where the difference between GLM Gaussian and OLS comes from. 
GLM Gaussian is (supposed to be) using WLS with pinv which should be the same as OLS with pinv, except WLS is run twice, I guess, in GLM (or more precisely one np.linalg.lstsq followed by one WLS with pinv).


user: josef-pkt
issue_4022: state: open
title: zt_ind_solve_power returns numpy array instead of float
body: `sm.stats.zt_ind_solve_power` returns an array if `nobs1` is less than 197. I'm sure 197 is not special; it's just the result of the other settings in this specific case. The [documentation](http://www.statsmodels.org/dev/generated/statsmodels.stats.power.zt_ind_solve_power.html) says that a float is returned, not an array. If an array is returned, should it be assumed that there is no answer for the `ratio`?
```
es = 0.20
alpha = 0.05
power = 0.80
for n1 in range(1, 401, 10):
	n1, sm.stats.zt_ind_solve_power(es, n1, alpha, power, ratio=None, alternative='two-sided')

(1, array([ 12926.]))
(11, array([ 7233.36610346]))
(21, array([ 14302.63047839]))
(31, array([ 294141.47907722]))
(41, array([ 424723.9626395]))
(51, array([ 653547.47554599]))
(61, array([ 1047796.24848788]))
(71, array([ 864465.36172393]))
(81, array([ 1457060.60164413]))
(91, array([ 1248130.6972059]))
(101, array([ 2165978.03108565]))
(111, array([ 1886292.83657938]))
(121, array([ 3004670.74975744]))
(131, array([ 4855495.1619719]))
(141, array([ 3969905.79322159]))
(151, array([ 6555037.82106867]))
(161, array([ 5454723.46003231]))
(171, array([ 9135790.240248]))
(181, array([ 15376233.87906924]))
(191, array([ 41926014.47320025]))
(201, 41.06351974728969)
(211, 13.277510022299376)
(221, 7.919027208755753)
(231, 5.642037224648649)
(241, 4.382048715544689)
(251, 3.5820930501426567)
(261, 3.029115166586838)
(271, 2.6240369376713617)
(281, 2.314520102141116)
(291, 2.070316800463505)
(301, 1.8727279560623094)
(311, 1.709566984615799)
(321, 1.5725585411403584)
(331, 1.4558815605038435)
(341, 1.3553226926244626)
(351, 1.267757015041292)
(361, 1.1908199627958316)
(371, 1.122686872917464)
(381, 1.0619267276423723)
(391, 1.0074094489317402)
```

Just for reference, with R using rpy2 pwr library, if I attempt this same thing, an error is thrown with the following:

```
>>> for i in range(2, 1000):
	pwr.pwr_2p2n_test(h=0.20, n1=i, sig_level=alpha, power=power, alternative="two.sided")[2][0]

	
Traceback (most recent call last):
  File "<pyshell#45>", line 2, in <module>
    pwr.pwr_2p2n_test(h=0.20, n1=i, sig_level=alpha, power=power, alternative="two.sided")[2][0]
  File "C:\Python35\lib\site-packages\rpy2\robjects\functions.py", line 178, in __call__
    return super(SignatureTranslatedFunction, self).__call__(*args, **kwargs)
  File "C:\Python35\lib\site-packages\rpy2\robjects\functions.py", line 106, in __call__
    res = super(Function, self).__call__(*new_args, **new_kwargs)
rpy2.rinterface.RRuntimeError: Error in uniroot(function(n2) eval(p.body) - power, c(2 + 1e-10, 1e+09)) : 
  f() values at end points not of opposite sign
```

```
>>> es = 0.20
>>> for i in range(196, 1000):
	pwr.pwr_2p2n_test(es, n1=i, sig_level=alpha, power=power, alternative="two.sided")[2][0]

	
Traceback (most recent call last):
  File "<pyshell#48>", line 2, in <module>
    pwr.pwr_2p2n_test(es, n1=i, sig_level=alpha, power=power, alternative="two.sided")[2][0]
  File "C:\Python35\lib\site-packages\rpy2\robjects\functions.py", line 178, in __call__
    return super(SignatureTranslatedFunction, self).__call__(*args, **kwargs)
  File "C:\Python35\lib\site-packages\rpy2\robjects\functions.py", line 106, in __call__
    res = super(Function, self).__call__(*new_args, **new_kwargs)
rpy2.rinterface.RRuntimeError: Error in uniroot(function(n2) eval(p.body) - power, c(2 + 1e-10, 1e+09)) : 
  f() values at end points not of opposite sign

>>> for i in range(197, 1000):
	pwr.pwr_2p2n_test(es, n1=i, sig_level=alpha, power=power, alternative="two.sided")[2][0]

	
49654.8108846919
21845.45271925996
14053.719626671102
```
user: jaradc
issue_4018: state: open
title: FAQ-D/TST/ENH: missing models in generic tests, e.g. pickle/remove_data and Wald inference
body: Some models need to be added to generic pickle tests, if they don't have their own tests.
Some models are also missing in generic test for wald inference and similar.

related: the two generic test modules in base need review.

related test helper functions #1909  which could also work as a substitute for generic tests #3820
user: josef-pkt
issue_4013: state: open
title: BUG: GLMResults get_prediction ignores linear keyword
body: reading the code

first we compute linear prediction
instead of returning the result if linear=True, the computation continue to compute and return prediction for linear=False.
user: josef-pkt
issue_4003: state: open
title: DOC/REL/Examples  new features or changes in 0.9
body: This is just a partial task list to keep track of a few items

e.g. changes in optimization

- changes and improvements in GLM.fit
   I never ran example for this #4001 for method in IRLS, #3646, #3257 for EIM in gradient optimization
   needs release notes and examples, 
   fit_irls changes default method (almost the same svd based algorithm but different numpy/LAPACK function)

- changes in discrete, llnull, start_params, needs docs release notes, and description of new options

- fit `method='minimize'` check documentation, release notes

- increase default maxiter in discrete and ARMA/ARIMA #4004


recent ENH merges

- hdrboxplot
- lilliefors exponential
- seasonal_decompose, 2d and trend extrapolation
- update sandbox TheilGLS
- GLM irls wls_method, minimal WLS -> optimization
- summary_col


user: josef-pkt
issue_3994: state: open
title: REF/SUMM: numpy lstsq changes default rcond in 1.14
body: https://github.com/numpy/numpy/pull/9582/files

independently of whether the change is better than the current default:

- we might have more discrepancies in rcond/threshold across linalg function, ie. algorithm option
   (main current candidate switching linalg method in GLM.fit IRLS)
- It will change results compared to our current behavior. But most likely we don't have unit tests where the change in rcond affects the results.
   e.g. #3990 sklearn classification examples are highly collinear, which seems to adversely affect GLM IRLS
- results will depend on numpy version in highly collinear cases.

However, the current rcond independent of nobs is also fragile #2628
old issue for differences in threshold: #1662 (but mainly difference svd(x) versus svd(x.T x)

issue trigered by
https://github.com/scipy/scipy/pull/7995#discussion_r143332191

ilayn: "These bounds are kind of standard. (well standard in the sense "that's what matlab does"). NumPy also picked these up for matrix_rank tolerance which should be inline with null basis column number. You can see more about it here https://www.ma.utexas.edu/documentation/lapack/node92.html
"
josef-pktd"
The lapack links says "p(m , n) is a modestly growing function of m and n"
My question is whether linear growth in m is not too much (e.g. sqrt(m) or log(M)?). statsmodels has some issues that constant in m (rcond independent of m) is too small.
"

TODO:
- check consistency across fit methods and matrix_rank
- check what p(m, n) (LAPACK) or in our case rcond(nobs) would look good in our applications. E.g. If something like law of large numbers in error accumulation holds, then it would be sqrt(nobs)
  good example to start might be the overparameterized categorical exog in #2628
- check whether we should stick with numpy default or override and try to have our own consistent rcond function

user: josef-pkt
issue_3990: state: open
title: Performance: Logit and GLM, and sklearn, IRLS slow with highly collinear exog
body: based on a discussion on the sklearn mailing list, and some timings shown there

GLM IRLS is very slow compared to the other optimizers in artificial datasets created with sklearn. The main problem seems to be that several singular values are in the neighborhood of 1e-14 or 1e-15.
(My first guess was that convergence criterion in GLM uses deviance, but the slowness is also with params)

If I add noise to exog to make it well conditioned, as in the following script, then IRLS and other optimizers are comparable. In a larger dataset nobs=1Million, GLM takes about 4 times longer than Logit, using current master.

(Note, timing in statsmodels include covariance estimation)

```
sklearn.LogisticRegression: l1 6.6286749839782715
sklearn.LogisticRegression: l2 7.15504789352417
QC check did not pass for 51 out of 51 parameters
Try increasing solver accuracy or number of iterations, decreasing alpha, or switch solvers
Could not trim params automatically due to failed QC check.  Trimming using trim_mode == 'size' will still work.
sm.Logit l1 4.817397832870483
sm.Logit l1_cvxopt_cp 26.204403162002563
sm.Logit newton 6.074285984039307
sm.Logit nm 135.2503378391266
m:\josef_new\eclipse_ws\statsmodels\statsmodels_py34_pr\statsmodels\base\model.py:511: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
  "Check mle_retvals", ConvergenceWarning)
sm.Logit bfgs 5.553915977478027
sm.Logit lbfgs 4.537201166152954
sm.Logit powell 271.670814037323
sm.Logit cg 8.214794874191284
sm.Logit ncg 7.019952058792114
sm.GLM.fit IRLS 20.715612173080444
sm.GLM L2: 157.6966769695282
sm.GLM.fit bfgs 19.05343985557556
sm.GLM.fit lbfgs 17.348237991333008
sm.GLM.fit newton 18.742539167404175
m:\josef_new\eclipse_ws\statsmodels\statsmodels_py34_pr\statsmodels\base\model.py:511: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
  "Check mle_retvals", ConvergenceWarning)
sm.GLM.fit nm 29.833044052124023
sm.GLM.fit powell 172.68626284599304
sm.GLM.fit cg 19.932121992111206
sm.GLM.fit ncg 18.5813729763031
sm.GLM.fit basinhopping 1227.4370460510254
```

----------------
```
# -*- coding: utf-8 -*-
"""
Created on Thu Oct  5 15:48:35 2017

scikit-learn mailing list
"""

import numpy as np
import sklearn.datasets
from sklearn.preprocessing import StandardScaler

#np.random.seed(987126)
X, y = sklearn.datasets.make_classification(n_samples=1000000,
n_features=50, random_state=987126) #123)
X0 = StandardScaler(copy=True, with_mean=True, with_std=True).fit_transform(X)
from statsmodels.tools.tools import add_constant
X += 0.5 * np.random.randn(*X.shape)
X = add_constant(X)
import time
tol = 0.0001
maxiter = 1000
DISP = 0


if 1: # sk.LogisticRegression
    import sklearn
    from sklearn.linear_model import LogisticRegression

    for method in ["l1", "l2"]: # TODO, add solvers:
        t = time.time()
        model = LogisticRegression(C=1, tol=tol, max_iter=maxiter,
penalty=method)
        model.fit(X0,y)
        print("sklearn.LogisticRegression:", method, time.time() - t)




if 1: # sm.Logit.fit_regularized
    from statsmodels.discrete.discrete_model import Logit
    for method in ["l1", "l1_cvxopt_cp"]:
        t = time.time()
        model = Logit(y,X)
        result = model.fit_regularized(method=method, maxiter=maxiter,
                                       alpha=1.,
                                       abstol=tol,
                                       acc=tol,
                                       tol=tol, gtol=tol, pgtol=tol,
                                       disp=DISP)
        print("sm.Logit", method, time.time() - t)

if 1: # sm.Logit.fit
    from statsmodels.discrete.discrete_model import Logit

    SOLVERS = ["newton", "nm",
"bfgs","lbfgs","powell","cg","ncg"] #,"basinhopping",]
    for method in SOLVERS:
        t = time.time()
        model = Logit(y,X)
        result = model.fit(method=method, maxiter=maxiter,
                           niter=maxiter,
                           ftol=tol,
                           tol=tol, gtol=tol, pgtol=tol,  # Hmmm.. needs to be reviewed.
                           disp=DISP)
        print("sm.Logit", method, time.time() - t)

if 1: # sm.GLM.fit
    from statsmodels.genmod.generalized_linear_model import GLM
    from statsmodels.genmod.generalized_linear_model import families
    for method in ["IRLS"]:
        t = time.time()
        model = GLM(y, X, family=families.Binomial(link=families.links.logit))
        result = model.fit(method=method, atol=tol,
maxiter=maxiter, full_output=False, disp=DISP, tol_criterion='params')
        print("sm.GLM.fit", method, time.time() - t)


if 1: # GLM.fit_regularized
    from statsmodels.genmod.generalized_linear_model import GLM
    from statsmodels.genmod.generalized_linear_model import families
    t = time.time()
    model = GLM(y, X, family=families.Binomial(link=families.links.logit))
    result = model.fit_regularized(method='elastic_net', alpha=1.0,
L1_wt=0.0, cnvrg_tol=tol, maxiter=maxiter)
    print("sm.GLM L2:", time.time() - t)



if 1: # GLM.fit
    # Hits division by zero.
    SOLVERS = ["bfgs","lbfgs", "newton", "nm",
"powell","cg","ncg","basinhopping",]
    from statsmodels.genmod.generalized_linear_model import GLM
    from statsmodels.genmod.generalized_linear_model import families
    for method in SOLVERS:
        t = time.time()
        model = GLM(y, X, family=families.Binomial(link=families.links.logit))
        result = model.fit(method=method,
#                           scale="X2",
#                            alpha=1.,
#                            abstol=tol,
#                            acc=tol,
#                            tol=tol, gtol=tol, pgtol=tol,
#                            maxiter=maxiter,
#                            #full_output=False,
                           
                           disp=DISP)
        print("sm.GLM.fit", method, time.time() - t)
```
------------------
user: josef-pkt
issue_3989: state: open
title: BUG/ENH: robust joint loc/scale estimation convergence problems
body: https://stackoverflow.com/questions/46606082/using-huber-scale-and-location-estimator-in-statsmodel

robust.scale.Huber fails to converge even with large maxiter

The data is trimodal (three separate parts) which might be the source of the problem.
AFAIR, I never looked at convergence issues there. One issue is that scale.Huber is not designed as a Model so we don't get much extra information out of it. 

IIRC, Huber-Dutter would be the equivalent model. But I cannot replicate the estimate of scale.Huber with RLM. Either I don't have the right norms or norm threshold parameters, or there is something more fundamentally different in the algorithm.

```
a=[4.3498776644415429, 16.549773154535362, 4.6335866963356445, 8.2581784707468771, 1.3508951981036594, 1.2918098244960199, 5.7349939516388453, 0.41663442483143953, 4.5632532990486077, 8.1020487048604473, 1.3823829480004797, 1.7848176927929804, 4.3058348043423473, 0.9427710734983884, 0.95646846668018171, 0.75309469901235238, 8.4689505489677011, 0.77420558084543778, 0.76506022382450845, 1.5673666392992407, 1.4109878442590897, 0.45592078018861532, 4.71748181503082, 0.65942167325205436, 0.19099796838644958, 1.0979997466466069, 4.8145761128848106, 0.75417363824157768, 5.0723603274833362, 0.30627007428414721, 4.8178689054947981, 1.5383475959362511, 0.7971041296695851, 4.689826268915076, 8.6704498595703274, 0.56825576954483947, 8.0383098149129708, 0.39400084281108422, 0.89827542590321019, 8.5160701523615785, 9.0413284666560934, 1.3590549231652516, 8.355489609767794, 4.2413169378427682, 4.8497143419119348, 4.8566372637376292, 0.80979444214378904, 0.26613505510736446, 1.1525345100417608, 4.9784132426823824, 1.0766360391211101, 1.9604545887151259, 0.77151237419054963, 1.2302626325699455, 0.846912462599126, 0.85852710339862037, 0.38035542024830299, 4.7586522644359093, 0.46796412732813891, 0.52933680009769146, 5.2521765047159708, 0.71915381047435945, 1.3502865819436387, 0.76647272458736559, 1.1206637428992841, 0.72560665950851866, 4.4248008256265781, 4.7984989298357457, 1.0696617588880453, 0.71104701759920497, 0.46986438176394463, 0.71008686283792688, 0.40698839770374351, 1.0015132141773508, 1.3825224746094535, 0.93256270304709066, 8.8896053101317687, 0.64148877800521564, 0.69250319745644506, 4.7187793763802919, 5.0620089438920939, 5.1710564773987233, 9.5341720525579809, 0.43052713463119635, 0.79288845392647533, 0.51059695992994469, 0.48295891743804287, 0.93370512281086504, 1.7493284310512855, 0.62744557356984221, 5.0965146009791704, 0.12615625248684664, 1.1064189602023351, 0.33183381198282491, 4.9032450273833179, 0.90296573725985785, 1.2885647882049298, 0.84669066664867576, 1.1481783837280477, 0.94784483590946278, 9.8019240792478755, 0.91501030105202807, 0.57121190468293803, 5.5511993201050887, 0.66054793663263078, 9.6626055869916065, 5.2628061618536908, 9.5905100705465696, 0.70369230764306401, 8.9747551552440186, 1.572014845182425, 1.9571634928868149, 0.62030418652298325, 0.3395356767840213, 0.48287760518144929, 4.7937042347984198, 0.74251393675618682, 0.87369567300592954, 4.5381205696031586, 5.2673192797619084]
```
user: josef-pkt
issue_3988: state: open
title: TST: checking option combinations, data
body: I think we have to become more systematic in checking option combinations
e.g. predict returning pandas Series needed several fixes

For data, e.g. in predict

- 1-d versus 2-d 
   both numpy arrays and pandas Series/DataFrames
- numpy versus pandas
- with or without formulas and going through patsy, in predict case also `transform` keyword
- other supported data structures in special cases, e.g. dict in predict

this sounds like something like 9 or more possibilities

I think we have to also think more often in other cases what option combinations are possible, and which might cause errors. (purely inherited possibilities or option combinations are less risky.)

user: josef-pkt
issue_3981: state: open
title: SUMM/Topic Week: gofplots probability plots
body: several open issues and PRs

it will be time consuming to work through all option combination

I was looking briefly at #3547 and AFAICS the current version does not allow us to turn off loc and scale fit
I tried some `normal` examples and fit=False and fit=True plots look the same when I use the class, but maybe (?) not when using the qqplot helper function (which doesn't make sense base on the code)

Thinking about fit cases again

- no fit at all, requires fully specified distribution
  compare data with given distribution, no standardization, no estimation, not even loc and scale
  expected: this should show clear discrepancies if there is only a loc shift or loc-scale transformation
  e.g. straight line qqplot but not at 45 degree line
- fit loc scale
  This is equivalent to full fit for loc-scale families, without shape parameters, e.g. normal.
  this seems to be the current default, or only solution if options are ignored.
  expected: normal data should always fit well. distributions with shape parameter should show discrepancy coming from the shape parameter, e.g. df in t-distribution
- full fit
  only relevant in distribution with shape parameters, otherwise it is just the previous fit loc scale case. 
  expectation: good fit if data is from the null distribution family

display axis, should be orthogonal to fit issue, but is not if we use standardized versions.

ppplots are always in terms of probabilities on axis, not much choice

qqplots
y - axis
- raw scale: don't transform data quantiles, available as prob_plots with raw scale y axis
- sample standardized y-axis, this is the current `sample percentiles` (?), standardized by estimated loc and scale  (to have this version might be a bug)
- theoretical standardized sample quantiles: I think this is missing and should be the case when fit is False, i.e. standardize by given loc-scale (which might be estimated and then coincide with the previous case)  `sample_quantiles` uses self.loc and self.scale, in contrast to sample_percentiles

x - axis in qq plot is easier and should use the corresponding distribution under the null
two possibilities
- use loc-scale standardized distribution as xlabels
- use raw distribution as xlabels

Once the *content* of the plot is defined, then the axis can be transformed e.g. nonlinearly to match some probability paper which would often/mainly affect axis labeling.
(The current `_fmt_probplot_axis` seems to do this and can be extended to Paul Hobson's newer work on axis scale in/for matplotlib.)

first task: create a pandas dataframe with the quantities for all options.
then start to check plots for all cases

and this needs unit tests (not smoke tests), otherwise I and the code will get confused again.

  
user: josef-pkt
issue_3967: state: open
title: different implementations of mackinnoncrit in coint vs adfuller
body: Can someone please explain why the `mackinnoncrit ` implementation is different in `coint` vs `adfuller`? In `adfuller` we have this as

`critvalues = mackinnoncrit(N=1, regression=regression, nobs=nobs)`

whereas in `coint` we have this as:

`mackinnoncrit(N=k_vars, regression=trend, nobs=nobs - 1)`

which results in a different set of critical values for the same input. I would actually expect these to be the same, since they are supposed to be used for similar purposes?

As an example of values I see please see screenshot below.


![image](https://user-images.githubusercontent.com/13929456/31129220-cd99acc8-a84b-11e7-9324-603860fe705d.png)


user: tsando
issue_3961: state: open
title: REF: MultinomialModel subclasses BinaryModel
body: Why does multinomial subclass binary
`class MultinomialModel(BinaryModel)`

AFAICS, MultinomialModel  defines all methods that BinaryModel, so it should reuse any of the binary model code. The only exception is `__init__` which has an explicit model class check to NOT do something in the multinomial case, and otherwise just delegates to super
user: josef-pkt
issue_3957: state: open
title: ENH: allpermutation iterator - multiset version
body: (2 hour coding exercise that took a bit longer with cleanup and unit tests)
Get all unique permutations from a list of values, where values can be repeated.

Main application two sample permutation test where values need only be binary. We can get all permutation for very small samples. 
maybe for #3576 and #2324 (code in scipy PR) and others
I don't know yet where this should go.

algorithm 1 of Aaron Williams, 2009

after translating this to python lists, I found other similar implementation that are closer to the original pointer arithmetic, which is absent in my version (Python lists are linked lists, AFAIR).

https://github.com/ekg/multipermute/blob/master/multipermute.py
answer by smitchr here which is misattributed to Takaoka instead of Williams, AFAICS:
https://stackoverflow.com/questions/19676109/how-to-generate-all-the-permutations-of-a-multiset

sympy has an implementation shown in the answer at
https://stackoverflow.com/questions/41210142/get-all-permutations-of-a-numpy-array
sympy has a lot more on combinatorics and iterators for it

----------------
```
# -*- coding: utf-8 -*-
"""
Created on Sat Sep 30 17:08:15 2017

Author: Josef Perktold
License: BSD-3
"""

def _shift(li, idx):
    """move element from position idx to front
    """
    val = li.pop(idx)
    li.insert(0, val)

    
def permutation(li, issorted=False, copy=True):
    """iterator for unique permutations of multiset

    This implements Algorithm 1 of Williams 2009 using a Python lists instead 
    of pointer arithmetic.

    Parameters
    ----------
    li : list
        list with elements for permutation. Tha algorithm needs
        the list sorted in non-increasing order. The list elements
        can be any type that implements comparisons and is sortable.
    issorted : boolean
        If False (default), then the list will be sorted.
    copy : bool
        If False, then the original list will be modified inplace.
        Warning, the yielded list is always an inplace modification and refers
        to the same object each time.

    Returns
    -------
    yields permuted lists. The yielded list is muted inplace.

    Notes
    -----
    Warning: This iterator can be long. The number of permutations
    is multinomial, N! / (n1! n2!, n3! ...)
    where N is the length of the list and ni are the counts for the
    unique values.

    References
    ----------
    Williams, Aaron. "Loopless generation of multiset permutations by prefix
    shifts." In Symposium on Discrete Algorithms (SODA), New York, USA. 2009.

    Examples
    --------
    >>> for i in permutation([1, 1, 1, 0]): print(i)
    ...
    [1, 1, 1, 0]
    [0, 1, 1, 1]
    [1, 0, 1, 1]
    [1, 1, 0, 1]

    >>> for i in permutation(['a', 'b', 'c', 'c']): print(i)
    ...
    ['c', 'c', 'b', 'a']
    ['a', 'c', 'c', 'b']
    ['c', 'a', 'c', 'b']
    ['c', 'c', 'a', 'b']
    ['b', 'c', 'c', 'a']
    ['c', 'b', 'c', 'a']
    ['a', 'c', 'b', 'c']
    ['c', 'a', 'b', 'c']
    ['b', 'c', 'a', 'c']
    ['a', 'b', 'c', 'c']
    ['b', 'a', 'c', 'c']
    ['c', 'b', 'a', 'c']

    >>> from scipy.misc import comb, factorial
    >>> f = factorial
    >>> f(4) / (f(1) * f(1) * f(2))
    12.0

    # Note to save each permuted list it needs to be copied.
    >>> p = [i.copy() for i in permutation([1, 1, 0, 0], issorted=True, copy=False)]
    >>> np.array(p)
    array([[1, 1, 0, 0],
           [0, 1, 1, 0],
           [1, 0, 1, 0],
           [0, 1, 0, 1],
           [0, 0, 1, 1],
           [1, 0, 0, 1]])
    >>> comb(4, 2)
    6.0

    """
    li = list(li)
    if copy is True:
        li = li.copy()
    if len(li) == 0:
        # empty list not handled below
        yield li
        return
    if issorted is False:
        li.sort(reverse=True)
    if li[0] == li[-1]:
        # shortcircuit
        yield li
        return

    n = len(li)
    i = n - 2  # second last element
    yield li # yield initial list
    while i + 2 < n or (i + 1 < n and li[i+1] < li[0]):
        if i + 2 < n and li[i] >= li[i+2]:
            beforek = i + 1
        else:
            beforek = i

        k = beforek + 1
        if li[k] < li[0]:
            i = 0
        else:
            i += 1
        _shift(li, k)

        yield li

        
import numpy as np
from scipy.misc import comb, factorial
from numpy.testing import assert_equal, assert_array_equal, assert_, assert_allclose

def test_permute_multiset():
    k1, k0 = 2, 3
    n = k0 + k1
    nc = comb(n, k1)
    mylist = [1]*k1 + [0]*k0
    
    res = [[1, 1, 0, 0, 0],
           [0, 1, 1, 0, 0],
           [1, 0, 1, 0, 0],
           [0, 1, 0, 1, 0],
           [0, 0, 1, 1, 0],
           [1, 0, 0, 1, 0],
           [0, 1, 0, 0, 1],
           [0, 0, 1, 0, 1],
           [0, 0, 0, 1, 1],
           [1, 0, 0, 0, 1]]
    
    p = [i.copy() for i in permutation(mylist.copy(), issorted=True, copy=False)]
    assert_array_equal(np.asarray(p).mean(0), k1 / n * np.ones(n))
    assert_array_equal(p, res)
    assert_equal(len(p), nc)
    
    # test string valued
    res = [['c', 'c', 'b', 'a'],
           ['a', 'c', 'c', 'b'],
           ['c', 'a', 'c', 'b'],
           ['c', 'c', 'a', 'b'],
           ['b', 'c', 'c', 'a'],
           ['c', 'b', 'c', 'a'],
           ['a', 'c', 'b', 'c'],
           ['c', 'a', 'b', 'c'],
           ['b', 'c', 'a', 'c'],
           ['a', 'b', 'c', 'c'],
           ['b', 'a', 'c', 'c'],
           ['c', 'b', 'a', 'c']]
    p = [i.copy() for i in permutation(['a', 'b', 'c', 'c'])]
    f = factorial
    count = f(4) / (f(1) * f(1) * f(2))
    assert_array_equal(p, res)
    assert_equal(len(p), count)
    
    # test special cases
    p0 = [i.copy() for i in permutation([1, 1, 1], issorted=True, copy=False)]
    assert_(p0 == [[1, 1, 1]])
    
    p0 = [i.copy() for i in permutation([], issorted=True, copy=False)]
    assert_(p0 == [[]])

if __name__ == '__main__':
    test_permute_multiset()
    
    k1, k0 = 2, 3 #4, 3
    nc = comb(k1 + k0, k1)
    print(nc, factorial(k1+k0))
    mylist = [1]*k1 + [0]*k0
    
    counter = 0
    for i in permutation(mylist):
        print(i)
        counter += 1
        if counter > nc + 3: break
    print('counter:', counter)
```

user: josef-pkt
issue_3954: state: open
title: SUMM: "topic week" consolidate stats for rates and proportion
body: I completely lost the overview.
There was a lot of discussion and work in 2015, and several more isolated items, issues and PRs since then.

If we ignore coming up with a "final" API and not try to do everything at once, then there are many relatively easy merges. Some code/functions still need to be collected that are not yet in PRs.

e.g. `proportion_confint_2sample` in analogue to proportion_confint and multinomial_proportions_confint
or some name similar to proportion_confint_2sample for independent samples and for paired samples.
or one single function `proportion_confint` with options 'one', 'paired' and 'indep'
(But for two sample we have more options also for which statistic, e.g. difference, ration, odds ratio, where some of this overlaps with 2x2 contingency tables, which don't distinguish between assumed data generating or sampling process.)

In terms of API and implementation we could go for a dual approach, simple function for easy use like proportion_confint, and the class based methods for more complex parts, e.g. permutation, bootstrap,... and all in one (test, confint and power).

current status in master: 
mostly in `stats.proportion`, collection of function without much relationship between them

## Parts, PRs, issues

...


## for unit tests
(which are partially difficult to find):

#3952 DescTools and epiR
#2605 PropCI
and more spread out across R packages with some references and links in issues or PRs

Stata also has a epitab subpackage with related functions, but more field specific API/terminology



user: josef-pkt
issue_3952: state: open
title: TST/ENH/Reference R package for proportions, epidemiology
body: Two packages that have a variety of statistical tests also for proportions and contingency tables

https://www.rdocumentation.org/packages/epiR/versions/0.9-87
https://www.rdocumentation.org/packages/DescTools/versions/0.99.19

This should be helpful for writing unit tests against given the large number of tests, ci and power computation

e.g. https://www.rdocumentation.org/packages/DescTools/versions/0.99.19/topics/BinomDiffCI
A collection for confidence intervals for difference of two proportions based on the same reference that I used. (I don't know what the status of my unit tests are)

Also there are quite a lot of hypothesis tests that I haven't seen yet.
user: josef-pkt
issue_3941: state: open
title: Example/Doc: discrete NFL prediction probabilities
body: (found by chance)
http://fivethirtyeight.com/features/how-to-play-our-nfl-predictions-game/
with historical and current data

sounds interesting for some examples
they have an asymmetric loss function for "scoring" the predictions (maybe the function is just highly nonlinear, mentions Brior score)

e.g. confidence interval sampling distribution of predictive probability
I tried something similar for Poisson.
E.g. bet that team A wins with at least 5 point difference.
except we don't have distribution for correlated difference of count variables (positive and negative integers like Skellam in scipy.)
user: josef-pkt
issue_3938: state: open
title: ENH/Performance: numerical differentiation, score_factor, score derivative
body: two ideas to better exploit the structure of the loglikelihood for possibly higher performance or higher accuracy

Currently we compute derivatives directly from the loglike.

- curse of dimensionality: In most current models, the center part is linear. This means we can use chain rule to compute the derivative in the score_factor and then just multiply by exog. We don't exploit this in numerical differentiation.
- often we have analytical score even when we are not able to get an analytical hessian. We currently don't have the setup for computing a numerical derivative hessian by differentiating the score.

However, our current numdiff functions might not have the correct broadcasting behavior to compute these numerical derivatives. The workaround that I had used in GLM with diag (?) is very expensive.

(The first point above is partially motivated by mailing list thread [SciPy-Dev] "Add feature to approx_fprime". The second point comes from count models and score or conditional moment tests.)

user: josef-pkt
issue_3937: state: open
title: Error `ValueError: f(a) and f(b) must have different signs` for anova power analysis
body: Hello,

I was trying to perform a power analysis using class FTestAnovaPower. However, when I tried to run method solve_power, I get the following error message `ValueError: f(a) and f(b) must have different signs`. What should I do to fix this error?

My statsmodels version is 0.6.1. Here's the complete snippet of the code
```python
from statsmodels.stats.power import FTestAnovaPower
f = FTestAnovaPower()
f.solve_power(nobs=17500, alpha=0.95, k_groups=7, power=0.8, effect_size=None)
```
Thanks!
user: mitbal
issue_3933: state: open
title: KDEMultivariateConditional 'bw' argument is required, contrary to docstring
body: The documentation for `statsmodels.nonparametric.kernel_density.KDEMultivariateConditional` says that the `bw` argument is optional:

https://github.com/statsmodels/statsmodels/blob/18983eaba57ae25dfb2d058c884aa86935b248d9/statsmodels/nonparametric/kernel_density.py#L373

But the code says it's required:

https://github.com/statsmodels/statsmodels/blob/18983eaba57ae25dfb2d058c884aa86935b248d9/statsmodels/nonparametric/kernel_density.py#L410

I'm not sure whether the code or the documentation should be changed in this case.
user: mdickinson
issue_3929: state: open
title: ENH: GeneralizedPoisson at alpha=0, score_test
body: see also #3887 general issues for count model improvements

It looks like GeneralizedPoisson is well defined at alpha=0, which is the Poisson special case

both score and hessian agree with poisson for the mean parameters
score and hessian w.r.t alpha looks fine

This means we can also get the score test at alpha=0 directly from the GPP model as a parameter restriction. We can also use directly one-sided tests for under or over dispersion, and I guess we also get confidence intervals under H0:alpha=0

example from test_discrete:TestGeneralizedPoisson_underdispersion
with under dispersion, GPP uses p=1

```
In [53]:
res3.params
Out[53]:
array([ 1.11363381, -0.53195835, -0.11228032])
In [60]:
res_p.model, res2.model
Out[60]:
(<statsmodels.discrete.discrete_model.Poisson at 0xcd26d30>,
 <statsmodels.discrete.discrete_model.GeneralizedPoisson at 0xcd32860>)

In [54]:
p0 = res3.params.copy()
p0[-1] = 0
res2.model.score(p0)
Out[54]:
array([ -7.11150894e-04,   9.01914996e-01,  -4.16418536e+01])

In [55]:
res2.model.hessian(p0)
Out[55]:
array([[-284.00071115, -389.098085  ,   41.64327594],
       [-389.098085  , -599.29283271,   68.71776778],
       [  41.64327594,   68.71776778, -222.68071367]])

In [58]:
res_p.model.score(p0[:-1])
Out[58]:
array([ -7.11150894e-04,   9.01914996e-01])

In [59]:
res_p.model.hessian(p0[:-1])
Out[59]:
array([[-284.00071115, -389.098085  ],
       [-389.098085  , -599.29283271]])
```
user: josef-pkt
issue_3925: state: open
title: SUMM: improve start_params and preliminary estimators
body: continuing discussion and investigation in #3578 and some parts of llnull PR #3921 
basic implementation for one component count models in #3928

currently NegativeBinomial, GeneralizedPoisson and NegativeBinomial use Poisson as preliminary estimator to get better start_params.

ZeroInflated models use a separate model method `_get_start_params` for a preliminary estimator
ZIP uses Poisson (but should use TruncatedPoisson when available), 
ZIGP uses ZIP
ZINBP also uses the main model NBP

The extra parameters compared to the simpler model are set to constant, e.g. 0.1 or zeros.

suggestions:

- add optimizer options for preliminary estimator, see 3578, preferably optim_kwds_prelim in kwargs
  - [x] implemented for model in discrete_model #3928
  - [ ] models in count_model. zero-inflated, and others ?
- start with llnull: I don't know how well it works (I might have checked some negbin cases)
  3921 added one step moment estimators for null params, the null MLE model is in the results class and not in the model class, so currently not available for model.fit
  - [x] implemented for Poisson in #3928, not for any other models
- use better preliminary start_params for extra parameters. The moment estimator in 3921 can be generalized to just use resid and mu and so could be used in any transitions e.g. from Poisson to GeneralizedPoisson or Negative Binomial. (I don't have an example or code for zero inflation start params yet, e.g. mixing probability could be roughly estimated from excess zeros, as in the zero-inflation/modification score or chisquare tests)
   - [x] implemented for NBP and GPP in #3928, not for any other models
   - [ ] zero-inflation needs also better start_params for inflation/mixing parameters

two objectives: get better defaults and provide more options to users.

 see also 
#3887 SUMM issue for count model follow-up
#3924 boundary, numerical problems that also affect optimization

some test cases are in various bug issues, and in current unit tests that show warnings, or that explicitly work around convergence problems.

another issue: `maxiter=35` is too small as a default, e.g. it is not changed if `method=bfgs`

user: josef-pkt
issue_3924: state: open
title: SUMM: clipping away from boundary in discrete
body: a general issue to get an overview of boundary problems in `discrete`

GLM/genmod has clipping in all or most places to keep away from boundaries 0 and 1. discrete models almost have no clipping with the exception of Probit, and in some cases the new count models.

The main problem is that if values are at the boundaries 0, 1 for binary and multinomial and 0 for count models, then some values result in infs or nans, and raise overflow or similar warnings.

Overflow in exp is a similar problem, but whether clipping would help to avoid exp(x) is inf is less clear.

see also my related summary for problems in open parameter spaces when we approach the limits https://groups.google.com/d/msg/pystatsmodels/r48gDjpgPBk/1BXmBfjBAQAJ

Two reasons why we need clipping:

- during optimization values are tried out that are too close to the boundary, or at the boundary if it is associated with exp overflow. This can happen even if the underlying model is fine with the data.
- in corner cases it is possible that an estimate approaches the boundary or is at the boundary, especially in combination with over- or underflow. E.g. in perfectly predicted cases the probability will be 1. If there are no events in count model, then the estimate for the mean could be zero.

related problems:

- variance estimate goes to zero
- extra shape parameter goes to zero but is numerically impossible, e.g. Negative Binomial dispersion parameter
- mixing probability (with Logit, Probit or similar) goes to zero or one.
- intentional violation, e.g. #2208 GLM without restrictive link (example log-binomial)

issues for basic discrete models:

#3337 negative binomial
#3635 MNLogit overflow and zeros
#3923 Logit overflow and zeros

#3863 alpha -> 0 in NegBin

We need to add numerical improvements and clipping into all models where this is a problem.
(i.e. compatify away from boundary)
`np.clip(temp, FLOAT_EPS, 1 - FLOAT_EPS)`
or use `np.nextafter(1, 0)` (used by Evgeny in count_model)






user: josef-pkt
issue_3923: state: open
title: BUG/ENH: overflow in Logit
body: https://stackoverflow.com/questions/46173061/statsmodels-throws-overflow-in-exp-and-divide-by-zero-in-log-warnings-and-ps

(I haven't looked at the provided data yet.)

AFAICS, we are not using the same or similar clipping in discrete binary models Logit/Probit as in GLM links or families
user: josef-pkt
issue_3920: state: open
title: FAQ-D/REF: setting attributes in wrapped results instance
body: this works:
`res_gp0._results._attach_nullmodel = True`

This does not work:
`res_gp0._attach_nullmodel = True`
because it attaches the attribute to the wrapper instead of the results instance itself.

I just puzzled for a while again why the second version did not have any effect on the computation.

Currently there is no public reason to set attributes in the results instance. It's only used for "cheating".

In the specific case, we need a better way to signal that we want to keep the null model.
user: josef-pkt
issue_3918: state: open
title: BUG: transparams and start_params not consistent
body: count models in discrete
It looks like start_params needs to be the transformed params if `_transparams=True` is used during the optimization

This happens in NegativeBinomial, and might also happen in the new count models NBP, GPP, ...

```
# use start_params, otherwise all nans
res_nb = NegativeBinomial(y, x, loglike_method='nb1').fit(start_params=[8, 0, 1], method='nm', maxiter=1000, 
                                    warn_convergence=False)
print(res_nb.summary())
sp = res_nb.params.copy()
sp[-1] = np.log(sp[-1])
res_nb = res_nb.model.fit(start_params=sp)
print(res_nb.summary())
```

The example has huge overdispersion `alpha=1700`, while the log(alpha) is then around 7.
The problem doesn't affect optimization much if alpha is around 1 or in another small range because the absolute mistake is not large. example is from mailing list link in #3476

user: josef-pkt
issue_3916: state: open
title: ENH: compare_score_test for variable addition
body: just another idea

if we have a full, unrestricted model available, then we can get score and cov from the unrestricted model.
The score_test under correctly specified likelihood is easy then to compute. That's simpler then constructing the score test from the restricted model using score_factor and hessian_factor, especially in the case with extra params as in GeneralizedPoisson, NegativeBinomial or zero-inflated models.
(We have one version in linear regression model and in GLM, but LEF has zero expected cross derivatives with variance and is therefore simpler, i.e. only mean parameters and only linear predictor, score_factor is univariate/scalar series.)

information needed is that we get the restricted parameter estimates. User can provide it, or we can default to filing with zeros for the simple case of variable addition, if we either use some properties of the data or the param_names to identify which parameters are zero under the restricted model.

example two-sample score test for count variables:
We have (in master or PR) standalone tests for binomial and poisson. However, if there is over-
 or under-dispersion, then the test does not have the correct distribution.
A simple case would be to estimate NegativeBinomial or GeneralizedPoisson and test for an indicator variable. Wald test is trivial, but score_test has often better small sample properties.

user: josef-pkt
issue_3915: state: open
title: ENH/Design GLM pattern, families, links, for non-LEF distributions
body: related similar issue #2107

The idea is to bring discrete and other future models closer to the GLM pattern of delegating to family and link classes, instead of hardcoding inside the models. This would be similar to what GAMLSS in R is doing.

There would be some additional function call overhead and generic code will in several cases be computationally less efficient than the special case models.
Right now I'm thinking more of the usage for post-estimation results where computational efficiency is not as important.

The current case are prediction, margins in the new count models and the required evaluation of distribution methods and computation of derivatives.
For example loglike is built into the model class and cannot use different exog or distribution parameters than those provided and hardcoded into the model. (Evgeny used an artificial model to manipulate the attached endog). In contrast families specify the distribution in terms of mu (mean) directly.

We also have the scipy distributions available and some similar classes in statsmodels.distributions, but they don't have the required methods like derivatives, and not the regression based parameterization in several cases, e.g. NegativeBinomial. 
In the gemod.families we have distributions that have the used parameterization and the derivatives and other methods that we needed so far. However, the families are currently only for one-parameter LEF.

Parametric duration models are another group of models on the wishlist that could follow a similar pattern. Accelerated failure time models AFT look very similar to discrete models with log-link.

Modified models, like zero-inflation, hurdle, truncated, censored, mixture models, need additional methods of the distribution that are not available in the "plain" discrete models. Delegation would make this more modular and reusable than adding special case methods like pmf or cdf to every model.



user: josef-pkt
issue_3913: state: open
title: nonlinear predict inference versus conditional moment test (cmt)
body: (just some notes, that I didn't realize before. I didn't check details of formulas yet, so I'm not sure how much the corresponding code will overlap.)

use case: predicted probabilities for count distributions versus conditional moment test

predicted probabilities for count distributions are a deterministic function of the parameters and exog. We can use delta method or endpoint transformation to get standard errors and confidence intervals.
(`predict_nonlinear` #2296 and similar to margins)

Testing whether predicted probabilities are compatible with observed probabilities uses a "residual", i.e. moment condition that expected difference between observed indicator function and predicted probability is zero. This is a random variable (endog is random) and not a deterministic function.
Test depends on two components, variance of extra moment condition and of their covariance to score-obs (and implied distribution of params) (using partitioned inverse of full moment conditions).
cm_test #2096, and chisquare tests #3904
Cameron Trivedi count book 2nd ed. p. 194 for predicted probabilities using cm-test (and OPG version with auxiliary regression)

Using derivative of moment condition or predict function overlaps between the two.



user: josef-pkt
issue_3910: state: open
title: statsmodels.formula.api.ols() Orthogonalization
body: Hello, I am running an ANOVA via the following code structure:

```
ols = statsmodels.formula.api.ols(model, data)
anova = statsmodels.api.stats.anova_lm(ols, typ=2)
```

 I noticed that depending on the order in which factors are listed in `model`, variance (and consequently the F-score) is distributed differently along the factors.

Supposing that my data looks like:

```
value , factor1 , factor2
    9 ,       A ,       L
    8 ,       A ,       L
    9 ,       A ,       M
    9 ,       A ,       M
    1 ,       B ,       N
    2 ,       B ,       N
    2 ,       B ,       O
    1 ,       B ,       O
```

What will happen to the shared variance between factor1 and factor2? Will it all be ascribed to the first factor listed in the expression? Does the model orthgonalize the second listed factor to the first, or do I have to do that manually? Would the above data warrant a multicollinearity warning?

user: TheChymera
issue_3905: state: open
title: FAQ-D/ENH/SUMM/TST: Monte Carlo verification and unit tests
body: (This is partially an old topic but I don't find an issue.)

- We need Monte Carlo studies to evaluate of verify small and large sample properties especially when we don't have reference numbers available from other packages.
- Parametric bootstrap uses mostly the same method but with a different interface and objective.
- A third similar usecase is power calculation based on Monte Carlo simulations.

It would be good to add more support for this

- make it easy to construct data generating processes, simulation models, maybe with some prefabricated ones (e.g. we had discussion for test cases for non-parametric function estimation)
  possibly allow a prespecified model or exog.
- code for running the Monte Carlo and collecting results, similar to bootstrap
- reporting: support summary tables, this might now be easier by just using pandas for the results, i.e. collect results into a data frame.


example use case: 
There are a large number of score tests for specification testing possible and available, but they are not available in other packages or not easy to find. In many cases I didn't find published studies to choose among a large number of similar diagnostic or specification test.
Related to this, we have several ways of generically constructing specification or hypothesis tests. Even when they are asymptotically equivalent, we don't have reference numbers or cannot compare them at high precision in smaller samples.
(I did something similar once as a basic generic check of scipy.stats' hypothesis test, construct "nice" cases under the null and simulate to check if size, rejection probability is correct.)
user: josef-pkt
issue_3904: state: open
title: ENH: chisquare diagnostic test for models
body: partition the (endog, exog) space and compute cell counts/proportions, use chisquare test as hypothesis tests, asymptotic covariance for chisquare statistic is not "trivial"

similar to Hosmer-Lemeshow test, or pearson chisquare test with MLE based on grouped data, but more general theory, i.e. applies to arbitrary types of endog and exog (continuous or discrete)

The JoE article looks good, I haven't read the Econometrica article

Andrews, Donald W. K. 1988a. “Chi-Square Diagnostic Tests for Econometric Models: Theory.” Econometrica 56 (6): 1419–53. doi:10.2307/1913105.

———. 1988b. “Chi-Square Diagnostic Tests for Econometric Models.” Journal of Econometrics 37 (1): 135–56. doi:10.1016/0304-4076(88)90079-6.

Manjón, M., and O. Martínez. 2014. “The Chi-Squared Goodness-of-Fit  Test for Count-Data Models.” Stata Journal 14 (4): 798–816.
(no access to this, but that's where I got the Andrews reference from)
https://www.stata.com/meeting/spain12/abstracts/materials/Majon_Martinez.pdf  slides about the Stata implementation
**update** article available at https://journals.sagepub.com/doi/pdf/10.1177/1536867X1401400406

One application: specification test for count models based on observed versus predicted frequencies, but Andrews JOE has many examples application for distributional tests.
The advantage is that it is very general and allow for any asymptotically normal sqrt(n) consistent estimators, and allows a wide range of methods for data dependent partitioning of (y, x).
(i.e. it would cover MLE, QMLE, M-estimators, GMM as long as we can get predictive counts from the model. Aside: There is also a draft version of GMM estimation of distributions for grouped data.)

#3887 follow-up, open issues when we have more count models to choose from, including diagnostics
#1288 binned gof test for Monte Carlo
#3897 Vuong test comparing non-nested models
https://github.com/statsmodels/statsmodels/issues/2041#issuecomment-249098368 Hosmer-Lemeshow related



user: josef-pkt
issue_3903: state: open
title: FAQ-D/REF: data handling for multiple exog, multi-part models
body: I have no idea what the best current approach is to handle a second exog.
AFAIK this has not been systematically added to model and might need refactoring.

The last case that Skipper added is for MixedLM
https://github.com/statsmodels/statsmodels/pull/2097/files#diff-65a46f8f38518560bd427a74998e9d24
and possibly relate "missing" handling by Kerby #3591

`MixedLM.from_formula` also has special, model specific code to handle the extra exog formula `re_formula`

cases: 

The problem at the moment is that zero-inflated PR #3755 does not have full support for this, and it's not obvious how to add it.
Other models like IV2SLS don't allow formulas for instruments, but there the instruments are not in the param_names.
Several other multipart models are in PRs or in planning.

user: josef-pkt
issue_3902: state: open
title: FAQ-D/SUMM: using instance methods and pickling
body: We have largely, but not completely avoided instance methods because of pickling and garbage collection.
For pickling, it seems now to be possible to pickle instance methods in python 3.5 and later (and maybe partially also in 3.4)

This means that we will be able to rethink our instance method policy as we drop older python versions. However, I don't have an overview over recent developments.

#839 garbage collection and reference cycles

older pickling issues
#2156 GEE
#603 nonparametric

#1729 usage of pickling for prediction

and the latest PR #3876 uses `copyreg` to allow pickling for multiprocessing
comments on pickling starting at https://github.com/statsmodels/statsmodels/pull/3876#issuecomment-323945982
"Both `PCA` and `KDEMultivariate` are blocking with pickle dump."

user: josef-pkt
issue_3901: state: open
title: ENH: OLS/WLS based on summary statistic, (cov_data)
body: (I don't find an issue, but the idea is old.)

This is similar to #3570 but for the specific case of adding a model class for linear regression based on summary statistics like the covariance matrix of the data. As variation of this the summary statistic can be a matrix decomposition of exog.

This can include multivariate OLS.

One usecase is to use this with non-standard covariance matrix, e.g. a robust #3230 or penalized #3197  cov_data estimate. 
For a sparse model it is not clear whether or when we need sparsified covariance or sparsified inverse covariance.

similar other multivariate methods can be based on "modified" covariance estimate.

user: josef-pkt
issue_3900: state: open
title: ENH: longitudinal two-part count models
body: (another one for the wishlist similar to #3899 )

(I only read a few abstracts)

examples could also be used to test cluster robust standard errors for zero-inflated and similar count models.

important link
several of "Statistical Modelling" journal articles have data in 
http://www.statmod.org/smij/archive.html

Farewell, V.t., D.l. Long, B.d.m. Tom, S. Yiu, and L. Su. 2017. “Two-Part and Related Regression Models for Longitudinal Data.” Annual Review of Statistics and Its Application 4 (1): 283–315. doi:10.1146/annurev-statistics-060116-054131.

Hall, Daniel B. 2000. “Zero-Inflated Poisson and Binomial Regression with Random Effects: A Case Study.” Biometrics 56 (4): 1030–39. doi:10.1111/j.0006-341X.2000.01030.x.

Hall, Daniel B, and Zhengang Zhang. 2004. “Marginal Models for Zero Inflated Clustered Data.” Statistical Modelling 4 (3): 161–80. doi:10.1191/1471082X04st076oa.

Min, Yongyi, and Alan Agresti. 2005. “Random Effect Models for Repeated Measures of Zero-Inflated Count Data.” Statistical Modelling 5 (1): 1–19. doi:10.1191/1471082X05st084oa.

Kong, Maiying, Sheng Xu, Steven M. Levy, and Somnath Datta. 2015. “GEE Type Inference for Clustered Zero-Inflated Negative Binomial Regression with Application to Dental Caries.” Computational Statistics & Data Analysis 85 (May): 54–66. doi:10.1016/j.csda.2014.11.014.


user: josef-pkt
issue_3899: state: open
title: ENH: bunching, heaping, stochastically rounded endog
body: The fair_pt dataset has counts bunched at 7 and 12
(see graph below which shows histogram and predicted probabilities for endog > 0, from my experimental notebook for zero inflation PR #3755)

another example are smoking studies where consumers often report multiples of 5, 10 or 20.

This is different from a fixed/deterministic rounding case because there we know the censoring limits and can use an interval censored model. (#664)

there are some article for "heaped count data" and related topics, but I didn't read anything.

Cummings, T. H., J. W. Hardin, A. C. McLain, J. R. Hussey, K. J. Bennett, and G. M. Wingood. 2015. “Modeling Heaped Count Data.” Stata Journal 15 (2): 457–79.
(gated pay article, I don't have access and no ungated copy in google search)

Bermúdez, Lluís, Dimitris Karlis, and Miguel Santolino. 2017. “A Finite Mixture of Multiple Discrete Distributions for Modelling Heaped Count Data.” Computational Statistics & Data Analysis 112 (August): 14–23. doi:10.1016/j.csda.2017.02.013.

Drechsler, Jörg, and Hans Kiesl. 2016. “Beat the Heap: An Imputation Strategy for Valid Inferences from Rounded Income Data.” Journal of Survey Statistics and Methodology 4 (1): 22–42. doi:10.1093/jssam/smv032.

Wang, Hao, and Daniel F. Heitjan. 2008. “Modeling Heaping in Self-Reported Cigarette Counts.” Statistics in Medicine 27 (19): 3789–3804. doi:10.1002/sim.3281.

Bar, Haim Y., and Dean R. Lillard. 2012. “Accounting for Heaping in Retrospectively Reported Event Data – a Mixture-Model Approach.” Statistics in Medicine 31 (27): 3347–65. doi:10.1002/sim.5419.

Heitjan, Daniel F., and Donald B. Rubin. 1991. “Ignorability and Coarse Data.” The Annals of Statistics 19 (4): 2244–53. doi:10.2307/2241929.

Heitjan, Daniel F., and Donald B. Rubin. 1990. “Inference from Coarse Data Via Multiple Imputation with Application to Age Heaping.” Journal of the American Statistical Association 85 (410): 304–14. doi:10.2307/2289765.

Schneeweiss, H., J. Komlos, and A. S. Ahmad. 2010. “Symmetric and Asymmetric Rounding: A Review and Some New Results.” AStA Advances in Statistical Analysis 94 (3): 247–71. doi:10.1007/s10182-010-0125-2.

Schneeweiss, H., and J. Komlos. 2009. “Probabilistic Rounding and Sheppard’s Correction.” Statistical Methodology 6 (6): 577–93. doi:10.1016/j.stamet.2009.06.005.





![zi_count_freq](https://user-images.githubusercontent.com/440735/29995556-041d8184-8fba-11e7-87c3-9e23d80bf6d7.png)

user: josef-pkt
issue_3897: state: open
title: ENH diagnostic/specification Vuong test for non-nested hypothesis
body: (I don't find a specific issue, nor experimental code in the sandbox. I thought there should be.)

for non-nested hypothesis the Vuong test is actually trivial
`ztest_paired(model1.loglikeobs(res1.params), model2.loglikeobs(res2.params), alternative=...)`
or
`ztest(model1.loglikeobs(res1.params) - model2.loglikeobs(res2.params), alternative=...)`

We need `loglikeobs` to compute the variance of the difference across observations, otherwise it is jut the log-likelihoodratio.

It gets more complicated if the models are overlapping, or parameters are on the boundary of the parameter space. 

application to Poisson versus ZeroInflatedPoisson, available in Stata zip and zinb, and in R pscl
(not clean theoretically, because they are nested in general, or unbound parameter space with Logit/Probit, see also #1357)

#1086 https://gist.github.com/jseabold/6617976

Cameron Trivedi count data 2nd edt. p. 198 secteion 5.4.2

Desmarais, B. A., and J. J. Harden. 2013. “Testing for Zero Inflation in Count Models: Bias Correction for the Vuong Test.” Stata Journal 13 (4): 810–35.

Wilson, Paul. 2015. “The Misuse of the Vuong Test for Non-Nested Models to Test for Zero-Inflation.” Economics Letters 127 (February): 51–53. doi:10.1016/j.econlet.2014.12.029.

One problem for zi models is that using Logit or Probit inflation model can have zero inflation only if linpred goes to inf (or -inf, depending on the definition of params). The edge case causes numerical and convergence problems. With clipping, the prob inflation is bound below > 0. So strictly speaking they are not nested, but the estimate in ZIModel will go to whatever lower bound we impose.)
We could have underinflation, but that is not implemented in the current ZI models. However, we will also get hurdle models which allow for more or fewer zeros than the main model, but it guess the model will be nested also in the implementation to the simple main model, Poisson, NegBin, ...

user: josef-pkt
issue_3895: state: open
title: BUG: NegativeBinomial and new count models override callback
body: related: #3706

The count models except for Poisson use a hardcoded `callback=lambda x: x` in the super(...).fit call.
This breaks the use of basinhopping, which has 3 arguments in the callback according to the exception traceback.

Either we drop this or we fix the callback to `callback=lambda *x, **kwds: x, kwds`
(only positional arguments are in basinhopping AFAICS, but we could as well be generic and add `**kwds`.

from https://github.com/statsmodels/statsmodels/pull/3755#issuecomment-325678812 :

I guess you copied the callback from NegativeBinomial.fit. Poisson fit doesn't override `callback`.
I have no idea why NegativeBinomial.fit overrides the user provided or None callback, except those things can easily slip through a code review.

The reason might be that DiscreteModel uses the callback (if None) to check for perfect separation which should only be for binary models right now. I don't see any other use of the callback.

user: josef-pkt
issue_3894: state: open
title: ENH: alias, shortcut method name for minimize options in fit
body: Just a user convenience to avoid having to look up and specify the full options.

We have now the generic "minimize" as a fit methods, but it requires two keywords. As convenience we could add an option that just converts the `method` into the full method keywords.
e.g. fit(method='dog-leg')
would be translated into fit(method='minimize'.... ?)

maybe not worth it if the documentation is easy to find.
user: josef-pkt
issue_3891: state: open
title: Naming of weights argument in WLS
body: Should we migrate the weights argument of WLS to be named freq_weights, like in GLM?  If so, what is the best path for doing that?

@jarvmiller 
user: kshedden
issue_3890: state: open
title: REF/ENH: verify valid model.__init__ keywords
body: e.g. #3889

using `**kwargs` is dangerous for usage because there is no error for wrong and misspelled keywords.

We already have `_init_keys` and we should be able to expand the usage for checking that only valid keywords are given by the user.
This would also enforce that `_init_keys` have the correct list, e.g. #3888 

We might have to distinguish different keyword types, e.g. `missing` will not be necessary if we recreate a model based on the attached data, endog, exog and similar.

user: josef-pkt
issue_3889: state: open
title: ENH: summary - add auxiliary init parameters
body: Some models like NegativeBinomial, NegativeBinomialP and GeneralizedPoisson have additional fixed parameters specified in `__init__`. The `summary` doesn't show those.

problem: models swallow extra keywords, and there is no indication that a different (default) model is estimated if a keyword is wrong or misspelled.  (Can we fix that without imposing too much extra effort for new models?)

```
>>> res_nb1 = NegativeBinomial(endog, exog, p=1).fit(method='newton')
>>> res_nb1.model.loglike_method
'nb2'
```

(I only realized that I used the wrong keyword, because the results are the same as
`res_nbp1 = NegativeBinomialP(endog, exog, p=2).fit(method='newton')`
)
user: josef-pkt
issue_3887: state: open
title: SUMM/ENH/BUG: count, GP-P, NB-P follow-up
body: there is a PR #3807 but no issue yet (AFAICS)

- [ ] docstring: I checked and fixed the class docstrings, but some method docstrings are still incomplete and I haven't checked them yet
  e.g.  NegativeBinomial.fit doesn't have full list of arguments in code, but then uses inherited generic `__doc__`  (needs templating)
- transparams: currently unused, untested, buggy in GPP and NBP, and turned off by default see #3918
- [x] init_kwds and llnull:
  - NB-P and GB-P don't have the `p` parameters in llnull AFAICS
  - two part models like ZIP need the second exog in creating the null model
- [x] llnull: more robust estimation, is already a separate issue -> fixed, enhanced in #3921
- margins
  - [x] one part models look fine, NB and NB-P is fixed and unit tested (for p=2)
  - truncated models: I guess we need extra computation for the mean and derivatives (i.e. it is not just exp(x b))
  - two part models like ZIP and hurdle: I guess this should raise NotImplementedError for now. 
     We have two exog, and several types of mean or conditional mean prediction. This will need larger enhancements.
     -> exception for ZI added in #3908, get_margeff() didn't raise for ZINB but I don't know what the result means.
- robustness to boundary cases, e.g. ZI-prob goes to zero. 
  maybe not much what we can do, but we could provide tools to identify problematic cases, one example score test based on simpler model
- start_params
   will need review and refactoring to see which models will provide good initial estimates (problem in ZIGPP and ZINBP)


enhancements

- goodness-of-fit measures, tests and graphics  
  -> partially added in #3908 as private functions
- generalize to allow more submodels in ZI, Hurdle
- score test for zero inflation and similar Jansakul and Hinde (2002) us identity/linear link for inflation model to allow that zero-inflation probability=0. (There are several other score test articles for various comparisons but I only skimmed time a few years ago.) 
 -> partially added in #3908 as private function
- Vuong test, see #3897
- prediction standard errors and confidence intervals, for which prediction and what kind of confidence intervals.  - see also #3915 problem with reusing loglikeobs, score_obs for predicted probabilties
- get_distribution: 
- add new predict options to old count models.

question:
Can we use a chisquare test for testing predicted versus empirical probabilities? E.g. corresponding to the histogram plot of aggregated counts or the corresponding cumulative probabilities.

References

Jansakul, N., and J. P. Hinde. 2002. “Score Tests for Zero-Inflated Poisson Models.” Computational Statistics & Data Analysis 40 (1): 75–96. doi:10.1016/S0167-9473(01)00104-9.

Jansakul, N., and John P. Hinde. 2008. “Score Tests for Extra-Zero Models in Zero-Inflated Negative Binomial Models.” Communications in Statistics - Simulation and Computation 38 (1): 92–108. doi:10.1080/03610910802421632.

Ridout, Martin, John Hinde, and Clarice G. B. DeméAtrio. 2001. “A Score Test for Testing a Zero-Inflated Poisson Regression Model Against Zero-Inflated Negative Binomial Alternatives.” Biometrics 57 (1): 219–23. doi:10.1111/j.0006-341X.2001.00219.x.


user: josef-pkt
issue_3884: state: open
title: REF: deprecate ARMA/ARIMA in favor of SARIMAX ?
body: related: 
roadmap for tsa #3115
old issue to compare of ARIMA and SARIMAX #2542

The long term plan is to deprecate ARMA and ARIMA to avoid code duplication and because those models are barely maintained. We will not be able to remove them fast after deprecation, but we need to start directing users to SARIMAX. There might be or is some functionality that does not have a close substitute yet, and we need to get a better overview of the different usecases.

Can we deprecate ARMA/ARIMA in favor of SARIMAX? What's missing?

- model.predict: we still have little out of the box support to predict on new datasets without reestimation. ARMA has predict in the model class, SARIMAX doesn't https://github.com/statsmodels/statsmodels/issues/3882#issuecomment-324451614
- ...
user: josef-pkt
issue_3863: state: open
title: ENH/BUG/FAQ: Negative binomial numerical precision at alpha=0
body: There is a problem with score and hessian if alpha in NegativeBinomial is close to zero.
example in #3811 using cpunish dataset.

AFAICS, alpha = 0 only produces nans in score and hessian. The problem is that computations are done in terms of 1 / alpha and alpha**(-2) and alpha**(-3)

However, some terms like those with digamma and log will be difficult to check for canceling terms.

It looks to me right now that NegativeBinomial does not work correctly or robustly in the neighborhood of alpha=0, i.e. close to Poisson.


user: josef-pkt
issue_3858: state: open
title: MICE doesn't support categorical data
body: Running MICE imputation on a dataframe with categorical data throws an error:
(DataFrame info is somewhat abridged)
```
In [27]: data.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index
Data columns (total 23 columns):
time                       int64
status                     category
age                        float64
event_age                  float64
sex                        category
...
dtypes: category(2), float64(12), int64(9)

In [28]: imp = mice.MICEData(data[:1000], perturbation_method='boot')
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-28-9f7c97b6ca1c> in <module>()
----> 1 imp = mice.MICEData(data[:1000], perturbation_method='boot')
 
/home/ohadl/anaconda3/lib/python3.6/site-packages/statsmodels/imputation/mice.py in __init__(self, data, perturbation_method, k_pmm, history_callback)
    254         self._cycle_order = [vnames[i] for i in ii]
    255
--> 256         self._initial_imputation()
    257
    258         self.k_pmm = k_pmm
 
/home/ohadl/anaconda3/lib/python3.6/site-packages/statsmodels/imputation/mice.py in _initial_imputation(self)
    291
    292         for col in self.data.columns:
--> 293             di = self.data[col] - self.data[col].mean()
    294             di = np.abs(di)
    295             ix = di.idxmin()
 
/home/ohadl/anaconda3/lib/python3.6/site-packages/pandas/core/generic.py in stat_func(self, axis, skipna, level, numeric_only, **kwargs)
   6195                                       skipna=skipna)
   6196         return self._reduce(f, name, axis=axis, skipna=skipna,
-> 6197                             numeric_only=numeric_only)
   6198
   6199     return set_function_name(stat_func, name, cls)
 
/home/ohadl/anaconda3/lib/python3.6/site-packages/pandas/core/series.py in _reduce(self, op, name, axis, skipna, numeric_only, filter_type, **kwds)
   2383         return delegate._reduce(op=op, name=name, axis=axis, skipna=skipna,
   2384                                 numeric_only=numeric_only,
-> 2385                                 filter_type=filter_type, **kwds)
   2386
   2387     def _reindex_indexer(self, new_index, indexer, copy):
 
/home/ohadl/anaconda3/lib/python3.6/site-packages/pandas/core/categorical.py in _reduce(self, op, name, axis, skipna, numeric_only, filter_type, **kwds)
   1823         if func is None:
   1824             raise TypeError("Categorical cannot perform the operation "
-> 1825                             "{op}".format(op=name))
   1826         return func(numeric_only=numeric_only, **kwds)
   1827
 
TypeError: Categorical cannot perform the operation mean
```
user: ohadle
issue_3857: state: open
title: ENH: standardized coefficients - params table
body: Add beta coefficients (coefficients for standardized data) as a post-estimation results method.

API: return similar to t_test with params table, summary, summary_frame

Most likely this can internally reuse t_test, we just need the linear transformation matrix to standardize.
Alternative would be similar to transformation in models, e.g. fit_constrained and returning a results instance.

Applies to linear model, but I never looked at the interpretation in GLM, discrete. So, I'm not sure whether it would be helpful to add in base instead of regression.

Issue #1131 discusses transformation to improve estimation but here it is just a post-estimation diagnostic.

I wrote this function already a long time ago, but never committed it anywhere. If the design in not orthogonal, then the beta coefficients are not a variance decomposition. I read some papers for importance statistics in that case, but those didn't look very popular.

related topic: SAS has a large number of extra statistics and diagnostics for the linear model.
user: josef-pkt
issue_3851: state: open
title: FAQ: How to do analysis of covariance with statsmodels
user: forwho
issue_3842: state: open
title: Correlation
user: Sarmentor
issue_3840: state: open
title: FAQ-D/REF/ENH: change remove_data to allow for common attribute names in results and model
body: It looks like the current base.model remove_data method always tries to remove model attributes also from the results instance.  AFAIR, this was initially written when we used references in the results instance for attributes in the model instance as a user convenience.

This seems to break, i.e. cause a warning, when the results instance has a cached attribute with the same name as a (regular) model attribute. `mu` in GLM has this pattern

see comments starting here https://github.com/statsmodels/statsmodels/issues/3563#issuecomment-318148289

None of the things that I tried with manipulating the lists of things to remove change anything in the warning. Making `GLMResult.mu` into a regular `property` or renaming it to mu2 removes the CacheWriteWarning.

- Either we refactor remove_data to allow the case where the results instance has a cached attribute with the same name as a model attribute.
- or, we disallow this case (i.e. separate what are model and what are results attribute, GLM should not attach mu to the model in fit_irls.)


user: josef-pkt
issue_3836: state: open
title: SUMM/ENH: functional data
body: We need to start collecting design ideas and references for functional data.

This is mainly for the case when the response variable endog is a function, but might include cases where one of the exog is a function.

example time series analysis if a seasonality is considered as a function, e.g. a large number of observations in one seasonal cycle
https://stackoverflow.com/questions/45284122/numeric-value-for-time-series-data-with-frequency-every-2-minutes
I recently read a nice paper with application to huricane severity data by Haerdle and co-authors, IIRC

other application that IIRC showed up (was asked for) in the past was yield curve time series

There are also several text books on it.

main tool: representing the functional variable as a spline

user: josef-pkt
issue_3835: state: open
title: Migration from 0.6.1 to 0.8.0 issues
body: It appears that ```fit_regularized``` return value is more different from `fit` return value in newer versions. The following example runs normal in 0.6.1, but fails in 0.8.0:

```
import numpy as np
import statsmodels.api as sm


Y = [1, 3, 4, 5, 2, 3, 4]
X = range(1, 8)
X = sm.add_constant(X)

results = sm.OLS(Y, X).fit()
print(results.resid)

results_2 = sm.OLS(Y, X).fit_regularized(cnvrg_tol=0.0001, alpha=1., L1_wt=0)
print(results_2.resid)
```

Error for the second print statement:
`AttributeError: 'RegularizedResults' object has no attribute 'resid'`

How can I achieve the similar functionality with `fit_regularized `as in statsmodels 0.6.1? Any help would be greatly appreciated.
user: K-Mike
issue_3829: state: open
title: some errors when using ARIMA to make prediction
body: Hi all,
      I am getting some Errors when forecasting on an ARIMA. I don't know where is the problem.Here is my code:
`#-*- coding: utf-8 -*-

#from __future__ import print_function
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import statsmodels.api as sm
import xlrd


#读取Excel数据
#discfile = 'data_test.xls'
discfile = 'forecast-python.xls'
bk = xlrd.open_workbook(discfile)
sh = bk.sheet_by_name("Sheet1")
for iCol in range(0, 1):
	data = sh.col_values(iCol)
	#data = zip(*data)
	GridID = int(data[0])
	testData = data[1:433]
	#testData = data[1:433]
	#data = pd.read_excel(discfile,index_col = 0,parse_cols = 0)

	#testData = np.array(testData,dtype=np.float)

	testData = pd.Series(testData)

	testData.index = pd.Index(sm.tsa.datetools.dates_from_range('2001','2432'))
	#testData.index = pd.Index(sm.tsa.datetools.dates_from_range('2001','2048'))


	#####输出文件路径
	PredictFilePath = "predict_value.txt"
	PFP = open(PredictFilePath,'w')


	# data=data['number']
	# data.head()

	testData.plot(figsize = (12,8))
	print(testData)

	#使用一阶差分
	diff_1 = testData.diff(1)
	diff1 = diff_1.dropna()

	###确认最小值
	arma_mod00 = sm.tsa.ARMA(diff1,(1,1)).fit()
	minAIC = arma_mod00.aic

	######计算p,q值
	for i in range(1, 5):
		for j in range(1, 5):
			try:
				arma_mod61 = sm.tsa.ARMA(diff1,(i,j)).fit()
				aic = arma_mod61.aic
				print(i,j)
				if aic < minAIC:
					p = i
					q = j
					minAIC = aic
			except Exception, e:
				print (0)
			#print(arma_mod61.aic,arma_mod61.bic,arma_mod61.hqic)

	####arma模型值
	arma_mod61 = sm.tsa.ARMA(diff1, (p, q)).fit()

	#用模型预测
	#predict_data = arma_mod61.predict('2048','2096',dynamic=False)
	predict_data = arma_mod61.predict('2432','2480',dynamic=False)
	print(predict_data)

	####还原预测值
	diff1_shift = predict_data.shift(48)
	print(diff1_shift)
	diff_recover_1 = predict_data.add(diff1_shift)
	diff_recover_1 = diff_recover_1.dropna() # 最终还原的预测值
	print('预测值')
	print(diff_recover_1)

	for i in range(0, 48):
		print >> PFP, int(diff_recover_1.values[i]),
	print >> PFP

PFP.close()
`
here is the sample data:(27230 is  the ID)
 
27370
--
1678
1678
1651
1636
1656
1663
1663
1700
1697
1730
1716
1826
2115
2963
4415
5223
6532
6296
5804
5275
4852
4602
4488
4395
4287
4414
4490
4365
4190
4279
4413
4201
4152
4293
4479
4589
4714
4343
4197
4196
3901
3615
3680
3394
3341
3181
3167
3070
1703
1648
1609
1568
1612
1633
1662
1661
1660
2008
2054
1902
2069
2736
3824
5163
5649
6064
5696
5605
5235
5128
4609
4341
4340
4467
4341
4407
4269
4191
4062
4014
4377
4546
4518
4480
4691
4516
4060
3765
3736
3545
3494
3380
3237
3051
2989
2932
1659
1646
1622
1585
1600
1615
1611
1655
1667
2027
1731
1948
2098
3006
4596
5563
6301
6083
5775
5620
5262
5190
4720
4769
5051
4970
5168
5309
5365
5137
5105
5053
5190
5345
5353
5455
5144
4777
4464
4686
4340
3993
3830
3695
3636
3437
3382
3312
1693
1698
1727
1697
1710
1724
1711
1739
1706
1711
1770
2001
2198
2874
4217
5212
6028
7042
6190
5972
5666
5473
5033
4780
4821
4918
4862
4880
4872
4849
4784
4721
4833
4892
4822
5126
4921
4705
4259
4070
3961
3830
3677
3685
3353
3236
3184
3142
1714
1754
1699
1653
1683
1693
1711
1796
1748
1714
1760
1984
2257
3299
3907
5090
5980
5932
5439
5371
5174
5115
4837
4736
4713
4885
4975
5154
5066
4941
4868
4768
4922
5276
5728
5689
5094
4577
5044
4270
3967
3787
3472
3592
3342
3202
3420
3040
1732
1717
1745
1718
1725
1747
1782
1808
1756
1740
1760
1911
2189
2984
4136
5242
5665
5669
5510
5156
5168
4890
4603
4373
4334
4509
4568
4614
4291
4412
4400
4338
4598
4655
4901
4844
5083
4606
4452
3898
3645
3489
3399
3299
3189
3080
3055
2993
1636
1656
1650
1622
1641
1729
1774
1752
1753
1725
1743
1913
2131
3122
4160
5540
6592
5999
5709
5452
4989
4971
4715
4680
4529
4702
5072
4928
4939
4592
4525
4547
4498
4812
5697
5594
5763
5467
5597
4743
4563
4392
4269
4141
3978
3663
3540
3541
1828
1916
1865
1868
1881
1849
1851
1861
1884
1861
1969
2054
2397
3437
5450
6933
8015
7565
7221
6733
6207
6046
5486
4981
4994
5048
5140
5057
4853
4958
5230
5473
5572
5677
6201
6116
6009
5983
5544
5241
4846
4647
4373
4284
4242
3941
4148
3727
1809
1832
1875
1873
1880
1895
1879
1911
1933
2005
1993
2110
2355
3256
5002
6405
7689
7492
6903
6195
6001
5961
5698
5381
5154
5461
5736
5634
5521
5692
5362
5417
5780
5765
5944
6153
5870
5912
5879
5123
4774
4657
4454
4342
4217
3879
3771
3605
1853
1852
1834
1841
1843
1877
1887
1874
2152
1931
2168
2201
2451
3724
5105
6492
8257
7434
7015
6565
6483
6378
5740
5448
5631
5741
5961
6244
6450
6105
5949
5846
6083
6155
6503
6277
6254
6357
5868
5056
4792
4559
4423
4292
4095
3839
3751
3644



And here is the error:
`Traceback (most recent call last):
  File "C:\Program Files (x86)\JetBrains\PyCharm 2017.1\helpers\pydev\pydevd.py", line 1591, in <module>
    globals = debugger.run(setup['file'], None, None, is_module)
  File "C:\Program Files (x86)\JetBrains\PyCharm 2017.1\helpers\pydev\pydevd.py", line 1018, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "D:/  ʿ о / ҵ  о /arima/arima.py", line 48, in <module>
    arma_mod00 = sm.tsa.ARMA(testData,(1,1)).fit()
  File "C:\Python27\lib\site-packages\statsmodels\tsa\arima_model.py", line 452, in __init__
    super(ARMA, self).__init__(endog, exog, dates, freq, missing=missing)
  File "C:\Python27\lib\site-packages\statsmodels\tsa\base\tsa_model.py", line 44, in __init__
    self._init_dates(dates, freq)
  File "C:\Python27\lib\site-packages\statsmodels\tsa\base\tsa_model.py", line 64, in _init_dates
    raise ValueError("Frequency inference failed. Use `freq` "
ValueError: Frequency inference failed. Use `freq` keyword.
`




user: hehehechenlina
issue_3820: state: open
title: TST generic test mixin for general use
body: Instead of adding models to the tests in base, we could reuse the check mixin class in the subfolder tests.
user: josef-pkt
issue_3816: state: open
title: ENH:  summary as_text option for separator lines
body: The VECM summary creates 10 tables. When they are combined to string, then the tables look crowded because there is no separator line between tables.
user: josef-pkt
issue_3812: state: open
title: TST: more unit tests for margins, get_margeff
body: Given that only Logit has margeff unit tests, we need better test coverage.

an idea is to add generic tests just for the effects itself (not standard errors), using basic equivalences or specific formulas.

e.g. for log-link, exponential mean function in count models
`(np.exp(res1.params[:-1].dot(res1.model.exog.T)).mean(0)*res1.params[:-1])[1:] - margin_gp1.margeff` 
is around 1e-14 in the generalized poisson example
(slicing truncates constant and extra GP dispersion parameter)

margin_gp1c_elc = res1c.get_margeff(method='eydx', dummy=True, count=True)
margin_gp1c_elc.margeff - res1c.params[1:-1].values
is at about atol=2e-14

another option is to use link functions
user: josef-pkt
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_3810: state: open
title: SUMM/REF: Review discrete_model
body: some issues while looking at discrete models, specifically countmodels for extension in GSOC 2017

- k_extra, extra parameters in margins, #3809 -> fixed for NB, GP in #3811, still open two-exog models
- uses endog_names instead of param_names
- fit calls result._get_robustcov_results, can we use the `results.__init__` code for this?
- ...
- TST: discrete margins has unit tests against Stata only for Logit, no other model has margeff unit tests
- ...
user: josef-pkt
issue_3808: state: open
title: TST: VAR unused test results
body: there is a Stata results file that has zero coverage
and the class CheckVAR has no line coverage in asserts and is also is not used

https://codecov.io/gh/statsmodels/statsmodels/src/e9fdba96c677426d031e84849fa5b53b23b144a5/statsmodels/tsa/vector_ar/tests/test_var.py
(found browsing the coverage results in #3804 that also shows modules that are not run in the test run, i.e. zero coverage files.)
user: josef-pkt
issue_3803: state: open
title: ENH: add predict probability to negative binomial
body: Poisson has a separate method
GeneralizedPoisson and other coming count models have prob as predict option.

there is nothing yet for negative binomial

this should be more directly available as result than going through get_distribution
see #106 for parameter conversion for negative binomial
user: josef-pkt
issue_3797: state: open
title: `i` index overwritten in _arma_predict_out_of_sample
body: https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/arima_model.py#L338

```
    if q:
        i = 0  # if q == 1
    else:
        i = -1

    for i in range(min(q, steps - 1)):
```

`i` is set and then promptly ignored.  Any idea what it is supposed to do?  If it helps, `q` here is an alias for `k_ma`.

user: jbrockmendel
issue_3793: state: open
title: ENH: seasonal unit root tests
body: just to park a link, question with references

https://stats.stackexchange.com/questions/287548/what-is-the-intuition-for-testing-seasonal-difference-with-ocsb-test-and-its-cor

we might need this for an automatic forecasting project.
Hyndman's auto.arima uses OCSB (after switching from using ???)

some references (downloaded but not read

HEGY and OCSB seem to be the main candidates, and maybe DHF
It looks like ADF has power for seasonal unit root (if I skimmed OCSB correctly), but cannot discriminate between one period unit root and seasonal unit root.)

Demetrescu, Matei, and Uwe Hassler. 2007. “Effect of Neglected Deterministic Seasonality on Unit Root Tests.” Statistical Papers 48 (3): 385–402. https://doi.org/10.1007/s00362-006-0343-6.

Depalo, D. 2009. “A Seasonal Unit-Root Test with Stata.” Stata Journal 9 (3): 422–38.

Dickey, D. A., D. P. Hasza, and W. A. Fuller. 1984. “Testing for Unit Roots in Seasonal Time Series.” Journal of the American Statistical Association 79 (386): 355–67. https://doi.org/10.2307/2288276.

Hassler, Uwe, and Paulo M. M. Rodrigues. 2004. “Seasonal Unit Root Tests Under Structural Breaks*.” Journal of Time Series Analysis 25 (1): 33–53. https://doi.org/10.1111/j.1467-9892.2004.00336.x.

Hylleberg, S., R. F. Engle, C. W. J. Granger, and B. S. Yoo. 1990. “Seasonal Integration and Cointegration.” Journal of Econometrics 44 (1): 215–38. https://doi.org/10.1016/0304-4076(90)90080-D.

Osborn, Denise R., A. P. L. Chui, Jeremy P. Smith, and C. R. Birchenhall. 1988. “Seasonality and the Order of Integration for Consumption.” Oxford Bulletin of Economics and Statistics 50 (4): 361–77. https://doi.org/10.1111/j.1468-0084.1988.mp50004002.x.


there are some articles to get p-values or critical values from response surface modelling (not read either)
Diaz-Emparanza (2014) has add-on/function for Gretl

Barrio Castro, Tomás del, Andrii Bodnar, and Andreu Sansó. 2017. “Numerical Distribution Functions for Seasonal Unit Root Tests with OLS and GLS Detrending.” Computational Statistics 32 (4): 1533–68. https://doi.org/10.1007/s00180-016-0688-9.

Diaz-Emparanza, Ignacio. 2014. “Numerical Distribution Functions for Seasonal Unit Root Tests.” Computational Statistics & Data Analysis, CFEnetwork: The Annals of Computational and Financial Econometrics, 76 (August): 237–47. https://doi.org/10.1016/j.csda.2013.03.006.

(general issue base on superficial skimming seems to be whether to remove a deterministic seasonal component before running seasonal unit test, 
i.e. seasonal unit root in addition to instead of as substitute for a deterministic seasonal component.)

user: josef-pkt
issue_3790: state: open
title: KDEMultivariate for unordered variables output depends on the numerical value of the unordered variable
body: Version: '0.8.0'

When using the default bandwidth selection with un-ordered variables in KDEMultivariate, the result depends on the numeric values of the training variables. 

Compare:
```
import statsmodels.api as sm
x = np.array([0,0,0,0,1])
kde_x = sm.nonparametric.KDEMultivariate(data=x, var_type="u")
print kde_x.pdf([0])
# output: 0.61561605356
```

With 
```
import statsmodels.api as sm
x = np.array([0,0,0,0,1]) * 1000
kde_x = sm.nonparametric.KDEMultivariate(data=x, var_type="u")
print kde_x.pdf([0])
# output: -183.58394644
```

This happens because the bandwidth is estimated as if the values were continuous. I suggest that this behavior is changed, or at the very least documented.

One suggestion for default might be to set the bandwidth to 1. This way the probability estimate overlaps with the MLE.

I'm happy to submit a pull request for this if people are ok with the default value.

user: ijmbarr
issue_3787: state: open
title: Attribute Existence Ambiguity
body: Moving part of the conversation from #3570 

>> moving away from `self.__dict__.update(...)` ?

> Why? What would be the problem? What would be the alternative?
I don't have a strong opinion but it worked well and don't remember ever having problems with it. We use it a lot. So any moving away would depend on a better alternative.
`self.__dict__.update(mlefit.__dict__)` looks like a simple way to copy over attributes.

>> and/or `setattr`

> no idea, I didn't know or remember that we use it. So I don't have currently an opinion.
We use getattr with a default if the attr doesn't exist pretty often.




I count 15 appearances of `__dict__.update` and ~50 `setattr`, excluding results and tests modules.

> What would be the problem?

The problem is that it is difficult to reason about what attributes exist where.

Take for instance `ModelData`.  Inside `__init__` it runs either `self.__dict__.update(arrays)` or `self.__dict__.update(kwargs)` to attach essentially everything (aside from `exog` and `endog` I'm not even sure what else is in here or why).  Then in `Model._handle_data` many of those kwargs then get popped off of `data.__dict__` to get attached to model instead.  The problem is that reasoning about what attributes exist where is difficult and shouldn't be.

`self.__dict__.update(mlefit.__dict__)` in GenericLikelihoodModelResults I'll live with as it is specifically intended to be user-obscure magic.  `DiscreteResults` I'd like to change to call `super(DiscreteResults, self).__init__` and avoid the `self.__dict__.update`, but that is a much lower priority than the ModelData/Model case.

The alternative I would advocate:

ModelData - Explicitly specify the attributes that are important (`exog`, `endog`, ??), and any other array-like objects or kwargs go into dictionaries like `data._arrays` and `data._init_kwargs`.  Then we know exactly what attributes exist, and exactly where to find any other auxiliary objects.

Model - Explicitly specify the parts of ModelData that should be copied over.  Anything else either goes into an auxiliary dictionary or is accessed via `model.data.foo`.

> We use getattr with a default if the attr doesn't exist pretty often.

If there are common attributes like this, let's initialize them to `None` from the get-go (in `__new__` if we have no other choice) and avoid any ambiguity as to what exists.



user: jbrockmendel
issue_3775: state: open
title: BUG/ENH VECM follow-up
body: An issue for open issues after merging VECM, to look at or decide before a release
extensions are separate issues #3135 #3133 

- ???
- check api.py and doc additions
- why does HypothesisTestResults and subclasses implement `__eq__` (LGTM warning)
- check behavior for not supported cases in coint test, e.g. LGTM Warning "Local variable 'jc' may be used before it is initialized." in coint_table.py
- standard pattern in plots (add `ax`), VECM follows VAR in some parts which does not have the "modern" statsmodels pattern and conventions in some parts
- unit tests are `yield` based, which might require adjustments when we switch to pytest and they deprecate/remove yield support.
- backwards compatibility? new methods `test_whiteness_new`, deprecate replace old methods?

other LGTM warnings seem mostly irrelevant (e.g. we circular import warnings with api.py which I think don't apply.)
https://lgtm.com/projects/g/statsmodels/statsmodels/rev/pr-8201a3203c0ae4914ad5563d67ab9b2fe56ddd2c
https://coveralls.io/builds/12153518/source?filename=%2Fhome%2Ftravis%2Fbuild%2Fstatsmodels%2Fstatsmodels%2Fstatsmodels%2Ftsa%2Fvector_ar%2Fvecm.py


**edit/update** still open (partial check)

- summary top table missing
- missing or incomplete docstrings, e.g. what is `cov_params_default`? class docstring has non-existing `cov_params` instead
- related to `cov_params_default`, the corresponding bse is `stderr_params`. There is no `params` attribute itself. What should be used for Wald tests? i.e. separate or joint components of all parameters. There is no joint params array in the fit method, and no joint Hessian. 
- ...
user: josef-pkt
issue_3774: state: open
title: ENH: QuantReg, missing start_params, convergence criteria
body: QuantReg doesn't allow setting a start_params, and convergence criteria are hard coded to max abs parameter difference.

There is already some commented out code for handling start_params, but it requires rewriting part of the IRLS loop

for convergence criteria see e.g. #2825 and older #931

triggered by #3370 which looks more likely caused by the convergence problems in QuantReg, than smaller improvements possible in the fit implementation.

aside: QuantReg computes the matrix_rank in fit.
user: josef-pkt
issue_3772: state: open
title: PERF/ENH: add _fit_cache for unchanging components in loglike
body: possible design: add a cache for components of loglike, score and hessian that don't change during estimation.

The previous idea was to drop terms like some other packages like SPSS. However, I think now it is easier to implement a cache. @jbrockmendel raised that idea in a issue or PR.

Currently we are doing something a bit similar in the linear models where we cache pinv or similar. However, there the use case was storing an intermediate result that can be used for a new endog.

Adding an explicit `_fit_cache` attribute would keep those intermediate results in a central place so they are easier to manage and delete if necessary.

Another partially related design is reparameterization during fit only, which we use in several models, e.g. NegativeBinomial and some tsa models to keep parameters in the valid range. There we use a flag that is set and reset in the fit method.

(Note we also have the `obs` methods for score and gradient and loglike, which we might not change if there are observation specific parts to be cached.)

user: josef-pkt
issue_3770: state: open
title: QuantReg runs 30x slower than quantile regression in R
body: Hi Guys, I face a challenge as following:

Response variables from a pandas DataFrame: **endog**, 5000 * 90
Explanatory variable: **exog**, a pandas Series, length = 90
---- session info ---
OS: macos
RAM: 16G
CPU: core i7
---------=-----------
`from statsmodels.tools import tools`
`from statsmodels.regression.quantile_regression import QuantReg`
`for i in range(5000):`
    `mod = QuantReg(endog.iloc[i].values, tools.add_constant(exog.values))`
    `res = mod.fit(q=0.5).params[1]`

Execution time is ~106s
------------=-----------
However, using quantile regression (rq function) in R, only spent
`user  system elapsed `
`6.249   0.374   3.770 `
--------=-----------
Is there any method to reduce the execution time of QuantReg in Python to be comparable to R?
user: solo7773
issue_3763: state: open
title: Compositional Data Analysis modules
body: This addresses #3560 and #3537 

Things that need to be touched up in this PR

- [ ] Docs
- [ ] Changelog

@josef-pkt not sure exactly where this module should belong - do you have any better recommendations?
user: mortonjt
issue_3762: state: open
title: ENH Survey methods GSOC
body: Below are the features added via this PR:

* Summary statistics for survey data
    * mean, total, ratio, and quantile
    * Each statistic’s standard error can be derived from the jackknife, bootstrap, or linearized/robust method. Methodology from STATA documentation is used
    * For the bootstrap and jackknife, you can promote privacy by retrieving the replicate weights as a numpy array. This way, you can attach it to the data, delete stratum and PSU info, and send it to others to analyze.

* Models for survey data
    * Uses StatsModels to create models that take into account info from a survey such as strata, cluster, weights, etc
    * Coefficients get linearized standard error using methodology specified in STATA and SAS, user is allowed to choose based on their preference for the assumptions and corrections made
    * For Jackknife standard errors, STATA methodology is used
* Contingency table for survey data
    * Essentially calculates a total-like statistic for your data grouped by one or two provided variables.
    * Tests of independence for complex survey data are calculated using methodology from STATA documentation

All results are tested against STATA currently. This isn’t just adding everything from STATA to Python; for example, STATA doesn’t calculate bootstrap weights, you can only provide them. It also doesn’t provide standard errors for quantiles. In the future, I am sure that features not supported by STATA will continue to be added (maybe from SAS or R).

Future work:
* Adding the right corrections for calculating SEs with replicate weights provided
* Add more tests of independence for SurveyTable
* Continue implementing sandwich methods such as 'jack-sandwich' for SurveyModel
* Create 'results' class to make output readable


Bugs:
* LRT and Pearson test statistic are off. This could be for one of two reason
    * _group_variances() gives the correct values, but they might be in the incorrect order. Depends on how STATA and Python decide which group indices to analyze first. For example, the variance in the (0,0) position of the matrix given in _group_variance() is the 0,1 group in the cell_prop matrix
    * An accumulation of rounding differences. Best would be to just print out the results in my test file
* jackknife summary statistics don't match w/ rep_weights provided. This is probably due to a misinterpretation of corrections to apply because `jdata` matches when jack rep weights are given and when they are calculated. The only thing done after `jdata` is build is multiplying by nh, fpc, etc
* SAS version of SE not matching. Least confident about where the bug could be here
user: jarvmiller
issue_3760: state: open
title: FAQ: scale: variance vs. standard deviation
body: In [scipy](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html) the parameter scale always denotes the standard deviation of a distribution.

In [statsmodels](http://www.statsmodels.org/stable/generated/statsmodels.regression.linear_model.RegressionResults.html?highlight=regressionresult) it seems to always denotes the variance. (Is this correct?)

Is there a reason that statsmodels diverts from the convention used in scipy for the parameter scale. Also, the latter is the convention in probability theory for [location-scale](https://en.wikipedia.org/wiki/Location%E2%80%93scale_family) families. I bet many users will implicitly assume that scale refers to the standard deviation of the error distribution and not its variance. (The standard Gaussian is, well, *scaled* by the parameter σ). Its at least what I did till I realised that the numbers just don't add up.

Just wanted to know if there is a motivation, a "change in perspective" that I missed (or, say, if it just happend to be so when the package was developed.)

Thanks.
user: 00tau
issue_3759: state: open
title: ENH: predict with weights, freq_weights and p_weights
body: related: 
margins with weights #2696
weighted sum/total #2851 

how do we predict a weighted average over the population?  similar to POM (potential outcome mean)

triggered by a WLS survey weights question
https://groups.google.com/d/msg/pystatsmodels/ZPVKynMo3ko/iUJcr9GdAwAJ

WLS does not have weights in predict. However, I think because of the linear structure we can just use a weighted average of the exog. But that will not work for nonlinear models, discrete and GLM.


user: josef-pkt
issue_3757: state: open
title: Survey methods gsoc 2017
user: jarvmiller
issue_3751: state: open
title: BUG ? ENH :  fit kewywords, constraints in llnull
body: see https://github.com/statsmodels/statsmodels/pull/3727#issuecomment-307472070

llnull is using `model._get_init_kwds()` for recreating the model, but does not use any fit_kwds and uses a plain fit.

possible problems

- fit_xxx: fit_regularized and fit_constraint are not used, This is not incorrect, if the constant is unconstrained and we only have constraints on the slope parameters
- fit constraints on extra parameters will not be taken into account, We don't have them yet, but GP-P in #3727 alllows to turn on the no-negativity constraints via exp transform. This forces and almost corner solution if the unconstrained/untransformed estimate would be negative (under dispersion GP)
- similar but less obvious: penalization on extra parameters and whether those should also be imposed on the Null constant-only model. (Ther is no case yet, but would be allowed by generic penalization mixin)


This is besides problems with convergence in the estimation of the null model.

user: josef-pkt
issue_3749: state: open
title: ENH: gof EDF discrete distributions MonteCarlo bootstrap
body: related #308

a reference (that I found by chance in a folder, related to old parameterization of GP-1 #3727)

This is similar to Stephens but for a new discrete distribution. critical values based on bootstrap and tabulated based on a Monte Carlo. It has the usual EDF test statistics but for discrete case, and a description of the simulation method. (I didn't read the article but looks like Stephens' style.)

No theorem, and I guess that the distribution is not (asymptotically) pivotal (*), so we need bootstrap in applications, the tabulated values will be only valid for the specific case.
(*) that's the case in continuous distribution if there is a shape parameter besides loc and scale. (But maybe it is loc and scale case because we only have two parameters, mean and dispersion factor, but it's not a loc-scale family in the sense of standardizing)

(It's not for the regression case, so it won't help for count regression.)



user: josef-pkt
issue_3747: state: open
title: BUG/ENH: transparams in loglike for non-newton methods
body: discrete.NegativeBinomial and I think several other models including GeneralizedPoisson in #3727 use internal transformation to impose non-negativity.
Because hessian is not define for the transformed dispersion parameter, the transform is only used for non-Newton methods, i.e. optimizers that don't use Hessian. 

The problem is that newton methods use a white-list `newton`, `ncg` which does not include the new access through generic scipy.optimize which allows the use of newer newton methods like dogleg.

We need to expand or come up with an alternative to the white-list.

Special case, in GeneralizedPoisson we can add an option to turn it off because the distribution is also defined for underdispersion (negative dispersion parameter, that is not too negative)


user: josef-pkt
issue_3745: state: open
title: Ordering of params in custom statespace model
body: Hi..
Just reopened the issue #3736 
I have some issues related to ordering of params. I have added a comment there.

Thanks
user: as4456
issue_3744: state: open
title: ENH: predictive margins versus teffects regression adjustment
body: Stata's documentation on margins has examples that compares ATE or ATT computed through the margin command with the ones computed using teffects.

That makes some nice comparison test cases for expanding margins. The direct RA regression uses an interaction effect between the binary treatment variable and all other regressors to compute separate regression coefficients for the two levels of the treatment effect.

Currently, our discrete margins do not support a multi-column effect. One difficulty is that it is not easy to get necessary information for multi-column effects. One possibility would be to use patsy through predict to construct the correct predictive design matrix. However, we still have problems recovering all multi-column effects and recreate a desired original dataframe.

(There was a related stackoverflow question for how to find out from the model and design_info to which variables/columns where used from the dataframe. There is no simple obvious way to recover that info.)

(There should be related issues to the implementation part.)
user: josef-pkt
issue_3743: state: open
title: ENH: diagnostics for WLS, and WLS version of GLM
body: (while looking at Stata's linktest #1479 documentation and our diagnostics)

two issues

- weights in regression diagnostic auxiliary regression
   - I guess the OLS based auxiliary regression are consistent but not efficient if we have weights
   - variable addition or score tests would use the weights from WLS in the score equations

- GLM has the WLS of the final step of the irls 
   - I think it should work to use resid_working and weights to reconstruct a WLS that corresponds to the final local approximation, and we can use WLS based tests for diagnostics, variable addition, link/nonlinearity test
   - this would be different (?) but asymptotically equivalent to using conditional moment and score tests directly, or tests using resid_pearson.

user: josef-pkt
issue_3742: state: open
title: df_model in mixed_linear_model
body: Last one for a while, scout's honor.

`MixedLMResults` has an `aic` [method](https://github.com/statsmodels/statsmodels/blob/master/statsmodels/regression/mixed_linear_model.py#L2417):

```
    @cache_readonly
    def aic(self):
        if self.reml:
            return np.nan
        if self.freepat is not None:
            df = self.freepat.get_packed(use_sqrt=False, has_fe=True).sum() + 1
        else:
            df = self.params.size + 1
        return -2 * (self.llf - df)
```

`bic` follows the [same pattern](https://github.com/statsmodels/statsmodels/blob/master/statsmodels/regression/mixed_linear_model.py#L2428).

The local `df` variable here behaves a lot like `df_model` does elsewhere (or in many cases, like `df_model+1`).  The class does not define a `df_model` attribute, but does define `df_resid` as the fairly-standard `self.df_resid = self.nobs - np_matrix_rank(self.model.exog)`.

Would it make sense to define `df_model` to match that `df`?  Is `df_resid` invariant to which of those three cases for `df` holds?

user: jbrockmendel
issue_3741: state: open
title: ENH: more support, conf_int, for conditional kde
body: Most of the discussion for conf_int and similar inference for kde is only for the univariate case.
Similar results could be provided for multivariate and conditional kde

see discussion starting here https://github.com/statsmodels/statsmodels/issues/1244#issuecomment-306053778


user: josef-pkt
issue_3723: state: open
title: ENH: fvalue  OLS/WLS summary without constant and robust cov_type
body: this is the enhancement part of #2444

We switch to wald fvalue in linear regression if the cov_type is not 'nonrobust', However, this does not work if the constant is implicit.

One alternative that would work is a score or LM test for linear restriction.

I'm not sure it's worth it. But it would provide a simple test for comparing group means `y ~ C(groups) - 1` against constant-only `y ~ 1`. 
Need cov_type robust score_test, which should be already be available for regression models but maybe without appropriate interface.

user: josef-pkt
issue_3722: state: open
title: Design: pandas add results to a dataframe
body: Just a generic idea.

In plots we have the ax keyword so users can add the plots to an existing axis. A bit similar, we could add additional results to a dataframe. It's less important for DataFrame or Series because users can merge the results to their own dataframe. So I'm not sure yet if it's worth it.  
analogue is add_trend to a dataframe.

example: 
- summary_frame for t_test: we can add optional multiple testing p-value corrections as a new column (that's not so easy in summary because of the fixed number of columns in params_table)

user: josef-pkt
issue_3721: state: open
title: ENH: add wald_joint, wald_joint_nonlinear
body: implementation idea for issue raised in several places, see e.g #3676

wald_joint, wald_joint_nonlinear: return joint distribution of linear or nonlinear transformation

these are not tests, but can be used for tests
similar to t_test but t_test treats the constraints as separate, vectorized constraints.

extras: return a WaldResults class that just has the wald methods of LikelihoodResults
(use a fake subclass of base class or, eventually (!), check outsourcing to mixin #3570)

(I don't know if this should be vectorized, but it would be 3 dimensional cov_params, in margins/predict we had the discussion, somewhere (?), about needing the joint distribution of two (non)linear transformation for each observation.)


user: josef-pkt
issue_3720: state: open
title: ENH: proportions_ztest, variance options, vectorize
body: this is an enhance part of #2734 (bugs are closed)

ENH: this only allows pooled variance, not a constrained mle variance (if value != zero).
ENH: we could vectorize this, I think, which would be useful for sample size or power calculations.
user: josef-pkt
issue_3719: state: open
title: REF: cleanup extend discrete Margins
body: 

- refactor const_idx detection, use generic instead of module specific code
   https://github.com/statsmodels/statsmodels/issues/2817#issuecomment-184485404
user: josef-pkt
issue_3718: state: open
title: ENH/REF: regression summary, use_t=False, fvalue
body: #3119 fvalue breaks if df are undefined (besides using int instead of float in string formatting)

Maybe (?) we should adjust the top table in summary if `use_t` is False. We could switch to chi2 distribution automatically instead of using f_value.

Currently use_t only affects the wald statistic, however the f_value is a standard ssr based test cov_type='nonrobust' but we switch to Wald if cov_type is a robust type.  

`use_t=False, cov_type='nonrobust'` could trigger ssr base chisquare test. (i.e. equivalent to chi2 base LR test similar to llnull)

will not be backwards compatible if users set use_t=False.


user: josef-pkt
issue_3717: state: open
title: ENH: summary add rows for wald_test_terms, multiparameter terms
body: (I thought I opened something like this already, but search is not successful)

I initially thought this when looking at Stock/Watson Introduction to Econometrics textbook. In the summary or summary_col (comparing models side by side) they add additional wald test results for combined effects, e.g. polynomial terms, similar to a type 3 Anova terms.

We have those now conveniently available with `wald_test_terms`, but we would need an option to add extra rows to summary and summary_col, or add an extra table to summary.  
see  #3702 and #3122 for dropping rows instead of adding rows.
and  #1637 for merging columns in LATEX,

user: josef-pkt
issue_3716: state: open
title: WIP/BUG/REF: correlation_tools - fix value in clip_evals, add `fudge` term
body: This was supposed to fix #3227  value ignore in clip_evals.

Include fix for non or slow convergence in `tests\test_corrpsd.py:test_corrpsd_threshold` (test run print convergence warnings. I tried up to 1000 iterations, but we only seem to approach the threshold in the eigenvalue for corr_nearest from below.
- add extra convergence check directly on the trial corr matrix
- add a fudge term 1e-16

Now, convergence is very fast in test_corrpsd_threshold with smallest eigenvalue possibly a bit above threshold.

However, unit test against R fail now. I'm not sure it's really testing the same things, e.g. R has ones on diagonal, i.e. corr_nearest, while the test cases uses cov_nearest with diagonal values slightly above one.
distance to original matrix is smaller in the new version than in the R reference numbers.  
This looks more like a reason to adjust the unit tests, than the code.

About the fudge term (I just made it up):
- it is only relevant for eigenvalues around zero. multiplicative fudge factor doesn't work if threshold is zero
- idea is to adjust or clip eigenvalues to something slightly higher than the target threshold, so we cross the threshold instead of just approaching it slowly. This might introduce a small distortion in the converged result, but works well and fast in the test cases.


status: currently fails in comparison to R, I need to get the changes out of the way, and haven't looked and decided yet whether to adjust the unit tests or the code.





user: josef-pkt
issue_3708: state: open
title: AttributeError: 'LikelihoodModelResults' object has no attribute 'mle_retvals'
body: I'm getting an `AttributeError` while fitting an ARIMA (I'm using version 0.8.0). Here's a (hopefully) reproducible example:


__Set up:__
```python
import numpy as np
from statsmodels.tsa.arima_model import ARIMA
from numpy.random import RandomState

rs = RandomState(42)
x = rs.rand(25)
```

Just for your validation, here's `x`:

    >>> x
    array([ 0.37454012,  0.95071431,  0.73199394,  0.59865848,  0.15601864,
            0.15599452,  0.05808361,  0.86617615,  0.60111501,  0.70807258,
            0.02058449,  0.96990985,  0.83244264,  0.21233911,  0.18182497,
            0.18340451,  0.30424224,  0.52475643,  0.43194502,  0.29122914,
            0.61185289,  0.13949386,  0.29214465,  0.36636184,  0.45606998])

__Fit:__
```python
arima = ARIMA(endog=x, order=(0,0,0), exog=None)
res = arima.fit(full_output=0)
```

And the error:

```
RUNNING THE L-BFGS-B CODE

           * * *

Machine precision = 2.220D-16
 N =            1     M =           12
 This problem is unconstrained.

At X0         0 variables are exactly at the bounds

At iterate    0    f=  1.44026D-01    |proj g|=  6.38378D-08

           * * *

Tit   = total number of iterations
Tnf   = total number of function evaluations
Tnint = total number of segments explored during Cauchy searches
Skip  = number of BFGS updates skipped
Nact  = number of active bounds at final generalized Cauchy point
Projg = norm of the final projected gradient
F     = final function value

           * * *

   N    Tit     Tnf  Tnint  Skip  Nact     Projg        F
    1      1      4      1     0     0   4.441D-08   1.440D-01
  F =  0.144026168071338

CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH

 Cauchy                time 0.000E+00 seconds.
 Subspace minimization time 0.000E+00 seconds.
 Line search           time 0.000E+00 seconds.

 Total User time 0.000E+00 seconds.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "//anaconda/lib/python2.7/site-packages/statsmodels/tsa/arima_model.py", line 979, in fit
    armafit.mle_retvals = mlefit.mle_retvals
AttributeError: 'LikelihoodModelResults' object has no attribute 'mle_retvals'
```

Looks like this might be an artifact of not including the `full_output` (from the [`LikelihoodModelResults` docstring](https://github.com/statsmodels/statsmodels/blob/master/statsmodels/base/model.py#L286)):

> full_output : bool, optional
>   Set to True to have all available output in the Results object's
>   mle_retvals attribute. The output is dependent on the solver.
>   See LikelihoodModelResults notes section for more information.

__I do not see this error when `full_output` is truthy.__
user: tgsmith61591
issue_3705: state: open
title: WIP/ TSTS/BUG/ENH Gee margins dummy
body: see #3704 missing support for dummy and count argument

adds unit tests for GEEMargins similar to Logit tests in #3696
no attempt to fix the issue (I didn't look at the details, missing `_derivative_predict`

needs rebase: 
- I branched off 3696 so it can be rebased after that is merged
- includes one incorrect monkey patching in unit tests, just to see what the results are

currently fails, this was written to check what's going on with GEE Margins

user: josef-pkt
issue_3704: state: open
title: BUG/ENH: GEE Margins do not support dummy and count
body: see https://github.com/statsmodels/statsmodels/pull/3696#issuecomment-303902539

dummy=True in GEE Margins raises
`AttributeError: 'GEE' object has no attribute '_derivative_predict'`

continuing example from 3696:
```
# check for GMM margins
import statsmodels.api as sm
groups = np.ones(x.shape[0], dtype=np.int32)
mod = sm.GEE(y, x, groups, family=sm.families.Binomial())
res_gee = mod.fit(cov_type='naive')
#mod._derivative_predict = mod._derivative_exog  # try monkey patching with wrong method
me_gee =  res_gee.get_margeff(at='mean', dummy = False)
me_gee_dummy =  res_gee.get_margeff(at='mean', dummy=True)
```

dummy=False works  
dummy=True is missing `_derivative_predict`

```
  File "...\statsmodels\genmod\generalized_estimating_equations.py", line 1733, in get_margeff
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    return GEEMargins(self, (at, method, atexog, dummy, count))
  File "...\statsmodels\genmod\generalized_estimating_equations.py", line 2528, in __init__
    self.get_margeff(*args, **kwargs)
  File "...\statsmodels\genmod\generalized_estimating_equations.py", line 2719, in get_margeff
    method, 1)
  File "...\statsmodels\discrete\discrete_margins.py", line 344, in margeff_cov_with_se
    pass
  File "...\statsmodels\discrete\discrete_margins.py", line 324, in margeff_cov_params
    else:
  File "...\statsmodels\discrete\discrete_margins.py", line 214, in _margeff_cov_params_dummy
    dfdb = dfdb.mean(0)
AttributeError: 'GEE' object has no attribute '_derivative_predict'
```
user: josef-pkt
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_3703: state: open
title: ENH/TST: GLM, discrete, saturated model, perfect prediction
body: corner case behavior if we have a model with perfect prediction by design

(I thought I already have an issue, but don't find it right now.  
trigger for issue right now: according to their documentation, SPSS handles zero cases in loglike and deviance separately, dropping terms instead of "fixing" them as we do)

examples: 
- contingency table, loglinear model with saturated exog
- computing saturated reference case for deviance, (I needed to add an epsilon to avoid corners)
- model with aggregated or averaged data using freq_weights or var_weights

Those would make good test cases for the currently open issues of zero division and similar in GLM families.

user: josef-pkt
issue_3693: state: open
title: FAQ: overview models by error structure
body: All our models are based on a conditional distribution of endog, y, given exog, x, $f(y_i | x_i)$ for observation i. The conditional distribution can be given by a prediction function plus an additive error term as in OLS, or directly as conditional distribution as in the discrete and generalized linear models.

Models differ in their assumption about wether observations are identically and independently distributed.

## identically and independently distributed observations

This covers most of our basic models including OLS, discrete models, GLM and RLM. 

However, in the exponential family only the conditional expection E(y|x) needs to be correctly specified. We can correct Wald inference by using robust sandwich standard errors that consistently estimate the covariance of the parameter estimates when heteroscedasticity or correlation is present.

## heteroscedasticity

Models that are able to use a variance that differs across observations are currently only WLS and GLS, and will include GLM with var_weights.

(Note: discrete models and GLM assume identical distribution, but the variance is a function of the mean and is therefore in general not constant across observations.

## cluster correlation and panel data

In the simple case we can have correlation within a cluster or group but no correlation across groups. 
- MixedLM for linear model with normal random effects.
- GEE for generalized linear models with correlation within groups based on estimating equations (GMM)
- (GLMMixed is still missing)

Crossed random effects or panel data with time and individual random effects are currently only supported by variance components in MixedLM.

## Serial Correlation

If we have data ordered by a a time axis, then observations can be correlated with past observations. All or most models in `tsa` assume specific structure for the serial correlation. 
GLSAR

## Spatial correlation

There is nothing specific to spatial correlation in statsmodels. see pysal for this.


TBC

user: josef-pkt
issue_3691: state: open
title: FAQ-D: terminology/spelling heteroscedastic versus heteroskedastic
body: "heteroscedastic" wins (so far)

We should standardize the spelling because it makes a search for the word easier.


(A while ago a read a blog post or something similar, usage is unclear, heteroscedastic is more American)
user: josef-pkt
issue_3690: state: open
title: ENH/DOC: reset_test
body: A general specification test mainly for functional form, extends easily to GLM and other models, either with an auxiliary regression or a variable addition score test.  
similar to link_test #1479

see https://github.com/statsmodels/statsmodels/issues/2041#issuecomment-276161907

I thought I already had a function, but it looks like it's misplaced and not in api, docs
https://github.com/statsmodels/statsmodels/blob/master/statsmodels/stats/outliers_influence.py#L86
2 unit tests against gretl in "statsmodels\regression\tests\test_glsar_gretl.py"  
I guess it should be in diagnostics, and not in outliers_influence

related
#5388 tools for binning, ...
user: josef-pkt
issue_3689: state: open
title: orig_endog vs endog_lagged vs...
body: Mostly @ChadFulton 

Many models in tsa have to truncate the given `endog`, and update `nobs` in accordance.  They could be more uniform in what names they use for both the truncated and un-truncated versions.


- var_model has `endog_lagged`
- arima_model calls the truncated version `self.endog`, then later accesses `self.data.endog` to access the original. (at least I think that's the motivation.  Less sure on this than the others)
- markov_autoregression has `orig_endog`.  It also updates `self.data.endog` so the previous method is not immediately applicable.
- `regression.linear_model.GLSAR` truncates `exog` and calls it `wexog`

I don't have any real preferences over these variants, but would like to settle on a standard.  It could be done with aliases to maintain backward-compatibility.  Thoughts?
user: jbrockmendel
issue_3687: state: open
title: API prediction methods in statespace SARIMAX
body: Hi all, 

I´m trying to understand the difference between the several prediction methods available in the statespace SARIMAX function:

- `predict`
- `get_prediction`
- `forecast`
- `get_forecast`

as has a different return object. Which one is the "canonical" method to get the prediction results, including confidence intervals?
user: ispmarin
issue_3686: state: open
title: DOC: improve structure and cross links or cross references
body: I went through the table of content for `stats` http://www.statsmodels.org/dev/stats.html
and there is no mention of other functionality/modules that have their own top level entry.

example: contingency tables and ANOVA  
e.g. mcnemar and cochrans_q are in stats "Non-Parametric Tests" and in contingency tables docs

also top level TOC ordering could be changed, e.g. put contingency tables and ANOVA next to stats.

example question: What is available for proportions? Is it possible to get a full list of what's available for different tasks, objectives, assumptions?

(docs still need to be updated for recent additions like MANOVA and CCA)


user: josef-pkt
issue_3676: state: open
title: SUM/DOC/FAQ:  overview documentation for parameter hypothesis tests in models
body: see #1971 for more topic pages in the documentation

(following a bit the discussion in Julia about adding ANOVA type tests
https://groups.google.com/d/msg/pystatsmodels/5w1OAr_OFQg/CiXu72opAQAJ )

We don't have an overview page or notebook in the documentation about what is available.

- wald test: all models
   - t_test vectorized single restriction hypothesis
      missing multiple testing p-value correction, e.g #3297
   - wald_test (and special case f_test) for joint hypothesis
   - wald_test_terms : list of hypothesis with single or joint restrictions
   - missing: pairwise, joint distribution of several hypothesis
   - nonlinear hypothesis ??? where is it, #2296 (still open)
   - multivariate tests: ?, MANOVA, ... 
- LR type test: only for linear model
   - compare_lr_test: two given models
   - anova_lm: standard ANOVA (F-tests) for linear models
   - llnull in discrete, likelihood models (LR test against constant-only model)
   - (GMM j-test, test for overidentifying restrictions
   - missing : extend to other models
   - WIP/missing: GLM deviance and pearson chisquare tests
   - missing: QMLE, M-estimator: LR-type test with dispersion in GLM, RLM (LR-type test in a PR)
   - - multivariate tests: ?, cointegration, rank tests
- LM/score test: in linear model, semi-hidden in GLM, correctly specified likelihood assumption
   - compare_lm_test in linear regression model, allows robust cov_type (IIRC)
   - score_test (in GEE and hiddedn in GLM)
   - diagnostic test: linear model only, includes missing variable LM tests
   - missing/WIP: full support based on prototype or special case versions.
- conditional moment tests
   - mainly specification tests, no proper support except some examples or special cases
- other ???
  - CvM tests ?
   - ...
- special cases
   - tsa : what do we have for hypothesis tests on parameters?
      - granger causality
      - ...
- test on functions of parameters, (not clear how this fits into a categorization)
   - exponential form (`eform`)
   - predictions, margins
   - ATE, ATT, ...
   - ...
   
check:
- AnovaRM: is it a wald test or LR-type (F) test?

Almost all of the above is for linear (or affine) hypothesis on the parameters associated with explanatory variables. 
Some specification tests reduce to added variable tests (i.e. hypothesis on mean parameters). Other score, LR or CM tests refer to "extra" parameters, e.g. heteroscedasticity tests, test for overdispersion in Poisson, test for zero-inflation, ....  

**tools:** 

- estimating models under linear restrictions so we can use LR or score tests: fit_constrained (only in GLM ?)
- zero-restriction omitting parameters: some models don't support general linear constraints, e.g. lags in ARMA and related models
- ...
- missing e.g. drop or add functions for models. stepwise, ...
- missing score and LR test equivalent of wald_test_terms, i.e. drop all terms one or more at a time.
- mostly missing estimation subject to nonlinear constraints
- one step estimation of constrained models: ? where is the helper function to compute constrained params

**other inferential**
- confidence intervals: in most cases included with the hypothesis tests
   - wald : standard, generically supported
   - profile loglikelihood (LR): version for GLM in a PR, others?, 
   - score confidence intervals
- power: nothing yet for models

**"Caveats"**

- add warnings if cov_type is not "nonrobust", and a LR or score test assumes correctly specified loglike.  
   Stata raises exception (IIUC the docs)
- possible problems with weights: only wald tests are generic,   
   other types of tests need to watch out for how weights need to be used. Initially not supported.






user: josef-pkt
issue_3674: state: open
title: WIP/ENH: add grouping (letter display) for pairwise comparison
body: see #3669 
form overlapping groups based on statistically significant pairwise comparison, or a distance matrix for significance

see docstring for details

has basic unit tests for 3 examples in Piepho 2004, no unit tests for options
API unclear, also might need restructuring (lazy letter assignment in results class?)
more convenience, pandas index, choosing alphabet, ... ?

Intended mainly as helper function to integrate with plot and summaries

location maybe temporary (multicomp is still in sandbox, and we cannot add it to stats.multicomp because of circular import if it's reused in multicomp and Tukey)

user: josef-pkt
issue_3670: state: open
title: fit_regularized().summary() shows as None
body: Hi,
I am running the following regression code:

```
Kara = sm.OLS(train.ix[:,0], train.ix[:, col_for_coeff]).fit_regularized(alpha=0.2, L1_wt=0.5)
print(Kara.summary())
```

The output is shown as 

```
None
```

There was hardly any incidents on Stackoverflow, except one small mention in the comments section on [this question.](http://stackoverflow.com/questions/40072870/statistical-summary-table-in-sklearn-linear-model-ridge) Any help?

System info:
OS: Windows 8.1
Python: 3.5.3
Statsmodels: 0.8.0
user: iceman121
issue_3669: state: open
title: ENH: grouping for pairwise comparison, tukey-hsd
body: http://stackoverflow.com/questions/43987651/tukey-test-grouping-and-plotting-in-scipy

see reference cited in R package multcompView
Piepho, Hans-Peter. 2004. “An Algorithm for a Letter-Based Representation of All-Pairwise Comparisons.” Journal of Computational and Graphical Statistics 13 (2): 456–66.

This looks just like the standard combine all non-significant pairs into groups, where groups can overlap and do not partition the set of groups. AFAIR several packages print a list for this.

I don't remember whether I ever made a function for this.

For the boxplot or letter labeling a group is assigned the combination of the letter identifier of all groups it belongs to.


user: josef-pkt
issue_3659: state: open
title: SUMM/Design: penalization - interpretable scale of penalization weight
body: Can we define penalization weights for regularized least squares or penalized maximum likelihood in a way that it has an interpretable scale?
Is it possible to define it in terms of an interpretable "prior"?

e.g. #3389 adjusted elastic net alpha to match R.
I don't remember what I do in Ridge and others, but the penalization weights does not take the residual variance in linear model Ridge/Theil into account.

The problem is that it is difficult to guess in which range the penal weight might correspond, for example, to a medium strong prior.
I switched datasets and the penalization weight for elastic net with L1_wt=0.9 that "looks" fine is alpha=1 in the first case and around alpha=1e-6 in the second case.
(examples for #3647 the second dataset HCW has 20 observation and 25 regressors and is underdetermined without regularization.)

related: Greenland, follow-up readings for #3561, has several articles for the connection between penalization and Bayesian priors, and some recommendation for how to choose priors.

for example: 
Quadratic penalization corresponds to a normal prior and would have the same interpretation as the params and cov_params (or as linear restrictions on them) if we use the correct scaling for combining loglikelihood and logprior.
LASSO or similar would have interpretable threshold rules.

related: Do we add scaling/zscoring the data or equivalently using data dependent prior scaling e.g. proportional to exog.var or exog.cov.  In that case coefficients are in scale of "beta" coefficients (standardized coefficients). 
IIRC, Greenland argues against it because we often have better prior information on the "natural" scale. (e.g. effect per pack of cigarettes, year of schooling, ...) 




user: josef-pkt
issue_3654: state: open
title: Question / having trouble using MixedLM
body: I actually have a few questions about MixedLM, but please also keep in mind that I'm new to statistics and `statsmodels`, so I might be doing something wrong. I have really scoured the internet for help before asking though. The tl;dr is that I'm having trouble getting convergence even with data that looks reasonable, and the "free=" kwarg doesn't seem to work at all with fixed params; nor have I been able to get penalties working (they ruin the convergence of things that would otherwise converge, but I thought Ridge was supposed to help convergence, not hurt it? And I do standardize when I try Ridge) -- but I can open that as another issue, because I might be doing something wrong here.

Here's code to reproduce the issue:
```
from itertools import groupby
from operator import itemgetter

import numpy
from statsmodels.regression.mixed_linear_model import MixedLM, MixedLMParams

# Using statsmodels version 0.8.0.
x = numpy.arange(50, 100, 0.5)

# Partition x into 3 groups. Yes, it's true that's not many groups, and
# the real situation I'm working with often has more groups, but not always;
# but it's still necessary to account for correlation within each group,
# even if there are just 3, and isn't that what a random effects model
# is for?
group_list = ['group1' if v < 67
              else 'group2' if v < 86
              else 'group3' for v in x]

# Set up y so that it has a group random intercept that correlates with x.
y = x * .8 + 10 \
    + numpy.random.normal(size=x.size, scale=2) \
    + [{'group1': -5, 'group2': 1, 'group3': 5}[g] for g in group_list]

# This is a rise over run of about 0.3 (that is, each group rises about 5
# units of y and there are three of them over an x range of 50, so
# 5. / (50. / 3) = 0.3). Additionally y rises 0.8 for every unit of x, so in terms of
# predicting y for unknown groups, one would predict something like
# y = (0.3 + 0.8)x. 

# Calculate the group medians so that can be included as a fixed param
# so that the correlation of the random intercept with the group median
# can be accounted for.
# cf. "Fitting Multilevel Models When Predictors and Group Effects Correlate"
# http://www.stat.columbia.edu/~gelman/research/unpublished/Bafumi_Gelman_Midwest06.pdf
medians_by_group = {
    group_id: numpy.median([xs for _, xs in vals])
    for group_id, vals
    in groupby(sorted(zip(group_list, x), key=itemgetter(0)),
               key=itemgetter(0))}

medians_arr = \
    numpy.array([medians_by_group[group_id]
                 for group_id in group_list])

# It's true I deliberately haven't standardized by dividing the x
# values by their mean or standard deviation; but when I do that
# it doesn't seem to have any effect anyway?
exog = numpy.vstack((numpy.broadcast_to([1.], x.shape),
                     x,
                     medians_arr)).T

# that is,
# X = [[1, x_1, group_1_median],
#      [1, x_2, group_1_median], ...

exog_re = numpy.full((x.size, 1), 1.0, dtype=numpy.float64)

# Guess a zero intercept and a 1:1 relationship between x and y.
naive_start_params = \
    MixedLMParams.from_components(fe_params=[0, 1, 0], cov_re=numpy.eye(1))

mod = MixedLM(endog=y, exog=exog, groups=group_list, exog_re=exog_re)

# lbfgs dies with a Singular matrix LinAlgError from mixed_linear_model.py
# line 1160, so it's not included here -- but why the error? None of the
# other methods have that problem.
try_methods = ['bfgs', 'powell', 'cg', 'basinhopping']

# 1. Try fitting in a variety of ways: all five methods just listed, and also
# see whether using start_params affects the convergence or not.
all_results = \
    {(method, use_start_params):
         mod.fit(method=method,
                 start_params=(
                     naive_start_params if use_start_params else None))
     for method in try_methods
     for use_start_params in (True, False)}

# For me, only powell and basinhopping converge, and they do so regardless
# of whether there are start_params or not; and also they both converge
# to basically the same values. See attempt_1_fit_fixed_effects.png plot below.

# 2. Try fitting with free= to hold y intercept at zero.
all_results_intercept_zero = \
    {method:
        mod.fit(method=method,
                start_params=naive_start_params,
                free=MixedLMParams.from_components(fe_params=[0, 1, 1],
                                                   cov_re=numpy.eye(1)))
     for method in try_methods}

# Again, only powell and basinhopping converge, and neither of them held
# the intercept at zero actually. In fact they converge to basically the
# same exact values as without free=. But this isn't unique to these
# methods; I see the same issue when bfgs does converge -- it ignores
# the free= request.
```

Here's the fit that powell and basinhopping find without free= specified:
![attempt_1_fit_fixed_effects](https://cloud.githubusercontent.com/assets/6463868/25952857/a9e3f72c-3616-11e7-95cb-f55124c73bd8.png)

So, I guess, to sum up: Why don't bfgs and cg converge; why does lbfgs die with an error; and why doesn't free= seem to have an effect (even with bfgs when it converges in other situations)? What's the correct way to hold a fixed parameter fixed?
user: joshua-g
issue_3653: state: open
title: Question to the API of MixedLM.from_formula
body: I have another question about the interface to the MixedLM.from_formula
function.  Sorry, if I may bother you again.

I have data which are linear in some `x`.

```
y_ijk(x) = a_ijk + b_ijk ⋅ x
```

I would like to model the parameters `a_ijk`, `b_ijk` by a "random
intercept" model:

```
a_ijk = μ + δ_i + α_j + β_k(j)
b_ijk = γ + η_i
with α_j and β_k(j) random
```

Then I guess the following call to MixedLM.from_formula is correct:

```
MixedLM.from_formula('y_ijk ~ i*x', groups='j', re_formula='1',
    vc_formula={'k': '0+k'}, data=df)
```

(I hope it's correct so far.) But what if I would like to model a
random slope, too?

```
a_ijk = μ + δ_i + α_j + β_k(j)
b_ijk = γ + η_i + α'_j + β'_k(j)
with α_j, β_k(j) and α'_j, β'_k(j) random
```

What if I may assume the slope is only random in `j`? (Which is how I
would like to model my data.)

```
a_ijk = μ + δ_i + α_j + β_k(j)
b_ijk = γ + η_i + α'_j
with α_j, β_k(j) and α'_j random
```

All my best guesses to MixedLM.from_formula resulted in fits which did
not even converge, or resulted in singular matrices, or produced output
which likely don't belong to the above models. My guess is, I don't
understand the API.

Thank you in advance.
user: 00tau
issue_3647: state: open
title: ENH: causal impact, synthetic control
body: see #3631 for background on the type of treatment effect estimator
also includes predictive models with inequality constraints on parameters, see #3632, #1211 for nnls

currently:
- one treated, many controls
- only outcome variables, no covariates for matching or prediction
- OLS based, serial dependence not modeled, see tsa based notebook and Google's causalimpact
- constraints and regularization, options
  - OLS, no constraints, no regularization
  - elastic net
  - nonnegativity constraints using nnls
  - simplex constraint, i.e. nonnegative and sum to one, using slsqp
  - with or without adding constant to the predictor
- no explicit tuning parameter selection, e.g. variable selection or choosing penalization weights, Needs an extra step for specification search for prediction, e.g. information criteria or cross-validation
- no inference, nor standard errors are included here. Analytic/Asymptotic formulas for HCW style are in draft stage. Otherwise, it needs wrapper for placebo permutation tests

background
- this started out as simplified synthetic control (Abadie et al), but removing covariates
- HCW use just outcome regression with variable/control selection, Follow-up articles use e.g. LASSO and weaken assumptions for consistent estimation of ATT
- Doudchenko, Imbens emphasize the role of constraints, and this is currently close to their approach

Also, `statsmodels.treatment` doesn't exist in master yet.

I haven't thought much about names for module and classes.

no nice wrapper things, this is currently pure numpy array based, no missing value handling, ...
Formulas might not be needed here because we only have one type of variables in the basic version, and we don't want to specify a list of names if we have a large number of controls.

 
user: josef-pkt
issue_3645: state: open
title: ENH: tool: recover params when projecting or absorbing
body: When we run OLS after absorbing part of the explanatory variables, then we get the params of the not-absorbed variables. The demean and detrend functions are specific examples but we also absorb in other places.

How to we get the params for the absorbed parameters?

(aside: where are the internal tool functions? I always have difficulty to remember and I don't think they are in the docs.)

Tool functions are often just a few lines to inline, but I don't always know or remember how to compute them or how to do it in a computational efficient way.
user: josef-pkt
issue_3644: state: open
title: ENH: panel robust cov_type, time and subgroups
body: 2 or multi cluster correlation where one of the components is time (with kernel/HAC correlation)

setup: time and cross-section are individuals in clusters with no between-cluster correlation
e.g. cross-section is individuals in states/provinces and the cross-sectional correlation is only within a state

I don't think this case is currently covered by any cov_type.  
All the panel cov_types assume correlation between all cross-sectional units, which is more general and covers within cluster correlation (i.e. a block structure of the contemporaneous correlation matrix), but there might be computational or precision/efficiency problems without imposing the restriction when the number of cross-sectional units is large.

example for data structure: survey data with individuals in states without aggregating to state averages
Bertrand, Marianne, Esther Duflo, and Sendhil Mullainathan. 2004. “How Much Should We Trust Differences-In-Differences Estimates?” The Quarterly Journal of Economics 119 (1): 249–75. doi:10.1162/003355304772839588.


related:
Thompson, Samuel B. 2011. “Simple Formulas for Standard Errors That Cluster by Both Firm and Time.” Journal of Financial Economics 99 (1): 1–10. doi:10.1016/j.jfineco.2010.08.016.
(I found the reference a few weeks ago but don't remember details. I think it had another case for panel robust cov.)

(another possibility: resurrect generic cov_type that would also handle distance based, e.g. spatial, correlation)





user: josef-pkt
issue_3643: state: open
title: ENH: use GMM for potential outcome prediction, synthetic response
body: (just a semi-random idea, not seen in literature)

as followup to #3631 diff-in-diff type of analysis, but with just one or a few treated units, similar to synthetic response, HCW and Google's causalimpact

(the third version of a notebook to try out some ideas is at https://gist.github.com/josef-pkt/a8e8181096a7a06a1413fe3d84f9a56d )

one problem for inference is how to get the standard errors and confidence interval for ATT.
ATT is a out of sample time averaged prediction error, where one part of the uncertainty comes from the parameter estimate.

For cross-sectional ATE, ATT and potential outcomes, I used GMM to get the standard errors, following the pattern in Stata. A similar idea might also apply here.
The main problem, AFAICS, when adding the ATT moment condition is that the moment conditions are block diagonal, i.e. one block with pre-treatment observation for estimation and one block for prediction during treatment periods. In the basic case with time series data, we need to use HAC weights and cov_type in both parts. However, I don't think we can use the current GMM estimator for the block diagonal structure. At least the mean and nobs computation need to be adjusted, so that the average is taken within the block and not over the entire array of moment conditions.

More generally, we need "effective nobs" for each column or moment condition when we have a block structure. One advantage in the current case is still that it is a two-stage or triangular, recursive problem, the first block is independent of the second block, but the second block depends on the parameter estimate and the cov_params of the first block.

eg. b0 are estimate parameters, b1 are parameters of the prediction error process including ATT, then

m0(b0)
m1(b0, b1)

d m0 / d b1 = 0
d m1 / d b0 != 0

(intuitively, the second block of moment conditions doesn't constrain the first block, there is no feedback)

This is a bit similar to testing extra moment conditions as in CMT or score tests. We estimate the parameters on one part, but we need the connection of the second part to the first part to get the proper inference for the second part. In that case it is adding moment conditions on the same observations, in the case here we are adding a new block (rows) of observations and additional moment conditions.

A bit related: 
What if we use a dummy for each observation in the treatment period, so we get a perfect fit and the parameters for the dummies are the prediction errors. Then we need to estimate the mean of prediction errors to get ATT, or fit another process to the treatment effect. Can GMM handle single observation dummies, the associated parameter cannot be consistently estimated, and we might run into problems with getting the weight and covariance matrix, even thought at the end we only want a fixed number of average parameters, which we can estimate consistently.
(e.g. Do outlier dummies work in the current GMM implementation or do we need to constrain the weight matrix?)

user: josef-pkt
issue_3639: state: open
title: Specific Seasonal ARMA order needs to be passed to x13_arima_analysis
body: Hi,
Is there a way to pass a specific seasonal ARMA order to x13_arima_analysis rather than it examining the "maxorder" tuple for model identification.
Lets say, I want to pass the following SARIMA spec: ARIMA(2,1,1)*(2,1,1) to x13_arima_analysis
I believe specific difference order could be passed to "diff" argument and setting the "maxdiff" argument as None.
Is there a way to pass a specific ARMA order also?

Thanks
user: as4456
issue_3634: state: open
title: ENH:  end-of-sample break test
body: (mainly to park a reference that was referenced by largely unrelated issues Conley and Taber #3631, too much to read now)

related to structural break testing, outlier and anomaly detection and control charts , see also #3295 
maybe close to one observation tests, see #2847


Andrews, D. W. K. 2003. “End-of-Sample Instability Tests.” Econometrica 71 (6): 1661–94. doi:10.1111/1468-0262.00466.

asymptotic theory when test statistic is not consistent i.e. we don't get law of large numbers for it. 
sample size goes to infinity, but test is on a fixed number of observations.

AFAIR, Greene mentioned something like this as aside in the text book.

user: josef-pkt
issue_3632: state: open
title: SUMM/FAQ: predictive models instead of causal/structural
body: What are the consequences if we only need consistent point/mean or interval prediction but do not need consistent parameter estimates?

(This is mainly an issue for categorizing use cases and interpretation of models. Main implementation consequence is that we don't always need inference, i.e. objective is predictive as in scikit-learn/machine learning.)

stable correlation instead of (causally) interpretable parameters

examples: 

- estimating auxiliary variables
  - propensity score for IPW
  - imputation, potential outcome in treatment effect (in some cases)
  - first stage instrumental variable, control function (not looked at details yet)
- ...

(related tools:  non- or semi-parametric (series) estimators, variable selection and penalization without regard to inference, perfect separation and singular design are mostly irrelevant, ...)
user: josef-pkt
issue_3631: state: open
title: SUMM: difference in differences, panel data and similar treatment effects
body: triggered by #3622

DiD in the simplest case is just the interaction effect of time and group dummies in OLS
What support to we need? What extensions and related models would be useful?

### asymptotics

- standard DiD: 2 time periods, T = 2, number of individuals in treated `n0`` and control/untreated `n0` both go to inf
- small sample corrections: number of time periods T and number of groups is small, similar to cluster robust cov_type  Donald and Lang 2007
- T small/fixed, n0 (number treaded) small and fixed, n1 -> inf  Conley and Taber 2011
- T (or only pre-treatment periods, T0,  goes to inf, n0 (number treaded) small and fixed, panel and time series
   - synthetic control Abadie, et al. 2010 and other years  (motivated for case studies with one treated unit)
   - Hsiao, Chin and Wang 2012 (HCW), and followup studies, expecially Li and Bell 2017 

### methods and approaches

- basic DiD: fixed effects OLS
- DiD with IPW: Abadie 2008 (semiparametric)  (This looks good for background and for specific application) Cross-sectional IPW with selection on observables is special case.
- DiD with matching: (I didn't read anything specific) Heckman, Todd and Ichimura, mentioned as comparison case in some newer studies that compare DiD-Matching with sythetic control and HCW
- synthetic control: matching-weights in pre-treatment, weighted average for potential outcome in treatment periods
- HCW: very simple to implement, OLS in pre-treatment and prediction of outcome variable in treatment years.  uses only outcome variable, important variable selection either AICC/BIC or Lasso and cross-validation
- estimated factor model: similar to HCW but with estimating the full factor model. Both synthetic control are motivated by factor model structure, but don't use and estimate it explicitly 
   - Bai (interactive fixed effects, not looked at, code in Julia), 
   - Xu 2017 uses factor model for prediction of potential outcome
- non-linear: not looked at, e.g. Logit/Probit, ... 

Doudchenko, Nikolay, and Guido W. Imbens.    (too long to print, not read, but 

### API and structure

??? eclectic

- DiD: packages in Stata, wrapper around OLS, adding IPW of difference regression, ... ?
- synthetic control: packages in Stata, main point estimating the weights.  
  No standard errors, testing through placebo tests.
- HCW: simple OLS but work on variable selection or penalization (limitation if number of pre-treatment periods is not long enough, and there are many potential controls.)


## References

(notes: 
There are many surveys, lecture notes, slides, summaries, textbook chapters that were not detailed enough for me when I tried in the past for DiD and related methods.
List below is incomplete, mostly the main articles that I looked at)

Abadie, Alberto, Alexis Diamond, and Jens Hainmueller. 2010. “Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program.” Journal of the American Statistical Association 105 (490): 493–505.

———. 2015. “Comparative Politics and the Synthetic Control Method.” American Journal of Political Science 59 (2): 495–510. doi:10.1111/ajps.12116.

Abadie, Alberto, and Javier Gardeazabal. 2003. “The Economic Costs of Conflict: A Case Study of the Basque Country.” The American Economic Review 93 (1): 113–32. doi:10.1257/000282803321455188.

Becker, Martin, and Stefan Klößner. n.d. “Fast and Reliable Computation of Generalized Synthetic Controls.”

Donald, Stephen G, and Kevin Lang. 2007. “Inference with Difference-in-Differences and Other Panel Data.” The Review of Economics and Statistics 89 (2): 221–33. doi:10.1162/rest.89.2.221.

Doudchenko, Nikolay, and Guido W. Imbens. 2016. “Balancing, Regression, Difference-In-Differences and Synthetic Control Methods: A Synthesis.” NBER Working Paper 22791. National Bureau of Economic Research, Inc. https://ideas.repec.org/p/nbr/nberwo/22791.html.

Gardeazabal, Javier, and Ainhoa Vega-Bayo. 2016. “An Empirical Comparison Between the Synthetic Control Method and HSIAO et Al.’s Panel Data Approach to Program Evaluation.” Journal of Applied Econometrics, January, n/a-n/a. doi:10.1002/jae.2557.

Hsiao, Cheng, H. Steve Ching, and Shui Ki Wan. 2012. “A Panel Data Approach for Program Evaluation: Measuring the Benefits of Political and Economic Integration of Hong Kong with Mainland China.” Journal of Applied Econometrics 27 (5): 705–40. doi:10.1002/jae.1230.

Kreif, Noémi, Richard Grieve, Dominik Hangartner, Alex James Turner, Silviya Nikolova, and Matt Sutton. 2016. “Examination of the Synthetic Control Method for Evaluating Health Policies with Multiple Treated Units.” Health Economics 25 (12): 1514–28. doi:10.1002/hec.3258.

Lechner, Michael. 2011. “The Estimation of Causal Effects by Difference-in-Difference Methods.” Foundations and Trends® in Econometrics 4 (3): 165–224. doi:10.1561/0800000014.

Li, Kathleen T., and David R. Bell. 2017. “Estimation of Average Treatment Effects with Panel Data: Asymptotic Theory and Implementation.” Journal of Econometrics 197 (1): 65–75. doi:10.1016/j.jeconom.2016.01.011.

Wang, Huiqiang. 2016. “Estimating the Health Impacts of Food Safety Interventions: Optimal Counterfactual Selections via Information Criteria in Small Samples.” Food Policy 63 (August): 44–52. doi:10.1016/j.foodpol.2016.06.005.

Xu, Yiqing. 2017. “Generalized Synthetic Control Method: Causal Inference with Interactive Fixed Effects Models.” Political Analysis 25 (1): 57–76. doi:10.1017/pan.2016.2.

## to get started

I played with simplified synthetic control in an example, and tried out what turned out to be the same HCW

- HCW: easy, use LASSO or elastic net to avoid overfitting, needs cross-validation or similar
  full best subset selection (of explanatory variables) will be computationally expensive with many controls
- synthetic control: see #3622 has a few tricky parts, 
  - nnls and slsqp seem to work fine, we have basinhopping as global optimizer (because outer optimization loop might have multiple local minima)
   - data handling requires work
   - versions with different constraints imposed, (Doudchenko and Imbens) 
      (aside: in a simple example without much noise imposing the full no-extrapolation restriction resulted in a single match, i.e. weight=1 for one control and zero for others. This didn't happen without sum to 1 constraint on weights/regression coefficients)
- support for explicit or implicit matching: covariance balance
- DiD with IPW: this looks mostly like an extension of the IPW method (Abadie 2005), 
  - extension for semi-nonparametric propensity score estimation.


- extra support for standard fixed effects DiD ???
- extensions: Xu and factor models look interesting, but data requirements might be too large.  
  (Also factor models on untreated as unsupervised component selection might miss features/components that  are closer to the controls, i.e. takes common factor structure more literal that synthetic control or HCW.)




user: josef-pkt
issue_3629: state: open
title: params is a method of emplikeAFT in aft_el.py, not an attribute
body: In aft_el.py, params is one of the methods for emplikeAFT (see the doc), and def in class AFTResults(OptAFT).

It throws an error for koul_and_mc.py in the example folder.
`new_params = model.fit().params` cann't get the params estimation, and  `new_params = model.fit().params` works.
user: ShuguangSun
issue_3626: state: open
title: ENH: fit_regularized has no get_prediction, cov_params
body: fit_regularized for elastic_net e.g. in OLS returns an instance of `RegularizedResults(Results)`
Because it inherits from base.Results no wald inference and none of the model specific results are available.

example: I was using `get_prediction` to get the confidence intervals for `OLS(...).fit()` but this does not work when switching to `OLS(...).fit_regularized(...)`
user: josef-pkt
issue_3623: state: open
title: How can ARIMA do one step predict with new test data?
body: I trained a ARIMA model with two weeks data.

Now I want to use the model to do one step predict with my new test data.

But I dont know how to feed my test data to the predict function.

```python
http_march_train = http_ratio['2017-03-01': '2017-03-14'] # collect every 15minutes
start = '2017-03-15 00:00'
end = '2017-03-15 23:45'
http_march_test = http_ratio[start: end]  

arima = smt.ARIMA(http_march_train, (3, 0, 3))
model = arima.fit()

# this is a dynamic forecast of the 15th day
model.predict(start = start, end = end)

# How can I feed the http_march_test data to predict to do an in sample one-step new forecast?


```
user: reminia
issue_3622: state: open
title: ENH: causal/treatment effect with synthetic control
body: see mailing list https://groups.google.com/d/msg/pystatsmodels/M35hci6Gl8A/hrZKrF0bEQAJ

based on several articles by Abadie and co-authors  
main use case: case studies for the impact of a large event with only one or just a few units under treatment

two R packages: Synth and MSCMT

---
semi-random pieces of information:

using nnls and slsqp for non-negativity and sum-to-one constraints
http://stackoverflow.com/questions/33385898/how-to-include-constraint-to-scipy-nnls-function-solution-so-that-it-sums-to-1

The MSCMT authors seem to have a lot of information about computational details
Becker, Martin, and Stefan Klößner. “Fast and Reliable Computation of Generalized Synthetic Controls.” working paper, Preliminary version: January 30, 2017  

Becker, Martin, and Stefan Klößner. n.d. “Computing Generalized Synthetic Controls with the R Package MSCMT.”




background reference
Doudchenko, Nikolay, and Guido W. Imbens. 2016. “Balancing, Regression, Difference-In-Differences and Synthetic Control Methods: A Synthesis.” NBER Working Paper 22791. National Bureau of Economic Research, Inc.   
https://ideas.repec.org/p/nbr/nberwo/22791.html.  
https://arxiv.org/abs/1610.07748  

Based on a quick browsing, they seem to point out the  "no-extrapolation" constraints in Abadie et al.
user: josef-pkt
issue_3621: state: open
title: warning message
body: Hello,
Could you explain me what the following message means. The source code that cause the message is below the warning message.

```
statsmodels\tsa\statespace\kalman_filter.py:1505: ValueWarning: Model has time-invariant 
state_intercept matrix, so the state_intercept argument to `predict` has been ignored.   
warn(warning % (name, name), ValueWarning)

_**mod.initialize_known(saved_state, saved_cov)
if not df_hist_load_used_with_model.empty: 
    sarimax_model_results = mod.smooth(saved_model_parameter)
sarimax_model_results.predict(start= 0, end= iend_index, exog= None)**_
```

user: fgilbert01
issue_3616: state: open
title: ENH/Design compare_pearson_chi2, variance assumptions
body: hypothesis test base on pearson_chi2 in GLM, and other LEF by extension

AFAIR from looking at Agresti's categorical book, the test just compares pearson_chi2 directly.

In an example I got negative difference, i.e. restricted/reduced model has lower pearson_chi2.
As in other case the problem seems to be that we don't use a common denominator when scaling or standardizing the residuals (resid_pearson) in the computation of pearson_chi2.

If I use a common denominator, then the results look much better
https://github.com/statsmodels/statsmodels/pull/3410#issuecomment-295823923

In simple one or two sample hypothesis tests, we would use the same variance assumption for full and reduced models. The same common denominator assumption is used in ssr based f-tests.  
However, here we have to scale each observation instead of dividing the ssr as with common scale or dispersion as denominator.

Need to find a reference, I don't remember seeing it, but it looks kind of "obvious".


The example in #3410 is not completely obvious because it looks at exposure, var_weights and freq_weights. pearson_chi2 with the original data has the correct sign. 
user: josef-pkt
issue_3614: state: open
title: ENH/BUG?  t_test inhomogenous restriction, diff missing
body: AFAICS while trying an example

if we have an affine restriction R * p = q, then it doesn't show the values for R*p - q

a simple case, I would like the table for the difference, e.g. `params[0] - 1`
I can get it by reversing the standardization: `tt.statistic * tt.sd`

(example is just trying out something with poisson exposure and might not make much sense yet)

```
In [92]:
res_poiss_s.params - np.log(xsum)
Out[92]:
array([ 0.77318973, -0.6061358 ,  0.11778304, -0.51081432])

In [98]:
tt = res_poiss_s.t_test((np.eye(4), np.log(xsum)))
tt
Out[98]:
<class 'statsmodels.stats.contrast.ContrastResults'>
                             Test for Constraints                             
==============================================================================
                 coef    std err          z      P>|z|      [0.025      0.975]
------------------------------------------------------------------------------
c0             2.5649      0.277      2.788      0.005       2.021       3.109
c1             1.7918      0.408     -1.485      0.138       0.992       2.592
c2             2.1972      0.333      0.353      0.724       1.544       2.851
c3             1.0986      0.577     -0.885      0.376      -0.033       2.230
==============================================================================

In [101]:
tt.statistic * tt.sd
Out[101]:
array([ 0.77318973, -0.6061358 ,  0.11778304, -0.51081432])
```
user: josef-pkt
issue_3613: state: open
title: ENH: GLM summary additional information
body: GLM summary could provide more information

- llnull or wald_null similar to discrete models
- Stata has dispersion, i.e. pearson_chi2 and deviance divided by df_resid. This is a more immediate measure of dispersion than unscaled values
- Stata also shows BIC, and AIC (if IRLS is not used)
user: josef-pkt
issue_3605: state: open
title: BUG: MixedLM.fit blocks optimizer keywords
body: I wanted to try the new minimize optimization with dogleg, but MixedLM uses a whitelist for `kwargs`.

using example from #3604

```
>>> mdf = md.fit(method='minimize', min_method = 'dogleg')
  File "...\statsmodels\regression\mixed_linear_model.py", line 1936, in fit
    raise ValueError("Argument %s not allowed for MixedLM.fit" % x)
ValueError: Argument min_method not allowed for MixedLM.fit
>>> Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
```
user: josef-pkt
issue_3604: state: open
title: Inconsistent results for mixed linear models under Python 2.7 and 3.4 
body: _statsmodels.mixedlm_ generates different results under Python 2.7 and Python 3.4. Moreover, in Python 3.4, the model does not converge when _re_formula_ is used with variable slope.

Here is a simple example:

```python
import statsmodels.formula.api as smf
import pandas as pd
import numpy as np
from copy import deepcopy

np.random.seed(seed=0)

data1 = pd.DataFrame(columns=['x','y','id'])
data2 = pd.DataFrame(columns=['x','y','id'])
data1['x'] = np.random.normal(size=100)
data1['y'] = deepcopy(data1['x'])*3 + np.random.normal(size=100)
data1['id'] = np.ones(100)
data2['x'] = np.random.normal(size=100)
data2['y'] = deepcopy(data2['x'])*2 + np.random.normal(size=100)
data2['id'] = np.ones(100)*2

data = pd.concat([data1,data2])
data = data.reset_index(drop=True)

md = smf.mixedlm('y ~ x', data, groups=data['id'], re_formula='~ x')
mdf = md.fit(method='cg')
print(mdf.summary())
````

Python 2.7 converges and generates the following results:

```
Mixed Linear Model Regression Results
==============================================================
Model:               MixedLM   Dependent Variable:   y        
No. Observations:    200       Method:               REML    
No. Groups:          2         Scale:                0.9636  
Min. group size:     100       Likelihood:           -286.7528
Max. group size:     100       Converged:            Yes      
Mean group size:     100.0                                    
--------------------------------------------------------------
                   Coef.  Std.Err.   z    P>|z| [0.025 0.975]
--------------------------------------------------------------
Intercept           -0.048    0.532 -0.091 0.928 -1.091  0.994
x                    2.631    1.673  1.572 0.116 -0.649  5.911
Intercept RE         0.558                                    
Intercept RE x x RE  1.660                                    
x RE                 5.597                                    
==============================================================
```

Python 3.4 does not converge and gives these results:

```
/usr/local/lib/python3.4/dist-packages/statsmodels/base/model.py:496: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
  "Check mle_retvals", ConvergenceWarning)
/usr/local/lib/python3.4/dist-packages/statsmodels/base/model.py:496: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
  "Check mle_retvals", ConvergenceWarning)
/usr/local/lib/python3.4/dist-packages/statsmodels/base/model.py:496: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
  "Check mle_retvals", ConvergenceWarning)
/usr/local/lib/python3.4/dist-packages/statsmodels/base/model.py:496: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
  "Check mle_retvals", ConvergenceWarning)
          Mixed Linear Model Regression Results
=========================================================
Model:              MixedLM Dependent Variable: y        
No. Observations:   200     Method:             REML     
No. Groups:         2       Scale:              0.9680   
Min. group size:    100     Likelihood:         -284.9909
Max. group size:    100     Converged:          No       
Mean group size:    100.0                                
---------------------------------------------------------
                    Coef.  Std.Err. z P>|z| [0.025 0.975]
---------------------------------------------------------
Intercept           -0.057                               
x                    2.620                               
Intercept RE         0.034                               
Intercept RE x x RE  0.129                               
x RE                 0.482                               
=========================================================
```

Which are pretty different for some of the parameters. Any ideas?
user: sosata
issue_3601: state: open
title: ENH: tools, reparameterizing linear constraints to zero constraints
body: just a reference before I loose it again

appendix A in the following contains formulas for transforming exog for a linear constraint into a partitioned zero constraint.
This is a similar case as the reparameterization for `fit_constrained` which imposes the constraint. (I haven't compared the formulas.)

Winkler, Anderson M., Gerard R. Ridgway, Matthew A. Webster, Stephen M. Smith, and Thomas E. Nichols. 2014. “Permutation Inference for the General Linear Model.” NeuroImage 92 (May): 381–97. doi:10.1016/j.neuroimage.2014.01.060.


user: josef-pkt
issue_3595: state: open
title: ENH: cov_type for user defined cov_params
body: Currently, there is no official way to use a user provided cov_params_default. We allow `cov_p` as arguments to some methods, but cannot set the default that is used in the wald tests.

Adding `cov_type='cov', cov_kwds={'cov': my_cov}` would allow setting cov_params_default in the "standard" way (without having to worry about manipulating the results instance)


use case: bootstrap standard errors created outside of the model
user: josef-pkt
issue_3590: state: open
title: SUMM: Bootstrap in regression models
body: Statsmodels still doesn't have any support for bootstrap except in a few isolated cases, e.g. bootstap cov in GenericLikelihoodModel and for impulse response functions in VAR/SVAR.

The arch package has some functions.
see also #2339 and #420 (I never figured out for myself how to start with it, except for several attempts at special cases.)

basic resampling and bootstrapping params is relatively easy. However, we need to be able to resample pivotal statistics to get higher order improvements, e.g. bootstrap t confidence intervals for regression.
(I don't know or remember how to do this, especially not a multi-parameter version, so it's not obvious and needs at least a prototype.)

To get started: (specification) robust bootstrap t confint based on sandwich cov_type, that then can be combined with wild or block bootstrap (in similar direction as studentized permutation tests #3571)

(triggered by http://stackoverflow.com/questions/43151401/bootstrapping-multiple-regression-parameters-in-python which is the first time I see a bootstrap question)

(Note: resampling, especially residuals, under the Null would often be better, similar to score tests are often better than wald tests, but then we need more support, and we can only use one Null at a time.)





user: josef-pkt
issue_3589: state: open
title: ENH: `expect` method for kernel density estimation
body: http://stats.stackexchange.com/questions/271297/expected-value-of-continuous-probability-distribution

interesting question: expectation method for kernel density
I don't think we have anything like that. scipy.stats.distributions has `expect` with integrate.quad.

Given that we have the discretized points when using fft, we could use standard discrete integration like trapezoid, which should work well enough in smooth cases, simple weighted average will be less precise
???
user: josef-pkt
issue_3585: state: open
title: Question about the possibility of speeding up in RegressionModel's fit method through using the pinv of X'X rather than pinv of X
body: I have noticed statsmodels solves the OLS using equation:
      1)beta = dot( np.linalg.pinv(X), y )                     
but since pinv(X) = dot( pinv( dot( X.T, X ) ), X.T ) (from wiki).
why not using equation
      2)beta = dot( pinv( dot( X.T, X ) ), y )
And this equation seems running fast than 1) since it solves the pinv of an Hermitian matrix.   
user: yc19920125
issue_3577: state: open
title: ENH/DOC: contingency tables, doc exception, no informative repr or str
body: http://www.statsmodels.org/devel/contingency_tables.html  shows an exception in the last case stratified 2x2

running test_nominal_association doesn't show the results in `__repr__` or `__str__`

```
>>> table.test_nominal_association()
<bunch object containing statsmodels results>

>>> print(table.test_nominal_association())
<bunch object containing statsmodels results>

>>> vars(table.test_nominal_association())
{'statistic': 7.3992462809917354, 'pvalue': 0.11623505314616689, 'df': 4}
```
user: josef-pkt
issue_3576: state: open
title: WIP ENH: stats._random.py add simulate conditional contingency tables
body: see #3557  (see also #3571 for similar idea for GLM in regression setting)

a brief exercise in simulating 2-dim contingency tables with fixed margins under independence. Given the independence assumption (probabilities are product of margins), this does not allow arbitrary probability distributions and hypothesis tests.

related #2932 for Monte Carlo p-values for unconditional multinomial sampling 
see #2936 for general sampling space assumptions in contingency tables, rates and proportions.

This turned out to be just a permutation structure, first version is based on Boyett, but then I added just plain permutation case.
Permutation needs the nobs arrays, but loops for one random sample are in numpython (i.e. in C, with shuffle and bincount.) Neither shuffle nor bincount can be vectorized, so we need a loop over samples in a Monte Carlo run. One function uses a generator to yield and infinite sequence of random tables.

In contrast Patefield loops over cells and computes a random count variable in each cell. That algorithm is faster for large nobs and not very large number of cells. However, the count distribution (conditional on left over items) is more difficult to compute, and I didn't figure out the shortcut that Patefield uses, which starts at the expected cell count and then loops to increase or decrease. This extra loop wouldn't be fast in python without some vectorization.

There is no application for this yet. The target would be Kim, Agresti 1997 and similar see #3557

extras

`p_table` : compute the probability pmf for a specific table, IIRC it's the pmf for a multivariate hypergeometric distribution. used in the unit tests to compare MC frequencies with theoretical probabilities.


This includes now `chisquare_contingency` as a permutation test for 2-dimensional contingency table based on Pearson's chisquare statistic. Partially tested against an example in an article.
user: josef-pkt
issue_3575: state: open
title: ENH: stats add Brunner Munzel test
body: two or k-sample comparison with unequal distributions, like Behrens-Fisher problem if variance differs.

If we have two samples with different distributions, variance, skewness and kurtosis, then comparing means, medians or center of location might be only partially interesting.
kruskal-wallis and Brunner-Munzel test something like 
P(y_1 < y_2) = 0.5 or 
P(y_1 < y_2) + 0.5 * P(y_1 = y_2) = 0.5 (with ties or discrete variables) 

see comment in #3571 for permutation tests (sample sizes < 10 in a group)

Neuhäuser, Markus. 2010. “A Nonparametric Two-Sample Comparison for Skewed Data with Unequal Variances.” Journal of Clinical Epidemiology 63 (6): 691–93. doi:10.1016/j.jclinepi.2009.08.026.

Brunner, Edgar, and Ullrich Munzel. 2000. “The Nonparametric Behrens-Fisher Problem: Asymptotic Theory and a Small-Sample Approximation.” Biometrical Journal 42 (1): 17–25. doi:10.1002/(SICI)1521-4036(200001)42:1<17::AID-BIMJ17>3.0.CO;2-U.

Fagerland, Morten W., and Leiv Sandvik. 2009. “Performance of Five Two-Sample Location Tests for Skewed Distributions with Unequal Variances.” Contemporary Clinical Trials 30 (5): 490–96. doi:10.1016/j.cct.2009.06.007.

this article (I didn't read) looks like a good overview of implicit and explicit assumptions and interpretation when stricter assumptions don't hold. There are articles that make a similar point
Fay, Michael P., and Michael A. Proschan. 2010. “Wilcoxon-Mann-Whitney or T-Test? On Assumptions for Hypothesis Tests and Multiple Interpretations of Decision Rules.” Statistics Surveys 4: 1–39. doi:10.1214/09-SS051.

Also, there are several more recent papers with application or variation of Brunner-Munzel to various setting including multiple hypothesis testing and multiple comparisons.

implementation: 
The basic Brunner-Munzel test with t-statistics similar to Welch-Satterthwaite looks relatively easy because scipy has midrank function, and we have the pattern for Welch-Satterthwaite.
Adding permutation looks also relatively straight forward, as in #3571 references.



user: josef-pkt
issue_3574: state: open
title: aic in ar/arima/...
body: Related to #3568, I am trying to simplify the AR[...] models and have them inherit some attributes as properties from a general case.  There are two inconsistencies I need help understanding.

The aic/bic/hqic are defined slightly differently in ARMAResults, ARResults, and VARResults.  The differences can be understood by just looking at aic:

[ARResults](https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/ar_model.py#L726): `aic = np.log(self.sigma2) + 2 * (df_model+1) / nobs`

[VARResults](https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/vector_ar/var_model.py#L1463): `aic  = ld + 2 * (neqs * df_model) / nobs`

Here `ld` is analogous to `np.log(self.sigma2)`.

[ARMAResults](https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/arima_model.py#L1452): `aic = -2 * llf + 2 * (df_model+1)`

In the CSS case, `llf` can be simplified to the point where this is a positive affine transform of the version in ARResults.

Finally the actual questions.

1) Is there a reason not to use a common definition between AR vs ARMA/ARIMA?

2) If we set neqs=1 in the VAR, we should end up with an AR, but there is an extra 1 unaccounted for.  Where did it go?


user: jbrockmendel
issue_3571: state: open
title: ENH: Score permutation test in models
body: (just an idea no reference)

In two or k sample case we can get the distribution for a "no effect" hypothesis by permuting the observations. In analogy this should also apply to models with explanatory variables, implemented as score variable addition test with permutation distribution for the test statistic.
In the simplest case with one categorical variable we get the analogue of a k-sample or anova type hypothesis test.

By reparameterization similar to fit_constrained, we can convert linear hypothesis into a variable addition test, e.g. using offset if we just fix the value of one params and the new variable or parameter would be the difference to the offset.

This would be a conditional test analogous to an independence test in a contingency table with fixed margins. (maybe there is something for the regression case in the R package for conditional hypothesis tests.)

(I'm thinking mainly of GLM binomial and poisson where bootstrap samples are more difficult to define.)




user: josef-pkt
issue_3570: state: open
title: ENH: InferenceMixin or NoDataModel
body: Suppose I have `params` and `cov_params` estimates, and maybe a predict function, and want to get the inference from a LikelihoodModelResults class.

Sometimes I faked a model for some functions or classes that require a model as argument in experimental code by using a DummyModel and attach whatever is needed.

It would be more useful if a reusable version is officially supported, which would also make it clearer what information is actually required (e.g. `.data`, `param_names`, `df_resid`, ...), and review if we really need all those in specific methods.


related: I once wrote or started a multivariate normal class with linear transformation that does something similar, but having it tied into model based inference is more specific and makes it easier to figure out what we need. (e.g. nonlinear tests and inference)

(random thoughts:
I think we don't want to actually move this to a Mixin class at the current stage because of the tight integration with Results and Models and a dual class hierarchy is simpler than a triple hierarchy, e.g. adding score_tests or likelihood ratio tests doesn't fit the simple Wald pattern. Those would also need different information.  (*)
The NoDataResults could initially be just Wald based inference, LR is possible with just adding llf, score would either require that the score is provided or that we have access to the full score_obs computation. (for score type tests: I have a version that uses moment conditions and their derivative in GMM style which IIRC works with summary statistics instead of data.) 
However, profile likelihood or score confidence intervals require a full model with data to compute large deviations from the point estimate.
)

(*) a Mixin that is only used for LikelihoodModelResults would essentially not change anything compared to the current structure.

user: josef-pkt
issue_3568: state: open
title: df_resid in ar/arima/var_model
body: I'm trying to merge some redundant and confusing-to-me bits of ar_model, arima_model, and var_model.  First a question that should be easy:

AR.fit and ARIMA.fit each take a `method` argument.  `method` is used to determine `self.nobs` dynamically, which in turn affects `self.df_resid` and so on down the line.  The options they recognize are slightly different, and I want to confirm that these are synonyms.

ARIMA.fit recognizes "css-mle", "mle", and "css".  It [sets](https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/arima_model.py#L937) `nobs = n_totobs - k_ar` if the method is "css", otherwise `nobs = n_totobs`.  AR.fit recognizes "cmle" and "mle", and [subtracts](https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/ar_model.py#L554) `k_ar` in the "cmle" case.  (VAR subtracts `k_ar` regardless).  So it looks very much like "cmle" is a synonym for "css".  Is this correct?


That was the easy part.  One more for now.  AR, ARIMA, and VAR each define df_model, df_resid, and the information criteria slightly differently.  My guess/hope is that the definition in AR contains a typo, in which case all of these can be merged into properties in a base class.

[VAR](https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/vector_ar/var_model.py#L864) and [ARIMA](https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/arima_model.py#L1385) both define `df_resid` as `self.nobs - self.df_model`.  But [AR](https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/ar_model.py#L696) defines it as `self.n_totobs - self.df_model`.  Should the AR version also be `nobs - df_model`?

Some suggestive evidence:

a) df_resid is defined but then never used in ar_model.py
b) AR [defines](https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/ar_model.py#L754) fpe as `((nobs+df_model)/(nobs-df_model))*self.sigma2`.  Having df_resid in the denominator here would line up with the definition in [VAR](https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/vector_ar/var_model.py#L1477).
c) AR divides by nobs-df_model in defining old_scale in the [bse](https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/ar_model.py#L715) method.

user: jbrockmendel
issue_3567: state: open
title: FAQ/Design/ENH: What if a model does not have any parameters that need to be estimated?
body: empty params or no estimated params

I guess that all or most of our models cannot handle "empty" models, i.e. models where there are no free parameters that need to be estimated.

possible usage is as a reference model where some parts are defined without free parameters, e.g. all parameters are restricted to fixed values.

AFAIR, I saw a stackoverflow question for ARIMA(0,1,0) i.e. just a random walk.

The usecase I was just thinking of are simple one sample score and LR tests or score and profile likelihood confidence intervals for e.g. GLM. The constant as the only parameter would be fixed, and the prediction would be based only on an offset variable. (analogue of one sample tests in statsmodels.stats)
(Two sample hypothesis tests are not a problem because we have one free parameter)

workaround: 
all statistics could be calculated based on a fake model or calling the corresponding methods that use params as argument directly.
However, we wouldn't be able to use generic code that just estimates a restricted model.

One possibility that might work, `fit(start_params=params_fixed, maxiter=0)` is supposed to return the results for fixed start_params, but it would have cov_params, wald statistics and df_resid for the one parameter case.

I'm not sure whether supporting these "empty-fit" models would justify the increased code complexity.


user: josef-pkt
issue_3566: state: open
title: Examples for out-of-sample forecast
body: Hi I am trying som of statsmodels Time series models and noticed that for some of them there are no instructions on how to obtain out-of-sample forecasts. I am particularly interested in SARIMAX and DVAR models. In the case of DVAR there is not even a  notebook example to follow.

user: fccoelho
issue_3563: state: open
title: FAQ: How to save only the model's prediction part?
body: Hi.
I'm currently trying to fit the OLS and using it for prediction.

`import statsmodels.api as sma`
`ols = sma.OLS(myformula, mydata).fit()`
`with open('ols_result', 'wb') as f:`
`pickle.dump(ols ,f)`

However, my data set is about 5GB and OLSResults takes about 6GB memory.
It's the problematic part because i want to share the model.predict() with other environments, but moving and loading 6GB file takes too much time.

`with open('ols_result', 'wb') as f:`
`pickle.dump(ols.predict ,f)`

This code does not solved my situation. (it also takes 6GB)

Is there any way to save only the predicting part of the model?

**edit** (Josef)

most of the discussion below (comments by me) miss the point of this example here which is that formula information cannot be pickled, and we need to recreate it as a hack.
as pointed out by by @bsaunders23 with link to #4174 
user: youjin2
issue_3562: state: open
title: Historical Decomposition of a VAR object
body: Is the "historical decomposition" part implemented in statsmodels similar to below:
http://stackoverflow.com/questions/36950491/historical-decomposition-in-r
https://estima.com/forum/viewtopic.php?f=4&t=655 
user: as4456
issue_3561: state: open
title: ENH: Firth's penalized logit, GLM
body: question on mailing list
general penalized discussions in #2282, #2293

This is a specific issue for implementing Firth's bias correcting penalization, which is different from the L1, L2, CAD penalization.
We could use L2 type penalization for complete separation. The original motivation for the Firth type penalization was bias correction. Penalization is related to logdet information matrix or hessian.

The bias correction can also be used for other models, mainly GLM/LEF, in a similar way, including multinomial.

todo; add references

implementation for Logit/GLM-Binomial (given my partial overview)
- adjust endog: similar to continuity correction or removing zeros in multinomial tests, we can adjust endog proportional to diagonal of hat matrix (*)
- adjust estimating equations and use IRLS or similar
- use penalized mixin, main problem might be the nonlinearity of the penalization function and the lack of explicit first and second derivatives. (SAS has some formulas, don't find right now, either for glm or for phreg)

(*) this article has a list of pseudo-responses for several standard GLM models (family-link combinations)
Kosmidis, Ioannis, and David Firth. 2009. “Bias Reduction in Exponential Family Nonlinear Models.” Biometrika 96 (4): 793–804. doi:10.1093/biomet/asp055.
(I didn't read it. I don't know if this is just for the IRLS/MLE optimization or if we can use other results like cov_params based on this. predict and resid_xxx is likely wrong if we just change endog.)

also in Kosmidis' thesis section 5.2.3 Refinement of the pseudo-data representation: Obtaining the bias-reduced estimates using already implemented software

**update** Kosmidis Firth 2009 show that the penalized likelihood representation only works under some assumptions on the link function which is satisfied by canonical links. In general, the bias correction is added to the estimating equation and we might have to solve that explicitly, i.e. IRLS/GEE/GMM style.


SAS
http://support.sas.com/documentation/cdl/en/statug/68162/HTML/default/viewer.htm#statug_logistic_details05.htm#statug.logistic.logisticdfirth  the only mention I find for logistic except for examples
http://support.sas.com/documentation/cdl/en/statug/68162/HTML/default/viewer.htm#statug_logistic_examples15.htm
and phreg
http://support.sas.com/documentation/cdl/en/statug/68162/HTML/default/viewer.htm#statug_phreg_details23.htm


user: josef-pkt
issue_3560: state: open
title: SUMM/ENH/SMEP: compositional data and statistics on a simplex
body: see also discussion in #3537

data is a proportions/shares/categorical distribution data or data where only relative magnitudes are observed

from what I can see browsing and reading various articles, we have three possible approaches
(I did only partial readings to get a rough overview.)

1. QMLE/GMM: use multinomial logit data with continuous response on the simplex instead of counts or binary/categorical response, single fraction case works with all binary reponse models, see main description comment in #3537, #2661, #2712 
2. transformed endog: e.g. logistic normal (similar to log normal) transform endog to map into R^{d-1} and use normal there, Aitchison type analysis, see discussion in #3537 with links to python bio-related packages
3. use distribution defined on simplex, e.g. extensions of dirichlet, or beta in the single proportion case.

related article (that I didn't read, I downloaded it in 2012 for the first time, maybe related to QMLE fraction work), 
seems to provide an overview for the one fraction case including Papke/Wooldridge on QMLE for a fraction.

Kieschnick, Robert, and B D McCullough. 2003. “Regression Analysis of Variates Observed on (0, 1): Percentages, Proportions and Fractions.” Statistical Modelling 3 (3): 193–213. doi:10.1191/1471082X03st053oa.


preliminary comments:

- to 1. QMLE is relatively clear, just needs implementation, we only get E(y | x) based parameter estimates and prediction
- to 2. usual problem with transforming endog, we estimate based on E(transform(y) | x). Interpretation similar to log-normal distribution. Statistics on the simplex is implied from the transformation. "Aitchison geometry" changes definition/metric for expectation and similar statistics on the simplex. (I'm not sure this is not building a house under a lamp post (mixed metaphor))
- to 3. Looks like heavily developed for use as Bayesian prior. grouped, nested and other versions to move away from independence assumptions built into dirchlet. AFAICS, no clear winner yet. (grouped or nested might (!) be similar to nested Logit in discrete choice models to work around the independence of irrelevant alternative assumption in multinomial logit or discrete choice models.)



user: josef-pkt
issue_3557: state: open
title: ENH: simulate contingency tables for exact monte carlo tests
body: related to #2931

For conditional tests the simulations are more difficult because we need to simulate contingency tables with given margins which is more difficult than simulating from an unconditional distribution like binomial, multinomial or poisson.

Kim, Donguk, and Alan Agresti. 1997. “Nearly Exact Tests of Conditional Independence and Marginal Homogeneity for Sparse Contingency Tables.” Computational Statistics & Data Analysis 24 (1): 89–104. doi:10.1016/S0167-9473(96)00038-2.

Patefield, W. M. 1981. “Algorithm AS 159: An Efficient Method of Generating Random R × C Tables with Given Row and Column Totals.” Journal of the Royal Statistical Society. Series C (Applied Statistics) 30 (1): 91–97. doi:10.2307/2346669.

IIRC I looked at articles from Berry Mielke in the past but don't remember and contingency tables with fixed margins looked to complicated given my limited interest, maybe this:
Long, Michael A., Kenneth J. Berry, and Paul W. Mielke. 2010. “Multiway Contingency Tables: Monte Carlo Resampling Probability Values for the Chi-Squared and Likelihood-Ratio Tests.” Psychological Reports 107 (2): 501–10. doi:10.2466/03.PR0.107.5.501-510.


user: josef-pkt
issue_3556: state: open
title: ENH: fft for discrete probabilities, multinomial p-value
body: https://github.com/huonw/sisfft-py   MIT licensed
http://www.maths.usyd.edu.au/u/uri/ Keich's website

older bagfft is in FAST in c++ without license http://www.maths.usyd.edu.au/u/uri/
bagfft link is dead

It looks like fft can be used to compute some of the test p-values for tests in large contingency tables. I skimmed some articles, but don't understand the details how fft is used. (There are many papers, but not very explicit in the computational details and computing the characteristic function (?) seems to involve more steps.)


theme: approximately exact conditional tests for contingency tables or multinomial distributions



user: josef-pkt
issue_3540: state: open
title: ENH: Allow specifying SARIMAX trend as part of observation equation
body: Currently, the trend in SARIMAX is specified as part of the state space form as part of the state intercept. This PR allows including it more similarly to an exogenous variable instead (so it would go into the observation intercept instead), by setting a new `trend_as_exog` argument to True.

This is already what was done if `hamilton_representation` was set to True; this argument just allows also doing it for the default "Harvey" representation.

As I note in the argument docstring, this doesn't really have much of an effect on anything (e.g. parameter estimates should be almost identical). It's basically just allows a bit more fine-tuned control over what the state space representation looks like.
user: ChadFulton
issue_3539: state: open
title: ENH: projected normal, GEE with circular response
body: related to #3537 regression with circular or angular response

projecting bivariate or multivariate endog on the circle seems to be the most popular and numerically easiest way for regression with a circular or angular response and linear regressors.

Based on a brief skimming Hall, Shen looks interesting as candidate for implementation or introduction to this type of regression models.

Hall, Daniel B., and Jing Shen. 2015. “Marginal Projected Multivariate Linear Models for Clustered Angular Data.” Australian & New Zealand Journal of Statistics 57 (2): 241–57. doi:10.1111/anzs.12114.

this paper introduce using bivariate normal projected circular regression
Presnell, Brett, Scott P. Morrison, and Ramon C. Littell. 1998. “Projected Multivariate Linear Models for Directional Data.” Journal of the American Statistical Association 93 (443): 1068–77. doi:10.2307/2669850.

(I don't have an overview of circular regression, but the main alternatives seem to have numerical problems, like multiplicity of local minima in MLE. 
So this seems to be a better start and is related to "shape" matrix in elliptical models, Tyler scatter.
)

user: josef-pkt
issue_3538: state: open
title: ENH/REF/SUMM: discrete choice - other packages
body: other project that develop discrete choice models in Python

https://github.com/timothyb0912/pylogit
see also brief mailing list discussion: see second half of this thread https://groups.google.com/d/msg/pystatsmodels/IoQckMN32zA/fk2n9PJ7BAAJ

more from Berkeley urban:

https://github.com/UDST/choicemodels/issues/1
https://github.com/ual/choicemodels/wiki/Project-notes

https://github.com/UDST/urbansim/blob/master/urbansim/models/dcm.py
https://github.com/ual/LCCM

user: josef-pkt
issue_3537: state: open
title: FAQ-D: compositional/multinomial/multivariate versus ordered logit
body: (semi-random design thoughts)

main issue: do we need to compute all possible "choices"?

- 1. In multinomial and discrete choice models the endog is multivariate binary, so it is possible in some cases to compute only the statistics/loglikelihood for the observed choice.
- 2. With compositional or multivariate continuous data we observe non-zero values for all "choices" and need to compute the full set of statistics/loglikelihood.
- 3. If we want to reuse multinomial as compositional quasi-likelihood models, then we also need the loglikelihood and score for all choices

In case 1. 
- MNLogit and similar need statistics for all choices to compute normalization, so we need to do already most of the computation for the full case
- ordered Logit, interval censored endog and similar define the probabilities independently from each other based on a latent variable. We don't need to compute anything about the non-observed choices.

so for 3. we have a trade-off between reusable model/code versus computational efficiency (compute only what's needed for the specific, narrowly defined model.)

compositional data: shares restricted to the simplex
projected data: multivariate random variable projected to the unit circle/sphere or simplex


user: josef-pkt
issue_3532: state: open
title: Coefficients in `summary()` and `params` differ for random effects (MixedLM)
body: Why is the result of the coefficients in `MixedLMResultsWrapper.summary()` and `MixedLMResultsWrapper.params` different for the variance of the random effects?

```
vcf = {'slice': '0+C(slice)'}
model = sm.MixedLM.from_formula('response ~ intensity + intensity:group',
    groups='subject', re_formula='1', vc_formula=vcf, data=df)
rfit = model.fit()

print(rfit.summary())
print(rfit.params)
```


```
In [14]: print(rfit.summary())
            Mixed Linear Model Regression Results
==============================================================
Model:                MixedLM   Dependent Variable:   response
No. Observations:     247       Method:               REML
No. Groups:           7         Scale:                0.0553
Min. group size:      13        Likelihood:           -41.2644
Max. group size:      65        Converged:            Yes
Mean group size:      35.3
--------------------------------------------------------------
                     Coef. Std.Err.   z    P>|z| [0.025 0.975]
--------------------------------------------------------------
Intercept            0.854    0.151  5.672 0.000  0.559  1.149
intensity            0.093    0.003 30.556 0.000  0.087  0.099
intensity:group[T.2] 0.027    0.004  6.901 0.000  0.019  0.035
Group RE             0.068    0.328
slice RE             0.198    0.341
==============================================================


In [15]: print(rfit.params)
Intercept               0.853810
intensity               0.092588
intensity:group[T.2]    0.027137
Group RE                1.229293
slice RE                3.587345
dtype: float64

In [16]: sm.version.full_version
Out[16]: '0.8.0.dev0+fc3584f'
```


user: 00tau
issue_3530: state: open
title: ENH:  directional or circular statistics ?
body: to add or not too add?

another topic that I spent a few months on to figure out the basics, but it's doubtful I'm interesting in more that brief visits to the topic


main reference
Mardia, K. V. 1972. Statistics of Directional Data. Probability and Mathematical Statistics. London, New York: Academic Press.

Mardia, Kanti V., and Peter E. Jupp, eds. 1999. Directional Statistics. Wiley Series in Probability and Statistics. Hoboken, NJ, USA: John Wiley & Sons, Inc. doi:10.1002/9780470316979.

I think I read the older book.
partially based on python translation of matlab fileexchange toolbox
There is also a python version by the matlab toolbox author, and some functions in scipy and in an astropy package.
Astropy uses `quantities` to keep track of units, which we would not want to use.

a bit related: statistics with complex valued data: #3528 

user: josef-pkt
issue_3529: state: open
title: Max lags < nobs on high number of data points
body: I am getting error saying "maxlag should be < nobs" when I ran SARIMAX with following data and order:

**train_data** :
 
[ 294188.4375     210114.990234   188622.728516    58837.6806641
   15483.6000977  123683.999512    80514.71875     53679.3706055       0.
       0.              0.              0.              0.              0.
       0.              0.              0.              0.              0.
       0.              0.              0.              0.              0.
       0.              0.              0.              0.              0.
       0.              0.              0.              0.              0.
       0.              0.              0.              0.              0.
       0.              0.              0.              0.              0.
       0.         167231.4375          0.         221264.945312   105288.5
   18581.2695312   65031.1308594   27870.4804688   18581.2695312
   21678.1503906  221035.375      241550.03125    353023.402344
  278454.226562   151743.5625     167231.4375     112428.421875
   27870.4794922   18578.1191406   27776.6396484       0.              0.
       0.              0.              0.              0.              0.
       0.              0.              0.              0.              0.
       0.              0.              0.              0.              0.
       0.              0.              0.              0.              0.
       0.              0.              0.              0.              0.
       0.              0.              0.              0.         133313.8125
       0.         130068.890625    71224.5703125  120774.          37162.5390625
  158639.046143   153429.673828   236746.669922   316328.418945
  286373.498901   413651.203125   260297.005859   314403.996094
  135210.029297   112028.429688    34873.2900391   29768.6308594
   52034.2792969       0.          16891.1992188       0.              0.
       0.              0.              0.              0.              0.
   92906.36            0.          46453.18        28810.12            0.
       0.              0.              0.              0.              0.
       0.              0.              0.              0.              0.
       0.              0.              0.              0.              0.
       0.              0.              0.              0.          63283.77
  134907.76            0.          78720.4        150164.         169580.44
   47419.7             0.         265750.5        308980.35       570382.45
  369837.76       329795.06       506430.05        65347.84       136118.96
   78771.4             0.              0.              0.              0.
       0.              0.              0.              0.              0.
   85551.25        46066.07            0.          58130.98            0.
       0.              0.              0.              0.              0.
       0.              0.              0.              0.              0.
       0.              0.              0.              0.              0.
       0.           3140.74         3407.64            0.           6857.19
    3208.79        55495.81       119335.49     ]

**non-seasonal order** :
(2, 0, 3)

**seasonal order** :
(1, 1, 1, 52)
user: pranavbahl2308
issue_3528: state: open
title: ENH: OLS with complex-valued data
body: original title: "Exception of RegressionResult.summary() when performing OLS with complex-valued data"

    import numpy as np
    import statsmodels.formula.api as sm
    
    x = np.linspace(0, 10, 100)
    # Edit: using a complex-valued x, for example "x = np.linspace(0, 10 + 5j, 100)" works too

    beta = 5. + 2j
    y = beta * x
    
    est = sm.OLS(y, x).fit()
    assert np.allclose(beta, est.params[0])  # OK
    
    print(est.summary())

I am performing an ordinary least square (OLS) on complex-valued data. The OLS regression works as expected. However calling the method 'summary()' raises the following exception: ``TypeError: ufunc 'chdtrc' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''``

statsmodels version: 0.8.0
numpy version: 1.11.2
Python 3.5.2
Windows 7 64 bits

user: nbedou
issue_3521: state: open
title: ENH: new convenience method fit_difficult
body: partially a follow-up to #3520 

AFAIR, Stata has something like "try harder" for difficult optimization problems in some models

We started thinking about switching in a squence of optimization methods and endogenous switching between methods, but this is currently messy. GLM has an option to use irls before fit_gradient.

A noninvasive way to get to this would be to introduce a new method `fit_difficult` that repeatedly calls the `fit` method. This should be a convenient "automatic" feature to try to identify and work around convergence problems, including non-positive definite hessian.
We could also add the option here about what to do with singular or near-singular score_obs, i.e. instead of just checking the exog, we check the full score array (including derivatives/score_factor and weights).
We could also switch to OPG if hessian is not invertible or not definite.

This would be mainly for models that use the scipy optimizers. I'm not sure about the other ones like RLM and QuantileRegression.

user: josef-pkt
issue_3520: state: open
title: FAQ/SUMM Design: What is the policy for estimation failures running many cases?
body: When we run many estimation problems in a loop, then it is possible that an estimation case does not work. In the loop we often don't want to stop if an exception or estimation failure occurs.

What is the general policy or what are the recommendations for this case?

the location where something can go wrong

- problems with the data like missing values or singular exog, or possibly also dtype issues
- optimization: convergence failure
- missing hessian, hessian not positive (semi-) definite
- other missing results (not clear if there are any)

Currently we don't have a general policy and stable API for many cases. Decisions are mostly case specific.
We refactored exceptions and warnings to be more specific, but unit test coverage for those is relatively low, and not guarantee yet that the exception or warning classes that are raised at specific failures are part of the stable API. Some should be.

How do we detect a failure:

- exceptions
- flags, attributes (with or without warning): e.g. `coverged` boolean, rank of `exog` or `cov_params`
- result not available (with or without warning): e.g. properties that are None or np.nan

How do we work around a failure:

- up to user: e.g. problems with data, insufficient data cleaning, better start_params
- inside statsmodels
  - work around missing results, e.g. protect summary, #3299 summary breaks if hessian is None
  - try harder to avoid exceptions, e.g. should we switch to OPG if hessian is not invertible, or should we regularize (pinv) as in linear regression models, more improvements to optimization like endogenous switching of algorithm
   - catch exceptions: a loop where we already to this is in the lag-search in ARIMA type models.

Can we identify the source of the failure:

- Are the messages from warnings or exceptions informative enough?
- diagnostics after failures ?
- general diagnostics for "validity", e.g. stricter convergence check including zero gradient, multicollinearity diagnostics, special case in tsa for stationarity assumption (e.g. exceptions or convergence failures in ARMA)

see sanity checks, possible transformations of data, general diagnostic checks.



 
user: josef-pkt
issue_3516: state: open
title: REF: deprecate f_test and related outside linear regression models
body: see #3119 for problem if df_xxx are not defined

df_xxx should be defined in general, but f_test shouldn't be valid and available by default. We have now wald_test as the proper generalized substitute.

I guess we should also deprecate fvalue outside of regression models instead of reinterpreting it like tvalues.

FAQ-D for use_t #1833

replacement for f_value and f_pvalue could be null_statistic/null_value and null_pvalue, but this still leaves it unclear whether we use likelihood ratio or wald based null test.  
Currently, fvalue uses llr (f-statistic based on ssr) or wald_test depending on cov_type='nonrobust'.
 
user: josef-pkt
issue_3509: state: open
title: ENH:  VARMA  causality invertibility constraints
body: maybe useful (I didn't look at it)

Roy, Anindya, Tucker S. McElroy, and Peter Linton. 2014. “Estimation of Causal Invertible VARMA Models.” arXiv:1406.4584 [Math, Stat], June. http://arxiv.org/abs/1406.4584.

There is also an older article for imposing stationarity, that I don't have access to
Ansley, Craig F., and Robert Kohn. 1986. “A Note on Reparameterizing a Vector Autoregressive Moving Average Model to Enforce Stationarity.” Journal of Statistical Computation and Simulation 24 (2): 99–106. doi:10.1080/00949658608810893.

AFAIK, we don't have stationarity restrictions for VAR/VECM parameter estimates.


user: josef-pkt
issue_3501: state: open
title: ENH: imputation, degrees of freedom, variance estimation
body: (random idea)

When we impute values, then they are not valid observations and will bias inference if we treat them as regular observations. 
Full imputation methods like MICE use random imputation to correct the inference for imputation.

possible case for simpler imputation is missing values and outlier adjustments in time series.

A simpler method to correct the inference would be to drop imputed values in the computation of some auxiliary statistics, like estimate of error variance, and correct degrees of freedom for the imputed values, i.e. count only actual values.
AFAIR, I haven't seen a reference that does this. However, this would be partially in analogy to dropping observations which we cannot easily do in time series analysis when we need observation with a fixed sequence or equal spaced index.

Implementation: We need to carry around either a mask or a weights array for non-imputed observations, and use it for computing the relevant statistics. Also effective nobs (and df_resid) will be different from nobs that counts all observations in the model (len(endog)).

If we have weights in the model, then we can just drop observations by setting the weights of them to zero. However, in tsa we don't have weights and imputing missing values or shrinking outliers can be done from the outside.

context: 
#3497 (simple outlier detection in tsa), 
#3492 binned equal spaced nonparametric regression, that would allow for weights

pandas has tsa imputation so users just impute values and then ignore that for estimation and inference

user: josef-pkt
issue_3500: state: open
title: How to model a mixed effects model with nested random effects
body: I would like to know if it is possible to fit a mixed random effects
model with two random effects in `statsmodels` if one of the random
factors is nested within the other.

A minimal example would look as follows: Given are two groups of `6`
random individuals. From each individual `5` random tissue samples have
been extracted.  Each tissue sample has been stimulated at different
intensity levels `x` ranging form `0` to `30`.  We would like to test if
there exist a difference between the groups in response to the stimulus.

So, we have two fixed factors: `group` and `intensity`.  And two random
factors: `subject` and `tissue sample`, where `tissue sample` is nested
within `subject`.

If `y_ijk (x_l)` denotes the response of tissue sample `k` of subject
`j` of group `i=0,1` at intensity `x_l`, we assume the following model
holds:

```
y_ijk ( x_l ) = μ + δ⋅i + α_i + β_i(j) + γ ⋅ x_k + ε_ijk
```

for an intercept `μ`, a group effect `δ`, a slope `γ`, and the random effects

`α_i    ~ N(0, σ^2_α)` for `i = 1,…,m`, and
`β_i(j) ~ N(0, σ^2_β)` for `j = 1,…,n`, and
`ε_ijk  ~ N(0, σ^2)` for `j = 1,…,n`.

The `α_1, …, α_m` are independent. For any given `i`, the `β_i(1), …,
β_i(n)` are independent, and all `ε_ijk` are independent.

**Q**: How do I model this mixed effects model with nested random
effects using `statsmodels`?

----

The above example as a minimal example in python:

```
from pandas import DataFrame

num_x = 15
num_s = 6
num_t = 5

x = np.repeat(np.linspace(0,30,num_x)[None], 2*num_s*num_t, axis=0).ravel()
group = np.hstack(( np.zeros(num_x*num_s*num_t, dtype = int),
                    np.ones(num_x*num_s*num_t, dtype = int)))
subject = np.repeat(np.arange(2*num_s), num_x*num_t)
tissue_sample = np.repeat(np.arange(2*num_s*num_t), num_x)

dat = DataFrame({'x':x, 'group':group, 'subject':subject, 'tissue_sample':tissue_sample})
dat.shape
dat.dtypes

factor_s = np.random.normal(scale=7, size=2*num_s)
factor_t = np.random.normal(scale=2, size=2*num_t*num_s)
error    = np.random.normal(scale=.3, size=2*num_x*num_s*num_t)

intercept = 5
group_effect = 6
slope = 0.3

dat['y'] = intercept + group_effect * dat.group + \
    factor_s[dat.subject] + factor_t[dat.tissue_sample] + \
    slope * dat.x + error

dat.shape
dat.dtypes
```

Plotting the data

```
# Plotting
import matplotlib
import matplotlib.pyplot as pt
import seaborn as sb

pt.close()
palette = sb.color_palette("muted", n_colors=2)
fig, ax = pt.subplots()
for variables,da in dat.groupby(['group', 'subject', 'tissue_sample']):
    ax.plot(da.x, da.y, '.-',
    c=palette[variables[0]],
    label='group %i' % group[0] )
ax.set_xlabel('Intensity')
ax.set_ylabel('Response')
```
user: 00tau
issue_3497: state: open
title: ENH: Add basic time series outlier functions
body: I've created a simple time series outlier detection model as I had proposed in #3285.

The processes is:
1. Remove any trend component using rlm then create 'residues'
2. Use a robust measure of dispersion (ie., MAD or IQR) on the residues to identify potential outliers.

This implements 
* isoutliers:  find outliers in a time 
* replace: replace outliers with NaN or an interpolated value from the neighbours.
* plot: chart the time series overlaying the trend line and the outliers as red stars  

While simplistic it work well enough and is a starting point. It might be interesting to extend this to use autoregression to create the residues instead of rlm.

As I'm new to creating code for statsmodels, I'm happy to talk any feedback on programming style and advice on what else is needed, e.g., additional documentation etc. 
user: Kevin-McIsaac
issue_3496: state: open
title: PoissonZiGMLE : predict not implemented?
body: I successfully run PoissonZiGMLE:

```
PZI = PoissonZiGMLE(df_zip['obs'],Xmat,offset=df_zip['offsetv'])
result = PZI.fit(maxiter = 1000)
print result.summary()
```

however when I try:

`result.predict(df_zip, offset=offsetv)`

I get this error:
```
---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
<ipython-input-83-bfcfbc12df01> in <module>()
----> 1 result.predict(df_zip, offset=offsetv)

/software/centos6/x86_64/canopy-1.7.4/Canopy_64bit/User/lib/python2.7/site-packages/statsmodels/base/model.py in predict(self, exog, transform, *args, **kwargs)
    747             exog = np.atleast_2d(exog)  # needed in count model shape[1]
    748 
--> 749         return self.model.predict(self.params, exog, *args, **kwargs)
    750 
    751 

/software/centos6/x86_64/canopy-1.7.4/Canopy_64bit/User/lib/python2.7/site-packages/statsmodels/base/model.py in predict(self, params, exog, *args, **kwargs)
    175         This is a placeholder intended to be overwritten by individual models.
    176         """
--> 177         raise NotImplementedError
    178 
    179 

NotImplementedError: 
```

Thanks!

user: chiaral
issue_3495: state: open
title: ENH: add AC autocorrelation robust cov_type 
body: some older methods in article use AC instead of HAC fore serial correlation without heteroscedasticity.

We or I mentioned it in some comments before, Stata has a AC version in a user package.
Since HAC is more robust that AC, it was not important that it is missing.

However: Using a pre-filter on time series will create serial correlation even if it is not in the original series. A moving average filter will introduce M-dependence.

context: non-parametric local estimation 
specifically #3492 where I think it is difficult or computationally expensive to get a HAC version, but should be easier to get an AC version for the residual variance estimate.

Related: variance of mean in one sample case with serial correlation for simple t-test or confidence intervals. There are explicit formulas, but we need generic implementation.

Note: HAC if exog is only a constant (ones) is the same as AC

user: josef-pkt
issue_3493: state: open
title: ENH: windowed regression using linalg ufunctions
body: one part of followup to #3492 binned local polynomial regression

In #3492 I copied the formulas from Fan Marron for local constant and local linear regression.
However, those are just explicitly solving a small linear system of equations for each observations.

We can do this more generally and fast by using the vectorized linear algebra functions.

I.e. use convolution (vectorized over moments - scipy signal instead of numpy ?) to get the moment equations for each observation, then use linalg solve or inv vectorized over observations.

compared to Savitzky-Golay this reverses the order of linalg solve/inv and the convolution. savgol does linalg first and then convolves. If we have weights as in the binned local linear regression, then we cannot convolve the linalg solution because it is observation or freq_weight specific.

difference to explicit solution in 3492 is that there we only solve for one of the parameters and don't need the full solution.
user: josef-pkt
issue_3492: state: open
title: WIP ENH: add binned local polynomial regression
body: this implements binning for local constant and local linear kernel regression following Fan and Marron 1994

It also adds a simple cross-validation with leave k-th bin out to choose the bandwidth
This undersmooths in small samples.

speed is very good, but there still might be a little bit of slack.
see https://groups.google.com/d/msg/pystatsmodels/KVmXLAJfMxU/lUaoV51oCwAJ

It works, but 
- API not settled
- options not tested
- needs better bandwidth choice in smaller samples, e.g. options for cross-validation setting, eg. evaluate at more points. 
   Maybe replace MSE with something less quadratic, to not be influence so much by large residuals.
- boundary handling
  If the half window is smaller than the number of explanatory variables, then the design matrix for the local regression is singular at the boundaries. I think this is the zero division warning when the window length was too small in the cross-validation, i.e. minimum window length is 3 for linear. 
- empty bins
  if the window length is small and there are empty bins, then the design matrix could also be singular if we don't have enough grid points with non zero weights.
- leave k-th out cross validation related to empty bins: Should we require k > window_length ?

This uses only convolution. Additional results like effective degrees of freedom are not available. (that's why I used cross-validation for choosing window length given that the algorithm is fast)
Auxiliary quantities for binned observations in Turlach and Wand 1996 use the local regression representation and I have not tried to derive those for a convolution version. (diagonal of hat/smoother matrix for local constant looks ok, but for local linear regression it requires that we solve analytically a partitioned inverse, which I didn't try.
**update** see #3493 for follow-up idea of combining convolution and linalg for regression, which should solve this issue because we can use the linear algebra version with x.T W x directly, which means we can also calculate cov_params (x.T W x)**(-1) and some assumption, like homoscedasticity, on the error variance.

As byproduct of cross-validation we get a out-of-sample (leave many bins out) estimate of the mean squared prediction error MSE). But it's not used yet outside of cross-validation.

My plan is to merge this as basic smoother, and leave additional bandwidth selection methods and results statistic for the future, when I have again a week or three available.
I also haven't looked yet for references for a convolution version for local polynomial regression with order higher than linear.

This is a companion to #3380 which focuses on variance estimation with almost equal spaced regressor and nonparametric mean function. 
One difference to #3380 and to Savitzky-Golay is that in this PR we handle frequency weights coming from the binning.



user: josef-pkt
issue_3491: state: open
title: exposure for linear models - endog is sum
body: This is mainly a conceptual issue for understanding exposure

related to discussion in
https://github.com/statsmodels/statsmodels/issues/2175#issuecomment-276165115
https://github.com/statsmodels/statsmodels/pull/3410#issuecomment-279194446

If endog is the sum of independent normal distributed variables, then we can compute the variance for the sum from the number of terms in the sum. This is the same as `exposure` in Poisson and GLM, i.e. aggregation property of LEF distributions.

I just saw that binned local polynomial regression of Fan and Marron uses endog as a sum
`endog = y * freq_weights`
The purpose is as computational shortcut, because it make some terms invariant.

This is the inverse analogue of `aweights` in Stats or var_weights that are justified when endog is an average.

This is not ENH. We don't need exposure in OLS/WLS because there are equivalent ways of modelling the same data. But we can use it for cross-checking with GLM exposure (once it is a generic feature and not an offset).



user: josef-pkt
issue_3490: state: open
title: Does Bayesian Change Point  fit in statsmodel? 
body: I've just completed an [analysis](https://github.com/Kevin-McIsaac/Nightlife/blob/master/9.1_Lockout_Law_Impact_on_Train_Usage.ipynb) of train data looking for impacts from Sydney's :lockout laws". For this analysis I found Bayesian Change Point to be highly effective. 

I did not find an implementation in any of the main repos. I found this [implemenation](https://github.com/hildensia/bayesian_changepoint_detection)  which worked increidably well for me. I think it would be useful for other users if this was included in a major package, to aid discovery, and perhaps the documentation and examples polished (which I can do) . 

Does this fit statsmodels or does if belong somewhere else, e.g., scikit learn.


user: Kevin-McIsaac
issue_3486: state: open
title: ENH: varying coefficents, models with a sparse penalized component
body: another application for being able to handle sparse categorical exog components in the models

similar to Theil/Ridge quadratic penalization but with sparse matrices.

example usecase: varying coefficient model but with neighborhood penalization on the difference, i.e. imposing smoothness or closeness in some distance measure.
e.g. outlier detection: assume sequential ordering of observation (#3484), introduce a dummy variable for each observation and penalize by second order difference.
However, this is nobs and nobs, nobs in dimension, so requires sparse approaches to be able to handle it.

We have many of the pieces
- GAM and TheilRidge have L2 penalization with general constraints "prior", but dense only
- MixedLM has some of the sparse structure for the case of variance components
- sparse regression is in the works, 
  - OLSAbsorb removes the categoricals which doesn't work for penalization
  - SparseOLS: having the entire exog in sparse might work for this, but inference might become dense  with larger than nobs, nobs cov_params, we still need reduced inference
  - adding a sparse component to standard models needs quite a bit of work


reference

Nyquist, Hans. 1991. “Restricted Estimation of Generalized Linear Models.” Journal of the Royal Statistical Society. Series C (Applied Statistics) 40 (1): 133–41. doi:10.2307/2347912.
section 7 on piecewise regression 


What I was thinking to get started

similar to link test (todo: I don't remember the details)
Use (pearson) residuals and regress on fittedvalues and observation dummies with sequential penalization. This is one sparse OLS. If there is no natural ordering or sequence of the observations, then try sorting by fitted values or some explanatory variable.
It's a similar idea to lowess scatter plot smoothing to see patterns in the residuals but a bit more formal and more flexible. 
see also #3380 for related nonparametric variance estimation.
(the same can be used for nonlinearity diagnostics, if we use varying coefficient instead of varying intercept.)


user: josef-pkt
issue_3485: state: open
title: ENH: hypothesis tests with up to three inequality constraints
body: (inequality constraints in wald test with multiple parameter is a difficult topic, open issues for it)

one computational detail

Shapiro, Alexander. 1985. “Asymptotic Distribution of Test Statistics in the Analysis of Moment Structures Under Inequality Constraints.” Biometrika 72 (1): 133–44. doi:10.2307/2336343.

page 140 (last part),141: special case  beta >= 0 for up to 2 or 5 constraints is not too difficult.
especially for 1 or 2 constraints we have simple formulas for the weight of the chisquare mixture.
There is additionally a correlation computation that I didn't look at.
But it seems to be independent of how many parameters we have.

i.e. we can 
- extent t_test to one-sided hypothesis, (is kind of easy)
- add one-sided wald test function that should support a few constraints, 
- more constraint in one-sided hypothesis test if the general code can be added.  
   Some references that use inequality constraints mention that Shapiro is fine up to 5, exact computation up to 10, and approximations beyond that. (I didn't look at any details

I didn't check how much this is applies to Wald, score/LM or LR test



user: josef-pkt
issue_3484: state: open
title: ENH: tools artificial one-dimensional ordering
body: (semi-random idea)

in time series we have sequential order, in cases like IPW and imputation we artificially order by a "treatment index" or some predictive closeness. For other cases we don't have an ordering index and need to compute neighborhood for each observation.
In some diagnostic test we still need to add sorting by provided exog variable.

General idea: There are many statistical methods that can be used with sequentially ordered observation. If we can create an artifial ordering , then we can extend the applicability of those methods.

This could apply more to exploratory data analysis and diagnostic checking where we don't need something that has complete statistical foundation and that can be a bit "ad-hoc".

example: jump or structural break detection, outlier detection or "time"-varying coefficients with local penalization
user: josef-pkt
issue_3471: state: open
title: ENH: nonparametric GLM, splines, variance function
body: mainly to park a reference 

related 
#2889 GLM dispersion models
splines, GAM, nonparametric, semiparametric, ...

Gijbels, I., I. Prosdocimi, and G. Claeskens. 2010. “Nonparametric Estimation of Mean and Dispersion Functions in Extended Generalized Linear Models.” TEST 19 (3): 580–608. doi:10.1007/s11749-010-0187-1.
using p-splines

IIRC, they have several articles, and I briefly looked at them a while ago, but I don't remember anything

user: josef-pkt
issue_3470: state: open
title: ENH: specification test multivariate to univariate - combining hypothesis tests
body: (random idea)

general helper function or pattern based on some specification tests for tsa VAR

There are many specification tests, like normality, autocorrelation, conditional heteroscedasticity/arch, based on univariate residuals of a model. For VAR, several specification tests use an orthogonal decomposition, either cholesky or eigen/singular value, and the apply the univariate tests and combine them into a single test statistic and p-value.

option: decomposition='chol' cholesky, eigenvalue, none

problem for test combination: 
- no decomposition doesn't create uncorrelated univariate tests
- more general: aside from normal distribution uncorrelated doesn't imply independence. Is uncorrelated enough or do we need independence? e.g. for tests of higher moments (I don't remember what assumptions are imposed in multivariate normality test. Doornik or Luetkepohl)

(scipy has some generic hypothesis test z-value or p-value combination functions.)

(related univariate and multivariate outlier detection or tests, variance estimation based on differencing for heteroscedasticity)
 
user: josef-pkt
issue_3462: state: open
title: REF deprecate lower case link classes
body: We still have several lower case link classes, several like log and logit are just dummy subclasses of the pep-8 conformant capitalized clases

#3370 changed the interface for links, so it requires specifying a link instance, which makes pep-8 class names more appropriate.

deprecate and make them go away in a few versions. AFAIR, we use lower case links heavily in examples, so they will be in common use.
user: josef-pkt
issue_3446: state: open
title: SUMM/ENH: LR type tests 
body: compare_lr, compare_jtest, ANOVA and related 
e.g. anova type tests base on deviance or pearson chisquare for GLM

I don't see a summary issue, but many individual issues

- #30 oldest issue for compare_xxx methods, still open
- #2673 test_lr, where we estimate both models
- #1358  RLM, implemented in PR #1341 
- GLM:
  - #825 request for anova_glm
  - #2688  more general model selection in GLM
  - #2688  lr-type for "quasi-"mle i.e. dispersion only, reference Cameron and Trivedi

- #1626 reference to Stata function and documentation
- #3363 misspecified models, sum or linear combination of chisquare random variables
- #2641  PR to move compare_lr_test higher up to base classes

general discussion on missing features in GLM, especially also related to scale handling

related profile likelihood

  
user: josef-pkt
issue_3434: state: open
title: DESIGN: AR, VAR, SVAR signature deprecations in 0.9
body: I think we need to start the deprecation process so we can make these models consistent with setting `order` in `__init__`.

Currently the signatures of `__init__` and `fit` in AR is:

- `sm.tsa.AR(self, endog, dates=None, freq=None, missing='none')`
- `sm.tsa.AR.fit(maxlag=None, method='cmle', ic=None, trend='c', transparams=True, start_params=None, solver='lbfgs', maxiter=35, full_output=1, disp=1, callback=None, **kwargs)`

and in VAR is:

- `sm.tsa.VAR(self, endog, dates=None, freq=None, missing='none')`
- `sm.tsa.VAR.fit(maxlags=None, method='ols', ic=None, trend='c', verbose=False)`

and in SVAR is:

- `sm.tsa.SVAR(self, endog, svar_type, dates=None, freq=None, A=None, B=None, missing='none')`
- `sm.tsa.SVAR.fit(A_guess=None, B_guess=None, maxlags=None, method='ols', ic=None, trend='c', verbose=False, s_method='mle', solver='bfgs', override=False, maxiter=500, maxfun=500)`

I propose that we do the following:

1. Add an optional `order` argument to the class constructor
2. If the `order` argument is not set, issue a `DeprecationWarning`
3. In the `fit` method, if `maxlag is not None` or if `ic is not None`, issue a `DeprecationWarning`.
4. Otherwise maintain existing behavior in `fit`, except that if `maxlag is None` but `order` was set in the class constructor, `maxlag` is set to that `order` value.
5. Change the docstring to note the deprecation.

Examples below. I don't know how deprecations are best performed - i.e. we can issue a `DeprecationWarning` but that is ignored by default in Python.

Also, I don't know if I should specify a version number? Maybe 0.10? Or is 0.9 okay?

**Examples**:

Start with

```python
import warnings
import statsmodels.api as sm
warnings.simplefilter('always')
endog = [0] * 10
```

**Construction**

```python
mod1 = sm.tsa.AR(endog)
```
```
ar_model.py:95: DeprecationWarning: Specifying the AR order in `fit` is deprecated.
Please specify it when constructing the model using the `order` argument
  " using the `order` argument", DeprecationWarning)
```

and no warning is issued if instead:

```python
mod2 = sm.tsa.AR(endog, order=1)
```

**Fitting**

Now, if `order` was not specified in construction, we will have the following behavior for `fit`:

```python
mod1.fit()
# Here no further warnings are issued, since this is a perfectly good `fit` call, even after the
# deprecation is complete, and they've been warned about needing to set `order` when they
# constructed the class
```

```python
mod1.fit(maxlag=2)
```
```
ar_model.py:557: DeprecationWarning: Specifying the AR order in `fit` using the `maxlag`
argument is deprecated and will be unavailable in a future version. Please specify the exact
order when constructing the model using the `order` argument
  " `order` argument", DeprecationWarning)
```

```python
mod1.fit(ic='aic')
```
```
ar_model.py:566: DeprecationWarning: Selecting the AR order by minimizing information criteria
in `fit` is deprecated and will be  unavailable in a future version. Please specify the exact order
when constructing the model using the `order` argument
  " the `order` argument", DeprecationWarning)
```

```python
mod1.fit(maxlag=2, ic='aic')
```
```
ar_model.py:557: DeprecationWarning: Specifying the AR order in `fit` using the `maxlag`
argument is deprecated and will be unavailable in a future version. Please specify the exact
order when constructing the model using the `order` argument
  " `order` argument", DeprecationWarning)
ar_model.py:566: DeprecationWarning: Selecting the AR order by minimizing information criteria
in `fit` is deprecated and will be  unavailable in a future version. Please specify the exact order
when constructing the model using the `order` argument
  " the `order` argument", DeprecationWarning)
```
user: ChadFulton
issue_3419: state: open
title: ENH:  nearest_cov_toeplitz
body: We have a collection of function to obtain a nearest positive definite or factored covariance or correlation matrix (added in #631 #1384 )

Similarly, the following article (reference in #3417) has a nearest_toeplitz algorithm

Tecuapetla-Gómez, Inder, and Axel Munk. 2016. “Autocovariance Estimation in Regression with a Discontinuous Signal and M-Dependent Errors: A Difference-Based Approach.” Scandinavian Journal of Statistics, January, n/a-n/a. doi:10.1111/sjos.12256.

looks like a simple iterative solution that combines projecting on positive semidefinite (via eigenvalue clipping) and projecting on toeplitz (looks like averaging diagonals and off diagonals).
(but I had thought that using autocovariances always provides a positive semi-definite toeplitz matrix)
user: josef-pkt
issue_3418: state: open
title: ENH: reverse TOST, testing equivalence, intervall null hypothesis
body: equivalence testing has equivalence as the alternative hypothesis to shift the "burden of proof"

However, I have seen now articles that use equivalence  `abs(params - value_null) <= delta` as the Null hypothesis. 
Do we have this already? 
Can we just reverse TOST?
Is it equivalent to score confidence intervals?

e.g.
https://github.com/statsmodels/statsmodels/issues/3417#issuecomment-277417297
another recent article (?) was using it similar to non-equivalence testing of models similar to GOF tests

I didn't look at any details, not read the articles.

related: one sided hypothesis testing in regression models #1193 (if only one parameter of interest)

user: josef-pkt
issue_3417: state: open
title: ENH: change or break point detection
body: in sample change point detection, many break points with piecewise constant mean

the following and citing literature look interesting and have R packages
but 86 pages is too long too print and read, mentions dynamic programming and sounds a bit like Bai/Perron, they mention exponential family and log-like, so it looks like it's extended to GLM 
Frick, Klaus, Axel Munk, and Hannes Sieling. 2014. “Multiscale Change Point Inference.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 76 (3): 495–580. doi:10.1111/rssb.12047.

The following allows for autocorrelated, m-dependent (MA) errors and has change point detection as an application
this is related to difference based variance and autocovariance estimation #3392
Tecuapetla-Gómez, Inder, and Axel Munk. 2016. “Autocovariance Estimation in Regression with a Discontinuous Signal and M-Dependent Errors: A Difference-Based Approach.” Scandinavian Journal of Statistics, January, n/a-n/a. doi:10.1111/sjos.12256.

related: 
- Chad has Bai/Perron from his first GSOC,  issue ?
- issue #3295 for predictive, out of sample anomaly detection in control chart context
- #2634 segmented regression, with connected lines or splines instead of jumps
- jump detection in non-parametrics regression, no issue yet, but was another application for difference based variance estimation #3392

user: josef-pkt
issue_3415: state: open
title: ENH: predict_cdf predict_prob GLM gof
body: (random idea for evaluating how well a distribution matches the data)

In GLM models like tweedie or gamma or Poisson, the error is not additive as in a gaussian model (linear or nonlinear least squares), instead we specify the distribution of endog given exog directly.
If we want to predict more than the mean, then we need a distribution of the predictions.

relatively simple would be the cdf(y | x), similar to discrete probability (pmf) in Poisson e.g. in Vuong test.
e.g. What's the population prediction for observing endog > some_number, which can compare model implied with data frequencies.

this would be mainly a wrapper around two things:
- #106 getting distribution instance after model fit (I don't know how much this is already available)
- #2712 nonlinear prediction

extras: 
- can we get something like a probability plot for this.
- for discrete or interval probabilities we can also create a classification or confusion table.
user: josef-pkt
issue_3407: state: open
title: TST get_prediction is missing unit tests for keyword options
body: #3406 fixes a bug with exog when weights is not defined

AFAICS, the unit test only verify the results for the basic case without arguments, but don't check that arguments are correctly handled.

for example `offset`: It looks like the code is correct, we use it in predict but not for covariance estimation (offset is assumed known not random). But no unit tests, so there is no guarantee it actually works or works correctly.
user: josef-pkt
issue_3400: state: open
title: Errors in statsmodels.nonparametric.kde.KDEUnivariate
body: **edit** the numpy compatibility has been resolved in 0.8, but the last comment might indicate some numerical problems, with the example as test case.


1. This code
> nobs = 300
> np.random.seed(1234)  # Seed random generator
> dens = sm.nonparametric.KDEUnivariate(np.random.normal(size=nobs))
> dens.fit()
> plt.plot(dens.cdf)
> plt.show()

generates error:
> /usr/local/lib/python3.5/dist-packages/statsmodels/nonparametric/kdetools.py in revrt(X, m)
>      18     if m is None:
>      19         m = len(X)
> ---> 20     y = X[:m/2+1] + np.r_[0,X[m/2+1:],0]*1j
>      21     return np.fft.irfft(y)*m
>      22 
> 
> TypeError: slice indices must be integers or None or have an __index__ method

May be this is correct?
> y = X[:int(m/2+1)] + np.r_[0,X[int(m/2+1):],0]*1j

2. This code
> dens.fit(kernel='epa', fft=False)

generates error:

> /usr/local/lib/python3.5/dist-packages/statsmodels/nonparametric/kde.py in kdensity(X, kernel, bw, weights, gridsize, adjust, clip, cut, retgrid)
>     332         # handle weights
>     333     if weights is None:
> --> 334         weights = np.ones(nobs)
>     335         q = nobs
>     336     else:
> 
> /usr/local/lib/python3.5/dist-packages/numpy/core/numeric.py in ones(shape, dtype, order)
>     190 
>     191     """
> --> 192     a = empty(shape, dtype, order)
>     193     multiarray.copyto(a, 1, casting='unsafe')
>     194     return a
> 
> TypeError: 'float' object cannot be interpreted as an integer

If in file `/usr/local/lib/python3.5/dist-packages/statsmodels/nonparametric/kde.py` the line 327: `nobs = float(len(X))` replace to `nobs = len(X)` and run `plt.plot(grid, kernel_density.evaluate(grid))`, generates next error:

> /usr/local/lib/python3.5/dist-packages/statsmodels/sandbox/nonparametric/kernels.py in isInDomain(xy)
>     171             """Used for filter to check if point is in the domain"""
>     172             u = (xy[0]-x)/self.h
> --> 173             return u >= self.domain[0] and u <= self.domain[1]
>     174 
>     175         if self.domain is None:
> 
> ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()


statsmodels.\_\_version\_\_ = 0.6.1
numpy.\_\_version\_\_ = 1.12.0

user: nikitavolkov21
issue_3399: state: open
title: ENH/Performance: Binning for local nonparametric estimation
body: Similar to the gridded kernel density estimation, we can also use binning for fast evaluation of local polynomial regression, kernel regression or similar.

related #3380 equal spaced versus unequal spaced local regression or smoothing

I found these mentioned in an article (but didn't look much at them. I also didn't search to see if there is anything newer.)

Fan, Jianqing, and James S. Marron. 1994. “Fast Implementations of Nonparametric Curve Estimators.” Journal of Computational and Graphical Statistics 3 (1): 35–56. doi:10.2307/1390794.

Hall, Peter, and M.P. Wand. 1996. “On the Accuracy of Binned Kernel Density Estimators.” Journal of Multivariate Analysis 56 (2): 165–84. doi:10.1006/jmva.1996.0009.

Turlach, B. A., and M. P. Wand. 1996. “Fast Computation of Auxiliary Quantities in Local Polynomial Regression.” Journal of Computational and Graphical Statistics 5 (4): 337–50. doi:10.2307/1390888.

user: josef-pkt
issue_3398: state: open
title: ENH/CLN orthogonal polynomials density estimation
body: What's the status of density estimation with orthogonal polynomials?
I don't see an open issue

3 "People also read" articles

Jiang, Min, and Serge B. Provost. 2011. “Improved Orthogonal Polynomial Density Estimates.” Journal of Statistical Computation and Simulation 81 (11): 1495–1516. doi:10.1080/00949655.2010.492781.

Jin, Tao, Serge B. Provost, and Jiandong Ren. 2016. “Moment-Based Density Approximations for Aggregate Losses.” Scandinavian Actuarial Journal 2016 (3): 216–45. doi:10.1080/03461238.2014.921640.

Provost, Serge B., and Hyung-Tae Ha. 2016. “Distribution Approximation and Modelling via Orthogonal Polynomial Sequences.” Statistics 50 (2): 454–70. doi:10.1080/02331888.2015.1053809.

user: josef-pkt
issue_3392: state: open
title: SUM/ENH: extensions and applications of differencing-based variance 
body: This is a companion to #3384 and follow-up to #3380 
(there should be several references to each topic)

missing in #3380: 

- edge windows: currently this truncates, i.e. 'valid' in terms of convolution
- inference: several papers have the asymptotic distribution of the variance estimate sqrt(nobs), normal
- not checked: repeated exog, e.g. discrete points, calculating variance for each repeated exog is better.
  (this is different from repeated measures or panel data with cluster correlation, sometimes not clear from abstracts)
- weights ???  e.g. needed for repeated exog, between groups estimator if individual observation is an average with different subsample/group sizes. (I didn't think about what would be needed, but it would break that we can use the same filter coefficients for all observations.)  
  Related: outlier robustness weights (Just rescaling the data is in WLS/OLS might not work for computing differences, but maybe weighted averaging of pseudo-residuals will work, similar to using MAD instead of average squares. Using MAD on pseudo-residuals looks also good in a few examples that I tried.).

application and extensions

- estimated variance for bandwidth selection in mean smoothing (not looked at) (e.g. scipy splines require user provided variance for smoothing)
- inference for mean function (not looked at): mean function and variance estimate have different trade-offs between bias and variance in bandwidth selection. separate variance estimate should improve inference (e.g. confidence intervals).
- variance function estimation, assuming parametric or non-parametric heteroscedasticity
- autocovariance, autocorrelation for m-dependent processes, need m-lag (similar to seasonal differencing)
- technical: allow choice of which pairs of observations are used, like m-lag or between group estimator don't have differencing across consecutive ordered observations.
- partial linear model: differencing removes effect of nonlinear component and we can estimate linear component with standard estimators at standard asymptotic rates.
- multi-dimensional, especially for two or low dimensional grid: useful for spatial components
- specification tests, e.g. for non-linear form   ???
- ...




user: josef-pkt
issue_3391: state: open
title: FAQ-D Performance: all pairwise, median, partition, U-statistics
body: statistics that are based on all pairs of observations have computational shortcuts for median and partition. However, those algorithms are pretty complicated.

One idea as simple compromise: create all unique pairs (for abs functions) with triu_indices and use numpy median or partition which don't require full sorting.

needs timing, should be faster than full sort, but still requires full memory for half of all pairs (nobs * (nobs +1) / 2 if main diagonal is included, and computing all pairs.

example:  
- median( abs(x_i - x_j))  for all i, j, with i!=j    (robust variance estimator)
- the corresponding standard u-statistic would be  abs(x_i - x_j).mean()

related to last example: single pass algorithm would save memory with iterator
How fast is python's reduce for iterators `reduce(add, (x[i] - x[j]) for i in range(nobs) for j in range(i))` or something like this? e.g. for one or many million pairs.

user: josef-pkt
issue_3388: state: open
title: ENH: heteroscedasticity robust standard errors in tsa VAR
body: interesting question (I thought about it before but never looked it up)
http://stats.stackexchange.com/questions/257580/heteroscedasticity-in-var-residuals

for VAR
Brüggemann, Ralf, Carsten Jentsch, and Carsten Trenkler. 2016. “Inference in VARs with Conditional Heteroskedasticity of Unknown Form.” Journal of Econometrics 191 (1): 69–85. doi:10.1016/j.jeconom.2015.10.004.

for VARMA
Boubacar Mainassara, Y., and C. Francq. 2011. “Estimating Structural VARMA Models with Uncorrelated but Non-Independent Error Terms.” Journal of Multivariate Analysis 102 (3): 496–505. doi:10.1016/j.jmva.2010.10.009.

Stata doesn't have any vce choice for VAR, AFAICS. Based on a brief google search, EViews and others also don't have it directly for VAR. (e.g. EViews recommendation on their forum is to use a generic system estimator for this, but that will not have the VAR post estimation features, like impulse response functions.)

AFAIK: HAC would not be valid because autocorrelation would render OLS estimation of VAR model inconsistent.

brief look at the two articles: HC sandwich has the standard form in terms of score and Hessian, but explicit calculation in VAR require lots of Kronecker products. 

user: josef-pkt
issue_3384: state: open
title: SUMM/FAQ: difference based variance estimators
body: We can estimate the variance based on the difference between observations y_i - y_j if they have the same mean, or we can remove the effect of a changing mean.

examples: 

- robust variance estimate: median of all pairwise differences for iid observations with outliers  
  reference ??? issue: ???  
  (because of median there is an efficient nobs log(nobs) instead of nobs**2 algorithm, but too complicated for me)
- nonparametric estimation based on differences with locally constant mean function #3380

variation: estimating (auto-)covariance based on differences


some usability comments based on the importance of the assumption (mainly #3380):
(from what I understand)

iid assumption
AFAICS: 
- (a) identical mean and independence are crucial assumptions, 
- (b) identical variance looks less important, we just get the average or median variance with heteroscedasticity.

because of (a) the outlier robust median pairwise difference will be less robust to violation of iid than other outlier robust estimators. e.g. all pairs combines differences across different lags which will not be identical if the are local effect (local changes in mean, or conditional differences, i.e. neighborhood effects)

nonparametric in #3380 
Hall, Kay and Titterington assume local constant mean, where "local" refers to the neighboring points in the sample. asymptotic assumption: design points, observations of explanatory variable get closer together so effect of mean disappears locally.
If we get this local constant behavior, than HKT differencing is optimal.

Polynomial differencing: 
Dette, Munk and Wagner assume local polynomial behavior. This works well if the polynomial corresponds to the local curvature, but is inefficient (asymptotically very inefficient) if HKT is appropriate, i.e. in the case of local constant mean.

related: small or large sample refers to neighborhood not to total sample size. An assumption on compact support for the explanatory variables is not "innocent".
e.g. estimate a sin wave on an interval with increasing number of points versus observing a sine wave where the support expands with the sample size and locally we still have the same number of points.

(extra: the same will apply for any asymptotic inference, but I ignored inference so far)








user: josef-pkt
issue_3380: state: open
title: WIP ENH: add nonparametric variance estimator based on differencing
body: This sounded useful when I stumbled on it, so I had to give it a try. It seems to work pretty well and the approach is becoming a bit more popular in recent years, although citation numbers are not very high.

y = f(x) + e  where e ~ (0, \sigma^2)
f(x) is an unknown function.

If we estimate f(x) nonparametrically with a optimal bandwidth, then the bias is too high to get a good variance estimate in small samples. The alternative is to smooth in such a way that we get a good estimate for the variance. One such estimator is based on differencing the series (ordered with respect to x).

implementation is based on description in 
Dette, H., A. Munk, and T. Wagner. 1998. “Estimating the Variance in Nonparametric Regression—what Is a Reasonable Choice?” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 60 (4): 751–64. doi:10.1111/1467-9868.00152.

but I read or skimmed also a few more articles.
This PR also provides the residual for the smoothed variance estimation, so we could use another nonparametric or semi-parametric estimator for estimating a variance function depending on x.

right now it's mainly 3 ways to calculate the same variance, the simplest and should be fastest is using convolve which assumes equal spaced x or sufficient smoothness. Unequal spaced x can be handled with a sparse window filter, where I took part of the sparse matrix handling from the current hp_filter.

unit tests: just compare 3 different implementation to produce exactly the same number. Estimated variance looks close to true, data generating variance.

Usage Recommendation based on Dette, Munk and Wagner:

- polynomial differencing with order 2 is pretty robust for different settings, i.e. use it if not much prior information about the shape of the underlying mean function
- polynomial differencing with order 2 or larger has good performance if the underlying mean function fluctuates strongly
- polynomial differencing might underestimate variance with low fluctuation models
- Hall, Kay and Titterington (HKT) optimal differencing works well with small fluctuations
- HKT has good large sample properties with increasing order
- HKT is not very good if fluctuations are large

small sample in Dette, Munk and Wagner is 25 observations, other small sample behavior for nobs <=100.

I think what this implies is that 
- for an original data series with possibly considerable mean fluctuation, we want to default to ("poly", 2)
- for residual series, e.g. residuals of a regression model, we want to use (HKT, 2) or increasing order with large nobs.

not yet checked: references on using this for nonparametric test of the mean function. (might be just like an f_test comparing ssr with var based on differencing estimator.
Dette, Munk 1998? 



user: josef-pkt
issue_3378: state: open
title: ENH - multiple response categorical variables
body: Here's my initial port of the [MRCV R](https://cran.r-project.org/web/packages/MRCV/MRCV.pdf) package into Python. It focuses on the functions from the [2004 Bilder and Loughin paper](https://scholar.google.com/citations?view_op=view_citation&hl=en&user=PpsqFosAAAAJ&citation_for_view=PpsqFosAAAAJ:9yKSN-GCB0IC). 

I haven't done proper documentation yet (though I do have docstrings on everything)...I'm hoping to get some initial impressions so I don't spend time documenting details that may change. I did start on a Jupyter notebook that walks through most of the functionality. It's incomplete but you can [see it here](https://github.com/rogueleaderr/statsmodels/blob/ENH-multiple-response-categorical-variables/statsmodels/stats/MRCV%20Table%20Documentation.ipynb)

I verified that all of the results match the output from the R library for the test dataset I'm using (which is from a survey conducted by my employer in the fall; the data is included as a new dataset). I tried to think of corner cases that would break the functions and covered all the ones I can think of. Please let me know if you think of any other corner cases.

The MRCV library provides three types of tests: 

1) Bonferroni corrected
2) Second order Rao Scott corrected
3) Bootstrapping

I've only done a partial implementation of the bootstrap method because at a first cut, it was much too slow to use in real time analysis (which is my primary use case). I'm hoping to find a way to make it faster and flesh it out in the future.

A couple implementation decisions I'm not 100% sure about:

1) All of the tests need to retain the original data to test for independence (i.e. you can't  just feed them a pre-tabulated table like you can with the existing Table class). So I introduced a new `Factor` class with some built-in data munging operations and metadata. I'm not sure if that was overkill (or if it should perhaps have a less generic name like `ContingencyTableFactor`.)

2) In theory, this methodology can be used on three MRCV's at a time (e.g. a table with one row variable and two column variables). I didn't have the bandwidth to handle that case right now. But I want to allow for the possibility in the future. So I'm making the `MRCVTable` constructor accept row_factors and column_factors as a list but then enforcing that each list has only one item. I know that's weird, but I'm not sure how else to do it that allows allowing extra factors in the future without requiring a breaking API change if I do.


Thanks for taking a look. I'm very open to whatever feedback you have. Also please let me know if you'd like me to do detailed documentation before you have a first review.

(Also I know the commits are a mess...I'll rebase once I get the thumbs up)

user: gjlondon
issue_3375: state: open
title: ENH API survey weights
body: (late night random thought, there are related issues but no time)

proposal/possible API for using prob_weights with survey data, and given that we don't have Stata_like prefix syntax.

define weights in a model `__init__` as
`prob_weights = Survey(weights, strata=None, cluster=None)`

where Survey is a class to hold information about survey sampling.

This uses the weights for weighted estimation of parameters and switches the default cov_type to the relevant survey sandwich, if cov_type='nonrobust' is unchanged. Issue warning that we are switching the default cov_type. Other cov_types are allowed and used. 
We don't override cov_type for prob_weights without survey instance, so in that case users can still use the non_robust or any other cov_type. 

basic version
strata is univariate and encodes the full partition of the population
cluster in case of nested clusters is the largest cluster unit i.e. top level in nesting

extra:
convenience: strata corresponds to a multi-index and we take the intersection
multi-dimensional clusters: in prob_weights Survey or required to be specified explicitly in cov_type?

more: 
survey panel data could also have time index and serial correlation also, so we would need survey + panel robust cov_type. (I guess that ends up using a combination of HC with strata correction and any of the correlation robust cov_types.


user: josef-pkt
issue_3372: state: open
title: SUMM/ENH: clustered data: a few large unordered clusters
body: The related old question is in #2136
There are several related questions on panel data, and panel and cluster robust standard errors.

Basic question: What estimation methods can we use if we have clustered data without time or other ordered structure, with within correlation (unobserved heterogeneity) and number of clusters G is small but number of observations per cluster M_g is large.
i.e. M_g asymptotics for G fixed.

In panel, longitudinal and possibly repeated measures data the within cluster observations are ordered, M_i is commonly the number of periods T_i.
(repeated measures data might not be ordered as in experiments with random timing of the treatment.)

we have e.g. #1201
- cluster robust cov_types G -> inf,   M fixed or also M -> inf, main asymptotics with respect to large G
- panel robust cov_types T_i -> inf, G fixed (maybe G -> inf), use time series properties

But we don't have anything corresponding to panel cov_types or models if observations in clusters are not ordered. 

Wooldrige cross-section, panel textbook 2010, chapter 20 on survey sampling, section 20.3.4 explicitly looks at this case.
references cited (I didn't check them)
Hansen 2007: G -> inf asymptotics if also M -> inf  see #1201
Moulton 1990: raises issue that cluster robust doesn't work with large clusters
Donald and Lang 2007: use between OLS but with nobs = G and t-distribution. (very conservative)
    this sounds a bit like repeated measures ANOVA. 
    Wooldridge sounds critical. If M_i are really large, then the group means or group specific regression already have large M asymptotics. recommends minimum distance estimator instead (I didn't check)
Loeb and Bound 1996: minimum distance based on regression by cluster ?

common side issue in Wooldridge: It's often possible to estimate (population or sample) average effects, average marginal or average partial effects, even if the full model is not identified.

aside: The description for the minimum distance approach sounds a bit like `fama_macbeth` although there the second stage is in a time series context. (I don't remember what Peterson criticized.)

There are also bootstrap articles for inference with a small number of clusters, but I don't remember if they work with large M.




user: josef-pkt
issue_3371: state: open
title: ENH: add var_weights to GLM
body: PR #2879 was intended to add weights but ended up adding mainly test cases and no new weights.

To make this more digestible it will be better to start with more limited target

- add var_weights in `GLM.__init__` 
- rename internally the current freq_weights to `iweights` and 
  - define iweights as product var_weights * freq_weights
- use None for weights that are not used (and add many `if xxx_weights are None:`

minimal result: replicate var_weights for case when endog is an average and match R and Stata for params_table. Family Gaussian with var_weights should also replicate WLS (except for use_t default).

extra results (those are the messy parts)

- loglike and similar: var_weights behaves differently from the other weight types. All others just multiply loglike_obs. 
- resid_pearson and similar: My guess: Those should include var_weights, but not freq and other weights  (the latter is unchanged from now). Alternatively we need wresid_xxx in analogy to WLS.
  maybe: resid (unweighted), wresid (var_weighted), resid_pearson and resid_anscombe (var_weighted).
  open: resid_deviance (it's likely that var_weighted is more useful).
  open: resid_xxx are cached attributes. Do we need a new resid method that can take arguments like use_weights, but without caching?

resids and other attributes are mainly intermediate results where the definition is mostly for convenience. The actual use and test cases need to go on some final results. E.g. outliers in anscombe, scale computation with resid_pearson and resid_deviance, LR and deviance tests for comparing models, resids for diagnostics and regression plots and so on.

score_factor, score_obs and hessian_factor: those might all be multiplicative with `iweights`.

question: Are there differences with respect to weight types between canonical and noncanonical link functions?


aside: stats.stackexchange has several questions on weights and glm for R. Several of those about the limitation for outlier robust weights and IPW.






user: josef-pkt
issue_3369: state: open
title: REF/ENH/DOC: arma_generate_sample API confusing
body: see http://stats.stackexchange.com/questions/255206/zero-lag-for-arma-model

Having to specify the lag polynomial for the ARMA process instead of the coeffs as in the regression is confusing.

It's "natural" to work with the lag polynomials in the context of looking at the processes, which I did when I wrote this, but it is not obvious in the context of estimation. I have seen the standalone arma_generate_sample function now mainly used for regression/estimation examples.

One possibility: add a new function like `arma_simulate` that uses ar_coeffs and ma_coeffs instead of the lag polynomials as arguments.


Related aside: Do we still have example for working with lag-polynomials? I don't see any in a brief search.
e.g. multiplicative Lag polynomials can be obtained by just multiplication of the corresponding polynomials.

user: josef-pkt
issue_3368: state: open
title: SUMM/ENH: granger causality tests
body: (we have a few issues open for granger causality tests for both the bivariate and the VAR version, and some changes in the VECM PR)

- #3241 enhancements to bivariate grangercausalitytests
- #379#issuecomment-6995396 loop for multiple testing, other problems might largely be addressed in VECM PR

- #1101 frequency domain version in nitime

new:

grangercausalitytests has unused add_constant keyword (raises if False)

**indirect granger causlity in VAR**

I never thought about the indirect effect before. I think we have mostly either  
`B does not directly granger cause A` or  
`B does not granger cause any of the other variables`, but there is also an indirect effect 
`B does not directly granger cause A and  
(either B does not Granger cause C or C does not directly granger cause A)`  
see the answer here
http://stats.stackexchange.com/questions/187706/does-the-granger-causality-test-in-the-vars-package-make-sense/192612#192612
It looks a bit messy to test for both no direct and no indirect effects especially when there are more than 3 variables in the VAR. (maybe it's some intersection-union test?)

user: josef-pkt
issue_3367: state: open
title: Logit returns nan p-value for some coefficients
body: Why do some Coef. have nan Std.Err, z, P>|z| and CI? 
I believe this is a bug.

```
                         Results: Logit
=================================================================
Model:              Logit            Pseudo R-squared: 0.192     
Dependent Variable: call_value       AIC:              13522.0051
Date:               2017-01-08 20:17 BIC:              13582.4183
No. Observations:   14068            Log-Likelihood:   -6753.0   
Df Model:           7                LL-Null:          -8361.4   
Df Residuals:       14060            LLR p-value:      0.0000    
Converged:          1.0000           Scale:            1.0000    
--------------------------------------------------------------------
       Coef.     Std.Err.       z       P>|z|      [0.025     0.975]
--------------------------------------------------------------------
x1    -0.2207         nan        nan       nan        nan        nan
x2     0.0377         nan        nan       nan        nan        nan
x3    -2.4384         nan        nan       nan        nan        nan
x4     1.3972         nan        nan       nan        nan        nan
x5     0.7828         nan        nan       nan        nan        nan
x6    -0.0561      0.1487    -0.3771    0.7061    -0.3476     0.2354
x7    -1.3261      0.5234    -2.5338    0.0113    -2.3520    -0.3003
x8     5.8230      6.9486     0.8380    0.4020    -7.7960    19.4421
x9     0.1786      0.1668     1.0712    0.2841    -0.1482     0.5055
=================================================================

```
user: chananshgong
issue_3363: state: open
title: SUMM/ENH: LR like test for misspecified models, sum of chisquare distribution
body: (I ran into this now 3,4 times in the last months for different applications. I might have some experiments from years ago.)

For QMLE or estimating equations or GMM under mis- or incompletely specified assumptions (no "true" likelihood)

Wald test with robust sandwich covariance has still standard chisquare distribution.
LR or anova type tests have asymptotically usually the distribution of a linear combination of chisquare distributions.

Although LR/anova type tests have non-standard distribution (and is not pivotal), it is still used in several areas of statistics. I don't remember having seen an actual application in econometrics.

- (outlier) robust estimator: I don't remember reference
- ANOVA with survey weights or marginal contingency tables #3334
- GEE: anova or lr test, see reference Fan and Zhang below


Fan, Chunpeng, and Donghui Zhang. 2014. “Robust Small Sample Inference for Generalised Estimating Equations: An Application of the Anova-Type Test.” Australian & New Zealand Journal of Statistics 56 (3): 237–55. doi:10.1111/anzs.12081.

AFAIR, the outlier robust literature referred to articles with the full linear combination of chisquare distribution, e.g. via characteristic function.
The Anova papers for contingency tables use 1st or 2nd order approximation.
The GEE article uses a 2nd order approximation (based on partial skimming of article)

Implementation
The weights in the linear combination of chisquare random variables are the eigenvalues of a data dependent matrix. This matrix is application specific and not always explicitly given in some articles (especially in econometrics when I looked at it in the past).
First order approximation in the special case of contingency tables can avoid computing the full matrix for the eigenvalues.

So we need a function just for the distribution, with p-values based on different degrees of approximation.
We also need the model specific code for the matrix for which the eigenvalues are computed.
We can also use special case functions in cases where there is a model specific shortcut that avoids the full matrix.

user: josef-pkt
issue_3361: state: open
title: ENH: LCA Latent class modelling : finite mixtures for multivariate categorical data
body: (I never heard about or looked at this)

stackoverflow question: http://stackoverflow.com/questions/41488795/what-is-the-proper-way-to-perform-latent-class-analysis-in-python

some information in this 
http://stats.stackexchange.com/questions/122213/latent-class-analysis-vs-cluster-analysis-differences-in-inferences

a blog example R and SAS
http://sas-and-r.blogspot.ca/2011/01/example-821-latent-class-analysis.html

UCLA examples use mostly MPlus, e.g. examples for a text book
http://www.ats.ucla.edu/stat/examples/alca/

pennstate has a webpage and authors of a recent book, and code for Stata and SAS
https://methodology.psu.edu/latentclassbook
documentation for it https://methodology.psu.edu/sites/default/files/software/lcastata/Stata%20LCA%20Plugin-v1.2b.pdf
refers to Latent Gold which has (what looks like) a good technical documentation
http://www.statisticalinnovations.com/user-guides/
http://www.statisticalinnovations.com/wp-content/uploads/LGtecnical.pdf

aside: section 3.3 of  Stata%20LCA%20Plugin-v1.2b.pdf
"The LCA Stata Plugin accommodates clusters and weights using the
pseudo-maximum-likelihood approach (Skinner, 1989; Vermunt & Magidson, 2005b, pp.
98-100). Under this approach, sampling weights are first standardized to have an average value
of 1 over all of the individuals being analyzed; they are then used as if they were frequency
weights in calculating the estimates. Clustering is ignored for estimation purposes, but is taken
into account in calculating standard errors by using a “robust” or “sandwich” style covariance
estimate.
"
related to survey sampling, analysis of complex survey data #3335

---

based on browsing for an hour:

this is mainly some version of multinomial logit with finite mixture random effects/parameters.
popular estimation methods seems to be a EM variant.
but other methods are possible http://www.john-uebersax.com/stat/faq.htm#est
using log-linear model mentioned there might be interesting

related: Sounds a bit like discrete choice model with random effects or random parameters with finite support mixing distribution.

The term "latent class models" for finite mixture random effect is also used in econometrics for standard univariate endog. This just a special case for random effects models, e.g. latent class poisson. It might be a bit more difficult if all parameters can change because the parameter space increases with k_params_random * k_support. 

Besides the finite mixture probability estimation, the main issue will be which "base" model to use for the multivariate categorical or ordered endog.
user: josef-pkt
issue_3359: state: open
title: SUMM/FAQ-D: Refactor scale handling, especially GLM, RLM
body: new working assumptions:
"`scale` is an estimated parameter that belongs into the results, and the model is not supposed to keep it as an attribute"
Choice of estimation method for `scale` should be `fit` argument `scale_type` like `cov_type` if needed.

(variance or dispersion function either depending on explanatory variables or provided as `var_weights` are a separate issue.)

This is inherent if scale is part of `params`, but should also apply if `scale` is a separate parameter either estimated after estimating the mean model (as in OLS) or concentrated out as part of the fit.

## Refactoring


#2073 issue on separate estimation of scale and nuisance parameters

linear regression models like OLS do not calculate scale in `fit`, it is part of the computations in the results class, but there is a scale keyword in `RegressionResults.__init__`

Currently, both GLM and RLM store `self.scale` as attribute in the model during fit.
for GLM:
#2399 GLM scale attribute
#2353 WIP for refactoring scale handling in GLM, but uses model.scale attribute.

AFAIK, scale does not enter the estimating equations or moment conditions for GLM/LEF, so GLM could even move the scale computation to the Results class like OLS/RegressionModel. #3357

general
#1594 `cov_params_default` versus separate `scale` (+ `normalized_cov_params`)

and more specific issues 

extra issue/question:
default scale_type between cov_params and llf/loglike: regression model uses concentrated out MLE scale in llf, but uses degrees of freedom corrected scale for cov_params.
similar choice for MixedLM is MLE versus REML, IIUC.
In OLS we have a separate possibility to provide small sample and degrees of freedom corrections in cov_type that only affects cov_params and associated hypothesis tests.


Does this affect future extensions to heteroscedastic MLE?
user: josef-pkt
issue_3357: state: open
title: Performance: GLM irls calculates scale in optimization loop
body: `self.scale = self.estimate_scale(mu)`

I don't think this is necessary inside the loop. I think it is not used anywhere in irls, but because it is attached to `self`, it is more difficult to tell that it is not used indirectly.
user: josef-pkt
issue_3338: state: open
title: ENH: QuantileRegression cluster robust standard errors
body: related to #1379  more cov_types for RLM, M-estimators in general

Arellano's slide referenced here https://github.com/statsmodels/statsmodels/issues/3335#issuecomment-268719056 http://www.cemfi.es/~arellano/survey-metrics-sep2014-tr.pdf
use quantile regression as one of the examples.
The section on cluster robust covariance for panel data has the results also for Quantile Regression.
(The example is not quite the same as we are using, I think he uses a uniform truncation kernel to estimate the local density.) 
The center part of the sandwich, i.e. the covariance of the score or moment condition, looks the same as for any other cases.

(I don't remember seeing correlation robust cov_params for quantile regression before.)

user: josef-pkt
issue_3337: state: open
title: divide by zero encountered in log
body: I have this warnings:

    /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/statsmodels/base/model.py:466: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
    "Check mle_retvals", ConvergenceWarning)
    /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/statsmodels/discrete/discrete_model.py:2012: RuntimeWarning: divide by zero encountered in log
      llf = coeff + size*np.log(prob) + endog*np.log(1-prob)


with this code:

    df = pd.read_csv('test.csv')
    features = ['unemployed', 'pp']
    X = (df[features] - df[features].mean())/df[features].std()
    y = df['num'].values
    X_sm = sm.add_constant(X[features].copy())
    model = sm.NegativeBinomial(y, X_sm, offset=np.log(df['population'].values+1))
    
    start = np.zeros(X_sm.values.shape[1]+1)
    start[0] = np.log(np.mean(y))
    res = model.fit(maxiter=1000, start_params=start, method='bfgs')
    
    print(res.summary())


and this version:

    '0.6.1'

can you help me?

[test.csv.zip](https://github.com/statsmodels/statsmodels/files/669539/test.csv.zip)

user: denadai2
issue_3336: state: open
title: FAQ: Can we use diagnostics and regression plots after other models than OLS?
body: not yet, or at least not verified.

Post-estimation results like residual diagnostics, regression plots and outlier-influence measures have been written for plain OLS with some new additions to linear time series models. 

Some of those are applicable to any type of residuals, but in many cases some parts are specific to OLS. However, even if the functions are applicable for other residuals or models, those results have not been verified and checking whether the functions are appropriate is still up to the user.


e.g. regression plots don't seem to work with penalized estimation even using penalized OLS
http://stats.stackexchange.com/questions/252733/partial-regression-plots-for-regularized-l2-linear-regression?noredirect=1#comment481531_252733
user: josef-pkt
issue_3335: state: open
title: SUMM/ENH: methods for complex survey, stratified sampling
body: another standard tool that is missing Stata surv prefix and methods, R package, SAS, ...

related:
- adding weights to models, specifically probability weights


...

References
------------

books (with some brief comments from spot checking some sections)

Lehtonen, Risto, and Erkki Pahkinen. 2004. Practical Methods for Design and Analysis of Complex Surveys. 2nd ed. Statistics in Practice. Chichester, West Sussex, England ; Hoboken, NJ: J. Wiley.
covers univariate statistics (e.g. mean), linear regression and logit/binomial.
looks medium advanced, long empirical examples, quite a bit of math and references for it (e.g. small sample behavior of Wald versus LR with correction, Wald test also has small sample correction and F distribution approximation, mentions GEE for cluster robust estimation)

Mukhopadhyay, Parimal. 2016. Complex Surveys: Analysis of Categorical Data. Singapore: Springer Singapore. http://link.springer.com/10.1007/978-981-10-0871-9.
looks more advanced, high fraction of math. might be a good reference for implementation
categorical data only

Heeringa, Steven, Brady T. West, and Patricia A. Berglund. 2010. Applied Survey Data Analysis. Chapman & Hall/CRC Statistics in the Social and Behavioral Sciences Series. Boca Raton, FL: Chapman & Hall/CRC.
more introductory, low requirement for statistical background, has some "Theory Boxes" and some sections with math (overall more summation signs than matrix algebra), uses Stats, SAS?, SUDAAN?

Lumley, Thomas. 2010. Complex Surveys: A Guide to Analysis Using R. Wiley Series in Survey Methodology. Hoboken, N.J: John Wiley.
application oriented towards users of R survey package, too little math but likely useful for software requirements and references, and examples and unit tests against survey package.

There are more books, especially older ones, that sound interesting, but I don't have access to them,
and the same for some chapters in handbooks.

Fuller, Wayne A. 2009. Sampling Statistics. Wiley Series in Survey Methodology. Hoboken, N.J: Wiley.
based on a brief looks this seems to cover the theory behind finite sample (traditional) survey sampling well. It has theorems and proofs, while most other books mostly just state formulas without derivation or proofs. (and I like Fuller's measurement error book)
After skimming and reading a few sections, I think this is the best book (for me) to get some basic background theory on survey sampling.

Handbook of Statistic 29A and 29B look like good overview articles but I haven't seen a single theorem or proof yet. (very different from handbook of econometrics series.)

two more maybe "classics"
Cochran, William Gemmell. 1977. Sampling Techniques. 3d ed. Wiley Series in Probability and Mathematical Statistics. New York: Wiley.
traditional survey sampling, first edition was 1953

Wolter, Kirk M. 2007. Introduction to Variance Estimation. New York: Springer. http://public.eblib.com/choice/publicfullrecord.aspx?p=372101.
refers to Cochran for details for formulas in first chapter, later chapter look more "modern" methods for variance estimation with survey samples

user: josef-pkt
issue_3334: state: open
title: FAQ/ENH: quasi-maximum-likelihood for contingency table and log-linear models
body: Can we get misspecification robust cov_params and hypothesis testing for contingency tables and log-linear models?

With application to `surv`ey analysis, and marginal models (in the sense of GEE).

triggered by this question on the mailing list
"Adding support for multipleresponse questions in contingency tables"
https://groups.google.com/d/msg/pystatsmodels/D99xd2GK3-8/0pWYCmuvCwAJ

The underlying application is about survey questions that allow multiple answers (pick all that apply). This will often result in correlation across choices.
The methods for this will be a different issue, here the question is about sandwich robust covariance matrix estimation and misspecification robust testing.


my initial impression:
The setup reminds me more of GEE and clogit/Conditional Logit as a general approach.
AFAICS, they estimate a reduced, marginal model and base their tests on something like LikelihoodRatio tests, which don't have a standard chisquare distribution, and they need the approximation to the distribution of the linear combination of chisquare random variables.
The alternative would be to use something like GEE, or log-linear model with cluster robust standard errors and use either wald or score tests which should still have a standard chisquare distribution, I guess. However wald tests are not very accurate and are often liberal (over reject) in small samples.
<end quote>

The main technical reference goes back to two articles by Rao and Scott on using multinomial or loglinear models for complex surveys. It's the first time that I look at the theoretical derivation for survey method, like `surv` prefix in Stata, which, IIRC, essentially says that it uses weighted estimation with a robust (HC1) covariance. 
With contingency tables or log-linear model it is a bit more difficult than using the standard implementation because we might have only the count and not the individual data. (That's back to the aggregation issue, aggregation and sufficient statistics are fine under LEF/GLM but not in the quasi-likelihood setting.) Rao/Scott and Bilder/Loughin have the sandwich form for the cov_params, and delta method for derived statistics, but the center of the sandwich is based on the covariance of the estimated probabilities (or something like that). 



## Some References

I did some selective reading and skimming, and there are more related articles

Agresti, Alan, and I-Ming Liu. 1999. “Modeling a Categorical Variable Allowing Arbitrarily Many Category Choices.” Biometrics 55 (3): 936–43. doi:10.1111/j.0006-341X.1999.00936.x.

AGRESTI, ALAN, and IVY LIU. 2001. “Strategies for Modeling a Categorical Variable Allowing Multiple Category Choices.” Sociological Methods & Research 29 (4): 403–34. doi:10.1177/0049124101029004001.

Bilder, Christopher R., and Thomas M. Loughin. 2007. “Modeling Association Between Two or More Categorical Variables That Allow for Multiple Category Choices.” Communications in Statistics - Theory and Methods 36 (2): 433–51. doi:10.1080/03610920600974419.

———. 2009. “Modeling Multiple-Response Categorical Data from Complex Surveys.” Canadian Journal of Statistics 37 (4): 553–70. doi:10.1002/cjs.10040.

Rao, J. N. K., and A. J. Scott. 1981. “The Analysis of Categorical Data From Complex Sample Surveys: Chi-Squared Tests for Goodness of Fit and Independence in Two-Way Tables.” Journal of the American Statistical Association 76 (374): 221–30. doi:10.2307/2287815.

———. 1984. “On Chi-Squared Tests for Multiway Contingency Tables with Cell Proportions Estimated from Survey Data.” The Annals of Statistics 12 (1): 46–60.

Scott, Alastair. 2016. “Rao-Scott Corrections and Their Impact.” Accessed December 21. http://ww2.amstat.org/sections/SRMS/Proceedings/y2007/Files/JSM2007-000874.pdf.

Thomas and Decady are other often cited authors in this area, but I haven't looked at anything of theirs yet.

user: josef-pkt
issue_3329: state: open
title: BUG/ENH: MANOVA, MultivariateOLS follow-up
body: issue to collect comments, bugs, enhancements as followup to merge of #3327

- [ ] The oldest pandas (0.12) on travisci fails, We don't support it anymore, but it looks like there is an unnecessary unicode: `KeyError: 'Wilks\xe2\x80\x99 lambda'`
user: josef-pkt
issue_3325: state: open
title: ENH: regression plots with categorical or dummy exog
body: http://stackoverflow.com/questions/41187037/matplotlib-non-alignment-of-the-dots-on-a-plot

The regression_plots where designed with continuous variables in mind. They don't look good or appropriate for categorical regressors.

An alternative to the jitter in the stackoverflow question would be to add boxplots.

Bei now, there should be enough premade plot functions in matplotlib, pandas and seaborn that we shouldn't need much code of our own.
user: josef-pkt
issue_3323: state: open
title: biased prediction with regularized/penalized estimators Logit
body: mainly an interesting question
https://mail.python.org/pipermail/scikit-learn/2016-December/000995.html
prediction in regularized Logistic Regression (scikit-learn) is biased.

**edit** There might be something fishy in scikit-learn or with the example. AFAICS, the estimating equation for the constant stays unchanged when other parameters are penalized at `(endog - fittedvalues).sum() = 0`. This means that the estimated probability should still be equal to the sample fraction as in the unpenalized case, i.e. the constant adjusts for the shrinkage in the other parameters.
(Aside: L1 or L2 penalization is additive in loglike and score. This is different from robust estimators where the moment condition itself is nonlinearly transformed and we need to compensate for the bias introduced by the nonlinear transformation, especially (or only?) when the distribution is asymmetric.)

I have looked mostly at the effect of regularization on parameter estimates, and I don't know what the general impact on prediction is, i.e. reduce variance but get a biased prediction.

main point here: What's the penalization target? In most cases we penalize toward params = 0, which implies in the logit/probit case to bias towards a prediction of 0.5. Does the intercept compensate?

For us: Can we use the offset to change the penalization target? 
two extremes: 
- without regularization affine reparameterization does not affect prediction.
- with fully specified "prior" we can choose an arbitrary penalization target depending on the parameterization.
- If we are limited to a specific implementation of the penalization target, like zero in LASSO or Ridge, then using offset and reparameterization affect the penalization target and the estimator is not affine equivariant/invariant. What would be a useful way to choose a "prior" implemented via transformation and offset?


user: josef-pkt
issue_3322: state: open
title: SUMM/Review/Refactor robust
body: statsmodels.robust never got a make-over. It's still largely the original code.

The basic usecases work quite well, but there are several not commonly used features that have problems or where the code can be improved. We need some PRs with incremental improvements because the big PRs have a tendency to stall. For example, I found bugs while working on enhancements, but then the bugfixes got stuck in the enhancement PRs.

These issues are for fixing and improving the existing code independently of additional enhancements and refactoring needed for extensions.

- RLM with perfect fit: several issues (starting at 55) and one PR
   - #2425 (a consequence for nan in linalg?)
- RLM scale options that don't work
- RLM fit starting values #1354, #2397
- norms: several bugs in the rho and other methods that are currently not used
  -[x] #1347 Hampel, but there are more bugs
- `estimate_location` is in norms, what's the appropriate location
- scale: this contains several estimators that are useful as standalone methods, but the code is more like a prototype version, e.g. hardcoded assumptions instead of options, missing convergence criteria or convergence options, recalculating constant numbers inside a loop method.
   - #455   what's the status here?
   - #550
   - #1440
   - `scale.Huber` is a possible candidate for enhancement makeover because it is the generic joint loc-scale estimator, similar to Huber-Dutter
- cov_type HC, HAC, cluster, ...

 

user: josef-pkt
issue_3321: state: open
title: BUG: RLM callable as scale_est option
body: Browsing the RLM code for the scale options related to #3319, I saw another bug.

```
    def _estimate_scale(self, resid):
        ...
        else:
            return scale.scale_est(self, resid)**2
```

the last should use `self`:  `self.scale_est(self, resid)`

It looks like fixing this would provide a general option for the scale estimate by being able to use a user provided callable. We might want to add df_resid and other things in the call as in the call to HuberScale. The the provided callable would need a `**kwds`

Also, AFAICS, the square `**2` shouldn't be there. Mad and HuberScale() don't have the square and scale is the standard deviation in RLM, AFAIR.

user: josef-pkt
issue_3316: state: open
title: GLM with Gamma family gives DomainWarning on v.0.8.0rc1
body: Running the example from [the docs](http://statsmodels.sourceforge.net/devel/examples/notebooks/generated/glm.html) on statsmodels 0.8.0rc1 gives a `DomainWarning`:
```
/Users/xxx/miniconda3/envs/xxx/lib/python3.5/site-packages/statsmodels/genmod/generalized_linear_model.py:244: DomainWarning: The inverse_power link function does not respect the domain of the Gamma family.
  DomainWarning)
```

Reproducible example in [notebook gist](https://gist.github.com/yoavram/00638efdb17976f23263b232d5485051).

Python 3.5 on macosx.
user: yoavram
issue_3315: state: open
title: ENH: maximum trimmed likelihood
body: I mentioned it in several issues but don't see a specific issue for it

see also #3273 and #3284 

This is the MLE version of least trimmed squares LTS or ELTS.
PR #452 contains a draft version `MaximumTrimmedLikelihood` (LTLikelihood? old name?) as subclass of LTS for the initial global random search.
At the time, I hadn't read much on it and it was mainly coded by analogy, or maybe I used Hadi Luceño.

(I don't remember about these references, but I had looked at those and similar)
Hadi, Ali S., and Alberto Luceño. 1997. “Maximum Trimmed Likelihood Estimators: A Unified Approach, Examples, and Algorithms.” Computational Statistics & Data Analysis 25 (3): 251–72. doi:10.1016/S0167-9473(97)00011-X.

Müller, Christine H., and Neyko Neykov. 2003. “Breakdown Points of Trimmed Likelihood Estimators and Related Estimators in Generalized Linear Models.” Journal of Statistical Planning and Inference 116 (2): 503–19. doi:16/S0378-3758(02)00265-3.

Čížek, Pavel. 2016. “Trimmed Likelihood-Based Estimation in Binary Regression Models.” Austrian Journal of Statistics 35 (2&3): 223–32.




user: josef-pkt
issue_3314: state: open
title: suggestion: add more options for naming conventions to the summary_frame methods
body: As follow-up to #3309, 
could we add some options for naming conventions to the summary_frame methods.
For instance, in the case of `MNLogit`, the `endog` column looks somethign like what's below:

```
> me.summary_frame().reset_index().head()
   endog        exog     dy/dx  Std. Err.         z      Pr(>|z|)  \
0  myvar=0      age100  0.134127   0.017388  7.713880  1.220494e-14   
1  myvar=0    age100sq -0.128643   0.016293 -7.895520  2.891064e-15   
2  myvar=0     married -0.002546   0.000580 -4.389987  1.133577e-05   
3  myvar=0        male  0.000640   0.000488  1.312193  1.894551e-01   
4  myvar=0  lnHHincome -0.003306   0.000346 -9.565278  1.119005e-21   
```

It might in this case be convenient for many users to have instead a column labeled `myvar` simply taking the values that appear after the "=" sign in the existing "endog" column.

There are, points out @josef-pkt in #3309, other multivariate endog cases which need  automatic name creation, e.g. "when we need the raveled params and associated statistics."

I doubt following is helpful, since the key parts would get done in other helper methods, but I wrapped summary_frame() as follows to get my column named "myvar":
```
def _mlogit_marginal_effects_summary_frame(me): 
    df=me.summary_frame().reset_index()
    endog=me.results._get_endog_name(None,None)[0]
    df[endog] = pd.Series(  df.endog.str.split('=').apply(pd.Series)[1])
    return(df)
```
user: cpbl
issue_3313: state: open
title: ENH/DOC: docs by topic for generic features
body: a bit similar to #650 but more focused on parts of models and results that are consistently shared across many models

for example I don't remember a quick link to have examples for this f_test, t_test, wald_test, and coming mv_test and score_test  I know it's explained for at least one of the models but it's not obvious where to look for it.
http://stats.stackexchange.com/questions/250672/scipy-statsmodel-wald-test-input

http://www.statsmodels.org/dev/examples/index.html#user is a bit like this but not enough.

(somewhere I have a toc-like list with generic pieces of our usage structure)


for cov_type I have a draft notebook and the main function has the relevant docstring
fit optimization ???
post-estimation
formulas look well covered http://www.statsmodels.org/dev/example_formulas.html and linked to from the main TOC page ("Basic Documentation").
...

I don't think docstring examples for generic methods will work and would be a huge maintenance burden (scipy.stats.distribution docstring templating is already pretty heavy), mainly because inherited methods are generic and we would need many different classes in the examples.

So, being able to link to central examples and explanation would be a useful substitute.

Stata is also linking for some of the generic/common features to a central explanation, e.g. for most of the generic fit and pre-fix options and for post-estimation.




user: josef-pkt
issue_3310: state: open
title: FAQ/DOC: hypothesis testing: what are the maintained assumptions
body: hypothesis test generally specify H0 and H1.

Additionally, it would be useful if we could also point out `"maintained assumptions"`. In general, what are the "maintained assumptions" is or might be difficult to specify, in some cases it might point to simple important restrictions for the applicability of a test.

e.g. "common variance" for t-test (when not using the unequal variance option), sphericity or similar for repeated measures or manova, or common covariance. likelihood ratio tests assume either correctly specified loglike or correctly specified up to scaling (in GLM, RLM?)

related: I added a sentence to the OLS summary() comments about the robustness of the cov_params

It should be possible to get to a more consistent doc string pattern for hypothesis test, beyond formal (numpy) docstring standard. 

(I don't know how easy this will be to add to the docstrings. Often we might have not be aware of the crucial maintained assumptions.) 

"maintained assumption" shouldn't be everything, it should cover the major deviations against which a test is not robust. e.g. t-test is pretty robust to distributional assumption, except for heavy tailed distributions and outliers, but it will usually not be robust to correlation across observations.
user: josef-pkt
issue_3306: state: open
title: ANOVA documentation clarity.
body: Hi,

I just have a small issue. I'm trying to find the p-value of an ANOVA test. The code worked perfectly and returned a DataFrame with the  final column labeled as "PR(>F)". When I went to double check if this was the same as a p-value, I noticed that the documentation for what is returned by the ANOVA function appears to be incomplete:

https://github.com/statsmodels/statsmodels/blob/master/statsmodels/stats/anova.py#L293

Cheers!
user: Jessime
issue_3302: state: open
title: ENH: heteroscedasticity, variance function with quantile regression - nonlinearity
body: (another random thought, a quick google search shows that there is some information, but I haven't read anything.)

In the illustrative quantile regression example, I used heteroscedasticity to get quadratic quantile functions. In general, outside of i.i.d. or non-normal (?) samples, the quantile function might be a non-linear function of the parameters. We can handle this, for example, with splines, however splines might run easily into crossing problems (example ?). If one source of the nonlinearity of the quantile function is in systematic, exog dependent heteroscedasticity, then a pre-weighting as in HETWLS might remove most of the non-linearity.

Question: Can we add a HetQuantileRegression that does the variance scaling before computing the Quantile Regression and then move back to original space?   (pre-whiten and recolor add variance function scaling to objective function. (viewed as an M-estimator with scaling, analog to RLM)

aside: RLM doesn't allow for prior variance functions or heteroscedasticity either.


user: josef-pkt
issue_3300: state: open
title: SUMM/ENH: gof tests, autocorrelation of point process, random arrival
body: What can we use to test whether a point or counting process is time homogeneous?

eg. 
- Poisson or counting process
- binary event with unequal probabilities, similar to runs test

I looked into this once http://stackoverflow.com/questions/18835678/test-for-poisson-process

Do we have the related gof tests, e.g. for exponential, uniform?

specific application:
I have a time series or sequence of observations with outliers. In initial stage time series properties are ignored, no strong autocorrelation, and outliers candidates are detected for individual observations.
Using the outlier indicator (e. g. value > threshold), I want a quick measure or test whether outliers are "bunching".

autocorrelation, autocovariance might be dominated by "base state"
autocross-product of indicator will just count event sequences (1*1=1, all others are zero) 

related (I have no idea about methods there):
in spatial, 2-D, this would be a uniform random field.

(for integer, Poisson AR regression we could test whether the lagged indicators have insignificant coefficient.)


user: josef-pkt
issue_3298: state: open
title: Power analysis for correlation
body: Hello I would like to run a power - or more importantly, sample size - analysis for correlation.

Basically I would like to know how many data points I would need to detect a correlation of 0.5 or above with a=0.05 and 1-b=0.8. I see statsmodels provides a function to [do this for a t-test](http://statsmodels.sourceforge.net/devel/generated/statsmodels.stats.power.tt_solve_power.html#statsmodels.stats.power.tt_solve_power). Is there any function to do it for Pearson's r? If not, could you perhaps assist me with writing and contributing one?

I found [an online calculator](http://www.sample-size.net/correlation-sample-size/) but it would be great not to rely on an external service.  
user: TheChymera
issue_3297: state: open
title: ENH: t_test_multiple, multiple comparison or testing for regression parameters
body: (another semi-random idea)

related to our missing multiple comparison functions from model results (Wald based), a companion to the standalone multiple testing in k-sample comparisons.

We are still missing automatic creation for contrast or restriction matrices for all_pairs and similar comparison based on params, cov_params and the formula information.

an idea to sidestep this part: add the multiple comparison correction and results for a user provided list or matrix of restrictions.
i.e. use restriction matrix as given as in t_test and wald_test, get the joint multivariate distribution similar to wald_test but then compute hypothesis tests for each sub-test. We could use a standard p-value correction as for multiple testing or, more accurately, use the numerical box probability as it is in the sandbox for multiple testing (theoretically more appropriate but possible integration warnings).

related idea #3295 applications when we want simultaneous confidence intervals for a small number of jointly t or normal distributed random variables.

Pallmann, Philip, and Ludwig A. Hothorn. 2016. “Analysis of Means: A Generalized Approach Using R.” Journal of Applied Statistics 43 (8): 1541–60. doi:10.1080/02664763.2015.1117584.
(not read but sounds good, I assume it's similar to the other Hothorn and co-authors methods.)

side question: comparison to a reference, all-pairs, to the mean
If we assume independence of the k-samples, then testing a set of linear combinations would have a known covariance structure. This could have a more accurate or efficient implementation for the p-values than the multivariate box probability for arbitrary covariance matrix. tukey_hsd is a special case like this.


user: josef-pkt
issue_3296: state: open
title: SUMM/ENH: control chart tools, online, outliers, anomalies
body: (I'm coming to this now in the context of outlier robust estimation.)

roughly

- phase 1: try to build a "clean" or in-process reference model.  
   Includes outlier identification, robust estimation, non-parametric or distribution free methods preferred
- phase 2: monitoring new observations based on reference model.   
   This is mainly establishing control limits, or confidence intervals for new (out-of-sample) observations 
- online updating: related to "self-starting control charts"  
   I ran, by accident, into some articles on online updating of robust estimators (called "recursive" estimators), e.g. M or S-estimators.

other issues

- multiplicity control, FDR and so on
- univariate, multivariate or blocked/nested.  
   e.g. We can combine multivariate outlier detection for common or correlated causes with univariate measures (and maybe univariate measures on principal components)  
  also affects whether we want multivariate series or vectorized processing of univariate series

There are some things available in Python, like https://github.com/bwghughes/spc, plus plotting packages, but, AFAICS, those are focused on phase 2 monitoring with just the simple traditional control limits.
Also specialized packages for example for anomaly detection for specific cases.

Stata and SAS/JMP have the basic control charts.

What I would like to see:

- a few basic cases for a full framework, mainly as pattern, and to see and unit test how things can fit together. E.g. a class with methods for all phases, maybe usable for cases that don't require "industrial strength" on-time monitoring.
- (phase 1): more outlier robust methods and outlier identification, missing tools and recipes. non-parametric, non-normal methods?
- (phase 2): more or better confidence intervals for prediction, some reusable helper function, e.g. in- and out-of-sample distribution of mahalanobis distance.
- getting started with more online estimation, update methods currently don't fit well into our main model setup.

Those things are of general interest and not only for control charts, except for the first point. Control charts look like an interesting use case.

related use case: Maybe it's not the process that goes wrong, maybe the model is wrong.
e.g. change point detection "out-of-sample", i.e. signal as early as possible that the world might have changed and that the model might be wrong. (e.g. dynamic prediction is systematically wrong)

more multivariate with emphasis on panel, spatial structure would be disease outbreak detection, scan statistics (?), ...


user: josef-pkt
issue_3295: state: open
title: ENH: out-of-sample cusum, k-step forecast, anomaly detection
body: an idea related to outlier/anomaly detection

(a bit related to cusum control charts *)

If we have rolling or online 1 to k-step ahead (dynamic) forecasts, then we could use the 1 to k CUSUM or just the k-sum prediction error to see if it is an outlier (outside a threshold). This would provide more power to detect multi-period outliers compared to only using one step ahead forecasts.

The simple case would be to assume that the k observations are independent and we can calculate the normal distribution for the k-sum. We have a small correlation in the residuals coming from using the same parameters that are random variables, i.e. with associated cov_params.
With serial correlation as in ARMA, the residuals would also be correlated in themselves, and the joint distribution of k prediction errors or residuals is more difficult. I don't think we have that yet.

(*) semi-random reference: 
http://www.jmp.com/support/help/Statistical_Details_for_CUSUM_Control_Charts.shtml#1165418

aside: outlier/anomaly or change point. 
Standard CUSUM test is used for change point detection. In this case we don't know why a sequence of observations doesn't correspond to the previous observation, or to the model estimated with those observations. Maybe it's an anomaly, maybe a structural change, maybe our model was wrong in the first place, ...

main question for code: Can we get the joint distribution of several predicted or forecast observations? Or of a linear combination of those predictions or residuals?

related: t_test evaluates several hypothesis separately. wald_test evaluates one joint hypothesis, which might work for a linear prediction. 
We don't have anything yet, that provides the joint distribution of several linear combination of parameters.

user: josef-pkt
issue_3288: state: open
title: SUMM: model selection criteria based on quadratic form (wald) of parameters
body: distance measure (beta - beta*) V(beta*) (beta - beta*)
has several applications (where I don't remember where I have seen it)


While looking for information/selection criteria I found this which uses a penalized quadratic form similar to BIC or AIC. It relies on the above quadratic form and doesn't need to have a likelihood function or objective function defined (GEE variable selection is used as an example)
Jiang, Wenxin, and Xiangyang Liu. 2004. “Consistent Model Selection Based on Parameter Estimates.” Journal of Statistical Planning and Inference 121 (2): 265–83. doi:10.1016/S0378-3758(03)00112-5.

AFAIR (but not where)

- model selection criteria (the above article is the first time I see it for this)
- diagnostic tests based on this form
- calculate one-step constrained estimates  #2092
- minimizing it has a name in econometrics, referenced next to GMM somewhere, minimum chisquare ?

In the form in the Jiang and Liu paper, it is the Wald statistic where the reference beta* and V(beta*) is from the largest model.

(There should be somewhere in an issue more details on the case of using one-step constrained estimation which would be useful as starting value for fully iterated estimation.)

(Maybe methods like this cannot beat the specialized or fully specified methods, but they would often be helpful as a quick check and starting point for fancier methods if those are available. Also, we already have all the Wald test infrastructure.)

Todo: find the cases and references where this is used (or clear this up if I don't remember correctly).

(related aside: IIRC hausman test uses two V in the quadratic form)

user: josef-pkt
issue_3285: state: open
title: TSA: outliers, robust estimation, inference
body: related to #943

943 and related discussion is on how to handle outliers. AFAICS, the main approach for outlier detection and handling for ARMA and similar models are based intervention models, and "oxford" dummy saturation.
AFAIK, this ignores the impact of the outlier search on inference.

Soren Johansen and Bent Nielsen have a recent or current sequence of papers that includes asymptotic theory and is closer to the robust literature, e.g.

Johansen, Søren, and Bent Nielsen. 2016. “Asymptotic Theory of Outlier Detection Algorithms for Linear Time Series Regression Models.” Scandinavian Journal of Statistics 43 (2): 321–48. doi:10.1111/sjos.12174.

Johansen, Søren, and Bent Nielsen. 2013. “Outlier Detection in Regression Using an Iterated One-Step Approximation to the Huber-Skip Estimator.” Econometrics 1 (1): 53–70. doi:10.3390/econometrics1010053.
http://www.mdpi.com/2225-1146/1/1/53

This is just a reference for now (so we find it again), I have no idea when we will get into this neighborhood. The priority should be to implement the traditional methods for outlier handling in tsa models with focus on prediction and less emphasis on inference. (Quality of forecast standard errors has not been a top priority for anyone either.)

an issue that might be relevant even for traditional methods is the possible difference in convergence or asymptotic distribution depending on one-step WLS, one-step Newton and iterated solutions. see #3273


as an example: unit root tests under outlier removal (just quick skimming of the second paper, 2013, above: it has the Brownian motion based distribution for the AR(1) coefficient for/under unit root as an example.)
(related: somewhere I have seen articles on robust VAR and cointegration)


user: josef-pkt
issue_3284: state: open
title: ENH:   wrapper/prefix mixin class for robust estimation using weights
body: This is an idea for a generic class that puts outlier robust estimation outside the standard models, or possibly follow the fit_xxx pattern.

requires that models handle frequency weights (or generic iweights), or that we can overwrite loglike, score and hessian with a weighted version (which should work for parameters and generic sandwich cov_type but maybe not for all extra results). 

RLM uses WLS at it's core for irls. Similar, multivariate regression would need a multivariate WLS or weighted mean in the core mean function calculation.
In both cases robustness is implemented through a weighted regression/estimation.

Mallow's type weights for GLM for example directly put weights on the (likelihood and) score function. 

This would work generically for most models, assuming we can calculate a robustness weight function. We could put the weight iteration into a wrapper class or a special `fit_robust` method and we don't need to rewrite every model to have a robust version.
The weight function could take `resid` as argument and whatever else is needed as keywords. 

possible problems: 

Messy scale separation: we might need robust estimators for scale/var, or more generally for nuisance or auxilliary parameters. Those might not be nicely exposed to being overwritten.

How much model or distribution specific information do we need in order to calculate the robustness weights, e.g. cutoffs or shape parameters for the robust weights which depend on the reference distribution?

GLM would be a good case to try this out because it already has frequency weights.

This is a bit similar to maximum trimmed likelihood, which is even easier because we don't need the weights, we just drop observations. MTLE is overall messier because we do the initial global optimum search.



user: josef-pkt
issue_3280: state: open
title: SUMM/ENH: multivariate hypothesis tests for covariance structure based on robust scatter
body: (another sequence of references, I put here just the last one I read)

Make hypothesis tests on the covariance/scatter matrix robust to some distributional assumptions
application to multivariate statistics, e.g. #2765

Tyler and other M-estimators for scatter or covariance will come with #3230
based on elliptical distribution, some discussion in #3266
a little bit similar to #2942 where the emphasize is on improved small sample properties

Tyler, David E. 1983. “Robustness and Efficiency Properties of Scatter Matrices.” Biometrika 70 (2): 411–20. doi:10.2307/2335555.

and related references e.g. by Bentley and coauthors for application in SEM (?)
but Tyler 1983 makes nicely the point here.
Note, Tyler scatter M-estimation was published later (1987).

Some multivariate tests for covariance structure might be very sensitive to deviation from the normal distribution of the sample e.g. in the direction of heavy-tailed distributions.
One possible solution that works in some cases is to correct the chi2 statistic by the kurtosis.
Tyler 1983 shows that using standard sample covariance is very inefficient with medium to large kurtosis, even though we can use the kurtosis correction to the test statistic.
He recommends instead using a robust M-estimator which does not loose much efficiency in the normal case but is more efficient for elliptically symmetric heavy tailed deviations from normality.
This applies to tests that don't rely on overall scale of the scatter matrix, eg. correlation pattern or relative magnitudes in the covariance/scatter matrix.

Also heavy tailed elliptically symmetric distribution is a different deviation from a standard model than a pure large outlier or contamination case (where we would want to use MCD, MVE or similar strong descending weight functions). 
There are a large number of recent and not so recent articles plugging high breakdown estimators into multivariate method. 

user: josef-pkt
issue_3278: state: open
title: SUMM/ENH: RLM/GLM nonlinear function optimization
body: We would like to extend RLM #3261 and GLM #2179 to nonlinear mean functions.
several other related issues that I didn't look for

We can use the default scipy optimizers. But it might be more performant to have something more specific, e.g. based on the new scipy leastsq functions. This would be something like a nonlinear equivalent of IRLS.

(parking this reference)
Gay, David M., and Roy E. Welsch. 1988. “Maximum Likelihood and Quasi-Likelihood for Nonlinear Exponential Family Regression Models.” Journal of the American Statistical Association 83 (404): 990–98. doi:10.2307/2290125.

They discuss approximating the "messy" part of the Hessian. This sounds a bit similar to OIM versus EIM (observed versus expected information matrix) and the expensive part of the current GLM hessian.
I'm a bit surprised in how few iterations their estimator converges, but they use mostly very small sample sizes.

Also, for RLM we still need to get score_obs and hessian or the `..._factor` versions if applicable.

user: josef-pkt
issue_3277: state: open
title: NaN p-values in anova_lm
body: When I fit two models and run anova_lm.. I get p-values of NaN for both models. However, when I use 'np.abs(table['df_diff'])' instead of 'table['df_diff']'  as an argumnet to the 'stats.f.sf' function, then I get the correct answer (as found in R's anova function). 

It is here in the anova_lm() function:

    if test == "F":
        table["F"] = table["ss_diff"] / table["df_diff"] / scale
        table[pr_test] = stats.f.sf(table["F"], table["df_diff"],
                             table["df_resid"])
        # for earlier scipy - stats.f.sf(np.nan, 10, 2) -> 0 not nan
        table[pr_test][table['F'].isnull()] = np.nan

Fixed code:

    if test == "F":
        table["F"] = table["ss_diff"] / table["df_diff"] / scale
        table[pr_test] = stats.f.sf(table["F"], np.abs(table["df_diff"]),
                             table["df_resid"])
        # for earlier scipy - stats.f.sf(np.nan, 10, 2) -> 0 not nan
        table[pr_test][table['F'].isnull()] = np.nan


user: ncullen93
issue_3276: state: open
title: SUMM: outlier robust estimation through data shrinkage
body: shrinking the endog is another principle that allows reuse of existing methods for robust regression. This is similar to winsorizing and an alternative to trimming or dropping outliers (e.g. #3273  #943).
I'm not sure how wide this applies, the cases I have seen are all for the linear model (e.g. with spline basis functions.)

It is mostly a computational tool (and looks a bit similar to working residuals in GLM-IRLS, except we don't need weights in the core model). Shrinkage is data dependent and iteratively updated so that the final estimating equation corresponds to the ones of an M-estimator.

Other possible applications: cross-validation and residual bootstrap where new observations are shrunk so they are not outliers anymore and don't mess up bootstrap or cross-validation criteria.

two articles, there are several that use the pseudo data, mainly for splines and additive models with a nonlinear component.

Oh, Hee-Seok, Jaeyong Lee, and Donghoh Kim. 2008. “A Recipe for Robust Estimation Using Pseudo Data.” Journal of the Korean Statistical Society 37 (1): 63–72. doi:10.1016/j.jkss.2007.05.001.

Oh, Hee-Seok, Douglas W. Nychka, and Thomas C. M. Lee. 2007. “The Role of Pseudo Data for Robust Smoothing with Application to Wavelet Regression.” Biometrika 94 (4): 893–904.

Alimadad, Azadeh, and Matias Salibian-Barrera. 2011. “An Outlier-Robust Fit for Generalized Additive Models With Applications to Disease Outbreak Detection.” Journal of the American Statistical Association 106 (494): 719–31. doi:10.1198/jasa.2011.tm09654.

Tharmaratnam, Kukatharmini, Gerda Claeskens, Christophe Croux, and Matias Salibián-Barrera. 2010. “S-Estimation for Penalized Regression Splines.” Journal of Computational and Graphical Statistics 19 (3): 609–25.
This is a nice paper, that got me started on this, but I don't remember how they are using it. 

(I haven't looked much into this, but it sounds very useful for implementing various extras like cross-validation for penalized estimation.)


user: josef-pkt
issue_3275: state: open
title: FAQ/ENH: RLM bias in constant with asymmetric distribution
body: RLM assumes in general a symmetric distribution otherwise we need to add a term into the estimating equations for Fisher consistency. For the latter case we (will) have special models for specific asymmetric distribution, e.g. robust estimation in GLM or Poisson.

However, it looks like (*) the asymmetric distribution only biases the estimate of the constant or intercept in the linear model. However, we need to demean the other explanatory variables to make them orthogonal to the constant (as far as I know or guess).

(*) I saw this mentioned in several places, but don't remember where.

related but different
#3273
Ruppert, David, and Raymond J. Carroll. 1980. “Trimmed Least Squares Estimation in the Linear Model.” Journal of the American Statistical Association 75 (372): 828–38. doi:10.2307/2287169.
In section 3 they separate out the constant because it is not affected by the initial estimator in contrast to the slope coefficients (asymptotically, see their section 4).

I'm not sure whether we want to do something, recommend demeaning to users or add a transformation internally, similar to question for #1131

roughly: 
GLM/LEFM estimate the mean function, conditional expectation of the endogenous variable, but the robust objective function and nonlinear transformation in the estimating equations has as consequence that the estimating equation does not have expected value of zero under asymmetric distribution.
user: josef-pkt
issue_3274: state: open
title: canonical correlation analysis and partial least squares
body: I wonder if there is any plan of including CCA and PLS in statsmodels. sklearn have a CCA implementation but didn't report tests results. I'd like to give CCA a try if this could be of interest to statsmodels.
user: yl565
issue_3273: state: open
title: SUMM: outlier robust, reweighted and two stage robust methods - inference
body: see also #3270

Dropping outliers or hard rejection weights is one of the easiest way to get high efficiency after using a high breakdown estimator in an initial stage. We can do the reweighting for robust estimation of covariances, in regression or MLE through trimming outliers. see #3256

One problem is that the distribution of the estimator and inference depend on the initial estimator, at least in some cases.

conclusion based on below:

- use fully iterated final stage (possibly with fixed scale), then standard error will not depend on initial conditions (as in MM and tau-estimators). For the special case of trimming we can use iterated ELTS or iterated EMTLE (efficient least trimmed squares or efficient maximum trimmed likelihood) part of it is in my LTS/ELTS PR based on Doornik, more theory and different trimming criteria are Cizek and references.
- If we want one-step or without convergence in the final stage, then we need to account for the initial estimator and the data dependence of the weights. One idea would be to use GMM similar to the IPW treatment effects case if the first stage as a estimating equation/moment condition representation.
- don't recommend that users drop outliers (warning by Ruppert and Carrol) if they want valid inference, unless they use a fully iterated estimator.

----

There are several references that mention it, but I never went through the details

Cizek ?

two older ones that show some of the details (I didn't read them)

Dollinger, Michael B., and Robert G. Staudte. 1991. “Influence Functions of Iteratively Reweighted Least Squares Estimators.” Journal of the American Statistical Association 86 (415): 709–16. doi:10.2307/2290402.

Ruppert, David, and Raymond J. Carroll. 1980. “Trimmed Least Squares Estimation in the Linear Model.” Journal of the American Statistical Association 75 (372): 828–38. doi:10.2307/2287169.

However, AFAIK, this does not affect all two-stage estimator, for example MM or tau estimators. In MM estimator we only use a scale/variance estimate from the initial stage and locally the asymptotic behavior is equivalent to an M-estimator. Tau estimator is equivalent to a data adaptive M-estimator.
(Also, we still have some orthogonality in the information matrix in LEF and elliptically symmetric models between first and second moment parameters.)

I'm not sure how this fits together. We need a warning at least for the mix-and-match reweighting results, when or if we add cov_params. 

related: influence in WLS
http://stackoverflow.com/questions/40621686/does-statsmodels-wls-have-get-influence-function/40644521#40644521
So far I have seen mainly influence measures that take weights as given, however, Dollinger and Staudte (1991), point out the influence that comes through endogenous or data dependent weights.

**update**
Ruppert and Carroll above only use trimming by a fixed fraction of observations, or dropping the tails of an initial quantile regression.

Dollinger and Staudte is a good read (pretty self contained and not too heavy in the influence function)
a piece of the above puzzle:
The dependence of the reweighting estimator on the initial estimator is only if we use a few IRLS steps. The effect goes away if we iteratively update the weights until convergence, if convergence condition is satisfied. A ~~Gauss-Newton step (GNR)~~ Newton-Raphson has even in one step the standard asymptotic result and does not depend on the initial estimator. GNR and IRLS differ in this.
They use M-estimator with smooth rho/norm functions in the theorems but say it extends to hard thresholding/trimming functions. Initially they assume scale is known, but it also extends to using a fixed scale estimate from the initial estimator. They don't have a result if the scale estimate is updated during IRLS as in Huber proposal 2.

maybe that corresponds to what Cizek (*) is arguing that including the trimming in the main estimator removes the dependence of an initial estimator on the cov_params. (I read a bit of his but did not try to understand any details)

(*) correct spelling has two accents, (without those Zotero only has working papers)
(IIRC I looked at the 2013 article for this point, i.e. similar to ELTS)

———. 2013. “Reweighted Least Trimmed Squares: An Alternative to One-Step Estimators.” TEST 22 (3): 514–33. doi:10.1007/s11749-013-0335-5.

Čížek, Pavel. 2006. “Least Trimmed Squares in Nonlinear Regression under Dependence.” Journal of Statistical Planning and Inference 136 (11): 3967–88. doi:10.1016/j.jspi.2005.05.004.
———. 2008a. “Robust and Efficient Adaptive Estimation of Binary-Choice Regression Models.” Journal of the American Statistical Association 103 (482): 687–96.
———. 2008b. “GENERAL TRIMMED ESTIMATION: ROBUST APPROACH TO NONLINEAR AND LIMITED DEPENDENT VARIABLE MODELS.” Econometric Theory 24 (6): 1500–1529. doi:10.1017/S0266466608080596.

———. 2016a. “Generalized Method of Trimmed Moments.” Journal of Statistical Planning and Inference 171 (April): 63–78. doi:10.1016/j.jspi.2015.11.004.
———. 2016b. “Trimmed Likelihood-Based Estimation in Binary Regression Models.” Austrian Journal of Statistics 35 (2&3): 223–32.
Neykov, N. M., P. Čížek, P. Filzmoser, and P. N. Neytchev. 2012. “The Least Trimmed Quantile Regression.” Computational Statistics & Data Analysis 56 (6): 1757–70. doi:10.1016/j.csda.2011.10.023.

**update 2**

Simpson, Douglas G., and Yuan-Chin Ivan Chang. 1997. “Reweighting Approximate GM Estimators: Asymptotics and Residual-Based Graphics.” Journal of Statistical Planning and Inference, Robust Statistics and Data Analysis, Part II, 57 (2): 273–93. doi:10.1016/S0378-3758(96)00049-3.
has some interesting details
I'm not sure whether their main result is very useful (if the initial estimator is approximately a GM estimator, then reweighting with GM produces an estimator that is approximately GM IIUC?)

However they have some interesting details on GM estimators:   
the three parameterization for the exog weights, some comments about relative performance in the simulation,  
asymptotic distribution including score and Hessian for the HC sandwich,  
diagnostic plots for weighted and unweighted residuals and fitted values, see also #808  
Newton-Raphson (which AFAIU looks the ~~same as Gauss-Newton GNR~~) one-step approximate GM estimator is asymptotically equivalent to full iteration.

(aside: I think I'm mixing up Newton-Raphson one-step and Gauss-Newton on-step
Davidson and MacKinnon use Gauss-Newton regression which I thought is equivalent to an extra WLS step, but there might be conditions like information matrix equality holds that might be required for this.
)

one-step NR: From what I know previously, the asymptotic equivalence between one-step and fully iterated Newton-Raphson requires that we start at a sqrt(nobs) consistent estimator. Dollinger and Staudte assume this throughout. So the NR one-step result would not apply to some high breakdown estimators that are converging slower than sqrt(nobs) like MVE and LMS. We need full iteration even with NR in those cases.

related aside: In some (?) article, they showed Monte Carlo results where one-step reweighting was doing surprisingly well based on an initial estimator with slower convergence rate (MVE, IIRC). This would be the case if the first step in an iterative reweighting estimator brings us already close to the fully iterated estimate in a small sample.

(to add a bit more to the one-step soup)

one-step fisher scoring versus newton-raphson

both have asymptotics in one-step equivalent to fully iterated under homoscedasticity and symmetric distribution (with M/GM estimators)
However, starting with a less that sqrt(n) initial estimator
- fisher scoring one-step get sqrt(n) asymptotics under asymmetry but not under heteroscedasticity
- newton-raphson one-step get sqrt(n) asymptotics under heteroscedasticity but not under asymmetry

i.e. robustness in different directions of distributional misspecification

Simpson, D. G., D. Ruppert, and R. J. Carroll. 1992. “On One-Step GM Estimates and Stability of Inferences in Linear Regression.” Journal of the American Statistical Association 87 (418): 439–50. doi:10.2307/2290275.

another related aside: Huber sandwiches are not HC White sandwiches.
My impression while reading early 1980's articles on robust estimators and M-estimators: 
The sandwiches usually don't take possible unspecified heteroscedasticity into account,  
i.e. E(x u u X) = E(u u) * E(x x). (similar to AC versus HAC).
They end up with an inv(X'X), which in itself is not robust. 
(roughly: the general form is H^{-1} V H^{-1}, but then the actual formulas in the theorems are simplified using additional assumptions which are specified but may not be needed.)

user: josef-pkt
issue_3271: state: open
title: ENH: tools dataframe for probability plot , ecdf, kaplan meier without censoring
body: another idea, I didn't check what we have already

I think it is essentially a distribution equivalent `weightstats` with one-sample, 2 and k-sample versions, based on empirical distribution function, either stepwise or interpolated.

The basics to support various plots would be a dataframe with edf, eppf for each sample and pooled sample and theoretical probabilities if available. (Large part of this should already be in probability plots.)
This allows different plots depending what is put on the axis.

comparing two distributions: an additional "effect" illustration is the shift plot, i.e. how to we get from one distribution to a second one. (*)
ks_test for confidence intervals, ...


(*) interesting aside: generalized skew-elliptical distributions have a sample selection interpretation under the assumption that the population distribution is (elliptically) symmetric.
Ma, Yanyuan, Marc G. Genton, and Anastasios A. Tsiatis. 2005. “Locally Efficient Semiparametric Estimators for Generalized Skew-Elliptical Distributions.” Journal of the American Statistical Association 100 (471): 980–89. doi:10.1198/016214505000000079.

sample selection shifts the distribution (non-constant shift, non-parametric)

user: josef-pkt
issue_3270: state: open
title: FAQ: How good are standard errors? How good is inference?
body: The basic models have established cov_params and related standard errors and inference. For these the usual limitations apply.

There are some additional models or approaches that do not have established standard errors or where reliable standard errors are not yet implemented, or where a model produces "generic" standard errors for the parameters that might not be valid for all options.

In general, the covariance of parameters are estimated under the assumption that the model is given and correctly specified. Some covariance estimators like those based on the sandwich form are robust to misspecification of the model in some directions. In some cases the asymptotic standard errors are not a good approximation to finite samples, where the required sample size depends on the specific model and modelling approach.

some areas where standard errors might be wrong:

- specification search and variable selection: Standard errors are mostly calculated ignoring any uncertainty about or search for the specification.  Two special cases are penalized estimation and semi-parametric fitting where better estimates of the covariance of the parameters might be available but not implemented. An example for semi-parametric fitting is the choice of the polynomial order or of the number of knots in splines, or other analogs of a bandwidth.

- multi-step estimation procedures: If the estimation procedure consists of several stages, then the effect of initial stages is not always included in the covariance estimate of the last model. We have three cases:
  - wrongly ignored: The standard errors are incorrect if the effect of initial stages are not taken into account. Specification search and variable selection is one example. Other examples are the effect on weighted regression if weights are determined by the data for example in inverse probability weighting (but see next).  Estimator are often combined in a sequential way in outlier robust regression. In some cases the covariance of the final stage estimator depends on the estimator in earlier stages
  - ignored but asymptotically irrelevant: In some cases we can estimate some parameters in an initial stage but those parameters are asymptotically uncorrelated with the parameters of interest. A common example for this is the estimation of parameters of the variance function in GLM or linear exponential family which asymptotically do not affect the covariance of the mean parameters because the information matrix is block-diagonal. This includes feasible generalized least squares estimators and correlation estimators in GEE.
  - explicitly taken into account: Some models like two-stage least squares, two-stage heckman estimation have well defined estimation stages that are correctly taken into account in the covariance of the final stage parameters. 

**Backwards Compatibility Warning** 
We will improve and change the default and the options for `cov_params` and related estimators without necessarily going through a deprecation stage in cases where an existing implementation use a very narrow interpretation or are based on very strict assumption so that they are incorrect for common use cases.
user: josef-pkt
issue_3266: state: open
title: SUMM/ENH: TModel use t-distribution for estimation
body: (coming back to t-distribution MLE after robust readings)

motivation: heavy-tailed distribution but not clear outliers

current: TModel, TArma in `miscmodels`, t-based covariance in outlier robust cov PR
sandbox.distribution has MVT as basic distribution class, including rvs for simulation, but no fit method.
sandbox.distribution also has a class MVElliptical, which was supposed to work as a superclass for various elliptically symmetric distributions, but AFAIR I didn't do much or anything with it.

(terminology: I'm not sure what to call the multivariate t scatter/cov/shape matrix Sigma. 
`cov_y = Sigma * df / (df - 2)` if df > 2
same terminology problem for all elliptical distributions. Taylor uses "scatter" where scaling factor is indeterminate. robust literature uses "shape" for cov scaled so that det(cov) = 1
)

extension:
- heterogeneous kurtosis or degrees of freedom (varying by variable), copulas
- skew-t (skew-elliptical distribution ?)

technical notes:

Property 4 in Lange, Little, Taylor 1989: Mahalanobis distance is F-distributed 
(I couldn't find that property while searching for it with google)
Appendix B has score for general elliptically symmetric models and expected information matrix for multivariate t model.
Another important property: estimates for mean parameter and cov, df are asymptotically uncorrelated, i.e. block-diagonal EIM. This holds for all elliptically symmetric models. (As a consequence we can concentrate on inference for mean parameters separately from cov, df as in GLM/LEF.)
They also mention generalized power exponential family as another elliptically symmetric distribution family that includes also short tails, but is computationally and analytically less tractable.

Lange, Kenneth L., Roderick J. A. Little, and Jeremy M. G. Taylor. 1989. “Robust Statistical Modeling Using the T Distribution.” Journal of the American Statistical Association 84 (408): 881–96. doi:10.2307/2290063.


user: josef-pkt
issue_3262: state: open
title: Repeated measures ANOVA
body: I wanted to verify whether among different measurements sessions, any are significantly different from the others. One way to do this would be to use a repeated-measures ANOVA, but I see that the [ANOVA implementation](http://statsmodels.sourceforge.net/devel/anova.html) in statsmodels does not support this.

I thought another way would be to use [MixedLM](http://statsmodels.sourceforge.net/devel/generated/statsmodels.regression.mixed_linear_model.MixedLM.html), which can deal with repeated measures, and then run an ANOVA on that model. I tried two approaches, using [this data](http://statsmodels.sourceforge.net/devel/generated/statsmodels.regression.mixed_linear_model.MixedLM.html):


```
import pandas as pd
import statsmodels.api as sm
import statsmodels.formula.api as smf

df = pd.read_csv("~/MixedLM_data.csv")
lm = smf.mixedlm("t ~ session", df, groups=df["subject"]).fit()
anova = sm.stats.anova_lm(lm)
```
This fails with `AttributeError: 'MixedLMResults' object has no attribute 'ssr'`.

```
import pandas as pd
import statsmodels.api as sm
import statsmodels.formula.api as smf

df = pd.read_csv("~/MixedLM_data.csv")
lm = smf.mixedlm("t ~ session", df, groups=df["subject"]).fit()
anova = lm.f_test([[0,1,-1,0,0,0],[0,1,0,-1,0,0],[0,1,0,0,-1,0],[0,1,0,0,0,-1]])
print(anova)
```
This succeeds, but I am unsure whether I understand the `.f_test()` syntax correctly. I would like to test the hypothesis that all sessions are equal.
I notice that an array of length 6 is required, though I have only 5 sessions. How come?

Can you help me out @kshedden ?
user: TheChymera
issue_3261: state: open
title: WIP: ENH Robust nonlinear estimators
body: (another PR to consolidate my robust regression code)

I added it for now to miscmodels because the models subclass GenericLikelihoodModel.

this includes a Huber-Dutter model class, see related #3258 (i.e. could be used for linear model with AR(1) errors and innovation outliers)

Also includes a few examples, classes for nonlinear functions like sigmoid and menten. Related, lmfit includes now more predefined nonlinear functions. We had related discussion around the time I wrote this in 2014. https://mail.scipy.org/pipermail/scipy-user/2014-March/035552.html
scipy.optimize leastsq also includes now some robust objective functions like Huber, since GSOC 2015

inference: AFAIR, I didn't check much, however we have now cov_type="HC0" which is the basic M-estimator sandwich also in GenericLikelihoodModel, and that should be the default for these type of functions. (RLM has an additional small sample correction based on Huber 197?)

analytical derivatives: I guess they can be build up from the different components
Hu, Hongchang. 2013. “Asymptotic Normality of Huber–Dutter Estimators in a Linear Model with AR(1) Processes.” Journal of Statistical Planning and Inference 143 (3): 548–62. doi:10.1016/j.jspi.2012.08.012.
has expressions for score and hessian (?) for Huber-Dutter that could be useful. (He uses a "nonstandard" robust norm function based on normal cdf, instead of Huber or bisquare.)

not clear (I never read about this): What do we use for global optimization, possibly many local optima with redescending weight functions? This might be more of a problem for generic nonlinear cases than for linear models. (Use basinhopping instead of the usual random starting values as in LTS?)

maybe still to come: simple and short class as prototype for RobustGMM, i.e. start with robust estimating equations which is useful if the estimating equations for mean and scale don't come from a consistent objective function.

user: josef-pkt
issue_3259: state: open
title: SUMM/ENH: endogenous or predetermined regressors with outliers (measurement errors)
body: This is a special issue coming out of additive outlier problem in time series or dynamic model, that has wider applicability.  (partial followup to #3258 RLM with AR errors)
("SUMM" issue, because I don't know if we can do anything specific yet.)

simplest case 

y_t = a y_{t-1} + b x_t + e

y_{t-1} might have a one time measurement error because of a outlier, special event, that does not effect future y's with the given dynamics.

similar: 
- multi-equation models with outliers in different equations
- endogenous explanatory variables with possible outliers
- tsa, VAR

robust multivariate estimators are a bit similar but treat variables with possible outliers symmetrically. It is closer to recent cell-wise oulier literature, but we assume we have more structure for the regressor with outliers. (Essentially, the proposal is to use an imputation solution, which has also been suggested for cell-wise outliers. see related to #3230 )

also related: explanatory variables with measurement errors, see #3187

specific in tsa: robust filtering Maronna, Martin, Yohai 2006 book section 8.6.2

idea: replace the noisy explanatory variable by it's conditional expectation
This is the same as in IV2SLS, simple measurement error models, imputation.
Then, we can handle innovation outliers with standard robust estimators.

question in analogy to IV2SLS, imputation or measurement error models: 
Do we need to adjust inference, cov_params for the extra stage of estimating the conditional expectation? Can we ignore it if we assume that there are only a few outliers?

(maybe: analogy to outlier dummies or outlier intervention models in tsa: if we estimate a parameter for the outlier effect, then inference would still be standard, AFAIK.)


 



user: josef-pkt
issue_3258: state: open
title: ENH:  robust/RLM equivalent of GLSAR or MLE with AR errors
body: what do we do if we have serially correlated errors and possible outliers?

i.e. linear regression model with AR errors plus outliers or heavy tailed distributions

see discussion at "RLM (again), with pre-whitening" (partially copied below)
https://groups.google.com/d/msg/pystatsmodels/vL-2gVMIqCs/fPrfQNGD7SwJ

objective: We want good estimates of the parameters on the regressors beta, and time series features are mostly a "nuisance". This is in contrast to the separate issue of outliers in time series analysis where the focus is on estimating the dynamics in spite of outliers (see ??? for outlier estimation in tsa)

However, we still have to distinguish at least innovation and additive ouliers (IO and AO)

- IO or heavy tailed innovations: looks like standard M- or Huber-Dutter type models should work without problems
- AO: this is an additional error to just an observation that doesn't affect future dynamics, i.e. similar to a measurement error that has only an effect in the current period

Denby and Martin provides a good comparison. M-estimators (with Huber or Bisquare weights) work for IO but not for AO. Generalized M-estimators (GM with extra weights for regressors) works for AO outliers with much smaller bias. I think this can be used as starting point.  see also #3259

Denby, Lorraine, and R. Douglas Martin. 1979. “Robust Estimation of the First-Order Autoregressive Parameter.” Journal of the American Statistical Association 74 (365): 140–46. doi:10.2307/2286743.

Handling only IO and heavy-tailed innovations would be a good and relatively simple way to get started.

---
my comments on the mailing list, slightly edited

Based on a quick search I only find this reference that sounds similar:
Coursey, Don, and Hans Nyquist. 1986. “A Procedure for Obtaining M-Estimates in Regression Models with Serially Dependent Errors.” Journal of Time Series Analysis 7 (4): 255–67. doi:10.1111/j.1467-9892.1986.tb00493.x.

However, in spite of the title, they only look at median (quantile) regression and look at stable distributions in the Monte Carlo, and have no theory.
Essentially, they put Cochran-Orcut iteration with a robust estimator, which is what I did also, if I guess correctly based on the message.

Almost all of robust statistics looks only at the iid case. There is some time series literature for pure time series analysis that I didn't look at much (except for outlier detection). So, far I only found a single working paper for sandwich robust covariance matrices with correlation across observations (like HAC/Newey-West).

Writing an RLM equivalent to GLSAR would only take a few hours, but figuring out whether the theory is correct and writing unit test usually takes me much longer (unless I have a corresponding function.in another package).
An alternative would be to use MLE with t-distribution with 3 or 5 degrees of freedom. In this case all the standard results for MLE would apply, and it would be reasonably robust to outliers or heavy tails but it wouldn't be a high breakdown estimator.

'---

Geodesy seems to be using robust estimators with (network, spatial) correlated error, e.g.

Guo, Jianfeng, Jikun Ou, and Haitao Wang. 2009. “Robust Estimation for Correlated Observations: Two Local Sensitivity-Based Downweighting Strategies.” Journal of Geodesy 84 (4): 243–50. doi:10.1007/s00190-009-0361-y.
Yang, Y., L. Song, and T. Xu. n.d. “Robust Estimator for Correlated Observations Based on Bifactor Equivalent Weights.” Journal of Geodesy 76 (6–7): 353–58. doi:10.1007/s00190-002-0256-7.

They work the other way, adjust the weighting/whitening in GLS to account for possible outliers. We cannot do this much yet, because most of our models don't have an appropriate weight option yet.
In their case the estimation problem is simpler because they assume that the clean covariance matrix of the error term is given by prior knowledge or a prior estimate. (The underlying structure looks a bit similar to measurement error models.)

I didn't find anything or not much in the statistics literature about robust estimation using the dependence structure into account, but there seems to be a lot more on the property of standard M-estimators under dependence (which sounds more like the analog to OLS + correlation consistent sandwich cov_params)


aside: if the time series properties dominate the GLSAR results, then it is necessary to take into account that there are different kinds of outliers in time series that depend on whether outlier have lasting effect, i.e. effect future observations through the dynamic structure or outliers themselves are correlated (e.g. outlier episodes that last a few periods).


Nevertheless, I think it should be possible to handle additive outliers or heavy tailed innovation case with correlation with RLM in analogy to GLS which shouldn't be too difficult.

'----

One article that sounds good
Hu, Hongchang. 2013. “Asymptotic Normality of Huber–Dutter Estimators in a Linear Model with AR(1) Processes.” Journal of Statistical Planning and Inference 143 (3): 548–62. doi:10.1016/j.jspi.2012.08.012.

He (?) estimates the AR(1) coefficient jointly with location coefficient beta and scale. This is similar to using MLE instead of GLS. However, it requires non-linear parameter estimation instead of IRLS which we currently don't have.
But the theory should be largely the same as for standard M-estimators with auxiliary scale.

(Huber-Dutter is just an M-estimators where both mean parameters and scale are derived from the same objective function. Most robust M-estimators specify the estimating equation for scale separately from the estimating equation for the mean. a bit like full MLE versus Feasible GLS.)


user: josef-pkt
issue_3257: state: open
title: GLM Fit with scipy optimizers
body: Hello,

So I was playing with the scipy optimizers and I've run across a few issues:

1. I'm not sure if `exposure` works...

```python
data = sm.datasets.fair.load_pandas()
data.data['Exposure'] = 1
data.data['Exposure'][1000:2000] = .5
data.data['Exposure'][2000:4000] = .2
data.data['Exposure'][4000:] = 0.1
model = smf.glm('affairs ~ age + rate_marriage', data=data.data, exposure=data.data['Exposure'],
                family=sm.families.Tweedie(link_power=0., var_power=1.5))
x = model.fit()
x2 = model.fit(method='lbfgs')
x3 = model.fit(method='newton')
```

`x.summary()` and `x3.summary()` give about the same results. `x2.summary()` gives you something really different... the params and z-scores are much different. 

2. I don't fully understand the following code, but if you use the scipy optimizers to fit the GLM, and the hessian is not positive-definate, the code refuses to attempt to invert the matrix. 

https://github.com/statsmodels/statsmodels/blob/master/statsmodels/base/model.py#L474

I think this behavior is correct (when you override this logic, the z-scores you get are just, well, weird.) However, the code sets `Hinv = None`, and this line of code then tries to divide `None / scale`, which returns an error. 

https://github.com/statsmodels/statsmodels/blob/master/statsmodels/genmod/generalized_linear_model.py#L939

Perhaps instead of `None`, it should return a matrix of the same shape as the hessian matrix, but with `np.nan` or 0 as the values?  
user: thequackdaddy
issue_3256: state: open
title: ENH: Reweighting estimators for outlier robust estimators
body: partially a follow-up to #3230 robust covariance, and related to ELTS for robust regression

Most high breakdown estimators, for example LTS and MCD, have low efficiency. It is common to add a reweighting step that uses a hard rejection rule for outliers based on a cutoff determined from the initial estimator.
This is build into some of the estimating functions, e.g. cov_ogk. However, there is a more systematic literature with different approaches and more explicit on the asymptotics and statistical properties.
a main reference for adaptive cutoffs is Gervini and Yohai 2002 for robust (univariate) regression.

TODO: implement this a bit more systematically so we can use it as a mix-and-match component.

Gervini, Daniel, and Victor J. Yohai. 2002. “A Class of Robust and Fully Efficient Regression Estimators.” The Annals of Statistics 30 (2): 583–616. doi:10.1214/aos/1021379866.

Marazzi, Alfio, and Victor J Yohai. 2004. “Adaptively Truncated Maximum Likelihood Regression with Asymmetric Errors.” Journal of Statistical Planning and Inference, Contemporary Data Analysis: Theory and Methods in, 122 (1–2): 271–91. doi:10.1016/j.jspi.2003.06.011.
this uses a a bit different adaptive cutoff rule than Gervini and Yohai 2002

and several more
also Cizek for adding the adaptive cutoff rule to the estimators directly










user: josef-pkt
issue_3250: state: open
title: ENH: penalized variance to prevent degenerate estimates, e.g. RLM
body: (a semi-random idea by analogy)

problem: perfect fit in RLM #1439 (and several older issues): MAD is zero of there are too many identical values (or observations in a subspace with resid=0)

In one of the original cases that raised the problem, the user ran RLM on many small samples, in some cases `mad` was zero.  If we assume that the different samples are roughly similar, then an appropriate model could be a Mixture/Random Effects model, and we would use the covariance information to "shrink" the coefficients to each other (as in panel data).
(maybe in the stype of "small area statistics") we could impose the overall information as prior/penalization on an individual sample. In that case, we wouldn't think that the Perfect Fit property of robust estimators should hold.

Since in GLM/LEF or linear model the variance is estimated separately and doesn't affect the asymptotics, we could just have a rough and a bit ad-hoc "improvement" of the scale estimate.
We would want either a user provided (from other samples) informative prior or a weak, not quite diffuse prior to keep us away from scale=0.
(I guess there should be a conjugate prior to simplify the calculations.)

Also: Bayesian language is easier for motivation, implementation will be in classical setting.

The same might be possible with other models where we estimate a variance directly (e.g. in some of the statespace models a stochastic component might become degenerate.)

a bit related estimation of regularize and robust covariance matrices #3197 #3221 (although there the focus is on covariance/correlation/off-diagonal parts, with some scale estimation)
user: josef-pkt
issue_3241: state: open
title: ENH: more options for granger causality
body: two possible enhancements (while thinking about granger causality test to answer some questions)
- the wald test "params" uses nonrobust cov: Now that we have it, adding `cov_type` as a fit option would be easy. (I never checked the literature for granger causality and sandwich cov.)
- add "`exog`": We could have other explantory variables when we test for the partial effect of the lags of a explanatory variable.

(a bit different: 
What about using the t-distribution as alternative to OLS? In finance application with possible heavy tails it is possible that the test is dominated by "outliers". related to outlier robust estimation.)

user: josef-pkt
issue_3238: state: open
title: ENH: gof, probability plots multivariate normal - small sample
body: where are the references?  
I read several references for better properties of probability plots than mahalanobis distance and chi-squared for small sample, but don't remember where.

usecase: I'm trying to make a pp-plot and a qq-plot for multivariate normality using chi2 distribution, but the upper area doesn't look very straight with nobs=100.  A clean example with nobs=1000 looks like a straight line. (application plots to illustrate multivariate outliers.)

related: multivariate normal gof tests are still missing e.g. #382
but we have some for VAR

Rencher and Christensen (2012) Methods of Multivariate Analysis, 3rd ed.,
section 4.4.2 page 106, 107 use Beta distribution (or F?) instead of chisquare
chisquare approximation for mahalanobis distance is "poor"

user: josef-pkt
issue_3237: state: open
title: Design/FAQ-D: global memoize
body: Do we want to memoize (store values) in global namespace?

It would be convenient, but it is risky when it is a global variable that never goes out of scope.
maybe: 
- store only latest or just a few values which still helps to avoid repeated calculation in loops
- try to do the memoize locally, i.e. attached to a model class.
- let users hold on to the "constants" and provide them as function argument 

example:
In robust we hardcoded some constants for the default values (e.g. constant that determine breakdown point or thresholds and constants for consistency at the normal distribution).

In other cases those constant depend on some parameters, e.g. below for truncated covariance.

```
def _coef_normalize_cov_truncated(frac, k_vars):
    # normalize cov_truncated (example ogk)
    ct = k_vars / stats.chi2.expect(lambda x: x, lb=0, ub=stats.chi2.ppf(frac, k_vars), args=(k_vars,)) 
    ct *= frac   # correction for using cov of truncated sample which uses nobs of subsample not full nobs
    return ct

class NormalizTruncCov(object):
    _cache = {}
    def __call__(self, frac, k_vars):
        return self._cache.setdefault((frac, k_vars), _coef_normalize_cov_truncated(frac, k_vars))

coef_normalize_cov_truncated = NormalizTruncCov()

print(coef_normalize_cov_truncated(0.9, 5))
coef_normalize_cov_truncated._cache


1.1782096495480963
{(0.9, 5): 1.1782096495480963}
```

user: josef-pkt
issue_3231: state: open
title: FIX: avoid zero division and log0 in loglike calculation (resolve conflicts)
body: This PR resolves the conflicts in #2301

user: bfcondon
issue_3230: state: open
title: WIP ENH: Outlier robust covariance
body: see #3221

getting started, some parts are almost clean
for other parts I need to collect functions from various scripts so I can start on a consistent collection

contains:
- stats.covariance gaussian rank correlation, quadrant correlation, and helper function to transform for consistency at the normal distribution (only regression tests)
- robust.covariance:
  - supporting function: tau_scale (currently wrong location, written to support OGK)
  - GK, OGK: verified agains R robustbase, but unit tests don't have numbers from R yet.
  - Tyler : this is currently just my old function, not verified or checked again
  - iterative weighted covariance: with t-distribution weights  (smoke test)
- scale estimators:
  - `_scale_iter`: basic iterative M-estimator for scale, IIRC, given `rho` function, approximate unit test
  - `trimmed_scale`: another old function, not reviewed again
  - ? I still have an m-scale function that uses scipy fsolve, maybe obsolete given the iterative solvers

problems to address
- perfect prediction (mad=0) again: When we use `mad` as initial estimate, we get scale=0 if more than half of the observations fit perfectly. see #1341
  - we need to handle the scale=0 special case in the calculations and add unit tests
  - we need options to choose the fraction of resid=0 at which we want scale=0 (comments in an article, which ? IIRC a tau-scale article), use trimmed variance as alternative to mad as starting value ?

Terminology
- Is scale std or var in robust? Are we consistent?  
  `scale_trimmed` returns standard deviation.  
  (difference scale in distribution is std (sigma), scale in regression is var (sigma**2)
- covariance, scatter, shape in elliptical distribution
  - `cov` either covariance or scatter matrix, as returned by functions, scaling depends on options or implementation details 
  - covariance: expected second moment, (of distribution parameter or random variable)
  - scatter: covariance that is not necessarily scale or has indeterminate scale, e.g. Tyler
  - shape: covariance matrix scale so that det(cov) = 1, i.e. covariance = scale * shape
- center of location: center, median, mean, loc. What is the generic term?
  - `mean` returned by functions for the center, i.e. mean or expected value of data excluding outliers  
    ok for symmetric distribution where mean is the center, e.g. median estimates the mean.
    Need to be more explicit for skewed distributions, but main value of interest is mean (expected value), but we also have median and location parameters which might differ.

user: josef-pkt
labels: superseded: Used for PRs with content that is superseded by a newer PR
issue_3227: state: open
title: BUG: clip_evals uses wrong indicator
body: ```
def clip_evals(x, value=0): #threshold=0, value=0):
    evals, evecs = np.linalg.eigh(x)
    clipped = np.any(evals < 0)
    x_new = np.dot(evecs * np.maximum(evals, value), evecs.T)
    return x_new, clipped
```

we are clipping based on value and not necessarily at 0
`clipped = np.any(evals < value)`

this is used as indicator in the functions to see whether clipping was necessary. So, that's incorrect if threshold (value) != 0.
(in examples, `value` was still tiny like 1e-15, IIRC, just to get positive definiteness.  So there is a small imprecision in the results because of this bug)

user: josef-pkt
issue_3224: state: open
title: SUMM/ENH: nonlinear transformation to normality - utility functions
body: #3220 We use transformation in several places to make the data or statistics closer to normally distributed.

However, we don't have a central collection of recipes or helper functions.

We have the delta method for inference based on normal distribution. It's used in a "hardcoded" way in some of the hypothesis tests and related, e.g. for proportions and correlation coefficients. 
We also have it for nonlinear prediction and for nonlinear transformation of parameters, e.g. `eform` in Logit and Poisson.
For transforming data we have Box-Cox and related transformation.

missing reference: quite a while ago I saw a reference that mentioned general variance stabilizing transformtion, maybe by integrating the variance function. I don't remember, and it was just in the context of discussing an implementation detail in a PR (?).

case right now: I'd like to use something like Fisher z-transform for spearman rho and kendalls tau to calculate confidence intervals (but I'm not sure I remember correctly, and don't have a helper function handy.)

aside for confidence intervals:
It looks like we go in different directions, transformation to normality or transformation from normality.
**from normality**: In the `eform` (exp(params)) case we assume params has a pretty close to normal distribution, and we we are interested in the non-normal transformed variable.
**to normality**: When using Fisher's z-transform we are interested in the original statistic, but we do the transformation to have a better approximation by the normal distribution.
End point transformation for confidence intervals needs to go in the right direction, i.e. we cannot directly reuse the `eform` transformation code.

example proportion
AFAICS: we don't use any transformation in statsmodels.stats.proportion
http://support.sas.com/documentation/cdl/en/imlsug/62558/HTML/default/viewer.htm#ugvartransform_sect6.htm lists a few available transformations.
We have some of them in the context of GLM and the discrete models. but not for pure stats, AFAIR.
related parameterized transforms or link functions for binomial proportion
http://stats.stackexchange.com/questions/10975/transforming-proportion-data-when-arcsin-square-root-is-not-enough

(barely related aside: Can we do a GLM like model with a Fisher z-transform link function to estimate a regression model for correlations? in analogy to logit for proportions. I guess that should be better than least squares, i.e. minimize MSE. Gaussian MLE has proper objective but full parametric assumption. ???)

user: josef-pkt
issue_3223: state: open
title: cross-validation with outliers - robust evaluation criteria
body: (I'm not sure where this fits yet, eval_measures)

When we want to do cross-validation for example for penalization in outlier robust estimation, then we need evaluation criteria that are not distorted by outliers.

example penalized covariance estimators:
Öllerer, Viktoria, and Christophe Croux. 2015. “Robust High-Dimensional Precision Matrix Estimation.” In Modern Nonparametric, Robust and Multivariate Methods, edited by Klaus Nordhausen and Sara Taskinen, 325–50. Springer International Publishing. doi:10.1007/978-3-319-22404-6_19.

use the robust covariance estimate of the validation sample, left-out sample in k-fold cross-validation, to calculate the likelihood distance.
implementation in scikit learn: the function `graph_lasso(emp_cov, alpha, cov_init=None, ...` can use a robust `emp_cov`, but for cross-validation every occurence of `emp_cov` needs to be replaced by a robust empirical covariance estimator.

The same will apply to other regularized covariance estimators like Tyler if cross-validation is used.

---

I never seen or looked at it in regression setting, but there are articles on robust penalized splines which will run into this problem. (I haven't read any of those.)
My guess is that we need to use something like median prediction error or a higher quantile of the prediction error, or of whatever observation wise distance measure we use. That would capture residual (y-) outliers, If we have a robust distance measure for the exog, then we could also ignore x-outliers in the evaluation sample (but that might not be necessary if we already ignore large prediction errors because x-outliers in the prediction sample by themselves will not distort the estimation, maybe, maybe not.)

user: josef-pkt
issue_3222: state: open
title: ENH: hac (long run covariance) with iweights (oulier robustness)
body: I'm looking at outlier robust covariance estimators #3221, but I'm using a time series.

Can we add observation weights capturing "inlyingness" to the HAC aggregation of auto-covariances?
If we just use threshold or hard-rejection weights, then this would correspond to HAC with missing observations.

This assumes that observations are outliers, instead we might think of outliers in the correlation (i.e. think of stacked data as in VAR).

(I'm not sure how this works and no reference. Triggered because I used a cointegration test dataset for the robust covariance calculation, i.e. some variables in the data are integrated I(1))

user: josef-pkt
issue_3221: state: open
title: SUMM/ENH: (outlier) robust covariance estimators
body: getting started with a wishlist for outlier robust multivariate location and scatter estimators
#3220  size (overall scaling)
- MCD in scikit-learn, not good with high contamination and large k_vars
- Tyler scatter: implemented including shrinkage, k_vars > nobs possible, breakdown point ? It's not in the usual list
- OGK: implemented, high breakdown point and works with large k_vars
- Olive Hawkins : matlab has it http://www.mathworks.com/help/stats/robustcov.html#bu48rby-1_1
  Olive is not cited much by the usual robust guys, but sounds interesting, e.g. DetMCD and others follow a bit his ideas of smart selection of initial sets
- DetMCD: does pretty well, a bit better than OGK and also works with larger k_vars
- S-, tau-, MM-estimators ???
- M-estimators: low or zero breakdown point, but useful for refinement step and for cases where we don't need high breakdown point.
- Stahel-Donahue: is often best estimator in smaller k_vars, not cheap, 
- MVE: comes back into fashion because it's low bias, as starting point for MM, tau or similar
  Maronna (?) argues in favor for this purpose. low efficiency of MVE doesn't matter in this case.
- other starting covariances or correlation converted to covariance estimators. i.e. use same idea as DetMCD to get starting candidates for cov estimators that are efficient and consistent at multivariate normal
- and there are more

aside:
My impression in reading the high breakdown literature again is that outliers are really observations from a different population. 
In contrast, references like Tyler and shrinkage estimators emphasize more general estimation problems because of small samples or heavy tailed distributions where we just want to get a better estimate that is less sensitive to "peculiarities" of the specific sample.

some of the main intended usages:
- weights for GLM, discrete and nonlinear models to downweight influential observations based on robust distance measure
- linear multivariate methods, e.g. multivariate regression, robust PCA, ...
- making covariance matrices less noisy (where we don't necessarily care about breakdown points as in #2942)

References:

I started to collect those at https://github.com/josef-pkt/misc/wiki/robust:-multivariate---References

user: josef-pkt
issue_3220: state: open
title: Reference: scaling of scatter matrix to get covariance
body: (parking a reference to computational detail)

how do we normalize a scatter matrix so that it is consistent for specific distribution, commonly the normal
cov = sigma = c scatter -> find "size" c

related: mad, iqr and similar have normalization constants
here it is for the multivariate case

Maronna et al text book on robust statistics 2006 section 6.3.2 on page 186
using chi2 distribution for mahalanobis distances we can calculat
c = median( {d_i}_i ) / chi2.ppf(0.5, k_vars)

this has also been used without reference in Maronna and Zamar 2002 on cov_ogk.
I didn't see anything mentioned for the "size" estimates in Tyler estimator for scatter in elliptical distribution.

There are several references (*2) for consistency and small sample scaling of MCD and similar but I didn't look carefully (brief browsing or skimming doesn't show any obvious answer)
Many articles just mention the scaling factors but they don't show the numbers or formulas.

Maronna, Ricardo A., Douglas Martin, and Víctor J. Yohai. 2006. Robust Statistics: Theory and Methods. Reprinted with corr. Wiley Series in Probability and Statistics. Chichester: Wiley.

Maronna, Ricardo A., and Ruben H. Zamar. 2002. “Robust Estimates of Location and Dispersion for High-Dimensional Datasets.” Technometrics 44 (4): 307–17. doi:10.1198/004017002188618509.

(*2)
Hardin, Johanna, and David M. Rocke. 2005. “The Distribution of Robust Distances.” Journal of Computational and Graphical Statistics 14 (4): 928–46. doi:10.1198/106186005X77685.
Pison, G., S. Van Aelst, and G. Willems. n.d. “Small Sample Corrections for LTS and MCD.” Metrika 55 (1–2): 111–23. doi:10.1007/s001840200191.

user: josef-pkt
issue_3215: state: open
title: CLN/BUG: status of densityorthopoly.py 
body: scipy has a deprecation discussion for some of the orthogonal polynomial functions. `scipy gh-6591`

"statsmodels\sandbox\nonparametric\densityorthopoly.py"
needs most likely a refactoring to use numpy.polynomial

no unit tests
one python 3 incompatibility, and one refacturing error.
otherwise runs still fine.

https://github.com/statsmodels/statsmodels/compare/master...josef-pkt:fix_densityorthopoly

last example that is printed is not orthonormal: Is this expected or something changed?

user: josef-pkt
issue_3212: state: open
title: Forecasting the unknown future
body: Hi, 
First of all thanks for putting this ARIMA summary online.
Second, I have a question. How would someone make predictions on the future (test set) if observed data are still unknown (NaN). Here is a sample code I tried https://github.com/kthouz/Misc/blob/master/arima_tuto.ipynb

user: kthouz
issue_3211: state: open
title: add 2 methods with continuity correction (CC)
body: 1. add 2 methods with continuity correction (CC) for Normal and Wilson methods respectively; 
2. add the paper of Newcombe on  single proportion  which is the mostly sited paper on this topic.
3. cosmetic changes.

user: Jiangtang
issue_3210: state: open
title: ENH: penalization structures: combination selection and targeted shrinkage
body: Sparse Laplacian Shrinkage combines a L1 based penalty and a quadratic informative penalty, similar to glm-net but with structured L2 penalization matrix

Sparse Laplacian Shrinkage is the first strand of literature that I ran into that uses this penalization structure

Huang, Jian, Shuangge Ma, Hongzhe Li, and Cun-Hui Zhang. 2011. “The Sparse Laplacian Shrinkage Estimator for High-Dimensional Regression.” The Annals of Statistics 39 (4): 2021–46. doi:10.1214/11-AOS897.

and references in #3209

This links L1-based variable selection with generalized Ridge as in TheilRidge or GAM/penalized splines, and general structured penalization.
I don't find an issue with the "munich" article. (e.g. penalization of categorical variables towards combining levels.)

user: josef-pkt
issue_3209: state: open
title: ENH: generalized Ridge penalization for multivariate linear model
body: Does it have a closed form solution? 
I guess so. Do we need to stack or can we work with the multivariate model?
If we need to stack, then we need to handle the reshaping, but otherwise could use a generic penalized linear model, preferably taking advantage of the block sparse structure in stacking.

example and application: 
VAR with Minnesota priors.
penalization based on spatial or network structure

I was looking at sparse VAR estimation using elastic net with a full structured quadratic penalization (params dot L dot params) where the smoothing target comes from a prior knowledge about the network structure.

As a first version just extending generalized Ridge to linear multivariate or SUR model would be a good starting point.

partially related references that triggered the issue

Jiang, X., X. Hu, W. Xu, and E. K. Park. 2015. “Predicting Microbial Interactions Using Vector Autoregressive Model with Graph Regularization.” IEEE/ACM Transactions on Computational Biology and Bioinformatics 12 (2): 254–61. doi:10.1109/TCBB.2014.2338298.
(not very clear in details, I didn't see them mentioning from where they get their graph laplacian "prior", only VAR(1). They are possibly cheating, predictive performance is much better than the penalization methods that don't use the graph "prior".)

method refers to this
Li, Caiyan, and Hongzhe Li. 2008. “Network-Constrained Regularization and Variable Selection for Analysis of Genomic Data.” Bioinformatics 24 (9): 1175–82. doi:10.1093/bioinformatics/btn081.

aside: quadratic informative prior is incorporated as artificial observations.

user: josef-pkt
issue_3208: state: open
title: ENH: multivariate (multiple endpoint) stats
body: We should get extension for one, 2 or k-sample tests to multivariate samples, i.e. the analogue of some of the t_test, ... to the multivariate case.

one reference (that I didn't read): equivalence tests for multivariate means

Hoffelder, Thomas, Rüdiger Gössl, and Stefan Wellek. 2015. “Multivariate Equivalence Tests for Use in Pharmaceutical Development.” Journal of Biopharmaceutical Statistics 25 (3): 417–37. doi:10.1080/10543406.2014.920344.


missing descriptive statistics
multivariate skewness and kurtosis - Mardia see #3280 for usage in correcting chi2 statistic

user: josef-pkt
issue_3202: state: open
title: ENH: Small Area Estimation
body: (parking another reference to a topic that we don't have, and where I don't know much)

also related to survey methods, which we don't have either (Stata's svy prefix and related methods)

Pfeffermann, Danny. 2013. “New Important Developments in Small Area Estimation.” Statistical Science 28 (1): 40–68. doi:10.1214/12-STS395.
sounds like a good recent overview, found while looking for expectiles and M-quantiles for #3183

another related issue: sampling from a finite population  (Wooldridge ?)

user: josef-pkt
issue_3201: state: open
title: FAQ panel data, GEE and MixedLM for econometricians
body: We should get some simple recipe notebooks to illustrate how GEE and MixedLM can be used by users with an econometrics background that are not familiar with the "statistics" models in GEE and MixedLM.

e.g. There was a question a while ago how to do simple two-way crossed effect, and I didn't find a quick example.
related #705

user: josef-pkt
issue_3200: state: open
title: ENH: GEE add panel-HAC cov_type
body: I'm reading the motivation for regularized sandwiches in GEE in Warton 2011 referenced here
https://github.com/statsmodels/statsmodels/issues/2942#issuecomment-246756851

cluster robust covariances don't work (well) if the number of observations per cluster is large compared to the number of clusters.
Warton proposes regularization, shrinking towards the working correlation/covariance matrix.

Instead, if the observations within a cluster are ordered, then this also goes in the direction of panel time series, large T, smaller n_groups, where we can use one of the panel HAC robust sandwich covariances.

The same should apply to GEE.
The advantage compared to GLM with various cov_types is that we get more options for working correlation instead of just independence, and we would get the associated efficiency gains.
Compared to linear model we get the additional families besides gaussian.

aside: Warton uses GEE for multivariate (endog) data, i.e. "groups" are different variables for the same observation.

user: josef-pkt
issue_3199: state: open
title: ENH: multivariate oulier prediction, Hotelling T2
body: related: #2847 for univariate case

interesting question
http://stats.stackexchange.com/questions/234837/is-there-a-hotellings-t2-function-for-multivariate-data-using-python
and link to blog post
https://blog.pivotal.io/data-science-pivotal/products/multivariate-time-series-forecasting-for-virtual-machine-capacity-planning

It's related to multivariate outlier detection, but we want to get a test statistic or prediction interval based on the multivariate normal distribution. e.g. what's the distribution of the predicted Mahalanobis distance of a new observation, or the confidence interval based on it.
The example is based on VAR predictions.

(I have not much background for this. I guess it's essentially 2 to 5 lines of code but which, and a lot more lines for unit tests.)
partially a guess: we can just use F or chi-square distribution on the Mahalanobis distance, and for those we should have the normal prediction distribution (expected value and covariance) for the VAR predictions already available.

related aside: textbooks on Multivariate Statistics have the mean and covariance tests for multivariate normal random variables. Those are not yet available in Python scipy or statsmodels.

user: josef-pkt
issue_3197: state: open
title: ENH: add penalized covariance/correlation and precision estimate
body: There are many cases where an estimated covariance does not have good small sample properties.

We need some regularization methods (beyond what scikit-learn has)
One of the main tasks will be to provide different covariance or correlation patterns as shrinkage targets.
e.g. correlation_tools has functions to force covariance matrices to be positive definite or have a factor structure..
In other cases we want to shrink for example towards one of the GEE covariance patterns.

related #2942 regularized cov_params/weights or covariance in GEE/GMM

(opened for now so I can park a script in the comments.)

user: josef-pkt
issue_3191: state: open
title: Convergence information being displayed during ARIMA.fit() regardless of disp value
body: ``` python
>>> sm.version.full_version
'0.8.0.dev0+bb1db2b'
```

``` python
arima_model = ARIMA(transformed_data, order=(model_order[0], model_order[1], model_order[2]))
arima_results = arima_model.fit(disp=False)  # or disp=-1
```

```
RUNNING THE L-BFGS-B CODE

           * * *

Machine precision = 2.220D-16
 N =            3     M =           10
 This problem is unconstrained.

At X0         0 variables are exactly at the bounds

At iterate    0    f=  7.16189D+00    |proj g|=  1.29181D-01

At iterate    5    f=  7.00504D+00    |proj g|=  1.47562D-03

At iterate   10    f=  7.00442D+00    |proj g|=  1.79098D-02

At iterate   15    f=  6.93115D+00    |proj g|=  2.95095D-01

At iterate   20    f=  6.84326D+00    |proj g|=  7.16658D-04

           * * *

Tit   = total number of iterations
Tnf   = total number of function evaluations
Tnint = total number of segments explored during Cauchy searches
Skip  = number of BFGS updates skipped
Nact  = number of active bounds at final generalized Cauchy point
Projg = norm of the final projected gradient
F     = final function value

           * * *

   N    Tit     Tnf  Tnint  Skip  Nact     Projg        F
    3     23     25      1     0     0   4.230D-07   6.843D+00
  F =   6.8432611444769238

CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL

 Cauchy                time 0.000E+00 seconds.
 Subspace minimization time 0.000E+00 seconds.
 Line search           time 0.000E+00 seconds.

 Total User time 0.000E+00 seconds.
```

user: philipobrien
issue_3187: state: open
title: SUMM/ENH: measurement errors with known distribution
body: (another topic where I don't have an overview and no ready answers)
#1097 for linear model (simple cross-sectional)

scipy also has orthogonal distance regression (where I never got beyond one toy example)

in the case of descriptive statistics: some comments
https://github.com/statsmodels/statsmodels/issues/3186#issuecomment-244441367

Most of repeated measures, cluster, panel still assume the exog are observed, but random intercept becomes similar for repeated measurements of endog.
multivariate models including models like VAR have noise in all variables. Factor analysis, latent variables and similar (statespace models, system of equations) are tools at least for the everything-is-normally distributed case.

special case: deconvolution with known second distribution.

How do these things fit together, and what are the functions and models that we need?

application area mostly engineering and natural sciences with "hard" measurements and known  measurement error of (real) instruments.
(I think it doesn't show up much in social sciences because there is a huge amount of "noise" everywhere and "endogeneity" problems are the main analogue of measurement error.)

semi-random related areas
- computer simulation with known simulation error, e.g. bootstrap. (like surface fitting for obtaining the asymptotic distribution or p-values)
- sampling from a finite population, survey sampling: weighting depends on sampling scheme to recover the noisily observed population statistics. (but weight is observation and not variable dependend)
- item response models: multiple measurements on underlying latent factors, (I don't know much about this)

user: josef-pkt
issue_3186: state: open
title: Per observation weigts for stats.weightstats.DescrStatsW
body: Currently weights are applied for each observation across all variables (ie: per column).

It would be nice to allow different weights for each observation/variable pair. 

user: drevicko
issue_3185: state: open
title: FAQ-D: faster linalg shortcuts
body: some faster linear algebra calculations than the textbook version
(we should collect some shortcuts so we don't have to remember or figure them out each time)
## trace

trace of a dot product (used for example in MLE of covariance matrix under normality)

```
>>> a = np.random.randn(5,5)
>>> b = np.random.randn(5,5)
>>> np.trace(b.T.dot(a))
1.5560338812520342
>>> (b*a).sum()
1.5560338812520342

>>> b = np.random.randn(5,3)
>>> a = np.random.randn(5,3)
>>> np.trace(b.T.dot(a))
1.0538616102922069
>>> (b*a).sum()
1.0538616102922076
```
## diagonal matrix

...
## vectorized quadratic forms (e.g. for prediction, t-test)

```
>>> s = np.random.randn(5,5)
>>> x = np.random.randn(3,5)
>>> for x_i in x: print(x_i.dot(s).dot(x_i))
-22.6038544773
-1.32502755596
-4.41106410399

>>> (x.dot(s) * x).sum(1)
array([-22.60385448,  -1.32502756,  -4.4110641 ])

# my first try using einsum
>>> np.einsum('ji,ik,jk -> j', x, s, x)
array([-22.60385448,  -1.32502756,  -4.4110641 ])
```

user: josef-pkt
issue_3181: state: open
title: ENH: outlier robust cov, transformation to normality, copulas
body: (semi-random ideas, while thinking about robust or quantile regression)

objective:
find inlier `exog` observations so we can identify outliers that are possibly influential observations. If we downweight or remove those influential observations, we can get robust estimators (RLM currently only handles endog/residual outliers)

problem: robust cov like MinCovDet in scikit-learn are designed for normal case and use Mahalanobis distance.

What if distributional assumption is strongly violated, e.g. discrete, binary or count data or strongly asymmetric distributions like log-normal?

-> treat restricted support variables separately
-> try transformation of marginals (i.e. univariate variables), e.g. ECDF/rank, inverse probability transform, marginal robust standardization
-> calculate robust correlation or covariance in transformed space, rank correlation, copula correlation with ECDF marginals.

specifically: 
helper function for transforming to normality via ECDF/rank
What is the outlier robustness behavior of a gaussian copula with ECDF marginals?

another possible idea: use robust regression for the exogs similar to MICE if outliers are just in some variables. (similar to idea for getting inverse covariance or partial correlation, e.g. for large number of variables.)

(I would like something that is fast and works with reasonable examples even if it doesn't have a very high breakdown point. Usage is mainly as pre-estimation tool, or can be iterated with main regression.)

user: josef-pkt
issue_3180: state: open
title: ENH: using weights for cross-validation
body: When we have arbitrary weights implemented, then we can use them also for cross-validation, i.e. set the weights for the left-out observations to zero, and predict for them.

The advantage might be minimal overhead for estimating the leave-k-obs-out solution.

(application: penalization splines for RLM type and quantile or expectile regression, where we already have weights for IRLS.)

user: josef-pkt
issue_3178: state: open
title: PHReg predicted hazard ratios size do not match with test set size
body: The prediction of the hazard ratios with the PHReg model is somehow constrained to the size of the train set, is this correct?

I don't see this behaviour in LifeLines python package https://github.com/CamDavidsonPilon/lifelines.

Small working example bellow.

```
import numpy as np
from statsmodels.duration.hazard_regression import PHReg

# Data
n, p = 500, 5
exog = np.random.normal(size=(n, p))
lin_pred = exog.sum(1)
endog = -np.exp(-lin_pred)*np.log(np.random.uniform(size=n))


# Train model - 200 observations
model = PHReg(endog[:200], exog[:200]).fit()
print model.summary()


# Test model - 300 observations
hr_pred = model.predict(exog[200:], pred_type='hr')

print len(hr_pred.predicted_values)
# Output: 200 (it should be 300, the same size as the exog[200:])
```

Thank you,

user: EmanuelGoncalves
issue_3176: state: open
title: FAQ/Reference: fitting unbalanced SUR with GEE
body: http://www.stata-journal.com/article.html?article=st0079
McDowell, A. 2004. “From the Help Desk: Seemingly Unrelated Regression with Unbalanced Equations.” Stata Journal 4 (4): 442–48.
#2500  sysreg requires balanced data (all equation have the same number of observations, identical index), also required by Stata's reg3 and R systemfit.

partially related: reg3 and systemfit allow unstructured or diagonal cov_resid. In analogy to GEE and spatial it would be possible to allow for richer "contemporaneous" correlation structures.

---

more general: SUR with unbalanced data
(I read something recently but don't remember the reference or which ones I read. 
Several versions for SUR that produce identical results in the balanced case but not in the unbalanced case, going back to Schmidt 1977
related: covariance or correlation calculations in GEE and cluster robust standard errors in unbalanced case)

Schmidt, Peter. 1977. “Estimation of Seemingly Unrelated Regressions with Unequal Numbers of Observations.” Journal of Econometrics 5 (3): 365–77. doi:10.1016/0304-4076(77)90045-8.

Baltagi, Badi H., Susan Garvin, and Stephen Kerman. 1989. “Further Monte Carlo Evidence on Seemingly Unrelated Regressions with Unequal Number of Observations.” Annales d’Économie et de Statistique, no. 14: 103–15. doi:10.2307/20075741.

Foschi, Paolo, and Erricos J. Kontoghiorghes. 2002. “Seemingly Unrelated Regression Model with Unequal Size Observations: Computational Aspects.” Computational Statistics & Data Analysis, Matrix Computations and Statistics, 41 (1): 211–29. doi:10.1016/S0167-9473(02)00146-9.
———. 2003. “Estimating Seemingly Unrelated Regression Models with Vector Autoregressive Disturbances.” Journal of Economic Dynamics and Control 28 (1): 27–44. doi:10.1016/S0165-1889(02)00105-7.

user: josef-pkt
issue_3173: state: open
title: ENH: AR (Hannan-Risannen) based specification/outlier search
body: The original Hannan-Risannen idea was to search for ARMA lag lengths by using faster OLS/AR estimation.

I've seen it mentioned somewhere (?) for using it for outlier detection.

This would be faster than looping over ARIMA MLE, if we want to, for example, test all possible outlier dummies.  It won't work well with very short time series.

Additionally, with OLS we could use standard penalization and regularization methods for variable (outlier, jump, ...) selection.

user: josef-pkt
issue_3172: state: open
title: FAQ-D: backporting to maintenance
body: notes based on backporting for 0.8

`git log --merges origin/master --pretty=oneline --abbrev-commit`

prepare .bat file for cherry-picking, assuming git log is in string `ss`:

```
print('\n'.join(s for s in ss.splitlines()[::-1])) # print in merge sequence
# edit: comment out undesired merge commits
# cherry-pick, if reversed merge list is stored in ss_rev string
cp_template = 'git cherry-pick %s -m 1'
print('\n'.join(cp_template % s[:7] for s in ss_rev.splitlines() if s[0] != '#'))
# save to bat file and run it on top of a maintenance branch checkout
```

user: josef-pkt
issue_3165: state: open
title: FAQ-D scipy.signal.lfilter has correct initial conditions
body: While looking at a branch (not PR) of Kevin, I saw that scipy.signal.lfilter now treats initial condition correctly. That is, lfilter without zi replicates recursive ARMA filtering or generate_sample without having to specify zi, with zero starting values. (I tried only for 1 lag ARMA)

AFAIR, this didn't work in the "old times", i.e. 6 to 8 years ago.

example:

```

import numpy as np
from scipy import signal

from statsmodels.tsa.arima_process import arma_generate_sample

x = np.arange(10.)

ar = [1, -0.9]
ma = [1, 0.1]
#ma = [1, 0]
y1 = arma_generate_sample(ar, ma, nsample=10, distrvs=lambda n: x)
expected = x.copy()
for i in range(1, 10):
    expected[i] = -ar[1] * expected[i - 1] + expected[i] + ma[1] * x[i - 1]

y2 = signal.lfilter(ma, ar, x)

print(np.column_stack((y1, y2, expected)))
```

produces:

```
[[  0.          0.          0.       ]
 [  1.          1.          1.       ]
 [  3.          3.          3.       ]
 [  5.9         5.9         5.9      ]
 [  9.61        9.61        9.61     ]
 [ 14.049      14.049      14.049    ]
 [ 19.1441     19.1441     19.1441   ]
 [ 24.82969    24.82969    24.82969  ]
 [ 31.046721   31.046721   31.046721 ]
 [ 37.7420489  37.7420489  37.7420489]]
```

user: josef-pkt
issue_3163: state: open
title: check that we `always` warn 
body: (I didn't find an open issue, but thought there should still be one.)

If we need `warnings.simplefilter('always')` in a unit test to check assert_raises, then it could be that the warning filter is not set correctly

e.g.
https://github.com/statsmodels/statsmodels/pull/3083/files#diff-8a6a1e1a044465488e2fd3b73e9ffac0R645

AFAIK, we still have no systematic checks that warnings are always raised (in cases where we want this).

I don't know of a specific "bug".

related #2493  some warnings in the test suite showed up depending on the versions used

user: josef-pkt
issue_3156: state: open
title: Logit/MNLogit unidentified parameter, no variation
body: http://stackoverflow.com/questions/38932129/mnlogit-regression-singular-matrix-error/38933654#38933654

One binary explanatory variable always has P(endog=1, ex=1) = 1

investigation: check eigenvalues and eigenvectors of Hessian, which in this case has 1 zero eigenvalue, with all weight on the offending exog.

I don't remember the details of the theory for this. What are the identification conditions?

Can we add some helper functions or code to identify these cases?
Note: just looking at exog itself doesn't show anything because it is well conditioned. We need some tools for partial separation checks.

This is related to perfect or partial separation. We have several issues open for that.
#516 #2282 #1705

In this case looking at the predicted values didn't make me suspicious enough

using the MNLogit version of the model, I got:

```
>>> fitted.max(0)
array([ 0.99796101,  0.94396116])
>>> fitted.min(0)
array([ 0.05603884,  0.00203899])
```

odds-ration looks informative (this is using Logit(...).fit_regularized)

```
>>> fitted[fitted[:,0] > 0.99, :].shape
(92, 2)
>>> np.exp(mod.params)
Intercept         1.840790e-01
goal_angle        1.031292e+00
goal_distance     1.210362e+00
np_distance       8.270958e-01
fp_distance       9.143966e-01
is_fast_attack    1.131565e+00
is_header         3.969808e-01
prev_tb           3.039952e+00
is_rebound        1.197888e+00
is_penalty        1.539024e+01
prev_cross        1.028040e+00
is_tb2            1.867687e+00
is_own_goal       8.252994e+11
is_cutback        9.073262e-01
asst_dist         9.930969e-01
dtype: float64
```

user: josef-pkt
issue_3154: state: open
title: FAQ/ENH: time series analysis in other Python packages
body: I don't have an overview anymore on python packages.

In the old times, there was just nitime

scot includes VAR for connectivity https://github.com/scot-dev/scot
    (I didn't look around much, but there might be some overlap, and maybe with VAR extensions that we don't have yet. VAR-ICA ?)
for example one thing we don't have for VAR/multivariate time series is multivariate Yule-Walker or Levinson-Durbin.
Interesting would be other packages that have sparse VAR  (#2933), e.g. network and connectivity analysis

user: josef-pkt
issue_3153: state: open
title: ENH: MLE loglike for multivariate gaussian model, covariance
body: We need helper functions or classes for the multivariate Gaussian model. We are starting to use it in several models, but a simple standalone class would be useful for rapid development, and maybe as Mixin.

multivariate means here; balance panel, or several endog variables with _paired_ observations (same nobs for all variables)
(avoids problem with unbalanced panel as in GEE.)

specific case: estimating covariance
After concentrating out the mean estimate, the loglike is a product of empirical and model defined covariances (a bit more complicated, check).

user: josef-pkt
issue_3152: state: open
title: SUMM: outlier robust descriptive statistics, including correlation, acf
body: partially a followup to #838

other statistics for which we should also get outlier robust estimators

cov

...

**acf, pacf**

Dürre, Alexander, Roland Fried, and Tobias Liboschik. 2015. “Robust Estimation of (Partial) Autocorrelation.” Wiley Interdisciplinary Reviews: Computational Statistics 7 (3): 205–22. doi:10.1002/wics.1351.
(overview article, found by chance, I didn't search systematically)
(part of outlier handling in time series analysis, more specific issues are for parametric intervention based models)
(I also have a draft for nonparametric trimming/winsorizing based of fft)

user: josef-pkt
issue_3151: state: open
title: ENH: Add significance test for elastic_net
body: This add the basics for handling the cov test described in 

http://statweb.stanford.edu/~tibs/ftp/covtest.pdf

for the elastic_net code.  I'm working on the tests now but I wanted to post this so that people can get a sense of the structure of things.  See the covTest R package for a reference on previous implementations.  @kshedden let me know if you have any thoughts.

One thing to note, this implements a separate CovTestResults class similar to WaldTestResults, at some point it might make sense to do some restructuring with constrast.py in general but that's probably beyond the scope of this PR. 

user: lbybee
issue_3150: state: open
title: ENH/REF: missing handling in post-estimation with exog
body: see #3149 #3087 for generic `Results.predict`  missing are "propagated"

a analogous problem will show up in other post-estimation methods that have an exog as argument, e.g. get_prediction, get_margins and similar.

We need to add this to a generic exog handling helper function for those methods.

impact: 
"propagate" sounds reasonable in prediction that only uses single row processing (i.e. not cross observation dependence). 
tsa or cluster/panel structure might cause additional problems with cross-observation nan propagation.
We don't have systematic tests (or any at all except the OLS case in #3149) for missing in post-estimation.

user: josef-pkt
issue_3148: state: open
title: BUG/ENH: SVAR  indeterminacy in sign of B
body: see https://github.com/statsmodels/statsmodels/issues/3128#issuecomment-238673265
and following comments

with scipy 0.8.0 the estimated B matrix in unit test has negative diagonal.

AFAICS, this is an equivalent solution to positive diagonal because only square is used
i.e. negative standard deviation (linear transformation) implies the same variance (diagonal covariance matrix in test case)

(Pauli replied on the mailing list that there shouldn't be a change in fmin related to step size or treatment of starting values.)

user: josef-pkt
issue_3145: state: open
title: BUG/ENH: KDEMultivariate cv_ls is fragile, local minima
body: least squares cross-validation in KDEMultivariate can produce widely different results after small changes in the data.
There is currently no option to influence the optimization with fmin in `_kernel_base.GenericKDE._cv_ls`

http://stats.stackexchange.com/questions/228811/problem-with-least-squares-cross-validation-in-statsmodels

a reference from a brief search http://www.stat.washington.edu/courses/stat527/s13/readings/Sheather_StatSci_2004.pdf
points to the possibility of multiple local minima

```
"""
http://stats.stackexchange.com/questions/228811/problem-with-least-squares-cross-validation-in-statsmodels
"""

import pandas as pd
import statsmodels.api as sm
#this is the original data with the y value of the last moved slightly
df_works = pd.DataFrame([[  538139.47293084,  3506243.03653242],
                         [  538150.09004195,  3506237.73434416],
                         [  538080.50079698,  3506151.83736346],
                         [  538124.70281065,  3506254.52157404],
                         [  538136.83381139,  3506245.77673011],
                         [  538155.19176266,  3506217.94499408],
                         [  538157.14459963,  3506216.01139807],
                         [  538120.47723684,  3506250.05577059],
                         [  538094.69172001,  3506211.3862365 ],
                         [  538140.71494773,  3506238.02450588],
                         [  538150.71773573,  3506242.5]])

kde = sm.nonparametric.KDEMultivariate(df_works, "cc", "cv_ls")
print("kde.bw", kde.bw)
kde = sm.nonparametric.KDEMultivariate(df_works, "cc", "cv_ml")
print("kde.bw", kde.bw)

df_works2 = df_works.copy()
df_works2.iloc[-1, -1] = 3506242.01951111

kde2 = sm.nonparametric.KDEMultivariate(df_works2, "cc", "cv_ls")
print("kde.bw", kde2.bw)

kde2 = sm.nonparametric.KDEMultivariate(df_works2, "cc", "cv_ml")
print("kde.bw", kde2.bw)
```

user: josef-pkt
issue_3144: state: open
title: FAC/DOC  fit_constrained is missing a notebook
body: see example here

http://stackoverflow.com/questions/29261018/how-to-add-sum-to-zero-constraint-to-glm-in-python/29266239#29266239

TODO:
to avoid patsy dropping collinear columns for categoricals, we need to build a dataframe and call patsy.dmatrix several times with each categorical variable.

user: josef-pkt
issue_3143: state: open
title: ENH/Reference: time series bootstrap VAR/VECM
body: (parking a topic with recent literature)

there are several papers related to parametric residual bootstrap and other time series bootstrap

Cavaliere, G., Rahbek, A. and Taylor, A. M. R. (2012), Bootstrap Determination of the Co-Integration Rank in Vector Autoregressive Models. Econometrica, 80: 1721–1740. doi:10.3982/ECTA9099
and several others by same authors
results look good in terms of size and power to identify correct cointegration rank (much better than asymptotics in small samples)
plain implementation should be just some short code, once we have the pieces in VECM and VAR

Special Issue: Recent developments in bootstrap methods for dependent data
http://onlinelibrary.wiley.com/doi/10.1111/jtsa.v36.3/issuetoc

user: josef-pkt
issue_3139: state: open
title: OLS params length should be predictable (collinearity in add_constant)
body: ## The problem:

In a loop over models I can't just do const, slope = res.params
## The reason:

For regression models with a single predictor and without a constant, model.fit().params has a length of 1, i.e. contains the slope. This is predictable. :smile: 
For models with a constant, it is length 2: the constant and the slope - although I instinctively expected them in the opposite order this is predictable :smile:
except when that constant happens to be 0... then it is length 1, i.e. not predictable :disappointed:
## The solution?:

If there is a constant, return a tuple of length 2, even if the constant is 0.

user: ghost
issue_3138: state: open
title: Question about the meaning of freq_weight in the file generalized_linear_model.py
body: Dear author,

I want to ask the meaning of parameter 'freq_weights'. I am trying to apply some weighted method on poission regression and negative binomial regression now, and the weights depend on every test sample. Similar to WLS and OLS, I want to add the weights. Could I achieve this goal just by changing the value of parameter 'freq_weights'? I think it's right, and just put this issue to confirm.

user: huaxiuyao
issue_3137: state: open
title: ENH: multivariate endog wald_test: helper functions/methods/class for ravel/vec params
body: How to we handle t_test and wald_test generically for models that have "naturally" 2-dimensional params?

examples: currently VAR, Multinomial, more are coming VECM, SUR, ...

one option would be to ravel/`vec` the params and store them in standard 1-D form.

another option that I'm thinking about in this issue is to support wald tests or general linear restriction tests for 2-D params and associated raveled or structural cov_params.

needed for current t_test and wald_test
- ravel params corresponding to cov_params
- param_names for raveled params

extras for special cases: sparse or structured cov_params, e.g. kronecker product
- possibility to avoid full cov_params ?
  - cov_params for a single equation (one univariate endog)
  - reduced cov_params before kronecker ?
  - special tests: MANOVA: tests of restrictions on rows or columns of params

user: josef-pkt
issue_3136: state: open
title: ENH: general/generic constraints mixin 
body: (just an idea while reading on restricted estimation in system of equations, VAR, VECM)

We can impose general, linear or nonlinear restrictions on parameters by transformation between the full parameters and a reduced set of unrestricted parameters. We do this in several classes already.

A generic, rapid development version would be to use a Mixin class similar to penalized that transforms the parameters before calling the inherited methods, loglike, score, hessian. Similar to the linear transformation code in fit_constrained and fit_transformed,  we could get the score, Hessian and cov_params of the full model by delta method.

This only works if fit uses generic nonlinear optimization, but works for all parameters not just constraints that can be obtained by transforming exog. So, it would cover some other cases than fit_constrained, which works for any optimizer but only for exog transformation, and requires offset for non-homogeneous constraints.

Given that we have examples of most of the pieces, this would most likely just require wrapping/wiring everything in one class.

user: josef-pkt
issue_3135: state: open
title: FAQ/SUMM  Inference in VECM, cointegrated systems
body: Starting a list to get an overview for different cases (currently written from memory)
large parts will become only relevant with extension, see #3133 

Without weakly exogeneous variables - mostly in Lutkepohl book
with weakly exogenous variables - Harbo et al 1998, Pesaran et al 2000
with combined or mixed restrictions - Boswijk/Doornik  
- cointegrating rank
  - without exog: 5 cases tabulated critical values, nuisance parameter free
  - with exog: 3 nuisance parameter free cases in Harbo et al, 5 cases in Pesaran 
    (where I still don't understand the two cases with unrestricted constant or trend)
  - problem: rank with exog is limited by number of included endogenous 
    (Harbo et all: it's an assumption, 
    somewhere else someone mentions that rank could be larger in the implementation, and alpha, beta would have full rank )
- `Pi` matrix - VECM without reduced rank  (name? params_ec ?)
  ???
- beta params_coeq
  - super consistent with (over)identifying restrictions, LR test is commonly used, Wald test ?
  - not super consistent if identification uses restrictions on alpha (params_adj) (Boswijk, Doornik)
  - no inference if not identified, examples in Pesaran et al 2000 used for specification search
- alpha (params_adj) 
  - given beta (fixed or super consistent) - standard sqrt(n) asymptotics
  - identifying restriction jointly in alpha and beta (Boswijk, Doornik case) ?
- adding extra lag:
  Lutkepohl mentions somewhere that we get standard asymptotics for testing if we add an extra lag that is not included in the restrictions for the test.  where and what?

not clear (I didn't try to understand yet):
- ARDL: Pesaran, I haven't looked at this. sounds like partial system with just a single equation 
- especially Boswijk, Doornik talk about mixed asymptotic distribution. cases where different variables have different rate of convergence, e.g. combining restrictions or hypothesess on alpha and beta.
  What are the practical consequences?

critical values for rank test references
MacKinnon, Haug, Michelis 1999 (updated from Pesaran et al 2000 working paper version)
Osterwald-Lenum 1992

user: josef-pkt
issue_3133: state: open
title: SUMM/ENH: Extensions to VECM
body: follow-up to #3070 GSOC project an PR with extensions that will mostly need more time or work after GSOC
- extensions based on level VAR for given parameters
  Aleks is already reusing some things like forecast, 
  this is mainly an interface issue
- extension based on difference VAR for given cointegrating vectors
  this is just a standard stationary VAR with exog or additional explanatory variables, the error correction term in this case
  useful for Wald test and similar on the short-run behavior, cointegrating vector parameters beta are super consistent under identificating restrictions and can be ignored for short-run analysis
- enhancements for VAR that can be reused for VECM (for fixed params_coeq)
  - VAR with zero restrictions and variable selection (lutkepohl, large sparse VAR)

the following require enhancement to VECM implementation
- exog, partial system with weakly exogenous variables
- estimation under overidentifying restriction on beta (params_coeq)
- estimation under restriction on constant and trend, similar to the previous for trend in params_coeq
- restrictions on alpha, params_adjust, with reestimation of params_coeq (beta)
  this includes restrictions for weak exogeneity and tests for them
- checks that we have enough identifying restrictions when they are user defined.
- hypothesis tests:
  - cotrending hypothesis (params_coeq, beta_trend = 0)

features that don't need changes to basic code
maybe part of GSOC
- cointegration test: required soon
- LR test for specification, e.g. 
  - trend, test trend=caseIV versus caseIII  see 

longer term:
Structural VECM, with simultaneous effects, (system of dynamic simultaneous equations)
SVAR needs also more work
## References
- restricted estimation switching algorithm: 
  Stata manual (uses Boswijk 1995), Boswijk 1995, Boswijk and Doornik 2004
- weak exogeneity, partial systems: 
  Pesaran, Shin, Smith 2000, Harbo, Johansen, Nielsen, Rahbeck 1998
- short run restriction VAR/VEC
  only Lutkepohl book, references for large sparse VAR

user: josef-pkt
issue_3132: state: open
title: ENH: global VARX VECX
body: https://sites.google.com/site/gvarmodelling/gvar

The matlab toolbox seems to have a compatible license

see also
https://groups.google.com/d/embed/msg/pystatsmodels/Cu_ijjbthLw/34a9khRACgAJ

The download includes an extensive user guide with Theory appendix

AFAICS, the toolbox is oriented for the global economy application, but all the tools should be generic.
E.g. the tools for structural VAR and VEC modelling could be available in statsmodels and the international macro/finance application could be a separate package.

related for us also network-VAR for large scale VAR modelling, including linkage tests and modelling,
spatial and panel data VAR

user: josef-pkt
issue_3129: state: open
title: FAQ/Notes: trend cases for cointegration
body: adjusted Stata notation

D y_t = a(b y_{t-1} + ci_0 + li t) + sum( G_i Dy_{t-i} ) + co + lo t   (1)

```
             Stata        Stata                        coeq   diff
case    included    excluded   Johansen    Z1      Z0
nc          -            ci li co lo      H2              -       -    
rc           ci             li co lo       H1*            ci       -              
c           ci, co          li lo           H1              -       co    
rt          ci li co         lo             H*              li       co    
t          ci li co lo      -               H              -       colo    
```

nc:  None, no constant E(y) = E(Dy) = 0
rc: restricted constant, constant only in cointegrating equation
c:   unrestricted constant, constant only in diff equation
rt:   restricted trend, trend only in co-eq, constant in diff
t:    unrestricted trend: nothing in co-eq, constant and trend in diff

SAS 14 page 2989 describes the same 5 cases directly analogous to Z1, Z0  (which are Z1 and Z2 in their notation)

The Eviews documentation describes imposing constraints by projecting out the constant/trend first, but documentation doesn't show explicit formulas.
(this sounds similar to Le Sage)

Eviews allows for user specified additional deterministic and exog, but mentions that they are not taken into account for the critical values of johansen's cointegration test. 

MacKinnon et al  ?  IIRC they also impose additional restrictions. They also have critical values for weakly exogenous variables.
MacKinnon et.al is based on the working paper version on Pesaran, Shin and Smith
From skimming PSS, the cases look the same. Also the description of the algorithm p.3030 might have the same projection, but maybe an additional projection of one part (???) 
Based on the description in Mackinnon compared to SAS and Stata it looks like there is an additional restriction in MacKinnon/PSS/Eviews, so two cases are non-overlapping, 
After struggling through the main sections of PSS:
I still don't see a difference in the estimation to the other version. However, in Appendix B they describe the Simulations for the critical values and it is the same as in MacKinnon et al.
**I think now that there is no diffference in the estimation and the test statistics, but the critical values are in two cases derived under additional restrictions that Johansen/Osterwald-Lenum don't impose.**
(to proof by example once we have the code for the different versions, or an example that can be compared to the articles.)

Osterwald-Lenum is the main reference for critical values in Johansen's cases. Eviews has both MacKinnon and OL critical values.

PcGive ?
only the three main cases ?

Lutkepohl jmulti ?

Lutkepohl book

user: josef-pkt
issue_3127: state: open
title: PERF/ENH: reusing parts of regressor matrix for specification search or testing
body: We are moving to have all options that affect the internal design/regressor matrix into the `__init__` of the models, instead of having those options in fit.

During specification search we want to change the regressors but there is a large overlap with columns that don't change.

example: in autolag for adfuller, I arranged the regressor matrix so we can drop columns at the end without rebuilding the columns before.

VAR/VECM and similar:  
- adjusting trend options leaves the part of the regressor matrix with lags unchanged
- dropping/reducing the number of lags can also drop sets of columns at the end when using the column ordering of `lagmat`

**possible solutions**
- 1. write specialized functions that do the specification search and model adjustments in an efficient way.
- 2. add keyword to `__init__` to allow that users (internal methods) provide part of generated regressors (e.g. array of lagged series in VAR/VECM)

(while thinking about or reviewing VECM #3070 )

user: josef-pkt
issue_3119: state: open
title: BUG: TypeError in contrast results summary()  if `df_denom` or `df_num` not passed for f test.
body: I think the F test itself should probably fail without these degrees of freedom, but it doesn't (although p-value is nan). Then a ValueError is raised when trying to create the summary because `%d` can't handle None / NaN values.

``` python
endog = np.log(sm.datasets.macrodata.load_pandas().data.realgdp) * 400
endog.index = pd.date_range('1959-01-01', '2009-07-01', freq='QS')
mod = sm.tsa.UnobservedComponents(endog, 'rtrend', cycle=True, stochastic_cycle=True, cycle_period_bounds=[4, 16])
res = mod.fit()

# This doesn't fail
out = res.f_test('sigma2.cycle=0, sigma2.trend=0')

# pvalue is nan
print(out.pvalue)

# Throws exception
print(out.summary())
```

Exception trace is:

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-359-c47c9700b50f> in <module>()
     10 print(out.pvalue)
     11 
---> 12 print(out.summary())

/Users/fulton/projects/statsmodels/statsmodels/stats/contrast.pyc in summary(self, xname, alpha, title)
    154             # TODO: create something nicer for these casee
    155             return '<F test: F=%s, p=%s, df_denom=%d, df_num=%d>' % \
--> 156                    (repr(self.fvalue), self.pvalue, self.df_denom, self.df_num)
    157         else:
    158             # generic

TypeError: %d format: a number is required, not float
```

user: ChadFulton
issue_3115: state: open
title: DESIGN: Roadmap for time series
body: Starting an issue to discuss the longer-term roadmap for `tsa` with some of my thoughts / questions. The major theme I have in mind is making `tsa` more consistent in terms of inputs, outputs, and methods. Obviously we haven't even released with state space models, so I am surely "getting ahead of myself".

Some specific thoughts:
- `TimeSeriesModel`: this was written a long time ago (e.g. includes the comment `#NOTE: Can drop this once we move to pandas >= 0.8.x`), and probably includes a lot of code that is no longer necessary with modern Pandas. Cleaning it up shouldn't be too much work, but redoing the date wrapping behavior might take some work, see below.
- Results wrapping: there are two concerns here that I've run into specifically:
  1. Handling dates for wrapping output is not general. It relies on internal state, and was written for `predict` and `forecast` and so is tied directly into those things. One issue is that when we have output arrays that could be associated with dates, they can't be wrapped right now because we can't set the internal state (without doing yet another set of property wrappers to call the right functions first).
  2. It doesn't really play well with multivariate models. The multivariate support that exists was bolted on to work specifically with the VAR class, and so is pretty tough to work with.
- `sm.tsa.AR` - is this still worth keeping around? It doesn't match the current Statsmodels modeling paradigm (i.e. the specification is in the `fit` method rather than the constructor) and there are two other classes that can fit AR models (`sm.tsa.ARIMA` and `sm.tsa.SARIMAX`).
- `sm.tsa.ARIMA` - is this still worth keeping around? I'm guessing it is, or at least parts of it are; I'm just trying to figure out how all the pieces of `tsa` fit together. I'm not very familiar with it, but I think a lot of its functionality is duplicated by `sm.tsa.SARIMAX` (except for CSS fitting?). My main concern is that I don't know if anyone is maintaining these models.
- Model selection: currently, this is handled by `maxlag` and `ic` arguments to `fit` in `sm.tsa.AR`, `sm.tsa.VAR`, and `sm.tsa.SVAR` but is not explicitly supported by `sm.tsa.ARMA`, `sm.tsa.ARIMA`, or any of the state space models. Since the current paradigm is that order should be specified in `__init__`, I think this should be moved out of the models themselves completely, and model selection should be handled with some kind of external helper function / class.

**Naive proposal**:

Here is an (ambitious, naive) proposal which is probably not the right way to go about things in many ways, but might be a start:
- Deprecate `sm.tsa.AR`, `sm.tsa.ARMA`, `sm.tsa.ARIMA` in v0.9 and replace them with new versions in v0.10, maintaining as much backwards compatibility as possible (but will definitely have to break things especially with `sm.tsa.AR`).
  1. Eliminate `ar_model.py`
  2. Extract the conditional-MLE / CSS components (and anything else not duplicated elsewhere) of `sm.tsa.ARMA` and `sm.tsa.ARIMA` and put them in a new semi-internal class (e.g. `sm.tsa.LeastSquaresARIMA`).
  3. Create new thin wrapper classes `sm.tsa.AR`, `sm.tsa.ARMA`, `sm.tsa.ARIMA` that delegate to `sm.tsa.LeastSquaresARIMA` (called if `method == 'css'`) and `sm.tsa.SARIMAX` (called if `method == 'mle'`).
- Deprecate `sm.tsa.VAR` in v0.9 and replace it with a new version in v0.10, maintaining as much backwards compatibility as possible.
  1. Rename `sm.tsa.VAR` into new semi-internal class (e.g. `sm.tsa.LeastSquaresVAR`).
  2. Create new thin wrapper class `sm.tsa.VAR` that by default delegates to `sm.tsa.LeastSquaresVAR`. Since `sm.tsa.VARMAX` is currently pretty far behind `sm.tsa.VAR` in terms of post-estimation (IRF, FEVD, etc.), I think it should raise a NotImplementedError if `method == 'mle'` until it catches up.
- I don't know what to do with `sm.tsa.SVAR`, so I guess we can just leave it alone, except for making some changes to make the interface consistent (e.g. eliminating `maxlags` order selection in `fit`).
- Enforce that all models are consistent in terms of:
  - Parameter names (notation for lags, exogenous coefficients, etc.)
  - A consistent set of methods in the model classes (e.g. `fit`, `predict`, `forecast`, `simulate`, `impulse_responses`) with consistent arguments and consistent output. Obviously allow for some methods to not be implemented, but require that they throw `NotImplementedError` rather than not having the method at all.
  - A consistent set of attributes and methods in the results classes
  - A consistent format for summary output.
- Construct a new model selection class that accepts a list of specifications and selects the best model on the basis of some IC. Potentially tie in with parallel estimation, etc.

user: ChadFulton
issue_3101: state: open
title: Pearson Chi2 Test Statistic in a GLM
body: I'm trying to compare 2 models using a Pearson Chi-2 type 3 statistical test. I've done this before in some expensive software, so was trying to figure out how to do it with python. 

That procedure followed (I'm fairly sure) what exists in [this paper](http://www.psychologie.uzh.ch/dam/jcr:ffffffff-b371-2797-0000-00000fda8f29/chisquare_diff_en.pdf). In essence, you calculate 2 values for a pair of nested models (nested meaning that one model has every parameter in the other model, and also has at least one additional parameter the other model doesn't.)

In python/statsmodels, I was thinking you could calculate it this way...

``` python
df_diff = res1.df_resid - res2.df_resid
chi2_diff = res1.pearson_chi2 - res1.pearson_chi2

from scipy.stats import chi2
1 - chi2.cdf(chi2_diff, df_diff)
```

I'm getting extremely large values that I don't believe are accurate. 

Looking through the literature, I found [this paper by Smyth](http://www.statsci.org/smyth/pubs/goodness.pdf). It uses this formula  when `scale == 1`:

![image](https://cloud.githubusercontent.com/assets/8186432/16709503/267776c2-45d8-11e6-8bbd-29a58e319ca4.png)

However, more generally, it uses (see Section 6, "Unknown Dispersion Parameter"):

![image](https://cloud.githubusercontent.com/assets/8186432/16709507/4b82a36a-45d8-11e6-8f67-243978b83003.png)

Should we be dividing by the scale parameter as well? If I understand the current code, we do not:

https://github.com/statsmodels/statsmodels/blob/master/statsmodels/genmod/generalized_linear_model.py#L1325

Thoughts? 

user: thequackdaddy
issue_3091: state: open
title: big, large distributed GLM - Performance
body: http://support.sas.com/resources/papers/HPGENSELECTscalability.pdf
refers to http://blog.revolutionanalytics.com/2012/10/allstate-big-data-glm.html

150 million observations and 70 degrees of freedom

based on SAS technical paper, exog has 70 column, i.e. df_model = 70 or 69

user: josef-pkt
issue_3089: state: open
title: FAQ-D: API consistency in tsa
body: #3028 cleans up VAR including changing to more descriptive names.

similar features are in AR, ARMA/ARIMA with exog and in statespace models.

What is the common API for time series models?
- basics common to all models or all LikelihoodModels
  - fit 
  - predict
  - summary  missing in AR
  ## \- wald tests, not supported across all models
- basics, specific to time series
  - lag order: created regressors
  - time index, datetime/period handling
  - forecast
  - post-estimation: impulse response, fevd, ...
- specific
  - exog, missing in VAR
  - trend are specified separately from exog in several models -> can we drop it, at least internally
    VECM uses deterministic instead of trend and will allow for seasonal dummies
  - 

user: josef-pkt
issue_3079: state: open
title: BUG/ENH: Tweedie parameterization and better handling of exposure, inclusion of 'perry' and 'taylor' methods.
body: Hello,

See #2915 for the cause of this. 

(I just realized that you probably wanted me to name the method 'gamma-glm', but I named it 'perry'. I also made a 'taylor' which is supposed to be WLS version of the same thing. If you want me to rename, let me know.)

I'm not sure this totally works. I first used the `fair` dataset to see how well/poorly this works (I generated sample weights, exposures. On a scale of 1-10, I'd give that a 7 as far as how good it felt. `taylor` and `perry` were very close, but the rest was quite different. 

When I ran this on my real dataset, my confidence fell to a 2. I'm getting much different estimates for p using all the different methods. So I might need to think this out more.

``` python
In [4]: model = sm.GLM(
   ...:     endog=np.array(data['PDIncurredLoss']),
   ...:     exog=bx,
   ...:     family=sm.families.Tweedie(var_power=1.639, link_power=0),
   ...:     exposure=np.array(data['PDExposure'])
   ...: )
   ...: 
   ...: res = model.fit(rtol=1e-2, tol_criterion='params', maxiter=10)
   ...: p = model.estimate_tweedie_power(res.mu)

In [5]: model.estimate_tweedie_power(res.mu, method='taylor')
Out[5]: 2.0409433864515334

In [6]: model.estimate_tweedie_power(res.mu, method='perry')
Out[6]: 1.1050768778823175

In [7]: model.estimate_tweedie_power(res.mu, method='brentq')
Out[7]: 1.6455822161384193

In [8]: res.summary()
Out[8]: 
<class 'statsmodels.iolib.summary.Summary'>
"""
                 Generalized Linear Model Regression Results                  
==============================================================================
Dep. Variable:                      y   No. Observations:              3569086
Model:                            GLM   Df Residuals:                  3569058
Model Family:                 Tweedie   Df Model:                           27
Link Function:                    log   Scale:                   2553.65026079
Method:                          IRLS   Log-Likelihood:                    nan
Date:                Sun, 26 Jun 2016   Deviance:                   7.7973e+07
Time:                        20:20:35   Pearson chi2:                 9.11e+09
No. Iterations:                    10                                         
```

On the bright side, all the methods feel like converge with a guess-and-check approach. So that keeps it above a 1. 

user: thequackdaddy
issue_3066: state: open
title: FAQ individual versus joint significance
body: a nice example here

http://stats.stackexchange.com/questions/218667/simple-linear-regression-p-values-and-the-aic

user: josef-pkt
issue_3039: state: open
title: ENH: linalg helper: orthogonal complement of matrix
body: (parking another helper function, so it doesn't get lost)

Written to understand orthogonal complement which is heavily used in VECM/cointegration.
Should also show up in canonical correlation, reduced rank regression and similar.
Also looks similar to reparameterization to include constraints (which I wrote without understanding the math). 

general story: econometrics books often refer to things without telling how to compute it.
Eg. 
constraints: w.l.o.g. we only consider this constraint, because the other forms can be obtained by reparameterization/transformation".

Related to here: normalization of cointegrating vectors to [eye, beta_2] requires a sequence of equations so that first variables with coefficients normalize to eye include elements of all cointegrating vectors. (But authors like Lutkepohl don't say how.)
My guess, not verified, is that below this will show up in the inv of the first part during normalization.
I need an example to check this.

```
# -*- coding: utf-8 -*-
"""
Created on Fri Jun 17 13:42:36 2016

Author: Josef Perktold
License: BSD-3


example from http://ltcconline.net/greenl/courses/203/Vectors/orthogonalComplements.htm

no reference for the use of SVD, but it should be "obvious"

"""

import numpy as np

def orthogonal_complement(x, normalize=True, threshold=1e-15):
    """Compute orthogonal complement of a matrix

    this works along axis zero, i.e. rank == column rank,
    or number of rows > column rank
    otherwise orthogonal complement is empty

    TODO possibly: use normalize='top' or 'bottom'

    """
    x = np.asarray(x)
    r, c = x.shape
    if r < c:
        import warnings
        warnings.warn('fewer rows than columns', UserWarning)

    # we assume svd is ordered by decreasing singular value, o.w. need sort
    s, v, d = np.linalg.svd(x)
    rank = (v > threshold).sum()

    oc = s[:, rank:]

    if normalize:
        k_oc = oc.shape[1]
        oc = oc.dot(np.linalg.inv(oc[:k_oc, :]))
    return oc


xxx = np.array([[1,0,1,0,2], [0,1,1,1,0], [1,1,1,1,1]]).T
ocn = np.round(orthogonal_complement(xxx, normalize=True), 2)
print(ocn)
print(orthogonal_complement(xxx, normalize=False))
print(ocn.T.dot(xxx))

"""
# some math courses seem to normalize at the bottom

>>> x2 = np.array([[1., 3, 0], [2, 1, 4]])
>>> np.round(orthogonal_complement(x2.T, normalize=True), 2)
array([[ 1.  ],
       [-0.33],
       [-0.42]])
>>> orthogonal_complement(x2.T, normalize=True)
array([[ 1.        ],
       [-0.33333333],
       [-0.41666667]])
>>> oc2 = orthogonal_complement(x2.T, normalize=True)
>>> oc2 / oc2[-1]
array([[-2.4],
       [ 0.8],
       [ 1. ]])
"""
```

user: josef-pkt
issue_3028: state: open
title: Var cleanup rebased
body: rebased version of #815  Skipper's VAR cleanup

see original PR for TODO and DONE list for this PR
see #2274 for a SUMM issue for VAR cleanup

many manually edited merge conflicts (many of the merge conflicts were in view numpy compatibility, adding type keyword, compatibility fixes by Pierre)

I cannot run the entire tests right now because matplotlib segfaults on my computer, in this case in `statsmodels.tsa.vector_ar.tests.test_var.TestVARResults.test_plot_cum_effects`

code analysis in Eclipse/PyDev still shows several red lines, i.e. bugs.

This should be compared to statespace VARMAX for consistency, especially names and shapes.

Do we have a glossary for names used in VAR? e.g. attributes for AR part and for deterministic/trend/exog and for combined/all regressors.

extras

`wald_test_terms` would be useful to test for lags or for variables (like granger causality)
t_test is still NotImplementedError: possible solution: reshape params, call super, reshape back (reshape needs to match cov_params)

user: josef-pkt
issue_3027: state: open
title: Design/ENH: check no asarray, or usage of asanyarray
body: #2240 drops asarray from detrend so it works with numpy ndarrays and with pandas.

This is potentially dangerous, alt least lists will not work anymore, other array subtypes (matrices, ...) might have problems, although unlikely in this case.

Also some other recent PRs use asanyarray.

One possibility is to replace asarray usage with a whitelist of allowed types and convert everything else. (xarray, dask, ... ?)

user: josef-pkt
issue_3025: state: open
title: ENH: add exog to VAR
body: (I thought I added this enhancement request already somewhere, but don't find it.)

VECM for given cointegrated vectors is a VAR in differences with cointegrating arrays as `exog`.

Since cointegrating vectors can be estimated superconsistenly, we can plug in the cointegrating arrays and get all the results from the standard VAR code, e.g. diagnostics.
It can also be used in a two-step procedure if we first estimate cointegrating vectors (beta params in Lutkepohl)
for diagnostics see Lutkepohl: A new Introduction ..., section 8.1

(Alternative: VECM can be converted to the VAR representation, but I don't remember now the asymptotics, e.g. for residual diagnostics. It should have the same residuals as the VECM representation.)

user: josef-pkt
issue_3024: state: open
title: ENH: produce critical values for johansen cointegration test
body: parking an example script, a draft for simulating critical values and p-values for johansen cointegration test

I wrote it mainly to check whether it works as simply as it sounded.
This is slow for runtime usage, but can help produce additional tables.
currently only for the simple case of no deterministic part, no constant and no trend, i.e. case 0

cross-ref PR #453 and associated issues

MacKinnon et al 1999 have 5 cases, and don't include 2 cases of Johansen, and Osterwald-Lenum.
Their tables also have weakly exogenous integrated variables and cite Pesaran for the theory.

aside: MacKinnon et al use locally normal cdf/ppf approximation for interpolation of p-values or critical values. looks useful and should be tried out for some cases.

```
# -*- coding: utf-8 -*-
"""
Created on Wed Jun 15 16:09:17 2016

Author: Josef Perktold
License: BSD-3
"""
import time
import numpy as np


def simulate_coint_johansen(k_vars, case=0, nobs=2000, n_rep=10000):
    """simulate Wiener process and Johansen's trace and max-eigenvalue

    This is for case 0 without constant nor trend.

    Parameters
    ----------
    k_vars : int
        number of stochastic trends k - r
    case : ???  int or string, or trend
        currently not used, only no constant, no trend implemented
    nobs : int
        number of observations in the random walk
    n_rep : int
        number of replication in the simulation

    Returns
    -------
    (needs to change, returns irrelevant information)
    res_e : ndarray
        simulated eigenvalues
    res_es : ndarray
        simulated cumulative sum of eigenvalues
    res_t : ndarray
        simulated trace

    References
    ----------
    Luetkepohl New introduction, p. 333 section 8.2.5
    MacKinnon, Haug, Michelis 1999,
       see table II page 571 column with k=0 (no weakly exogenous variables)

    TODO:
    calulate for all k_vars <= k_vars with the same simulation
       maybe with cholfactor
    change returns
    """

    res_e = []
    res_es = []
    res_t = []

    for i in range(n_rep):
        dx = np.random.randn(nobs + 1, k_vars)
        dx[0] = 0
        x = dx.cumsum(0)
        x_m1 = x[:-1]
        dx0 = dx[1:]

        mat1 = x_m1.T.dot(dx0)
        mat2 = x_m1.T.dot(x_m1)
        #mat = mat1.T.dot(np.linalg.solve(mat2, mat1))
        mat = mat1.T.dot(np.linalg.inv(mat2).dot(mat1))

        trace = mat.trace()
        evals = np.sort(np.linalg.eigvalsh(mat))[::-1]
        evals_sum = evals.cumsum()

        res_e.append(evals)
        res_es.append(evals_sum)
        res_t.append(trace)

    res_e = np.sort(res_e, axis=0)
    res_es = np.sort(res_es, axis=0)
    res_t = np.sort(res_t, axis=0)

    return res_e, res_es, res_t


def crit_value_coint_johansen(k_vars, case=0, alpha=0.05, value=None, nobs=2000,
                              n_rep=10000):
    """
    Parameters
    ----------
    k_vars : int
        number of stochastic trends k - r
    case : ???  int or string, or trend
        currently not used, only no constant, no trend implemented
    alpha : float
        significance level for critical values, e.g. 0.05. The returned
        critical values are the 1-alpha quantiles.
    value : tuple of float
        tuple of max eigenvalue and trace statistic for which the p-value
        will be calculated

    nobs : int
        number of observations in the random walk
    n_rep : int
        number of replication in the simulation
    Returns
    -------
    crit_max, crit_trace
        critical values for the given significance level
    pvalue_max, pvalue_trace
        pvalues for the given max eigenvalue and trace statistic
    """

    res_e, res_es, res_t = simulate_coint_johansen(k_vars, case=case,
                                                   nobs=nobs, n_rep=n_rep)
    idx = int(n_rep * (1 - alpha))
    crit_max, crit_trace = res_es[idx, 0], res_t[idx]
    if value is None:
        return crit_max, crit_trace
    else:
        pvalue_max = (res_e[:,0] >= value[0]).mean()
        pvalue_trace = (res_t > value[1]).mean()
        return crit_max, crit_trace, pvalue_max, pvalue_trace


examples = [] #'ex1'
    # simulate Wiener process

if 'ex1' in examples:
    nobs, kvars = 1000, 11

    n_rep=10000

    res_e = []
    res_es = []
    res_t = []

    t0 = time.time()
    for i in range(n_rep):
        dx = np.random.randn(nobs + 1, kvars)
        dx[0] = 0
        x = dx.cumsum(0)
        x_m1 = x[:-1]
        dx0 = dx[1:]

        mat1 = x_m1.T.dot(dx0)
        mat2 = x_m1.T.dot(x_m1)
        #mat = mat1.T.dot(np.linalg.solve(mat2, mat1))
        mat = mat1.T.dot(np.linalg.inv(mat2).dot(mat1))

        trace = mat.trace()
        evals = np.sort(np.linalg.eigvalsh(mat))[::-1]
        evals_sum = evals.cumsum()

        res_e.append(evals)
        res_es.append(evals_sum)
        res_t.append(trace)

    t1 = time.time()
    res_e = np.sort(res_e, axis=0)
    res_es = np.sort(res_es, axis=0)
    res_t = np.sort(res_t, axis=0)

    print(t1 - t0)
    print(res_e[int(n_rep*0.95)])
    print(res_es[int(n_rep*0.95)])
    print(res_t[int(n_rep*0.95)])


crit_mhm = 17.8, 24.28
np.random.seed(987456)
t0 = time.time()
res3 = crit_value_coint_johansen(3, value=crit_mhm)
t1 = time.time()
print(t1 - t0)
print(res3)

from numpy.testing import assert_allclose

assert_allclose(res3[:2], crit_mhm, rtol=0.02)
assert_allclose(res3[2:], [0.05, 0.05], atol=0.003)
```

user: josef-pkt
issue_3009: state: open
title: cointegration VAR/VECM with mixed I(0) I(1)
body: Luetkepohl page 250 last full paragraph

"Generally, any stationary variables in the system must be placed in the upper r-dimensional subvector of y_t. ..."

stationary I(0) variables are cointegrated with themselves.

Lutkepohl uses a definition for I(1) system that allows for I(0) variables.

This is related to an example that got me confused while writing the `johansen` test. The presence of I(0) variables "messes up" the test.

The ordering in the quote above is related to the identifying normalization for the cointegrating vectors.

(I'm not sure yet what this implies, but it will be needed at some point. If we do purely statistical normalization, then the ordering of variables in the VAR/VECM system is up to the user. In that case, this could be considered as usage question and not as implementation issue.)

Should johansen test just be used for the subsystem with integrated variables? 
However, the short-run dynamics of the integrated variables also depends on the stationary variables in the system, so I guess we still need them.

user: josef-pkt
issue_3006: state: open
title: FAQ-D: convention for type, shape of confidence intervals
body: (while reading #2953)

Do we have a consistent convention for the confidence intervals returned by methods and functions, tuple, ndarray, shape?

user: josef-pkt
issue_2973: state: open
title: TST MultiComparison   check if only two groups
body: followup to #2948 bugfix for 2-sample case

It looks like two sample special case is not sufficiently unit tested.

user: josef-pkt
issue_2971: state: open
title: glmnet elastic net - warm start ?
body: Just a thought, I don't want to check right now

Does GLM elastic net use warm start on weights for irls?

and a glmnet related stats.stackexchange question with link to scikit-learn issue
http://stats.stackexchange.com/questions/100672/possible-to-evaluate-glm-in-python-scikit-learn-using-the-poisson-gamma-or-twe/214159#214159

user: josef-pkt
issue_2963: state: open
title: REF/ENH:  PHReg cov_type
body: (Found while browsing the code for #2924)

PHReg currently defines two cov_types `nonrobust` and `cluster` (AFAICS) but it doesn't use the standard keywords. Also it could reuse the existing code.

Stata has nonrobust, robust (HC0 or HC1) and cluster robust, plus bootstrap, ... For cluster robust it seems to default to the usual number of cluster correction that can be turned off.

I guess we should add HCx and 2-cluster version, but I don't think times series sandwiches like HAC will make sense for PHReg.

user: josef-pkt
issue_2960: state: open
title: Weighted KDE doesn't seem to work
body: I noticed that the weights options for nonparametric.KDEUnivariate seems to be ignored. Here is a MWE.

```
import statsmodels.api as sm
import numpy as np
from scipy.stats import norm

pdf = lambda x: norm.pdf(x, loc=-25, scale=25) 
parameter = np.random.uniform(-100, 100, 1000)
posterior = pdf(parameter)
centers = np.linspace(-100, 100, 1000)
nbins = int(2. * len(parameter)**(1./3.))


def statsmodels_kde(parameter=parameter, posterior=posterior, bw='scott'):
    kde_func = sm.nonparametric.KDEUnivariate(parameter)
    kde_func.fit(kernel='gau', weights=posterior, bw=bw, fft=False)
    kde = kde_func.evaluate(centers)
    return kde


def statsmodels_unewighted_kde(parameter=parameter, posterior=posterior, bw='scott'):
    kde_func = sm.nonparametric.KDEUnivariate(parameter)
    kde_func.fit(kernel='gau', bw=bw, fft=True)
    kde = kde_func.evaluate(centers)
    return kde


# Checks with plots
import matplotlib.pyplot as plt

plt.hist(parameter, nbins, weights=posterior, normed=True, label='samples')
plt.plot(centers, pdf(centers), label='true pdf', lw=5, alpha=0.7, ls='-.')

plt.plot(centers, statsmodels_kde(), lw=5, ls='-', color='blue', label='sm')  
plt.plot(centers, statsmodels_unewighted_kde(), lw=5, ls='--', label='sm unweighted')

plt.xlabel('x')
plt.ylabel('p(x)')
plt.legend(loc='best', frameon=False)
plt.show()
```

![figure_1](https://cloud.githubusercontent.com/assets/3758193/15421749/4f3b4936-1eb8-11e6-814f-62044601ede7.png)

As you can see, the Gaussian weights are ignored, resulting in a uniform distribution. Am I calling the function in the wrong way? I see that it's [undocumented here](http://statsmodels.sourceforge.net/devel/generated/statsmodels.nonparametric.kde.KDEUnivariate.fit.html#statsmodels.nonparametric.kde.KDEUnivariate.fit). Maybe it hasn't been implemented yet?

user: andrewfowlie
issue_2957: state: open
title: ENH: FDR based confidence intervals
body: just a reference (I didn't read anything)

companion to #2955 simultaneous confidence intervals

We want to get the confidence interval corresponding to hypothesis tests and p-values.
The same can be applied to p-value correction for multiple testing.

Here is an article with the FDR corrected multiple confidence intervals

Benjamini, Yoav, Daniel Yekutieli, Don Edwards, Juliet Popper Shaffer, Ajit C. Tamhane, Peter H. Westfall, and Burt Holland. 2005. “False Discovery Rate: Adjusted Multiple Confidence Intervals for Selected Parameters [with Comments, Rejoinder].” Journal of the American Statistical Association 100 (469): 71–93.

This might be useful for example for outlier identification, and nice associated plots.

user: josef-pkt
issue_2955: state: open
title: SUMM/FAQ-D: simultaneous confidence intervals
body: (another missing piece in hypothesis testing plus everything. And I don't have an overview of theory, approaches, ...)

How can we construct simultaneous confidence intervals for several parameters?
Can we use exact or Monte Carlo probabilities or bootstrap to get better small sample results?
How do we choose the shape of the confidence region?

(triggered by this https://github.com/statsmodels/statsmodels/issues/2895#issuecomment-219918032 and PR #2953 )

what we have, AFAIR
- Sheffe linear prediction confint (added by Kerby, and it took me a while to understand the max)
- Tukey confidence interval for pairwise comparison
- some bootsrap intervals in tsa (impulse response function IIRC) where I never checked whether they are simultaneous or pointwise
- multinomial probabilities, new in above issue and PR
- maybe: standard p-value, Bonferroni correction for outliers

We have a lot more on hypothesis tests, p-values, p-value correction, permutation max-t pvalues (not in statsmodels).

I don't have time to read, so just two references as starting point, emphasis on bootstrap

Beran, Rudolf. 1990. “Refining Bootstrap Simultaneous Confidence Sets.” Journal of the American Statistical Association 85 (410): 417–26. doi:10.2307/2289778.
summarizes Sheffe, bootstrap and introduces double bootstrap for simultaneous confidence interval

Zhilova, Mayya. 2015. “Simultaneous Likelihood-Based Bootstrap Confidence Sets for a Large Number of Models.” arXiv:1506.05779 [Math, Stat], June. http://arxiv.org/abs/1506.05779.
This is long but looks interesting. Based on skimming a few pages, IIUC.
 uses frequency weights for bootstrap resampling.
 uses correction for alpha (nominal size) for individual confidence intervals (*)
 is advertised as being robust to local model misspecification

(*) this looks relatively easy to get a meaningful shape of the confidence interval that can be small sample corrected.

general issues (partially in analogy to rates and proportion single parameter confidence intervals)

**nuisance parameters and "exact" size**

Fisher's exact test and similar guarantee the size for all parameters (and are as consequence conservative)
Berger/Boos guarantees the size, but uses restricted parameter space.
E-tests use MLE estimate of nuisance parameters

If we don't maintain "exact" size, then one question is what's the worst case parameter and how bad (liberal) is the size or coverage. (there are a few articles on confidence coefficient which is the worst case coverage probability. It's zero for Wald, if there is no correction for the boundary observation. Binomial x=0 implies uncorrected Wald confidence interval [0, 0] is one point. However in practice most implementation use a correction for this case.

**Wald or score parameters for confidence intervals**

score intervals have usually better properties, maintain size better than Wald. 
However, we need to search over the parameter space to find the score confidence limits.

One problem for bootstrap or Monte Carlo probabilities in the above is that searching the parameter space is expensive because we need to run the simulation for each values of the parameters.
Related bootstrap for hypothesis tests on a coefficient in regression models is often more accurate if the model is simulated under the null (imposed restriction) than if we use the unrestricted model. However, if the restrictions change or we want to test a different parameter, then we would have to run the simulations again. (articles on bootstrap for heteroscedasticity robust standard errors by ...)
So, simulation approaches for this type of hypotheses or confidence intervals only works when there is one or at most a few parameters or hypothesis.

Aside: multinomial might be an interesting case because of the GLM type variance differences across parameters if they are not all the same under the null or estimated.

How do we get started?

Right now adjusting the bonferroni correction for individual confidence intervals looks the most likely theoretically reasonable and computationally feasible approach.
Plus some simultaneous analogues of single parameter bootstrap confidence intervals. ???

user: josef-pkt
issue_2951: state: open
title: ENH: callback functions for generated regressors
body: my guess is that this might be similar to regression trees, but I never looked at those details.

The basic idea: We want to select variables from a large possible list. Explanatory variables are generated from a smaller set of existing variables. Instead of creating all variables at once, we create them on the fly.

For large sparse VARs we need to create lagged variables.
In large dummy sets we can create sparse dummy variables as needed, and not stored.
For standard regression, we want to include higher order terms like polynomials, splines or interactions.

question/option: do we impose marginality on terms or not, i.e. if higher order or interaction terms are included, then lower order terms also need to be included.

example: ISIS: ultra-high sure independence screening
- start with regular ISIS or similar on existing main variables
- cycle through higher order terms for each main variable

example: coordinate descend for penalized variable selection

If we can fit things into memory, then it wouldn't be worth it as long as we have to cycle several times, I guess.

user: josef-pkt
issue_2950: state: open
title: ENH: RLM add elastic net or fit_regularized
body: Jianqing Fan is in recent articles mixes penalized estimation, variable selection with RLM Huber loss.

another application, IIUC
use RLM for variance or correlation estimation, with penalized LASSO or SCAD we can get robust sparse correlation and covariance matrices.

my guess is that RLM can use the same pattern as GLM.fit_regularized, after it get's offset, and maybe the the scipy optimizer.

user: josef-pkt
issue_2949: state: open
title: REF: decouple optimization from loglike, and/or add kwds
body: just another idea for partial decoupling of loglike and optimization:
- add kwds/kwargs that are used in the optimization call to loglike/score/hessian 
- check existing attribute before creating the lambda function for the optimization  (allow objective function to be defined in attribute of a model, or as string by method name)
  but avoid instance methods

usecase for first is setting options for loglike during optimization. This is currently implemented by an attribute for transformation that are only used during optimization.

another possibility: drop constant terms during optimization. (We always maximize full loglike, while other packages don't include terms that are independent of the parameters, e.g. use deviance instead of loglike)
(that's only for a bit better performance, except in Tweedie case.)

todo: find related issues 

user: josef-pkt
issue_2947: state: open
title: SUMM: unbalanced panel data, missing observation longitudinal - IPW
body: (I didn't find an issue yet, but this got mentioned in several issues.)

In unbalanced panels or longitudinal data, we can ignore the missingness only if it is completely at random MCAR, or exogenous.

If observations are misssing at random, i.e. we can model them based on additional explanatory variables, then one approach is to use weighting to adjust the estimation for MAR.
If only a few observations are missing, then they can be imputed.

There is some literature in econometrics and a large literature in statistics.
The simpler case is if some individual drop out, consecutive missing observation at the end. More complicated is if there are intermittent missing values.

main infrastructure: observation specific probability weights in the models
Kerby mentioned it for GEE

References
....
(I read on this a few weeks ago, but don't find anything right now. Some files with likely names below.)

"Missing Data Methods for Generalized Linear Models.pdf"
"Multiple imputation for missing data in a longitudinal cohort study a tutorial based on a detailed case study involving imputation of missing outcome.pdf"
"Comparison of Four Methods for Handing Missing Data in Longitudinal Data Analysis through a Simulation Study_OJS_2014123113391029.pdf"
"Efficient augmented inverse probability weighted estimation in missing data problems.pdf"
"Weighted Generalized Estimating Functions for Longitudinal Response and Covariate Data That Are Missing at Rand.pdf"

user: josef-pkt
issue_2946: state: open
title: FAQ/ENH:  limitation of predictive mean matching pmm
body: http://statisticalhorizons.com/predictive-mean-matching

Morris, Tim P., Ian R. White, and Patrick Royston. "Tuning multiple imputation by predictive mean matching and local residual draws." BMC medical research methodology 14, no. 1 (2014): 1.
http://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-14-75

(There are a few related articles that I looked at a few weeks ago, but don't remember.)

One task is to provide diagnostic tools to see how much variation is actually used, and how many neighbors are available.
Similar issue in IPW is if there are observations that don't have close matches with opposite treatment (probability near 0 or 1)

user: josef-pkt
issue_2945: state: open
title: ENH: frailty models with penalization SCAD LASSO
body: (just to park another reference at the moment)
related to #2374 penalized MLE Mixin

In the simulations SCAD outperforms LASSO and similar.

book chapter
Androulakis, Emmanouil, Christos Koukouvinos, and Filia Vonta. 2013. “Penalized Likelihood Methodology and Frailty Models.” In Statistical Models and Methods for Reliability and Survival Analysis, edited by Vincent Couallier, Léo Gerville-Réache, Catherine Huber-Carol, Nikolaos Limnios, and Mounir Mesbah, 45–60. John Wiley & Sons, Inc. http://onlinelibrary.wiley.com/doi/10.1002/9781118826805.ch4/summary.

user: josef-pkt
issue_2944: state: open
title: ENH: ultrahigh screening with SCAD or adaptive LASSO
body: (where is my code, I don't see it in #2374, not committed anywhere?)

main references are Jianqing Fan and coauthors

the following two are very recent, and the Wang/Leng article seems to have a good updated algorithm, theory and references for it. I didn't look at the article yet.
(I was using mainly GLM/MLE models but I guess those results extend at least in local approximation.)

Wang, Xiangyu, and Chenlei Leng. 2016. “High Dimensional Ordinary Least Squares Projection for Screening Variables.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 78 (3): 589–611. doi:10.1111/rssb.12127.

Yang, Yuehan, and Lan Wu. 2016. “Nonnegative Adaptive Lasso for Ultra-High Dimensional Regression Models and a Two-Stage Method Applied in Financial Modeling.” Journal of Statistical Planning and Inference 174 (July): 52–67. doi:10.1016/j.jspi.2016.01.011.
I didn't look at it, but looks good to see how to implement things with adaptive LASSO

user: josef-pkt
issue_2943: state: open
title: SUMM: foward search variables and observations - high breakdown - ultrahigh screening
body: (a mixed bag of issues, without overview or references for actual algorithms)
triggered by looking at some literature for robust transformation (box-cox issue and PR)
https://github.com/statsmodels/statsmodels/issues/1309#issuecomment-217048374
Atkinson and Riani have many articles but not a large number of citations. 2 articles below

two issues for both adding variables and adding observations
1. intial dataset for a "clean" submodel
2. criteria for expanding and shrinking model - add observations and/or variables

two examples

LTS/ELTS: the old standard way is by starting with random base sets. This is inefficient and only "stochastically correct". Newer methods start with carefully selected base sets of observations (looks a bit complicated to code). Alternative, start with inliers in exog space by using a (robust) Mahalonibis distance.
ELTS (not LTS) adds observations based on closeness with respect to normal distribution as reference.

ultra-high dimensional screening: Initial random set, and then adding based on correlation or score statistic for added variables, shrinking based on SCAD or similar.

Oxford (Hendry et al) general to specific approach goes the other way, start with saturated model and then drop irrelevant variables or outlier dummies.

example (don't remember reference for approximately this algorithm)
(idea is to get algorithms into master that work but don't necessarily have a reference implementation or follow exactly some reference in the literature, and leave refinements, replacing parts for later.)

MM-estimator based on exog-inliers
- approximately robust Mahalonibis inliers: without fully robust and cheap, general distance estimate use standardization by robust variance and use Euclidean distance (ignore correlation) based only on continuous variables, try to keep enough discrete variables. (recent literature on mixed data distances ?)
- use x-inliers with RLM to be robust to y-outliers to get initial "clean" set of observations
- use ELTS base on this initial subset of observations, and then full MM-estimator

I think this should be high breakdown in **most** cases (no guarantees) and have very high efficiency.

Atkinson and Riani use a clean intial set and then add observations or blocks of observations that are close in "model-distance". It wasn't clear to me how they select the initial clean set.

related to time series, outlier, anomaly detection
https://github.com/statsmodels/statsmodels/issues/2873#issuecomment-209286703
The search process and masking problem is a similar issue to uncorrelated data, but we cannot just remove observations if we want to use a simple implementation of time series models.
If we have a long enough time series, then we can work on contiguous subsamples or windows. Markov-Switching models might also be a good way to identify a "dominant" regime.
Once we have a likely clean model, we can expand and add a very large number of outliers, and then in a second round trim all the ones that are not significant, or are penalized away.
(i.e. outlier detection as variable selection problem)

forward selection criteria:
I looked at several article (don't remember the references) that use variation on variable addition score tests. I don't think we want to end up with stepwise regression and just add many of them in each step so we have a good chance to nest the true model, and just have to remove, say 50% overhead.
(That's the basic idea in Jianqin Fan's ultrahigh dimensional screening with high probability for oracle, IIRC)
(Older approaches that just use unconditional correlation are not very useful if there is a large amount of correlation between variables.)

References (to be filled in)

Atkinson, Anthony C., and Marco Riani. 2002. “Forward Search Added‐variable T‐tests and the Effect of Masked Outliers on Model Selection.” Biometrika 89 (4): 939–46. doi:10.1093/biomet/89.4.939.

Riani, Marco. 2004. “Extensions of the Forward Search to Time Series.” Studies in Nonlinear Dynamics & Econometrics 8 (2). doi:10.2202/1558-3708.1208.

user: josef-pkt
issue_2942: state: open
title: ENH: GMM/QIF(GEE) regularized weight matrix
body: I don't see an open issue for this, and just found a statistics article for it for QIF-GEE

To get started: using principal components

Cho and Qu use principal components on a subset of moment conditions in estimating the correlation structure in GEE type models.
Cho, Hyunkeun, and Annie Qu. 2015. “Efficient Estimation for Longitudinal Data by Combining Large-Dimensional Moment Conditions.” Electronic Journal of Statistics 9 (1): 1315–34. doi:10.1214/15-EJS1036.

There are several articles in econometrics for this

The older article in econometrics
Doran, Howard E., and Peter Schmidt. "GMM estimators with improved finite sample properties using principal components of the weighting matrix, with an application to the dynamic panel data model." Journal of Econometrics 133, no. 1 (2006): 387-409.

and a lot more that is on my target list
Carrasco, Marine. "A regularization approach to the many instruments problem." Journal of Econometrics 170, no. 2 (2012): 383-398.

implementation in GMM

Initially I thought of using the existing pattern for the weight matrix methods to include regularization, which doesn't need any new infrastructure.
However, based on a brief look Cho and Qu start with principal components of the moment conditions to reduce the number of moment conditions. This is equivalent to adjusting the weightmatrix but has the advantage of making the system, dimension of arrays, in the estimation smaller.
I never went through this articles to check actual implementation.

**edit**
possible use for VAR cov_resid https://github.com/statsmodels/statsmodels/issues/3260#issuecomment-258735064
user: josef-pkt
issue_2941: state: open
title: SUMM/FAQ-D: optimization algorithms for complex objective functions
body: What optimization algorithms are available if the objective function or/and estimating equations consist of several parts difficult parts?

I started to skim several articles that describe various optimization algorithms for general or specific multi-part objective functions. We are using some algorithms on a model specific basis, but I think we should eventually move to a more consistent, systematic pattern.

(a brief overview based on what I remember and a reference I just found)
This is not a classification, because algorithms overlap
- IRLS
- scipy optimizers, newton or pseudo-newton, and others
- popular in machine learning (mostly for large number of explanatory variables)
  - gradient descend, coordinate descend, ...
- backfitting as in GAM (not needed if we have a linear basis)
- two-stage or iterative: GLS, GEE, dispersion modelling. (for block-diagonal information matrix)
- alternating projections
- local quadratic approximation, e.g. quadratic solution to L1/SCAD non-differentiability at zero
- ...

References

Fan, Yanqin, Sergio Pastorello, and Eric Renault. 2015. “Maximization by Parts in Extremum Estimation.” The Econometrics Journal 18 (2): 147–71. doi:10.1111/ectj.12046.
(I didn't read it, but looks interesting and I like the authors)

Related

The above are optimization algorithms that in most cases find the (at least asymptotically equivalent) solution. In some cases it is easier, necessary of faster to simplify the model or us an approximate solution.
- Simplified models, e.g. composite models, "marginal" models like GEE, or GMM
- Local Approximation: e.g. one step estimators, especially for diagnostics and computing variation on an existing solution.

....

user: josef-pkt
issue_2936: state: open
title: FAQ/SUMM: sampling spaces for rates and proportions
body: Hypothesis tests and confidence intervals are derived under some assumption on the sampling distribution, or on the conditional sampling distribution in the case of conditional tests.

In the case of asymptotic distribution of a test statistic, the sampling distribution enters in the assumed form of the variance. see for example #2689 

In the case of exact hypothesis test and confidence interval, we explicitly use the corresponding sampling distribution.

This is an attempt to provide an overview from this perspective (see e.g. 2nd part of comment https://github.com/statsmodels/statsmodels/issues/2607#issuecomment-217950653 )

contingency tables is mixing up or combining sampling distributions depending on the use case and hypothesis test
## binomial

```
- k-sample (k >0) test for a binary random variable, i.e. a single proportion
  e.g. contingency table 2xK with independent sampling in second dimensions
- GLM binomial
- most of what is in statsmodels.stats.proportion
```
## multinomial

```
- proportions for more than 2 category levels, total number of observations fixed
  e.g. histogram counts
  e.g. contingency tables conditional on total sample size
- new code in statsmodels.stats._multinomial
- conditional Poisson
```
## poisson

```
- no fixed number of observations
```
## hypergeometric

```
 - contingency table with row and column totals fixed
 - nothing in statsmodels ?
 - Fisher's exact in scipy.stats
```

< list needs to be filled >

user: josef-pkt
issue_2933: state: open
title: ENH: large sparse VAR
body: question on mailing list for sparse VAR
https://groups.google.com/d/msg/pystatsmodels/NWurUtjX9Qg/98xpoTIuBQAJ

reference
Davis, Richard A., Pengfei Zang, and Tian Zheng. 2015. “Sparse Vector Autoregressive Modeling.” Journal of Computational and Graphical Statistics 0 (ja): 1–53. doi:10.1080/10618600.2015.1092978.
(ahead of print)

some references that I downloaded, but I only read part of Basu/Michailidis 2015

Basu, Sumanta, and George Michailidis. 2015. “Regularized Estimation in Sparse High-Dimensional Time Series Models.” The Annals of Statistics 43 (4): 1535–67. doi:10.1214/15-AOS1315.

Dowell, J., and P. Pinson. 2016. “Very-Short-Term Probabilistic Wind Power Forecasts by Sparse Vector Autoregression.” IEEE Transactions on Smart Grid 7 (2): 763–70. doi:10.1109/TSG.2015.2424078.

Melnyk, Igor, and Arindam Banerjee. 2016. “Estimating Structured Vector Autoregressive Model.” arXiv:1602.06606 [Math, Stat], February. http://arxiv.org/abs/1602.06606.

Nicholson, William B., Jacob Bien, and David S. Matteson. 2014. “Hierarchical Vector Autoregression.” arXiv:1412.5250 [Stat], December. http://arxiv.org/abs/1412.5250.

Nicholson, William, David Matteson, and Jacob Bien. 2015. “VARX-L: Structured Regularization for Large Vector Autoregressions with Exogenous Variables.” arXiv:1508.07497 [Stat], August. http://arxiv.org/abs/1508.07497.

Wong, Kam Chung, Ambuj Tewari, and Zifan Li. 2016. “Regularized Estimation in High Dimensional Time Series under Mixing Conditions.” arXiv:1602.04265 [Cs, Stat], February. http://arxiv.org/abs/1602.04265.

user: josef-pkt
issue_2932: state: open
title: ENH: stats for multinomial samples
body: exact monte carlo p-values for multinomial proportions

user: josef-pkt
issue_2931: state: open
title: ENH  hypothesis tests, confint for multinomial proportions (exact)
body: companion to issue #2895

Cai, Yong, and K. Krishnamoorthy. "Exact size and power properties of five tests for multinomial proportions." Communications in Statistics—Simulation and Computation® 35, no. 1 (2006): 149-160.
(only 7 citation in Google scholar, but looks good to me.)

The probability space for multinomial distribution is too large for exact calculation except in very small samples. For box/interval probabilities there are some faster algorithms that don't look easy to implement. (At least I didn't try to figure out the details of the algorithms in two recent articles.)

As alternatives: 
- The references in #2895 Sison and Glaz and by May and Johnson use an approximation that is based on truncated Poisson and Edgeworth expansion. My implementation works for some examples but not yet for others. It calculates box probabilities and confint are appropriate if probabilities are roughly the same across bins.
- Monte Carlo probabilities: That's much easier to implement and seems to work very well, and is not very slow. It is also more general than just having box probabilities and can be used for p-values based on some test statistic. Cai and Krishnamoorthy use exact calculation for small samples based on chisquare test, and recommend Monte Carlo for larger samples. In the small sample the Monte Carlo also agrees at about 3 decimals with the exact p-values when using 100,000 replications.

Related: 
They also have "Nass test" which is the chisquare test with corrected distribution (scaled chisquare with adjusted degrees of freedom) which is doing very well in small, but not tiny, samples.  (small in multinomial chisquare test refers to expected number of observation in each bin).
Also, compared to binomial proportion both standard chisquare test and exact test work better if there are more bins, with smaller liberal resp. conservative deviation from size.
LR (Q) test has a quite distored size.

Status
I wrote some function that mostly work, 
I'm using a semi-generic function to calculate multinomial probabilities by simulation based on an indicator callback function.

(I haven't looked yet what R packages are doing in this area.)

user: josef-pkt
issue_2929: state: open
title: BUG/ENH: edgeworth expansion for discrete random variables
body: There seems to be a problem in the Edgeworth expansion in statsmodels.distributions for this case.

The example is Poisson where all cumulants are equal to the rate lambda.
The numbers using ExpandedNormal are very large, either something weird or something that forces a density.
I have no idea whether the problem is in the computation or the theory, because I never looked at Edgeworth expansion for discrete distribution before.

Below compares ExpandedNormal, scipy.stats.poisson and a handwritten edgeworth expansion based on mvsk.

```
import statsmodels.stats.moment_helpers as smmh
from statsmodels.distributions.edgeworth import ExpandedNormal
```

```
>>> ExpandedNormal(16 * np.ones(4)).pdf(x)
55.122275327263743
>>> x = 12
>>> ExpandedNormal(16 * np.ones(4)).pdf(x)
55.122275327263743
>>> stats.poisson.pmf(x, 16)
0.066128872394649665

>>> cum_poi
array([ 16.,  16.,  16.,  16.])
>>> mvsk16 = smmh.mnc2mvsk(smmh.mc2mnc(smmh.cum2mc(cum_poi)))
>>> pdf_edgeworth_mvsk_simple(x, mvsk16)
0.065953825954002607
```

where the explicit function is based on articles in and written for #2895

```
_norm_pdf_C = np.sqrt(2*np.pi)
def _norm_pdf(x):
    return np.exp(-x**2/2.0) / _norm_pdf_C

def pdf_edgeworth_mvsk_simple(x, mvsk):
    """pdf of Edgeworth expansion based on mean, variance, skew and kurtosis

    This uses explicit power polynomial formulas.
    The moments in `mvsk` are tuple unpacked and all calculations are 
    elementwise, so numpy broadcasting applies.

    """
    m, v, skew, kurt = mvsk

    s = np.sqrt(v)
    xs = (x - m) / s
    del x, v, m

    xs2 = xs * xs
    xs3 = xs2 * xs
    xs4 = xs2 * xs2

    pdf = (1 +
           skew / 6 * (xs3 - 3 * xs) +
           kurt / 24 * (xs4 - 6 * xs2 + 3) +
           skew**2 / 72 * (xs3 * xs2 - 15 * xs4 + 45 * xs2 - 15))
    pdf *= _norm_pdf(xs) / s

    return pdf
```

aside: cumulants are additive: the cumulants of a sum of independent random variables is the sum of the cumulants

user: josef-pkt
issue_2926: state: open
title: ENH: Add a generic plot_predict function
body: ## ENV
- Python 2.7
- np.**version**: 1.11.0
- pd.**version**: 0.18.0
- scipy.**version**: 0.17.0
- matplotlib.**version**:  1.3.1
- statsmodels.**version**: 0.5.0
## WHERE

I tried the [ARMA code](http://statsmodels.sourceforge.net/devel/examples/notebooks/generated/tsa_arma_0.html) and I got error in `In[21]`

```
fig, ax = plt.subplots(figsize=(12, 8))
ax = dta.ix['1950':].plot(ax=ax)
fig = arma_mod30.plot_predict('1990', '2012', dynamic=True, ax=ax, plot_insample=False)
```

And the `plot_predict` [doc](http://statsmodels.sourceforge.net/0.6.0/generated/statsmodels.tsa.arima_model.ARMAResults.plot_predict.html)
## WHAT ERROR

In my jupyter-notebook, it shows

```
In[21]
    fig, ax = plt.subplots(figsize=(12, 8))
    ax = dta.ix['1950':].plot(ax=ax)
    fig = arma_mod30.plot_predict('1990', '2012', dynamic=True, ax=ax, plot_insample=False)

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-28-2a5da9c756f0> in <module>()
      1 fig, ax = plt.subplots(figsize=(12, 8))
      2 ax = dta.ix['1950':].plot(ax=ax)
----> 3 fig = arma_mod30.plot_predict('1990', '2012', dynamic=True, ax=ax, plot_insample=False)

/usr/lib/python2.7/dist-packages/statsmodels/base/wrapper.pyc in __getattribute__(self, attr)
     33             pass
     34 
---> 35         obj = getattr(results, attr)
     36         data = results.model.data
     37         how = self._wrap_attrs.get(attr)

AttributeError: 'ARMAResults' object has no attribute 'plot_predict'
```

I am new to `statsmodels`. If this is a bug of it, thank you for fixing it out!

user: GoingMyWay
issue_2924: state: open
title: Phreg predict
body: See #2894 

This should map to what was mentioned, there are still some issues with the MICE tests but I wanted to get this up.

user: lbybee
issue_2915: state: open
title: REF: Pearson estimate of scale parameter/Tweedie parameter with weights
body: I'm a little confused over how to interpret the Pearson estimates for the scale parameter and a shape parameter in a frequency dataset. 

The Pearson estimate of scale is as follows:

![formula](https://chart.googleapis.com/chart?cht=tx&chl=%5Cphi_%7Bpearson%7D%20%3D%20%5Cfrac%7B%5Csum%7B%5Cfrac%7Bw_i%20%28Y_i%20-%20%5Cmu_i%29%5E2%7D%7BV%28%5Cmu_i%29%7D%7D%7D%7Bdf_%7Bresid%7D%7D)

If I have 1 record representing 1 year of a policy and it has 1 claim, then the frequency (which is the same as endog and Y) equals 1.

If I split that record into 2 separate 6-month records, then I now have 1 record with a frequency of 2 and a separate record with a frequency of 0. (The claim only occurs once... the only difference is how the data is laid out). Mu is the same between all the scenarios. It becomes pretty intuitive to see that you will get a different estimate of phi based off of my (somewhat) arbitrary splits in the policy. 

Is there an adjustment that we could do to the Pearson estimate so that 1 12-month record and 2 six-month records, with 1 annual claim in each scenario, would give the same estimate of scale?

user: thequackdaddy
issue_2914: state: open
title: FAQ: number of cores used in statsmodels, and how to restrict it
body: Statsmodels is using joblib in a few places for parallel processing where it's under our control. Current usage is mainly for bootstrap and it is not used in the models directly.

However, some of the underlying Blas/Lapack libraries in numpy/scipy also use mutliple cores. This can be efficient for linear algebra with large arrays, but it can also slow down the operations especially when we want to use parallel processing on a higher level.

How can we restrict the number of cores used by the linear algebra libraries?

This depends on which linear algebra library is used. see mailing list thread 
https://groups.google.com/d/msg/pystatsmodels/Lz9-In0pgPk/BtcYsj_ABQAJ
- `openblas`: try setting the environment variable `OMP_NUM_THREADS=1`
- `Accelerate` on OSX, set `VECLIB_MAXIMUM_THREADS`
- `mkl` in anaconda 
  
  ```
  import mkl
  mkl.set_num_threads(1)
  ```

user: josef-pkt
issue_2913: state: open
title: SUMM/REF  refactor link argument in family
body: Currently the `family.__init__` takes the link class but not an instance.

This causes problems with parametric links because we need to specify an extra argument in the `family.__init__` for the link parameter.
For example, I don't see how it's possible to use a Power link with user specified power parameter except by writing a subclass, which will break in the valid link check.
see discussion for Tweedie in https://github.com/statsmodels/statsmodels/pull/2872/files#r61099271

This is the original refactoring that switched from using link instances to using link classes as argument
https://github.com/statsmodels/statsmodels/commit/5917fcc6640df3840c564c0f50cfe86c34ccd77e

related issues:
- #1509 same issue as described here (the part above is duplicate of it)
- #28 a very old issue with the same problem
- #1778 power link not allowed for Gaussian
- #2351 issue for scale not for link, but uses a path based on a link isinstance check
- #2872 add Tweedie: PR will follow old pattern but this will need refactoring based on outcome of this issue.
- #1880  PR to make valid link check optional

---

We can keep link classes as argument during a deprecation period while we switch to link instances.

Note: we don't want to provide instances in statsmodels to avoid that multiple models use the same instance. Based on a comment in the old commit this might have been one of the problems with the initial instance based implementation. (similar to scipy distribution "global" instances)

user: josef-pkt
issue_2908: state: open
title: ENH: MLE for heteroscedasticity with WLS
body: Just an idea how to get the MLE version of GLSHet.  
(while thinking about the difference between loglike and deviance. The latter doesn't contain the relevant part of the loglike for variance estimation.)

WLS has the full normal loglikelihood, AFAIU. So we can maximize `llf` over variance/weights parameters. i.e. use an optimization loop over WLS.
(Kerby did something similar for estimating the NegativeBinomial parameter with GLM.
at the end of this notebook http://nbviewer.jupyter.org/urls/umich.box.com/shared/static/0lgt5635uo6rhakjj9d5.ipynb
just evaluating on some grid.)

small issue: the concentrated linear regression loglike removes the scale, i.e. factor to multiply variance/weight function.

user: josef-pkt
issue_2906: state: open
title: FAQ: modelling seasonality in time series
body: We have seasonal effects in SARIMAX, but not in the original ARMA, #247

other alternatives
- use deterministic seasonality through exog.
- other tsa models like exponential smoothing versions, ....  see PR #2892
- prefilter seasonality, seasonal adjustment
- ...

Possible problem: in statespace framework long seasonality is memory intensive in the dense representation

a few links

http://robjhyndman.com/hyndsight/seasonal-periods/
http://robjhyndman.com/hyndsight/longseasonality/
https://m.reddit.com/r/MachineLearning/comments/4bwug8/time_series_analysis_help/

http://stackoverflow.com/questions/36826883/python-sum-of-sine-fits (not directly for seasonal, OLS with sin, cos exog.

user: josef-pkt
issue_2902: state: open
title: FAQ: namespace, eval_env in formulas, from_formula
body: How do we update the eval environment when we call patsy within a function, but the formula uses functions from the outer scope?

http://stackoverflow.com/questions/36798992/namespace-issues-when-calling-patsy-within-a-function/36803612

incrementing depth levels in `eval_env` keyword.

user: josef-pkt
issue_2900: state: open
title: BUG/ENH: implement formula interface for t_test in MixedLMResults
body: Implement formula interface for t_test in MixedLMResults

https://github.com/statsmodels/statsmodels/blob/master/statsmodels/regression/mixed_linear_model.py#L2266

Note: may have to drop extra coefficients if vc_formula was used as the t-test doesn't accept these.

user: bburan
issue_2899: state: open
title: multipletests/fdrcorrection0 returning corrected array of nans
body: It seems if I pass in an array with any nan values, the pvals_corrected is filled with only nans. 
The functions I've used:
`statsmodels.sandbox.stats.multicomp.multipletests(p,alpha=0.05,method='fdr_bh')`
`statsmodels.sandbox.stats.multicomp.fdrcorrection0(p,alpha=0.05)`

---
the following is based on the discussion below:

If we have missing values, nans, in the uncorrected p-values then we can remove them and assign the results to the original position of a pval-corrected array, i.e.

```
mask = np.isfinite(p)
pval_corrected = np.empty(p.shape)
pval_corrected.fill(np.nan) 
pval_corrected[mask] = multipletests(p[mask],method='fdr_bh')[1]
```

When using pandas, we need to go through numpy ndarrays to apply the mask

```
import numpy as np
import pandas as pd
import statsmodels.stats.multitest as multitest

p = np.array([[0.0, 0.1], [0.8, np.nan], [0.05, 0.1]])
p_df = pd.DataFrame(p, index=["a", "b", "c"], columns=["d", "e"])

mask = np.isfinite(p)
pval_corrected = np.full(p.shape, np.nan)
pval_corrected[mask] = multitest.multipletests(p[mask], method='fdr_bh')[1]

pval_corrected_df = pd.DataFrame(pval_corrected, index=p_df.index, columns=p_df.columns)
print pval_corrected_df
```

In this case hypothesis tests with missing or nan pvalues are not counted for the total number of hypothesis tests in the multiple testing pvalue correction.

If we want to count all missing/nan for the total number of tests, but don't want them to be rejected, then we can replace missing/nan pvalues by one instead of removing them.

user: mgxd
issue_2898: state: open
title: ENH: GMM-GLM dispersion Poisson/NegativeBinomial - optimal instruments
body: just parking a reference:

issue is replicating GLM with GMM, specifically with dispersion modelling, related to Quadratic Estimating Equations.

In Tweedie PR #2872, issue #2858 the question was how to find the moment condition for the dispersion parameters, specifically the "instrument" in front of the conditional moment restriction for dispersion/variance.
The approach we finally took is directly from a reference for Pearson function estimator.
The more general question is what are the optimal, or at least better than ad-hoc, instruments.

I just saw that Cameron and Trivedi count data 2nd ed. have sections (11.7) on QEE and GMM for moment estimators for Poisson/NegativeBinomial type count models.
expression for optimal instruments look too complicated, but it's a more specific example than Chamberlain and Newey.

---

topic: models that include functions for 2nd moment (variance and correlation functions)
GLSHet, Garch, QIF, GEE, GMM, double exponential, ...
Mixed Models are also in this category but they are just MLE where all the work is computational

user: josef-pkt
issue_2897: state: open
title: ENH: distribution based on series expansion - Poisson
body: (while cleaning up a pile of papers, I found this again)

Cameron, A. Colin, and Per Johansson. "Count data regression using series expansions: with applications." Journal of Applied Econometrics 12.3 (1997): 203-223.

section 11.2 in count data book 2nd edition describes series expansion Poisson and chapter describes also other "flexible" methods.  (not read)

chapter in book mentions BIC for selecting number of terms, and inference based on assumption of true model, i.e. standard MLE asymptotics for finite number of parameters.

This follows series approach in Gallant and Nychka (squared sum of series expansion)

this differs from the orthogonal Poisson series expansion of Best, Rayner, Thas (smooth tests)
(where I didn't find out whether it can be used in regression context)

user: josef-pkt
issue_2894: state: open
title: BUG/REF PHReg.predict returns Bunch
body: PHReg.predict returns a Bunch with predicted and standard errors which breaks writing generic code on top of it
https://github.com/statsmodels/statsmodels/pull/2884#issuecomment-210511548

it would also break generic cross validation or similar.

The pattern that we get slowly into is that predict stays just the mean prediction/expected values and a new method `get_prediction` returns the full prediction results.

One possibility would be to add a new keyword `return_se=False` which could be reused and wrapped by `get_prediction`

user: josef-pkt
issue_2892: state: open
title: ENH: New Seasonal TimeSeries Model:TBATS/BATS/MTBATS
body: A new timeseries forecast/decompose model based on Rob.J.Hyndman's paper.
http://robjhyndman.com/papers/complex-seasonality/

It is the same as the tbats model of R's forecast package.

Feature:
1. Long Seasonality forecast (such as minutes data)
2. Complex seasonality based on trigonometric transform(such as multiple seasonality/non-interger periods)
3. BoxCox transform
4. arma error
5. exog support

work:
- [x] TBATS Model
- [ ] BATS Model
- [ ] MTBATS Model
- [ ] MBATS Model
- [ ] Documents
- [ ] Related
  - [ ] Test case
  - [ ] IPython notebook examples
  - [ ] related datasets
- [ ] Base Tools
  - [x] Fourier Seasonal tools
  - [ ] Fourier forecast exog generate tool
  - [x] Model auto select tool
  - [x] Fourier K auto select tool

user: Leoyzen
issue_2890: state: open
title: ENH: dispersed contingency tables, counts, rates and proportions
body: When I was reviewing contingency tables, loglinear models and one or two sample tests for rates and proportions, I saw that all inference relies on the correctly specified model in terms of absence of unobserved heterogeneity, correlation or similar sources of over or under dispersion.

related:
#2718 #2723 two sample hypothesis test and confidence intervals for Poisson rates
#2743 #2605 #2608 two sample hypothesis test and confidence intervals for proportions

and others
#2418 (merged) and others for contingency tables
#2688 loglinear model (no general issue, works with GLM-Poisson, examples in my repo)
#2889 SUMM dispersion modelling in GLM

one possibility is to use negative binomial as the relevant distribution.
possible problem with having only aggregate data

Efron, Bradley. 1986. “Double Exponential Families and Their Use in Generalized Linear Regression.” Journal of the American Statistical Association 81 (395): 709–21. doi:10.2307/2289002.
has a contingency table example

Mütze, Tobias, Axel Munk, and Tim Friede. 2016. “Design and Analysis of Three-Arm Trials with Negative Binomially Distributed Endpoints.” Statistics in Medicine 35 (4): 505–21. doi:10.1002/sim.6738.
(I have not yet looked at it)

Sudhir Paul & Khurshid Alam (2014): Testing equality of two negative binomial
means in presence of unequal over-dispersion parameters: a Behrens–Fisher problem analog,
Journal of Statistical Computation and Simulation, DOI: 10.1080/00949655.2014.955025

user: josef-pkt
issue_2889: state: open
title: SUMM/ENH: dispersion modelling in GLM, exponential family
body: (this is triggered by some readings on dispersion estimation as followup to Tweedie #2858 #2872)

the question is how do we estimate dispersion parameters or data (exog not mean) varying variance functions.

Basic idea:
1. underlying distribution for fixed dispersion is LEF, mean and dispersion are orthogonal (asymptotically independent)
2. we can just use any moment conditions on the variance and covariance of errors to estimate a dispersion function.

The question is how do we choose the moment condition or estimating function for the dispersion parameters.

example: 
Tweedie: In this case we could use full MLE but that is expensive because we need to compute the compound Poisson distribution.
Similar in several other distributions like normal that have explicit variance and dispersion in the likelihood function.
Poisson or Binomial with general dispersion are not likelihood models and we need something else to define dispersion.

Instead we can use some moment definition based on Pearson or deviance residuals. E.g. used to estimate overdispersion.

analog to feasible generalized least squares FGLS: 
use a model for the variance for squared residuals to estimate the parameters
e.g. double exponential model https://cran.r-project.org/web/packages/dglm/dglm.pdf

GEE does something similar to estimate the parameters of the working correlation matrix.

statistics: 
various approaches to create "fake" likelihood models based on approximations to some general class of distributions. (extended quasi-likelihood, exponential dispersion models, double exponential models, ...)

econometrics: two main approaches AFAICS
1. model the source of dispersion: e.g. explicitly model latent variables for unobserved heterogeneity or heteroscedasticity. e.g. Poisson with different mixing distributions, Probit with heteroscedasticity in the underlying normal latent variable
2. Just use LEF and orthogonality of mean and 2nd moments to estimate with "ad-hoc" moment or orthogonality condition for variance or correlation. Mean parameter are estimation is robust to variance misspecification, just use sandwiches for inference. Same idea in GEE.
   (question: are there optimial instruments and efficient moment conditions for GMM.)
   Gourieroux et al., Wooldridge, Cameron and Trivedi

Another related approach is to find a non LEF distribution that has the desired properties and extra parameters. The advantage of this and item 1. for econometrics above is that we have a full probability model that we can use for prediction, instead of just estimating parameters or effects.
### Getting started
- two step or iterative estimation similar to FGLS on top of current GLM
  - extra parameter as in NegativeBinomial or Tweedie variance function
  - dispersion as function of explanatory variable GLMHet in analogy to GLSHet
    need `var_weights` from #2879

user: josef-pkt
issue_2888: state: open
title: BUG/Review GLM scale type negative binomial
body: Why does GLM negative binomial default to pearson estimate of overdispersion?

unit tests for GLM negative binomial use `Committee` which has reference number scale=0.385...

scaletype is set in `fit` i.e. possible problem if `fit` is called repeatedly with different scale types

AFAICS: we can use `scale` argument in `fit` to get cov_type dispersed through the choice of pearsnon or deviance dispersion estimate
Note: pearson dispersion is abbreviated with `"X2"` which would be better as `pearson`, `dev` might also be better spelled out as `deviance`.

AFAIK, there is currently no usage of choosing scale/scaletype in `fit` 
My initial thought was to implement over/under dispersion through `cov_type`
-> merge scale into cov_type or keep separate?

user: josef-pkt
issue_2886: state: open
title: ENH: GAS models, generalized autoregressive score models
body: This looks like a very interesting approach for general nonlinear/non-normal time series models, including point processes

(mentions Poisson ARMA type models as special case, but most applications are in finance)

http://www.gasmodel.com/gaspapers.htm
"The GAS model also goes by the name of Dynamic Conditional Score (DCS) model, Score Driven (SD) model, or Dynamic Score (DySco) model."

I only read the article introducing GAS model, but given the list of articles in 2015 and 2016 this looks like it's becoming very popular.
Main advantage: we can use MLE to estimate models where similar models could before only be estimated by simulation (MCMC, particle filter)
Creal, Drew, Siem Jan Koopman, and André Lucas. 2013. “GENERALIZED AUTOREGRESSIVE SCORE MODELS WITH APPLICATIONS” Journal of Applied Econometrics 28 (5): 777–95. doi:10.1002/jae.1279.
working paper version is 2008

The GAS paper only talks about MLE, but given that they start with score equations, I think this should generalize to QMLE and GMM.
In LEF, score only depends on first moment or first two moments, not the entire distribution. 

user: josef-pkt
issue_2885: state: open
title: ENH: transfer impulse response function, lagged exog
body: a bit related #881 dynamic models, where some or most of the dynamics comes from delayed response to changes in exog.

What tools do we have for interpretation? e.g  impulse response function for exog instead of shock.

---

application where I just look for something related:
mortality count models in response to air pollution:
e.g. higher observed air pollution immediately increases hospitalization and death rate or count, but the effect is partially shifting death to earlier periods. The long term (cumulative rate for one month) net effect _might_ be small.
I bumped into this by accident right now (via time series analysis for GLM/Poisson)
http://pubs.healtheffects.org/view.php?id=348
Public Health and Air Pollution in Asia (PAPA): Coordinated Studies of Short-Term Exposure to Air Pollution and Daily Mortality in Four Cities

(cities and countries publish air pollution data which sounds like interesting datasets, e.g. available for Montreal. but needs merging/aggregation and cleaning, and linking with other data)

user: josef-pkt
issue_2883: state: open
title: SUMM/ENH:  model averaging estimators
body: an issue to start with a new topic: we don't have any model averaging or ensemble methods
I never looked at it, but Bruce Hansen started several years ago. 

What's the theory? What's interesting beyond machine learning?

Liu, Chu-An. 2015. “Distribution Theory of the Least Squares Averaging Estimator.” Journal of Econometrics 186 (1): 142–59. doi:10.1016/j.jeconom.2014.07.002.

DiTraglia, Francis. 2015. “Using Invalid Instruments on Purpose: Focused Moment Selection and Averaging for GMM, Second Version.” SSRN Scholarly Paper ID 2642977. Rochester, NY: Social Science Research Network. http://papers.ssrn.com/abstract=2642977.

and Bruce Hansen's articles, and a (google scholar) citation list for the above look like good starting points.

user: josef-pkt
issue_2882: state: open
title: ENH: penalization, shrinkage to subspace
body: (issue mainly to park an article that might have good general theory in MLE context, not read yet)

related PR #1665 TheilGLS, generalized Ridge for linear model

Hansen, Bruce E. 2016. “Efficient Shrinkage in Parametric Models.” Journal of Econometrics 190 (1): 115–32. doi:10.1016/j.jeconom.2015.09.003.

I was thinking about something similar also in a GMM framework (but didn't look for literature)

also related: combining estimators from different samples (as in distributed computing or small area analysis)
in the Theil Goldberger interpretation the "prior" is the estimation result of a previous sample

user: josef-pkt
issue_2875: state: open
title: ENH: score tests and weights
body: sub issue for #2849  weights in GLM (and MLE models)

I need to check this:
variable addition test might need multiplication by weights, i.e. call whiten in WLS, GLS

score/LM test for variable addition uses `score_factor * exog_new`
score_factor has weights (corresponding to wresid, i.e. sqrt(weights), while exog_new has no weights. Do we need equivalent of `wexog`?

main question to check weights, squared weights, square-root of weights, 
When do we use which one?

user: josef-pkt
issue_2873: state: open
title: ENH: Outlier robust estimation using weights
body: An idea for a new wrapper model when we have weights available in the estimation.

Use IRLS as in RLM on top of current MLE models.
e.g. a simple implementation would use an existing model and model class and repeatedly fit it with weights defined by the RLM norms (for outliers in endog or residuals) and similar weighting function for exog outliers, i.e. influental observations in x.
This could use a small maxiter in the inside optimization, or even maxiter=1 as in RLM, although that converges in one step.
More efficient implementation should reuse starting values, hessian, maybe one-step estimators. (or use BHHH style gradient WLS.) 

two papers as starting point for search in GEE context:

Preisser, John S., and Bahjat F. Qaqish. 1999. “Robust Regression for Clustered Data with Application to Binary Responses.” Biometrics 55 (2): 574–79. doi:10.1111/j.0006-341X.1999.00574.x.

Lee, Donghwan, Youngjo Lee, Myunghee Cho Paik, and Michael G. Kenward. 2013. “Robust Inference Using Hierarchical Likelihood Approach for Heavy-Tailed Longitudinal Outcomes with Missing Data: An Alternative to Inverse Probability Weighted Generalized Estimating Equations.” Computational Statistics & Data Analysis 59 (March): 171–79. doi:10.1016/j.csda.2012.10.013.

user: josef-pkt
issue_2871: state: open
title: Matching: tools and estimator (propensity score or covariance balanced)
body: #858 and PR #2288 propensity score matching
#2443 SUMM issue treatment effects

a few more references to complement #858 

in python:
https://github.com/MacHu-GWU/ctmatching-project  uses scikit-learn neighbors
https://github.com/laurencium/CausalInference   treatment effects, several kinds including matching
https://github.com/kellieotto/ModelMatch old python code before switching to R, has also the Dehejia-Wahba Lalonde data

list of software
http://www.biostat.jhsph.edu/~estuart/propensityscoresoftware.html (Stuart)
found in comments at http://sas-and-r.blogspot.ca/2010/04/example-734-propensity-scores-and.html
and an article 
Stuart, Elizabeth A. 2010. “Matching Methods for Causal Inference: A Review and a Look Forward.” Statistical Science 25 (1): 1–21. doi:10.1214/09-STS313.

Besides the specific propensity score matching in PR #2288 we need more generic tools for matching and stratification, and diagnostic, graphical tools for evaluating the match, e.g. covariate or propensity score balancing across matches and treatments.

user: josef-pkt
issue_2870: state: open
title: ENH: spatial GLM
body: just to park a reference to pysal and possible extensions for it to GLM or Poisson
https://github.com/TaylorOshan/pysal/blob/GIS596_gwr/pysal/contrib/gwr/glm.py

(maybe GSOC)

GLM and Poisson with more cross-observation correlation, different from GEE.
possibly sparse matrices

(I didn't look at any details, but there might be tools or patterns to imitate)

user: josef-pkt
issue_2867: state: open
title: save calibration parameter (mle_model_results.MLEResults)
body: Hello,

I am using SARIMAX model to forecast hourly electric load. When, I used 1 year of hourly historical data, the fit of the model is long so I want to be able to save in a file my SARIMAX model results to be able to forecast in a later session.
I would like to know if it is possible to save the model results that will only contain the optimized calibration parameter. The file created by "sarimax_model_results.save(cur_w_dir + OUTPUT_FILE_SARIMAX_PARAMETER, remove_data=False)" is 600 MByte large. I would like to know if we can make it smaller.

user: fgilbert01
issue_2861: state: open
title: ENH: doubly robust GEE
body: (mainly a reference, and possibly a reference for unit tests)

https://cran.r-project.org/web/packages/drgee/index.html

http://www.degruyter.com/view/j/em.2015.4.issue-1/em-2014-0021/em-2014-0021.xml
Zetterqvist, Johan, and Arvid Sjölander. 2015. “Doubly Robust Estimation with the R Package Drgee.” Epidemiologic Methods 4 (1): 69–86. doi:10.1515/em-2014-0021.

I don't have access to the article and didn't find an ungated version.

Might only add longitudinal to what can be done with GLM/GMM.
In skimming abstracts from other articles, often only the independence working correlation is used in double robust GEE type models. In that case, we can replicate using GLM/GMM directly.

(Note IPW requires that probability weights, or at least var_weights are implemented. #2455 so far can only use linear outcome model because WLS was the only model with weights. GLM has now freq_weights added in #2835.)

user: josef-pkt
issue_2860: state: open
title: ENH:  GLM family for estimating equations `EE`
body: experimental idea

I think we can reuse large parts of GLM with IRLS with just estimating equations defined by mean/link and variance function.

This is similar to Quasi/Pseudo MLE in R but without defining a quasi-likelihood objective function.
We could set loglike and whatever else is invalid to nan or None. (nan is nicer, None is safer because it raises for sure. We need to check that there is no linear algebra that can break with nans.)

Also, it would only work with irls as optimizer.

Possible usecases would be to see how far we can implement something like tweedie #2858 without specifying loglike. 
Another case would be to add arbitrary variance function.

Conceptually this would be similar to the GEE/GMM approach, but for the independent observation case that is handled in GLM.
Compared to GMMGLM we might gain some additional postestimation results that are already available in GLM but where there is nothing specific in GMM.

user: josef-pkt
issue_2851: state: open
title: FAQ/ENH: weighted sums Stata's total
body: What's the pandas/statsmodels equivalent of Stata's total function?

http://www.stata.com/manuals14/rtotal.pdf

(I just saw this while looking for other things related to weights #2849 and didn't look at details.)

It's mainly useful for survey probabilities and weighted population means and totals. This is standalone and maybe similar to `stats` weighted statistics with more types of weights.

The related usage in models is for predict_at and margins with frequency or probability weights. (open question and not checked for freq_weights in models in #2849. )

user: josef-pkt
issue_2849: state: open
title: followup GLM weights, Stata and Poisson
body: freq_weights for GLM has been merged in #2835 
the summ issue is #2848

the PR #2835 was testing compared to the model with expanded, repeated data
I spot checked against Stata

issues to check
- optimization: what's the behavior if weights are None
- residuals and their usage: unclear, see below
- applicability or reusability for other weight types
- margins: not checked yet at all.
  Stata has a list of `marginsnotok` after fweight. How do weights affect `margins`? Currently we only have margins for change in mean, equivalent to `predict`. Aggregation, i.e. averaged margins and margins at average exog, would need to take weights into account.

compared to SUMM issue, I looking specifically at the comparison to Poisson and GLM-Poisson in Stata
## residuals and similar

checking anscombe and deviance residuals, they both ~~agree~~ with Stata fweights
correction only anscombe agrees, deviance residuals: we use weighted, Stata uses unweighted, AFAICS. 
Calling `model.family.resid_dev` without freq_weights produces the same deviance residuals as Stata, i.e. it is unweighted.
Using aweights in Stata produces exactly the same resid_dev as using fweights.

related issue: fittedvalues is on a per observation basis, i.e. not weighted. checking an example we have the same fittedvalues with freq_weights, as Stata has with `aweights` (should be the same as with `fweights`)
Maybe predict should get weights. AFAICS, WLS predict does not allow weights. Not needed in predicting mean but would be needed for prediction/confidence interval.
-> check `get_prediction` behavior for se and intervals.

Currently, with PR #2835, the definitions are a bit inconsistent
e.g. `resid_reponse` is weighted, `endog - fittedvalues` are unweighted.

possible solution: make all `resid_xxx` unweighted, add `wresid_xxx` but don't cache them. (analogous to wresid). (What we need depends on the use case.)
## reusing for other weights

comparing fweights to Stata `aweights`:
- params are the same
- bse differ by sqrt(nobs / wnobs) -> need to reset wnobs
- Stata docs mention that aweights and pweights use fweights after normalizing the weights to nobs.
- not clear for auxiliary results: e.g. resid_pearson should take account of var_weights but not of freq_weights, I guess.
- reweighting freq_weights as use to sum to nobs agrees with Stata aweights results for bse, llf, deviance and pearson ( `(resid_pearson**2).sum()` )

pweights:
- I'm not able to get the same bse as Stata, they don't differ by a constant factor for HC1 in any version with weights as in the aweight equivalent experiments. `params` agree and are the same in all version of weights.
  **update** the issue here is not pweights itself. The calculation of HC1 assumes freq_weights, and  using freq_weights normalized to sum to nobs is not the same as aweights 
- llf, pearson and deviance in Stata with pweights look like they correspond to fweights (larger sample) while reported nobs and df_resid uses the small nobs=number of rows.
## cov_type
- using freq_weights: HC1 corresponds to Stata fweights `vce(robust)` with the usual correction ( for bse: by factor `sqrt((wnobs - 1) / nobs)`)

user: josef-pkt
issue_2848: state: open
title: SUMM: weights - which kinds
body: Stata has now 4 kinds of weights fweights, aweights, pweights and iweights. Probability weights, pweights, have been added after Stata 12 (which was my first Stata). "U" manual has useful information.

We have `weights` and `freq_weights`
My prefered terminology is to be more explicit than Stata, e.g. freq_weights, var_weights and prob_weights.  I don't know about what to do with importance weights iweights yet. Maybe for outliers and robust estimation.

We don't impose some of the restriction on weights as Stata (eg. non-integer freq_weights) and we don't normalize var_weights in WLS like Stata does or did for aweights.
## current status
- WLS/GLS: uses `weights` for `var_weights`
- GEE uses `weights` for `prob_weights` according to docstring
- GLM will get `freq_weights` in #2835 
- stats weightstats uses `weights` for `freq_weights`
## Implication of weights in models and results

For estimation of the parameters most of the weights can be combined. There are some differences for estimating cov_params, and especially robust cov_params.
- `freq_weights`
  - they produce identical results as repeating the observations (if freq_weights are integers).
  - unclear for now: residuals for diagnostics and impact on leave one observation out statistics.
- `var_weights`
  - ...
- `prob_weights`
  - useful for treatment effects estimation and survey adjustments, and correcting for missing at random, MAR, observations
- `importance_weights`
  - ??? 
  - RLM? robust GLM?
## Implementation
- we shouldn't have too many redundant calculations if there are no weights
- adjusting all the postestimation features and options will require work. Add warnings for unchecked things.
- figure out where or when we can combine different weights, and what we need to keep track of when the implications and calculations differ across weights.
## issues and PRs

see also issue label `topic-weights`
- #2840 GLM/LEF weights and aggregation
- #2646 weights in GMM
- #2701 SUMM for several applications of weights 
- ...

user: josef-pkt
issue_2847: state: open
title: ENH: one observation t-test against reference sample
body: http://stackoverflow.com/questions/35788140/scipy-stats-ttest-1samp-hypothesis-testing-for-comparing-previous-performance-to

two sample t-test where one sample has only one point, i.e. is one additional point consistent with the previous sample.

I read some articles in medical statistics about this. I don't remember anything.
I think the point was not to include the new observation in calculating the variance and mean.

(sounds similar to outlier test)

would be good to add, or have a ready example if something else can be reused for it.

user: josef-pkt
issue_2843: state: open
title: How to use mutli-column functions in Patsy which return categorical output?
body: I am dealing with a situation where each item can have 1 or 2 labels both of which come from the same set. In this case I only want to have 1 set of categorical variables which encode if the values of each level are present or absent with respect to the reference. I also want to inversely weight the values of these categorical variables based on how many labels are present. 
E.g. Let my categories be `{A, B, C}`

``` python
data = pd.DataFrame({"col1": ["A", "A", "B"], "col2": ["O", "B", "C"], "num_vals": [1,2,2]})

def inverse_val(x):
    return 1.0/x

# Using categorical coding will not give the correct output
X = patsy.dmatrix("(C(col1) + C(col2)):inverse_val(num_vals)", data, return_type="dataframe")
```

I tried to solve this issue using the following code:

``` python
def multival(*x, **kwargs):
  #raise Exception("Not Implemented")
  levels, reference = kwargs.get("levels", None), kwargs.get("reference", None)
  weights = kwargs.get("weights", None)
  if len(x[0].shape) != 1:
    raise Exception("Mismatching Shapes. All arrays should be 1d and should have the same shape")
  for k in x:
    if k.shape != x[0].shape:
      raise Exception("Mismatching Shapes. All arrays should be 1d and should have the same shape")
  if levels is None:
    levels = np.sort(np.unique(np.hstack(x))) # Sort the unique values and then use this ordering as levels
  if reference is None:
    reference = levels[0]
  #print "Levels: %s, reference: %s" % (levels, reference)
  levels = levels[levels != reference] # Remove reference from levels
  level_len = len(levels)
  #print x[0].shape[0], level_len
  out = np.zeros((x[0].shape[0], level_len))
  for i, v in enumerate(levels):
    # print i, v
    for col in x:
      out[np.where(np.array(col) == v), i] = 1
  #print "Created matrix with shape: ", out.shape
  colnames = ["T.%s" % k for k in levels]
  if weights is not None:
    weights = weights.values
    return pd.DataFrame(out, columns=colnames).divide(weights, axis=0)

  return pd.DataFrame(out, columns=colnames)

X = patsy.dmatrix("multival(col1, col2, weights=num_vals)", data, return_type="dataframe")
```

There is no way of identifying the column names generated by patsy:

E.g. The names are like this:

``` python
multival(col1, col2, weights=num_vals, reference='O')[0]
multival(col1, col2, weights=num_vals, reference='O')[1]
multival(col1, col2, weights=num_vals, reference='O')[2]
```

I would much rather prefer column names like the following:

``` python
multival(col1, col2, weights=num_vals, reference='O')[T.0]
multival(col1, col2, weights=num_vals, reference='O')[T.1]
multival(col1, col2, weights=num_vals, reference='O')[T.2]
```

OR if I am passing the levels variable:

``` python
multival(col1, col2, weights=num_vals, reference='O')[T.A]
multival(col1, col2, weights=num_vals, reference='O')[T.B]
multival(col1, col2, weights=num_vals, reference='O')[T.C]
```

Is there a way to achieve this in patsy ?

user: napsternxg
issue_2842: state: open
title: TST/Performance cluster robust in almost cross-section
body: follow-up question to #2840 for a specific usecase

Suppose we have a cross-section where a few units/individuals have repeat observations. 
Does the current cluster robust covariances work well and correctly in this case?

I guess it should work correctly, but will be slow.
Cluster of size 1 should work if the array shapes stay correct, but I don't think there are unit tests.
If we loop over all clusters/groups, then this can be a large number in almost cross-section, that doesn't take advantage that most are single element groups.

TODO: check this and possibly add unit tests

user: josef-pkt
issue_2841: state: open
title: Building design matrices - Best practice
body: Sirs,

So I've been experimenting with the freq_weights fork that we've put together. Overall, I've very pleased with the speed. I threw about 3 million records with 120 parameters at statsmodels. The fit times for a poisson frequency and a poisson loss cost model was about 3-5 minutes. I limited the number of iterations to 10. This is on anaconda 3.5 with the mkl driver. 

However, I've come to thinking that I need a better way to build design matrices. What I'm doing now is painfully slow. 

A few questions...
- Do you like patsy? Initially I was playing with it but I've come across a few things I don't like. 
- There is no support for multithreading. (As far as I can see) but coming up with the exog datasets seems embarassingly parallel.
- I'm not sure if writing a formula is really the most efficient way to produce design matrices. Its easy for a first try, but it seems that you're limited to what patsy is designed to do. If you want a more complex grouping or a piecewise function your model, you essentially have to write it yourself as far as I can tell. 
- My data live in big Microsoft SQL databases.. they can be hundreds of gigs in size. I obviously don't want to load a 100GB SQL table into memory, so how would you handle this? write helper functions that download only a few columns at a time and then build the design matrix, and then fit? This feels unpleasant to me... lots of steps that aren't really doing anything. 
- So this is a long point, but I'll try to make as succinct as I can. For log-link models (which are inherently multiplicative) I need to create relativities for implementation. We often use splines and n-degree polynomials to model effects. For example, I might have separate columns in my design matrix which are Driver Age, Driver Age *\* 2, Driver Age *\* 3, etc. When I give the model to my IT department, they really want to see factor levels and relativities. So  with driver age, what I need is to create a design matrix with 1 record for each unique possible driver age (maybe 16 to 100) and the rest of the columns at the null/base value. Then I can build predictions for each of these rows and then divide the predicted value at each driver age by say the predicted value for a 50 year old. I think this is very similar to what is meant in [this](https://github.com/pydata/patsy/issues/64) patsy issue. How do you guys handle this? 

user: thequackdaddy
issue_2840: state: open
title: weights: repeat versus mean/variance in LEF family
body: (issue with just some notes, see weights isssues: ....)

frequency weights version variance weights.
GLM in R, and aweights in Stata have the interpretation that the observation is the mean of a group of samples with the same exog.

```
Property 11.3.7 (Taking sample means)
Let Y1,...,Ym be a sample of m independent copies of the random variable Y, and
let Y=(Y1+···+Ym)/m be the sample mean. If Y is a member of an exponential dispersion 
subfamily with fixed functions b(·) and c(·;·) and with parameters θ and ψ, then Y is in the 
same exponential dispersion subfamily with parameters θ and ψ/m, if this pair of 
parameters is allowed.
```

page 303: "Modern Actuarial Risk Theory - Using R" Springer

Similar results apply for Poisson, similar for Binomial but that has a limitation.

---

Difference in use case, but can be largely equivalent

frequency weights: this is just to save memory and computation where we combine observations with identical exog **and** identical endog.
This does not impose any assumption, as long as we combine "full" observations, e.g. identical clusters or panel units if there is within correlation.

var_weights for mean: we are combining observations with identical exog, but with non-identical endog, under some assumptions on the individual realisations. Standard assumption is i.i.d. And based on above we stay in the same exponential subfamily.
`weights` in this case are just given by how many i.i.d. observations are used in calculating the mean.

AFAICS, mean is just a special case, another version would be aggregating the samples as in Poisson or Binomial with person-years.
The main application is for life-tables or similar tables that publish only average or aggregate results for prespecified "strata", i.e. groups of observations with identical exog.

Given the strong assumptions in this case, we can get different results if we don't impose these assumptions, e.g. by using robust covariance matrix.

---

related to #2835 and #2831 about cov_type HC with exposure versus freq_weights.

user: josef-pkt
issue_2838: state: open
title: adding restrictions on parameters in VAR
body: is there a possibility to add restrictions on the VAR coefficients?

**edit**
The question showed up again on the mailing list. And another related large VAR question is somewhere in issue comments. (don't find it right now)

I opened a new issue related to this for large sparse VAR, which is more complicated than just adding some restrictions because it needs to make efficient use of memory. #2933

user: ayat-rashad
issue_2831: state: open
title: FAQ cov_type HC with Poisson and exposure and weights
body: **edit** this agrees with Stata
I had already the Stata results included, just hadn't used it for a test.
bse agrees with Stata in the test case at rtol=1e-12, except for the `nobs / (nobs - 1)` factor.

I don't understand the results
see first comment (which I will remove later)

cov_type=HC has standard errors that are 10 times larger than nonrobust

GLM Poisson and discrete Poisson produce the same results

user: josef-pkt
issue_2829: state: open
title: REF/ENH: extend cov_type to work with weights
body: (starting a separate issue for this)
#2805 implements frequency weights for GLM, but it doesn't work for robust covariances

one issue is restrictive assumption for nobs, k_params, see #2827 but doesn't apply for HC

_debugging HC0 for GLM Poisson:_
using `_get_sandwich_arrays(results)`
hessian looks correct, relative difference approximately 0.002 (numerical noise ?)
xu, i.e. scoreobs looks also fine, one is the repeat (group sum) version of the other in replicated sample.

but xu.T @ xu differs by a large amount 

Thinking incorrectly:
xu has the weights. so xu **2 has the weights in squares.
we need sqrt(weights)  -> check weighted covariance in stats
WLS.whiten uses sqrt(weights), so xu = wexog \* wresid has weights not square weights.
(but score_obs uses exog and also just has weights ???)
time for some math

in any case: if I divide xu by sqrt(weights) (or sqrt(exposure)), then the two versions agree at 3e-11

We run again into the fundamental difference across different definitions of weights.
freq_weights are additive in mean and variance.
weighted average (linear combination of random variables) has weights square in covariance calculations compared to mean calculations.
(same problem and "thinko" I ran into in weight_stats)

user: josef-pkt
issue_2828: state: open
title: BUG/ENH: generic cov_type white_simple uses hardcoded use_correction
body: related to missing HCx for x > 0

`use_correction=False` is used in call to cov_white_simple, i.e. HC0 with no options

this is the generic execution path for models except linear models OLS/WLS

user: josef-pkt
issue_2827: state: open
title: REF/ENH: cov_type assumes nobs k_params from score_obs
body: The small sample correction define nobs and k_params from the shape of xu

```
if use_correction:
    nobs, k_params = xu.shape
    ....
```

this doesn't generalize, work correctly for ddof (absorbed params, for params that are not in score (*)) and not for frequency weights.

(*) how would this handle an overidentified case in GMM (GMM still has it's own sandwich code)

looked at because of #2805 freq_weights in GLM: `cov_type = 'HC0'` doesn't work in the current setting. However, the bse are not proportionally wrong, so it's not the small sample correction here.
actually: currently `use_correction` is always false for HC in GLM see #2828

user: josef-pkt
issue_2826: state: open
title: Performance: GLM
body: see https://github.com/statsmodels/statsmodels/pull/2805#issuecomment-185919194
for a GLM-Poisson example (numbers before bugfix of max_start_irls)

The time seems to be spent mainly in the Hessian calculation which takes 6.4 seconds in the example, total fit time with max_start_irls=3 and 3 newton iterations is 25 seconds.
IRLS with the WLS cov_params takes only 1.2 seconds.
with max_start_irls=5, newton takes only 13 seconds and 1 iteration.

link.deriv2 still uses numerical derivatives (fix in glmnet/penalized PR IIRC)

calculating hessian_factor is not optimized for special cases, e.g. for canonical link we could skip the second part (oim = eim, numerically difference is around 8th significant digit, eyeballing in example)
calculating hessian with `observed=False` takes 0.003 seconds, in contrast to oim 6.5 s

`method='bfgs', skip_hessian=True` raises an exception when normalized_cov_params is None in a calculation.

I guess we should get at least OPG or EIM as option for the optimization.
outer product of gradient `opg` should essentially be identical to irls.
expected information matrix, eim is already available but not as option to optimizer.

user: josef-pkt
issue_2825: state: open
title: SUMM/Design: consistency in convergence criteria
body: based on PR #2803  GLM uses absolute tolerance on deviance
see comments there for discussion

current behavior 

model specific estimators:
- GLM: atol on deviance
- RLM is similar and the convergence criterion is on the value of the objective function, but it has an option for choosing it `conv='dev'`.
  (RLM was largely modeled the same way as GLM)
- GLSAR uses max rtol for params
- MixedLM uses max atol for fe params
- Quantile regression uses max atol for params
- (GLSHet uses fixed number of iterations, open issue IIRC)

generic optimizers:
- our `newton` method: docstring says relative, but code is absolute max atol for params
- scipy optimizers are mixed according to our docstrings, rtol or tol on gradient, we use defaults, users can change

related aside: 
using only relative tolerance causes convergence problem in perfect fit. Our current GLM works fine with it, e.g. in a saturated loglinear model (Poisson or Binomial)
comment lists R, SAS and Stata
`https://github.com/JuliaStats/GLM.jl/issues/117#issuecomment-168544306`

user: josef-pkt
issue_2824: state: open
title: ENH: get design_info from exog if attached, design_info separate from formula interface
body: I just saw a usecase where we have exog in the model (non formula interface) that is the return of a call to patsy dmatrices. (current unit tests of #2805)

We could reuse this in the same way as in `from_formula`, e.g. use the names, allow transform in predict and so on.

separate, related issue: I was thinking for a while how we can use and manipulate the `design_info` without having to go through the formula treatment.

some desired methods would be `drop` and `subset` of variables, columns or terms and maintaining and reusing the related formula information. 
`patsy.dmatrices` return `DesignMatrix` which is a subclass of np.ndarray, but returns ndarrays on slicing or other operations. However, it looks like np.repeat keeps the DesignMatrix class.

user: josef-pkt
issue_2822: state: open
title: DOC: Build process followup
body: Followup to https://github.com/statsmodels/statsmodels/pull/2816
- [x] include `@savefig` decorators on all the example plots
- [x] figure out why `duration.rst` breaks the execution on `index.rst` (and possibly elsewhere)
- [x] integrate into Travis to auto-biuld
- [x] cache the notebook execution step
- [x] integrate statsmodels styling / template into HTML notebooks
- [x] verify the output, make sure all the examples are executed and work
- [x] fixup any compatibility things (including `make.bat`!)
- [ ] Pull in wiki examples?
- [ ] partial builds similar to pandas `make.py --single`

user: TomAugspurger
issue_2815: state: open
title: ENH: Add self exciting threshold autoregression (SETAR) model
body: This is the code from three years ago, see #958 for more details.

The main issue to fix here is unit tests, which do not cover everything in the model.

user: ChadFulton
issue_2812: state: open
title: BUG/ENH: 2760 simulation start time
body: Fix #2760 
Draft commit. Only simulate function of Kalman Filter modified by adding a start_time parameter.

user: ngaurav
issue_2811: state: open
title: ENH/REF/Review: PHReg has cluster robust cov_type
body: uses `model.groups is not None` as indicator if cov_type='cluster'
uses nobs loop to aggregate cov

Do we need to standardize, generalize add small sample (small number of cluster) correction, or is it fine this way?

(I just saw this by chance, and don't know details)

user: josef-pkt
issue_2804: state: open
title: SUMM: GLM feature wishlist
body: Hello, 

So long story short, I'm an actuary looking to do some GLM modeling in python. I have some experience with R, but am open to new things. I've been watching this project for a bit but have been too lazy/impatient to get some features implemented. 

So here's a list of features in the GLM that I'm considering putting some PR's together for. I appreciate any advice you can give about statsmodels philosophy before I write too much and its all rejected. I don't want to waste your time with my PR's, so the more advice the better. 
1. QuasiPoisson. This is a TODO in the GLM family module. I don't think its that hard. My experimenting seems to show that literally copying Poisson is most of the work.
2.  The Pearson residual for Poisson/QuasiBinomial
3.  User-specified weights
4.  Tweedie

I'd appreciate your input before I start this. Note, I'm an actuary, not a programmer, so I'd need some handholding on some things.

Thanks.

user: thequackdaddy
issue_2802: state: open
title: ENH: combining hypothesis tests, pvalues or zvalues
body: (I thought I opened an issue once, but don't find any

related also to Metaanalysis e.g. #1575
scipy.stats has two methods to combine zvalues (Fisher combination of p-values)
partially related: estimating FDR for large number of hypotheses #2297 or multiple testing p-value correction

I ran now into this for the first time in the econometrics literature in the context of panel data unit root tests
(I haven't read anything except for a few abstracts, and don't know any details)

one early reference commonly cited is 
Choi, In. 2001. “Unit Root Tests for Panel Data.” Journal of International Money and Finance 20 (2): 249–72. doi:10.1016/S0261-5606(00)00048-6.

Demetrescu, M., Hassler, U. and Tarcolea, A.-I. (2006), Combining Significance of Correlated Statistics with Application to Panel Data. Oxford Bulletin of Economics and Statistics, 68: 647–663. doi: 10.1111/j.1468-0084.2006.00181.x

refers to this as basis
Hartung, J. (1999), A Note on Combining Dependent Tests of Significance. Biom. J., 41: 849–855. doi: 10.1002/(SICI)1521-4036(199911)41:7849::AID-BIMJ8493.0.CO;2-T

partially found via 
Costantini, M. and Lupi, C. (2013), A Simple Panel-CADF Test for Unit Roots. Oxford Bulletin of Economics and Statistics, 75: 276–296. doi: 10.1111/j.1468-0084.2012.00690.x
and the corresponding R package for CADF  (punitroots seems to be abandoned) 
https://cran.r-project.org/web/packages/CADFtest by Lupi

This looks also interesting because, based on the abstract and list of referenced or cited literature, we get a connection of statistics with large number of tests (MRI, genetics) with panel data econometrics, and spatial econometrics should be similar. 

user: josef-pkt
issue_2799: state: open
title: ddof for models
body: This is for a generic argument to adjust the df_resid and df_model

The main use case is ddof for absorbed variables, e.g. demeaned exog and endog. 
See #2568 and #2774

Absorbing only works for linear model, but we might be doing something similar for discrete, GLM and other nonlinear/nongaussian models. E.g. absorb into `offset` and adjust df_resid. 
However, in those cases the default is `use_t=False` which doesn't depend on df_resid, but it's used in some small sample corrections.

For interpretation and example for OLS see #2774, with possibility for unit tests against Stata's `areg` to check definition of extras. The main comparison I used is OLS with dummies versus OLS absorbed with ddof.

This can be a separate PR from #2774

see #285 for connection to cov_type, cov_params

user: josef-pkt
issue_2791: state: open
title: ENH: predict, get_prediction with log or transformed endog
body: How do we get prediction and similar for models with transformed endog?
example `log(y) ~ ...` estimated with OLS

related to issue #1449    #1480
also related: use Poisson instead, several articles and FAQ
also GLM-Gamma is close to lognormal model

specific point: correcting the prediction of the original so we get E(y | x)  or E(y | ?) for population.
We have an explicit expression for the correction under lognormality assumption.
in general we can use a _smearing_ estimate

(I went through this some time ago, but don't know where the functions are. I don't see a specific issue. I just bumped into this blog again.)
see for example http://davegiles.blogspot.ca/2014/12/s.html
In equation (3) he uses the average exp residuals.
AFAIR, I averaged over sample x_i, which might (!) be somewhere in Wooldridge.

It's not clear yet how to connect this to the models (especially we don't know which transformation the user has used), but we can provide at least some helper function.
It fits most likely in the topic of various predict functions and methods. (PR #2296 )

user: josef-pkt
issue_2790: state: open
title: Added Phillips-Perron Test
body: Added an implementation of Phillips-Perron test to tsa.stattools .
@josef-pkt please take a look .

user: dhpiyush
issue_2788: state: open
title: Using trained (AR-XYZ) models with new data
body: Hello I do timeseries forecasting with TSA, AR, ARIMA, ARMA. What I would like to be doing is, training the models with a subset of the data I have to get the weights and then try the models on new unseen data. I could not figure out a way to do this in a clean way, since the training data + fitting results seem to be calculated in the fit() function and directly used in the predict() function.

Could you give me a hint how to do this or where to find this in the documentation? Maybe there is a way replacing the internal endogen values and reacalculate using the old weights?

Thanks
David

user: Acertainuser
issue_2784: state: open
title: SUMM/REF: grouputils
body: we need an inventory and current status of various panel data helper functions

I just saw that the pandas version Grouping in statsmodels.tools.grouputils already has the sparse_dummy function as a method. It uses the original version, I added it again in PR #2783 with extra dtype option.
- #2783 multiway demeaning
- issue groups versus multi-index: Bug/limitaion in current #2783 where the implementation assumes that two-way groups uniquely specify all obserations like in a multi-index. This is in general not required for groups. (Groups uniquely identify an observation with individual-time two way effects, wide format is 2d for each variable with individual and time identifiers as rows and columns.)
- #1133 panel data PR uses pandas internally, but doesn't have multiway fixed effects, AFAIR

user: josef-pkt
issue_2783: state: open
title: ENH: helper function: pandas multi-way group demean
body: parking a recipe for pandas group_demean based on answer by Jeff Reback on pydata mailing list

PR #2774  has the iteration loop for multi-way group-demeaning based on numpy ndim arrays. panel data in the PR will also be the main usage.

```
np.random.seed(1234)
df = DataFrame({'A' : np.random.randint(0,10,size=100), 'B' : np.random.randn(100), 'D' : np.random.randn(100)})
df[['B', 'D']] -= df.groupby('A')[['B', 'D']].transform('mean')
df[df.A==3].B.mean()
```

user: josef-pkt
issue_2782: state: open
title: EeNH: fisher exact with confidence interval
body: Just saw this by cChance
https://github.com/JuliaStats/HypothesisTests.jl/blob/master/src/fisher.jl

user: josef-pkt
issue_2778: state: open
title: ENH: Follow-up Fixed Effects estimators, identification, time-invariant effects
body: D. F. McCaffrey, J. R. Lockwood, K. Mihaly, and T. R. Sass "A review of Stata commands for fixed-effects estimation in normal linear models"

They comment that it is possible and some packages do estimate time invariant effects even when individual effects are absorbed.
In general, absorbing time constant individual effect also absorbs all other time constant effects.
It seems possible to separate these effects if we have groups of appropriately identical individuals.
(I didn't try to figure out the details.)

In general: With arbitrary crossed factors, two- or multiway fixed effects it is easy to run into identification problems, e.g. when we have too few observations in some combinations.
This won't show up much in almost balanced data like in finance or macro-economics. 
It is mentioned in approaches for matched microdata like worker-firms or student-teacher studies.

This will need to be handled "eventually" or maybe it drops out with some of the generic colinearity handling (e.g. related case: interaction effect with categoricals in empty cells which are currently just zero by pinv or default linear algebra).

Most likely the first step is to use sparse Ridge with tiny penalization to get approximately the equivalent of a pinv solution, i.e. we get a solution but not all coefficients are identified by the data (estimable).

user: josef-pkt
issue_2777: state: open
title: ENH: link between treatment effects and Fixed Effects estimators
body: Kosuke Imai, In Song Kim   "On the Use of Linear Fixed Effects Regression Estimators for Causal Inference" http://imai.princeton.edu/research/FEmatch.html  
with associated R package https://cran.r-project.org/web/packages/wfe/index.html

shows algebraic equivalence between some matching, IPW, repeated measures treatment effects estimators and fixed effects estimators with binary treatment.

I haven't read it yet.
This should be interesting when we get treatment effects and Panel/FE merged.

(This might also be useful for the link of FE estimators and repeated measures, within ANOVA.)

user: josef-pkt
issue_2774: state: open
title: ENH: large categorical, OLSAbsorb with sparse projection
body: see #2568 for some design discussion, and references to different algorithms

We are partialing out fixed effects in panel data, or any categorical factor variable with many levels.

This includes currently only a sparse version for general multi-way factors. 

Warning: some results in returned OLS are still wrong (fvalue, llf, ...)
see docstrings and comments for other current definitions and limitations.

currently missing: function that creates sparse dummy representation uses currently a dense intermediate array

see also #2711 for OLSSparse, is currently SparseOLS

module names and paths are not settled yet. 
The projection class needs to be public.

---

Design issues
- exog, wexog: currently I have the transformed exog in wexog using whiten
  However, several things use the original exog. 
  - OLS.loglike doesn't use wexog/wresid, consequence of aic, bic
  - fittedvalues and resid are for original data in model, i.e. exog, endog (ignore absorbed effects)
  - wresid, scale, ssr, ess, mse_resid, rsquared, rsquared_adj
    are fine, those use wexog, transformed in current case
  - bse, wald inference, t_test, pvalues are fine, (except for parameter slicing and constant definition)
- df_model:
  - > still need to override
  - mse_model, fvalue (nonrobust) is incorrect without adjusting df_model
- predict, fittedvalues, residuals
  The WLS version matches almost everything with the main exception in predict fittedvalues and resid. Those are designed for the values on the original scale and variable without taking the predictive effect of fixed effects into account. For predict we would have to solve for the fixed effects first before we can do any prediction.
  see comment below https://github.com/statsmodels/statsmodels/pull/2774#issuecomment-173362025
  - fittedvalues, residuals, override and retro-fit, because absorbed wresid is ols resid:
  
  ```
  >>> np.max(np.abs(res_ols.fittedvalues - (res_absorb.model.endog - res_absorb.wresid)))
  2.6645352591003757e-15
  >>> np.max(np.abs(res_ols.resid - res_absorb.wresid))
  2.6645352591003757e-15
  ```
  - predict: we need an option, doing full prediction is more expensive (not much more expensive after we have the fixed effects params, and we can store those)
- inference, standard errors for fixed effects
  I don't see a way to calculate those without creating the full sparse inv(x'x) where we can restrict x to the fixed effects. I think inv(x'x) might not be very sparse for a general case. We could store only the diagonal if we have memory problems, but we still need to calculate for example LU xtx.solve(np.eye) in chunks of np.eye columns. Can the argument for LU solve be a dense low memory dtype (e.g. bool)?
  Orme/Yamagata. 2014: robust F-test for joint hypothesis that all individual effects are zero

---

applicability of different absorb methods
- demeaning in nd array form assumes unique multi index given by two-way effect (in my current version)
  looks similar to this http://stats.stackexchange.com/questions/124662/group-fixed-effects-not-individual-fixed-effects-using-plm-in-r?rq=1

---

References

Orme, Chris D., and Takashi Yamagata. 2014. “A Heteroskedasticity-Robust F-Test Statistic for Individual Effects.” Econometric Reviews 33 (5-6): 431–71. doi:10.1080/07474938.2013.824792.

Related
- transform models: this is similar to using an internal transformation of the model
- ddof for OLS, for OLS with absorbing by users  

user: josef-pkt
issue_2773: state: open
title: ENH: Statespace: In-sample prediction using smoothed or filtered states.
body: Currently the `predict` command returns one-step-ahead predictions for in-sample non-dynamic predictions. Specifically this means that the prediction for time t is based on the estimated _states_ at time t-1 ("predicted" states).

In some cases we may want predictions for time t based on the estimated states at time t ("filtered" states) or estimated states using data from the full sample ("smoothed" states). In particular, if there is missing data in `endog`, it is best estimated using the smoothed states.

This has no effect on forecasting or dynamic prediction, since the predicted / filtered / smoothed states only differ when new data is introduced.

user: ChadFulton
issue_2772: state: open
title: ENH: exog_other or exog_transform in models
body: (just another semi-random idea for a general API)

gam/penalized splines, polynomials, segmented regression, ... are not supported by patsy.
We don't get access to the stateful transform.

One possibility to include those is to use `exog_other = [list of instances]` to combine standard linear terms with more complex terms, where instance is one of the supported cases.
This would mean that we implement something similar to patsy's stateful transformations but restricted to a few cases where we need a special fit method. 
Patsy's stateful transform are only relevant to create the design matrix but do not affect the estimation method, e.g. standard unpenalized splines. In contrast, we need to know additional properties of the transformation, penalization, knot search or similar.

This has the advantage that we can implement specific fit and predict methods.

(OLSAbsorb is a bit similar. We can either just take the absorbed exog and ddof and calculate the rest, or we can absorb inside the model, then we can also get additional information about the absorbed categoricals. A difference is that aborb is more like weights, it affects all exog, and can be added as a simple argument containing the factors.)

user: josef-pkt
issue_2771: state: open
title: ENH: create sparse dummy matrix
body: Useful for sparse regression with fixed effects or categorical variables.

see 
#2710 sparse support in models
#2568 absorbing fixed effects

opening to park my draft version, that I had misplaced, discussion scipy-user around nov 30, 2011
see first comment.
I don't find the script for sparse cluster robust sandwich covariance.

user: josef-pkt
issue_2765: state: open
title: SUMM/ENH: hypothesis tests for covariance/correlation matrices
body: another topic with pieces all over the place

related: test for one or two correlations, see issue #667 (partially a duplicate)

Here it is tests on the entire correlation/covariance matrix
- sphericity, corr = identity: e.g. used in repeated measures and panel data, related #2756
- rank, largest or smallest eigenvalue: cancor, multivariate, Johansen cointegration, weak instrument tests
  - #1111 cancor
  - ... ?

References

Ledoit, Olivier, and Michael Wolf. "Some hypothesis tests for the covariance matrix when the dimension is large compared to the sample size." Annals of Statistics (2002): 1081-1102.
improved performance and asymptotics when data has large `k_vars` relative to `nobs`

user: josef-pkt
issue_2764: state: open
title: SUMM/ENH: grouped time survival
body: standard Cox PHReg works in continuous time but with the possibility of ties.
The alternative data structure or assumption is that we observe the data only at fixed time periods or over fixed intervals. We also get this structure if we bin time into discrete intervals. 

The main task is to get the data preparation functions. 
After that large parts of the analysis can be done with GLM, Logit/Binomial #2763 or Poisson #1764 
Related: It's possible to model the baseline hazard with splines (Stata user contributed which? Ref?)

One difference is whether we have
- individual data where we can use for example cluster robust standard errors for individual unobserved heterogeneity (what Stata does), or
- summary data as in life tables

see #1764 Poisson for some discussion

see also  #2386 Conditional Logit for tie handling 

user: josef-pkt
issue_2763: state: open
title: ENH/FAQ: grouped Cox PHReg and Logit/Binomial
body: (mainly parking a reference)

D'Agostino, Ralph B., Mei‐Ling Lee, Albert J. Belanger, L. Adrienne Cupples, Keaven Anderson, and William B. Kannel. "Relation of pooled logistic regression to time dependent Cox regression analysis: the Framingham Heart Study." Statistics in medicine 9, no. 12 (1990): 1501-1515.

the main interesting part is the appendix

this is similar to #1764  Cox as Poisson model

My impression is that the main difference is whether we use log or logit link, but I didn't try to understand all the math/derivation. Cox and pooled Logit are essentially the same if the length of the time intervals and the probabilities go to zero. (D'Agostino compare risk ratio with odds ratio which are very close in the two models.)

One difference of Logit to count models like Binomial or Poisson is whether we have individual data or "strata", i.e. summary statistics of individuals with identical explanatory variables.

The main work is in the data handling as in the Poisson equivalent of Cox PHReg.

see also #2386 Conditional Logit to handle ties.

user: josef-pkt
issue_2762: state: open
title: TukeyHSD: No method to get group1 and group2 to match res.std_pairs
body: I wanted to write  a function to export the TukeyHSDResult to a CSV file, however there doesn't seem to be any rhyme or reason to the output from res.groups...

Using res.groupsunique I can calculate the combinations possible (e.g. for 6 groups I know there should be 15 unique combinations which corresponds to the len(res.std_pairs) as well as the summary output) but when I call res.groups I get a list of 52 strings, with each group label repeated several times. I would understand if it was double in length of the combinations, or even double in length plus the redundant A --> A (or A <-- A) comparisons, but even then I would expect the length of the list to be 42, not 52

What logic is producing the strange behavior of res.groups? Has anyone already come up with a function which I can use to produce a proper length list, e.g. of tuples like (group1_label, group2_label) ?

user: trevstanhope
issue_2758: state: open
title: SUMM: Fixed Effects in panel data
body: This is for the case when the parameter space grows with increasing sample size, i.e. the incidental parameter problem.  I don't have an overview of the theory (when does it matter, when not).

related: 
#2568 (absorb FE)
#2745 (move pandas stats models)
#2756 diagnostic tests, for various cases, including FE

some references that I just found

Stock, James H., and Mark W. Watson. "Heteroskedasticity‐robust standard errors for fixed effects panel data regression." Econometrica 76.1 (2008): 155-174.

This seems to differ from cluster robust standard errors because of incidental parameter. 
Cluster robust sandwich allow arbitrary correlation for the covariance, but assume finite parameter space in the mean function, AFAICS.

I added two links to absorb issue for computational approach
Note: #2568 absorbing FE is a purely computational issue. However, we have the related issue for conditioning and removal of FE from the analysis, e.g. whitening, demeaning, differencing, ...

user: josef-pkt
issue_2756: state: open
title: ENH: diagnostic tests for panel data (LM test for autocorrelation, heteroscedasticity)
body: add diagnostic tests for panel data - LM test for serail correlation, heteroscedasticity, cross-sectional correlation and similar.

I ran into breusch-pagan test for panel data. The tests have a similar structure as the ones for OLS, but go in more directions and have to watch out for incidental parameter problem when removing fixed effects (one way or two way).

some links
http://www.princeton.edu/~otorres/Panel101.pdf  Panel Data Analysis Fixed and Random Effects using Stata (v. 4.2): overview of Stata in 2007, second part has diagnostic tests, several by users.
http://www.stata-journal.com/sjpdf.html?articlenum=st0039 "Testing for serial correlation in linear panel-data models"  Stata buitin xtserial, implements Wooldridge 2002
http://www.stata-journal.com/sjpdf.html?articlenum=st0004 "Residual diagnostics for cross-section time series regression models"  Baum, available at ssc, after one-way fixed effects regression

recent paper for heteroscedasticity
Juhl, Ted, and Walter Sosa-Escudero. "Testing for heteroskedasticity in fixed effects models." Journal of Econometrics 178 (2014): 484-494.
(no citation in Google, but "standard" implementation for robust tests Davidson MacKinnon and Wooldridge artificial regression, looks good.)

Some recent papers also look at two-way fixed effect (individual and time FE)

Baltagi, Baltagi and Li have several papers but I never got far in reading those.

---

partially related to FE modelling in panel data or more generally, 
e.g. #2568 (absorb FE) and #2745 (move pandas stats models)
and conditional estimation Chamberlain style remove FE or RE.

user: josef-pkt
issue_2753: state: open
title: SUMM:  GLM versus R glm: what's missing
body: (It's the first time I go over the R glm documentation, and understand enough.)

We are close to having essentially all the features that are in R glm. A few things are missing:

several extra functions in R have methods that work with glm results objects.
- anova:  anova (3 versions stats, car and MASS) work also for glm and either do Wald or LR tests. stats anova also has `rao` score test. 
  status: type 3 wald is available, compare_xxx is not, score_test is not advertised and still experimental
  my loglinear notebook has examples but no helper functions, 
- diagnostics: lmtest has several tests that work with glm objects. I didn't see any specialized tests yet, like Hosmer-Lemeshow gof
  http://stat.ethz.ch/R-manual/R-patched/library/stats/html/influence.measures.html also handles glm
  status: we have residuals and all/most basics but no unit tested diagnostic functions (some might work already, I haven't checked). We are not yet holding on to the last irls-WLS results for diagnostics
- quasi: dispersion and "foreign" variance functions
  status: we have sandwiches and some things work or may work by "misusing" current implementation, but no official API nor unit tests
  see #2889 for general dispersion modelling e.g. double exponential 
- more families, links: we have what's in R stats, some difficult ones in packages like Tweedie are missing. we have cloglog but not loglog (which R misnames cloglog)
- weights: R glm has variance weights, but I didn't see frequency or case weights
- effects/margins: I haven't checked yet what R has available. our margins follow Stata
- ... ?

Todo: add links to open issues
- tool/utility functions: 
  - I'm missing some helper functions for looking at dispersion, e.g. calculate negativebinomial coefficient from poisson or negbin residuals.
  - parameter conversion functions, e.g. to scipy distributions (for simulation)

user: josef-pkt
issue_2743: state: open
title: ENH: hypothesis testing for proportions matched case-control
body: This is a followup for various proportion test, confidence intervals, power, e.g. #2605

In paired samples we have 1 to 1 matching between two groups of individuals (e.g. 1 treatment - 1 control). The standard test is McNemar for this. (We have some extension in contingency tables where I haven't checked in which way this extends from 2x2 tables.)

The extension is 1:M matching, where one treatment individual is matched with M >= 1 control individuals. This looks also similar to the 1:M matching in propensity score matching.

The Model for this is ConditionalLogit (which in my current PR only allows 1 event per "group" or panel)
Stata has examples for different methods and power/sample size calculations in `power mcc` documentation.

(I have read almost nothing specifically about 1:M case-control studies, but looks useful.)

Note:

The setup is a bit different because in what's defined as "matched case control studies" uses to outcome to sample, e.g. https://onlinecourses.science.psu.edu/stat507/node/48 ("retrospective")
I ran into this somewhere before, maybe Agresti, and I don't know where or when this makes a difference.  The tests and calculation themselves are the same as a McNamara extension.

(In propensity score matching we match for the treatment and not the final outcome.)

user: josef-pkt
issue_2735: state: open
title: ENH: binom_test 2-sided is minlike, add centered
body: (I didn't realize this before, but I ran into problems explaining binom_test in a notebook.)

`binom_test` from scipy.stats is reused in statsmodels but it's not a centered test, it's minlike in Fay's terminology

from exactci documentation:

```
minlike: sum probabilities of all likelihoods equal or less than observed
central: double minimum one-sided p-value
blaker: combine smaller observed tail probability with opposite tail not greater than observed tail
```

The following function replicates the results from exactci "central"

```
def binom_test_centered(count, nobs, prop=0.5):
    """two-sided centered binomial test"""
    prob_low = stats.binom.cdf(count, nobs, p_null)
    prob_upp = stats.binom.sf(count - 1, nobs, p_null)
    return 2 * min(prob_low, prob_upp)
```

Note "minlike" looks more powerful, scipy binom_test matches R's binom.test (verified)
However we need central for consistency with one-sided tests and equivalence tests.

(Note Fisher's exact test as implemented is also minlike)

**follow-up**

there is currently a function `binom_test_reject_interval` which is for central binomial and not for minlike. This matches several functions in R, AFAIR, that also don't take into account that the standard binom_test is minlike. The notebook has plots for the difference in power between the two.
R package MESS looks like it has the correct power for minlike binomial test, `power.binom.test`.

user: josef-pkt
issue_2731: state: open
title: ENH/REF: sample size for proportions and other discrete
body: for proportions power is a sawtooth function of the sample size.
For sample size calculations we should calculate the smallest nobs such that the power for all large nobs is at least the required threshold.

I saw this in a repo (?) on github in python, and Julious and Campbell Tutuorial in Biostatistics statistics in Medicin 2012 check the next 10 nobs for this, and "criticize" PASS for providing the smallest nobs that goes above the power threshold, even when the power for the next nobs is again below the threshold.

AFAICS, there is not sample size calculation for binomial proportions available yet, although there are some power functions implemented.

user: josef-pkt
issue_2730: state: open
title: ENH: LM-test for heteroscedasticity in binary response models Logit, Probit
body: I'm not sure what the status here is, I just found an older script of mine with a function

```
def het_binary(results, exog_var, method='all'):
    '''Lagrange Multiplier (score) test for heterogeneity in binary models

    Test a homoscedastic model against a variance function for the latent
    variable of the form exp(Z gamma) where the `exog_var` Z are the
    explanatory variables for the heteroscedasticity.
```

which (unfortunately) has 4 or 5 different versions, and I don't know of any reference numbers.
(based on Greene, Wooldridge, and maybe Davidson-MacKinnon

http://www.statalist.org/forums/forum/general-stata-discussion/general/5564-heteroskedasticity-test-for-logit-and-logistic-models
http://www.stata.com/manuals13/rhetprobit.pdf

related to issues and PR on score/LM and conditional moment tests

---

two blog articles for R
http://www.r-bloggers.com/the-problem-with-testing-for-heteroskedasticity-in-probit-models/
https://diffuseprior.wordpress.com/2012/11/19/the-heteroskedastic-probit-model/

Shazam for LM test Davidson-MacKinnon  with Shazam output (useful for unit test)
http://www.econometrics.com/intro/logit3.htm
data: http://www.econometrics.com/intro/school.htm

another blog article in favor of LM test
http://davegiles.blogspot.ca/2011/05/gripe-of-day.html
but he has things mostly in "unreadable" Eviews workfiles http://web.uvic.ca/~dgiles/downloads/binary_choice/index.html

user: josef-pkt
issue_2729: state: open
title: ENH: rsquared for more models
body: Eviews shows Rsquared after ARMA  (question on mailing list)

UCLA stats has an rsquared function after RLM for Stata
http://www.ats.ucla.edu/stat/stata/faq/rregr2.htm

There is also an issue for Rsquared as evaluation/cross-validation measure

There is also a large list of Pseudo-Rsquared where we don't have much yet.

user: josef-pkt
issue_2726: state: open
title: WIP  ENH: add blaker proportion confint
body: initial version, still failures in root finding

This adds exact Blaker confidence intervals for a one sample Binomial proportion. It should be added as option to the existing function before merge.
Blaker confidence interval has smaller length than Clopper-Pearson and is contained in it. It doesn't have minimal length in contrast to other refinements of exact confidence intervals, but it produces nested confidence intervals (the confidence interval with a larger significance level is contained in the one with the smaller significance level), which is not the case for the other exact refinements.

The same approach also works for Poisson and other confidence intervals.

There are some R packages that implement it, probci and exactci. The second also for Poisson.

**Implementation notes**

Blaker does an inward grid search for the confidence limits. This can be slow because it needs a fine grid to avoid missing small non-monotonicities.
I'm using scipy.optimize.brentq which should be faster, but doesn't find the outermost limit (zero of the function).
There are two commented out test cases that fail and I didn't investigate yet. I thought we get the non-monotonicities only in exceptional cases.

Lecoutre and Poitevineau 2016 show that a different function is monotonic and has only one zero in the relevant range. I partially implemented the function for the upper confidence interval but not their approach, again using brentq instead of their several steps of finding the zero.

user: josef-pkt
issue_2725: state: open
title: paired t-test?
body: This is either a "implement such a test" or a "document it better":

I'm currently looking for the statsmodel equivalent of (from http://statistic-on-air.blogspot.de/2009/07/paired-students-t-test.html)

```
a = c(12.9, 13.5, 12.8, 15.6, 17.2, 19.2, 12.6, 15.3, 14.4, 11.3)
b = c(12.7, 13.6, 12.0, 15.2, 16.8, 20.0, 12.0, 15.9, 16.0, 11.1)

t.test(a,b, paired=TRUE)

    Paired t-test

data: a and b
t = -0.2133, df = 9, p-value = 0.8358
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
    -0.5802549 0.4802549
sample estimates:
mean of the differences
    -0.05
```

I googled for "statsmodels t-test paired" and only found a [ttost_paired](http://statsmodels.sourceforge.net/devel/generated/statsmodels.stats.weightstats.ttost_paired.html#statsmodels.stats.weightstats.ttost_paired). Is there simply no paired t-test or am I just not good enough in statistics to find the right test? In the latter case, for the sake of people like me: it would be nice if then the documentation could be adjusted :-)

user: jankatins
issue_2722: state: open
title: ENH: score confidence interval for GLM, discrete
body: score tests are doing quite well overall, score confidence intervals are emphasized in some of the two sample proportion comparisons and are doing also reasonably well for medium sample sizes.

Calculation: see references on two sample proportions.
Essentially we are calculating two one-sided score-tests and search for the Null hypothesis parameter.
In terms of implementation and effort required this should be similar to profile likelihood confidence intervals. An efficient implementation might be able to use the offset and doesn't need to change exog in each case, if we want the confidence interval for a single parameter. (I guess changing the constant for affine restrictions would work in the `constrained` transformed model, but requires adjusting the constraints transformation but not of the exog, AFAICguess.)

The advantage of score confidence intervals should be that we can use robust covariance matrices to handle overdispersion or heteroscedasticity or correlation. 
example http://www.inside-r.org/packages/cran/pairwiseCI/docs/pairwiseCImethodsCount
the mention dispersion problems with profile confidence interval that are handled for the specific proportional overdispersion case by QuasiPoisson (using deviance profile which I never heard but is just scaled loglikelihood, I guess).

user: josef-pkt
issue_2721: state: open
title: ENH: effect measures: NNT number needed to treat
body: see also #2215 for relative risk/risk ratio

NNT is the inverse of the risk difference (difference in proportion or rate)
confidence interval might have two segments going to +inf and -inf which can be displayed in inverse scale (i.e. relabel plot with confidence interval for risk difference)
see Altman, BMJ 1998 Confidence intervals for the number needed to treat

Zhu and Wu 2014 "Confidence interval estimation for the number of patient-years needed to treat" Pharmaceutical Statistics 2014
they calculate the confint for the case of Poisson or Negative Binomial regression, based on asymptotic normality
this is different from the literature on two sample Poisson comparison #2718, which doesn't have explanatory variables, but we have exact and approximately exact hypothesis tests and confidence intervals available. 

---

in general: we need a setup for calculating various effect measures and provide the standard errors and confidence intervals, and maybe hypothesis tests

Poisson `eform` calculates relative risk, but for NNT we need risk difference which is not automatically available. see Zhu and Wu 2014 reference above

user: josef-pkt
issue_2718: state: open
title: ENH: 2 sample comparison of poisson rates, including exact
body: http://stackoverflow.com/questions/33944914/implimentation-of-e-test-for-poisson-in-python/34013250#34013250

We can compare rates (positive but not smaller than one) of two samples similar to two sample proportions tests and confidence intervals.

http://stackoverflow.com/questions/33944914/implimentation-of-e-test-for-poisson-in-python/34013250#34013250
#2605 confint for proportions,
#2607 exact tests

there are also other issues for proportions, exact conditional, mid-p, approximate or unconditional exact. The structure seems to be similar in rates and proportions cases.
E-test: uses MLE or constrained MLE for unconditional test with (assumed) exact distribution
There should be also the unconditional exact and berger_boos, but I didn't look for them yet.

R stats seems to have conditional exact, some others are in various R packages.
some R related papers by Michael P. Fay. ref?

below is a copy of my current script file

```
'''Test for ratio of Poisson intensities in two independent samples

Author: Josef Perktold
License: BSD-3

destination statsmodels

'''


from __future__ import division
import numpy as np
from scipy import stats


# copied from statsmodels.stats.weightstats
def _zstat_generic2(value, std_diff, alternative):
    '''generic (normal) z-test to save typing

    can be used as ztest based on summary statistics
    '''
    zstat = value / std_diff
    if alternative in ['two-sided', '2-sided', '2s']:
        pvalue = stats.norm.sf(np.abs(zstat))*2
    elif alternative in ['larger', 'l']:
        pvalue = stats.norm.sf(zstat)
    elif alternative in ['smaller', 's']:
        pvalue = stats.norm.cdf(zstat)
    else:
        raise ValueError('invalid alternative')
    return zstat, pvalue


def poisson_twosample(count1, exposure1, count2, exposure2, ratio_null=1,
                      method='score', alternative='2-sided'):
    '''test for ratio of two sample Poisson intensities

    If the two Poisson rates are g1 and g2, then the Null hypothesis is

    H0: g1 / g2 = ratio_null

    against one of the following alternatives

    H1_2-sided: g1 / g2 != ratio_null
    H1_larger: g1 / g2 > ratio_null
    H1_smaller: g1 / g2 < ratio_null

    Parameters
    ----------
    count1: int
        Number of events in first sample
    exposure1: float
        Total exposure (time * subjects) in first sample
    count2: int
        Number of events in first sample
    exposure2: float
        Total exposure (time * subjects) in first sample
    ratio: float
        ratio of the two Poisson rates under the Null hypothesis. Default is 1.       
    method: string
        Method for the test statistic and the p-value. Defaults to `'score'`.
        Current Methods are based on Gu et. al 2008
        Implemented are 'wald', 'score' and 'sqrt', see Notes
    alternative : string
        The alternative hypothesis, H1, has to be one of the following

           'two-sided': H1: ratio of rates is not equal to ratio_null (default)
           'larger' :   H1: ratio of rates is larger than ratio_null
           'smaller' :  H1: ratio of rates is smaller than ratio_null

    Returns
    -------
    stat, pvalue two-sided 

    not yet
    #results : Results instance
    #    The resulting test statistics and p-values are available as attributes.


    Notes
    -----
    'wald': method W1A, wald test, variance based on separate estimates
    'score': method W2A, score test, variance based on estimate under Null
    'wald-log': W3A
    'score-log' W4A
    'sqrt': W5A, based on variance stabilizing square root transformation

    References
    ----------
    Gu, Ng, Tang, Schucany 2008: Testing the Ratio of Two Poisson Rates,
    Biometrical Journal 50 (2008) 2, 2008

    '''

    # shortcut names
    y1, n1, y2, n2 = count1, exposure1, count2, exposure2
    d = n2 / n1
    r = ratio_null
    r_d = r / d

    if method in ['score']:
        stats = (y1 - y2 * r_d) / np.sqrt((y1 + y2) * r_d)
    elif method in ['wald']:
        stats = (y1 - y2 * r_d) / np.sqrt(y1 + y2 * r_d**2)
    elif method in ['sqrt']:
        stats = 2 * (np.sqrt(y1 + 3 / 8.) - np.sqrt((y2 + 3 / 8.) * r_d))
        stats /= np.sqrt(1 + r_d)

    return _zstat_generic2(stats, 1, alternative)
    pvalue = stats.norm.sf(np.abs(stat))*2
    return stat, pvalue


from numpy.testing import assert_allclose

# testing against two examples in Gu et al

print('\ntwo-sided')
# example 1
count1, n1, count2, n2 = 60, 51477.5, 30, 54308.7

s1, pv1 = poisson_twosample(count1, n1, count2, n2, method='wald')
pv1r = 0.000356
assert_allclose(pv1, pv1r*2, rtol=0, atol=5e-6)
print('wald', s1, pv1 / 2)   # one sided in the "right" direction

s2, pv2 = poisson_twosample(count1, n1, count2, n2, method='score')
pv2r = 0.000316
assert_allclose(pv2, pv2r*2, rtol=0, atol=5e-6)
print('score', s2, pv2 / 2)   # one sided in the "right" direction

s2, pv2 = poisson_twosample(count1, n1, count2, n2, method='sqrt')
pv2r = 0.000285
assert_allclose(pv2, pv2r*2, rtol=0, atol=5e-6)
print('sqrt', s2, pv2 / 2)   # one sided in the "right" direction

print('\ntwo-sided')
# example2
# I don't know why it's only 2.5 decimal agreement, rounding?
count1, n1, count2, n2 = 41, 28010, 15, 19017
s1, pv1 = poisson_twosample(count1, n1, count2, n2, method='wald', ratio_null=1.5)
pv1r = 0.2309
assert_allclose(pv1, pv1r*2, rtol=0, atol=5e-3)
print('wald', s1, pv1 / 2)   # one sided in the "right" direction

s2, pv2 = poisson_twosample(count1, n1, count2, n2, method='score', ratio_null=1.5)
pv2r = 0.2398
assert_allclose(pv2, pv2r*2, rtol=0, atol=5e-3)
print('score', s2, pv2 / 2)   # one sided in the "right" direction

s2, pv2 = poisson_twosample(count1, n1, count2, n2, method='sqrt', ratio_null=1.5)
pv2r = 0.2499
assert_allclose(pv2, pv2r*2, rtol=0, atol=5e-3)
print('sqrt', s2, pv2 / 2)   # one sided in the "right" direction

print('\none-sided')
# example 1 onesided
count1, n1, count2, n2 = 60, 51477.5, 30, 54308.7

s1, pv1 = poisson_twosample(count1, n1, count2, n2, method='wald', alternative='larger')
pv1r = 0.000356
assert_allclose(pv1, pv1r, rtol=0, atol=5e-6)
print('wald', s1, pv1)   # one sided in the "right" direction

s2, pv2 = poisson_twosample(count1, n1, count2, n2, method='score', alternative='larger')
pv2r = 0.000316
assert_allclose(pv2, pv2r, rtol=0, atol=5e-6)
print('score', s2, pv2)   # one sided in the "right" direction

s2, pv2 = poisson_twosample(count1, n1, count2, n2, method='sqrt', alternative='larger')
pv2r = 0.000285
assert_allclose(pv2, pv2r, rtol=0, atol=5e-6)
print('sqrt', s2, pv2)   # one sided in the "right" direction

print('\none-sided')
# example2 onesided
# I don't know why it's only 2.5 decimal agreement, rounding?
count1, n1, count2, n2 = 41, 28010, 15, 19017
s1, pv1 = poisson_twosample(count1, n1, count2, n2, method='wald',
                            ratio_null=1.5, alternative='larger')
pv1r = 0.2309
assert_allclose(pv1, pv1r, rtol=0, atol=5e-4)
print('wald', s1, pv1)   # one sided in the "right" direction

s2, pv2 = poisson_twosample(count1, n1, count2, n2, method='score',
                            ratio_null=1.5, alternative='larger')
pv2r = 0.2398
assert_allclose(pv2, pv2r, rtol=0, atol=5e-4)
print('score', s2, pv2)   # one sided in the "right" direction

s2, pv2 = poisson_twosample(count1, n1, count2, n2, method='sqrt',
                            ratio_null=1.5, alternative='larger')
pv2r = 0.2499
assert_allclose(pv2, pv2r, rtol=0, atol=5e-4)
print('score', s2, pv2)   # one sided in the "right" direction
```

user: josef-pkt
issue_2717: state: open
title: SUM Proportional hazards regression
body: The list below covers various issues with small/moderate complexity relating to proportional hazards regression:
- [ ] If entry times are used, add something to the summary output indicating that this is so.
- [ ] When stratification is present, add a message noting that observations in strata with no events are dropped and are not counted in the reported sample size (perhaps report the number of dropped strata and/or dropped samples).
- [ ] Allow entry times to be provided as a variable name when using formulas.
- [ ] Add handling for exact ties in the survival times (see #2386)
- [ ] Add support for time varying covariates.
- [ ] Add support for case weights
- [ ] Remove regularization code when #2385 is merged
- [ ] Frailty models:
  http://support.sas.com/documentation/cdl/en/statug/63962/HTML/default/viewer.htm#statug_phreg_sect036.htm

user: kshedden
issue_2715: state: open
title: SUM  KDE followup
body: some extra issues while reviewing #2318 
- normal reference for Scott/Silverman with non-gaussian kernel as in https://github.com/statsmodels/statsmodels/pull/1271/files  i.e. for `Epanechnikov`
- additional kernel constants are in old version (useful for plugin, standard errors, ... ?)
- Results class with caching, Grid/Interpolation distribution. but evaluation of points in #2318 looks already fast
- the old multivariate kde has conditional which is still missing in the new version

user: josef-pkt
issue_2714: state: open
title: ENH: binned count data, ordinal Logit
body: (random find)

http://onlinelibrary.wiley.com/doi/10.1002/sim.6495/full
McGinley, JS, Curran, PJ, and Hedeker, D (2015), A novel modeling framework for ordinal data defined by collapsed counts. Statist. Med., 34, 2312–2324. doi: 10.1002/sim.6495.
(not read or downloaded yet)
found through authors website, includes online appendix http://curran.web.unc.edu/manuscripts/

Might follow the same pattern as cumulative link models or interval censored regression. It might be easier than cumulative link model if bin bounds are known. 

user: josef-pkt
issue_2712: state: open
title: SUMM:  QMLE, Logit, Poisson with continuous data
body: I have problems finding various pieces for fractional data analysis with Logit/Probit/Binomial and related Poisson with continuous data.
question on stackoverflow in connection with penalization
http://stackoverflow.com/questions/33845539/modelling-probabilities-in-a-regularized-logistic-regression-model-in-python/33850446#33850446

**main point:** It works but I don't find examples or recipes, and I don't think there are explicit unit tests. There should be enough unit tests for the robust covariance HC0.
#2074 FAQ for LEF robustness
#2642 FAQ overview models by type of response, `endog` data

**fractional/proportion data**
- #2040 main issue, GEE example for Papke/Wooldridge, closed #2044 (enable continuous data)
- #2538 almost duplicate, request for examples/docs
- #2661 more towards stats, treatment effects for proportions

Stata 14 has explicit method `fracreg`, but I didn't read enough to understand what it adds to just using the regular models, with cov_type HC

**Poisson positive continuous data**
- #1151  older main issue, not updated since I started to look at QMLE, 
  we should get more references, pages on internet and articles with discussion 
  (??? I looked at several papers in economics, applied econometrics)

**general issues**
- #2041 score tests, including robust and conditional moment tests
- #1777 GLM with foreign variance function
- #1790 GMM version of GLM models, also other related GMM issues
- combination with penalization, I never checked, see above stackoverflow question

user: josef-pkt
issue_2711: state: open
title: WIP ENH: SparseOLS
body: see #2710

This has a first working version for SparseOLS with cov_type HC0 or nonrobust.

two main problems
- Results subclasses LikelihoodModelResults which doesn't provide all the same results as RegressionResults -> try subclassing RegressionResults
- generic cov_type handling needs refactoring in `LikelihoodModelResults.__init__`, and there is a stale cached bse if I don't empty the cache explicitly. Other cov_types are missing.

other issues:
- Data handling, `model.data` is just a Dummy class that is needed for now for `param_names` (used in t_test), RegressionResults also need the has_constant information, e.g. for rsquared, and maybe needs other information in the data instance.
- this should be easy to convert to SparseWLS by just adding a `whiten` method

tested currently against OLS for the main information that is also in the params table

user: josef-pkt
issue_2710: state: open
title: SUMM/ENH: sparse matrix support in models
body: see mailing list
https://groups.google.com/d/msg/pystatsmodels/fOQtn5RKxzI/SX6UjBYaCwAJ  "OLS with sparse exog"
https://groups.google.com/d/msg/pystatsmodels/A2ZBopoCynY/ikoR-X3vCgAJ "Robust standard errors and memory usage"
- [ ] Data handling: need a dataclass for sparse exog, pandas integration, param_names
- [ ] categoricals, patsy's formulas are dense only, new pandas has sparse get_dummies
- [ ] core calculations, see below
- [ ] Results post estimation, robust covariance matrices, see SparseOLS as example (HC0 only)

**core calculations**
(copied from mailing list)
- MixedLM uses it for the covariance calculations and penalization (I guess)
- For linear models like OLS, the optimizer itself needs to be for sparse, Etienne used sparse.linalg.lsqr.
- For discrete and GLM, I think only predict needs to be changed to calculate the linear prediction with sparse exog if we use a dense Hessian and standard optimization. (Hessian could be sparse but I don't think there is a sparse non-linear optimizer.)
- There is also an old PR for sysreg which uses sparse block matrices or sparse Kronecker products to handle balanced panel data or models with multivariate endog.

One design question that is not clear to me yet is whether we can retrofit the existing models for sparse or add new models. I think that right now there are too many gaps in the supporting infrastructure that it's better or easier to write new models.

I will clean up SparseOLS (currently in a Gist) or SparseWLS  to get a basic model with sparse support. GLM with scipy optimizers would be another good candidate, score_obs and Hessian can be calculated from the 1-D factors.

user: josef-pkt
issue_2708: state: open
title: ENH: histogram distribution fitting, poisson, semiparametric exponential distribution
body: see #2297 for implementation and use

local FDR is based on Efron Statistical Science 2008, with reference to R package

that paper refers to

Efron, Bradley; Tibshirani, Robert. Using specially designed exponential families for density estimation. Ann. Statist. 24 (1996), no. 6, 2431--2461. doi:10.1214/aos/1032181161. http://projecteuclid.org/euclid.aos/1032181161.

histogram density fitting with Poisson is attributed to Lindsey 1974.  Sounds similar to the Astro application.

Efron/Tibshirani use a KDE as carrier density, but not in the FDR paper. I've also read other articles with polynomial exponential models, but MLE not histogram based. The basic approach is easy, the GOF statistics and bandwidth choice in Efron/Tibshirani looks a bit more difficult.

This is related to series approximation and semi-nonparametric density estimation (Gallant/Tauchen ?) but it looks like it generates proper density in an easier way. (exp instead of squares, if I remember semi-nonparametric correctly.)

user: josef-pkt
issue_2706: state: open
title: FAQ   datahandling for discrete choice Conditional Logit in long form
body: just a link found by chance
http://stackoverflow.com//questions/17688155/complicated-for-me-reshaping-from-wide-to-long-in-pandas

this looks like a required format for `ConditionalLogit`, but comments refer to asclogit in Stata, i.e. `CLogit`

user: josef-pkt
issue_2705: state: open
title: FAQ-D: going from prototype or basic model to full model
body: related what's required #1970  (doesn't provide much information yet)
(I thought I wrote already a list of "What's in a model and results?" but don't find it.)

Let's assume we want to build a new maximum likelihood model. For quick prototyping we can subclass `GenericLikelihoodModel` which tries to fill in some of the missing elements in a generic way.
Full MLE model subclass `LikelihoodModel` and `LikelihoodModelResuts` which requires some additional model specific parts.

The Base Model

The minimal part of any model is the loglikelihood function and a predict method, if prediction applies. LikelihoodModel provides data and formula handling and optimization,  the LikelihoodModelResults class provides inference based on Wald tests and a few extra methods.
(TODO: what else do we have generic? summary?)

These inherited methods are still mainly designed for the basic `(endog, exog, keywords)` models, and will need adjustments in models that follow a different pattern.
(TODO: explain what to do in two part models, and figure out which support is missing)

The basic model should then provide wald parameter inference, standard errors, tvalues and pvalues, and prediction.
general Wald tests, `t_test`, `wald_test`, are inherited but some things need to be available, e.g. t_test raises if `df_resid` is not defined.
## The full model
- wrapper class for results: The wrapper class for a results class provides the pandas interface in the returns and attributes. For example the parameters `params` are returned as a pandas.Series if the original data was in a pandas DataFrame or Series.

(more later)

user: josef-pkt
issue_2704: state: open
title: Parallel version of Formula models possibly by USING spark
body: I have been using statsmodels for training some Formula based models of the type `y ~ X1 + X2` and am struggling with the memory requirements and speeds of statsmodels. My data usually consists of number of records of the order of 500 million and features around 100-200. Because, I have a lot of RAM I am able to train my models in 20-30 hours. However, when I monitor the python process I observe that python ends of taking a lot of memory and the operation is really slow. 

A major reason I am using `statsmodels` rather than `sklearn` or other ML libraries, which support parallel model training, is because of the amazing R style formula models and summary statistics (e.g. std. err, p-value, confidence interval, AIC, BIC, R-squared, etc.) generated by statsmodels' `summary()` function. I am aware that there are parallel ways to train models and feel that statsmodels should add support for that. Also, process for generating summary statistics for statsmodels should be parallelized so that it can be used for big datasets. 

I have also started using Apache Spark recently and have found that they support the `glm` module from R as part of their `SparkR` package. However, that module also has just coefficient summary for the model. I believe there is this great opportunity for statsmodels to support parallel model training and summary statistics and spark can also be used as a dependency if needed. 

If there are already plans to add in these features do share that. 

user: napsternxg
issue_2701: state: open
title: SUMM/ENH rare events, unbalanced sample, matching, weights, tail
body: see also #505 with some links to stackoverflow questions in comments
my current view (without knowing literature): http://stackoverflow.com/questions/33605979/statsmodels-logistic-regression-class-imbalance/33611826#33611826

Instance sampling in credit scoring: An empirical study of sample size and balancing
http://www.sciencedirect.com/science/article/pii/S0169207011001403
answer and comments here by one of the authors http://www.analyticbridge.com/forum/topics/handling-imbalanced-data-when-building-regression-models

Gary King and Langche Zeng, 2001, Logistic Regression in Rare Events Data
http://pan.oxfordjournals.org/content/9/2/137.abstract
referenced in answer here http://stats.stackexchange.com/questions/6067/does-an-unbalanced-sample-matter-when-doing-logistic-regression

an overview of methods related to machine learning
http://www.ele.uri.edu/faculty/he/PDFfiles/ImbalancedLearning.pdf

King/Zeng looks the most popular on stats.stackexchange and might be a good starting point for the statistics problem, e.g. for inference.

There is also a relationship to matching (like propensity score matching) but subsampling or oversampling uses the endogenous variable, and to case-control matching which matches on outcome, IIRC.  (related aside: Agresti categorcal data analysis has comments in several sections about sampling, fixed margin counts, based on the "endogenous" variable. Odds ratio is identified in Logit, but not in other models, but I didn't try to figure out the details.)
#2442 proposes to implement weights without worrying about interpretation i.e. implement first and add support for different interpretation or kinds of weights later.

specifically, I need them for IPW in #2455

user: josef-pkt
issue_2699: state: open
title: ENH/BUG   postmerge review prediction, eform, nonlinear transformation
body: #2151 added several things but without a lot of review, and without advertising.

(basic test coverage looks fine, but maybe not for extra options.)
- [ ] `params_transform_univariate`  `transform` keyword is not implemented, incomplete docstring
    IIRC, the idea was to allow for transforms that are like links but not coded as inverse.

user: josef-pkt
issue_2698: state: open
title: REF/ENH:  GLM-Binomial/Bernoulli  count versus probability
body: I just saw that GLM binomial fittedvalues returns probabilities not counts.
Binomial is actually Bernoulli with frequency weights.

The difference will be for prediction and prediction confidence intervals and margins depending on whether we want results for proportions or counts.

possible solution: add frequency weights explicitly and introduce different "Binomial/Bernoulli" families, one for count data and one for proportion results.

alternative (maybe easier): add count option to predict and related

If we allow generic multiplicative exposure, then I think it is equivalent to specifying a frequency weight in prediction for general GLM.

user: josef-pkt
issue_2696: state: open
title: SUMM/ENH: margins, ATE and ATET for stratified contingency tables
body: Agresti categorical data analysis, 2nd, 2003,  section 2.3 on stratified 2x2xK tables 
has good discussion on other explanatory (confounding) variables and Simpson's paradox

He doesn't mention (as far as I read) average treatment effect and average treatment effect on the treated.

AFAICT, this would be calculating a marginal table but not with observed frequencies along the confounder (K) dimension, but with marginal weights corresponding to population or treated.
i.e. the same what we would be doing in Logit/Binomial to calculate ATE (in parametric model)

**addition**
The more general issue is in how to handle frequency weights in the models for calculating Margins and POM (potential outcome mean). Instead of adjusting the exog for predict, we need to adjust the weights to calculate POM, assuming no unobserved confounders.

example: from Agresti 2003: 2x2x2x2 contingency table with seatbelt use, gender, location and injury. I'm using a GLM-Binomial example right now.

Qu. What would have been the injury proportion of women if they had the same seatbelt use and location as man, or the population average seatbelt use and location?
Qu. What's the ATE of seatbelt use, population averaged over confounders.
(This is similar to case in economics when we have a conditional individual effect and a composition effect. change in price versus change in quantities to get total expenditures)

Related: inverse probability weighting and survey weighting.

see also #2698  prediction in GLM-Binomial. If we have "exposure" or "fweights" as option in predict, then it would be easy to aggregate prediction with different weights.

user: josef-pkt
issue_2691: state: open
title: REF: outsourced joint hypothesis tests based on normal distribution
body: Another vague idea that shows up in different places

wald_test and t_test are methods of LikelihoodModelResults, we cannot reuse it after other methods or functions. Parallelized t_tests are easy, as in contrast results, because they only require diagonal but could require linear combinations.  Joint F or chisquare test are more complex to reimplement each time, and we have to loop since they are not vectorized.

usecase score/LM test for variable addition in the CvM (?) version
roughly: I have a set of variables, including multicolumn variables like categorical factors, that we want to test for variable addition. 
References for CvM (?) type tests are for specification tests with multiple alternatives, when we want to have each test robust to the other alternatives. I cannot find reference or comment for them right now. (context of #2096 conditional moment tests)

example right now: model selection in log-linear model with stepwise forward regression adding interaction terms

user: josef-pkt
issue_2689: state: open
title: FAQ-D: GLM, loglinear, contingency table - Pearson versus standardized Pearson
body: #2418 add contingency table, and there is a question on what the distribution is for Pearson and for standardized (Pearson) residuals

Agresti: Categorical Data Analysis, Second Edition, 2003
section 9.3.3 correspondence between loglinear and logit residuals

loglinear uses Poisson denominator (variance), Logit uses Binomial denominator (variance)
standardized Pearson residuals  `r = e / sqrt(1 - h_i)`  h_i diagonal of hat matrix, in loglinear (poisson) is the same as the binomial Pearson residual.

Related Agresti (section 8.6.7) summarizes and refers to Lang 1996 for the connection Poisson versus Multinomial sampling 

user: josef-pkt
issue_2688: state: open
title: ENH/Note: model selection in GLM and log-linear models - LR, Pearson/Score
body: mainly a note for future reference

Agresti Categorical Data Analysis 2002? 
section 9.2.3, loglinear model comparison statistic, p 363 and following. section before is on stepwise forward or backward selection
end of section 9.2.4 uses multiple testing correction (Bonferroni style - the other one where I don't remember the guys name)

LR test, difference of deviance is additive and tests are independent along a path of nested models.

Comparing Pearson chisquare statistic is not necessarily positive when comparing nested models. Denominator should be kept constant (similar to keeping scale/variance/weight constant in LR-test, j-test or similar) equ. 9.4
related: Smyth "Pearson's Goodness of Fit Statistic as a Score Test Statistic"  Score test against the saturated alternative. 
(Note: when we have a fixed number of categories and no continuous variables, then saturated model has a fixed finite number of parameters independent of sample size, i.e. standard MLE theory applies.)

AFAICS: this is standard result for MLE/GLM, but is well described in the log-linear model section in Agresti.

user: josef-pkt
issue_2687: state: open
title: SUMM: post-estimation support for categoricals with formulas
body: patsy has added full categorical infor in  design_info see #2292
http://patsy.readthedocs.org/en/latest/API-reference.html#patsy.DesignInfo.factor_infos
and `term_codings`

```
>>> rr = patsy.dmatrix('x', {'x': ' a b c d e f'.split()})

>>> rr.design_info
DesignInfo(['Intercept', 'x[T.b]', 'x[T.c]', 'x[T.d]', 'x[T.e]', 'x[T.f]'], 

factor_infos={EvalFactor('x'): FactorInfo(factor=EvalFactor('x'), type='categorical'
, state=<factor state>, categories=('a', 'b', 'c', 'd', 'e', 'f'))}, 

term_codings=OrderedDict([(Term([]), [SubtermInfo(factors=(), contrast_matrices={}, 
um_columns=1)]), (Term([EvalFactor('x')]), [SubtermInfo(factors=(EvalFactor('x'),), 
contrast_matrices={EvalFactor('x'): ContrastMatrix(array([[ 0.,  0.,  0.,  0.,  0.
],
       [ 1.,  0.,  0.,  0.,  0.],
       [ 0.,  1.,  0.,  0.,  0.],
       [ 0.,  0.,  1.,  0.,  0.],
       [ 0.,  0.,  0.,  1.,  0.],
       [ 0.,  0.,  0.,  0.,  1.]]), ['[T.b]', '[T.c]', '[T.d]', '[T.e]', '[T.f]'
])}, num_columns=5)])]))
```

and (dropping magic methods):

```
>>> dir(rr.design_info)
[ '_repr_pretty_', 'builder', 'column_name_indexes', 'column_names', 'describe', 
'design_info', 'factor_infos', 'from_array', 'linear_constraint', 'slice', 'subset', 
'term_codings', 'term_name_slices', 'term_names', 'term_slices', 'terms']
```

It looks like we have all information to support
- `pairwise wald/t test`, and other premade contrasts (which ?)
- `margins` for categorical explanatory variables and  (issue ?)
- predictive means LSMeans/predict_at  #2167
- stepwise regression, and LR and score tests for categorical terms

Related: something similar for stateful transforms, and multicolumn transformation (np.vander) are not yet available.

user: josef-pkt
issue_2678: state: open
title: ENH: data augmentation for Ridge in pinv
body: just an idea while taking a break

pinv can have problems with singular matrices (e.g. empty cells in categorical interactions) 
#2628 and mailing list https://groups.google.com/d/msg/pystatsmodels/TmxnSHAJltg/zDTSDlkPCAAJ (noise instead of zero)

The alternative to increasing the threshold for singular value cutoff, is to add a small Ridge penalization.

Since we don't have the product matrix X'X, we could use data augmentation 
`exog = vstack(exog, 1e-15 * eye(..)` and `endog = vstack(endog, zeros(...))` just for calculating params and normalized_covparams.

I didn't check if it makes sense or would be easier than full Ridge.

user: josef-pkt
issue_2677: state: open
title: WIP ENH: optimize knot location, Segmented Regression
body: see #2634

Use case: This is a complementary method to penalized splines, where we want to carefully place a few knots instead of penalizing a large number of knots. The main advantage is in cases where we have clear breaks like in piecewise linear regression. This uses power splines and assumes that the estimated function is continuous but not smooth.

This is still largely "research code". I'm trying to figure out how to make this computationally efficient and robust in the optimization.

to the optimization: I guess we can have many local optima. My initial idea was to optimize a knot at a time, restricting it to be within the interval given by the neighboring knots. (article on knot selection in splines, ref?). Because of a mistake in the scipy documentation for `brent` I actually use the lower and the center point of the bracket which actually works better. Using `fminbound` to force the interval gives worse results.

One consequence/problem with `brent` is that it doesn't preserve the increasing order or knots.
It works well in the examples, but it doesn't "feel" robust.

Another option is to increase the number of knots sequentially. This also works well in the example, but we still have the unsorted knot problem.

TODO:
- compatibility fixes: I'm working currently only with python 3.4 and numpy 1.9. Final version needs to have compatibility fixes.
- no predict yet: We need to be able to create the spline/segment basis for an exog in prediction.
  Similar to stateful transform in patsy.
- pandas integration, xnames, ...: currently extra `data` information is not transferred to created models
- no `model.__class__.__init__` keywords yet, should be relatively easy to add
- Other exog: currently uses or assumes the full model: projecting on other exog would be computationally more efficient. (p-inverse of one block in partitioned matrix, instead of full p-inverse)
- selecting number of knots: sequential addition of knots also provides, ssr, aic, ... for choosing the number of knots. A proper hypothesis test would have to take the optimization of knot location into account (multiple testing, sup test)

user: josef-pkt
labels: superseded: Used for PRs with content that is superseded by a newer PR
issue_2673: state: open
title: ENH: test_LR, compare_LR with internal estimation, lr_anova
body: I'm testing sequential variable addition.  LR F-test version 

compare_lr and GMM.jtest have the problem that we should keep some auxiliary estimates, like scale or weightmatrix as fixed. In the current version that's up to the user because the user provides both models.

variation:
- adjust llf, or F-statistic to adjust for scale  (see open issues of adding scale as argument)
- reestimate model with the restriction on the auxiliary parameters
- add lr version of restriction tests

In application for variable selection there is still the multiple testing problem.

user: josef-pkt
issue_2664: state: open
title: Perf:  check memory layout for efficient access or avoiding copies
body: We never explicitly check or optimize for memory layout. Some iterative loops assume that we have Fortran contiguous arrays, e.g. autolag in  tsa.stattools, but it's not verified not unit tested.

np.column_stack changed behavior after numpy 1.6 and does not always return F-contiguous any more.
see https://mail.scipy.org/pipermail/numpy-discussion/2015-October/074031.html

We should check at least some loops that they really do have the right memory to avoid copying.

user: josef-pkt
issue_2663: state: open
title: BUG/ENH kruskal as MultiComparison
body: The status of `MultiComparison.krusal` is unclear, the main function for users is in scipy.stats

The intention in MultiComparison was to reuse the standard tools for the rank versions.

Looking at a bug in scipy, I get similar results with nan.
https://github.com/scipy/scipy/issues/4017 and the linked 3611 (which is specific to the scipy cython implementation)

I didn't check what's going on with `MultiComparison.krusal`

```
>>> x = [1.] * 163 + [2.] * 81 + [3.] * 40 + [4.] * 6 + [5.] * 2
>>> y = [1.] * 1007 + [2.] * 362 + [3.] * 99 + [4.] * 27 + [5.] * 13
>>> from scipy import stats
>>> stats.mannwhitneyu(x,y)
(193405.5, nan)
C:\programs\WinPython-64bit-3.4.3.1\python-3.4.3.amd64\lib\site-packages\scipy\stats\stats.py:4093: RuntimeWarning: invalid value encountered in sqrt
  sd = np.sqrt(T*n1*n2*(n1+n2+1)/12.0)
>>> import scipy
>>> scipy.__version__
'0.15.1'
>>> len(x)
292
>>> len(y)
1508
>>> groups = [0] * len(x) + [1] * len(y)
>>> xy = np.concatenate((x, y))
>>> len(xy)
1800

>>> co = sm.stats.multicomp.MultiComparison(xy, groups)
>>> co.kruskal()
MultiComparison.kruskal
0 1 109.399413175 nan nan False
nan
nan
m:\josef_new\eclipse_ws\statsmodels\statsmodels_py34\statsmodels\sandbox\stats\multicomp.py:868: RuntimeWarning: invalid value encountered in sqrt
  se = np.sqrt(f * np.sum(1. / groupnobs[[i,j]] )) #np.array([8,8]))) #Fixme groupnobs[[i,j]] ))
```

user: josef-pkt
issue_2662: state: open
title: BUG: GMM DistQuantilesGMM is broken and unfinished
body: DistQuantilesGMM  is a victim of the previous round of GMM and GMMResults refactoring.

It was written based on articles for estimating distributions, and doesn't have the GMM pattern of individual observations. I'm not sure what the status is.

Since I wrote this I learned that I could use indicator functions for the individual observations.
Without individual observations we cannot compute the standard empirical covariance of the moment conditions, but it's possible to work without it if we can get it from somewhere else as in the distribution estimation. 

One problem is that I didn't find any other implementation to write unit test, but it should now be possible to test the with observation version using standard GMM implementation with the summary statistics only version. We might run into similar problems with other minimum distance estimators that are not M-estimators.

user: josef-pkt
issue_2661: state: open
title: ENH: fractional data, treatment effect in paired proportions (not potential)
body: (just a random find)

http://nbviewer.ipython.org/github/stitchfix/Algorithms-Notebooks/blob/master/mult-test-blog/free_throw_analysis.ipynb

The outcome variable is a paired/bivariate proportion depending on observed binary treatment, and we observe many independent observations of those. proportion is not constant across observations.
Analogous to paired t-test but with proportions.

It's not treatment effects literature where treatment is either one or the other but not both, and where we need to use potential outcomes.

Assume independence conditional on unobserved individual true probabilities, i.e. individual specific constant.

What's the best small sample test for testing for no effect. What's the best small sample estimate for average marginal effect size.

Underlying model is binomial GLM but with paired data and unobserved heterogeneity. Because of binomial, we also have heteroscedasticity. Alternative, Quasi-MLE for "continuous binary model".

Note if the outcome variable were binary, then we could use ConditionalLogit, but I guess it cannot be mapped into it. (Conditional Logit conditions out a single fixed effect constant per individual. It might be possible under the assumption that the treatment effect (change) is constant across individuals, but the base level can be a fixed effect.)

Note: I don't think this maps into the testing two proportions case or contingency tables, because there the underlying observation is binary, unless stratified contingency tables or 2 by k tables can be reused for this, where strata or the k column units are the individuals.

Using Fisher's (?) transform or similar to transform to paired z or t test will be the easiest but accurate only with large enough samples, and I guess not too close to the 0,1 boundaries.

aside question: Is there multiplicative heteroscedasticity version for Binomial similar to Poisson?

user: josef-pkt
issue_2660: state: open
title: why sm.add_constant does not work here?
body: ![sm_add_constant](https://cloud.githubusercontent.com/assets/1667488/10555635/e1e608e2-7437-11e5-96fd-ed45bafb0139.PNG)

it is weird sometimes it works and sometimes it doesn't

user: songhuiming
issue_2657: state: open
title: FAQ-D, check:  chaining, nesting complex step derivatives
body: Can complex step derivatives be nested or chained?

I guess not, as far as I understand the approach.

not verified

user: josef-pkt
issue_2654: state: open
title: ENH: composite models - composite (partial) likelihood (general theory)
body: see also #1950  and issues on conditional moment tests and GMM

combining marginal and conditional likelihood functions, or their estimating equations, without specifying the full likelihood. Similar idea as in GEE and GMM but partial likelihood based.

Varin, Cristiano, Nancy Reid, and David Firth. "An overview of composite likelihood methods." Statistica Sinica 21.1 (2011): 5-42.
http://www3.stat.sinica.edu.tw/sstest/j21n1/j21n11/j21n11.html
(315 citations in Google)

composite likelihood refers to the general approach, not a specific model. Similar to partial likelihood, limited information MLE, ...

The asymptotic theory follows directly from M-estimation with misspecified likelihood. Or could be represented as exactly identified GMM model. (cov_params has the usual sandwich form).

(Aside: a difference in approach: Here we start with a fictitious full likelihood but it's too complicated to estimate the full model or we want to skip specifying some parts for specification robustnes. So, we take as much as we can from the likelihood. In constrast, GMM is more often: Let's find some orthogonality conditions and estimate. Composite likelihood provides a partial likelihood based way to find those moment conditions.)

Example applications in the paper are more oriented towards panel and spatial models where we cannot model or estimate the large joint correlated model, e.g. because of the size of the required matrices, or because it would require very high dimensional numerical integration.)

(I have read only parts of the article, and found it again because I was looking at composite GMM.)
One section refers to the comparison with GEE1 and GEE2, which I had read some time ago.

(some more comments after reading)

Then provides a good overview with many references that sound interesting. The general part sounds "general", but the examples and issues are mostly on combining similar marginal or conditional models, not combining models for different submodels as in what I was looking at in GMM, e.g. `heckman` type two stage or two equation models.

A more typical example would be correlated binary models or correlated countmodels, with temporal, spatial or cluster correlation. A popular approach for these case is to use pairwise models, e.g. use pairwise numerical integration for random effects instead of one very large dimensional integration.

To implement this we still need to have the bivariate models firs.

about asymptotics: M-estimator or estimating equation asymptotics are standard with fixed, finite number of parameters. Some models are not in this setting, analogy for example with incidental parameter problem in panel data with fix length but increasing number of units.

The paper mentions also many other technical issues, the we run into in different places, and don't have yet in statsmodels, parameter averaging/combination, small sample precision of covariance of moment condition (center of sandwich), one-step estimation for e.g. jackknife, ...

(aside: sandwich robust covariance refering to Heagerty and Lumley sound like flattop or uniform kernel on overlapping or non-overlapping windows.)

"big data"  in some cases the motivation is to handle large datasets, where for example full MixedGLM wouldn't work.

user: josef-pkt
issue_2653: state: open
title: ENH: ARMA allow higher order trend
body: trend in fit of ARMA only has options 'c' and 'nc'.  As a convenience, we could add also higher order trends. 

related `trend` should be `__init__` keyword #1049

Main advantage and work is that for predict and forecasting the trend would be internally generated instead of requiring users to go through `exog`.

It doesn't do anything additional to what can be done already explicitly through `exog`, but if we have a trend keyword I don't see any reason why we shouldn't extend it.

see for example stackoverflow question with pandas example
http://stackoverflow.com/questions/33067208/arma-forecasting-problems-with-trend-elimination

user: josef-pkt
issue_2649: state: open
title: ENH: KDE for discrete, contingency tables
body: see https://groups.google.com/d/msg/pystatsmodels/F7n-zcZPoEw/YfGVpLbiBwAJ

KDEMultivariate allows for discrete variables in the mixed type setting. However, it looks inefficient if there are only discrete variables.

My guess is that faster code could use the observed frequency or counts in the contingency table directly.

Qu: What are good distance measures for predicted probabilities in cross-validation
(e.g. gof test for Poisson, Hosmer/Lemeshow for Logit.

user: josef-pkt
issue_2648: state: open
title: Wrong error message when fitting a VAR model to a small dataset
body: If attempting to fit a statsmodels.VAR model to a DataFrame that is too small, during the choleskey factorization, it will return the error 
"ValueError: array must not contain infs or NaNs" that ultimately stems from numpy\lib\function_base.py", line 613, in asarray_chkfinite
    "array must not contain infs or NaNs"

An assertion could have been made at an earlier point to capture this error, and give a descriptive roor message.

user: Kristian60
issue_2646: state: open
title: ENH: GMM with observation weights, heteroscedasticity
body: followup to my comments here and related comments https://github.com/statsmodels/statsmodels/issues/2640#issuecomment-146280375

GMMW would be an analog to WLS. It is a relatively trivial extension, since weights can already be incorporated into the moment conditions, and it is implicitly used for example in GMMGLM.
This makes it also possible to write a unit test directly against WLS and GMM.

extensions for this would be pre- or simultaneously calculating the weights like in GLSHet given a model for the variances, or as in GLM as a function of the mean.
Calculating variances as additional moment condition would go into the direction of 2-parameter GLM/LEF and further for the GEE usecase into QIF (quadratic inference function)

And it would be a simple case to work towards more general whitening of correlation from the moments.

One possible complication is that we might want to use nonrobust GMM-weight matrix and cov_params as special case.
Some of this might only work for GMM with IV structure (models with `elementary zero function` in Davidson, MacKinnon, which are generalization of residuals in least squares models and GLM)

main direct applications
- IV with heteroscedasticity
- generic observation weights to use in robust RLM equivalents (downweight outliers or influential observations, just a possible connection: truncated regression for heavy tailed distributions or infinite variance distributions Eric Renault ?)

user: josef-pkt
issue_2642: state: open
title: FAQ: models by reponse type - what's available
body: Here is a brief overview (which should be extended for the documentation)
## continuous response
- continuous response on real line 
  - linear models OLS, WLS
  - generalized linear models GLM with Gaussian family (nonlinearity with known link function)
  - robust linear models RLM  (if outliers)
  - panel data : GEE, MixedLM   (rest is TODO or WIP), fixed effects via dummies
  - time series: GLSAR and all models in tsa: AR, ARIMA, VAR, SARIMAX, ...
- positive continuous response
  - GLM with Gaussian or Gamma family and exp or other positive link
  - Poisson, NegativeBinomial (?), GLM with those families,  with robust standard errors (Quasi-MLE)
  - truncated, Tobit: WIP
- proportion, continuous data on interval
  - Logit, Probit, GLM with Binomial family and robust standard errors  (Quasi-MLE)
  - BetaRegression (WIP)
## binary response
- Logit, Probit, GLM-Binomial and various link functions
- correlated data structure (panel or cluster): GEE
## multinomial response
- MNLogit
- GEE
- contingency table
- as WIP: ConditionalLogit, CLogit, OrderedLogit, ...
## Count Data
- Poisson, GLM-Poisson, NegativeBinomial, GLM-NegativeBinomial, the last has fixed dispersion parameter
- GLM-Binomial: count with known upper bound, e.g. k successes in n trials
- Zero-Inflated (experimental, WIP)
## Multivariate and Multi-equation models
- time series: VAR, VARMAX, ...
- WIP: heckman, binary treatment
- Instrumental variables (does it belong here?)
- MANOVA, MultivariateOLS
## Generic - Do it yourself
- GenericLikelihoodModel: for MLE
- Generalized method of moments, GMM  (Poisson in test suite)

user: josef-pkt
issue_2641: state: open
title: ENH: Move compare_lr_test to LikelihoodModelResults
body: See #2436
Closes #2440 

Part of my unfinished GSoC(another PR pending)

user: saketkc
issue_2640: state: open
title: ENH/REF/SUMM   GMM round 3: cleanup and more enhancements
body: Time to finish up and move it out of the sandbox.

I didn't seen a bug about wrong numbers since the last round.

TODO:
- [ ] Documentation: write at least basic explanation and documentation for options, publish notebooks
- [ ] API another review and some adjustments
- [ ] some refactoring
  - [ ] maxiter=0 ? see #2637 
  - [ ] weight methods: more kernels for HAC, and reuse `sandwich_covariance`
  - [ ] data and names handling, e.g. `param_names` see #2633 
  - [ ] check if score is more accurate and faster using equation for 1st derivative (using derivative of  moment conditions Dg W g)
- [ ] check test coverage of inherited methods (e.g. I recently had to fix t_test that raised exception)

unclear yet
- separation of `cov_type` from weights method: there is no option even though we have separate calculations
  e.g. AFAIR: maxiter=1 uses initial weightmatrix, but calculated cov_params based on covariance of moments after the fit.  This is the main usecase.

Extensions and internal usage
- [ ] hypothesis testing: 
  - [ ] check Wald test, 
  - [ ] add score/LM test, 
  - [ ] refactor compare Q for fixed weights

Internal usage will not be necessary for GMM itself, but provides cases to check how well it works
- write models that use GMM: I don't have any current plans for this, make IVPoisson official (out of test suite)
- use GMM for specific inference questions: That's my current objective for internal use of GMM
  - [ ] get GMM representation of a model, e.g. easy for GLM
  - [ ] conditional moment tests on top of GMM representation of a model
  - [ ] covariance matrix of combined equations, example treatment effects, McFadden has some other examples for two stage estimation (e.g. `heckman`)

user: josef-pkt
issue_2639: state: open
title: ENH/SUMM: One-Step estimators (one optimization step from a consistent start)
body: topic: computationally cheap tools

McFadden reference here
https://github.com/statsmodels/statsmodels/issues/2637#issuecomment-145311645
also points out one-step estimators, similar to Davidson and MacKinnon in their textbook

example cases: going a step from constrained to unconstrained estimate or reverse.
Doing one-step has the same asymptotic properties at 1st order as full optimization.

Useful for example if we want to have simple tests for variable or constraint selection as in variable selection approaches. It might also be useful in other cases where speed matters because we need to do many calculations, maybe bootstrap.

Another possible usecase is to get initial conditions for the full optimization. (But maybe if we implement the setup for one-step, then it might not be to difficult to iterate to convergence for this case. That depends on how we implement this.)

user: josef-pkt
issue_2638: state: open
title: ENH: helper: ordered multinomial logit parameter transformation for discrete mixture
body: A simplified version of multinomial logit with monotone choice parameterization.

e.g. we have 5 classes (discrete points on the real line) and we don't know the location of the points nor the probabilities. We want to estimate a discrete mixture distribution for a sample or as one component (mixing probability) in a larger model.

To get a unique parameterization of the "states", we can impose monotonicity x0 < x1 < ... < xk where xi are the support points.

parameterization c0, c1, ..., ck such that
x0 = c0
x1 = x0 + np.exp(c1)
...
xk = x{k-1}  + np.exp(ck)

This is what I used for ordered Logit/Probit, but is more widely useful, for example to implement mixed effects models with finite support mixing distribution.

user: josef-pkt
issue_2637: state: open
title: ENH: score/Lagrange Multiplier test for GMM
body: This seems to be missing in current GMM code, I don't remember if or what I read about lagrange multiplier tests in GMM.

At the end here
https://github.com/josef-pkt/misc/blob/gmm_glm/notebooks/ex_gmm_glm.ipynb
I compare score test in GLM with test of overidentifying restriction in GMM, but that's not the same as a score test since we don't keep the parameters fixed. Or if we put zeros in the weight matrix, then we don't get the unrestricted values (i.e. with full weight matrix).

related: maxiter=0 changes the parameters from the starting values, AFAICS in an example
(I thought that I have an issue that maxiter=0 keeps params unchanged, e.g. #1728)

related issues:
#2041 score tests, conditional moment tests
#2096 PR for score tests, lagrange multiplier tests and conditional moment tests (PR for translating theory into code)
#1096 review and more fixes for GMM  (issue sounds outdated)

user: josef-pkt
issue_2636: state: open
title: Design: Parameter classes for general nonlinear models?
body: (Just a vague idea about how this can be incorporated in a generic way.)

Kerby uses parameter classes in several model, mainly to handle various transformations.
mlefit uses parameter classes including starting values and bounds
various mcmc packages all require predefined parameters (with prior distributional assumptions)

Should we go into this in a generic systematic way?

use cases
- nonlinear least squares, and MLE for nonlinear prediction function - similar to mlefit
- generically support distributions estimation with multiple parameters - e.g. a MLE wrapper around scipy.stats.distributions
- generic estimators like GMM with arbitrary nonlinear moment conditions, estimating equations
- modular way to impose constraints on parameters

requirements
- link function: explicitly given by users not defaults based on bounds as mlefit
- constraints:  set some params to a fixed value during optimization, attach transform_params to the parameter class
- linear predictor: we don't want to just have scalar parameters, but defining linear, or even nonlinear predictors for the parameter (similar to expectation mu in GLM, and two parameters in beta regression, ZIP, ...)

partially going into the direction of R's GAMLSS and of mlefit at the same time

some of it would just be outsourcing the transform_params that are currently used.
For single parameter models it will be overkill, i.e. unnecessary complexity.

user: josef-pkt
issue_2634: state: open
title: ENH/DOC: segmented regression
body: https://groups.google.com/d/msg/pystatsmodels/YztBPF9gr7M/81rw0hp0GDMJ
http://datascience.stackexchange.com/questions/8266/is-there-a-library-that-would-perform-segmented-linear-regression-in-python

Is there anything specific about this? 
For known break points, this can just be modeled by an interaction with a indicator function (0 before, 1 after the break) or a linear spline. The first approach has a jump, the second approach results in a connected piecewise regression line

patsy has splines

possible: What are useful tests or measures for deciding on where to set the break points if they are not known?

Chad had in previous GSOC preparation (?) an implementation of Bai and/or Andrew and Ploberger. That's for structural breaks but should apply in a similar way for breaks in individual terms. (What's the distribution of the sup test for a knot in a linear spline?)
or just use cross-validation or information criteria to decide on when to stop adding breaks.
(AFAIR: recursive dynamic programming subdivision in Bai as implemented by Chad)

user: josef-pkt
issue_2633: state: open
title: BUG/ENH: GMM, allow None exog and instruments in generic GMM
body: working with the Gamma example for GMM in Greene.

Here we only have a single data variable, e.g. just an endog

fit and summary work if we create fake exog and instruments to provide the shape, but it doesn't work if we set exog and instrument to None.

ENH: provide better default support when we don't have the endog, exog, instrument pattern for the data and model.

problems: how and where do we infer and set k_params and k_moments, and how do we set the param names?  (summary/SimpleTable truncate params list if names are not full length)

(edit) param_names handling in GMM looks a bit outdated, needs check with current implementation of generic attributes in `model.data`. We have now param_names separate from xnames, IIRC.

(edit 2) I'm not sure anymore it's a code or documentation problem (and missing informative ValueErrors). If I set k_moments and k_params correctly, then everything seems to work, even with GMM(y, None, None, ...).

```
>>> model2 = my_GMM2(X, None, None)
>>> res_c2 = model2.fit(start_params=[2.05831079,  0.06579918], inv_weights=np.eye(4), maxiter=2, optim_method='nm', wargs=dict(centered=True))
Optimization terminated successfully.
         Current function value: 0.001803
         Iterations: 30
         Function evaluations: 60
Optimization terminated successfully.
         Current function value: 0.105310
         Iterations: 37
         Function evaluations: 66
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "m:\josef_new\eclipse_ws\statsmodels\statsmodels_py34\statsmodels\sandbox\regression\gmm.py", line 679, in fit
    self._fix_param_names(params, param_names=None)
  File "m:\josef_new\eclipse_ws\statsmodels\statsmodels_py34\statsmodels\sandbox\regression\gmm.py", line 531, in _fix_param_names
    if len(params) < len(xnames):
TypeError: object of type 'NoneType' has no len()
```

user: josef-pkt
issue_2632: state: open
title: ENH: overdispersed Binomial, Beta-Binomial
body: (this showed up already several times, but I didn't see an issue)

see lecture slides at https://www.statistics.ma.tum.de/personen/professorinnen/claudia-czado/forschung/lecture-slides/
for some issues and diagnostics

Note: 
Binomial count data can be over- or under (?) dispersed

Binary, Bernoulli data cannot be over or under dispersed, but could have heteroscedasticity in the underlying latent variable (Stata's hetprobit)

Issue for binomial count data is similar to Poisson, BetaBinomial would be analog of NegativeBinomial.

see also #2890
user: josef-pkt
issue_2631: state: open
title: ENH/SUMM: Gamma regression
body: We have Gamma distribution as family for GLM

Greene 5th edition has Gamma distribution (without explanatory variables) as GMM example with comparison to MLE. The appendix includes an illustration of optimization problems.
(I have now a working version for the GMM, but it requires `nm` as optimizer.)

Czado has some slides on Gamma regression in the GLM setting
https://www.statistics.ma.tum.de/personen/professorinnen/claudia-czado/forschung/lecture-slides/

we have open issues for computational problems with GLM-Gamma.

stats.stackexchange When to use gamma
http://stats.stackexchange.com/questions/67547/when-to-use-gamma-glms

user: josef-pkt
issue_2628: state: open
title: REF: pinv threshold, linear regression for almost singular
body: based on a stackoverflow question

because of numerical precision problems the pinv singular treatment doesn't kick in early enough for essentially singular design matrices.

The below script return huge non-singular numbers for about a third of the runs.
Note: we are printing the warning in `summary()`

for example

```
>>> np.linalg.pinv(mod.exog).dot(mod.endog)
array([ 2.5      ,  3.59375  ,  4.296875 ,  2.9140625,  3.8046875])
>>> np.linalg.pinv(mod.exog, rcond=1.5e-15).dot(mod.endog)
array([ 1.6,  2.6,  3.6,  3.4,  4.4])
```

`qr` produces similar ill determined results

possible solution:
- change threshold for pinv
- provide more methods to handle/detect near singular (see other issues, including penalized and automatic dropping of collinear variables)

```
# -*- coding: utf-8 -*-
"""
Created on Wed Sep 16 00:44:55 2015

Author: http://stackoverflow.com/questions/32599159/robustness-issue-of-statsmodel-linear-regression-ols-python


"""
import numpy as np
import pandas as pd
from statsmodels.formula.api import ols

seed = np.random.randint(100000)
#seed = 62802  #df_model=4, bad numbers
#seed = 46269  #df_model=4, bad numbers38680
print(seed)
np.random.seed(seed)


nbData = 600
rand1 = np.random.uniform(size=nbData)
rand2 = np.random.uniform(size=nbData)
a = 1 * (rand1 <= (1.0/3.0))
b = 1 * (((1.0/3.0)< rand1) & (rand1< (4/5.0)))
c = 1-b-a
d = 1 * (rand2 <= (3.0/5.0))
e = 1-d
weigths = [1,2,3,1,2]
y = a+2*b+3*c+4*d+5*e
df = pd.DataFrame({'y':y, 'a':a, 'b':b, 'c':c, 'd':d, 'e':e})

mod = ols(formula='y ~ a + b + c + d + e - 1', data=df)
res = mod.fit()
print(res.summary())

# try changing pinv threshold
print(np.linalg.pinv(mod.exog).dot(mod.endog))
print(np.linalg.pinv(mod.exog, rcond=1.5e-15).dot(mod.endog))
```

user: josef-pkt
issue_2627: state: open
title: ENH: mediation analysis - triply robust estimators, inverse probability weights
body: Follow-up after readings for #2352

mediation analysis decomposes average treatment effect into a direct and an indirect mediated effect.

Tchetgen, Eric J. Tchetgen, and Ilya Shpitser. "Semiparametric theory for causal mediation analysis: efficiency bounds, multiple robustness and sensitivity analysis." The Annals of Statistics 40.3 (2012): 1816-1845.

adds mediation effect to doubly robust treatment effects estimators and is similar to PR #2455
2 out of 3 equations need to be correctly specified. 

The pattern is the same as in PR #2455 if the mediator is also binary, we get an additional IPW. 
I haven't looked at cases where treatment or mediator are continuous.

Tchetgen et al p. 1826, 1827 for the specification including the 3 moment conditions (if we use GMM like in the ATE case)

Stata doesn't have mediation effect build in AFAIK.
I haven't seen an implementation for Tchetgen at al approach.
R: 
Kerby's #2352 compares to package `mediation`
another approach is followed in package `medflex`

(I only read two papers and skimmed around a bit, but don't have an overview yet.)

user: josef-pkt
issue_2626: state: open
title: FAQ: ANOVA versus linear model (OLS)
body: When are ANOVA and linear model approaches equivalent? 
beyond simple one-way/k-sample comparisons

My guess is that an ANOVA approach is only useful in balanced orthogonal designs.

beyond that we need more specific definitions, e.g. type2, type3, type4 sum of squares or partial effects.

reference I just found

http://www.utdallas.edu/~herve/abdi-contrasts2010-pretty.pdf
http://www.utdallas.edu/~herve/R_comp4Abdi_ExperimentalDesign.pdf

user: josef-pkt
issue_2625: state: open
title: ENH: reduced rank vector models
body: `B_full = M * B_reduced`

example: MNLogit reduced to stereotype Logit, `slogit` in Stata

but this also has the same structure as cointegrating regression (VAR).

see slogit references, and similar multivariate Logit models in R 

user: josef-pkt
issue_2624: state: open
title: ENH: stats proportion_confint   add midp exact confint
body: related to two sample proportion tests, midp are popular, almost maintain size and less conservative than exact.

Add mid-p Clopper-Pearson also to the existing proportion_confint

(Stata uses it in st epitab AFAICS)

I haven't seen formula yet that matches the current beta implementation.

**Edit**
(What does the last sentence above mean?)

There are more possibilities for confidence intervals in proportion_confint, see comment below.

user: josef-pkt
issue_2621: state: open
title: ENH: diagnostic plot partial predict confidence interval
body: trying to evalute nonlinear prediction of a nonparameteric component, e.g. term composed of splines or polynomials.

For diagnostics we can use the linear prediction (even in GLM) since we are interested in nonlinearities in the effect of an individual predictor and not in the nonlinearity coming from the link function.

For the full prediction we can calculate confidence interval for the conditional expectation (of the linear term). 

We can also look at partial prediction that only includes one variable or one term, similar to CCPR plots in the linear model. We also have partial predict functions available or under review (e.g. in GAM PR).

question: Can we get confidence intervals for the partial prediction?

E.g. simple case: use cov_params/t_test just for the corresponding columns.
other possibility: Should we plot the full prediction confidence interval but centered at the partial prediction?

aside: CCPR and similar don't allow for multi-column terms.

user: josef-pkt
issue_2620: state: open
title: MixedLM summary: z or t?
body: In this page:

http://statsmodels.sourceforge.net/stable/examples/notebooks/generated/mixed_lm_example.html

statsmodels and R are compared side by side. However in the summary for fixed effects what R calls t value in statsmodels is labeled as z; accordingly the P>|z| should probably be P>|t|

Is there a reason for that?

user: uliano
issue_2619: state: open
title: ENH: treatment effect for proportions, risk ratio, odds ratio
body: parking a reference

Tu, Chunhao, and Woon Yuen Koh. "Comparison of adjusted and unadjusted marginal probabilities for estimating binomial parameters in observational studies." Journal of Statistical Computation and Simulation 84.12 (2014): 2644-2651.

Not enough details to understand their implementation (requires guessing). Simulation show that unadjusted statistics for proportions are better than inverse probability weighted.

no citation, older articles in this area have few citations.

However, this and their references look useful to make the connection between Logit/GLM and treatment effects/IPW and two sample hypothesis tests (see #2605 )

user: josef-pkt
issue_2610: state: open
title: SUMM: Wrapper / data attach method consistency
body: It seems to me that there are some consistency and naming issues with the way wrapping and metadata attachment are done; I don't know how to address them all at once, but here is a list of current behavior with some notes about issues. The main issues I feel like exist are:
- `attach_*` methods weren't designed with multivariate series in mind; problems include:
  - inconsistent squeezing, some functions not defined for 2d data
  - inconsistent behavior with respect to column names (currently 2d DataFrames get column names, but 1d series do not - this behavior is tested in unit tests, so I'm not sure if anything relies on it)
  - no method for handling tiled or repeated columns (for example cov_params for 3 endog series, should be something like [y1_lower, y2_lower, y3_lower, y1_upper, y2_upper, y3_upper] or [y1_lower, y1_upper, y2_lower, y2_upper, y3_lower, y3_upper]).
- I'm not sure what the future of the `attach_*` functions are - I didn't want to add new ones for my uses because it seems like a proliferation of one-off attach functions isn't a good idea.
- `wrapper.py` inconsistent behavior with respect to passing arguments. Easy fix to match existing behavior (if that's what is desired).

It's also possible that I just don't fully understand the design plan of metadata attaching; if that's so, I'd be grateful for that information also.
#### `wrapper.py`

Here, just a consistency issue: wrappers for attributes allow passing arguments in the wrapper specification (see  https://github.com/statsmodels/statsmodels/blob/32e739dfb7beed07e1f94eda631ade301c01e42f/statsmodels/base/wrapper.py#L38) but this is not allowed for method wrappers (https://github.com/statsmodels/statsmodels/blob/32e739dfb7beed07e1f94eda631ade301c01e42f/statsmodels/base/wrapper.py#L92).

Presumably this is just because someone needed it for attributes (`attach_general_columns`) but not for methods; it's an easy fix.
#### Current `attach_*` methods

I feel like there is room for improvement here in terms of consistency and functionality.

The following `attach_*` methods exist (and only do anything if we have Pandas data):

**`attach_columns`**
Returns a series or DataFrame, with parameter names (`self.param_names`) as index (row names). Used in:
- `base/model.py` to wrap the following attributes: `params, bse, pvalues, tvalues, conf_int`
- `regression/linear_model.py` to wrap the following attributes: `chisq, HC0_se - HC3_se`

**`attach_columns_eq`**
Returns a DataFrame with exog names as index (row names), ynames as columns. Used only in `var_model.py` for attributes: `bse, params, pvalues, tvalues, stderr`

**`attach_cov`**
Returns a DataFrame with parameter names as both column and row names. Used only for `cov_params` in various models.

**`attach_cov_eq`**
Returns a DataFrame with ynames as both column and row names. Used only in `var_model.py` for attribute `sigma_u, sigma_u_mle`.

**`attach_rows`**
Returns a Series or DataFrame with row labels attached. Used in:
- `base/model.py` for `resid, fittedvalues`
- `discrete/discrete_model.py` for `resid_*` attributes
- `genmod/generalized_estimating_equations.py` for `centered_resid`
- `genmod/generalized_linear_model.py` for `resid_*` attributes
- `regression/linear_model.py` for `sresid`, `weights`, `wresid`, `norm_resid`

It has a couple of issues with it:
- Inconsistent squeezing
- The DataFrame version sets the column names to ynames, the Series version does not set a `name`.

**`attach_dates`**:
Returns a TimeSeries object with `self.predict_dates` as index.

**`attach_generic_columns`**
Returns a Series object with index (row names) equal to the value of an arbitrary attribute of the data object, where the attribute name is passed as an additional argument to the wrapper specification (see below) Used in:
- `regression/mixed_linear_model.py`: for `bse_re`, `fe_params`, `bse_fe`

**`attach_generic_columns_2d`**
Returns a DataFrame object with with index (row names) equal to the value of an arbitrary attribute of the data object, where the attribute name is passed as an additional argument to the wrapper specification. In principle it allows the specification of different column names, but in practice all wrapping code will use the row names as the column names too. Used in:
- `regression/mixed_linear_model.py`: for `cov_re` and `cov_re_unscaled`

user: ChadFulton
issue_2608: state: open
title: WIP ENH: exact test equality of two proportions
body: see #2607 

This PR implements an unconditional exact test for the equality of two independent proportions, either based on full max or on Berger/Boos confidence interval test.

verified and unit tested with published example and website with online calculation at print precision.

work in progress, PR so I don't forget this and can go back to other things.

main Todos
- API
- optimize code, currently unit test are a bit slow
  - streamline code: currently expensive repeated or unnecessary calculations
  - use special properties of test like monotonicity to shortcut calculations
  - better optimization than pure grid search, looks smooth in example but many local maxima
    I didn't see any spikes yet that are described in several references. 

extend:
- basic extension is to allow for non-zero difference in the null

user: josef-pkt
issue_2607: state: open
title: ENH: (almost) exact hypothesis tests for proportions and contingency tables
body: "Never use Fisher's exact test !"

follow-up to #2605 confidence intervals for two sample proportions

unconditional exact tests guarantee the size (less or equal to nominal size) but are more powerful, i.e. less conservative in most cases than conditional exact tests like Fisher's exact.

Exact tests use the (assumed) population distribution, either binomial or multinomial or similar, to evaluate the probability of observing samples that have an at least as extreme test statistic as the observed one.

Aside: I haven't seen it mentioned anywhere yet. Assuming binomial or multinomial distribution is not innocent, the distributional assumption could be incorrect if we have for example over dispersion because of unobserved heterogeneity and/or correlation (with possible under dispersion). Some examples (applications) sound at least "fishy" to my (e.g. incidence counts of contagious disease.)

As prototype I have test for equality of two independent proportions, similar to http://www4.stat.ncsu.edu/~boos/exact/   (results identical at the 4 displayed decimals.)
(currently brute force only and not optimized, arrays in size of cardinality of sample space)
- 1. asymptotic (score) test
- 2. exact distribution but nuisance parameter replaced by constraint MLE
- 3. sup/max over space of nuisance parameters, or Berger and Boos subspace (confidence interval), max of 1.
- 4. E+M Loyd and Shan articles (not implemented yet), max of 2. 

variation: using different test statistics in 1. for 2., 3., and 4.

other approximations: 
- mid-p p-values
- small sample and continuity corrections   (using Yates is even worse than Fisher's exact)

target sample size
- with some optimizations we should be able to handle unconditional exact for small to moderate sample, up to a few hundred observations
- for larger samples we can use some approximations, or bootstrap simulations instead of full enumeration.

target null and alternative
- testing for equality is relatively simple, 2 values are the same under the Null
- testing for pre-specified difference is needed for inferiority/superiority and equivalence tests and for (score) confidence intervals. analytical expression for constraint MLE are in the literature (roots of low degree polynomial)
- alternatives one and two sided

target tests and sampling
- 2 independent proportions - 2x2 contingency table with one fixed margin
- paired sample - reduced sample space because only divergent results matter
- 2x2 multinomial - only total number of observations fixed
- larger than 2x2: 2xK, 2x2xK, ... sample space blows up fast
- Poisson: exact comparison of intensities - no fixed sample size  (article with Berger/Boos sup test)

target results

we should get for simple tests what we have in t_test/wald_test after estimating models, plus power and sample size calculation.
- effect size
- pvalue
- confidence interval of effect size
- maybe asymptotic standard error of effect size
- power

connection to models
- analogous tests and results based on asymptotics in GLM and binary discrete models.

...

(my short wishlist)

user: josef-pkt
issue_2606: state: open
title: ENH: Design   score confidence interval
body: We can construct confidence intervals by inverting score tests.
This is the third type beside Wald test and LR (or profile LR) test based confidence intervals.

Agresti argues that score CI are in many case better than Wald CIs. This is similar to the observation that score tests are in many cases better sized than Wald tests.

Agresti, Alan. 2011. “Score and Pseudo-Score Confidence Intervals for Categorical Data Analysis.” Statistics in Biopharmaceutical Research 3 (2): 163–72. doi:10.1198/sbr.2010.09053.
http://www.tandfonline.com/doi/abs/10.1198/sbr.2010.09053

(found while looking at confidence intervals for two proportions, odds and risk ratio in #2605 )

related #2041 SUMM issue for score tests (and more general conditional moment tests)

note for implementation: 

Score CI might not be very cheap because we need to calculate the restricted models to find the threshold parameter values.
It might be ok for simple restrictions (Similar to inverting LR tests) but more difficult if we have more complicated, possibly nonlinear restrictions. Odds ratio might still be just a simple restriction on parameters in Logit, but maybe already nonlinear because of exp(beta).

user: josef-pkt
issue_2599: state: open
title: SUMM: frequency weights, contingency tables, exog with discrete support
body: objective: better support for datasets that only include categorical `exog`

motivation: examples like in this http://data.princeton.edu/wws509/stata/c6s5.html have only categorical exog where the data is given in the form of contingency tables.

This would be relatively easy to support if the models had frequency weights `fweights`.
Related support for the data is mostly available in pandas.

related issues:

...

user: josef-pkt
issue_2596: state: open
title: ENH: optimization and nested models
body: (just an idea for the wishlist)

example single index models or GLM with nonparametric links or GAM with linear components inside.

`E(y | x) = link_inv(sum_k(f_k(X_k * b_k)))`

link_inv is a known or fixed mean or inverse link function to map to domain of distribution parameter.
f_k  non or semiparametric function, e.g. in spline or polynomial basis
`X_k * b_k`  linear predictor for k'th component

This is just nested linear models, assuming f_k use linear basis space, with a GLM link on top.
Could be estimated by GAM with parameter restrictions, or as nested nonlinear model with linear components in each stage besides top link.  (nested irls or penalized irls?)

user: josef-pkt
issue_2591: state: open
title: ENH: more tsa plots
body: Do we have a plot for spectral density or periodogram?

I wrote a 4 axis plot a long time ago with scatter of time series, acf, pacf and spectral density/periodogram. 

???

user: josef-pkt
issue_2588: state: open
title: SUMM: diagnostic tests, measures and plots
body: (this is another topic that is missing consistent implementation across models.)
(I might add a `topic_diagnostic` label, but don't want to go through issues right now. Search shows many open isssues.)

This is mainly about checking underlying assumption and specification tests.
Postestimation in general is more about interpretation of model.

**main tasks, design**
- implementation of functions and methods across models (or check generalization and applicability)
- easy access for users (e.g. `get_diagnostics`, `get_plots`, `check_sanity`)
- documentation (suggest possible remedies? maybe not that)

main issues by models
- linear regression: ok
- tsa: PR by Chad for some models
## \- GLM/discrete: essentially nothing

issues by topic

...

user: josef-pkt
issue_2586: state: open
title: ENH: deterministic starting sets for robust/resistant estimators
body: (parking an issue and reference)

FastXXX methods for resistant estimators use random starting sets (implemented in PR)

Recent research showed how to select a few deterministic datasets instead

early on are articles by Olive, and a few others

now there is also:

Mia Hubert, Peter Rousseeuw, Dina Vanpaemel, Tim Verdonck, The DetS and DetMM estimators for multivariate location and scatter, Computational Statistics & Data Analysis, Volume 81, January 2015, Pages 64-75, ISSN 0167-9473, http://dx.doi.org/10.1016/j.csda.2014.07.013.
(http://www.sciencedirect.com/science/article/pii/S0167947314002175)

user: josef-pkt
issue_2585: state: open
title: ENH: out of core GAM
body: for wishlist, just parking a reference

Wood, S. N., Goude, Y. and Shaw, S. (2015), Generalized additive models for large data sets. Journal of the Royal Statistical Society: Series C (Applied Statistics), 64: 139–155. doi: 10.1111/rssc.12068
http://onlinelibrary.wiley.com/doi/10.1111/rssc.12068/abstract

function bam in R mgcv

https://stat.ethz.ch/R-manual/R-devel/library/mgcv/html/bam.html

(sounds like just a special case of out-of-core GLM fitted with PIRLS)

user: josef-pkt
issue_2583: state: open
title: ENH: diagnostics  AR(1) for unequal spaced points
body: (Just an idea, I have not seen it in any references.)

To identify or check for local nonlinearities in the residual, we could calculate a local AR(1) coefficient for series with unequal spaced time points.  "Time points" would be just a sorted array of an `exog`.

Idea is similar to Durbin Watson statistic for time series.

Besides a global nonlinearity measure, it could also be useful for locally adaptive non or semi parametric regression, splines, local polynomial or kernel regression.

I was looking at knot selection in splines with unequal local curvature.

Code is or was in GEE for continuous time AR(1) coefficient estimation.  More general spatial correlation as in pysal.

user: josef-pkt
issue_2582: state: open
title: Distributed Lag models for GLM
body: Gasparrini, A., B. Armstrong, and M. G. Kenward. "Distributed lag non-linear models." Statistics in medicine 29.21 (2010): 2224.

https://scholar.google.ca/scholar?cites=1113922094832206321&as_sdt=2005&sciodt=0,5&hl=en
with R package dlnm https://cran.r-project.org/web/packages/dlnm/index.html

I didn't know distributed lag models are still pretty popular for applied work.

another (new and now citation but with R package)
Obermeier, Viola, et al. "Flexible distributed lags for modelling earthquake data." Journal of the Royal Statistical Society: Series C (Applied Statistics) 64.2 (2015): 395-412.
gated article that I don't have access to. thesis of Obermeier is ungated
Main point, AFAICS, is that they write it on top of `mgcv` for GAM and splines, which should be possible to immitate after our GAM/splines merge.

user: josef-pkt
issue_2580: state: open
title: SUMM/Follow-up: GAM and penalized splines
body: #2435 PR with development discussion

**update**
#5370 latest version of GAM PR
#5296 previous PR with comments on most of the changes to get it to work correctly


## TODO:

- [ ] k-fold cross-validation needs more careful review and refactoring


- [ ] check and enable recreating model, e.g. llnull currently fails in GAM for discrete Logit (generic failure in penal ?) We also might want to have a new instance of the model for cross-validation or other penalization selection.  
old item, I'm not sure what the status of this is


## Extension
- [ ] tensor splines
- [ ] Wood's confidence intervals
- [ ] simultaneous confidence intervals (Kerby general PR and comment)

user: josef-pkt
issue_2577: state: open
title: Add method for generating ARMA predictions for new data, unrelated to training data
body: It seems that the only way to use a trained ARMA model on new data is to use this internal function:  `_arma_predict_out_of_sample`. Furthermore, it's not obvious how to do so. 

I would assume that for an ARMA(p,q) model, there are max(p.q) previous datapoints required to generate a new prediction. So If I had an AR(2,3) model I'd need to pass in 3 datapoints to get one new output sample.  There is also some information about past "residuals" that is not completely clear how to generate. An API like the following (perhaps with extra required parameters such as residuals....) would be quite useful to me:

`def arma_model.predict_new_data(prev_datapoints, num_samples_to_generate=1)`

where `prev_datapoints` would be a list of length 3, and `if num_samples_to_generate > 1:` then the "dynamic" stuff would be turned on automatically.

See:
http://stackoverflow.com/questions/18616588/arma-out-of-sample-prediction-with-statsmodels
http://stats.stackexchange.com/questions/76160/im-not-sure-that-statsmodels-is-predicting-out-of-sample

user: eraoul
issue_2576: state: open
title: ENH diagnostic tests (het_xxx) - input checks and constant
body: see mailing list https://groups.google.com/d/msg/pystatsmodels/PkG7l7CuxiA/5WzhQl56JQAJ

Currently het_white and het_breusch_pagan assume that there is a constant in the exog.

het_breusch_pagan also assumes full rank exog, even though we could check the rank in the auxiliary regression, use df_model as in het_white.

more robustness to user input would be relatively costless to add, since we get the information from the auxiliary regression.

user: josef-pkt
issue_2572: state: open
title: FAQ/ENH: cross validation performance measures (GLM)
body: So far we have very little support for cross-validation build in. GAM PR #2435 adds cross-validation for penalized GLM.

related issues #1577 #2027 #1282
https://github.com/statsmodels/statsmodels/pull/2435#issuecomment-127379173

General question:  What are the performance criteria for general model, especially generalized linear models?

We have `tools.eval_measures` which only includes measures targeted toward continuous variables.
sklearn http://scikit-learn.org/stable/modules/classes.html#module-sklearn.metrics has a large number for classification and others, but not count variables.

a few links to related discussions
https://stat.ethz.ch/R-manual/R-devel/library/boot/html/cv.glm.html doesn't say much about `cost`, use thresholded 0-1 prediction for Binomial/Binary
http://stats.stackexchange.com/questions/48811/cost-function-for-validating-poisson-regression-models related question for Poisson, comment mentions `cv.glmnet` uses deviance
http://stackoverflow.com/questions/23904179/calculate-cross-validation-for-generalized-linear-model-in-matlab  matlab question, not discussion about `cost` function (example for interface)
http://www.uni-kiel.de/psychologie/rexrepos/posts/crossvalidation.html binary only uses Brier score
(looks like a nice collection of examples)

**general**
(my intuition, no references, I don't remember reading anything specific)

objective: estimation of model versus prediction

In prediction like scikit learn or Kaggle competition, the objective for cross-validation is based on a subject/application specific cost/utility function. This does not need to be consistent with the objective function of the estimator. (But then maybe we should change the objective of the estimator also for parameters and not just for hyper parameters.)

For model estimation, I think we should use the same criteria for the hyper parameters as for the standard parameters. In the GLM case this would be deviance or loglike. In the linear model case it is mean squared error which is equivalent to the loglike (for fixed standard deviation ?).
Another possibility (I'm not sure) would be to look at the score_obs/estimating equations.

**Implementation for GLM**

In general, we don't have access to loglike_obs to evaluated at observations that are not in the estimation sample. In the case of GLM, we have it defined in the `family` which, I think, could be easily reused.

user: josef-pkt
issue_2571: state: open
title: ENH: auto ARIMAX,  structural breaks and diagnostics
body: A brief search on forecasting in TSA found this

http://stats.stackexchange.com/questions/68261/performance-evaluation-of-auto-arima-in-r-and-ucm-on-one-dataset
where auto.arima fails because of structural breaks, and a bug in R arima in initialization.

related:
order and differencing selection in ARMA: #1224 #1557

However, this could be extended to checking for outliers and structural breaks.
@ChadFulton  is adding diagnostic tests for SARIMAX and statespace models, including a structural break test.
The Oxford econometricians, Hendry, ..., use search over sets of dummy variables to identify outliers. Now, with working exog in ARIMAX it would be relatively easy to add diagnostics checks and tests through exog (with Wald tests, for example).

(more elaborate: use penalized MLE to do variable selection on outlier dummies and trend breaks or similar, another candidate for SCAD penalized MLE)

user: josef-pkt
issue_2569: state: open
title: ENH: weightstats CompareMeans add from_data and summary methods
body: see http://stackoverflow.com/questions/31768464/confidence-interval-for-t-test-difference-between-means-in-python

to make the interface more convenient in weightstats CompareMeans, we could add `from_data` and `summary` methods.

summary could print a summary table (1 row) similar to `LikelihoodResults.t_test` and `Contrast`

Note: weightstats CompareMeans was written before Contrast results gained the summary and other enhancements, and we never checked for adding similar improvements.

user: josef-pkt
issue_2568: state: open
title: Absorbing fixed effects
body: The absorption technique for estimating models with fixed effects that is available in Stata and SAS is not currently part of statsmodels. It is quite useful when one wants to include many fixed effects but is not interested in their individual estimates.

The Stata documentation on this technique ("areg") is [here](http://www.stata.com/manuals13/rareg.pdf). Discussion of the SAS implementation is [here](http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#statug_glm_sect032.htm).

See also @josef-pkt's discussion [here](https://groups.google.com/d/msg/pystatsmodels/KDgh7_O7vkw/n04NOWO9IwAJ). 

user: stoffprof
issue_2564: state: open
title: SUMM/Design: State space: Postestimation
body: In some ways this is a state space companion to #2150.

We probably want to do the following:
1. [x] Implement a `get_prediction` / `get_forecast` or similar (I'm not sure what conclusions were reached in #2150 / #719) to get a full results object with confidence intervals, etc. (rather than the point predictions which are given by `predict` and `forecast`).
2. [ ] Implement a new set of functions for impulse response functions (IRFs, FEVDs, dynamic multipliers, etc. as appropriate)
3. [ ] Figure out how to better implement a wide range of diagnostic tests (i.e. rather than just adding every test possible as a new method to the results object).

user: ChadFulton
issue_2559: state: open
title: ENH/TST: multivariate time series analysis
body: I just saw this looking for multivariate autocorrelation

should be useful for references and unit tests

https://cran.r-project.org/web/packages/MTS/

```

Version:    0.33
Imports:    Rcpp, fGarch, mvtnorm
LinkingTo:  Rcpp
Published:  2015-02-12
Author: Ruey S. Tsay
Maintainer: Ruey S. Tsay <ruey.tsay at chicagobooth.edu>
License:    Artistic License 2.0
NeedsCompilation:   yes
Materials:  README
In views:   TimeSeries
CRAN checks:    MTS results
```

user: josef-pkt
issue_2557: state: open
title: FAQ/Design: centering confidence intervals (acf, correlation, ..)
body: the acf confidence interval is centered around the actual values instead of around zero.
However, AFAIU, the confidence interval is only valid for the assumption that the correlation of the stochastic process is zero.

see comments in
https://github.com/statsmodels/statsmodels/pull/2552#issuecomment-124016973

similar story for pearson correlation

user: josef-pkt
issue_2551: state: open
title: SARIMAX or ARIMA in python doesnt handle NANs in exogenous variables
body: I am not sure if its just part of the source code that doesnt handle NANs but I've noticed if I input a variable with NANs in a dataframe as part of the exogenous variables, it will not properly fit the seasonal model.

exog_df = pd.DataFrame([])
exog_df['SFO3'] = Blodgett_gs_residual_df['SFO3']
exog_df['VPD'] = Blodgett_gs_residual_df['VPD']       #   this variable has NAN values inside the column
exog_df['PAR'] = Blodgett_gs_residual_df['PAR']
exog_df['Ta'] = Blodgett_gs_residual_df['Ta']
exog_df.index = Blodgett_gs_residual_df.index

wue_model_data = sm.tsa.SARIMAX(endog=Blodgett_gs_residual_df['WUE'],exog=exog_df,order=(2,15,1)).fit()

  File "/Users/JasonDucker/Documents/forJason/plot_blodget_demo.py", line 404, in <module>
    wue_model_data = sm.tsa.SARIMAX(endog=Blodgett_gs_residual_df['WUE'],exog=exog_df,order=(2,15,1)).fit()#,time_varying_resgression=True,mle_regression=False,enforce_stationarity=False,enforce_invertibility=False).fit()

  File "/Users/JasonDucker/anaconda/lib/python2.7/site-packages/statsmodels-0.7.0-py2.7-macosx-10.5-x86_64.egg/statsmodels/tsa/statespace/mlemodel.py", line 182, in fit
    start_params = self.start_params

  File "/Users/JasonDucker/anaconda/lib/python2.7/site-packages/statsmodels-0.7.0-py2.7-macosx-10.5-x86_64.egg/statsmodels/tsa/statespace/sarimax.py", line 902, in start_params
    params_exog = np.linalg.pinv(exog).dot(endog)

  File "/Users/JasonDucker/anaconda/lib/python2.7/site-packages/numpy/linalg/linalg.py", line 1585, in pinv
    u, s, vt = svd(a, 0)

  File "/Users/JasonDucker/anaconda/lib/python2.7/site-packages/numpy/linalg/linalg.py", line 1327, in svd
    u, s, vt = gufunc(a, signature=signature, extobj=extobj)

  File "/Users/JasonDucker/anaconda/lib/python2.7/site-packages/numpy/linalg/linalg.py", line 99, in _raise_linalgerror_svd_nonconvergence
    raise LinAlgError("SVD did not converge")

LinAlgError: SVD did not converge

Anyone know a fix to this problem with the SARIMAX or ARIMA function? Thanks for the time and help!

user: jad10d
issue_2549: state: open
title: ENH: summary for additional or auxiliary results
body: Some models use auxiliary estimates that are not always included in summary.

example GLSAR doesn't show AR coefficients because it uses the inherited RegressionResults without changes. Same for GLSHet.   see also #1805
Do we show pearson_chisquare or scale estimate in GLM summary?

models where auxiliary estimates are always included like GEE have the extra information in summary.

user: josef-pkt
issue_2548: state: open
title: ENH/Design history class for iterative solvers in models
body: https://github.com/statsmodels/statsmodels/pull/2269/files#r34956113

A possible alternative would be to use a simple history subclass of python list that has a as_dataframe attribute, which could also add rho and maybe other things to the history DataFrame. But this would have to be introduced consistently for all histories in models with specialized iterative solvers.

user: josef-pkt
issue_2538: state: open
title: EX/DOC: movie ratings as a Binomial or proportion regression
body: Movie ratings would be a nice example for Binomial and for QMLE with binary or Bernoulli models. As far as I have seen it's used by several data science or online courses.

see http://stackoverflow.com/questions/31420175/pandas-patsy-statsmodels-linear-reg-issue-passing-in-categorical-variable-d  and my answer for the idea.

related: There should be issues for QMLE with Logit and Probit. I find #2074
Stata 14 added specific models for this, and also added Beta regression. 
I have the examples "somewhere", when I looked at Beta regression
Reference Wooldridge and others

user: josef-pkt
issue_2527: state: open
title: ENH: GenericLikelihoodWrapper for scipy.stats distribution fit
body: see http://stackoverflow.com/questions/31359017/how-to-get-error-estimates-for-fit-parameters-in-scipy-stats-gamma-fit/31365801#31365801

my current script:

```
# -*- coding: utf-8 -*-
"""
Created on Sun Jul 12 03:44:40 2015

Author: Josef Perktold
License: BSD-3

"""

import numpy as np
from statsmodels.base.model import GenericLikelihoodModel

from scipy.stats import gamma
shape = 12; loc = 0.71; scale = 0.0166
data = gamma.rvs(shape, loc=loc, scale=scale, size=1000)
params = gamma.fit(data) # params close to but not the same as (shape, loc, scale) 
# HOW TO ESTIMATE/GET ERRORS FOR EACH PARAM?

print(params)
print('\n')


class Gamma(GenericLikelihoodModel):

    nparams = 3

    def loglike(self, params):
        return gamma.logpdf(self.endog, *params).sum()


res = Gamma(data).fit(start_params=params)
res.df_model = len(params)
res.df_resid = len(data) - len(params)
print(res.summary())
print('\n')

res.t_test(([1, 0, 0], [10]))
res.t_test('par0=10')

import statsmodels.api as sm
res_glm = sm.GLM(data, exog=np.ones(len(data)), 
                 family=sm.families.Gamma(link=sm.families.links.identity)).fit()
print(res_glm.summary())

print(gamma.mean(*params))
print(gamma.var(*params))
```

GLM Gamma only has mean parameter and separately estimated scale parameter. Note: Pearson chisquare as scale estimate does not correspond to the variance of endog. 

user: josef-pkt
issue_2516: state: open
title: Generalized Linear Mixed Model
body: Poisson and Binomial with cannonical links working. Non cannonical links not working yet. Gaussian not working. 

~~**edit (Josef)**~~
- ~~we need to squash the first 4 commits before merge, up to 0685f2f572115230e2076c676884fc00787f0a5d, to remove redundant copy of modules.~~

user: gjhunt
issue_2508: state: open
title: ENH: confidence interval for Logit, seaborn
body: I haven't looked at the details. This is mainly a reference to a (down stream) use case.

seaborn has bootstrap confidence intervals for logistic regression, which is slow. We should have them available without bootstrap (via endpoint transformation of linear prediction confidence intervals)

https://github.com/mwaskom/seaborn/issues/347#issuecomment-60974777
http://tomwallis.info/2014/10/29/high-level-plotting-in-python/

user: josef-pkt
issue_2505: state: open
title: Source of Anderson-Darling statistic formulas?
body: It would be nice if the source of the Anderson-Darling approximations formulas was documented.
Thanks.

user: anntzer
issue_2503: state: open
title: SUMM: old pull requests
body: An inventory of PRs that have been waiting for a long time
### The biggies
- [ ] #304 Nonlinear LS
- [ ] #361 sysreg -> rebased in #2500 
- [ ] #452 robust new estimators
- [ ] #786 repeated measures anova
- [ ] #1120 Conditional Logit and discrete choice -> rebased #1605
- [ ] #1133 Panel Data
- [ ] #1341 robust cleanup, (nonlinear) M-estimators ?
- [x] #1489 exponential smoothing
- [x] #1665 Theil/Ridge penalized linear model
- [ ] #1673 Restricted GLS (not so big)
- [ ] #2139 Tobit rebased

and more recent ones

missing

nonlinear tsa Chads first GSOC
### graphics
- [ ] #1388 facet plots
### major refactoring and enhancements
- [ ] #308 GOF in sandbox
- [ ] #973 #1678 #1687 nonparametric
- [ ] #1433 probability plots
### several smaller functions and classes

BDS test, Johansen cointegration, ...

---

the rest is 2014 or newer

more in sandbox which also has outdated code and models, that should be deleted or replaced
.

user: josef-pkt
issue_2502: state: open
title: ENH: State space models with HAC covariance matrices
body: Placeholder / question issue. I don't have a citation for this yet, but would be interested if anyone else runs across one.

user: ChadFulton
issue_2501: state: open
title: ENH: optimization model attribute for complex_step ok
body: In margins we use complex step numdiff in a try except clause.

Doing this in optimization is too expensive, but we could store an attribute after the first try.

(random thought for general nonlinear models nlls, ...)

user: josef-pkt
issue_2500: state: open
title: Sysreg rebased
body: A rebased version ot the sysreg PR #361

One hacky extra commit (I might clean it) for getting `example_sysreg3.py` and unit tests to run.

I didn't check anything else, may conflicts during rebase, that I guess I fixed correctly.

Might still not be fully python 3 compatible, especially examples

user: josef-pkt
issue_2495: state: open
title: REF/ENH: GMM refactor cov_type
body: GMMResults defines `cov_params` method that is inconsistent with inherited method and breaks t_test. #2446 

just using the inherited cov_params and renaming the current GMMResults cov_params breaks the unit tests. 

The missing piece is an official way to set the `cov_type` and the corresponding `cov_params_default`. The best, I guess, would be to follow the newer pattern with `cov_type` in `fit`.

The question is what keywords to use. The structure for the options is rather different from the (quasi)-likelihoodmodels
- default GMM: We always have sandwiches, definition depends on whether we used the optimial weighting matrix, and what we use for the covariance of the moment conditions ("meat").

cases:
- (assumed) optimal weighting matrix -> cov_type needs to follow the same options as weighting matrix (no extra options available ?)
- exactly identified -> weighting matrix is irrelevant (except for numerical properties),  we need assumptions for covariance of moment conditions.
- suboptimial weighting matrix -> need separate options for weighting matrix and cov_type. cov_type is "double sandwich" and we need assumptions for covariance of moment conditions.

This means we need a separate cov_type argument, additional to weighting matrix (`weights_method`) and `has_optimal_weights`, for the latter two cases.

and we need `weights_kwds` and `cov_kwds` for cluster and hac, and panel, ...

(Stata gmm has separate `wmatrix(wmtype, wmoptions)` and `vce(vcetype, vceoptions)`)

user: josef-pkt
issue_2494: state: open
title: Please add support to datetime in regressions
body: I noticed, it's not possible to do a simple OLS when the independent variable is a series of datetime points.

This makes unreasonably hard to add a trend line to a time series, which should be a trivial thing...

user: fccoelho
issue_2491: state: open
title: REF/ENH generic cov_type handling, use _get_robustcov_results method
body: https://github.com/statsmodels/statsmodels/pull/2356#issuecomment-117770227

(copied here, there is more discussion on implementing a different set of cov_types in statespace MLE models)

There is a bug/mistake/limitation in LikelihoodModelResults.**init**
This should check first for an available _get_robustcov_results or get_robustcov_results method before using the generic one.

I works for example for RegressionResults, because that doesn't put cov_type as keyword argument into the super(...).**init** call.

user: josef-pkt
issue_2485: state: open
title: REF: Quantile Regression: handling of zero residuals is asymmetric
body: currently we treat zero residuals as positive

after fixing #2377 zeros are moved to the positive part, then they are multiplied by (1-q) in the `where`, which already had a weak inequality.

```
mask = np.abs(resid) < .000001
resid[mask] = ((resid[mask] >= 0) * 2 - 1) * .000001
resid = np.where(resid < 0, q * resid, (1-q) * resid)
```

Given that we don't use linear programming in Quantile Regression, this will be only for the corner case when we have many exactly fitting observations because of the underlying data. (AFAICS)

In #2377 I suggested to use penalization instead, but this won't be in that fix, because I would have to write more unit tests to think about it. or verify it.

(Related: similar to the smoothed L1 penalization, I would like to get a smoothed version of Quantile Regression into RLM.)

user: josef-pkt
issue_2484: state: open
title: REF/Style: avoid Stata names
body: I just saw that Stata doesn't like first authors more generally

beside adfuller, there is also `hsheather` for the bandwidth method in QuantReg (Hall is not the first name of Sheather, I guess)

user: josef-pkt
issue_2483: state: open
title: FAQ/BUG: Quantile regression: small sample, bse are nan
body: found while investigating #2377 (with the proposed fix replacing sign by `>=` )

the standard errors are nan, and there are RuntimeWarnings related to the kernel estimation.

What's the minimum size for the kernel calculation of cov_params?
the following needs to be smaller than q, e.g. q = 0.5
`chamberlain` and `bofinger` require at least 4 observations, `hall_sheather` (`hsheather`) needs 8

```
>>> import statsmodels.regression.quantile_regression as qr
>>> qr.chamberlain(4, 0.5)
0.48999099613501351
>>> qr.hall_sheather(8, 0.5)
0.48577951310258921
>>> qr.bofinger(4, 0.5)
0.49086103316125856
```

```
import numpy as np
from statsmodels.api import QuantReg

X = np.array([[1, 0], [0, 1], [0, 2.1]], dtype=np.float64)
y = np.array([0, 1, 2], dtype=np.float64)

res = QuantReg(y, X).fit(0.5)

X = np.array([[1, 0], [0, 1], [0, 2.1], [0, 3.1]], dtype=np.float64)
y = np.array([0, 1, 2, 3], dtype=np.float64)

res = QuantReg(y, X).fit(0.5)
```

prints:

```
m:\josef_new\eclipse_ws\statsmodels\statsmodels_py34\statsmodels\regression\quantile_regression.py:238: RuntimeWarning: invalid value encountered in less_equal
  kernels['epa'] = lambda u: 3. / 4 * (1-u**2) * np.where(np.abs(u) <= 1, 1, 0)
m:\josef_new\eclipse_ws\statsmodels\statsmodels_py34\statsmodels\regression\quantile_regression.py:208: RuntimeWarning: invalid value encountered in double_scalars
  d = np.where(e > 0, (q/fhat0)**2, ((1-q)/fhat0)**2)
>>> print(res.summary())
                         QuantReg Regression Results                          
==============================================================================
Dep. Variable:                      y   Pseudo R-squared:               0.9839
Model:                       QuantReg   Bandwidth:                         nan
Method:                 Least Squares   Sparsity:                          nan
Date:                Mon, 29 Jun 2015   No. Observations:                    4
Time:                        10:32:09   Df Residuals:                        2
                                        Df Model:                            2
==============================================================================
                 coef    std err          t      P>|t|      [95.0% Conf. Int.]
------------------------------------------------------------------------------
x1                  0        nan        nan        nan           nan       nan
x2             0.9677        nan        nan        nan           nan       nan
==============================================================================
m:\josef_new\eclipse_ws\statsmodels\statsmodels_py34\statsmodels\stats\stattools.py:72: UserWarning: omni_normtest is not valid with less than 8 observations; 4 samples were given.
  "samples were given." % int(n))
C:\programs\WinPython-64bit-3.4.3.1\python-3.4.3.amd64\lib\site-packages\scipy\stats\_distn_infrastructure.py:1728: RuntimeWarning: invalid value encountered in greater
  cond1 = (scale > 0) & (x > self.a) & (x < self.b)
C:\programs\WinPython-64bit-3.4.3.1\python-3.4.3.amd64\lib\site-packages\scipy\stats\_distn_infrastructure.py:1728: RuntimeWarning: invalid value encountered in less
  cond1 = (scale > 0) & (x > self.a) & (x < self.b)
C:\programs\WinPython-64bit-3.4.3.1\python-3.4.3.amd64\lib\site-packages\scipy\stats\_distn_infrastructure.py:1729: RuntimeWarning: invalid value encountered in less_equal
  cond2 = cond0 & (x <= self.a)
```

user: josef-pkt
issue_2482: state: open
title: FAQ/Design: OLS fvalue if there is no constant
body: followup to #2477.

If cov_type is not `nonrobust`, then we use a waldtest to calculate the fvalue.

If there is no constant, not even an implicit constant, then the fvalue is calculated as a wald test that **all** coefficients are zero, i.e. the Null hypothesis is that the `endog` is zero.

Is this desired?

(after #2477, we set fvalue to nan if there is an implicit constant.)

related general question: What do we do about consistency across models?
All or most likelihood models calculate llnull independently of the presence of a constant. 
If there is no (explicit or implicit) constant in the model, then the Null model, with constant only, is not nested in the in the full model.

user: josef-pkt
issue_2481: state: open
title: ARIMAX with innovations from a non-Gaussian distribution
body: A nice extension to the ARIMAX fit functionality would be to be able to choose a non-Gaussian distribution for the innovations (for example a t-distribution).

user: MishaVeldhoen
issue_2474: state: open
title: FAQ: ARIMAX versus Stata
body: When estimating an ARIMAX(p, 1, q), Stata differences also the `exog` variables. The statsmodels version does not difference the `exog`. 

In order to replicate the behavior of Stata, we need to `diff` the exog ourselves.
Note: When we diff, then we need to preserve the initial observation which is `nan`, since it will be truncated during estimation.

Numpy np.diff drops the invalid initial observation. pandas DataFrame diff keeps the initial observation as missing.

The following replicates the Stata results. (I'm using ndarrays in the model, but the same should work with pandas.

```
mod111 = sm.tsa.ARIMA(np.asarray(data_sample['loginv']), (1,1,1), 
                   #exog=np.asarray(data_sample[['loggdp', 'logcons']]))   # exog in levels
                   exog=np.asarray(data_sample[['loggdp', 'logcons']].diff()))

res111 = mod111.fit(disp=1, solver='bfgs', maxiter=5000)
exog_full_d = data[['loggdp', 'logcons']].diff()
res111.predict(start=197, end=202, exog=exog_full_d.values[197:])
```

user: josef-pkt
issue_2469: state: open
title: REF: pickling formula information using patsy
body: This is a follow up to #1263 

We "fixed" the formula pickling by rebuilding from the model (exog) from the original dataframe.
However, patsy has pickling support now https://github.com/pydata/patsy/pull/66

I haven't looked at the details

user: josef-pkt
issue_2466: state: open
title: TST: more unit tests for X13 , TravisCI
body: #2412  fixed python 3 compatibility, no additional unit tests,

unit tests are not run on TravisCI because we don't install x13 there

user: josef-pkt
issue_2465: state: open
title: ENH: add dispersion method to GLM, especially for negative binomial
body: I don't think we have an easy way to recover the negative binomial dispersion parameter from an estimated GLM Poisson or NegativeBinomial estimate, results instance.

Since the NegativeBinomial dispersion parameter is taken as fixed and has to be provided by the user, it would be easier for users to use a two-stage method with an initial `fit` that can be used to obtain the dispersion parameter.

user: josef-pkt
issue_2464: state: open
title: MixedGLM Likelihood Inference
body: I'm posting this issue as a place to discuss and think about our strategy for likelihood-based inference for generalized linear mixed models. Right now I've got a working likelihood function for a MixedGLM class on this branch on [my statsmodels fork](https://github.com/gjhunt/statsmodels/tree/glmm/statsmodels/regression). 

Firstly a suite of tests needs to be written to test this likliehood function. Preliminary tests seem to agree with R, however. 

We will need to implement a score function for the likelihood. Right now we are using Laplace approximation to calculate the integrals for the likelihood function. We'll have to think whether or not we can use the same laplace approximation for the score. If these integrals don't fit the form of the laplace approximation then do we need to use some more complicated quadrature or do we go to numerical derivatives. In the former case -- if we are going to implement something more complicated than Laplace -- then we probably should re-write the log-likelihood so that it uses this method too. The seemingly popular choice is Gauss-Hermite. Some investigation into R needs to be done to determine how they do this. 

Another issue we'll have to think about is efficiency. We should start thinking about timing the runtime for various methods. This will probably come especially into play when we start trying to get the MLEs for the various parameters. However a couple of kinks need to be worked out before we can do this. Along these lines are refactoring the code currently which is probably a bit more human-readable than time efficient. 

Associated with the MLEs is that we need to determine how to deal with the scale parameter of the overdispersed exponential family. Currently we are just treating these as fixed. Its not clear what precisely glmer in R is doing but it seems like they are trying to do some sort of profiling out of sigma. Although its not clear that anyone currently knows how to deal with the scale parameter. [glmer github issue](https://github.com/lme4/lme4/issues/63)

user: gjhunt
issue_2463: state: open
title: FAQ: using sympy for statistics
body: @unpingco has a nice set of notebooks, for example
https://github.com/statsmodels/statsmodels/issues/2462

user: josef-pkt
issue_2462: state: open
title: FAQ: using statespacemodel for Macroeconomics
body: see
http://nbviewer.ipython.org/gist/ChadFulton/fbce8efd41fcf271b316/

user: josef-pkt
issue_2460: state: open
title: ENH: Regression Discontinuity Design
body: Besides Matching, IPWeighting we also need Regression Discontinuity Design for estimating causal or treatment effects

Two introductory overview papers for applications in epidemiology and health

Bor, Jacob, Ellen Moscoe, Portia Mutevedzi, Marie-Louise Newell, and Till Bärnighausen. 2014. “Regression Discontinuity Designs in Epidemiology.” Epidemiology (Cambridge, Mass.) 25 (5): 729–37. doi:10.1097/EDE.0000000000000138.

Moscoe, Ellen, Jacob Bor, and Till Bärnighausen. 2015. “Regression Discontinuity Designs Are Underutilized in Medicine, Epidemiology, and Public Health: A Review of Current and Best Practice.” Journal of Clinical Epidemiology 68 (2): 132–43. doi:10.1016/j.jclinepi.2014.06.021.

main references for economics will be articles by Imbens, ...

user: josef-pkt
issue_2459: state: open
title: ENH: Instrumental Variables IV and Logit, treatment effect
body: (mainly to park two references I haven't read)
interesting because of a new application area for IV estimation

Mendelian Randomization is IV estimation where genes are used as IVs

First paper is a bit similar to treatment effects literature by estimating or testing specific quantities instead of getting a consistent estimate for all parameters.
(based on very brief skimming of the article)

Burgess, Stephen, and CRP CHD Genetics Collaboration. 2013. “Identifying the Odds Ratio Estimated by a Two-Stage Instrumental Variable Analysis with a Logistic Regression Model.” Statistics in Medicine 32 (27): 4726–47. doi:10.1002/sim.5871.

Boef, Anna G. C., Olaf M. Dekkers, and Saskia le Cessie. 2015. “Mendelian Randomization Studies: A Review of the Approaches Used and the Quality of Reporting.” International Journal of Epidemiology 44 (2): 496–511. doi:10.1093/ije/dyv071.

user: josef-pkt
issue_2458: state: open
title: GMM for missing data
body: Missing at random MAR and GMM
missing in instrumental variables

Chaudhuri, Saraswata, and David K. Guilkey. 2015. “GMM with Multiple Missing Variables.” Journal of Applied Econometrics, January, n/a – n/a. doi:10.1002/jae.2444.
http://onlinelibrary.wiley.com/doi/10.1002/jae.2444/abstract

(most likely interesting for implementation or for connection MAR-GMM to statistical methods, and I know Saraswata)

user: josef-pkt
issue_2456: state: open
title: FAQ-D/Design: Do we have access to original of transformed exog?
body: (I'm not sure how the current, data handling together with formulas works for this)

When we have the designmatrix, `exog`, given, then in some cases we have the transformed variable that can span several columns. That's all we need to know for estimation in the standard cases.

However for some results it's necessary to have access to the original array. We have `exog_orig`, but - does it contain the untransformed variable?
- do we have the name to find the right column(s)?
- do we have the information when some observations have been dropped because of missing values?

use cases - examples
- spline or polynomial basis: for plotting, predict and marginal effects we should go back to the original (1-dimensional data, or multidimensional if splines or polynomials are multivariate)
- Imputation: this goes through the original formula, AFAIR
- for categorical we have separate issue about the encoding used so we can construct contrast matrices like all-pairs comparison without users having to provide additional information (has been changed in patsy to make more info available)

related question:

What do we do if we or users don't go through the formula?

user: josef-pkt
issue_2454: state: open
title: REF/ENH: add estimating equations or score_obs to all models (OLS)
body: (I don't find an issue, but I commented on this a few times)

We have now `score_obs` for most likelihood models, and keep adding them as part of the consistent set of methods. However, we don't have the equivalent of non-likelihood models, e.g. OLS.

Possible ambiguity especially from the treatment of the variance of observations:

example OLS: score of MLE divides by scale (error variance). estimating equation can ignore scale

use case right now:
combining moment conditions of several models into a joint GMM estimator.
For Logit, Probit, I can just reuse score_obs, but there is no equivalent for OLS/WLS

other use cases:
score_test and conditional moment tests.

user: josef-pkt
issue_2453: state: open
title: ENH: GMM with instrument callback, optimal instruments
body: Just an idea while writing a subclass of GMM:

optimal instruments may/will depend on the estimated parameters. Currently, the instruments are always fixed.

One example is nonlinear least squares replication by GMM where the instruments are the Jacobian of the function (or something like this).
- iterative optimization: update instruments together with a `GMM.fit` loop. I doubt it's computationally efficient to iterate `gmm.fit` (inner loop) until convergence for each values of the instruments (outer loop)
- joint optimization: using instrument as a function or callback. In this case we will need to separate optimization from inference. AFAIR, the instruments are treated as fixed when we calculate weight and covariance matrix (derivative of moment conditions does not include derivative of instruments which are assumed fixed)

context is roughly (what I remember) (with special reference to treatment effects)

We want to use weights for the observations in the moment conditions. With two stage estimation we can treat "nuisance" parameters as fixed (like variance, heteroscedasticity in LEF models), or we can estimate those parameters jointly in one big GMM problem.
For example GEE treats covariance parameters as fixed for the GMM/GEE inference.

In treatment effects with inverse propensity score weighting, we can treat the weights as fixed or include the estimating equations of the treatment selection model in GMM.

The theory roughly AFAIU (might not apply to all models within the treatment effects category)
With parametric LEF (QMLE) models the variance or weight parameters are separable and we can treat them as fixed (and use sandwich cov_params). However, assumption in treatment effects literature is semiparametric, the mean function can be misspecified in double robust estimators. Additionally, taking effect of first stage estimators into account might improve small sample properties.
(Stata calculates standard errors based on joint moment conditions. Some references mention that we need to correct for first stage estimation. Topel and similar, which can be done automatically in joint GMM moment conditions.)

user: josef-pkt
issue_2452: state: open
title: Mixed LM optimization GSOC 2015
body: I'm opening this issue for discussion of possible optimization strategies for MixedLM.

The starting point for this work is PR #2363.

First, information gathering:
- Get a rough estimate of where we stand relative to R in terms of total run time.  My experience is that we lag behind LMER by a factor of 3-5 but this is based on very limited examples.
- Get a comparison of the iteration count to convergence for MixedLM and R.  If we are doing more iterations then it is either a problem with the optimizer or we are being too conservative about calling convergence.
- Use the profiler to get a sense of where we are spending time.  I would focus on optimizing the loglikelihood first.  Anything that works there can probably be used in the score function as well.
- Build a set of examples (could be simulated) and a timing script so that when we make changes we know if it really helps.

Ideas for improvement:
- Avoid creating many small temporary arrays by creating re-usable storage for temporaries and using the out keyword parameter, especially in numpy.dot.  One easy place to do this is smw_solver which creates a solver function that is called multiple times (hence storage can be re-used).  Also consider using scipy blas dger to do in-place outer product updates (we have many).  I don't know if there is a good way to use the profiler to see if overhead for creating/destroying temporaries is a significant factor in overall run times.
- Try using sparse matrices for cov_aug and cov_aug_inv (only relevant when variance components are used).  Also sparse matrices could be used for exog_re especially when using a single group to accommodate sparse designs (like random effects ANOVA).
- Find another approach for _gen_dV_dParam : this function helps a lot with readability and extensibility but it might be quite expensive with deeply nested loops and many temporary objects.  Unfortunately it's complex to redesign and still maintain clarity.  The returned arguments are vectors for the regular random effects but are matrices of varying sizes for variance components.
- Avoid re-parsing the formula for each group in from_formula.
- Adjust tuning parameters in the optimizers.  I have noticed in bfgs and lbfgs that sometimes it stops with failed convergence but then when you start again from the ending point it rapidly converges.  Rather than running the optimizer multiple times, can we adjust the convergence criteria to be more strict so that it converges in one call?  I wonder if the issue is that the likelihood is flat so it thinks it is stuck but really needs to just keep going.

@josef-pkt @saketkc 

user: kshedden
issue_2451: state: open
title: ENH/Design: Roy model in python - treatment effects
body: mainly to park a link:

https://github.com/grmToolbox/package   Roy model (MIT licensed, so license compatible)
related to treatment effect see #2443 
(i haven't looked at the details of Roy model yet.)

structural economic models versus semiparametric reduced form, statistical models
see https://groups.google.com/d/msg/pystatsmodels/vJOvEu1UmVw/ikP01CmJn0kJ
Gelman's blog and discussion there.

partially related: We use a similar choice based terminology and utility maximization based restriction on the parameterization in the multinomial models, especially for nested models.

potential outcome is the utility given the discrete choice
However, in randomized trials it's not the "agent" that chooses the treatment (except for non-compliance). 

user: josef-pkt
issue_2449: state: open
title: [Bug/Doc] lowess returns nan and does not warn if there are too few neighbors
body: **Statsmodels version**: 0.6.1
**Issue**: when the # of unique values in X is less than the number of unique neighbors needed to compute LOWESS, nans are returned but no error or warning is given.
**Expected behavior**: a warning should be raised that LOWESS cannot be computed due to lack of sufficient unique neighbors.

(Workaround: you could jitter all the x values by some tiny epsilon to create more unique values, and then rerun lowess(y, x).)

``` python
import statsmodels as sm
print sm.__version__ # '0.6.1'
from sm.nonparametric.smoothers_lowess import lowess
import numpy as np

# initializing data
x = np.random.choice(range(0, 3), 100) # only 3 unique values: {0, 1, 2}
y = np.random.choice(np.arange(0, 1, 0.1), 100)

preds = lowess(y, x)[:, 1] # Slicing to get the predicted y values
# no warning or exception raised, despite the fact that all predictions are nan
print preds
>>> array([ nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
        nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
        nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
        nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
        nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
        nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
        nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
        nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
        nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,  nan,
        nan])

# Trying again, with a few more unique x values
x = np.random.choice(range(0, 5), 100) # now, with 5 unique values
y = np.random.choice(np.arange(0, 1, 0.1), 100)
print preds
>>> array([ 0.4558193 ,  0.4558193 ,  0.4558193 ,  0.4558193 ,  0.4558193 ,
        0.4558193 ,  0.4558193 ,  0.4558193 ,  0.4558193 ,  0.4558193 ,
        0.4558193 ,  0.4558193 ,  0.4558193 ,  0.4558193 ,  0.4558193 ,
        0.4558193 ,  0.4558193 ,  0.4558193 ,  0.4558193 ,  0.4558193 ,
        0.4558193 ,  0.4558193 ,  0.42940657,  0.42940657,  0.42940657,
        0.42940657,  0.42940657,  0.42940657,  0.42940657,  0.42940657,
        0.42940657,  0.42940657,  0.42940657,  0.42940657,  0.42940657,
        0.42940657,  0.42940657,  0.42940657,  0.42940657,  0.42940657,
        0.42940657,  0.42940657,  0.42940657,  0.40486366,  0.40486366,
        0.40486366,  0.40486366,  0.40486366,  0.40486366,  0.40486366,
        0.40486366,  0.40486366,  0.40486366,  0.40486366,  0.40486366,
        0.40486366,  0.40486366,  0.40486366,  0.40486366,  0.40486366,
        0.40486366,  0.40486366,  0.40486366,  0.41932243,  0.41932243,
        0.41932243,  0.41932243,  0.41932243,  0.41932243,  0.41932243,
        0.41932243,  0.41932243,  0.41932243,  0.41932243,  0.41932243,
        0.41932243,  0.41932243,  0.41932243,  0.41932243,  0.41932243,
        0.41932243,  0.41932243,  0.41932243,  0.48931377,  0.48931377,
        0.48931377,  0.48931377,  0.48931377,  0.48931377,  0.48931377,
        0.48931377,  0.48931377,  0.48931377,  0.48931377,  0.48931377,
        0.48931377,  0.48931377,  0.48931377,  0.48931377,  0.48931377])
```

user: carlshan
issue_2448: state: open
title: ARIMAResults conf interval
body: Hello, i am using ARIMAResults to forecast and i am intersted in the conf interval. I would like to know how it is computed. I appreciate very much any response.
Thanks!

user: huig-
issue_2447: state: open
title: memory for FDR
body: http://stackoverflow.com/questions/30764325/memory-efficient-benjamini-hochberg-fdr-correction-using-numpy-h5py

I did one round of reducing temporaries in multipletesting, but maybe I didn't check FDR.

(Note: I don't see much to design for huge memory in this) 
alternatives in PR by Kerby and in nipy

user: josef-pkt
issue_2445: state: open
title: DOC: documenting internal helper functions
body: We are writing helper functions quite often, but I don't know how to find them.

case right now:
cov_params for a two stage estimator that needs to take the correlation between first stage and second stage parameters into account. (first stage is Probit in my case, second stage is linear)
(There should be something simple based on score_obs and hessian if we have both models available)

user: josef-pkt
issue_2443: state: open
title: SUMM: treatment effects, Inverse probability weights
body: see discussion in https://groups.google.com/d/msg/pystatsmodels/WtqcXF3KtJI/wVH4HrHtHVMJ

target: get equivalent to Stata's `teffects`  
## References

mix between statistics, econometrics and Stata (articles and documentation)

,,,

Related PRs:
propensity score matching
mediation analysis

My guess is that we start a new subdirectory, maybe `causal` or `treatment`. `treatment` is better. `causal` is too vague and means also many other things like granger_causality.

user: josef-pkt
issue_2442: state: open
title: ENH: weights in MLE for internal use
body: To get started with weights for MLE, I suggest we just add them, but don't support initially any official interpretation. I'm thinking mainly of discrete models and GLM. The same applies to quantile regression, which can also add "user weights".

Specifically, I'm currently not interested in adding nobs correction or similar for frequency weights. Those require more work and a definition what different weights actually mean.

use cases: robust estimation to downweight outliers, and (I'm currently looking at) inverse probability weights, IPW, for treatment effect estimation. Weights in QuantReg are needed for Quantile treatment effects estimated using IPW (Firpo 2007).

difference and similarity to WLS: 
Weights in WLS have the interpretation of heteroscedasticity and reweight the residuals. In contrast, in MLE we put directly a weight in front of the individual log likelihood contribution. 
IRLS in both RLM and GLM use WLS just for weighted parameter estimation which is similar to the MLE weights suggested here.

Official support for weights can come as soon someone writes the unit tests and clarifies different weight options.

**edit**

Actually, in one parameter LEF/GLM models, the inverse weights correspond to rescaling the residuals and have directly the interpretation of a multiplicative heteroscedasticity as in WLS.
This should be equivalent to the variance function manipulation that I did for GLM to use a "foreign" variance function (example: Poisson with approximate Bernoulli/Binomial variance function for risk ratio estimation). A difference is in that changing the variance function directly is more flexible because it is not restricted to a multiplicative factor.)

user: josef-pkt
issue_2441: state: open
title: QuantReg loglike()?
body: I'm running statsmodels 0.6.1.  The documentation for QuantReg suggests that it has a loglike() function, see, e.g.:

http://statsmodels.sourceforge.net/stable/generated/statsmodels.regression.quantile_regression.QuantReg.html

But when I run it, it raises a "NotImplementedError" exception.  (The analogous call to WLS works fine, so I don't think I've made a mistake invoking it).  Is the documentation wrong, or should I expect loglike() to work?

Thank you

user: wfbradley
issue_2438: state: open
title: backfitting using offset
body: mainly an idea for some implementation details

GAM or partially additive models can be fitted by components using backfitting.
This is very similar to the more recent group coordinate descend algorithms.

We can fit any GAM with specified basis functions as one large single optimization problem, however for some parts like choosing penalization weights or for plots it might be useful to have a "marginal" model that just has one component and all other terms are in the offset.
This will be a mixture between backfitting/group coordinate descend and IRLS or standard optimization.

user: josef-pkt
issue_2437: state: open
title: formula and time series operators
body: how do we handle time series operators in formulas?

there is an open issue somewhere using pandas lag operator

---

Stata has lag and diff operators

I found by chance the dynlm package for R, used in AER examples
includes lag and diff operators and formula functions for season and lags

http://cran.r-project.org/web/packages/dynlm/dynlm.pdf
problems with prediction
http://stats.stackexchange.com/questions/6758/1-step-ahead-predictions-with-dynlm-r-package

user: josef-pkt
issue_2436: state: open
title: GenericLikelihoodModelResults for MixedLM
body: I was looking into implementing likelihood ratio tests which already exists for [linear model](https://github.com/statsmodels/statsmodels/blob/master/statsmodels/regression/linear_model.py#L1563)

Rather than simply duplicating it, I wanted to understand how this should be implemented so that such tests are available to other likelihood models too. [For e.g. [GLM](http://stackoverflow.com/questions/23803236/likelihood-ratio-test-statsmodels)] And GenericLikelihoodModelResults seems to be an appropriate class, probably designed with this in mind?

cc @josef-pkt @kshedden 

user: saketkc
issue_2429: state: open
title: smooth priors detrending
body: It appears that statsmodels doesn't have a function for smooth priors detrending as described by Akaike here: https://statistics.stanford.edu/sites/default/files/AND%20ONR%2040.pdf

Is this observation correct? I checked statsmodels.tsa.tsatools.detrend() and I am only given the options of 'constant' ,'linear', or 'quadratic'.

user: ghost
issue_2427: state: open
title: faq-d: splines b-splines versus truncated power splines
body: miscellaneous notes, for internal usage (mainly so I don't forget)

Binder, Harald, and Willi Sauerbrei. 2008. “Increasing the Usefulness of Additive Spline Models by Knot Removal.” Computational Statistics & Data Analysis 52 (12): 5305–18. doi:10.1016/j.csda.2008.05.009.
http://www.sciencedirect.com/science/article/pii/S0167947308002740#
section 4. going from b-splines to truncated power splines for unequal spaced knots.

(equal spaced knots is just a diff operator, IIRC) 

The idea (that I have in mind) is actually the opposite of their paper. estimate truncated power splines with knot deletion, and then convert to b-splines for better numerical stability.

user: josef-pkt
issue_2423: state: open
title: REF/Design llnull and reduced model for parameter restrictions/transformations
body: I ran into a problem with the inherited llnul with linear restriction penalized estimators but I think the problem is more general.

llnull creates a constant only model and takes the attributes from the main model. There might be attributes that are only relevant for the original parameterization.
examples are parameter transformations and restrictions on parameters.

my current case: In penalized MLE, I penalize linear restriction `\gamma = L \beta` where L is (r, k_params) array (linear transformation). L is attached to the model as `restriction` (current name) and is transferred to the llnull constant only model, where it raises an exception. 
The number of (stochastic) restrictions `r` can be smaller or larger than `k_params` and is not directly relevant.

**related issues** for parameter transformation, restrictions, penalizations and reduced model creation.

**possibilities:** 
- add `_get_null_model` or similar as internal method to model
- automatically adjust extra attributes, need to know how and in which cases. (e.g. drop columns of restriction matrix, if the reduced matrix still makes sense.)
- overwrite `llnull` in results class: that would be the simplest in our current pattern, except I don't have a specialized results class, I'm subclassing a model but not yet a corresponding results class.
- ???

user: josef-pkt
issue_2422: state: open
title: Include documentation for method used in confidence interval generation with QuantReg
body: I've been digging through the source for this class and I can't seem to find where the confidence intervals for the quantile regression coefficients are calculated. Nor can I find what method is used for the minimization procedure in finding the regression coefficients (simplex?, MLE?) so I'm really not sure.

I think this would be good to include in the documentation for when users need to write up their analyses. 

user: pbreach
issue_2421: state: open
title: REF: use average loglike in optimization
body: I start to think that it will be better for optimization to switch to average loglike instead of total loglike.

For example the gradient in poisson can be huge, both because of `exp` but also because of the sum in cases with larger nobs.

Currently, we only use average loglike for discrete fit_regularized because fmin_slsqp had problems with the unscaled version.

One problem besides backwards compatibility is that llf, aic, ... for likelihood ratio test needs sum not average.

user: josef-pkt
issue_2420: state: open
title: ENH: add thresholding methods to _penalties
body: for reuse in coordinate wise solvers, like coordinate descend, we need the thresholding functions, then it will be easy to switch penalization functions for the optimizer.

user: josef-pkt
issue_2419: state: open
title: ENH: PIRLS penalized IRLS for GLM and similar
body: I ran across a reference that does something similar to my penalized PR #2374 
Use quadratic approximation to penalization function to use a standard optimizer

http://link.springer.com/article/10.1007%2Fs11634-015-0205-y
Oelker, Margret-Ruth, and Gerhard Tutz. 2015. “A Uniform Framework for the Combination of Penalties in Generalized Structured Models.” Advances in Data Analysis and Classification, April, 1–24. doi:10.1007/s11634-015-0205-y.

They reuse penalized iterated reweighted least squares from mgcv package in R. mgcv is our the main reference for the GAM and penalized spline GSOC project.

(I didn't check yet, but they mention local quadratic approximation referring to Fan and Li, not fixed quadratic approximation that I use.)

I still looks only useful for "smaller" problems k_vars not too large, and coordinate descend should still be better for larger k_vars if the penalization is univariate (or in groups). 

user: josef-pkt
issue_2417: state: open
title: Design/TST/REF   clear cache after results.__init__
body: related to adding more consistency and "discipline" in creating results instances, with a view on being able to recreate results instances.

Putting values directly into the `_cache` was a backdoor that I used a few times in experimental scripts.
We need unit tests to check that this doesn't happen in our actual code.

Maybe we should get an internal `_clear_cache` method, since `_cache` is in most cases not just a dictionary where we could just do `results_cache = {}`

user: josef-pkt
issue_2414: state: open
title: NeweyWest Robust standard errors
body: In the following CSV:
https://dl.dropboxusercontent.com/u/11592107/TestNeweyWest.csv

there exists two series, cons8 and lag_dmeancay. Consider the regression
    cons8 = c + lag_dmeancay

in Statsmodels,

``` python
import statsmodels as sm
import pandas as pd

df = pd.read_csv('TestNeweyWest.csv')

X = df['lag_dmeancay']
y = df.ix['cons8']
X = sm.add_constant(X)

model = sm.OLS(y, X)
results = model.fit(cov_type='HC3')
```

Produces the results

```
cons          = 0.0371   (SE: 0.001)
lag_dmeancay  = 0.3149   (SE: 0.043) 
```

While running the same regression in R:

``` R
library(dynlm)
library(sandwich)
library(lmtest)
library(plyr)
df <- read.csv("TestNeweyWest.csv", header = TRUE, sep = ';')
ols <- lm(cons8 ~ lag_dmeancay, data = df)
coeftest(ols, vcov = NeweyWest)
```

This produces 

```
cons          = 0.0371   (SE: 0.011)
lag_dmeancay  = 0.3149   (SE: 0.358) 
```

Both results should be HC robust using the mehtods of Newey, W. K., & West, K. D. (1987). Hypothesis testing with efficient method of moments estimation. International Economic Review, 777-787.

While estimated parameters are consistent, standard errors in R are tenfold of those in statsmodels. Ive tried using HAC with various maxlags, HC0 through HC3. I cant seem to figure out how to get consistent estimates.

user: Kristian60
issue_2407: state: open
title: FAQ/FAQ-D: results of derived or transformed models
body: Some models or fit_xxx methods return a results instance where some results have been modified. The modification are in several cases based on an auxiliary model or different parameterization or on parameter restrictions that are not defined within the model that is associated with the results instance.

Typical example: `fit_constrained` where we estimate a model under constraints but return the results for the original parameterization where the model itself doesn't incorporate the constraint.

**What works differently in a derived results instance like this?**
- loglike, score, hessian are for the original model
  - score will not be zero
  - hessian will not be the same as the inverse of appropriately scale `normalized_cov_params` or `nonrobust` `cov_params_default`
  - both score and hessian correspond to the unrestricted model evaluated at the restricted parameters as in a score or Lagrange multiplier test
  - hessian will not be invertible if restrictions where used to impose identification or nonsingular design. (eg. when dropping perfectly collinear variables or using full dummy sets with parameter restrictions)  We might still be able to use `pinv` but I'm not sure when that is (theoretically) appropriate.
- k_xxx, degrees of freedom will be based on auxiliary model 
  - those should depend on k_vars and k_constraints, i.e. effective counts and dof
- predict should/will work fine in the original exog space and original parameterization
- wald_tests: should work without problems
  - this is based on the assumption that the reduction/transformation is treated as based on known constraints.   (e.g. pinv in OLS is the regularized solution)
  - no concept of estimability is incorporated in the implementation (AFAIU)
  - cov_params should have all restrictions incorporated, cov_params has less than full rank

**dangerous things**

generic and inherited code that might do the wrong thing

not known yet (insufficient test coverage for non-main results, not enough example cases)

candidates:
- code that uses score and hessian as if they were for the restricted model
  - get_covparams_robust ?
- code that uses `exog.shape[1]` for df_model or the number of effective parameters
- (internal) if we get code that does something with `mle_retvals` and `mle_settings`

user: josef-pkt
issue_2406: state: open
title: REF: NegativeBinomial start_params might fail
body: NegativeBinomial is using a Poisson model to get the start_params

However, that might fail, it uses default optimizer 

I added `skip_hessian=True, full_output=False` to the poisson fit for start_params in NegtiveBinomial
c3791cbf8b81b452a0cc01515f9732e37a901aa2 in PR #2374 

There might be other code that needs similar review and changes.

user: josef-pkt
issue_2405: state: open
title: ENH: RestrictionMixin, TransformationMixin
body: Following the pattern of Mixins that modify loglike and related methods, I started with a Mixin that does parameter transformation to impose for example nonlinear restrictions and restriction on parameters that are not exog related.

The competing design of building the transformation and restrictions directly into a LikelihoodModel is used in several places.
The advantage of a Mixin is that we can extend existing models to include parameter restrictions and transformations. The mixin modifies loglike and similar on top of existing methods and wouldn't collide with existing transformation.
The second advantage is that we can separately develop and experiment with code in how to handle this more generically, without starting to change existing models.

Basic structure of the mixin is easy, but it needs a lot more work to handle all occurences of `params`, and to separate estimated params from transformed/expanded params, relevant for inference and for summary.

*_Difference to `fit_constrained`:  *_
`fit_constrained` and similar create an auxiliary model and transform the exog array making a copy of the relevant data (or a view if submatrix is aligned, I guess). The parameter of the auxiliary model is transformed into the original parameter space (this could apply also to the ConstrainedMixin), but the auxiliary model is just a standard model without restrictions.

related - still TODO: we need a more consistent patterns for Transformation and nonlinear function classes

---

Getting the details done is too much distraction right now, so here's the script (based on PenalizedMixin, run on python 3.4)

```
# -*- coding: utf-8 -*-
"""
Created on Wed May 13 20:15:34 2015

Author: Josef Perktold
License: BSD-3
"""

import numpy as np
from statsmodels.discrete.discrete_model import Poisson


class ConstrainedMixin(object):

#    def __init__(self, *args, penal=None, **kwds):
#        super(ConstrainedMixin, self).__init__(*args, **kwds)
#        
#        if penal is None:
#            self.penal = smpen.SCADSmoothed(0.1, c0=0.0001)
#        else:
#            self.penal = penal
#        self.pen_weight = len(self.endog) #100.

    def set_constraints(self, transform):
        self.transform = transform

    def loglike(self, params, transform=True):

        if transform:
            params = self.transform.expand(params)

        llf = super(ConstrainedMixin, self).loglike(params)
        return llf

    def score(self, params, transform=True):    
        if transform:
            loglike = self.loglike
        else:
            loglike = lambda p: self.loglike(p, transform=transform)

        from statsmodels.tools.numdiff import approx_fprime
        return approx_fprime(params, loglike)

    def score_todo(self, params, transform=True):
        if transform:
            params = self.transform.expand(params)
        sc = super(ConstrainedMixin, self).score(params)
        return sc

    def hessian(self, params, transform=True):
        if transform:
            loglike = self.loglike
        else:
            loglike = lambda p: self.loglike(p, transform=transform)

        from statsmodels.tools.numdiff import approx_hess
        return approx_hess(params, loglike)

    def hessian_todo(self, params, transform=True):
        # add transform derivatives
        if transform:
            params = self.transform.expand(params)
        sc = super(ConstrainedMixin, self).hessian(params)
        return sc


class PoissonConstrained(ConstrainedMixin, Poisson):
    pass

#class LogitConstrained(ConstrainedMixin, Logit):
#    pass
#    
#class ProbitConstrained(ConstrainedMixin, Probit):
#    pass


class ExampleTransform(object):

    def __init__(self, **kwds):
        self.__dict__.update(kwds)

    def expand(self, params):
        """take reduced constrained params and return full params 
        """
        params_new = np.zeros(self.k_params)
        params_new[self.mask] = params
        params_new[~self.mask] = self.func(params)
        return params_new



class ExampleTransform2(object):

    def __init__(self, k_params, **kwds):

        self.k_params = k_params
        self.change_index = [1, 2]
        self.func = lambda p: [(p[0] + np.sqrt(p[3])) / 3, 2 * p[3]]


        mask = np.ones(k_params, dtype=np.bool_)
        mask[self.change_index] = False
        self.keep_index = mask #np.arange(k_params)

        self.__dict__.update(kwds)

    def expand(self, params):
        """take reduced constrained params and return full params 
        """
        params_new = np.zeros(self.k_params)
        params_new[self.keep_index] = params
        params_new[self.change_index] = self.func(params_new)
        return params_new


def dummy_func(*args, **kwds):
    pass


# simulate data
np.set_printoptions(suppress=True)
np.random.seed(987865)

nobs, k_vars = 100, 5
k_nonzero = 4
x = (np.random.rand(nobs, k_vars) + 0.5* (np.random.rand(nobs, 1)-0.5)) * 2 - 1
x *= 1.2
x[:, 0] = 1
beta = np.zeros(k_vars)
beta[:k_nonzero] = 1. / np.arange(1, k_nonzero + 1)
linpred = x.dot(beta)
mu = np.exp(linpred)
y = np.random.poisson(mu)

res0 = Poisson(y, x).fit()


modr = PoissonConstrained(y, x)
transf = ExampleTransform2(x.shape[1])
modr.set_constraints(transf)
resr = modr.fit(start_params = res0.params[transf.keep_index], callback=dummy_func)

#example is adjusted so the difference is not too large
#transf.expand(res0.params[transf.keep_index])

p = transf.expand(resr.params)
print(p)
print(res0.model.loglike(p))
print(resr.model.loglike(p, transform=False))
print(resr.llf)
print(res0.llf)

# todo LR test
```

user: josef-pkt
issue_2404: state: open
title: BUG/REF  don't check perfect separation for Poisson and others
body: The perfect separation check is in DiscretModel and even checks Poisson.

AFAIK, the perfect separation check is only designed for binary models right now, and shouldn't check any other models.

I ran into it because my model, a subclass of Poisson, didn't have a correct `cdf` method

user: josef-pkt
issue_2402: state: open
title: FAQ: modelling and forecasting seasonal data
body: (Note: please add information, I (Josef) don't have an overview)

e.g.
http://stats.stackexchange.com/questions/152280/forecasting-values-based-on-day-of-week-and-hour

One alternative is to work with fixed effects and have different intercepts for each hour and DOW. i.e. something like results = sm.ols(formula='Count ~ Epoch + C(DoW) + C(Hour)', data=df).fit() using patsy to convert DoW and Hour into time dummies

Another alternative would be to use splines to model a fixed seasonal pattern http://patsy.readthedocs.org/en/latest/spline-regression.html e.g. `bs(Hour, df=4)` for hours. If there are more observations in each `season`, then periodic splines would be more appropriate, or another periodic polynomial basis.

SARIMA would work better for changing seasonal patterns, and is available in statsmodels master but not yet in a released version https://github.com/statsmodels/statsmodels/blob/master/examples/notebooks/statespace_sarimax_stata.ipynb

other methods:

... EWMA (PR)

user: josef-pkt
issue_2401: state: open
title: REF/Design: mle_settings, replicating fit 
body: I'm writing unit tests at high precision (e.g. rtol=1e-10) for `_fit_xxx` in PR #2374
`ENH/REF: add LikelihoodModel._fit_collinear, adjust _fit_zeros and tests …`   f3adefc
- not all models define `mle_settings` (only those that use scipy optimizers) -> add to all
- rename ? `mle_settings` to more general `optim_settings`, and corresponding `mle_retvals`
- optimizer tolerance limits replicability if the fit optimizer settings are not the same
  e.g. I use `res.mle_setting['start_params']` to use the same starting values, but not yet the optimizer method, without those the estimated params only agree on a few decimals or rtol

user: josef-pkt
issue_2400: state: open
title: FAQ-D: debugging problems with results attributes
body: When writing statsmodels code that works with results instances that are not `self`, we need to watch out that we don't have the wrapper.

I just reset the scale

`res.scale = something`

and debugged for an hour, why cov_params didn't change, even though `res.scale` showed the correct `something`

**Problem:**

Setting an attribute of a wrapped results instance, adds this attribute to the wrapper and not to the underlying result instance, `res._results`

Make sure we have the right `res._results` to manipulate.

user: josef-pkt
issue_2399: state: open
title: GLM: scale attribute in model
body: in GLM scale is attached to the model, and the attribute is used for calculating the results, e.g. bse

This makes it difficult to recreate results with different params and normalized_cov_params
see #2393 

user: josef-pkt
issue_2396: state: open
title: design: new attributes `params_derived` and similar?
body: just parking a possible suggestion for a name

`params_derived` or similar are params that are part of likelihood definition if estimated by full MLE, but may not be estimated by `fit` with other parameters.

example: variance/scale and dispersion parameters
open issues on how to include scale of MLE or of QMLE in generic results code

details are not clear yet

`hessian_inv` versus `normalized_cov_params` versus `cov_params_default`
We need to know which is which for calculating sandwiches of transformed models

user: josef-pkt
issue_2395: state: open
title: ENH/TST fit_constrained and cov_type
body: see also #1093 recreating models

there are currently no unit tests for `fit_constrained` when `cov_type` is also chosen (at least I don't see any in `discrete/tests/test_constrained.py`.
i don't know whether it works or not
Stata results file for Poisson contains an example with `vce(robust)` in `results_poisson_constrained.py` and one for GLM/Logit in `results_glm_logit_constrained.py`

we need the same or similar pattern for other `fit_xxx` as mentioned in #1093

user: josef-pkt
issue_2393: state: open
title: SUMM: creating and recreating results instances
body: We need to create new results instances in several methods, and we could drop some code duplication (where fit is overridden just to get the right results instance)

related #1093 was main issue to recreate or clone models

need to collect issues, and get more general solution, 
Right now, we have some specific solutions in various new methods or methods under development, latest installment `PenalizedMixin`, `fit_regularized` and various `fit_reparameterized`
PR #2374 #2385  for penalized
no PR yet for the `fit_reparameterized` variants, similar to poisson and glm `fit_constrained`

one possible solution is `fit(maxiter=0)` which needs to be consistently implemented and unit tested
Done for GLM: #1728 reopened method='nm' works, default method (irls) doesn't have params (is None)

**update**
#2374 not has a `LikelihoodModel._fit_zeros` method that implements creating a results instance and modifying it based on a auxiliary transformed/reduced regression with generic tests for OLS, WLS, Logit, Poisson, NegativeBinomial, GLM and RLM. skips GEE, other models are not yet in base `test_generic_methods`. Two methods that use this are `PenalizedMixin.fit` and `LikelihoodModel._fit_collinear` with additional unit tests.

This is now a working solution and a good starting point for more refactoring for consistency across models and for finding a less hacky solution.

One possible direction is to provide similar or pieces of it for similar use cases, e.g. if we already have the parameter estimates from a different (non-scipy) optimizer like coordinate descend, then we don't need an auxiliary model and could just adjust the covariance of params (`normalized_cov_params`/`hessian_inv` and `cov_params_default`)

**one main problem:**

retrofitting an existing results instance is fragile: 
It is difficult to avoid cached attributes and updating all required attributes. (This was one major reason to move `cov_type` into `fit`)
I ran into problems going the `fit(maxiter=0)` and adjust attributes route, in PR #2374 08c919d18a2499cdf5a5c33e7a8077f5866c3330 especially with RLM, and I haven't started with robust cov_types yet in other models.

two possibilities:
- don't retrofit, make it possible to create a new results instance
- move all but the basic code out of `results.__init__` into an `_initialize` method, that can be called to "cleanly" retrofit a results instance

user: josef-pkt
issue_2392: state: open
title: SUMM: using Mixins for modifying likelihood models or M-estimators
body: proposal in the wiki
https://github.com/statsmodels/statsmodels/wiki/SMEP-D:-Multiple-Inheritance-and-Modifier-Mixins-for-LikelihoodModels

and initial design issue #2370

this SUMMary issue is to collect related issues for specific cases and discuss some implementation details

currently, I have two mixin draft version, MixedMixin, and Penalized Mixin
### penalized
#2385 PR add elastic net
#2374 PR add SCAD to _penaltiies
#2293 general request for maximum penalized likelihood, (Logit separation case)

...
### Implementation
- signatures of methods, especially `__init__`
  - Should we override method just to get a informative signature, instead of `*args, **kwds`?
  - possible complication if we have other positional arguments besides endog and exog
  - maybe python compatibility: I'm using python 3.4 now, location where `*args` are permitted has changed
- return classes
  - we need to find a more flexible and more standardized way how return classes are determined and how return instances are created. several open issues ???
  - do we need specific results classes, ResultsMixins?  
    new methods to present specific results, maybe not necessary for Penalized but needed for Mixed.
- code location
  - where do we put the mixin classes, currently I'm planning on a new module for each mixin in base
    i.e. `statsmodels.base.mixed.MixedMixin` `statsmodels.base.penalized.PenalizedMixin` but for example the penalized mixin is just one class with about 40 lines and will only get a bit larger. 
    (We have `statsmodels.base._penalties` for the penalty functions, which also might have to be public for use in `model.__init__` as in GLM and RLM.)

user: josef-pkt
issue_2389: state: open
title: ENH/BUG: GLM is missing loglikeobs
body: I need loglike obs for panel data, current case for GLMMixed defined via Mixin.

sandwiches and GEE (IIRC) only need scoreobs, but not loglikeobs.

user: josef-pkt
issue_2387: state: open
title: REF: generic extra_params handling, versus mean parameters
body: We need to separate mean parameters from the full `params` to evaluate the mean function (conditional expectation E(y | x).

current code: 
- I added a extra keyword to handle NegativeBinomial extra parameter in some of the generic code.
- other models mainly in tsa have explicit list of how params is composed. 
  Composition is hard coded by design.

my current problem:

I'm using MixedMixin to modify likelihood models. We get additional parameter in `params` for the MixedProbit. Some of the results methods or cached attributes break because of
`np.dot(exog, params)` where `params` is longer than the number of exog columns.

Either we need to know and use the number of extra params or need a index or mask for mean parameters. This is relevant also for other cases when we want to change a model to have different kinds of parameters, e.g. if we allow for modeling of the variance in heteroscedastic models.
(two parameter GLM/LEF) or share code with multi component models like for example zero-inflated Poisson.

user: josef-pkt
issue_2386: state: open
title: Conditional Logit and PHReg with exact tie handling
body: based on the discussion on the mailing list, the `exact` tie handling uses the probabilities from a conditional logit (Chamberlain) model. 

R survival has clogit implemented as a special case using phcox with small number of discrete time periods and exact tie handling.

This could also be added to our PHReg.

user: josef-pkt
issue_2384: state: open
title: SUMM:  post estimation results for mixed, random effects models
body: What do we want? Can we have a consistent policy, including linear MixedLM?
(Issue created while looking at #2382 )
- specification tests together with other mixed models, 
  LR, LM test, possible problem: variances are on the boundary of parameter space
  (LM test with one parameter possibly on the boundary is ok (one-sided chisquare))
- best prediction including individual random effects, ? 

references:
- vignette for mlogit in R, section 6 http://cran.r-project.org/web/packages/mlogit/vignettes/mlogit.pdf
  for discrete choice mixed logit

user: josef-pkt
issue_2382: state: open
title: Review/Bugs: discrete choice mixed logit
body: see PR #1605

below are my comments about the status of MXLogit copied from the mailing list

**Summary**
- two problems or bugs
  - overparameterized mean of random effects -> I removed extra means in my experimental changes
  - hard lower bound on variance is absorbing for optimization, score, hessian become zero
    --> I use exp(std) loglink instead
- missing unit tests
- refactoring
  - fit is returns inherited results class not the model specific results class
  - model specific results class calls model.fit in `__init__`
  - looks like it's still inheriting GenericLikelihoodModel
- numerical problems
  - can we get analytical score or hessian?
  - can we use gaussian quadrature for multiple, independent random effects?
- enhancements
  - score_obs and connect to generic code that uses score_obs
  - panel data / groups ?
  - correlated random effects
  - non-normal random effects distributions
  - specification tests together with other mixed models, LR, LM test, boundary of parameter space

not clear:
- parameters are 1-d unlike MNLogit, endog is essentially multivariate  
  - do we have generic results code that covers this case? residuals, predict, wald_test (formula)

similar code patterns to MixedGLM?  GSOC 2015 project

---

Question about Mixed Logit in GSOC 2013 
What would be the steps for using that code? Assume I'm not that savvy with git, which is true.

Theoretically it's just checking out or downloading the branch.

However, it's a year behind master and won't have the newer parts of statsmodels. (And we cannot import two different versions of statsmodels at the same time).
It looks like I converted the branch to common python 2 - python 3 codebase last year. 

Do I understand correctly that the code appears to be working, though there are no unit tests so there are no guarantees? Is there any other way in which it is "unfinished"?

About terminology since we just had a question about Mixed Logit, Poisson, GLM and a new GSOC project:
The project in 2013 was all about variations on Multinomial Logit or discrete choice models. I'm not sure how well it would work for binary Logit, which would be just a special case.

The emphasis of the GSOC project was to get the core estimation parts. I guess there is also work left to get formula integration and more post estimation results.

Based on running the mxlogit example (on python 3.4), it looks like mxlogit still has bugs (e.g. hessian has one row and column of zeros *), and the return class is not properly connected. 

(\* After dropping the zero column in hessian, the parameter standard errors, bse, look like they agree at around 2 decimals/digits with R except for the standard deviation of the random component. It looks like a duplicated intercept in the parameter list, specifically I think the random effects distributions should have mean zero.)

Besides redundant random effects mean, there is also a problem with the lower bound for variances.

After changing these two things, I'm getting reasonably close to R mnlogit.
2 to 3 digits on mean parameters and statistically significant variances.
bse is close at two digits, but maybe a bit larger than R but one bse differs by 14 %
maximized loglikelihood llf ours: -197.687  R: -197.26

I guess we are down to numerical approximation errors in numerical integration of random effects, and numerical derivatives in score and hessian. I don't have R available right now to check how much the results in R change with changing one of the optimization or numerical integration options.
I only checked two examples for which the results are in the PR. It needs more examples to see how robust this is.

MixedGLM will have similar issues, and we will have to look at numerical issues more this summer during GSOC.

Josef

clogit looks good, but Ana ran out of time for mxlogit, it was added at the very end of GSOC, AFAIR.

another point: 
Ana worked with 1-d parameter arrays which should make most of the generic postestimation results like t_test and similar work without changes. (discrete.MNLogit has 2-d parameters which are still a headache.)

and one more:
Numerical integration uses Halton sequences which could be generalized to other distribution than Gaussian. There is no version using Gaussian quadrature for integration, but it looks like the design is modular enough that only one method needs to be enhanced to support any finite point approximation to a distribution of random effects. 

Based on this, I don't think it would take long to get mxlogit into a usable, tested state. Some improvements (like analytical score and hessian and more features)  would be more open ended.

PS: 
After being puzzled by the random effects integration, I don't think mxlogit has been extended to panel data.
AFAICS by partially skimming the vignette for R's mnlogit http://cran.r-project.org/web/packages/mlogit/vignettes/mlogit.pdf ,
this is a Logit random utility model with unobserved cross-sectional heterogeneity, not for repeated observations of the choice for each individual.
https://github.com/statsmodels/statsmodels/pull/1605/files#diff-1804646219b3397b00cc482dc933321f

partial follow up:
Aside from data handling, adding panel data might be only a change in one line, or a few with proper support.
np.multiply.reduceat(cdf, group_idx) in cdf_average
where group_idx = np.concatenate(([0], np.where(groups[1:] != groups[:-1])[0] + 1))
assuming data is sorted by group indicator.
(and ignoring possible numerical problems with repeatedly multiplying tiny numbers.)

Looks like this pattern would work also for MixedGLM, AFAICS.

user: josef-pkt
issue_2380: state: open
title: ENH: measures for multicollinearity
body: additional measures for multicollinearity and enhanced variance inflation factor

relevant issues
#2376  treatment of constant for variance inflation factor
#1669 vif for ridge regression, base for non-loop calculation
#2157 general request for multicollinearity measures
#6249 problems with implicit or no constant
#2378 condition number, scaling

related issues
#1744 diagnostic messages including multicollinearity
#2062 use reduced column space for singular design matrices, I started to work on this with `fit_transformed` `fit_constrained` pattern
#1534 partially related for dropping singular, perfectly collinear columns
#4582 sequential dropping of large vif columns

Note:
Definition of "perfect collinearity" and "singular", "reduced rank" design matrices are subject to floating point noise and have different thresholds. see #1662

user: josef-pkt
issue_2379: state: open
title: factor encoding for deviation from mean
body: random thought: I ran into this a few times, but haven't tried yet

What is the coding for categorical variables to get deviation from mean in unbalanced dataset?

user: josef-pkt
issue_2378: state: open
title: normalized condition number
body: We report the condition number in RegressionResults as ratio of largest to smallest eigenvalue of exog.

Greene 5th edt, page 57 mentions sqrt with exog standardized to have unit length, refering to Belsley Kuh and Welsh.  This sounds like the usual thresholds are for standardized exog, not the actual exog.

related to other measures for multicollinearity
Check whether we should add normalized condition number.


see also #553 about choosing a threshold
user: josef-pkt
issue_2376: state: open
title: statsmodels.stats.outliers_influence.variance_inflation_factor is wrong
body: In the source code, 
the line below doesnt take the constant term (interception) into account. 
 r_squared_i = OLS(x_i, x_noti).fit().rsquared

For example, data is:
                     mpg cyl  disp  hp drat    wt  qsec vs am gear carb
Mazda RX4           21.0   6 160.0 110 3.90 2.620 16.46  0  1    4    4
Mazda RX4 Wag       21.0   6 160.0 110 3.90 2.875 17.02  0  1    4    4
Datsun 710          22.8   4 108.0  93 3.85 2.320 18.61  1  1    4    1
Hornet 4 Drive      21.4   6 258.0 110 3.08 3.215 19.44  1  0    3    1
Hornet Sportabout   18.7   8 360.0 175 3.15 3.440 17.02  0  0    3    2
Valiant             18.1   6 225.0 105 2.76 3.460 20.22  1  0    3    1
Duster 360          14.3   8 360.0 245 3.21 3.570 15.84  0  0    3    4
Merc 240D           24.4   4 146.7  62 3.69 3.190 20.00  1  0    4    2
Merc 230            22.8   4 140.8  95 3.92 3.150 22.90  1  0    4    2
Merc 280            19.2   6 167.6 123 3.92 3.440 18.30  1  0    4    4
Merc 280C           17.8   6 167.6 123 3.92 3.440 18.90  1  0    4    4
Merc 450SE          16.4   8 275.8 180 3.07 4.070 17.40  0  0    3    3
Merc 450SL          17.3   8 275.8 180 3.07 3.730 17.60  0  0    3    3
Merc 450SLC         15.2   8 275.8 180 3.07 3.780 18.00  0  0    3    3
Cadillac Fleetwood  10.4   8 472.0 205 2.93 5.250 17.98  0  0    3    4
Lincoln Continental 10.4   8 460.0 215 3.00 5.424 17.82  0  0    3    4
Chrysler Imperial   14.7   8 440.0 230 3.23 5.345 17.42  0  0    3    4
Fiat 128            32.4   4  78.7  66 4.08 2.200 19.47  1  1    4    1
Honda Civic         30.4   4  75.7  52 4.93 1.615 18.52  1  1    4    2
Toyota Corolla      33.9   4  71.1  65 4.22 1.835 19.90  1  1    4    1
Toyota Corona       21.5   4 120.1  97 3.70 2.465 20.01  1  0    3    1
Dodge Challenger    15.5   8 318.0 150 2.76 3.520 16.87  0  0    3    2
AMC Javelin         15.2   8 304.0 150 3.15 3.435 17.30  0  0    3    2
Camaro Z28          13.3   8 350.0 245 3.73 3.840 15.41  0  0    3    4
Pontiac Firebird    19.2   8 400.0 175 3.08 3.845 17.05  0  0    3    2
Fiat X1-9           27.3   4  79.0  66 4.08 1.935 18.90  1  1    4    1
Porsche 914-2       26.0   4 120.3  91 4.43 2.140 16.70  0  1    5    2
Lotus Europa        30.4   4  95.1 113 3.77 1.513 16.90  1  1    5    2
Ford Pantera L      15.8   8 351.0 264 4.22 3.170 14.50  0  1    5    4
Ferrari Dino        19.7   6 145.0 175 3.62 2.770 15.50  0  1    5    6
Maserati Bora       15.0   8 301.0 335 3.54 3.570 14.60  0  1    5    8
Volvo 142E          21.4   4 121.0 109 4.11 2.780 18.60  1  1    4    2

For regression, 
fit <- lm(mpg~disp+hp+wt+drat, data=mtcars)
vif( fit ) should give 
    disp       hp       wt     drat 
8.209402 2.894373 5.096601 2.279547 

But vif( df[:, ['disp', 'hp', 'wt','drat'].values(), 0 )   will give 37.1154 instead of 8.209

user: littlecuocuo
issue_2373: state: open
title: Design: register a transform in fit for results
body: Just an idea, I don't know yet whether this is feasible.

*_use case_:*
In penalized estimation, the penalization is often calibrated or predefined for standardized coefficients, standardized data (rescaled exog). 

If the user uses a transformation prior to creating the model, then it might still be of interest to have the results, tests and reports for the original scale. That is we `fit` in one parameterization or scaling and have the Results using a different one.

This is a generalization of `fit_constrained` and `fit_transformed` that allows user to use only the second half of the transformation.

**Problem with this**:

While it is easy to transform params and cov_params, there are many results that directly depend on the attached exog, like fittedvalues, resid, rsquared and similar. The attached exog would have the wrong scaling, if the user rescaled it.

`fit_constrained` and `fit_transformed` use an auxiliary model that contains the transformed data, but still has the original data in the main model.

(Doesn't sound good anymore.)

**correction** 

If only exog is transformed but not endog, then fittedvalues and resid would still be independent of linear or affine transformation of exog. 
Predict, Margins and others would depend on the original scaling.

user: josef-pkt
issue_2371: state: open
title: online and streaming stats and recursive model fit
body: Two packages in Julia:

https://github.com/johnmyleswhite/StreamStats.jl
https://github.com/joshday/OnlineStats.jl

both MIT licensed, and include (approximate) Logit and similar

user: josef-pkt
issue_2370: state: open
title: Design: modifiers for loglike, score, hessian
body: Specific case: Where and how do we add penalization terms to the loglike, score and hessian?

Another case that is already in use is calling a `transform_params` to separate parameters to be optimized from internal representation. `transform_params` are currently mainly used for internal reparameterization (inequality constraints with interior solution) and yet not much for imposing equality constraints. The former are only relevant in optimization, the latter are also relevant for inference and other results.

The general task is that we need a wrapper that modifies the loglike and others on entry and exit. It would be a typical case for a decorator, except we don't want to hardcode a decorator in there.

see also my message in
https://groups.google.com/d/msg/pystatsmodels/qWJHRtPidaI/g8jpOJ4i7gYJ

The main purpose is for optimization with the standard scipy optimizers, but we need it also for inference, e.g. calculating the sandwiches for cov_params.
Optimization that uses other optimization methods can add a new fit method, but that still leaves robust cov_params as open question.

possible designs
- use subclasses and Mixin, e.g. `class PoissonPenalized(PenalizedMixin, Poisson)`
  - (+) doesn't require changes to existing models
  - (-) needs a new class for every existing class
- add optimizer options
  - current discrete fit_regularized adds separate optimizer that includes penalty, specific return classes
  - weaker option: outsource optimizers (PR) and allow overwriting functions/methods that are used in optimization, e.g. the lambda functions used in the optimizers based on loglike, score and hessian
  - (-) does not provide access to penalties/reparameterization for inference.
- build it into all models
  - check in each method, loglike, score, hessian, whether a "modifier" should be applied, similar to some of the current `trans_params`
  - call overhead: just one additional `if` check in the methods
  - needs a way to define the option to use it. current internal flags is not "nice" and doesn't generalize. Use keyword options instead. For that we need `**kwds` to be used by optimizers.
  - (-) needs to replicate code in every method
  - possible: refactor current loglike to leading underscore and add new loglike as wrapper. (similar to scipy.stats.distributions)
- combinations of the above.

other possibilities
-  "prefix" class with delegation, `mod_m = PenalizedMEstimator(modpr, smpen.SCADSmoothed(0.1, c0=0.0001))`
  - problem It's difficult to add all the necessary methods of the model class to the the prefix class

user: josef-pkt
issue_2368: state: open
title: ENH: estimating ARMA parameters from step response function
body: see mailing list discussion "ARMAX, SISO dynamic model identification"
ARMAX, SISO dynamic model identification

"Since the step input is observed, the estimation problem is not really ARMA. 
For given x (step function) this is just a ARX (except our ARX has X in mean function) or distributed lag model and can just be estimated by OLS (i.e. linear regression if we ignore initial conditions).
"

```

import numpy as np
from scipy import signal
from statsmodels.regression.linear_model import OLS

ar_coeffs = [0.9391]
ma_coeffs = [0.0609, -0.1, 0.2]
ar = np.concatenate(([1], -np.asarray(ar_coeffs)))
ma = np.concatenate(([1], ma_coeffs))

step = np.ones(200.)# * (1 + ar[1:].sum())
step[:100] = 0.

y = signal.lfilter(ma, ar, step)
print(y[95:110])

res = OLS(y[1:], np.column_stack((y[:-1], step[1:], step[:-1]))).fit()
print(res.summary())

res2 = OLS(y[2:], np.column_stack((y[1:-1], step[2:], step[1:-1], step[:-2]))).fit()
print(res2.summary())

res3 = OLS(y[3:]+0.00*np.random.randn(len(y)-3), np.column_stack((y[2:-1], step[3:], step[2:-1], step[1:-2], step[:-3]))).fit()
print(res3.summary())



from statsmodels.tsa.tsatools import lagmat, lagmat2ds

def arma_from_response(outvar, invar, order, exog=None):
    """Estimate ARMA parameters from a response function

    Parameters
    ----------
    outvar : array_like, 1-D
        endog, response of system to the input given by invar
    invar : array_like, 1-D or 2-D ?
        input to system, for example impulse or step function. This replaces
        the error term in an ARMA process
    order : tuple of 2 int
        order of the ARMA process, length of lag polynomials without leading 1.
    exog : array_like or None
        If exog is not None, then it will be prepended to the regressors in the
        auxiliary OLS regression. This can be used to capture the effect of
        other explanatory variables.
        Note: This is not the same definition as the X in ARMAX

    Returns
    -------
    res_ols : OLSResults instance
        results of the auxiliary regression.


    Notes
    -----
    Todo:
    - add exognames, current parameter names in summary are not informative - added
    - should we allow for weights, i.e. use WLS
    - should we use a robust cov_params_default?
    - reuse lagmat for computing with several orders, e.g max_order = (4, 3)
    - check if MISO works, assumes currently identical lag length for all inputs
    - ...

    References
    ----------

    ??? Is anybody else doing something like this?

    """
    outvar = np.asarray(outvar)
    invar = np.asarray(invar)

    lags = lagmat2ds(np.column_stack((outvar, invar)), *order)
    endog = lags[:, 0]
    regressors = lags[:, 1:]
    if exog is not None:
        regressors = np.column_stack((exog, regressors))
    res = OLS(endog, regressors).fit()
    names = res.model.exog_names
    names[k_exog : k_exog + order[0]] = ['ar_L%i' % i for i in range(1, order[0]+1)]
    names[k_exog + order[0] :] = ['ma_L%i' % i for i in range(order[1] + 1)]
    return res

#outvar, invar, exog

order = (1, 3)
outvar = y#[90:120]
invar = step#[90:120]
exog = None

##lags = lagmat2ds(np.column_stack((outvar, invar)), *order)
##endog = lags[:, 0]
##regressors = lags[:, 1:]
##res_ols = OLS(endog, regressors).fit()
##print(res_ols.params)
print(arma_from_response(outvar, invar, order).params)
```

results of last two print statements

```
                           OLS Regression Results                            
==============================================================================
Dep. Variable:                      y   R-squared:                       1.000
Model:                            OLS   Adj. R-squared:                  1.000
Method:                 Least Squares   F-statistic:                 7.465e+31
Date:                Tue, 21 Apr 2015   Prob (F-statistic):               0.00
Time:                        19:39:01   Log-Likelihood:                 6102.9
No. Observations:                 197   AIC:                        -1.220e+04
Df Residuals:                     192   BIC:                        -1.218e+04
Df Model:                           5                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|t|      [95.0% Conf. Int.]
------------------------------------------------------------------------------
x1             0.9391   2.22e-16   4.22e+15      0.000         0.939     0.939
x2             1.0000   8.62e-15   1.16e+14      0.000         1.000     1.000
x3             0.0609   1.22e-14      5e+12      0.000         0.061     0.061
x4            -0.1000   1.22e-14  -8.21e+12      0.000        -0.100    -0.100
x5             0.2000   9.23e-15   2.17e+13      0.000         0.200     0.200
==============================================================================
Omnibus:                        2.302   Durbin-Watson:                   0.035
Prob(Omnibus):                  0.316   Jarque-Bera (JB):               27.296
Skew:                          -0.260   Prob(JB):                     1.18e-06
Kurtosis:                       1.252   Cond. No.                         308.
==============================================================================

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[ 0.9391  1.      0.0609 -0.1     0.2   ]
```

user: josef-pkt
issue_2367: state: open
title: Design: flexible number of returns
body: What are the functions that have a variable number of returns based on the keyword arguments?

acf return see #2366

**design options**
- keep variable number of returns
- always return Bunch or Results if we need variable number of returns (current trend in pattern for many functions/methods that need more flexibility and can have lazily evaluated extra results)
- fill returns with `None`, so the number of returns remains unchanged (proposed in #2366 )

Any changes in number and types of returns will break backwards compatibility, and we should settle on a pattern so we have to break it only once

user: josef-pkt
issue_2362: state: open
title: ENH: postestimation for ANCOVA type regression models, predicted means
body: this is an integration issue to bring different things together for one type of analysis.

The model is as in ANCOVA with one or several factor variables and their interactions and some continuous (confounder or control) variables.

We need hypothesis tests, prediction, plots and "post-hoc" analysis (like pairwise comparisons)
We have many of those already but not applicable or specialized for this use case.

several of those tasks should be beginner friendly, for example the plots.

---

example pairwise comparison after OLS or GLM similar to tukeyhsd

My initial plan was to recover the contrast matrices for the restrictions based on the estimated parameters and the encoding used in the creation of the factor/dummy columns in the design matrix.

Nathaniel's idea (discussed with him at pycon) was to use prediction to get the relevant means and contrasts. This has the advantage that by going through patsy, the prediction is independent of the actual encoding of the factor. The main problem is that, in order to call the design matrix for new values of the exog, we need to have enough information to create them in the same ways as they were defined in the original data of the user (i.e. exog_orig)

user: josef-pkt
issue_2361: state: open
title: ENH: interpolators, smoothers and splines
body: The type of models that have only a few (for memory consumption) basis functions, can be estimated and used by our regular models, or by looking at the (nobs, nobs) "kernel" matrix.

Examples: 
Kernel Ridge Regression in sandbox 
scikit smooth
http://mail.scipy.org/pipermail/scipy-dev/2015-April/020617.html  with link to code on scipy-central
(has a nice class structure that could be used in a similar way for subclassing with choice of basis function)

creating a projection matrix based on splines.
Use case: Matthews on scipy-dev: having a single 1d exog x, but many endog y.

Using patsy's bsplines doesn't handle boundaries properly. problems with extrapolation. That will need more work.
(Also using the bspline in a formula with OLS works fine for fitting, but I had an exception in the predict. traceback will follow in next comment.)

```
>>> spb2 = patsy.dmatrix("bs(x, df=10, degree=3, include_intercept=True) - 1", {'x': np.arange(0,10, 0.25)})
>>> project = spb2.dot(np.linalg.pinv(spb2[::4]))
>>> y = np.sin(np.linspace(-np.pi, np.pi, 10))
>>> y2 = np.sin(2*np.linspace(-np.pi, np.pi, 10))
>>> y3 = np.sin(1.5*np.linspace(-np.pi, np.pi, 10))
>>> pred = project.dot(np.column_stack((y,y2,y3)))
>>> plt.plot(np.arange(0,10, 0.25)[::4], np.column_stack((y,y2,y3)), "o")
[<matplotlib.lines.Line2D object at 0x000000000D032D30>, <matplotlib.lines.Line2D object at 0x000000000D032B38>, <matplotlib.lines.Line2D object at 0x000000000D032438>]
>>> plt.plot(np.arange(0,10, 0.25), pred, "+-")
[<matplotlib.lines.Line2D object at 0x000000000D080D68>, <matplotlib.lines.Line2D object at 0x000000000D080518>, <matplotlib.lines.Line2D object at 0x000000000D083080>]
>>> plt.ylim(-1.2, 1.2)
(-1.2, 1.2)
>>> plt.show()
```

user: josef-pkt
issue_2360: state: open
title: REF: use pandas categoricals instead of np.unique
body: we had a recent case where using pandas was faster than using np.unique

with pandas 0.15 it will be possible to merge groups (all unique tuples with several groups)
this is used in cluster robust standard errors for 2 clusters

my guess is that this will be for statsmodels 0.9 or 0.10 
(maybe early as optional implementation)

here is an example (by Jeff as discussed at pycon)

```
In [118]: df = DataFrame({'A' : list('aabcdea'),'B' : list('bbdefgb'), 'C' : range(7)})

In [119]: df
Out[119]: 
   A  B  C
0  a  b  0
1  a  b  1
2  b  d  2
3  c  e  3
4  d  f  4
5  e  g  5
6  a  b  6

In [120]: i = df.set_index(['A','B']).index

In [121]: i
Out[121]: 
MultiIndex(levels=[[u'a', u'b', u'c', u'd', u'e'], [u'b', u'd', u'e', u'f', u'g']],
           labels=[[0, 0, 1, 2, 3, 4, 0], [0, 0, 1, 2, 3, 4, 0]],
           names=[u'A', u'B'])

In [122]: pd.Categorical(i.values).codes
Out[122]: array([0, 0, 1, 2, 3, 4, 0], dtype=int8)

In [123]: i.values
Out[123]: 
array([('a', 'b'), ('a', 'b'), ('b', 'd'), ('c', 'e'), ('d', 'f'),
       ('e', 'g'), ('a', 'b')], dtype=object)


In [125]: pd.Categorical(i.values).codes.astype('i8')
Out[125]: array([0, 0, 1, 2, 3, 4, 0])
```

user: josef-pkt
issue_2359: state: open
title: linear restriction, overparameterized models and patsy
body: Patsy currently does not provide an factor/categorical encoding that does not drop columns that are already linear dependent available in "earlier" factor columns
https://github.com/pydata/patsy/issues/60
https://github.com/pydata/patsy/issues/42
(discussed with Nathaniel at pycon 2015)

I ran into this while looking at a stackoverflow question
http://stackoverflow.com/questions/29261018/how-to-add-sum-to-zero-constraint-to-glm-in-python
Two problems there, patsy does create full dummy sets and statsmodels doesn't implement `fit_constrained` for all models.

user: josef-pkt
issue_2358: state: open
title: ENH: anova_lm type 1: options for term sequence
body: see https://github.com/pydata/patsy/issues/27#issuecomment-93097379

copied here from Nathaniel

For reference, the issue is that patsy does impose some constraints on column order: specifically, it groups together terms so that those which contain the same combination of continuous factors go together, and then within each group it puts lower-order interactions before higher-order interactions. The user request was that they wanted to do a type-I anova, and statsmodels only supported (don't know if this is still true?) type-I anovas where each column was entered from left to right. So they wanted to in particular have some categorical terms, then some continuous terms, then some categorical terms, which violates that "grouping" constraint above.

My current feleing (also expressed more thoughtfully in that thread) is that the best solution is just for statsmodels type-I anova code to support explicit specification of what order you want to enter the terms in. Doing this in patsy is hard because (a) I actually think the current behaviour is nicer for most use cases, so am reluctant to de-optimize user experience in general just to improve type-I anovas (which are almost never the right thing anyway, and rarely used outside of introductory classes), and (b) it's not clear that patsy can fix this entirely, since in general type-I anovas might want almost any ordering of columns, and patsy can't really support that without extreme contortions. Allowing y ~ a + x + b would not be too hard, but y ~ a:b + a:x + a + b:x would be very difficult and intrusive (and it's not even clear how it would work).

user: josef-pkt
issue_2357: state: open
title: BUG/ENH quantreg, add starting values, try to prevent cycle
body: If quantreg ends up in a convergence cycle, then it's not possible to try to get out of it by changing the starting values.

http://stats.stackexchange.com/questions/145548/statsmodels-quantreg-convergence-cycle-warning

commented code: we cannot choose beta/params. we would have to change the initial xstar, i.e. wexog (not quite since it doesn't use sqrt for weighting `wx * x`) during iteration

477c92916d907ea0d5754a858edb997b00af4595  in PR #896

original comment about cycle
https://github.com/statsmodels/statsmodels/pull/655#issuecomment-19872975

user: josef-pkt
issue_2355: state: open
title: ENH: add empirical autocovariance and autocorrelation for multivariate time series VAR
body: while looking at #2354, I don't see any empirical acf and acovf for VAR.
 VARProcess has implied autocovariance as `acf`

related but different: multiple univariate acf #891

I found `_compute_acov(x, nlags=1)` in var_model.py, and there is a sample_acov method in VARResults. This has no options besides nlags, always demeans.

user: josef-pkt
issue_2350: state: open
title: TST more unit tests for GLM - discrete
body: test_glm.py has `CheckComparisonMixin`  but it is not used for all families, and doesn't check enough of the attributes.

example: GLM `llnull` has no test coverage, but glm and discrete should coincide, see #2349
(I just checked it for a Logit example)

for a possible problem for GLM llnull in gaussian family, see next comment

user: josef-pkt
issue_2349: state: open
title: NegativeBinomial vs summary2(): like Hayek vs Keynes
body: I don't know why but summary2() is _not_ getting along with NegativeBinomial.

rhDNase2.txt

```
"id" "trt" "fev" "count" "time"
493301 1 28.8 0 168
493303 1 64 0 169
493305 0 67.2 2 168
493309 1 57.6 0 168
493310 0 57.6 0 171
....
```

bug.py

```
import statsmodels.api as sm
import pandas

print(sm.version.full_version)

cystic = pandas.read_csv("./rhDNase2.txt", sep=" ", index_col=False)

model = sm.GLM.from_formula("count ~ C(trt) + fev", data=cystic, family=sm.families.NegativeBinomial())
model.exog_names[1] = "trt"
fit = model.fit()
print(fit.summary()) #works
print(fit.summary2())#->TypeError: loglike() got an unexpected keyword argument 'scale'
```

output

```
[kousu@galleon A4]$ python bug.py 
0.6.1
                 Generalized Linear Model Regression Results                  
==============================================================================
Dep. Variable:                  count   No. Observations:                  645
Model:                            GLM   Df Residuals:                      642
Model Family:        NegativeBinomial   Df Model:                            2
Link Function:                    log   Scale:                   1.40673520006
Method:                          IRLS   Log-Likelihood:                -914.54
Date:                Fri, 03 Apr 2015   Deviance:                       775.69
Time:                        01:11:12   Pearson chi2:                     903.
No. Iterations:                     8                                         
==============================================================================
                 coef    std err          z      P>|z|      [95.0% Conf. Int.]
------------------------------------------------------------------------------
Intercept      1.1639      0.175      6.663      0.000         0.822     1.506
trt           -0.2740      0.132     -2.079      0.038        -0.532    -0.016
fev           -0.0165      0.003     -6.172      0.000        -0.022    -0.011
==============================================================================
Traceback (most recent call last):
  File "bug.py", line 12, in <module>
    print(fit.summary2())#->TypeError: loglike() got an unexpected keyword argument 'scale'
  File "/usr/lib/python3.4/site-packages/statsmodels/genmod/generalized_linear_model.py", line 1124, in summary2
    xname=xname, yname=yname, title=title)
  File "/usr/lib/python3.4/site-packages/statsmodels/iolib/summary2.py", line 137, in add_base
    info = summary_model(results)
  File "/usr/lib/python3.4/site-packages/statsmodels/iolib/summary2.py", line 293, in summary_model
    out[key] = func(results)
  File "/usr/lib/python3.4/site-packages/statsmodels/iolib/summary2.py", line 283, in <lambda>
    info['LL-Null:'] = lambda x: "%#8.5g" % x.llnull
  File "/usr/lib/python3.4/site-packages/statsmodels/tools/decorators.py", line 94, in __get__
    _cachedval = self.fget(obj)
  File "/usr/lib/python3.4/site-packages/statsmodels/genmod/generalized_linear_model.py", line 984, in llnull
    return self.family.loglike(self._endog, self.null, scale=self.scale)
TypeError: loglike() got an unexpected keyword argument 'scale'
[kousu@galleon A4]$ 
```

user: kousu
issue_2348: state: open
title: distance covariance and correlation
body: Just wanted to park your gist here since I came across it this week, looking for similar. I don't know if it's something we want to include or if it belongs in scipy or scikit-learn (I don't think it's already there?).

https://gist.github.com/josef-pkt/2938402

There's a O(n) memory and O(n*log(n)) algorithm here that wouldn't be too hard to implement (and put into Cython)

http://arxiv.org/pdf/1410.1503v1.pdf

user: jseabold
issue_2346: state: open
title: pairwise_tukeyhsd - retrieve "reject" values
body: I was thinking about how to get the numeric values of "reject" column.

```
In [140]: print(tukey_result.summary())
Multiple Comparison of Means - Tukey HSD,FWER=0.05
=============================================
group1 group2 meandiff  lower   upper  reject
---------------------------------------------
ger    tbw   14.1319  -3.3327 31.5966 False 
ger    tww   30.2923  11.3796  49.205  True 
tbw    tww   16.1604  -0.9708 33.2915 False 
---------------------------------------------
```
1. `tukey_result.reject` returns boolean results the same for attribute `reject2`. Where I can find the numeric float values of `p`?
2. why the simple `tukey_result.summary()` does not print the summary table?

Cheers,
Arnaldo. 

user: arnaldorusso
issue_2343: state: open
title: Gee fixed covstruct
user: gjhunt
issue_2342: state: open
title: Can't compute BIC on high-dimensional time series
body: Hi,

I'm trying to use StatsModels on fMRI timeseries data. 

You can download the data here http://www.nimh.nih.gov/labs-at-nimh/research-areas/clinics-and-labs/chp/research-articles.shtml You have to remove the _146387.txt when saving the file.

from scipy import io

data=io.loadmat('NV_COS_data.mat')
timeseries_mat=data['NVdata']['timeseries'][0][0][0][0]
timeseries_mat = timeseries_mat.T

import statsmodels
model = statsmodels.tsa.vector_ar.var_model.VAR(timeseries_mat)
results=model.fit(5)
print results.bic

It works for array sizes up to model = statsmodels.tsa.vector_ar.var_model.VAR(timeseries_mat[:,0:130]) 
timeseries_mat[:,0:129] still works.

I get the error  
....
   ld = logdet_symm(self.sigma_u_mle)
  File "/usr/local/lib/python2.7/dist-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/tools/linalg.py", line 213, in logdet_symm
    c, _ = linalg.cho_factor(m, lower=True)
  File "/usr/lib/python2.7/dist-packages/scipy/linalg/decomp_cholesky.py", line 132, in cho_factor
    check_finite=check_finite)
  File "/usr/lib/python2.7/dist-packages/scipy/linalg/decomp_cholesky.py", line 30, in _cholesky
    raise LinAlgError("%d-th leading minor not positive definite" % info)
numpy.linalg.linalg.LinAlgError: 130-th leading minor not positive definite

Any help would be appreciated. 

Maria

user: mariakesa
issue_2341: state: open
title: ENH: complex step derivative
body: It's very accurate for first derivatives but doesn't work for all functions.

eg. scipy.stats.norm.cdf doesn't work, but `erf` does
we can fix Probit and similar models for this (although we know the derivative of the cdf already)

```
>>> (0.5+0.5*special.erf((1+ 1e-8 * 1j)/np.sqrt(2))).imag / 1e-8  - stats.norm.pdf(1)
8.3266726846886741e-17
```

collection of function extended to complex so they work with complex step derivatives
see fortran version here
http://mdolab.engin.umich.edu/content/guide-complex-step-derivative-approximation-0

found in here http://blogs.mathworks.com/cleve/2013/10/14/complex-step-differentiation/
with more references

user: josef-pkt
issue_2340: state: open
title: Granger Causality Tests
body: Hi, 
Thanks in advance. Any insight into this would be greatly appreciated (if bug or interpretation).

I am using the grangercausalitytests in statsmodels. As a test, I have two identical time-series X1 and X2 (they are identical). I would expect to verify that with a given large set of lags that X2 is granger causal X1 or the opposite (If I understand the function correctly).

The p-values I get for the ssr-ftest is very high (between 0.8 and 1.0) indicating that the null hypothesis is not true and the alternative hypothesis likely is. Since the null hyphothesis is only not true if p-value is less than a critical value (say < 0.05), and true if the p-value is high. In my case for the above example, the null hypothesis is therefore true, which according to the documentation is that x2 is NOT granger causal to x1. Given that they are the same time series with lag 10 or more, I am surprised by this result. 

Please help. Any insight into this would be greatly appreciated (if bug or interpretation).

The doc on the function from the website:

"The Null hypothesis for grangercausalitytests is that the time series in the second column, x2, does NOT Granger cause the time series in the first column, x1. Grange causality means that past values of x2 have a statistically significant effect on the current value of x1, taking past values of x1 into account as regressors. We reject the null hypothesis that x2 does not Granger cause x1 if the pvalues are below a desired size of the test."

user: kudva
issue_2339: state: open
title: Initial version ``statsmodels.resampling`` with Efron's bootstrap
body: Here's a start on the `statsmodels.resampling` package. At the moment it's simple -- one function to estimate confidence intervals of an arbitrary statistic (using a generator for memory efficiency).

user: edschofield
issue_2338: state: open
title: nan handling in X of ARIMAX ?
body: question based on fixing the arimax test in PR #2337 : 

Do we have proper missing handling for initial conditions for exog in ARIMAX?.  
nan was introduced by `exog.diff()` (by user/test not by ARIMA)

I assume that if we have ARIMA with differencing, then we loose an initial endog observation, so the initial exog observation is irrelevant, but we (need to) have checks for length, pandas indices, missing, ...

user: josef-pkt
issue_2331: state: open
title: MixedLM from_formula, confusing *args, **kwargs
body: A user trying to use cov_struct in MixedLM as keyword argument
http://stackoverflow.com/questions/29147151/python-cov-struct-attribute-for-mixedlm

The signature does not indicate what's allowed and required.

Not clear: do we need to keep awful signatures to be consistent within the class hierarchy?

Related: issue ??? where we don't see incorrect keywords if we use `*args` and `**kwargs` without verification

user: josef-pkt
issue_2328: state: open
title: ENH: API for outlier_test in OLSResults
body: I'm running the outlier test with plain numpy arrays as data, and the return of outlier_test is just an array with 3 columns. This should be made more convenient and informative, maybe return a results class, and an option to show only the top results.
(some of this is redundant if pandas has been used for the model data) 

Also, the function in outlier_influence has  `labels` and `order` options but those are not available in the OLSResults method.

(there should be related issues for extending outlier_test to other models.)

user: josef-pkt
issue_2326: state: open
title: Bug?  initial conditions in statespace_local_linear_trend.ipynb
body: The plot and fittedvalues show zero for the first observation. `llf_obs` is much larger for the first two observations.

Are these a consequence of default initialization of the statespace model?
Are they, or can the initial observation be ignored in the calculation of the Hessian?

Hessian is not positive definite and bse and rest shows nans in summary table

```
In [22]:mod.endog, res.fittedvalues(), res.llf_obs
Out[22]:
(array([[ 6.96129605,  7.04141166,  7.05272105,  6.9902565 ,  6.76272951,
          6.8134446 ,  6.68959927,  6.56385553,  6.41345896,  6.47697236,
          6.31173481,  6.31896811,  6.34388043,  6.4035742 ,  6.29341928,
          6.29341928,  6.41673228,  6.36475076,  6.48157713,  6.59850903,
          6.47543272,  6.44888939,  6.39859493,  6.18208491,  6.1737861 ,
          6.08904488,  6.00141488,  6.08221891,  5.99146455,  6.06610809,
          5.98141421,  6.07073773,  6.02827852,  5.93753621]]),
 array([[ 0.        ,  6.96129602,  7.1215273 ,  7.10650987,  7.01524931,
          6.73584183,  6.72001095,  6.61683616,  6.47777271,  6.30642515,
          6.36428315,  6.24584327,  6.24318122,  6.29795578,  6.39614529,
          6.30939695,  6.27742018,  6.4070299 ,  6.39333111,  6.50453335,
          6.65531308,  6.54603077,  6.45799668,  6.37394338,  6.12442775,
          6.06270173,  5.99480849,  5.91557807,  6.01160178,  5.96905913,
          6.04539687,  5.9846274 ,  6.06166943,  6.04229634]]),
 array([-7.82671804, -7.82669382,  0.90475446,  0.73236801, -1.53847562,
         1.06730176,  1.30591105,  1.21809647,  1.15615694, -0.00851262,
         1.22045091,  1.09972596,  0.87594676,  0.82856237,  0.85669728,
         1.33745943,  0.44326465,  1.2659223 ,  0.98580229,  0.93705686,
        -0.16130164,  0.90880972,  1.18463664, -0.36919037,  1.23563485,
         1.31697859,  1.34734057,  0.05289226,  1.33044589,  0.90964764,
         1.15824796,  1.00318757,  1.29732337,  0.83699252]))
```

user: josef-pkt
issue_2324: state: open
title: Adding fast permutation tests
body: Recently, I've been developing optimized permutation tests and have been attempting to push them into scipy.  I have a permutation t-test PR [here](https://github.com/scipy/scipy/pull/4440) and a permutation f-one way ANOVA PR [here](https://github.com/scipy/scipy/pull/4519)

I'm beginning to think that these algorithms may not be appropriate for scipy, because they were designed strictly for contingency tables - and the algorithm essentially breaks when attempting to account for non-uniform 2D arrays.

But these permutation algorithms offer 100x speedup over the naive implementation of the permutation test with the proper matrix libraries installed.  

Would these algorithms fit better in statsmodels?

user: mortonjt
issue_2314: state: open
title: REF: change wendog -> transformed_endog or something perhaps model-specific
body: The name `wendog` is an old holdover from `scipy.stats.models` and is really not informative, especially in the case of multinomial models where it's the dummied multinomial response. We can keep an alias `_wendog` around for cases where it would facilitate code re-use.

user: jseabold
issue_2305: state: open
title: ENH: More informative error message in fit_regularized Hessian inversion fail
body: See #659.

user: jseabold
issue_2304: state: open
title: ENH: Regularize the intercept?
body: Should we regularize the intercept? Shouldn't it be in or out of the model? We have data on the (first) non-varying column in `model.data.const_idx`.

user: jseabold
issue_2295: state: open
title: ENH: diagnostic plots for count regression
body: some ideas in
http://stats.stackexchange.com/questions/70558/diagnostic-plots-for-count-regression/70581#70581

user: josef-pkt
issue_2294: state: open
title: weighted average prediction
body: for average effect calculations we might want to choose weights to reflect a target population

example: `balanced` in LSMEANS or Stata's contrast

see also http://stackoverflow.com/questions/28686966/weighting-pandas-dataframe-with-auxiliary-variables
"raking" to get weights, and
http://stackoverflow.com/questions/28679026/weighting-values-based-on-variables

`margins` doesn't have weighted mean for reduction

user: josef-pkt
issue_2293: state: open
title: penalized nonlinear MLE, generalized linear model and discrete models
body: http://stats.stackexchange.com/questions/139306/penalized-ml-estimation-of-non-linear-probit
see Firth and Heinze, also Gelman, adding a prior to make an infinite parameter in perfect separation to be finite

---

(there doesn't seem to be another issue yet for implementing maximum penalized likelihood and similar

two references that I finally read

Fan, Jianqing, and Runze Li. 2001. “Variable Selection via Nonconcave Penalized Likelihood and Its Oracle Properties.” Journal of the American Statistical Association 96 (456): 1348–60. doi:10.1198/016214501753382273.
Fan, Jianqing, and Jinchi Lv. 2010. “A SELECTIVE OVERVIEW OF VARIABLE SELECTION IN HIGH DIMENSIONAL FEATURE SPACE.” Statistica Sinica 20: 101–48.

Kerby started with coordinate descent as optimization method which is widely recommended as prefered choice.
Fan/Li use local quadratic approximation, which might be easy to implement and "should" work well for "smaller" problems

also penalization functions: Fan/Li introduce SCAD which doesn't penalize large values of the parameters. (Kerby started with a penalization function module)

---

Implementing penalized likelihood or penalize M-estimators when using scipy optimizers instead of our own, specifically written optimization

just an idea  
add a Mixin class that adds the penalization term to loglike, score and maybe hessian.
Automatically adjust the default cov_params to the sandwich form. sandwich cov_params in Fan/Li looks standard, same structure as in my Ridge/TheilGLS PR.
Fan/Li use covariance of score_obs which demeans, and I don't think our sandwiches have this (only relevant if mean is not zero. Mean score is zero by construction in exactly identified models, i.e anything except GMM?)

user: josef-pkt
issue_2292: state: open
title: ENH: extracting categorical info from patsy
body: we need the information how categorical exog are encode
either for exog created by formula, or as a general description (_)
(_) we should define meta information for params and exog independently of whether `from_formula` was used or not. 

for example predict is able to convert exog with stateful transform including categoricals. How does this work? I haven't checked.

use cases:
- Margins: currently not implemented for categoricals except for binary (dummy)
- predict at: similar to margins case
  we need to fix the categorical to one level for all observations, e.g. set on dummy to 1 and the other dummy variables to zero.
- all pairs t_test

user: josef-pkt
issue_2288: state: open
title: Propensity Score matching
body: This is a Work In Progress of the Propensity Score matching algorithm: #858 

user: jburroni
issue_2282: state: open
title: Logit: strong but not perfect (?) separation
body: see http://stackoverflow.com/questions/28568623/statsmodels-logistic-regresion-type-problems/28590006#28590006

optimization fails to converge, most observations are predicted close to 0 or 1, standard errors of some parameters are huge, eigenvalues of `exog` don't have a problem (in rescaled version)
but we don't get perfect separation, and no perfect separation exception.

GLM with IRLS has all `params` at around `1e+14`

user: josef-pkt
issue_2280: state: open
title: DOCs case insensitive operating system pca/PCA
body: another case for problem on windows in creating the documentation because filenames don't distinguish in capitalization 

pca.PCA
pca.pca

pca function is missing in my html docs, similar to links lower case and capitalized.

also pca is both function and module name which we mostly try to avoid

(just a complaint and as reminder because I spent some time looking for the problem why pca is not in my rendered documentation, I don't know if we can do anything here)

user: josef-pkt
issue_2276: state: open
title: Performance: VAR select_order
body: I think we can avoid the repeated creation of `get_var_endog` by using a single QR decomposition and update the R matrix.

This should speed up order selection.

user: josef-pkt
issue_2274: state: open
title: SUMM   VAR cleanup and refactor
body: (some preliminary notes on VAR, while looking at bugs and finding more of those.)

several open issues, bugs, non-informative names, insufficient test coverage

However, I think the overall structure is pretty good, well separated, modularized classes, although it can get confusing to trace the flow. (see non-orthogonal irf changes #???)

The only structural problem is that VAR.fit has arguments that are supposed to be in `__init__` to avoid "stale state", i.e. changing and messing up the model while result instances hold on to the model. (#769, #2273)

Skipper started refactoring in #815 
It fixes several bugs, but needs work, and more test coverage. It also has a SUMMary in the description

another renaming suggestion

cov_alpha -> cov_ar_coeffs or cov_ar_params    relevant for dynamics, e.g. irf issue #1636

cleaning up VAR, and SVAR and related is "quite a bit of" work.

Since we don't have a maintainer or developer with enough time, it would be better for these kind of changes to work in incremental PRs, which are easier to finish, review and merge.
except in this case we already have a refactoring branch
(and I have the same problem with `robust`)

user: josef-pkt
issue_2273: state: open
title: REF  VAR, trend needs to be in `__init__` - wrong model state
body: calling fit twice with different trend, first default, then trend='nc', raises exception for

no code and traceback, because copy pasting from notebooks is annoying !! (especially when I'm tired). Where's my spyder?

this is (maybe partially) a duplicate of #769  with example `ic` instead of trend

user: josef-pkt
issue_2262: state: open
title: BUG: Kalman filter with failures with single precision input
body: Inconsistent results across platforms. Test failures in:
- test_kalman:TestClark1987Single
- test_kalman:TestClark1987SingleComplex
- test_representation:TestClark1987Single
- test_representation:TestClark1987SingleComplex

These pass on at least one system (Mac OS X 10.9.5, Homebrew Python 2.7.6) but fail on another (Travis' Linux, Anaconda).

Need to:
1. [ ] Identify point of failure (overflow? byproduct of recursions? something else?)
2. [ ] Add substantially more unit tests (existing tests are very basic)
3. [ ] Widen test platforms / machines

Follows up on #2252.

user: ChadFulton
issue_2256: state: open
title: Design: handling of transformed parameters
body: There are different problems when we reparameterize for optimization

for SARIMAX: https://github.com/statsmodels/statsmodels/issues/#2252#issuecomment-74252465
for MixedLM: #2231

frequently used design (Skipper's code)  set transform_params flag as attribute for likelihood calculation.

in fit_restricted or fit_transformed, I create a separate transformed model, but it's because of the data transformation and not just parameter transformation.

There is also a constraint fit with parameter transformation for restrictions in GenericLikelihoodModel. IIRC there is no choice in that case once the transformation, restriction has been set, and results params, bse,... are for transformed params.

user: josef-pkt
issue_2248: state: open
title: REF naming  IV, IV2SLS use `regressor` not `exog`
body: I'm still thinking about naming for endog on right hand side `y = b1 y + b2 x`

suggestion 

signature has `IV2SLS(endog, regressor, instruments, ...)`

but docstring says `regressors are stored and attached as`exog``

not very nice but at least the signature is better, and we can still reuse all existing automatic data handling

user: josef-pkt
issue_2239: state: open
title: The fit_regularized docs are a bit unclear and unfinished
body: The documentation for the `trim_mode` helper keywords look off/unfinished, copy-pasted below. 

Without looking at the code my guess is that `size_trim_tol` is ignored if `trim_mode` is 'auto'. Is this not the case?

`auto_trim_tol` looks unfinished. What is it's relation to `qc_tol`?

I think we need to be very explicit about what exactly this is doing.

Also, all of the "above" notes should be "below"?

It would also be great to have some references to the literature in the docstring for this. Just naively running things, I almost always get the QC check failed warning. I have no idea what this means and if we're not going to give guidance on what should happen if the QC check failed (we should somewhere no?), then we should link to some relevant literature. (Related to #1455.)

```
trim_mode : 'auto, 'size', or 'off'
    If not 'off', trim (set to zero) parameters that would have been
    zero if the solver reached the theoretical minimum.
    If 'auto', trim params using the Theory above.
    If 'size', trim params if they have very small absolute value
size_trim_tol : float or 'auto' (default = 'auto')
    For use when trim_mode == 'size'
auto_trim_tol : float
    For sue when trim_mode == 'auto'.  Use
```

user: jseabold
issue_2235: state: open
title: ENH: add `eform` exponentiated parameters
body: (I thought I already opened a specific issue, but don't find one.)

application for example for risk ratio #2215
implementation is already largely done in #2151

copying my comment from
https://github.com/statsmodels/statsmodels/issues/2208#issuecomment-71389696

You are using `exp(params)` in your notebook. Stata optionally allows this as and option like irr or or as an eform option. I looked into it (creating a eform params table) and it's pretty easy to do, but wasn't sure about all the different names and interpretation exp(params) has across models in Stata.
If you are interested and have some ideas about an interface, then this could be added as a post-estimation method or function for statsmodels 0.8.
(the exp params table is a bit inconsistent: the parameters and standard errors are from the transformation, the conf_int is by endpoint transformation, and the t/z statistic and p-value is for params==0 or exp(params) == 1)

user: josef-pkt
issue_2231: state: open
title: ENH/Design mixedlm params versus summary
body: I'm checking t_test and wald_test for mixed_lm using the examples/notebooks/mixed_lm_example

summary shows different params for the random effects than the attribute in the second example. design or bug?

```
md = smf.mixedlm("Weight ~ Time", data, groups=data["Pig"], re_formula="~Time")
mdf = md.fit()
print(mdf.summary())

              Mixed Linear Model Regression Results
=================================================================
Model:               MixedLM    Dependent Variable:    Weight    
No. Observations:    861        Method:                REML      
No. Groups:          72         Scale:                 6.0374    
Min. group size:     11         Likelihood:            -2217.0475
Max. group size:     12         Converged:             Yes       
Mean group size:     12.0                                        
-----------------------------------------------------------------
                       Coef.  Std.Err.   z    P>|z| [0.025 0.975]
-----------------------------------------------------------------
Intercept              15.739    0.550 28.609 0.000 14.661 16.817
Time                    6.939    0.080 86.927 0.000  6.783  7.095
Intercept RE           19.493    1.572                           
Intercept RE x Time RE  0.294    0.154                           
Time RE                 0.416    0.033                           
=================================================================

In [21]: mdf.params
Out[21]:
Intercept                 15.738766
Time                       6.939014
Intercept RE               1.796876
Intercept RE x Time RE     0.027083
Time RE                   -0.261114
dtype: float64

In [18]: mdf.t_test(np.eye(len(mdf.params)))
Out[18]:
<class 'statsmodels.stats.contrast.ContrastResults'>
                             Test for Constraints                             
==============================================================================
                 coef    std err          z      P>|z|      [95.0% Conf. Int.]
------------------------------------------------------------------------------
c0            15.7388      0.550     28.609      0.000        14.661    16.817
c1             6.9390      0.080     86.927      0.000         6.783     7.095
c2             1.7969      0.640      2.808      0.005         0.543     3.051
c3             0.0271      0.063      0.433      0.665        -0.096     0.150
c4            -0.2611      0.013    -19.386      0.000        -0.288    -0.235
==============================================================================
```

user: josef-pkt
issue_2220: state: open
title: SUMM: test coverage
body: Let's target 0.8 for some improvement in test coverage.
We need a list of worst offenders in modules that are (almost) "core"
- [ ] #1920 GEE covariance_type, maybe outdated
- [ ] #782 emplike/originregress

`stats`
- [ ] #812 gof, almost no coverage
- [ ] gaps in ...
  - [ ] contrast 87% maybe higher, some lines are untested, `Contrast` has test (where? usage?)
  - [ ] sandwich covariance, 79%, some unused code (HCx) without tests, some untested options
  - [ ] base.py 62% contains only a results class, untested summary and similar
  - [ ] other between 80-90% with some parts/lines/options not covered by unit tests
- [ ] descriptive_stats very low: obsolete or not?

we should get an automatic list of core modules with coverage  (list of all files is too noisy)

user: josef-pkt
issue_2218: state: open
title: REF:  cleanup wald test contrast df_xxx
body: Bad naming !

ContrastResults uses df_denom for the p-value
`self.pvalue = self.dist.sf(self.statistic, df_denom)`

This should be a serious bug. The relevant lines show that they have test coverage. 
(This got me worried for some time. test failure in `statsmodels.base.tests.test_generic_methods.TestGeneric....test_ftest_pvalues` if I make a change to df in chi2, no failure in regression or discrete tests, i.e. only conistency checks with verified pvalues)

However, df_denom in chi2 is not the same as df_denom in the F-test. It's the same as df_num.

https://github.com/statsmodels/statsmodels/commit/dbbf05a02d9490fe4c33d3ef0bf5c4fef9350db0#diff-b165b4bd4e10edd0dfb485e47c562b2cL1518

my problem: I never know which is the numerator and which the denominator
switch to generic `df_constraints` instead of `df_num`

aside: df_constraints for wald test are defined in terms of shape. Using rank should be safer. Also we don't check for "estimability" which I still haven't fully figured out.

user: josef-pkt
issue_2217: state: open
title: ENH: asymptotic power for local alternatives
body: mainly an idea while looking at what should be in the wald test results (wald_test_terms) and whether we need to attach the scipy.stats distribution by default.

Most econometrics articles on wald, score and conditional moment tests have local power formulas.
It's mainly keeping track of the non-centrality parameter and defining "effect size" or local alternative.
We have the required distributions in scipy.stats and they are already used in the current `power` functions.

**edit**
#1758 is the more general issue, when we also want to use approximation for non-local alternatives, and take different variances and Hessian between null and alternative as in GLM into account.

user: josef-pkt
issue_2216: state: open
title: ENH: permutation tests
body: just two links for now
https://github.com/pbstark/permute  (*)
https://github.com/scipy/scipy/issues/4399

(*) readme has link to two books with associated code and data

user: josef-pkt
issue_2215: state: open
title: ENH: estimate risk ratio in binomial model
body: based on the comments in #2208 

`risk ratio` is a useful effects measure that is often used in medical, epidemiological studies, and I guess other areas.

It makes a good use, example and test case for several extensions on top of GLM or Logit, and there is a large literature for it.
- eform: exponentiated parameters as in Stata, `rr` is eform in log-binomial model
- predictive margins: average treatment effect, ATTreated, ATUntreated see Bender Kuss 2010  (NNT, EIN, NNE)  (partially specific to a binary exog, not as general as DiscreteMargins)
- log-binomial model for out-of domain GLM/LEF as in #2208 
- comparison of different models, e.g. Poisson with robust standard errors because of variance misspecification.

Bender Kuss 2010
Ralf Bender, Oliver Kuss, Methods to calculate relative risks, risk differences, and numbers needed to treat from logistic regression, Journal of Clinical Epidemiology, Volume 63, Issue 1, January 2010, Pages 7-8, ISSN 0895-4356, http://dx.doi.org/10.1016/j.jclinepi.2009.07.007.

Bender Kuss respond to this article by Austin, which derives the population average RR:

Peter C. Austin, Absolute risk reductions, relative risks, relative risk reductions, and numbers needed to treat can be obtained from a logistic regression model, Journal of Clinical Epidemiology, Volume 63, Issue 1, January 2010, Pages 2-6, ISSN 0895-4356, http://dx.doi.org/10.1016/j.jclinepi.2008.11.004.

(They take the average of the probability given treated and given untreated separately and then divide. I would have calculated the RR directly for each observation and then average the RR. 
I've seen also in another article the same averaging as Austin.)

and to finish the list:
the response of Austin to Bender and Kuss with reference to the econometrics literature for the difference between ATE and ATT
Peter C. Austin, Different measures of treatment effect for different research questions, Journal of Clinical Epidemiology, Volume 63, Issue 1, January 2010, Pages 9-10, ISSN 0895-4356, http://dx.doi.org/10.1016/j.jclinepi.2009.07.006.

user: josef-pkt
issue_2208: state: open
title: GLM domain violations with non-canonical links
body: IRLS produces results with no warnings when the fitted means are outside the family domain.  It's not clear that the solution is actually an MLE or if the standard errors, etc. are meaningful.  An example is below.

See also #1932, #2037

```
import statsmodels.api as sm
import numpy as np

np.random.seed(823)

n = 1000
exog = np.random.normal(size=(n, 3))
exog[:, 0] = 1

true_params = np.r_[-0.5, 0.1, -0.1]
lin_pred = np.dot(exog, true_params)

endog_mean = np.exp(lin_pred)

endog = 1*(np.random.uniform(size=n) < endog_mean)

lnk = sm.families.links

model_irls = sm.GLM(endog, exog, family=sm.families.Binomial(link=lnk.log))
result_irls = model_irls.fit(method="IRLS")

model_gradient = sm.GLM(endog, exog, family=sm.families.Binomial(link=lnk.log))
result_gradient = model_gradient.fit(start_params=result_irls.params, method="newton")

print(result_irls.summary())
print(result_gradient.summary())

print("Gradient at IRLS solution:")
print(model_gradient.score(result_gradient.params))

# Some predicted values are out of range:                                                                                 
print(result_irls.predict().max())
```

user: kshedden
issue_2207: state: open
title: Trend in tsa models: standardize
body: trend polynomials like `t**range(order)` are numerically pretty bad if `t` are integers of `range(nobs)`.
Instead we should either normalize `t` so it is in [-1,-1] or use numpy's orthogonal polynomials, e.g. `chebvander`. The first still allows the "usual" interpretation, the second takes a bit more effort in interpretation.

thought about while review kalman filter SARIMAX, but it also is in ARIMA and VAR. It should be consistently handled if it is changed.
I didn't check the current implementation.

user: josef-pkt
issue_2206: state: open
title: VAR missing handling
body: see https://groups.google.com/d/msg/pystatsmodels/MjW4PgSg9pc/A21FG2Br5k4J

I think it should be relatively ease to drop nans in the constructed y and x matrices, where x includes the all the lagged AR terms.

user: josef-pkt
issue_2205: state: open
title: OLS/WLS with multivariate endog: what breaks
body: I'm surprised 2-D endog breaks only in summary:
http://stackoverflow.com/questions/28122820/statsmodel-multivariate-ols-error-matrices-are-not-aligned

the dot product could be fixed with a transpose.

We might actually get some things out of the current models for multivariate y. Worth a try.
tvalues, bse won't work though.

proper solutions: #2203 and sysreg models

user: josef-pkt
issue_2203: state: open
title: add `regression.special_linear_model` 
body: I have been thinking for a while to add some shortcut and stripped down linear models into regression. 
Stripped down means that not all functionality and results are available, i.e. "limited" models.

One possible suggestion for the module name is `special_linear_model`.
I thought of `simplified` as a name but it's a misnomer.

examples: 
- OLS based on summary statistic (e.g. covariance matrix)
- parallel OLS: multivariate endog with shortcuts

user: josef-pkt
issue_2198: state: open
title: use start_params in `arma_order_select_ic`
body: if max_ar or max_ma are large, then there might be problems in the optimization.
There might also be other problems, for example
http://stackoverflow.com/questions/28056601/the-computed-initial-ar-coefficients-are-not-stationary

(I never run examples for this)

One reasonably simple improvement that could work quite well
`arma_order_select_ic` checks models of increasing order but it doesn't use the parameters from the smaller model as `start_params` for the larger model.

one possible implementation
cache the parameters in a dictionary with keys (k_ar, k_ma) and use the first of the following that is available
1) (k_ar, k_ma - 1)
2) (k_ar - 1, 0)
3) (0, k_ma - 1) 

2) handles pure AR models and the initial order selection path of an ARMA model
3) handles pure MA models.

This should both be more robust and faster, more robust if the initial model AR(1) (or AR(0)?) is easy too estimate without stationarity problems.

user: josef-pkt
issue_2190: state: open
title: mosaicplot should handle crosstab and frequency-count DataFrames
body: I have this dataset, in frequency-count form:
`speeddatting.csv`

```
DecisionM,IntelligentM,freq
0, 5, 9
0, 6, 21
0, 7, 35
0, 8, 35
0, 9, 14
0, 10, 10
1, 5, 11
1, 6, 12
1, 7, 30
1, 8, 48 
1, 9, 27
1, 10, 16
```

I want to make a mosaic plot of it. This should be easy; the data is already counted for us. For reference, I am looking for this:
![expected output](http://i.imgur.com/SXqbmMr.png)

But this does not work:

```
import pandas
import statsmodels
import matplotlib.pyplot as plt
from statsmodels.graphics.mosaicplot import mosaic

D = pandas.read_csv("speeddating.csv")
print(D)
```

`mosaic()` ignores the `freq` column (I guess I never told it to look there, so fair enough), decides there are equal amounts (i.e. 1) of each case, and I see
![fail#1](http://i.imgur.com/Ts8zORP.png)

The docs for `mosaic()` don't give any insight (and by the way, currently the "Using a DataFrame as source," section isn't being parsed as code by your Sphinx run: http://statsmodels.sourceforge.net/devel/generated/statsmodels.graphics.mosaicplot.mosaic.html).

I tried reformatting

```
G = {(I,d) : D.loc[(D.DecisionM == d) & (D.IntelligentM==I), "freq"].iloc[0] for d in [0,1] for I in range(5,10)}

mosaic(G)
plt.show()
```

But this is awkward to write and comes out wrong because dictionaries are unordered (I see that I could fix that with the `index` parameter, but how to use that is unclear to me and just makes the code above even more awkard).
![fail#2](http://i.imgur.com/3pFUtVH.png)

The way I got that first (correct) image was computationally expensive: I rewrote the DataFrame in the form `mosaic()` wants and let it crunch it up again:

```
import itertools
flatten = lambda L: list(itertools.chain.from_iterable(L))

G2 = flatten([[(I,d)]*D.loc[(D.DecisionM == d) & (D.IntelligentM==I), "freq"].iloc[0] for d in [0,1] for I in range(5,10)])
G2 = pandas.DataFrame(G2)
G2.columns = ["IntelligentM", "DecisionM"]
print(G2)

mosaic(G2,["IntelligentM", "DecisionM"])
plt.show()
```

To be clear, G2 looks like:

```
     IntelligentM  DecisionM
0               5          0
1               5          0
2               5          0
3               5          0
4               5          0
5               5          0
6               5          0
7               5          0
8               5          0
9               6          0
10              6          0
11              6          0
12              6          0
13              6          0
14              6          0
15              6          0
16              6          0
17              6          0
18              6          0
19              6          0
20              6          0
21              6          0
22              6          0
23              6          0
24              6          0
25              6          0
26              6          0
27              6          0
28              6          0
29              6          0
..            ...        ...
212             8          1
213             8          1
214             8          1
215             9          1
216             9          1
217             9          1
218             9          1
219             9          1
220             9          1
221             9          1
222             9          1
223             9          1
224             9          1
225             9          1
226             9          1
227             9          1
228             9          1
229             9          1
230             9          1
231             9          1
232             9          1
233             9          1
234             9          1
235             9          1
236             9          1
237             9          1
238             9          1
239             9          1
240             9          1
241             9          1

[242 rows x 2 columns]
```

This is even more verbose than the last because `mosaic()` demands labelled columns. It's also slow. I want to avoid having to do this, but it's the only way I've found so far.

Further, sometimes contingency data comes in crosstab form; in fact I can generate it from the above:

```
In [4]: pandas.crosstab(G2.IntelligentM, G2.DecisionM)
Out[4]: 
DecisionM      0   1
IntelligentM        
5              9  11
6             21  12
7             35  30
8             35  48
9             14  27
```

But `mosaic()` fails badly on this:

```
In [7]: mosaic(pandas.crosstab(G2.IntelligentM, G2.DecisionM),["IntelligentM","DecisionM"])
...
KeyError: "['IntelligentM' 'DecisionM'] not in index"
```

Can mosaic() gain a `freq=` param that shortcircuits `[_normalize_dataframe()](https://github.com/statsmodels/statsmodels/blob/b235b7bff9890f4b6280f3c0f4a57c605f3b81e0/statsmodels/graphics/mosaicplot.py#L322)` and something similar to handle crosstabs? Or maybe it would be better--if less elegant--to be split to mosaic_contigency(), mosaic_items(), mosaic_crosstab().
contingency mode would operate on dicts `{(var1, var2): count}`, item mode would operate on tables `(var1, var2, count)` and crosstab mode would operate on matrices `[var1, var2]: count`.

I'm on statsmodels 0.6.1 on Python3.

user: kousu
issue_2183: state: open
title: SUMMary/ENH  probability plots - design and open issues
body: We have an open PR #1433 to improve the gofplots Probability plots
I find it difficult to figure out what the various fit, standardization and axis label (which scaling) options are.

some notes on it:
- 1) Probplots are general/generic and should work for all distributions
- 2) As in GOF tests, the null hypothesis can be either fully specified or for the set of distribution, (e.g. is it N(0, 1) or is it normally distributed for any loc and scale, or in between is it normal with mean=0 and arbitrary scale)
- 3) The location/scale distributions with, for example, standardized data `(x - mean) / std` is only a special case, but the most common usage.
- 4) probability plots can be used as "estimator", e.g. normal case - which line fits the bulk of the observations in the qq space.

for display, one of the relevant choices are the labels/units of the axes
- 5) pp-plot: straightforward, always in [0,1]
- 6) qq-plot: either original units of raw data or standardized units for loc=0, scale=1 (which is not necessarily the same as standardizing by sample mean and sample variance.
  It looks like in 0.6 we have standardized axis lables, in PR #1433 we get original scale axis labels. I think the latter is more common and contains more information.
- 7) related adjustment of plots through plotting positions.

adding lines and supporting elements
- options for lines - either fully specified location, or estimated location of line. We don't necessarily want the line based on MLE parameters if the data is contaminated with outliers or for other reasons, i.e. we don't always want to assume that the Null hypothesis hold for the line.

**compared to R**
qqnorm plots standardized x-axis  (`Theoretical Quantiles'), qqplot uses raw scale for x-axis  (I imported`distr`and`distrMod`), 
both plot raw scale on y-axis (`Sample Quantiles')
qqplot seems to take only "frozen", completely specified distribution (as far as I have figured out, docstring is not very clear)

`distrMod` has pointwise and simultaneous confidence bands ("exact" and asymptotic) for qqplot, e.g.
`b = qqbounds(x3, Norm(15, sqrt(30)), alpha=0.95,n=30, withConf.pw = TRUE, withConf.sim  = TRUE, exact.sCI=FALSE ,exact.pCI= FALSE, nosym.pCI = FALSE)`
I assume it uses the assumption of no estimated parameters and fully a-priori specified distribution. Simultaneous confidence bands are based on ks_test with known parameters..

(to be continued)

**background**
#4 merge PR looks like the early qq-plot with discussion on adding line.
#412 merged PR with discussion about previous refactoring and enhancements by Paul Hobson

**current issues**
#1433  PR
#1299 initial bug issue

https://github.com/phobson/statsmodels/pull/1   other changes, with some discussion, obsolete closed
#1407 #1414 disabled unit tests
## Other Issues and Enhancements
- [ ] #1297 k-sample probability plot, in analogy to k-sample gof test
- [ ] #1307 chisquare probability plot for multivariate normal, 
    extension to general elliptical distribution would require the level set distributions instead of chisquare
- [ ] #1119 confidence intervals and other extensions, see "compared to R" above
- special cases, e.g. normal plot that can use distribution specific properties - no issue ?

user: josef-pkt
issue_2181: state: open
title: Creating binomial model from formula reverses prediction direction
body: If a GLM is fit using a formula, the direction of the prediction is reversed so that positive coefficients indicate a higher likelihood of a False or zero value in the dependent variable.  See discussion [here](https://groups.google.com/forum/?hl=en#!topic/pystatsmodels/Z6hHQJ9lHqc).

An example:

``` python
import numpy as np
import statsmodels as sm
import statsmodels.api

x = 3*np.random.normal(size=1000)
prob = 1 / (1 + np.exp(-x))                                                                                    
y = np.random.uniform(size=1000) < prob

model = sm.api.GLM(y, x, family=sm.api.families.Binomial()).fit()
print(model.summary())

model = sm.api.GLM.from_formula(b'y~x', data={'x': x, 'y': y}, family=sm.api.families.Binomial()).fit()
print(model.summary())
```

In the first model, the coefficient of x is positive, as it should be (higher x value makes a True y more likely).  In the second model, the coefficient of x is erroneously negative.

Also, although "positive coefficient means True/1 prediction" is probably what most people would expect, it should still be explicitly documented.

user: BrenBarn
issue_2179: state: open
title: GLM/LEF for non-linear mean function
body: We need to extend the LEF/GLM models to not be GLinearM.

I thought there are open issues, but I didn't see any.

design idea (some notes for it)

Instead of moving to general nonlinear mean function `E(y | x) = m(x, params)`, we can move to a combination link-mean function `E(y | x) = inverse_link(m(x, params))`.

One main point: reuse current link properties, and current `score_factor` to extend it by just replacing the inner part which is currently the linear prediction `linpred = exog dot params + offset`
The advantage is in applying chain rule for derivatives.
If link is identity, then this reduces to the econometrics general mean function.

user: josef-pkt
issue_2177: state: open
title: ENH: internal method `_get_exog_predict`
body: related to margins and predict

we have different ways of defining exogs to be used in Margins and in prediction. To separate/modularize code and make it reusable and consistent we should have a generic method for the models or function.

for prediction see PRs #2172 and #2151 
for Margins see #2154 
for contrasts (?) see #2167 

Note: this is outside of tsa, currently only single equation/exog, but need more flexible handling in subclasses

user: josef-pkt
issue_2175: state: open
title: REF: separate exposure from offset
body: Offset is a generic, fixed additive term that can be added to any model that has a linear core (exog \* params)
Exposure is currently we treated essentially the same way as offset after taking logs, and is only supported if the link function is `log`, i.e. exponential mean model.

I'm getting convinced over the last few times that I looked at it, that we should treat exposure as a generic multiplicative term for any link function, which only coincides with offset in the log-link case.

(Mainly) Kerby refactored offset and exposure for 0.6 to make it more efficient and fixed some bugs or inconsistencies.

We need to collect some usecases for multiplicative exposure. 
example, I guess: Poisson would have multiplicative exposure with any mean function not just with log-link

user: josef-pkt
issue_2167: state: open
title: LSMeans, Contrasts  in SAS, Stata
body: marginal means when we have categorical variables
evaluating margins at predefine population distributions or survey distributions
marginal means for balanced sample if actual estimation sample is unbalance

Similar to DiscreteMargins and wald_anova with emphasis on categorical explanatory variables

SAS has LSMEANS  (IIRC), in Stata large parts are in `contrats`, but also in margins and predict

I never worked with examples for this, but it looks pretty important based on SAS and Stata documentation.

Where do we get started ?

user: josef-pkt
issue_2164: state: open
title: Margins, handle offset and exposure
body: AFAICS, I lost my overview again

DiscreteMargins ignores offset and exposure, but maybe not everywhere. ?
#2158 Margins for GEE adds offset_exposure to some of the internal derivative calculations

we need to 
- check current code for consistency in treatment of offset and exposure, unit tests
- decide on what behavior we want and refactor, adding options for users

my guess:

We need to be able to account for `offset` because it will change where we evaluate, e.g. with fixing some params a part would just be moved from exog to offset but would disappear from margin calculations

In most cases users will be interested in exposure=1, to get a standardized marginal effect. However, if exposure reflects the "distribution of the exog population", then we would want to use exposure as a weighting factor for `"overall"`, average treatment/marginal effect.

user: josef-pkt
issue_2163: state: open
title: check for valid misspelled keywords in models
body: Do we have a mechanism to warn the user that an argument passed to a model was not used?

A misspelled (nonexistent) argument simply disappears into *kwargs.

It is not easy to do automatically because we use `**kwargs` for cases when different classes handle parts of the keywords, eg. optional or required arrays in sub classes and `missing` and similar data handling in base classes. Additionally, we have the generic `from_formula` which has a different signature and keywords than `__init__`.

I guess the easiest will be to attach `valid_kwds` to a class, so we can check generically. We already added `init_keys` to be able to recreate a model which can be reused.

related: some other functions/methods also don't have a check for misspelled keywords. e.g cov_kwds in fit, plotting options ?

user: josef-pkt
issue_2157: state: open
title: diagnostic for nonlinear: multicollinearity
body: http://stackoverflow.com/questions/27651702/checking-for-multicollinearity-in-python

I don't think we have any ready made standalone functions either for an exog directly, of for the results (score_obs ?) of an estimation result of discrete or GLM.

see also #1534 for some problems

**update**

We should add multi-collinearity and collinearity/singular measures to `get_diagnostic`.
Those can be directly based on X, score_obs, hessian and/or cov_params.
We can have multicollinearity or near-singularity problems in the estimation results in non-linear models even when hessian inversion does not fail (which raises a warning). 

However, adding them only to get_diagnostic will not make them available for results.summary(). 
Summary should add a extra_text warning for small eigenvalues and large condition number also for non-linear models.

user: josef-pkt
issue_2155: state: open
title: ENH: delta method
body: add a standalone, reusable function with options

multivariate version is used in discrete_margins, we had brief discussion, my version is in miscmodels

one issues is how to handle different multi-dimensional use cases

Margins needs vectorized for `at='all'`, in the basic usecase we would need only `se`, i.e. diagonal.
I think there should be a computational short cut to calculate only the diagonal of vectorized version.

We still need the entire covariance matrix for hypothesis test that involve more than a single effect.

I guess it's:
`se = (jac.dot( cov_params) * jac).sum(-1)` 
where `jac` is `(nobs, k_params, k_params)` or  `(nobs, k_effects, k_params)`

user: josef-pkt
issue_2154: state: open
title: SUMM/Review  Margins
body: (DiscreteMargins is one of the main parts of traditional models regression/discrete/genmod/robust that I didn't go through completely in details.)

I was trying to figure out how some basic things are calculated in Discretemargins, but it's rather confusing. It needs an outline of the main elements. My plan was to see how it can be extended and how much overlap there is with extended prediction.

Note: one main difference of margins to predict is in the dimension. A full problem is has 2d effects/prediction `(nobs, k_params/k_effects)` which would imply a 3d summary table. Margins with `at='all'` would be this case, but is not fully implemented. 
To get to 2d tables: Margins reduced over nobs, prediction looks only at univariate effect/predicted value.
### for future
- can we extend `method`? Stata also handles `predict` inside Margins
  different `dy`, `ey`, `dx`, `ex` are just rescalings, predict requires different effect function 
- where do we need to add other mean functions? extension to GLM with `link` function, in discrete models link or mean function is hard coded
- what's the relationship to average treatment effect? Wooldridge,  Stata has separate manual for treatment effect that I barely looked at.
- code sharing with other parts (predict_at ?), general pattern, refactoring (DiscreteMargins is it's own results class and uses `_reset` cache).
- robustness (to misspecification): is using sandwich robust cov_params enough?
- extension to `terms`, i.e. multi-column exog
- Stata has hypothesis tests that compares margins, I don't know much about this yet.
### Outline
- (1) data/exog handling
- (2) calculating effects and distribution (covariance) of effects, defining effects and derivatives for delta method
- (3) results and user access to them

to (2)
- `_effects_at` calculates average effect for `overall`, why does it select first row for `at` other than `overall` and `all`? Isn't it an error if it has more than one row?

to be continued, ...
### Issues

(needs more cross-links to other issues)
#930 Margins for GLM,  GEE has margins but maybe untested
#745 Margins for multi-column exog
#2150 related SUMMary issue for prediction

---

issues while reading code
- unit test coverage, AFAICS only Logit is tested, I'm using Poisson as an example to play with
- open TODOs in comments
- `transform` in atexog as in `predict`? in `_get_margeff_exog` `if atexog is not None`
- const_idx is now available model.data
- from the results I don't see how to get the information about which columns were treated as dummy (I haven't tried count exog).
- for cross-checking or reuse it would be useful to have access to the `exog` that were used in calculating the results
- naming: in `margeff_cov_params`: `derivative` argument is misleading since it uses the effect function and not the derivative of the effects function, but it takes jacobian if array.
- There is currently no analytical derivative of effects function possible/implemented
- DRYing `model._derivative_exog` is model specific, but all have the same boiler plate of `ey` `ex` transformation/scaling
### reusable code
- exog at or `atexog` handling: `_get_margeff_exog` should be reusable by a `predict_at` (Kerby plot function), we can also add transform/patsy handling from `predict` (which I will already need in different places for predict) to margins `atexog` handling.
- detection of dummy and count exog: eventually we need a pattern/convention for this, nonparametric product kernel an MICE use also the exog type
- delta method doesn't work for 'all', needs vectorized version. see #2155

user: josef-pkt
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_2153: state: open
title: design: pattern for summary of other results
body: related open issues for summary: https://github.com/statsmodels/statsmodels/issues?q=is%3Aopen+is%3Aissue+label%3Acomp-io+summary

example t_test, currently I'm working on prediction results and parameter transformations (eventually nonlinear_test)

layout: header-text + table(s) + footer-text
similar to summary of Results classes
- t_test reuses params_table but doesn't have header and footer
- also Results.summary doesn't add footer to repr_html #1448 
- we need some explanatory text in footer, e.g. confidence interval method for nonlinear transformation
- for t_test we should also have other versions of params_table (p-value correction instead of conf_int), part of #742

related: rename `summary_frame` -> `frame` ?
in cases where we want to use `frame` to pandas wrap results prefixing with `summary_` sounds superfluous and verbose

user: josef-pkt
issue_2152: state: open
title: Issues with UnivariateKDE.evaluate
body: I'm trying to understand what's causing some surprising behavior with the `KDEUnivariate` class.

``` python
import numpy as np
from statsmodels.nonparametric.kde import KDEUnivariate
x = np.random.randn(20)
kde = KDEUnivariate(x)
```

Specifically, the `kde.evaluate` method seems unable to reproduce the `kde.density` property:

``` python
kde.fit()
np.testing.assert_array_almost_equal(kde.density, kde.evaluate(kde.support))
```

```
AssertionError: 
Arrays are not almost equal to 6 decimals

(mismatch 99.21875%)
 x: array([ 0.001018,  0.000991,  0.000969,  0.000953,  0.000942,  0.000937,
        0.000937,  0.000943,  0.000953,  0.000969,  0.00099 ,  0.001015,
        0.001046,  0.001082,  0.001123,  0.00117 ,  0.001221,  0.001278,...
 y: array([  3.280454e-04,   3.543112e-04,   3.824256e-04,   4.124967e-04,
         4.446365e-04,   4.789620e-04,   5.155945e-04,   5.546600e-04,
         5.962887e-04,   6.406158e-04,   6.877806e-04,   7.379272e-04,...
```

This is a subtle difference:

``` python
import matplotlib.pyplot as plt
import seaborn
plt.plot(kde.support, kde.density, lw=5)
plt.plot(kde.support, kde.evaluate(kde.support), "--", lw=5)
```

![kde](https://cloud.githubusercontent.com/assets/315810/5519987/c83a0594-893a-11e4-9391-2362eaf359f1.png)

It seems to be because `evaluate()` is not picking up the `fft` option:

``` python
kde.fit(fft=False)
np.testing.assert_array_almost_equal(kde.density, kde.evaluate(kde.support))
```

Although, this fails a stricter test:

``` python
np.testing.assert_array_equal(kde.density, kde.evaluate(kde.support))
```

```
AssertionError: 
Arrays are not equal

(mismatch 68.0%)
 x: array([  3.280454e-04,   7.088984e-04,   1.425063e-03,   2.664943e-03,
         4.636106e-03,   7.503213e-03,   1.129811e-02,   1.583096e-02,
         2.065008e-02,   2.509707e-02,   2.847294e-02,   3.028022e-02,...
 y: array([  3.280454e-04,   7.088984e-04,   1.425063e-03,   2.664943e-03,
         4.636106e-03,   7.503213e-03,   1.129811e-02,   1.583096e-02,
         2.065008e-02,   2.509707e-02,   2.847294e-02,   3.028022e-02,...
```

More troubling is that trying to use `evaluate` with a non-gaussian kernel raises an exception:

``` python
kde.fit(kernel="tri", fft=False)
kde.evaluate(kde.support)
```

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-16-bb57467307a4> in <module>()
      1 kde.fit(kernel="tri", fft=False)
----> 2 kde.evaluate(kde.support)

/Users/mwaskom/anaconda/lib/python2.7/site-packages/statsmodels/nonparametric/kde.pyc in evaluate(self, point)
    256         """
    257         _checkisfit(self)
--> 258         return self.kernel.density(self.endog, point)
    259 
    260 

/Users/mwaskom/anaconda/lib/python2.7/site-packages/statsmodels/sandbox/nonparametric/kernels.pyc in density(self, xs, x)
    181         xs = np.asarray(xs)
    182         n = len(xs) # before inDomain?
--> 183         xs = self.inDomain( xs, xs, x )[0]
    184         if xs.ndim == 1:
    185             xs = xs[:,None]

/Users/mwaskom/anaconda/lib/python2.7/site-packages/statsmodels/sandbox/nonparametric/kernels.pyc in inDomain(self, xs, ys, x)
    168             return (xs, ys)
    169         else:
--> 170             filtered = filter(isInDomain, zip(xs, ys))
    171             if len(filtered) > 0:
    172                 xs, ys = zip(*filtered)

/Users/mwaskom/anaconda/lib/python2.7/site-packages/statsmodels/sandbox/nonparametric/kernels.pyc in isInDomain(xy)
    163             """Used for filter to check if point is in the domain"""
    164             u = (xy[0]-x)/self.h
--> 165             return u >= self.domain[0] and u <= self.domain[1]
    166 
    167         if self.domain is None:

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
```

user: mwaskom
issue_2150: state: open
title: SUMM: prediction results (outside tsa)
body: Bringing together notes for different cases of prediction results. This does not include conditional forecasting as in TSA, but includes non-linear and non-normal models.

example, simplest case OLS: predict mean, confidence interval for predicted mean, confidence interval for predicting a new observation, see #719 
extends to WLS with weights or estimated variance function HetGLS or (non-existing) HetMLE.

issue for GLM #938
### models, cases
- linear prediction  x \* params and residual is normal (or t distributed)
- non-linear: single index/link function with linear part: 
  we can separate link transformation from properties of linear prediction
  - models as in GLM, discrete and if 
  - the endog has been transformed by user, e.g. `'log(y) = x * beta'` with OLS for log-normal model
- non-linear: general non-linear mean function
  - non-linear least squares, ... (not covered yet out of the box)
- multi-parameter models: 
  - simplest case estimated variance function, HetGLS, for prediction this might be similar to implied variance functions as in LEF/GLM family
  - BetaRegression
- panel/mixed : not clear, BLUE or similar, with or without random effect estimate
- multi-equation models: not clear, follow Stata just predict mean and confidence interval for mean
  - different equations: like simultaneous equations, heckman
- multivariate endog: related to multi-equation models but equations have the same structure
  - Multinomial Logit and similar
  - multivariate endog as in balanced panel, SUR,  (not yet available) similar to tsa.VAR without time series properties
### Which Results
- predicted mean, linear or non-linear
- confidence interval
- distribution, used in MICE for random sampling from predictive distribution. 
  - In discrete case we can also predict 
- higher moments, skew, kurtosis, expectation for limited range, tail probabilities (extreme value and risk analysis) 
- `predict_at` and average treatment effect, ATE, and similar
- Margins: derivatives/changes with respect to explanatory variables in case of non-linear models, already implemented for discrete (and GLM ?)

problem: **convolution**

We have two sources of uncertainty, one coming from parameter uncertainty, one coming from the assumed distribution of endog or residual.
It will be relatively straightforward to treat them separately, but the convolution is difficult to compute except in the simple normal model with additive error.

This will cause a difference in the implementation/API pattern between linear model and all other models, RegressionResults prediction interval for new observations is calculated as convolution, but we won't have it in other models. We can get it by MonteCarlo or Bootstrap for other models but that should be separate methods since they are not cheap. (I ran some experimental functions for this for the Poisson case.)
### Implementation

for some interface comment see #719 

In terms of internal implementation, confidence intervals for new observations are different, inference and confidence interval on mean prediction is the same as testing, e.g. t_test can calculate everything for predict_mean.

Stata has model specific predict and lincom, nlcom and test, testnl.

In terms of algorithm we have three cases for inference on function of parameters
- (1) linear in parameters: implemented in t_test (lincom/test in Stata)  linear transformation of normal or t-distribution
- (2) univariate non-linearity: univariate transformation of a univariate normal distributed result, 
  example: inverse link transformation of linear prediction, non-linear transformation of single parameter, 
  special feature: confidence interval with endpoint transformation
- (3) non-linear function of multivariate parameters: here we cannot use 1-D endpoint transformation, the only available method is the delta method, (besides bootstrap or similar)

(2) is the main tool for prediction in GLM and discrete, one-parameter LEF (if variance is estimated then it is still asymptotically uncorrelated with mean, block diagonal information matrix. I guess NB1 and BetaRegression won't fit into this.)

other 

in-sample versus out-of-sample: the main difference here is that we have a corrected residual estimate, in-sample is currently mostly covered by influence_outliers

user: josef-pkt
issue_2147: state: open
title: Use better equation names in MNLogit results
body: Now that we're treating factor Y inputs, we need to use the equation names for the columns of the results parameters.

(**Edit**: I think the above is misleading. AFAICT, we don't currently, and maybe never did, allow factors for Y. This is related to #2049 and #1210. I abandoned the PR #2197 because it introduced too much complexity. Could be worth revisiting at some point.)

user: jseabold
issue_2146: state: open
title: REF/ENH: outliers_influence summary_frame options
body: I think we should make some things optional

`dfbetas` could be optional, and we could add, as an option, the `predict_mean` .... ci from the `summary_table` **function** to the DataFrame 

IIRC (check): the residual variance estimate differs, has a correction term, for the exog that where used in estimation compared to general exog in predict. So this would be useful independent of enhancements to predict in #719

user: josef-pkt
issue_2145: state: open
title: helper function pattern extra methods that may return pandas with index
body: looking at possible solution for prediction standard errors #719

What data/interface handling do we need for an extended predict function? 
(besides the actual calculation code)
- converting exog is in base.model.Results.predict going through patsy if necessary
- row index if exog is DataFrame/Series - this seems to be missing in current `predict`

to the latter:

OLSResults.outlier_test returns DataFrame with row_labels if available
We might have it also in other methods or auxiliary functions and classes but I don't have an overview.

We may need helper functions or private methods that are able to wrap arrays similar to wrappers but outside of predefined model Results.

---

for predict we have two cases, exog used in estimation and exog given as argument.
in outliers_influence we only calculate at the exog used in estimation.

user: josef-pkt
issue_2144: state: open
title: REF: separate definition of mse_resid and scale
body: largely a followup to #2137

we use `mse_resid` as the estimate of the residual scale in several places, e.g. sandbox `wls_prediction_std` and in outliers_influence for prediction standard errors.

`scale` is the same number but not used much, I guess.

todo: review, streamline, refactor

scale is more generally used, it is used in most other models. `mse_resid` is linear regression specific.

Note: GLM/LEF, Poisson also have a distinction between scale as it is assumed by the model, and actual scale defined by, for example, the pearson residual.

scale handling in GLM for QMLE with dispersed variance is also still not very clear or clean.

user: josef-pkt
issue_2141: state: open
title: expanding_brentq docs?
body: It looks like there's a conflict between the docs for the signs of `start_low` and `start_upp` and what the code actually checks. I assume the code is right.

user: jseabold
issue_2139: state: open
title: Tobit skipper rebase temp
body: Skippers branch, formerly PR #447

interactive rebase that removed two commits with the test results files.
From what I can see is that the large files are not included in any commit here

user: josef-pkt
issue_2136: state: open
title: cluster robust cov_params in long panel - singular cov_params
body: case of long panel, T > N

I was trying to use the 'within' example from the repeated measures anova PR
https://github.com/statsmodels/statsmodels/pull/786/files#diff-10

With cluster robust standard errors the bse has nans and cov_params has reduced rank.

In the literature there are comments about rank reduced covariances if there are more observations per cross-sectional unit than cross-sectional units, T > N.
but I never looked at an example before.

What should we do in this case?
ignore for now (user needs to use it correctly)
warn
???

user: josef-pkt
issue_2133: state: open
title: Handle category dtype from pandas
body: Pandas now handles categorical and ordinal variables within the blocks of a DataFrame. Will need to write some code to handle these appropriately. Right now we do get a fairly informative error message though. As an example (Python 3, pandas 0.15.2)

```
[~/]
[55]: from io import BytesIO

[~/]
[56]: from urllib.request import urlopen

[~/]
[57]: dta = pd.read_stata(BytesIO(urlopen("http://www.stata-press.com/data/r13/fullauto.dta").read()))
```

rep77 is ordinal and foreign is categorical

user: jseabold
issue_2130: state: open
title: WIP  REF: anova_lm
body: see #2123 and #1773 

This is not yet intended as "serious" PR

I had made some changes to anova_lm to extend anova_lm to GLM, but then decided to take a different route. There will be separate PR with `wald_test_terms` or `wald_anova` as generic method for ANOVA type 3 kind of analysis.

user: josef-pkt
issue_2126: state: open
title: Updating QR
body: Scipy PR https://github.com/scipy/scipy/pull/4249

user: jseabold
issue_2123: state: open
title: BUG/DOC: anova_lm robust not supported everywhere
body: AFAICS, the `robust` option in keyword is only supported for `anova2_lm_single` and `anova3_lm_single`   but the doc string doesn't say so.

F-tests based on ssr are not robust to heteroscedasticity.

related enhancement issue #1773

user: josef-pkt
issue_2122: state: open
title: GenericLikelihoodModel: add option for which results class
body: The superclass fit methods return a hard coded results class. This means we have to copy over the results to a new instance if we want to return a model specific results class.

I'm running into this right now in BetaRegression PR #2030, and I have somewhere (didn't find it) an issue for improvements in discrete.

GenericLikelihoodModel would be a good case where we can try out adding an option for which Results class is created.

user: josef-pkt
issue_2113: state: open
title: smaller memory footprint of models
body: We currently carry around some data that might not be strictly necessary. E.g., `orig_endog`, `orig_exog`, etc. We probably shouldn't do this by default. We should also make the attached `data` instance the first-class citizen and have `endog`, `exog`, etc. only has properties of the models that point to the `data` attributes. 

user: jseabold
issue_2108: state: open
title: iterated lowess is wrong
body: Consider

```
In [1]: sm.nonparametric.lowess([0] * 10 + [1] * 10, np.arange(20), it=2)[:, 1].T
Out[1]: 
array([ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
        0.        ,  0.        ,  0.        ,  0.26162324,  0.45476355,
        0.56315913,  0.70972324,  1.        ,  1.        ,  1.        ,
        1.        ,  1.        ,  1.        ,  1.        ,  1.        ])
```

(either with the old, 0.6.0 version, or the latest "nan-fixed" version)
This is likely incorrect as the output array isn't even symmetric around 0.5.

Compare with R's result:

```
> lowess(c(rep(0, 10), rep(1, 10)), iter=2)
$x
 [1]  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20

$y
 [1] 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000
 [7] 0.00000000 0.03796574 0.29511209 0.44982749 0.55017251 0.70488791
[13] 0.96203426 1.00000000 1.00000000 1.00000000 1.00000000 1.00000000
[19] 1.00000000 1.00000000
```

which is symmetric.

Note that the issue doesn't appear if `iter=1` (statsmodels gives the same result as R).

user: anntzer
issue_2107: state: open
title: SUMM  REF make discrete more like GLM, added properties
body: GLM has now a lot of nice generic features that makes it easy to write new semi generic functions on top of it.

discrete is still lacking support for several extensions, there are several issues open to refactor and enhance the discrete models, or MLE models in general (incoming BetaRegression will follow or already follows a similar pattern)

Latest example #2018 to support generic residual plots and added variable plots.

refactorings (still need to collect numbers for specific issues)
- resid and fitted_values
- attach `link` used in mean function (single index model for mean E(y|x)
  BetaRegression has two link functions
- missing offset
- exposure is stored in `log` form
- score_factor, hessian_factor or similar to allow for score_tests
- ...

more general:
- better support for recreating models and fit, using the same options but different endog or exog
  this was mostly added for `fit_constrained`

user: josef-pkt
issue_2092: state: open
title: fit_constrained: inefficiency of QMLE
body: extremum estimator
`max_b S(b) subject to a(b) = 0`

MLE is efficient, but QMLE (general M-estimator) is not.

```
It is well-known that in many contexts (e.g., see Ferguson [1958] for the
maximum likelihood case) minimum chi-square estimation is an asymptotically
optimal method of imposing constraints on a vector of estimated parameters.
It turns out that the restricted efficient GMM estimator ST is asymptotically
equivalent to the minimum chi-square estimator bT, which was defined in equation
(2.8), while the restricted quasi-maximum likelihood estimator is inefficient
relative to the other two estimators.
```

Newey West 1987: Hypothesis Testing with Efficient Method of Moments Estimation

see reference to Chamberlain for minimum chisquare to impose constraint.
We/I have already a similar function for the linear case, or linearized case, I guess that's the same.

(I doubt it's important but we might get some results as a "side effect", i.e. while targeting other functionality.)

user: josef-pkt
issue_2091: state: open
title: LM / score tests:  `omitted` test missing in diagnostic
body: I don't see a simple lm or score test for omitted variables in stats.diagnostics.

There is compare_lm_test, however it requires two results instances instead of a results instance and extra variables for a variable addition score/LM test.

most of the current diagnostic tests are implemented as simple variable addition tests 
with `lm_statistic =  nobs * rsquared`

see other issues for generic version for variable addition/omitted test
#2041 for SUMMary

(aside: I needed it as a test case for a generic GMM based conditional moment test.)

user: josef-pkt
issue_2085: state: open
title: modify the appearance of the ACF PLOT function when using subplots
body: Hi!
I'm using statsmodels (0.6.0) on Python 2.7, and i'd like to plot several autocorrelation functions  as subplots. Each subplot represents the autocorrelation for different data on each month so i'd like to compare them visually. 

Now, the problem is that unlike other functions (which I easily subplot using Matplolib), here I am not able to add a lagend, add xlabel or use other common features in plotting. It seems that `ACF` plot in statsmodels has fixed and "anchored" properties, like the title "Autocorrelation" cannot be removed (or at least I couldn't find a way). The reason i'd like to remove it is that it appears on the head of each subplot and covers the axis of the plots above it.

Here is the subplots figure that I got:
![figure_1](https://cloud.githubusercontent.com/assets/8626306/4964906/1842c884-6774-11e4-84b8-a9367b8c0fd4.png)

Moreover, It would be helpful to add legends to mark each month and add "Lag" on the bottom of the figure. 
the following functions didn't work:

```
plt.legend(['January'] ,loc='upper right', fontsize = 'x-small')
plt.xticks(visible=False)
plt.xlabel('Lag')
```

Does anybody know if such changes are feasible and if yes how to do it?  

P.S
for the figure above I used the following:

```
fig, axes = plt.subplots(5,1)
fig = sm.graphics.tsa.plot_acf(ts1['2013/01/01 00:00:00':'2013/01/02 23:59:59'], lags= 86400, ax=axes[0])
fig = sm.graphics.tsa.plot_acf(ts2['2013/02/01 00:00:00':'2013/02/02 23:59:59'], lags= 86400, ax=axes[1])
fig = sm.graphics.tsa.plot_acf(ts3['2013/03/01 00:00:00':'2013/03/02 23:59:59'], lags= 86400, ax=axes[2])
fig = sm.graphics.tsa.plot_acf(ts4['2013/04/01 00:00:00':'2013/04/02 23:59:59'], lags= 86400, ax=axes[3])
fig = sm.graphics.tsa.plot_acf(ts5['2013/05/01 00:00:00':'2013/05/02 23:59:59'], lags= 86400, ax=axes[4])
```

user: aviPython
issue_2079: state: open
title: heckman covergence problems with MLE
body: just saw a reference in Stata mailing list by chance

http://www.stata.com/statalist/archive/2005-03/msg00607.html

see PR

user: josef-pkt
issue_2077: state: open
title: FAQ/ENH/DOC: regression, model equivalent to statistical tests
body: Many of the one, two or k-sample tests can be calculated also through regression models with appropriate choice of the design matrix.

It would be good to have a list compiling some examples.
-> TODO

As possible enhancement this might provide additional options for the hypothesis test.

example 
t-test with unequal variance assumption would be GLS or heteroscedasticity robust OLS. The actual test might have different small sample adjustments, like the degrees of freedom for t-test with unequal variance and unequal sample size.

Another example: Cochran-Armitage test is a score test in a binomial model, with application for example in dose-response models.
Using the standard models we could reuse cluster robust standard errors to get a cluster correlation robust version of Cochran-Armitage. (The change to the test statistic is actually just a change in the variance estimate.) 
Dennis Boos On Generalized Score Tests, The American Statistician Vol. 46 (4) 1992 Example 4 on p. 331

another example: Mann–Whitney–Wilcoxon rank sum and rank regression
R. Chen, T. Chen, N. Lu, H. Zhang, P. Wu, C. Feng & X.M. Tu (2014) Extending
the Mann–Whitney–Wilcoxon rank sum test to longitudinal regression analysis, Journal of Applied
Statistics, 41:12, 2658-2675, DOI: 10.1080/02664763.2014.925101
http://dx.doi.org/10.1080/02664763.2014.925101
uses functional response GEE (whatever that is)

another example: McNemar test and extension with generalized score test in GEE
Klingenberg, B. and Agresti, A. (2006), Multivariate Extensions of McNemar's Test. Biometrics, 62: 921–928. doi: 10.1111/j.1541-0420.2006.00525.x
http://onlinelibrary.wiley.com/doi/10.1111/j.1541-0420.2006.00525.x/abstract
based on abstract, I didn't browse the article

user: josef-pkt
issue_2074: state: open
title: FAQ: Which models provide robust mean function estimation?
body: I would like to estimate the conditional expectation E(y | x) that is robust to other misspecifications. Which models in statsmodels can I use? What are the properties and advantages of each?

Suppose we have a mean function `E(y | x) = m(x, b)` where m is an known function and we want to estimate the parameters `b`. `m` is the inverse link in GLM or GEE terminology.

All distributions in the linear exponential family can estimate the parameters `b` consistently without requiring any additional assumptions. However, efficiency of the estimate depends on the specific distribution or more precisely it's implied weighting function.
Also, if we don't want to make variance or distributional assumptions, then we need to use robust covariance matrices for inference.

Caveat: Since we only require that we estimate the mean function, we cannot use other properties of the distribution for prediction. For example the implied distribution of future observations might be incorrect, even though the expectation is consistently estimated.

This robustness applies to all currently implemented families and links in `genmod`, GLM and GEE, and to the corresponding distributions in `discrete` and `regression`.
- normal/OLS: for data on the real line R, implied variance function is constant, independent of mean
  (GLM family Gaussian, OLS)
- Bernoulli : for data bound in a compact interval, specifically set {0,1} or real interval [0,1]. Example    for first is binary response and for second it is proportions. variance function p \* (1 - p)  where p = m = E(y | x) ? check
- Binomial : similar to Bernoulli for integer or real data between zero and known upper bounds. check
  Note: robust cov most likely does not work
- Poisson : for nonnegative data, nonnegative integers or positve real line R_+ (including zero)
  implied variance function, equal or proportional to m
- NegativeBinomial, Negbin 1: nonnegative data as in Poisson, variance function m + c m**2 ???
- geometric : special case of NegativeBinomial
- ...
- gamma
- inverse...

user: josef-pkt
issue_2073: state: open
title: two step estimation, robustness, and nuisance parameters like scale
body: follow-up to #1776 especially part 2, estimating equations independent of scale model
and score tests robust to unspecified covariance see SUMM #2041 
also #1788 to full MLE including variance estimation

in GEE the contrast is between GEE1 and GEE2
similar would apply to other models

Linear exponential family LEF: 
- We estimate the mean model consistently independent of the specification of the variance, correlation assumptions
- If we model the variance and estimate it simultaneously by MLE (or GEE2), then the mean model depends on the correct specification of the variance, correlation.
- not clear to me: weighted least squares is robust and we can use sandwiches even after using a variance estimate from a first stage. This is also what GEE (GEE1) is doing.
  Iterated GLS is in some cases equivalent to MLE. How does this match up with different robustness results.

consequences:

df_model include variance in count or not, see #1802 and other issues ?

keep distinction in LEF family for MLE versus QMLE,   OLS versus Normal linear model 

2 stage estimators don't have always the same properties as one stage estimators.

This might not be relevant for pure scale estimation. 
A stronger example is: NegativeBinomial (NegBin 2) with fixed or pre- (initial stage) estimated shape parameter still estimates the mean parameters (beta) consistently even if variance assumption is misspecified. (Wooldridge 4.2)
The Negbin 2 MLE (joint estimation of location and shape/dispersion parameter by MLE) "is generally inconsistent if the distribution is at all misspecified, even if E(y|x) is correctly specified" (quote p.378). Negbin 1 is not in LEF even with fixed shape parameter. NegBin 2  
Wooldridge end of section 4.1)

follow-up:
If one estimator is robust and the other one is efficient, then we could use Hausman test. Eg. test MLE of Negbin against Poisson parameter estimates with Negbin as Null.
Does this work? I haven't seen any references.
Actually, I saw it a year ago:
Cameron Trivedi 1998 p. 181 mention Hausman test of NB2 against Poisson, but it has no power because the Hessian has a zero off-diagonal block (referencing Holly 1982 and 1987 )

user: josef-pkt
issue_2072: state: open
title: REF: method to get (non-default) robust cov from results.
body: instead of just `get_robustcov_results` we should have (also) `get_robustcov` that returns the covariance matrix and not a results instance.

Usage: To write robust score tests we don't need the parameter estimates for the full model but we need the score and hessian and either score_obs or robustcov for the full model.
related: #2054

roughly
- fit_restricted estimates the restricted parameters but has the full model
- if we have robust cov_params of the unrestricted model evaluated at the restricted parameters, then we can plug it into the score/LM_test formula for M-estimators. (new method to come)
  We get HC, HAC or cluster robust score/LM tests with exactly the same definition as in cov_type.

**possible problem**, looking at `base.covtype.get_robustcov_results`:
We are also adjusting other things in the results instance for small  sample correction like degrees of freedom.
Small sample correction might not apply correctly if we have the full model, but estimation is based on a smaller model (n_params estimated is not the same as n_params in full model)

This won't be relevant asymptotically or with chisquare based score tests, df_resid is used in scaling factor in all models (in contrast to Stata which uses it only for the linear regression).
It might be possible to use use_t=True for small sample correction, however in several models or cases Monte Carlo articles showed that score test is often conservative and using t or F would  make it even more conservative. Wald test are often liberal and overreject, where t or F goes in the right direction.

We might need different small sample corrections for score/LM tests than for Wald tests.

possible solution for implementation problem:
`base.covtype.get_robustcov_results` could return a dict with keys of results instance that should be adjusted or added, instead of adjusting or adding it directly. Needs one extra step to fix the results instance but would make the function work without requiring a results instance.

user: josef-pkt
issue_2071: state: open
title: Time varying covariates support for cox model
body: It would be great to add this feature. How complicated is this to implement? I've had difficulty finding a paper with implementation details. Tried looking at the R `survival` package but they seem to do it in C.

Could try solving the cox as a Poisson as mentioned in this [thread link](https://github.com/statsmodels/statsmodels/issues/1764), but I imagine you'd want to create a separate survival model for that.

user: lminer
issue_2070: state: open
title: cov_type non positive definite e.g. cluster robust, flat-top kernel
body: http://www.statalist.org/forums/forum/general-stata-discussion/general/240792-missing-standard-errors-when-running-2d-cluster-with-fixed-effects

A. Colin Cameron, Jonah B. Gelbach, and Douglas L. Miller: Robust Inference with Multi-way Clustering
mention that the covariance could have negative values on diagonal with fixed effects.

Same can happen with HAC kernels that don't smooth enough as in Newey-West, e.g. flat or flat-top kernels. 

recommendation: force, convert to positive definite (we have several functions), or only use relevant submatrix.

user: josef-pkt
issue_2068: state: open
title: ENH: add generalized IRF and FEVD for VAR
body: forecast error variance decomposition using the Generalized Impulse Response Function of Pesaran and Shin (1998)  http://www.sciencedirect.com/science/article/pii/S0165176597002140

see #2067 and referenced mailing list discussion for details

application in spillover analysis 
The example works according to the WINRATS reference.

It looks like we need an option to choose and calculate internally the full scaled sigma_u (covariance of innovations)

Needs to be checked for auxiliary changes in other parts than just `orth_irfs`, e.g. confidence intervals.
Even after #2067 choosing the P matrix for the (non)orthogonalization won't be fully supported, I guess.

user: josef-pkt
issue_2063: state: open
title: FAQ: Which sandwich, robust covariance should I use?
body: ### possible heteroscedasticity

for the choice in case of possible heteroscedasticity in the linear model Long and Ervin recommend HC3
Long, J. Scott, and Laurie H. Ervin. 2000. “Using Heteroscedasticity Consistent Standard Errors in the Linear Regression Model.” The American Statistician 54 (3): 217–24. doi:10.2307/2685594.
rule: don't pretest, just use it if you are not sure that the variance is constant

more recent: Davidson MacKinnon recommend wild bootstrap
### autocorrelation robust

???
### cluster, panel data

see large T versus large N asymptotics
use small sample correction depending on the number of clusters, possibly use t/F instead of normal/chisquare
references ???
### related

FAQ #1923 comparison to Stata

user: josef-pkt
issue_2062: state: open
title: ENH: Reduced exog column space, svd, qr
body: tools, infrastructure and pattern for supporting perfectly collinear `exog`

we need a helper function to get the reduced full rank exog either using svd or pivoting QR (now available in minimum scipy IIRC)
pivoting QR might will not drop sequentially (reading the docstring), we might need explicit check instead for colinear columns instead.
#1061  colinear exog in auxiliary regression: we only need the rank but using pinv on full, possibly singular exog might be wasteful

drop collinear columns in regression, and nan results
we have infrastructure for nandot and nans in the t_test, but it might not be generically accessible (only for fit_regularized in discrete?)
this is user facing.

use reduced exog for non-linear MLE optimization, internal reparameterization.
can be user facing with nans and reduced rank QR, or purely internal with SVD.

user: josef-pkt
issue_2056: state: open
title: regularization, coordinate descend and instrumental variables
body: I've seen now several articles that recommend coordinate descend as the best algorithm for L1 (or jointly L1-L2) regularized linear or generalized linear models. 

Here is an application to 2SLS for IV estimation

Wei Lin, Rui Feng & Hongzhe Li
Regularization Methods for High-Dimensional Instrumental Variables Regression With an Application to Genetical Genomics
DOI:    10.1080/01621459.2014.908125
http://www.tandfonline.com/doi/abs/10.1080/01621459.2014.908125
currently accepted manuscript

Might be interesting for the combination of traditional econometrics and large regularized data problems.

user: josef-pkt
issue_2055: state: open
title: score test, wald test with null on the boundary
body: boundary of parameter space are not so scary after all

application: mixed effects with zero variance, zero-inflated and overdispersed count models, mixture probability

Likelihood Ratio, Score, and Wald Tests in a Constrained Parameter Space
Geert Molenberghs and Geert Verbeke
The American Statistician, Vol. 61, No. 1 (Feb., 2007), pp. 22-27
Published by: Taylor & Francis, Ltd. on behalf of the American Statistical Association
Article Stable URL: http://www.jstor.org/stable/27643833

(I don't think this applies to the 'not identified under the alternative' cases.)

score_test for a single parameter is easy, just a 0.5. 0.5 mixture of chisquare or a one-sided test with normal distribution.
multivariate test for theta > 0 also looks "digestable" 

special cases to figure out the pattern are the count models where there is a large number of articles.

user: josef-pkt
issue_2054: state: open
title: score_tests cluster robust, sandwiches
body: see summary issue #2041 and partially related #2050 

score test with cluster robust standard errors. And by analogy other sandwiches will/should follow the same pattern

Guo, X., Pan, W., Connett, J. E., Hannan, P. J. and French, S. A. (2005), Small-sample performance of the robust score test and its modifications in generalized estimating equations. Statist. Med., 24: 3479–3495. doi: 10.1002/sim.2161
http://onlinelibrary.wiley.com/doi/10.1002/sim.2161/abstract

article is about robust Wald and score test in GEE, both are implemented
I never looked at the details of GEE score test.
From the article it looks like we have the same or similar small sample correction in the score test as we have for cluster robust cov_params.  (small = small number of clusters)

(aside: bias corrected cluster robust cov_params are similar/analogous to HC1 or HC2. We should try to get CR1, CR2, ...
IIRC, bias_corrected in GEE by Mancl and deRouen is CR2 ?
)

related: bootstrap to improve small sample performance.
There is a recent literature on cluster robust testing with unbalanced panels.
e.g.
MacKinnon and Webb "Wild Bootstrap Inference for Wildly Different Cluster Sizes"
https://ideas.repec.org/p/qed/wpaper/1314.html
MacKinnon has the related paper on confidence intervals.
AFAICS, they don't mention much of the statistics literature

They call CR1 and a version of CR2  CV1 and CV2.

computational problem for CV2 (and GEE exchangeable): The survey data with clustering by State has 40,000 observations in one cluster. This would require arrays with shape (40,000, 40,000).

user: josef-pkt
issue_2052: state: open
title: Neyman C(alpha) test or conditional score tests
body: another generic testing approach, related to #2050

When we have a composite alternative, e.g. heteroscedasticity and intertemporal dependence, then we might want to estimate them separately.
Might/should apply also with poisson versus zeroinflation and overdispersion (negative binomial)
Beta regression has also mean and variance that are asymptotically not independent (hessian is not block diagonal). Testing the mean specification depends on the variance assumption

references
score test under misspecified alternative
Jaggia Trivedi 1994
Davidson MacKinnon 1991
Bera Yoon 1993
Godfrey Orme 2001

related to #2050 
Robust tests in LEF can in many/all cases treat mean and variance specification separately, robustness only affects the covariance matrix (sandwiches).
Here we cannot separate different deviations from the null from each other, either we test all at the same time or we need to condition on the left out possible alternatives.

user: josef-pkt
issue_2050: state: open
title: one-step estimation, gauss-newton regression, influence, score tests
body: This is a generic implementation issue. I'm not sure yet how everything fits together

Main point we exploit a local quadratic approximation of the loglikelihood (objective) function or a linear approximation of the estimating equation or score function to get fast calculations for local deviations of the model.

example score tests or lm tests: Davidson MacKinnon use the GNR or it's nonlinear variants (Logit) for specification tests

we had the discussion before on other issues about using the linear approximation for diagnostic, influence and outlier measures (GLM and GEE)

example for general principle is also here (found by chance)
http://support.sas.com/documentation/cdl/en/statug/63962/HTML/default/viewer.htm#statug_logistic_sect051.htm
in the section on diagnostic see the description of the augmented model for calculating dfbeta

The main implementation choice is which estimator for the covariance of the score (cov_params) is used, OPG (outer product of gradient), observed hessian or expected hessian

I guess using hessian is one Newton optimization step, using expected hessian is one Fisher scoring step, some of the Davidson MacKinnon auxiliary regression cases use the OPG.

variation: for the robust case we need the sandwich variant  (somewhere)

related topic for my literature search: robust score tests, robust to misspecification of the alternative or of higher moments.

user: josef-pkt
issue_2049: state: open
title: Make sure strings/factors work as y for binary models
body: See #2048 for the general approach.

user: jseabold
issue_2041: state: open
title: SUMM: score tests and similar
body: I need a summary issue for the score_test, lagrange multiplier test, artificial regression, moment restriction tests

They are all similar or asymptotically the same with different assumption on robustness to misspecification. There are many open issues for this
#1163 basic issue
#1756 conditional moment restriction
#2054 score_tests with robust covariance assumptions

related:

RESET test for GLM, Logit, Probit (example also in Papke Wooldridge for proportion response model)

special cases
overdispersion test in Poisson (and related count models)

**implementation***

what do we have available for a test?

two (or three) cases:
- if we have a fully specified model: we can use score_test directly
  we have score_obs, score, hessian w.r.t. all parameters that we want to test
  no distinction in handling score from parameters used in estimation from those under test
  e.g. variable addition tests
- we only have a partial model without fully specified alternative in the model: we can use artificial regression, "scores" defined outside of the model or conditional moment tests, or embed in GMM
  - the only generic setup I can see is conditional moment tests `cm_test`
  - testing covariance assumption in mean models: a large battery of LM test for OLS, but in a similar way should be extendable to entire LEF, I have it for GLM reusing the methods from the link and family.
  - special case tests: e.g. score tests for poisson versus zero inflated or hurdle model.
- general tests on properties: Hausman, Information Matrix equality, ...
  can be formulated as LM test in linear case, but I don't know how they fit in "generically"

**issues on infrastructure**
#2072  get_robustcov to be reused for score tests

artificial regression: Gauss-Newton regression of Davidson-MacKinnon
generic "one-step" estimators  (see for example Newey McFadden 1994 Handbook of Econometrics
-> reuse Newton optimizer, iterative WLS results ?

**multiple hypothesis, multiple testing**
just an idea: using normal based score test for single restriction
simplest case: we have many possibly omitted variables that we want to test
we can test all of them at the same time - standard score test, like `results.wald_test`
we could also test the individual restrictions/additions - like `results.t_test`
questions: correlation of the omitted variables, jointly or in loop?
for example Pagan and Vella 1989 report separate t-statistics for a list of moment restrictions (omitted variables or heteroscedasticity)
Stata also reports separate tests with optional p-value correction in some of the estat specification tests after regress

Where are the score_test examples that I already did?

https://github.com/statsmodels/statsmodels/pull/1781/files#diff-724e15f4019b2a24906f6b271d67e370R433
Poisson overdispersion are not in repo


**edit**
variable addition score test: reduced rank #5364
We need to handle the case when some added variables are already implicitly included in the restricted model. GLM.score_test uses the full number of variables and not the rank of the addition.
(found while running an example for GAM in #5296 )
user: josef-pkt
issue_2039: state: open
title: hypothesis testing in beta regression
body: a followup to PR #2030

As reference, the latest version for parameter tests in Beta regression
http://www.tandfonline.com/doi/abs/10.1080/00949655.2012.700456

score test is doing pretty well, it maintains mostly the size
Wald tests are very liberal
bootstrap based tests and small sample corrected tests do well, but not much better than the score test

for generic score_test we need to handle two `score_factor` and two `exog`.
Otherwise it's the usual `score_factor * exog`

user: josef-pkt
issue_2038: state: open
title: Optimizer newton, check for inf and nans
body: debugging a case for beta regression

the function value is inf and the hessian is nan
fails with `LinAlgError: Singular matrix`

```
(Pdb) oldparams
array([   1.5829879 ,   -0.83957631,    1.485248  ,   -0.35448903,
       -169.71926289,    2.99451304])
(Pdb) f(oldparams)
inf
(Pdb) hess(oldparams)
array([[ nan,  nan,  nan,  nan,  nan,  nan],
       [ nan,  nan,  nan,  nan,  nan,  nan],
       [ nan,  nan,  nan,  nan,  nan,  nan],
       [ nan,  nan,  nan,  nan,  nan,  nan],
       [ nan,  nan,  nan,  nan,  nan,  nan],
       [ nan,  nan,  nan,  nan,  nan,  nan]])
```

example from miscmodels.test_beta

```
from statsmodels.genmod.families import links
model = "methylation ~ gender + CpG"
Z = patsy.dmatrix("~ age", tb.methylation)

# fails with `bfgs` and `newton` with default start_params
mod3 = Beta.from_formula(model, tb.methylation, exog_precision=Z, link_precision=links.log())
res3d = mod3.fit(method='newton', ridge_factor=0)
```

user: josef-pkt
issue_2027: state: open
title: Predictive accuracy measures for count data
body: similar to #1577 for binary data

measures of goodness of fit or of the accuracy of the predictive distribution for count data.

references 

Czado C, Gneiting T, Held L. 2009. Predictive model assessment for count data. Biometrics65:1254–61

Gneiting, Tilmann, and Matthias Katzfuss. 2014. “Probabilistic Forecasting.” Annual Review of Statistics and Its Application 1 (1): 125–51. doi:10.1146/annurev-statistics-062713-085831.

user: josef-pkt
issue_2016: state: open
title: Add bw_method to sm.graphics.violinplot + default colors
body: If someone wants to take up #1226 and finish it.

user: jseabold
issue_2015: state: open
title: Prediction confidence intervals GLM and discrete
body: confidence intervals for the mean (expected value) of non-linear, single-index models

ci can be obtained by applying the inverse link function to the linear prediction ci. 
They don't need the delta method, and Stata uses a mix of delta method and transformation of bounds.

maybe that's what this is about http://stackoverflow.com/questions/26053810/python-statsmodels-adding-confidence-intervalls-to-binary-logistic-regression

I thought I opened an issue for Poisson prediction but cannot find it

see also #1349 for linear models
and #1449 for general non-linear transformation

---

about Stata:

UCLA stats has an example
http://www.ats.ucla.edu/stat/stata/faq/adjustpr.htm
which uses adjust, which I never heard about.
I can match those numbers with just the model.cdf of the confidence intervals for the linear prediction.

`adjust` was removed and replaced by margins in Stata 11, which only uses delta method but not endpoint-transformation confidence intervals.
http://statalist.1588530.n2.nabble.com/adjust-vs-margins-revisited-td5355569.html

However, I read somewhere (???) in the Stata manual about endpoint transformation confidence intervals for predefined transformation, exp for odds ratio and similar in Poisson or other exponential/log-linear models.

user: josef-pkt
issue_2011: state: open
title: OLS summary   add explicit is-singular in the top table
body: Given that we have now a wider userbase, the default behavior of pinv is not really useful for the unwary.

see for example
http://stats.stackexchange.com/questions/116825/different-output-for-r-lm-and-python-statsmodel-ols-for-linear-regression

see also #1448 not all users see the warnings in the bottom text

user: josef-pkt
issue_2005: state: open
title: BUG emplike `emplikeAFT.predict` uses endog
body: I don't think this can be right

```
    def predict(self, params, endog=None):
        if endog is None:
            endog = self.endog
        return np.dot(endog, params)
```

The results class doesn't define predict and doesn't inherit, so I guess it has never been used.

separation between the model class emplikeAFT and the results class AFTResults looks strange. 
fitting is params of the results class.

user: josef-pkt
issue_2002: state: open
title: Indicate that offset/exposure was used in GLM summary
body: Currently there is nothing in the printed summary of the model fit to inform/remind you that an offset was used when fitting the model.

user: kshedden
issue_2001: state: open
title: GLM from_formula pass auxiliary variables by name
body: We allow auxiliary variables (offset, exposure, etc.) to be passed by name in a few from_formula's (GEE, PHReg).  Currently this isn't implemented for GLM.from_formula.

user: kshedden
issue_1997: state: open
title: TST coverage cov_type robust covariance
body: trying to get an overview over which cases of models and cov_types, and implementations are tested

TODO: increase coverage
Stata: `glm` has HAC, `prais` has HC and cluster
## linear models
#### OLS

regression\tests\test_robustcov.py  essentially complete coverage overall, not all cases tested in `fit`
regression\tests\test_cov.py  test for consistency of HC0 attributes, with `cov_p` argument in t_test, f_test
also some HC in tests against Gretl

for robust_covariance functions with OLS
stats\tests\test_sandwich.py  cov_hac_simple, cov_cluster_2groups
stats\tests\test_panel_robustcov.py cov_nw_panel, cov_nw_groupsum and cov_cluster
#### WLS

regression\tests\test_robustcov.py   HCx and cluster, no `fit`
#### GLS

None
## discrete and GLM

discrete\tests\test_sandwich_cov.py
**Poisson, GLMPoisson**  cluster and HC1   (against Stata)
**NegativeBinomial** cluster   (against Stata)
cross-check
Logit - GLM-Logit   cluster
(Probit - GLM-Probit disabled, hessians don't agree very well)
**OLS - GLMGauss** cluster, HC0

user: josef-pkt
issue_1971: state: open
title: DOC: Topic section in documentation
body: We need some central location for documenting features that are common to most models, and/or have a generic implementation.
### robust covariances

main candidate right now are robust covariances: There is no systematic documentation of the options for it, it's spread out over various docstrings.
### Wald tests

Wald tests are our basic inference tool, it's completely generic. This should also have a central description place
### Optimization options

Has good description in docstrings of fit functions, FAQ for optimization problems
### general common features

predict, ... ???
### other things that are partially available or WIP

fit_constrained, 
fit_regularized
generic score/LM tests
diagnostics extended to other models than OLS

user: josef-pkt
issue_1970: state: open
title: FAQ-D/SUMM: required attributes of models and results
body: We don't have a list of what attributes models and results need to define to be able to use generic methods.

The requirements are increasing from version to version. These requirements are usually inherited when we go through the regular data handling procedure, but might be left out if a new model follows a different pattern.

eg. we need extra attributes for wald_test (f_test, t_test)

k_constant
exog_names  (which are params_names in general)
df_resid, df_model
#632
#436

**edit** we should collect a checklist

- recreating a model init_keys, e.g. for llnull, example #3888
- score_obs for sandwiches
user: josef-pkt
issue_1967: state: open
title: NegativeBinomial, score, hessian inconsistent definition if _transparams = True
body: see #1943

score is with respect to transformed params
hessian is with respect to untransformed params, but correctly evaluated at the transformed value

not really a bug, because optimization is designed around this, but not nice. 
And we need to watch out that the optimizers don't use the inconsistent score/hessian at the same time.

user: josef-pkt
issue_1966: state: open
title: FAQ-D: How do we add robust covariances to a new model?
body: see ???

The outer part: Hessian
The inner part: score_obs  (currently jac is old name, linear model uses exog and resid)
The structure information: time, groups, ...
The small sample corrections: default options

How is it connected?

TBA

user: josef-pkt
issue_1963: state: open
title: ENH: BigModels for out of core, limited memory estimation
body: see https://groups.google.com/d/msg/pystatsmodels/CICsroXZmrs/USkmpBJ9SbIJ
and some follow-up threads like
https://groups.google.com/d/msg/pystatsmodels/RgnLmXE0F7o/A-84qD9RNBMJ

R has biglm and bigglm

user: josef-pkt
issue_1962: state: open
title: ENH: allow cov_type=None to skip Hessian/cov_params calculation
body: I was looking into subset selection, selecting features or checking many different parameterizations.

In many of these cases we don't need the `cov_params` which is mainly for inference, we need some goodness of fit measures and maybe the params.

related see #1943 that only removes duplicate calculation of Hessian

user: josef-pkt
issue_1953: state: open
title: ENH: newton with initial hessian
body: Does any of the newton type estimators allow that we provide an initial estimate of the hessian?

This would be useful if we want to investigate different models or datasets (like leave-one-out) that start out with the full model for which we already have all the results.

application: influence and outlier diagnostics, either full or one-step estimation after dropping some data points.

user: josef-pkt
issue_1951: state: open
title: MixedLM  reuse _smw_solve in special cases
body: just an idea for possible performance improvements while staring at some linear algebra

`_smw_solve`  is called very often to invert the residual covariance matrix 

The calculations depend on exog_re

In some special cases like random constant the exog_re are the same across all units or only depend on the length for the individual exog_re.

`_smw` could be converted into a class that has a multiply/whiten method and keep the repeated calculations around. (similar to chol_factor, chol_solve or others)

related to #1948 

user: josef-pkt
issue_1950: state: open
title: ENH: full MLE for composite models ? ZeroInflated, Heckman
body: add examples for how to reuse existing models to create composite models

examples 
ZeroInflated Poisson or NegativeBinomial  (miscmodel, Skipper's gist)
Heckman sample selection: PR #1935  (more difficult pattern than ZIP)

question: Is this relevant?
ZIP is a mixture distribution where we don't observe the class membership.
In censored regression we know whether censoring occured.
(truncation versus censoring)

user: josef-pkt
issue_1949: state: open
title: ENH: generic MLE for GLS models
body: Create a generic class for rapid development and to establish a pattern for this type of models.
based on non-linear optimization and either full or concentrated log-likelihood.

partially a duplicate of #874 MLE and REML for FGLS model with heteroscedasticity

see #1948 for covariance structures

high level general approach, e.g.
Linear and Generalized Linear Mixed Models and Their Applications, section 1.3
http://link.springer.com/book/10.1007%2F978-0-387-47946-0

Implementation 

idea: similar to GEE allow for parameterized `cov_struct` or variance function as in GLM that handles the details, 

related questions:
What are the conditions that iterated FGLS and MLE are equivalent? In which cases do we have equivalence?   (only asymptotic equivalence are numerical equality?)

user: josef-pkt
issue_1948: state: open
title: SUMM: tools for residual covariance structure 
body: **objective:** collect the tools, functions or classes, for handling non-i.i.d. covariance structures

The idea is to get reusable components for a mix-and-match system as complement to the (more efficient) model specific implementations.

**models:** 
- panel/longitudinal oriented: GEE, MixedLM, Panel
- other structures in WLS/GLS group: linear models with AR, heteroscedastic or SUR residuals.

**estimators:**
- two-step/iterative: covariance of error term is calculated in a separate step, GEE, Panel, HetGLS, ...
- one step estimator - MLE or REML: currently only MixedLM  see also #874

for _two-step_ estimators we will need mainly the estimation of the covariance parameters and a `whiten` method, 

see WLS/GLS cases  GLSAR, GLSHet
some functions are in GEE (and a messy incomplete trial version in the sandbox)

for the _one-step_ estimators, MLE, REML, we need also the log-determinant, and derivatives for numerical optimization and `cov_params`

see for example Mixed Effects model:`_smw_solve` and `_smw_logdet` for `s*I + A*B*A'` 
what we need is something like `statsmodels.stats.correlation_tools.FactoredPSDMatrix` for `diag + A*A'`

**related**
- #1646 
- there should be more comments or issues in the context of GEE
- GLSAR Prais-Whinston ?  (AR(1) only)
- covariance tools in tsa, e.g. correlation implied by AR, ARMA

user: josef-pkt
issue_1945: state: open
title: FAQ-D   inplace negation
body: Is there a way to negate a boolean, or to change the sign of a float inplace?

np.logical_not(arr, out=arr)
np.negative(arr, out=arr)

might be useful for negating loglike and similar during optimization.

(asked on numpy mailing list)

user: josef-pkt
issue_1944: state: open
title: Hessian is not cached, cov_type
body: Do we need repeated calls to `model.hessian` with the final parameters?

Hessian can be expensive to calculate (e.g. complex derivative in MixedLM would take more than a second) and we currently don't cache it.

However, in the results the hessian or a rescaled version is usually stored as `normalized_cov_params`

For the sandwiches, `hessian` is currently created in `fit` (e.g. LikelihoodModel.fit for nonlinear optimization using scipy) and calculated again in  `get_robustcov_results`

The one in `super().fit` could be for the wrong parameterization and could be skipped. see #1943 

If we use `cov_type="OPG"` (which is not implemented except for GenericLikelihoodModel), then we wouldn't need a Hessian at all.

Implementation:
implicit caching could be tricky because it's needed in several steps of the fit and results calculations.
Results.hessian is a method not an attribute, so we would need a separate cached attribute.

It might be easier to check whether `normalized_cov_params` or a scaled version of it could have a definition and alias that quarantees that it's the hessian (second derivative of the objective function). 
This would be different from the current use as unscaled cov_params, although overlapping in most cases.
for example in QuantReg `normalized_cov_params` is still the default cov_params which depends on the `cov_type` (called `vcov` in this case)

user: josef-pkt
issue_1941: state: open
title: REF/ENH: predict autodetect transform 
body: If I use a numpy array as exog in predict but the model was created with a formula, then patsy doesn't find the column names.

We could switch to `transform=False` in this case.  I don't know if there are any ambiguous cases.

related #540

```
>>> fitted0 = res_gex.predict(res_gex.model.exog_li[0])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new2_py27\statsmodels\statsmodels\base\model.py", line 698, in predict
    exog)
  File "build\bdist.win32\egg\patsy\highlevel.py", line 278, in dmatrix
  File "build\bdist.win32\egg\patsy\highlevel.py", line 156, in _do_highlevel_design
  File "build\bdist.win32\egg\patsy\build.py", line 889, in build_design_matrices
  File "build\bdist.win32\egg\patsy\build.py", line 158, in eval
  File "build\bdist.win32\egg\patsy\eval.py", line 480, in eval
  File "build\bdist.win32\egg\patsy\eval.py", line 463, in _eval
  File "build\bdist.win32\egg\patsy\compat.py", line 133, in call_and_wrap_exc
  File "build\bdist.win32\egg\patsy\eval.py", line 122, in eval
  File "<string>", line 1, in <module>
  File "build\bdist.win32\egg\patsy\eval.py", line 41, in __getitem__
  File "build\bdist.win32\egg\patsy\eval.py", line 41, in __getitem__
ValueError: field named CpG not found.


>>> fitted0 = res_gex.predict(res_gex.model.exog_li[0], transform=False)
```

user: josef-pkt
issue_1936: state: open
title: FAQ-D writing unit tests, getting reference results
body: (I'm adding this here because I have no idea whether our developer documentation is very explicit.)

**getting results from Stata**

see PR comment for what I am doing:
https://github.com/statsmodels/statsmodels/pull/1935#issuecomment-53361910

unit tests are written taking existing unit tests as examples.
TODO:
We should have better base classes and helper functions for unit tests. E.g. results from Stata and test methods to compare with Stata are partially independent of model.

user: josef-pkt
issue_1935: state: open
title: Heckman
body: Added a module to estimate the Heckman selection model using the Heckman 2-step. A test module checks the parameter estimates that this module creates in an example dataset against the parameter estimates that Stata creates. If this happens to merge well into the existing statsmodels code, I'd like to add an MLE estimation method as well. Stata currently has the option to estimate the Heckman model by either the 2-step or by MLE.

user: bert9bert
issue_1932: state: open
title: Gamma GLM fails easily with reciprocal link
body: The gamma GLM seems to fail easily during optimization when using the canonical reciprocal link.  It works OK with clean simulated data, but fails with real data sets or complicated models.  With the exponential link it is more stable.  I will put together some examples.

user: kshedden
issue_1930: state: open
title: SUMM: consequence of k_constant=0, check consistent handling
body: (no time right now to collect issues)

we have several places and open issues where we don't handle the absence of a constant correctly or consistently, eg. in defining df_model, ...
- [ ] GLM uses `- 1` in df_model
- [ ] GEE I'm also just adding now `self.df_model = self.exog.shape[1] - 1  # assumes constant`.
    Complication in GEE is that it also handles linear constraints by parameter transformation, and exog is the transformed matrix.  (need to adjust k_constant in this case ?)


some of the related issues
#1664 general definition of nobs, df_xxx, k_xxx
#1723 df_xxx in models and results
user: josef-pkt
issue_1928: state: open
title: GEE: question behavior with singular exog, k_exog versus rank
body: I don't know what the behavior is, or what it should be.
followup question to defining df_resid #1918 

in several other models that don't have panel structure, we define df_resid using the `rank(exog)` and not `exog.shape[1]`

GEE uses as denominator for the scale estimate `nobs - exog[0].shape[1]`.
Stata has df_m and df_pearson

I'm adding df_resid to GEE, with `df_resid = nobs - exog[0].shape[1]` so it corresponds to the scale calculation.

user: josef-pkt
issue_1925: state: open
title: GEE: several results with different covariance, slow fit.
body: usability question: about short circuiting GEE.fit with known parameters

based on unit test for `TestGEEMultinomialCovType`
includes 3 fit calls with difference cov_types plus a few asserts (that don't calculate much)
(need better example for timing)

running the test without start_params: Ran 1 test in 5.228s
running the test with start_params:      Ran 1 test in 3.191s

one second with known start_params looks too much.

GEEResults is all set up to create a new results instance from an existing one.
`get_robust_cov_results` could be made to work as in RegressionModel.

user: josef-pkt
issue_1923: state: open
title: FAQ: robust covariances in statsmodels compared to Stata
body: statsmodels provides largely the same types of robust covariances that Stata provides. 
However, statsmodels provides the same set of covariance options in several of the main models, currently linear regression, discrete models and GLM. Several models, like RLM, GEE and quantile regression provide their own model specific robust covariance choices.

Stata provides the basic covariance types across models, however many covariance types are only available in some estimation function. For example, HAC is in `newey` and as `vce` option in GLM.

The default options for the covariances for small sample correction differ in several cases between statsmodels and Stata. (And in some cases Stata including user contributed packages has different defaults across estimation functions.)

(TODO: We need to bring the explanations for different models and covariance types together somewhere.)

Note: There are still several open issues to change and improve the availability of robust covariance types in statsmodels.

user: josef-pkt
issue_1922: state: open
title: REF/DOC robust covariance, cov_type in fit
body: PR #1870 adds generic function and adds robust covariance choice to fit

I'm pretty sure it's correctly connected based on unit tests and examples.
Inference based on the robust covariance is only unit tested on a few examples and is incomplete.
But there are also several parts separately **unit** tested instead of the various/all model results.
(add example to generic tests after GEE cov_type PR #1916 is in)

TODO:
- documentation: both a rst file with summary and inclusion in the individual docstrings
  question: copy to each docstring (templated), or just point to central docstring?
  similar issue as for optimization option in discrete models.
- check fit signatures for consistency: some models `fit` have explicit keywords `cov_type`, `cov_kwds`, some have it hidden in `**kwargs`
  All final classes should have explicit keywords. The base model classes have it in `**kwargs` because they don't apply to all cases. (e.g. NegativeBinomial needs to process cov_type after the fit parameters are not transformed any more)
- check and decide on where `use_t` should be changed or attached. I started with it as part of `cov_kwds` and then moved it to an explicit keyword, next to `cov_type`, `cov_kwds`.
  `use_t` could or should be removed from `cov_kwds` (but we would need to add it everywhere separately.)

more:
- regression still has it's own method `get_robustcov_results`. It should be possible to use the generic version now.
- LikelihoodModelResults has `_get_robustcov_results` which is currently only for internal use, while `get_robustcov_results` is tested that it correctly creates a new instance.
- cov_kwds: are currently not checked for extra or misspelled kwds, those are silently ignored.
  (it's not completely easy: need to pop, check if empty at end, but verify we didn't miss any)

ENH:
- model specific cov_types: no cases yet. Should be possible by using model specific `_get_robustcov_results` method that takes care of some cov_types and delegates to generic/super for the standard ones. example overdispersed discrete.Poisson.

user: josef-pkt
issue_1921: state: open
title: ENH: Heckman Correction
body: I'm thinking of contributing a small module to perform Heckman correction for selection bias. If I were to do so, what classes and APIs should my code inherit, and where should I put the module?

user: bert9bert
issue_1920: state: open
title: GEE: covariance_type  almost no test coverage
body: based on a search in test_gee, there are no unit tests that verify/assert the numbers for the bias_reduced covariance.

naive covariance has one test against OLS

user: josef-pkt
issue_1913: state: open
title: FAQ: How stable is statsmodels? Do new versions break backwards compatibility?
body: the short version:

We are careful with changing established or heavily use models and functions, with at least one year of deprecation if there are changes that break backwards compatibility.

Recently added code or code that hasn't seen much usage, and where we didn't get any feedback might be changed, if it is really necessary, without (long) deprecation.

Internal code, attributes and methods, and the implementation might be changed at any time. 
The boundary between public API and internal API is not well defined.

Code in the sandbox can be changed at any time. 
This does not include code that is located in a sandbox module but has a public import path through one of the non-sandbox APIs. Examples are statsmodels.stats.multitesting and tukey_hsd.

user: josef-pkt
issue_1912: state: open
title: covariance estimators based on inner sandwich
body: Provide estimates of covariance matrices as they are defined in the inner part of the sandwich robust covariance estimators as standalone function.

This was raised before by Kevin for a HAC Newey-West covariance.  issue ?
We also need cluster based covariance matrices.

IIRC, also for GMM I need the inner part separately from the outer part.

related 
issue #1109 normalization by nobs or degrees of freedom

user: josef-pkt
issue_1911: state: open
title: ENH: GEE summary additions
body: add a wald test for the "null" model, i.e. all slope coefficients are zero?
(GEE also has score test, but in other models we use likelihood ratio or wald test.

summary doesn't show time, should it?

user: josef-pkt
issue_1910: state: open
title: REF: GEE covariance types, compute on demand
body: GEE currently computes all three types of covariance matrices in the model in `_covmat` at the same time.

Instead we could calculate it only based on the `cov_type` that has been chosen by the user in `fit`.

This might also apply to other models that precalculate different covariance types.

(In my robust covariance code, I calculate robust covariances in the results instance.)

user: josef-pkt
issue_1909: state: open
title: TST: streamlining unit tests, assert and helper functions
body: just an idea while comparing equivalent models in a script/notebook, see #1825 

Currently we write the assert in methods that can be inherited. However, this does not allow us to reuse the same kind of asserts or comparisons in other test functions or test classes.

For example, in the generic tests in base I added methods that verify that t/F or normal/chisquare are used, e.g. for p-values and confidence intervals.

Another candidate would be to compare the main attributes between two models.

(unit tests make up a lot of code, that is barely reviewed or refactored)

user: josef-pkt
issue_1908: state: open
title: add sanity check method to models
body: silly user mistake, using the wrong data #1907 

besides the open issues that we have on dtype checking (for example with object arrays) there are other problems that we could either explicitly check for, or we add helper functions or methods that do the basic sanity checks of the data for the user.

FAQ: Why does my model not work?
Answer: run `model.sanity_check`

(we also have an issue for statistical "sanity" check, i.e. run a battery of diagnostics on the model or results.)

user: josef-pkt
issue_1907: state: open
title: input checking, corner case, convergence problem GEE  "long panel"
body: Below there is the traceback from a GEE Binomial fit example (from my notebook comparing GEE, GLM and discrete).

I just discovered that my problem was because I used count data in GEE, instead of the binaries data. There is no problem with the binary data.

So the main problem, I guess, is to fail with an informative message, or be able to do some basic checking on the data.

```
endog.values
array([  0.,   0.,   3.,   4.,   6.,  18.,  11.,  39.,  29.,  58.,  53.,
        12.,  44.,  18.,   1.,   1.,   0.,   1.,   6.,   2.,   1.,   0.,
         0.,   0.,   0.,   2.,  11.,   4.,   0.,   7.,   7.,   5.,  12.,
         1.])
```

```
fam = families.Binomial()
ind = Independence()
mod1 = GEE(endog, exog, group, cov_struct=ind, family=fam)
res_gee = mod1.fit() #start_params=res_glm_logit_clu.params)
#print res_gee.summary()
print(res_gee.bse)
---------------------------------------------------------------------------
LinAlgError                               Traceback (most recent call last)
<ipython-input-136-768f8c8ca52d> in <module>()
      2 ind = Independence()
      3 mod1 = GEE(endog, exog, group, cov_struct=ind, family=fam)
----> 4 res_gee = mod1.fit() #start_params=res_glm_logit_clu.params)
      5 #print res_gee.summary()
      6 print(res_gee.bse)

e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new2_py27\statsmodels\statsmodels\genmod\generalized_estimating_equations.pyc in fit(self, maxiter, ctol, start_params, params_niter, first_dep_update, covariance_type)
    851         for itr in range(maxiter):
    852 
--> 853             update, score = self._update_mean_params()
    854             if update is None:
    855                 warnings.warn("Singular matrix encountered in GEE update",

e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new2_py27\statsmodels\statsmodels\genmod\generalized_estimating_equations.pyc in _update_mean_params(self)
    646             score += np.dot(dmat.T, vinv_resid)
    647 
--> 648         update = np.linalg.solve(bmat, score)
    649 
    650         self.fit_history["cov_adjust"].append(

C:\Programs\Python27\lib\site-packages\numpy\linalg\linalg.pyc in solve(a, b)
    326     results = lapack_routine(n_eq, n_rhs, a, n_eq, pivots, b, n_eq, 0)
    327     if results['info'] > 0:
--> 328         raise LinAlgError, 'Singular matrix'
    329     if one_eq:
    330         return wrap(b.ravel().astype(result_t))

LinAlgError: Singular matrix
```

user: josef-pkt
issue_1900: state: open
title: corner case: corr_nearest_factor optimization fragile, non-convergence
body: see comment in issue #1850

possibility of convergence cycles, i.e. non-convergence
can it hang?

user: josef-pkt
issue_1896: state: open
title: WLS, GLS   Check if cov_type works
body: No test cases for GLS, 

OLS is tested 
WLS is tested for HC and cluster

user: josef-pkt
issue_1882: state: open
title: TST: GLM binomial, check score_obs and hessian for frequency weights
body: I don't know if weights is handled correctly in binomial.

Needs test for the weights case.
consequence whether robust covariance is correct in this case, PR #1870

related Probit with counts http://stackoverflow.com/questions/25253133/replicate-probit-regression-in-python-and-spss
does it work in GLM or not?

user: josef-pkt
issue_1878: state: open
title: links probit, CDFLink  precision loss in Hessian
body: comparing hessian in GLM with hessian of Probit, the agreement is not very high

Are there numerical problems in the numerical second derivatives of the link function?

```
>>> hess_glm_probit = res_glm_probit_clu.model.hessian(res_probit_clu.params)
>>> print(hess_probit - hess_glm_probit)
[[ 0.03198082 -0.02195936  0.0191278 ]
 [-0.02195936  0.02408385  0.02408385]
 [ 0.0191278   0.02408385 -0.00904406]]
>>> print(hess_probit / hess_glm_probit)
[[ 0.99975476  1.0007156   0.99958227]
 [ 1.0007156   0.99793767  0.99793767]
 [ 0.99958227  0.99793767  1.00047687]]
>>> hess_glm_probit
array([[-130.40437081,  -30.68664568,  -45.79012885],
       [ -30.68664568,  -11.67795402,  -11.67795402],
       [ -45.79012885,  -11.67795402,  -18.96560781]])

>>> print(hess_probit / hess_glm_probit - 1)
[[-0.00024524  0.0007156  -0.00041773]
 [ 0.0007156  -0.00206233 -0.00206233]
 [-0.00041773 -0.00206233  0.00047687]]
```

in contrast, agreement in score_obs is at floating point precision

```
>>> sc_probit = res_probit_clu.model.jac(res_probit_clu.params)
>>> sc_glm_probit = res_glm_probit_clu.model.score_obs(res_probit_clu.params)
>>> print(np.max(np.abs(sc_probit - sc_glm_probit)))
4.4408920985e-16
```

part of cross-checking in #1870 between GLM and other models

user: josef-pkt
issue_1876: state: open
title: TST: add GenericLikelihoodModel to generic tests, e.g. pickle
body: see #1873  GenericLikelihoodModel couldn't be pickled and it has been changed as a side effect of unrelated changed (use_t and distribution for inference and conf_int

GenericLikelihoodModel doesn't have generic test as those in statsmodels base for pickling, and consistency of names.

(might not all apply in the general form)

user: josef-pkt
issue_1874: state: open
title: Kalman Filter on univariate ARIMA Time Series forecast results
body: Hi everyone,
I was wondering if anyone can help me with the KalmanFilter class in statsmodels.
I have been trying to improve my forecast models using pykalman library.
But I recently noticed that it the transition matrix using Expectation–maximization algorithm (also gives other parameters to compute the Kalman Gain matrix was not rigorously adequate with ARIMA model.
I recently saw various things on statsmodels (http://statsmodels.sourceforge.net/stable/_modules/statsmodels/tsa/kalmanf/kalmanfilter.html#KalmanFilter.T) but I was wondering how can we know our Kalman Filter parameters automatically.Mainly the transition matrix using Kalman Filter in statsmodels.

user: babacar91
issue_1871: state: open
title: New version of the Univariate KDE estimator
body: For my own research, I developed a Univariate KDE module, with a focus on managing boundary conditions. You can find the documentation there:

http://pyqt-fit.readthedocs.org/en/latest/

the package contains more than just the KDE, but I would like to extract the univariate KDE estimator  and include it in statsmodels, Of course, I would perform the necessary renaming and any other structural change that would be deemed necessary for inclusion in statsmodels.

Compared to the statsmodels implementation, I decomposed the KDE evaluation in 4 parts for improved modularity:
- the main KDE object (to provide a uniform API)
- the kernel object
- the bandwidth calculation
- the KDE estimation method

I believe this separation is important to be able to test new methods or kernels. You can follow the tutorial to see how the various parts interact (although I don't talk much about the kernel).

All the modules have documentations and the testing coverage is about 97%, and I believe the structure should provide sufficient flexibility for anyone to implement their own algorithms, either for bandwidth, kernel or for the method itself.

Please, tell me what you think about it,

user: PierreBdR
issue_1862: state: open
title: DOC/TST: graphics add notebooks to visually inspect if options work
body: visual "crowd-testing" ?

We only have smoke test for the plots.
One addition that works both as tests and as documentation would be to add notebooks that just illustrate different options for the plots.

and it would make reviewing a PR for graphics easier
See for example #1861

user: josef-pkt
issue_1849: state: open
title: adfuller, autolag is wasting memory
body: see thread at https://groups.google.com/d/msg/pystatsmodels/iU3yiWpM2gM/uSacbHcUv3sJ

first point: autolag keeps all regression model/results for the optimal lag search.

user: josef-pkt
issue_1848: state: open
title: REF: emplike OriginResults repeated calculation
body: OriginResults el_test and conf_int_el repeatedly fit the same OLS model that was already fit in the model instance fit

`self.model.fit()`

`self.model` refers to the (auxiliary) OLS model, not to the underlying `ELOriginRegress` model instance

`ELOriginRegress._-init__` doesn't call super, no automatic data handling.
The auxiliary OLS regression model will have the super call.

user: josef-pkt
issue_1846: state: open
title: FAQ, FAQ-D: How to I verify numerically my analytical derivatives
body: use numdiff

TODO: example

user: josef-pkt
issue_1845: state: open
title: BUG: emplike weights.sum < 1, enforce sum close to 1
body: investigating #1316 and #1831 

I'm not able to replicate the failures, it always converges for me. 
However, checking `el_test` for different values, I also get weird/wrong results.

In one case `weights` are mostly zero
in other cases they sum is equal to one within numerical error.

```
>>> llr, pval, w, p = tt.res1.el_test(0.784, [2], ret_params=True)
>>> w.sum()
0.085400220469679738
>>> w
array([  4.86330832e-17,   7.35626704e-17,   4.71643799e-17,
         1.99456822e-17,   5.58759305e-17,   5.64207446e-17,
         5.00890999e-17,   4.38986001e-17,   4.75506879e-17,
         5.11990713e-17,   6.11848967e-17,   7.36822444e-17,
         8.40161017e-17,   8.04144292e-17,   3.74126620e-17,
         4.12366246e-17,   3.82284014e-17,   3.64449122e-17,
         3.26098741e-17,   3.38589133e-17,   8.54002205e-02])
```

my current script (I did many other things in interpreter directly)

```
import numpy as np
from scipy import linalg, stats

from statsmodels.regression.linear_model import OLS
from statsmodels.emplike.tests import test_regression as tr

tt = tr.TestRegressionNM()

print linalg.svdvals(tt.res1.model.exog)

sig = 0.05
r0 = stats.chi2.ppf(1 - sig, 1)

do_all = False
if do_all:
    llr_ = [([b] + list(tt.res1.el_test(b, [2]))) for b in np.linspace(0.6, 1.29, 31)]
    llr = np.array(llr_)

print tt.res1.el_test(0.784, [2])

llr, pval, w, p = tt.res1.el_test(0.784, [2], ret_params=True)
print w.sum()
llr, pval, w, p = tt.res1.el_test(0.8, [2], ret_params=True)
print w.sum()
```

user: josef-pkt
issue_1843: state: open
title: stats: add more functions and hypothesis tests based on summary statistics
body: There is no convenient way to calculate t-tests based on sample statistics only.

http://stackoverflow.com/questions/24904822/scipy-stats-ttest-ind-without-array-python

The current, generic functions like `_tstat_generic` don't contain an additional layer that, for example. calculates the standard deviation of the test statistic.

We could add some "usable" functions that can be used just with the summary statistics from the data, `(nobs, mean, std)` for each sample.

user: josef-pkt
issue_1833: state: open
title: FAQ-D: use_t choosing between normal and t distribution
body: see also #1759 for user FAQ

How do we implement the choice between normal and t distribution, or between F and chisquare distribution for parameter inference?

Result instances define an attribute `use_t` that all methods are supposed to use when choosing the distributions.

distribution of parameters is used in:
- pvalues
- conf_int
- t_test
- wald_test

hardcoded f distribution
- f_pvalue
- f_test
- anova_lm
- compare_f_test

hardcoded chisquare distribution
- LR-test like pvalue for `llnull`, `compare_lr_test`

not yet checked: Margins, prediction, ...

user: josef-pkt
issue_1818: state: open
title: ENH/BUG/DOC: discrete Model get_margeff don't use offset exposure
body: marginal effect do not take offset or exposure into account.
same for `_derivative_predict`

no exposure can be interpreted as a "standardized" or rate effect for unit exposure.
However, offset would be necessary, to get the correct reference points (for `at`).

I don't see this documented.

user: josef-pkt
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_1815: state: open
title: ENH: corner case, summary, verify converged score
body: We should verify that the score is actually close to zero at optimum.

In some cases it will be part of the convergence criteria, in other cases the score might not be very close to zero.

cases
see compare_lm_test score fails in #1787
optimization method='nm' has large score in some examples that I checked in the past

make it part of the summary warnings/notes ?

user: josef-pkt
issue_1814: state: open
title: FAQ-D  Docs cross-linking in docs and See Also
body: in `See Also` of docstrings we can directly cross-link  to any function, method or class that is included in the documentation.

**modules:** most modules don't have a rst page, so we cannot cross-link directly.
We can cross-link explicitly by creating a link.

example:

in `glm.rst`

```
.. _families:

Families
^^^^^^^^
```

and in GLM docstring:

```
    See also
    --------
    statsmodels.genmod.families.family 
    :ref:`families`
```

the first doesn't create a link, the second creates a link to the section header in the docs

user: josef-pkt
issue_1813: state: open
title: TST: add unittest for equivalent models
body: Stata has some overview tables with equivalent special cases across models.

We don't and have only limited unit tests, mainly what I have written to tests secondary results, 

affects mainly GLM versus discrete, and GLM versus OLS, and RLM versus OLS.

tsa.ARMA now also allows OLS as special case. (has unit tests IIRC)

there are also unit tests for replicating OLS with GMM, and IV2SLS with GMM. 
Do we have Poisson versus GMM?

user: josef-pkt
issue_1809: state: open
title: variable selection, stepwise regression
body: just adding a link to examples and datasets

http://www4.stat.ncsu.edu/~boos/var.select/

http://www4.stat.ncsu.edu/~boos/var.select/fas.bes.html contains forward and backward selection results in SAS and R

user: josef-pkt
issue_1808: state: open
title: tsa acf ccf different defaults for unbiased
body: Why are the defaults different for acf and ccf?

http://stackoverflow.com/questions/24601405/why-do-statsmodelss-correlation-and-autocorrelation-functions-give-different-re

can we fix this?

user: josef-pkt
issue_1806: state: open
title: ENH: diagnostic regression summary - panel version
body: Looking at an example, the autocorrelation tests a the bottom of the summary data assume that the entire sample is one ordered (time) series.

With panel data (consecutive observations within panel/group) we should use only within autocorrelation.

We don't have any information about the structure of the data in OLS or similar, time series/serial versus cross-section versus panel.

BTW currently we have in summary Durbin-Watson for autocorrelation, why not ....?

user: josef-pkt
issue_1805: state: open
title: inheritance subclasses of RegressionResults
body: while browsing the docs

`gmm.IVRegressionResults`  shows HC and other results, 

Are those methods and attributes correct for subclasses of RegressionResults?
IV, GLSAR, GLSHet, ...

Is there a good way to find test coverage for a specific model?  

user: josef-pkt
issue_1802: state: open
title: FAQ: Wrong value of BIC, AIC for linear regressions and OLS
body: The standard definition of BIC is

BIC = -2 log(L*) + k log(n)

where k is the number of parameters of the _full statistical model_. When assuming a model with Gaussian errors, 

y = f(x) + e, e~N(0, s)

k is the number of parameters of f _plus 1_ for the (unobserved) variance s of the Gaussian errors. BIC methods in statsmodels always ignore this +1, which results in BICs that are systematically shifted by -log(n). Although this does not have any effect on model selection, it is confusing and, strictly speaking, incorrect. IMHO it should be fixed.

**Update**

statsmodels follows the least squares and quasi-likelihood tradition and only takes mean parameters into account in the number of parameters for information criteria like AIC and BIC. Scale parameters and other auxiliary parameters that are not directly estimated as part of the main optimization are not included. This default does not change.

Starting with statsmodels version 0.13 (PR #7693), a new results method `info_criteria` is included for GLM and for linear regression that allows users to adjust the number of parameters in the information criteria calculation, using `dk_params` (for "delta k_params").

Full Maximum Likelihood method such as those used in `discrete` models estimate scale or dispersion parameters jointly with mean parameters and both types of parameters are counted in the information criteria computation.
(Generalize Poisson and NegativeBinomialP have an extra power parameter that is not estimated and is not taken into account for information criteria.)

user: seeslab
issue_1796: state: open
title: SUMM `outlier_test` everywhere
body: We have 32 open issues on searching for `outliers`.

So far we have an `outlier_test` method only for OLS.

We need generic functions or a common pattern to be able to add it to most models.
should be easy based on predefined "standardized" residuals, (standardized, studentized, anscombe, ...)

(there should be some plot functions in the examples.)

Bonus: 
Outlier test and plots should be able to use observations that have not been used in the fit.
This is useful to check observations that have been trimmed.

user: josef-pkt
issue_1793: state: open
title: columns with zeros
body: see https://groups.google.com/d/msg/pystatsmodels/tW1ldvNN6QU/42LPWxSE2pwJ

see #1627 for consequence with transformations
see #1792 for bug in constant detection

copy it here (I cannot copy right now without extra lines, double spacing)

user: josef-pkt
issue_1791: state: open
title: ENH: GLM profile confidence intervals
body: This creates profile confidence intervals for individual (single) parameters using offset to fix a parameter.

Example GLM binomial

test case: compares with regular conf_int, only tried for GLM binomial case
see test case for example

user: josef-pkt
issue_1790: state: open
title: everything GMM,  GLMGMM
body: Similar to GLMMLE #1788 I think we can get generic GMM for GLM families and links

Score_obs are moment conditions, and we should be able to just plug them into IVGMM, similar to the examples for Poisson.

Essentially, all we are doing is using the estimating equations from the families, similar to GEE but without the GEE covariance modeling, but with instrumental variables under the assumption of some orthogonality conditions.

see #1742 for IVPoisson or GMMPoisson as already worked out special case.

user: josef-pkt
issue_1789: state: open
title: robust variance of score/score_obs
body: Is the robust variance of the score (for score/LM tests) just the inner part of the sandwich?
I think so.   `Hinv @ (score_obs.T @ score_obs) @ Hinv` or `Hinv @ S @ Hinv`
where S is the heteroscedasticity, HAC or cluster robust covariance of the score_obs (or moment conditions in GMM)

See also `RegressionResults.compare_lm_test`

Make it separately accessible from `sandwich_covariance.py`.
I think it's also the same that Kevin needed for TSA and added HAC/Newey-West separately in `tsa`.

user: josef-pkt
issue_1788: state: open
title: ENH: GLM MLE with scale
body: I'm reading through the SPSS algorithm manual for GLZM (generalized linear model)

They also have ordered multinomial and Tweedie distribution.

The have the full formulas for derivatives and second derivatives, including the score and hessian for the scale estimate.

Should be relatively straightforward:
Create a GLMMLE class that includes the scale estimate in the params and only implements the families with a scale estimate in the loglikelihood function. (not over/under dispersed)
There is no advantage to replicating the scale=1 models, Logit, Probit, Poisson,
also we have already NegativeBinomial in discrete as full MLE but without link functions.

It's essentially similar to the `miscmodels` that are just MLE models with extra parameters, but taking advantage of the family and link structure of GLM.
The main difference would be that the extra parameters are included in params, loglike, score, hessian, and so on.
It would also be possible to parameterize the extra scale parameter as a function of the data (e.g modeled heteroscedasticity).

---

Related issue:
We want to use the nonlinear LikelihoodModel fit also for distributions that don't have extra parameters to make it easier to extend to nonlinear predictor (non just single index link function)

user: josef-pkt
issue_1787: state: open
title: corner case: score compare_lm_test: is_nested fails for Longley
body: compare_lm_test checks whether the restricted model is nested by checking the score.

Longley fails, score for Longley at the parameter estimate is large.
I don't see why, except Longley is a nasty case.
I tried to rescale endog but it didn't help.

(the constrained version had the last or the first two variables dropped.

```
from statsmodels.genmod.tests.test_glm import TestGlmGaussian

tt = TestGlmGaussian()
tt.test_compare_OLS()

from statsmodels.regression.linear_model import OLS
resd = OLS(tt.data.endog, tt.data.exog[:, 2:]).fit()
#tt.resd.compare_lm_test(resd, demean=False)
# raise
#ValueError: Restricted model is not nested by full model.
```

found while trying to use it as unit test of `GLM.score_test` versus `RegressionResults.compare_lm_test` PR #1781 

user: josef-pkt
issue_1786: state: open
title: degrees of freedom in score/LM test linear model compare_lm_test
body: testing `GLM.score_test` against `RegressionResults.compare_lm_test`

`RegressionResults.compare_lm_test` uses denominator nobs instead of df_resid in the calculation of the scale (it doesn't use precalculated scale

`GLM.score_test` uses df_resid in the scale

user: josef-pkt
issue_1784: state: open
title: REF: add exog argument to score, score_obs and hessian
body: Just a thought for making score_tests easier.
I'm writing `score_test` for GLM and need the score (score_obs) at other exog than self.exog. see PR #1781
The score falls nicely into a `factor * exog` pattern, so it would be easy to allow other exog.
see #1775 

`score(params, scale=None, exog=None)`

and the same for hessian.

user: josef-pkt
issue_1782: state: open
title: REF genmod variancefunction don't subclass
body: The genmod.varfuncs don't subclass `VarianceFunction`   oversight because not needed?

I tried to add a generic numerical derivative in PR #1781 but it wasn't inherited from the super class
https://github.com/josef-pkt/statsmodels/commit/6433db8747c19ff8016d63c51df84e78c7fbb6bf

user: josef-pkt
issue_1778: state: open
title: BUG family Gaussian.loglike power link
body: dead code

`Power` link is not allowed  (why not?)
`self.fitted` is not defined
doesn't take into account that `scale` is an argument and can be fixed different from MLE

```
if isinstance(self.link, L.Power) and self.link.power == 1:
        # This is just the loglikelihood for classical OLS
            nobs2 = Y.shape[0]/2.
            SSR = ss(Y-self.fitted(mu))
            llf = -np.log(SSR) * nobs2
            llf -= (1+np.log(np.pi/nobs2))*nobs2
            return llf
```

user: josef-pkt
issue_1777: state: open
title: GLM with "foreign" variance function
body: examples, unit tests

do we get the same results as with HetGLS if family is gaussian?

essentially do WLS, or feasible WLS, with GLM.

(GLM has a lot of mix-and-estimate options, and I don't think we have tried most of the "weird" combinations yet.)

user: josef-pkt
issue_1776: state: open
title: Design: fixing scale or nuisance parameters in estimation, concentrated likelihood
body: very vague

**part 1**
Some nested model comparison tests assume that we take some "nuisance parameter" as constant across models, usually it's recommended to use them from the unconstrained model.

examples: 
`compare_lr_test` should fix the scale, similar 
GMM compare objective function value at two parameters/models, with fixed the weight matrix
other similar ones for general classes of estimators

**part 2**
nuisance parameters that don't affect the "estimating equations", that can be canceled ordropped from the derivative of the loglikelihood function or score.

example: scale in linear model and in GLM

problem 1 we don't need them in the optimization, but we need them in the loglike value at the estimate, `llf`.  (Only relevant if we assume likelihood is correctly specified. We cannot use llf, if we assume only the estimating equations or moment conditions are correct. The latter are independent of the dropped parameters, like scale.)

problem 2 linear models use a concentrated likelihood where scale is implicitly calculated.

specific question mainly to part 2: 
- If we want to have both likelihood model (correctly specified) and quasi-likelihood (over-/underdispersed) model for GLM, how do we define `loglike` and `score`, and have `MLE` versus `M-estimator` in the same class.
- Which score do we need for score test, with or without dispersion/scale factor?
- If we want to get the loglike for arbitrary parameters, then the concentrated loglike always adjusts the scale. good or bad?

Comment: I guess in most cases. which way it is used, is irrelevant, as long as we use it consistently, with the same or matching definitions.

user: josef-pkt
issue_1775: state: open
title: ENH: score_factor_obs  for LEF and similar
body: In many models the `score_obs` decompose into

`f(u) * x`

where u are the residuals `u = y - fitted`, and `x` are exog or instruments.
f(u) depends on y, x and parameters

suggestion: add `score_factor = f(u)` to Results or Model classes.

advantage: we can calculate new score for different x (or instruments z in IV) 

Kevin used this for LM test in linear models, where `f(u) = u`

LEF/GLM have this structure, but it's not general enough for a `base` model

user: josef-pkt
issue_1774: state: open
title: generic anova tables for models other than ols
body: `stats.anova.anova_lm` calculates various ANOVA tables
It uses a sequence of wald, f tests.

The same pattern could be extended to work with other models.
especially the code for getting the terms should be a useful pattern to extend to other usecases.

(I just looked at a few more details in the implementation, I still never tried to fully understand it.)

user: josef-pkt
issue_1773: state: open
title: ENH: anova_lm, robust covariance with other sandwiches
body: `stats.anova.anova_lm` allows for heteroscedasticity robust covariances accessed in helper function `_get_covariance`

This can be extended to other robust covariances. (once they have a clean interface)

**update**

The way robust covariance matrices has changed since it became a `fit` option

`anova_lm` needs to be updated, `cov_params` (via `cov_params_default`) now returns the chosen cov_type, and there is the additional option `use_t`

anova_lm has a `test='Chisq'` which is currently not implemented, for type 3 anova `f_test` can be replaced by `wald_test`. (I tried that out and it works and honors `use_t`)

related: anova 3 works for GLM if `ssr` is ignored and `wald_test` is used. see #825

user: josef-pkt
issue_1767: state: open
title: Failure time == 0 in Cox model
body: There seem to be differences in how R and Stata handle failure times equal to zero (or more generally, failure times equal to entry times).  We currently agree with R (at least in the failure time == 0 case, I haven't checked with entry times yet).

Here is a relevant discussion:

http://www.stata.com/support/faqs/statistics/time-and-cox-model/

I'm not sure which is right.  We'll stick with the R convention for now.

user: kshedden
issue_1765: state: open
title: FAQ-D: Directory structure: Where should I put a new model?
body: (meta-issue to discuss location of models and directory structure)

**The main models**
- base
- regression
- discrete
- genmod
- robust
- duration    a more general group of models than survival models
- multivariate
- panel      not yet merged
- treatment      not yet merged
- tsa
  - vector_ar
  - filters
  - interp
  - statespace
- nonparametric
- emplike
- miscmodels

**other statistics and support**
- stats
- distributions
- graphics

**Meta tools**
- imputation  first method MICE
- resampling    (not yet)
- formula

**data and IO**
- iolib
- datasets

**mostly internal**
- compat
- tools

**other**
- sandbox

user: josef-pkt
issue_1764: state: open
title: Cox PHReg as Poisson model
body: interesting relationship

Stata manual has a description and example how to estimate PHReg with a Poisson model in the ststcox.pdf

This needs stsplit  to get repeated observations for each individual. (e.g. split on failure times, an individual gets a record for all intervals prior to failure)

Also, if there is within/across individual heteroscedasticity, then the robust covariance takes repeated observations into account. 
I didn't look at the details, the reference for vce(robust) is Lin and Wei JASA 1989 http://www.jstor.org/stable/2290085

One usage illustrated in the Stata example is to estimate piecewise linear hazard rate (with time interval dummies)

(I didn't see what the relationship between Poisson and PHReg is if there are ties, maybe it's Breslow.)

user: josef-pkt
issue_1759: state: open
title: FAQ: should I use normal or t and chisquare of f distribution (`use_t`)
body: ???

user: josef-pkt
issue_1758: state: open
title: ENH: more power: for LikelihoodModelResults
body: Wooldridge section 12.6.4 on local asymptotic power for M-estimators.

looks easy to add to t_test and f_test cases (in ContrastResults ?)
just uses normality, either noncentral chisquare or noncentral F test 
I would only start with "observed" power, I don't know if effect sizes and sample size calculation would be easy.

Wooldridge refers to 
Andrews, Donald W. K. 1989. “Power in Econometric Applications.” Econometrica 57 (5): 1059–90. doi:10.2307/1913623.
which inverts the power function
Looks like equal power (type 2 error) intervals or contours (ellipsoids)
also has direction of largest and direction of smallest power for linear combinations.
Andrews only requires a consistent estimator with asymptotic normality

related: we don't have "observed power" (given the test results for the data) connected or readily available
some articles were criticizing reporting observed ex post power.

**edit**

the interest for sample size calculations is often in alternatives that are not local/contiguous alternatives. 
Self, Steven G., and Robert H. Mauritsen. "Power/sample size calculations for generalized linear models." Biometrics (1988): 79-86.
They use the calculations based on local alternatives, but have comments about it's applicability.

related problem for GLM: the variance function also depends on the mean and will differ between null and alternative.

see papers by Shieh that take estimation properties under the alternative into account, e.g.
Shieh, Gwowen. "On power and sample size calculations for Wald tests in generalized linear models." Journal of statistical planning and inference 128.1 (2005): 43-59.
and related literature.

user: josef-pkt
issue_1756: state: open
title: ENH: add generic test for (conditional) moment restrictions
body: Another generic test class for M-estimators that define `score_obs` or `estimating equations` (first order conditions)

Cameron Trivedi Microeconometrics has a general derivation for m-test in section 8.2.3
one page that looks like it can be translated directly into code.

Wooldridge and Davidson/MacKinnon have more elaborate examples and description but focus a lot on how to calculate it through auxiliary regression. (CT also have sections on those but shorter, AFAICS)

user: josef-pkt
issue_1755: state: open
title: ENH: Quasi-Likelihood Ratio test for overdispersed GLM, discrete
body: If we have misspecification in the form of simple under/overdispersion, like in GLM or Poisson model with scale not equal to 1, then the Quasi-LR test can still be used to test competing nested models.

see end of section 7.5.1 in Cameron/Trivedi Microeconometrics, which reference 
Wooldridge, J. M. (2002), Econometric Analysis of Cross Section and Panel Data, p 370

for general QMLE the LR statistic is weighted sum of chi-squares distributed
Cameron Trivedi Microeconometrics section 8,5,3
(I have already written the distribution for weighted sum of chi-squares random variables, the problem is getting the case specific weights, which are eigenvalues of some matrix products)

user: josef-pkt
issue_1753: state: open
title: ENH: add generic score, LM test
body: looks like this is actually pretty easy to add this generically.

We need to be able to build the unrestricted model with access to score_obs and hessian evaluated at some params (already available in discrete), and we need to be able to estimate the restricted model.

linear restrictions can be estimated with fit_constrained and we get the restricted parameters in the unconstrained parameterization.

Cameron Trivedi Microeconometrics equ. 7.25 p. 235 for correctly specified likelihood
just the standard definition

`LM = - score_obs(params_constrained) dot invHessian(params_constrained) dot score_obs(params_constrained)  / nobs`

equ 7.29 and 7.38 (section 7.5.1) with sandwich form if likelihood is misspecified and for general M-estimators.

**edit** 7.38 looks "strange" to me, the term in the outer matrix inverse should be rank deficient. Maybe the outer inverse is a typo? Wooldridge Panel 2010 2nd ed. p. 423 equ. (12.68) doesn't have the outer inverse, and looks otherwise the same.

related: 
Both Cameron/Trivedi and Davidson/MacKinnon argue that the LM/score tests based on outerproduct of gradient or auxiliary regression have low power in small samples.

user: josef-pkt
issue_1752: state: open
title: ENH: MAIC for lag selection in unit root tests
body: It should be relatively easy to add new criteria for `autolag` lag length selection, since the basic code already exists.

Lag Length Selection and the Construction of Unit Root Tests with Good Size and Power
Serena Ng and Pierre Perron http://www.jstor.org/stable/2692266
they use a modified AIC

new paper in case of volatility
http://ideas.repec.org/p/cwl/cwldpp/1844.html

lecture notes for brief overview http://eurequa.univ-paris1.fr/membres/Ahamada/cours/nsss.pdf

related to question: Do we need `maxlag` argument for all unit root tests? PR #1499
it looks like the answer should be yes.

user: josef-pkt
issue_1750: state: open
title: GLM, discrete overdispersed cov_params (robust)
body: specific issue for adding a scale parameter to GLM and discrete models.

related 
#1095 QMLE
#1158 summary issue for robust covariances

IIRC, there are more comments and references spread over different issues

reference among several others

Cameron Trivedi Microeconometrics section 5.7.4 on Generalized Linear Models
explicitly says
common estimate for scale factor is sum of squared pearson residuals divided by (nobs - k_vars)   (i.e. divided by df_resid)

I got slightly strange numbers when I tried this a while ago, and find the equivalent result in Stata. (maybe because I looked at Stata's poisson and not glm)

Note: our GLM has a hidden, untested (?) option to choose the calculation for the scale parameter.

user: josef-pkt
issue_1749: state: open
title: ENH/REF: fit_constrained followup
body: see #1714 for things that were left open

---

another one:
- homogeneous constraints R beta = 0 don't need offset and could be used with any model, needs some `if` branching in `fit_constrained_wrap` to skip offset.
  -> try testcase for discrete_Logit

user: josef-pkt
issue_1748: state: open
title: Design: patsy stateful transform - replicability - multiple imputation
body: Patsy's stateful transforms can be or are data dependent.

main question: how can we get a comparable parameterization if we work with different datasets, e.g. subsets or different completed sets in multiple imputation.

Frank's current MICE version uses formulas to recreate the models during the imputation.
The Stata manual on multiple imputation mentions several checks and warnings when individual models create different parameterization, if some observations or some variables change. We don't have the same automatic behavior in our model estimation as Stata, but we might run into similar problems.

I think what we need are super-stateful transforms or transforms that have the state parameterized, i.e. fixed for the creation of the design matrix.

Example:
- categorical variables: If we have a changing set of level of a categorical.variable, then the columns, factor coding, will change. In this case we have problems anyway because some parameters are not identified across models.
- linear data transformation: for example standardization affects a single variable, but the transformation depends on mean and standard deviation of the given exog.
- polynomial for continuous variables: several possible versions
  see also https://github.com/pydata/patsy/issues/20
  - (x, x**2) is not stateful, but can be badly scaled, parameterized
  - np.polynomial.vander would be good if we rescale the variable, state is the linear transformation into the domain, for example  (0, 1)
  - orthogonalized (on the data) transform is a complicated stateful transform, that has vander followed by  QR

user: josef-pkt
issue_1747: state: open
title: cov_type parametric for GLS sandwich 
body: similar to robust covariance matrices, 

example Stata xtpcse issue #1207 

We use an estimator like OLS or GLSAR that doesn't take some correlation or heteroscedasticity into account, but calculate the standard errors from the correct formula under the assumption of a specific covariance structure:

`cov_params = (x'x)^(-1) x' sigma x (x'x)^(-1)`

where `sigma = E(u_i, u_j)`    for observations i and j with a specific structure
e.g. sigma = cross-sectional correlation in panel data.

user: josef-pkt
issue_1744: state: open
title: FAQ, Design: helpful summary, diagnostic hints
body: this is mainly to add a link 

discussion how to use ols and linear regression
http://forums.udacity.com/questions/100155297/how-much-different-do-you-get-r2-by-using-ordinary-least-squares#ud359

some points
- warning text at end of summary about multicollinearity or singularity is useful (dummy variable trap)
- rescaling to reduce multicollinearity, high condition number
- separate model building/development stage from production code: When selecting variables and building the model it is useful to work with a full statsmodels.OLS (or whatever other Model). Prediction or estimating only parameters (on cloudserver with short timeout) can be much more efficiently done with direct calculation.

TODO:
- We have an issue to provide more out of the box diagnostic, but I don't find the issue now.
- We are missing the function to identify collinear columns in exog, recent stackoverflow question. http://stackoverflow.com/questions/23848003/detecting-mulicollinear-or-columns-that-have-linear-combinations-while-modelli
- automatic dropping of collinear variables: wish was articulated before to do the same as Stata.  (issue ?)
  I don't like it to be a default, but we need better singularity options and easily accessible transforms.

user: josef-pkt
issue_1743: state: open
title: Design:  groups as observation, score_obs, and base_models
body: (unsolved mysteries - vague thoughts on general classes of models)

**background**

What's the most general statistical model?

I thought it's M-Estimators, but I recently read about the distinction between M-estimators and Extremum estimators. 
With M-estimators we have the contribution of an individual observation to the objective (e.g. loglike or loss) function, with general extremum estimators we don't have individual contributions.

GMM is an extremum estimator, but not an M-estimator except in special cases (?).
GEE is not even extremum estimator (if it's not in GMM form), we solve directly the estimating equations.
robust M-estimators, can also be defined with different estimating equations for loc and scale, so there is no obvious objective function for minimization. (I think in most cases they can still be found so that M-estimation theory can be used, even if the calculations are in terms of robust estimating equations.)

**Specific: score_obs in panel/longitudinal data**

If I want to define score_obs for general models (I would like to add it to LikelihoodModel), then what happens if our basic observation is a group of observations instead of individual observations.

E.g. Mixed models where we integrate for each cross-sectional unit over the unobserved heterogeneity, to get the marginalized likelihood contribution of a cross-sectional unit and not of one observation.

I think this user question had a similar structure
https://groups.google.com/d/msg/pystatsmodels/k6qV6eHM2hg/8z6GdziikngJ

How is the usage of score_obs for score/Lagrange Multiplier tests and for robust sandwich covariance matrices affected?

We don't need to worry about this now, but adding functionality based on score_obs means that some future models will have problems with the existing "pattern".

user: josef-pkt
issue_1742: state: open
title: add GMMPoisson (IVPoisson)
body: Stata has `ivpoisson` which has both the additive and multiplicative error model.
Besides GMM it uses a second method.

GMMPoisson is essentially finished, but it's in the test suite as a test case for GMM, and I have the notebook as usage example for GMM.
I don't remember if it has the full extra results. Unit tests are against generic GMM in Stata not against ivpoisson.

move it to discrete_models.

Stata also has ivprobit, which could follow the same pattern.
Stata doesn't have ivlogit, which, IIRC, would also be easy but doesn't have additive IV structure. I guess it would need to subclass GMM instead of IVGMM.

As bonus, with `instruments=exog` (no endog in explanatory variables), we would have sandwich robust covariances.

user: josef-pkt
issue_1741: state: open
title: Performance: drop constant terms in loglike during MLE optimization
body: I just read in the SAS manual that they drop the constant terms during optimization but report the full loglikelihood for the final results.

If we have a switch similar to what we use for parameter transformation, then we also could speed up the optimization.
many references show the constant, but until now we always construct the full loglike.

related: 
I haven't tried to replace loglike and score with sum(loglike_obs) and sum(score_obs) because I also worry that in many cases it would add repeated computation for the same constants or terms.

user: josef-pkt
issue_1740: state: open
title: Likelihood Ratio-Based Confidence Intervals, profile-likelihood
body: As Kerby pointed out on the mailing list, once we have constrained estimation, we can invert the LR test to obtain confidence intervals for individual parameters

for example for GLM
http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#statug_genmod_sect036.htm

user: josef-pkt
issue_1739: state: open
title: GLM ANOVA style analysis
body: just saw this by chance in the SAS documentation

http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#statug_genmod_sect035.htm

type 2 and type 3 ANOVA analysis for GLM based on LR tests, as a complement to Wald tests.

I also saw generalized score tests for GEE/GLM in the SAS documentation. (but don't find it anymore right now)

user: josef-pkt
issue_1737: state: open
title: GLM results statstics, diagnostics
body: see #929 for same for GEE

some information in Stata and SAS manuals, but I cannot match up those two.
They possibly use different definitions, e.g. approximation to the hat matrix.

some discussion in
https://github.com/statsmodels/statsmodels/pull/1734#issuecomment-45049596
and follow-up

Skipper added a reference to an influence approximation to an issue (but I don't remember which or where we discussed this.)

Both SAS and Stata have "standardized" residuals, divided by (1 - diag_hat_matrix).

user: josef-pkt
issue_1735: state: open
title: wishlist: generic acceleration of EM-like iterative algorithms
body: e.g. "2012 John M. Chambers award for best statistical software for our R package, turboEM, which provides a suite of convergence acceleration schemes for EM and MM algorithms (jointly with Jennifer Bobb)" which is GPL licensed.  Is there something like this for Python?  I code-searched "squarem" and "turboem" in statsmodels and scipy but I didn't find anything.  Aitken's method for fixed point is available in scipy.

user: argriffing
issue_1732: state: open
title: FAQ-D: recreating a model, keywords
body: How can we create a new model with maybe different data that uses the same keywords in the model creation as a given model?

for motivation see #1093 

usage

```
    init_kwds = model._get_init_kwds()
    mod_new = self.__class__(endog_new, exog_new, **init_kwds)
```

Warning `_get_init_kwds` returns a reference to the attributes attached to the model. If those are mutable, for example arrays like offset, exposure, weights, then changes to them will also change the underlying arrays of the given model which will cause problems or hard to find errors.

not sure if we need the copy if we want to modify the init_kwd arrays, TODO: more unit tests

```
    import copy
    init_kwds = copy.copy(self._get_init_kwds())
    del init_kwds['offset'] 
    mod_constr = self.__class__(endog, exogp_st, offset=offset, **init_kwds)
```

How does it work?

Each model needs to register the names of extra keyword arguments in the `__init__` of the model class in `_init_keys`.
The `base.Model` registers all keywords that have been handed to it in super calls. This happens for example with `weights` in WLS or `offset` and `exposure` in count models and GLM.

`_get_init_kwds` returns a dictionary with the model attributes as values. In some cases it needs to transform or recreate the original array, for example exposure is internally stored after taking the log.

As an example, GLM has to add the `family` to the `_init_keys`: we have in `GLM.__init__`:

```
# register kwds for __init__, offset and exposure are added by super
self._init_keys.append('family')
```

then we have

```
>>> mod1._init_keys
['exposure', 'offset', 'family', 'family']
>>> mod1._get_init_kwds()
{'offset': None, 'family': <statsmodels.genmod.families.family.Binomial object at 0x052E7B30>, 'exposure': None}
```

**Warning:** This is a new feature and has been used only in a few tested cases so far. 
No comprehensive test suite and not all models have registered their init_keys yet.

user: josef-pkt
issue_1731: state: open
title: trend not correct in VAR
body: The trend always starts at 1, instead of taking into account the pre-sample variables and starting at `k_ar + 1`. Our estimates do not match stata/gretl in this case for the constant.

user: jseabold
issue_1729: state: open
title: FAQ, FAQ-D: I want to store (pickle) my model for prediction
body: After we have estimated a model and obtained the results, we might want to pickle the results and model to use it later for prediction. To avoid storing all the data that was used in the estimation and in the generation of the results, we have added a `remove_data` method to the results instance, additional to `save` and `load` methods that do the pickling and unpickling.

`remove_data`  will (is supposed to) remove all arrays that have a length of nobs, and in some cases additional attributes. The resulting "data-less" results and model instance will not be able to calculate any new results that need those arrays. However, predict and most inference only needs the stored parameters and covariance matrix, and will still be functioning correctly.

Note: 

**FAQ-D: How does it work, and what do we need to do to support it?**

`remove_data` only removes attributes and attached arrays if they have been registered and does not try to determine automatically which attributes to delete. We have arrays or attributes in three main parts
- `model._data_attr` registers a list of names for attributes of the `model` and the `model.data`. The default are endog and exog and the corresponding references in `.data`
- `results.._data_attr` registers a list of names for attributes of the `model` instance
- `results.data_in_cache` registers cached attributes of the results instance, that are _stored_ in the cache and not directly as attributes.

Removing attributes or cache keys in `remove_data` is protected against non-existence and doesn't raise an error if a registered name or key cannot be found.

The main methods are in `base.model.Model`. The registered lists are added in the `__init__` of model and results classes.

Unit tests for one case of each group of classes and where new models can be added are in `/base/tests/test_shrink_pickle.py`

user: josef-pkt
issue_1727: state: open
title: confidence intervals through inverting tests of linear constraints - profile likelihood
body: Kerby on the mailing list:

Another application for linear constraints is constructing confidence sets by inverting tests.  For example, if you want a CI for the ratio of two regression coefficients b1/b2, you can test the null hypothesis b1/b2 = r (equivalent to constraining b1 - r*b2 = 0 and testing against a general alternative), and the confidence set consists of all r for which the test does not reject.  You can construct CI's for any rational function of the parameters this way (i.e. any quotient of two linear functions of the parameters).  We could provide a convenience function that would hide the details.  More exotic but also useful is 2d confidence sets for two rational functions of the parameters.

user: josef-pkt
issue_1726: state: open
title: LikelihoodModel versus GenericLikelihoodModel - move methods
body: Vincent brought this up once.

LikelihoodModel could get some of the methods from the GenericLikelihoodModel.
Some of those are good enough now.

Currently, I would like numerical derivatives for score_obs and Hessian in GLM which are not implemented explicitly yet.
Score and Hessian are not used in the estimation, but would be useful for post-estimation. Stata has `predict score` to get `score_obs`

user: josef-pkt
issue_1725: state: open
title: DOC: add a Glossary, define terminology
body: I'm thinking of a glossary of names that we use in statsmodels, not so much as a glossary for statistics terms (there are many on the web).

similar to the list of attributes in the docstring of the Results classes, and based on more consistent namings and definitions, see https://github.com/statsmodels/statsmodels/wiki/SMEP-D:-Naming-Convention and issues on naming convention, and naming convention in the docs.

This would be low maintenance, since we don't change names often, or at all.

sphinx rendering would allow for heavy use of cross-links with `see also`

user: josef-pkt
issue_1724: state: open
title: FAQ-D hasconst, k_constant, const_idx
body: How do I know if a model has a constant?

What do I need to adjust to handle both cases, when there is a constant and when there is no constant?

user: josef-pkt
issue_1723: state: open
title: FAQ-D   df_resid df_model in results and in model
body: (developer FAQ) (answer not settled yet)

Why are the df_xxx different in the results from those in the model?

Where do I set df_xxx?

user: josef-pkt
issue_1722: state: open
title: optimization retvals, fit history - missing, consistency
body: I'm trying to run the same code with discrete_model Poisson and GLM.
- discrete_models have `mle_retvals`, 
  summary shows `converged` boolean, 
  but we don't have `iterations` available, (we have f_calls from the optimizer)
- GLM has `fit_history`,  
  summary shows number of iterations but not converged boolean

the converged function in GLM doesn't distinguish stopping because of maxiter from early stopping because converged.

user: josef-pkt
issue_1721: state: open
title: constraint formula expression from constraint matrix
body: an idea for the summary with constrained estimation

we should be able to recreate a string expression from the constraint_matrix, constraint_values, so we can add it to the summary string.

user: josef-pkt
issue_1720: state: open
title: code review: GLM
body: - why do we calculate `pinv_wexog` in `GLM.initialize`?  It's not used AFAICS.
  `normalized_cov_params` is calculated with it, but that's also unused.
- fit "state" attached to the model: several fit results are attached to model (e.g. mu). It's part of the design for the iterative estimation. I don't see a problem, but it would be better to detach those from the model, once they are attached to the results instance.

naming
- family `predict` versus `fitted`: not clear which is which. ?  
  (difficult because statistics and econometrics differ, link versus inverse-link)

user: josef-pkt
issue_1719: state: open
title: GLM aic hasconst
body: it looks to me that GLMResults.aic assume hasconst = 1

```
    def aic(self):
        return -2 * self.llf + 2*(self.df_model+1)
```

has `hasconst = 0` cleanup ever been extended to GLM ?

user: josef-pkt
issue_1718: state: open
title: FAQ:  statsmodels for Stata users
body: What are some of the equivalent or almost equivalent functions in statsmodels compared to those in Stata, for some of the related estimation and post-estimation options.
## Post-Estimation
### estat
### margins
### Testing

Stata after estimation: test, lincom, lrtest, testnl,  suest/hausman? contrasts?
- `test` joint hypothesis f_test (RM), wald_test (RM), 
          individual hypothesis t_test (RM)
- `lincom` t_test  (RM)
- `lrtest` compare_lr_test (RM)
### Prediction
- `predict` predict (RM)
- cached attributes: fittedvalues, resid, ...

(RM) = method of results instance

user: josef-pkt
issue_1717: state: open
title: cov_params reduced rank - consequences
body: a followup to constrained estimation:
If constraints are imposed without changing the parameter space, then the cov_params has reduced rank.

If we use pinv that drops singular vectors, or we use PCR with reduced number of components, then we get cov_params with reduced rank.

any consequences of this?

e.g. compare_lr_test and compare_f_test uses the difference in df_resid which should still be correctly set. 

**Bigger conceptual question:**

What's the meaning of the wald test in the constrained model?

We are imposing the constraint, so cov_params includes the constraints. The only thing that we can meaningfully test is in the direction where we don't have the constraints, for example on unconstrained parameters.

To test the constraints that are already imposed we cannot use the cov_params and wald_test, but we can use the score/lagrange multiplier test. (issues #2041)

I'm not sure this is clear when we have a full model in hand that already has imposed some constraints, params, cov_params and so on are for the full parameter space but the values are in a subspace.

user: josef-pkt
issue_1716: state: open
title: REF: update estmat2nparray [ci skip]
body: this is the Stata file I'm currently using to dump results

user: josef-pkt
labels: superseded: Used for PRs with content that is superseded by a newer PR
issue_1715: state: open
title: ENH: model transformation and constant, and penalization
body: I'm again just running into convergence failures that are only on TravisCI.
This time with Poisson.fit_constrained.

I never manage to solve the convergence failure for one testcase for NegativeBinomial.fit_regularized on TravisCI.

If we do a transformation of the data to make it behave nicer, then we run into two related problems.

Penalized/Regularized estimators are not invariant to linear transformation. Penalization is on one parameter space, and with any transformation we need to also transform the penalization.

If we use a transformation that affects all columns, we don't know which is the constant. 

user: josef-pkt
issue_1712: state: open
title: ENH: summary doesn't show offset and exposure
body: trying out different Poisson regression (with and without exposure/offset), I cannot see from the summary() whether offset or exposure has been used.

user: josef-pkt
issue_1710: state: open
title: TST: streamlining some tests
body: Redundant checking of the same thing several times.

example I just saw:

A test that tests one method, but does the estimation from the default start_params, instead of short-cutting the estimation.

```
def test_poisson_predict():
    #GH: 175, make sure poisson predict works without offset and exposure
    data = sm.datasets.randhie.load()
    exog = sm.add_constant(data.exog, prepend=True)
    res = sm.Poisson(data.endog, exog).fit(method='newton', disp=0)
```

I think we have many **small** inefficiencies in the test suite, but it starts to add up.

"Unit tests are code" and will eventually need some maintenance.

related: similar change in tsa where estimation sometimes didn't converge well enough for testing a specific method, see #1706 

user: josef-pkt
issue_1707: state: open
title: rename base fit to _fit ?
body: Currently just an idea for refactoring:

If we rename the basic  `LikelihoodModel.fit` to `_fit`, then subclasses could call the optimization directly instead of going through the `super` calls.

In discrete_models the fit and fit_regularized methods skip classes in super calls see #1617 

Note after outsourcing of the optimization code it's easier to call the optimizers, but I guess it would still be useful to get the full LikelihoodModel fit features without extra code.

user: josef-pkt
issue_1705: state: open
title: FAQ corner case: perfect predictability, perfect separation
body: Since I was just reading the section in the Stata manual for Probit

A question that was just asked on the LinkedIn Econometrics group

```
Logistic Regression Drops Variable of Interest

Hello. I am running a logistic regression, but one of the variables of interest has no observations 
in one of the categories of the dependent variable. So, basically, for x=1, y is always 1. For x=0, 
y can be both 0 or 1. When i run this model, logistic regression essentially omits this variable 
because of perfect predictability. But this is a variable of interest. A friend of mine suggested 
that if i introduce a random error term into x, then we can preserve the coefficient by "tricking" 
stata into estimating this coefficient. So, basically, instead of x, i now have the regression with 
a new variable, x1 = x + random normal with mean zero, and std div small but greater than zero. 
This does indeed give me a coefficient! The question I have for this group is whether this is a 
fairly well known technique or not. I searched Google but did not find any discussions about it, 
so hoping that you can help me.
```

I think I don't like the automatic "cleaning" of the data by Stata. But we should be able to provide tools to get around corner cases like perfect separation. (After users got the exception.)

user: josef-pkt
issue_1704: state: open
title: t_test `can't test a constant constraint`
body: I'm trying to use t_test to get the covariance for a degenerate linear transformation, but patsy complains.

question: Should this be allowed?

Note: for a while now I have been using `t_test` to get the results for linear transformations and not just for "t-tests"

```
    tt = res3.t_test(transf3.transf_mat, transf3.constant.squeeze())
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new2_py27\statsmodels\statsmodels\base\model.py", line 1050, in t_test
    LC = DesignInfo(self.model.exog_names).linear_constraint(r_matrix)
  File "build\bdist.win32\egg\patsy\design_info.py", line 220, in linear_constraint
  File "build\bdist.win32\egg\patsy\constraint.py", line 392, in linear_constraint
  File "build\bdist.win32\egg\patsy\constraint.py", line 58, in __init__
ValueError: can't test a constant constraint

>>> transf3.transf_mat
array([[ 0.,  1.,  0.],
       [ 0.,  0.,  1.],
       [ 1.,  0.,  0.],
       [-0., -0., -0.],
       [-0., -0., -0.]])
>>> transf3.constant.squeeze()
array([ 0.,  0.,  0.,  0.,  1.])
>>> res3.model.exog.shape
(20, 3)
```

example is for constraint estimation, the covariance matrix in the constraint full model (parameterized for original exog) is degenerate or rank reduced.

```
>>> cov_params3 = transf3.transf_mat.dot(res3.cov_params()).dot(transf3.transf_mat.T)
>>> np.sqrt(np.diag(cov_params3))
array([ 0.59285072,  0.31334777,  0.30669526,  0.        ,  0.        ])

>>> print res3_ols.bse
[ 0.59285072  0.31334777  0.30669526]
```

(example based on "dirty" development script for estimation with linear constraints)

user: josef-pkt
issue_1702: state: open
title: Not FAQ (but would be a good one): resources for learning stats with statsmodels
body: From ML: "What are some good tutorials or resources (besides the documentation) to learn statistics and econometrics through the Python statistics model (statsmodels)?"

user: jseabold
issue_1693: state: open
title: ENH: compare_lr_test for subset/partitioned regression
body: lrtest in Stata allows to compare a model with a list of models that are a partition of the sample space.

```
. logit vote age moinc dependents
    . estimates store All
    . logit vote age moinc dependents if county==1
    . estimates store A1
    . logit vote age moinc dependents if county==2
    . estimates store A2
    . logit vote age moinc dependents if county==3
    . estimates store A3
    . lrtest (All) (A1 A2 A3), df(7)

. logit vote age moinc dependents c.age#i.county c.moinc#i.county c.dependents#i.county
. estimates store A5
. lrtest (All) (A5)
```

user: josef-pkt
issue_1687: state: open
title: Semilinear updates
body: This is an update of the PR #1652, with commits simplified to it is easy to see what has been changed.

It fixes some small bugs in the semi-parametric regression and adds unit tests and with data sets for them. See previous closed PR for details.

user: Padarn
issue_1685: state: open
title: Naming: GLM    replace eta and mu or alias
body: I definitely **hate** `eta` and `mu` (and almost all greek letters).
I didn't look at this in 2 weeks and won't remember. :(

```
    @cache_readonly
    def fittedvalues(self):
        return self.mu
```

What's this,  the linear or the non-linear prediction ?

not even completely sure after reading the docstring for mu

user: josef-pkt
issue_1680: state: open
title: ENH grid of scatterplots
body: see last part of question http://stackoverflow.com/questions/23784399/how-to-visualize-a-nonlinear-relationship-in-a-scatter-plot

Where do we have a plot like this?
I know I made one already once, with lowess and ellipsoids but only lower triangual part.

without lowess 
https://picasaweb.google.com/106983885143680349926/Joepy?noredirect=1#5663446443722234690
https://picasaweb.google.com/106983885143680349926/Joepy?noredirect=1#5683071363613315698
make removal of space between plots in grid optional

user: josef-pkt
issue_1678: state: open
title: Updates to univariate kde - all kernels and weights
body: This is an 'almost finished' PR which updates the fft binning univariate kde. I have changed the routine based on the algorithm given in

`Wand, M. P. and Jones, M. C. (1995). Kernel Smoothing.`

It now works for any kernel type (by computing the kernel fft explicitly), and for weighted samples. I have also written some unit tests for this (compares fft and non-fft versions for the same example).

In the case of the Gaussian kernel, the code no longer uses the 'exact' fft of the kernel, but the results are very close, and the code still passes all the unit tests. 

A few other small changes:
- Now the kernels in `/sandbox/nonpaprametric/kernels.py` return `0` instead of `nan` when out of sample.
- Some documentation has been cleaned up.

I did this because I was thinking about implementing the 2D version, and thought I may as well have a go at cleaning this up first.

EDIT: Still want to compare a few extra things between this implementation and the previous one - worth making sure it is isn't slower for example.

user: Padarn
issue_1675: state: open
title: TST: SVAR  missing unit tests
body: SVAR only has unit tests for A and B

The included Stata .do file, works, but doesn't have the options that replicate the statsmodels version. To replicate with the small sample corrections and `use_t` is

`svar gdp cons inv, aeq(A) beq(B) lags(1/3) var dfk small`

results look like high agreement with visual comparison

---

related issues
#701
#671

:

user: josef-pkt
issue_1673: state: open
title: ENH: Restricted gls
body: WIP moving RLS out of the sandbox

added minimal tests
no refactoring yet

**Performance:** 
uses GLS: builds full (nobs, nobs) sigma and cholsigminv

**Names**
k_constr, ... k_xxx instead of `nconstraint`
name of class: RestrictedGLS
rwexog, ... are misnomers because they are moment matrices for moment equations

open: names for constraints R, q

**design**
- what do we want to cache in the model?   reusable model for residual bootstrap?
- `cov_type` default, move sandwich `wrnorm_cov_params` to results (reuse inherited code)
  (maybe: need to cache/attach `pinvX` for HCxx_se)
- from_formula for constraints
- other lazy attributes of model, e.g. diag of hatmatrix

user: josef-pkt
issue_1672: state: open
title: FAQ: My design matrix is singular, can I still use the results?
body: by estimator.
If it doesn't raise an exception, then not all parameters are identified

OLS
Using pinv we get a generalised inverse solution. Not all linear combinations are estimable, reference Matthew?
Fittedvalues and prediction are unique, independent of how the parameters have been chosen from the set of possible solutions.

RestrictedLS

....

user: josef-pkt
issue_1670: state: open
title: formula in predict: robustness to misspelling
body: (Why is "Intercept" capitalized and not "const"?)

what I wanted:

```
>>> res.predict({'Intercept':np.zeros(4), 'time_day':np.arange(1,5), 'country':['a']*4}, transform=True) - res.params[0]
array([  0.00000000e+00,   3.33333333e-01,   3.33066907e-16,
        -3.33333333e-01])
```

What I did:

```
>>> res.params
Intercept             7.500000e-01
C(country)[T.b]      -6.661338e-16
C(country)[T.c]      -2.500000e-01
bs(time_day, 3)[0]    8.888889e-01
bs(time_day, 3)[1]   -2.222222e-01
bs(time_day, 3)[2]   -3.333333e-01
dtype: float64

>>> res.predict({'intercept':np.ones(4), 'time_day':np.arange(1,5), 'country':['a']*4}, transform=True)
array([ 0.75      ,  1.08333333,  0.75      ,  0.41666667])
>>> res.predict({'intercept':np.zeros(4), 'time_day':np.arange(1,5), 'country':['a']*4}, transform=True)
array([ 0.75      ,  1.08333333,  0.75      ,  0.41666667])

>>> res.predict({'intercept':np.zeros(4), 'time_day':np.arange(1,5), 'country':['a']*4}, transform=True) - res.params[0]
array([  0.00000000e+00,   3.33333333e-01,   3.33066907e-16,
        -3.33333333e-01])
```

no warning that I misspelled intercept.

user: josef-pkt
issue_1669: state: open
title: ENH: variance inflation factor
body: variance inflation factor in outlier_influence uses the definition to calculate vif for one parameter instead of using linalg to calculate all of them at once

see my answer and example script which includes vif for Ridge Regression
http://stackoverflow.com/questions/23660120/variance-inflation-factor-in-ridge-regression-in-python/23665644#23665644

user: josef-pkt
issue_1668: state: open
title: ENH collect tools for linear restrictions, constraints, contrasts 
body: so far they are mainly in t_test and f_test plus the restricted estimation in GEE.

extra: naming convention

r_matrix, q_matrix is not really good   (although I always remember it), used also in TheilGLS
RLS #1667 uses `constr`and `param`

user: josef-pkt
issue_1667: state: open
title: ENH: Restricted Least Squares
body: We still have RLS in the sandbox, which still works without problems.

Needs unit tests and cleanup to fit better into the class hierarchy.
Robust cov_params are attached to model and HCxx_se raises exception (same as my TheilGLS)

user: josef-pkt
issue_1664: state: open
title: consistent definitions of nobs, n_xxx, k_xxx
body: we need more consistent definitions and usages of the descriptive attributes, as we add more models and more general cases or functionality.

example:
I'm still using code like
`nobs, k_exog = self.model.exog.shape`  (or more often k_vars instead of k_exog)
which describe the data, but are not always what we really need.

In the OLS case we have (assuming non-singular exog):
`k_params = df_model + k_constant = k_exog = self.model.exog.shape[1] = rank(exog)`
However, none of these equalities need to hold exept `k_exog = self.model.exog.shape[1]`
Moreover, df_model is used in several places that don't use necessarily the same definition. (example where I had the problems is small sample correction to df for cluster robust standard errors.)

(related: Stata defines some of these, like df, in terms of rank(cov_params).)

Similarly, once we allow for case weights, then effective nobs is not equal to exog.shape[0]. (related: Stata does the weight rescaling so nobs = sum(weights).)
I guess we have this already in time series models that condition on the initial observations.

Which attributes to we have guaranteed to be available in all models?

See Also:
for consistency of names
http://statsmodels.sourceforge.net/devel/dev/naming_conventions.html#variable-names
for getting a start on consistent definitions and names across models
https://github.com/statsmodels/statsmodels/wiki/SMEP-D:-Naming-Convention

user: josef-pkt
issue_1663: state: open
title: FAQ: Performance tips
body: ### formula convenience has a cost
- If you repeatedly need to use the same transformation or the same formula generated matrix of explanatory variables, then keep them around instead of requiring patsy to repeat the same transformation multiple times.
  
  CON: `predict` doesn't know how to do the transformation and you have to use transformed variables.
  
  links
  http://nbviewer.ipython.org/gist/jiffyclub/cd8e26542758e4e20436 (note, the formula timing includes the exp and log transformation while the OLS timing does not.)
  http://nbviewer.ipython.org/gist/fscottfoti/1cec32e70a60e440ebce

user: josef-pkt
issue_1662: state: open
title: pinv rcond and multicollinearity, different cutoffs 
body: I spend some time debugging why my Ridge/TheilGLS cannot replicate OLS.

The example uses Longley data following an example in R MASS lm.ridge.
condition number is bad

If I solve the moment equation with pinv, I get a "regularized" solution. If we use pinv/svd on the original data (as does OLS), then we get an unregularized solution.
`rcond` kicks in with `pinv(x.T.dot(x))`, but not with `pinv(x)`

`lm` in R gives the same unregularized solution as statsmodels OLS

aside: I got this because I use TheilGLS and not a more efficient direct implementation of Ridge with SVD

```
>>> np.linalg.pinv(x.T.dot(x)).dot(x.T.dot(y))
array([  9.36081012e-05,   2.17314250e-01,   2.13408612e-02,
         4.33978774e-03,  -1.70034590e+00,   1.17385704e-01,
        -3.10581300e-01])
```

```
>>> np.linalg.pinv(x).dot(y)
array([  2.94685636e+03,   2.63527247e-01,   3.64829137e-02,
         1.11610505e-02,  -1.73702984e+00,  -1.41879853e+00,
         2.31287851e-01])
>>> np.linalg.solve(x.T.dot(x), x.T.dot(y))
array([  2.94685632e+03,   2.63527246e-01,   3.64829135e-02,
         1.11610504e-02,  -1.73702984e+00,  -1.41879851e+00,
         2.31287844e-01])
>>> np.linalg.inv(x.T.dot(x)).dot(x.T.dot(y))
array([  2.94685632e+03,   2.63527246e-01,   3.64829135e-02,
         1.11610504e-02,  -1.73702984e+00,  -1.41879851e+00,
         2.31287845e-01])
```

```
>>> np.linalg.pinv(x.T.dot(x), rcond=0).dot(x.T.dot(y))
array([  2.94666457e+03,   2.63524239e-01,   3.64819282e-02,
         1.11606065e-02,  -1.73702745e+00,  -1.41869855e+00,
         2.31252583e-01])
```

part of the script that defines the model:

```
import numpy as np
import pandas

from statsmodels.regression.linear_model import OLS
from statsmodels.datasets import longley

ll = longley.load_pandas()
names = 'TOTEMP  GNPDEFL     GNP  UNEMP  ARMED     POP  YEAR'.split()
s = [1000, 1, 1000, 10, 10, 1000, 1]
rescale = pandas.Series(data=s, index=names)
data = ll.data / rescale

formula = 'GNPDEFL ~'+ ' + '.join(['GNP', 'UNEMP', 'ARMED', 'POP', 'YEAR', 
                                   'TOTEMP'])
res_ll_ols = OLS.from_formula(formula, data).fit()

x = res_ll_ols.model.exog
y = res_ll_ols.model.endog
```

numerical precision problems, but the main point 
`rcond` depends on whether we use `x` or `x.T x`  
(square root of eigenvalues versus eigenvalues)

the singular values (square root of eigenvalues of `x.T x` ) don't look so bad

```
>>> np.linalg.svd(x)[1]
array([  8.15814890e+03,   4.56186582e+02,   3.24418001e+02,
         1.34253673e+02,   2.13324009e+00,   1.36760693e+00,
         2.11512471e-04])
```

condition number in terms of eigenvalues are worse but smallest eigenvalue is not too bad.

```
>>> np.linalg.svd(x)[1]**2
array([  6.65553934e+07,   2.08106198e+05,   1.05247039e+05,
         1.80240487e+04,   4.55071327e+00,   1.87034873e+00,
         4.47375255e-08])


>>> np.linalg.eigvalsh(x.T.dot(x))[::-1]
array([  6.65553934e+07,   2.08106198e+05,   1.05247039e+05,
         1.80240487e+04,   4.55071327e+00,   1.87034873e+00,
         4.47353505e-08])
>>> np.linalg.svd(x.T.dot(x))[1]
array([  6.65553934e+07,   2.08106198e+05,   1.05247039e+05,
         1.80240487e+04,   4.55071327e+00,   1.87034873e+00,
         4.47404782e-08])
>>> sv = np.linalg.svd(x)[1]
>>> sv.max()/sv.min()
38570533.694658495
>>> (sv.max()/sv.min())**2
1487686069490786.2
>>> np.linalg.cond(x)
38570533.694658607
>>> np.linalg.cond(x.T.dot(x))
1487587887911603.5
```

user: josef-pkt
issue_1661: state: open
title: FAQ: How is statsmodels better than R
body: (a place to complain when I'm getting frustrated with R)
### statsmodels is largely more "discoverable"

example: I estimated this model, but what post-estimation functions are defined for it
`dir(res)` or

```
> predict(res)
Error in UseMethod("predict") : 
  no applicable method for 'predict' applied to an object of class "ridgelm"
```

oops: I inherited this method, but maybe it doesn't apply for this case.

user: josef-pkt
issue_1660: state: open
title: DOC: advertise t_test for linear transformation
body: FAQ: How can I get standardized beta coefficients?

FAQ: I want to linearly transform my exog for the estimation for numerical reasons, but I'm not really interested in the transformed `params`. I would like to get the params table in my original scale.

answer: use t_test

example: scale exog to have variance = 1

motivation: 
penalization is not affine/linear equivariant (?), it depends on scale. (It would be scale invariant if we choose the penalization proportional to scale.)
some optimizers are not scale invariant

Can we make the params table swappable in the summary()?

user: josef-pkt
issue_1659: state: open
title: cov_type - overwriting in subclasses
body: just a note about possible implementation 

overwrite method that chooses and dispatches for robust cov calculation in subclass
check for cov_type in specific set
if not, then delegate to super

not sure yet how to template the docstrings

examples: 
Poisson: simple over (or under ?) dispersion
OLS: fix scale  (opposite of Poisson)
Ridge: sandwich or plain/`simplified` (choice might be more difficult to implement in combination with robust)

user: josef-pkt
issue_1653: state: open
title: Add (pred_table) to appropriate GLM results
body: Confusion matrices have been requested also for GLM. Currently now only for Binomial discrete choice model results.

user: jseabold
issue_1649: state: open
title: FAQ: examples with optimization failures
body: as followup to recommendation for optimizers, we should collect here some examples

not just failures but also difficult cases

user: josef-pkt
issue_1646: state: open
title: ENH covariance for one-way groups
body: what's the best way to get one-way covariance for example in unbalanced panel?

After reshape and filling with nans we could use masked arrays or pandas.

data:   x = 2d data,    groups is 1d integer group indicator

similar, possibly different shapes 
in the panel hac versions
I also have written an extract all pairs version somewhere (for AR1 ?)

similar use case as in gee and mixed without pattern
trying to figure out random coefficient similar to Stata's xtrc  
(GLS instead of MLE or REML in Mixed )
SUR is another case, but there we assume balanced panel

user: josef-pkt
issue_1645: state: open
title: IC in ARIMA
body: Parking this here. Right now, AFAIK we agree with most major statistical packages out of convention. This may or may not make theoretical sense. Worth exploring perhaps at some point.

We had some discussions about this in #324, #1262, and #1370. I just came across the 'tsa' package in R, which has an explicit note in their `arimax` function. "In the computation of the AIC, the number of parameters excludes the noise variance."

Tangentially related #1224.

user: jseabold
issue_1641: state: open
title: FAQ: choosing optimizer
body: **Note update**: 
statsmodels supports now all scipy optimizers through the scipy.optimize minimize function. The list below has not been updated. Several scipy optimizers (especially bfgs) have changed behavior across scipy versions, and new optimizer's have been added to scipy.

**Which optimizer should I choose, especially when I run into problems?**

some notes on the optimizers that we can choose for non-linear optimization in models
see also #1649 "FAQ: examples with optimization failures"
related: parameter transformation to make the optimization problem better behaved, see #1131 

our current defaults that work well in cases that are not too badly behaved:
- discrete:  `newton` for fit, and `slsqp` for fit_regularized
- generic MLE: `bfgs`
- models with their own IRLS or other iterative solvers: GLM, RLM, GEE, ...

With the exception of basinhopping, all optimizers are local optimizers and might converge only to a local optimum.
## Newton

`"newton"` is a simple solver implemented in statsmodels. It includes now a ridge factor to reduce problems with near singular hessian.
works well for discrete models, few iterations, uses Hessian
problems: breaks if Hessian is not positive definite. Doesn't have line search and might break if gradient is very large at the starting values.
## Bfgs

works well and fast if we are close enough to the minimum. Finds an optimum with very small gradient.
problems: can diverge or fail to converge if gradient is large, or shape of function is not "nice" during the optimization.
## Nelder-Mead, `nm`

the most robust optimizer in scipy. converges even from not very good starting values
Problems: takes a large number of iterations. Can get stuck in almost local minima or stops just short of minimum. Doesn't use gradient for convergence check.
## slsqp

used in fit_regularized, constraint solver that works well in quadratic problems.
Problems: possibly platform issues
## Iteratively reweights least squares, IRLS

hardcoded optimizer in GLM and RLM.
works usually well and converges in few iterations.
Problem: possible convergence problem for some starting values. However, we have seen it so far only in "cooked" examples.
## lbfgs

fmin_l_bfgs_b without constraints has only recently been added.
Seems to converge fast and more robustly than bfgs.
my impression from some examples: doesn't bring the gradient as close to zero as bfgs (which might be important when we use the score for hypothesis testing)
## Others

**fmin_cg**: not used much
possible platform and version problems, can require a considerable number of iterations. 

**powell**: used in  `emplike` as alternative to `nm`
supposed to be robust, but I haven't seen much improvement compared to `nm` and seems to be less robust than `nm`. (based on a very limited set of examples)
## Global

basinhopping
currently the only global optimizer, recently connected, requires recent scipy.
We have little experience with it so far. It seems to be working very well.
## Rootfinder

used only for one-dimensional search AFAIK

fsolve: can be pretty nonrobust, maybe problems in emplike
brentq: works well but requires predefined bounds
expanding brentq: initial expanding search for bound and then brentq
    Seems to work very well, "home made" specific algorithm in power calculation and generic ppf calculation in scipy.stats

I don't have any or much experience with any of the others.
## Unknown

not enough information
for example Kerby's new models

user: josef-pkt
issue_1640: state: open
title: ENH: add offset to all non-linear models
body: offset makes sense in all non-linear models, since, in contrast to linear models, we cannot subtract an offset from endog.

usecase: beside obvious offset, use for fixing some linear coefficients

user: josef-pkt
issue_1632: state: open
title: visualizing marginal effects
body: I just saw a nice example in a paper that visualizes marginal effects and uncertainty over the ranges of the independent variables. Would be a nice, easy visualization addition to the marginal effects class I think. 

user: jseabold
issue_1631: state: open
title: underdispersion NegativeBinomial boundary solution
body: What happens if we estimate a NegativeBinomial and we have underdispersion relative to Poisson?

The best estimate should be the edge case alpha=0 which corresponds to Poisson.
But ln(alpha) would go to -inf, and the untransformed version might break with alpha < 0.

(I didn't make up an example yet.)

user: josef-pkt
issue_1627: state: open
title: subset is applied before data transformations rather than after
body: I would expect that if I have transformations in my data, they would be applied before the subsetting is done. This does not appear to be the case. For example, in the case below, I would expect the number of observations to be 4 and not 3:

```
In [216]:
example = pd.DataFrame({'x':[1,2,3,4,5], 'y':[1,2,3,4,5]})
```

```
In [218]:
model = ols("""
    y ~ x.shift()
""", data=example, subset=example.index[1:])

fit = model.fit()

print fit.summary()
```

```

                            OLS Regression Results                            
==============================================================================
Dep. Variable:                      y   R-squared:                       1.000
Model:                            OLS   Adj. R-squared:                  1.000
Method:                 Least Squares   F-statistic:                 5.071e+30
Date:                Wed, 23 Apr 2014   Prob (F-statistic):           2.83e-16
Time:                        15:23:08   Log-Likelihood:                 102.40
No. Observations:                   3   AIC:                            -200.8
Df Residuals:                       1   BIC:                            -202.6
Df Model:                           1                                         
==============================================================================
                 coef    std err          t      P>|t|      [95.0% Conf. Int.]
------------------------------------------------------------------------------
Intercept      1.0000   1.38e-15   7.24e+14      0.000         1.000     1.000
x.shift()      1.0000   4.44e-16   2.25e+15      0.000         1.000     1.000
============================================================
```

user: cancan101
issue_1626: state: open
title: composite models and compare_lr LRtest
body: http://www.stata.com/manuals13/rlrtest.pdf

Stata's lrtest allows composite models for example when different segments of the observations are estimated separately and compared with the pooled model.

(old similar version based of F based Chow-test in sandbox regression onewaygls.)

using composite models implies that also auxiliary parameters (scale) differ across models. (example Logistic doesn't have any.)

user: josef-pkt
issue_1625: state: open
title: TST:  HAC robust covariances for GLM
body: I just saw that Stata provides `vce(hac)`  for GLM. It doesn't for explicit MLE models like nbreg.

useful for writing unit tests, currently only `cluster` robust covariances have unit tests for Poisson.

user: josef-pkt
issue_1624: state: open
title: NegativeBinomial df_model df_resid, extra parameter
body: I run into a problem that df_model and df_resid don't count the alpha shape parameter in NegativeBinomial, but the generic fit_regularized does. #1623 

(fit_regularized uses number of non-zero params, generic Discrete uses rank(exog))

df_resid is irrelevant for the normal based asymptotic cov_params and bse
the only place I have seen it so far is in `aic` and in `bic` and NegativeBinomial overwrites both to add the shape parameter back in.

```
    @cache_readonly
    def aic(self):
        # + 1 because we estimate alpha
        return -2*(self.llf - (self.df_model + self.k_constant + 1))

    @cache_readonly
    def bic(self):
        # + 1 because we estimate alpha
        return -2*self.llf + np.log(self.nobs)*(self.df_model +
                                                self.k_constant + 1)
```

but it's added in also for `loglike_method = 'geometric'` which doesn't have the extra params.

I think df_model and df_resid should be changed to take the extra shape parameters into account, since they affect the mean estimation.

(based on my reading of the test files Stata defines df_m based on exog without constant, but uses all params in aic, auxiliary parameters are stored separately `e(k aux)`)

extra:
DiscreteResults aic and bic do not take into account that there might not be a constant in the model, but it's always subtracted 
`self.df_model = float(np_matrix_rank(self.exog) - 1)`
so this cancels

user: josef-pkt
issue_1622: state: open
title: formula convention for multi equation models
body: (while briefly looking at conditional and nested logit again)

do we want lists instead of a piped merged string?
`formula = [list of formulas]`

```
formula = ['y1 ~ x1 + x3',
                 'y2 ~ x1 + x2 + x5']
```

user: josef-pkt
issue_1621: state: open
title: offset in formula
body: Looking at #1486 I see that in R you can specify an offset in a formula

```
y ~ x1 + x2 + offset(x3)
```

Looks useful...

user: jseabold
issue_1620: state: open
title: FAQ: does statsmodels provide any code for bayesian estimation?
body: To add to FAQ. Statsmodels seems to be becoming a popular tag for pymc/bayes questions on stackoverflow.

user: jseabold
issue_1618: state: open
title: Cancer dataset license?
body: @j-grana6 Can you clear this up? Is this data from US gov't agency? Having `???` for copyright isn't ideal.

user: jseabold
issue_1617: state: open
title: design: super skipping classes fit_regularized
body: just an observation to comment on the current code:

Poisson.fit_regularized calls super but skips CountModel.fit_regularized
`cntfit = super(CountModel, self).fit_regularized(`

trying to use the own class in super results in a missing `mle_retvals`
in NegativeBinomial
`cntfit = super(NegativeBinomial, self).fit_regularized(` raises exception in `summary` because of missing `mle_retvals`
doing the same as in Poisson works: `cntfit = super(CountModel, self).fit_regularized(`

using the inherited `CountModel.fit_regularized` also worked except for start_params
see https://github.com/statsmodels/statsmodels/issues/1453#issuecomment-37097078

user: josef-pkt
issue_1609: state: open
title: refactor exposure in count models
body: Problem with exposure: we don't store "exposure" we store log(exposure)

I think that's not a good design, storing a different variable under the original name

found while working on #1221,  #1093 and #1608

user: josef-pkt
issue_1605: state: open
title: Dcm logit rebased
body: rebased version of PR #1120

last commits 2014 fixed py3.3 compatibility

user: josef-pkt
labels: superseded: Used for PRs with content that is superseded by a newer PR
issue_1604: state: open
title: IRLS robustness in GLM
body: Duplicating the failing example from #443.

```
import numpy as np
from numpy.testing import assert_almost_equal, assert_equal

import statsmodels.api as sm

y2 = np.array('0 1 0 0 0 1'.split(), int)
wt = np.array([50,1,50,1,5,10])
y2 = np.repeat(y2, wt)
x2 = np.repeat([0,0,0.001,100,-1,-1], wt)

sm.GLM(y2, sm.add_constant(x2), family=sm.families.Binomial()).fit()

sm.GLM(y2, sm.add_constant(x2), family=sm.families.Binomial()).fit(start_params=[-4, -5])
```

The former fails silently. The latter takes log(x) with x < 0.

user: jseabold
issue_1597: state: open
title: folder structure - location of models
body: Where are new models supposed to go?

what is the type of category that determines subdirectories?
### Panel, Longitudinal

location for mixed ?:
is in regression in the PR 
the logical choice would be in `panel` which will be created by the panel PR, but the panel PR also spreads the models across directories, panel, regression and discrete_model

GEE is in genmod

what's supposed to be in regression besides `linear_model`?  and non-linear least squares?

Stata has all panel structure econometrics with `xt` prefixed names and mixed effects with `me` (except for `mixed` itself).
### Other

GMM is currently in sandbox.regression, but I'd like to move it to `statsmodels.base`

Where does panel-tsa or tsa-panel go?  
(maybe split by tools, "forecasting" instead of "predict")

user: josef-pkt
issue_1595: state: open
title: model comparison notebook
body: Do we have a notebook that does model comparison? If not, might be nice to cook one up showing how to use the methods provided by, say, the linear regression models.

user: jseabold
issue_1594: state: open
title: Should `scale` and `normalized_cov_params` be removed from `LikelihoodModelResults`?
body: Most likelihood models do not naturally have `scale` or `normalized_cov_params` concepts, and these are only applicable to a subset of additive error models.

Some of the applications of these simply **fake** the result and set `normalized_cov_params` to the actual covariance of the parameters and `scale=1` - it seems to have these as attributes is then misleading.

It would be more natural for the model inheriting from `LikelihoodModelResults` to provide a covariance estimator if required.

These are only actually used in the following lines,

``` python
 if cov_p is None:
            if scale is None:
                scale = self.scale
            if hasattr(self, 'cov_params_default'):
                cov_p = self.cov_params_default
            else:
                cov_p = self.normalized_cov_params * scale
```

but then derived classes have the attributes `scale` and `normalized_cov_params`.  

This also appears to have the side effect that `cov_params` has to be a function rather than just an attribute property, so that the somewhat awkward situation arises where the unfortunate mixed syntax is required

``` python
>>> res.params
>>> res.bse
>>> res.tvalues
>>> res.cov_params()
```

I think the correct behavior is to have the subclass explicitly pass the parameter covariance so that 

``` python
def __init__(self, model, params, normalized_cov_params=None, scale=1.):
```

would become

```
def __init__(self, model, params, cov_params=None):
```

and then the magic currently in `cov_params()` will not be required.

The other use of `cov_params` can be moved to a either a private function or a function that is more appropriately named such as `cov_linear_restriction` which is what the computational side of `cov_params` appears to compute.

user: bashtage
issue_1591: state: open
title: tools: check R and Stata export helpers
body: Vincent has converted it to package PR ?
I have changes in my local copy for Stata, and maybe also for R

user: josef-pkt
issue_1590: state: open
title: predict functions as standalone functions
body: usecase similar to the case we discussed when we introduced clear data and pickle.  estimate once, predict often (without model)

most current models outside of tsa would just need the patsy processing for data conversion and a linear or non-linear E(y|x) function

optional confidence intervals and standard errors if we have the cov_params.

can we pickle a closure, nested function created in a Resultsclass?

user: josef-pkt
issue_1584: state: open
title: truncating long variable names in summary tables
body: We should look into this (and options setting, etc. for when it kicks in), but right now a really long variable name gets printed in a full summary table. It should trigger truncation a la pandas. Truncation could also be smart so it's the shortest still identifiable string as stata does.

user: jseabold
issue_1581: state: open
title: Add seasonal unit root tests
body: For arrays we'll have to take a period argument. 

There is some code to deal with pandas freqs -> period already. Used in seasonal_decomp and more in the exponential smoothing PR.

Rarely used now: Dickey-Hasza-Fuller (DHF)

Seems to be the most popular: Hylleberg-Granger-Engle-Yoo (HEGY) 

user: jseabold
issue_1578: state: open
title: FAQ: Why doesn't my model match R
body: Different contrast, reference category handling.

Did they both converge? Any warnings? Perfect prediction? Multicollinearity? 

user: jseabold
issue_1577: state: open
title: classification performance measures
body: Add classification performance statistics

```
def precision(pred_table):
    """
    Precision given pred_table. Binary classification only. Assumes group 0
    is the True.

    Analagous to (absence of) Type I errors. Probability that a randomly
    selected document is classified correctly. I.e., no false negatives.
    """
    tp, fp, fn, tn = map(float, pred_table.flatten())
    return tp / (tp + fp)


def recall(pred_table):
    """
    Precision given pred_table. Binary classification only. Assumes group 0
    is the True.

    Analagous to (absence of) Type II errors. Out of all the ones that are
    true, how many did you predict as true. I.e., no false positives.
    """
    tp, fp, fn, tn = map(float, pred_table.flatten())
    try:
        return tp / (tp + fn)
    except ZeroDivisionError:
        return np.nan


def accuracy(pred_table):
    """
    Precision given pred_table. Binary classification only. Assumes group 0
    is the True.
    """
    tp, fp, fn, tn = map(float, pred_table.flatten())
    return (tp + tn) / (tp + tn + fp + fn)


def fscore_measure(pred_table, b=1):
    """
    For b, 1 = equal importance. 2 = recall is twice important. .5 recall is
    half as important, etc.
    """
    r = recall(pred_table)
    p = precision(pred_table)
    try:
        return (1 + b**2) * r*p/(b**2 * p + r)
    except ZeroDivisionError:
        return np.nan
```

Also missing ROC curve.

user: jseabold
issue_1576: state: open
title: t_test and f_test reprs
body: We should revisit this. The current tests repr don't look so hot IMO. We can do better now.

```
print results.f_test('income = education')
<F test: F=array([[ 0.06823286]]), p=0.795203994108, df_denom=42, df_num=1>
```

user: jseabold
issue_1575: state: open
title: ENH: Meta analysis
body: Has any attempts been made to implement meta-analysis
http://en.wikipedia.org/wiki/Meta-analysis#cite_note-Manual-42

Basically it is a way to combine multiple statistic samples by a similar factor. I am doing brief research on this for someone but I would like to start an implementation. The only problem is I am not good with creating the classes so it will be an hassle to whoever converts it :(.

We may need an section on Effect size measurements

http://en.wikipedia.org/wiki/Effect_size#Hedges.27_g

Which include
*Correlation (pearson) already in statsmodels
*Coefficient of determination (R^2)
*Eta-squared
*Cohen F^2 (I think statsmodel has this)
*Hedges's g

Related:

https://github.com/statsmodels/statsmodels/issues/771

user: ccsv
issue_1573: state: open
title: summary vs. summary2
body: We need to come to some decision on this before 0.6.0. Having both is unseemly ;)

user: jseabold
issue_1570: state: open
title: Warnings. How much is too much?
body: Do we really need this in every single regression output?

```
Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
```

Similarly the kurtosis test Warning from scipy.stats when nobs <= 20. Should we silence that? Yes, your inference is only valid if underlying assumptions are met, but do we need to see the underlying assumptions every time we do something? Conversely, how many statistical tests are there were the underlying assumptions aren't met and there isn't a warning? My guess is 90% of them. I think we should document these kinds of things _once_ and be done with it instead of hitting the user over the head with them.

user: jseabold
issue_1569: state: open
title: Experiment: OLS from moment matrix
body: (random thoughts while looking at iteration, and thinking of out of memory computation)

For OLS we should be able to get all results just based on the moment matrix `[x,y]'[x,y]` and nobs.

fit is just one QR, the only work would be to see what needs to be overwritten in the the RegressionResults, endog, exog, fittedvalues and resid wouldn't be available.
Anything that would require model.data would also break, unless it's "faked".

WLS or GLS would mostly work the same way based on the transformed data.

(Unfortunately it wouldn't work for any other models, like discrete or GLM, or even FGLS)

user: josef-pkt
issue_1537: state: open
title: diagnostic, gof for Poisson and count data
body: nice answer and overview here, with references for R
http://stats.stackexchange.com/questions/70558/diagnostic-plots-for-count-regression

this may be interesting  (not poisson), linked to in another stackoverflow question
rms: Regression Modeling Strategies
http://cran.r-project.org/web/packages/rms/index.html

Stata poisson postestimation lists only `estat gof` and `linktest` as specific diagnostic tests.


test overdispersion in Poisson
https://stats.stackexchange.com/questions/66586/is-there-a-test-to-determine-whether-glm-overdispersion-is-significant

user: josef-pkt
issue_1534: state: open
title: Multicollinearity in logit?
body: It looks like R is dropping some variables here or something? The R results indicate NaNs in the regression, and we don't match the results. See my comments there.

https://stackoverflow.com/questions/22716521/why-cant-statsmodels-reproduce-my-r-logistic-regression-results/22719638

user: jseabold
issue_1529: state: open
title: ENH: solution_path method for computing solution path of regularized fits
body: This PR adds a method called "solution_path" to base.LikelihoodModel.  The solution path is the set of parameter estimates obtained as the penalty weight parameter is varied.  The method works with any model that has a "fit_regularized" method.  It calls that method repeatedly to construct the solution path, using bisection to focus computational effort on the places where parameter estimates become zero. 

user: kshedden
issue_1522: state: open
title: test precision iv2sls hausman
body: precision too high

currently the only failure on
http://nipy.bic.berkeley.edu/builders/statsmodels-py2.x-sid-sparc/builds/106/steps/shell_7/logs/stdio

```
======================================================================
FAIL: statsmodels.sandbox.regression.tests.test_gmm.TestIV2SLSSt1.test_hausman
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildslave/nd-bb-slave-sparc-sid/statsmodels-py2_x-sid-sparc/build/venv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/buildslave/nd-bb-slave-sparc-sid/statsmodels-py2_x-sid-sparc/build/venv/local/lib/python2.7/site-packages/statsmodels-0.6.0-py2.7-linux-sparc64.egg/statsmodels/sandbox/regression/tests/test_gmm.py", line 644, in test_hausman
    assert_allclose(hausm[1], res2.hausman['DWHp'], rtol=1e-10, atol=0)
  File "/home/buildslave/nd-bb-slave-sparc-sid/statsmodels-py2_x-sid-sparc/build/venv/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 1183, in assert_allclose
    verbose=verbose, header=header)
  File "/home/buildslave/nd-bb-slave-sparc-sid/statsmodels-py2_x-sid-sparc/build/venv/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 644, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-10, atol=0

(mismatch 100.0%)
 x: array(4.1227010410623496e-16)
 y: array(4.12270104038e-16)
```

user: josef-pkt
issue_1512: state: open
title: Poisson non-existence of MLE
body: `On the existence of the maximum likelihood estimates in Poisson regression`
http://www.sciencedirect.com/science/article/pii/S0165176510000832

from a quick read :

`if rank(exog[endog>0,:) < exog.shape[1]: then params -> - inf`

assuming `rank(exog) == exog.shape[1]` which we need for discrete_model.Poisson

my guess is that the same applies for GLM Poisson, but `rank(exog) < exog.shape[1]` might be handled by pinv in WLS

I thought I already opened an issue once for perfect separation in Poisson, but cannot find any.

user: josef-pkt
issue_1511: state: open
title: Poisson start_params
body: I'm running into optimization problems with large endog values

I think we can just use OLS of the loglinear model for the start_params.
I get good results in the examples, converges the same as GLM Poisson.

(semi-outsource start_params calculation into its own method)

(note: there is an open issue to add link function for the actual used one, e.g. Log for Poisson)   `start_params = pinv(exog).dot(link_inverse(endog))`

currently Poisson has no special start_params code

user: josef-pkt
issue_1509: state: open
title: GLM families link classes
body: Why can I not use the class?

it works with the instance `sm.families.links.log`

```
>>> mod_glm = sm.GLM(y, x, family=sm.families.Gaussian(link=sm.families.links.Log))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new2_py27\statsmodels\statsmodels\genmod\families\family.py", line 408, in __init__
    self.link = link()
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new2_py27\statsmodels\statsmodels\genmod\families\family.py", line 60, in _setlink
    raise ValueError(errmsg % (`self.links`, link))
ValueError: Invalid link for family, should be in [<class 'statsmodels.genmod.families.links.log'>, <class 'statsmodels.genmod.families.links.identity'>, <class 'statsmodels.genmod.families.links.inverse_power'>]. (got <statsmodels.genmod.families.links.Log object at 0x06443830>)
```

user: josef-pkt
issue_1508: state: open
title: stats.weightstats  doc improvement or more standalone functions
body: make it easier to find things or add more standalone wrapper functions, like ztest, or allow ttest_ind to handle 1 sample case

I just saw this in a repo:

```
[t,p]=sst.ttest_1samp(keepers, .5)
#[t2,p2]=smws.ztest(keepers, value=.5, alternative='larger') smws is off in the values... unclear why
```

It took me a few minutes, and checking the documentation (! oh no) to find where the one sample ttest is.

ztest is easy to find, but that's normal based and not the same as scipy.stats.ttest_1samp

```
>>> smws.DescrStatsW(keepers).ttest_mean(value=0.5)
(-2.2679833508703919, 0.035181338574102126, 19.0)
>>> stats.ttest_1samp(keepers, .5)
(array(-2.2679833508703915), 0.035181338574102161)
```

user: josef-pkt
issue_1498: state: open
title: Is there a function to generate a train set and test set from DataFrame？
body: I find sandbox>tool has cross_validate.py, and there are many functions from sklearn include split , are there any update?

user: cloga
issue_1496: state: open
title: bean plot - improve docstring, maybe transpose
body: see Nick Cox's comments here

http://stats.stackexchange.com/questions/88558/beanplots-plus-sign

typo in "Independent-Indpendent"

user: josef-pkt
issue_1495: state: open
title: Mdependent
body: I have added one new covariance structure that may be called by the user using MDependent(m) as their covstruct argument for the Generalized Estimating Equations class. Here is a brief explanation of this structure:

1) It assumes the correlation between all observations within m time steps from each other and in the same cluster is a constant r.
2) It assumes all other correlations between observations is 0.
3) The number m may range from 0 to positive infinity.

Locally tested for replicability, equivalence for m = inf to Exchangeable case, and m = 0 to Independence case.

user: chengfulun
issue_1490: state: open
title: robust GLM families
body: another hint in Hastie Pregiborn, chapter 6 in Chambers Statistical analysis in S

on page 229 they briefly show how families can be adjusted to maximize a robust criterion. In the example the scaled deviance is transformed by the Huber function.

The description doesn't go a lot into details, but it might not be very difficult, since it uses the same framework.

However, one part that they don't mention at all is whether the covariance of the parameter estimate needs to be adjusted. My guess is that we get the adjustment automatically from the WLS regression, but I don't know if we get the correct sandwich form as in RLM. (we won't get the small sample adjustment that is used in RLM).

related: most of the robust literature that I looked at recently used weighted maximum likelihood or similar to correct for exog, x,  outliers.

user: josef-pkt
issue_1488: state: open
title: offset for score, Lagrange multiplier test for GLM, discrete, single-index models
body: We don't exploit `offset` which is available in GLM and in discrete_model yet.

Hastie, Pregiborn, p.222 in chapter 6 in Chambers Statistical Analysis in S mention using a previous fit as offset  `y ~ dose + age + offset(prior.fit)`

This looks like a cheaper way of generating the score for a score or LM test for adding additional variables. analogous to a residual regression after OLS or similar.

exposure cannot be used that way because it might not be additive, see also #1486 

creating score tests this way also sounds similar to the `linktest` in Stata, see #1479 
The linktest doesn't fix the coefficient for the linear prediction of the previous estimate.

user: josef-pkt
issue_1487: state: open
title: formula Terms in results analysis
body: (reading part of GLM chapter in Chambers Statistical Analysis in S)

Currently we mainly support inference on individual parameters (except anova_lm and compare_xxx)

In some cases, like polynomial(x, degree) or categorical variables, the effect of a variable combines several coefficients. We have the terms available if the user uses the formula.

for example we could provide type 3 anova tables for terms additional to the params table.  
(I don't like the sequential anova table, Hastie and Pregiborn, chapter 6, p.216, say "the components here are not as meaningful" refering to `summary.aov` in "unbalanced" designs. Poisson seems to be "balanced" because of the multiplicative structure. check this)

The anova table for GLM is based on deviance residuals in S.

We can reuse score or wald tests for terms by building the appropriate restriction matrix.

user: josef-pkt
issue_1486: state: open
title: GLM: review  offset and exposure
body: just some things that are unclear and potentially bugs 
(while briefly browsing the code)
- in GLM.fit   offset and exposure are exclusive not additive
- GLMResuts.llf special cases NegativeBinomial (I don't see why) and does not reuse mu and ignores offset and exposure for negbin.

not clear to me:  
- exposure in `GLM._check_inputs` is always log transformed, should this depend on link ?

Does offset and exposure have a meaning outside of count or Poisson? 
But they can be used to fix any parameters, but we don't take advantage of that yet AFAIK.

---

Other
`mu` is attached to model and not to results. (?)
Can we get score_obs/jac from the last WLS results?

user: josef-pkt
issue_1480: state: open
title: non-linear models with link functions
body: functional transformation to linearize a model are not innocent, 
e.g. log(y) = x b instead of y = exp(x b)  or   y = x**b
even nonlinear least squares might have the wrong heteroscedacticity

JMCS Silva, S Tenreyro The log of gravity, The Review of Economics and statistics, 2006 - MIT Press
http://scholar.google.ca/scholar?cluster=18284523183521199275&hl=en&as_sdt=0,5&sciodt=0,5

and a reason to finally get something with Box-Cox transformation.

related #1449 

user: josef-pkt
issue_1479: state: open
title: diagnostic: link test - nonlinearity test for many model 
body: see Stata 'r.pdf' manual

estimate y with `exog_test = (fittedvalues, fittedvalues**2)`
reject correct specification if fittedvalues**2 is significant

there might be something similar for OLS already, but Stata's link test works for discrete models, e.g. after Logit.
For example for Logit we have to run a `Logit(endog, exog_test).fit(...)` again.
(do we have all information to run model and fit with the same options again?)

if rejected: either some missing non-linearities in the explanatory variables or incorrectly specified link function

**edit**  don't use fittedvalues, use `predict(linear=True)`, fittedvalues should be refactored to return the transformed, expected mean or estimated mean function E(y|x).

user: josef-pkt
issue_1476: state: open
title: DOC missing notebooks - stats
body: we don't have any notebooks for the possibly popular `stats` like  multicomparison (tukey_hsd), multitesting, power, ...
(only on my blog, and I'm slow in writing more posts)

user: josef-pkt
issue_1475: state: open
title: Add backcasting functions
body: Will be useful for things like #1287 and can be used with filters that require data before and after an observation. Targeting an ARMA-based one for now.

user: jseabold
issue_1466: state: open
title: DOC examples, define exog_names
body: add exog_names to the documentation example that use numpy arrays. for example in GLM

I think it would be good to have a mix of ndarrays, pandas and formula examples.
The first GLM example uses ndarrays.

We can also check if some additional examples should switch to pandas or formulas.

aside: we still need to get a recommended way to set exog_names, see comments in #1465 

user: josef-pkt
issue_1464: state: open
title: outlier tests for univariate data?
body: Do we have anything? Cf. https://stackoverflow.com/questions/22354094/pythonic-way-of-detecting-outliers-in-one-dimensional-observation-data

There's a link to a nice test in the comments.

user: jseabold
issue_1459: state: open
title: perfect prediction: OLS produces very different results on different systems
body: OLS in statsmodels produces different results when run on OSX or Ubuntu. Code:

```
ython 2.7.5+ (default, Sep 19 2013, 13:48:49) 
[GCC 4.8.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import statsmodels
>>> import statsmodels.api as stats
>>> import numpy as np
>>> responses = np.asarray([[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 2.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 3.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 4.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 5.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 6.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 7.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 8.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 9.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 10.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 11.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 12.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 13.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 14.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 15.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 16.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 17.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 18.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 19.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 20.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 21.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 22.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 23.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 24.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 25.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 26.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 27.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 28.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 29.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 30.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 31.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 32.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 33.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 34.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 35.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 36.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 37.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 38.0], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 39.0]])
>>> predictors = np.asarray([[1.0, 0.0, 1.0, 0.0], [1.0, 0.0, 1.0, 1.0], [1.0, 0.0, 1.0, 2.0], [1.0, 0.0, 1.0, 3.0], [1.0, 0.0, 1.0, 4.0], [1.0, 0.0, 1.0, 5.0], [1.0, 0.0, 1.0, 6.0], [1.0, 0.0, 1.0, 7.0], [1.0, 0.0, 1.0, 8.0], [1.0, 0.0, 1.0, 9.0], [1.0, 0.0, 1.0, 10.0], [1.0, 0.0, 1.0, 11.0], [1.0, 0.0, 1.0, 12.0], [1.0, 0.0, 1.0, 13.0], [1.0, 0.0, 1.0, 14.0], [1.0, 0.0, 1.0, 15.0], [1.0, 0.0, 1.0, 16.0], [1.0, 0.0, 1.0, 17.0], [1.0, 0.0, 1.0, 18.0], [1.0, 0.0, 1.0, 19.0], [1.0, 1.0, 1.0, 20.0], [1.0, 1.0, 1.0, 21.0], [1.0, 1.0, 1.0, 22.0], [1.0, 1.0, 1.0, 23.0], [1.0, 1.0, 1.0, 24.0], [1.0, 1.0, 1.0, 25.0], [1.0, 1.0, 1.0, 26.0], [1.0, 1.0, 1.0, 27.0], [1.0, 1.0, 1.0, 28.0], [1.0, 1.0, 1.0, 29.0], [1.0, 1.0, 0.0, 30.0], [1.0, 1.0, 0.0, 31.0], [1.0, 1.0, 0.0, 32.0], [1.0, 1.0, 0.0, 33.0], [1.0, 1.0, 0.0, 34.0], [1.0, 1.0, 0.0, 35.0], [1.0, 1.0, 0.0, 36.0], [1.0, 1.0, 0.0, 37.0], [1.0, 1.0, 0.0, 38.0], [1.0, 1.0, 0.0, 39.0]])
>>> def calc_p(idx):
...  fit = stats.regression.linear_model.OLS(responses[:, idx], predictors)
...  pval = fit.fit().pvalues[1]
...  return -np.log10(pval)
... 
>>> [calc_p(i) for i in xrange(12)]

```

Results (OSX):
[0.048289753396822455, 0.048289753396822455, 0.048289753396822455, 0.048289753396822455, 0.048289753396822455, 0.048289753396822455, 0.048289753396822455, 0.048289753396822455, 0.048289753396822455, 0.048289753396822455, -0.0, 1.8705516032101939]

Results(Ubuntu):
[0.076789855682299193, 0.076789855682299193, 0.076789855682299193, 0.076789855682299193, 0.076789855682299193, 0.076789855682299193, 0.076789855682299193, 0.076789855682299193, 0.076789855682299193, 0.076789855682299193, 0.39363496152986921, 0.15508352013515456]

user: chartl
issue_1456: state: open
title: sandwiches, robust cov for fit_regularized
body: discrete_models have a second fit function.

I never looked at how to implement the cov_type options for fit_regularized.
Main issue is most likely the nan in the cov for trimmed parameters.

user: josef-pkt
issue_1455: state: open
title: DOC add examples for discrete_model  fit_regularized
body: There are examples inside the source examples, but no notebook in the documentation examples

There are many other things missing from the doc examples, I guess.

user: josef-pkt
issue_1450: state: open
title: add missing values to faq
body: This comes up frequently. The formula interfaces could have better kwarg definitions too.

https://stackoverflow.com/questions/22234589/ignoring-missing-values-in-multiple-ols-regression-with-statsmodels
https://stackoverflow.com/questions/19223408/statsmodel-api-logit-valueerror-array-must-not-contain-infs-or-nans

user: jseabold
issue_1449: state: open
title: predict does not attempt to invert transformations to left hand side
body: Let's say I have a model like this:

```
model = "log(y) ~ s"
fit = ols(model, data=data).fit()
```

Presumably I am trying to predict `y` from `s`. I am taking the log of the left hand side so that I believe that I now have a linear model. Ultimately though, I want the forecast value of y for a given value of s.

If I use `fit.predict(s)`, what I am given are predictions for `log(y)` as opposed to `y` itself. Is there any way (for some subset of transformations for which the inverse is known) to tell the `predict` that I would like to predict `y`? Something like: `fit.predict(s, 'y')`

user: cancan101
issue_1448: state: open
title: _repr_html_ for ols fit summary does not show warning for multicollinearity
body: Running the following:

```
N = 1000
e4 = numpy.random.normal(size=N)
e5 = numpy.random.normal(size=N)
e6 = numpy.random.normal(size=N)

y = e4 + e5

data4 = pd.DataFrame({'a':e4,'b':e4+e5/1000,'y':y})
fit = ols("y ~ a + b", data4).fit()
```

I get the following:

```
In [139]:

print fit.summary()
                            OLS Regression Results                            
==============================================================================
Dep. Variable:                      y   R-squared:                       1.000
Model:                            OLS   Adj. R-squared:                  1.000
Method:                 Least Squares   F-statistic:                 2.641e+26
Date:                Thu, 06 Mar 2014   Prob (F-statistic):               0.00
Time:                        00:48:23   Log-Likelihood:                 25520.
No. Observations:                1000   AIC:                        -5.103e+04
Df Residuals:                     997   BIC:                        -5.102e+04
Df Model:                           2                                         
==============================================================================
                 coef    std err          t      P>|t|      [95.0% Conf. Int.]
------------------------------------------------------------------------------
Intercept  -1.087e-13   6.36e-14     -1.710      0.088     -2.33e-13   1.6e-14
a           -999.0000   6.18e-11  -1.62e+13      0.000      -999.000  -999.000
b           1000.0000   6.18e-11   1.62e+13      0.000      1000.000  1000.000
==============================================================================
Omnibus:                        4.445   Durbin-Watson:                   1.986
Prob(Omnibus):                  0.108   Jarque-Bera (JB):                4.519
Skew:                          -0.159   Prob(JB):                        0.104
Kurtosis:                       2.918   Cond. No.                     2.01e+03
==============================================================================

Warnings:
[1] The condition number is large, 2.01e+03. This might indicate that there are
strong multicollinearity or other numerical problems.
```

but this `fit.summary()._repr_html_()` shows no such warning:

``` html
<table class="simpletable">
<caption>OLS Regression Results</caption>
<tr>
  <th>Dep. Variable:</th>            <td>y</td>        <th>  R-squared:         </th>  <td>   1.000</td> 
</tr>
<tr>
  <th>Model:</th>                   <td>OLS</td>       <th>  Adj. R-squared:    </th>  <td>   1.000</td> 
</tr>
<tr>
  <th>Method:</th>             <td>Least Squares</td>  <th>  F-statistic:       </th>  <td>2.641e+26</td>
</tr>
<tr>
  <th>Date:</th>             <td>Thu, 06 Mar 2014</td> <th>  Prob (F-statistic):</th>   <td>  0.00</td>  
</tr>
<tr>
  <th>Time:</th>                 <td>00:49:04</td>     <th>  Log-Likelihood:    </th>  <td>  25520.</td> 
</tr>
<tr>
  <th>No. Observations:</th>      <td>  1000</td>      <th>  AIC:               </th> <td>-5.103e+04</td>
</tr>
<tr>
  <th>Df Residuals:</th>          <td>   997</td>      <th>  BIC:               </th> <td>-5.102e+04</td>
</tr>
<tr>
  <th>Df Model:</th>              <td>     2</td>      <th>                     </th>      <td> </td>    
</tr>
</table>
<table class="simpletable">
<tr>
      <td></td>         <th>coef</th>     <th>std err</th>      <th>t</th>      <th>P>|t|</th> <th>[95.0% Conf. Int.]</th> 
</tr>
<tr>
  <th>Intercept</th> <td>-1.087e-13</td> <td> 6.36e-14</td> <td>   -1.710</td> <td> 0.088</td> <td>-2.33e-13   1.6e-14</td>
</tr>
<tr>
  <th>a</th>         <td> -999.0000</td> <td> 6.18e-11</td> <td>-1.62e+13</td> <td> 0.000</td> <td> -999.000  -999.000</td>
</tr>
<tr>
  <th>b</th>         <td> 1000.0000</td> <td> 6.18e-11</td> <td> 1.62e+13</td> <td> 0.000</td> <td> 1000.000  1000.000</td>
</tr>
</table>
<table class="simpletable">
<tr>
  <th>Omnibus:</th>       <td> 4.445</td> <th>  Durbin-Watson:     </th> <td>   1.986</td>
</tr>
<tr>
  <th>Prob(Omnibus):</th> <td> 0.108</td> <th>  Jarque-Bera (JB):  </th> <td>   4.519</td>
</tr>
<tr>
  <th>Skew:</th>          <td>-0.159</td> <th>  Prob(JB):          </th> <td>   0.104</td>
</tr>
<tr>
  <th>Kurtosis:</th>      <td> 2.918</td> <th>  Cond. No.          </th> <td>2.01e+03</td>
</tr>
</table>
```

user: cancan101
issue_1440: state: open
title: robust scale, generalize options for scale-norm
body: robust.scale.Huber looks interesting, even has axis.
(I never read the details)

equivalence of `robust.scale.Huber` and `RLM`
difference might be different convergence criteria

```
>>> chem = np.array([2.20, 2.20, 2.4, 2.4, 2.5, 2.7, 2.8, 2.9, 3.03,
...             3.03, 3.10, 3.37, 3.4, 3.4, 3.4, 3.5, 3.6, 3.7, 3.7, 3.7, 3.7,
...             3.77, 5.28, 28.95])
>>> rscale.huber(chem)
(array(3.2054980819923693), array(0.6736526001047897))

>>> ressc = sm.RLM(chem, np.ones(len(chem))).fit(scale_est=rscale.HuberScale(d=1.5))
>>> ressc.params
array([ 3.205])
>>> ressc.scale
0.67354141871569495
```

scale.Huber has norm option for mean, but in all robust.scale M-estimators the scale-norm is Huber's function. (clipped squares)

user: josef-pkt
issue_1439: state: open
title: RLM robust estimator Exact Fit Property
body: `Exact Fit Property`: if it is possible to exactly fit at least half of the points, then an estimator with EFP returns the exact fit to these points.

Reference: Maronna, Martin and Yohai 2006  book section 5.10
I have seen it in several parts of the book, in the description of the estimators, there are special cases for scale=0.

see #55 check if we have this, 
I think we do after my PR #1341 if `scale_est='mad'` but not with HuberScale. 

user: josef-pkt
issue_1435: state: open
title: GenericLikelihoodModel use jac for score 
body: design is not clear

I have a subclass of GenericLikelihoodModel, where I add the `jac` method.
However, score doesn't use it automatically.

I need to add

```
    def score(self, params):
        return self.jac(params).sum()
```

document or refactor

user: josef-pkt
issue_1418: state: open
title: REF: add cov_type robust covariance to fit method
body: https://github.com/josef-pkt/statsmodels/tree/REF_covtype_fit

misc notes:
- wls_prediction_std needs to be added to unit tests for robust cov.
  after the first commit, I have a scale that is much too large. I only saw it because the confidence intervals in regression_plots.plot_fit looked too wide.
- also wls_prediction_std needs to be checked that it uses the robust cov

user: josef-pkt
issue_1412: state: open
title: regressionplots combo-plot ?
body: I thought we have a combo plot for regression results? But I didn't find any.

A graph with several subplots that works as quick visual summary. residuals, ....

user: josef-pkt
issue_1411: state: open
title: test precision failures robustcov and generic
body: from http://nipy.bic.berkeley.edu/builders/statsmodels-py2.x-wheezy-sparc/builds/96/steps/shell_4/logs/stdio

tests look strange 0 == 0 with rtol only

```
======================================================================
FAIL: statsmodels.base.tests.test_generic_methods.TestGenericGLM.test_ftest_pvalues
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildslave/nd-bb-slave-sparc-wheezy/statsmodels-py2_x-wheezy-sparc/build/venv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/buildslave/nd-bb-slave-sparc-wheezy/statsmodels-py2_x-wheezy-sparc/build/venv/local/lib/python2.7/site-packages/statsmodels-0.6.0-py2.7-linux-sparc64.egg/statsmodels/base/tests/test_generic_methods.py", line 78, in test_ftest_pvalues
    assert_allclose(pvals, res.pvalues, rtol=5e-10)
  File "/home/buildslave/nd-bb-slave-sparc-wheezy/statsmodels-py2_x-wheezy-sparc/build/venv/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 1181, in assert_allclose
    verbose=verbose, header=header)
  File "/home/buildslave/nd-bb-slave-sparc-wheezy/statsmodels-py2_x-wheezy-sparc/build/venv/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 644, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=5e-10, atol=0

(mismatch 100.0%)
 x: array([ 0.,  0.,  0.,  0.])
 y: array([ 0.,  0.,  0.,  0.])

======================================================================
FAIL: statsmodels.base.tests.test_generic_methods.TestGenericNegativeBinomial.test_ftest_pvalues
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildslave/nd-bb-slave-sparc-wheezy/statsmodels-py2_x-wheezy-sparc/build/venv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/buildslave/nd-bb-slave-sparc-wheezy/statsmodels-py2_x-wheezy-sparc/build/venv/local/lib/python2.7/site-packages/statsmodels-0.6.0-py2.7-linux-sparc64.egg/statsmodels/base/tests/test_generic_methods.py", line 78, in test_ftest_pvalues
    assert_allclose(pvals, res.pvalues, rtol=5e-10)
  File "/home/buildslave/nd-bb-slave-sparc-wheezy/statsmodels-py2_x-wheezy-sparc/build/venv/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 1181, in assert_allclose
    verbose=verbose, header=header)
  File "/home/buildslave/nd-bb-slave-sparc-wheezy/statsmodels-py2_x-wheezy-sparc/build/venv/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 644, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=5e-10, atol=0

(mismatch 100.0%)
 x: array([ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
        0.        ,  0.00060013,  0.02261474,  0.01573902,  0.        ])
 y: array([ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
        0.        ,  0.00060013,  0.02261474,  0.01573902,  0.        ])

======================================================================
FAIL: statsmodels.base.tests.test_generic_methods.TestGenericRLM.test_ftest_pvalues
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildslave/nd-bb-slave-sparc-wheezy/statsmodels-py2_x-wheezy-sparc/build/venv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/buildslave/nd-bb-slave-sparc-wheezy/statsmodels-py2_x-wheezy-sparc/build/venv/local/lib/python2.7/site-packages/statsmodels-0.6.0-py2.7-linux-sparc64.egg/statsmodels/base/tests/test_generic_methods.py", line 78, in test_ftest_pvalues
    assert_allclose(pvals, res.pvalues, rtol=5e-10)
  File "/home/buildslave/nd-bb-slave-sparc-wheezy/statsmodels-py2_x-wheezy-sparc/build/venv/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 1181, in assert_allclose
    verbose=verbose, header=header)
  File "/home/buildslave/nd-bb-slave-sparc-wheezy/statsmodels-py2_x-wheezy-sparc/build/venv/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 644, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=5e-10, atol=0

(mismatch 100.0%)
 x: array([ 0.,  0.,  0.,  0.])
 y: array([ 0.,  0.,  0.,  0.])

======================================================================
FAIL: statsmodels.regression.tests.test_robustcov.TestOLSRobustHacLarge.test_tests
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildslave/nd-bb-slave-sparc-wheezy/statsmodels-py2_x-wheezy-sparc/build/venv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/buildslave/nd-bb-slave-sparc-wheezy/statsmodels-py2_x-wheezy-sparc/build/venv/local/lib/python2.7/site-packages/statsmodels-0.6.0-py2.7-linux-sparc64.egg/statsmodels/regression/tests/test_robustcov.py", line 52, in test_tests
    assert_allclose(pval, res2.pvalues, rtol=5 * rtol)
  File "/home/buildslave/nd-bb-slave-sparc-wheezy/statsmodels-py2_x-wheezy-sparc/build/venv/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 1181, in assert_allclose
    verbose=verbose, header=header)
  File "/home/buildslave/nd-bb-slave-sparc-wheezy/statsmodels-py2_x-wheezy-sparc/build/venv/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 644, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=5e-10, atol=0

(mismatch 100.0%)
 x: array([ 0.        ,  0.03651568,  0.        ])
 y: array([ 0.        ,  0.03651568,  0.        ])
```

user: josef-pkt
issue_1401: state: open
title: fit.summary prints time of summary call rather than time of fit
body: What is the purpose of printing the time when `summary` is called as opposed to the time when the `fit` was initially done?

user: cancan101
issue_1393: state: open
title: influence deleting covariate pattern
body: If we have case weights when several observations have identical data, then a simple influence diagnostic could delete the entire pattern.

Similar if several observations have the same exog pattern, then we could delete all of them for the influence measure.

reference 
Dupont, William D., and William Dudley Dupont. Statistical modeling for biomedical researchers: a simple introduction to the analysis of complex data. Cambridge University Press, 2009.
in the 1st edition 2002: table 1.5.6 page 187 with Stata, residual analysis for Multiple logistic regression
http://www.cambridge.org/ca/academic/subjects/medicine/epidemiology-public-health-and-medical-statistics/statistical-modeling-biomedical-researchers-simple-introduction-analysis-complex-data-2nd-edition

user: josef-pkt
issue_1392: state: open
title: out-of-memory computation figure out pattern setup - multipletests
body: reported privately:

example multipletests

> The data I used was of the medium size,  5453**2 elements

then multipletests fails with memory error

There are some possibilities to optimize the algorithm for using lower memory, but that will not go very far.

We should get some examples that illustrate how we can work with out of memory arrays, memorymaps, pytables, ... whatever. I never tried any of those.

In the multipletests case, batching is not so easy because we need to work with the sorted array, and use the index to put the results back into the original order.
Otherwise, multipletests is a rather simple case, that only uses plain numpy (ufunc), no linear algebra or other messier things.

user: josef-pkt
issue_1391: state: open
title: zero values in Poisson GLM / perfect prediction in GLM
body: Are we guarding against these things?

https://stackoverflow.com/questions/21785049/statsmodels-poisson-glm-different-than-r/21788917#21788917

```
import numpy as np
import pandas as pd
import statsmodels.formula.api as smf
import statsmodels.api as sm
from scipy.stats.stats import pearsonr

data = pd.read_csv('https://gist.github.com/jseabold/9008683/raw/250ff03e6a3ab3514fe4c0e2a95d4b5fcab9aad1/AT_Austria.csv')
```

What's going on here? The parameters are wrong, it converges to a big negative number

```
Model = smf.glm('Data~Origin+Destination+Dij', data=data, offset=np.log(data['Offset']), family=sm.families.Poisson()).fit()
```

This matches R's GLM (default solver fails, bfgs does something at least), but isn't necessarily converged.

```
mod = smf.poisson('Data~Origin+Destination+Dij', data=data, offset=np.log(data['Offset'])).fit(method='bfgs')
print mod.mle_retvals['converged']
```

R gives these warnings

```
Warning messages:
1: glm.fit: fitted rates numerically 0 occurred 
2: glm.fit: fitted rates numerically 0 occurred 
```

user: jseabold
issue_1389: state: open
title: kde nonparametric : automatic bandwidth selection, sample size dependent
body: A good very recent overview 
the recommendations depend on the sample size 
It would also be useful to have recommendation for a method choice as part of the documentation, not just in this case.

specifically they are not very positive on cross-validation in either small or in large samples.

Heidenreich, Nils-Bastian, Anja Schindler, and Stefan Sperlich. "Bandwidth selection for kernel density estimation: a review of fully automatic selectors." AStA Advances in Statistical Analysis 97.4 (2013): 403-433.

We have several methods implemented in statsmodels, but I don't have any overview.

user: josef-pkt
issue_1382: state: open
title: inequality constraints in regression
body: some references (for the robust RLM case)
http://scholar.google.com/scholar?cites=17895094206680664513&as_sdt=2005&sciodt=0,5&hl=en

Question: if we impose a constraint and want to test inequalities, are we running into the parameter on the boundary of the parameter space problem for inference?
see for example #1211 

maybe not because the underlying parameter space is larger, and we just want to test an inequality constraint.

the distribution is non-standard, chi-bar-squared, a mixture of chisquared, but it doesn't look impossible
A. Shapiro: Towards a Unified Theory of Inequality Constrained Testing in Multivariate Analysis
http://www.jstor.org/stable/1403361 

This looks like a more understandable introduction
Molenberghs, Geert, and Geert Verbeke. 2007. “Likelihood Ratio, Score, and Wald Tests in a Constrained Parameter Space.” The American Statistician 61 (1): 22–27. doi:10.1198/000313007X171322.

user: josef-pkt
issue_1379: state: open
title: M-estimators: heteroscedasticity and correlation robust standard errors
body: For implementation it looks like we could reuse sandwich covariance.

For RLM: H1 is the analog of OLS "nonrobust" covariance
I haven't figured out if H3 is HC, it has a summation term that looks similar to HC0, but it doesn't look HC robust in an example. 
edit: H3 is not HC: all three specialize to the OLS variance for squared loss.
summation includes psi_deriv times exog, and not psi times exog
see Huber, Peter J. 1973. “Robust Regression: Asymptotics, Conjectures and Monte Carlo.” The Annals of Statistics 1 (5): 799–821.
(In an example having heteroscedasticity robust cov didn't seem very important, since RLM has endogenous weighting, discounting outliers, i.e. high variance, noisy observations.)

I didn't find any references for HC for RLM, but there are for autocorrelated and spatially correlated, which just look like HAC with a truncated uniform kernel.
Cui, Hengjian, Xuming He, and Kai W. Ng. 2004. “M-Estimation for Linear Models with Spatially-Correlated Errors.” Statistics & Probability Letters 66 (4): 383–93. doi:10.1016/j.spl.2003.10.018.
Fan, Jun, Ailing Yan, and Naihua Xiu. 2014. “Asymptotic Properties for M-Estimators in Linear Models with Dependent Random Errors.” Journal of Statistical Planning and Inference. doi:10.1016/j.jspi.2013.12.005. http://www.sciencedirect.com/science/article/pii/S0378375813003078.

(The implementation by analogy doesn't look very difficult, but the theory looks hairy and the assumptions for the proofs are pretty restrictive.)

all the robust statistics books seem to be assume iid., or independence/uncorrelatedness of x and the variance of the error u, and of u across observations.
Note there is the assumption that the error distribution is contaminated which allow for some iid distortion.
Maronna, Marting Yohai have a chapter on time series models, that I didn't read and I guess assumes correct specification except for the contamination.

user: josef-pkt
issue_1376: state: open
title: summary() unit tests  mock date time
body: The summary prints the date and time and is difficult to test

If we mock the datetime, then we can just use a simple assert_equal on the string.
(and we need an example that is robust to numerical precision differences across platforms and versions, no 1e-10 in results summary)

https://pypi.python.org/pypi/freezegun
I thought I saw another library a short time ago, but don't remember.

user: josef-pkt
issue_1373: state: open
title: Minimum Distance Estimators and Tests
body: In contrast to assumptions for model inference where we assume we want to estimate and test the "true" model, some approaches emphasis finding the best "approximating" model.

Much of those are based on minimum distance. 

example (IIRC) AIC is not consistent (choosing the true number of components asymptotically), but tries to find the best approximating model. BIC is consistent.

similar Vuong test
not available yet in statsmodels, Skipper has Poisson versus ZIP
https://gist.github.com/jseabold/6617976

I added some distance measures to the sandbox but didn't use them yet for model identification, estimation or testing.

Also, I never looked at the details for functional or distributional specification tests where a parametric model is compared to a non-parametric (kernel based) model. I guess it might be similar. (statsmodels.sandbox.nonparametric)

It would be nice to have generic classes and function that collect the relevant statistical results (what's the distribution of a MD estimator?), besides specific cases like the Poisson Vuong test.

related:
equivalence tests, TOST, chisquare test without exact null hypothesis
(in large samples all exact null hypothesis are rejected.)

user: josef-pkt
issue_1372: state: open
title: RLM supporting functions 
body: RLM norms define default parameters, but it's not clear how they are/were chosen.
SAS mentions that theirs are at 95% relative efficiency for normal model (except for Hampel and Welch). But they use 85% efficiency parameters in other examples, cases.

A helper function to calculate relative efficiency for given norm and parameters would be helpful in choosing those, or even as information for default values.
other possibility: can we get the influence function for a norm.

(I don't know how to calculate these two. 

related: additional diagnostic plots, ... for RLM results

user: josef-pkt
issue_1371: state: open
title: outlier tests and FDR multiple testing
body: Just a thought (while thinking about outliers in RLM)

The outlier test is testing an observation (predicted y not predicted mean x*beta).
Under the normal assumption, we have an expected number/fraction of wrong "outliers". That is observation in the tail even if the null hypothesis of no outliers is correct.
For FDR, we should then assume we have a fraction alpha of cases where the hypothesis is False. This would give us better pvalue correction, less conservative than standard FDR, more precise than FDR that estimates the fraction of False cases.

Is this argument correct? I never saw an argument like that.

user: josef-pkt
issue_1362: state: open
title: robust HuberScale call signature
body: For a public function the call signature is not very nice

I would prefer something like `HuberScale()(data, ddof)`
nobs and df_resid can be calcualted from len(data) and ddof

user: josef-pkt
issue_1358: state: open
title: likelihood ratio type tests, compare_xxx for nested models
body: comment for RLM: 
Hampel et al 1986 p 347   use scale from full model in estimation of reduced model, but doesn't matter asymptotically
(SAS for rsquared estimates both mean and sigma for null model (only constant), but uses scale of full model for standardizing residuals in comparison).

similar question shows up in other models and tests, where difference might not even be positive, e.g. GMM or Hausman test.

Implementation:

current compare_lr and compare_ftest just use the statistics from two independently estimated models.
Instead, use model information that we have in the restricted model, and reestimate with some constraints.

small issue: what to do if the fit options differ?
We raise if really inconsistent.
We reestimate based on options in full model

user: josef-pkt
issue_1356: state: open
title: ENH add wald_null to base LikelihoodModelResults
body: In analogy to fvalue and llnull, I would like to add wald_null to the LikelihoodModelResults
For now I just add it to RLMResults, see #1341 but it should be moved.

In contrast to fvalue and llnull, the only assumption is asymptotic normality or the parameter estimate, which we have in all models. It will also work as replacement for llnull and fvalue in summary(). (see Stata)

**open issue**:

The code relies on const_idx, this can be incorrect in
- 2d params with several constants, for example MNLogit
- implicit constant: needs the projection approach of Kevin
- tsa AR, ARMA  I'm not sure wald test for constant makes much sense ?

What's the result if the regression is through the origin, i.e. there is no constant?

user: josef-pkt
issue_1355: state: open
title: BF: use proper scale into HuberScale estimator instead of stand_mad.
body: In the IRLS algorithm, Huber's scale update is:

"sigma^{n+1} = 1/n_a \* sum(\chi(r_i/s)) \* s *_ 2,"

where "s = sigma^{n}".

Actual statsmodels' master branch uses "s = mad(r)" by default instead and there is no possibility to change this. In the literature, mad(r) is sometimes used directly to perform the scale update (I guess for code simplification purpose), but I cannot find any reason why we would use mad(r) in combination with an HuberScale estimator.

For backward compatibility reasons, MAD stay the default init scale estimator in HuberScale iterations when None is provided.

user: VirgileFritsch
issue_1354: state: open
title: ENH: restarting RLM
body: currently RLM always starts with the OLS (WLS without weights) estimate. 
I had already thought about adding starting_weights, but didn't try it.

I just found:

```
Descending psi-functions are tricky, especially when the starting values
for the iterations are nonrobust. Residuals that are accidentally large
because of the poor starting parameters then may stay large forever
because they exert zero pull. It is therefore preferable to start with a
monotone psi, iterate to death, and then append a few (1 or 2) iterations
with the nonmonotone psi.
```

Huber 1981, p.191,192
Huber and Ronchetti 2009, p.186

MM estimators only fix the scale in the M-estimation, AFAICS.

Huber and Ronchetti have some discussion on uniqueness, and on non-convergence if scale=0 in towards the end of section 7.8.

user: josef-pkt
issue_1353: state: open
title: DOC: overview of post-estimation results
body: One of the nice features of the Stata documentation is the section on post-estimation results. We don't have currently a good overview what's available after the different models. Much of the usage is visible only in the documentation examples.

I think it would be very useful to have post-estimation sections and links in the documentation.

For example a general overview page at the top level next to http://statsmodels.sourceforge.net/devel/gettingstarted.html
and then a list of specific post-estimation results in the modules or models.
It would also be a good place to provide a structured overview over the Results attributes and methods

What do we have (not written in `GitHub Flavored Markdown`) :

basic
~~~~
fittedvalues
resid
resid_pearson
...

goodness of fit
~~~~~~~~~~~~
rsquared, pseudo-rsquared
information criteria aic, bic, ...
wald/f-test against constant only model

parameter inference and testing
~~~~~~~~~~~~~~~~~~~~~~~~~
parameter table: bse, confint, tvalues pvalues
t_test: single hypothesis, vectorized
wald/f_test: joint hypothesis
comparison of nested models: likelihood ratio, lagrange mutliplier tests

prediction
~~~~~~~~
predict
tsa: special case

Margins, interpreting nonlinear parameters
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
(discrete and GEE only)

Graphics
~~~~~~~~
gof and regression plots

Diagnostic Measures and Specification Tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

...

---

see mailing list thread https://groups.google.com/d/msg/pystatsmodels/FJhSjb7Es5c/4mrc5HyNKFoJ

the docstring for the `get_influence` method has a missing `See Also`
http://statsmodels.sourceforge.net/devel/generated/statsmodels.regression.linear_model.OLSResults.get_influence.html
We can see what's actually available in the `stats` documentation
http://statsmodels.sourceforge.net/devel/stats.html#outliers-and-influence-measures

There are some additional post_estimation results that are not obvious to find (missing See Also in the docstrings), for example the regression plots, probability plots, and diagnostic tests after OLS. In many case we have examples but the full list is only available in the specific documentation.
for example http://statsmodels.sourceforge.net/devel/graphics.html  (see the two related sections goodness-of-fit and regression plots)
or http://statsmodels.sourceforge.net/devel/stats.html#residual-diagnostics-and-specification-tests

user: josef-pkt
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_1351: state: open
title: ENH: RLM more results statistics
body: add more results statistics: 

(pseudo) rsquared, SAS has a definition
Wald test for constant only: analogy to other models
... ?

**edit:** PR #1341 includes some of those, pseudo-rsquared, ...

number of downweighted options would be nice, what should be thresholds
**edit:** maybe a histogram of weights

add to summary() see #1350

What additional results can we get based on the last WLS regression? 
(todo: store wls_results)

user: josef-pkt
issue_1350: state: open
title: review RLM summary, `__repr__` ?
body: summary with HuberScale prints

`Scale Est.:        <statsmodels.robust.scale.HuberScale object at 0x06409FD0>`
which makes the summary table ugly and much too long

add a `__repr__` to HuberScale, or
replace by a short string in summary

norm just prints the class name

```
>>> res_rlm2.fit_options['scale_est']
<statsmodels.robust.scale.HuberScale object at 0x06409FD0>
>>> res_rlm2.fit_options['norm']
'HuberT'
```

user: josef-pkt
issue_1349: state: open
title: prediction confidence intervals, after other than OLS
body: http://stackoverflow.com/questions/21397082/statsmodels-plotting-mean-confidence-intervals-based-on-heteroscedastic-consiste

last point in comments to my answer: how to get confidence interval for mean prediction

`y_predict = x * beta, beta ~ N(params, cov_params)`

should work for all (linear) models, including RLM, and any sandwiches covariances.

example I posted to mailing list a while ago: t_test already produces confint for mean prediction.

user: josef-pkt
issue_1345: state: open
title: Missing Data Handling in TSA
body: It's a bit trickier in TSA I think. What if a NaN occurs in the middle of an endogenous or exogenous variable? Just dropping isn't enough I don't think. Do we need a check in TSA to see that dates are consecutive? Should we even provide a drop option? It makes sense at the beginning or end of a series but not anywhere else I don't think.

user: jseabold
issue_1341: state: open
title: REF/ENH RLM and robust scale for almost perfect prediction
body: see #55

This makes the code more robust for the cases where we have almost perfect prediction, or where mad of the sample or residual is zero.

There are also small fixes to missing asarray and integer division in robust/scale

The code adds some extra calculation `if scale == 0:`

Explicit, verifying test for `HuberScale` are still missing. I get the same results with an independent new implementation.
New unit tests include a "sanity" check for all norm, scale combinations: 
no exceptions, params is reasonably (no nan, close to median) estimated in RLM in all examples
a few nans in bse, and in standalone scale estimates for some corner cases

**update**

This now includes also some other fixes and enhancements:
- BUG robust.norms.Hampel.rho was incorrect
- BUG robust.norms.Hampel.psi had a `sign` error 
  (this should have caused problems in existing code, for some values in the negative range)
- REF robust.norms.TukeyBiweight shift function so that rho(0) == 0
- ENH: add rsquared, aic, bic, deviance to RLMResults (based on SAS manual)
- ENH: compare_lrtype  LR/Ftest like test for nested models

for scale
- BUG: robust.scale.Huber use mu if given

**TODO**
- regression test for robust scale with almost perfect prediction is outdated, 
  needs another check and new regression numbers.
- add comment to compare_lrtype 
  Maronna, Martin and Yohai 2006 p.106 use empirical cdf for chi2_factor 
  problem
  MMY don't divide by df
  (only test case against SAS has df=1, so we cannot verify)
  Hampel et all 2005 (1986?) p.346 divide by df and by nobs

user: josef-pkt
issue_1335: state: open
title: names in robust, rename, standardize
body: "nonstandard" names

init, initscale -> `start_scale` for iterative solver ?
`M` -> `norm` in `RLM.__init__`
scale ?
`cov` -> `cov_type` in `RLM.fit`
`mu` -> `center` in  `scale.Huber`    maybe, maybe not

I'm trying not to think about `rho`'s and `psi`'s, awful but standard robust lingo

user: josef-pkt
issue_1328: state: open
title: Feature: access all fit results in one single `pd.Series`
body: I'm using OLS on some geospatial timeseries, so I have a large number (i.e., 180x360) of models which I fit.

Is there a way to easily store the results in some pandas data structure? I can see that the Results class has a `save` method, but this uses pickle to save only one model's results. I want to look at global maps (i.e., `pcolormesh()`) off all the coefficients and some of the other result parameters (R², t, ...). This would be easy if I would not have to manually (in the loop over all models) access every single value I want to store. I'm thinking of a `Results.to_pandas()` method which returns _one_ `pd.Series` object holding all the information from the Results class, because then I would just have to access this one `Series` and could add it to an existing `Panel` in my loop.

user: andreas-h
issue_1316: state: open
title: Test  emplike.tests.test_regression.TestRegressionNM.test_ci_beta2 failing
body: The test  statsmodels.emplike.tests.test_regression.TestRegressionNM.test_ci_beta2
is failing in Fedora 20 with the following versions

```
[1] sm.version.full_version
'0.6.0.dev-b401d4d'

[2] sm.api.show_versions()
INSTALLED VERSIONS
------------------
Python: 2.7.5.final.0
OS: Linux 3.12.7-300.fc20.x86_64 #1 SMP Fri Jan 10 15:35:31 UTC 2014 x86_64
byteorder: little
LC_ALL: None
LANG: es_ES.utf8

Statsmodels
===========

Installed: 0.6.0.dev-b401d4d (statsmodels)

Required Dependencies
=====================

cython: 0.19.2 (/usr/lib64/python2.7/site-packages/Cython)
numpy: 1.8.0 (/usr/lib64/python2.7/site-packages/numpy)
scipy: 0.12.1 (/usr/lib64/python2.7/site-packages/scipy)
pandas: 0.10.0 (/usr/lib64/python2.7/site-packages/pandas)
    dateutil: 1.5 (/usr/lib/python2.7/site-packages/dateutil)
patsy: 0.2.1 (/usr/lib/python2.7/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 1.3.0 (/usr/lib64/python2.7/site-packages/matplotlib)
cvxopt: Not installed

Developer Tools
================

IPython: 0.13.2 (/usr/lib/python2.7/site-packages/IPython)
    jinja2: 2.7.1 (/usr/lib/python2.7/site-packages/jinja2)
sphinx: 1.1.3 (/usr/lib/python2.7/site-packages/sphinx)
    pygments: 1.6 (/usr/lib/python2.7/site-packages/pygments)
nose: 1.3.0 (/usr/lib/python2.7/site-packages/nose)
virtualenv: 1.10.1 (/usr/lib/python2.7/site-packages)
```

And the stack trace is

```
======================================================================
ERROR: statsmodels.emplike.tests.test_regression.TestRegressionNM.test_ci_beta2
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/spr/devel/statsmodels/build/lib.linux-x86_64-2.7/statsmodels/emplike/tests/test_regression.py", line 150, in test_ci_beta2
    beta2ci = self.res1.conf_int_el(2, lower_bound=.59, upper_bound=2.2,  method='nm')
  File "/home/spr/devel/statsmodels/build/lib.linux-x86_64-2.7/statsmodels/regression/linear_model.py", line 2224, in conf_int_el
    upper_bound)
  File "/usr/lib64/python2.7/site-packages/scipy/optimize/zeros.py", line 463, in brenth
    r = _zeros._brenth(f,a, b, xtol, maxiter, args, full_output, disp)
ValueError: f(a) and f(b) must have different signs

----------------------------------------------------------------------
```

user: sergiopasra
issue_1313: state: open
title: Entropy calculation returns ~0. Problem with scipy.integrate.quad
body: Came across this. Not sure what's going on yet. Some kind of underflow? I haven't done any digging.

```
x = stats.norm.rvs(100, 7, size=5000)
density = sm.nonparametric.KDEUnivariate(x)
density.fit()
density.entropy
```

Isolated problem

```
from scipy import stats, integrate
import statsmodels.api as sm

x = stats.norm.rvs(100, 7, size=5000)
density = sm.nonparametric.KDEUnivariate(x)
density.fit()
density.entropy

def entr(x, s):
    pdf = kern.density(s, x)
    return pdf * np.log(pdf + 1e-12)

kern = density.kernel
a, b = -np.inf, np.inf
integrate.quad(entr, a, b, args=(x,))
```

user: jseabold
issue_1310: state: open
title: summary test date time problem
body: looks like a possible solution to unit test summary strings
https://pypi.python.org/pypi/frozendate

user: josef-pkt
issue_1308: state: open
title: Autoregressive Spectral density estimator
body: Is there an equivalent for an Autoregressive Spectral density estimator in Statsmodel that is similar to:

http://stat.ethz.ch/R-manual/R-patched/library/stats/html/spec.ar.html

The reason for this other than using it to estimate Autoregressive values is it can be used to auto detect periods for seasonal functions

http://stats.stackexchange.com/questions/1207/period-detection-of-a-generic-time-series/1214#1214

user: ccsv
issue_1307: state: open
title: add chisquare probability plot for multivariate normal
body: first version 

https://github.com/josef-pkt/statsmodels/compare/probplot_enh

branch also adds step function option to probplots

user: josef-pkt
issue_1297: state: open
title: probability plots with multiple data series
body: mainly an idea while thinking of k-sample goodness of fit tests and plots:

can we plot several data series on the same qq or pp plot?

one application would be if we have several series, by category for example
another application would be a k sample comparison where the reference distribution is the edf of the pooled sample.

user: josef-pkt
issue_1289: state: open
title: sf, pvalue for weighted sum of chisquare
body: For several hypothesis tests the asymptotic distribution is the same as a weighted sum of chisquare random variables.

For some tests (don't remember which) the weights depend on the eigenvalues of a matrix. Others have a simple form like 1/(j*(j+1)) (anderson darling k_sample asymptotic distribution).

I mostly finished the survival function to calculate p-values. Should go in distributions or in stats. ?

user: josef-pkt
issue_1288: state: open
title: binned edf test ?  cdf version of chisquare test ? proportions
body: I'm not sure what I want here.

The use case: 
I have a monte carlo of a distribution and want to compare it with a theoretical distribution, either cdf or sf.
The theoretical distribution is expensive to calculate, so I don't want to calculate the cdf/sf at all sample points to run a ks_test or similar ECDF test.

What I would like is to have hypothesis tests for joint and pointwise equality of distributions using the cdf/sf at a few values (10 to 50 instead of nobs=20000).

chisquare test version:

```
>>> efreq = -np.diff(np.concatenate((esf, [0])))
>>> tfreq = -np.diff(np.concatenate((tsf, [0])))
>>> stats.chisquare(efreq * nobs, tfreq * nobs)
(10.534192022890188, 0.30899244414970101)
```

where esf and tsf are empirical and theoretical survival function respectively.

using pointwise confidence interval for proportion

```
>>> from statsmodels.stats.proportion import proportion_confint
>>> nobs = 20000
>>> pci = proportion_confint(tsf * nobs, nobs, 0.05)
>>> ((pci[0] < esf) & (esf < pci[1])).all()
False
>>> pci = proportion_confint(tsf * nobs, nobs, 0.01)
>>> ((pci[0] < esf) & (esf < pci[1])).all()
True
```

binom_test and proportions_ztest are not vectorized to allow multiple hypothesis

```
>>> prop.binom_test(esf * nobs, nobs * np.ones(len(tsf)), tsf)
Traceback (most recent call last):
  File "<pyshell#44>", line 1, in <module>
    prop.binom_test(esf * nobs, nobs * np.ones(len(tsf)), tsf)
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new2_py27\statsmodels\statsmodels\stats\proportion.py", line 384, in binom_test
    pval = stats.binom_test(count, n=nobs, p=prop)
  File "C:\Programs\Python27\lib\site-packages\scipy\stats\morestats.py", line 1033, in binom_test
    raise ValueError("Incorrect length for x.")
ValueError: Incorrect length for x.
>>> prop.proportions_ztest(esf * nobs, nobs * np.ones(len(tsf)), tsf)
Traceback (most recent call last):
  File "<pyshell#45>", line 1, in <module>
    prop.proportions_ztest(esf * nobs, nobs * np.ones(len(tsf)), tsf)
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new2_py27\statsmodels\statsmodels\stats\proportion.py", line 580, in proportions_ztest
    raise NotImplementedError(msg)
NotImplementedError: more than two samples are not implemented yet
```

looping

```
>>> r = np.array([prop.proportions_ztest(e, n, value=t) for (e,n,t) in zip(esf * nobs, nobs * np.ones(len(tsf)), tsf)])
>>> pandas.DataFrame(np.round(np.column_stack((esf, tsf, r)), 6), columns='esf tsf zstat pval'.split())
       esf       tsf     zstat      pval
0  1.00000  1.000000       inf  0.000000
1  0.94355  0.943928 -0.231824  0.816674
2  0.47545  0.470093  1.517125  0.129235
3  0.13535  0.135336  0.005904  0.995289
4  0.03325  0.030465  2.196859  0.028031
5  0.00680  0.006037  1.312614  0.189313
6  0.00130  0.001108  0.753831  0.450951
7  0.00020  0.000193  0.068221  0.945610
8  0.00005  0.000032  0.350501  0.725963
9  0.00000  0.000005      -inf  0.000000
```

user: josef-pkt
issue_1284: state: open
title: linear algebra, projection helper functions
body: where do we put them?

a module with lightweight imports for quick reuse, in stats or tools
- function to identify constant in linear_model (Kevin)
- projection, orthogonal projection in IV/GMM (partialling out variables)
- partial correlation
- multiple correlation 
- linear constraint transformation and checks, in GEE, not yet: Kevin's explanation

user: josef-pkt
issue_1283: state: open
title: outliers_influence - generalize
body: statsmodels.stats.outliers_influence was written with only OLS in mind as application
main class is OLSInfluence

Parts of the calculations will extend to other models. Others will need a model category specific implementation.

related 
 #373 streamline calculation for sigma_notobsi to work without loop

related
 #574 multiple correlation, I also have other measures like all partial correlation that can be used to investigate multicollinearity of exog.
 #1284 there are also several linear algebra or projection helper functions, that should be collected in one place

user: josef-pkt
issue_1282: state: open
title: Feature: add Log-Likelihood to eval_measures.py 
body: For convenience can someone add Log-Likelihood to eval_measures.py? It would make it more convenient to use  eval_measures.py  to produce model summaries.

user: dfrusdn
issue_1276: state: open
title: Multiple Imputation
body: Add intelligent way of dealing with missing data. For example, multiple imputation.  Currently both R and Stata offer this. 

See for example: 
- http://sites.stat.psu.edu/~jls/mifaq.html
- http://www.stata.com/manuals13/mi.pdf
- http://r.iq.harvard.edu/docs/amelia/amelia.pdf
- http://www.stat.ucla.edu/~yajima/Publication/mipaper.rev04.pdf

A general overview on dealing with missing data: http://www.uvm.edu/~dhowell/StatPages/More_Stuff/Missing_Data/Missing.html

Does something like this belong in statsmodels or in scikit learn? Scikit currently has an imputer but it is very rudimentary (mean filling).

user: cancan101
issue_1272: state: open
title: Cronbach's alpha implementation (with code)
body: Cronbach's alpha is a measurement of internal consistency.

http://www.ats.ucla.edu/stat/spss/faq/alpha.html

https://en.wikipedia.org/wiki/Cronbach%27s_alpha

I am new to python but I have working code that can be improved (see below) but I do not know where to insert it for Statsmodel

```
import numpy as np


def svar(X):
    n = float(len(X))
    svar=(sum([(x-np.mean(X))**2 for x in X]) / n)* n/(n-1.)
    return svar


def CronbachAlpha(itemscores):
    itemvars = [svar(item) for item in itemscores]
    tscores = [0] * len(itemscores[0])
    for item in itemscores:
       for i in range(len(item)):
          tscores[i]+= item[i]
    nitems = len(itemscores)
    print "total scores=", tscores, 'number of items=', nitems

    Calpha=nitems/(nitems-1.) * (1-sum(itemvars)/ svar(tscores))

    return Calpha

###########Test################
itemscores = [[ 4,14,3,3,23,4,52,3,33,3],
              [ 5,14,4,3,24,5,55,4,15,3]]
print "Cronbach alpha = ", CronbachAlpha(itemscores)

```

I would suggest merging the two functions and change it for numpy arrays before implementing it.

user: dfrusdn
issue_1265: state: open
title: Duplicate Plots, notebook displays twice
body: I am starting IPython Notebook with `ipython notebook --pylab=inline` and then
plotting using `plot_acf`. I am currently getting the resulting plots duplicated:

![image](https://f.cloud.github.com/assets/51059/1798296/b1bb9b3a-6b3f-11e3-9b1f-27c08dfe56ed.png)

What am I doing wrong here?

Standard pandas plotting using `returns_sq.plot()` works fine (only one resulting plot).

user: cancan101
issue_1257: state: open
title: SUMM: GEE followup
body: (an issue to collect possible or required changes to GEE, that should be reviewed after merge)
- #1256 access to default covariance
- `bse` is not defined (using default covariance), my mistake: bse is inherited
- t_test requires df_resid which is not defined, use_t is also newer than GEE
- add `get_margeff` to Results class, has only smoke test
- raise exceptions for misspelled covariance_types (not done for argument in fit)
- inherited attributes that are not available, needs overwriting `llf` see comment. (*)

(*) currently wrong inheritance, GEEResults is not a "LikelihoodModelResults", we still need a new baseclass (also for GMM, ...), `llf` based results are not available

**naming convention**

with new areas like robust covariances across models, we need to decide on a consistent naming scheme

`cov_type`, `covariance_type`, ...
`covariance_robust` `robust_covariance`, ... postfix qualifier see comment 1 in #1264

user: josef-pkt
issue_1244: state: open
title: kde standard error and pointwise confidence intervals
body: based on Bruce Hansen's notes this should be easy to add (for fixed bandwidth)
http://www.ssc.wisc.edu/~bhansen/718/NonParametrics1.pdf

They might not be very good, but better than nothing.

A bit better but trickier: inverting an asymptotic t-test for confidence intervals.
non-trivial because variance depends on alternative (like in proportions tests), nonlinear rootfinding.

user: josef-pkt
issue_1241: state: open
title: sandbox kernels speed up inDomain
body: inDomain uses pythons filter to select the neighborhood for bounded kernels.

this would be much faster with np.searchsorted, I guess

kernels.CustomKernel doesn't store the data, so we would need a `is_sorted` flag to indicate that we can use searchsorted.

statsmodels.nonparametric.KDEUnivariate stores the data, so it could sort it once.

This is independent of PR #973 but will create merge conflicts if done in another PR at the same time.

user: josef-pkt
issue_1237: state: open
title: old branch misc_io
body: https://github.com/statsmodels/statsmodels/commits/misc_io

Two of the commits might still be interesting, the rest has been rewritten/included in other pull requests

DOC: edit example_gls, fix add_constant, add table to notebook
REF: SimpleTable right align cells, add helper function num2str

Review so we can delete branch from main repo

user: josef-pkt
issue_1236: state: open
title: summary: descriptive statistics refactor
body: There is an old branch in the main repo
https://github.com/statsmodels/statsmodels/commits/desc-stats
with the last two commits improving the descriptive summary statistics

review and merge, or decide whether we want to drop descriptive (and let pandas handle it)

user: josef-pkt
issue_1232: state: open
title: More informative doctrings in from_formula models
body: When a model requires specification of multiple arguments, it is difficult to know which ones to input when looking at the docstring for the formula-based models.

Compare the call structure of ``smf.gee''

```
Definition: smf.gee(cls, formula, data, subset=None, *args, **kwargs)
```

To the paramater definition of ``sm.GEE''

```
Parameters
----------
endog : array-like
    1d array of endogenous response values.
exog : array-like
    A nobs x k array where `nobs` is the number of
    observations and `k` is the number of regressors. An
    intercept is not included by default and should be added
    by the user. See `statsmodels.tools.add_constant`.
groups : array-like
    A 1d array of length `nobs` containing the cluster labels.
time : array-like
    A 2d array of time (or other index) values, used by some
    dependence structures to define similarity relationships among
    observations within a cluster.
family : family class instance
    The default is Gaussian.  To specify the binomial
    distribution family = sm.family.Binomial(). Each family can 
    take a link instance as an argument.  See 
    statsmodels.family.family for more information.
covstruct : CovStruct class instance
    The default is Independence.  To specify an exchangeable
    structure covstruct = sm.covstruct.Exchangeable().  See 
    statsmodels.covstruct.covstruct for more information.
offset : array-like
    An offset to be included in the fit.  If provided, must be
    an array whose length is the number of rows in exog.
constraint : (ndarray, ndarray)
   If provided, the constraint is a tuple (L, R) such that the 
   model parameters are estimated under the constraint L * 
   param = R, where L is a q x p matrix and R is a
   q-dimensional vector.  If constraint is provided, a score
   test is performed to compare the constrained model to the 
   unconstrained model.
missing : str 
    Available options are 'none', 'drop', and 'raise'. If 'none', no nan 
    checking is done. If 'drop', any observations with nans are dropped.
    If 'raise', an error is raised. Default is 'none.'
```

user: vincentarelbundock
issue_1224: state: open
title: ARIMA order selection
body: Could include some tools to suggest orders / make automatic order selection suggestions a little easier. E.g., in SAS
- [SCAN method](http://support.sas.com/documentation/cdl/en/etsug/60372/HTML/default/viewer.htm#etsug_arima_sect031.htm) cf. old mailing list post on this
- [ESACF method](http://support.sas.com/documentation/cdl/en/etsug/60372/HTML/default/viewer.htm#etsug_arima_sect029.htm) 
- [MINIC method](http://support.sas.com/documentation/cdl/en/etsug/60372/HTML/default/viewer.htm#etsug_arima_sect030.htm)

SAS and gretl also expose the [AUTOMDL](http://support.sas.com/documentation/cdl/en/etsug/60372/HTML/default/viewer.htm#etsug_x12_sect016.htm) statement in X13-ARIMA. I have some code for this one already.

user: jseabold
issue_1223: state: open
title: summary() extra information for discrete models
body: The summary() for Poisson and NegativeBinomial doesn't mention that `exposure` was used.

There might be additional "diagnostics" that we might add to some of the models.

user: josef-pkt
issue_1220: state: open
title: missing in extra data (example sandwiches, robust covariances)
body: see comments #863 

The sandwich calculations require additional arrays like groups or time.

If patsy or statsmodels `missing='drop'` removes the rows with missing, then the data of the user doesn't match up with the cleaned endog, exog.

Currently the user needs to drop rows to get consistent matching data for all variables. (it's just a `dropna()` when using pandas)

Instead we can drop internally, if `missing='drop'` and we have the index of the dropped rows. 

user: josef-pkt
issue_1217: state: open
title: AIC vs AICc
body: Any reason that the `summary` for linear models uses the regular AIC (https://github.com/statsmodels/statsmodels/blob/a14d989183d812da8cfecc3f2a752aa6eef44b7b/statsmodels/regression/linear_model.py#L1602) rather than the corrected AICc (https://github.com/statsmodels/statsmodels/blob/c30c29a2c2d2ac11c730d21c19a0556839535ff3/statsmodels/tools/eval_measures.py#L333) ?

user: cancan101
issue_1216: state: open
title: emplike/OLS convergence checks in conf_int_el and test_el
body: OLSResults conf_int_el and test_el
don't have checks for successful convergence, rootfinding.

In a Debian test run the returned results is out of bounds
The test fails only on some machines.

Instead of returning wrong results we should try again for better convergence or raise an exception if a root cannot be found

```
statsmodels_0.5.0+git13-g8e07d34-1~nd13.04+1_i386.build FAILED  18:05.47 real, 671.50 user, 186.72 sys, 1147360 out
FAIL: statsmodels.emplike.tests.test_regression.TestRegressionNM.test_ci_beta2
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/tmp/buildd/statsmodels-0.5.0+git13-g8e07d34/debian/python-statsmodels/usr/lib/python2.7/dist-packages/statsmodels/emplike/tests/test_regression.py", line 149, in test_ci_beta2
    assert_almost_equal(beta2ci, self.res2.test_ci_beta2, 6)
  File "/usr/lib/python2.7/dist-packages/numpy/testing/utils.py", line 452, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
  File "/usr/lib/python2.7/dist-packages/numpy/testing/utils.py", line 812, in assert_array_almost_equal
    header=('Arrays are not almost equal to %d decimals' % decimal))
  File "/usr/lib/python2.7/dist-packages/numpy/testing/utils.py", line 645, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Arrays are not almost equal to 6 decimals

(mismatch 50.0%)
 x: array([ 0.60120459,  0.        ])
 y: array([ 0.60120459,  2.18470794])
```

user: josef-pkt
issue_1212: state: open
title: f_value OLS WLS don't match
body: I'm writing tests between WLS and OLS with user-weighting.  in PR #1189

The fvalues differ.  Should they?

second part: When I use robust covariances (HC1), the fvalue also differ, even though the cov_params is the same. The only difference in the code can come from the constant detection.
There must be something strange or not robust there.

I'm using one of the exog as weights (inverse: `weights=exog[...]`)

user: josef-pkt
issue_1211: state: open
title: Non-Negative Least Squares
body: Add  non-negative least squares to statsmodels.

There is currently an implementation in scipy (http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.nnls.html)

user: cancan101
issue_1210: state: open
title: dtypes, check for object arrays in endog
body: patsy converts object array string endog into dummy variables which are 2d endog

reported on mailing list
help on textbook ANOVA example
https://groups.google.com/forum/#!topic/pystatsmodels/_rnHIUnx5dM

user: josef-pkt
issue_1209: state: open
title: hat matrix, HC2, HC3 for WLS versus Stata
body: It's not clear what definition of the hatmatrix Stata uses with `aweights`

testing robust HCx for WLS against Stata regress with aweights

HC1 matches after trivial change (resid -> wresid)
HC2 and HC3 differ, because the diagonal of the hat matrix is different, after regress with aweights (Stata at least divides one more time by weights.)

making the trivial change resid -> wresid, exog -> wexog, I get the same results with WLS or OLS after rescaling/weighting

```
>>> resw = WLS(dtapa_endog, add_constant(dtapa_exog[['value', 'capital']], prepend=False), weights=1./dtapa_exog['value']).fit()
>>> resw_h2 = resw.get_robustcov_results(cov_type='HC2', use_t=True)
>>> weights_sqrt=np.sqrt(1./np.asarray(dtapa_exog['value']))
>>> resww.params
array([ 0.11694307,  0.10410757, -9.27233358])
>>> resww.bse
array([ 0.00555499,  0.01306514,  3.38487221])
>>> resww_h2 = resww.get_robustcov_results(cov_type='HC2', use_t=True)
>>> resww_h2.bse
array([ 0.00770722,  0.00995744,  2.36858154])
>>> resw_h2.bse
array([ 0.00770722,  0.00995744,  2.36858154])
```

however Stata gets

```
. regress invest mvalue kstock [aw=1/mvalue], vce(hc2)
(sum of wgt is   7.3221e-01)

Linear regression                                      Number of obs =     200
                                                       F(  2,   197) =   36.35
                                                       Prob > F      =  0.0000
                                                       R-squared     =  0.7728
                                                       Root MSE      =  35.178

------------------------------------------------------------------------------
             |             Robust HC2
      invest |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
      mvalue |   .1169431   .0138133     8.47   0.000     .0897021    .1441841
      kstock |   .1041076   .0210187     4.95   0.000     .0626569    .1455582
       _cons |  -9.272334   6.059023    -1.53   0.128    -21.22121    2.676539
------------------------------------------------------------------------------
```

I like our results better.
Note Stata has a big increase in standard errors between weighted and unweighted for the cases of HC2 and HC3, but not for HC1 which doesn't depend on hat matrix.

There will be other cases where we can test against Stata for specific models with non-spherical variance, Panel, HetGLS, GLSAR, ...

user: josef-pkt
issue_1207: state: open
title: more panel robust, contemporaneous correlation xtpcse
body: trying to figure out what xtpcse in Stata is doing

based on the calculations (I didn't see a reference yet) 
xtpcse in the case below assumes that we have a common covariance matrix across time periods. In contrast, cluster robust with time as group index does not assume a contemporaneous covariance matrix that is common across periods (heteroscedasticity is included)
(The inner E(u u') is based on covariance across periods in xtpcse versus the outer product for each period separately in cluster.)

example case Grunfeld from unittests in #1189 

`xtpcse invest mvalue kstock`

contemporaneous correlation only, balanced panel

`E(u u') = Sigma_N kron I_T`, where 
Sigma is contemporaneous correlation of errors, similar to SUR
I_T is identity matrix

playing in the interpreter to build the sandwich, I get the same results as xtpcse

```
>>> exog = res.model.exog
>>> resid2d = res.resid.reshape((10, -1)).T
>>> c = np.dot(resid2d.T, resid2d) / resid2d.shape[0]
>>> S_inner = sum([np.dot(exog[time==ti, :].T, np.dot(c, exog[time==ti, :])) for ti in np.unique(time)])
>>> np.sqrt(np.diag(np.dot(res.normalized_cov_params, np.dot(S_inner, res.normalized_cov_params))))
array([ 0.00721244,  0.02788621,  6.78096485])
>>> _ - [.0072124, .0278862, 6.780965]   # compare with Stata results for bse
array([  3.76733978e-08,   1.30352255e-08,  -1.52535000e-07])
```

setup for OLS

```
    import statsmodels.datasets.grunfeld as gr

    dtapa = gr.data.load_pandas()
    #Stata example/data seems to miss last firm
    dtapa_endog = dtapa.endog[:200]
    dtapa_exog = dtapa.exog[:200]
    res = OLS(dtapa_endog, add_constant(dtapa_exog[['value', 'capital']],
              prepend=False)).fit()

    #time indicator in range(max Ti)
    time = np.asarray(dtapa_exog[['year']])
```

xtpcse can do more, can also estimate AR(1) autocorrelation.
The pure contemporaneous correlation version is closer to SUR and ShortPanel (with time as group index)

**Aside**: as comparison, cluster robust with time index has larger standard errors (10% to 50%)

```
>>> res_cltc = res.get_robustcov_results(cov_type='cluster', groups=time, use_correction=True)
>>> res_cltc.bse
array([  7.90949350e-03,   3.86723375e-02,   1.02728909e+01])
>>> res_cltc.bse / [.0072124, .0278862, 6.780965]
array([ 1.09665208,  1.38679123,  1.51496003])
```

user: josef-pkt
issue_1203: state: open
title: general: use df_resid of results instance for inference
body: we have df_resid in results and in models

e15f26b in #1189  also adjusts res.model.df_resid. This is **not** good, because we don't want to change the underlying model which can be attached to several robust covariance results instances

Instead we should change all methods in the Results classes that access model.df_resid to access the results df_resid instead.

One possible problem: results.df_resid is not defined for base.LikelihoodModel,
it's a cached attribute in RegressionResults
GLM and RLM set it as simple attribute in `__init__`
GenericLikelihoodModel also sets it in `__init__`

update: L1CountResults and similar L1xxxResults also modify .model.df_resid in the `Results.__init__`
also the same in GenericLikelihoodModelResults and ARResults

Do we have any stale state or state spill-over problems?

user: josef-pkt
issue_1201: state: open
title: cluster robust inference, degrees of freedom
body: issue specifically about degrees of freedom and `use_t`
more general issue #1099

cluster and panel robust standard errors
"Which ones are we talking about?"

Cameron, A. Colin, and Douglas L. Miller. "Robust inference with clustered data." Handbook of empirical economics and finance (2010): 1-28.
working paper version http://www.econstor.eu/bitstream/10419/58373/1/635883198.pdf

roughly: with large number of cluster (G of them) asymptotics and variables that don't vary within cluster (L of them), the degrees of freedom is G-L.
Stata uses G-1
using t-distribution with small df works much better in small G case, then normal distribution.
https://github.com/statsmodels/statsmodels/pull/1189#issuecomment-29026705

Not clear to me: Inference for variables where we do have a lot of within variation. There we still have number of observations per cluster (N_g or T) to provide information.  both N and T -> inf
see also 
Christian B. Hansen, Asymptotic properties of a robust variance matrix estimator for panel data when is large, Journal of Econometrics, Volume 141, Issue 2, December 2007, Pages 597-620, ISSN 0304-4076, http://dx.doi.org/10.1016/j.jeconom.2006.10.009.
(http://www.sciencedirect.com/science/article/pii/S0304407606002089)

under mixing conditions we don't need full cluster robust (we are using HAC kernels)
convergence rate is sqrt(N*T)

mentioned in Cameron/Miller
Under difference-in-difference with few treated clusters, the degrees of freedom should be determined by the number of treated clusters not by the total number of clusters.
If we want to support this, then we need an extra option because we don't know what the treated clusters are.

same question for all three version of panel/cluster robust sandwiches that we have right now.

I will add Stata's `df = G - 1` as default, so I can finish up the unit tests.
ivreg2 with cluster uses normal without df (which I also get with `use_t=False`)

user: josef-pkt
issue_1197: state: open
title: power: 2 or k samples with different variance under alternative
body: Skipper wrote an explicit sample size or power calculation for 2 proportions, that matches R base stats and Stata.
https://groups.google.com/d/msg/pystatsmodels/yerxcPsI2bc/prvlH0MUDCcJ
R `pwr` uses the effect size approach and produces the same result as my version in statsmodels.

Because the variance depends on the mean (as in linear exponential family), the variances for the two samples are not the same under the alternative. 
Check details and see if there are general results.

This doesn't fit in the current effect size based power calculations, since the variance is absorbed into the effect size, and there is no argument included that allows for additional variance specification.

Since the rootfinding class is completely generic and allows extra arguments, it shouldn't be difficult to add this.
However, maybe solving for the effect size is not clear if the variance differs.

There are still two different cases:

normal or t distribution: variance is an extra parameter
proportions, binary, count data in LEF family: variance depends on the mean and we don't have a separate variance parameter. 

The second case might actually fit quite well into the current framework. 

user: josef-pkt
issue_1194: state: open
title: diagnostic tests: results instance with intermediate results
body: In cases where some tests like het_breushpagan rejects the Null of no heteroscedasticity it would be interesting to look at which auxiliary variables are significant in the artificial direction.

This would help in getting an idea in which "alternative direction" the Null is rejected.
In this case, what would be likely candidates for including in estimating a variance function.

(not only for linear_model, I looked at a case with heteroscedasticity after estimating a Poisson  model)

user: josef-pkt
issue_1193: state: open
title: Results.t_test allow one sided option
body: t_test currently doesn't allow a choice for one-sided tests.

see stats.weightstats for options for a t-test

does TOST make sense for a regression based hypothesis (multivariate parameter)?

Can we extend wald_test to allow some one-sided hypothesis?
Not sure, I skipped reading the literature on testing inequalities in regression or MLE models so far.

user: josef-pkt
issue_1191: state: open
title: add `force` to methods ?
body: Stata is sometimes refusing to calculate something when it might be statistically incorrect, but has a `force` option to return the calculation anyway.

example: ???

Just thought about it in the case of FGLS when we don't calculate the correct "forecast" or "forecast error"  issue #554 :
In the case of GLSHet the current prediction error for WLS should still be reasonably correct, but for GLSAR we do not take the conditioning on past (other) values into account. (We can use AR or ARMA in that case.)
So, GLSAR would need to black out the prediction errors inherited from GLS or WLS.

user: josef-pkt
issue_1190: state: open
title: Diagnostics, Specification tests after discrete and GLM
body: (intended as umbrella issue)

We need the diagnostics and specification tests for all models, similar to stats.dignostics, stats.influence_outliers and graphics.regressionplots which is currently mainly (only) for OLS

several textbooks list hypothesis tests, plots and descriptive measures for the results for this, GLM, Logit, Probit
- regressionplots: seems to apply in a very similar way to discrete models including GLM
- influence_outliers: several statistics are similar or unchanged, but set is smaller
  mostly based on residuals
- diagnostics: specification tests

related issues:

???

---

Starting with diagnostics, 
I have some tests for heteroscedasticity after binary models, Logit and Probit, and after Poisson in uncommitted scripts.
I haven't checked yet how much we can reuse current tests like the het_xxx and acorr_xxx

user: josef-pkt
issue_1184: state: open
title: missing resid: resid_pearson, ...
body: Poisson is missing resid_pearson, resid_xxx 

I didn't check the other discrete Models.
BinaryResults has resid_pearson, resid_dev and resid_response
Logit and Probit have model specific resid_generalized

We can just add the genmod.family to the discrete models that are exponential models.
For Poisson for example, the family defines resid_anscombe and resid_dev

user: josef-pkt
issue_1183: state: open
title: GEE bias-corrected robust covariances
body: comparison of several bias correction methods for robust covariance matrix in GEE
(SAS seems to have them)

Chunpeng Fan , Donghui Zhang & Cun-Hui Zhang (2013) A Comparison of Bias-Corrected Covariance
Estimators for Generalized Estimating Equations, Journal of Biopharmaceutical Statistics, 23:5, 1172-1187, DOI:
10.1080/10543406.2013.813521
To link to this article: http://dx.doi.org/10.1080/10543406.2013.813521

Note: includes comment about the difference of using F distribution instead of chisquare distribution which makes the tests more conservative or less liberal

user: josef-pkt
issue_1177: state: open
title: linear model conditional bootstrap
body: add bootstrap to OLS (and other linear models) that is numerically efficient (no unnecessary repeated calculations.

see #1099  wild bootstrap for robust bse, pvalues, t_test
see #718 reusing model instance

Looks easy to do, just connect the pieces.

For HC3 or other robust methods that use hat_diag, we would need to save and reuse hat_diag. 
It should be attached to the model in the first place. use lazy evaluation with cached attribute. 
hat matrix does not depend on estimated parameters, (at least not outside twostage or iterative FGLS)

user: josef-pkt
issue_1175: state: open
title: diagnostic tests with robust cov  Wald instead of LM 
body: Stata regress postestimation mentions Wald tests instead of LM test that don't require homoscedasticity assumption.

For example Breush Godfrey acorr test already has a F-test option
http://statsmodels.sourceforge.net/devel/generated/statsmodels.stats.diagnostic.acorr_breush_godfrey.html
but I never seen the connection with robust standard errors before, and I don't know if that version is actually correct with unspecified heteroscedasticity.

closely related #1168  

user: josef-pkt
issue_1172: state: open
title: Explore using libgretl if configured at build time
body: Gretl provides library access to many models. There may actually already be Python bindings too.

user: jseabold
issue_1168: state: open
title: robust covariances, specification tests, nonlinear least squares
body: two additional articles for robust sandwich covariances and for specification tests, with non-linear least squares as special case.

includes sandwiches, information matrix test, wald and lagrange multiplier tests and tests for autocorrelation and heteroscedasticity (as those in `diagnostics`)

Domowitz, Ian, and Halbert White. 1982. “Misspecified Models with Dependent Observations.” Journal of Econometrics 20 (1) (October): 35–58. doi:10.1016/0304-4076(82)90102-6.

White, Halbert, and Ian Domowitz. 1984. “Nonlinear Regression with Dependent Observations.” Econometrica 52 (1) (January 1): 143–161. doi:10.2307/1911465.

user: josef-pkt
issue_1166: state: open
title: Quantile Regression calculates robust cov in fit
body: Quantile Regression calculates robust cov in fit

Based on a quick browsing of the code, I think the calculations could be moved to the results class as in RLM, even if the cov arguments are still in fit.

related #696

user: josef-pkt
issue_1165: state: open
title: Inference for Quantile Regression
body: followup to #696
specifically what inference can we use after estimation.

question that I'm looking at right now is whether QuantReg should use t or normal distribution. (I'm adding a `use_t` option to the Results classes.)
QuantRegResults currently inherits the t distribution from the RegressionResults AFAICS. As an M-estimator I would expect to see the normal distribution.

Also quantreg is not in the generic `base/tests/test_shrink_pickle.py`, which I use for unittests for a few generic methods.

Also I'm standardizing the use of robust cov_params. see #1158 

user: josef-pkt
issue_1163: state: open
title: Lagrange multiplier, score tests
body: Lagrange multiplier, score tests is missing as a general test

part of GEE

for Linear Model with robust covariance
https://github.com/bashtage/statsmodels/commit/c0d24a989f130c6ef21621c3b158921579e737de

user: josef-pkt
issue_1162: state: open
title: log likelihood vs. negative log likelihood, score vs. negative score
body: This is a nitpick, but the code sometimes uses the signs of `loglike` and `score` too loosely for my taste.  For example `score = lambda params: -self.score(params) / nobs` in https://github.com/statsmodels/statsmodels/blob/master/statsmodels/base/model.py#L356

In the kalman filter code https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/kalmanf/kalman_loglike.pyx#L121

```
loglike = -.5 *(loglikelihood + nobs*log(sigma2))
loglike -= nobs/2. * (log(2*pi) + 1)
```

I'm pretty sure that `loglike` is really a negative log likelihood or some kind of penalized negative log likelihood.

In https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/kalmanf/kalmanfilter.py#L83

```
Returns
-------
likelihood
        The negative of the log likelihood
    history or priors, history of posterior
        If history is True.
```

this makes me want to just disregard variable names altogether.

Throughout https://github.com/statsmodels/statsmodels/blob/master/statsmodels/discrete/discrete_model.py the member functions `loglike` and `score` of the `Model` subclasses really do mean the log likelihood and the score function, but on the other hand the `f` and `score` arguments of the `_fit_mle_*` functions mean the negative log likelihood and the negative score function respectively.

user: argriffing
issue_1158: state: open
title: SUMM: robust covariances: connect to models, default options
body: there are several issues open for robust covariances:

specific here a comparison to Stata for possible options

related issues: 
#44 add cov_p as argument to f_test and t_test, closed
#276 replacing cov inplace or create new instance or choose in `fit()`
#1418 and PR #1867 and #1870 add cov_type to fit method
#1922 followup issue to #1870 for details of interface in `fit` (and other todos)

model specific issues:
#1738 GLM (reference, and score_obs)
#1074 add robust cov to WLS (and others) (includes references)
#1095 robust cov for QMLE, #1750 same question GLM related
#863 (cluster) robust for MLE models
#1209 HCx after WLS, definition of hatmatrix
#1906 and PR #1916 GEE refactoring cov_type to fix a bug, should be refactored to follow same pattern
#1104 fvalue is wrong with unspecified heteroscedasticity  - replaced by Wald test

```
      llnull likelihood ratio test for nonzero slope coefficients not yet replace by Wald test
```
#1016 refactor into cached attributes (among unrelated issues)

see also https://groups.google.com/d/msg/pystatsmodels/t5DmAO62lwQ/ZeY54VgSl6UJ

**cov_types**

which types of robust covariance matrices do we support, 
Besides those currently in statsmodels.stats.sandwich_covariance:
- Poisson maybe others needs a special case: overdispersed, only scale is misspecified
- `ivreg2` documentation also mentions `AC`  i.e. HAC without H, only autocorrelation robust.
  not sure ivreg2 allows `bw(#)` and `robust bw(#)`
- xtpcse uses in one case something similar to SUR see #1207
- HC4 ?
- CR0, CR1, ... ?   see #1099

**Stata**

`regress` uses HC1 by default after `robust`
`ivreg2` uses HC1 with option `small` and `HC0` without `small`
`regress robust` and `ivreg2 small` use t distribution in parameter table
`ivreg2` default (without small) uses normal distribution in parameter table
reference for generic MLE sandwiches  
   U:20.21.2 Correlated errors, other sections in U, and P:_robust

**Implementation**

I made a start in https://github.com/josef-pkt/statsmodels/compare/robust_cov  PR #1189
creating a new results instance in a results method
However, it would be better if the `cov_params_default` were set in the results `__init__`
problem: I don't have access to the Results Wrapper from inside the results when I create a new results instance, or how do we get access?
-> add to fit method as main access and selection point

I think `cov_type='nonrobust'` should be treated the same way as the robust covariances, currently we never store `normalized_cov_params * scale`

RLM and QuantileReg currently specify the type of robust cov in fit.

**Econometrics**
- subclasses of RegressionResults (GLSAR, GLSHet, IV2SLS, ...) : for which are the results are valid?
- result statistics and diagnostic tests : which ones are still valid?
  I needed to change the calculations for `fvalue` but there might be more.
  Likelihood ratio tests, compare_f
- Extensions beyond OLS: Can I just use wresid for WLS, (F)GLS? 
  yes, to get cov_params and parameter inference, I'm not sure about other statistical results
  see issue #1209

user: josef-pkt
issue_1141: state: open
title: ENH: model fitting interface for simultaneous calculation of loglike and score
body: > Is there a parameter fitting interface for simultaneously computing log likelihood
> and score, analogous to the scipy optimization interfaces
> for simultaneously calculating f(x) and f'(x)?

https://github.com/statsmodels/statsmodels/issues/1129#issuecomment-26918436

user: argriffing
issue_1137: state: open
title: hausman specification test
body: see discussion starting here https://github.com/statsmodels/statsmodels/pull/1105#issuecomment-26804196

currently in GMM and Panel pull requests

should go into stats.diagnostics with proper options
option sigma/scale is missing in `spec_hausman`

Stata has a more general function `suest` that doesn't assume one of the estimators is efficient and allows for correlation between the estimators.

user: josef-pkt
issue_1132: state: open
title: Models after estimation with reduced information (GMM)
body: How do we use models/estimators that are only designed for estimating some parameters but do not have a fully specified model, e.g. when "nuisance" parameters are removed for estimation?

example: 
individual specific heterogeneity that is differenced out

In GMM I have the additional problem that the moment conditions might not correspond to a predictive model or implement one directly. (example Poisson/Exponential with multiplicative heterogeneity).

Some Quasi- or Pseudo likelihood models might be similar to the GMM case.

Do we try to recover a full model or just stick with parameter estimation?

user: josef-pkt
issue_1131: state: open
title: automatic transformation, scaling or scaling checks or preprocessing
body: I ran into an example with some badly scaled variables (Cameron Trivedi count data similar to randhi).

It took me a while to figure out that the bad scaling was the reason for numerical problems:
- very large condition number, almost singular
- fmin_bfgs goes out of range (huge numbers)
- fmin ("nm") was still ok, but not very precise

Stata doesn't seem to have a problem with the original unscaled data. As far as I know, Stata treats the constant differently, but I haven't read that they also rescale.
I haven't tried if demeaning would be enough in this case.

In my case the user wouldn't see part of the rescaling gmm instruments or first stage of IV2SLS.
For other cases, we would have to transform the parameters to adjust for any scaling.

At least a warning would be useful to point out the problem to users.

---

An idea for stronger numerical stabilisation: use pinv like transformation (analog to linear model)

If the underlying exog dependent part is linear `x dot beta`, then we can transform the exog before fitting and retransform after fitting.
Instead of just rescaling exog, we can orthogonalize exog, which is just a linear, possibly rank reduced, transformation of the data and the linear parameters, beta.
This would correspond to using the analog of a PCA (SVD) regression internally during the optimization, but we report the results in the original exog space.

From my previous experiments, OLS with pinv produces numerically unstable results only in very extreme cases of bad scaling or almost perfect multicollinearity. (one NIST case)
Non-linear optimization like in the discrete models are numerically much less robust.

related for Hessian: #953 linalg.inv in Newton fails relatively easily

user: josef-pkt
issue_1130: state: open
title: ENH: k-sample trend tests
body: k-sample test with Null hypthesis that mean or median are the same and alternative hypothesis that the mean or median are ordered, increasing or decreasing.

see https://groups.google.com/d/msg/pystatsmodels/PLD-QdF7OxI/9UI982MbsBoJ

Cuzick's trend test
https://gist.github.com/josef-pkt/7035711
I have a cleaned up version.

(to go in statsmodels.stats.nonparametric, but I don't see where I have started the code for that, I thought I did.)

user: josef-pkt
issue_1129: state: open
title: ENH: multinomial logistic regression memory usage
body: Multinomial logistic regression is implemented in `statsmodels` as `statsmodels.discrete.discrete_model.MNLogit`.  After training, this model allows observations consisting of multiple quantitative features to be softly classified.  In contrast to hard classification which guesses a single class assignment for each query observation, soft classification guesses a distribution over class assignments.  Multiple supervised observations are used for the training stage which estimates parameter values of the model using observations annotated with hard classifications.

My issue is that one of the intermediate matrices constructed during training does not fit into RAM.  For my problem, the orders of magnitude of three numbers are relevant.  I have a large number of observations (~1e8), a medium number of classes (~1e4), and a small number of features (~10).  Two important matrices do fit in RAM.  The first is the data vector X, also called exogenous by statsmodels with its econometric roots, which has size (large \* small).  The second matrix that fits in RAM is the Hessian matrix which has size ((medium \* small) \* (medium \* small)).  On the other hand, the matrix that temporarily stores the soft classification of each observation in the training set does not fit in RAM.  This matrix has size (large \* medium).

I suspect that the explicit construction of this large matrix that stores intermediate results during training could be avoided.

user: argriffing
issue_1128: state: open
title: panel conditional Poisson
body: opening an issue for it, since it took me a while to find the commit again

https://github.com/jseabold/statsmodels/commit/e427d270278b2ecaa3175b8d78ba89e372f4a335

conditions out the individual heterogeneity, similar to using deviation from group mean or differences in linear panel data models.

Also exists for Logit Chamberlain's Conditional Logit. 
(different from discrete choice clogit McFadden, Stata has two different functions)

aside: using GMM we can remove multiplicative heterogeneity from Poisson (Mullahy 1997, Boes 2010) (I have a simple version as gmm test case)

user: josef-pkt
issue_1126: state: open
title: White specification test, sanity test, robust standard errors
body: I thought I already added White's general (mis-)specification test somewhere, but I don't find it in the statsmodels source.
Test is based on comparing robust and non-robust cov_params.

reminded me because I saw this:

King, Gary, and Margaret Roberts. "How Robust Standard Errors Expose Methodological Problems They Do Not Fix." (2012).
http://scholar.google.ca/scholar?cluster=8159045491591613177&hl=en&as_sdt=0,5&sciodt=0,5#

need a place to store this reference

Beck, Nathaniel. "Sweeping fewer things under the rug: tis often (usually?) better to model than be robust." (2012).
http://scholar.google.ca/scholar?cluster=7791028302104669808&hl=en&as_sdt=0,5#

user: josef-pkt
issue_1120: state: open
title: WIP: Conditional Logit and Mixed Logit
body: Add dcm_base.py with common code for discrete choice models and one file por each model:
- Conditional logit (dcm_clogit.py) 
- Mixed logit or random-coefficients model (dcm_mxlogit.py)

user: AnaMP
labels: superseded: Used for PRs with content that is superseded by a newer PR
issue_1119: state: open
title: quantiles, pp-plot
body: In the probability plots PR #412, I briefly mentioned extensions to using different estimators and to add confidence intervals
(I don't find an issue for this.)

"
Following up on some references and hints, I thought about possible extension for the future

confidence intervals for quantiles: just a thought, and I don't know how much sense it makes in probability plots
using robust estimation (RLM) for the regression line, if loc and scale are not given
using qqplot for estimation, NIST handbook and dataplot discusses this, but there is a non graphic oriented theory for it.

my basic thought was that: This pull request is very useful for users that are used to visually interpreting different versions of the probability plot, and I would like to add some formal statistical tests related to this (over time and besides the other gof tests).
"
https://github.com/statsmodels/statsmodels/pull/412#issuecomment-8017764

a misplaced (?) branch https://github.com/josef-pkt/statsmodels/compare/quantile

From a discussion on scipy issues https://github.com/scipy/scipy/pull/2991
scipy has a different default for the plotting position, sm: a=0, scipy: a=0.3175

user: josef-pkt
issue_1114: state: open
title: ENH: MANCOVA
body: tests for multivariate linear model

see
https://groups.google.com/d/msg/pystatsmodels/FzvPMwOZeIU/-g_-7K2jBtQJ
and issue #1111 

user: josef-pkt
issue_1112: state: open
title: Ordering in Type I ANOVA
body: Trying to compute Type I ANOVA tests on a model:

``` python
from statsmodels.stats.anova import anova_lm
import statsmodels.formula.api as sm
import pandas as pd
cps = pd.read_csv("http://www.mosaic-web.org/go/datasets/cps.csv")
mod1 = sm.ols("wage ~ married + age + educ", data=cps)
fit1 = mod1.fit()
anova_lm(fit1, typ=1)
```

which is working beautifully. However, I'd like to compare this with `married` (which is a categorical variable) coming in last in the ANOVA test. I had thought that this would do the trick:

``` python
mod2 = sm.ols("wage ~ age + educ + married", data=cps)
fit2 = mod2.fit()
anova_lm(fit2, typ=1)
```

however, patsy always puts the categorical variables first in the design matrix.

Based on a conversation on the [mailing list](https://groups.google.com/forum/#!topic/pystatsmodels/ZvsyZag3xaw), it looks like the most viable option to control order is to add an `order` argument to `anova_lm` (and its 'sub-fuctions').

``` python
statsmodels.__version__
'0.5.0'
patsy.__version__
'0.1.0'
```

user: carsonfarmer
issue_1111: state: open
title: missing tools: canonical correlation and ranktest
body: http://ideas.repec.org/c/boc/bocode/s456865.html

used by ivreg for underidentification tests
I guess it should also be useful for sysreg

Johansen's cointegration test is also a test for matrix rank, but I didn't try to understand the details.

This would fit in statsmodels stats.

(Related other location for regression helpers is in statsmodels.tools and spread out
Where do I put a `partial` function for partialling out variables in linear regression?)

user: josef-pkt
issue_1110: state: open
title: residual diagnostics for Instrumental Variables estimation
body: based on the article for ivreg2 Baum, Schaffer, Stillman

RESET test needs to be adjusted
`ivreset` uses Pesaran-Taylor (1999) and Pagan-Hall (1983)

autocorrelation test: Box-Pierce, Durbin's h and Breush-Godfrey are invalit
`ivactest` uses Cumby-Huizinga (1992)

check what should be in `summary` of IV2SLS

user: josef-pkt
issue_1109: state: open
title: REF: robust_covariance normalizations
body: The inner S matrix does not divide by `nobs` in the current implementation.
This is not a bug, since the nobs cancel or are handled in the outer part of the sandwiches.

`robust_covariance` uses `results.normalized_cov_params` in the outer part which also doesn't have division by nobs.

However, it is easy to get inconsistent handling of the `nobs` or `(nobs - k_vars)` normalization. 
(I didn't check and didn't have the nobs for GMM HAC, while I use the nobs normalized S in the other cases in GMM.)

Check for general use of the pieces in robust_covariances and decide on one definition.

user: josef-pkt
issue_1104: state: open
title: regression result fvalue assumptions ? sandwiches ?
body: In IV2SLS, the inherited fvalue is not the same as the value calculated from the f_test.
The result from the f_test agree with Stata, fvalue of the results instance doesn't.

The results instance uses
`fvalue = self.mse_model/self.mse_resid`

the Wald test fvalue is

```
restriction = np.eye(len(res1.params))
res_f = res1.f_test(restriction[:-1]) # without constant
res_f.fvalue
```

It looks like there are two different definitions for `fvalue` used that coincide in the linear model. 

Which hypothesis test is the `fvalue` of the regression result? Is it a Likelihood Ratio test?

I guess (not checked yet), the same discrepancy will occur if we use sandwich, robust cov_params in the linear model.

**update**

The same is true for LL-Null versus Wald test in discrete models:

Stata reports in `summary` (in a Poisson example)
`LR chi2(1)      =       0.00`  after non-robust, and 
`Wald chi2(1)    =       0.00`  after robust covariance (cov_cluster in example)

need to adjust also the label in summary() in discrete_model

for implementation:
In some versions of negative binomial we have an extra parameter. In the Wald test this needs to be excluded additional to the constant.

user: josef-pkt
issue_1101: state: open
title: granger causality in the frequency domain
body: There is an implementation in nitime 

based on Geweke

Breitung, Jörg, and Bertrand Candelon. 2006. “Testing for Short- and Long-run Causality: A Frequency-domain Approach.” Journal of Econometrics 132 (2) (June): 363–378. doi:10.1016/j.jeconom.2005.02.004.

thread at http://mail.scipy.org/pipermail/nipy-devel/2011-August/006640.html

user: josef-pkt
issue_1100: state: open
title: Panel Probit model, GMM, MLE, Mixed
body: Another issue to connect some models

GMM estimation of Panel Probit 
Bertschek, Irene, and Michael Lechner. 1998. “Convenient Estimators for the Panel Probit Model.” Journal of Econometrics 87 (2) (December): 329–371. doi:10.1016/S0304-4076(98)00008-6.

Green's response to show that MLE and simulated MLE works

Greene, William. 2004. “Convenient Estimators for the Panel Probit Model: Further Results.” Empirical Economics 29 (1) (January 1): 21–47. doi:10.1007/s00181-003-0187-z.

The GMM version is close to the GEE version, but looks largely at nonparametric optimal instruments (Newey 1993 Handbook of Statistics).

The MLE version with random effects is close to discrete choice mixed model. (Ana's GSOC)

another approach: copulas
Nikoloulopoulos, Aristidis K., Harry Joe, and N. Rao Chaganty. 2011. “Weighted Scores Method for Regression Models with Dependent Data.” Biostatistics 12 (4) (October 1): 653–665. doi:10.1093/biostatistics/kxr005.

dependency is modeled as normal copula with discrete marginal distribution
compares with GEE, but has a full multivariate probability model.
there now several articles with copulas with discrete margins
a short course https://lred.uea.ac.uk/web/computing/dependence-modelling-using-multivariate-copulas-programme

user: josef-pkt
issue_1099: state: open
title: revisit robust covariance, standard errors (cluster, spatial)
body: reference (not read yet)

Vogelsang, Timothy J. 2012. “Heteroskedasticity, Autocorrelation, and Spatial Correlation Robust Inference in Linear Panel Models with Fixed-effects.” Journal of Econometrics 166 (2) (February): 303–319. doi:10.1016/j.jeconom.2011.10.001.

This combines heteroscedasticity, autocorrelation, cluster and spatial correlation robustness.

I made some notes in the code, but only allow one kernel dimension (HAC Newey West type), IIRC.

pysal has gmm with Kelejian and Prucha.

GEE PR allows for multidimensional "time" index that can also be used for spatial correlation, based on a distance function.

Also AFAIR, I don't have different kernels implemented. See for example Andrews Econometrica 1991
Recently popular: flat-top kernels. e.g Politis IIRC ?

Vogelsang 2012 has a good overview of underlying assumptions for the different versions.

Vogelsang's analysis uses large T asymptotics, t -> inf,   n fixed or also n -> inf
references C. B. Hansen 2007 for the extension of fixed T, large n panel/cluster robust covariance to the large T case.

I didn't  see anything in Vogelsang for two or multi cluster robust standard errors.
That should be robust to within (time-auto-) correlation and to cross-sectional/spatial correlation. (do we need both n -> inf and T -> inf ?  ) 
Do we always need a HAC/kernel weighting in the large dimension (n or T -> inf)?
see CB Hansen 2007 for this
n->inf, T->inf   cluster robust cov (Arellano) only converges with sqrt(n) in general, with strong mixing converges with sqrt(nT)
n fixed, T->inf: not a standard distribution because there is no truncation/kernel for HAC for within cluster autocorrelation.

one problem with Vogelsang's fixed-b asymptotic is that it has non-standard distribution for p-values.
The difficulty in the model assumption is the presence of fixed effects and time dummies. (in Vogelsang.)

---

(There are several other github issues to improve robust cov and sandwiches.)

aside: scikit-learn has distance based (auto)correlation for Gaussian Processes.

user: josef-pkt
issue_1098: state: open
title: random coefficient - mixed model
body: (some notes and connections because I never got any intuition for "Mixed" models.)

mixed models "are" random coefficient panel models

The vignette of R package plm has a comparison of econometric and statistics models.

In the normal-normal model, the model conditional on the variances, covariances is "just" a stochastic restriction model (similar to generalized Ridge, conjugate Bayesian normal equations).

The approaches differ in the estimation of the variance, covariance estimation and decomposition, separate step (moment equations), MLE, REML.

Al Kadiri, Carroll, Wand have a brief summary. Penalized splines as mixed model for longitudinal data. 1)

assumptions, special cases:

Al Kadiri, Carroll, Wand assume uncorrelated random coefficients (different random effects are not correlated), but have an unspecified, unstructured within cluster/panel covariance matrix of the random error.

variance component models: "random coefficients for the constant" with uncorrelated random effect.

1) Al Kadiri, M., R.J. Carroll, and M.P. Wand. 2010. “Marginal Longitudinal Semiparametric Regression via Penalized Splines.” Statistics & Probability Letters 80 (15–16) (August 1): 1242–1252. doi:10.1016/j.spl.2010.04.002.

user: josef-pkt
issue_1096: state: open
title: GMM fixes and improvements
body: (some notes to figure out where GMM was left off)

**bold** and after; update 2015-10-04

required changes:
- fmin  method="nm" is not working well enough, `bfgs` works well    **DONE** `bfgs` doesn't for plain gamma distribution
- code and arguments of methods are inconsistent in use of weights or inverse weights **???**\* DONE but needs another check
- `fit` is pretty useless without options (e.g. maxiter),   **DONE** `fit` has been refactored to be main access point
  drop it and use `fititer` as main method? (`fititer` doesn't return results instance)
- `calc_weightmatrix` is missing kernels, and degrees of freedom or small sample corrections (HCi, HAC)
  connect S matrix calculation from `stats.sandwich_covariance`  **TODO**
## 

**IVGMM** with `start_weight = Z' Z`    (instrument Z)   **DONE**
- 1 iteration: same as IV2SLS  (except currently nonlinear optimization)
- >= 2 iterations: should be IV2SLS with nonparametric heteroscedasticity
- option to avoid sandwich cov_params ? 
    (i.e. assume start_weight is correctly specified optimal weights)
- use explicit linear solution in linear regression model

**CondGMM**      **TODO?** no optimal instruments code

GMM with conditional moment restrictions, similar to IVGMM, but with additional methods to help construct optimal instruments or approximations to them.
(not yet started)

**Comparison to Stata**     **Mostly DONE**  #1105 added test suite against Stata

Since there are so many options it is difficult to figure out how to match reference numbers with Stata.
In some cases, I can match Stata to many digits, in other cases the results are just in the same neighborhood.
- Stata doesn't center moments by default. I do, following the recommendation in Bruce Hansen's textbook
  (without centering I can match the params of Stata with HAC weights, but not the standard errors. 
  Still another option with a different defaults?
- Optimal weights or not. According to the Stata manual the cov_params is always estimated under the assumption
  of non-optimal weights (W != S^{-1}). My default is to assume `has_optimal_weights=True` and use S reestimated after the final iteration.

user: josef-pkt
issue_1095: state: open
title: Add to estimators of V[B] for QMLE
body: I'm looking mainly at Poisson QMLE right now, but I think this holds for several of the maximum likelihood estimators we have if the model is not parametrically correctly specified and I assume GLM too (any linear exponential family). For example, if Poisson is used as a QMLE then we need to provide covariance matrices and post-estimation tests that are robust to the (lack of) distributional assumptions. Right now everything assumes that the models is fully correct.

I'm not sure how much (yet) the sandwich stuff we have already will work here, at least for the covariance, and just needs to be an available option.

documentation for what Stata uses: search help for `_robust` with section Maximum likelihood estimators in `p.pdf`

user: jseabold
issue_1093: state: open
title: clone method, information to reconstruct model, results
body: see #1841 for duplicate issue with notes/comments by Kerby

just an idea for some use cases
- **llnull** currently used in some models in summary for likelihood ratio test
  for example Poisson  
- **bootstrap** attached as a method: _conditional on exog_: we would like to get a new instance of the model that we can reuse internally for running bootstrap replication.
  We need to be able to clone, recreate a new instance of the model and then refit with new endog. Simple in OLS when we just have endog and exog. but we might need more information about what the user used in the signature in some models (if we want to write generic code).
- **results with robust cov_params**: (my latest thought on design)
  make it possible for users to specify which cov_params to use and all statistics in the results instance are based on this cov_params. Instead of changing cov_params in the given result instance, we create and return a new result instance.
- **diagnostic tests** example link-test see #1479 (with comments)

---

data meta information about original model

If we want to get the original names and possibly row names or row indices, then we also need access to that information. I don't know how that works. 
The use cases above are for internal use of a cloned model.
A case for getting the original names of the DataFrame see https://github.com/statsmodels/statsmodels/pull/1529/files#diff-b7b680f28b6de7c43a2dc7afb511f1c4R99

user: josef-pkt
issue_1083: state: open
title: TukeyHSDResults do not keep the pairs tested
body: For the result to be self contained, it should keep the set of pairs used for the testing.

I'm not sure if accessing the pairs through the MultiComparison is correct. If the mc object can change then we would expect the result to keep the original pairs.

Also, it would be nice to use the actual levels instead of using the triu_indices. At least for the report

user: jburroni
issue_1080: state: open
title: repeated calculations for score, hessian
body: joint calculation of hessian and gradient was discussed and removed in #409 

clogit (by Ana) also has a more elaborate calculation for xbetas, that are repeated.

We need to check if we gain much by trying to streamline the calculations during the optimization.

(and we have a vbench essentially ready for usage)

user: josef-pkt
issue_1074: state: open
title: robust cov    WLS, RLM
body: extensions to robust cov_params
- add to other models
- possibilities for bias reduction

First paper is an overview, small sample bias can be pretty large
one conclusion: (wild) bootstrap looks like the best bet to have small/no bias in small samples. Other methods will work well in some cases. H1 through Hx...

MacKinnon, James G. 2013. “Thirty Years of Heteroskedasticity-Robust Inference.” In Recent Advances and Future Directions in Causality, Prediction, and Specification Analysis, edited by Xiaohong Chen and Norman R. Swanson, 437–461. Springer New York. http://link.springer.com/chapter/10.1007/978-1-4614-1653-1_17.

no access to book, alternative link:
http://www.econstor.eu/bitstream/10419/67778/1/661568482.pdf

Furno, Marilena. 1997. “A Robust Heteroskedasticity Consistent Covariance Matrix Estimator.” Statistics 30 (3) (January): 201–219. doi:10.1080/02331889708802610.
http://www.tandfonline.com/doi/citedby/10.1080/02331889708802610#tabModule
for robust estimators like RLM, based on WLS, our RLM has it's own sandwich (that I haven't compared yet

Cragg, John G. 1992. “Quasi-Aitken Estimation for Heteroskedasticity of Unknown Form.” Journal of Econometrics 54 (1–3) (October): 179–201. doi:10.1016/0304-4076(92)90105-Z.
http://www.sciencedirect.com/science/article/pii/030440769290105Z
basic idea get a FGLS estimate with parametric (misspecified) heteroscedasticity, and then use a HC robust cov_params for the misspecified heteroscedasticity.

last two look like a justification for using HC on wresid in WLS, GLS, ...
also maybe GLM, 
check: compare RLM sandwich, with WLS sandwich on last regression in iteration

other references: use bootstrap to correct the bias in the cov estimates

related: problems in general to estimate covariance matrices with low bias and little noise (e.g. for GMM weight matrices, for GEE longitudinal)

user: josef-pkt
issue_1065: state: open
title: VAR Summary Statistics Broken on Linear Dependent data w/ Non-Positive Definite Matrix
body: I uploaded a modified example notebook - https://gist.github.com/tmarthal/6287916
http://nbviewer.ipython.org/6287916

Showing that on a successful auto-correlation with two linearly dependent series (i.e. one series is just a multiple of another), the results object calculations break due to the  result coefficientss not being a positive definite matrix. Specifically the output is problematic in the _stats_table() function of the statsmodels.tsa.vector_ar.util package. 

user: tmarthal
issue_1062: state: open
title: diagnostic outlier_test - use studentized residuals
body: A refactoring or enhancement proposal

the outlier test currently uses `get_influence` to get the studentized residuals

GLM and RLM already have the studentized residuals or standardized residuals as attributes (check which), that can be directly used in the `outlier_test`.

discrete and GLM have other types of residuals. If we know their asymptotic distribution, then we can also use other outlier tests with those other types of residuals.

the current outlier_test uses the t-distribution, while in other cases we will only have the normal distribution.
(another case for `use_t` ?)

user: josef-pkt
issue_1061: state: open
title: diagnostic test with categorical
body: Many or most of the current diagnostic tests just follow a standard version that assumes continuous variables.

What happens if there are dummy variables, categorical variables with dummy coding.

example `linear_lm`
- There is no non-linearity introduced if we take powers of a dummy variable.
  `(0, 1)**2 = (0, 1)`
- The dof in the p-value calculation will (?) be wrong if the added variables are not linearly independent of the existing ones.

Note: for the second case: OLS already calculates the rank of exog, df_model, and the number of extra linearly independent variables that are added is just `mod_large.df_model - mod_small.df_model`

Need more unit tests against cases like these.

Also, this is not mentioned in the docstrings.

---

related: Instrument in GMM based on polynomials
We can use squares and products and higher polynomial terms to get an approximation to the efficient instruments. pinv wouldn't care much but the linear algebra (matrix inverses) might not work well if the matrices are singular.

possible solution: Use SVD/PCA dropping zero eigenvalues, similar to what pinv does. This will remove the redundant variables, the number of instruments is smaller, and we also don't count redundant variables when we just look at the array `shape`
This could just use the `princomp` PCA class, or the SVD version, it has been unused until now.

user: josef-pkt
issue_1054: state: open
title: diagnostic tests - compared to R
body: two links that provide summaries of R functions for model diagnostics
http://www.statmethods.net/stats/rdiagnostics.html
http://rstudio-pubs-static.s3.amazonaws.com/2134_a31c4c71a01c472eb6081432e80689b2.html

We have essentially everything, except *), and more
ncvTest is just Breush-Pagan http://www.inside-r.org/node/86862

*) missing:
- Global Validation of Linear Model Assumptions http://www.jstor.org/stable/30047462
  never heard of this one, but might be a useful addition
- ceresplots whatever they are http://www.inside-r.org/packages/cran/car/docs/ceresPlots

some diagnostics are extended to GLM
need to check what we can generalize or use for other models

possibility: use R examples to improve documentation for our functions

user: josef-pkt
issue_1048: state: open
title: doc structure for offline
body: - pdf file has "Table of Content" as header, -> change header in top page.
  latex build also has option to drop top level headers
- examples are outside of doc tree -> last time I moved them inside the doc tree during htmlhelp build, don't know if this still works, (and I don't have nbviewer installed for notebook rendering
- in chm the initial pages, `Basic Documentation` look unsorted without any extra hierarchy level

user: josef-pkt
issue_1033: state: open
title: docs: PDF of stable & dev downloads?
body: Hello,
would it be possible to provide PDFs for download and offline reading on the page

http://statsmodels.sourceforge.net/

like here:
http://pandas.pydata.org/developers.html

Thanks in advance.

user: timmie
issue_1016: state: open
title: refactoring: code review linear models
body: A few items while reading linear_model.py

There are several errors and many warnings reported by pylint. (no bugs AFAICS)
Some code would benefit from a refactoring.

we can open specific issues for the details
- RegressionModel has no whiten method (pylint): should it raise `NotImplementedError`, methods that need to be defined by subclasses
- sandwich:  
  - they define `het_scale` as public attribute: no explanation, temp variable?, private?
    **update**: They are described in the class docstring for HCx_se, and are the adjusted squared residuals.
  - should they use only whitened attributes ?
  - make the robust covariances into lazy `cache_readonly` properties
- `norm_resid` is unused and untested (according to comment)
- silencing other pylint messages, e.g. weights not defined because indirectly attached
- scale : Is argument in `RegressionResults.__init__` (ignored), and as cached attribute
- more ?

user: josef-pkt
issue_1013: state: open
title: nonlinear models: Maximum Likelihood Curve/Model Fitting in Python
body: see http://stackoverflow.com/questions/17979563/maximum-likelihood-curve-model-fitting-in-python
`This question was voluntarily removed by its author`

essentially MIA, and miscmodels.nonlinls.py looks dead

nonlinls in my nonlinearls2 branch looks ok, requires only minor changes.
However, there is still the GSOC 2012 project branch waiting for cleanup.

nonlinls is based on least-squares, and should be almost identical to MLE under the assumption of normally distributed errors (same relationship as in linear_model)

I don't find a version that uses MLE optimization directly.

user: josef-pkt
issue_998: state: open
title: unit tests for summary()
body: summary and summary2 only have smoke tests

see also #503

user: josef-pkt
issue_989: state: open
title: AR mlefit is attached to model and not results
body: shouldn't this be in the results instance?

user: josef-pkt
issue_973: state: open
title: Additions to Univariate KDEs
body: I've added 'cdf_eval', 'icdf_eval', 'variance' and 'variance_eval' functions to the UnivariateKDEs. 

I realise that the format of these changes may not be what is wanted in the long run, but I figured starting a pull request would help push discussion along so some choice is made.

ref: https://github.com/statsmodels/statsmodels/issues/904

user: Padarn
issue_961: state: open
title: weightstats add skew and kurtosis, and moments
body: weightstats.DescrStatsW has only mean and variance and no higher moments

http://stackoverflow.com/questions/17588085/kurtosis-skewness-of-a-bar-graph-python

user: josef-pkt
issue_960: state: open
title: structural change test, update of Hansen
body: From a brief look this seems to be an update and expansion on Hansen's tables for the p-values of the sup and other (?) test for structural breaks.

http://www.degruyter.com/view/j/jem.2013.2.issue-1/jem-2012-0014/jem-2012-0014.xml?format=INT

Approximate p-Values of Certain Tests Involving Hypotheses About Multiple Breaks
Alastair Hall / Nikolaos Sakkas

Citation Information: Journal of Econometric Methods. Volume 2, Issue 1, Pages 53–67, ISSN (Online) 2156-6674, ISSN (Print) 2194-6345, DOI: 10.1515/jem-2012-0014, March 2013

user: josef-pkt
issue_958: state: open
title: Add SETAR model
body: - See https://gist.github.com/ChadFulton/5962854 for a very first draft.
- See http://www.ssc.wisc.edu/~bhansen/progs/joes_99.html for a link to the implementation paper; Hansen (1999)
- See https://github.com/statsmodels/statsmodels/wiki/SETAR-Model for a longer set of notes / problems with the implementation. Written in a wiki to hopefully prevent cluttering up this issue.

Branch (empty so far) is at https://github.com/ChadFulton/statsmodels/tree/setar

user: ChadFulton
issue_954: state: open
title: pandas for contingency table
body: (I don't know how to do anything quick with pandas.)

We need some recipes and helper functions, and if we have level label, then we can add them to nice summary printouts for contingency table hypothesis tests.

current examples with numpy
https://github.com/statsmodels/statsmodels/pull/950/files#L3R113 SAS with binary letter levels for cochrans_q   (2x2xk table)
and in inter_rater (two helper functions)

individual observations -> count or frequency weights in long format -> contingency table (as array)

user: josef-pkt
issue_953: state: open
title: use pinv in optimize newton
body: I think it is safe to use pinv instead of inv in the newton iteration

`method='bfgs'` finishes without problems

```
>>> print sm.Logit(endog, exog.astype(float)).fit().summary()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new2\statsmodels\statsmodels\discrete\discrete_model.py", line 1186, in fit
    disp=disp, callback=callback, **kwargs)
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new2\statsmodels\statsmodels\discrete\discrete_model.py", line 164, in fit
    disp=disp, callback=callback, **kwargs)
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new2\statsmodels\statsmodels\base\model.py", line 357, in fit
    hess=hess)
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new2\statsmodels\statsmodels\base\model.py", line 405, in _fit_mle_newton
    newparams = oldparams - np.dot(np.linalg.inv(H),
  File "C:\Python26\lib\site-packages\numpy\linalg\linalg.py", line 445, in inv
    return wrap(solve(a, identity(a.shape[0], dtype=a.dtype)))
  File "C:\Python26\lib\site-packages\numpy\linalg\linalg.py", line 328, in solve
    raise LinAlgError, 'Singular matrix'
numpy.linalg.linalg.LinAlgError: Singular matrix
```

user: josef-pkt
issue_943: state: open
title: outliers in time series, winsorizing, frequency domain filtering
body: use an ARMA filter to winsorize outlying observations

I don't know how to use a central (back and forward) filter in time domain, but it should be easy in frequency domain.

(Just a thought while fixing TArma as testcase in miscmodels, 
With TArma I get almost the same coefficients as with ARMA but smaller bse when the DGP has t-distributed innovations.)

user: josef-pkt
issue_942: state: open
title: Statsmodels Grangercausalitytests
body: grangercausalitytests ([df.X, df.Y], maxlag, addconst=True, verbose=True)

This test returns the following error: "TypeError: cannot concatenate 'str' and 'int' objects"

FYI: df is a dataframe with the columns X and Y.

Can we change the input constraint? 

---

JP edit:
I'm adding `asarray`

However: 
output is difficult to read, this should be converted to an instance of a result class

it has a verbose option instead of a summary text in a result instance

If anyone is really interested in the auxiliary regression results, then more informative parameter/variable names would be useful. (I'm not sure this is worth the effort in the various diagnostic tests that use auxiliary regression.)

user: FCJW
issue_941: state: open
title: add conditional logit model
body: **Description**

In contrast to MNLogit, here the variables differ by choice, but the parameters are the same across choices (at least for some .

special case: rectangular structure: all choices have the same variables, but different values

**Interface**

???
see also issue #940 for data handling in general multi-equation models

user: josef-pkt
issue_940: state: open
title: data handling for multiple equation models
body: The data handling in the `__init__` of the base.Model is completely oriented towards single equation models, it assumes a single endog and a single exog.

system of equation models and versions of discrete choice models that work with different exog for each choice are better served with multiple exog, and in the former case possibly with multiple endog.

possibilities without rewriting `handle_data` to allow for lists/tuples of endog and exog:

 0) rewrite `handle_data` to handle tuples of exog
 1) call `handle_data` multiple times, and save them in make `model.data` attribute into a list
 2) instantiate several base model classes, that work as stub models mainly to hold the data and metainformation across equations
 3) possible in some cases: merge all exog into one big exog (same for endog if it applies, not sure if `handle_data` can handle 2d endog)
 4) ignore exog and use data (argument) directly, and map equations to data columns by a dictionary (this seems to be what I had in mind for nested logit).
Or maybe better, use exog argument but just as collection of data variables without any direct relation to the actual design matrices for each equation.

 3) is the simplest solution where it can be used, as for example with panel data.
- Panel, longitudinal data: can be stacked vertically with unbalanced panels and horizontally in SUR type of balanced panels
- sysreg: standard simultaneous equation models, stacking doesn't work, internally we have a possible sparse block structure
- conditional logit (different exog per choice) could be horizontally stacked, or use long format where however several rows would correspond to one equation (Greene's data csv file format)
  
  Horizontal stacking could require inefficient array copying since we always need to break up the exog into the individual exogs for each choice. But horizontal stacking might be worth a try. Supporting code like k_vars and exog_names would still be wrong without splitting.

1) calling `handle_data` several times: 
 This looks pretty flexible. It requires to include endog each time which would be redundant in the conditional logit case. Also none of the generic support code would know how to handle several `data` elements in a model. 

 The model would be fully responsible to handle the list of `data` instances for supporting code.

 One extra problem is that `missing` handling would in most cases need to drop observations across calls to `handle_data`. (It might be possible by given all other exog to the first call to `handle_data` as extra arrays.)

4) dictionary plus data
 This might actually be the easiest for the user, especially if the dictionary can be replaced by a formula. Similar to case 1) we would have to call `handle_data` or `handle_formula_data` several times, building the design matrices internally similar to `from_formula`. 
## 

One followup problem, that we have the number of variables different from the number of parameters, does already exist in MNLogit and in ARMAX (and in VAR/SVAR to some extend), but these cases have more structure than general multi-equation models.

user: josef-pkt
issue_939: state: open
title: summary() formatting strings helper functions
body: We have one helper function for float or g formatting.

In some cases we have integer formatting that could be NaN or None. For example, I had a case when df_model was None, and I got an exception in the summary method.

Instead, we could have automatic formatting also for this case. Maybe the only cases so far are df_model or df_resid.

Alternative, check in summary whether df_model and df_resid are integers, and don't print them if not.

user: josef-pkt
issue_938: state: open
title: confidence interval for mean prediction of mean
body: for genmod GLM/GEE

SAS uses just the that `x * params` is a linear transformation of normal distributed random variable params.

http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#statug_genmod_sect039.htm#statug.genmod.genmodclpv

get confidence interval for `x beta`, then transform interval bounds with inverse link, to get confidence interval if `linear=False`

would be easy to add, except we still have no established pattern for stochastic information for predict

user: josef-pkt
issue_937: state: open
title: sharing code: Mixins, new Results classes
body: followup to issue #929 and PR #928 (adding GEE)

common code for GEE and GLM Results might be better implemented in (or moved to) a new intermediate level results class or mixin.

genmod and discrete share many feature but don't share much code.

Margins and Influence are separate classes, but only available for and attached to one model or model category (discrete and OLS resp).

_Get features for free by sharing code_

user: josef-pkt
issue_931: state: open
title: review: convergence criteria
body: I don't have an overview for the convergence criteria across models, and how they can be set by the user. RLM, GLM, FGLS, QuantileRegression discrete and MLE models.
- maxiter
- tolerance on objective function, e.g. deviance in GLM
- tolerance on the parameter estimates

an overview table for what criteria are available and what the defaults are would be useful.

related: do the model.fit warn if maxiter has been reached? GLM does not AFAICS

user: josef-pkt
issue_930: state: open
title: add margins to genmod
body: discrete has margins now as a largely standalone discrete.discrete_margins.

Check whether we cannot simply reuse it for the genmod model(s), 
see also issue #929 for GEE results

user: josef-pkt
issue_929: state: open
title: GEE result statistics, diagnostics
body: PR #928 will add GEE for generalized linear models to statsmodels.

currently we are missing some post-estimation results
Stata http://www.ats.ucla.edu/stat/stata/library/gee.htm
R gee http://www.inside-r.org/packages/cran/gee/docs/summary.gee (I don't see what post-estimation results they provide)
- goodness of fit measures: Stata uses chi2 wald statistic, and Hausman’s specification test
- testing of linear hypothesis: need generic Wald test (as complement to f_test and t_test)  
  Stata has `lincom` linear hypothesis tests, and `test` Wald tests separately, and equivalents for non-linear hypothesis (I don't know yet what's the difference)
- margins: Stata also has margins, which we have in discrete but not yet in genmod

I don't know what residual diagnostics, influence and outlier measures are available.
genmod.GLM has different kinds of residuals, but no supporting influence and outlier measures and tests yet.

user: josef-pkt
issue_926: state: open
title: discrete has no aic, bic in summary
body: discrete has no aic, bic in summary

user: josef-pkt
issue_925: state: open
title: Design overview for models
body: Working on `statsmodels.base` mainly because of GenericLikelihoodModel, I have a hard time gaining an overview over where everything is defined.

We need some code inspection utilities to get an updated list of what is defined in the different models and where. 
It's important to see on which level of our class hierarchy are attributes, properties and methods defined, so we know where to add new models. And we know which attributes an methods need to be defined or checked if the inherited version is correct.

There are several open issues with specific items.

One special case for some future extension are models where `n_params != exog.shape[1]`
ARMAX is a main case but I haven't looked at yet, whether it has a solution for parameter (and names) handling that can be generalized to other models.

user: josef-pkt
issue_921: state: open
title: parameter restrictions where to define df_model, xnames, ...  
body: I'm reviewing the GenericLikelihoodModel, and try to fix so it works again out of the box.

One problem I run into is when I set a parameter restriction in the fit method. During initialization I don't know yet how many parameters will be estimated. `fit` can fix some parameters at predefined values, and as a consequence the actual number of `params` can change.

Since now the initialization code for xnames, df_model, df_resid, and so on, is in the `__init__` and `initialize` which is called by `__init__`, the default code paths don't work correctly.

I get exceptions at various places, when df_model, xnames, ... are not defined or not defined correctly.

One option is to force that the restrictions are in `__init__`, so that the correct number of parameters is already known.

The other option is a fall back that fixes up the missing or incorrect meta attributes during fit.

I don't really like the first option, since it makes it more difficult to write fit methods with different constraints. However, the problem is a bit similar to the one we had when we moved `order` from `fit` to `__init__` in ARMA.

The parameters in my cases are extra parameters that are not associated with an `exog`.  The two cases I'm looking at are fixing the location in distribution estimation (like floc in scipy.stats) and fixing extra shape parameters such as the degrees of freedom in a maximum likelihood estimation with the t distribution.

The same problem will show up when we get non-linear least squares with or without restrictions.

user: josef-pkt
issue_916: state: open
title: el_aft strange opt code
body: ```
class OptAFT(_OptFuncts):
    """
...
    """
    def __init__(_OptFuncts):
        pass
```

pylint complains

this is later subclassed and overwrites `__init__`

```
class AFTResults(OptAFT):
    def __init__(self, model):
        self.model = model
```

user: josef-pkt
issue_914: state: open
title: GenericLikelihoodModel test cases
body: followup to #432 

currently we have test cases for GenericLikelihoodModel in miscmodels.count but with relatively low precision IIRC

Vincent has written an initial version of NegativeBinomial as a basic GenericLikelihoodModels. This would make a good test case for the test suite.

So that we can assure that the Generic model works reasonably well by default.

user: josef-pkt
issue_904: state: open
title: KDEUnivariate improvements and consistency - cdf and icdf
body: I think there are a couple of small issues with the KDEUnivariate class right now, in particular with the cdf and icdf. Although some of these could be by design

1) As I mentioned in issue #885 there is currently a discrepancy between the icdf and cdf fields. The icdf returns the sample icdf (that is, quantiles defined by the input data), while cdf returns the cdf corresponding to the smoothing kernel.

2) The cdf of a KDEUnivariate object looks like it is a data field, but can take a very long time to calculate if the sample is large. Because it is designed to look like a data field, this can happen without warning (I thought I'd hit some sort of infinite loop).

3) As mentioned by josef-pkt also in issue #885, the bandwidth of the kernel you would use for a kernel estimate of a cdf is different (I think it scaled like n^(1/3) rather than n^(1/5)). Currently this is not what you would get using the cdf.

I'd be interested in contributing here, but I would want to be consistent with how want to see this working.

user: Padarn
issue_891: state: open
title: acf, acovf for multiple univariate (vectorized)
body: related to issue #322 
acovf has an explicit check that `ndim` is 1, and raises a ValueError if not, to handle pandas objects/series
see 
https://github.com/statsmodels/statsmodels/commit/bcdb0250696d7079190f7c45eedc5bede6312f9c

my impression is that this should work vectorized for 2d along one axis
I don't remember which versions/shapes I checked a long time ago.

user: josef-pkt
issue_889: state: open
title: SimpleTable conversions uses as_ while Pandas conversions uses to_. It should be the same
body: Convertion of  pandas object or statmodels objects into [html, latex,..] should be perform using the same message/selector.

user: jburroni
issue_888: state: open
title: KDEUnivariate reuse binned array for repeated fit
body: Currently KDEUnivariate performs the binning on each call to fit.

If we want to repeatedly fit, for example for cross-validation, then we can reuse the already binned array.

We don't have a cross-validation code yet for KDEUnivariate.

(see #887 for the case of too much caching.)

user: josef-pkt
issue_881: state: open
title: dynamic linear regression
body: http://www.inside-r.org/packages/cran/dynlm/docs/dynlm

```
The interface and internals of dynlm are very similar to lm, but currently dynlm offers three advantages 
over the direct use of lm: 
1. extended formula processing, 
2. preservation of time series attributes, 
3. instrumental variables regression (via two-stage least squares).
```

similar distributed lag models

main question, I think, is whether we gain something by using time series explicitly as in ARMA, or requiring `tsset` (Stata)

It might be possible to reuse large parts of the current tsa code for time/sequence handling. lag, diff and so on might be handled by pandas also?

user: josef-pkt
issue_879: state: open
title: support for contingency table
body: see discussion in #878 

mostly a wrapper around formulas, pandas crosstabs and statistical tests.

formula interface to pandas might not be so easy if patsy does different preprocessing, eg convert bool to dummy. For simple cases (2x2 or 2xk) it might be easier to just use numpy after patsy.

**update** Nov 2015
#2418 PR merged contingency table support

user: josef-pkt
issue_878: state: open
title: formula: boolean endog, contingency table
body: I have two binary variables for which I want to see how they relate to each other, so I tried correlations and a simple regression and found the following error when I tried to get the regresison summary: 

```
print df[["connection", "published"]].corr()

            connection  published
connection      1.0000     0.0335
published       0.0335     1.0000

res = sm.ols("published ~ connection", df).fit()
res.summary()


---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-12-864f3205c5bd> in <module>()
----> 1 res.summary()

C:\portabel\Python27\lib\site-packages\statsmodels\regression\linear_model.pyc in summary(self, title, xname, yname, alpha, float_format)
   1198         jb, jbpv, skew, kurtosis = jarque_bera(self.wresid)
   1199         omni, omnipv = omni_normtest(self.wresid)
-> 1200         dw = durbin_watson(self.wresid)
   1201         condno = cond(self.model.wexog)
   1202         eigvals = eigvalsh(np.dot(self.model.wexog.T, self.model.wexog))

C:\portabel\Python27\lib\site-packages\statsmodels\stats\stattools.pyc in durbin_watson(resids)
     27     """
     28     diff_resids = np.diff(resids,1)
---> 29     dw = np.dot(diff_resids,diff_resids) / \
     30         np.dot(resids,resids);
     31     return dw

ValueError: matrices are not aligned
```

[Actually I would like to get some kind of contingency table and chi squared test, but up to now I haven't found how to do that in statsmodels]

user: jankatins
issue_877: state: open
title: diagnostic plots for regression - another version
body: see plot at bottom of

http://support.sas.com/documentation/cdl/en/statug/63347/HTML/default/viewer.htm#statug_glm_sect056.htm

for one way analysis

looks better than the one in NIST/dataplot

user: josef-pkt
issue_876: state: open
title: qqplot for multivariate elliptical distributions
body: One visual tool to inspect outliers in multivariate normal samples is to plot Mahalonobis distance against the corresponding distribution. I've seen plot for normal in papers.

I guess this generalizes if we plot against the non-linear transformation for elliptic distributions (like t distribution). I haven't seen that yet.

user: josef-pkt
issue_874: state: open
title: REML for FGLS type models
body: MLE and REMLE for heteroscedastic model, or more generally for models with covariance structures equivalent to FGLS, but with MLE or REMLE instead of iterative least squares.

This paper looks reasonable to implement:

Smyth, Gordon K. 2002. “An Efficient Algorithm for REML in Heteroscedastic Regression.” Journal of Computational and Graphical Statistics 11 (4) (December): 836–847. doi:10.1198/106186002871.
http://www.jstor.org/stable/1391164

user: josef-pkt
issue_873: state: open
title: pggls and ShortPanel
body: I just saw pggls in plm, 
http://www.inside-r.org/packages/cran/plm/docs/summary.pggls

this looks close to what I have in my ShortPanel, need to write example and test against it, if it's close enough

user: josef-pkt
issue_872: state: open
title: heteroscedasticity robust cov for small samples
body: This looks good for small samples

Cai, Li, and Andrew F. Hayes. 2008. “A New Test of Linear Hypotheses in OLS Regression Under Heteroscedasticity of Unknown Form.” Journal of Educational and Behavioral Statistics 33 (1) (March 1): 21–40. doi:10.3102/1076998607302628.

http://jeb.sagepub.com/content/33/1/21

uses eigenvalue decomposition to transform to close to standard t-test with Satterthwaite degrees of freedom for heterogeneous variance. (I read only parts)

user: josef-pkt
issue_866: state: open
title: regression test for constant variance - add Levene or Brown-Forsythe
body: we have
`statsmodels.stats.diagnostic.HetGoldfeldQuandt` 
which is based on the F-test.

However, the F-test for equality of variances is very non-robust to deviation from normality (much more so than means).

Levene or Brown-Forsythe would be a robuster alternative.

see also page 3 of http://www.stat.ucdavis.edu/%7Eaganguly/STA108/6.Diagnostics_in_R.doc

there is also no reason to restrict to only 2 subsets, e.g. allow one-way heterogeneity by specifying groups.

http://support.sas.com/documentation/cdl/en/statug/63347/HTML/default/viewer.htm#statug_glm_a0000000869.htm

main question is whether we should adjust the degrees of freedom in the equal variance tests to take account of the estimated regression parameters (if not, then we can just call Levene or similar with the residuals.)

user: josef-pkt
issue_865: state: open
title: robust: Alternative to LTS
body: Park, Yousung, Daeyoung Kim, and Seongyong Kim. 2012. “Robust Regression Using Data Partitioning and M-Estimation.” Communications in Statistics - Simulation and Computation 41 (8): 1282–1300. doi:10.1080/03610918.2011.598994.

http://www.tandfonline.com/doi/abs/10.1080/03610918.2011.598994

looks good in simulations, at least when outliers are bunched together, but looks also good in example datasets

related: polemic against LTS and similar robust with random starting sets
www.math.siu.edu/olive/pphoax.pdf‎

user: josef-pkt
issue_864: state: open
title: dtypes promotion
body: Currently we are doing very little dtype checking and upcasting in statsmodels. Some of it is however implicitly done in functions that we call

To avoid lower precision calculations and be upwards compatible, something like

```
x = np.asarray(x, >=np.float64)
```

would be good, something that converts ints, bool and lower precision to float64 but leaves higher precision float and complex double alone.

Sebastian's answer (numpy mailing list)
`np.result_type(arr, np.float64)` uses the usual numpy promotion rules.
But it doesn't do the "asarray" part. Its still the closest thing I can
think of right now.

requires numpy 1.6

addition: performance hint

`astype` makes copy of array even if dtype is already correct `https://github.com/scipy/scipy/issues/4203#issuecomment-64773299` (don't spam scipy issues)

user: josef-pkt
issue_863: state: open
title: cluster robust standard errors for Logit, Probit and GMM
body: http://www.tandfonline.com/doi/abs/10.1198/jbes.2010.07136#.Ua5-o8rcPIU
Cameron, A. Colin, Jonah B. Gelbach, and Douglas L. Miller. 2011. “Robust Inference With Multiway Clustering.” Journal of Business & Economic Statistics 29 (2): 238–249. doi:10.1198/jbes.2010.07136.

I used this when I implemented cluster robust standard errors, but I didn't look at the second part where they use it for M- and GMM-Estimators.

user: josef-pkt
issue_862: state: open
title: statistical tests with cluster robust sandwich covariance 
body: example Levene test for correlated clusters
(overall best results are with trimmed mean as location measure)

Iachine, Ivan, Hans Chr. Petersen, and Kirsten O. Kyvik. 2010. “Robust Tests for the Equality of Variances for Clustered Data.” Journal of Statistical Computation and Simulation 80 (4): 365–377. doi:10.1080/00949650802641841.

http://www.tandfonline.com/doi/abs/10.1080/00949650802641841#.Ua5yhsrcPIU

looks good, robust behavior in Monte Carlo
exploiting equivalence of Anova type tests and linear regression

(part of my literature review on robust hypthesis tests for comparing variances)

user: josef-pkt
issue_861: state: open
title: TOST vectorized
body: jburroni uses the vectorized ttest in CompareMeans

question: Do the tost versions, or the ttest with `value` work vectorized?

The main question is whether we can specify  `value` or tost limits as an array, list.

user: josef-pkt
issue_860: state: open
title: Nonlinear fit using lmfit
body: I am a pandas user and contributor, but somewhat new to statsmodels. I tried your NonlienarLS for a few weeks, and I think it is incomplete. Specifically, I like the features in the lmfit project: parameters with bounds and mathematical constraints between them. I wrote a function to apply lmfit optimization on a DataFrame or Series.

I come to this as an experimental scientist, not a statistician, so maybe my needs are somewhat different that the goals of statsmodels. And maybe you are not interested in making lmfit an optional dependency. But consider reading [the docstring of my implementation](https://github.com/danielballan/mr/blob/master/mr/core/fitting.py#L60) and let me know if this could become a useful pull request.

Thanks.

user: danielballan
issue_858: state: open
title: Implement Propensity Score Matching
body: It would be nice to have a propensity score matching implementation. 
We can follow the description given at: 

Dehejia, Rajeev. “Practical Propensity Score Matching: a Reply to Smith and Todd.” Journal of Econometrics 125, no. 1–2 (March 2005): 355–364. doi:10.1016/j.jeconom.2004.04.012.
Dehejia, Rajeev H, and Sadek Wahba. “Propensity Score-matching Methods for Nonexperimental Causal Studies.” Review of Economics and Statistics 84, no. 1 (2002): 151–161.

Also,
"Other software packages promise: "Provides functions for multivariate and propensity score matching and for finding optimal balance based on a genetic search algorithm. A variety of univariate and multivariate metrics to determine if balance has been obtained are also provided."

For example:

http://cran.r-project.org/web/packages/Matching/index.html
http://cran.r-project.org/web/packages/MatchIt/index.html
http://cran.r-project.org/web/packages/optmatch/index.html
http://cran.r-project.org/web/packages/cem/index.html

but 
"None of this code is BSD compatible, so we can't just translate it,
but going through these packages' docs may help give an idea of what
is to be done."

user: jburroni
issue_852: state: open
title: Multiple Comparisons
body: In PR #849 I merged Tyler's enhancements to tukeyhsd into master (see also #809 #789 #824 )

some things left to do
- another round of proofreading docstrings, and cleaner example script
- add p-values to tukeyhsd results
- refactoring, some comments in the PRs and issue
- move out of sandbox
- unit tests for multicomp module (currently only tukeyhsd is really verified)

further enhancements
- plot for all pairs
- other pairwise comparisons 
  (Dunnet should be easy, proportions_pairwise needs better results class)
- more is a never-ending story 
  https://github.com/statsmodels/statsmodels/wiki/SMEP:-Multi-Comparisons

user: josef-pkt
issue_851: state: open
title: var_plots Datetimeindex
body: what's the pandas requirement for var_plots

with pandas 0.7., I get errors when running the doc build
(I didn't clean the plot directories before build)

```
  File "c:\python26\lib\site-packages\matplotlib\sphinxext\plot_directive.py", line 472, in run_code
    exec code in ns
  File "<string>", line 1, in <module>
  File "E:\Josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new2\statsmodels\docs\source\plots\var_plots.py", line 17, in <module>
    mdata.index = pandas.DatetimeIndex(quarterly)
AttributeError: 'module' object has no attribute 'DatetimeIndex'
E:\Josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new2\statsmodels\docs\source\vector_ar.rst:153: WARNING: Exception occurred in plotting var_plot_acorr
 from E:\Josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new2\statsmodels\docs\source\plots/var_plot_acorr.py:
Traceback (most recent call last):
  File "c:\python26\lib\site-packages\matplotlib\sphinxext\plot_directive.py", line 472, in run_code
    exec code in ns
  File "<string>", line 1, in <module>
  File "E:\Josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new2\statsmodels\docs\source\plots\var_plots.py", line 17, in <module>
    mdata.index = pandas.DatetimeIndex(quarterly)
AttributeError: 'module' object has no attribute 'DatetimeIndex'
```

user: josef-pkt
issue_850: state: open
title: Groups/Factors minimal size per level
body: (Just a thought, I don't know yet where it applies)

When we calculate based on groups or strata, then we need minimum size in the calculations (minimum size could be zero - keep empty levels, or one, or larger)

We need unit tests to check those cases.

user: josef-pkt
issue_847: state: open
title: stats inconsistent test options
body: weightstats is growing more options, functions and method.
- Since I wrote them with different applications in mind, some options still vary across functions.
- There is also still some duplicate code that should be refactored.
- across modules the naming for the alternative is not consistent (e.g. 2-sided, two-sided), sometimes allowing aliases
- ttest use two or three functions, ztest uses only one function for one-sample, two independent samples, and two paired sample.

latest changes are in PR #784
changes and addition for normal distributions where partially based on being able to use them for testing proportion, correlation, and other statistics where we use a normal approximation.

user: josef-pkt
issue_846: state: open
title: GAM prediction and splines
body: Hello,

I use GAM frequently in R (and as well through rpy2). I am looking for a pythonic implementation to get rid of rpy2 and all memory issues linked to to rpy2.

I checked the gam implementation (not tested yet), but I am missing some functionality to be able to use this implementation.
1. Prediction function. Once the gam model is build a new set of x-data should be processed to result in an prediction result (like gam.predict in R).
2. Possibility to retrieve the smoothed splines, to report or implement them independently (not that easy in R as well).

Keep on doing the nice work !!!!

Thanks,

Luc

user: ldkphd
issue_844: state: open
title: Docs: docstrings for attributes (not possible)
body: Attributes cannot have a docstring themselves in python

http://stackoverflow.com/questions/3051241/how-to-document-class-attributes-in-python

In the model we work largely with descriptors and properties which can hold their own docstring.

I'm writing a results class for a statistical test that just assigns all results as attributes. The only way to document them is in the class docstring, unless we want to convert them all to properties.

(candidate for a statsmodels docstandard in developer docs, when we have one, and we settle on a pattern)

user: josef-pkt
issue_842: state: open
title: Class factory for NegativeBinomial
body: This will also apply to Tobit (and maybe ARIMA). I'm starting to like more that most options go into the Model instantiation and the fit methods are as uniform as possible. Could possible help in writing Pipeline-style code for bootstrapping etc. like scikit-learn, though I need to think more about whether and how it generalizes. Something like

```
class _NegativeBinomialBase(LikelihoodModel):
    ...

class NegativeBinomialNB1(_NegativeBinomialBase):
    def loglike(self, params):
        ...

...

def NegativeBinomial(..., loglike_method="nb1"):
    if loglike_method == "nb1":
        return NegativeBinomialNB1(...)
```

user: jseabold
issue_839: state: open
title: garbage collection instance methods
body: see thread at http://mail.scipy.org/pipermail/scipy-dev/2013-May/018823.html

assigning an instance method in the `__init__` of a class creates a reference cycle which prevents python from immediately garbage collecting an instance that is not referenced anymore. 

We have this for example in an external bootstrap or model choice loop.

Currently, the only class where we still do this is discrete.NegativeBinomial, I think.

user: josef-pkt
issue_837: state: open
title: MNLogit: choice of reference category
body: currently the first category in alphanumerical order is used as reference category in MNLogit.

Users can change their labels or categories to get the desired reference category as the first in alphanumerical order, but it would be more convenient to have an option for this.

Related: using formulas: patsy automatically converts string arrays to dummy variables, which are not handled by MNLogit for endog

see discussion at https://groups.google.com/d/msg/pystatsmodels/6Aj0YJChsmI/lBAQqtPBnpYJ

user: josef-pkt
issue_827: state: open
title: Error in logistic regression with "cloglog"
body: In trying to reproduce the code results from Dobson & Barnett "An
Introduction to GLMs" in Python, I have run into an inconsistency between
the results from R and from Python. While the results for logistic
regression with statsmodels match the R-results for the logit and probit
link functions, the results for the cloglog link are inconsistent.

I have looked at the Python code in statsmodels, and it seems correct to me,
so I am a bit dumbfounded.
Also, the statsmodels link only works for "cloglog", but crashes for
"CLogLog".
## Python Code:

```
'''Solution to Exercise 7.3.1 (Beetle Mortality) in Dobson & Bartlett:
"Introduction to GLMs"

author: thomas haslwanter
date:   may 2013
ver:    1.0

'''

# Standard libraries
import numpy as np
import pandas as pd
import statsmodels.api as sm

if __name__ == '__main__':
    data = np.array([[1.6907, 59, 6],
        [1.7242, 60, 13],
        [1.7552, 62, 18],
        [1.7842, 56, 28],
        [1.8113, 63, 52],
        [1.8369, 59, 53],
        [1.861, 62, 61 ],
        [1.8839, 60, 60]])

    df = pd.DataFrame(data)
    df.columns = ['x', 'n', 'y']

    # fit the model
    df['tested'] = df['n']
    df['killed'] = df['y']
    df['survived'] = df['tested'] - df['killed']

    model_logit = sm.GLM.from_formula('survived + killed ~ x', data=df, family=sm.families.Binomial(sm.families.links.logit)).fit()
    print 'Logit:'
    print df['n']*(1-model_logit.fittedvalues)

    model_cll = sm.GLM.from_formula('survived + killed ~ x', data=df, family=sm.families.Binomial(sm.families.links.cloglog)).fit()
    print 'CLogLog:'
    print df['n']*(1-model_cll.fittedvalues)
```
## Python Results:

Logit:
0     3.457461
1     9.841672
2    22.451378
3    33.897635
4    50.095822
5    53.290913
6    59.222159
7    58.742961
CLogLog:
0     2.365465
1    12.557569
2    27.787944
3    36.431779
4    49.561865
5    51.380562
6    57.100959
7    57.057706
## R code:

```
y=c(6,13,18,28,52,53,61,60)
n=c(59,60,62,56,63,59,62,60)
x=c(1.6907,1.7242,1.7552,1.7842,1.8113,1.8369,1.8610,1.8839)
n_y=n-y
beetle.mat=cbind(y,n_y)
res.glm1=glm(beetle.mat~x, family=binomial(link="logit"))
res.glm3=glm(beetle.mat~x, family=binomial(link="cloglog"))
fitted.values(res.glm1)*n
fitted.values(res.glm3)*n
```
## R results:

> fitted.values(res.glm1)_n
>         1         2         3         4         5         6         7         8 
>  3.457461  9.841672 22.451378 33.897635 50.095822 53.290913 59.222159 58.742961 
> fitted.values(res.glm3)_n
>        1        2        3        4        5        6        7        8 
>  5.58945 11.28068 20.95422 30.36944 47.77642 54.14273 61.11331 59.94723 

---

For completeness, I also attach the outputs of the Python and R model parameters: they are the same for logit, but
different for cloglog:

```
                 Generalized Linear Model Regression Results                  
==============================================================================
Dep. Variable:      ['I(n - y)', 'y']   No. Observations:                    8
Model:                            GLM   Df Residuals:                        6
Model Family:                Binomial   Df Model:                            1
Link Function:                  logit   Scale:                             1.0
Method:                          IRLS   Log-Likelihood:                -18.715
Date:                Sat, 11 May 2013   Deviance:                       11.232
Time:                        19:09:38   Pearson chi2:                     10.0
No. Iterations:                     7                                         
==============================================================================
                 coef    std err          t      P>|t|      [95.0% Conf. Int.]
------------------------------------------------------------------------------
Intercept     60.7175      5.181     11.720      0.000        50.563    70.871
x            -34.2703      2.912    -11.768      0.000       -39.978   -28.563
==============================================================================
                 Generalized Linear Model Regression Results                  
==============================================================================
Dep. Variable:      ['I(n - y)', 'y']   No. Observations:                    8
Model:                            GLM   Df Residuals:                        6
Model Family:                Binomial   Df Model:                            1
Link Function:                cloglog   Scale:                             1.0
Method:                          IRLS   Log-Likelihood:                -27.058
Date:                Sat, 11 May 2013   Deviance:                       27.917
Time:                        19:09:38   Pearson chi2:                     25.1
No. Iterations:                     8                                         
==============================================================================
                 coef    std err          t      P>|t|      [95.0% Conf. Int.]
------------------------------------------------------------------------------
Intercept     37.5589      2.943     12.764      0.000        31.791    43.326
x            -21.5240      1.676    -12.843      0.000       -24.809   -18.239
==============================================================================
```

and for R:
    > res.glm1

```
Call:  glm(formula = beetle.mat ~ x, family = binomial(link = "logit"))

Coefficients:
(Intercept)            x  
     -60.72        34.27  

Degrees of Freedom: 7 Total (i.e. Null);  6 Residual
Null Deviance:      284.2 
Residual Deviance: 11.23        AIC: 41.43 
> res.glm3

Call:  glm(formula = beetle.mat ~ x, family = binomial(link = "cloglog"))

Coefficients:
(Intercept)            x  
     -39.57        22.04  

Degrees of Freedom: 7 Total (i.e. Null);  6 Residual
Null Deviance:      284.2 
Residual Deviance: 3.446        AIC: 33.64 
> 
```

user: thomas-haslwanter
issue_826: state: open
title: precision cloglog
body: My guess is that cloglog link should use scipy.special.log1p and scipy.special.expm1 to get better precision when p is very small

```
>>> from scipy import special
>>> special.log1p(-1e-10)
-1.00000000005e-10
>>> np.log(1-1e-10)
-1.000000082790371e-10

>>> 1 - np.exp(-1e-10)
1.000000082740371e-10
>>> - special.log1p(-1e-10)
1.00000000005e-10
```

user: josef-pkt
issue_825: state: open
title: ANOVA for GLM
body: anova_lm is NOT supported by GLM
e.g. glm_model = sm.GLM.from_formula('weight~group', family=sm.families.Gaussian(), data=df).fit()

but is supported by OLS
e.g. model = sm.OLS.from_formula('weight~group', data=df).fit()

It would be nice (and more consistent with R) if it were possible to define the ANOVA such that both models are supported (perhaps by using the deviance instead of ssr).

If there are no changes in the implementation, at least the docu of anova_lm should mention that it only works for OLS, not for GLM.

user: thomas-haslwanter
issue_815: state: open
title: Clean up VAR
body: Still a work in progress. Likely take another week or two given I have other obligations this week.
## Renames/deprecations

sigma_u -> cov_resid
sigma_u_mle -> cov_resid_mle
endog -> Y
endog_lagged -> X
detomega -> det_cov_resid
_chol_sigma_u -> _chol_cov_resid
_zz -> _XTX
_cov_alpha -> _cov_params_ex_trend
_cov_sigma -> _cov_cov_resid
intercept -> trend_coefs

Only in call signatures and internally
omega -> cov_resid (or cov_resid_mle depending on context)

For VARX:
X <- contains RHS variables
exog <- contains all exogenous variables

signature for VARX
VAR(Y, ..., exog)

Improves the documentation a bit, though there's still some work to do. Added the VAR notebook. Cleaned up some plotting. Add some functions to the statsmodels.tsa.var namespace through var.api.
## Closed Issues
#99 - Added conf_int with wrappers.
#380 - Fix IRF plotting settings.
#769 - Stale model issue. Issue exists but behavior raises a warning and is deprecated. Added select_order_fit, deprecated maxlags and lag order selection in methods except for select_order, which is now a class method of VAR.
#757 - Legend in forecast plots.
#756 - Initial values in forecasting, addressed. Use of y is deprecated forecast function is not imported into var namespace, if someone needs it.
#234 - cov_params with wrapper.
## TODO
- [ ] #142 - Forecasts for trend != 'c' and tests.
- [ ] #379 - Causality tests (also normality and whiteness)
- [ ] #671 - In progress. t_test, f_test
- [x] #136 - Set verbose to false.
- [ ] #435 - VAR Predict method
- [ ] Refactor SVAR

Turn all the smoke tests into real tests. 
Clean up the var.rst for refactor. 

user: jseabold
issue_813: state: open
title: VAR forecast should allow for a `start_date` keyword
body: so it doesn't plot the whole series if you're really only interested in behavior towards the end.

user: jseabold
issue_812: state: open
title: test coverage: statsmodels\stats\gof  no unittests
body: `powerdiscrepancy` and other functions in `gof` have no unit tests

I never found an R function for it. `powerdiscrepancy` was partially checked against a book.

user: josef-pkt
issue_811: state: open
title: FAQ/FAQ-D: statistical tests: sign convention for 2-sample comparison
body: **edit: summary**

current convention for two sample tests considers "sample2" as the reference case (reference case is the second sample)
with signature `function(sample1, sample2, value=....)`

```
H0: stat(sample1) - stat(sample2) - value = 0
versus
H1:  stat(sample1) - stat(sample2) - value != 0
```
and analogous one-sided and TOST versions

variation for ratio

```
H0: stat(sample1) - stat(sample2) * value = 0
versus
H1:  stat(sample1) - stat(sample2) * value != 0
```
or
```
H0: stat(sample1) / stat(sample2)  - value = 0
versus
H1:  stat(sample1) / stat(sample2)  - value != 0
```

-------------------
**original comment**
stat_test(sample1, sample2) or
stat_test(sample0, sample1)

do we use
`stat(sample1) - stat(sample2)`
or `stat(sample2) - stat(sample1)`
or `stat(sample1) - stat(sample0)`    (gof.chisquare_effectsize)

I just changed proportion.proportion_effectsize to the first version to match R pwr.

This needs a consistency check and possible refactoring across all statistical tests.

`chisquare_effectsize(p0, p1)` is the same sequence as R pwr, but looks reversed to me compared to proportions effects size

It's also for one sample comparison, where one of them is the hypothesized value.
In two sample tests, we also have cases with `sample1, sample2, value`, where the null hypothesis is
`stat(sample1) - stat(sample2) - value = 0`    (where `value` is sometimes also called `diff` .)

user: josef-pkt
issue_808: state: open
title: RLM needs influence statistics
body: Recreated the influence statistics for a notebook, but RLM should have weighted influence stats.

```
from statsmodels.formula.api import rlm
dta = sm.datasets.statecrime.load_pandas().data

rob_crime_model = rlm("murder ~ urban + poverty + hs_grad + single", data=dta, 
                      M=sm.robust.norms.TukeyBiweight(3)).fit(conv="weights")

weights = rob_crime_model.weights
idx = weights > 0
X = rob_crime_model.model.exog[idx]
ww = weights[idx] / weights[idx].mean()
hat_matrix_diag = ww*(X*np.linalg.pinv(X).T).sum(1)
resid = rob_crime_model.resid
resid2 = resid**2
resid2 /= resid2.sum()
nobs = int(idx.sum())
hm = hat_matrix_diag.mean()
rm = resid2.mean()
```

user: jseabold
issue_799: state: open
title: rank based multiple tests
body: looks interesting, rank based k-sample tests similar to gof tests

see also reference to Gao Alvo and Gao Alvo Chen Li 2008
rank-based one-way analysis, k-sample comparisons

Rank-based multiple test procedures and simultaneous confidence intervals
Frank Konietschke, Ludwig A. Hothorn, and Edgar Brunner
Source: Electron. J. Statist. Volume 6 (2012), 738-759. 

http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.ejs/1336049813

user: josef-pkt
issue_798: state: open
title: statistics for cov_params
body: http://biogeme.epfl.ch/doc/01logitpython.html

shows a table with statistics for the covariance matrix, including both standard and robust covariances

might be interesting, I've never seen it before

user: josef-pkt
issue_790: state: open
title: multiple comparison Sheffe
body: Sheffe's method also seems to have quite a good reputation
http://en.wikipedia.org/wiki/Scheff%C3%A9%27s_method

user: josef-pkt
issue_788: state: open
title: ENH: errorbar argument for interaction_plot function
body: I'd like to say things like:

`interaction_plot(x, trace, y, error_func=scipy.stats.sem)`

For each of the cells error bars would then be drawn as obtained by applying what is passed via the `error_func` parameter. This should be quite easy to get. 
What do people think?

user: dengemann
issue_786: state: open
title: WIP: repanova
body: Initial work-in-progress code for repeated-measures ANOVA.

user: rogerlew
issue_782: state: open
title: emplike\originregress essentially zero test coverage
body: I'm checking which modules have low test coverage.

originregress does not have unit test (according to test coverage report)

what's the status of this?

user: josef-pkt
issue_771: state: open
title: weighted stats class for pandas objects
body: Might be nice to support pandas objects in DescrStatsW with some kind of wrapper.

user: jseabold
issue_769: state: open
title: Stale model with VAR
body: Another ones of these.

```
def get_macrodata():
    data = sm.datasets.macrodata.load().data[['realgdp','realcons','realinv']]
    names = data.dtype.names
    nd = data.view((float,3))
    nd = np.diff(np.log(nd), axis=0)
    return nd.ravel().view(data.dtype)

import statsmodels.api as sm
from statsmodels.tsa.vector_ar import util
data = get_macrodata()
mod = sm.tsa.VAR(data)
res = mod.fit(2)

res2 = mod.fit(10, ic='fpe')

print res2.model.exog_names
# this is wrong
print res.model.exog_names
print res.exog_names
```

What to do? With ARIMA we moved the order to the model instantiation and we don't have lag-order selection for ARMA. My inclination would be to do the same for VAR and make the lag-order selection a function. The other thing I thought of is to make a Model.copy() and return a copy of the model attached to VARResults, but this might be too heavy. 

user: jseabold
issue_757: state: open
title: VAR plotting legends
body: The legend for forecast_interval needs some work.

http://statsmodels.sourceforge.net/devel/vector_ar.html#forecasting

user: jseabold
issue_756: state: open
title: VAR Forecasting requires initial values
body: The VAR forecast method 1) requires initial values when by default it should use what we have available in the model and 2) doesn't allow the passing of pandas objects. In fairness, I'm not sure how many of our predict/forecast methods allow pandas objects. Need to check.

user: jseabold
issue_745: state: open
title: margins and tests for variables that are more than once in exog
body: What do we need to calculate margins, dy/dx (x), and tests (f_test, t_test, Wald test) with respect to a variable that enters in more than one terms?

examples:
- interaction of categorical variables (handled by patsy ?)
- splines and similar (more my area)

example linear spline (segmented regression) and maybe polynomial:
margins: slope at different points (stateful transform should define derivative ? right derivative in at kinks)
tests: slope is zero or constant over an interval, at a point ?

If there are other exog, then we also need to know how columns match to "terms", which columns represent out variable, and what are the other variables (at mean, ...)

user: josef-pkt
issue_743: state: open
title: caseweights in linear models
body: (I'm opening this specifically for linear models, because in WLS we have potentially different interpretation of weights.)

Using WLS with weights as case weights, I cannot replicate what we would get with expanding the arrays to all observations (np.repeat) or with the t-test in stats.weightstats which uses that caseweight interpretation.

We still need to get the interpretation of weights cleared up, Stata has 3 kinds, SAS in some formulas has two kinds.

specific case: in a simple two-sample mean comparison, WLS with weights as caseweights produces the same estimate for the weighted means and mean differences, but not the correct t-values.   (not "correct" for the case weight interpretation)

both RLM and GLM might need extra support from WLS if their weights definitions are expanded.

related: 
- PR #504 with discussion in #501 made changes to match some WLS results to Stata and R, #504 left GLS as an open issue
- issue #505 is a general issue to check and make it consistent everywhere
- issue #481 asks specifically for case weights for maximum likelihood models

---

links to Stata:
case weights are called frequency weights, while WLS test suite is written for aweights
http://www.stata.com/help.cgi?weights
also http://www.ats.ucla.edu/stat/stata/faq/weights.htm

description for aweights http://www.stata.com/support/faqs/statistics/analytical-weights-with-linear-regression/

user: josef-pkt
issue_742: state: open
title: ENH ContrastResults
body: the ContrastResults for f_test and t_test still look awful

for t_test results we should also get confidence intervals so it matches the params table

```
>>> tt = res.t_test(cc)
>>> print summary_params((tt, tt.effect, tt.sd, tt.tvalue, tt.pvalue, np.ones((4,2))), yname='slopes', xname=['seg>%d'%ii for ii in range(4)])
==============================================================================
                 coef    std err          t      P>|t|      [95.0% Conf. Int.]
------------------------------------------------------------------------------
seg>0          0.7054      0.059     11.964      0.000         1.000     1.000
seg>1          0.3199      0.089      3.604      0.000         1.000     1.000
seg>2          0.9765      0.062     15.737      0.000         1.000     1.000
seg>3          1.0327      0.076     13.572      0.000         1.000     1.000
==============================================================================
>>> 
```

**update**
The table for t_test results has been added in PR #1189
- add multiple testing p-value correction, see comment below for Stata
  - for the table it would be better to drop the confidence interval and add columns with corrected p-values
  - one usecase: pairwise comparison of levels within a term. (I have experimental version.)

user: josef-pkt
issue_738: state: open
title: caseweights when test function doesn't support it
body: weightstats explicitly calculates statistics with taking weights into account, the "repeat" array is never used.

I dropped levene as method, because scipy.stats.levene doesn't support weights. The cheap (in terms of code) way out is to create the full repeat array and use it for those functions.

SPSS manual describes that they are doing this (after rounding of the weights to integers).

possible problem: because of the rounding nobs of expanded array might not be the same as round(sum(weights)) if weights are not integers. Maybe add or take away one random observation.

user: josef-pkt
issue_720: state: open
title: documenting _results, getting through the wrapper
body: If a developer wants ndarrays in the results, then it's possible to access them through `._result`.

This should be documented in the developer docs somewhere.

user: josef-pkt
issue_718: state: open
title: backdoors, fit new y
body: we had a long thread about whether endog/Y should be in the `__init__` or in the `fit` method.
https://groups.google.com/d/msg/pystatsmodels/Fkwoqbgmey8/c2F4CO0muRkJ

We didn't change the method where engog is given, because of too much working avoiding stale state, but we changed that pinv_wexog is cached.
I also added a section to the documentation about "stale" state in repeated calls to `fit` with different arguments.

During the first GSOC we had the discussion about creating instances (and whether it's cheap) and decided to create a new instance for each WLS iteration in GLM and RLM.

However with the wrapper and checking code in `__init__`, the instance creation is not so cheap anymore. (no actual timing)

Currently, Vincent is adding quantile regression with an `inline` iteratively reweighted least squares loop.
In GLSHet, I call the `fit` method repeatedly without creating new instances.

I think we should add a semi-official method for fitting with new y. (semi-official: public but add Warnings in docstring about proper use.), along the lines of this
https://groups.google.com/d/msg/pystatsmodels/Fkwoqbgmey8/zxjbXnc_n4kJ
and GLSHet, and GLSAR iterative loop.

Besides the nipy use case, which is actually multivariate or panel model, the main usecase would be parametric bootstrap.

(BTW: PanelAR1 should handle the nipy usecase in a vectorized way, but there might not be a way to estimate with fixed rho.
edit: No, it doesn't. It's not truly mulivariate endog with same exog, it either assumes different exog or/and identical parameters across panel-units.)

user: josef-pkt
issue_717: state: open
title: GLSAR  users are supposed to call iterative_fit ?   API
body: I just looked at GLSAR docstring. It still says experimental.

fit of the super class GLS is not overwritten. The only way to estimate the AR is when iterative_fit is called. ???

There are several other tickets for GLSAR.

finish up

user: josef-pkt
issue_705: state: open
title: panel data multiway error component
body: Davis, Peter. 2002. “Estimating Multi-way Error Components Models with Unbalanced Data Structures.” Journal of Econometrics 106 (1) (January): 67–95. doi:10.1016/S0304-4076(01)00087-2.
http://www.sciencedirect.com/science/article/pii/S0304407601000872

lots of linear algebra (instead of loops) without calculating (n, n) matrices

The creation of sparse indicator/dummy matrices is in grouputils. Needs many helper functions to work through the actual linear algebra calculations.

related PR #690 and issue #646 

related for two way random effects

Baltagi, Badi H., Seuck H. Song, and Byoung C. Jung. 2002. “A Comparative Study of Alternative Estimators for the Unbalanced Two-way Error Component Regression Model.” Econometrics Journal 5 (2): 480–493. doi:10.1111/1368-423X.t01-1-00094.
http://onlinelibrary.wiley.com/doi/10.1111/1368-423X.t01-1-00094/full

user: josef-pkt
issue_701: state: open
title: bug, pylint vector_ar irf
body: line 379 and 447    `svar` not defined

```
        elif svar:
            irfs = self.svar_irfs
```

From a quick reading both methods should raise errors, not tests, no examples ?

---

second bug

svar_model line 734: pylint opt_B on line 738 not defined

```
                        opt_A = mean_AB[:split]
                        opt_A = mean_AB[split:]                      # should be opt_B   ???
                    ma_coll[i] = SVAR(sim, svar_type=s_type, A=A_pass,
                                 B=B_pass).fit(maxlags=k_ar,\
                                 A_guess=opt_A, B_guess=opt_B).\
                                 svar_ma_rep(maxn=T).cumsum(axis=0)
```

looks like easy fix, but no tests ???

user: josef-pkt
issue_699: state: open
title: paired t-test with correlation
body: In the power calculations for paired t-test some articles explicitly consider correlation.

Does it matter for the paired t-test?
ENH: see where it should be added to the power calculations.

user: josef-pkt
issue_697: state: open
title: Rethinking the class hierarchy - base models - estimation methods
body: It's not clear what the super class for quantile regression should be. PR #655 
Also, kernel regression does not yet have a results class. What would be the most appropriate base class.

LikelihoodModelResults defines the f_test, t_test, ... which are based on the (asymptotic) normality of the estimated parameters. We get that also in many models that don't define a likelihood, llf, and associated attributes and methods, e.g.
`RLM.loglike: raise NotImplementedError`

I think we should have "cleaner" basemodels that are specific to an estimation method, so we don't have to overwrite methods or attributes as `NotImplementedError` (which should actually be a `DoesNotApplyError`)

for example:
- move methods that are based on distribution of params to the base Model
- create specific base method classes, Likelihood, M-Estimation, GMM, ...
- move all attributes and methods to the appropriate base method class, e.g. likelihoodratio test
- inherit from the right base method classes, eg. robust from MEstimationResults not from LikelihoodModelResults

my guess is that we can expand M-Estimation to quantile regression and minimum distance estimators, not just "traditional" robust estimators.

user: josef-pkt
issue_696: state: open
title: quantile regression is an M-estimator
body: based on the first few pages of
http://onlinelibrary.wiley.com/doi/10.1111/j.1467-9868.2012.01044.x/abstract
Zhou, Zhou, and Xiaofeng Shao. 2013. “Inference for Linear Models with Dependent Errors.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 75 (2): 323–343. doi:10.1111/j.1467-9868.2012.01044.x.

The structure of quantile regression in PR #655 is very close to RLM (reweighted least squares).

There might be some parts where we can exploit the relationship.

For example, RLM assumes independent errors. What's the equivalent of a HAC covariance for RLM.  
Vincent added various kernel estimators to quantile regression (following Stata). Do these have wider application?

user: josef-pkt
issue_693: state: open
title: correlation_structures
body: unfinished cleanup (errors, undefined variables, imports) after copy paste in 
statsmodels\sandbox\panel\correlation_structures.py

mainly in the AR versions

user: josef-pkt
issue_688: state: open
title: grouputils - unfinished, errors
body: see also #660 #687 

this was partially written for sandwich_covariance and for short_panel to keep internally track of the indices.

I added methods and functions as I needed them, but there are several unfinished methods with errors in grouputils.

clean up

user: josef-pkt
issue_687: state: open
title: sandwich covariance code cleanup
body: The internal code of sandwich_covariance hasn't been given a cleanup review.

Besides inconsistent function arguments (indices), the code hasn't been updated to use grouputils and some optimized functions.

The test coverage is also missing some options in the robust covariance functions.

user: josef-pkt
issue_684: state: open
title: interrater, collection of tests and measures
body: nipy has intraclass correlation coefficient
https://github.com/nipy/nipype/pull/340

the original paper and some matlab fileexchange version have 6 versions of ICC, IIRC.

fleiss' kappa is still missing the standard deviation and other inference statistics

see #591 for first part: Cohen's kappa

user: josef-pkt
issue_682: state: open
title: _isdummy function in discrete_model.py can incorrectly identify dummy variables under certain condition
body: Looking at full version 0.5.0.dev-1bbd4ca, the current code identifies a dummy variable when it has only 1 and 0 values.  A particular cross-validation or sample of data might result in a categorical variable that happens to always be 0 or always be 1. _isdummy will incorrectly classify the variable in that case.

An alternative, and arguably more pythonic, logic might be:

np.all( (X==1) | (X==0), axis=0)

This may also eliminate the need for calculating the remainder variable as the code currently does.

user: upandacross
issue_678: state: open
title: taylor diagram ?
body: A plot to compare performance of different models.

Would this be interesting for statsmodels?

description
http://www-pcmdi.llnl.gov/about/staff/Taylor/CV/Taylor_diagram_primer.htm

matplotlib implementation
http://old.nabble.com/Taylor-diagram-(2nd-take)-p33364690.html

(I'm just saw this through a question on the matplotlib mailing list. I've never heard of Taylor diagram.)

user: josef-pkt
issue_671: state: open
title: VAR missing t_test, f_test
body: I don't see t_test, f_test in the VAR results.

see also issue #99 

We need to systematically check that we have all the "usual" methods and attributes.

user: josef-pkt
issue_667: state: open
title: ENH: statistical tests for variances, and covariance matrices
body: There is a large list of one to k sample tests on variances, and covariances.

IIRC then they are all missing except for Levene and Bartlett in scipy.stats.

Do we have F_test for variances? http://en.wikipedia.org/wiki/F-test_of_equality_of_variances
others for exampe:
http://en.wikipedia.org/wiki/Hartley%27s_test

For covariances and correlations there are other tests that are mostly missing.

sphericity tests for traditional repeated measures ANOVA?

user: josef-pkt
issue_666: state: open
title: sample size determination for fractions, ratios or quantiles
body: general issue: we want to get the distribution of quantiles, order statistics
scipy has binom_test
I mentioned it briefly in the discussion on qqplot improvement

specific:
we need selection criteria for the number of bootstrap or permutation samples, and an error estimate on the result (quantile, p-value) that we get from the bootstrap. 
SAS has it, I have seen some references for bootstrap (where ?) and the basic formula is simple but missing.

user: josef-pkt
issue_665: state: open
title: robust covariance estimators for nonlinear regression
body: I could use a heteroscedasticity robust cov_params for non-linear least squares.

Can we just plug the jacobian into the formula for the linear model?
find references

In the MLE case, is heteroscedasticity captured by the sandwich cov_hjjh ?

user: josef-pkt
issue_664: state: open
title: models with interval censored, binned data
body: If we only observe discretized `endog` for an underlying continuous distribution, then we can use MLE with the pmf based on the bins, `diff(cdf(bin_edges))`.

This is relatively easy to do for distributions that have a cdf that is easy to calculate.

From the observed structure it's a discrete distribution, the underlying model is continuous.

I think this could go into `statsmodels.discrete`. It's very similar to count models, except count models impose/assume non-negativity of the observed endog.

user: josef-pkt
issue_660: state: open
title: groups, grouputils, pandas, patsy
body: How to get a group or categorical variable as integer index from pandas:
see https://groups.google.com/d/topic/pystatsmodels/1emtXMNm_iY/discussion
patsy has the code to write the version check for pandas > 0.9

grouputils and groups at a few other places (e.g. k-sample tests) use integer arrays to identify levels or individuals/groups in a categorical.
None of them are currently connected to pandas or patsy.

user: josef-pkt
issue_659: state: open
title: Decrease abolute tolerance in allclose to detect perfect separation earlier
body: Toy example parked for consideration. Haven't thought about or looked at this.

```
yes = [1, 1, 0, 0.]
data = [[1, 0, 1],[1, 0, 1],[1, 2, 1],[1, 1, 1.]]
sm.Logit(yes, data).fit(method="bfgs") # warning, why are we inverting when we have analytic results?
sm.Logit(yes, data).fit(method="powell") # PerfectSeparationError
sm.Logit(yes, data).fit() # LinAlgError
```

**Edit**: The last now raises a PerfectSeparationError. BFGS still does not.
user: jseabold
issue_654: state: open
title: DOC: `incomplete` examples, was: review GLSAR example
body: PR #652 is doing some pep-8 cleaning of example
there are some left over code comments

example_glsar needs a review

I did some rewriting of GLSAR, but might not have checked the doc example

user: josef-pkt
issue_653: state: open
title: documentation: add brief description of general features of models
body: I think we should add a new section with a brief descriptions features that are the same across models.

The case I have in mind is the description of the new nan/missing value handling. 
It's described in the docstrings (I guess), but I would like to add a section BEWARE to it.
The advantages of cleaning first: You need to do it only once, if the same model is estimated repeatedly (no repeated checking and array copying). When we run regression with different number of exog variables/columns, then the number of observations can differ after removing missing observations in each exog. This is good in some cases, but not when we want to compare (nested) models.

other features ?
- pickle for prediction :? 
- constant handling : cleaned  up in 0.5
- t versus normal distribution in tests/postestimation inference

I don't remember if or where those are already described.

user: josef-pkt
issue_650: state: open
title: documentation: add topics/comparison pages
body: Compared to several other packages, sklearn for example, our documentation is still too much "library" style (like numpy, scipy)

see issue #635 for some discussion and suggestions

Currently, we have very few pages that present and compare different estimators that solve a similar problems. We have some of it in the examples that Vincent rewrote (e.g. OLS versus GLS versus RLM), and there are some overview pages (my diagnostic tests summary which is rather deep down in the toc tree.)

scikit-learn has overview pages also as module docstring (AFAICS), however this would not be useful for us to compare linear model versus rlm, or discrete model versus glm.

some of it could be done by expanding the examples and add more structure on top of it.

What would also be useful is a structured cross-index by statistical task or data. This should just be a commented index  without duplicating much of the information.
Good example is also the UCLA index, examples and FAQ.

examples: 
binary data: discrete Logit, Probit, GLM several links
time series prediction: ?
linear model and assumptions (more like the econometrics textbook structure plus RLM)
categorical analysis : now covered with examples

This should also link to relevant statistical tests and graphics.
(It would also help to identify gaps in what we have.)

Similar, I floated once the idea of a FAQ in the documentation, but nothing has happened yet. (and we don't get many faq's)

Caveat: 
Sounds like a lot more work than current manpower, and will also be work to keep maintained and updated.

user: josef-pkt
issue_647: state: open
title: GLSHet  status ?
body: What's the status with GLSHet?
It's not in the docs AFAICS.

there is issue #181 

similar are all options for GLSAR included?

no tests, there is one example file

Stata: as far as I have seen, Stata only has `arch` for this case, which uses MLE as estimator .
R: I haven't figured out what's similar, maybe an option/formula to `lm`

bias-correction: Greene mentions a less biased or bias corrected version for log link in the var estimation
    not implemented

user: josef-pkt
issue_646: state: open
title: ENH: mixed effects / repeated measures
body: How is this evolving in the moment? Looking at the sandbox there does not seem to be too much change.
I've got a couple of use cases for which I ideally need to report 'repeated measures' analysis
Also for quite some time I've been thinking about implementing a general solution applicable to to cognitive neuroscience / physiological data.
So typically it should handle scenarios like: Y ~ within_a x within_b x between_j. Y can be button press latencies, error rates, etc., or brain responses. In that case Y has to be estimated at location l,  time t (and frequency f).
I might give this a shot if no one is deeply devoted to this during the next weeks or so.

user: dengemann
issue_635: state: open
title: developer docs - how to add a new model
body: It came up in discussions a few times, but we don't have any explicit documentation how to write a new model class for statsmodels.

some information is spread across mailing list, current documentation and examples, but a dedicated section in the developer documentation would be more useful for new contributors.

user: josef-pkt
issue_632: state: open
title: missing attributes/methods in subclasses
body: We have several backwards incompatible changes in 0.5, it might be possible to make it less so. 
For example by defining default attributes and methods.

Some adjustments are missing in the sandbox/examples, and they currently raise exceptions.

I haven't looked into the following yet. It could also be that they don't inherit correctly

```
Traceback (most recent call last):
  File "C:\Programs\Python33\Lib\site-packages\statsmodels-0.5.0-py3.3-win32.egg\statsmodels\sandbox\examples\example_sysreg.py", line 47, in <module>
    grun_res = grun_mod.fit()
  File "C:\Programs\Python33\lib\site-packages\statsmodels-0.5.0-py3.3-win32.egg\statsmodels\sandbox\sysreg.py", line 234, in fit
    sur_fit = SysResults(self, beta, self.normalized_cov_params)
  File "C:\Programs\Python33\lib\site-packages\statsmodels-0.5.0-py3.3-win32.egg\statsmodels\sandbox\sysreg.py", line 373, in __init__
    normalized_cov_params, scale)
  File "C:\Programs\Python33\lib\site-packages\statsmodels-0.5.0-py3.3-win32.egg\statsmodels\base\model.py", line 947, in __init__
    super(LikelihoodModelResults, self).__init__(model, params)
  File "C:\Programs\Python33\lib\site-packages\statsmodels-0.5.0-py3.3-win32.egg\statsmodels\base\model.py", line 772, in __init__
    self.initialize(model, params, **kwd)
  File "C:\Programs\Python33\lib\site-packages\statsmodels-0.5.0-py3.3-win32.egg\statsmodels\base\model.py", line 778, in initialize
    self.k_constant = model.k_constant
AttributeError: 'SUR' object has no attribute 'k_constant'
```

```
Traceback (most recent call last):
  File "C:\Programs\Python33\Lib\site-packages\statsmodels-0.5.0-py3.3-win32.egg\statsmodels\sandbox\examples\ex_mixed_lls_re.py", line 152, in <module>
    print(res.t_test([1]))
  File "C:\Programs\Python33\lib\site-packages\statsmodels-0.5.0-py3.3-win32.egg\statsmodels\base\model.py", line 1134, in t_test
    LC = DesignInfo(self.model.exog_names).linear_constraint(r_matrix)
AttributeError: 'OneWayMixed' object has no attribute 'exog_names'
```

user: josef-pkt
issue_628: state: open
title: nonparametric Alternatives to product kernels
body: In the discussion on the nonparametric work of George and Ralph's GSOC, I mentioned elliptical multivariate kernels (like scipy's gaussian_kde) as alternative to the product kernels with independent bandwidth. (location ?)

By chance I found a good reference:

http://sfb649.wiwi.hu-berlin.de/fedc_homepage/xplore/ebooks/html/spm/spmhtmlnode18.html

This looks like a good discussion of "radial-symmetric" (elliptical) versus product kernel with diagonal and full covariance matrices.

user: josef-pkt
issue_624: state: open
title: ENH: parameterized link functions for GLM, discrete, 
body: example Box-Cox transformation with unknown parameter

reference (that I just found again)
A. Scallan,  R. Gilchrist,  M. Green "Fitting parametric link functions in generalised linear models"
http://www.sciencedirect.com/science/article/pii/0167947384900318

see also citations (13 only)
http://scholar.google.ca/scholar?cites=2729147818703607070&as_sdt=2005&sciodt=0,5&hl=en

user: josef-pkt
issue_614: state: open
title: nonparametric: TestRegCoefC  need checking, better tests
body: We should return the p-value and attach relevant variables (I added the bootstrap t_dist in my branch)

My first example reports "Not Significant" even though the parameters are large. Another run has significant at 10% without pivot, and at 1% with pivot. looks fine.
The test statistic and the bootstrap distribution have small values in my example. Should there be a normalization also in the pivot=False case?

the test only checks whether "Not Significant" is reported or not, but not any numbers

---

versions of the test

Racine, J. (1997): "Consistent Significance Testing for Nonparametric Regression" is not cited in the book
Book, R package np, and
Racine, J.S., J. Hart, and Q. Li (2006), “Testing the significance of categorical predictor variables
in nonparametric regression models,” Econometric Reviews, 25, 523-544.
use wild bootstrap 

Racine, J.S., J. Hart, and Q. Li (2006) first bootstrap method is residual as the one implemented here.

Book has other tests in section 12.3.4, looks a bit similar to TestFForm.
- given that most pieces and the basic structure is available, it might be possible to offer more choices 
  without a big amount of extra work

user: josef-pkt
issue_612: state: open
title: nonparametric TestFForm
body: I fixed some errors and it runs now.

some notes for refactoring
- don't run bootstrap in `__init__`, use separate method
- Li/Racine mention that the test statistic is asymptotically normally distributed, although slow to converge.
  It might still be a good check, especially in larger sample where bootstrap will be slow.
  Add a method that allows choice for p-values based on aympt.normal or bootstrap.
- wild bootstrap could be outsourced, useful for other applications.
- for the test of a model estimated with OLS, it would be useful to have a separate exog.
  my example is one explanatory variable with polynomial function, 
  OLS has full vander matrix, but I think the kernel parts only need one variable, the other ones are 
  extra work. In my case the bw (by 'cv_ls') for the constant was 0, I don't know if this is a general 
  feature/result.
- having an extra `estimator` function looks redundant, TestFForm doesn't need the params, 
  instead we could just use the fitted values or predict of the Null model.
- given that it uses a wild bootstrap (similar would be parametric bootstrap) the exog is not
  changed during the bootstrap loop. This could be used for optimization of the bootstrap loop
  for these cases.
- what should be attached? I added the bootstrap results, so I can look at them after it's finished.

---

This can be used for various models, and it would be useful to add wrappers for it, e.g. for OLS.
I don't know if it would handle WLS if we use wendog and wexog.

user: josef-pkt
issue_609: state: open
title: regression plots for nonlinear, noparametric models
body: While working on kernel regression version, I'm trying to figure out which plots can provide information about the fit with multivariate x and nonlinear nonparametric functions.

Is there an analog of the linear regression and residual plots for this?
Can some of those be reused? haven't tried yet.

special cases SingleIndex and SemiLinear models.

some example plots are in my statsmodels/examples scripts ex_kernel_xxx

user: josef-pkt
issue_608: state: open
title: kernel regression models - review shape of endog
body: In Kernel regression models, I have mainly seen the problems in SingleIndex and SemiLinear models, endog is 2d (nobs, 1)

This means that predicted, fitted values also need to be 2d, for example dot(exog, b)[:,None]

In KernelCensoredModel, the censoring also need to be 2d (nobs, 1) if it's not a scalar.

I think it would be better to switch to endog as 1d, and as consequence the others could also be 1d.
I don't have a full overview yet, so I don't know if there are places where a 2d endog is more appropriate.

Note: I think we need tests for KernelCensoredModel, SingleIndex and SemiLinear for various shapes of exog before changing this.

---

related: some internal arrays have observations in columns. Is this useful ?

user: josef-pkt
issue_607: state: open
title: Problems nonparametric SemiLinearModel
body: similar problems as in SingleIndexModel Issue #605 

getting it to run in https://github.com/josef-pkt/statsmodels/commit/5f886079b7e3b3fd94447fe601bfe871c2f65e32
with example script in following commit

main problem: keeping track of required shapes of arrays to avoid exceptions, errors in broadcasting

shapes of arrays, e.g. endog is 2d, needs a review for all KernelReg and subclasses

also as in other KernelReg based models: need results statistics, especially for linear part

user: josef-pkt
issue_606: state: open
title: LOO and other cross-validation iterators
body: Skipper wrote a simplified LOO iterator in nonparametric.

However, for Kernel regression and variants it requires a double iterator. Use loo iterator that returns index instead.

related: If the nobs is large, then we might choose the bandwidth only on a subsample instead of the full sample (for example leave every second observation out).
In this case making it easy to switch a cross-validation iterator would be helpful.

user: josef-pkt
issue_605: state: open
title: Problems nonparametric SingleIndexModel
body: based on PR #562 

SingleIndex has only one nonparametric function
- number of bw should be 1
- vartype should also be just 1d, and 'c' by default or only
  (reshaping exog depends on len(vartype) which caused shape exceptions because I had 
  len(vartype)=1 but the real K/k_vars was 3)
- bw argument missing, might be useful for fixed user given bw
  would require a `_est_b_bw` method that only maximizes `b`
- the model doesn't use the dot product x*b, uses elementwise product and does Kernel regression over all variables

in my first example the fit is awful, maybe bandwidth choice or optimization problems or something in my example

the first fix is in https://github.com/josef-pkt/statsmodels/commit/04c80d8c2c274d91ce241ed464c95f1595d17cec
results look more reasonable now; not clean, just getting it to work

user: josef-pkt
issue_604: state: open
title: KernelReg - loc_linear: possible LOO speedup?
body: In the current design, loc_linear calculates the estimation for the full endog, exog given as argument. The leave-one-out loop is outside, calling `_loc_linear`

An alternative possibly faster way for making the LOO loops is by precalculating all used arrays for the full dataset, and then use a leave-one-out sum in the actual loop (for example by indexing with boolean with one false, or dot product with column of ones and one zero).

I would expect this to be faster because we calculate less within the loop, but I'm not sure the gain will be much and haven't tried it yet.

---

maybe I'm wrong: loc_linear has all calculations done for each left out i

I guess related to issue #598 

user: josef-pkt
issue_603: state: open
title: instance methods and pickle - nonparametric
body: SingleIndexModel.func is an instance method that will not pickle by default.

If we need to pickle an instance, then we can use the recipe in the negative binomial PR #584 which seems to work.

user: josef-pkt
issue_602: state: open
title: Design of nonparametric Regression
body: Notes based on PR #562 for possible redesign of nonparametric regression

model classes in PR #562: KernelReg, KernelCensoredReg, SingleIndexModel, SemiLinear

Model Classes
- fit and predict are merged: 
  - should KernelReg have fit or predict method, or both
  - SingleIndexModel, SemiLinear should have both, fit to estimate the parameters, predict to predict
  - fit (on full data) is necessary for diagnostic results, but only predict is necessary for limited prediction
  - move bandwidth calculation from `__init__` to `fit` ? It should be there, but there might be 
    methods that don't work until fit has been called. 
  - to predict we would have then model.fit().predict(exog_predict) as in other models, but which would
    be a bit of a detour in the KernelReg case. 
    SingleIndexModel, SemiLinear have also other parameters to estimate and would be more in line 
    with other models.
- options for nonlinear optimization, currently just uses optimize.fmin (hard coded) 

Result Classes
- we need a results instance to hold results about the fit, fitted values, resid, estimated error variance,  
  testing, marginal effects/derivative (currently returned by fit)
- results should be lazy, in case we only want to predict, (for example using KernelReg as smoother)
- check similar to other models, which methods should be in model and which in results class.
- as with other models, the results classes could be rather minimal at the beginning and we can add
  additional statistics and methods (for diagnostics) as they become available.

partially aside: 
KernelReg could be used in GAM (generalized additive model) and similar just as a smoother instead of stand alone. With results classes I think it would be easy to add whatever we need for these cases, if and when we get to them.

Also, the structure of the classes should be flexible enough that we can have a consistent interface with other, not kernel based, nonparametric regression models and smoothers. (splines, polynomials, ...)

user: josef-pkt
issue_600: state: open
title: KernelCensoredReg unfinished?
body: My impression is that KernelCensoredReg is not correct or finished.
- I added W to `_est_loc_linear` in my branch. Fit with `bw='cv_ls'` looks ok in plot, with random censoring.
- `bw='aic'` uses the inherited version which doesn't take account of censoring, - disable for now.
- only loc_linear is implemented, not loc_constant
- I haven't looked at other inherited parts yet.
- Only one short test

some comments about this are in pull request #562 

I didn't find an R implementation to test against.

user: josef-pkt
issue_599: state: open
title: BUG? bw='aic' has negative bandwidth with small absolute value
body: bug or feature?

based on example
https://github.com/josef-pkt/statsmodels/commit/6018ccea87f61326142fc5e19412bfe71001345a

bandwidths with bw='aic' are:

```
[     -0.02202225  204484.99229957]
[     -0.02202225  221554.27383839]
```

user: josef-pkt
issue_598: state: open
title: ENH: kernel methods with full matrix
body: Mainly a thought after reading some of the pull request #562 kernel methods

Some of the other kernel representation use a full projection matrix, similar to scipy rbf and Jonathan Stickel's smoothers.

y_predict = K(x,x) \* y   or maybe y_predict = K(x,x) \* K(y,y) or something like this

This avoids the loops but requires (nobs, nobs) matrices
This might be a faster alternative to some of the PR kernel methods for small to medium sized data, nobs < 500 or nobs < 1000

Need references and details, I don't see how it works with Racine/Li in the few pages I looked at.

aside: look into including Jonathan Stickel's smoothers again. Now we have an optional dependency on cvxopt.

user: josef-pkt
issue_596: state: open
title: add interquartile range, or scoreatpercentiles to statsmodels
body: interquartile range is used in several places.

nonparametric uses axis argument

We won't have efficient scoreatpercentiles until we drop support for older numpy and scipy

user: josef-pkt
issue_595: state: open
title: kernelreg doesn't finish with nobs=1000
body: I'm running an example with KernelReg.

with nobs=999 or nobs=1001 the fit returns after a few seconds. with nobs=1000, it doesn't return after several minutes (then I killed it)

(I don't understand the code well enough to see where this might hang.)

example not committed yet. based on rebased PR#562

user: josef-pkt
issue_585: state: open
title: Add visually weighted regression plots to graphics module
body: Quite useful and visually attractive: http://blog.enthought.com/general/visualizing-uncertainty/

user: rgommers
issue_579: state: open
title: Unify data handling
body: Unify and clean up tools/data.py and base/data.py. This is confusing. It looks like most of the code in tools/data.py is not used anymore except for the type checks, which should probably just be in base/data.py. There are still import checks for things that are now required as well.

user: jseabold
issue_574: state: open
title: multiple correlation
body: I'd never heard this terminology before. Might be nice to have some kind of convenience function for this  like the smc function in the psych package. Kind of a leave-one-variable-out R-squared. We may already have something similar around somewhere.

http://stackoverflow.com/questions/13452353/multiple-correlation-in-python
http://cran.r-project.org/web/packages/psych/

user: jseabold
issue_566: state: open
title: switch to QR by default for linear models
body: We have the idea for a while to switch to QR instead of pinv as the default for linear models (and possibly by implication for RLM and GLM)

I don't think we can do it before we have a proper singular matrix handling.

It's a pretty big backwards incompatible jump, that will need several rounds of deprecation, and might need adjustments at several places in our existing code. 

It might also be a problem that the test suites (ours and others) doesn't test for singular cases but might implicitly rely on the pinv behavior.

user: josef-pkt
issue_557: state: open
title: Add Theil-Sen estimator
body: Was asked about adding this. Seems simple enough.

http://en.wikipedia.org/wiki/Theil%E2%80%93Sen_estimator

user: jseabold
issue_555: state: open
title: RLM weighted results
body: RLM does not return weighted results. Ie., I think the expected results for resid, wexog, etc. should use the estimated weights. This would mean updated wexog and pinv_wexog in fit I believe so that we can get things like the hat matrix and influence statistics etc.  Consider this 

```
import pandas
from statsmodels.formula.api import rlm

dta = pandas.read_csv("http://www.stat.ufl.edu/~aa/social/csv_files/statewide-crime-2.csv")
dta = dta.set_index("State", inplace=True).dropna()
dta.rename(columns={"VR" : "crime",
                    "MR" : "murder",
                    "M"  : "pctmetro",
                    "W"  : "pctwhite",
                    "H"  : "pcths",
                    "P"  : "poverty",
                    "S"  : "single"
                    }, inplace=True)

crime_model = rlm("murder ~ pctmetro + poverty + pcths + single", data=dta, 
                                    M=sm.robust.norms.TukeyBiweight()).fit()

weights = crime_model.weights
hat_matrix_diag = weights*(crime_model.model.exog * crime_model.model.pinv_wexog.T).sum(1)
resid2 = (weights * crime_model.resid)**2
```

Compare to what you get with Stata with predict res and predict hat.

http://www.ats.ucla.edu/stat/stata/webbooks/reg/chapter4/statareg_self_assessment_answers4.htm

user: jseabold
issue_553: state: open
title: multicollinearity warning too sensitive?
body: I've yet to run a regression in the last week where I don't get this warning.

```
dta = pandas.read_csv("http://www.stat.ufl.edu/~aa/social/csv_files/statewide-crime-2.csv")
dta = dta.set_index("State", inplace=True).dropna()
dta.rename(columns={"VR" : "crime",
                    "MR" : "murder",
                    "M"  : "pctmetro",
                    "W"  : "pctwhite",
                    "H"  : "pcths",
                    "P"  : "poverty",
                    "S"  : "single"
                    }, inplace=True)

from statsmodels.formula.api import ols

mod = ols("murder ~ pctmetro + poverty + pcths + single", data=dta).fit()
print mod.summary()
```

user: jseabold
issue_551: state: open
title: outlier test - special case univariate mean variance
body: robust loc and scale estimation and outlier tests
thread at https://groups.google.com/d/topic/pystatsmodels/nsz5xJqCHbg/discussion

besides the older standard test and what we can do as special case in regression on a constant:

ELTS: efficient least trimmed squares
my impression (from the robustness literature) is that starting small has better robustness properties than trimming starting with the full sample.  I guess this is also the case in the simple univariate mean, variance case.
(I never looked at special case of just mean instead of regression, except for test on NIST handbook.)

need t-test for a single observation, given estimate of mean and variance.

for pvalue correction I think 2step fdr would be the best, but doesn't have corrected pvalues, only reject/don't reject.

user: josef-pkt
issue_550: state: open
title: robust Huber fails to converge
body: ```
>>> yy2 = [150.4, 28.8, 46.6, 40.2, 46.5]


>>> rob.Huber()(yy2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new\statsmodels\statsmodels\robust\scale.py", line 146, in __call__
    return self._estimate_both(a, scale, mu, axis, est_mu, n)
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new\statsmodels\statsmodels\robust\scale.py", line 190, in _estimate_both
    raise ValueError('joint estimation of location and scale failed to converge in %d iterations' % self.maxiter)
ValueError: joint estimation of location and scale failed to converge in 30 iterations
```

possible application: robust loc and scale estimation for outlier tests
thread at https://groups.google.com/d/topic/pystatsmodels/nsz5xJqCHbg/discussion

user: josef-pkt
issue_545: state: open
title: copying arrays and array order
body: If we ever need it.

pythons copy.copy, copy.deepcopy might change array order (numpy)

http://mail.scipy.org/pipermail/numpy-discussion/2012-October/064285.html

user: josef-pkt
issue_540: state: open
title: Predict and Patsy
body: Currently, if you fit the model with a formula then you need to feed back in data to predict that's of a similar type (has **getitem** with column name I assume). This isn't a deal breaker, but it might be nice to do some kind of guessing here if we can, allowing levels in type of categories and allowing array-like input that is coerced to something that Patsy expects. E.g.

```
import pandas
import numpy as np
from statsmodels.formula.api import ols
url = "http://www.ats.ucla.edu/stat//mult_pkg/faq/general/lgtrans.csv"
data = pandas.read_csv(url)
model = "write ~ C(female) + np.log(math) + np.log(read)"
test_scores = ols(model, data=data).fit()
```

This won't work

```
X = ['female', 65, 70]
test_scores.predict(X)
```

Nor will this

```
X = [[1, 65, 70]]
X = pandas.DataFrame(X, columns=["female", "math", "read"])
test_scores.predict(X)
```

But this does

```
X = [['female', 65, 70]]
X = pandas.DataFrame(X, columns=["female", "math", "read"])
test_scores.predict(X)
```

user: jseabold
issue_538: state: open
title: polyfit with options
body: A convenience class for a common use case

```
PolyFit(endog, exog, base='cheby', method='ols', method_args=None, ...)

Parameters
---------------

base : string for polynomial class
    use np.polynomial
method : string in ('ols', 'wls', 'rlm', 'elts', 'quantile', ...)  #GLSAR ?
    which estimation method to use

```

or better, move `method` to `fit(..)`

needs to create the full exog, np.vander for given base
needs to create the full p.exog in predict
rest is inherited, delegated to corresponding method.

shows up every once in a while on mailing list or stack overflow
would be nice to show off different estimators

user: josef-pkt
issue_536: state: open
title: onewaygls
body: idea: to allow for unchanging params/exog in onewaygls, I could just remove them first.

regress endog and exog that are subject to change in parameters on exog for which params are assumed constant.

still in sandbox

possibly obsolete or only a convenience class for one-way ANCOVA.
classes for one sample, two-sample, k-sample tests ?

user: josef-pkt
issue_525: state: open
title: DOC: documenting identical information across classes, methods and attributes
body: a full class docstring with good overview information

http://statsmodels.sourceforge.net/devel/generated/statsmodels.regression.linear_model.RegressionResults.html#statsmodels.regression.linear_model.RegressionResults

class docstring that just points out the difference:

http://statsmodels.sourceforge.net/devel/generated/statsmodels.regression.linear_model.OLSResults.html#statsmodels.regression.linear_model.OLSResults

I think this is a good split overall

things to improve:

1) given that we don't have problems with docstrings in cached attributes anymore, we should add those.
I would prefer to have an indication that they use attribute access, without the misleading `()`. This should be indicated in the 1st line of the description (`CA` `cached attribute` or similar)

2) more explanatory information:
I was looking at this again during the recent rsquared definition discussion
Stata has a good FAQ pages about missing rsquared and how to calculate it and how (not) to interpret it, for example
http://www.stata.com/support/faqs/statistics/r-squared/
http://www.stata.com/support/faqs/statistics/two-stage-least-squares/
As we clarify issues like these it would be useful to have a more extensive documentation, but not necessarily in the individual doc strings, for example just a link to another page for details (technical details that we have never written).

My target would be to have a more central location for a more extensive documentation of parts that are common to many models and results classes.

user: josef-pkt
issue_516: state: open
title: MNLogit perfect separation
body: Add perfect separation warning to MNLogit.

user: jseabold
issue_505: state: open
title: Weights
body: Make sure weights are correctly handled throughout the models. This includes GLM, RLM, ANOVA, and the discrete choice models. I think it also might make sense to have weights objects. It might also be interesting to see how far we can get with those provided by PySAL, but I haven't spoken with their developers since the summer. Many of their estimators are just duplicating ours. We should make it easy for them to use our code.

user: jseabold
issue_502: state: open
title: DOC: use bibtex based bibliographies
body: There are sometime references to literature:
http://statsmodels.sourceforge.net/devel/regression.html#technical-documentation

To unify, a bibtex-based bibliography could be used:
http://sphinxcontrib-bibtex.readthedocs.org/en/latest/index.html

user: timmie
issue_487: state: open
title: Style: Naming Conventions
body: Use mask in name for boolean indexes and idx for integer indexes. Not sure about where exactly this is used but sounds like a good style convention to get used to. Need to add to style docs and maybe do some cleanup internally as these are noticed.

user: jseabold
issue_481: state: open
title: add case weights to likelihood models
body: For some datasets with a large number of observations but many cases with identical exog, a representation of the data with case weights is more efficient.

It is partially allowed in GLM, but I think, we should add it consistently across models.

(aside: RLM doesn't allow for weights, weights are an internal part of the optimization procedure, IRLS)

user: josef-pkt
issue_459: state: open
title: Discrete choice models predict method has no standard error option
body: I'm in need of the estimate of the standard error of a logit prediction.

user: jseabold
issue_455: state: open
title: allow maxiter in HuberScale without raising exception
body: Virgile on mailing list:

Still concerning the use of a `HuberScale` scale estimate within IRLS, I
found in Huber's Robust Statistics book than the scale at step (m+1) should
be updated using a formula that does not require an iterative scheme.

According to the book, 
`sigma_(i+1)**2 = 1/(n*h)*sum(chi(r/sigma_i)*sigma_i**2`, 
as mentioned in the docstring of the
`HuberScale` class (in robust/scale.py). As far as I understand, this would
mean that HuberScale(maxiter=1) should be used in the context of IRLS, but
in that case, a "convergence not reached" exception occurs. My questions are
then: should the convergence check be removed in order to afford the use of
`HuberScale` with maxiter=1? Or is there any good reason to used an
iterative estimation of scale (say HuberScale(maxiter=1000)) within the IRLS
algorithm?

user: josef-pkt
issue_452: state: open
title: WIP: ENH:LTS, MTL least trimmed squares and ETS and MM-E
body: extensions to robust estimation

least trimmed squares
efficient least trimmed squares
mm-estimation
maximum trimmed likelihood

basic models work and look good in MonteCarlo and some examples from papers

no unit tests yet
incomplete integration with RLM
doesn't work with dummy variables yet: if subsets exog is singular, then it doesn't draw extra starting cases, and  it breaks (raises exception) in the Poisson case.
no large sample split for FastLTS yet

user: josef-pkt
labels: superseded: Used for PRs with content that is superseded by a newer PR
issue_451: state: open
title: make newton more robust to hessian
body: if the hessian is singular, then newton raises an exception:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-josef_new\statsmodels\base\model.py", line 670, in fit
    disp=disp, callback=callback, **kwargs)
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-josef_new\statsmodels\base\model.py", line 291, in fit
    hess=hess)
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-josef_new\statsmodels\base\model.py", line 341, in _fit_mle_newton
    newparams = oldparams - np.dot(np.linalg.inv(H),
  File "C:\Python26\lib\site-packages\numpy\linalg\linalg.py", line 445, in inv
    return wrap(solve(a, identity(a.shape[0], dtype=a.dtype)))
  File "C:\Python26\lib\site-packages\numpy\linalg\linalg.py", line 328, in solve
    raise LinAlgError, 'Singular matrix'
numpy.linalg.linalg.LinAlgError: Singular matrix
```

we could use `pinv` instead of `inv` (will be a bit slower)
or we could force the hessian to be positive definite (clip zero or negative eigenvalues, both inv and pinv invert hessian with negative eigenvalues)

(according to an PR for scipy, using eigh is faster than svd for symmetric matrices)

one possibility: 
use try except with inv and switch to eigh if inv fails 
(will be faster than always checking eigenvalues, but that doesn't detect negative eigenvalues)

user: josef-pkt
issue_449: state: open
title: optimization methods - BHHH, newton and ncg
body: (I'm working with GenericLikelihoodModel with a mixture distributions that has too many components which are not identified, I guess.
Hessian breaks down (not positive definite))

some random thoughts, I don't know the details well enough:

Is our newton really different from scipy's ncg?  
    ncg gave good results in Vincent's example for GenericLikelihoodModel for negative binomial

If we give the outerproduct of Jacobian as the hessian to fmin_ncg, isn't this then essentially the same as BHHH?
Can we get BHHH or something close to it almost for free?

see also issue #429 "LikelihoodModel optimization needs to be generalized"
and maybe we also need more flexibility in defining the Hessian that is used in the call to the optimization routines. (fake hessian for the optimization instead of the hessian for the final results statistics.)

user: josef-pkt
issue_436: state: open
title: Review changes to f_test and t_test
body: I just saw this and I don't know if I like it

https://github.com/statsmodels/statsmodels/commit/c4b0c13f1885d09e888559ffa4e58c2059b358b4

deprecating q_matrix
using patsy even if array ?

user: josef-pkt
issue_435: state: open
title: VARResults is missing a predict method.
body: VARResults is missing a predict method. It's hard to tell what VARResults.forecast actually does, and it looks like VARModel.predict is broken out of the box.

user: jseabold
issue_433: state: open
title: ENH: Semiparametric forecast intervals (application of KDE)
body: I was looking at this by chance

http://onlinelibrary.wiley.com/doi/10.1002/for.1185/abstract
Jason J. Wu Semiparametric forecast intervals DOI: 10.1002/for.1185

I think basic idea could be had very fast, fancy version in the paper needs more.

basically: 
- get the forecast errors from a parametric model (or LOO error in iid case)
- estimate cdf of error distribution by KDE (George added estimation of cdf not just pdf)
- get quantiles, confidence intervals from the non-parametric distribution around the parametric estimate

useful if the usual normal error assumption for forecast intervals is not very good.

fancy stuff in the paper (as far as I have seen and didn't look at details):
- kernel method is Adjusted Nadaraya-Watson: I don't know how much this differs from the "non-adjusted", and what the weights are in the kernel estimation
- minimum length forecast intervals (possibly disjoint) instead of just (alpha/2, 1-alpha/2) quantiles

user: josef-pkt
issue_420: state: open
title: Bootstrapping
body: See about incorporating this code into a bootstrapping framework. It's BSD licensed.

https://bitbucket.org/cevans/bootstrap/src

[Edit: Updated repo https://github.com/cgevans/scikits-bootstrap/]

user: jseabold
issue_414: state: open
title: SummaryTable is brutally cumbersome
body: Having added summary tables to AR, ARIMA, and now marginal effects for discrete choice variables, I have to say that doing this is brutally difficult and rather annoying. There's got to be a better way or a refactor that could sort this out.

user: jseabold
issue_411: state: open
title: fittedvalues in discrete
body: I think fittedvalues in discrete should be the default predict, i.e. np.exp(x_b) instead of x_b.

(Just took me half an hour to figure out fittedvalues is not inherited.)

genmod GLM also has fittedvalues = mu which is predict linear=False, AFAICS

user: josef-pkt
issue_404: state: open
title: optimal step size for numerical derivatives
body: It occurred to me that it might be worth it to try and calculate some optimal step size when calculating numerical derivatives. One for the case where the scales are similar and one for each variable when the scales might be wildly different.

SAS has some notes here, where we might set eta according to the scale of the data. http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#statug_nlmixed_sect027.htm

This provides guidance for the complex derivatives step-size and indicates there may be accompanying Python code somewhere (don't know license): http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.141.8002&rep=rep1&type=pdf

Other choices might include Richardson Extrapolation (I think this is what R does)

http://en.wikipedia.org/wiki/Richardson_extrapolation

user: jseabold
issue_391: state: open
title: Extend results stuff for GLM
body: Many of the diagnostic and results stuff have been written for linear regression models. These should be extended to work for GLM. E.g., anova, outlier tests, leverage and outlier diagnostics. It's often a simple change but most won't work out of the box as they are.

user: jseabold
issue_390: state: open
title: ANOVA model comparison printing
body: It might be nice in addition to the returned dataframes to have some pretty printing to show which model formula corresponds to which model. They're readily available in the models and it's obvious from how anova_lm is called, but it still may be helpful.

user: jseabold
issue_382: state: open
title: Doornik-Hansen test for multivariate normality
body: This seems to be the preferred test for VAR residual normality testing. Do we have it anywhere?

http://economics.ouls.ox.ac.uk/12773/1/normal2.pdf
http://lists.wfu.edu/pipermail/gretl-users/2007-November/001954.html

user: jseabold
issue_380: state: open
title: IRF plotting from VARResults
body: AFAICT, there's no way to set figsize for plotting.irf_grid_plot from VARResults.irf.plot. plot_params is also ignored in irf_grid_plot. There are probably still a few other lurking issues. This code needs to be looked over again after the changes made last summer...

user: jseabold
issue_379: state: open
title: VAR.test_causality tests
body: It looks like VAR still has never had proper correctness tests written. There is some test data for test_causailty, but I'm not sure where they came from - they look like maybe regression / smoke tests only. The results for test_causality are slightly different than Stata's and I'd like to figure out why.

user: jseabold
issue_376: state: open
title: marginal effects atexog argument should use names
body: Right now atexog for margeff in the discrete choice models expects a dictionary with column numbers mapped to values. This is a but cumbersome. This should also accept names mapped to values and maybe even something nicer. Formulas? Pandas.Series/DataFrames. Plain ndarrays?

user: jseabold
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_375: state: open
title: Marginal Effects need to be wrapped
body: Currently marginal effects from the discrete choice models don't pass through the results wrappers.

user: jseabold
issue_373: state: open
title: Closed form studentized residuals external
body: I can't remember whether we are using the loop still, but I just came across a closed form solution for leave one out studentized residuals

```
ols_model.resid * ((ols_model.df_resid-1)/(ols_model.ssr * (1 - infl.hat_matrix_diag) - ols_model.resid**2))**.5
```

user: jseabold
issue_365: state: open
title: summary method for multi-equation systems
body: we need some iolib.summary helper functions for multi-equation system that cannot be reshaped to 2d like VAR and multinomial.

Parts of an IRC thread

this currently work, but doesn't look nice (based on example_sysreg.py

```
>>> from statsmodels.iolib import summary as summ
>>> summ.summary_params(grun_rires)
<class 'statsmodels.iolib.table.SimpleTable'>
>>> params_names = ['f%d_%s'%(fn,en) for fn in range(4) for en in ['const', 'value', 'capital']]
>>> print summ.summary_params(grun_rires, xname=params_names)
```

summ.summary_params_2dflat    needs an extension to work with flattened parameters
The main part that is missing right now is if equations don't have the same number of parameters/exog
VAR, multinomial, and maybe parts of systems have the same number of parameters in each equation, so we can reshape the parameter vector to 2d
summary_params_2dflat works from the 2d paramter array, instead we need something similar that knows when to switch equations
The problem for simultaneous equation system will also be that the regressor and parameter names differ across equations. Then putting in in 2d form will not make much sense, I guess

another possibility is to create separate tables for each equation and then concatenate them
in linear model Results summary:  
 smry.add_table_params(self, yname=yname, xname=xname, alpha=.05, use_t=True)

In this case you would have to do the loop in the system Results summary() method, instead of delegating it to a iolib.summary helper function

<7,1josef|2> Alexandre I think copying and adjusting the linear model summary should work with a loop for add_table_params
<4,1alexandreyc_> josef|2, ok but the problem is that self is not a single regression results instance, so I should create one for each equation?
<7,1josef|2> ok, I see the problem with the self now. It looks like it won't work without adjusting iolib.summary functions

<7,1josef|2> summary_params function in iolib seems to also accept tuples instead of result instances `results, params, std_err, tvalues, pvalues, conf_int = results`
it's not tested and likely doesn't work
except for name handling (?)  it would also be possible just to create a Dummy instance that has params tvalues pvalues, confint as attributes

I think you could try to pass a tuple instead of self in smry.add_table_params(self, yname=yname, xname=xname, alpha=.05
the only problem is that conf_int is supposed to be a function called with conf_int(alpha)

Now, I'm not sure it's really worth the effort to try to work around the missing helper functions, it might be easier just to create a flat version of summary_params_2dflat
summary_params_2dflat looks easy to copy and adjust, 
   with full exog_names   (repeated so it's the same length as the flat params)
   endog names would be the names of the equation,  
   then we also need to pass in where to slice, e.g. list of number of exogs across equations

For now sysmodels don't handle variable and equation names. In fact I should have a list with number of exogs, actually I was thinking that df_model was enought
You are assuming a constant in each equations like linear_model, it's enough for knowing the number of exogs in each equation

user: josef-pkt
issue_362: state: open
title: move yule_walker
body: It doesn't make any sense to have yule_walker in regression.linear_model. I think we should deprecate this import and move it to tsa.ar_model.

user: jseabold
issue_357: state: open
title: stats.multitest.multipletests needs expanded docs
body: The methods are unfinished and it might be nice to have some formulas to look at. I don't know that we need to talk about the test coverage in the notes section either.

user: jseabold
issue_349: state: open
title: sandbox: cleanup or remove outdated modules
body: I just saw that there is still old code in the sandbox, that might not run anymore with current versions

e.g. using np.unique1d in try_catdata

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=659405

user: josef-pkt
issue_338: state: open
title: bincount2d, binnedstatistic
body: scipy 0.11 has a new binned_statistic function that generalizes np.histogramdd and uses the same algorithm.

We can use it as compat file, or with additional enhancements. 

Similar code could be used to create a bincount2d, bincountdd from integer arrays, which would not have to do the quantization step.

currently my approach to multi-indices is unique with reverse-index for binning ndarrays, after creating a structured array view, followed by bincount.

mainly for internal use, example grouputils and prediction table for multinomial (PR #337 ), when we already have integer categories

user: josef-pkt
issue_332: state: open
title: Strange behavior with pacf yule-walker unbiased
body: Need to dig in further, but to replicate.

``` python
import statsmodels.api as sm
from statsmodels.datasets import macrodata
x = np.diff(macrodata.load().data['realgdp'])
pc = sm.tsa.pacf(x, nlags=199, method='ywbiased')
print pc[101]
-600.36057976934592
```

user: jseabold
issue_323: state: open
title: Normal Linear Model, llf
body: I always had problems matching llf of the linear models.

I ran into this again, and finally figured out that the scale estimate is MLE based with denominator nobs, instead of the denominator in results.scale which is (nobs - k_vars)

```
>>> stats.norm.logpdf(res.resid, scale=np.sqrt((res.resid**2).mean())).sum()
-59.216496084705149
>>> res.llf
-59.216496084705149
```

Even though the scale estimate is different, I think llf is exactly the same as Stata reports (IIRC)

At several places I needed a normal linear model that does not concentrate out the scale/sigma_squared estimate. We should get a basic generic Normal Model class for this.

user: josef-pkt
issue_310: state: open
title: REF: handling of sigma in GLS
body: see #295 handling of scalar sigma

I think the right solution should avoid cholsigmainv for sigma.ndim 0 or 1

future: FGLS might need a different way to handle sigma, using (nobs,nobs) matrices is not very efficient
WLS and GLSAR avoids going through the sigma handling in GLS.init
but that should be a different issue

user: josef-pkt
issue_308: state: open
title: sandbox: refactor GOF class and add exponential
body: I added GOF for exponential, and proper GOF class for normal when parameters are estimated

implements a different Stephen's paper

has some test coverage, but not full because R or Stata don't have the equivalent or only partially

user: josef-pkt
issue_306: state: open
title: use RandomState for random numbers
body: We should follow sklearn's example and use RandomState for any random variable or random process generation.

makes it easier to initialize with seed and will work better with parallel processing.

user: josef-pkt
issue_304: state: open
title: WIP Nonlinearls
body: The code generated for testing NIST nonlinear models. Some points to note:
1. NISTtest_generator.py creates two python modules for each NIST model.
2. The 'modelname_testclass.py' includes the statsmodels nonlinear classes for the model with and without the analytical jacobian feature and also a base class 'TestNonlinearLS' for setup and storing the correct results from the NIST data file.
3. The 'test_modelname.py' uses the above module to create a derived class for adding tests.
4. This way the templated files need not be recreated for adding more tests. Any tests for a NIST model can be added to its respective 'test_modelname.py' 

Though most of the tests fail for now, but the objective here was to first get the correct method of writing tests. Also after going through some manual checking, I noticed some of the calculated parameter values match upto 3 decimal places.

user: divyanshubandil
issue_301: state: open
title: WIP: ENH PanelAR1: add Prais-Winston for 1st obs, same as Stata
body: work in progress (opened so I don't lose sight of it)

AR1Panel verifies against Stata for given rho

needs refactoring

user: josef-pkt
issue_300: state: open
title: BUG: incorrect CDF in CDFLink - no tests
body: see PR #299  missing ppf

a quick browsing of the source shows that there are no tests for probit cauchy or other CDFLink models

user: josef-pkt
issue_296: state: open
title: Performance:  sequential regression
body: cases where we calculate a sequence of regression problems that add variables (in a predetermined sequence) are candidates for optimization

example lag determination tsa.stattools._autolag
- does fortran order make a difference
- any advantages of using cholesky or QR?

(do we need a performance label?)

user: josef-pkt
issue_290: state: open
title: GLSAR rho > 1
body: using macrodataset

Stata reports a rho>1 for both Prais-Winsten and Cochran-Orcutt

prais realinv realgdp realint

e(rho) =  1.006439250608449

This shouldn't be feasible/possible. Check when I have implemented Prais-Winsten.

Test against Gretl for AR(1) errors are in growth rates. Stata looks close in that case (not yet checked in details

user: josef-pkt
issue_285: state: open
title: ddof in regression, and use-t
body: `xtgls  invest mvalue kstock, corr(ar1)`   FGLS for timeseries-cross-section

estimates scale = ssr / nobs instead of ssr / (nobs - k_vars)

and reports z-values, i.e. normal distribution based

`xtregar` with `re` reports z-values and chi2, with `fe` it reports t-values and F

In PanelAR1, I have the same params (using the same rho as stata) but bse and cov_params differ because the scale is normalized in a different way. 
I'm returning a OLS regression, the same would be with GLS.

In the documentation for another estimator (don't remember which), Stata has a user option to use `nobs` or `(nobs - k_vars)` as denominator in the scale calculation, referring to Greene that both cases are used and no clear advantage to either one.

-> need a possibility to change ddof for scale, e.g. `ssr / (nobs - k_vars - ddof)`  with possibility to set `ddof = - k_vars`
-> need possibility to switch between t and normal distribution (and f and t)   use_t as in summary.

(I thought I opened already an issue for ddof with pre-estimation removal of groups means for fixed effects models, but didn't find it.)

user: josef-pkt
issue_284: state: open
title: ShortPanel, too few individuals, singular
body: The Grunfeld example has fewer firms than time periods

estimating an unrestricted covariance matrix results in a reduced rank matrix and causes a failure of the Cholesky decomposition.

Stata produces numbers, but prints a warning "not converged"

```
>>> mod = ShortPanelGLS2(res.model.endog, res.model.exog, firm_id)
>>> res_p = mod.fit()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-josef_new\statsmodels\sandbox\panel\panel_short.py", line 120, in fit
    self.cholsigmainv_i = np.linalg.cholesky(np.linalg.pinv(sigma_i)).T
  File "C:\Python26\lib\site-packages\numpy\linalg\linalg.py", line 531, in cholesky
    Cholesky decomposition cannot be computed'
numpy.linalg.linalg.LinAlgError: Matrix is not positive definite -         Cholesky decomposition cannot be computed

>>> sigma_i = mod.get_within_cov(mod.res_pooled.resid)
>>> np.linalg.eigvals(sigma_i)
array([ 118599.13113534,   28749.15411872,   10884.5047154 ,
          1567.26735086,     455.28185451,     283.0793974 ,
           118.27313134,      47.77001344,      44.56802788,
            34.72886823,       5.18697788,       0.        ,
            -0.        ,       0.        ,       0.        ,
            -0.        ,      -0.        ,      -0.        ,
            -0.        ,       0.        ])
```

user: josef-pkt
issue_282: state: open
title: FAQ-D SMWarnings
body: we would need some custom warnings that are set to always warn on first import of statsmodels. The warning level shouldn't be reset after the initial import by statsmodels (e.g. in later imports of some modules) so we don't affect changed settings by user.

example I have right now is 

lag in `cov_nw_panel`: if lag is too large then we cannot use the largest lags, but we still have the truncated lagged sum.
I have it right now as exception, but I don't think that's useful. lag=a_large_number could just mean, take all available lags, as in python

```
>>> range(5)[:10]
[0, 1, 2, 3, 4]
```

However, in this example the kernel would still depend on the specified lag, so it would be a truncated kernel, any problems with that?

user: josef-pkt
issue_281: state: open
title: base data: instance check in base class
body: statsmodels.base.data.ModelData mixes subclassing and instance check in the super class ModelData

This should have a proper separation, e.g. no pandas usage in ModelData

for example: overwrite`_get_names` in subclasses

user: josef-pkt
issue_277: state: open
title: (statistical) test results classes
body: a long postponed decision: what do statistical tests return?

for example: https://groups.google.com/d/topic/pystatsmodels/Th8s05AqOGA/discussion
(found by chance again)

see also https://github.com/statsmodels/statsmodels/issues/268 for `__repr__` and `__str__`

my code for statistical tests, e.g. diagnostics, still uses several different patterns

user: josef-pkt
issue_251: state: open
title: add api.py to all subdirectories
body: see https://groups.google.com/group/pystatsmodels/browse_thread/thread/9cdfd56fd659659a?hl=en

I think it will be possible to do it in 0.4.1 without breaking backwards compatibility if we don't remove imports from the `__init__.py`

Remove all imports from `__init__.py` for release 0.5.0

I haven't looked at subdirectories at lower levels, eg. genmod.families which doesn't have code that isn't needed by GLM. An api.py is really not necessary in this case, unless we add generic functions in there (e.g. pre-specified transformation that I want also for other models and methods)

One other exception, directories for only internal use have an empty `__init__.py` already, e.g.

statsmodels\base                 has only the test runner in the `__init__.py`
statsmodels\tsa\base           completely empty
statsmodels\compatnp         completely empty

user: josef-pkt
issue_249: state: open
title: docs sandbox.rst out of date
body: http://statsmodels.sourceforge.net/devel/sandbox.html

many functions that are listed, are superseded by pandas.

some new models and cleaned-up old models (e.g. renamed mixed, ...) are missing

user: josef-pkt
issue_222: state: open
title: Extend f_test with SSR for (non-?)linear model
body: Given a hypothesis encapsulated by an r_matrix, you can back out the sum of squared residuals of the alternative model without fitting it. Can be interesting in the case of ANOVA-type analyses when we compare hypotheses for two nested models without fitting either of them. Right now, I am calculating this as part of type II ANOVA, but it would be nice to push this up a level. Also, clean-up ContrastResults repr while we're at it. We can do better than that.

user: jseabold
issue_203: state: open
title: GLM Binomial Trials
body: Should we keep data_weights that was dropped in commit 304ac9b? Or make Binomial family take a trials keyword. Use case, you have the success _rate_, but you also have a different number of trials for each observation in a separate variable. It's more convenient to not have to convert to success, failure.

user: jseabold
issue_201: state: open
title: table_dist needs review
body: stats.table_dist is currently only used by lilliefors (and parts are indirectly tested)

in PR #200 I made the creation of the polyinomial interpolators lazy. This needs another review for lazy and redundant calculations. and example in module needs to be converted to test

needs another usecase besides lillifors to see whether it's sufficiently general (2d only)

user: josef-pkt
issue_189: state: open
title: Fitting VAR processes from multiple sequences
body: Hello,
Not sure if this is the right forum to ask this - or if you would prefer this type of discussion on the mailing list. 
It seems like the interface to models, like VAR is designed to be used with a single sequence. I didn't see an obvious way to fit a model to multiple sequences. (i.e. concatenating along time axis doesn't make sense here as we would have discontinuities at the end of each sequence). 
Has this issue come up before? Is anyone else interested in fitting a model from many sequences?

Thanks for making this project available to the community.
Graham

user: gwtaylor
issue_179: state: open
title: Add modified newton-raphson
body: Include a scalar step-size to newton's method. There is some discussion of this in Numerical Recipes 9.7.

user: jseabold
issue_153: state: open
title: test_var: fevd is a print statement
body: this was a test error because 2to3 seems to not have translated the print statement.

No idea why it wasn't translated, and no idea what that test is supposed to do

```
class CheckFEVD(object):

    fevd = None

    #---------------------------------------------------------------------------
    # FEVD tests

    def test_fevd_plot(self):
        if not have_matplotlib():
            raise nose.SkipTest

        self.fevd.plot()
        close_plots()

    def test_fevd_repr(self):
        print self.fevd
```

user: josef-pkt
issue_146: state: open
title: singular exog - check
body: I'm not sure anymore what the behavior with singular (reduced rank) exog matrices is. Needs some investigation (again).

for OLS or linear model:
I'd like to warn and add a comment in summary, but I don't want to kill it and create warning noise. Some of the diagnostic tests don't care about singular exog, since they just need the rsquared or a joint test. df_resid (?) uses rank, so this works also with reduced rank.
(example is when dummy or constant is squared and added as additional variable)

What's the effect of using pinv (and bounding smallest eigenvalue away from zero) on the cov_params ? Can we still detect singular exog ex post (df_resid), or should we check and warn in fit? 

related: Until we replace pinv with svd or similar we have redundant calculations to check singularity, eigenvalues and rank.

user: josef-pkt
issue_143: state: open
title: add histogram boxplot
body: Tony Yu has a recipe for a histogram strip chart, it's a kind of boxplot with colors for histogram frequency

http://sourceforge.net/mailarchive/message.php?msg_id=28750326

attachment is at http://www.mail-archive.com/matplotlib-users@lists.sourceforge.net/msg23263.html

original thread http://www.mail-archive.com/matplotlib-users@lists.sourceforge.net/msg18761.html

This would make a nice addition to graphics/boxplots

user: josef-pkt
issue_142: state: open
title: VAR.forecast_cov and plot only work for trend='c'
body: running example ex_var.py with trend = 'nc' or 'ct' raises an exception in plot_forecast and forecast_cov

```
>>> res_nc.forecast_cov(3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-josef\scikits\statsmodels\tsa\vector_ar\var_model.py", line 1056, in forecast_cov
    omegas = self._omega_forc_cov(steps)
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-josef\scikits\statsmodels\tsa\vector_ar\var_model.py", line 1203, in _omega_forc_cov
    B = self._bmat_forc_cov()
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-josef\scikits\statsmodels\tsa\vector_ar\var_model.py", line 1241, in _bmat_forc_cov
    return np.vstack((upper, self.params.T, lower))
  File "C:\Python26\lib\site-packages\numpy\core\shape_base.py", line 214, in vstack
    return _nx.concatenate(map(atleast_2d,tup),0)
ValueError: array dimensions must agree except for d_0
```

user: josef-pkt
issue_136: state: open
title: VAR: print statements and plots
body: Trying out some examples (following some pandas examples form a notebook by Wes):

In python I don't expect an assignment to print anything, unless it's a warning.
Also, I would expect a test to return a numerical result.

```
>>> c = res.test_causality('m1', 'realgdp')
Granger causality f-test
=============================================================
   Test statistic   Critical Value          p-value        df
-------------------------------------------------------------
         1.110648         3.011286            0.330  (2, 579)
=============================================================
H_0: ['realgdp'] do not Granger-cause m1
Conclusion: fail to reject H_0 at 5.00% significance level

>>> res.test_whiteness()
FAIL: Some autocorrelations exceed 0.1414 bound. See plot
>>> w = res.test_whiteness()
FAIL: Some autocorrelations exceed 0.1414 bound. See plot
```

user: josef-pkt
issue_133: state: open
title: tsa in sandbox, check example
body: sandbox/tsa/example_arma.py  had the old plot acf code

check what's still useful to keep, and look at FIXME after merging Ralf's PR 122

user: josef-pkt
issue_132: state: open
title: sandbox stats runs: runstest and some nonparametric tests
body: this module has several statistical tests (besides some draft code for runs distributions). There are some verified tests for the tests in the module, they should be moved to the test directory.

median_test_ksample doesn't have a reference and doesn't say whether it's tested. I don't remember where I got the description, but it just looks like a standard chisquare test.

the links in the docs page don't work http://statsmodels.sourceforge.net/stats.html#non-parametric-tests 

user: josef-pkt
issue_130: state: open
title: attach diagnostic tests to result instance
body: summary currently has autocorrelation and normality tests, but they are just available in the summary tables. Heteroscedasticity, outlier and (non)linearity diagnostics are missing.

I might also be nice to attach them as methods that cover several tests, .test_acorr, .test_het, .test_xxx? and also to attach the corresponding diagnostic regression plots.

Also doc examples for the whole enchilada, estimate, test, plot, alternative methods in the style of UCLA stats or a chapter in a textbook like Greene would be useful to see what's available and how to run them as user.

user: josef-pkt
labels: topic-post_estim: post-estimation results, margins, prediction, inference, diagnostic
issue_127: state: open
title: add iteration and permutation helper functions
body: Ralf needs a substitute for itertools.combinations in functional plots, which is not available in python 2.5

similar there are permutation and other combinatorial functions that we need for permutation tests, 

I think they can go together, but where, tools, resampling, next to cross_val iterators...?

Alex has a permutation function by Fernando in mne-python, I have a non-duplicate group permutation somewhere not committed yet.

user: josef-pkt
issue_126: state: open
title: tukeyplot check
body: tukeyplot is not in the sandbox, but it is only preliminary code.  Check what the status is

user: josef-pkt
issue_106: state: open
title: ENH/SUMM: add rvs or distributions to families
body: I have problems generating random variables for nbinom NegativeBinomial GLM example, because it takes two parameters. I'm not sure I got the correct parameterization for gamma.

It would be useful to directly have an rvs method in the families, or alternatively the scipy.stats distribution attached.

The distribution would also be useful for a new method predict_distribution.

Should scipy.stats be imported lazily, or is it already imported anyway?


**update**
#7535 add proper `get_distribution` to GLM and families
**update2**
#7929 add `get_distribution` to discrete count models  to model and results classes
#7707 overview what's available for post-estimation

open design question
- method in results instance ?
- separate out `convert_params`  (currently defined in discrete.NegativeBinomialP)
- direct access to implied statistics, e.g. Poisson results has `predict_prob`, new count models have `predict(... which="prob")`

user: josef-pkt
issue_99: state: open
title: VAR is missing conf_int
body: In analogy to MNLogit it should be a 3d matrix matrix with the confidence interval as the last axis.

n_regressors by neqs by 2

user: josef-pkt
issue_93: state: open
title: perfect fit, check corner case behavior of models
body: check the behavior of all models under perfect fit, e.g. linear model with zero noise

one of the robust experts mentioned in an abstract that perfect fit might not be a misfeature (reference?)

user: josef-pkt
issue_92: state: open
title: cox regression: possible convergence failure
body: For future reference:
Similar to the perfect separation case in Logit, it looks like optimization of loglike in Cox hazard model can go off to infinity.

http://www.jstor.org/stable/2676848

I only read the abstract, Google has cited by 41

user: josef-pkt
issue_87: state: open
title: summary: change diagnostic table for regression
body: the summary table has Jarque-Bera test for normality on the right sub-table. It should be on the left side and a autocorrelation test on the right side below durbin-watson.

add info text if normality test or autocorrelation test is below 5% or 10%. We don't have p-values for Durbin-Watson, but it's less powerful anyway.

other possible changes ....?

user: josef-pkt
issue_84: state: open
title: add Nonlinear leastsquares
body: This should be in release 0.4

(trying out milestones)

user: josef-pkt
issue_73: state: open
title: rename rlm.M
body: finding the norm doing dir(rlmmodelinstance) is not easy

a more informative name instead of M would be better.

user: josef-pkt
issue_63: state: open
title: internal/design adjustments in discretemod
body: Original Launchpad bug 642240: https://bugs.launchpad.net/statsmodels/+bug/642240
Reported by: josef-pktd (joep).

a few notes on discrete models, while looking how nested logit fits in
- nparams
  aic, bic, ... calculations need the number of params self.df_model+self.model.J-1, 
  I need it also for a default starting values and other things in ru2nmnl
  If we set nparams in the model, then bic, aic don't need to be special cased for MNL. We could use df_model as default if nparams doesn't exist in the model
- in MNL.J and .K are not very informative when reading, nchoices and nexog would be more informative
  in ru2nmnl, I started to use nchoices, I don't think I have any nexog. 
- llnull: I don't think this will work in nested logit, and maybe not in clogit. Unfortunaltely the the arguments to model.**class**.**init** are more complicated than the generic (endog, exog)
  --> possibility to override in model or in a Results subclass ?
- llnull: do we need the null model for anything else ? currently null model is not attached
- Rsquare: for MNL type models, SAS (proc MDC) has six definitions additional to McFadden, they are all easy to calculate given llnull

extras not checked yet 
Confusion/classification matrix ?

(I expect to add more as I work my way through this.)

user: wesm
issue_60: state: open
title: fvalue and mse_model are -inf if only 1 regressor
body: Original Launchpad bug 440151: https://bugs.launchpad.net/statsmodels/+bug/440151
Reported by: josef-pktd (joep).

the tests for dimension of exog, endog for model fail because -if is not equal -inf.
The source I think is if exog is only 1d or one column, then the df_model = 0, which, I think, causes a zero division for fvalue and mse_model

It is necessary to fix test and to check whether the -inf result is actually correct.
I did the initial correction in the test for fvalue, but commented it out again, because we need to check first if -inf is correct.

In the example, the df_model should be 1 instead of 0.
I think we need a `hasconstant` indicator so we can use
the correct df if there is only 1 (non-constant) regressor.

If we correct df_model by not subtracting 1, then we won't get -inf in the
case of this ticket, and we can ignore that -inf != -inf with numpy <= 1.3

user: wesm
issue_59: state: open
title: GLS: missing standard errors/confidence interval for fittedvalues/predict
body: Original Launchpad bug 428911: https://bugs.launchpad.net/statsmodels/+bug/428911
Reported by: josef-pktd (joep).

I didn't find any results for getting the prediction error for fittedvalues or for predictions made with GLS.predict. It would be nice to have standard errors that can be used to plot confidence intervals together with the fitted values.

This should be relatively easy for iid case in OLS, but maybe not so easy for heteroscedastic case (GLS, WLS) or other non-i.i.d. cases 
(GLSAR doesn't have conditional or one-step ahead prediction yet.)

user: wesm
issue_55: state: open
title: perfect fit (in rlm)
body: Original Launchpad bug 790770: https://bugs.launchpad.net/statsmodels/+bug/790770
Reported by: josef-pktd (joep).

look at what happens in the models, model results when we have a perfect fit, and decide what to do in this cases
example for rlm, but there might be similar problems in other cases

mailinglist 2011-05-31
for rlm having a zero residual doesn't cause an exception anymore, but having a perfect fit can produce nans in some results (scale=0 and 0/0 division)

```
import numpy as np
import scikits.statsmodels.api as sm
x = np.array([0, 0.96, 2.18])
levels = np.array([2.8, 2.8, 2.8])
res1 = sm.RLM(endog=levels,exog=sm.add_constant(x),M=sm.robust.norms.HuberT()).fit()
print res1.params
print res1.sresid
print res1.scale

>>> print res1.params
[  2.77555756e-17   2.80000000e+00]
>>> print res1.sresid
[ nan  nan  nan]
>>> print res1.scale
0.0
```

user: wesm
issue_37: state: open
title: GenericLikelihoodModel and fittedvalues
body: Original Launchpad bug 717510: https://bugs.launchpad.net/statsmodels/+bug/717510
Reported by: josef-pktd (joep).

GenericLikelihoodModel and its results do not define fittedvalues 

Check what minimal methods a subclass has to implement, so we can have some fittedvalues. The meaning of fittedvalues is not obvious in some non-linear or non-normal models.

Is defining a predict method in the model subclass enough? Is there some generic support?

example Markov switching model with AR terms in each state: estimation with GenericLikelihoodModel seems to work fine although it's slow, but how do I get prediction, fitted values and residuals?

user: wesm
issue_30: state: open
title: add compare methods to Results
body: Original Launchpad bug 677947: https://bugs.launchpad.net/statsmodels/+bug/677947
Reported by: josef-pktd (joep).

see thread starting Oct17 2010, Comparing models

proposal by Thomas Wiecki

http://bazaar.launchpad.net/~josef-pktd/statsmodels/statsmodels-josef-experimental-gsoc/revision/2186

For now I made two methods out of it compare_f_test and
compare_lr_test both attached to regression results
I think, compare_lr_test will go higher up in the class hierarchy so
that it is also available for maximum likelihood models.

I like the more explicit names, but shorter names might be possible

In the simple example, compare_f_test produces exactly the same result
as f_test with the corresponding restriction matrix, as it should for
linear models. Likelihood Ratio test results are close to f_test but I
don't have an exact test for it yet.

user: wesm
issue_28: state: open
title: make link argument in Family._setlink more flexible
body: Original Launchpad bug 422327: https://bugs.launchpad.net/statsmodels/+bug/422327
Reported by: josef-pktd (joep).

setlink checks for class instance, both in general and for specific instances and raises a ValueError

if not isinstance(link, L.Link):
 validlink = link in self.links

especially this includes the use of Power with different values (reported by PierreGM)

Proposal:

convert Exception to warning: since it's up to the user to decide which links to use, using the "wrong" links might not make much statistical sence but shouldn't break the functioning of the model.

make check for specific links by **name** attribute

allow for user specified links, not just links that are derived from the predefined links, e.g. extension to BoxCox transformation.

user: wesm