Repository: statsmodels/statsmodels

Issue:
summary() params table split conf_int
If the params table is rendered as html, then it doesn't look nice that confidence interval is a single column.

for example
http://nbviewer.ipython.org/github/nipunreddevil/PythonAnalyticsEdge/blob/master/week_2/lecture/baseball.ipynb?create=1

single column conf_int was optimized for string/text printing.
I guess it will also be messed up in Latex.



Discussion:
adding prio-high

I saw a presentation about statsmodels, and the html rendering of confint columns looks bad

How would you render it in HTML? Also, based on my cursory look into `iolib/*.py`, it's not that easy to switch a cell's representation based on the output format?

(I'm trying to catch up with some issues)

This doesn't require changing the rendering itself because that is handled by `SimpleTable`.
The task here is to split the columns into two columns, keeping the numeric formatting, but the header for confidence interval needs to change. The header needs to be split into two corresponding to the two columns. AFAIR, the header was my reason to use a single column, but at the time html view of summary wasn't very popular yet (and I didn't see how bad it looks).

https://github.com/statsmodels/statsmodels/blob/master/statsmodels/iolib/summary.py#L474
is making the combined string for lower and upper confidence limite which is further down added to the params_table.  There might be other places, that I didn't see right now.

@josef-pkt I see. If we are splitting the header for confint, what should each header say? Will it look decent in the print/text context?

I'm not sure about the header, maybe we try the same as what `summary2` uses, i.e. `[0.025    0.975]`
alpha / 2 and (1 - alpha) / 2


Resolving patch:
diff --git a/statsmodels/iolib/summary.py b/statsmodels/iolib/summary.py
--- a/statsmodels/iolib/summary.py
+++ b/statsmodels/iolib/summary.py
@@ -39,7 +39,7 @@ def summary(self, yname=None, xname=None, title=0, alpha=.05,
 
     Returns
     -------
-    Defualt :
+    Default :
     returns='print'
             Prints the summarirized results
 
@@ -450,13 +450,12 @@ def summary_params(results, yname=None, xname=None, alpha=.05, use_t=True,
 
     #Dictionary to store the header names for the parameter part of the
     #summary table. look up by modeltype
-    alp = str((1-alpha)*100)+'%'
     if use_t:
         param_header = ['coef', 'std err', 't', 'P>|t|',
-                        '[' + alp + ' Conf. Int.]']
+                        '[' + str(alpha/2), str(1-alpha/2) + ']']
     else:
         param_header = ['coef', 'std err', 'z', 'P>|z|',
-                        '[' + alp + ' Conf. Int.]']
+                        '[' + str(alpha/2), str(1-alpha/2) + ']']
 
     if skip_header:
         param_header = None
@@ -468,35 +467,12 @@ def summary_params(results, yname=None, xname=None, alpha=.05, use_t=True,
 
     exog_idx = lrange(len(xname))
 
-    #center confidence intervals if they are unequal lengths
-#    confint = ["(%#6.3g, %#6.3g)" % tuple(conf_int[i]) for i in \
-#                                                             exog_idx]
-    confint = ["%s %s" % tuple(lmap(forg, conf_int[i])) for i in \
-                                                             exog_idx]
-    len_ci = lmap(len, confint)
-    max_ci = max(len_ci)
-    min_ci = min(len_ci)
-
-    if min_ci < max_ci:
-        confint = [ci.center(max_ci) for ci in confint]
-
-    #explicit f/g formatting, now uses forg, f or g depending on values
-#    params_data = lzip(["%#6.4g" % (params[i]) for i in exog_idx],
-#                       ["%#6.4f" % (std_err[i]) for i in exog_idx],
-#                       ["%#6.3f" % (tvalues[i]) for i in exog_idx],
-#                       ["%#6.3f" % (pvalues[i]) for i in exog_idx],
-#                       confint
-##                       ["(%#6.3g, %#6.3g)" % tuple(conf_int[i]) for i in \
-##                                                             exog_idx]
-#                      )
-
     params_data = lzip([forg(params[i], prec=4) for i in exog_idx],
                        [forg(std_err[i]) for i in exog_idx],
                        [forg(tvalues[i]) for i in exog_idx],
                        ["%#6.3f" % (pvalues[i]) for i in exog_idx],
-                       confint
-#                       ["(%#6.3g, %#6.3g)" % tuple(conf_int[i]) for i in \
-#                                                             exog_idx]
+                       [forg(conf_int[i,0]) for i in exog_idx],
+                       [forg(conf_int[i,1]) for i in exog_idx]
                       )
     parameter_table = SimpleTable(params_data,
                                   param_header,
diff --git a/statsmodels/iolib/tableformatting.py b/statsmodels/iolib/tableformatting.py
--- a/statsmodels/iolib/tableformatting.py
+++ b/statsmodels/iolib/tableformatting.py
@@ -96,7 +96,7 @@
 fmt_params = copy.deepcopy(fmt_base)
 
 fmt3 = dict(
-            data_fmts = ["%s", "%s", "%8s", "%s", "%23s"],
+            data_fmts = ["%s", "%s", "%8s", "%s", "%11s", "%11s"],
             )
 fmt_params.update(fmt3)
 


Test patch:
diff --git a/statsmodels/discrete/tests/results/mn_logit_summary.txt b/statsmodels/discrete/tests/results/mn_logit_summary.txt
--- a/statsmodels/discrete/tests/results/mn_logit_summary.txt
+++ b/statsmodels/discrete/tests/results/mn_logit_summary.txt
@@ -1,49 +1,49 @@
 ==============================================================================
-       y=1       coef    std err          z      P>|z|      [95.0% Conf. Int.]
+       y=1       coef    std err          z      P>|z|      [0.025      0.975]
 ------------------------------------------------------------------------------
-const         -0.2794      0.612     -0.457      0.648        -1.479     0.920
-x1            -0.0119      0.034     -0.350      0.727        -0.079     0.055
-x2             0.2946      0.093      3.152      0.002         0.111     0.478
-x3            -0.0257      0.007     -3.873      0.000        -0.039    -0.013
-x4             0.0897      0.069      1.297      0.195        -0.046     0.225
+const         -0.2794      0.612     -0.457      0.648      -1.479       0.920
+x1            -0.0119      0.034     -0.350      0.727      -0.079       0.055
+x2             0.2946      0.093      3.152      0.002       0.111       0.478
+x3            -0.0257      0.007     -3.873      0.000      -0.039      -0.013
+x4             0.0897      0.069      1.297      0.195      -0.046       0.225
 ------------------------------------------------------------------------------
-       y=2       coef    std err          z      P>|z|      [95.0% Conf. Int.]
+       y=2       coef    std err          z      P>|z|      [0.025      0.975]
 ------------------------------------------------------------------------------
-const         -1.7392      0.720     -2.414      0.016        -3.151    -0.327
-x1            -0.0941      0.039     -2.415      0.016        -0.170    -0.018
-x2             0.3838      0.108      3.559      0.000         0.172     0.595
-x3            -0.0230      0.008     -2.914      0.004        -0.038    -0.008
-x4             0.2451      0.080      3.053      0.002         0.088     0.402
+const         -1.7392      0.720     -2.414      0.016      -3.151      -0.327
+x1            -0.0941      0.039     -2.415      0.016      -0.170      -0.018
+x2             0.3838      0.108      3.559      0.000       0.172       0.595
+x3            -0.0230      0.008     -2.914      0.004      -0.038      -0.008
+x4             0.2451      0.080      3.053      0.002       0.088       0.402
 ------------------------------------------------------------------------------
-       y=3       coef    std err          z      P>|z|      [95.0% Conf. Int.]
+       y=3       coef    std err          z      P>|z|      [0.025      0.975]
 ------------------------------------------------------------------------------
-const         -3.0091      1.069     -2.815      0.005        -5.104    -0.914
-x1            -0.1115      0.057     -1.965      0.049        -0.223    -0.000
-x2             0.5641      0.159      3.557      0.000         0.253     0.875
-x3            -0.0156      0.011     -1.391      0.164        -0.038     0.006
-x4             0.0687      0.119      0.577      0.564        -0.165     0.302
+const         -3.0091      1.069     -2.815      0.005      -5.104      -0.914
+x1            -0.1115      0.057     -1.965      0.049      -0.223      -0.000
+x2             0.5641      0.159      3.557      0.000       0.253       0.875
+x3            -0.0156      0.011     -1.391      0.164      -0.038       0.006
+x4             0.0687      0.119      0.577      0.564      -0.165       0.302
 ------------------------------------------------------------------------------
-       y=4       coef    std err          z      P>|z|      [95.0% Conf. Int.]
+       y=4       coef    std err          z      P>|z|      [0.025      0.975]
 ------------------------------------------------------------------------------
-const         -6.5607      0.870     -7.540      0.000        -8.266    -4.855
-x1            -0.0949      0.043     -2.183      0.029        -0.180    -0.010
-x2             1.2668      0.128      9.897      0.000         1.016     1.518
-x3            -0.0109      0.008     -1.315      0.189        -0.027     0.005
-x4             0.3067      0.088      3.475      0.001         0.134     0.480
+const         -6.5607      0.870     -7.540      0.000      -8.266      -4.855
+x1            -0.0949      0.043     -2.183      0.029      -0.180      -0.010
+x2             1.2668      0.128      9.897      0.000       1.016       1.518
+x3            -0.0109      0.008     -1.315      0.189      -0.027       0.005
+x4             0.3067      0.088      3.475      0.001       0.134       0.480
 ------------------------------------------------------------------------------
-       y=5       coef    std err          z      P>|z|      [95.0% Conf. Int.]
+       y=5       coef    std err          z      P>|z|      [0.025      0.975]
 ------------------------------------------------------------------------------
-const         -6.0760      0.774     -7.851      0.000        -7.593    -4.559
-x1            -0.0967      0.039     -2.479      0.013        -0.173    -0.020
-x2             1.3377      0.116     11.493      0.000         1.110     1.566
-x3            -0.0198      0.008     -2.631      0.009        -0.035    -0.005
-x4             0.3193      0.080      4.001      0.000         0.163     0.476
+const         -6.0760      0.774     -7.851      0.000      -7.593      -4.559
+x1            -0.0967      0.039     -2.479      0.013      -0.173      -0.020
+x2             1.3377      0.116     11.493      0.000       1.110       1.566
+x3            -0.0198      0.008     -2.631      0.009      -0.035      -0.005
+x4             0.3193      0.080      4.001      0.000       0.163       0.476
 ------------------------------------------------------------------------------
-       y=6       coef    std err          z      P>|z|      [95.0% Conf. Int.]
+       y=6       coef    std err          z      P>|z|      [0.025      0.975]
 ------------------------------------------------------------------------------
-const        -10.5973      0.957    -11.076      0.000       -12.473    -8.722
-x1            -0.1447      0.042     -3.475      0.001        -0.226    -0.063
-x2             2.0395      0.141     14.479      0.000         1.763     2.316
-x3            -0.0129      0.008     -1.612      0.107        -0.029     0.003
-x4             0.4576      0.085      5.352      0.000         0.290     0.625
+const        -10.5973      0.957    -11.076      0.000     -12.473      -8.722
+x1            -0.1447      0.042     -3.475      0.001      -0.226      -0.063
+x2             2.0395      0.141     14.479      0.000       1.763       2.316
+x3            -0.0129      0.008     -1.612      0.107      -0.029       0.003
+x4             0.4576      0.085      5.352      0.000       0.290       0.625
 ==============================================================================
diff --git a/statsmodels/discrete/tests/test_discrete.py b/statsmodels/discrete/tests/test_discrete.py
--- a/statsmodels/discrete/tests/test_discrete.py
+++ b/statsmodels/discrete/tests/test_discrete.py
@@ -1332,7 +1332,7 @@ def test_issue_339():
     cur_dir = os.path.dirname(os.path.abspath(__file__))
     test_case_file = os.path.join(cur_dir, 'results', 'mn_logit_summary.txt')
     test_case = open(test_case_file, 'r').read()
-    np.testing.assert_(smry == test_case[:-1])
+    np.testing.assert_equal(smry, test_case[:-1])
 
 def test_issue_341():
     data = sm.datasets.anes96.load()
diff --git a/statsmodels/regression/tests/test_regression.py b/statsmodels/regression/tests/test_regression.py
--- a/statsmodels/regression/tests/test_regression.py
+++ b/statsmodels/regression/tests/test_regression.py
@@ -951,16 +951,16 @@ def test_summary():
 \\textbf{Df Model:}         &           6      & \\textbf{                     } &             \\\\
 \\bottomrule
 \\end{tabular}
-\\begin{tabular}{lccccc}
-                  & \\textbf{coef} & \\textbf{std err} & \\textbf{t} & \\textbf{P$>$$|$t$|$} & \\textbf{[95.0\\% Conf. Int.]}  \\\\
+\\begin{tabular}{lcccccc}
+                  & \\textbf{coef} & \\textbf{std err} & \\textbf{t} & \\textbf{P$>$$|$t$|$} & \\textbf{[0.025} & \\textbf{0.975]}  \\\\
 \\midrule
-\\textbf{GNPDEFL}  &      15.0619  &       84.915     &     0.177  &         0.863        &      -177.029   207.153       \\\\
-\\textbf{GNP}      &      -0.0358  &        0.033     &    -1.070  &         0.313        &        -0.112     0.040       \\\\
-\\textbf{UNEMP}    &      -2.0202  &        0.488     &    -4.136  &         0.003        &        -3.125    -0.915       \\\\
-\\textbf{ARMED}    &      -1.0332  &        0.214     &    -4.822  &         0.001        &        -1.518    -0.549       \\\\
-\\textbf{POP}      &      -0.0511  &        0.226     &    -0.226  &         0.826        &        -0.563     0.460       \\\\
-\\textbf{YEAR}     &    1829.1515  &      455.478     &     4.016  &         0.003        &       798.788  2859.515       \\\\
-\\textbf{constant} &   -3.482e+06  &      8.9e+05     &    -3.911  &         0.004        &      -5.5e+06 -1.47e+06       \\\\
+\\textbf{GNPDEFL}  &      15.0619  &       84.915     &     0.177  &         0.863        &     -177.029    &      207.153     \\\\
+\\textbf{GNP}      &      -0.0358  &        0.033     &    -1.070  &         0.313        &       -0.112    &        0.040     \\\\
+\\textbf{UNEMP}    &      -2.0202  &        0.488     &    -4.136  &         0.003        &       -3.125    &       -0.915     \\\\
+\\textbf{ARMED}    &      -1.0332  &        0.214     &    -4.822  &         0.001        &       -1.518    &       -0.549     \\\\
+\\textbf{POP}      &      -0.0511  &        0.226     &    -0.226  &         0.826        &       -0.563    &        0.460     \\\\
+\\textbf{YEAR}     &    1829.1515  &      455.478     &     4.016  &         0.003        &      798.788    &     2859.515     \\\\
+\\textbf{constant} &   -3.482e+06  &      8.9e+05     &    -3.911  &         0.004        &     -5.5e+06    &    -1.47e+06     \\\\
 \\bottomrule
 \\end{tabular}
 \\begin{tabular}{lclc}