Repository: statsmodels/statsmodels

Issue:
as_latex produces invalid latex
Version: 0.9.0
Function: statsmodels.iolib.summary.Summary.as_latex
Bug: Function outputs a string, as requested, but the string isn't valid LaTeX. Two problems arise when using print(as_latex()). Firstly, there are not spaces between all '$'s, and as LaTeX interprets $$ as a distinct command, the output fails to process. Secondly, \hline will appear outside the tabular environment, and again fail to render. LaTeX analysis done with TeXMaker: not yet verified to fail on other platforms.

Sample printed text:
\begin{table}
\caption{Results: Ordinary least squares}
\begin{center}
\begin{tabular}{llll}
\hline
Model:              & OLS                  & Adj. R-squared:     & 0.365      \\
Dependent Variable: & Degree of Automation & AIC:                & -110.6122  \\
Date:               & 2019-01-02 12:10     & BIC:                & -67.9402   \\
No. Observations:   & 527                  & Log-Likelihood:     & 65.306     \\
Df Model:           & 10                   & F-statistic:        & 31.28      \\
Df Residuals:       & 517                  & Prob (F-statistic): & 3.09e-47   \\
R-squared:          & 0.377                & Scale:              & 0.046581   \\
\hline
\end{tabular}
\end{center}
\hline
\begin{center}
\begin{tabular}{lcccccc}
\hline
                                   &  Coef.  & Std.Err. &    t    & P$>$$|$t$|$ &  [0.025 &  0.975]  \\
\hline
\hline
\end{tabular}
\begin{tabular}{lrrrrrr}
Interacting With Computers         &  0.1504 &   0.0135 & 11.1270 &      0.0000 &  0.1239 &  0.1770  \\
Finger Dexterity                   &  0.0152 &   0.0086 &  1.7653 &      0.0781 & -0.0017 &  0.0321  \\
Originality                        & -0.0766 &   0.0122 & -6.2508 &      0.0000 & -0.1006 & -0.0525  \\
Negotiation                        &  0.0886 &   0.0157 &  5.6522 &      0.0000 &  0.0578 &  0.1195  \\
Social Perceptiveness              & -0.0605 &   0.0137 & -4.4101 &      0.0000 & -0.0875 & -0.0336  \\
Total Employees                    &  0.0068 &   0.0106 &  0.6421 &      0.5211 & -0.0141 &  0.0278  \\
Mean Hourly Pay                    &  0.0331 &   0.0161 &  2.0576 &      0.0401 &  0.0015 &  0.0647  \\
College Required                   &  0.0156 &   0.0222 &  0.7033 &      0.4822 & -0.0280 &  0.0592  \\
Susceptibility to Machine Learning &  0.0234 &   0.0103 &  2.2673 &      0.0238 &  0.0031 &  0.0437  \\
Years of Education                 & -0.1031 &   0.0234 & -4.3980 &      0.0000 & -0.1491 & -0.0570  \\
\hline
\end{tabular}
\end{center}
\hline
\begin{center}
\begin{tabular}{llll}
\hline
Omnibus:       & 36.820 & Durbin-Watson:    & 1.513   \\
Prob(Omnibus): & 0.000  & Jarque-Bera (JB): & 82.840  \\
Skew:          & 0.381  & Prob(JB):         & 0.000   \\
Kurtosis:      & 4.787  & Condition No.:    & 8       \\
\hline
\end{tabular}
\end{center}
\end{table}


Discussion:
I don't find any related changes for this part of the code in recent years. AFAICS, the substitution remained unchanged since at least 0.8, but more likely much longer
https://github.com/statsmodels/statsmodels/commit/426dab05ed07610f08b61a69c2ee19cab7a7d7aa#diff-544bd25aaa1abbe545a65f6c053fab4cL881

Is it possible that LATEX process `$$` as two separate `$`, so it never showed up as problem?
(I don't have an easy and fast way to check LATEX right now.
I (we ?) usually use pdflatex directly. 
RStudio does what you suggest automatically, but still complains about misplaced \noalign.
"misplaced \noalign" ?

You mean the `\hline` between the tables is invalid (but ignored as non-fatal error)?

```
\end{tabular}
\end{center}
\hline
\begin{center}
\begin{tabular}{lcccccc}
```

I haven't checked yet which code produces that.
For both of the programs that I tested, it causes a fatal error, but yes.
Are you using `summary()` or `summary2()`?

I don't find any `\hline` in `summary`, but they are in `summary2`. There they are explicitly added, but after searching the code for half an hour I didn't find any code that adds hlines to `summary`.

The substitution code producing `$$` is shared or the same in both summary versions.
in `summary2.Summary().as_latex` the fix looks trivial, i.e. there is an extra `\hline` when joining the tables
```
        tab = [x.as_latex_tabular() for x in simple_tables]
        tab = '\n\\hline\n'.join(tab)
```
fixing the `$$` is less obvious.
The substitutions are sorted, but replacing `$$` by `$ $` would need to be at the end.
One possibility would be to switch to an ordered data structure instead of a dictionary

Resolving patch:
diff --git a/statsmodels/iolib/table.py b/statsmodels/iolib/table.py
--- a/statsmodels/iolib/table.py
+++ b/statsmodels/iolib/table.py
@@ -450,7 +450,8 @@ def as_latex_tabular(self, center=True, **fmt_dict):
         if center:
             formatted_rows.append(r'\end{center}')
 
-        return '\n'.join(formatted_rows)
+        # Replace $$ due to bug in GH 5444
+        return '\n'.join(formatted_rows).replace('$$', ' ')
 
 
     def extend_right(self, table):


Test patch:
diff --git a/statsmodels/regression/tests/test_regression.py b/statsmodels/regression/tests/test_regression.py
--- a/statsmodels/regression/tests/test_regression.py
+++ b/statsmodels/regression/tests/test_regression.py
@@ -979,7 +979,7 @@ def test_summary_as_latex():
 \\bottomrule
 \\end{tabular}
 \\begin{tabular}{lcccccc}
-                  & \\textbf{coef} & \\textbf{std err} & \\textbf{t} & \\textbf{P$>$$|$t$|$} & \\textbf{[0.025} & \\textbf{0.975]}  \\\\
+                  & \\textbf{coef} & \\textbf{std err} & \\textbf{t} & \\textbf{P$> |$t$|$} & \\textbf{[0.025} & \\textbf{0.975]}  \\\\
 \\midrule
 \\textbf{GNPDEFL}  &      15.0619  &       84.915     &     0.177  &         0.863        &     -177.029    &      207.153     \\\\
 \\textbf{GNP}      &      -0.0358  &        0.033     &    -1.070  &         0.313        &       -0.112    &        0.040     \\\\