Add an 'escape' keyword to DataFrame.to_latex
State: closed

closes #6472

The new keyword prevents the conversion from escaping column names

Currently, a sequence like $e^x$ gets properly formatted when you display a dataframe in an ipython notebook, but gets escaped when converted to latex.


Labels:
name: Enhancement
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
can you add a test or 2 for this? and add a release notes mention under enhancements mentioning #6472

Sure, if you think that the approach taken is valid in principle.

@jseabold I think you wanted this kw?

Hmm, it seems to me, you might just want control over the characters you escape, right? Or more to the point, you won't to escape all characters except `$` because it has a different meaning in this context whether it's escaped or not, while some might need to stay escaped. Do any of of the other characters meaning change context? I know for the tilde I was going through and replacing the `\\textilde` with `\sim` recently. Maybe a skip_replace keyword that _removes_ characters from the set

```
'[{}]'.format(re.escape(string.punctuation))
```

?

Or whatever else is currently in there. 

I'm by no means an expert and I don't think I understood all you've written. But I think to be most accurate one would just need to disable escaping for characters nested between two `$`-characters.

Currently `^$^$` 
will be 
`\textasciicircum\$\textasciicircum\$`
while it should be
`\textasciicircum$^$`

So by _removing_ characters from the escape-set you will not get the expected result. Apart from this, specifying a removal-set does not seem future-proof, as the set may suddenly change and you would constantly need to add additional characters to the removal-set when updating to a newer version.

IMHO if somebody wants to have control over the output he should be able to just disable the escaping and take care of it himself, this way one gets maximum control.

Or is there something I missed?

Thinking about it again... I might want to have a citation somewhere (`\cite{TheInternet2014}`) where I would not want the `\` to be escaped despite not being nested between `$`.

I was thinking the removal set would be a user option, not hard-coded, but what you say makes sense. We could use regex to not escape between matching `$` pairs. Should we also include `$$` pairs, `\(..\)`, and `\[...\]`? I mainly use `$` in tables.

Supposing you could have this skip escaping in equation environments behavior, wouldn't this reduce the need for fiddling? An escape-all or escape-none approach seems like you're going to have to fiddle at some point either way. 

Right, so it sounds like we're on the same page? I'll see if I can do this quickly unless your comfortable with regex and want to give it a shot.

I should actually be doing research, so I'm happy if I don't have to take care of it. I just wanted to share my solution (call it workaround if you like :) ) to this problem.

Time spent away from research on open source is ok. I checked. ;)

Yeah, that's a trickier regex than I have time for right now.

That's what I was telling myself as well all the time and the reason for the deadline approaching quite fast meanwhile ;)

I guess it's quite hard to get a regex right which is actually able to distinguish between the use of a `\` in a latex-command and in another context. And both may appear in the same content which is being converted in one single `to_latex` call.
Therefore I would say it is best to be able to completely turn off escaping. This requires "fiddling" but at least one is able to have full control over the output.
On the other hand it leaves room for improvements on the (still default) escape-mode turned on, which could implement a "only escape characters like: ^° etc. when they are not nested within two $"-blackbox-intelligence which will probably be what people want in 95% of the cases.

@m-kuhn how's this coming?

AFAIK, the regex in python isn't really stateful (which it is elsewhere IIUC), so it's pretty hard to make a regex that distinguishes characters only when they're nested in `$...$`. Not escaping everything is definitely one way to go, and is probably worth it for this special case as a stop-gap. But it definitely shouldn't be the default. It's very rare that I actual write latex math mark-up in a pandas object.

There's probably a pretty simple solution that doesn't involve regex that just walks the text and keeps track if its in a math environment or not. It won't be very fast, but if you're writing a table big enough for this to be an issue you're probably "doing it wrong."

What do projects like [asciitable](http://cxc.harvard.edu/contrib/asciitable/) do? I don't think SimpleTable in statsmodels handles this either.

As I understand it:
- Merging this branch is ok, as long as escaping is not turned off by default.
- The default "intelligent escaping" way could be optimized in the future, but that's outside the scope of this pull request and will require somebody with more in-depth knowlege of latex than me.

I can push the release notes and a small test as soon as I get the ok for this PR.

@m-kuhn this seems fine, can you just add a release note and a small test (and rebase). If later you want to come back and do a more involved PR I think we'd be open to it.

Added release note and a simple test

use `u(...)` to specify the unicode (in test_format.py)

Thanks for the pointer. Hope that fixes it now...

@m-kuhn can you rebase and force push again

Sure, just waiting for Travis to give its feedback about the current version. Compatibility with python 3.2 was quite demanding.

ahh...ok...that's what I was looking for....

I eventually removed the unicode `u()` formatting for the multiline string containing the escape sequences... I didn't find a way to make it behave the same way in 2.x and 3.x. However, as it's all latin1 supported it shouldn't be a problem.

@m-kuhn excellect

@jseabold ok on this?

@jseabold you ok with this?

In principal, as a stop-gap, sure. I'm not wild about the setting an attribute rather than the passing an argument, but I see that pattern is used elsewhere. 

There should probably be a separate issue about conditional formatting depending on whether you are in a math environment or not. It might actually be easiest to remove the math with regex, _then_ apply the escaping, and stick it all back together now that I'm thinking about it again, instead of trying to do the regex replace in one go, forcing the regex to be stateful when that's not easy to do in Python. 

Thanks for the feedback.
When opening a separate issue, please make sure to also mention and keep in mind that commands may appear outside math environments (E.g. `\cite{jseabold2014}`)

@jseabold how is this setting an attribute? it IS a passed keyword (the setting is done on an internal class only; not exposed to the user).

Yeah, I realize it's internal only. Compare the escape keyword vs. the recently added longtable keyword. Escape is set as an attribute in `__init__` only to use it later in only one method. Not exactly "explicit is better than implicit" but admittedly nit-picky.

oh i c....yeh...prob needs some cleanup at some point...ok will merge this