Repository: ipython/ipython

Issue:
& escaped to &amp; in tex ?
Reported on list.

Hi all (most probably Matthias),

This is the first time I have had to use nbconvert, and I have noticed 
that the ampersand symbol used in the eqnarray environment to align 
equations is not replaced properly in the latex output. In the latex 
file I still have &amp;=&amp; (which can't be compiled by latex), while 
in the JSON file I have the correct form, &=&. Is this a known issue? If 
so, what is the magic fix?

Cheers,
Zoltán

ping @jdfreder 



Discussion:
Can you be more specific of what you are trying to convert (and which ipython version you are using)?
If I enter in a markdowncell

``` latex
\begin{eqnarray}
1 &=& 2*3\\
41 &=& 3*6
\end{eqnarray}
```

this (currently) remains unchanged when converted to latex and compiles fine (ipython 1.1.0).
I haven't met an `&amp` yet. 

Oh, sorry. The ipython version is the latest from github (version 2.0). In fact, I believe that for version 1.1 nbcovert wasn't even part of ipython, it was a separate tool.
If I take your example, then my latex output is 

```
\begin{eqnarray}
1 &amp;=&amp; 2*3\\
41 &amp;=&amp; 3*6
\end{eqnarray}
```

I do have the ampersands in the output. I invoked nbconvert as 

python nbconvert --to latex Untitled0.ipynb

without any extra switches. Untitled0.ipynb contains only the cell that you had, nothing more. 

Strange! Which version of pandoc do you have installed?
I cannot reproduce with ipython master and pandoc 1.9.4.1. and 1.12.0.1

> python nbconvert --to latex Untitled0.ipynb

I guess you mean ipython ...

Btw. the incorporation of nbconvert into ipython was THE major milestone for the 1.0 release.

My pandoc version is 1.9.4.2. 
Yes, I meant ipython:)

Can you provide your ipynb file (although it might be really simple).
I don"t understand why pandoc uses html escapes here!

Hi Jakob,

Sure. Here is my notebook file, and also the output of

ipython notebook --to latex

Cheers,
Zoltán

On 23/09/13 11:28, Jakob Gager wrote:

> Can you provide your ipynb file (although it might be really simple).
> I don"t understand why pandoc uses html escapes here!
> 
> —
> Reply to this email directly or view it on GitHub 
> https://github.com/ipython/ipython/issues/4251#issuecomment-24907807.
> 
> {
>  "metadata": {
>   "name": ""
>  },
>  "nbformat": 3,
>  "nbformat_minor": 0,
>  "worksheets": [
>   {
>    "cells": [
>     {
>      "cell_type": "markdown",
>      "metadata": {},
>      "source": [
>       "\begin{eqnarray}\n",
>       "1 &=& 2_3\\\n",
>       "41 &=& 3_6\n",
>       "\end{eqnarray}"
>      ]
>     },
>     {
>      "cell_type": "code",
>      "collapsed": false,
>      "input": [],
>      "language": "python",
>      "metadata": {},
>      "outputs": []
>     }
>    ],
>    "metadata": {}
>   }
>  ]
> }

> I don"t understand why Pandoc uses html escapes here!

@jakobgager Maybe it's not recognizing that latex environment?  Strange that it isn't escaping incorrectly on yours.  Do you have the branch we were working on earlier, including your fix?  I know my fix will not parse this correctly (as you mentioned, it doesn't do inline latex)

Link to #4234  and #3503 

I tried with pandoc 1.10.1 and checked also the pandoc Json and pandoc clearly recognizes the `\begin{}` and `\end{}` as raw latex (block). Raw latex is passed through by pandoc when converting markdown to latex.
From my point of view, there is absolutely no reason to get any html escapes here.

@jdfreder I tried with ipython 1.1.0 and current master - not with any "advanced" markdown treatment. But maybe we should add such a check as well, because if we convert to html first this might appear in #3503.

@v923z Can you try reinstalling ipython (doesn't matter if master or 1.1.0) and convert again?

@v923z Can you try reinstalling ipython (doesn't matter if master or 1.1.0) and convert again?

Hm. I thought it had been established that the problem originates from pandoc. As for re-installation, I do that every morning when I check out the new code from github. But there is another machine, on which I installed ipython completely from scratch just two days ago (pulled code from github, version 2.0.0), and the problem still persists. pandoc version is 1.9.1.1.

For now, I can simply post-process the output (I have to remove &amp; only, I presume), so I can live with this. On the other hand, I guess, this has to be sorted out sooner or later...

Well I'm still not perfectly sure it is a pandoc issue.
Can you try in a shell:

``` bash
> echo '\begin{eqnarray}1&=&2\end{eqnarray}' | pandoc -f markdown -t json
```

Yes, here is the output:

`[{"docTitle":[],"docAuthors":[],"docDate":[]},[{"RawBlock":["latex","\\begin{eqnarray}1&=&2\\end{eqnarray}"]}]]`

So pandoc works as expected! Found as raw latex and no html escape with the _&_.
:question::question::question: still no idea

Maybe just for completion, can you try in a shell:

``` bash
> echo '\begin{eqnarray}1&=&2\end{eqnarray}' | pandoc -f markdown -t latex
```

That leaves the input unchanged. Is it what's supposed to happen?

Yes, exactly!

I downgraded pandoc to 1.9.1.1 but still get correct conversion (with ipython master).
The only way I get pandoc (I tried now with 1.9.1.1) to produce something similiar is

``` bash
> pandoc -f markdown -t html
\\begin{eqnarray}
1 &=& 2*3\\\\
\\end{eqnarray}
```

This leads to

```
<p>\begin{eqnarray}1 &amp;=&amp; 2*3\\ \end{eqnarray}</p>
```

Looks similar but has some distinct differences, and most important is a conversion to html and **not** to latex.

Maybe we have to start more fundamentally, which python, OS and shell are you using?

> Maybe we have to start more fundamentally, which python, OS and shell are you using?

The OS is either Mint Nadia, or ubuntu 12.04, and I ran the commands from the bash shell. The python version is 2.7, but I am not sure why that should matter. If you say that you don't see this problem anywhere else, then it's really weird...

So far I tried with:
- ubuntu 12.10 - python 2.7 - ipython 1.1.0 and master - pandoc 1.12.1
- ubuntu 13.04 - python 2.7 - ipython 1.1.0 - pandoc 1.10.1
- scientific linux 6 - python 2.6 - ipython 1.1.0 - pandoc 1.9.4.1

-> all worked fine
BUT!!! I finally managed to reproduce your issue with
- xubuntu 12.04 - python 2.7 - ipython master - pandoc 1.9.1.1  
  (all required packages are taken from the repository > sphinx, jinja2, pygments, tornado, pyzmq)

So maybe it is related to the "old" jinja2?? (asking @jdfreder )

The problem persists with an pandoc update (and jinja update). So no pandoc problem in the end!

Ok the problem comes from the citation2latex filter -> Only appear in master (not in 1.1.0)
The funny thing is it works with 12.10, but not with 12.04  
12.04:

``` python
In [1]: from IPython.nbconvert.filters import citation2latex

In [2]: s = "\\begin{eqnarray}\n1 &=&2 \\\\\n2 &=& 4\n\\end{eqnarray}"

In [3]: citation2latex(s)
Out[3]: u'\\begin{eqnarray}\n1 &amp;=&amp;2 \\\\\n2 &amp;=&amp; 4\n\\end{eqnarray}'
```

12.10:

``` python
In [1]: from IPython.nbconvert.filters import citation2latex

In [2]: s = "\\begin{eqnarray}\n1 &=&2 \\\\\n2 &=& 4\n\\end{eqnarray}"

In [3]: citation2latex(s)
Out[3]: '\\begin{eqnarray}\n1 &=&2 \\\\\n2 &=& 4\n\\end{eqnarray}'
```

I tried updating lxml but no success - any ideas @jdfreder, @ellisonbg 

Jakob, I completely lost the thread on the other issue: this particular problem with the ampersand is not related to the missing eqnarray environment in the html output, is it?

No, the other issue IS a pandoc issue :smile: 

It looks like lxml is designed to replace `&` with `&amp;`:
http://stackoverflow.com/questions/4972210/escaping-characters-in-a-xml-file-with-python

Looking at citation.py, all of the text is processed through the lxml logic.  

The strange thing is that with 12.10 and 13.04 lxml does not replace `&` with `&amp;`, maybe some default argument has changed?

`recover=True` maybe

Following your link, this might be a good starting point!
I'll try tomorrow.
Actually I tried to upgrade lmxl using `pip install --upgrade lxml`
 but this didn't changed a anything.

If this is the case, wouldn't it make more sense to test for whether & is replaced, and then clean up the output, if it is? 

I prefer to correct this bug at its origin instead of just concealing its effects. If there is really no way around than a subsequent testing and correction could be applied, however. 

As @jdfreder noted lxml always replaces `&` with `&amp;`. The "correct" conversions I saw with 12.10 and 13.04 are due to the test VE (there was no lxml installed) and the try...else implementation in citation2latex -> my mistake!
So the issue is present with all versions of lxml.

@v923z in the end your replace approach might be the best solution...
we could use the sax library

``` python
from lxml import html
s = "\\begin{eqnarray}\n1 &=&2 \\\\\n2 &=& 4\n\\end{eqnarray}"
t1 = html.fragment_fromstring(s, create_parent='div')
html.tostring(t1, method='html')
from xml.sax.saxutils import unescape
unescape(html.tostring(t1))
```

gives

```
'<div>\\begin{eqnarray}\n1 &=&2 \\\\\n2 &=& 4\n\\end{eqnarray}</div>'
```

# However 

I discovered something more severe in this context - if someone has the completely insane idea of writing something like `1<2` this would break the lxml approach

``` python
from lxml import html
s = "\\begin{eqnarray}\n1 &<&2 \\\\\n2 &=& 4\n\\end{eqnarray}"
t1 = html.fragment_fromstring(s, create_parent='div')
html.tostring(t1, method='html')
from xml.sax.saxutils import unescape
unescape(html.tostring(t1))
```

gives

```
'<div>\\begin{eqnarray}\n1 &</div>'
```

or with our methods:

``` python
citation2latex("1<2 is obviously true")
```

gives

```
u'1'
```

So I recommend to rewrite the `citation2latex` filter - pinging @jdfreder, @ellisonbg.


Resolving patch:
diff --git a/IPython/nbconvert/filters/citation.py b/IPython/nbconvert/filters/citation.py
--- a/IPython/nbconvert/filters/citation.py
+++ b/IPython/nbconvert/filters/citation.py
@@ -9,9 +9,17 @@
 #-----------------------------------------------------------------------------
 
 #-----------------------------------------------------------------------------
-# Code
+# Imports
 #-----------------------------------------------------------------------------
+from IPython.utils.py3compat import PY3
+if PY3:
+    from html.parser import HTMLParser
+else:
+    from HTMLParser import HTMLParser
 
+#-----------------------------------------------------------------------------
+# Functions
+#-----------------------------------------------------------------------------
 
 __all__ = ['citation2latex']
 
@@ -32,41 +40,72 @@ def citation2latex(s):
     Any HTML tag can be used, which allows the citations to be formatted
     in HTML in any manner.
     """
-    try:
-        from lxml import html
-    except ImportError:
-        return s
-
-    tree = html.fragment_fromstring(s, create_parent='div')
-    _process_node_cite(tree)
-    s = html.tostring(tree, encoding='unicode')
-    if s.endswith('</div>'):
-        s = s[:-6]
-    if s.startswith('<div>'):
-        s = s[5:]
-    return s
+    parser = CitationParser()
+    parser.feed(s)
+    parser.close()
+    outtext = u''
+    startpos = 0
+    for citation in parser.citelist:
+            outtext += s[startpos:citation[1]]
+            outtext += '\\cite{%s}'%citation[0]
+            startpos = citation[2] if len(citation)==3 else -1
+    outtext += s[startpos:] if startpos != -1 else ''
+    return outtext
 
+#-----------------------------------------------------------------------------
+# Classes
+#-----------------------------------------------------------------------------
+class CitationParser(HTMLParser):
+    """Citation Parser
 
-def _process_node_cite(node):
-    """Do the citation replacement as we walk the lxml tree."""
+    Replaces html tags with data-cite attribute with respective latex \\cite.
     
-    def _get(o, name):
-        value = getattr(o, name, None)
-        return '' if value is None else value
+    Inherites from HTMLParser, overrides:
+     - handle_starttag
+     - handle_endtag
+    """
+    # number of open tags
+    opentags = None
+    # list of found citations
+    citelist = None
+    # active citation tag
+    citetag = None
+
+    def __init__(self):
+        self.citelist = []
+        self.opentags = 0
+        HTMLParser.__init__(self)
     
-    if 'data-cite' in node.attrib:
-        cite = '\cite{%(ref)s}' % {'ref': node.attrib['data-cite']}
-        prev = node.getprevious()
-        if prev is not None:
-            prev.tail = _get(prev, 'tail') + cite + _get(node, 'tail')
-        else:
-            parent = node.getparent()
-            if parent is not None:
-                parent.text = _get(parent, 'text') + cite + _get(node, 'tail')
-        try:
-            node.getparent().remove(node)
-        except AttributeError:
-            pass
-    else:
-        for child in node:
-            _process_node_cite(child)
+    def get_offset(self):
+        # Compute startposition in source
+        lin, offset = self.getpos()
+        pos = 0
+        for i in range(lin-1):
+            pos = self.data.find('\n',pos) + 1
+        return pos + offset
+        
+    def handle_starttag(self, tag, attrs):
+        # for each tag check if attributes are present and if no citation is active
+        if self.opentags == 0 and len(attrs)>0:
+            for atr, data in attrs:
+                if atr.lower() == 'data-cite':
+                    self.citetag = tag
+                    self.opentags = 1
+                    self.citelist.append([data, self.get_offset()])
+                    return
+                
+        if tag == self.citetag:
+            # found an open citation tag but not the starting one  
+            self.opentags += 1
+  
+    def handle_endtag(self, tag):
+        if tag == self.citetag:
+            # found citation tag check if starting one
+            if self.opentags == 1:
+                pos = self.get_offset()
+                self.citelist[-1].append(pos+len(tag)+3)
+            self.opentags -= 1
+        
+    def feed(self, data):
+        self.data = data
+        HTMLParser.feed(self, data)


Test patch:
diff --git a/IPython/nbconvert/filters/tests/test_citation.py b/IPython/nbconvert/filters/tests/test_citation.py
--- a/IPython/nbconvert/filters/tests/test_citation.py
+++ b/IPython/nbconvert/filters/tests/test_citation.py
@@ -9,15 +9,13 @@
 #-----------------------------------------------------------------------------
 # Imports
 #-----------------------------------------------------------------------------
-
 from ..citation import citation2latex
 from nose.tools import assert_equal
 
 #-----------------------------------------------------------------------------
 # Tests
 #-----------------------------------------------------------------------------
-
-test_md = """
+test_md = {"""
 # My Heading
 
 Lorem ipsum dolor sit amet, consectetur adipiscing elit. Phasellus ac magna non augue
@@ -26,14 +24,13 @@
 Lorem ipsum dolor sit amet, consectetur adipiscing elit
 <em data-cite="takluyver">Thomas</em>. Quisque iaculis ligula ut ipsum mattis viverra.
 
-<p>Here is a plain paragraph that should be unaffected.</p>
+<p>Here is a plain paragraph that should be unaffected. It contains simple
+relations like 1<2 & 4>5.</p>
 
 * One <cite data-cite="jdfreder">Jonathan</cite>.
 * Two <cite data-cite="carreau">Matthias</cite>.
 * Three <cite data-cite="ivanov">Paul</cite>.
-"""
-
-test_md_parsed = """
+""":  """
 # My Heading
 
 Lorem ipsum dolor sit amet, consectetur adipiscing elit. Phasellus ac magna non augue
@@ -42,18 +39,112 @@
 Lorem ipsum dolor sit amet, consectetur adipiscing elit
 \cite{takluyver}. Quisque iaculis ligula ut ipsum mattis viverra.
 
-<p>Here is a plain paragraph that should be unaffected.</p>
+<p>Here is a plain paragraph that should be unaffected. It contains simple
+relations like 1<2 & 4>5.</p>
 
 * One \cite{jdfreder}.
 * Two \cite{carreau}.
 * Three \cite{ivanov}.
-"""
+""",
+
+# No citations
+r"""The quick brown fox jumps over the lazy dog.""": 
+r"""The quick brown fox jumps over the lazy dog.""",
+
+# Simple inline
+r"""Foo <cite data-cite=asdf>Text</cite> bar""":
+r"""Foo \cite{asdf} bar""",
+
+# Multiline
+r"""<cite data-cite=ewqr>Text
+</cite>Foo""":
+r"""\cite{ewqr}Foo""",
+
+# Nested tags
+r"""<div><div data-cite=Foo><div>Text</div></div></div> Bar""":
+r"""<div>\cite{Foo}</div> Bar""",
+
+# Including Maths
+r"""Foo $3*2*1$ <div data-cite=Foo>Text</div> Bar""":
+r"""Foo $3*2*1$ \cite{Foo} Bar""",
+
+# Missing end tag
+r"""<cite data-cite=asdf>Test Foo""":
+r"""\cite{asdf}""",
+
+r"""<cite data-cite=asdf><cite>Test Foo""":
+r"""\cite{asdf}""",
+
+r"""<cite data-cite=asdf><cite>Test</cite> Foo""":
+r"""\cite{asdf}""",
+
+# Multiple arguments
+r"""<cite width=qwer data-cite=asdf>Test</cite> Foo""":
+r"""\cite{asdf} Foo""",
+
+# Wrong capitalization
+r"""<CITE data-cite=asdf>Test</cite> Foo""":
+r"""\cite{asdf} Foo""",
+
+r"""<cite DATA-CITE=asdf>Test</cite> Foo""":
+r"""\cite{asdf} Foo""",
+
+# Wrong end tag
+r"""<asd data-cite=wer> ksjfs </asdf> sdf ds """:
+r"""\cite{wer}""",
+
+r"""<asd data-cite=wer>""":
+r"""\cite{wer}""",
+
+# Invalid tag names
+r"""<frog> <foo data-cite=wer></foo>""":
+r"""<frog> \cite{wer}""",
+
+# Non-nested tags
+r"""<strong> <h1> <cite data-cite=asdf></cite>Test</strong> Foo </h1>""":
+r"""<strong> <h1> \cite{asdf}Test</strong> Foo </h1>""",
+
+# LXML errors
+r"""Foo
+\begin{eqnarray}
+1 & <cite data-cite=bar>bar1</cite> \\
+3 & 4 \\
+\end{eqnarray}""":
+r"""Foo
+\begin{eqnarray}
+1 & \cite{bar} \\
+3 & 4 \\
+\end{eqnarray}""",
+
+r"""
+1<2 is true, but 3>4 is false.
+
+$1<2$ is true, but $3>4$ is false.
+
+1<2 it is even worse if it is alone in a line.""":
+r"""
+1<2 is true, but 3>4 is false.
+
+$1<2$ is true, but $3>4$ is false.
+
+1<2 it is even worse if it is alone in a line.""",
+
+r"""
+1 < 2 is true, but 3 > 4 is false
+
+$1 < 2$ is true, but $3 > 4$ is false
+
+1 < 2 it is even worse if it is alone in a line.
+""":
+r"""
+1 < 2 is true, but 3 > 4 is false
+
+$1 < 2$ is true, but $3 > 4$ is false
+
+1 < 2 it is even worse if it is alone in a line.
+"""}
 
 def test_citation2latex():
     """Are citations parsed properly?"""
-    try:
-        from lxml import html  #analysis:ignore
-    except ImportError:
-        assert_equal(test_md, citation2latex(test_md))
-    else:
-        assert_equal(test_md_parsed, citation2latex(test_md))
+    for input, output in test_md.items():
+        yield (assert_equal, citation2latex(input), output)