Repository: ipython/ipython

Issue:
Nbconvert [windows]: Inconsistent line endings in markdown cells exported to latex 
I'm getting what looks like inconsistent line endings in the output of markdown cells, which is causing latex to complain about runaway arguments and paragraphs ending early.

The exact error is:

```
! Paragraph ended before \ttl@straight@i was complete
```

And looking at the offending line in the `.tex` file (in emacs):

```
\subsection{ Text that doesn't have line breaks in it in the markdown cell^M^M    
 but does seem to have two CRLF's (judging by the ^M's), and a LF here.}
```

Taking the `^M`'s out will allow pdflatex to produce the output, but there are still ^M's scattered throughout the document. 

As a guess it's that [Pandoc explicitly expects UTF-8](http://johnmacfarlane.net/pandoc/README.html#using-pandoc), and somehow the subprocess is piping in CP-850 or similar windows encoding and pandoc is outputting some combination of the two line endings.

I've tried to recreate the issue just with Pandoc, inputting various encoding schemes but I couldn't recreate it.

Alternatively, it may be the utf-8 encoding casting that occurs in `IPython/nbconvert/utils/pandoc.py` which is adding LF but not removing the CRLFs ( but I'm still not sure why there are any line breaks as the original cell doesn't have them).

Sys info.

```
{'codename': 'An Afternoon Hack',
 'commit_hash': '02da31c',
 'commit_source': 'installation',
 'default_encoding': 'cp850',
 'ipython_path': 'C:\\Python27\\lib\\site-packages\\IPython',
 'ipython_version': '1.0.0',
 'os_name': 'nt',
 'platform': 'Windows-7-6.1.7601-SP1',
 'sys_executable': 'C:\\Python27\\python.exe',
 'sys_platform': 'win32',
 'sys_version': '2.7.3 |EPD_free 7.3-2 (32-bit)| (default, Apr 12 2012, 14:30:3
 ) [MSC v.1500 32 bit (Intel)]'}
```



Discussion:
Did you use heading in Markdown cell or a Header cell to produce that ?

Markdown cell

On Wed, Aug 28, 2013 at 4:47 PM, Matthias Bussonnier <
notifications@github.com> wrote:

> Did you use heading in Markdown cell or a Header cell to produce that ?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/ipython/ipython/issues/4122#issuecomment-23394581
> .

Then can you try in a header cell. Headers in markdown is a complicated things to get right, an I'm not sure we will support it outside if headings cells.

Envoyé de mon iPhone

Le 28 août 2013 à 11:58, mrmagooey notifications@github.com a écrit :

> Markdown cell 
> 
> On Wed, Aug 28, 2013 at 4:47 PM, Matthias Bussonnier < 
> notifications@github.com> wrote: 
> 
> > Did you use heading in Markdown cell or a Header cell to produce that ? 
> > 
> > — 
> > Reply to this email directly or view it on GitHubhttps://github.com/ipython/ipython/issues/4122#issuecomment-23394581 
> > . 
> > 
> > —
> > Reply to this email directly or view it on GitHub.

Same thing in a header cell, the combination of line endings is again tripping up pdflatex. Removing the ^M next to the LF lets latex move to the next issue which is that latex can't find the images referenced in the markdown cells (using the `![caption](files/etc..)` notation, which I should probably file a separate issue for).

Hum, weird, header cell should not have any line ending, as text should be forced to be one line, and `\section{``}` added automatically.

`![caption](...)` is a known issue. Someone have to write a MD parser that is smart enough to detect those and grab them from the file system. But this is a more general issue.
I is difficult to solve, as `![]()` syntax refers to url that you might not want to download as they are dynamic.

So using a Ubuntu VM with a link to the Windows 7 host filesystem the notebook files are happily converted to pdfs, this is the interim solution I'm using and it appears to work quite well.

Is there an open issue for the markdown `![caption](...)` problem? I have a potential pull request on it.

I was able to reproduce the CR + CR + LF line endings on my machine with both Python 2.7 and Python 3.3 on Windows 7 x64.  It looks like you only have the problem when the text is enclosed in a fragile latex environment that doesn't support line endings.  In other words, the latex failure is expected but not the new lines that are causing it.

How did you get the new lines in the heading blocks via the Markdown cell (without modifying the tex by hand)?  I can't even manage to get them in (nor should I be able to, without editing the tex by hand, like @Carreau mentioned above).  

Now, I can make my tex fail, if I type line breaks into a heading cell.  However, the line breaks aren't supported in heading cells, and should be removed.  In the notebook, when I deselect the cell, the line breaks are not visible (proper behavior).  BUT, when I export that same file using nbconvert, the line breaks make it all the way through to the tex.  **This is the bug.**  Now I think you were onto something in your original post, the CR +CR + LF line endings may be confusing the code that is supposed to remove all line endings in headers.  _I will look into this right now._

If you're willing to, please post the notebook, or just the Markdown portion of it, that is causing the trouble so I can try it on my machine.  

> open issue for the markdown `![caption](...)` problem?

Not that I know of

It looks like the sphinx template was the only template that wasn't removing new lines from headers.  #4181 should fix this problem with the heading cells (but doesn't do anything for the markdown).  

We use pandoc to produce our markdown output.  If you are producing headers with new lines in them through the Markdown cells, it is _probably_ a pandoc bug.

All of the latex templates are being refactored right now, bugs like this should be much easier to fix in the future (see #4112).

So it seems like any header cell over ~72 characters is getting a newline inserted into it, happening for me on both a recent master of IPython on Ubuntu 12.04 and on the pypi distributed version on Windows 7. On Windows the line breaks are of a `CRLF` type and that seems to cause latex to stop, however on Ubuntu latex will ignore the `LF`. To test this I merely have a new notebook with increasing heading sizes. 

I'm not sure where these newlines are getting inserted, so the above commit may fix it.

@mrmagooey can you check that you can export without failure now that #4181 is merged?  I'm no longer having trouble on Winows or Linux (using 100+ chaacters in a header in both the markdown and builtin headers).  However, I do still see a rogue `CR` at the end of the headers on the Windows machine.  

Scratch that, if I put spaces in the long headers, it still has problems on the Windows installation.  Looking...


Resolving patch:
diff --git a/IPython/nbconvert/utils/pandoc.py b/IPython/nbconvert/utils/pandoc.py
--- a/IPython/nbconvert/utils/pandoc.py
+++ b/IPython/nbconvert/utils/pandoc.py
@@ -15,6 +15,7 @@
 
 # Stdlib imports
 import subprocess
+from io import TextIOWrapper, BytesIO
 
 # IPython imports
 from IPython.utils.py3compat import cast_bytes
@@ -64,6 +65,6 @@ def pandoc(source, fmt, to, extra_args=None, encoding='utf-8'):
             "http://johnmacfarlane.net/pandoc/installing.html"
         )
     out, _ = p.communicate(cast_bytes(source, encoding))
-    out = out.decode(encoding, 'replace')
-    return out[:-1]
+    out = TextIOWrapper(BytesIO(out), encoding, 'replace').read()
+    return out.rstrip('\n')
 


Test patch:
diff --git a/IPython/nbconvert/tests/files/notebook2.ipynb b/IPython/nbconvert/tests/files/notebook2.ipynb
--- a/IPython/nbconvert/tests/files/notebook2.ipynb
+++ b/IPython/nbconvert/tests/files/notebook2.ipynb
@@ -123,6 +123,14 @@
      ],
      "prompt_number": 10
     },
+    {
+     "cell_type": "heading",
+     "level": 2,
+     "metadata": {},
+     "source": [
+      "Here is a very long heading that pandoc will wrap and wrap and wrap and wrap and wrap and wrap and wrap and wrap and wrap and wrap and wrap and wrap"
+     ]
+    },
     {
      "cell_type": "markdown",
      "metadata": {},
diff --git a/IPython/nbconvert/tests/test_nbconvertapp.py b/IPython/nbconvert/tests/test_nbconvertapp.py
--- a/IPython/nbconvert/tests/test_nbconvertapp.py
+++ b/IPython/nbconvert/tests/test_nbconvertapp.py
@@ -103,6 +103,20 @@ def test_post_processor(self):
             assert os.path.isfile('notebook1.tex')
             assert os.path.isfile('notebook1.pdf')
 
+    @dec.onlyif_cmds_exist('pandoc')
+    def test_spurious_cr(self):
+        """Check for extra CR characters"""
+        with self.create_temp_cwd(['notebook2.ipynb']):
+            self.call('nbconvert --log-level 0 --to latex notebook2')
+            assert os.path.isfile('notebook2.tex')
+            with open('notebook2.tex') as f:
+                tex = f.read()
+            self.call('nbconvert --log-level 0 --to html notebook2')
+            assert os.path.isfile('notebook2.html')
+            with open('notebook2.html') as f:
+                html = f.read()
+        self.assertEqual(tex.count('\r'), tex.count('\r\n'))
+        self.assertEqual(html.count('\r'), html.count('\r\n'))
 
     @dec.onlyif_cmds_exist('pandoc')
     def test_png_base64_html_ok(self):
@@ -114,7 +128,6 @@ def test_png_base64_html_ok(self):
             with open('notebook2.html') as f:
                 assert "data:image/png;base64,b'" not in f.read()
 
-
     @dec.onlyif_cmds_exist('pandoc')
     def test_template(self):
         """