Repository: ipython/ipython

Issue:
MathJax rendering problem in `%%latex` cell
The following screenshot shows the rendered output of one of our example notebooks.  The odd thing is that it works fine when the `nabla` comes back through a `Latex` display object.  See our example notebook "Part 5 - Rich Display System" for the examples.

![screen shot 2013-07-10 at 3 47 21 pm](https://f.cloud.github.com/assets/27600/778514/c6980318-e9b2-11e2-85f2-c8b26c9925e5.png)



Discussion:
Ahh, I see the problem.  In a `%%latex` cell the ``\\` at the end of the line is being collapsed into the `\nabla` on the following line.  Looks like we are removing newlines in `%%latex` cells now?

Nope, that's not something we have ever done.  I believe this is the same issue we have always had that forced the use of `\\\` due to escape issues.  We (@ahmadia, I think) have fixed it in markdown in _some_ cases, but I don't think the same fix is applied to the output of a latex cell.

But the odd thing is that when we pass the same latex source to the Latex
object it works fine.

On Wed, Jul 10, 2013 at 4:18 PM, Min RK notifications@github.com wrote:

> Nope, that's not something we have ever done. I believe this is the same
> issue we have always had that forced the use of \\ due to escape issues.
> We (@ahmdia, I think) have fixed it in markdown in _some_ cases, but I
> don't think the same fix is applied to the output of a latex cell.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/ipython/ipython/issues/3604#issuecomment-20780590
> .

## 

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com

crazy, maybe we did add `''.join` somewhere that I'm not aware of.  Will investigate.

My guess is we don't handle escaping backslashes properly in the cell magics

Just to chip in, I didn't touch anything besides the javascript for
Markdown cell-rendering.  I definitely didn't touch any of the Python code
for handling cell magics.

On Wed, Jul 10, 2013 at 7:32 PM, Min RK notifications@github.com wrote:

> My guess is we don't handle escaping backslashes properly in the cell
> magics
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/ipython/ipython/issues/3604#issuecomment-20781082
> .

I think it's the input transform, etc. machinery.  When a line ends with `\`, the following newline is not added.

I believe this is due to [`assemble_logical_lines`](https://github.com/ipython/ipython/blob/d99b6cf73dbd3678b66f6deee97796aac91f0e4e/IPython/core/inputtransformer.py#L165) being called on the cell, which it should not be when it is a cell magic.  @takluyver any thoughts on how to prevent transformers from being applied when it is a cell magic?

Notebook [illustrating](http://nbviewer.ipython.org/5971323) the problematic behavior.  It's definitely the input transformer that interprets trailing `\` as a line continuation. I don't know how to add the logic that zero transformers should be applied (other than the cell magic transform itself) when it is a cell magic.

@ahmadia thanks for reminding me.  After diving further into code, it's apparently totally unrelated to that (though the workaround is actually the same).

Hasn't there been other cases where the transformation logic has been
applied to cell magics?  It seems like cell magics should never have their
contents touched and that the relevant code paths for that are too subtle
perhaps?

On Wed, Jul 10, 2013 at 4:55 PM, Min RK notifications@github.com wrote:

> I think it's the input transform, etc. machinery. When a line ends with \,
> the following newline is not added.
> 
> I believe this is due to assemble_logical_lineshttps://github.com/ipython/ipython/blob/d99b6cf73dbd3678b66f6deee97796aac91f0e4e/IPython/core/inputtransformer.py#L165being called on the cell, which it should not be when it is a cell magic.
> @takluyver https://github.com/takluyver any thoughts on how to prevent
> transformers from being applied when it is a cell magic?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/ipython/ipython/issues/3604#issuecomment-20781894
> .

## 

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com

I'm trying to read and understand the input splitter and transformer logic (this is really the domain of @takluyver and @fperez), but yes, I think that the understanding of cell magics is that _zero_ transforms should be applied. Part of what makes this tricky is that the cell magic is itself implemented as an input transform.


Resolving patch:
diff --git a/IPython/core/inputsplitter.py b/IPython/core/inputsplitter.py
--- a/IPython/core/inputsplitter.py
+++ b/IPython/core/inputsplitter.py
@@ -486,17 +486,19 @@ def __init__(self, line_input_checker=True, physical_line_transforms=None,
         if physical_line_transforms is not None:
             self.physical_line_transforms = physical_line_transforms
         else:
-            self.physical_line_transforms = [leading_indent(),
+            self.physical_line_transforms = [
+                                             leading_indent(),
                                              classic_prompt(),
                                              ipy_prompt(),
                                              strip_encoding_cookie(),
+                                             cellmagic(end_on_blank_line=line_input_checker),
                                             ]
         
         self.assemble_logical_lines = assemble_logical_lines()
         if logical_line_transforms is not None:
             self.logical_line_transforms = logical_line_transforms
         else:
-            self.logical_line_transforms = [cellmagic(end_on_blank_line=line_input_checker),
+            self.logical_line_transforms = [
                                             help_end(),
                                             escaped_commands(),
                                             assign_from_magic(),
diff --git a/IPython/core/inputtransformer.py b/IPython/core/inputtransformer.py
--- a/IPython/core/inputtransformer.py
+++ b/IPython/core/inputtransformer.py
@@ -227,6 +227,8 @@ def _tr_help(line_info):
 def _tr_magic(line_info):
     "Translate lines escaped with: %"
     tpl = '%sget_ipython().magic(%r)'
+    if line_info.line.startswith(ESC_MAGIC2):
+        return line_info.line
     cmd = ' '.join([line_info.ifun, line_info.the_rest]).strip()
     return tpl % (line_info.pre, cmd)
 
@@ -272,7 +274,8 @@ def escaped_commands(line):
                               [a-zA-Z_*][\w*]*        # Variable name
                               (\.[a-zA-Z_*][\w*]*)*   # .etc.etc
                               )
-                              (\?\??)$                # ? or ??""",
+                              (\?\??)$                # ? or ??
+                              """,
                               re.VERBOSE)
 
 def has_comment(src):
@@ -328,7 +331,14 @@ def cellmagic(end_on_blank_line=False):
     line = ''
     while True:
         line = (yield line)
-        if (not line) or (not line.startswith(ESC_MAGIC2)):
+        # consume leading empty lines
+        while not line:
+            line = (yield line)
+        
+        if not line.startswith(ESC_MAGIC2):
+            # This isn't a cell magic, idle waiting for reset then start over
+            while line is not None:
+                line = (yield line)
             continue
         
         if cellmagic_help_re.match(line):
diff --git a/IPython/core/interactiveshell.py b/IPython/core/interactiveshell.py
--- a/IPython/core/interactiveshell.py
+++ b/IPython/core/interactiveshell.py
@@ -1672,7 +1672,13 @@ def _get_exc_info(self, exc_tuple=None):
         
         return etype, value, tb
     
-
+    def show_usage_error(self, exc):
+        """Show a short message for UsageErrors
+        
+        These are special exceptions that shouldn't show a traceback.
+        """
+        self.write_err("UsageError: %s" % exc)
+    
     def showtraceback(self,exc_tuple = None,filename=None,tb_offset=None,
                       exception_only=False):
         """Display the exception that just occurred.
@@ -1698,7 +1704,7 @@ def showtraceback(self,exc_tuple = None,filename=None,tb_offset=None,
                 # line, there may be SyntaxError cases with imported code.
                 self.showsyntaxerror(filename)
             elif etype is UsageError:
-                self.write_err("UsageError: %s" % value)
+                self.show_usage_error(value)
             else:
                 if exception_only:
                     stb = ['An exception has occurred, use %tb to see '


Test patch:
diff --git a/IPython/core/tests/test_inputsplitter.py b/IPython/core/tests/test_inputsplitter.py
--- a/IPython/core/tests/test_inputsplitter.py
+++ b/IPython/core/tests/test_inputsplitter.py
@@ -446,6 +446,24 @@ def test_syntax_multiline_cell(self):
                 out = isp.transform_cell(raw)
                 # Match ignoring trailing whitespace
                 self.assertEqual(out.rstrip(), out_t.rstrip())
+    
+    def test_cellmagic_preempt(self):
+        isp = self.isp
+        for raw, name, line, cell in [
+            ("%%cellm a\nIn[1]:", u'cellm', u'a', u'In[1]:'),
+            ("%%cellm \nline\n>>>hi", u'cellm', u'', u'line\n>>>hi'),
+            (">>>%%cellm \nline\n>>>hi", u'cellm', u'', u'line\nhi'),
+            ("%%cellm \n>>>hi", u'cellm', u'', u'hi'),
+            ("%%cellm \nline1\nline2", u'cellm', u'', u'line1\nline2'),
+            ("%%cellm \nline1\\\\\nline2", u'cellm', u'', u'line1\\\\\nline2'),
+        ]:
+            expected = "get_ipython().run_cell_magic(%r, %r, %r)" % (
+                name, line, cell
+            )
+            out = isp.transform_cell(raw)
+            self.assertEqual(out.rstrip(), expected.rstrip())
+        
+        
 
 #-----------------------------------------------------------------------------
 # Main - use as a script, mostly for developer experiments
diff --git a/IPython/core/tests/test_inputtransformer.py b/IPython/core/tests/test_inputtransformer.py
--- a/IPython/core/tests/test_inputtransformer.py
+++ b/IPython/core/tests/test_inputtransformer.py
@@ -259,6 +259,9 @@ def transform_checker(tests, transformer, **kwargs):
           (u'hello', None),
           (None , u_fmt("get_ipython().run_cell_magic({u}'bar', {u}'123', {u}'hello')")),
           ],
+         [(u'a=5', 'a=5'),
+          (u'%%cellmagic', '%%cellmagic'),
+          ],
        ],
        
        escaped =