Repository: ipython/ipython

Issue:
IPython(3)Lexer fails to tokenize properly on source with magic functions
Cross-issue from jupyter/nbconvert#105.

The `IPython3Lexer` and `IPythonLexer` fail to properly tokenize on source with magic functions as seen in the example code below.

```
>>> from IPython.lib import lexers
>>> lexer = lexers.IPython3Lexer()                                                                                                                                                               
>>> fragment = "%%time\nfor i in range(5):\nprint(n)"
>>> other_fragment = "for i in range(5):\nprint(n)"
>>> list(lexer.get_tokens(fragment))
[(Token.Operator, '%%'), (Token.Keyword, 'time'), (Token.Text, '\nfor i in range(5):\nprint(n)\n')]
>>> list(lexer.get_tokens(other_fragment))                                                                                                                                                       
[(Token.Keyword, 'for'), (Token.Text, ' '), (Token.Name, 'i'), (Token.Text, ' '), (Token.Operator.Word, 'in'), (Token.Text, ' '), (Token.Name.Builtin, 'range'), (Token.Punctuation, '('), (Token.Literal.Number.Integer, '5'), (Token.Punctuation, ')'), (Token.Punctuation, ':'), (Token.Text, '\n'), (Token.Name.Builtin, 'print'), (Token.Punctuation, '('), (Token.Name, 'n'), (Token.Punctuation, ')'), (Token.Text, '\n')]
```

The expected behavior is that the remaining part of the source (`for i in range(5): print(n)`) would be tokenized properly so that pygments can syntax highlight appropriately in `nbconvert`. Looking at the test suite, it appears there isn't actually a test case in `test_lexer.py` to test for this specific situation.

I can take a stab at this.

cc: @minrk 



Discussion:
The problem is in [`IPython/lib/lexers.py`](/ipython/ipython/blob/master/IPython/lib/lexers.py#L54):

``` python
ipython_tokens = [
  (r"(?s)(\s*)(%%)(\w+)(.*)", bygroups(Text, Operator, Keyword, Text)),
  (r'(?s)(^\s*)(%%!)([^\n]*\n)(.*)', bygroups(Text, Operator, Text, using(BashLexer))),
...
]
```

If a cell begins in `%%`, the entire cell is treated as text. Bash has been special-cased in one form ([which isn't documented](http://ipython.readthedocs.org/en/stable/interactive/magics.html#cellmagic-bash)); [all other cell magics](http://ipython.readthedocs.org/en/stable/interactive/magics.html#cell-magics) will cause the cells to not be rendered.

A simple fix would be to add tokens above these for each of the built-in cell magics, handling the cell content as appropriate (along the lines of using `BashLexer` above).

A more intelligent fix would be to have each cell magic handler provide its own token, and have `lexers.py` collate and insert them before `ipython_tokens`. This would have the benefit of enabling custom cell magics to define how they get highlighted.

@pythonian4000 Want to submit a PR?

I probably will, but only after I've submitted my PhD thesis :stuck_out_tongue_winking_eye: (I encountered the bug while in the midst of final formatting).

A friendly ping to _Dr._ @pythonian4000 :) or @captainsafia  if they wanted to revisit submitting a fix for this? As mentioned in the linked Issue from nbsphinx, this also affects notebooks that get rendered to Sphinx, put on readthedocs, etc... 

Ping?

Resolving patch:
diff --git a/IPython/lib/lexers.py b/IPython/lib/lexers.py
--- a/IPython/lib/lexers.py
+++ b/IPython/lib/lexers.py
@@ -34,7 +34,9 @@
 import re
 
 # Third party
-from pygments.lexers import BashLexer, PythonLexer, Python3Lexer
+from pygments.lexers import (
+    BashLexer, HtmlLexer, JavascriptLexer, RubyLexer, PerlLexer, PythonLexer,
+    Python3Lexer, TexLexer)
 from pygments.lexer import (
     Lexer, DelegatingLexer, RegexLexer, do_insertions, bygroups, using,
 )
@@ -51,19 +53,6 @@
            'IPythonPartialTracebackLexer', 'IPythonTracebackLexer',
            'IPythonConsoleLexer', 'IPyLexer']
 
-ipython_tokens = [
-  (r"(?s)(\s*)(%%)(\w+)(.*)", bygroups(Text, Operator, Keyword, Text)),
-  (r'(?s)(^\s*)(%%!)([^\n]*\n)(.*)', bygroups(Text, Operator, Text, using(BashLexer))),
-  (r"(%%?)(\w+)(\?\??)$",  bygroups(Operator, Keyword, Operator)),
-  (r"\b(\?\??)(\s*)$",  bygroups(Operator, Text)),
-  (r'(%)(sx|sc|system)(.*)(\n)', bygroups(Operator, Keyword,
-                                       using(BashLexer), Text)),
-  (r'(%)(\w+)(.*\n)', bygroups(Operator, Keyword, Text)),
-  (r'^(!!)(.+)(\n)', bygroups(Operator, using(BashLexer), Text)),
-  (r'(!)(?!=)(.+)(\n)', bygroups(Operator, using(BashLexer), Text)),
-  (r'^(\s*)(\?\??)(\s*%{0,2}[\w\.\*]*)', bygroups(Text, Operator, Text)),
-  (r'(\s*%{0,2}[\w\.\*]*)(\?\??)(\s*)$', bygroups(Text, Operator, Text)),
-]
 
 def build_ipy_lexer(python3):
     """Builds IPython lexers depending on the value of `python3`.
@@ -92,6 +81,36 @@ def build_ipy_lexer(python3):
         aliases = ['ipython2', 'ipython']
         doc = """IPython Lexer"""
 
+    ipython_tokens = [
+       (r'(?s)(\s*)(%%capture)([^\n]*\n)(.*)', bygroups(Text, Operator, Text, using(PyLexer))),
+        (r'(?s)(\s*)(%%debug)([^\n]*\n)(.*)', bygroups(Text, Operator, Text, using(PyLexer))),
+        (r'(?is)(\s*)(%%html)([^\n]*\n)(.*)', bygroups(Text, Operator, Text, using(HtmlLexer))),
+        (r'(?s)(\s*)(%%javascript)([^\n]*\n)(.*)', bygroups(Text, Operator, Text, using(JavascriptLexer))),
+        (r'(?s)(\s*)(%%js)([^\n]*\n)(.*)', bygroups(Text, Operator, Text, using(JavascriptLexer))),
+        (r'(?s)(\s*)(%%latex)([^\n]*\n)(.*)', bygroups(Text, Operator, Text, using(TexLexer))),
+        (r'(?s)(\s*)(%%pypy)([^\n]*\n)(.*)', bygroups(Text, Operator, Text, using(PerlLexer))),
+        (r'(?s)(\s*)(%%prun)([^\n]*\n)(.*)', bygroups(Text, Operator, Text, using(PyLexer))),
+        (r'(?s)(\s*)(%%pypy)([^\n]*\n)(.*)', bygroups(Text, Operator, Text, using(PyLexer))),
+        (r'(?s)(\s*)(%%python)([^\n]*\n)(.*)', bygroups(Text, Operator, Text, using(PyLexer))),
+        (r'(?s)(\s*)(%%python2)([^\n]*\n)(.*)', bygroups(Text, Operator, Text, using(PythonLexer))),
+        (r'(?s)(\s*)(%%python3)([^\n]*\n)(.*)', bygroups(Text, Operator, Text, using(Python3Lexer))),
+        (r'(?s)(\s*)(%%ruby)([^\n]*\n)(.*)', bygroups(Text, Operator, Text, using(RubyLexer))),
+        (r'(?s)(\s*)(%%time)([^\n]*\n)(.*)', bygroups(Text, Operator, Text, using(PyLexer))),
+        (r'(?s)(\s*)(%%timeit)([^\n]*\n)(.*)', bygroups(Text, Operator, Text, using(PyLexer))),
+        (r'(?s)(\s*)(%%writefile)([^\n]*\n)(.*)', bygroups(Text, Operator, Text, using(PyLexer))),
+        (r"(?s)(\s*)(%%)(\w+)(.*)", bygroups(Text, Operator, Keyword, Text)),
+        (r'(?s)(^\s*)(%%!)([^\n]*\n)(.*)', bygroups(Text, Operator, Text, using(BashLexer))),
+        (r"(%%?)(\w+)(\?\??)$",  bygroups(Operator, Keyword, Operator)),
+        (r"\b(\?\??)(\s*)$",  bygroups(Operator, Text)),
+        (r'(%)(sx|sc|system)(.*)(\n)', bygroups(Operator, Keyword,
+                                                using(BashLexer), Text)),
+        (r'(%)(\w+)(.*\n)', bygroups(Operator, Keyword, Text)),
+        (r'^(!!)(.+)(\n)', bygroups(Operator, using(BashLexer), Text)),
+        (r'(!)(?!=)(.+)(\n)', bygroups(Operator, using(BashLexer), Text)),
+        (r'^(\s*)(\?\??)(\s*%{0,2}[\w\.\*]*)', bygroups(Text, Operator, Text)),
+        (r'(\s*%{0,2}[\w\.\*]*)(\?\??)(\s*)$', bygroups(Text, Operator, Text)),
+    ]
+
     tokens = PyLexer.tokens.copy()
     tokens['root'] = ipython_tokens + tokens['root']
 


Test patch:
diff --git a/IPython/lib/tests/test_lexers.py b/IPython/lib/tests/test_lexers.py
--- a/IPython/lib/tests/test_lexers.py
+++ b/IPython/lib/tests/test_lexers.py
@@ -127,3 +127,50 @@ def testIPythonLexer(self):
             (Token.Text, '\n'),
         ]
         self.assertEqual(tokens, list(self.lexer.get_tokens(fragment)))
+
+        fragment = '%%writefile -a foo.py\nif a == b:\n    pass'
+        tokens = [
+            (Token.Operator, '%%writefile'),
+            (Token.Text, ' -a foo.py\n'),
+            (Token.Keyword, 'if'),
+            (Token.Text, ' '),
+            (Token.Name, 'a'),
+            (Token.Text, ' '),
+            (Token.Operator, '=='),
+            (Token.Text, ' '),
+            (Token.Name, 'b'),
+            (Token.Punctuation, ':'),
+            (Token.Text, '\n'),
+            (Token.Text, '    '),
+            (Token.Keyword, 'pass'),
+            (Token.Text, '\n'),
+        ]
+        self.assertEqual(tokens, list(self.lexer.get_tokens(fragment)))
+
+        fragment = '%%timeit\nmath.sin(0)'
+        tokens = [
+            (Token.Operator, '%%timeit\n'),
+            (Token.Name, 'math'),
+            (Token.Operator, '.'),
+            (Token.Name, 'sin'),
+            (Token.Punctuation, '('),
+            (Token.Literal.Number.Integer, '0'),
+            (Token.Punctuation, ')'),
+            (Token.Text, '\n'),
+        ]
+
+        fragment = '%%HTML\n<div>foo</div>'
+        tokens = [
+            (Token.Operator, '%%HTML'),
+            (Token.Text, '\n'),
+            (Token.Punctuation, '<'),
+            (Token.Name.Tag, 'div'),
+            (Token.Punctuation, '>'),
+            (Token.Text, 'foo'),
+            (Token.Punctuation, '<'),
+            (Token.Punctuation, '/'),
+            (Token.Name.Tag, 'div'),
+            (Token.Punctuation, '>'),
+            (Token.Text, '\n'),
+        ]
+        self.assertEqual(tokens, list(self.lexer.get_tokens(fragment)))