Repository: ipython/ipython

Issue:
Exceptions thrown in cell with multiline strings
It seems that when an exception is raised from a cell with a multiline string, we get a code-red error.

Try:

```
raise Exception(r'''foo
''')
```

It will report:

```
ERROR: An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line string', (1, 0))

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-15-8d28bd905cfc> in <module>()
      1 raise Exception(r'''foo
----> 2 ''')

Exception: foo
```

The first three lines are worrying for the casual user.



Discussion:
I'm still seeing this in a current version of iPython. The following cell in a python 3 notebook generates an error whenever an exception is thrown:

```py
print(arg="""
""")
```

gives me:

```
ERROR:root:An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line string', (1, 0))
```

However adding a comment avoids the error. This works:

```py
print(arg="""
""")
# """
```
So I ran into this today and dug a little; the issue here is that the way `ultratb.py` is using the `tokenize` module doesn't play nicely with multiline `"""` strings. In particular, when using a multiline string as an argument, the error is reported at the *last* line, whereas it's the *first* line for (say) a `\`-continued `"` string.

I would be happy to send a PR to squelch the error in this particular case, unless there are objections.
Ah, and on thinking about this a little more, the reason that single-line strings work well has nothing to do with Python, and everything to do with this snippet:
https://github.com/ipython/ipython/blob/f35cf047006e1398816c697fee19c28de77d9f1d/IPython/core/inputtransformer.py#L173-L191
One side-effect of this line is that `\`-continued strings are assembled before execution, meaning that `ultratb` won't mistakenly start parsing from the **end** of such a string.
Thanks @craigcitro for investigating. I think it's fine to silence the error in ultratb.

Resolving patch:
diff --git a/IPython/core/ultratb.py b/IPython/core/ultratb.py
--- a/IPython/core/ultratb.py
+++ b/IPython/core/ultratb.py
@@ -121,7 +121,7 @@
 from IPython.utils import py3compat
 from IPython.utils.data import uniq_stable
 from IPython.utils.terminal import get_terminal_size
-from logging import info, error
+from logging import info, error, debug
 
 import IPython.utils.colorable as colorable
 
@@ -952,10 +952,15 @@ def linereader(file=file, lnum=[lnum], getline=linecache.getline):
             #  - see gh-6300
             pass
         except tokenize.TokenError as msg:
+            # Tokenizing may fail for various reasons, many of which are
+            # harmless. (A good example is when the line in question is the
+            # close of a triple-quoted string, cf gh-6864). We don't want to
+            # show this to users, but want make it available for debugging
+            # purposes.
             _m = ("An unexpected error occurred while tokenizing input\n"
                   "The following traceback may be corrupted or invalid\n"
                   "The error message is: %s\n" % msg)
-            error(_m)
+            debug(_m)
 
         # Join composite names (e.g. "dict.fromkeys")
         names = ['.'.join(n) for n in names]


Test patch:
diff --git a/IPython/core/tests/test_ultratb.py b/IPython/core/tests/test_ultratb.py
--- a/IPython/core/tests/test_ultratb.py
+++ b/IPython/core/tests/test_ultratb.py
@@ -2,6 +2,7 @@
 """Tests for IPython.core.ultratb
 """
 import io
+import logging
 import sys
 import os.path
 from textwrap import dedent
@@ -372,3 +373,28 @@ def eggs(f, g, z=globals()):
         handler(*sys.exc_info())
     buff.write('')
 
+
+class TokenizeFailureTest(unittest.TestCase):
+    """Tests related to https://github.com/ipython/ipython/issues/6864."""
+
+    def testLogging(self):
+        message = "An unexpected error occurred while tokenizing input"
+        cell = 'raise ValueError("""a\nb""")'
+
+        stream = io.StringIO()
+        handler = logging.StreamHandler(stream)
+        logger = logging.getLogger()
+        loglevel = logger.level
+        logger.addHandler(handler)
+        self.addCleanup(lambda: logger.removeHandler(handler))
+        self.addCleanup(lambda: logger.setLevel(loglevel))
+
+        logger.setLevel(logging.INFO)
+        with tt.AssertNotPrints(message):
+            ip.run_cell(cell)
+        self.assertNotIn(message, stream.getvalue())
+
+        logger.setLevel(logging.DEBUG)
+        with tt.AssertNotPrints(message):
+            ip.run_cell(cell)
+        self.assertIn(message, stream.getvalue())