Repository: xonsh/xonsh

Issue:
Promp_toolkit: Can't continue from exception
Hey,

<!--- Provide a general summary of the issue in the Title above -->

In the following, I'm using the unterminated subprocess command `$(` as an example.

With readline:
```
 > env XONSH_DEBUG=1 xonsh --shell-type readline                                                                                                                                               
~ 
🐮 > $XONSH_DEBUG
1
~ 
🐮 > $(

~ 
🐮 > xonsh: To log full traceback to a file set: $XONSH_TRACEBACK_LOGFILE = <filename>
Traceback (most recent call last):
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/xonsh/base_shell.py", line 473, in compile
    code = self.execer.compile(src, mode="single", glbs=self.ctx, locs=None)
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/xonsh/execer.py", line 135, in compile
    tree = self.parse(input, ctx, mode=mode, filename=filename, transform=transform)
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/xonsh/execer.py", line 96, in parse
    tree, input = self._parse_ctx_free(input, mode=mode, filename=filename)
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/xonsh/execer.py", line 235, in _parse_ctx_free
    raise original_error from None
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/xonsh/execer.py", line 220, in _parse_ctx_free
    debug_level=(self.debug_level > 2),
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/xonsh/parsers/base.py", line 543, in parse
    tree = self.parser.parse(input=s, lexer=self.lexer, debug=debug_level)
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/xonsh/ply/ply/yacc.py", line 335, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/xonsh/ply/ply/yacc.py", line 1203, in parseopt_notrack
    tok = call_errorfunc(self.errorfunc, errtoken, self)
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/xonsh/ply/ply/yacc.py", line 194, in call_errorfunc
    r = errorfunc(token)
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/xonsh/parsers/base.py", line 3282, in p_error
    self._parse_error("no further code", None)
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/xonsh/parsers/base.py", line 676, in _parse_error
    raise err
  File "<string>", line None
SyntaxError: None: no further code
```

and I'm able to continue the session, totally fine.

However, with prompt toolkit, `$(` yields:
```
🐮 > env XONSH_DEBUG=1 xonsh --shell-type prompt_toolkit                                                                                                                                         

Unhandled exception in event loop:
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/eventloop/posix.py", line 154, in _run_task
    t()
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/eventloop/context.py", line 115, in new_func
    return func(*a, **kw)
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/application/application.py", line 376, in redraw
    self._redraw()
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/application/application.py", line 444, in _redraw
    self.renderer.render(self, self.layout)
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/renderer.py", line 546, in render
    layout.container.preferred_height(size.columns, size.rows).preferred)
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/layout/containers.py", line 245, in preferred_height
    for c in self._all_children]
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/layout/containers.py", line 245, in <listcomp>
    for c in self._all_children]
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/layout/containers.py", line 639, in preferred_height
    return self.content.preferred_height(width, max_available_height)
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/layout/containers.py", line 245, in preferred_height
    for c in self._all_children]
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/layout/containers.py", line 245, in <listcomp>
    for c in self._all_children]
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/layout/containers.py", line 2179, in preferred_height
    return self.content.preferred_height(width, max_available_height)
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/layout/containers.py", line 1362, in preferred_height
    dont_extend=self.dont_extend_height())
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/layout/containers.py", line 1380, in _merge_dimensions
    preferred = get_preferred()
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/layout/containers.py", line 1357, in preferred_content_height
    self.get_line_prefix)
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/layout/controls.py", line 561, in preferred_height
    height += content.get_height_for_line(i, width, get_line_prefix)
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/layout/controls.py", line 174, in get_height_for_line
    to_formatted_text(get_line_prefix(lineno, 0)))
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/shortcuts/prompt.py", line 807, in _get_line_prefix
    return self._get_continuation(prompt_width, line_number, wrap_count)
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/shortcuts/prompt.py", line 791, in _get_continuation
    prompt_continuation, style='class:prompt-continuation')
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/formatted_text/base.py", line 37, in to_formatted_text
    'Expecting string, got: %r' % (value[0][0], )

Exception Expecting string, got: Token
Press ENTER to continue...                                                                                                                                                                       

Unhandled exception in event loop:
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/eventloop/coroutine.py", line 92, in step_next
    new_f = coroutine.send(f.result())
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/application/run_in_terminal.py", line 104, in _run_in_t
    app._redraw()
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/application/application.py", line 444, in _redraw
    self.renderer.render(self, self.layout)
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/renderer.py", line 546, in render
    layout.container.preferred_height(size.columns, size.rows).preferred)
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/layout/containers.py", line 245, in preferred_height
    for c in self._all_children]
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/layout/containers.py", line 245, in <listcomp>
    for c in self._all_children]
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/layout/containers.py", line 639, in preferred_height
    return self.content.preferred_height(width, max_available_height)
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/layout/containers.py", line 245, in preferred_height
    for c in self._all_children]
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/layout/containers.py", line 245, in <listcomp>
    for c in self._all_children]
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/layout/containers.py", line 2179, in preferred_height
    return self.content.preferred_height(width, max_available_height)
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/layout/containers.py", line 1362, in preferred_height
    dont_extend=self.dont_extend_height())
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/layout/containers.py", line 1380, in _merge_dimensions
    preferred = get_preferred()
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/layout/containers.py", line 1357, in preferred_content_height
    self.get_line_prefix)
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/layout/controls.py", line 561, in preferred_height
    height += content.get_height_for_line(i, width, get_line_prefix)
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/layout/controls.py", line 174, in get_height_for_line
    to_formatted_text(get_line_prefix(lineno, 0)))
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/shortcuts/prompt.py", line 807, in _get_line_prefix
    return self._get_continuation(prompt_width, line_number, wrap_count)
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/shortcuts/prompt.py", line 791, in _get_continuation
    prompt_continuation, style='class:prompt-continuation')
  File "/home/dominic/dotfiles/venvs/xonsh/.venv/lib/python3.6/site-packages/prompt_toolkit/formatted_text/base.py", line 37, in to_formatted_text
    'Expecting string, got: %r' % (value[0][0], )

Exception Expecting string, got: Token
Press ENTER to continue...    
```

But I'm unable to continue as the same exception is thrown over and over

<!--- If you have a question along the lines of "How do I do this Bash command in xonsh"
please first look over the Bash to Xonsh translation guide: https://xon.sh/bash_to_xsh.html
If you don't find an answer there, please do open an issue! -->

## xonfig

```
+------------------+----------------------+
| xonsh            | 0.8.11               |
| Git SHA          | fe5c016b             |
| Commit Date      | Feb 27 11:19:35 2019 |
| Python           | 3.6.5                |
| PLY              | 3.11                 |
| have readline    | True                 |
| prompt toolkit   | 2.0.9                |
| shell type       | readline             |
| pygments         | None                 |
| on posix         | True                 |
| on linux         | True                 |
| distro           | debian               |
| on darwin        | False                |
| on windows       | False                |
| on cygwin        | False                |
| on msys2         | False                |
| is superuser     | False                |
| default encoding | utf-8                |
| xonsh encoding   | utf-8                |
| encoding errors  | surrogateescape      |
+------------------+----------------------+
```

## Expected Behavior
<!--- Tell us what should happen -->
I expect the session to continue, as with readline.



Discussion:
Thanks for reporting @deeuu!
This no longer seems to happen for me on v0.9.5.  Please reopen if it continues to be a problem!
@scopatz Still a nightmare for me!

Any incomplete statement yields the same exception.

Here's my latest `xonfig`

```
+------------------+----------------------+
| xonsh            | 0.9.5                |
| Git SHA          | 3fb4943a             |
| Commit Date      | May 31 15:43:52 2019 |
| Python           | 3.6.5                |
| PLY              | 3.11                 |
| have readline    | True                 |
| prompt toolkit   | 2.0.9                |
| shell type       | prompt_toolkit2      |
| pygments         | None                 |
| on posix         | True                 |
| on linux         | True                 |
| distro           | arch                 |
| on darwin        | False                |
| on windows       | False                |
| on cygwin        | False                |
| on msys2         | False                |
| is superuser     | False                |
| default encoding | utf-8                |
| xonsh encoding   | utf-8                |
| encoding errors  | surrogateescape      |
+------------------+----------------------+
```
What does your `.xonshrc` look like?
Hey @scopatz 

The culprit:
```
$MULTILINE_PROMPT = ''
```

Is this related to #507 ?

Note to @deeuu clear your `.xonshrc` before reporting an issue ;-)
haha, well you can also run with `--no-rc` too.

Interesting about the issue with `$MULTILINE_PROMPT`.  I can confirm that this is happening on ptk2 now too.
It is probably unrelated to #507 because PTK has changed so much since then

Resolving patch:
diff --git a/xonsh/ptk2/shell.py b/xonsh/ptk2/shell.py
--- a/xonsh/ptk2/shell.py
+++ b/xonsh/ptk2/shell.py
@@ -252,8 +252,8 @@ def continuation_tokens(self, width, line_number, is_soft_wrap=False):
         width = width - 1
         dots = builtins.__xonsh__.env.get("MULTILINE_PROMPT")
         dots = dots() if callable(dots) else dots
-        if dots is None:
-            return [(Token, " " * (width + 1))]
+        if not dots:
+            return ""
         basetoks = self.format_color(dots)
         baselen = sum(len(t[1]) for t in basetoks)
         if baselen == 0:


Test patch: