Repository: xonsh/xonsh

Issue:
"Unhandled exception" when "cd ~"
With an empty configuration file, when doing a simple `cd ~`, the following backtrace is displayed in the terminal:
```
Unhandled exception in event loop:
  File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/eventloop/utils.py", line 72, in schedule
    func()
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 456, in redraw
    self._redraw()
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 523, in _redraw
    self.context.run(run_in_context)
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 509, in run_in_context
    self.renderer.render(self, self.layout)
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/renderer.py", line 568, in render
    layout.container.preferred_height(size.columns, size.rows).preferred,
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 324, in preferred_height
    dimensions = [
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 325, in <listcomp>
    c.preferred_height(width, max_available_height) for c in self._all_children
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 792, in preferred_height
    return self.content.preferred_height(width, max_available_height)
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 324, in preferred_height
    dimensions = [
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 325, in <listcomp>
    c.preferred_height(width, max_available_height) for c in self._all_children
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 2595, in preferred_height
    return self.content.preferred_height(width, max_available_height)
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 1620, in preferred_height
    return self._merge_dimensions(
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 1648, in _merge_dimensions
    preferred = get_preferred()
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 1613, in preferred_content_height
    return self.content.preferred_height(
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/controls.py", line 639, in preferred_height
    content = self.create_content(width, height=1)  # Pass a dummy '1' as height.
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/controls.py", line 787, in create_content
    cursor_position=translate_rowcol(
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/controls.py", line 770, in translate_rowcol
    return Point(x=get_processed_line(row).source_to_display(col), y=row)
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/controls.py", line 716, in get_processed_line
    processed_line = transform(i, get_line(i))
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/lexers/pygments.py", line 319, in get_line
    for num, line in generator:
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/lexers/pygments.py", line 270, in create_line_generator
    yield from enumerate(split_lines(list(get_text_fragments())), start_lineno)
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/lexers/pygments.py", line 268, in get_text_fragments
    yield _token_cache[t], v
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/lexers/pygments.py", line 141, in __missing__
    result = "class:" + pygments_token_to_classname(key)
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/styles/pygments.py", line 66, in pygments_token_to_classname
    parts = ("pygments",) + token

Exception can only concatenate tuple (not "NoneType") to tuple
Press ENTER to continue...
```
Python: 3.8.2
xonsh: 0.9.14
Arch linux


Discussion:
Hi @marchandfab, thanks for reporting. This seems to be [prompt_toolkit3](https://github.com/prompt-toolkit/python-prompt-toolkit) upstream bug triggered by your pygments style configuration.
Can you reproduce the bug if you start `xonsh --no-rc`?
What is your Xonsh configuration?
I'm on the same versions, and after an update I am getting the same exception. Happens when running with --no-rc also.

I tried rolling back to prompt_toolkit3=3.0.3 and pygments=2.5.2, but still same error. Rolling back xonsh from 0.9.14 to 0.9.13 with either version of prompt_toolkit did fix it for me.

Bisecting xonsh, the first commit that I can make fail is 6082b61f
Thanks for investigating the issue @Briaoeuidhtns. It seems to be related to `$LS_COLORS` environment variable once again.
Is the issue still present on current master/HEAD? If so, does the issue go away if you remove `$LS_COLORS` from the environment before you start Xonsh?
It is still present at master/HEAD.

I actually didn't have $LS_COLORS set, but unset it just to be sure, no change.

Oddly enough, explicitly setting colors does seem to fix it, but only if done before launching. For instance, if I run `$LS_COLORS = $LS_COLORS.default_settings` from my working xonsh version before launching the one built off of current master, everything works as expected.
@Briaoeuidhtns good digging :)  @bobhy Any idea what could be the going on here. 
Preliminarily, I can't repro with either ptk 3.0.3 or 3.0.4.  What version
of pygments, python etc?  Can anybody else confirm?

On Thu, Mar 12, 2020, 5:07 AM Morten Enemark Lund <notifications@github.com>
wrote:

> @Briaoeuidhtns <https://github.com/Briaoeuidhtns> good digging :) @bobhy
> <https://github.com/bobhy> Any idea what could be the going on here.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/xonsh/xonsh/issues/3474#issuecomment-598080195>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAPBIZG3AXPBUHGMVUSJBMTRHCQ6DANCNFSM4LF3ORVA>
> .
>

prompt-toolkit 3.0.4, Pygments 2.6.1, python 3.8.2, running in a clean venv.
<details>
<summary>Everything</summary>
<pre>
appdirs==1.4.3
attrs==19.3.0
black==19.10b0
certifi==2019.11.28
cfgv==3.1.0
chardet==3.0.4
click==7.1.1
codecov==2.0.16
coverage==5.0.3
distlib==0.3.0
entrypoints==0.3
filelock==3.0.12
flake8==3.7.9
identify==1.4.11
idna==2.9
mccabe==0.6.1
more-itertools==8.2.0
nodeenv==1.3.5
packaging==20.3
pathspec==0.7.0
pluggy==0.13.1
pre-commit==2.1.1
prompt-toolkit==3.0.4
py==1.8.1
pycodestyle==2.5.0
pyflakes==2.1.1
Pygments==2.6.1
pyparsing==2.4.6
pytest==5.3.5
pytest-cov==2.8.1
pytest-flake8==1.0.4
pytest-timeout==1.3.4
PyYAML==5.3
regex==2020.2.20
requests==2.23.0
six==1.14.0
toml==0.10.0
typed-ast==1.4.1
urllib3==1.25.8
virtualenv==20.0.10
wcwidth==0.1.8
xonsh==0.9.14
</pre>
</details>

I can't reproduce on an arch docker with the same versions, but I'm not really sure what that leaves if @marchandfab is also having the same issue
Thanks for the additional info.   I've got a solid repro now, you had it right in the first report.  My previous checkin can return `None` for a token when trying to colorize a file system object (directory or file),  pygments doesn't like that.  Working on a fix now...

FYI, simple repro with `xonsh 0.9.14' and `prompt_toolkit 3.0.3`, does not depend on empty `xonshrc`.but does depend on timing:

```bash
# (invoke xonsh with normal .xonshrc)
> $LS_COLORS={}
> xonsh --no-rc
> ls work # if 'work' is a subdirectory name, will crash when type 'k'
```
It will not crash (for me) if I swap 2nd and 3rd commands, and does colorize the file name with a value not in $LS_COLORS (by definition), so there's a second bug lurking in there...

Resolving patch:
diff --git a/xonsh/pyghooks.py b/xonsh/pyghooks.py
--- a/xonsh/pyghooks.py
+++ b/xonsh/pyghooks.py
@@ -1355,8 +1355,8 @@ def _set_ndef_for_color_token(self, ttype):
 Initialized by XonshStyle."""
 
 
-@events.on_lscolors_change
-def on_lscolors_change_p(key, oldvalue, newvalue, **kwargs):
+# define as external funcition so tests can reference it too.
+def on_lscolors_change(key, oldvalue, newvalue, **kwargs):
     """if LS_COLORS updated, update file_color_tokens and  corresponding color token in style"""
     if newvalue is None:
         del file_color_tokens[key]
@@ -1364,6 +1364,9 @@ def on_lscolors_change_p(key, oldvalue, newvalue, **kwargs):
         file_color_tokens[key] = color_token_by_name(newvalue)
 
 
+events.on_lscolors_change(on_lscolors_change)
+
+
 def color_file(file_path: str, mode: int) -> (Color, str):
     """Determine color to use for file as ls -c would, given stat() results and its name.
     Parameters
@@ -1379,6 +1382,7 @@ def color_file(file_path: str, mode: int) -> (Color, str):
     Bugs
     ----
     * doesn't handle CA (capability)
+    * doesn't handle LS TARGET mapping.
     """
 
     lsc = builtins.__xonsh__.env["LS_COLORS"]
@@ -1425,7 +1429,7 @@ def color_file(file_path: str, mode: int) -> (Color, str):
     elif stat.S_ISDOOR(mode):
         color_key = "do"  # bug missing mapping for FMT based PORT and WHITEOUT ??
 
-    ret_color_token = file_color_tokens.get(color_key, None)
+    ret_color_token = file_color_tokens.get(color_key, Text)
 
     return ret_color_token, color_key
 


Test patch:
diff --git a/tests/test_ptk_highlight.py b/tests/test_ptk_highlight.py
--- a/tests/test_ptk_highlight.py
+++ b/tests/test_ptk_highlight.py
@@ -22,9 +22,9 @@
 from xonsh.platform import ON_WINDOWS
 from xonsh.built_ins import load_builtins, unload_builtins
 from xonsh.execer import Execer
-from xonsh.pyghooks import XonshLexer, Color, XonshStyle
+from xonsh.pyghooks import XonshLexer, Color, XonshStyle, on_lscolors_change
 from xonsh.environ import LsColors
-from xonsh.events import events
+from xonsh.events import events, EventManager
 from tools import DummyShell
 
 
@@ -143,15 +143,29 @@ def test_nested():
     )
 
 
-@skip_if_on_windows
-def test_path(tmpdir, xonsh_builtins):
+# can't seem to get thie test to import pyghooks and define on_lscolors_change handler like live code does.
+# so we declare the event handler directly here.
+@pytest.fixture
+def events_fxt():
+    return EventManager()
 
+@pytest.fixture
+def xonsh_builtins_ls_colors(xonsh_builtins, events_fxt):
+    x = xonsh_builtins.__xonsh__
     xonsh_builtins.__xonsh__.shell = DummyShell()  # because load_command_cache zaps it.
     xonsh_builtins.__xonsh__.shell.shell_type = "prompt_toolkit2"
     lsc = LsColors(LsColors.default_settings)
     xonsh_builtins.__xonsh__.env["LS_COLORS"] = lsc  # establish LS_COLORS before style.
     xonsh_builtins.__xonsh__.shell.shell.styler = XonshStyle()  # default style
 
+    events.on_lscolors_change(on_lscolors_change)
+    
+    yield xonsh_builtins
+    xonsh_builtins.__xonsh__ = x
+
+@skip_if_on_windows
+def test_path(tmpdir, xonsh_builtins_ls_colors):
+    
     test_dir = str(tmpdir.mkdir("xonsh-test-highlight-path"))
     check_token(
         "cd {}".format(test_dir), [(Name.Builtin, "cd"), (Color.BOLD_BLUE, test_dir)]
@@ -166,6 +180,25 @@ def test_path(tmpdir, xonsh_builtins):
         check_token(test_dir, [(Name.Constant, test_dir)])
 
 
+@skip_if_on_windows
+def test_color_on_lscolors_change(tmpdir, xonsh_builtins_ls_colors):
+    """Verify colorizer returns Token.Text if file type not defined in LS_COLORS"""
+
+    lsc = xonsh_builtins_ls_colors.__xonsh__.env["LS_COLORS"]    
+    test_dir = str(tmpdir.mkdir("xonsh-test-highlight-path"))
+
+    lsc['di'] = ('GREEN',)
+
+    check_token(
+        "cd {}".format(test_dir), [(Name.Builtin, "cd"), (Color.GREEN, test_dir)]
+    )
+
+    del lsc['di']
+    
+    check_token(
+        "cd {}".format(test_dir), [(Name.Builtin, "cd"), (Text, test_dir)]
+    )
+
 @skip_if_on_windows
 def test_subproc_args():
     check_token("cd 192.168.0.1", [(Text, "192.168.0.1")])