Repository: xonsh/xonsh

Issue:
Fails to install on python3.9b3
https://github.com/pypa/virtualenv/pull/1855/checks?check_run_id=765097416

```
 running install
  ERROR: /tmp/pip-install-h6jiugwr/xonsh/xonsh/parsers/base.py:3281: Symbol '__new_parser___tok' used, but not defined as a token or a rule
  Removed xonsh/parser_table.py
  Removed xonsh/__amalgam__.py
  Removed xonsh/completers/__amalgam__.py
  Removed xonsh/history/__amalgam__.py
  Removed xonsh/prompt/__amalgam__.py
  Building lexer and parser tables.
  ...
      Parser(
    File "/tmp/pip-install-h6jiugwr/xonsh/xonsh/lazyasd.py", line 79, in __call__
      return obj(*args, **kwargs)
    File "/tmp/pip-install-h6jiugwr/xonsh/xonsh/parsers/v35.py", line 38, in __init__
      super().__init__(
    File "/tmp/pip-install-h6jiugwr/xonsh/xonsh/parsers/base.py", line 505, in __init__
      self.parser = yacc.yacc(**yacc_kwargs)
    File "/tmp/pip-install-h6jiugwr/xonsh/xonsh/ply/ply/yacc.py", line 3434, in yacc
##[error]      raise YaccError('Unable to build parser')
  xonsh.ply.ply.yacc.YaccError: Unable to build parser
```


Discussion:
Hi @gaborbernat, thanks for reporting.
I can't, however, confirm the issue. Xonsh installs just fine as pulled from master in docker python:3.9.0b3 and it even works except for several parser test failures pulled from my python3.9 branch (#3588).
Edit: Xonsh 0.9.18 setup even works from PyPI (`pip install xonsh`) without any issues. The shell is, however, unusable due to python/cpython#20312.
```
<user>@dabe55511e94 /xonsh python3.9 # xonfig
+------------------+----------------------+
| xonsh            | 0.9.18.dev14         |
| Git SHA          | 96a7ef78             |
| Commit Date      | May 22 20:31:16 2020 |
| Python           | 3.9.0                |
| PLY              | 3.11                 |
| have readline    | True                 |
| prompt toolkit   | 2.0.4                |
| shell type       | prompt_toolkit       |
| pygments         | 2.6.1                |
| on posix         | True                 |
| on linux         | True                 |
| distro           | unknown              |
| on darwin        | False                |
| on windows       | False                |
| on cygwin        | False                |
| on msys2         | False                |
| is superuser     | True                 |
| default encoding | utf-8                |
| xonsh encoding   | utf-8                |
| encoding errors  | surrogateescape      |
+------------------+----------------------+
```
The GitHub action I've linked shows this to be the case on that Ubuntu and deadsnakes 🤔
I can now reproduce this on Python 3.9.0b4.
For me the error message reads "ERROR: /xonsh/xonsh/parsers/base.py:3284: Symbol '__peg_parser___tok' used, but not defined as a token or a rule".
The issue is, that we now have a new keyword in Python 3.9. Its name used to be `__new_parser__` and was renamed to `__peg_parser__` recently (see python/cpython#20814).

Resolving patch:
diff --git a/xonsh/lexer.py b/xonsh/lexer.py
--- a/xonsh/lexer.py
+++ b/xonsh/lexer.py
@@ -484,6 +484,9 @@ def split(self, s):
     @property
     def tokens(self):
         if self._tokens is None:
+            kwlist = kwmod.kwlist[:]
+            if PYTHON_VERSION_INFO >= (3, 9, 0) and PYTHON_VERSION_INFO < (3, 10):
+                kwlist.remove("__peg_parser__")
             t = (
                 tuple(token_map.values())
                 + (
@@ -505,7 +508,7 @@ def tokens(self):
                     "ATDOLLAR_LPAREN",  # @$(
                     "ERRORTOKEN",  # whoops!
                 )
-                + tuple(i.upper() for i in kwmod.kwlist)
+                + tuple(i.upper() for i in kwlist)
             )
             self._tokens = t
         return self._tokens
diff --git a/xonsh/parsers/base.py b/xonsh/parsers/base.py
--- a/xonsh/parsers/base.py
+++ b/xonsh/parsers/base.py
@@ -2169,7 +2169,9 @@ def apply_trailers(self, leader, trailers):
             return leader
         p0 = leader
         for trailer in trailers:
-            if isinstance(trailer, (ast.Index, ast.Slice, ast.ExtSlice)):
+            if isinstance(
+                trailer, (ast.Index, ast.Slice, ast.ExtSlice, ast.Constant, ast.Name)
+            ):
                 p0 = ast.Subscript(
                     value=leader,
                     slice=trailer,
diff --git a/xonsh/parsers/v36.py b/xonsh/parsers/v36.py
--- a/xonsh/parsers/v36.py
+++ b/xonsh/parsers/v36.py
@@ -130,7 +130,10 @@ def p_argument_test_or_star(self, p):
 
     def p_argument_kwargs(self, p):
         """argument : POW test"""
-        p[0] = ast.keyword(arg=None, value=p[2])
+        p2 = p[2]
+        p[0] = ast.keyword(
+            arg=None, value=p2, lineno=p2.lineno, col_offset=p2.col_offset
+        )
 
     def p_argument_args(self, p):
         """argument : TIMES test"""
@@ -145,7 +148,10 @@ def p_argument(self, p):
 
     def p_argument_eq(self, p):
         """argument : test EQUALS test"""
-        p[0] = ast.keyword(arg=p[1].id, value=p[3])
+        p3 = p[3]
+        p[0] = ast.keyword(
+            arg=p[1].id, value=p3, lineno=p3.lineno, col_offset=p3.col_offset
+        )
 
     def p_comp_for(self, p):
         """comp_for : FOR exprlist IN or_test comp_iter_opt"""


Test patch: