Repository: xonsh/xonsh

Issue:
Expression `2>0` not parsed correctly
When I enter the expression `2>0` in a Xonsh shell, I would expect it to evaluate it as a boolean and print `True` - that is what Python does. Instead, Xonsh attempts to parse it as a subprocess command and gives an error:
```
xonsh.tools.XonshError: xonsh: subprocess mode: command not found: 2>
```

I'll try to work on a fix for this issue and make a PR.

## xonfig

```
+------------------+-------------------------+
| xonsh            | 0.13.3                  |
| Git SHA          | c63f75ef                |
| Commit Date      | Nov 14 13:22:10 2022    |
| Python           | 3.9.13                  |
| PLY              | 3.11                    |
| have readline    | True                    |
| prompt toolkit   | 3.0.29                  |
| shell type       | prompt_toolkit          |
| history backend  | sqlite                  |
| pygments         | 2.11.2                  |
| on posix         | True                    |
| on linux         | True                    |
| distro           | fedora                  |
| on wsl           | False                   |
| on darwin        | False                   |
| on windows       | False                   |
| on cygwin        | False                   |
| on msys2         | False                   |
| is superuser     | False                   |
| default encoding | utf-8                   |
| xonsh encoding   | utf-8                   |
| encoding errors  | surrogateescape         |
| xontrib          | []                      |
| RC file 1        | /home/yaxollum/.xonshrc |
+------------------+-------------------------+
```

## For community
⬇️  **Please click the 👍 reaction instead of leaving a `+1` or 👍  comment**


Discussion:
codecov-commenter: # [Codecov](https://codecov.io/gh/xonsh/xonsh/pull/5013?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=xonsh) Report
> Merging [#5013](https://codecov.io/gh/xonsh/xonsh/pull/5013?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=xonsh) (ed03594) into [main](https://codecov.io/gh/xonsh/xonsh/commit/a587e7ec8c92f3640a4f3a62a52845c3f75c81f7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=xonsh) (a587e7e) will **decrease** coverage by `2.43%`.
> The diff coverage is `100.00%`.

```diff
@@            Coverage Diff             @@
##             main    #5013      +/-   ##
==========================================
- Coverage   66.89%   64.45%   -2.44%     
==========================================
  Files         129      129              
  Lines       23785    23793       +8     
  Branches     4488     4490       +2     
==========================================
- Hits        15910    15335     -575     
- Misses       6649     7206     +557     
- Partials     1226     1252      +26     
```

| Flag | Coverage Δ | |
|---|---|---|
| macOS-latest | `?` | |
| ubuntu-latest | `64.45% <100.00%> (+0.01%)` | :arrow_up: |
| windows-latest | `?` | |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=xonsh#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/xonsh/xonsh/pull/5013?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=xonsh) | Coverage Δ | |
|---|---|---|
| [xonsh/lexer.py](https://codecov.io/gh/xonsh/xonsh/pull/5013/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=xonsh#diff-eG9uc2gvbGV4ZXIucHk=) | `93.11% <100.00%> (+0.20%)` | :arrow_up: |
| [xonsh/parsers/completion\_context.py](https://codecov.io/gh/xonsh/xonsh/pull/5013/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=xonsh#diff-eG9uc2gvcGFyc2Vycy9jb21wbGV0aW9uX2NvbnRleHQucHk=) | `89.73% <100.00%> (-0.38%)` | :arrow_down: |
| [xonsh/tokenize.py](https://codecov.io/gh/xonsh/xonsh/pull/5013/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=xonsh#diff-eG9uc2gvdG9rZW5pemUucHk=) | `56.95% <100.00%> (+0.08%)` | :arrow_up: |
| [xonsh/macutils.py](https://codecov.io/gh/xonsh/xonsh/pull/5013/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=xonsh#diff-eG9uc2gvbWFjdXRpbHMucHk=) | `0.00% <0.00%> (-80.00%)` | :arrow_down: |
| [xonsh/winutils.py](https://codecov.io/gh/xonsh/xonsh/pull/5013/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=xonsh#diff-eG9uc2gvd2ludXRpbHMucHk=) | `0.00% <0.00%> (-47.55%)` | :arrow_down: |
| [xonsh/platform.py](https://codecov.io/gh/xonsh/xonsh/pull/5013/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=xonsh#diff-eG9uc2gvcGxhdGZvcm0ucHk=) | `46.62% <0.00%> (-19.65%)` | :arrow_down: |
| [xonsh/completers/path.py](https://codecov.io/gh/xonsh/xonsh/pull/5013/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=xonsh#diff-eG9uc2gvY29tcGxldGVycy9wYXRoLnB5) | `56.41% <0.00%> (-15.39%)` | :arrow_down: |
| [xonsh/lazyimps.py](https://codecov.io/gh/xonsh/xonsh/pull/5013/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=xonsh#diff-eG9uc2gvbGF6eWltcHMucHk=) | `57.37% <0.00%> (-14.76%)` | :arrow_down: |
| [xonsh/procs/readers.py](https://codecov.io/gh/xonsh/xonsh/pull/5013/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=xonsh#diff-eG9uc2gvcHJvY3MvcmVhZGVycy5weQ==) | `47.46% <0.00%> (-10.60%)` | :arrow_down: |
| [xonsh/xoreutils/\_which.py](https://codecov.io/gh/xonsh/xonsh/pull/5013/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=xonsh#diff-eG9uc2gveG9yZXV0aWxzL193aGljaC5weQ==) | `27.56% <0.00%> (-10.26%)` | :arrow_down: |
| ... and [33 more](https://codecov.io/gh/xonsh/xonsh/pull/5013/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=xonsh) | |

:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=xonsh)


Patch:
diff --git a/news/fix-pymode-ioredirect.rst b/news/fix-pymode-ioredirect.rst
new file mode 100644
index 00000000..71972769
--- /dev/null
+++ b/news/fix-pymode-ioredirect.rst
@@ -0,0 +1,23 @@
+**Added:**
+
+* <news item>
+
+**Changed:**
+
+* <news item>
+
+**Deprecated:**
+
+* <news item>
+
+**Removed:**
+
+* <news item>
+
+**Fixed:**
+
+* Expressions like ``2>1`` are now parsed correctly as Python code instead of being treated like special io-redirection operators.
+
+**Security:**
+
+* <news item>
diff --git a/xonsh/lexer.py b/xonsh/lexer.py
index a9c904ab..a0f55feb 100644
--- a/xonsh/lexer.py
+++ b/xonsh/lexer.py
@@ -253,9 +253,25 @@ def handle_redirect(state, token):
     typ = token.type
     st = token.string
     key = (typ, st) if (typ, st) in token_map else typ
-    yield _new_token(token_map[key], st, token.start)
+    new_tok = _new_token(token_map[key], st, token.start)
     if state["pymode"][-1][0]:
+        if typ == IOREDIRECT:
+            # Fix Python mode code that was incorrectly recognized as an
+            # IOREDIRECT by the tokenizer (see issue #4994).
+            # The tokenizer does not know when the code should be tokenized in
+            # Python mode. By default, when it sees code like `2>`, it produces
+            # an IOREDIRECT token. This call to `get_tokens` makes the
+            # tokenizer run over the code again, knowing that it is *not* an
+            # IOREDIRECT token.
+            lineno, lexpos = token.start
+            for tok in get_tokens(
+                token.string, state["tolerant"], tokenize_ioredirects=False
+            ):
+                yield _new_token(tok.type, tok.value, (lineno, lexpos + tok.lexpos))
+        else:
+            yield new_tok
         return
+    yield new_tok
     # add a whitespace token after a redirection, if we need to
     next_tok = next(state["stream"])
     if next_tok.start == token.end:
@@ -358,7 +374,7 @@ def handle_token(state, token):
         yield _new_token("ERRORTOKEN", m, token.start)
 
 
-def get_tokens(s, tolerant):
+def get_tokens(s, tolerant, pymode=True, tokenize_ioredirects=True):
     """
     Given a string containing xonsh code, generates a stream of relevant PLY
     tokens using ``handle_token``.
@@ -366,8 +382,10 @@ def get_tokens(s, tolerant):
     state = {
         "indents": [0],
         "last": None,
-        "pymode": [(True, "", "", (0, 0))],
-        "stream": tokenize(io.BytesIO(s.encode("utf-8")).readline, tolerant),
+        "pymode": [(pymode, "", "", (0, 0))],
+        "stream": tokenize(
+            io.BytesIO(s.encode("utf-8")).readline, tolerant, tokenize_ioredirects
+        ),
         "tolerant": tolerant,
     }
     while True:
@@ -406,7 +424,7 @@ class Lexer:
 
     _tokens: tp.Optional[tuple[str, ...]] = None
 
-    def __init__(self, tolerant=False):
+    def __init__(self, tolerant=False, pymode=True):
         """
         Attributes
         ----------
@@ -419,12 +437,15 @@ class Lexer:
         tolerant : bool
             Tokenize without extra checks (e.g. paren matching).
             When True, ERRORTOKEN contains the erroneous string instead of an error msg.
+        pymode : bool
+            Start the lexer in Python mode.
 
         """
         self.fname = ""
         self.last = None
         self.beforelast = None
         self._tolerant = tolerant
+        self._pymode = pymode
         self._token_stream = iter(())
 
     @property
@@ -442,7 +463,7 @@ class Lexer:
 
     def input(self, s):
         """Calls the lexer on the string s."""
-        self._token_stream = get_tokens(s, self._tolerant)
+        self._token_stream = get_tokens(s, self._tolerant, self._pymode)
 
     def token(self):
         """Retrieves the next token."""
diff --git a/xonsh/parsers/completion_context.py b/xonsh/parsers/completion_context.py
index dd5a132c..04d34350 100644
--- a/xonsh/parsers/completion_context.py
+++ b/xonsh/parsers/completion_context.py
@@ -350,7 +350,7 @@ class CompletionContextParser:
 
         self.error = None
         self.debug = debug
-        self.lexer = Lexer(tolerant=True)
+        self.lexer = Lexer(tolerant=True, pymode=False)
         self.tokens = tuple(self.used_tokens | self.artificial_tokens)
 
         yacc_kwargs = dict(
diff --git a/xonsh/tokenize.py b/xonsh/tokenize.py
index e03608dd..5127286c 100644
--- a/xonsh/tokenize.py
+++ b/xonsh/tokenize.py
@@ -374,6 +374,7 @@ ContStr = group(
     StringPrefix + r'"[^\n"\\]*(?:\\.[^\n"\\]*)*' + group('"', r"\\\r?\n"),
 )
 PseudoExtras = group(r"\\\r?\n|\Z", Comment, Triple, SearchPath)
+PseudoTokenWithoutIO = Whitespace + group(PseudoExtras, Number, Funny, ContStr, Name_RE)
 PseudoToken = Whitespace + group(
     PseudoExtras, IORedirect, Number, Funny, ContStr, Name_RE
 )
@@ -859,7 +860,7 @@ def tokopen(filename):
         raise
 
 
-def _tokenize(readline, encoding, tolerant=False):
+def _tokenize(readline, encoding, tolerant=False, tokenize_ioredirects=True):
     lnum = parenlev = continued = 0
     numchars = "0123456789"
     contstr, needcont = "", 0
@@ -993,7 +994,9 @@ def _tokenize(readline, encoding, tolerant=False):
             continued = 0
 
         while pos < max:
-            pseudomatch = _compile(PseudoToken).match(line, pos)
+            pseudomatch = _compile(
+                PseudoToken if tokenize_ioredirects else PseudoTokenWithoutIO
+            ).match(line, pos)
             if pseudomatch:  # scan for tokens
                 start, end = pseudomatch.span(1)
                 spos, epos, pos = (lnum, start), (lnum, end), end
@@ -1131,7 +1134,7 @@ def _tokenize(readline, encoding, tolerant=False):
     yield TokenInfo(ENDMARKER, "", (lnum, 0), (lnum, 0), "")
 
 
-def tokenize(readline, tolerant=False):
+def tokenize(readline, tolerant=False, tokenize_ioredirects=True):
     """
     The tokenize() generator requires one argument, readline, which
     must be a callable object which provides the same interface as the
@@ -1152,12 +1155,19 @@ def tokenize(readline, tolerant=False):
 
     If ``tolerant`` is True, yield ERRORTOKEN with the erroneous string instead of
     throwing an exception when encountering an error.
+
+    If ``tokenize_ioredirects`` is True, produce IOREDIRECT tokens for special
+    io-redirection operators like ``2>``. Otherwise, treat code like ``2>`` as
+    regular Python code.
     """
     encoding, consumed = detect_encoding(readline)
     rl_gen = iter(readline, b"")
     empty = itertools.repeat(b"")
     return _tokenize(
-        itertools.chain(consumed, rl_gen, empty).__next__, encoding, tolerant
+        itertools.chain(consumed, rl_gen, empty).__next__,
+        encoding,
+        tolerant,
+        tokenize_ioredirects,
     )
 
 


Test patch:
diff --git a/tests/test_lexer.py b/tests/test_lexer.py
index 3f3804d6..c707c1cc 100644
--- a/tests/test_lexer.py
+++ b/tests/test_lexer.py
@@ -477,3 +477,36 @@ def test_tolerant_lexer(s):
     lexer.input(s)
     error_tokens = list(tok for tok in lexer if tok.type == "ERRORTOKEN")
     assert all(tok.value in s for tok in error_tokens)  # no error messages
+
+
+@pytest.mark.parametrize(
+    "s, exp",
+    [
+        ("2>1", [("NUMBER", "2", 0), ("GT", ">", 1), ("NUMBER", "1", 2)]),
+        ("a>b", [("NAME", "a", 0), ("GT", ">", 1), ("NAME", "b", 2)]),
+        (
+            "3>2>1",
+            [
+                ("NUMBER", "3", 0),
+                ("GT", ">", 1),
+                ("NUMBER", "2", 2),
+                ("GT", ">", 3),
+                ("NUMBER", "1", 4),
+            ],
+        ),
+        (
+            "36+2>>3",
+            [
+                ("NUMBER", "36", 0),
+                ("PLUS", "+", 2),
+                ("NUMBER", "2", 3),
+                ("RSHIFT", ">>", 4),
+                ("NUMBER", "3", 6),
+            ],
+        ),
+    ],
+)
+def test_pymode_not_ioredirect(s, exp):
+    # test that Python code like `2>1` is lexed correctly
+    # as opposed to being recognized as an IOREDIRECT token (issue #4994)
+    assert check_tokens(s, exp)