Repository: xonsh/xonsh

Issue:
iTerm2 shell integration with xonsh
In case you haven't seen it, iTerm2 has some nice [shell integration](https://iterm2.com/shell_integration.html) that enables some nice features.  These work by echoing some [special escape sequences](https://groups.google.com/forum/#!msg/iterm2-discuss/URKCBtS0228/rs5Ive4PCAAJ) in the shell. I have written a Python library called [iterm2-tools](https://github.com/asmeurer/iterm2-tools) that has an [API](https://iterm2-tools.readthedocs.org/en/latest/shell_integration.html) for these sequences, and implemented integration with [IPython](https://iterm2-tools.readthedocs.org/en/latest/ipython.html). 

I would like to implement integration with xonsh. This is effectively the only thing keeping me from using Xonsh in day-to-day work. 

The integration by printing special sequences before and after the prompt, and before and after the output of a command. The prompt integration is easy: you generally just need to wrap your `$PROMPT` with them. The tricky part is the sequence that has to be printed before the command output. 

[Here](https://github.com/gnachman/iterm2-website/tree/master/source/misc) are the integrations with other shells. Bash implements this using the `DEBUG` trap (bash also uses `PROMPT_COMMAND` instead of editing `PS1` directly). zsh has `preexec` functions. tcsh has `postcmd` (I guess). And fish I'm not sure about (in each case, look how the `;C` sequence is being echoed). 

I looked a little bit in xonsh and didn't see any obvious ways to do this without editing xonsh itself. I figured it would be easier to ask here than dig too hard, though. 

Secondly, would you like for some or any of this to be part of xonsh? I'd like to use `iterm2-tools` as a dependency. We could either make it so that xonsh does this automatically if iterm2-tools is installed (technically it could work in terminals other than iTerm2, such as the now defunct FinalTerm, where the integration codes originated). Otherwise, I could add it to iterm2-tools and people would need to add a couple of lines to their .xonshrc to enable it. I don't care very strongly either way about this. I mostly just want to get it working. 

## For community
⬇️  **Please click the 👍 reaction instead of leaving a `+1` or 👍  comment**



Discussion:
Hi @asmeurer - This would be awesome to have! And you are right that there would not be an easy way to do this without editing xonsh itself.  We could add `preprecmd()` and `postpostcmd()`, but I am not sure that is a great option either.  

I kinda would like it if it just worked automatically if `iterm2-tools` was installed.  

Very much in favor of this. 

Why prepre?

Oh, yeah, sorry.  Since we inherit from Python's `cmd.Cmd` class, we already have a precmd and postcmd function.

err methods.

A thought I had was some way to hook into the parser. From what I can tell (please correct me if I am wrong), xonsh works by parsing the xonsh code and translating it into Python, the execing that Python (I guess subprocess calls are processed into some kind of subprocess function call and the code is execed in a namespace with that function). What we want to do is inject some code in the beginning of this pipeline. 

Is that too deep of an abstraction? 

By the way, the way IPython does it seems pretty fine to me https://ipython.org/ipython-doc/2/config/callbacks.html. 

I like the IPython approach too with registering callbacks.  This should probably happen at what the xonsh terms the "shell" level, which calls the parser, exec'r etc.  Ideally this would go into the BaseShell, but in the worst case it would need to be implemented in ReadlineShell and PromptToolkitShell. The parser and the execer are for the language independent of any notion of a terminal or shell.  

Just took a little look at this. Any reason we can't just use `precmd` directly (i.e., have some way to register callbacks and define `precmd` so that it calls those callbacks)?  We could even make the existing precmd and postcmd functions be the default callbacks. 

Also, what would be the xonshic way to implement the configuration. Is there some way to access the shell object from within the shell (like `get_ipython()` for IPython)? If so, we could just have the callbacks be a list on that object and let users modify that list at will. Otherwise there should be a function (in what namespace?) to access/modify them.

Hi @asmeurer allowing precmd and postcmd to call registered callbacks is probably the right way to go. 

Also, I believe that you can access the Shell instance as `__xonsh_shell__`.

That is, the xonsh shell gets placed into builtins

Awesome. So I'm thinking something like

``` patch
diff --git a/xonsh/base_shell.py b/xonsh/base_shell.py
index ef4c281..385d39e 100644
--- a/xonsh/base_shell.py
+++ b/xonsh/base_shell.py
@@ -101,6 +101,13 @@ class BaseShell(object):
         self.buffer = []
         self.need_more_lines = False
         self.mlprompt = None
+        self.precmd_callbacks = [self.get_default_precmd()]
+        self.postcmd_callbacks = []
+
+    def get_default_precmd(self):
+        def _default_precmd(line):
+            return line if self.need_more_lines else line.lstrip()
+        return _default_precmd

     def emptyline(self):
         """Called when an empty line has been entered."""
@@ -109,7 +116,14 @@ class BaseShell(object):

     def precmd(self, line):
         """Called just before execution of line."""
-        return line if self.need_more_lines else line.lstrip()
+        for callback in self.precmd_callbacks:
+            line = callback(line)
+        return line
+
+    def postcmd(self, stop, line):
+        for callback in self.postcmd_callbacks:
+            stop = callback(stop, line)
+        return stop

     def default(self, line):
         """Implements code execution."""
```

I would open a pull request, but I'm having issues with the tests. No matter how I try to run them, I get a bunch of errors, or it hangs, or both. 

Hi @asmeurer - This looks great to me.  The only adjustment that I would make is that the `default_precmd()` can live directly on the class.  There is no reason to have a factory for this that I can see.  There is typically only one BaseShell instance, and the binding issues are the same here as if you just had `default_precmd()` as a method. 

I'm not following what you're suggesting. Are you saying it should be a static method? I just set it as a method because it was the most convenient way to do it, not for any OO reasons (subclasses shouldn't really be overriding default_precmd; if they want to add precmds they should just add them in their constructor). 

I was just saying that I don't see the need for the `get_default_precmd()` since `defeault_precmd()` could be a method itself. The getter doesn't seem to add anything, but maybe I am missing some binding issue here...

I see. I'll add `default_precmd` as a function (but not a method, since subclasses shouldn't be overriding it, but rather just modifying the callbacks list). 

Yeah, a regular function is fine too.

I guess I did it that way because the default precmd needs to know about the shell instance (to see `needs_more_lines`). But it's probably not a bad idea to have all precmds take in the shell as an argument, so I'll just do it that way.

I looked at this but I'm unsure what to do. It looks like the prompt_toolkit shell doesn't subclass from Cmd. Rather, it has its own system (which does not include postcmd hooks). Further, both BaseShell and Cmd define precmd (the readline shell uses the BaseShell precmd because it inherits from it first). 

How should I proceed with this? Personally, it would be simpler if there were no multiple inheritance and BaseShell inherited from Cmd. But I don't know if the prompt_toolkit shell uses its own logic rather than Cmd for a reason. 

Also unclear on tests. I didn't see any existing tests for BaseShell (or did I miss them?). Is there a good set of example tests to show the testing pattern I should be using? 

Yeah, the BaseShell, ReadlineShell, and PrompToolkitShell are not tested explicitly yet.  (Also somehow the test suite exploded to ~135k test...not sure what that is about.) If you want to add some tests, then starting a new `test_base_shell.py` file is a good idea.  We use a simple, procedural style with test generators as often as not.  the `test_execer.py` file is a nice and small example.

How would you do the precmd stuff? 

What do you mean exactly? It should probably go into the BaseShell one way or the other.

It's confusing because the ptk shell has its own cmdloop (the readline shell uses the one from Cmd). For instance, it doesn't have postcmd. It would be way simpler if either BaseShell subclassed from Cmd, or nothing did. But I don't know maybe there's a reason ptk has to do things quite differently. 

I see.  I think that the ptk shell is simply incomplete in this regard then.  Adding a postcmd() to base shell should probably be done and then having the ptk call this method via self.postcmd(), like it calls precmd(), seems like the right thing to do. The readline postcmd() method will then additionally need to call super().postcmd(). Then it seems like these hooks could be put in the BaseShell.

Should this be a xontrib, making use of the hypothetical #1489?

Yes to both.  Though I am in favor of having this be a xontrib that part of xonsh, rather than a 3rd party one.

Yes, this is just blocking on a better hook system, specifically for precmd. I was going to implement it myself, but I got bogged down in the implementation differences between the two shells and the refactoring that implied. 

`precmd` would also be useful for enabling `direnv` envirnoments (that modify your env in accordance to directory's `.envrc`). 

@icarito You want [`on_chdir`](http://xon.sh/events.html#on-chdir-olddir-str-newdir-str-none)

@astronouth7303 nice! I'll try it!

@asmeurer precmd is in v0.4.7 (from #1728). For some reason, it's not appearing in the online documentation. (Will investigate later.)

for my specific need looks like I'd need to write the equivalent of https://github.com/direnv/direnv/blob/master/shell_fish.go but it's overkill for me at this time, I've spent way too much time already on this

but nice to talk to you, thanks for all the xonsh! (silly pun)

Looks like iterm2 and FinalTerm's shell integration was merged into python_prompt_toolkit: 
- https://github.com/jonathanslenders/python-prompt-toolkit/issues/148 
- https://github.com/jonathanslenders/python-prompt-toolkit/pull/299

 I think that means it should work here in xonsh. Could someone advise me on the steps to activate it?
I would like to see this as well. Anyone have this working?
I don't think anyone is working on this at the moment, but we'd still love to have this if someone wants to put in a PR!
I wrote [this](https://gist.github.com/roblabla/a458db34c85bd2f57a3254e7d8a066bc), which can be sourced from a xonshrc to get basic iTerm2 integration. It's largely ported from the fish integration, with a few changes here and there to make it integrate into xonsh.

It's missing some functionality I don't need (like printing the user_data) - if someone wants to pick it up and turn it into a xontrib, feel free.
I didn't test it, but make sure the shell timing functionality works correctly. My experience with iterm2_tools is that if you don't flush the before prompt sequence, it is buffered and the timings are wrong. 

People here may also be interested in this https://gitlab.com/gnachman/iterm2/issues/7827 (I haven't had time to work on it so if anyone here wants to pick it up feel free). 
The shell timing functionality *seems* to work correctly here. Running `sh -c 'sleep 5'` results in a running time of `00:05.112`.

The only feature that's sometimes broken is "Select Command Output" (selecting the wrong command's output), and I think that's an iTerm2 bug since I sometimes have it in bash too.
I realized a bug triggered by my script: It seems that when using it, xonsh messes up the line editing. Sometimes, instead of going to the next line, it will start overwriting the current line. This is very similar to what happens with bash when you put invisible characters in your `$PS1` without surrounding them with `\[` and `\]`, so I was wondering if there was anything I needed to do to make XonSH aware that I am printing characters that aren't actually rendered?

here's an example of what happens: https://asciinema.org/a/vPtpsHzbi2B8xUmLfPu3QCdiR (don't mind the traceback at the start, that's unrelated).
For the readline backend I believe it should be the same as in bash (\\[ and \\] are readline things, not bash). For prompt-toolkit, you need to use ZeroWidthEscape tokens for the formatting. Note if you use pygments you have to do something like https://github.com/asmeurer/mypython/blob/a9c35650a366b359ccce625dbccdd2ea86669570/mypython/mypython.py#L68-L82 to work around an issue in prompt-toolkit 2.0.
Thanks @roblabla! It would be great for this to become a xontrib (possibly in the xonsh repo here itself)
I would but I'm still having issues fixing the problem outlined in the [comment above](https://github.com/xonsh/xonsh/issues/374#issuecomment-535965059). The `\[` do not seem to work with the readline backend (which I am using). My prompt looks like this:

```
\[\]\[\]\[
\]roblabla@roblab ~ $ \[\]
```

So $PROMPT does not seem to be escaping the `\[`s.
Reading up on the [readline manpages](http://man7.org/linux/man-pages/man3/readline.3.html), various variables use `\1` and `\2` to escape terminal control sequence, but $PROMPT probably doesn't use those variables since \1 and \2 aren't escaped either.

**EDIT**: Ah, I got it, it's literally the character `0x1`, and not the characters `\` followed by `1`.

Surrounding the prompt characters with `\001`/`\002` does not work either, I still have the same problem. I updated the gist with what I currently have.
Using prompt toolkit is a bit more annoying. I need to print the escape sequence `"\033]133;B\007"` after the prompt, but I see no way of doing this. `$PROMPT` doesn't allow escape sequences with prompt_toolkit (see #1403), and there is no `post_prompt_print_hook` (the `post_prompt` hook is run after the user presses enter, not after the prompt is printed).
Can you call a function that prints it? 
We could add a `post_prompt_print_hook` if needed
@scopatz that would certainly help - I think it'd require changes to prompt_toolkit?
hmm maybe it would, I am not sure about that. PRs welcome here and/or to prompt-toolkit.  It would be great if they had a callback mechanism!
I don't see why it would. I use it just fine here (other than the pygments workaround I mentioned) https://github.com/asmeurer/mypython/blob/7b33931631305565352a1d23986705e38588a063/mypython/mypython.py#L692-L721
@asmeurer the problem is, $PROMPT does not take a `[(style, str)]`. Instead, it takes a string that it formats, and it doesn't escape ANSI escape sequences. When those unescaped ANSI escape sequences reach prompt_toolkit, it simply throws them out the window.

E.G. if I do `$PROMPT += "\033]133;A\007"`, my prompt will end up printing `^[]133;A^G` as raw text - everything gets escaped.

So I can't simply put the escape sequences in xonsh' $PROMPT, so I need a way to print things after the $PROMPT is printed.

Furthermore, `print_formatted_text(FormattedText([('[ZeroWidthEscape]', "\033]133;A\007")]))` prints `?]133;A` in the terminal instead of the raw escape sequence. You can run that directly in xonsh to see the results.
> E.G. if I do $PROMPT += "\033]133;A\007", my prompt will end up printing ^[]133;A^G as raw text - everything gets escaped.
> So I can't simply put the escape sequences in xonsh' $PROMPT

~@roblabla please try #3781~ UPD: don't worry :)
I don't think that PR actually fixes this. 
I don't use iterm2 right now (don't really use macos really) so I can't test this. Sorry.
Yeah, I think it just enables it, but doesn't implement actual support
Too bad, I'd really love to see support for this sometime in the future 🙄
Pull requests very welcome! Since I don't own a Mac this is something that I don't own the capability to work on.  (I wish they would port iTerm2 to Linux)
Heh, don't expect an iterm2 port to happen, it's all objective-c and swift, with deep integration into MacOS' GUI toolkit.

I'll try to see if I can boot my old MacOS and see if I can adapt my old gist to work with the recently merged PR.
The PR helps a lot! There's just one small problem now: Setting $PROMPT from an `on_pre_prompt` hook doesn't affect the prompt immediately - instead, it takes effect on any prompt shown *after* the current one. E.G. with the following hook:

```python
@events.on_pre_prompt
def preprompt():
    if not "HELLO_FROM_PREPROMPT" in $PROMPT:
        $PROMPT += "HELLO_FROM_PREPROMPT"
```

The first shell will **not** have HELLO_FROM_PREPROMPT, only the second prompt will have it:

```
[roblabla@roblab ~]$ xonsh
roblabla@roblab ~ $
roblabla@roblab ~ $ HELLO_FROM_PREPROMPT>
```

This delayed effect is a problem: we need to inject metadata for each prompt. Now, in theory we could avoid it by printing the pre-prompt directly instead of setting it in the prompt. I'd like to avoid this, however. Setting the prompt is much cleaner.

-----

I also managed to identify a bug in my script that `prompt_toolkit` made a lot more apparant than `readline`: The hostname I was acquiring through `$(hostname)` contained a `\n`. This hostname was then inserted within an OSC escape sequence. The `\n` caused prompt_toolkit to stop treating it as an OSC (causing its text to be rendered), and readline to stop treating the OSC as invisible characters (causing weird bugs like the one [shown here](https://asciinema.org/a/vPtpsHzbi2B8xUmLfPu3QCdiR)). Fixing this solved all the issues I had wrt invisible character escape sequences not working in readline!

----

So once $PROMPT can be set from `on_pre_prompt`, my script should be upstreamable within xonsh, and should work flawlessly with both the readline and prompt-toolkit backend. The code is still available [in the gist](https://gist.github.com/roblabla/a458db34c85bd2f57a3254e7d8a066bc).
I have created this https://github.com/jnoortheen/xontrib-iterm2

It works well on readline shell. 
![image](https://user-images.githubusercontent.com/6702219/148605638-a59f5b6b-a6ef-4193-888a-a0d4e4b96a88.png)


But getting some escaping errors with ptk shell. 
![image](https://user-images.githubusercontent.com/6702219/148605724-414a459d-5fb5-4b69-afc8-a9b5433a9186.png)

We can close this issue once the ptk issues are fixed.

Resolving patch:
diff --git a/xonsh/ptk_shell/shell.py b/xonsh/ptk_shell/shell.py
--- a/xonsh/ptk_shell/shell.py
+++ b/xonsh/ptk_shell/shell.py
@@ -43,7 +43,6 @@
 except ImportError:
     HAVE_SYS_CLIPBOARD = False
 
-ANSI_OSC_PATTERN = re.compile("\x1b].*?\007")
 CAPITAL_PATTERN = re.compile(r"([a-z])([A-Z])")
 Token = _TokenType()
 
@@ -84,19 +83,6 @@ def tokenize_ansi(tokens):
     return ansi_tokens
 
 
-def remove_ansi_osc(prompt):
-    """Removes the ANSI OSC escape codes - ``prompt_toolkit`` does not support them.
-    Some terminal emulators - like iTerm2 - uses them for various things.
-
-    See: https://www.iterm2.com/documentation-escape-codes.html
-    """
-
-    osc_tokens = ANSI_OSC_PATTERN.findall(prompt)
-    prompt = ANSI_OSC_PATTERN.sub("", prompt)
-
-    return prompt, osc_tokens
-
-
 def _pygments_token_to_classname(token):
     """Converts pygments Tokens, token names (strings) to PTK style names."""
     if token and isinstance(token, str):
@@ -358,7 +344,6 @@ def singleline(
             "refresh_interval": refresh_interval,
             "complete_in_thread": complete_in_thread,
         }
-
         if env["ENABLE_ASYNC_PROMPT"]:
             # once the prompt is done, update it in background as each future is completed
             prompt_args["pre_run"] = self.prompt_formatter.start_update
@@ -427,16 +412,6 @@ def _get_prompt_tokens(self, env_name: str, prompt_name: str, **kwargs):
         except Exception:  # pylint: disable=broad-except
             print_exception()
 
-        p, osc_tokens = remove_ansi_osc(p)
-
-        if kwargs.get("handle_osc_tokens"):
-            # handle OSC tokens
-            for osc in osc_tokens:
-                if osc[2:4] == "0;":
-                    env["TITLE"] = osc[4:-1]
-                else:
-                    print(osc, file=sys.__stdout__, flush=True)
-
         toks = partial_color_tokenize(p)
 
         return tokenize_ansi(PygmentsTokens(toks))
@@ -447,7 +422,7 @@ def prompt_tokens(self):
             carriage_return()
             self._first_prompt = False
 
-        tokens = self._get_prompt_tokens("PROMPT", "message", handle_osc_tokens=True)
+        tokens = self._get_prompt_tokens("PROMPT", "message")
         self.settitle()
         return tokens
 


Test patch:
diff --git a/tests/test_ptk_shell.py b/tests/test_ptk_shell.py
--- a/tests/test_ptk_shell.py
+++ b/tests/test_ptk_shell.py
@@ -7,7 +7,7 @@
 
 # verify error if ptk not installed or below min
 
-from xonsh.ptk_shell.shell import tokenize_ansi, remove_ansi_osc
+from xonsh.ptk_shell.shell import tokenize_ansi
 from xonsh.shell import Shell
 
 
@@ -99,33 +99,6 @@ def test_tokenize_ansi(prompt_tokens, ansi_string_parts):
         assert token[1] == text
 
 
-@pytest.mark.parametrize(
-    "raw_prompt, prompt, osc_tokens",
-    [
-        # no title
-        ("test prompt", "test prompt", []),
-        # starts w/ title
-        ("\033]0;TITLE THIS\007test prompt", "test prompt", ["\033]0;TITLE THIS\007"]),
-        # ends w/ title
-        ("test prompt\033]0;TITLE THIS\007", "test prompt", ["\033]0;TITLE THIS\007"]),
-        # title in the middle
-        ("test \033]0;TITLE THIS\007prompt", "test prompt", ["\033]0;TITLE THIS\007"]),
-        # title + iTerm2 OSC exapmle
-        (
-            "test \033]0;TITLE THIS\007prompt \033]133;A\007here",
-            "test prompt here",
-            ["\033]0;TITLE THIS\007", "\033]133;A\007"],
-        ),
-    ],
-)
-def test_remove_ansi_osc(raw_prompt, prompt, osc_tokens):
-    checked_prompt, removed_osc = remove_ansi_osc(raw_prompt)
-    assert prompt == checked_prompt
-    assert len(removed_osc) == len(osc_tokens)
-    for removed, ref in zip(removed_osc, osc_tokens):
-        assert removed == ref
-
-
 @pytest.mark.parametrize(
     "line, exp",
     [