Repository: xonsh/xonsh

Issue:
Please, add 'prompt_toolkit' to general requirements.
I'm having errors in 'abbrevs' and a lot of things printed at the terminal startup.
I use Manjaro Linux and neither the system package, nor the "Arch User Repository" package, nor the package in the "pip" requires the installation of "prompt_toolkit" by DEFAULT.
If possible, as it is undoubtedly an important package, add as required dependency.
In a last look at all the documentation, I found this syntax in the installation instructions for linux: "xonsh[ptk,linux]". I would like to say that, even for those who have a basic knowledge of python, this syntax is a bit obscure. If possible, it would be good to clarify the point.
Thank you very much for this wonderful project.


Discussion:
I would protest that the extras syntax is obscure for Pythonistas. It is extremely common and used by many packages with optional dependencies.

The rest of it is up for debate. It is arguable whether various packages should or should not include it. For example, I maintain the containers: `xonsh/xonsh` and `xonsh/action` do not include PTK because the former is meant to be a minimal base package and the latter is meant for automation. `xonsh/interactive` however does include ptk (as well as pygments) because it is meant for humans.

I believe the general logic is to allow xonsh to be used with reduced dependencies (readline only, without ptk+pygments). The way python packaging works is that you can't specify configurations with less dependencies--you can specify extras to extend functionality with new dependencies, but to specify a configuration with less dependencies you'd need a new package.

As with most FOSS, who is responsible for what packages where is a complex question--I think we directly maintain just the PyPI, Conda, and AppImage packages.
On conda-forge a lot of packages are split into xxx-base/xxx to handle the case with optional dependencies. For example [matplotlib ](https://github.com/conda-forge/matplotlib-feedstock/blob/master/recipe/meta.yaml#L20) and seaborn.

I don't think I have seen similar concepts on PyPI but maybe we should think about that. `xonsh` could be a meta package with all dependencies and `xonsh-base` the bare minimum. 
> I don't think I have seen similar concepts on PyPI but maybe we should think about that. `xonsh` could be a meta package with all dependencies and `xonsh-base` the bare minimum.

That's an interesting idea. I don't remember seeing such packages at PyPI either.

As for making _prompt_toolkit_ a general "hard" requirement, I also vote "nay". While it certainly is beneficial to interactive users, it is absolutely unnecessary for non-interactive use.
I don't use Manjaro myself, but since it's an Arch-based distro, I'm almost sure, that their Xonsh package lists _python-prompt_toolkit_ as an optional dependency.
Is Xonsh to the point that prompt_toolkit should be a hard requirement for *interactive* use?  As in, you get a run-time error if you invoke Xonsh interactively and don't have it?  And the configuration wizard offers a warning and / or help if you choose a color scheme or otherwise imply you're interested in interactive use but don't have PTK?

[PTK supports dumb terminal mode](https://github.com/prompt-toolkit/python-prompt-toolkit/pull/1035) if you are running in a really restricted environment, but it works fine on all the mainline environments that you'd use interactively. Maybe readline shell (and the maintenance burden) has outlived its usefulness...
I think we have users that do use the readline shell?

The main use of the readline interface is that it is lighter than the ptk interface. (Although honestly, if weight is a concern, you should probably not use xonsh.)
The basic reasoning here is that xonsh should be able to run on the standard library-only, so that if you have Python, you have  your shell
@scopatz  What about if we vend PTK similar to what we do with PLY. 

We could use the user installed PTK if it is available and recent enough. Otherwise fall back the version we vend. 

It would be a annoyin to maintain, but it would improve user experience on systems where PTK is not installed. I just don't know if it would be considered bad practice. 
I think that would be OK to do.  Annoying to maintain, but it might be worth it.
@scopatz I have been looking at vending PTK. 

I place it `xonsh/vended_ptk/prompt_toolkit`, and then I add the `vended_ptk` folde to the python path if "prompt_toolkit" is not installed. 

``` python
def vend_prompt_toolkit_if_not_available():
    spec = importlib.util.find_spec("prompt_toolkit")
    if spec is None:
        sys.path.append(str(pathlib.Path(__file__).with_name("vended_ptk").resolve()))
```

Do you know of an other way to import a vended subpackages as it were a toplevel module in `sys.modules`. I don't like the use of `__file__` in this implementation. 
Is there a meaningful difference between "vending" and "primary distributables include dependencies"?
> Do you know of an other way to import a vended subpackages as it were a toplevel module in sys.modules.

Change all of the imports in the vended version?
@scopatz That sounds like more risky thing to maintain, if we forget new imports when updating. 

But of course, it is an easy search/replace to do. 

I made a PR #3659  with the first implementation I came up with. What strategy do you prefer?
I think I like the function approach you used here
I'm sorry to be so late to the party, I just saw the checkin for the vended ptk, and I really don't like this idea.  I suggest reconsidering the approach of distributing 2 packages, the xonsh-min with readline and (just plain) xonsh that has a hard dependency on PTK.

1. Right now is an inopportune time to lock in a version of PTK.  My proposed fix for prompt-toolkit/python-prompt-toolkit#1158 is still pending and affects Xonsh users.  I expect it will get fixed in the mainline, but years from now, we could continue to get reports of this symptom and the response will be "known issue, upgrade PTK", which is kind of petty and rude.  Note that a user could easily get the vended version of PTK inadvertently if we don't warn (stridently enough) on startup, or the user doesn't happen to grovel over the install log.

2. But there's never a really good time to take this approach.  
  a) Imagine how the Xonsh team would feel if another project chose to vend Xonsh.  Does the team really want to stand pat on the project's features and reputation as of now?  How patient and understanding will the team be years hence about bug reports for long-fixed issues?
  b) It's more technical debt for Xonsh team to maintain and it's somebody else's design and bugs!   Shouldn't we add a test run with this version of PTK on top of the run with latest and greatest?  Shouldn't we be running PTK unit tests as well?  (`test` tree isn't even checked in...)  What's the schedule for refreshing the vended version?  Can we resist the temptation to fix bugs that affect Xonsh in the vended version, and will the team always be good about offering them upstream?  Why waste precious cycles even worrying about these issues?
  c) our incorporation of `ply` is an exception, not a precedent.  It's an orphaned project and we can and do test its integration with Xonsh (thoroughly).


Thanks for the feedback @bobhy. We are open to discuss other approaches. 

But let me clarify the idea a little more.  Xonsh only uses the builtin prompt_toolkit if the user didn't install their own. All package managers that distribute xonsh should still include prompt-toolkit as dependency. This will ensure user always get the newest. 
However, when installing from source or pypi xonsh will nicely with no dependencies or if prompt-toolkit is too old. 

We will still update the builtin prompt-toolkit with regular intervals. Updating is easy. We simply copy all code into a folder. 
It will of course still be annoying. But updating will only be very important if there are new stuff in PTK which we can't live without. Like your [fixes](https://github.com/prompt-toolkit/python-prompt-toolkit/issues/1158). Once they are in we will of course update the builtin prompt-toolkit. 
I still fail to see why "vending" is better then a hard dependency. As @bobhy pointed out, it's an acceptable evil in case of PLY, but it really is an exception.
OK, I've gotten some hands-on experience with the vended install, but it's
only confirming my opinion this is the wrong direction to invest team
resources...

1) the vended install does not install pygments.  So I don't get the "best"
experience and that reduces the benefit of the convenience install.
2) the vended PTK does not show up in `pip list`.  Despite this, I find it
works OK, and that I can override the vended package by installing a real
one from pypi.  This is unexpected and confusing behavior unless you are a
student of the Xonsh project, it should be documented to reassure users who
will be confused by it later.
3) we're making it harder for new users to get the best Xonsh experience
right off the bat.  I think we agree that configuration is
Pygments+(latest)PTK and PTK shell?  if so, the only way to get them is to
(read the instructions) and install Pygments and PTK from pypi before
installing Xonsh.  If you install and run Xonsh without them, you do not
get any warning but you do get a less-than-optimal experience and have to
go digging to figure out why.  It's great that Xonsh 'just works' in a
variety of environments, but we should put our thumb on the scales to
ensure that users get the best experience that their environment can
support (while giving them options to ratchet down by configuration if they
need to)

What are the runtime environments we should cater for? IMhO:

1. Mainline -- VT100/Xterm console, big 3 OS's  Xonsh + pygments + ptk.
This runs fine on my Raspberry PI, I don't think we need to worry about
resource-constrained systems.
2 Emacs or other not-quite-standard consoles: Xonsh + readline (+
Pygments?).  The canonic case for interactive use but can't use PTK for
some reason.
3 Embedded -- non-interactive, neither readline nor ptk.  Only option is to
invoke Xonsh with `-c <command line>`

However many environments we agree on, there should be a separate pypi
package that installs it in one go.

I think this is work worth doing and hereby volunteer to do it and the
documentation if you decide to go in this direction.


On Wed, Aug 12, 2020 at 6:26 AM David Strobach <notifications@github.com>
wrote:

> I still fail to see why "vending" is better then a hard dependency. As
> @bobhy <https://github.com/bobhy> pointed out, it's an acceptable evil in
> case of PLY, but it really is an exception.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/xonsh/xonsh/issues/3622#issuecomment-672789603>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAPBIZDZBWPACDRNOEBALP3SAJU4VANCNFSM4N6KMO2Q>
> .
>

I do agree with many of your points. But readline doesn't work well for Windows and a fundamental goal is that xonsh should work without dependencies. 

However I am not against a warning. For example as part of the welcome screen to new users, or when users users run `xonfig`

An alternative approach is xonsh-base/xonsh pypi packages. But I haven't seen that concept in pypi. 
Maybe it's time we revisit the dependency policy, then.

"Vending" is a crude way to bundle dependencies. Maybe we should look at other approaches, like:

* `pip download`, which grabs all the package files you need recursively (apt has similar tooling)
* zipapp, appimage, etc, which allow packaging an app and its dependencies into a single file bundle
* nuitka, which compiles python into native software (does PyPy have a precompiled mode?)

Encouraging the use of these forms of packaging in use cases where dependencies are unwanted, while allowing maintainers to just specify dependencies and allowing dependencies to work in mainstream uses, I think is a good thing.
I don't want to make this into such a big project! All I really want to do is make it much easier for new users to get off on the right foot if they have a mainstream environment.  I think that's the crux of OP's issue and that it can be resolved with only a little work and better documentation.

1) Add a `extras_require` tag `[full]` (or other tag we like better) that gives you `python-prompt-toolkit` and `pygments`.  
2) Document that the mainline install is now: `pip install xonsh [full]`.  Short and sweet.  I think the team recommends prefers install from PyPI if possible?  (Also, can't specify extras in platform package managers like `apt`)
3) Rewrite user documentation [Getting Started](https://xon.sh/#getting-started) so it includes the recommended default install and generally flows better.  The "big three" environments are enough alike that we can probably collapse them into one section with only a few exceptions called out. 

If this is acceptable, I'll get going on it.

Going forward, we could do some refactoring and repackaging but it rapidly turns into a can of worms if the above works for now.
Sounds like a good couple of plans @astronouth7303 @bobhy 
... Um, I'm going to drop the vended PTK and wcwidth packages from `setup.py` with this PR. But I think someone should revert #3659 eventually.
I don't think we should drop the vended versions until there is an pyinstaller etc solution available

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -440,5 +440,5 @@ def make_events():
 def setup(app):
     from xonsh.pyghooks import XonshConsoleLexer
 
-    app.add_lexer("xonshcon", XonshConsoleLexer)
+    app.add_lexer("xonshcon", XonshConsoleLexer())
     app.add_css_file("custom.css")
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -412,6 +412,7 @@ def main():
         "linux": ["distro"],
         "proctitle": ["setproctitle"],
         "zipapp": ['importlib_resources; python_version < "3.7"'],
+        "full" : ["ptk", "pygments", "distro"]
     }
     skw["python_requires"] = ">=3.6"
     setup(**skw)


Test patch: