Repository: xonsh/xonsh

Issue:
autocompletions support for generators
<!--- Provide a general summary of the issue in the Title above -->
<!--- If you have a question along the lines of "How do I do this Bash command in xonsh"
please first look over the Bash to Xonsh translation guide: https://xon.sh/bash_to_xsh.html
If you don't find an answer there, please do open an issue! -->

[PromptToolkit](https://python-prompt-toolkit.readthedocs.io/en/master/pages/asking_for_input.html#a-custom-completer) lets you return generators from autocompleters. But xonsh requires one to return `set`. It beats the purpose of having the env variable `COMPLETION_QUERY_LIMIT` . All the functions will be running and that is not efficient. 

## xonfig

<details>

```
+------------------+----------------------+
| xonsh            | 0.9.24               |
| Git SHA          | 74543ea9             |
| Commit Date      | Oct 10 15:12:47 2020 |
| Python           | 3.8.6                |
| PLY              | 3.11                 |
| have readline    | True                 |
| prompt toolkit   | 3.0.3                |
| shell type       | prompt_toolkit       |
| pygments         | 2.7.2                |
| on posix         | True                 |
| on linux         | True                 |
| distro           | manjaro              |
| on darwin        | False                |
| on windows       | False                |
| on cygwin        | False                |
| on msys2         | False                |
| is superuser     | False                |
| default encoding | utf-8                |
| xonsh encoding   | utf-8                |
| encoding errors  | surrogateescape      |
| on jupyter       | False                |
| jupyter kernel   | None                 |
| xontrib 1        | abbrevs              |
| xontrib 2        | argcomplete          |
| xontrib 3        | autovox              |
| xontrib 4        | back2dir             |
| xontrib 5        | cmd_done             |
| xontrib 6        | hist_navigator       |
| xontrib 7        | jedi                 |
| xontrib 8        | kitty                |
| xontrib 9        | notifypy             |
| xontrib 10       | pdb                  |
| xontrib 11       | prompt_ret_code      |
| xontrib 12       | vox                  |
| xontrib 13       | voxapi               |
+------------------+----------------------+

```

</details>

## Expected Behavior
<!--- Tell us what should happen -->
xonsh should enable returning autocompleters with yield support as well. 

## Current Behavior
<!--- Tell us what happens instead of the expected behavior -->
<!--- If part of your bug report is a traceback, please first enter debug mode before triggering the error
To enter debug mode, set the environment variable `XONSH_DEBUG=1` _before_ starting `xonsh`.
On Linux and OSX, an easy way to to do this is to run `env XONSH_DEBUG=1 xonsh` -->
1. not able to use generators as completers
2. not able to style the autocompletion menu. eventhough the shell supports it.


I think it is this way to support the compatibility with readline. At this point, does xonsh need to support readline? as it now ships with `prompt_toolkit`.

## For community
⬇️  **Please click the 👍 reaction instead of leaving a `+1` or 👍  comment**



Discussion:
> I think it is this way to support the compatibility with readline. At this point, does xonsh need to support readline? as it now ships with `prompt_toolkit`.

If I remember correctly (it was many years ago) we did this to allow combining results from many completers and de-duping them.  But that certainly doesn't mean we can't `yield from` those sets (or explore a different implementation).

Dropping readline support is a larger discussion issue, but I don't think it needs to block this.  Worst case, we can just exhaust the generator by dumping it into a set for the readline backend.
> If I remember correctly (it was many years ago) we did this to allow combining results from many completers and de-duping them. But that certainly doesn't mean we can't `yield from` those sets (or explore a different implementation).
> 

I think it is better to use `yield from` from the sets and let ptk handle the rest. Atleast when the shell is PTK we can add wrapper around the existing functions that return sets. 


> Dropping readline support is a larger discussion issue, but I don't think it needs to block this. Worst case, we can just exhaust the generator by dumping it into a set for the readline backend.

But I just wanted to know the reasons for keeping readline with the project. I will create an issue maybe where one can list what it brings to the table.


@daniel-shimon I want to work on this. My aim is to make the completer iterate only upto necessary number of completions (keeping set as well as current sequential returns). Please point me where to start.
Cool!
So the code you'll need to change is in `xonsh/completer.py`.
Currently the completers' results are saved in a `set`, I recommend saving a list of their results and then use `itertools.chain` to chain them and `itertools.islice` to take as many as you need

Resolving patch:
diff --git a/xonsh/completer.py b/xonsh/completer.py
--- a/xonsh/completer.py
+++ b/xonsh/completer.py
@@ -78,13 +78,55 @@ def complete(
             (prefix, line, begidx, endidx, ctx),
         )
 
-    def complete_from_context(self, completion_context, old_completer_args=None):
-        trace = XSH.env.get("XONSH_TRACE_COMPLETIONS")
-        if trace:
-            print("\nTRACE COMPLETIONS: Getting completions with context:")
-            sys.displayhook(completion_context)
-        lprefix = 0
-        completions = set()
+    @staticmethod
+    def _format_completion(
+        completion,
+        completion_context,
+        completing_contextual_command: bool,
+        lprefix: int,
+        custom_lprefix: bool,
+    ) -> tp.Tuple[Completion, int]:
+        if (
+            completing_contextual_command
+            and completion_context.command.is_after_closing_quote
+        ):
+            """
+            The cursor is appending to a closed string literal, i.e. cursor at the end of ``ls "/usr/"``.
+            1. The closing quote will be appended to all completions.
+                I.e the completion ``/usr/bin`` will turn into ``/usr/bin"``
+                To prevent this behavior, a completer can return a ``RichCompletion`` with ``append_closing_quote=False``.
+            2. If not specified, lprefix will cover the closing prefix.
+                I.e for ``ls "/usr/"``, the default lprefix will be 6 to include the closing quote.
+                To prevent this behavior, a completer can return a different lprefix or specify it inside ``RichCompletion``.
+            """
+            closing_quote = completion_context.command.closing_quote
+            if not custom_lprefix:
+                lprefix += len(closing_quote)
+            if closing_quote:
+                if isinstance(completion, RichCompletion):
+                    if completion.append_closing_quote:
+                        completion = completion.replace(
+                            value=completion.value + closing_quote
+                        )
+                else:
+                    completion = completion + closing_quote
+
+        completion = list(apply_lprefix([completion], lprefix))[0]
+
+        if (
+            isinstance(completion, RichCompletion)
+            and completion.append_space
+            and not completion.value.endswith(" ")
+        ):
+            # append spaces AFTER appending closing quote
+            completion = completion.replace(value=completion.value + " ")
+
+        return completion, lprefix
+
+    @staticmethod
+    def generate_completions(
+        completion_context, old_completer_args, trace: bool
+    ) -> tp.Iterator[tp.Tuple[Completion, int]]:
         for name, func in XSH.completers.items():
             try:
                 if is_contextual_completer(func):
@@ -124,60 +166,56 @@ def complete_from_context(self, completion_context, old_completer_args=None):
                 else:
                     lprefix = 0
 
-            if res is None or len(res) == 0:
+            if res is None:
+                continue
+
+            items = []
+            for comp in res:
+                items.append(comp)
+                yield Completer._format_completion(
+                    comp,
+                    completion_context,
+                    completing_contextual_command,
+                    lprefix or 0,
+                    custom_lprefix,
+                )
+
+            if not items:  # empty completion
                 continue
 
             if trace:
                 print(
-                    f"TRACE COMPLETIONS: Got {len(res)} results"
+                    f"TRACE COMPLETIONS: Got {len(items)} results"
                     f" from {'' if is_exclusive_completer(func) else 'non-'}exclusive completer '{name}':"
                 )
-                sys.displayhook(res)
-
-            if (
-                completing_contextual_command
-                and completion_context.command.is_after_closing_quote
-            ):
-                """
-                The cursor is appending to a closed string literal, i.e. cursor at the end of ``ls "/usr/"``.
-                1. The closing quote will be appended to all completions.
-                    I.e the completion ``/usr/bin`` will turn into ``/usr/bin"``
-                    To prevent this behavior, a completer can return a ``RichCompletion`` with ``append_closing_quote=False``.
-                2. If not specified, lprefix will cover the closing prefix.
-                    I.e for ``ls "/usr/"``, the default lprefix will be 6 to include the closing quote.
-                    To prevent this behavior, a completer can return a different lprefix or specify it inside ``RichCompletion``.
-                """
-                closing_quote = completion_context.command.closing_quote
-                if not custom_lprefix:
-                    lprefix += len(closing_quote)
-
-                def append_closing_quote(completion: Completion):
-                    if isinstance(completion, RichCompletion):
-                        if completion.append_closing_quote:
-                            return completion.replace(
-                                value=completion.value + closing_quote
-                            )
-                        return completion
-                    return completion + closing_quote
-
-                res = map(append_closing_quote, res)
-
-            completions.update(apply_lprefix(res, lprefix))
+                sys.displayhook(items)
+
             if is_exclusive_completer(func):
                 # we got completions for an exclusive completer
                 break
 
-        # append spaces AFTER appending closing quote
-        def append_space(comp: Completion):
-            if (
-                isinstance(comp, RichCompletion)
-                and comp.append_space
-                and not comp.value.endswith(" ")
-            ):
-                return comp.replace(value=comp.value + " ")
-            return comp
-
-        completions = map(append_space, completions)
+    def complete_from_context(self, completion_context, old_completer_args=None):
+        trace = XSH.env.get("XONSH_TRACE_COMPLETIONS")
+        if trace:
+            print("\nTRACE COMPLETIONS: Getting completions with context:")
+            sys.displayhook(completion_context)
+        lprefix = 0
+        completions = set()
+        query_limit = XSH.env.get("COMPLETION_QUERY_LIMIT")
+
+        for comp in self.generate_completions(
+            completion_context,
+            old_completer_args,
+            trace,
+        ):
+            completion, lprefix = comp
+            completions.add(completion)
+            if query_limit and len(completions) >= query_limit:
+                if trace:
+                    print(
+                        "TRACE COMPLETIONS: Stopped after $COMPLETION_QUERY_LIMIT reached."
+                    )
+                break
 
         def sortkey(s):
             return s.lstrip(''''"''').lower()


Test patch: