Repository: ipython/ipython

Issue:
ipython==8.8.0 has weird tab completion bug
<!-- This is the repository for IPython command line, if you can try to make sure this question/bug/feature belong here and not on one of the Jupyter repositories. 

If it's a generic Python/Jupyter question, try other forums or discourse.jupyter.org.

If you are unsure, it's ok to post here, though, there are few maintainer so you might not get a fast response. 

-->

    self.my_attr.my_method(self.f<tab>
Becomes:

    self.my_attr.my_method(self.my_attr.my_method(self.foo
It looks like it would grab and prepend the front part of the line.

## Workaround
Rolling back to 8.7.0 solves it.

## Repro
Unfortunately, I failed to reproduce it with simple setup with a simple class and method and attribute.

But I easily reproduce this bug at my work projects (which are huge... and have so much complicated dependencies). I think this bug wouldn't manifest itself unless the code base is complicated enough.

I've confirmed 8.7.0 doesn't have this issue.


Discussion:
@suewonjp thank you for opening this issue. A reproducer would really help. Based on your example I tried:

```python
class A:
    def my_method(self):
        pass
class B:
    my_attr = A()
    field = 1
    def test(self):
        self.my_attr.my_method(self.f<tab>
```

but it worked correctly, completing `field`.

Also, would you mind checking if changing [`IPCompleter.evaluation`](https://ipython.readthedocs.io/en/stable/api/generated/IPython.core.completer.html#IPython.core.completer.Completer.evaluation) has any influence on the behaviour you see?
Thank you for your reply.
I think I'll identify the problem commit by git binary search or something when I have time for that at work.

> Also, would you mind checking if changing IPCompleter.evaluation has any influence on the behaviour you see?

I'll try
It's likely related to completer refactor, so I would not spend time on bisecting IPython history. A reproducible example is all we need :)
>  reproducible example is all we need :)

Yes. I know. But I already tried several test scenarios and all failed to reproduce it.

Anyway, the problem commit is 4b0aed94df9573566e87941eec4e23bd58727097
I've confirmed the issue appears at the commit and doesn't appear at its previous commit.

Now I can investigate the issue side by side with two git worktrees (one uses 4b0aed9, the other uses 8.7.0) and compare internal states of the two.

I think this approach is a good option since it seems like the issue is only reproducible at my work place. 

It depends on whether I can find time to do that at work, though. (I have upcoming deadline for my current tasks so...)
The issue was gone by setting True for `IPCompleter.use_jedi` option. I've been using `IPCompleter.use_jedi = False` for a long while due to the issue at #11653

And with debug setting enabled, I've got the following.

![image](https://user-images.githubusercontent.com/5332080/218301932-d848fd88-e9db-4e52-b0d2-c2d9128b42cb.png)

`assert str.st<tab>` becomes `assert assert str.st*` and `assert assert str.st<tab>` becomes `assert assert assert.st*` and so on... just it tries grabbing things before it on the line endlessly.

Honestly, I doubt this may help you fix the issue, but it's what I've got so far. I hope I could share more helpful clues later...

@krassowski 
I've come up with a very minimum setup which can reproduce the issue (I think)

- CD to ipython repository
- Update it to the latest main branch
- Create a virtualenv
- Install ipython itself with `pip install -e .`
- Create a fresh new profile (just in case)
    - `ipython profile create debug`
- Start a session
    - `ipython --profile=debug`
- Disable use_jedi
    - `%config IPCompleter.use_jedi = False`
- Try a simple tab autocompletion
    - `assert str.st<tab>`

I could reproduce the issue with the procedure above on WSL and Mac. So the issue is not dependent on platforms at all.
I hope this would help.
Ok, so previously `IPCompleter.python_matches` (`attr_matches` to be precise) was returning: `["str.startswith", "str.strip"]` but now it is `["assert str.startswith", "assert str.strip"]`.

Previously `expr = "str"`, now `expr = "assert str"`. Larger expr is good for evaluation, but we need to trim it so that the prefix reconciliation algorithm does not go crazy (or change the reconciliation algorithm, but this may not be possible).

PR on the way.


Resolving patch:
diff --git a/IPython/core/completer.py b/IPython/core/completer.py
--- a/IPython/core/completer.py
+++ b/IPython/core/completer.py
@@ -1162,11 +1162,36 @@ def attr_matches(self, text):
             raise
         except Exception:
             # Silence errors from completion function
-            #raise # dbg
             pass
         # Build match list to return
         n = len(attr)
-        return ["%s.%s" % (expr, w) for w in words if w[:n] == attr]
+
+        # Note: ideally we would just return words here and the prefix
+        # reconciliator would know that we intend to append to rather than
+        # replace the input text; this requires refactoring to return range
+        # which ought to be replaced (as does jedi).
+        tokens = _parse_tokens(expr)
+        rev_tokens = reversed(tokens)
+        skip_over = {tokenize.ENDMARKER, tokenize.NEWLINE}
+        name_turn = True
+
+        parts = []
+        for token in rev_tokens:
+            if token.type in skip_over:
+                continue
+            if token.type == tokenize.NAME and name_turn:
+                parts.append(token.string)
+                name_turn = False
+            elif token.type == tokenize.OP and token.string == "." and not name_turn:
+                parts.append(token.string)
+                name_turn = True
+            else:
+                # short-circuit if not empty nor name token
+                break
+
+        prefix_after_space = "".join(reversed(parts))
+
+        return ["%s.%s" % (prefix_after_space, w) for w in words if w[:n] == attr]
 
     def _evaluate_expr(self, expr):
         obj = not_found


Test patch:
diff --git a/IPython/core/tests/test_completer.py b/IPython/core/tests/test_completer.py
--- a/IPython/core/tests/test_completer.py
+++ b/IPython/core/tests/test_completer.py
@@ -543,6 +543,7 @@ def test_greedy_completions(self):
         """
         ip = get_ipython()
         ip.ex("a=list(range(5))")
+        ip.ex("d = {'a b': str}")
         _, c = ip.complete(".", line="a[0].")
         self.assertFalse(".real" in c, "Shouldn't have completed on a[0]: %s" % c)
 
@@ -561,14 +562,14 @@ def _(line, cursor_pos, expect, message, completion):
             _(
                 "a[0].",
                 5,
-                "a[0].real",
+                ".real",
                 "Should have completed on a[0].: %s",
                 Completion(5, 5, "real"),
             )
             _(
                 "a[0].r",
                 6,
-                "a[0].real",
+                ".real",
                 "Should have completed on a[0].r: %s",
                 Completion(5, 6, "real"),
             )
@@ -576,10 +577,24 @@ def _(line, cursor_pos, expect, message, completion):
             _(
                 "a[0].from_",
                 10,
-                "a[0].from_bytes",
+                ".from_bytes",
                 "Should have completed on a[0].from_: %s",
                 Completion(5, 10, "from_bytes"),
             )
+            _(
+                "assert str.star",
+                14,
+                "str.startswith",
+                "Should have completed on `assert str.star`: %s",
+                Completion(11, 14, "startswith"),
+            )
+            _(
+                "d['a b'].str",
+                12,
+                ".strip",
+                "Should have completed on `d['a b'].str`: %s",
+                Completion(9, 12, "strip"),
+            )
 
     def test_omit__names(self):
         # also happens to test IPCompleter as a configurable