Repository: ibis-project/ibis

Issue:
ux: problems with ibis' `_` convenience API for deferred attribute resolution
#3804 introduced a nice way to be able chain several operations that use new columns not originally part of in input expression:


```python
import pandas as pd
import ibis
from ibis import _

table = ibis.memtable(pd.DataFrame({"x": list("12213")}))
(
   table.mutate(a=2 * _.x, b=_.x.cast("float64"))
   .group_by(_.a)
   .aggregate(count=_.b.count(), mean_b=_.b.mean())
   .order_by(_.mean_b)
).execute()
```

This makes it very easy to complex logic in functions that add or rename columns and the use the `.pipe` method to chain them in particular.

The choice of `_` as the name of the importable deferred attribute resolver is nice because it leads to quite concise and readable code. However I also find it problematic from an UX point of view for the following reasons:

- `a)` it's not easily googleable: someone not familiar with this idiom will have a hard time googling or stackoverflowing for it.
- `b)` it can conflict with the `_` variable of Jupyter notebooks (that store the value of the last executed cell) and can therefore lead to very confusing error messages for the unsuspecting users that do exploratory data analysis with Ibis in a Jupyter notebook with multiple cells.
- `c)` even in regular, non-interactive Python code it can conflict with the common idiom of assigning ignored function call results to a dummy `_` variable to express that we do not need a variable for an ancillary value. E.g.:

```python
a, _ = function_returning_a_pair_of_values()  # ignore the second value
```

```python
for _ in range(n_trials):
    if attempt():
        break
else:
    raise Exception(f"{n_trials} consecutive failed attempts")
```

## Possible solutions

- Solution for `a)` only could be to change the default name of the singleton while allowing to keep on using the `_` idiom later in the code:

```python
from ibis import deferred_resolver as _

# continue as previously
```

At least people reading this code for the first time will understand that `_` is some kind of deferred attribute resolver just by reading the code: no complex googling or random documentation scanning.

I don't have good solutions for  `b)` and `c)`. Here are ideas:

### Suggest (by convention) to name the resolver `c` instead of `_`

I picked up `c` for "column" because most of the time the attribute lookup matches a column look-up. However a variable named `c` might still frequently conflict with user code. Adapting the original example would be lead to code that looks like

```python
import pandas as pd
import ibis
from ibis import deferred_resolver as c


table = ibis.memtable(pd.DataFrame({"x": list("12213")}))
(
   table.mutate(a=2 * c.x, b=c.x.cast("float64"))
   .group_by(c.a)
   .aggregate(count=c.b.count(), mean_b=c.b.mean())
   .order_by(c.mean_b)
).execute()
```

or we could be even more creative with the **𓅠** unicode symbol for the ibis (bird) hieroglyph:

```python
import pandas as pd
import ibis
from ibis import deferred_resolver as 𓅠


table = ibis.memtable(pd.DataFrame({"x": list("12213")}))
(
   table.mutate(a=2 * 𓅠.x, b=𓅠.x.cast("float64"))
   .group_by(𓅠.a)
   .aggregate(count=𓅠.b.count(), mean_b=𓅠.b.mean())
   .order_by(𓅠.mean_b)
).execute()
```

The latter suggestion is more like a joke because:

- it can break code editors / readers that do not render non-ascii unicode symbols properly;
- it's cumbersome to type such code without setting some kind of OS-level custom keyboard mapping / [user defined code snippets](https://code.visualstudio.com/docs/editor/userdefinedsnippets).


Discussion:
Side UX problem: interactively inspecting the `_` in an IPython / Jupyter session is not easy because the `__str__` / `__repr__` of the `Deferred` class is not informative:

```python
>>> from ibis import _
>>> _
<ibis.expr.deferred.Deferred object at 0x11ff40df0>
>>> _.a
<ibis.expr.deferred.Deferred object at 0x122210640>
>>> (_.a.b.c * 2) == _.other
<ibis.expr.deferred.Deferred object at 0x124c8d3f0>
```

I think it might be worth subclassing `Deferred` for the attribute resolver to customize the `__repr__` to have something like:

```python
>>> from ibis import _
>>> _
<ibis.expr.deferred.DeferredAttributeResolver object at 0x11ff40df0>
>>> _.a
<ibis.expr.deferred.DeferredAttributeResolver for name 'a'>
```

Not sure what the class and `__repr__` for more complex values such as `(_.a.b.c * 2) == _.other` should be. Maybe it's fine to keep the generic `Deferred` class with a non-informative `__repr__`.

> ### Suggest (by convention) to name the resolver `c` instead of `_`

I agree that underscore should be a choice of the user and not the default symbol. I've also done a little bit of exploration around what it could look like if there was more of a schema object that would also allow some autocompletion for convenience. This was just in a toy dataframe library that I've used for exploring ideas.

https://github.com/jcmkk3/opossom/blob/main/docs/expression-namespace.ipynb
> I've also done a little bit of exploration around what it could look like if there was more of a schema object that would also allow some autocompletion for convenience. This was just in a toy dataframe library that I've used for exploring ideas.

```python
t = make_schema(penguins)
(
    penguins
    .assign(bill_depth_cm=t.bill_depth_mm / 100)  # Column name is offered as autocomplete
    .loc[t.bill_depth_cm > 0.185]  # Uses a column that was not previously in schema
    .shape
)
```

This is neat! Maybe ibis could try to mimic this API instead of using a deferred resolver singleton API? But that requires the user to wrap every expr explicitly with a deferred resolver whenever they anticipate the need to access lazily defined names in chained expressions.

Alternatively, all Ibis `expr` would expose a `__getattr__` that automatically fallback to generating a deferred expression if the requested name is not already present at initial `getattr` time.
Hey @ogrisel, these are good points, thanks for bringing them up.  I agree that `_` is problematic for the reasons you mentioned.  (Thanks also @jmckk for chiming in.)

To add some other possible approaches, let me say that we've been considering similar functionality for join expressions, which need to refer to both the left and the right tables being joined.  The obvious choice here are something like `L` and `R` (or `_L` or `L_`); this might suggest an equivalent `C` (or `X`) as a replacement for `_`.  AFAIK these single capital letter symbols wouldn't conflict with any mainstream Python idioms or platforms.

This still doesn't make them searchable, but we want to keep the identifier extremely short anyway; one or two characters at most, so maybe this is not possible.  Although we should come up with a memorable and searchable name that is part of its docstring and mentioned everywhere we mention this feature.

A Unicode character is clever but I agree, it's a non-starter if it can't be typed on a standard keyboard in every locale :)

As to your side UX problem, in my opinion, the repr for an Ibis expression in general should generate an identical (or equivalent) Ibis expression string, like a quine.  This would open up some really interesting use-cases, but is also a large amount of work.  We might be able to do something more helpful in this smaller case, which may be easier and also push us in this more general direction.
I may be missing something that makes this not possible, but one thing that I've been wondering with the underscore API is if there can be some auto-completion help. As it stands now, it makes things not much better than writing a SQL expression in a string since there is no guidance offered by the object.

It won't be able to give type based filtering of the completion candidates at first so it would have to offer the union of all the possible methods on each column expression type. After the first method, it should be able to offer some filtering of subsequent methods, however, because each method can only return a subset of the column types.

```python
# This auto-completes, but it is up to the user to use a method that works on their column type
# No less risky than the current underscore API, but with some auto-completion
penguins.mutate(species=_.species.split(" ")[0])

# Only string methods are offered as auto-completion
# after using `split` since `split` knows that it can only return a string object
penguins.mutate(species=_.species.split(" ")[0].upper())
```
I am not sure how to do proper tab completion for the deferred field resolver. The getattr/setattr trick in opposum will work for sequences of multiple statements in interactive programming setups but won't work in chained multi-mutate settings.

I think the only way to get it to work properly would be through mixed syntactic analysis and connected table inspection which sounds challenging. Valid completions depend on the lexical scope at the time of completion, not just on the state of the interactive Python interpreter.
> I am not sure how to do proper tab completion for the deferred field resolver. The getattr/setattr trick in opposum will work for sequences of multiple statements in interactive programming setups but won't work in chained multi-mutate settings.
> 
> I think the only way to get it to work properly would be through mixed syntactic analysis and connected table inspection which sounds challenging. Valid completions depend on the lexical scope at the time of completion, not just on the state of the interactive Python interpreter.

Here's a very hacky way to get the behavior that I had in mind. I feel like there's probably a better/cleaner way to get a similar behavior, especially with some of the more advanced typing features.

```python
from __future__ import annotations
from typing import Self, Any

class Column:
    def __init__(self, transforms: list[dict[str, list[Any]]]):
        self._transforms = transforms

    def name(self, name: str) -> Self:
        return self.__class__(self._transforms + [{"name": [name]}])


class StrColumn(Column):
    def split(self, sep: str) -> Self:
        return self.__class__(self._transforms + [{"split": [sep]}])

    def upper(self) -> Self:
        return self.__class__(self._transforms + [{"upper": []}])

class  NumColumn(Column):
    def sum(self) -> Self:
        return self.__class__(self._transforms + [{"sum": []}])

    def mean(self) -> Self:
        return self.__class__(self._transforms + [{"mean": []}])


class AllColumn(Column):
    def split(self, sep: str) -> StrColumn:
        return StrColumn(self._transforms).split(sep)

    def upper(self) -> StrColumn:
        return StrColumn(self._transforms).upper()

    def sum(self) -> NumColumn:
        return NumColumn(self._transforms).sum()

    def mean(self) -> NumColumn:
        return NumColumn(self._transforms).mean()


class ColumnMaker:
    def __getattribute__(self, name: str) -> AllColumn:
        return AllColumn([{"column_name": name}])


col = ColumnMaker()
```

![image](https://user-images.githubusercontent.com/45919695/202803079-ab9ee3f8-e04c-4882-be14-f7c0d3b7ac09.png)
![image](https://user-images.githubusercontent.com/45919695/202803141-90a99477-14e8-4461-83a4-79432758c480.png)

You can see how it starts with providing completions for all of the different methods on columns, then after I use a method that is specific to a certain type, it only completes those since it knows a more specific type.
> You can see how it starts with providing completions for all of the different methods on columns, then after I use a method that is specific to a certain type, it only completes those since it knows a more specific type.

Interesting. I suppose that the type hints will make this work even when no interactive Python shell backs the code editor (e.g. editing a `.py` file in VS Code, and not just for `.ipynb` files).

However, that won't solve the problem of suggesting the right column names in when typing expressions in a chained expressions with previously added column names (e.g. via `mutate` or `agg`).

Resolving patch:
diff --git a/ibis/expr/api.py b/ibis/expr/api.py
--- a/ibis/expr/api.py
+++ b/ibis/expr/api.py
@@ -65,6 +65,7 @@
     'date',
     'desc',
     'decompile',
+    'deferred',
     'difference',
     'e',
     'Expr',
@@ -1031,4 +1032,27 @@ def get_backend(expr: Expr | None = None) -> BaseBackend:
 intersect = ir.Table.intersect
 difference = ir.Table.difference
 
-_ = Deferred()
+_ = deferred = Deferred()
+"""Deferred expression object.
+
+Use this object to refer to a previous table expression in a chain of
+expressions.
+
+!!! note "`_` may conflict with other idioms in Python"
+
+    See https://github.com/ibis-project/ibis/issues/4704 for details.
+
+    Use `from ibis import deferred as <NAME>` to assign a different name to
+    the deferred object builder.
+
+Examples
+--------
+>>> from ibis import _
+>>> t = ibis.table(dict(key="int", value="float"), name="t")
+>>> expr = t.group_by(key=_.key - 1).agg(total=_.value.sum())
+>>> expr.schema()
+ibis.Schema {
+  key    int64
+  total  float64
+}
+"""


Test patch: