Repository: ibis-project/ibis

Issue:
Inconsistencies and *possible* improvements when working with columns
I've been enjoying exploring Ibis now that I've gotten over the hump of getting it setup (thanks for the help!). I've been trying to record my exploration while I learn it. One thing that I've noticed is some inconsistencies in the capabilities of the different ways that a user can specify column transformations. I tried to summarize them in the table below, along with a slightly more lengthy explanation in [this gist](https://gist.github.com/jcmkk3/691fc7e40ec5f165f2f5a8b8bd8f076a).

---

## Comparison Matrix

| Feature                                      | TA Attr | TA Item | Magic Attr | Magic Item | lambda Attr | lambda Item | def Attr | def Item |
|:---------------------------------------------|:-------:|:-------:|:----------:|:----------:|:-----------:|:-----------:|:--------:|:--------:|
| Select a column with valid python identifier | ✔️     | ✔️      | ✔️        | ✔️         | ✔️         | ✔️          | ✔️      | ✔️       |
| Select a column with spaces or other symbols | ❌     | ✔️      | ❌        | ✔️         | ❌         | ✔️          | ❌      | ✔️       |
| Auto completion for column names             | ✔️     | ❌      | ❌        | ❌         | ❌         | ❌          | ❌      | ❌       |
| Auto completion for common methods           | ✔️     | ✔️      | ❌        | ❌         | ❌         | ❌          | ❌      | ❌       |
| Auto completion for type specific methods    | ✔️     | ❌      | ❌        | ❌         | ❌         | ❌          | ❌      | ❌       |
| Expression can return a result standalone    | ✔️     | ✔️      | ❌        | ❌         | ❌         | ❌          | ❌      | ❌       |
| Column can be renamed using `.name`[^1]      | ✔️     | ✔️      | ✔️        | ✔️         | ❌         | ❌          | ❌      | ❌       |
| Column can be renamed using kwargs[^2]       | ✔️     | ✔️      | ✔️        | ✔️         | ✔️         | ✔️          | ✔️      | ✔️       |
| Can reference columns created during query   | ❌     | ❌      | ✔️        | ✔️         | ✔️         | ✔️          | ✔️      | ✔️       |

[^1]: Specific to assigning to a variable and renaming when using it in a query

[^2]: Can only be a valid python identifier unless using dictionary unpacking `**{}`

---

Some of these restrictions seem understandable, but some seem a bit more arbitrary and might be possible to improve. Ideally, it would be nice for the different styles to be as interchangeable as possible.

For instance, it would be nice for the *Table Attached, item based* style to give auto completion of the column names (like pandas) and give auto completion for methods of that column type. These table attached methods are a bit more wordy, but they do have the most opportunity for autocompletion since they know more about the table.

I also think that both *Table Attached* styles should allow referencing columns that don't exist on the attached table. This would simplify things for the user so they do not have to switch to the magic underscore or call back styles if they are trying to manipulate a column that was created during the current query.

It seems like the *magic underscore* style could get the more generic method auto completion so the user at least doesn't need to remember every method.

Lambdas could potentially get a similar treatment. I've seen lambdas autocomplete in vscode before, but I think it is language server and typing dependent.

I think this last one might be a bug, but magic underscores and call backs don't seem to be able to overwrite existing columns in a `mutate` with the `name` method. It works fine when using kwargs or if the name is different than any existing columns. See the gist for an example of this.


Discussion:
Hey @jcmkk3 -- thanks for the great writeup!  It's really helpful to be able to spot potential opportunities for improvements.

> For instance, it would be nice for the Table Attached, item based style to give auto completion of the column names (like pandas) and give auto completion for methods of that column type. These table attached methods are a bit more wordy, but they do have the most opportunity for autocompletion since they know more about the table.

The column names should be doable but I'll have to look up how to offer dict-like keys as completion options because I can't remember how that works.

> I also think that both Table Attached styles should allow referencing columns that don't exist on the attached table. This would simplify things for the user so they do not have to switch to the magic underscore or call back styles if they are trying to manipulate a column that was created during the current query.

I understand the ergonomics of this and it makes sense, but it also seems like it opens the door to some typo footguns.


> It seems like the _magic underscore_ style could get the more generic method auto completion so the user at least doesn't need to remember every method.

Do you have any suggestions on which methods are sufficiently generic for this?  We don't want to offer invalid completions and the deferred nature of the operator precludes looking up valid methods.



> Lambdas could potentially get a similar treatment. I've seen lambdas autocomplete in vscode before, but I think it is language server and typing dependent.

Yeah, I think this is an LSP thing.

> I think this last one might be a bug, but magic underscores and call backs don't seem to be able to overwrite existing columns in a `mutate` with the `name` method. It works fine when using kwargs or if the name is different than any existing columns. See the gist for an example of this.

That looks like a bug.
> The column names should be doable but I'll have to look up how to offer dict-like keys as completion options because I can't remember how that works.

This is done in #4635.

Resolving patch:
diff --git a/ibis/expr/analysis.py b/ibis/expr/analysis.py
--- a/ibis/expr/analysis.py
+++ b/ibis/expr/analysis.py
@@ -229,7 +229,6 @@ def get_mutation_exprs(exprs: list[ir.Expr], table: ir.Table) -> list[ir.Expr |
     # name does not exist in the original table.
     # Given these two data structures, we can compute the mutation node exprs
     # based on whether any columns are being overwritten.
-    # TODO issue #2649
     overwriting_cols_to_expr: dict[str, ir.Expr | None] = {}
     non_overwriting_exprs: list[ir.Expr] = []
     table_schema = table.schema()
diff --git a/ibis/expr/types/relations.py b/ibis/expr/types/relations.py
--- a/ibis/expr/types/relations.py
+++ b/ibis/expr/types/relations.py
@@ -3,7 +3,6 @@
 import collections
 import functools
 import itertools
-import operator
 import sys
 import warnings
 from typing import (
@@ -590,16 +589,21 @@ def mutate(
         import ibis.expr.analysis as an
         import ibis.expr.rules as rlz
 
-        exprs = [] if exprs is None else util.promote_list(exprs)
-        for name, expr in sorted(mutations.items(), key=operator.itemgetter(0)):
+        def ensure_expr(expr):
+            # This is different than self._ensure_expr, since we don't want to
+            # treat `str` or `int` values as column indices
             if util.is_function(expr):
-                value = expr(self)
+                return expr(self)
             elif isinstance(expr, Deferred):
-                value = expr.resolve(self)
+                return expr.resolve(self)
             else:
-                value = rlz.any(expr).to_expr()
-            exprs.append(value.name(name))
+                return rlz.any(expr).to_expr()
 
+        exprs = [] if exprs is None else util.promote_list(exprs)
+        exprs = [ensure_expr(expr) for expr in exprs]
+        exprs.extend(
+            ensure_expr(expr).name(name) for name, expr in sorted(mutations.items())
+        )
         mutation_exprs = an.get_mutation_exprs(exprs, self)
         return self.select(mutation_exprs)
 


Test patch:
diff --git a/ibis/tests/expr/test_table.py b/ibis/tests/expr/test_table.py
--- a/ibis/tests/expr/test_table.py
+++ b/ibis/tests/expr/test_table.py
@@ -238,11 +238,37 @@ def test_projection_array_expr(table):
 
 
 def test_mutate(table):
-    one = table.f * 2
-    foo = (table.a + table.b).name('foo')
-
-    expr = table.mutate(foo, one=one, two=2)
-    expected = table[table, foo, one.name('one'), ibis.literal(2).name('two')]
+    expr = table.mutate(
+        [
+            (table.a + 1).name("x1"),
+            table.b.sum().name("x2"),
+            (_.a + 2).name("x3"),
+            lambda _: (_.a + 3).name("x4"),
+            4,
+            "five",
+        ],
+        kw1=(table.a + 6),
+        kw2=table.b.sum(),
+        kw3=(_.a + 7),
+        kw4=lambda _: (_.a + 8),
+        kw5=9,
+        kw6="ten",
+    )
+    expected = table[
+        table,
+        (table.a + 1).name("x1"),
+        table.b.sum().name("x2"),
+        (table.a + 2).name("x3"),
+        (table.a + 3).name("x4"),
+        ibis.literal(4).name("4"),
+        ibis.literal("five").name("'five'"),
+        (table.a + 6).name("kw1"),
+        table.b.sum().name("kw2"),
+        (table.a + 7).name("kw3"),
+        (table.a + 8).name("kw4"),
+        ibis.literal(9).name("kw5"),
+        ibis.literal("ten").name("kw6"),
+    ]
     assert_equal(expr, expected)