Repository: ibis-project/ibis

Issue:
ux: unexpected cryptic `IbisTypeError` on expression `ibis.greatest(t.value, ibis._.lagged_value)`
## Reproducer

Setup:

```python
>>> import ibis
... import pandas as pd
... 
... t = ibis.memtable(
...     pd.DataFrame(
...         {
...             "value": [1, 2, 3, 4, 5],
...             "group": ["a", "b", "b", "b", "a"],
...         }
...     )
... )
```

I am then trying to evaluate:

```python
>>> (
...     t.group_by(t.group)
...     .order_by(t.value)
...     .mutate(lagged_value=t.value.lag())
...     .mutate(lagged_value=ibis.coalesce(ibis._.lagged_value, 0))
...     .mutate(greatest=ibis.greatest(t.value, ibis._.lagged_value))
... ).execute()
```

I get the following exception:

```python-traceback
IbisTypeError: The datatype of value <ibis.expr.deferred.Deferred object at 0x11baad480> cannot be inferred, try passing it explicitly with the `type` keyword.
```
<details>

```python-traceback
Traceback (most recent call last):
  Input In [6] in <cell line: 2>
    .mutate(greatest=ibis.greatest(t.value, ibis._.lagged_value))
  File ~/code/ibis/ibis/expr/api.py:245 in wrapper
    return fn(self, *args, **kwargs)
  File ~/code/ibis/ibis/expr/types/generic.py:141 in greatest
    return ops.Greatest([self, *args]).to_expr()
  File ~/code/ibis/ibis/common/grounds.py:26 in __call__
    return cls.__create__(*args, **kwargs)
  File ~/code/ibis/ibis/common/grounds.py:105 in __create__
    kwargs = cls.__signature__.validate(*args, **kwargs)
  File ~/code/ibis/ibis/common/annotations.py:271 in validate
    this[name] = param.validate(value, this=this)
  File ~/code/ibis/ibis/common/annotations.py:148 in validate
    return self.annotation(arg, this=this)
  File ~/code/ibis/ibis/expr/rules.py:88 in __call__
    result = super().__call__(*args, **kwargs)
  File ~/mambaforge/envs/dev/lib/python3.10/site-packages/toolz/functoolz.py:304 in __call__
    return self._partial(*args, **kwargs)
  File ~/code/ibis/ibis/expr/rules.py:106 in nodes_of
    values = tuple_of(inner, arg, **kwargs)
  File ~/mambaforge/envs/dev/lib/python3.10/site-packages/toolz/functoolz.py:304 in __call__
    return self._partial(*args, **kwargs)
  File ~/code/ibis/ibis/common/validators.py:166 in container_of
    return type(inner(item, **kwargs) for item in arg)
  File ~/code/ibis/ibis/common/validators.py:166 in <genexpr>
    return type(inner(item, **kwargs) for item in arg)
  File ~/code/ibis/ibis/expr/rules.py:88 in __call__
    result = super().__call__(*args, **kwargs)
  File ~/mambaforge/envs/dev/lib/python3.10/site-packages/toolz/functoolz.py:304 in __call__
    return self._partial(*args, **kwargs)
  File ~/code/ibis/ibis/expr/rules.py:222 in value
    arg = literal(None, arg)
  File ~/code/ibis/ibis/expr/rules.py:88 in __call__
    result = super().__call__(*args, **kwargs)
  File ~/mambaforge/envs/dev/lib/python3.10/site-packages/toolz/functoolz.py:304 in __call__
    return self._partial(*args, **kwargs)
  File ~/code/ibis/ibis/expr/rules.py:190 in literal
    raise com.IbisTypeError(
IbisTypeError: The datatype of value <ibis.expr.deferred.Deferred object at 0x11baad480> cannot be inferred, try passing it explicitly with the `type` keyword.
```

</details>

Note that the exception is raised right away when forming the expression:

```python
>>> ibis.greatest(t.value, ibis._.lagged_value)
Traceback (most recent call last):
...
IbisTypeError: The datatype of value <ibis.expr.deferred.Deferred object at 0x11baad480> cannot be inferred, try passing it explicitly with the `type` keyword.
```

Problems:

- I find this exception cryptic because I have no idea where to pass the `type` keyword in my expression by looking at the traceback
- I find this exception unexpected because if I call `ibis.greatest` on all deferred arguments, then there is no problem anymore and Ibis can generate a valid SQL query that outputs the expected outcome:

```python
>>> (
...     t.group_by(t.group)
...     .order_by(t.value)
...     .mutate(lagged_value=t.value.lag())
...     .mutate(lagged_value=ibis.coalesce(ibis._.lagged_value, 0))
...     .mutate(greatest=ibis.greatest(ibis._.value, ibis._.lagged_value))
... ).execute()
   value group  lagged_value  greatest
0      1     a             0         1
1      5     a             1         5
2      2     b             0         2
3      3     b             2         3
4      4     b             3         4
```

## Note

Ideally, I would have liked to write this code without the chained calls to mutate as follows:

```python
t.group_by(t.group).order_by(t.value).mutate(
    c=ibis.greatest(t.value, ibis.coalesce(t.value.lag(), 0))
).execute()
```

but then I would have faced #4722 first.


Discussion:
Not sure if this is a bug or not. At least the error message could be improved if it is justified (which I don't think it is given that the deferred alternative works).
Hey @ogrisel, you're finding some good stuff.  This issue is because the `singledispatch` mechanism we're using to detect `Deferred` objects only looks at the first argument, so when the first argument uses an `_` expression, it works, but if only a subsequent argument does, then it won't use that codepath.
This is what I suspected.
This isn't a blocker for 4.0.0, so I'm moving this issue off of that milestone.

Resolving patch:
diff --git a/ibis/expr/rules.py b/ibis/expr/rules.py
--- a/ibis/expr/rules.py
+++ b/ibis/expr/rules.py
@@ -232,6 +232,12 @@ def value(dtype, arg, **kwargs):
     """
     import ibis.expr.operations as ops
 
+    if isinstance(arg, Deferred):
+        raise com.IbisTypeError(
+            "Deferred input is not allowed, try passing a lambda function instead. "
+            "For example, instead of writing `f(_.a)` write `lambda t: f(t.a)`"
+        )
+
     if not isinstance(arg, ops.Value):
         # coerce python literal to ibis literal
         arg = literal(None, arg)


Test patch:
diff --git a/ibis/tests/expr/test_table.py b/ibis/tests/expr/test_table.py
--- a/ibis/tests/expr/test_table.py
+++ b/ibis/tests/expr/test_table.py
@@ -1809,3 +1809,12 @@ def test_pivot_longer_no_match():
             values_to="rank",
             values_transform=_.cast("int"),
         )
+
+
+def test_invalid_deferred():
+    import ibis
+
+    t = ibis.table(dict(value="int", lagged_value="int"), name="t")
+
+    with pytest.raises(com.IbisTypeError, match="Deferred input is not allowed"):
+        ibis.greatest(t.value, ibis._.lagged_value)