Repository: ibis-project/ibis

Issue:
bug: .substitute does not support mappings with `None` keys
### What happened?

```python
import ibis
t = ibis.examples.penguins.fetch()
t.year.substitute({2007: 9999, None: -1})

File [~/Documents/projects/scg/social/.venv/lib/python3.11/site-packages/ibis/expr/types/generic.py:470](https://file+.vscode-resource.vscode-cdn.net/Users/nickcrews/Documents/projects/scg/social/notebooks/~/Documents/projects/scg/social/.venv/lib/python3.11/site-packages/ibis/expr/types/generic.py:470), in Value.substitute(self, value, replacement, else_)
    468 expr = self.case()
    469 if isinstance(value, dict):
--> 470     for k, v in sorted(value.items()):
    471         expr = expr.when(k, v)
    472 else:

TypeError: '<' not supported between instances of 'NoneType' and 'int'
```

I would expect the NULLs to get replaced with -1, similar to .fillna

### What version of ibis are you using?

6.2.0

### What backend(s) are you using, if any?

NA

### Relevant log output

_No response_

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:
For a workaround, you can use:

```
In [7]: from ibis.interactive import *

In [8]: t = ibis.examples.penguins.fetch()

In [9]: t.year.fillna(-1).substitute({2007: 9999})
Out[9]:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ SimpleCase(IfNull(year, -1), IfNull(year, -1)) ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ int64                                          │
├────────────────────────────────────────────────┤
│                                           9999 │
│                                           9999 │
│                                           9999 │
│                                           9999 │
│                                           9999 │
│                                           9999 │
│                                           9999 │
│                                           9999 │
│                                           9999 │
│                                           9999 │
│                                              … │
└────────────────────────────────────────────────┘
```
To be clear, I think the `None` should work.
Oh, I realized this also means that I can't use ibis expressions as keys, which I want to do using a lookup from struct, eg
```python
m = {
    ibis.struct({"lat": 99, "lon": 88}) : "cats",
    ibis.struct({"lat": 77, "lon": 66}) : "dogs",
}
```
Generally speaking, any object that overrides `==` (`__eq__`) to return anything except a `bool` will not work well with Python dictionaries. `dict`s rely on `__eq__` implementations for resolving hash collisions, and there's no way to use the result of `expr1 == expr2`--which is another expression in ibis's case--to resolve the collision.

A long time ago I asked on python-ideas about a `__hash_eq__` method that would allow library authors to implement equality specifically used for hash collision resolution but it didn't get much traction (I also didn't actively pursue it).

We **do** implement `__hash__` for `Expr` (the base class of all ibis expressions), but it's limited.

You can construct a `dict` with ibis expressions as keys, but if any of those keys are equivalent (but not pointing to the same object) you'll get a `TypeError`:

```
In [4]: from ibis.interactive import *

In [5]: x = ibis.array([1, 2, 3])

In [6]: params = {x: 1}

In [7]: y = ibis.array([1, 2, 3])

In [8]: params[x]
Out[8]: 1

In [9]: params[y]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[9], line 1
----> 1 params[y]

File ~/src/ibis/ibis/expr/types/core.py:132, in Expr.__bool__(self)
    131 def __bool__(self) -> bool:
--> 132     raise ValueError("The truth value of an Ibis expression is not defined")

ValueError: The truth value of an Ibis expression is not defined
```

You can't see the call stack here, but what's happening is that the first lookup succeeds because I used the same object as a key. Python has a fast path in the `dict` implementation for this pattern.

In the second case, `x` and `y` are distinct objects (they have different addresses in memory), but they hash to the same value. Python needs a way to determine whether `x` and `y` are "truly" equal, and `__eq__` is the way that's done. Then `x.__eq__(y)` is called which returns an expression, followed by a call to `__bool__`. We override `__bool__` to fail because it doesn't make sense in the context of ibis to convert expressions to a single bool.

The main reason we made expressions hashable is to allow passing in `params` to `compile` and `execute` without having to unwrap the expressions to operations. For example:

```python
param = ibis.param("int64")  # expression
expr.execute(params={param: 42})
```

A slightly nuclear option is to remove `__eq__` (and probably `__ne__`, for symmetry) and have `eq` and `ne` methods that construct expressions. I'm not super in favor of that.
Ah, I see, this is a much larger can of worms than I thought. Thanks for the great explanation! [Here is the proposal you mention](https://mail.python.org/pipermail/python-ideas/2019-May/056553.html). You are right not many bites there.

> Then x.__eq__(y) is called which returns an expression

This feels like a bad idea, but throwing it out there. Perhaps we should move to another issue, we are getting away from .substitute(). Could __eq__ return a literal python `True` or `False` in cases where we can compute equality at "compile time" (like this), and the normal ibis boolean Expression in other cases?

- x.lower() == x -> Ibis expr, because we dont know this until runtime
- x.lower() == x.lower() -> True, because the expressions are structurally equivalent and .lower() is a pure, determinisitic function

This would require a lot more computation overhead at every __eq__ call, traversing the entirety of both left and right operands.

The easier solution relevant to .substitute would be if you could pass in an iterable of (key,val) pairs, but then this is the exact same thing as .cases. This actually makes me think that these two methods should get merged, and one deprecated.
Can you describe your use case (maybe in another issue?) for why you want ibis expressions to be hashable?

Maybe there's a solution to your problem that doesn't require this :)

Resolving patch:
diff --git a/ibis/expr/types/generic.py b/ibis/expr/types/generic.py
--- a/ibis/expr/types/generic.py
+++ b/ibis/expr/types/generic.py
@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+import contextlib
 from typing import TYPE_CHECKING, Any, Iterable, Literal, Sequence
 
 from public import public
@@ -683,12 +684,18 @@ def substitute(
         │ torg   │           52 │
         └────────┴──────────────┘
         """
-        expr = self.case()
         if isinstance(value, dict):
-            for k, v in sorted(value.items()):
-                expr = expr.when(k, v)
+            expr = ibis.case()
+            try:
+                null_replacement = value.pop(None)
+            except KeyError:
+                pass
+            else:
+                expr = expr.when(self.isnull(), null_replacement)
+            for k, v in value.items():
+                expr = expr.when(self == k, v)
         else:
-            expr = expr.when(value, replacement)
+            expr = self.case().when(value, replacement)
 
         return expr.else_(else_ if else_ is not None else self).end()
 


Test patch:
diff --git a/ibis/backends/tests/test_generic.py b/ibis/backends/tests/test_generic.py
--- a/ibis/backends/tests/test_generic.py
+++ b/ibis/backends/tests/test_generic.py
@@ -1608,3 +1608,19 @@ def test_sample_with_seed(backend):
     df1 = expr.to_pandas()
     df2 = expr.to_pandas()
     backend.assert_frame_equal(df1, df2)
+
+
+@pytest.mark.broken(
+    ["dask"], reason="implementation somehow differs from pandas", raises=ValueError
+)
+def test_substitute(backend):
+    val = "400"
+    t = backend.functional_alltypes
+    expr = (
+        t.string_col.nullif("1")
+        .substitute({None: val})
+        .name("subs")
+        .value_counts()
+        .filter(lambda t: t.subs == val)
+    )
+    assert expr["subs_count"].execute()[0] == t.count().execute() // 10
diff --git a/ibis/backends/tests/test_string.py b/ibis/backends/tests/test_string.py
--- a/ibis/backends/tests/test_string.py
+++ b/ibis/backends/tests/test_string.py
@@ -1022,8 +1022,7 @@ def test_array_string_join(con):
 
 
 @pytest.mark.notimpl(
-    ["mssql", "mysql", "pyspark", "druid", "oracle"],
-    raises=com.OperationNotDefinedError,
+    ["mssql", "mysql", "druid", "oracle"], raises=com.OperationNotDefinedError
 )
 def test_subs_with_re_replace(con):
     expr = ibis.literal("hi").re_replace("i", "a").substitute({"d": "b"}, else_="k")
@@ -1031,7 +1030,6 @@ def test_subs_with_re_replace(con):
     assert result == "k"
 
 
-@pytest.mark.notimpl(["pyspark"], raises=com.OperationNotDefinedError)
 def test_multiple_subs(con):
     m = {"foo": "FOO", "bar": "BAR"}
     expr = ibis.literal("foo").substitute(m)
diff --git a/ibis/tests/expr/test_value_exprs.py b/ibis/tests/expr/test_value_exprs.py
--- a/ibis/tests/expr/test_value_exprs.py
+++ b/ibis/tests/expr/test_value_exprs.py
@@ -792,13 +792,21 @@ def test_substitute_dict():
 
     result = table.foo.substitute(subs)
     expected = (
-        table.foo.case().when("a", "one").when("b", table.bar).else_(table.foo).end()
+        ibis.case()
+        .when(table.foo == "a", "one")
+        .when(table.foo == "b", table.bar)
+        .else_(table.foo)
+        .end()
     )
     assert_equal(result, expected)
 
     result = table.foo.substitute(subs, else_=ibis.NA)
     expected = (
-        table.foo.case().when("a", "one").when("b", table.bar).else_(ibis.NA).end()
+        ibis.case()
+        .when(table.foo == "a", "one")
+        .when(table.foo == "b", table.bar)
+        .else_(ibis.NA)
+        .end()
     )
     assert_equal(result, expected)