Repository: ibis-project/ibis

Issue:
bug: merging selections combines filters in incorrect way
### What happened?

I do a filter-mutate-filter, and the order needs to stay in that order to be correct, but ibis appears to jumble the order of these steps:

```python
import ibis
from ibis import _

ibis.options.interactive = True

t = ibis.read_parquet(
    "https://github.com/NickCrews/election-data/releases/download/v2024-04-05_0/cleaned.parquet"
)
t = t.cache()
t = t.filter(_.votes != "*").mutate(votes=_.votes.cast("int64")).filter(_.votes > 0)
# occasionally, but not always, I get
# ConversionException: Conversion Error: Could not convert string '*' to INT64
t.votes.mean()
```

This gives the sql

```sql
SELECT
  AVG("t1"."votes") AS "Mean(votes)"
FROM (
  SELECT
    "t0"."year",
    "t0"."date",
    "t0"."state_po",
    "t0"."county_name",
    "t0"."county_fips",
    "t0"."jurisdiction_name",
    "t0"."jurisdiction_fips",
    "t0"."district",
    "t0"."office",
    "t0"."magnitude",
    "t0"."special",
    "t0"."stage",
    "t0"."precinct",
    "t0"."writein",
    "t0"."candidate",
    "t0"."party_detailed",
    "t0"."mode",
    CAST("t0"."votes" AS BIGINT) AS "votes",
    "t0"."readme_check",
    "t0"."county_fips2",
    "t0"."jurisdiction_fips2"
  FROM "ibis_cache_hys22fcdbnbmpdoasprivdb3pm" AS "t0"
  WHERE
    "t0"."votes" <> '*' AND CAST("t0"."votes" AS BIGINT) > CAST(0 AS TINYINT)
) AS "t1"
```

I think we need to keep these as two separate relations. I think this is somewhat related to https://github.com/ibis-project/ibis/pull/9014#issuecomment-2065547342

### What version of ibis are you using?

main

### What backend(s) are you using, if any?

duckdb

### Relevant log output

_No response_

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:
Thanks for the report @NickCrews -- we'll probably disable merging filters across selects to avoid this happening.  In the interim (and we won't release 9.0 until we sort this out), you can probably use `try_cast` to handle this without the failure mode.
The produced expression then the sql specific `Select` operations looks correct to me. I understand that this data-dependent case works if the filters are not fused, but ibis expressions shouldn't have assumptions about the underlying data.

The minimal problem is the following:

```py
t.filter(_.votes != "*").filter(_.votes.cast("int64") > 0)
```

gets reduced to

```py
t.filter(_.votes != "*", _.votes.cast("int64") > 0)
```

and the latter fails due to the fallible cast operation. 

By not fusing the operations these two will behave differently depending on the data which might be considered a downside as well. This is not something we can make foolproof since the user may use the compound predicate directly `t.filter(_.votes != "*", _.votes.cast("int64") > 0)`.

The predicate which should be really used for this case is:
```py
t.filter((_.votes != "*").ifelse(_.votes.cast("int64") > 0), False)
```

Or rather a new API allowing to chain ands more conveniently:
```py
t.filter((_.votes != "*").and_then(_.votes.cast("int64") > 0)
```





Resolving patch:
diff --git a/ibis/backends/sql/compiler.py b/ibis/backends/sql/compiler.py
--- a/ibis/backends/sql/compiler.py
+++ b/ibis/backends/sql/compiler.py
@@ -27,6 +27,7 @@
     rewrite_capitalize,
     sqlize,
 )
+from ibis.config import options
 from ibis.expr.operations.udf import InputType
 from ibis.expr.rewrites import replace_bucket
 
@@ -453,7 +454,12 @@ def translate(self, op, *, params: Mapping[ir.Value, Any]) -> sge.Expression:
         # substitute parameters immediately to avoid having to define a
         # ScalarParameter translation rule
         params = self._prepare_params(params)
-        op, ctes = sqlize(op, params=params, rewrites=self.rewrites)
+        op, ctes = sqlize(
+            op,
+            params=params,
+            rewrites=self.rewrites,
+            fuse_selects=options.sql.fuse_selects,
+        )
 
         aliases = {}
         counter = itertools.count()
diff --git a/ibis/backends/sql/rewrites.py b/ibis/backends/sql/rewrites.py
--- a/ibis/backends/sql/rewrites.py
+++ b/ibis/backends/sql/rewrites.py
@@ -205,6 +205,7 @@ def sqlize(
     node: ops.Node,
     params: Mapping[ops.ScalarParameter, Any],
     rewrites: Sequence[Pattern] = (),
+    fuse_selects: bool = True,
 ) -> tuple[ops.Node, list[ops.Node]]:
     """Lower the ibis expression graph to a SQL-like relational algebra.
 
@@ -216,6 +217,8 @@ def sqlize(
         A mapping of scalar parameters to their values.
     rewrites
         Supplementary rewrites to apply to the expression graph.
+    fuse_selects
+        Whether to merge subsequent Select nodes into one where possible.
 
     Returns
     -------
@@ -240,7 +243,10 @@ def sqlize(
     )
 
     # squash subsequent Select nodes into one
-    simplified = sqlized.replace(merge_select_select)
+    if fuse_selects:
+        simplified = sqlized.replace(merge_select_select)
+    else:
+        simplified = sqlized
 
     # extract common table expressions while wrapping them in a CTE node
     ctes = frozenset(extract_ctes(simplified))
diff --git a/ibis/config.py b/ibis/config.py
--- a/ibis/config.py
+++ b/ibis/config.py
@@ -47,6 +47,9 @@ class SQL(Config):
 
     Attributes
     ----------
+    fuse_selects : bool
+        Whether to fuse consecutive select queries into a single query where
+        possible.
     default_limit : int | None
         Number of rows to be retrieved for a table expression without an
         explicit limit. [](`None`) means no limit.
@@ -55,6 +58,7 @@ class SQL(Config):
 
     """
 
+    fuse_selects: bool = True
     default_limit: Optional[PosInt] = None
     default_dialect: str = "duckdb"
 


Test patch: