Repository: ibis-project/ibis

Issue:
bug: Ibis 4.0 dev generating incorrect SQL
### What happened?

I believe Ibis 4.0 dev (latest master branch as of 17-NOV-2022 A.D.) has introduced a bug in the generation of SQL.  It seems to be introducing a Cartesian product for the Ibis expression executed here:

https://github.com/prmoore77/ibis-olap-aggregation/blob/fd8bb2f356003807fe04e296fbf0ed216f411f39/main.py#L66

In version 3.2.0 - the generated SQL is:
```
SELECT t0.product_node_name
     , t0.product_level_name
     , t0.sum_sales_amount
     , t0.sum_unit_quantity
     , t0.distinct_customer_count
     , t0.count_of_fact_records
FROM
  (SELECT t2.product_node_name AS product_node_name
        , t2.product_level_name AS product_level_name
        , t2.ancestor_node_sort_order AS ancestor_node_sort_order
        , sum(t1.sales_amount) AS sum_sales_amount
        , sum(t1.unit_quantity) AS sum_unit_quantity
        , count(DISTINCT t1.customer_id) AS distinct_customer_count
        , count('*') AS count_of_fact_records
   FROM sales_facts AS t1
   JOIN
     (SELECT t3.ancestor_node_id AS ancestor_node_id
           , t3.ancestor_node_natural_key AS ancestor_node_natural_key
           , t3.ancestor_node_name AS ancestor_node_name
           , t3.ancestor_level_name AS ancestor_level_name
           , t3.ancestor_is_root AS ancestor_is_root
           , t3.ancestor_is_leaf AS ancestor_is_leaf
           , t3.ancestor_level_number AS ancestor_level_number
           , t3.ancestor_node_sort_order AS ancestor_node_sort_order
           , t3.descendant_node_id AS descendant_node_id
           , t3.descendant_node_natural_key AS descendant_node_natural_key
           , t3.descendant_node_name AS descendant_node_name
           , t3.descendant_level_name AS descendant_level_name
           , t3.descendant_is_root AS descendant_is_root
           , t3.descendant_is_leaf AS descendant_is_leaf
           , t3.descendant_level_number AS descendant_level_number
           , t3.descendant_node_sort_order AS descendant_node_sort_order
           , t3.net_level AS net_level
           , concat(lpad(cast('-' AS TEXT), (t3.ancestor_level_number - cast(1 AS SMALLINT)) * cast(7 AS SMALLINT), cast('-' AS TEXT)), t3.ancestor_level_name) AS product_level_name
           , concat(lpad(cast('-' AS TEXT), (t3.ancestor_level_number - cast(1 AS SMALLINT)) * cast(7 AS SMALLINT), cast('-' AS TEXT)), t3.ancestor_node_name) AS product_node_name
      FROM product_aggregation_dim AS t3) AS t2 ON t1.product_id = t2.descendant_node_natural_key
   GROUP BY t2.product_node_name
          , t2.product_level_name
          , t2.ancestor_node_sort_order
   ORDER BY t2.ancestor_node_sort_order) AS t0
```

In "master" - the generated SQL is:
```
SELECT t0.product_node_name
     , t0.product_level_name
     , t0.sum_sales_amount
     , t0.sum_unit_quantity
     , t0.distinct_customer_count
     , t0.count_of_fact_records
FROM
  (SELECT t2.product_node_name AS product_node_name
        , t2.product_level_name AS product_level_name
        , product_aggregation_dim.ancestor_node_sort_order AS ancestor_node_sort_order
        , sum(t1.sales_amount) AS sum_sales_amount
        , sum(t1.unit_quantity) AS sum_unit_quantity
        , count(DISTINCT t1.customer_id) AS distinct_customer_count
        , count(*) AS count_of_fact_records
   FROM product_aggregation_dim
      , sales_facts AS t1
   JOIN
     (SELECT t3.ancestor_node_id AS ancestor_node_id
           , t3.ancestor_node_natural_key AS ancestor_node_natural_key
           , t3.ancestor_node_name AS ancestor_node_name
           , t3.ancestor_level_name AS ancestor_level_name
           , t3.ancestor_is_root AS ancestor_is_root
           , t3.ancestor_is_leaf AS ancestor_is_leaf
           , t3.ancestor_level_number AS ancestor_level_number
           , t3.ancestor_node_sort_order AS ancestor_node_sort_order
           , t3.descendant_node_id AS descendant_node_id
           , t3.descendant_node_natural_key AS descendant_node_natural_key
           , t3.descendant_node_name AS descendant_node_name
           , t3.descendant_level_name AS descendant_level_name
           , t3.descendant_is_root AS descendant_is_root
           , t3.descendant_is_leaf AS descendant_is_leaf
           , t3.descendant_level_number AS descendant_level_number
           , t3.descendant_node_sort_order AS descendant_node_sort_order
           , t3.net_level AS net_level
           , concat(lpad(cast('-' AS TEXT), (t3.ancestor_level_number - cast(1 AS SMALLINT)) * cast(7 AS SMALLINT), cast('-' AS TEXT)), t3.ancestor_level_name) AS product_level_name
           , concat(lpad(cast('-' AS TEXT), (t3.ancestor_level_number - cast(1 AS SMALLINT)) * cast(7 AS SMALLINT), cast('-' AS TEXT)), t3.ancestor_node_name) AS product_node_name
      FROM product_aggregation_dim AS t3) AS t2 ON t1.product_id = t2.descendant_node_natural_key
   GROUP BY t2.product_node_name
          , t2.product_level_name
          , product_aggregation_dim.ancestor_node_sort_order
   ORDER BY product_aggregation_dim.ancestor_node_sort_order ASC) AS t0
```

The key difference is the CROSS JOIN (using comma notation) in this part of the master version:
```
   FROM product_aggregation_dim
      , sales_facts AS t1
   JOIN
```

There is no WHERE clause joining the two tables, causing a Cartesian product.

### What version of ibis are you using?

Latest master branch wheel (generated by using ci/prepare.sh script)

### What backend(s) are you using, if any?

DuckDB

### Relevant log output

```sh
n/a - SQL shown above in the "What Happened?" section...
```


### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:
@prmoore77 Can you show the ibis expression that produced the SQL, even if it's just the output of

```
print(ibis_expr)
```
Sure @cpcloud - here is the print:
```
print(sales_fact_aggregation)

r0 := AlchemyTable: sales_facts
  product_id    !int32
  customer_id   !string
  date_id       !date
  unit_quantity !decimal(18, 3)
  sales_amount  !decimal(18, 3)
r1 := AlchemyTable: product_aggregation_dim
  ancestor_node_id            string
  ancestor_node_natural_key   int32
  ancestor_node_name          string
  ancestor_level_name         string
  ancestor_is_root            boolean
  ancestor_is_leaf            boolean
  ancestor_level_number       int32
  ancestor_node_sort_order    int64
  descendant_node_id          string
  descendant_node_natural_key int32
  descendant_node_name        string
  descendant_level_name       string
  descendant_is_root          boolean
  descendant_is_leaf          boolean
  descendant_level_number     int32
  descendant_node_sort_order  int64
  net_level                   int32
r2 := Selection[r1]
  selections:
    r1
    product_level_name:      StringConcat(LPad('-', length=r1.ancestor_level_number - 1 * 7, pad='-'))
    product_node_name:       StringConcat(LPad('-', length=r1.ancestor_level_number - 1 * 7, pad='-'))
r3 := InnerJoin[r0, r2] r0.product_id == r2.descendant_node_natural_key
r4 := Aggregation[r3]
  metrics:
    sum_sales_amount:        Sum(r0.sales_amount)
    sum_unit_quantity:       Sum(r0.unit_quantity)
    distinct_customer_count: CountDistinct(r0.customer_id)
    count_of_fact_records:   CountStar(sales_facts)
  by:
    product_node_name:        r2.product_node_name
    product_level_name:       r2.product_level_name
    ancestor_node_sort_order: r2.ancestor_node_sort_order
  sort_keys:
    asc r2.ancestor_node_sort_order
Selection[r4]
  selections:
    product_node_name:       r4.product_node_name
    product_level_name:      r4.product_level_name
    sum_sales_amount:        r4.sum_sales_amount
    sum_unit_quantity:       r4.sum_unit_quantity
    distinct_customer_count: r4.distinct_customer_count
    count_of_fact_records:   r4.count_of_fact_records
```
Please see the readme in the root of the [repo](https://github.com/prmoore77/ibis-olap-aggregation/tree/fd8bb2f356003807fe04e296fbf0ed216f411f39) to see the steps to run the code (you have to generate a duckdb database in one of the steps)...
Yep! I'm going to try and reproduce this without data, which will make it faster to debug.

Resolving patch:
diff --git a/ibis/expr/analysis.py b/ibis/expr/analysis.py
--- a/ibis/expr/analysis.py
+++ b/ibis/expr/analysis.py
@@ -3,6 +3,7 @@
 import functools
 import operator
 from collections import Counter
+from typing import Mapping
 
 import toolz
 
@@ -19,22 +20,21 @@
 # compilation later
 
 
-def sub_for(node, substitutions):
-    """Substitute subexpressions in `expr` with expression to expression
-    mapping `substitutions`.
+def sub_for(node: ops.Node, substitutions: Mapping[ops.node, ops.Node]) -> ops.Node:
+    """Substitute operations in `node` with nodes in `substitutions`.
 
     Parameters
     ----------
-    expr : ibis.expr.types.Expr
-        An Ibis expression
-    substitutions : List[Tuple[ibis.expr.types.Expr, ibis.expr.types.Expr]]
-        A mapping from expression to expression. If any subexpression of `expr`
-        is equal to any of the keys in `substitutions`, the value for that key
-        will replace the corresponding expression in `expr`.
+    node
+        An Ibis operation
+    substitutions
+        A mapping from node to node. If any subnode of `node` is equal to any
+        of the keys in `substitutions`, the value for that key will replace the
+        corresponding node in `node`.
 
     Returns
     -------
-    ibis.expr.types.Expr
+    Node
         An Ibis expression
     """
     assert isinstance(node, ops.Node), type(node)
@@ -50,6 +50,11 @@ def fn(node):
     return substitute(fn, node)
 
 
+def sub_immediate_parents(op: ops.Node, table: ops.TableNode) -> ops.Node:
+    """Replace immediate parent tables in `op` with `table`."""
+    return sub_for(op, {base: table for base in find_immediate_parent_tables(op)})
+
+
 class ScalarAggregate:
     def __init__(self, expr):
         assert isinstance(expr, ir.Expr)
diff --git a/ibis/expr/operations/relations.py b/ibis/expr/operations/relations.py
--- a/ibis/expr/operations/relations.py
+++ b/ibis/expr/operations/relations.py
@@ -147,7 +147,7 @@ def __init__(self, left, right, predicates, **kwargs):
         # TODO(kszucs): predicates should be already a list of operations, need
         # to update the validation rule for the Join classes which is a noop
         # currently
-        import ibis.expr.analysis as L
+        import ibis.expr.analysis as an
         import ibis.expr.operations as ops
 
         # TODO(kszucs): need to factor this out to appropiate join predicate
@@ -169,7 +169,7 @@ def __init__(self, left, right, predicates, **kwargs):
             old = right
             new = right = ops.SelfReference(right)
             predicates = [
-                L.sub_for(pred, {old: new}) if isinstance(pred, ops.Node) else pred
+                an.sub_for(pred, {old: new}) if isinstance(pred, ops.Node) else pred
                 for pred in predicates
             ]
 
@@ -358,17 +358,20 @@ def __init__(self, table, selections, predicates, sort_keys, **kwargs):
         )
 
     def order_by(self, sort_exprs):
-        from ibis.expr.analysis import shares_all_roots
+        from ibis.expr.analysis import shares_all_roots, sub_immediate_parents
 
         keys = rlz.tuple_of(rlz.sort_key_from(rlz.just(self)), sort_exprs)
 
         if not self.selections:
-            if shares_all_roots(keys, self.table):
+            if shares_all_roots(keys, table := self.table):
+                sort_keys = tuple(self.sort_keys) + tuple(
+                    sub_immediate_parents(key, table) for key in keys
+                )
                 return Selection(
-                    self.table,
+                    table,
                     self.selections,
                     predicates=self.predicates,
-                    sort_keys=self.sort_keys + keys,
+                    sort_keys=sort_keys,
                 )
 
         return Selection(self, [], sort_keys=keys)
@@ -474,18 +477,21 @@ def schema(self):
         return sch.Schema(names, types)
 
     def order_by(self, sort_exprs):
-        from ibis.expr.analysis import shares_all_roots
+        from ibis.expr.analysis import shares_all_roots, sub_immediate_parents
 
         keys = rlz.tuple_of(rlz.sort_key_from(rlz.just(self)), sort_exprs)
 
-        if shares_all_roots(keys, self.table):
+        if shares_all_roots(keys, table := self.table):
+            sort_keys = tuple(self.sort_keys) + tuple(
+                sub_immediate_parents(key, table) for key in keys
+            )
             return Aggregation(
-                self.table,
-                self.metrics,
+                table,
+                metrics=self.metrics,
                 by=self.by,
                 having=self.having,
                 predicates=self.predicates,
-                sort_keys=self.sort_keys + keys,
+                sort_keys=sort_keys,
             )
 
         return Selection(self, [], sort_keys=keys)
diff --git a/ibis/expr/types/groupby.py b/ibis/expr/types/groupby.py
--- a/ibis/expr/types/groupby.py
+++ b/ibis/expr/types/groupby.py
@@ -19,9 +19,7 @@
 import types
 from typing import Iterable, Sequence
 
-import toolz
-
-import ibis.expr.analysis as L
+import ibis.expr.analysis as an
 import ibis.expr.types as ir
 import ibis.expr.window as _window
 import ibis.util as util
@@ -36,7 +34,7 @@
             types.FunctionType,
             types.LambdaType,
             types.MethodType,
-            getattr(types, 'UnboundMethodType', None),
+            getattr(types, "UnboundMethodType", None),
         ),
     )
 )
@@ -49,6 +47,8 @@ def _get_group_by_key(table, value):
         return value(table)
     elif isinstance(value, Deferred):
         return value.resolve(table)
+    elif isinstance(value, ir.Expr):
+        return an.sub_immediate_parents(value.op(), table.op()).to_expr()
     else:
         return value
 
@@ -61,8 +61,7 @@ def __init__(
     ):
         self.table = table
         self.by = [_get_group_by_key(table, v) for v in util.promote_list(by)] + [
-            _get_group_by_key(table, v).name(k)
-            for k, v in sorted(expressions.items(), key=toolz.first)
+            _get_group_by_key(table, v).name(k) for k, v in expressions.items()
         ]
         self._order_by = order_by or []
         self._having = having or []
@@ -207,7 +206,7 @@ def projection(self, exprs):
         windowed_exprs = []
         for expr in util.promote_list(exprs):
             expr = self.table._ensure_expr(expr)
-            expr = L.windowize_function(expr, w=w)
+            expr = an.windowize_function(expr, w=w)
             windowed_exprs.append(expr)
         return self.table.projection(windowed_exprs)
 


Test patch:
diff --git a/ibis/tests/sql/snapshots/test_sqlalchemy/test_no_cart_join/out.sql b/ibis/tests/sql/snapshots/test_sqlalchemy/test_no_cart_join/out.sql
new file mode 100644
--- /dev/null
+++ b/ibis/tests/sql/snapshots/test_sqlalchemy/test_no_cart_join/out.sql
@@ -0,0 +1,3 @@
+SELECT ancestor_node_sort_order, 1 AS n 
+FROM facts AS t0 JOIN (SELECT t2.ancestor_level_name AS ancestor_level_name, t2.ancestor_level_number AS ancestor_level_number, t2.ancestor_node_sort_order AS ancestor_node_sort_order, t2.descendant_node_natural_key AS descendant_node_natural_key, concat(lpad('-', (t2.ancestor_level_number - 1) * 7, '-'), t2.ancestor_level_name) AS product_level_name 
+FROM products AS t2) AS t1 ON t0.product_id = t1.descendant_node_natural_key GROUP BY ancestor_node_sort_order ORDER BY ancestor_node_sort_order ASC
\ No newline at end of file
diff --git a/ibis/tests/sql/test_sqlalchemy.py b/ibis/tests/sql/test_sqlalchemy.py
--- a/ibis/tests/sql/test_sqlalchemy.py
+++ b/ibis/tests/sql/test_sqlalchemy.py
@@ -1140,3 +1140,32 @@ def test_to_sqla_type_array_of_non_primitive():
     assert result_name == expected_name
     assert type(result_type) == type(expected_type)
     assert isinstance(result, sa.ARRAY)
+
+
+def test_no_cart_join(con, snapshot):
+    facts = ibis.table(dict(product_id="!int32"), name="facts")
+    products = ibis.table(
+        dict(
+            ancestor_level_name="string",
+            ancestor_level_number="int32",
+            ancestor_node_sort_order="int64",
+            descendant_node_natural_key="int32",
+        ),
+        name="products",
+    )
+
+    products = products.mutate(
+        product_level_name=lambda t: ibis.literal('-')
+        .lpad(((t.ancestor_level_number - 1) * 7), '-')
+        .concat(t.ancestor_level_name)
+    )
+
+    predicate = facts.product_id == products.descendant_node_natural_key
+    joined = facts.join(products, predicate)
+
+    gb = joined.group_by(products.ancestor_node_sort_order)
+    agg = gb.aggregate(n=ibis.literal(1))
+    ob = agg.order_by(products.ancestor_node_sort_order)
+
+    out = str(con.compile(ob).compile(compile_kwargs=dict(literal_binds=True)))
+    snapshot.assert_match(out, "out.sql")