Repository: ibis-project/ibis

Issue:
bug: correlated subquery with join generates incorrect SQL
This Ibis expression (using the TPC-H database; a simplified version of TPC-H query 2) appears to be valid, but does not generate correct SQL:

```
    part = con.table("part")
    supplier = con.table("supplier")
    partsupp = con.table("partsupp")

    q = part.join(partsupp, part.P_PARTKEY == partsupp.PS_PARTKEY)
    q = q[
        part.P_PARTKEY,
        partsupp.PS_SUPPLYCOST,
    ]
    subq = partsupp.join(supplier, supplier.S_SUPPKEY == partsupp.PS_SUPPKEY)
    subq = subq.materialize()
    subq = subq[subq.PS_PARTKEY == q.P_PARTKEY]

    q = q[q.PS_SUPPLYCOST == subq.PS_SUPPLYCOST.min()]
```


### Expected Result

```sql
SELECT t0."P_PARTKEY",
       t0."PS_SUPPLYCOST"
   FROM base.part AS t1
   JOIN base.partsupp AS t2 ON t1."P_PARTKEY" = t2."PS_PARTKEY") AS t0
WHERE t0."PS_SUPPLYCOST" =
    (SELECT min(t3."PS_SUPPLYCOST") AS MIN
        FROM base.partsupp AS t2
        JOIN base.supplier AS t6 ON t6."S_SUPPKEY" = t2."PS_SUPPKEY") AS t3
     WHERE t3."PS_PARTKEY" = "P_PARTKEY"
```

### Actual Result

The generated SQL is:

```sql
SELECT t0."P_PARTKEY",
       t0."PS_SUPPLYCOST"
FROM
  (SELECT t1."P_PARTKEY" AS "P_PARTKEY",
          t2."PS_SUPPLYCOST" AS "PS_SUPPLYCOST"
   FROM base.part AS t1
   JOIN base.partsupp AS t2 ON t1."P_PARTKEY" = t2."PS_PARTKEY") AS t0
WHERE t0."PS_SUPPLYCOST" =
    (SELECT min(t3."PS_SUPPLYCOST") AS MIN
     FROM
       (SELECT [...]
        FROM base.partsupp AS t2
        JOIN base.supplier AS t6 ON t6."S_SUPPKEY" = t2."PS_SUPPKEY") AS t3,
       (SELECT t1."P_PARTKEY" AS "P_PARTKEY",
               t2."PS_SUPPLYCOST" AS "PS_SUPPLYCOST"
        FROM base.part AS t1
        JOIN base.partsupp AS t2 ON t1."P_PARTKEY" = t2."PS_PARTKEY")
     WHERE t3."PS_PARTKEY" = "P_PARTKEY")
```

The subquery above does an extra join with a copy of the outer query, whereas it should just reference the existing value in the outer query.  


Discussion:
Possible starting point:

Compare the output of one of the stringish backends like `impala` and see whether the SQL is correct. If not: then it's likely an ibis-wide issue, if it is, then it's likely ibis's use of SQLAlchemy that is buggy.
Impala output of the test query:
```sql
SELECT t0.*
FROM (
  SELECT t1.`p_partkey`, t2.`ps_supplycost`
  FROM part t1
    INNER JOIN partsupp t2
      ON t1.`p_partkey` = t2.`ps_partkey`
) t0
WHERE t0.`ps_supplycost` = (
  SELECT min(t3.`ps_supplycost`) AS `min`
  FROM (
    SELECT *
    FROM partsupp t2
      INNER JOIN supplier t6
        ON t6.`s_suppkey` = t2.`ps_suppkey`
  ) t3
  WHERE t3.`ps_partkey` = `p_partkey`
)
```
@gforsyth ~That looks correct. So then there's probably an issue with our use of sqlalchemy.~

It's not clear this code is correct.
Does this bug occurs with another dataset? I know this is a longshot but...
We're pretty deep into this and it appears to be related to mis- or non-use of `.as_scalar()` and `.correlate()` methods in `.where` usage.

Resolving patch:
diff --git a/ibis/backends/base/sql/alchemy/registry.py b/ibis/backends/base/sql/alchemy/registry.py
--- a/ibis/backends/base/sql/alchemy/registry.py
+++ b/ibis/backends/base/sql/alchemy/registry.py
@@ -77,7 +77,7 @@ def _varargs_call(sa_func, t, expr):
 
 
 def get_sqla_table(ctx, table):
-    if ctx.has_ref(table):
+    if ctx.has_ref(table, parent_contexts=True):
         ctx_level = ctx
         sa_table = ctx_level.get_ref(table)
         while sa_table is None and ctx_level.parent is not ctx_level:


Test patch:
diff --git a/ibis/tests/sql/test_sqlalchemy.py b/ibis/tests/sql/test_sqlalchemy.py
--- a/ibis/tests/sql/test_sqlalchemy.py
+++ b/ibis/tests/sql/test_sqlalchemy.py
@@ -622,6 +622,87 @@ def test_sort_aggregation_translation_failure(self):
 
         self._compare_sqla(expr, ex)
 
+    def test_where_correlated_subquery_with_join(self):
+        # GH3163
+        # ibis code
+        part = ibis.table([("p_partkey", "int64")], name="part")
+        partsupp = ibis.table(
+            [
+                ("ps_partkey", "int64"),
+                ("ps_supplycost", "float64"),
+                ("ps_suppkey", "int64"),
+            ],
+            name="partsupp",
+        )
+        supplier = ibis.table([("s_suppkey", "int64")], name="supplier")
+
+        q = part.join(partsupp, part.p_partkey == partsupp.ps_partkey)
+        q = q[
+            part.p_partkey,
+            partsupp.ps_supplycost,
+        ]
+        subq = partsupp.join(
+            supplier, supplier.s_suppkey == partsupp.ps_suppkey
+        )
+        subq = subq.projection([partsupp.ps_partkey, partsupp.ps_supplycost])
+        subq = subq[subq.ps_partkey == q.p_partkey]
+
+        expr = q[q.ps_supplycost == subq.ps_supplycost.min()]
+
+        # sqlalchemy code
+        part = sa.table("part", sa.column("p_partkey"))
+        supplier = sa.table("supplier", sa.column("s_suppkey"))
+        partsupp = sa.table(
+            "partsupp",
+            sa.column("ps_partkey"),
+            sa.column("ps_supplycost"),
+            sa.column("ps_suppkey"),
+        )
+
+        part_t1 = part.alias("t1")
+        partsupp_t2 = partsupp.alias("t2")
+
+        t0 = (
+            sa.select([part_t1.c.p_partkey, partsupp_t2.c.ps_supplycost])
+            .select_from(
+                part_t1.join(
+                    partsupp_t2,
+                    onclause=part_t1.c.p_partkey == partsupp_t2.c.ps_partkey,
+                )
+            )
+            .alias("t0")
+        )
+
+        partsupp_t2 = partsupp.alias("t2")
+        supplier_t5 = supplier.alias("t5")
+        t3 = (
+            sa.select([partsupp_t2.c.ps_partkey, partsupp_t2.c.ps_supplycost])
+            .select_from(
+                partsupp_t2.join(
+                    supplier_t5,
+                    onclause=supplier_t5.c.s_suppkey
+                    == partsupp_t2.c.ps_suppkey,
+                )
+            )
+            .alias("t3")
+        )
+
+        ex = (
+            sa.select([t0.c.p_partkey, t0.c.ps_supplycost])
+            .select_from(t0)
+            .where(
+                t0.c.ps_supplycost
+                == (
+                    sa.select([sa.func.min(t3.c.ps_supplycost).label("min")])
+                    .select_from(t3)
+                    .where(t3.c.ps_partkey == t0.c.p_partkey)
+                    .as_scalar()
+                )
+            )
+        )
+
+        self._compare_sqla(expr, ex)
+
     def _compare_sqla(self, expr, sqla):
         context = AlchemyContext(compiler=AlchemyCompiler)
         result_sqla = AlchemyCompiler.to_sql(expr, context)