Repository: tobymao/sqlglot

Issue:
Invalid SQL output from optimize function
Hello guys, I have been using the optimizer function, and I _think_ I have found a strange behaviour which is causing the optimize function to output an invalid query.

Whilst I found this isse on a trickier query, I have tried to create a simple example which illustrates the issue for reproducibility. Here it is:

```sql
WITH cte1
            AS (SELECT col1,
                       col2
                  FROM c1.d1.t1
                ),
     cte2
            AS (
                SELECT col1
                        , ARRAY_AGG(DISTINCT col3) AS col3_arr
                  FROM c1.d1.t2
                  WHERE col4 = 1
                  GROUP BY col1
                )
SELECT IF(cte1.col1 IN UNNEST(cte2.col3_arr), 1, 0) AS col5
FROM cte1
LEFT OUTER JOIN cte2 
USING (col1)
```

When I call optimize on this as follows:
```python
ast = parse_one(sql, dialect='bigquery')
print(optimize(ast,dialect='bigquery').sql(dialect='bigquery',pretty=True))
```

I get this: 

```sql
SELECT
  IF(`t1`.`col1` IN UNNEST(`cte2`.`col3_arr`), 1, 0) AS `col5`
FROM `c1`.`d1`.`t1` AS `t1`
```

However this is not valid sql - the CTE which is being referenced in the optimized query - `cte2` - is no longer defined, and the join relation has also been completely lost.

Is this an issue, or am I doing something wrong here?

Thanks guys!

_Originally posted by @rorynormaness in https://github.com/tobymao/sqlglot/discussions/3208_


Discussion:
I can look into this one. ETA is probably later today or tomorrow.

Resolving patch:
diff --git a/sqlglot/optimizer/scope.py b/sqlglot/optimizer/scope.py
--- a/sqlglot/optimizer/scope.py
+++ b/sqlglot/optimizer/scope.py
@@ -800,10 +800,9 @@ def walk_in_scope(expression, bfs=True, prune=None):
         if (
             isinstance(node, exp.CTE)
             or (
-                _is_derived_table(node)
-                and isinstance(node.parent, (exp.From, exp.Join, exp.Subquery))
+                isinstance(node.parent, (exp.From, exp.Join, exp.Subquery))
+                and (_is_derived_table(node) or isinstance(node, exp.UDTF))
             )
-            or isinstance(node, exp.UDTF)
             or isinstance(node, exp.UNWRAPPED_QUERIES)
         ):
             crossed_scope_boundary = True


Test patch:
diff --git a/tests/fixtures/optimizer/pushdown_projections.sql b/tests/fixtures/optimizer/pushdown_projections.sql
--- a/tests/fixtures/optimizer/pushdown_projections.sql
+++ b/tests/fixtures/optimizer/pushdown_projections.sql
@@ -79,6 +79,9 @@ WITH y AS (SELECT MAX(1) AS _ FROM x AS x) SELECT 1 AS "1" FROM y AS y;
 WITH y AS (SELECT a FROM x GROUP BY a) SELECT 1 FROM y;
 WITH y AS (SELECT 1 AS _ FROM x AS x GROUP BY x.a) SELECT 1 AS "1" FROM y AS y;
 
+WITH cte AS (SELECT col FROM t) SELECT IF(1 IN UNNEST(col), 1, 0) AS col FROM cte;
+WITH cte AS (SELECT t.col AS col FROM t AS t) SELECT CASE WHEN 1 IN (SELECT UNNEST(cte.col)) THEN 1 ELSE 0 END AS col FROM cte AS cte;
+
 --------------------------------------
 -- Unknown Star Expansion
 --------------------------------------
diff --git a/tests/test_optimizer.py b/tests/test_optimizer.py
--- a/tests/test_optimizer.py
+++ b/tests/test_optimizer.py
@@ -505,6 +505,9 @@ def test_file_schema(self):
         )
 
     def test_scope(self):
+        ast = parse_one("SELECT IF(a IN UNNEST(b), 1, 0) AS c FROM t", dialect="bigquery")
+        self.assertEqual(build_scope(ast).columns, [exp.column("a"), exp.column("b")])
+
         many_unions = parse_one(" UNION ALL ".join(["SELECT x FROM t"] * 10000))
         scopes_using_traverse = list(build_scope(many_unions).traverse())
         scopes_using_traverse_scope = traverse_scope(many_unions)