Repository: apache/calcite
Pull request #4226: [CALCITE-6873] FilterProjectTransposeRule should not push the Filter …
Author: login: NobiGo
Labels: name: LGTM-will-merge-soon
description: Overall PR looks OK. Only minor things left.
Created: 2025-03-05T07:22:50Z | Merged: 2025-03-07T01:43:21Z

…past the Project when the Filter contains a Subquery with correlation


Diff:
diff --git a/core/src/main/java/org/apache/calcite/rex/RexUtil.java b/core/src/main/java/org/apache/calcite/rex/RexUtil.java
index 2d1e78fb3e9b..d30a9d8a2bf9 100644
--- a/core/src/main/java/org/apache/calcite/rex/RexUtil.java
+++ b/core/src/main/java/org/apache/calcite/rex/RexUtil.java
@@ -2826,6 +2826,21 @@ private CorrelationFinder() {
     @Override public Void visitCorrelVariable(RexCorrelVariable var) {
       throw Util.FoundOne.NULL;
     }
+
+    @Override public Void visitSubQuery(RexSubQuery subQuery) {
+      if (!deep) {
+        return null;
+      }
+
+      for (RexNode operand : subQuery.operands) {
+        operand.accept(this);
+      }
+
+      if (!RelOptUtil.getVariablesUsed(subQuery.rel).isEmpty()) {
+        throw Util.FoundOne.NULL;
+      }
+      return null;
+    }
   }
 
   /** Shuttle that fixes up an expression to match changes in nullability of
diff --git a/core/src/test/java/org/apache/calcite/test/RelOptRulesTest.java b/core/src/test/java/org/apache/calcite/test/RelOptRulesTest.java
index baf8fd71832e..42d5f6149f50 100644
--- a/core/src/test/java/org/apache/calcite/test/RelOptRulesTest.java
+++ b/core/src/test/java/org/apache/calcite/test/RelOptRulesTest.java
@@ -521,6 +521,28 @@ private static boolean skipItem(RexNode expr) {
     }
   }
 
+  /** Test case for
+   * <a href="https://issues.apache.org/jira/browse/CALCITE-6873">[CALCITE-6873]
+   * FilterProjectTransposeRule should not push the Filter past the Project
+   * when the Filter contains a Subquery with correlation</a>. */
+  @Test void testFilterProjectTransposeRule2() {
+    final String sql = "select * from (select deptno from emp) as d\n"
+        + "where NOT EXISTS (\n"
+        + "  select count(*) from emp e where e.deptno = d.deptno)";
+    sql(sql)
+        .withRule(CoreRules.FILTER_PROJECT_TRANSPOSE)
+        .checkUnchanged();
+  }
+
+  @Test void testFilterProjectTransposeRule3() {
+    final String sql = "select * from (select deptno from emp) as d\n"
+        + "where NOT EXISTS (\n"
+        + "  select count(*) from emp e)";
+    sql(sql)
+        .withRule(CoreRules.FILTER_PROJECT_TRANSPOSE)
+        .check();
+  }
+
   /** Test case for
    * <a href="https://issues.apache.org/jira/browse/CALCITE-6031">[CALCITE-6031]
    * Add the planner rule that pushes the Filter past a Sample</a>. */
diff --git a/core/src/test/resources/org/apache/calcite/test/RelOptRulesTest.xml b/core/src/test/resources/org/apache/calcite/test/RelOptRulesTest.xml
index 153864230c54..ca64c718e6ba 100644
--- a/core/src/test/resources/org/apache/calcite/test/RelOptRulesTest.xml
+++ b/core/src/test/resources/org/apache/calcite/test/RelOptRulesTest.xml
@@ -5245,6 +5245,57 @@ LogicalProject(EMPNO=[$0])
           LogicalFilter(condition=[=($cor0.DEPTNO, $0)])
             LogicalProject(TWICEDEPTNO=[*($0, 2)])
               LogicalTableScan(table=[[CATALOG, SALES, DEPT]])
+]]>
+    </Resource>
+  </TestCase>
+  <TestCase name="testFilterProjectTransposeRule2">
+    <Resource name="sql">
+      <![CDATA[select * from (select deptno from emp) as d
+where NOT EXISTS (
+  select count(*) from emp e where e.deptno = d.deptno)]]>
+    </Resource>
+    <Resource name="planBefore">
+      <![CDATA[
+LogicalProject(DEPTNO=[$0])
+  LogicalFilter(condition=[NOT(EXISTS({
+LogicalAggregate(group=[{}], EXPR$0=[COUNT()])
+  LogicalProject($f0=[0])
+    LogicalFilter(condition=[=($7, $cor0.DEPTNO)])
+      LogicalTableScan(table=[[CATALOG, SALES, EMP]])
+}))], variablesSet=[[$cor0]])
+    LogicalProject(DEPTNO=[$7])
+      LogicalTableScan(table=[[CATALOG, SALES, EMP]])
+]]>
+    </Resource>
+  </TestCase>
+  <TestCase name="testFilterProjectTransposeRule3">
+    <Resource name="sql">
+      <![CDATA[select * from (select deptno from emp) as d
+where NOT EXISTS (
+  select count(*) from emp e)]]>
+    </Resource>
+    <Resource name="planBefore">
+      <![CDATA[
+LogicalProject(DEPTNO=[$0])
+  LogicalFilter(condition=[NOT(EXISTS({
+LogicalAggregate(group=[{}], EXPR$0=[COUNT()])
+  LogicalProject($f0=[0])
+    LogicalTableScan(table=[[CATALOG, SALES, EMP]])
+}))])
+    LogicalProject(DEPTNO=[$7])
+      LogicalTableScan(table=[[CATALOG, SALES, EMP]])
+]]>
+    </Resource>
+    <Resource name="planAfter">
+      <![CDATA[
+LogicalProject(DEPTNO=[$0])
+  LogicalProject(DEPTNO=[$7])
+    LogicalFilter(condition=[NOT(EXISTS({
+LogicalAggregate(group=[{}], EXPR$0=[COUNT()])
+  LogicalProject($f0=[0])
+    LogicalTableScan(table=[[CATALOG, SALES, EMP]])
+}))])
+      LogicalTableScan(table=[[CATALOG, SALES, EMP]])
 ]]>
     </Resource>
   </TestCase>


Reviews:
user: login: zabetak
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T08:41:47Z
user: login: NobiGo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T08:48:49Z
user: login: zabetak
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T09:04:53Z
user: login: NobiGo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T09:26:18Z
user: login: asolimando
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T10:34:37Z
user: login: NobiGo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T10:38:58Z
user: login: asolimando
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T10:50:50Z
user: login: zabetak
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T11:40:19Z
user: login: NobiGo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T01:25:21Z
user: login: zabetak
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-06T08:24:15Z
user: login: zabetak
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-06T08:27:15Z
user: login: asolimando
body: LGTM, thanks @NobiGo!
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-06T12:20:08Z

Review comments:
diff_hunk: @@ -213,7 +213,8 @@ protected FilterProjectTransposeRule(
   public interface Config extends RelRule.Config {
     Config DEFAULT = ImmutableFilterProjectTransposeRule.Config.of()
         .withOperandFor(Filter.class,
-            f -> !RexUtil.containsCorrelation(f.getCondition()),
+            f -> !RexUtil.containsCorrelation(f.getCondition())
+                && !RexUtil.SubQueryFinder.containsSubQuery(f),
path: core/src/main/java/org/apache/calcite/rel/rules/FilterProjectTransposeRule.java
user: login: zabetak
body: What about uncorrelated sub-query?
created_at: 2025-03-05T08:40:35Z
author_association: MEMBER
diff_hunk: @@ -213,7 +213,8 @@ protected FilterProjectTransposeRule(
   public interface Config extends RelRule.Config {
     Config DEFAULT = ImmutableFilterProjectTransposeRule.Config.of()
         .withOperandFor(Filter.class,
-            f -> !RexUtil.containsCorrelation(f.getCondition()),
+            f -> !RexUtil.containsCorrelation(f.getCondition())
+                && !RexUtil.SubQueryFinder.containsSubQuery(f),
path: core/src/main/java/org/apache/calcite/rel/rules/FilterProjectTransposeRule.java
user: login: NobiGo
body: Currently, The rule can't push down uncorrelated sub-queries either, because this can be achieved by removing the sub-query through other plans. Even if the sub-query is not removed, the subsequent plans can't be pushed down and will still be fine. 
Correlated sub-queries definitely cannot be pushed down. If pushed down, problems will occur in the subsequent plans. Therefore, here I did not differentiate whether the specific sub-queries contain correlation or not.
created_at: 2025-03-05T08:48:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -213,7 +213,8 @@ protected FilterProjectTransposeRule(
   public interface Config extends RelRule.Config {
     Config DEFAULT = ImmutableFilterProjectTransposeRule.Config.of()
         .withOperandFor(Filter.class,
-            f -> !RexUtil.containsCorrelation(f.getCondition()),
+            f -> !RexUtil.containsCorrelation(f.getCondition())
+                && !RexUtil.SubQueryFinder.containsSubQuery(f),
path: core/src/main/java/org/apache/calcite/rel/rules/FilterProjectTransposeRule.java
user: login: zabetak
body: The new condition mainly prevents filter push-down for **non-correlated** sub-queries. 

I get the impression that correlated sub-queries are already detected just by using `RexUtil.containsCorrelation`. The `CorrelationFinder` seems to traverse also sub-query expressions. Am I missing something?
created_at: 2025-03-05T09:04:53Z
author_association: MEMBER
diff_hunk: @@ -213,7 +213,8 @@ protected FilterProjectTransposeRule(
   public interface Config extends RelRule.Config {
     Config DEFAULT = ImmutableFilterProjectTransposeRule.Config.of()
         .withOperandFor(Filter.class,
-            f -> !RexUtil.containsCorrelation(f.getCondition()),
+            f -> !RexUtil.containsCorrelation(f.getCondition())
+                && !RexUtil.SubQueryFinder.containsSubQuery(f),
path: core/src/main/java/org/apache/calcite/rel/rules/FilterProjectTransposeRule.java
user: login: NobiGo
body: The new condition mainly prevents filter push-down for correlated sub-queries.

The RexUtil.containsCorrelation cannot detect all the correlations in the Subquery.

Because in RexVisitorImpl (CorrelationFinder):
```
  @Override public R visitSubQuery(RexSubQuery subQuery) {
    if (!deep) {
      return null;
    }

    R r = null;
    for (RexNode operand : subQuery.operands) {
      r = operand.accept(this);
    }
    return r;
  }
```
Correlation not only exists in operands but also may exist in Rel.

For example: 
EXISTS subquery operands are empty, and the query in Rel.
created_at: 2025-03-05T09:26:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -213,7 +213,8 @@ protected FilterProjectTransposeRule(
   public interface Config extends RelRule.Config {
     Config DEFAULT = ImmutableFilterProjectTransposeRule.Config.of()
         .withOperandFor(Filter.class,
-            f -> !RexUtil.containsCorrelation(f.getCondition()),
+            f -> !RexUtil.containsCorrelation(f.getCondition())
+                && !RexUtil.SubQueryFinder.containsSubQuery(f),
path: core/src/main/java/org/apache/calcite/rel/rules/FilterProjectTransposeRule.java
user: login: asolimando
body: Shouldn't we be providing a comprehensive function that handles both and use it here then?
created_at: 2025-03-05T10:34:37Z
author_association: MEMBER
diff_hunk: @@ -213,7 +213,8 @@ protected FilterProjectTransposeRule(
   public interface Config extends RelRule.Config {
     Config DEFAULT = ImmutableFilterProjectTransposeRule.Config.of()
         .withOperandFor(Filter.class,
-            f -> !RexUtil.containsCorrelation(f.getCondition()),
+            f -> !RexUtil.containsCorrelation(f.getCondition())
+                && !RexUtil.SubQueryFinder.containsSubQuery(f),
path: core/src/main/java/org/apache/calcite/rel/rules/FilterProjectTransposeRule.java
user: login: NobiGo
body: @asolimando @zabetak Based on the above discussion, I have enhanced RexUtil.containsCorrelation. If the test goes well, I will update the PR again. Maybe we can only push down the non-correlated sub-queries.
created_at: 2025-03-05T10:38:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -213,7 +213,8 @@ protected FilterProjectTransposeRule(
   public interface Config extends RelRule.Config {
     Config DEFAULT = ImmutableFilterProjectTransposeRule.Config.of()
         .withOperandFor(Filter.class,
-            f -> !RexUtil.containsCorrelation(f.getCondition()),
+            f -> !RexUtil.containsCorrelation(f.getCondition())
+                && !RexUtil.SubQueryFinder.containsSubQuery(f),
path: core/src/main/java/org/apache/calcite/rel/rules/FilterProjectTransposeRule.java
user: login: asolimando
body: Thanks @NobiGo, I feel like it's the right approach since all the callers of that function will benefit from this enhancement in a similar way.
created_at: 2025-03-05T10:50:50Z
author_association: MEMBER
diff_hunk: @@ -2826,6 +2826,21 @@ private CorrelationFinder() {
     @Override public Void visitCorrelVariable(RexCorrelVariable var) {
       throw Util.FoundOne.NULL;
     }
+
+    @Override public Void visitSubQuery(RexSubQuery subQuery) {
+      if (!deep) {
+        return null;
+      }
+
+      for (RexNode operand : subQuery.operands) {
+        operand.accept(this);
+      }
+
+      if (!RelOptUtil.getVariablesUsed(subQuery.rel).isEmpty()) {
+        throw Util.FoundOne.NULL;
path: core/src/main/java/org/apache/calcite/rex/RexUtil.java
user: login: zabetak
body: From a quick look it seems that `RelOptUtil.getVariablesUsed` on its own does exactly what you want. There is no need to call it from here. 

In fact, there is also `org.apache.calcite.plan.RelOptUtil#notContainsCorrelation` which might be exactly what you are looking for.
created_at: 2025-03-05T11:40:07Z
author_association: MEMBER
diff_hunk: @@ -2826,6 +2826,21 @@ private CorrelationFinder() {
     @Override public Void visitCorrelVariable(RexCorrelVariable var) {
       throw Util.FoundOne.NULL;
     }
+
+    @Override public Void visitSubQuery(RexSubQuery subQuery) {
+      if (!deep) {
+        return null;
+      }
+
+      for (RexNode operand : subQuery.operands) {
+        operand.accept(this);
+      }
+
+      if (!RelOptUtil.getVariablesUsed(subQuery.rel).isEmpty()) {
+        throw Util.FoundOne.NULL;
path: core/src/main/java/org/apache/calcite/rex/RexUtil.java
user: login: NobiGo
body: `RelOptUtil.getVariablesUsed` can collect all the variables used in the Filter operator, including those from the input plan and the filter's Condition. Here, we only need to focus on the variables within the Condition.
`RelOptUtil.notContainsCorrelation` determines whether a specific correlation is included. Here, we need to check whether a correlation exists.
Did I misunderstand something?
created_at: 2025-03-06T01:25:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2826,6 +2826,21 @@ private CorrelationFinder() {
     @Override public Void visitCorrelVariable(RexCorrelVariable var) {
       throw Util.FoundOne.NULL;
     }
+
+    @Override public Void visitSubQuery(RexSubQuery subQuery) {
+      if (!deep) {
+        return null;
+      }
+
+      for (RexNode operand : subQuery.operands) {
+        operand.accept(this);
+      }
+
+      if (!RelOptUtil.getVariablesUsed(subQuery.rel).isEmpty()) {
+        throw Util.FoundOne.NULL;
path: core/src/main/java/org/apache/calcite/rex/RexUtil.java
user: login: zabetak
body: My bad, I rushed a bit through the code and didn't realize that it was a full-blown RelNode visitor. The changes here do make sense now.
created_at: 2025-03-06T08:24:15Z
author_association: MEMBER

PR comments:
user: login: sonarqubecloud[bot]
created_at: 2025-03-07T01:26:06Z
body: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4226) **Quality Gate passed**  
Issues  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4226&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=4226&issueStatuses=ACCEPTED)

Measures  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=4226&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [64.3% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4226&metric=new_coverage&view=list)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=4226&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=4226)


author_association: NONE