Repository: apache/incubator-gluten
Pull request #8946: [GLUTEN-8945][VL] Pull out duplicate projections for HashProbe and FilterProject
Author: login: zml1206
Labels: name: VELOX
Created: 2025-03-10T08:03:49Z | Merged: 2025-03-19T09:57:31Z

## What changes were proposed in this pull request?

(Fixes: \#8945)
```
select t3.* from
(select c1, c2 as a,c2 as b from t1) t3
left join t2
on t3.c1 = t2.c1
```
Before PR:
ERROR: An unloaded lazy vector cannot be wrapped by two different top level vectors.
```
ProjectExecTransformer [c1#22, a#30, b#31]
 - BroadcastHashJoinExecTransformer [c1#22], [c1#24], LeftOuter, BuildRight, false
   :- ProjectExecTransformer [c1#22, c2#23 AS a#30, c2#23 AS b#31]
   :   - FileScanTransformer parquet spark_catalog.default.t1[c1#22,c2#23] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/Users/zml/Desktop/git_hub/incubator-gluten/spark-warehouse/org.a..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c1:int,c2:int> NativeFilters: []
    - BroadcastQueryStage 0
       - ColumnarBroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)),false), [plan_id=772]
          - ^(3) FilterExecTransformer isnotnull(c1#24)
             - ^(3) FileScanTransformer parquet spark_catalog.default.t2[c1#24] Batched: true, DataFilters: [isnotnull(c1#24)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/Users/zml/Desktop/git_hub/incubator-gluten/spark-warehouse/org.a..., PartitionFilters: [], PushedFilters: [IsNotNull(c1)], ReadSchema: struct<c1:int> NativeFilters: [isnotnull(c1#24)]
```
After PR:
`c2 as a, c2 as b` was posted out to the project outside the join.
```
+- == Final Plan ==
ProjectExecTransformer [c1#22, c2#23 AS a#30, c2#23 AS b#31]
 - BroadcastHashJoinExecTransformer [c1#22], [c1#24], LeftOuter, BuildRight, false
   :- ProjectExecTransformer [c1#22, c2#23]
   :   - FileScanTransformer parquet spark_catalog.default.t1[c1#22,c2#23] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/Users/zml/Desktop/git_hub/incubator-gluten/spark-warehouse/org.a..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c1:int,c2:int> NativeFilters: []
    - BroadcastQueryStage 0
       - ColumnarBroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)),false), [plan_id=772]
          - ^(3) FilterExecTransformer isnotnull(c1#24)
             - ^(3) FileScanTransformer parquet spark_catalog.default.t2[c1#24] Batched: true, DataFilters: [isnotnull(c1#24)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/Users/zml/Desktop/git_hub/incubator-gluten/spark-warehouse/org.a..., PartitionFilters: [], PushedFilters: [IsNotNull(c1)], ReadSchema: struct<c1:int> NativeFilters: [isnotnull(c1#24)]

```
## How was this patch tested?

UT.




Diff:
diff --git a/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxRuleApi.scala b/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxRuleApi.scala
index 3b15fa2263de..566cffd1bf18 100644
--- a/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxRuleApi.scala
+++ b/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxRuleApi.scala
@@ -96,6 +96,7 @@ object VeloxRuleApi {
     injector.injectPostTransform(_ => PushDownInputFileExpression.PostOffload)
     injector.injectPostTransform(_ => EnsureLocalSortRequirements)
     injector.injectPostTransform(_ => EliminateLocalSort)
+    injector.injectPostTransform(_ => PullOutDuplicateProject)
     injector.injectPostTransform(_ => CollapseProjectExecTransformer)
     injector.injectPostTransform(c => FlushableHashAggregateRule.apply(c.session))
     injector.injectPostTransform(c => InsertTransitions.create(c.outputsColumnar, VeloxBatch))
@@ -179,6 +180,7 @@ object VeloxRuleApi {
     injector.injectPostTransform(_ => PushDownInputFileExpression.PostOffload)
     injector.injectPostTransform(_ => EnsureLocalSortRequirements)
     injector.injectPostTransform(_ => EliminateLocalSort)
+    injector.injectPostTransform(_ => PullOutDuplicateProject)
     injector.injectPostTransform(_ => CollapseProjectExecTransformer)
     injector.injectPostTransform(c => FlushableHashAggregateRule.apply(c.session))
     injector.injectPostTransform(c => InsertTransitions.create(c.outputsColumnar, VeloxBatch))
diff --git a/backends-velox/src/main/scala/org/apache/gluten/extension/PullOutDuplicateProject.scala b/backends-velox/src/main/scala/org/apache/gluten/extension/PullOutDuplicateProject.scala
new file mode 100644
index 000000000000..7e6958bc4248
--- /dev/null
+++ b/backends-velox/src/main/scala/org/apache/gluten/extension/PullOutDuplicateProject.scala
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.extension
+
+import org.apache.gluten.execution.{BroadcastHashJoinExecTransformer, FilterExecTransformer, LimitExecTransformer, ProjectExecTransformer}
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute, AttributeMap, AttributeSet, PredicateHelper}
+import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution._
+
+import scala.collection.mutable.ArrayBuffer
+
+/**
+ * Velox does not allow duplicate projections in HashProbe and FilterProject, this rule pull out
+ * duplicate projections to a new project outside.
+ */
+object PullOutDuplicateProject extends Rule[SparkPlan] with PredicateHelper {
+  override def apply(plan: SparkPlan): SparkPlan = plan.transformUp {
+    case l @ LimitExecTransformer(p: ProjectExecTransformer, _, _) =>
+      val pullOutAliases = new ArrayBuffer[Alias]()
+      val newChild = rewriteProject(p, AttributeSet.empty, pullOutAliases)
+      if (pullOutAliases.isEmpty) {
+        l
+      } else {
+        outerProject(l.copy(child = newChild), l.output, pullOutAliases)
+      }
+    case p @ ProjectExecTransformer(_, child: ProjectExecTransformer) =>
+      val pullOutAliases = new ArrayBuffer[Alias]()
+      val newChild = rewriteProject(child, AttributeSet.empty, pullOutAliases)
+      val aliasMap = AttributeMap(pullOutAliases.map(a => a.toAttribute -> a))
+      val newProjectList = p.projectList.map(replaceAliasButKeepName(_, aliasMap))
+      ProjectExecTransformer(newProjectList, newChild)
+    case f @ FilterExecTransformer(_, child: ProjectExecTransformer) =>
+      val pullOutAliases = new ArrayBuffer[Alias]()
+      val newChild = rewriteProject(child, f.references, pullOutAliases)
+      if (pullOutAliases.isEmpty) {
+        f
+      } else {
+        outerProject(f.copy(child = newChild), f.output, pullOutAliases)
+      }
+    case bhj: BroadcastHashJoinExecTransformer
+        if bhj.streamedPlan.isInstanceOf[ProjectExecTransformer] =>
+      val pullOutAliases = new ArrayBuffer[Alias]()
+      val newStreamedPlan = rewriteProject(
+        bhj.streamedPlan.asInstanceOf[ProjectExecTransformer],
+        bhj.references,
+        pullOutAliases)
+      if (pullOutAliases.isEmpty) {
+        bhj
+      } else {
+        val newBhj = bhj.joinBuildSide match {
+          case BuildLeft => bhj.copy(right = newStreamedPlan)
+          case BuildRight => bhj.copy(left = newStreamedPlan)
+        }
+        outerProject(newBhj, bhj.output, pullOutAliases)
+      }
+  }
+
+  private def outerProject(
+      child: SparkPlan,
+      output: Seq[Attribute],
+      pullOutAliases: ArrayBuffer[Alias]): ProjectExecTransformer = {
+    val aliasMap = AttributeMap(pullOutAliases.map(a => a.toAttribute -> a))
+    val newProjectList = output.map(attr => aliasMap.getOrElse(attr, attr))
+    ProjectExecTransformer(newProjectList, child)
+  }
+
+  /**
+   * If there are duplicate projections and not refer to parent, only the original attribute is kept
+   * in the project.
+   */
+  private def rewriteProject(
+      project: ProjectExecTransformer,
+      references: AttributeSet,
+      pullOutAliases: ArrayBuffer[Alias]): SparkPlan = {
+    val projectList = project.projectList
+    val duplicates = AttributeSet(
+      projectList
+        .collect {
+          case attr: Attribute if !references.contains(attr) => attr
+          case a @ Alias(attr: Attribute, _)
+              if !references.contains(a) && !references.contains(attr) =>
+            attr
+        }
+        .groupBy(_.exprId)
+        .filter(_._2.size > 1)
+        .map(_._2.head))
+    if (duplicates.nonEmpty) {
+      val newProjectList = projectList.filter {
+        case a @ Alias(attr: Attribute, _)
+            if !references.contains(a) && duplicates.contains(attr) =>
+          pullOutAliases.append(a)
+          false
+        case _ => true
+      } ++ duplicates.filter(!project.outputSet.contains(_)).toSeq
+      val newProject = project.copy(projectList = newProjectList)
+      // If the output of the new project is the same as the child, delete it to simplify the plan.
+      if (newProject.outputSet.equals(project.child.outputSet)) {
+        project.child
+      } else {
+        newProject
+      }
+    } else {
+      project
+    }
+  }
+}
diff --git a/backends-velox/src/test/scala/org/apache/gluten/execution/VeloxHashJoinSuite.scala b/backends-velox/src/test/scala/org/apache/gluten/execution/VeloxHashJoinSuite.scala
index 2023ad97ebab..9ec2ec33a68e 100644
--- a/backends-velox/src/test/scala/org/apache/gluten/execution/VeloxHashJoinSuite.scala
+++ b/backends-velox/src/test/scala/org/apache/gluten/execution/VeloxHashJoinSuite.scala
@@ -21,7 +21,7 @@ import org.apache.gluten.sql.shims.SparkShimLoader
 
 import org.apache.spark.SparkConf
 import org.apache.spark.sql.Row
-import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference}
 import org.apache.spark.sql.execution.{ColumnarBroadcastExchangeExec, ColumnarSubqueryBroadcastExec, InputIteratorTransformer}
 import org.apache.spark.sql.execution.exchange.{BroadcastExchangeExec, ReusedExchangeExec}
 
@@ -195,4 +195,71 @@ class VeloxHashJoinSuite extends VeloxWholeStageTransformerSuite {
           }
         })
   }
+
+  test("pull out duplicate projections for HashProbe and FilterProject") {
+    withTable("t1", "t2", "t3") {
+      Seq((1, 1), (2, 2)).toDF("c1", "c2").write.saveAsTable("t1")
+      Seq(1, 2, 3).toDF("c1").write.saveAsTable("t2")
+      Seq(1, 2, 3).toDF("c1").write.saveAsTable("t3")
+      // test HashProbe, pull out `c2 as a,c2 as b`.
+      val q1 =
+        """
+          |select tt1.* from
+          |(select c1,c2, c2 as a,c2 as b from t1) tt1
+          |left join t2
+          |on tt1.c1 = t2.c1
+          |""".stripMargin
+      val q2 =
+        """
+          |select tt1.* from
+          |(select c1, c2 as a,c2 as b from t1) tt1
+          |left join t2
+          |on tt1.c1 = t2.c1
+          |limit 1
+          |""".stripMargin
+      val q3 =
+        """
+          |select tt1.* from
+          |(select c1, c2 as a,c2 as b from t1) tt1
+          |left join t2
+          |on tt1.c1 = t2.c1
+          |left join t3
+          |on tt1.c1 = t3.c1
+          |""".stripMargin
+      Seq(q1, q2, q3).foreach {
+        runQueryAndCompare(_) {
+          df =>
+            {
+              val executedPlan = getExecutedPlan(df)
+              val projects = executedPlan.collect {
+                case p @ ProjectExecTransformer(_, _: BroadcastHashJoinExecTransformer) => p
+              }
+              assert(projects.nonEmpty)
+              val aliases = projects.last.projectList.collect { case a: Alias => a }
+              assert(aliases.size == 2)
+            }
+        }
+      }
+
+      // test FilterProject, only pull out `c2 as b`.
+      val q4 =
+        """
+          |select c1, c2, a, b from
+          |(select c1, c2, c2 as a, c2 as b, rand() as c from t1) tt1
+          |where c > -1 and b > 1
+          |""".stripMargin
+      runQueryAndCompare(q4) {
+        df =>
+          {
+            val executedPlan = getExecutedPlan(df)
+            val projects = executedPlan.collect {
+              case p @ ProjectExecTransformer(_, _: FilterExecTransformer) => p
+            }
+            assert(projects.nonEmpty)
+            val aliases = projects.last.projectList.collect { case a: Alias => a }
+            assert(aliases.size == 1)
+          }
+      }
+    }
+  }
 }


Reviews:
user: login: rui-mo
body: Thanks.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T11:37:55Z
user: login: zml1206
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T02:30:31Z
user: login: zml1206
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T02:30:36Z
user: login: rui-mo
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T10:32:50Z
user: login: zml1206
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T10:36:14Z

Review comments:
diff_hunk: @@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.extension.columnar
+
+import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.execution.{BroadcastHashJoinExecTransformerBase, LimitExecTransformer, ProjectExecTransformer}
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute, AttributeMap, AttributeSet, PredicateHelper}
+import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution._
+
+import scala.collection.mutable.ArrayBuffer
+
+/**
+ * Velox does not allow duplicate projections in hash probe, this rule pull out duplicate
+ * projections.
+ */
path: gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/PullOutDuplicateProject.scala
user: login: rui-mo
body: Perhaps elaborate on what was the plan like, and what is the new plan like.
created_at: 2025-03-12T11:34:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.extension.columnar
+
+import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.execution.{BroadcastHashJoinExecTransformerBase, LimitExecTransformer, ProjectExecTransformer}
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute, AttributeMap, AttributeSet, PredicateHelper}
+import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution._
+
+import scala.collection.mutable.ArrayBuffer
+
+/**
+ * Velox does not allow duplicate projections in hash probe, this rule pull out duplicate
+ * projections.
+ */
+object PullOutDuplicateProject extends Rule[SparkPlan] with PredicateHelper {
+  override def apply(plan: SparkPlan): SparkPlan = plan.transformUp {
+    case bhj: BroadcastHashJoinExecTransformerBase =>
+      val pullOutAliases = new ArrayBuffer[Alias]()
+      val streamedPlan = rewriteStreamPlan(bhj.streamedPlan, bhj.references, pullOutAliases)
+      if (pullOutAliases.isEmpty) {
+        bhj
+      } else {
+        val aliasMap = AttributeMap(pullOutAliases.map(a => a.toAttribute -> a))
+        val newProjectList = bhj.output.map(attr => aliasMap.getOrElse(attr, attr))
+        val (newLeft, newRight) = bhj.joinBuildSide match {
+          case BuildLeft => (bhj.left, streamedPlan)
+          case BuildRight => (streamedPlan, bhj.right)
+        }
+
+        val newBhj =
+          BackendsApiManager.getSparkPlanExecApiInstance.genBroadcastHashJoinExecTransformer(
+            bhj.leftKeys,
+            bhj.rightKeys,
+            bhj.hashJoinType,
+            bhj.joinBuildSide,
+            bhj.condition,
+            newLeft,
+            newRight,
+            bhj.genJoinParametersInternal()._2 == 1)
+        ProjectExecTransformer(newProjectList, newBhj)
+      }
+  }
+
+  def rewriteStreamPlan(
path: gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/PullOutDuplicateProject.scala
user: login: rui-mo
body: Please add comments for how stream plan is rewritten.
created_at: 2025-03-12T11:37:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.extension.columnar
+
+import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.execution.{BroadcastHashJoinExecTransformerBase, LimitExecTransformer, ProjectExecTransformer}
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute, AttributeMap, AttributeSet, PredicateHelper}
+import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution._
+
+import scala.collection.mutable.ArrayBuffer
+
+/**
+ * Velox does not allow duplicate projections in hash probe, this rule pull out duplicate
+ * projections.
+ */
+object PullOutDuplicateProject extends Rule[SparkPlan] with PredicateHelper {
+  override def apply(plan: SparkPlan): SparkPlan = plan.transformUp {
+    case bhj: BroadcastHashJoinExecTransformerBase =>
+      val pullOutAliases = new ArrayBuffer[Alias]()
+      val streamedPlan = rewriteStreamPlan(bhj.streamedPlan, bhj.references, pullOutAliases)
+      if (pullOutAliases.isEmpty) {
+        bhj
+      } else {
+        val aliasMap = AttributeMap(pullOutAliases.map(a => a.toAttribute -> a))
+        val newProjectList = bhj.output.map(attr => aliasMap.getOrElse(attr, attr))
+        val (newLeft, newRight) = bhj.joinBuildSide match {
+          case BuildLeft => (bhj.left, streamedPlan)
+          case BuildRight => (streamedPlan, bhj.right)
+        }
+
+        val newBhj =
+          BackendsApiManager.getSparkPlanExecApiInstance.genBroadcastHashJoinExecTransformer(
+            bhj.leftKeys,
+            bhj.rightKeys,
+            bhj.hashJoinType,
+            bhj.joinBuildSide,
+            bhj.condition,
+            newLeft,
+            newRight,
+            bhj.genJoinParametersInternal()._2 == 1)
+        ProjectExecTransformer(newProjectList, newBhj)
+      }
+  }
+
+  def rewriteStreamPlan(
path: gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/PullOutDuplicateProject.scala
user: login: zml1206
body: Added.
created_at: 2025-03-13T02:30:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.extension.columnar
+
+import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.execution.{BroadcastHashJoinExecTransformerBase, LimitExecTransformer, ProjectExecTransformer}
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute, AttributeMap, AttributeSet, PredicateHelper}
+import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution._
+
+import scala.collection.mutable.ArrayBuffer
+
+/**
+ * Velox does not allow duplicate projections in hash probe, this rule pull out duplicate
+ * projections.
+ */
path: gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/PullOutDuplicateProject.scala
user: login: zml1206
body: 
Added in pr description.
created_at: 2025-03-13T02:30:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.extension
+
+import org.apache.gluten.execution.{BroadcastHashJoinExecTransformer, FilterExecTransformer, LimitExecTransformer, ProjectExecTransformer}
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute, AttributeMap, AttributeSet, PredicateHelper}
+import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution._
+
+import scala.collection.mutable.ArrayBuffer
+
+/**
+ * Velox does not allow duplicate projections in HashProbe and FilterProject, this rule pull out
path: backends-velox/src/main/scala/org/apache/gluten/extension/PullOutDuplicateProject.scala
user: login: rui-mo
body: > Velox does not allow duplicate projections in HashProbe and FilterProject

Could you provide a reference in Velox for this description, and is there any other operation having similar limitation?
created_at: 2025-03-18T10:31:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.extension
+
+import org.apache.gluten.execution.{BroadcastHashJoinExecTransformer, FilterExecTransformer, LimitExecTransformer, ProjectExecTransformer}
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute, AttributeMap, AttributeSet, PredicateHelper}
+import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution._
+
+import scala.collection.mutable.ArrayBuffer
+
+/**
+ * Velox does not allow duplicate projections in HashProbe and FilterProject, this rule pull out
path: backends-velox/src/main/scala/org/apache/gluten/extension/PullOutDuplicateProject.scala
user: login: zml1206
body: https://github.com/facebookincubator/velox/issues/12484#issuecomment-2704265309, 
I have only found these two so far, I am not sure about the others.
created_at: 2025-03-18T10:36:14Z
author_association: CONTRIBUTOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-03-10T08:04:06Z
body: https://github.com/apache/incubator-gluten/issues/8945
author_association: NONE
user: login: github-actions[bot]
created_at: 2025-03-10T08:04:23Z
body: Run Gluten Clickhouse CI on x86
author_association: NONE
user: login: github-actions[bot]
created_at: 2025-03-10T08:26:20Z
body: Run Gluten Clickhouse CI on x86
author_association: NONE
user: login: github-actions[bot]
created_at: 2025-03-13T02:30:14Z
body: Run Gluten Clickhouse CI on x86
author_association: NONE
user: login: github-actions[bot]
created_at: 2025-03-13T13:54:16Z
body: Run Gluten Clickhouse CI on x86
author_association: NONE