Repository: apache/datafusion-comet
Pull request #1556: feat: introduce hadoop mini cluster to test native scan on hdfs
Author: login: wForget
Created: 2025-03-19T08:23:39Z | Merged: 2025-03-25T22:08:27Z

## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->

Closes #1515.

## Rationale for this change

test native scan on hdfs

## What changes are included in this PR?

<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->

introduce hadoop mini cluster to test native scan on hdfs

## How are these changes tested?

Successfully run `CometReadHdfsBenchmark` locally (tips: build native enable hdfs: `cd native && cargo build --features hdfs`)


Linked issue #123: Add sqllogictest in ci to verify execution results against vanilla spark sql.
### What is the problem the feature request solves?

[sqllogictest](https://github.com/hydromatic/sql-logic-test) is a commonly used data driven approach to verify query engine.  We can use this to verify the results of comet are same as  vanilla spark sql.

### Describe the potential solution

_No response_

### Additional context

_No response_
Linked issue comments: user: login: liurenjie1024
created_at: 2024-02-27T07:12:07Z
body: I'm happy to take this is we have reached consensus on this.
author_association: NONE
user: login: viirya
created_at: 2024-02-27T07:16:06Z
body: Hmm, I guess the question is, how do you compare with Spark result in sqllogictest?

For example, in current unit test or e2e test like CometTPCDSQuerySuite, we compare Comet query result with Spark result generated in runtime by disabling Comet or golden filed generated by Spark.

But for sqllogictest, as I saw its usage in DataFusion, it works like comparing actual output with previous output (in sqllogictest file) to check output consistency. So it cannot be used to run Spark in runtime to get expected output.
author_association: MEMBER
user: login: liurenjie1024
created_at: 2024-02-27T07:21:57Z
body: >But for sqllogictest, as I saw its usage in DataFusion, it works like comparing actual output with expected output (in sqllogictest file). So it cannot be used to run Spark in runtime to get expected output.

Why? We can use spark connect.
author_association: NONE
user: login: viirya
created_at: 2024-02-27T07:24:10Z
body: > Why? We can use spark connect.

Can sqllogictest run Spark connect? And how do you compare it in sqllogictest?
author_association: MEMBER
user: login: liurenjie1024
created_at: 2024-02-27T07:25:23Z
body: > > Why? We can use spark connect.
> 
> Can sqllogictest run Spark connect?

There are two ways to do this:

1. We can use sqllogictest as a library.
2. We develop a test executor for sqllogictest.

Please note that the link I provided is java's implementation of sqllogictest.
author_association: NONE
user: login: liurenjie1024
created_at: 2024-02-27T07:26:05Z
body: >  And how do you compare it in sqllogictest?

We just run the slt files against both vanilla spark and comet, and they both should be ok.
author_association: NONE
user: login: viirya
created_at: 2024-02-27T07:57:21Z
body: I'm not sure if this makes sense. I tend to think this doesn't make too much sense at this moment.

* It is not something ready to use. As you said, we need to implement a sqllogictest test executor for Spark to talk to Spark connect. I think this makes thing complicated. For the test, we need Spark server, Spark connect client, sqllogictest text executor, sqllogictest... too many components which increase maintenance effort. 
* The slt files need to be run with both Spark and Comet. Although we have unit tests also doing like that, the number is limited.
* We will soon have Spark test pipelines which cover Spark compatibility which leverage existing Spark tests. I think Spark test itself has most correct and highest coverage to make sure Comet's Spark compatibility. If there are very corner cases which are missing in Spark tests, I think these cases should be less. We probably can add the tests directly into Spark or just in Comet as unit test.

author_association: MEMBER
user: login: liurenjie1024
created_at: 2024-02-27T08:07:23Z
body: Sounds reasonable, if we can port spark tests in this repo, it would be enough. Thanks for sharing!
author_association: NONE

Diff:
diff --git a/native/core/Cargo.toml b/native/core/Cargo.toml
index bc91972239..e964b68694 100644
--- a/native/core/Cargo.toml
+++ b/native/core/Cargo.toml
@@ -68,7 +68,7 @@ datafusion-comet-proto = { workspace = true }
 object_store = { workspace = true }
 url = { workspace = true }
 parking_lot = "0.12.3"
-datafusion-comet-objectstore-hdfs = { path = "../hdfs", optional = true}
+datafusion-comet-objectstore-hdfs = { path = "../hdfs", optional = true, default-features = false, features = ["hdfs"] }
 
 [dev-dependencies]
 pprof = { version = "0.14.0", features = ["flamegraph"] }
diff --git a/pom.xml b/pom.xml
index 259760d51f..a057bef126 100644
--- a/pom.xml
+++ b/pom.xml
@@ -58,6 +58,7 @@ under the License.
     <protobuf.version>3.25.5</protobuf.version>
     <parquet.version>1.13.1</parquet.version>
     <parquet.maven.scope>provided</parquet.maven.scope>
+    <hadoop.version>3.3.4</hadoop.version>
     <arrow.version>16.0.0</arrow.version>
     <codehaus.jackson.version>1.9.13</codehaus.jackson.version>
     <spotless.version>2.43.0</spotless.version>
@@ -447,6 +448,13 @@ under the License.
         <version>5.1.0</version>
       </dependency>
 
+      <dependency>
+        <groupId>org.apache.hadoop</groupId>
+        <artifactId>hadoop-client-minicluster</artifactId>
+        <version>${hadoop.version}</version>
+        <scope>test</scope>
+      </dependency>
+
     </dependencies>
 
   </dependencyManagement>
diff --git a/spark/pom.xml b/spark/pom.xml
index d28b5d29e9..80150d0242 100644
--- a/spark/pom.xml
+++ b/spark/pom.xml
@@ -141,6 +141,30 @@ under the License.
       <artifactId>arrow-c-data</artifactId>
       <scope>test</scope>
     </dependency>
+    <dependency>
+      <groupId>org.apache.hadoop</groupId>
+      <artifactId>hadoop-client-minicluster</artifactId>
+      <scope>test</scope>
+      <exclusions>
+        <!-- hadoop clients are provided by spark -->
+        <exclusion>
+          <artifactId>hadoop-client-api</artifactId>
+          <groupId>org.apache.hadoop</groupId>
+        </exclusion>
+        <exclusion>
+          <artifactId>hadoop-client-runtime</artifactId>
+          <groupId>org.apache.hadoop</groupId>
+        </exclusion>
+        <exclusion>
+          <artifactId>snappy-java</artifactId>
+          <groupId>org.xerial.snappy</groupId>
+        </exclusion>
+        <exclusion>
+          <artifactId>junit</artifactId>
+          <groupId>junit</groupId>
+        </exclusion>
+      </exclusions>
+    </dependency>
   </dependencies>
 
   <build>
diff --git a/spark/src/test/scala/org/apache/comet/WithHdfsCluster.scala b/spark/src/test/scala/org/apache/comet/WithHdfsCluster.scala
new file mode 100644
index 0000000000..49124d63e5
--- /dev/null
+++ b/spark/src/test/scala/org/apache/comet/WithHdfsCluster.scala
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet
+
+import java.io.{File, FileWriter}
+import java.net.InetAddress
+import java.nio.file.Files
+import java.util.UUID
+
+import scala.collection.JavaConverters._
+
+import org.apache.commons.io.FileUtils
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hadoop.hdfs.MiniDFSCluster
+import org.apache.hadoop.hdfs.client.HdfsClientConfigKeys
+import org.apache.spark.internal.Logging
+
+/**
+ * Trait for starting and stopping a MiniDFSCluster for testing.
+ *
+ * Most copy from:
+ * https://github.com/apache/kyuubi/blob/master/kyuubi-server/src/test/scala/org/apache/kyuubi/server/MiniDFSService.scala
+ */
+trait WithHdfsCluster extends Logging {
+
+  private var hadoopConfDir: File = _
+  private var hdfsCluster: MiniDFSCluster = _
+  private var hdfsConf: Configuration = _
+  private var tmpRootDir: Path = _
+  private var fileSystem: FileSystem = _
+
+  def startHdfsCluster(): Unit = {
+    hdfsConf = new Configuration()
+    // before HADOOP-18206 (3.4.0), HDFS MetricsLogger strongly depends on
+    // commons-logging, we should disable it explicitly, otherwise, it throws
+    // ClassNotFound: org.apache.commons.logging.impl.Log4JLogger
+    hdfsConf.set("dfs.namenode.metrics.logger.period.seconds", "0")
+    hdfsConf.set("dfs.datanode.metrics.logger.period.seconds", "0")
+    // Set bind host to localhost to avoid java.net.BindException
+    hdfsConf.setIfUnset("dfs.namenode.rpc-bind-host", "localhost")
+
+    hdfsCluster = new MiniDFSCluster.Builder(hdfsConf)
+      .checkDataNodeAddrConfig(true)
+      .checkDataNodeHostConfig(true)
+      .build()
+    logInfo(
+      "NameNode address in configuration is " +
+        s"${hdfsConf.get(HdfsClientConfigKeys.DFS_NAMENODE_RPC_ADDRESS_KEY)}")
+    hadoopConfDir =
+      Files.createTempDirectory(s"comet_hdfs_conf_${UUID.randomUUID().toString}").toFile
+    saveHadoopConf(hadoopConfDir)
+
+    fileSystem = hdfsCluster.getFileSystem
+    tmpRootDir = new Path("/tmp")
+    fileSystem.mkdirs(tmpRootDir)
+  }
+
+  def stopHdfsCluster(): Unit = {
+    if (hdfsCluster != null) hdfsCluster.shutdown(true)
+    if (hadoopConfDir != null) FileUtils.deleteDirectory(hadoopConfDir)
+  }
+
+  private def saveHadoopConf(hadoopConfDir: File): Unit = {
+    val configToWrite = new Configuration(false)
+    val hostName = InetAddress.getLocalHost.getHostName
+    hdfsConf.iterator().asScala.foreach { kv =>
+      val key = kv.getKey
+      val value = kv.getValue.replaceAll(hostName, "localhost")
+      configToWrite.set(key, value)
+    }
+    val file = new File(hadoopConfDir, "core-site.xml")
+    val writer = new FileWriter(file)
+    configToWrite.writeXml(writer)
+    writer.close()
+  }
+
+  def getHadoopConf: Configuration = hdfsConf
+  def getDFSPort: Int = hdfsCluster.getNameNodePort
+  def getHadoopConfDir: String = hadoopConfDir.getAbsolutePath
+  def getHadoopConfFile: Path = new Path(hadoopConfDir.toURI.toURL.toString, "core-site.xml")
+  def getTmpRootDir: Path = tmpRootDir
+  def getFileSystem: FileSystem = fileSystem
+
+  def withTmpHdfsDir(tmpDir: Path => Unit): Unit = {
+    val tempPath = new Path(tmpRootDir, UUID.randomUUID().toString)
+    fileSystem.mkdirs(tempPath)
+    try tmpDir(tempPath)
+    finally fileSystem.delete(tempPath, true)
+  }
+
+}
diff --git a/spark/src/test/scala/org/apache/spark/sql/benchmark/CometReadBenchmark.scala b/spark/src/test/scala/org/apache/spark/sql/benchmark/CometReadBenchmark.scala
index 5f40ab2d74..fcfbf6074f 100644
--- a/spark/src/test/scala/org/apache/spark/sql/benchmark/CometReadBenchmark.scala
+++ b/spark/src/test/scala/org/apache/spark/sql/benchmark/CometReadBenchmark.scala
@@ -24,13 +24,15 @@ import java.io.File
 import scala.collection.JavaConverters._
 import scala.util.Random
 
+import org.apache.hadoop.fs.Path
 import org.apache.spark.TestUtils
 import org.apache.spark.benchmark.Benchmark
+import org.apache.spark.sql.{DataFrame, SparkSession}
 import org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader
 import org.apache.spark.sql.types._
 import org.apache.spark.sql.vectorized.ColumnVector
 
-import org.apache.comet.CometConf
+import org.apache.comet.{CometConf, WithHdfsCluster}
 import org.apache.comet.CometConf.{SCAN_NATIVE_COMET, SCAN_NATIVE_DATAFUSION, SCAN_NATIVE_ICEBERG_COMPAT}
 import org.apache.comet.parquet.BatchReader
 
@@ -40,7 +42,7 @@ import org.apache.comet.parquet.BatchReader
  * benchmark-org.apache.spark.sql.benchmark.CometReadBenchmark` Results will be written to
  * "spark/benchmarks/CometReadBenchmark-**results.txt".
  */
-object CometReadBenchmark extends CometBenchmarkBase {
+class CometReadBaseBenchmark extends CometBenchmarkBase {
 
   def numericScanBenchmark(values: Int, dataType: DataType): Unit = {
     // Benchmarks running through spark sql.
@@ -71,6 +73,7 @@ object CometReadBenchmark extends CometBenchmarkBase {
         sqlBenchmark.addCase("SQL Parquet - Comet Native DataFusion") { _ =>
           withSQLConf(
             CometConf.COMET_ENABLED.key -> "true",
+            CometConf.COMET_EXEC_ENABLED.key -> "true",
             CometConf.COMET_NATIVE_SCAN_IMPL.key -> SCAN_NATIVE_DATAFUSION) {
             spark.sql(s"select $query from parquetV1Table").noop()
           }
@@ -79,6 +82,7 @@ object CometReadBenchmark extends CometBenchmarkBase {
         sqlBenchmark.addCase("SQL Parquet - Comet Native Iceberg Compat") { _ =>
           withSQLConf(
             CometConf.COMET_ENABLED.key -> "true",
+            CometConf.COMET_EXEC_ENABLED.key -> "true",
             CometConf.COMET_NATIVE_SCAN_IMPL.key -> SCAN_NATIVE_ICEBERG_COMPAT) {
             spark.sql(s"select $query from parquetV1Table").noop()
           }
@@ -118,6 +122,7 @@ object CometReadBenchmark extends CometBenchmarkBase {
         sqlBenchmark.addCase("SQL Parquet - Comet Native DataFusion") { _ =>
           withSQLConf(
             CometConf.COMET_ENABLED.key -> "true",
+            CometConf.COMET_EXEC_ENABLED.key -> "true",
             CometConf.COMET_NATIVE_SCAN_IMPL.key -> SCAN_NATIVE_DATAFUSION) {
             spark.sql("select sum(id) from parquetV1Table").noop()
           }
@@ -126,6 +131,7 @@ object CometReadBenchmark extends CometBenchmarkBase {
         sqlBenchmark.addCase("SQL Parquet - Comet Native Iceberg Compat") { _ =>
           withSQLConf(
             CometConf.COMET_ENABLED.key -> "true",
+            CometConf.COMET_EXEC_ENABLED.key -> "true",
             CometConf.COMET_NATIVE_SCAN_IMPL.key -> SCAN_NATIVE_ICEBERG_COMPAT) {
             spark.sql("select sum(id) from parquetV1Table").noop()
           }
@@ -244,6 +250,7 @@ object CometReadBenchmark extends CometBenchmarkBase {
         benchmark.addCase("SQL Parquet - Comet Native DataFusion") { _ =>
           withSQLConf(
             CometConf.COMET_ENABLED.key -> "true",
+            CometConf.COMET_EXEC_ENABLED.key -> "true",
             CometConf.COMET_NATIVE_SCAN_IMPL.key -> SCAN_NATIVE_DATAFUSION) {
             spark.sql("select sum(c2) from parquetV1Table where c1 + 1 > 0").noop()
           }
@@ -252,6 +259,7 @@ object CometReadBenchmark extends CometBenchmarkBase {
         benchmark.addCase("SQL Parquet - Comet Native Iceberg Compat") { _ =>
           withSQLConf(
             CometConf.COMET_ENABLED.key -> "true",
+            CometConf.COMET_EXEC_ENABLED.key -> "true",
             CometConf.COMET_NATIVE_SCAN_IMPL.key -> SCAN_NATIVE_ICEBERG_COMPAT) {
             spark.sql("select sum(c2) from parquetV1Table where c1 + 1 > 0").noop()
           }
@@ -300,6 +308,7 @@ object CometReadBenchmark extends CometBenchmarkBase {
         sqlBenchmark.addCase("SQL Parquet - Comet Native DataFusion") { _ =>
           withSQLConf(
             CometConf.COMET_ENABLED.key -> "true",
+            CometConf.COMET_EXEC_ENABLED.key -> "true",
             CometConf.COMET_NATIVE_SCAN_IMPL.key -> SCAN_NATIVE_DATAFUSION) {
             spark.sql("select sum(length(id)) from parquetV1Table").noop()
           }
@@ -308,6 +317,7 @@ object CometReadBenchmark extends CometBenchmarkBase {
         sqlBenchmark.addCase("SQL Parquet - Comet Native Iceberg Compat") { _ =>
           withSQLConf(
             CometConf.COMET_ENABLED.key -> "true",
+            CometConf.COMET_EXEC_ENABLED.key -> "true",
             CometConf.COMET_NATIVE_SCAN_IMPL.key -> SCAN_NATIVE_ICEBERG_COMPAT) {
             spark.sql("select sum(length(id)) from parquetV1Table").noop()
           }
@@ -352,6 +362,7 @@ object CometReadBenchmark extends CometBenchmarkBase {
         benchmark.addCase("SQL Parquet - Comet Native DataFusion") { _ =>
           withSQLConf(
             CometConf.COMET_ENABLED.key -> "true",
+            CometConf.COMET_EXEC_ENABLED.key -> "true",
             CometConf.COMET_NATIVE_SCAN_IMPL.key -> SCAN_NATIVE_DATAFUSION) {
             spark
               .sql("select sum(length(c2)) from parquetV1Table where c1 is " +
@@ -363,6 +374,7 @@ object CometReadBenchmark extends CometBenchmarkBase {
         benchmark.addCase("SQL Parquet - Comet Native Iceberg Compat") { _ =>
           withSQLConf(
             CometConf.COMET_ENABLED.key -> "true",
+            CometConf.COMET_EXEC_ENABLED.key -> "true",
             CometConf.COMET_NATIVE_SCAN_IMPL.key -> SCAN_NATIVE_ICEBERG_COMPAT) {
             spark
               .sql("select sum(length(c2)) from parquetV1Table where c1 is " +
@@ -403,6 +415,7 @@ object CometReadBenchmark extends CometBenchmarkBase {
         benchmark.addCase("SQL Parquet - Comet Native DataFusion") { _ =>
           withSQLConf(
             CometConf.COMET_ENABLED.key -> "true",
+            CometConf.COMET_EXEC_ENABLED.key -> "true",
             CometConf.COMET_NATIVE_SCAN_IMPL.key -> SCAN_NATIVE_DATAFUSION) {
             spark.sql(s"SELECT sum(c$middle) FROM parquetV1Table").noop()
           }
@@ -411,6 +424,7 @@ object CometReadBenchmark extends CometBenchmarkBase {
         benchmark.addCase("SQL Parquet - Comet Native Iceberg Compat") { _ =>
           withSQLConf(
             CometConf.COMET_ENABLED.key -> "true",
+            CometConf.COMET_EXEC_ENABLED.key -> "true",
             CometConf.COMET_NATIVE_SCAN_IMPL.key -> SCAN_NATIVE_ICEBERG_COMPAT) {
             spark.sql(s"SELECT sum(c$middle) FROM parquetV1Table").noop()
           }
@@ -452,6 +466,7 @@ object CometReadBenchmark extends CometBenchmarkBase {
         benchmark.addCase("SQL Parquet - Comet Native DataFusion") { _ =>
           withSQLConf(
             CometConf.COMET_ENABLED.key -> "true",
+            CometConf.COMET_EXEC_ENABLED.key -> "true",
             CometConf.COMET_NATIVE_SCAN_IMPL.key -> SCAN_NATIVE_DATAFUSION) {
             spark.sql("SELECT * FROM parquetV1Table WHERE c1 + 1 > 0").noop()
           }
@@ -460,6 +475,7 @@ object CometReadBenchmark extends CometBenchmarkBase {
         benchmark.addCase("SQL Parquet - Comet Native Iceberg Compat") { _ =>
           withSQLConf(
             CometConf.COMET_ENABLED.key -> "true",
+            CometConf.COMET_EXEC_ENABLED.key -> "true",
             CometConf.COMET_NATIVE_SCAN_IMPL.key -> SCAN_NATIVE_ICEBERG_COMPAT) {
             spark.sql("SELECT * FROM parquetV1Table WHERE c1 + 1 > 0").noop()
           }
@@ -501,6 +517,7 @@ object CometReadBenchmark extends CometBenchmarkBase {
         benchmark.addCase("SQL Parquet - Comet Native DataFusion") { _ =>
           withSQLConf(
             CometConf.COMET_ENABLED.key -> "true",
+            CometConf.COMET_EXEC_ENABLED.key -> "true",
             CometConf.COMET_NATIVE_SCAN_IMPL.key -> SCAN_NATIVE_DATAFUSION) {
             spark.sql("SELECT * FROM parquetV1Table WHERE c1 + 1 > 0").noop()
           }
@@ -509,6 +526,7 @@ object CometReadBenchmark extends CometBenchmarkBase {
         benchmark.addCase("SQL Parquet - Comet Native Iceberg Compat") { _ =>
           withSQLConf(
             CometConf.COMET_ENABLED.key -> "true",
+            CometConf.COMET_EXEC_ENABLED.key -> "true",
             CometConf.COMET_NATIVE_SCAN_IMPL.key -> SCAN_NATIVE_ICEBERG_COMPAT) {
             spark.sql("SELECT * FROM parquetV1Table WHERE c1 + 1 > 0").noop()
           }
@@ -587,3 +605,51 @@ object CometReadBenchmark extends CometBenchmarkBase {
     }
   }
 }
+
+object CometReadBenchmark extends CometReadBaseBenchmark {}
+
+object CometReadHdfsBenchmark extends CometReadBaseBenchmark with WithHdfsCluster {
+
+  override def getSparkSession: SparkSession = {
+    // start HDFS cluster and add hadoop conf
+    startHdfsCluster()
+    val sparkSession = super.getSparkSession
+    sparkSession.sparkContext.hadoopConfiguration.addResource(getHadoopConfFile)
+    sparkSession
+  }
+
+  override def runCometBenchmark(mainArgs: Array[String]): Unit = {
+    try {
+      super.runCometBenchmark(mainArgs)
+    } finally {
+      stopHdfsCluster()
+    }
+  }
+
+  override def readerBenchmark(values: Int, dataType: DataType): Unit = {
+    // ignore reader benchmark for HDFS
+  }
+
+  // mock local dir to hdfs
+  override protected def withTempPath(f: File => Unit): Unit = {
+    super.withTempPath { dir =>
+      val tempHdfsPath = new Path(getTmpRootDir, dir.getName)
+      getFileSystem.mkdirs(tempHdfsPath)
+      try f(dir)
+      finally getFileSystem.delete(tempHdfsPath, true)
+    }
+  }
+  override protected def prepareTable(
+      dir: File,
+      df: DataFrame,
+      partition: Option[String]): Unit = {
+    val testDf = if (partition.isDefined) {
+      df.write.partitionBy(partition.get)
+    } else {
+      df.write
+    }
+    val tempHdfsPath = getFileSystem.resolvePath(new Path(getTmpRootDir, dir.getName))
+    val parquetV1Path = new Path(tempHdfsPath, "parquetV1")
+    saveAsParquetV1Table(testDf, parquetV1Path.toString)
+  }
+}


Reviews:
user: login: wForget
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-19T08:24:59Z
user: login: wForget
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-19T08:26:49Z
user: login: kazuyukitanimura
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-20T18:39:00Z
user: login: parthchandra
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-20T22:06:03Z
user: login: wForget
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-21T02:26:31Z
user: login: wForget
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-21T02:38:07Z
user: login: wForget
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-21T03:04:01Z
user: login: wForget
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-21T03:05:50Z
user: login: wForget
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-21T03:07:49Z
user: login: kazuyukitanimura
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T17:49:58Z
user: login: parthchandra
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T23:07:07Z
user: login: wForget
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-24T02:11:13Z
user: login: kazuyukitanimura
body: My only last question is whether we should enable native exec for the microbenchmarks
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T18:15:27Z
user: login: wForget
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-25T02:08:25Z
user: login: ariel-miculas
state: COMMENTED
author_association: NONE
submitted_at: 2026-02-20T09:05:05Z
user: login: wForget
state: COMMENTED
author_association: MEMBER
submitted_at: 2026-02-26T01:05:31Z

Review comments:
diff_hunk: @@ -68,7 +68,7 @@ datafusion-comet-proto = { workspace = true }
 object_store = { workspace = true }
 url = { workspace = true }
 parking_lot = "0.12.3"
-datafusion-comet-objectstore-hdfs = { path = "../hdfs", optional = true}
+datafusion-comet-objectstore-hdfs = { path = "../hdfs", optional = true, default-features = false, features = ["hdfs"] }
path: native/core/Cargo.toml
user: login: wForget
body: disable `try_spawn_blocking ` to avoid native thread hanging
created_at: 2025-03-19T08:24:59Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet
+
+import java.io.{File, FileWriter}
+import java.net.InetAddress
+import java.nio.file.Files
+import java.util.UUID
+
+import scala.collection.JavaConverters._
+
+import org.apache.commons.io.FileUtils
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hadoop.hdfs.MiniDFSCluster
+import org.apache.hadoop.hdfs.client.HdfsClientConfigKeys
+import org.apache.spark.internal.Logging
+
+trait WithHdfsCluster extends Logging {
path: spark/src/test/scala/org/apache/comet/WithHdfsCluster.scala
user: login: wForget
body: most copy from [kyuubi](https://github.com/apache/kyuubi/blob/master/kyuubi-server/src/test/scala/org/apache/kyuubi/server/MiniDFSService.scala)
created_at: 2025-03-19T08:26:49Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet
+
+import java.io.{File, FileWriter}
+import java.net.InetAddress
+import java.nio.file.Files
+import java.util.UUID
+
+import scala.collection.JavaConverters._
+
+import org.apache.commons.io.FileUtils
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hadoop.hdfs.MiniDFSCluster
+import org.apache.hadoop.hdfs.client.HdfsClientConfigKeys
+import org.apache.spark.internal.Logging
+
+trait WithHdfsCluster extends Logging {
path: spark/src/test/scala/org/apache/comet/WithHdfsCluster.scala
user: login: kazuyukitanimura
body: Let's leave a comment that this was taken from kyuubi
created_at: 2025-03-20T18:39:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -447,6 +448,13 @@ under the License.
         <version>5.1.0</version>
       </dependency>
 
+      <dependency>
+        <groupId>org.apache.hadoop</groupId>
+        <artifactId>hadoop-client-minicluster</artifactId>
path: pom.xml
user: login: parthchandra
body: Do we need this dependency instead? https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-minicluster/3.3.4
Not sure what the difference is as long as both allow us to spin up a miniDFSCluster
created_at: 2025-03-20T21:59:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -63,6 +65,7 @@ object CometReadBenchmark extends CometBenchmarkBase {
         sqlBenchmark.addCase("SQL Parquet - Comet") { _ =>
           withSQLConf(
             CometConf.COMET_ENABLED.key -> "true",
+            CometConf.COMET_EXEC_ENABLED.key -> "true",
path: spark/src/test/scala/org/apache/spark/sql/benchmark/CometReadBenchmark.scala
user: login: parthchandra
body: Why is this needed? 
created_at: 2025-03-20T22:04:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -447,6 +448,13 @@ under the License.
         <version>5.1.0</version>
       </dependency>
 
+      <dependency>
+        <groupId>org.apache.hadoop</groupId>
+        <artifactId>hadoop-client-minicluster</artifactId>
path: pom.xml
user: login: wForget
body: My understanding is that `hadoop-client-minicluster` has fewer dependencies, and it depends on `hadoop-client-runtime` which is a shaded hadoop client (to avoid introducing conflicts)
created_at: 2025-03-21T02:26:31Z
author_association: MEMBER
diff_hunk: @@ -447,6 +448,13 @@ under the License.
         <version>5.1.0</version>
       </dependency>
 
+      <dependency>
+        <groupId>org.apache.hadoop</groupId>
+        <artifactId>hadoop-client-minicluster</artifactId>
path: pom.xml
user: login: wForget
body: https://github.com/apache/hadoop/blob/trunk/hadoop-client-modules/hadoop-client-minicluster/pom.xml

`hadoop-client-minicluster` seems to be a fat jar of hadoop mini cluster, so is it more suitable as a dependency for testing?
created_at: 2025-03-21T02:38:06Z
author_association: MEMBER
diff_hunk: @@ -63,6 +65,7 @@ object CometReadBenchmark extends CometBenchmarkBase {
         sqlBenchmark.addCase("SQL Parquet - Comet") { _ =>
           withSQLConf(
             CometConf.COMET_ENABLED.key -> "true",
+            CometConf.COMET_EXEC_ENABLED.key -> "true",
path: spark/src/test/scala/org/apache/spark/sql/benchmark/CometReadBenchmark.scala
user: login: wForget
body: CometBenchmarkBase sets COMET_EXEC_ENABLED to false by default, while native scan requires COMET_EXEC_ENABLED=true

https://github.com/apache/datafusion-comet/blob/main/spark/src/test/scala/org/apache/spark/sql/benchmark/CometBenchmarkBase.scala#L53-L54

https://github.com/apache/datafusion-comet/blob/52241085181449e4bdb3345b035bf28fef5b6c34/spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala#L213
created_at: 2025-03-21T03:04:01Z
author_association: MEMBER
diff_hunk: @@ -63,6 +65,7 @@ object CometReadBenchmark extends CometBenchmarkBase {
         sqlBenchmark.addCase("SQL Parquet - Comet") { _ =>
           withSQLConf(
             CometConf.COMET_ENABLED.key -> "true",
+            CometConf.COMET_EXEC_ENABLED.key -> "true",
path: spark/src/test/scala/org/apache/spark/sql/benchmark/CometReadBenchmark.scala
user: login: wForget
body: Line 68 is indeed redundant configuration, I will remove it.
created_at: 2025-03-21T03:05:50Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet
+
+import java.io.{File, FileWriter}
+import java.net.InetAddress
+import java.nio.file.Files
+import java.util.UUID
+
+import scala.collection.JavaConverters._
+
+import org.apache.commons.io.FileUtils
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hadoop.hdfs.MiniDFSCluster
+import org.apache.hadoop.hdfs.client.HdfsClientConfigKeys
+import org.apache.spark.internal.Logging
+
+trait WithHdfsCluster extends Logging {
path: spark/src/test/scala/org/apache/comet/WithHdfsCluster.scala
user: login: wForget
body: Thanks, added
created_at: 2025-03-21T03:07:49Z
author_association: MEMBER
diff_hunk: @@ -58,6 +58,7 @@ under the License.
     <protobuf.version>3.25.5</protobuf.version>
     <parquet.version>1.13.1</parquet.version>
     <parquet.maven.scope>provided</parquet.maven.scope>
+    <hadoop.version>3.3.4</hadoop.version>
path: pom.xml
user: login: kazuyukitanimura
body: Does it make sense to use the same hadoop version as what Spark uses? I.e. should we use maven profile to change versions?
created_at: 2025-03-21T17:49:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -447,6 +448,13 @@ under the License.
         <version>5.1.0</version>
       </dependency>
 
+      <dependency>
+        <groupId>org.apache.hadoop</groupId>
+        <artifactId>hadoop-client-minicluster</artifactId>
path: pom.xml
user: login: parthchandra
body: Ah, I did not know that. It doesn't matter which one we use then.
created_at: 2025-03-21T23:07:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -58,6 +58,7 @@ under the License.
     <protobuf.version>3.25.5</protobuf.version>
     <parquet.version>1.13.1</parquet.version>
     <parquet.maven.scope>provided</parquet.maven.scope>
+    <hadoop.version>3.3.4</hadoop.version>
path: pom.xml
user: login: wForget
body: Just for testing, I think we don't need to specify hadoop version for each spark version, and we introduced a shaded hadoop client which doesn't seem to cause conflicts.
created_at: 2025-03-24T02:11:13Z
author_association: MEMBER
diff_hunk: @@ -71,6 +73,7 @@ object CometReadBenchmark extends CometBenchmarkBase {
         sqlBenchmark.addCase("SQL Parquet - Comet Native DataFusion") { _ =>
           withSQLConf(
             CometConf.COMET_ENABLED.key -> "true",
+            CometConf.COMET_EXEC_ENABLED.key -> "true",
path: spark/src/test/scala/org/apache/spark/sql/benchmark/CometReadBenchmark.scala
user: login: kazuyukitanimura
body: I am not sure if we should enable `COMET_EXEC_ENABLED` as it will mix the scan benchmark and exec benchmark
created_at: 2025-03-24T18:14:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -71,6 +73,7 @@ object CometReadBenchmark extends CometBenchmarkBase {
         sqlBenchmark.addCase("SQL Parquet - Comet Native DataFusion") { _ =>
           withSQLConf(
             CometConf.COMET_ENABLED.key -> "true",
+            CometConf.COMET_EXEC_ENABLED.key -> "true",
path: spark/src/test/scala/org/apache/spark/sql/benchmark/CometReadBenchmark.scala
user: login: wForget
body: > I am not sure if we should enable `COMET_EXEC_ENABLED` as it will mix the scan benchmark and exec benchmark

It seems difficult to benchmark only scan anyway. If we disable exec conversion, it may introduce the performance loss of ColumnarToRow.
created_at: 2025-03-25T02:08:23Z
author_association: MEMBER
diff_hunk: @@ -68,7 +68,7 @@ datafusion-comet-proto = { workspace = true }
 object_store = { workspace = true }
 url = { workspace = true }
 parking_lot = "0.12.3"
-datafusion-comet-objectstore-hdfs = { path = "../hdfs", optional = true}
+datafusion-comet-objectstore-hdfs = { path = "../hdfs", optional = true, default-features = false, features = ["hdfs"] }
path: native/core/Cargo.toml
user: login: ariel-miculas
body: @wForget any idea what causes the native thread to hang when `try_spawn_blocking` is used?
created_at: 2026-02-20T09:05:05Z
author_association: NONE
diff_hunk: @@ -68,7 +68,7 @@ datafusion-comet-proto = { workspace = true }
 object_store = { workspace = true }
 url = { workspace = true }
 parking_lot = "0.12.3"
-datafusion-comet-objectstore-hdfs = { path = "../hdfs", optional = true}
+datafusion-comet-objectstore-hdfs = { path = "../hdfs", optional = true, default-features = false, features = ["hdfs"] }
path: native/core/Cargo.toml
user: login: wForget
body: > [@wForget](https://github.com/wForget) any idea what causes the native thread to hang when `try_spawn_blocking` is used?

Sorry, I didn’t investigate this issue more thoroughly.
created_at: 2026-02-26T01:05:31Z
author_association: MEMBER

PR comments:
user: login: codecov-commenter
created_at: 2025-03-19T09:09:35Z
body: ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/1556?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 58.39%. Comparing base [(`f09f8af`)](https://app.codecov.io/gh/apache/datafusion-comet/commit/f09f8af64c6599255e116a376f4f008f2fd63b43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`cb7d0b2`)](https://app.codecov.io/gh/apache/datafusion-comet/commit/cb7d0b209a0880eeb1c3965fe1f984b7175b4c9d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 96 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff              @@
##               main    #1556      +/-   ##
============================================
+ Coverage     56.12%   58.39%   +2.26%     
- Complexity      976      977       +1     
============================================
  Files           119      122       +3     
  Lines         11743    12217     +474     
  Branches       2251     2280      +29     
============================================
+ Hits           6591     7134     +543     
+ Misses         4012     3951      -61     
+ Partials       1140     1132       -8     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/datafusion-comet/pull/1556?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: kazuyukitanimura
created_at: 2025-03-25T22:08:36Z
body: Merged thanks @wForget 
author_association: CONTRIBUTOR