Repository: apache/datafusion-comet
Pull request #1702: perf: Add memory profiling
Author: login: andygrove
Labels: name: performance
Created: 2025-05-01T13:46:25Z | Merged: 2025-05-05T15:59:55Z

## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->

Closes https://github.com/apache/datafusion-comet/issues/1701

## Rationale for this change

<!--
 Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
 Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
-->

We need a way to profile memory.

## What changes are included in this PR?

<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->

Logging of JVM and native memory utilization on each call to `execute_plan`. After running TPC-H locally with 8GB executor memory and 8GB off-heap memory, I see these stats:

```
JVM_MEMORY: { heapUsed: 2696, heapCommitted: 4394, nonHeapUsed: 142, nonHeapCommitted: 160 }
NATIVE_MEMORY: { resident: 11531.53 }
```

When running with multiple iterations, I see overall memory exceeding 16GB. If this were running in a container environment then I would expect to see the executor be killed due to OOM.

```
JVM_MEMORY: { heapUsed: 4217, heapCommitted: 7440, nonHeapUsed: 145, nonHeapCommitted: 165 }
NATIVE_MEMORY: { resident: 18406.37 }
```

## How are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->


Linked issue #123: Add sqllogictest in ci to verify execution results against vanilla spark sql.
### What is the problem the feature request solves?

[sqllogictest](https://github.com/hydromatic/sql-logic-test) is a commonly used data driven approach to verify query engine.  We can use this to verify the results of comet are same as  vanilla spark sql.

### Describe the potential solution

_No response_

### Additional context

_No response_
Linked issue comments: user: login: liurenjie1024
created_at: 2024-02-27T07:12:07Z
body: I'm happy to take this is we have reached consensus on this.
author_association: NONE
user: login: viirya
created_at: 2024-02-27T07:16:06Z
body: Hmm, I guess the question is, how do you compare with Spark result in sqllogictest?

For example, in current unit test or e2e test like CometTPCDSQuerySuite, we compare Comet query result with Spark result generated in runtime by disabling Comet or golden filed generated by Spark.

But for sqllogictest, as I saw its usage in DataFusion, it works like comparing actual output with previous output (in sqllogictest file) to check output consistency. So it cannot be used to run Spark in runtime to get expected output.
author_association: MEMBER
user: login: liurenjie1024
created_at: 2024-02-27T07:21:57Z
body: >But for sqllogictest, as I saw its usage in DataFusion, it works like comparing actual output with expected output (in sqllogictest file). So it cannot be used to run Spark in runtime to get expected output.

Why? We can use spark connect.
author_association: NONE
user: login: viirya
created_at: 2024-02-27T07:24:10Z
body: > Why? We can use spark connect.

Can sqllogictest run Spark connect? And how do you compare it in sqllogictest?
author_association: MEMBER
user: login: liurenjie1024
created_at: 2024-02-27T07:25:23Z
body: > > Why? We can use spark connect.
> 
> Can sqllogictest run Spark connect?

There are two ways to do this:

1. We can use sqllogictest as a library.
2. We develop a test executor for sqllogictest.

Please note that the link I provided is java's implementation of sqllogictest.
author_association: NONE
user: login: liurenjie1024
created_at: 2024-02-27T07:26:05Z
body: >  And how do you compare it in sqllogictest?

We just run the slt files against both vanilla spark and comet, and they both should be ok.
author_association: NONE
user: login: viirya
created_at: 2024-02-27T07:57:21Z
body: I'm not sure if this makes sense. I tend to think this doesn't make too much sense at this moment.

* It is not something ready to use. As you said, we need to implement a sqllogictest test executor for Spark to talk to Spark connect. I think this makes thing complicated. For the test, we need Spark server, Spark connect client, sqllogictest text executor, sqllogictest... too many components which increase maintenance effort. 
* The slt files need to be run with both Spark and Comet. Although we have unit tests also doing like that, the number is limited.
* We will soon have Spark test pipelines which cover Spark compatibility which leverage existing Spark tests. I think Spark test itself has most correct and highest coverage to make sure Comet's Spark compatibility. If there are very corner cases which are missing in Spark tests, I think these cases should be less. We probably can add the tests directly into Spark or just in Comet as unit test.

author_association: MEMBER
user: login: liurenjie1024
created_at: 2024-02-27T08:07:23Z
body: Sounds reasonable, if we can port spark tests in this repo, it would be enough. Thanks for sharing!
author_association: NONE

Diff:
diff --git a/common/src/main/scala/org/apache/comet/CometConf.scala b/common/src/main/scala/org/apache/comet/CometConf.scala
index 4b4510fc29..652673b9cb 100644
--- a/common/src/main/scala/org/apache/comet/CometConf.scala
+++ b/common/src/main/scala/org/apache/comet/CometConf.scala
@@ -233,6 +233,12 @@ object CometConf extends ShimCometConf {
       defaultValue = true,
       notes = Some("stddev is slower than Spark's implementation"))
 
+  val COMET_MEMORY_PROFILING: ConfigEntry[Boolean] = conf("spark.comet.memory.profiling")
+    .doc("Enable logging of JVM and native memory statistics.")
+    .internal()
+    .booleanConf
+    .createWithDefault(false)
+
   val COMET_MEMORY_OVERHEAD: OptionalConfigEntry[Long] = conf("spark.comet.memoryOverhead")
     .doc(
       "The amount of additional memory to be allocated per executor process for Comet, in MiB, " +
diff --git a/native/Cargo.lock b/native/Cargo.lock
index ce7dc7f7f9..0b3e8f86b8 100644
--- a/native/Cargo.lock
+++ b/native/Cargo.lock
@@ -992,6 +992,7 @@ dependencies = [
  "parquet",
  "paste",
  "pprof",
+ "procfs",
  "prost",
  "rand 0.9.1",
  "regex",
@@ -999,6 +1000,7 @@ dependencies = [
  "snap",
  "tempfile",
  "thiserror 2.0.12",
+ "tikv-jemalloc-ctl",
  "tikv-jemallocator",
  "tokio",
  "url",
@@ -2998,6 +3000,31 @@ dependencies = [
  "unicode-ident",
 ]
 
+[[package]]
+name = "procfs"
+version = "0.17.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cc5b72d8145275d844d4b5f6d4e1eef00c8cd889edb6035c21675d1bb1f45c9f"
+dependencies = [
+ "bitflags 2.9.0",
+ "chrono",
+ "flate2",
+ "hex",
+ "procfs-core",
+ "rustix 0.38.44",
+]
+
+[[package]]
+name = "procfs-core"
+version = "0.17.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "239df02d8349b06fc07398a3a1697b06418223b1c7725085e801e7c0fc6a12ec"
+dependencies = [
+ "bitflags 2.9.0",
+ "chrono",
+ "hex",
+]
+
 [[package]]
 name = "prost"
 version = "0.13.5"
@@ -3863,6 +3890,17 @@ dependencies = [
  "ordered-float",
 ]
 
+[[package]]
+name = "tikv-jemalloc-ctl"
+version = "0.6.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f21f216790c8df74ce3ab25b534e0718da5a1916719771d3fec23315c99e468b"
+dependencies = [
+ "libc",
+ "paste",
+ "tikv-jemalloc-sys",
+]
+
 [[package]]
 name = "tikv-jemalloc-sys"
 version = "0.6.0+5.3.0-1-ge13ca993e8ccb9ba9847cc330696e02839f328f7"
diff --git a/native/core/Cargo.toml b/native/core/Cargo.toml
index e50226935e..fa725cb74e 100644
--- a/native/core/Cargo.toml
+++ b/native/core/Cargo.toml
@@ -40,6 +40,7 @@ parquet = { workspace = true, default-features = false, features = ["experimenta
 futures = { workspace = true }
 mimalloc = { version = "*", default-features = false, optional = true }
 tikv-jemallocator = { version = "0.6.0", optional = true, features = ["disable_initial_exec_tls"] }
+tikv-jemalloc-ctl = { version = "0.6.0", optional = true, features = ["disable_initial_exec_tls", "stats"] }
 tokio = { version = "1", features = ["rt-multi-thread"] }
 async-trait = { workspace = true }
 log = "0.4"
@@ -70,6 +71,9 @@ url = { workspace = true }
 parking_lot = "0.12.3"
 datafusion-comet-objectstore-hdfs = { path = "../hdfs", optional = true, default-features = false, features = ["hdfs"] }
 
+[target.'cfg(target_os = "linux")'.dependencies]
+procfs = "0.17.0"
+
 [dev-dependencies]
 pprof = { version = "0.14.0", features = ["flamegraph"] }
 criterion = { version = "0.5.1", features = ["async_tokio"] }
@@ -82,7 +86,7 @@ datafusion-functions-nested = { version = "47.0.0" }
 [features]
 default = []
 hdfs = ["datafusion-comet-objectstore-hdfs"]
-jemalloc = ["tikv-jemallocator"]
+jemalloc = ["tikv-jemallocator", "tikv-jemalloc-ctl"]
 
 # exclude optional packages from cargo machete verifications
 [package.metadata.cargo-machete]
diff --git a/native/core/src/execution/jni_api.rs b/native/core/src/execution/jni_api.rs
index 2dff70ee2a..3156fc0457 100644
--- a/native/core/src/execution/jni_api.rs
+++ b/native/core/src/execution/jni_api.rs
@@ -68,6 +68,10 @@ use crate::execution::shuffle::{read_ipc_compressed, CompressionCodec};
 use crate::execution::spark_plan::SparkPlan;
 use log::info;
 use once_cell::sync::Lazy;
+#[cfg(target_os = "linux")]
+use procfs::process::Process;
+#[cfg(feature = "jemalloc")]
+use tikv_jemalloc_ctl::{epoch, stats};
 
 static TOKIO_RUNTIME: Lazy<Runtime> = Lazy::new(|| {
     let mut builder = tokio::runtime::Builder::new_multi_thread();
@@ -126,6 +130,8 @@ struct ExecutionContext {
     pub explain_native: bool,
     /// Memory pool config
     pub memory_pool_config: MemoryPoolConfig,
+    /// Whether to log memory usage on each call to execute_plan
+    pub memory_profiling_enabled: bool,
 }
 
 /// Accept serialized query plan and return the address of the native query plan.
@@ -151,6 +157,7 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_createPlan(
     task_attempt_id: jlong,
     debug_native: jboolean,
     explain_native: jboolean,
+    memory_profiling_enabled: jboolean,
 ) -> jlong {
     try_unwrap_or_throw(&e, |mut env| {
         // Init JVM classes
@@ -231,6 +238,7 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_createPlan(
             debug_native: debug_native == 1,
             explain_native: explain_native == 1,
             memory_pool_config,
+            memory_profiling_enabled: memory_profiling_enabled != JNI_FALSE,
         });
 
         Ok(Box::into_raw(exec_context) as i64)
@@ -359,6 +367,41 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_executePlan(
         // Retrieve the query
         let exec_context = get_execution_context(exec_context);
 
+        // memory profiling is only available on linux
+        if exec_context.memory_profiling_enabled {
+            #[cfg(target_os = "linux")]
+            {
+                let pid = std::process::id();
+                let process = Process::new(pid as i32).unwrap();
+                let statm = process.statm().unwrap();
+                let page_size = procfs::page_size();
+                println!(
+                    "NATIVE_MEMORY: {{ resident: {:.0} }}",
+                    (statm.resident * page_size) as f64 / (1024.0 * 1024.0)
+                );
+
+                #[cfg(feature = "jemalloc")]
+                {
+                    // Obtain a MIB for the `epoch`, `stats.allocated`, and
+                    // `atats.resident` keys:
+                    let e = epoch::mib().unwrap();
+                    let allocated = stats::allocated::mib().unwrap();
+                    let resident = stats::resident::mib().unwrap();
+
+                    // Many statistics are cached and only updated
+                    // when the epoch is advanced:
+                    e.advance().unwrap();
+
+                    // Read statistics using MIB key:
+                    let allocated = allocated.read().unwrap() as f64 / (1024.0 * 1024.0);
+                    let resident = resident.read().unwrap() as f64 / (1024.0 * 1024.0);
+                    println!(
+                        "NATIVE_MEMORY_JEMALLOC: {{ allocated: {allocated:.0}, resident: {resident:.0} }}"
+                    );
+                }
+            }
+        }
+
         let exec_context_id = exec_context.id;
 
         // Initialize the execution stream.
diff --git a/spark/src/main/scala/org/apache/comet/CometExecIterator.scala b/spark/src/main/scala/org/apache/comet/CometExecIterator.scala
index c657aa7104..4685722398 100644
--- a/spark/src/main/scala/org/apache/comet/CometExecIterator.scala
+++ b/spark/src/main/scala/org/apache/comet/CometExecIterator.scala
@@ -19,6 +19,8 @@
 
 package org.apache.comet
 
+import java.lang.management.ManagementFactory
+
 import org.apache.spark._
 import org.apache.spark.internal.Logging
 import org.apache.spark.network.util.ByteUnit
@@ -57,6 +59,7 @@ class CometExecIterator(
     extends Iterator[ColumnarBatch]
     with Logging {
 
+  private val memoryProfilingEnabled = CometConf.COMET_MEMORY_PROFILING.get()
   private val nativeLib = new Native()
   private val nativeUtil = new NativeUtil()
   private val cometBatchIterators = inputs.map { iterator =>
@@ -92,7 +95,8 @@ class CometExecIterator(
       memoryLimitPerTask = getMemoryLimitPerTask(conf),
       taskAttemptId = TaskContext.get().taskAttemptId,
       debug = COMET_DEBUG_ENABLED.get(),
-      explain = COMET_EXPLAIN_NATIVE_ENABLED.get())
+      explain = COMET_EXPLAIN_NATIVE_ENABLED.get(),
+      memoryProfilingEnabled)
   }
 
   private var nextBatch: Option[ColumnarBatch] = None
@@ -130,6 +134,22 @@ class CometExecIterator(
   def getNextBatch(): Option[ColumnarBatch] = {
     assert(partitionIndex >= 0 && partitionIndex < numParts)
 
+    if (memoryProfilingEnabled) {
+      val memoryMXBean = ManagementFactory.getMemoryMXBean
+      val heap = memoryMXBean.getHeapMemoryUsage
+      val nonHeap = memoryMXBean.getNonHeapMemoryUsage
+
+      def mb(n: Long) = n / 1024 / 1024
+
+      // scalastyle:off println
+      println(
+        "JVM_MEMORY: { " +
+          s"heapUsed: ${mb(heap.getUsed)}, heapCommitted: ${mb(heap.getCommitted)}, " +
+          s"nonHeapUsed: ${mb(nonHeap.getUsed)}, nonHeapCommitted: ${mb(nonHeap.getCommitted)} " +
+          "}")
+      // scalastyle:on println
+    }
+
     nativeUtil.getNextBatch(
       numOutputCols,
       (arrayAddrs, schemaAddrs) => {
diff --git a/spark/src/main/scala/org/apache/comet/Native.scala b/spark/src/main/scala/org/apache/comet/Native.scala
index ef58e1c9c7..17d92e13d9 100644
--- a/spark/src/main/scala/org/apache/comet/Native.scala
+++ b/spark/src/main/scala/org/apache/comet/Native.scala
@@ -66,7 +66,8 @@ class Native extends NativeBase {
       memoryLimitPerTask: Long,
       taskAttemptId: Long,
       debug: Boolean,
-      explain: Boolean): Long
+      explain: Boolean,
+      memoryProfilingEnabled: Boolean): Long
   // scalastyle:on
 
   /**


Reviews:
user: login: comphead
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T16:13:49Z
user: login: andygrove
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-01T16:18:33Z
user: login: kazuyukitanimura
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T16:40:22Z
user: login: comphead
body: lgtm thanks @andygrove 
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-02T15:10:05Z
user: login: kazuyukitanimura
body: pending ci
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-02T18:48:50Z
user: login: comphead
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-02T21:49:50Z
user: login: comphead
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-02T21:52:24Z
user: login: andygrove
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-03T15:45:42Z

Review comments:
diff_hunk: @@ -359,6 +365,21 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_executePlan(
         // Retrieve the query
         let exec_context = get_execution_context(exec_context);
 
+        // memory profiling is only available on linux
+        if exec_context.memory_profiling_enabled {
+            #[cfg(target_os = "linux")]
path: native/core/src/execution/jni_api.rs
user: login: comphead
body: should the if statement be inside cfg? 
created_at: 2025-05-01T16:13:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -359,6 +365,21 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_executePlan(
         // Retrieve the query
         let exec_context = get_execution_context(exec_context);
 
+        // memory profiling is only available on linux
+        if exec_context.memory_profiling_enabled {
+            #[cfg(target_os = "linux")]
path: native/core/src/execution/jni_api.rs
user: login: andygrove
body: I did that originally, but the compiler complained about the unused variable `memory_profiling_enabled` on non-Linux platforms.
created_at: 2025-05-01T16:18:33Z
author_association: MEMBER
diff_hunk: @@ -130,6 +134,21 @@ class CometExecIterator(
   def getNextBatch(): Option[ColumnarBatch] = {
     assert(partitionIndex >= 0 && partitionIndex < numParts)
 
+    if (memoryProfilingEnabled) {
+      val memoryMXBean = ManagementFactory.getMemoryMXBean
+      val heap = memoryMXBean.getHeapMemoryUsage
+      val nonHeap = memoryMXBean.getNonHeapMemoryUsage
+
+      def mb(n: Long) = n / 1024 / 1024
+
+      // scalastyle:off println
+      println(
+        "JVM_MEMORY: { " +
+          s"heapUsed: ${mb(heap.getUsed)}, heapCommitted: ${mb(heap.getCommitted)}, " +
+          s"nonHeapUsed: ${mb(nonHeap.getUsed)}, nonHeapCommitted: ${mb(nonHeap.getCommitted)} " +
+          "}")
path: spark/src/main/scala/org/apache/comet/CometExecIterator.scala
user: login: kazuyukitanimura
body: Perhaps `logInfo` ?
If println is preferred, we may have to add back `// scalastyle:on println`
created_at: 2025-05-01T16:39:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -359,6 +367,41 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_executePlan(
         // Retrieve the query
         let exec_context = get_execution_context(exec_context);
 
+        // memory profiling is only available on linux
+        if exec_context.memory_profiling_enabled {
+            #[cfg(target_os = "linux")]
+            {
+                let pid = std::process::id();
+                let process = Process::new(pid as i32).unwrap();
+                let statm = process.statm().unwrap();
+                let page_size = procfs::page_size();
+                println!(
+                    "NATIVE_MEMORY: {{ resident: {:.0} }}",
+                    (statm.resident * page_size) as f64 / (1024.0 * 1024.0)
+                );
path: native/core/src/execution/jni_api.rs
user: login: comphead
body: ```suggestion

```
created_at: 2025-05-02T21:49:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -359,6 +367,41 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_executePlan(
         // Retrieve the query
         let exec_context = get_execution_context(exec_context);
 
+        // memory profiling is only available on linux
+        if exec_context.memory_profiling_enabled {
+            #[cfg(target_os = "linux")]
+            {
+                let pid = std::process::id();
+                let process = Process::new(pid as i32).unwrap();
+                let statm = process.statm().unwrap();
+                let page_size = procfs::page_size();
+                println!(
+                    "NATIVE_MEMORY: {{ resident: {:.0} }}",
+                    (statm.resident * page_size) as f64 / (1024.0 * 1024.0)
+                );
+
+                #[cfg(feature = "jemalloc")]
+                {
+                    // Obtain a MIB for the `epoch`, `stats.allocated`, and
+                    // `atats.resident` keys:
+                    let e = epoch::mib().unwrap();
+                    let allocated = stats::allocated::mib().unwrap();
+                    let resident = stats::resident::mib().unwrap();
+
+                    // Many statistics are cached and only updated
+                    // when the epoch is advanced:
+                    e.advance().unwrap();
+
+                    // Read statistics using MIB key:
+                    let allocated = allocated.read().unwrap() as f64 / (1024.0 * 1024.0);
+                    let resident = resident.read().unwrap() as f64 / (1024.0 * 1024.0);
+                    println!(
path: native/core/src/execution/jni_api.rs
user: login: comphead
body: wondering should we feed the metrics to log or to plan metrics
created_at: 2025-05-02T21:52:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -359,6 +367,41 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_executePlan(
         // Retrieve the query
         let exec_context = get_execution_context(exec_context);
 
+        // memory profiling is only available on linux
+        if exec_context.memory_profiling_enabled {
+            #[cfg(target_os = "linux")]
+            {
+                let pid = std::process::id();
+                let process = Process::new(pid as i32).unwrap();
+                let statm = process.statm().unwrap();
+                let page_size = procfs::page_size();
+                println!(
+                    "NATIVE_MEMORY: {{ resident: {:.0} }}",
+                    (statm.resident * page_size) as f64 / (1024.0 * 1024.0)
+                );
+
+                #[cfg(feature = "jemalloc")]
+                {
+                    // Obtain a MIB for the `epoch`, `stats.allocated`, and
+                    // `atats.resident` keys:
+                    let e = epoch::mib().unwrap();
+                    let allocated = stats::allocated::mib().unwrap();
+                    let resident = stats::resident::mib().unwrap();
+
+                    // Many statistics are cached and only updated
+                    // when the epoch is advanced:
+                    e.advance().unwrap();
+
+                    // Read statistics using MIB key:
+                    let allocated = allocated.read().unwrap() as f64 / (1024.0 * 1024.0);
+                    let resident = resident.read().unwrap() as f64 / (1024.0 * 1024.0);
+                    println!(
path: native/core/src/execution/jni_api.rs
user: login: andygrove
body: These metrics are for the overall process, so not specific to any particular plan. This feature just allows us to watch the executor logs and see when memory starts to approach the limit. I hope this will help us debug the cause of OOMs, although it will still primarily be a manual process.
created_at: 2025-05-03T15:45:42Z
author_association: MEMBER

PR comments:
user: login: andygrove
created_at: 2025-05-01T14:25:11Z
body: @mbutrovich I'd like to get your feedback on this PR. 
author_association: MEMBER
user: login: codecov-commenter
created_at: 2025-05-01T15:08:06Z
body: ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/1702?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `37.50000%` with `10 lines` in your changes missing coverage. Please review.
> Project coverage is 58.69%. Comparing base [(`f09f8af`)](https://app.codecov.io/gh/apache/datafusion-comet/commit/f09f8af64c6599255e116a376f4f008f2fd63b43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`077f5af`)](https://app.codecov.io/gh/apache/datafusion-comet/commit/077f5afb72907ca7d08f368165ce1a70fe40dd86?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 177 commits behind head on main.

| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion-comet/pull/1702?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ain/scala/org/apache/comet/CometExecIterator.scala](https://app.codecov.io/gh/apache/datafusion-comet/pull/1702?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fcomet%2FCometExecIterator.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9jb21ldC9Db21ldEV4ZWNJdGVyYXRvci5zY2FsYQ==) | 23.07% | [9 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/1702?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff              @@
##               main    #1702      +/-   ##
============================================
+ Coverage     56.12%   58.69%   +2.56%     
- Complexity      976     1143     +167     
============================================
  Files           119      129      +10     
  Lines         11743    12615     +872     
  Branches       2251     2359     +108     
============================================
+ Hits           6591     7404     +813     
- Misses         4012     4038      +26     
- Partials       1140     1173      +33     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/datafusion-comet/pull/1702?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: andygrove
created_at: 2025-05-02T18:52:02Z
body: Moving to draft while I experiment with jemalloc-specific logging
author_association: MEMBER
user: login: andygrove
created_at: 2025-05-02T18:56:05Z
body: I added some `jemalloc` specific logging:

```
NATIVE_MEMORY_JEMALLOC: { allocated=561605760, resident=3749654528 }
NATIVE_MEMORY_JEMALLOC: { allocated=360603312, resident=3691925504 }
NATIVE_MEMORY_JEMALLOC: { allocated=161601808, resident=3181555712 }
NATIVE_MEMORY_JEMALLOC: { allocated=14660072, resident=3086413824 }
NATIVE_MEMORY_JEMALLOC: { allocated=14271320, resident=3075936256 }
```

@mbutrovich does this seem useful?
author_association: MEMBER
user: login: andygrove
created_at: 2025-05-02T19:00:01Z
body: I'm not sure how to reconcile the different resident numbers:

```
JVM_MEMORY: { heapUsed: 2027, heapCommitted: 3778, nonHeapUsed: 121, nonHeapCommitted: 131 }
NATIVE_MEMORY: { resident: 4702 }
NATIVE_MEMORY_JEMALLOC: { allocated: 13, resident: 2816 }
```

edit: updated to use consistent MB formatting

edit: resident memory is close to heapUsed + jemalloc resident, but is this the correct interpretation?
author_association: MEMBER
user: login: andygrove
created_at: 2025-05-05T15:59:45Z
body: Thanks for the reviews @comphead @kazuyukitanimura and @mbutrovich (offline review). I'm going to go ahead and merge this and then iterate on improving it once I have used it to debug a real issue.
author_association: MEMBER
user: login: mbutrovich
created_at: 2025-05-07T14:18:24Z
body: Not the end of the world, but this introduces warnings on macOS when using jemalloc:
```
| use tikv_jemalloc_ctl::{epoch, stats};
|                         ^^^^^  ^^^^^
|
= note: `#[warn(unused_imports)]` on by default
```
author_association: CONTRIBUTOR