Repository: apache/datafusion
Pull request #15924: fix: overcounting of memory in first/last.
Author: login: ashdnazg
Labels: name: common
description: Related to common crate
name: functions
description: Changes to functions implementation
Created: 2025-05-02T14:09:18Z | Merged: 2025-05-14T20:53:45Z

## Which issue does this PR close?
- Closes #15923.

## Rationale for this change

When aggregating first/last list over a column of lists, the first/last
accumulators hold the necessary scalar value as is, which points to the
list in the original input buffer.

This results in two issues:

1) We prevent the deallocation of the input arrays which might be
significantly larger than the single value we want to hold.

2) During aggreagtion with groups, many accumulators receive slices of the
same input buffer, resulting in all held values pointing to this buffer.
Then, when calculating the size of all accumulators we count the buffer
multiple times, since each accumulator considers it to be part of its own
allocation.


## What changes are included in this PR?

The PR copies/compacts scalar values held by the first/last operators, such that they no longer hold the entire input buffer, thereby solving both 1 & 2.

## Are these changes tested?

Two tests are added.

## Are there any user-facing changes?

There's a new `compact` function, which might or might not be desired to be `pub`. I thought it might be useful somewhere else, but YMMV.

Linked issue #15923: overcounting of memory in first/last.
### Describe the bug

When aggregating first/last list over a column of lists, the first/last
accumulators hold the necessary scalar value as is, which points to the
list in the original input buffer.

This results in two issues:

1) We prevent the deallocation of the input arrays which might be
significantly larger than the single value we want to hold.

2) During aggreagtion with groups, many accumulators receive slices of the
same input buffer, resulting in all held values pointing to this buffer.
Then, when calculating the size of all accumulators we count the buffer
multiple times, since each accumulator considers it to be part of its own
allocation.

While 1 can be tolerated, 2 can easily result in an OOM for a small input with a few thousand groups.

Diff:
diff --git a/datafusion/common/src/scalar/mod.rs b/datafusion/common/src/scalar/mod.rs
index 5c020d1f6398a..e6c9b904353a1 100644
--- a/datafusion/common/src/scalar/mod.rs
+++ b/datafusion/common/src/scalar/mod.rs
@@ -3435,49 +3435,80 @@ impl ScalarValue {
                 .sum::<usize>()
     }
 
-    /// Performs a deep clone of the ScalarValue, creating new copies of all nested data structures.
-    /// This is different from the standard `clone()` which may share data through `Arc`.
-    /// Aggregation functions like `max` will cost a lot of memory if the data is not cloned.
-    pub fn force_clone(&self) -> Self {
+    /// Compacts the allocation referenced by `self` to the minimum, copying the data if
+    /// necessary.
+    ///
+    /// This can be relevant when `self` is a list or contains a list as a nested value, as
+    /// a single list holds an Arc to its entire original array buffer.
+    pub fn compact(&mut self) {
         match self {
-            // Complex types need deep clone of their contents
-            ScalarValue::List(array) => {
-                let array = copy_array_data(&array.to_data());
-                let new_array = ListArray::from(array);
-                ScalarValue::List(Arc::new(new_array))
+            ScalarValue::Null
+            | ScalarValue::Boolean(_)
+            | ScalarValue::Float16(_)
+            | ScalarValue::Float32(_)
+            | ScalarValue::Float64(_)
+            | ScalarValue::Decimal128(_, _, _)
+            | ScalarValue::Decimal256(_, _, _)
+            | ScalarValue::Int8(_)
+            | ScalarValue::Int16(_)
+            | ScalarValue::Int32(_)
+            | ScalarValue::Int64(_)
+            | ScalarValue::UInt8(_)
+            | ScalarValue::UInt16(_)
+            | ScalarValue::UInt32(_)
+            | ScalarValue::UInt64(_)
+            | ScalarValue::Date32(_)
+            | ScalarValue::Date64(_)
+            | ScalarValue::Time32Second(_)
+            | ScalarValue::Time32Millisecond(_)
+            | ScalarValue::Time64Microsecond(_)
+            | ScalarValue::Time64Nanosecond(_)
+            | ScalarValue::IntervalYearMonth(_)
+            | ScalarValue::IntervalDayTime(_)
+            | ScalarValue::IntervalMonthDayNano(_)
+            | ScalarValue::DurationSecond(_)
+            | ScalarValue::DurationMillisecond(_)
+            | ScalarValue::DurationMicrosecond(_)
+            | ScalarValue::DurationNanosecond(_)
+            | ScalarValue::Utf8(_)
+            | ScalarValue::LargeUtf8(_)
+            | ScalarValue::Utf8View(_)
+            | ScalarValue::TimestampSecond(_, _)
+            | ScalarValue::TimestampMillisecond(_, _)
+            | ScalarValue::TimestampMicrosecond(_, _)
+            | ScalarValue::TimestampNanosecond(_, _)
+            | ScalarValue::Binary(_)
+            | ScalarValue::FixedSizeBinary(_, _)
+            | ScalarValue::LargeBinary(_)
+            | ScalarValue::BinaryView(_) => (),
+            ScalarValue::FixedSizeList(arr) => {
+                let array = copy_array_data(&arr.to_data());
+                *Arc::make_mut(arr) = FixedSizeListArray::from(array);
             }
-            ScalarValue::LargeList(array) => {
-                let array = copy_array_data(&array.to_data());
-                let new_array = LargeListArray::from(array);
-                ScalarValue::LargeList(Arc::new(new_array))
+            ScalarValue::List(arr) => {
+                let array = copy_array_data(&arr.to_data());
+                *Arc::make_mut(arr) = ListArray::from(array);
             }
-            ScalarValue::FixedSizeList(arr) => {
+            ScalarValue::LargeList(arr) => {
                 let array = copy_array_data(&arr.to_data());
-                let new_array = FixedSizeListArray::from(array);
-                ScalarValue::FixedSizeList(Arc::new(new_array))
+                *Arc::make_mut(arr) = LargeListArray::from(array)
             }
             ScalarValue::Struct(arr) => {
                 let array = copy_array_data(&arr.to_data());
-                let new_array = StructArray::from(array);
-                ScalarValue::Struct(Arc::new(new_array))
+                *Arc::make_mut(arr) = StructArray::from(array);
             }
             ScalarValue::Map(arr) => {
                 let array = copy_array_data(&arr.to_data());
-                let new_array = MapArray::from(array);
-                ScalarValue::Map(Arc::new(new_array))
-            }
-            ScalarValue::Union(Some((type_id, value)), fields, mode) => {
-                let new_value = Box::new(value.force_clone());
-                ScalarValue::Union(Some((*type_id, new_value)), fields.clone(), *mode)
+                *Arc::make_mut(arr) = MapArray::from(array);
             }
-            ScalarValue::Union(None, fields, mode) => {
-                ScalarValue::Union(None, fields.clone(), *mode)
+            ScalarValue::Union(val, _, _) => {
+                if let Some((_, value)) = val.as_mut() {
+                    value.compact();
+                }
             }
-            ScalarValue::Dictionary(key_type, value) => {
-                let new_value = Box::new(value.force_clone());
-                ScalarValue::Dictionary(key_type.clone(), new_value)
+            ScalarValue::Dictionary(_, value) => {
+                value.compact();
             }
-            _ => self.clone(),
         }
     }
 }
diff --git a/datafusion/functions-aggregate/src/first_last.rs b/datafusion/functions-aggregate/src/first_last.rs
index ec8c440b77e5f..530b4620809be 100644
--- a/datafusion/functions-aggregate/src/first_last.rs
+++ b/datafusion/functions-aggregate/src/first_last.rs
@@ -752,7 +752,7 @@ where
 
     fn size(&self) -> usize {
         self.vals.capacity() * size_of::<T::Native>()
-            + self.null_builder.capacity() / 8 // capacity is in bits, so convert to bytes 
+            + self.null_builder.capacity() / 8 // capacity is in bits, so convert to bytes
             + self.is_sets.capacity() / 8
             + self.size_of_orderings
             + self.min_of_each_group_buf.0.capacity() * size_of::<usize>()
@@ -827,9 +827,14 @@ impl FirstValueAccumulator {
     }
 
     // Updates state with the values in the given row.
-    fn update_with_new_row(&mut self, row: &[ScalarValue]) {
-        self.first = row[0].clone();
-        self.orderings = row[1..].to_vec();
+    fn update_with_new_row(&mut self, mut row: Vec<ScalarValue>) {
+        // Ensure any Array based scalars hold have a single value to reduce memory pressure
+        row.iter_mut().for_each(|s| {
+            s.compact();
+        });
+
+        self.first = row.remove(0);
+        self.orderings = row;
         self.is_set = true;
     }
 
@@ -888,7 +893,7 @@ impl Accumulator for FirstValueAccumulator {
         if !self.is_set {
             if let Some(first_idx) = self.get_first_idx(values)? {
                 let row = get_row_at_idx(values, first_idx)?;
-                self.update_with_new_row(&row);
+                self.update_with_new_row(row);
             }
         } else if !self.requirement_satisfied {
             if let Some(first_idx) = self.get_first_idx(values)? {
@@ -901,7 +906,7 @@ impl Accumulator for FirstValueAccumulator {
                 )?
                 .is_gt()
                 {
-                    self.update_with_new_row(&row);
+                    self.update_with_new_row(row);
                 }
             }
         }
@@ -925,7 +930,7 @@ impl Accumulator for FirstValueAccumulator {
         let min = (0..filtered_states[0].len()).min_by(|&a, &b| comparator.compare(a, b));
 
         if let Some(first_idx) = min {
-            let first_row = get_row_at_idx(&filtered_states, first_idx)?;
+            let mut first_row = get_row_at_idx(&filtered_states, first_idx)?;
             // When collecting orderings, we exclude the is_set flag from the state.
             let first_ordering = &first_row[1..is_set_idx];
             let sort_options = get_sort_options(self.ordering_req.as_ref());
@@ -936,7 +941,9 @@ impl Accumulator for FirstValueAccumulator {
                 // Update with first value in the state. Note that we should exclude the
                 // is_set flag from the state. Otherwise, we will end up with a state
                 // containing two is_set flags.
-                self.update_with_new_row(&first_row[0..is_set_idx]);
+                assert!(is_set_idx <= first_row.len());
+                first_row.resize(is_set_idx, ScalarValue::Null);
+                self.update_with_new_row(first_row);
             }
         }
         Ok(())
@@ -1226,9 +1233,14 @@ impl LastValueAccumulator {
     }
 
     // Updates state with the values in the given row.
-    fn update_with_new_row(&mut self, row: &[ScalarValue]) {
-        self.last = row[0].clone();
-        self.orderings = row[1..].to_vec();
+    fn update_with_new_row(&mut self, mut row: Vec<ScalarValue>) {
+        // Ensure any Array based scalars hold have a single value to reduce memory pressure
+        row.iter_mut().for_each(|s| {
+            s.compact();
+        });
+
+        self.last = row.remove(0);
+        self.orderings = row;
         self.is_set = true;
     }
 
@@ -1289,7 +1301,7 @@ impl Accumulator for LastValueAccumulator {
         if !self.is_set || self.requirement_satisfied {
             if let Some(last_idx) = self.get_last_idx(values)? {
                 let row = get_row_at_idx(values, last_idx)?;
-                self.update_with_new_row(&row);
+                self.update_with_new_row(row);
             }
         } else if let Some(last_idx) = self.get_last_idx(values)? {
             let row = get_row_at_idx(values, last_idx)?;
@@ -1302,7 +1314,7 @@ impl Accumulator for LastValueAccumulator {
             )?
             .is_lt()
             {
-                self.update_with_new_row(&row);
+                self.update_with_new_row(row);
             }
         }
 
@@ -1326,7 +1338,7 @@ impl Accumulator for LastValueAccumulator {
         let max = (0..filtered_states[0].len()).max_by(|&a, &b| comparator.compare(a, b));
 
         if let Some(last_idx) = max {
-            let last_row = get_row_at_idx(&filtered_states, last_idx)?;
+            let mut last_row = get_row_at_idx(&filtered_states, last_idx)?;
             // When collecting orderings, we exclude the is_set flag from the state.
             let last_ordering = &last_row[1..is_set_idx];
             let sort_options = get_sort_options(self.ordering_req.as_ref());
@@ -1339,7 +1351,9 @@ impl Accumulator for LastValueAccumulator {
                 // Update with last value in the state. Note that we should exclude the
                 // is_set flag from the state. Otherwise, we will end up with a state
                 // containing two is_set flags.
-                self.update_with_new_row(&last_row[0..is_set_idx]);
+                assert!(is_set_idx <= last_row.len());
+                last_row.resize(is_set_idx, ScalarValue::Null);
+                self.update_with_new_row(last_row);
             }
         }
         Ok(())
@@ -1382,7 +1396,13 @@ fn convert_to_sort_cols(arrs: &[ArrayRef], sort_exprs: &LexOrdering) -> Vec<Sort
 
 #[cfg(test)]
 mod tests {
-    use arrow::{array::Int64Array, compute::SortOptions, datatypes::Schema};
+    use std::iter::repeat_with;
+
+    use arrow::{
+        array::{Int64Array, ListArray},
+        compute::SortOptions,
+        datatypes::Schema,
+    };
     use datafusion_physical_expr::{expressions::col, PhysicalSortExpr};
 
     use super::*;
@@ -1772,4 +1792,60 @@ mod tests {
 
         Ok(())
     }
+
+    #[test]
+    fn test_first_list_acc_size() -> Result<()> {
+        fn size_after_batch(values: &[ArrayRef]) -> Result<usize> {
+            let mut first_accumulator = FirstValueAccumulator::try_new(
+                &DataType::List(Arc::new(Field::new_list_field(DataType::Int64, false))),
+                &[],
+                LexOrdering::default(),
+                false,
+            )?;
+
+            first_accumulator.update_batch(values)?;
+
+            Ok(first_accumulator.size())
+        }
+
+        let batch1 = ListArray::from_iter_primitive::<Int32Type, _, _>(
+            repeat_with(|| Some(vec![Some(1)])).take(10000),
+        );
+        let batch2 =
+            ListArray::from_iter_primitive::<Int32Type, _, _>([Some(vec![Some(1)])]);
+
+        let size1 = size_after_batch(&[Arc::new(batch1)])?;
+        let size2 = size_after_batch(&[Arc::new(batch2)])?;
+        assert_eq!(size1, size2);
+
+        Ok(())
+    }
+
+    #[test]
+    fn test_last_list_acc_size() -> Result<()> {
+        fn size_after_batch(values: &[ArrayRef]) -> Result<usize> {
+            let mut last_accumulator = LastValueAccumulator::try_new(
+                &DataType::List(Arc::new(Field::new_list_field(DataType::Int64, false))),
+                &[],
+                LexOrdering::default(),
+                false,
+            )?;
+
+            last_accumulator.update_batch(values)?;
+
+            Ok(last_accumulator.size())
+        }
+
+        let batch1 = ListArray::from_iter_primitive::<Int32Type, _, _>(
+            repeat_with(|| Some(vec![Some(1)])).take(10000),
+        );
+        let batch2 =
+            ListArray::from_iter_primitive::<Int32Type, _, _>([Some(vec![Some(1)])]);
+
+        let size1 = size_after_batch(&[Arc::new(batch1)])?;
+        let size2 = size_after_batch(&[Arc::new(batch2)])?;
+        assert_eq!(size1, size2);
+
+        Ok(())
+    }
 }
diff --git a/datafusion/functions-aggregate/src/min_max.rs b/datafusion/functions-aggregate/src/min_max.rs
index af178ed675284..d06483bd079e4 100644
--- a/datafusion/functions-aggregate/src/min_max.rs
+++ b/datafusion/functions-aggregate/src/min_max.rs
@@ -645,19 +645,29 @@ fn min_max_batch_struct(array: &ArrayRef, ordering: Ordering) -> Result<ScalarVa
             }
         }
     }
-    // use force_clone to free array reference
-    Ok(extreme.force_clone())
+
+    Ok(extreme)
 }
 
 macro_rules! min_max_struct {
     ($VALUE:expr, $DELTA:expr, $OP:ident) => {{
         if $VALUE.is_null() {
-            $DELTA.clone()
+            let mut delta_copy = $DELTA.clone();
+            // When the new value won we want to compact it to
+            // avoid storing the entire input
+            delta_copy.compact();
+            delta_copy
         } else if $DELTA.is_null() {
             $VALUE.clone()
         } else {
             match $VALUE.partial_cmp(&$DELTA) {
-                Some(choose_min_max!($OP)) => $DELTA.clone(),
+                Some(choose_min_max!($OP)) => {
+                    // When the new value won we want to compact it to
+                    // avoid storing the entire input
+                    let mut delta_copy = $DELTA.clone();
+                    delta_copy.compact();
+                    delta_copy
+                }
                 _ => $VALUE.clone(),
             }
         }


Reviews:
user: login: ashdnazg
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-02T14:36:22Z
user: login: alamb
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T20:37:26Z
user: login: alamb
body: Thank you @ashdnazg 
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T20:38:46Z
user: login: chenkovsky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T00:02:14Z
user: login: ashdnazg
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T06:52:40Z
user: login: chenkovsky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T07:49:42Z
user: login: ashdnazg
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T08:57:57Z
user: login: chenkovsky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T09:11:15Z
user: login: ashdnazg
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T09:12:02Z
user: login: ashdnazg
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T09:15:12Z
user: login: alamb
body: Thanks @ashdnazg  -- this looks good to me. The only potential issue I see is that the typo in the test -- otherwise it looks good to me. I'll fire off some benchmarks to doubel check
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T15:27:34Z
user: login: eshed-flarion
state: COMMENTED
author_association: NONE
submitted_at: 2025-05-12T15:31:12Z
user: login: alamb
body: Thanks @ashdnazg  -- assuming the performance tests look good I think this one looks good 
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T15:41:07Z

Review comments:
diff_hunk: @@ -3415,6 +3415,100 @@ impl ScalarValue {
                 .map(|sv| sv.size() - size_of_val(sv))
                 .sum::<usize>()
     }
+
+    /// Compacts the allocation referenced by `self` to the minimum, copying the data if
+    /// necessary.
+    ///
+    /// This can be relevant when `self` is a list or contains a list as a nested value, as
+    /// a single list holds an Arc to its entire original array buffer.
+    pub fn compact(&mut self) {
+        match self {
+            ScalarValue::Null
+            | ScalarValue::Boolean(_)
+            | ScalarValue::Float16(_)
+            | ScalarValue::Float32(_)
+            | ScalarValue::Float64(_)
+            | ScalarValue::Decimal128(_, _, _)
+            | ScalarValue::Decimal256(_, _, _)
+            | ScalarValue::Int8(_)
+            | ScalarValue::Int16(_)
+            | ScalarValue::Int32(_)
+            | ScalarValue::Int64(_)
+            | ScalarValue::UInt8(_)
+            | ScalarValue::UInt16(_)
+            | ScalarValue::UInt32(_)
+            | ScalarValue::UInt64(_)
+            | ScalarValue::Date32(_)
+            | ScalarValue::Date64(_)
+            | ScalarValue::Time32Second(_)
+            | ScalarValue::Time32Millisecond(_)
+            | ScalarValue::Time64Microsecond(_)
+            | ScalarValue::Time64Nanosecond(_)
+            | ScalarValue::IntervalYearMonth(_)
+            | ScalarValue::IntervalDayTime(_)
+            | ScalarValue::IntervalMonthDayNano(_)
+            | ScalarValue::DurationSecond(_)
+            | ScalarValue::DurationMillisecond(_)
+            | ScalarValue::DurationMicrosecond(_)
+            | ScalarValue::DurationNanosecond(_)
+            | ScalarValue::Utf8(_)
+            | ScalarValue::LargeUtf8(_)
+            | ScalarValue::Utf8View(_)
+            | ScalarValue::TimestampSecond(_, _)
+            | ScalarValue::TimestampMillisecond(_, _)
+            | ScalarValue::TimestampMicrosecond(_, _)
+            | ScalarValue::TimestampNanosecond(_, _)
+            | ScalarValue::Binary(_)
+            | ScalarValue::FixedSizeBinary(_, _)
+            | ScalarValue::LargeBinary(_)
+            | ScalarValue::BinaryView(_) => (),
+            ScalarValue::FixedSizeList(arr) => {
+                let taken = arrow::compute::take(
+                    arr.as_ref(),
+                    &Int32Array::from_iter_values([0]),
+                    None,
+                )
+                .unwrap();
+                *Arc::make_mut(arr) = cast::as_fixed_size_list_array(&taken).clone();
path: datafusion/common/src/scalar/mod.rs
user: login: ashdnazg
body: It's probably possible to make it more performant, but I don't want to invest into that unless we know it's necessary.
created_at: 2025-05-02T14:36:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -3415,6 +3415,100 @@ impl ScalarValue {
                 .map(|sv| sv.size() - size_of_val(sv))
                 .sum::<usize>()
     }
+
+    /// Compacts the allocation referenced by `self` to the minimum, copying the data if
+    /// necessary.
+    ///
+    /// This can be relevant when `self` is a list or contains a list as a nested value, as
+    /// a single list holds an Arc to its entire original array buffer.
+    pub fn compact(&mut self) {
+        match self {
+            ScalarValue::Null
+            | ScalarValue::Boolean(_)
+            | ScalarValue::Float16(_)
+            | ScalarValue::Float32(_)
+            | ScalarValue::Float64(_)
+            | ScalarValue::Decimal128(_, _, _)
+            | ScalarValue::Decimal256(_, _, _)
+            | ScalarValue::Int8(_)
+            | ScalarValue::Int16(_)
+            | ScalarValue::Int32(_)
+            | ScalarValue::Int64(_)
+            | ScalarValue::UInt8(_)
+            | ScalarValue::UInt16(_)
+            | ScalarValue::UInt32(_)
+            | ScalarValue::UInt64(_)
+            | ScalarValue::Date32(_)
+            | ScalarValue::Date64(_)
+            | ScalarValue::Time32Second(_)
+            | ScalarValue::Time32Millisecond(_)
+            | ScalarValue::Time64Microsecond(_)
+            | ScalarValue::Time64Nanosecond(_)
+            | ScalarValue::IntervalYearMonth(_)
+            | ScalarValue::IntervalDayTime(_)
+            | ScalarValue::IntervalMonthDayNano(_)
+            | ScalarValue::DurationSecond(_)
+            | ScalarValue::DurationMillisecond(_)
+            | ScalarValue::DurationMicrosecond(_)
+            | ScalarValue::DurationNanosecond(_)
+            | ScalarValue::Utf8(_)
+            | ScalarValue::LargeUtf8(_)
+            | ScalarValue::Utf8View(_)
+            | ScalarValue::TimestampSecond(_, _)
+            | ScalarValue::TimestampMillisecond(_, _)
+            | ScalarValue::TimestampMicrosecond(_, _)
+            | ScalarValue::TimestampNanosecond(_, _)
+            | ScalarValue::Binary(_)
+            | ScalarValue::FixedSizeBinary(_, _)
+            | ScalarValue::LargeBinary(_)
+            | ScalarValue::BinaryView(_) => (),
+            ScalarValue::FixedSizeList(arr) => {
+                let taken = arrow::compute::take(
+                    arr.as_ref(),
+                    &Int32Array::from_iter_values([0]),
+                    None,
+                )
+                .unwrap();
+                *Arc::make_mut(arr) = cast::as_fixed_size_list_array(&taken).clone();
path: datafusion/common/src/scalar/mod.rs
user: login: alamb
body: I think you might be able to use the very recently added `ScalarValue::force_copy` that @chenkovsky added in https://github.com/apache/datafusion/pull/15667 (like 10 minutes ago):

https://github.com/apache/datafusion/blob/7e898620461a4580422a63688351325e3eeb7e89/datafusion/common/src/scalar/mod.rs#L3441

It does the same thing for the same reason as I understand
created_at: 2025-05-06T20:37:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -3415,6 +3415,100 @@ impl ScalarValue {
                 .map(|sv| sv.size() - size_of_val(sv))
                 .sum::<usize>()
     }
+
+    /// Compacts the allocation referenced by `self` to the minimum, copying the data if
+    /// necessary.
+    ///
+    /// This can be relevant when `self` is a list or contains a list as a nested value, as
+    /// a single list holds an Arc to its entire original array buffer.
+    pub fn compact(&mut self) {
+        match self {
+            ScalarValue::Null
+            | ScalarValue::Boolean(_)
+            | ScalarValue::Float16(_)
+            | ScalarValue::Float32(_)
+            | ScalarValue::Float64(_)
+            | ScalarValue::Decimal128(_, _, _)
+            | ScalarValue::Decimal256(_, _, _)
+            | ScalarValue::Int8(_)
+            | ScalarValue::Int16(_)
+            | ScalarValue::Int32(_)
+            | ScalarValue::Int64(_)
+            | ScalarValue::UInt8(_)
+            | ScalarValue::UInt16(_)
+            | ScalarValue::UInt32(_)
+            | ScalarValue::UInt64(_)
+            | ScalarValue::Date32(_)
+            | ScalarValue::Date64(_)
+            | ScalarValue::Time32Second(_)
+            | ScalarValue::Time32Millisecond(_)
+            | ScalarValue::Time64Microsecond(_)
+            | ScalarValue::Time64Nanosecond(_)
+            | ScalarValue::IntervalYearMonth(_)
+            | ScalarValue::IntervalDayTime(_)
+            | ScalarValue::IntervalMonthDayNano(_)
+            | ScalarValue::DurationSecond(_)
+            | ScalarValue::DurationMillisecond(_)
+            | ScalarValue::DurationMicrosecond(_)
+            | ScalarValue::DurationNanosecond(_)
+            | ScalarValue::Utf8(_)
+            | ScalarValue::LargeUtf8(_)
+            | ScalarValue::Utf8View(_)
+            | ScalarValue::TimestampSecond(_, _)
+            | ScalarValue::TimestampMillisecond(_, _)
+            | ScalarValue::TimestampMicrosecond(_, _)
+            | ScalarValue::TimestampNanosecond(_, _)
+            | ScalarValue::Binary(_)
+            | ScalarValue::FixedSizeBinary(_, _)
+            | ScalarValue::LargeBinary(_)
+            | ScalarValue::BinaryView(_) => (),
+            ScalarValue::FixedSizeList(arr) => {
+                let taken = arrow::compute::take(
+                    arr.as_ref(),
+                    &Int32Array::from_iter_values([0]),
+                    None,
+                )
+                .unwrap();
+                *Arc::make_mut(arr) = cast::as_fixed_size_list_array(&taken).clone();
path: datafusion/common/src/scalar/mod.rs
user: login: alamb
body: though perhaps this approach is better (modify the existing scalar rather than force a copy 🤔 )

In any event I think we should not have both functions if we can avoid it
created_at: 2025-05-06T20:38:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -3415,6 +3415,100 @@ impl ScalarValue {
                 .map(|sv| sv.size() - size_of_val(sv))
                 .sum::<usize>()
     }
+
+    /// Compacts the allocation referenced by `self` to the minimum, copying the data if
+    /// necessary.
+    ///
+    /// This can be relevant when `self` is a list or contains a list as a nested value, as
+    /// a single list holds an Arc to its entire original array buffer.
+    pub fn compact(&mut self) {
+        match self {
+            ScalarValue::Null
+            | ScalarValue::Boolean(_)
+            | ScalarValue::Float16(_)
+            | ScalarValue::Float32(_)
+            | ScalarValue::Float64(_)
+            | ScalarValue::Decimal128(_, _, _)
+            | ScalarValue::Decimal256(_, _, _)
+            | ScalarValue::Int8(_)
+            | ScalarValue::Int16(_)
+            | ScalarValue::Int32(_)
+            | ScalarValue::Int64(_)
+            | ScalarValue::UInt8(_)
+            | ScalarValue::UInt16(_)
+            | ScalarValue::UInt32(_)
+            | ScalarValue::UInt64(_)
+            | ScalarValue::Date32(_)
+            | ScalarValue::Date64(_)
+            | ScalarValue::Time32Second(_)
+            | ScalarValue::Time32Millisecond(_)
+            | ScalarValue::Time64Microsecond(_)
+            | ScalarValue::Time64Nanosecond(_)
+            | ScalarValue::IntervalYearMonth(_)
+            | ScalarValue::IntervalDayTime(_)
+            | ScalarValue::IntervalMonthDayNano(_)
+            | ScalarValue::DurationSecond(_)
+            | ScalarValue::DurationMillisecond(_)
+            | ScalarValue::DurationMicrosecond(_)
+            | ScalarValue::DurationNanosecond(_)
+            | ScalarValue::Utf8(_)
+            | ScalarValue::LargeUtf8(_)
+            | ScalarValue::Utf8View(_)
+            | ScalarValue::TimestampSecond(_, _)
+            | ScalarValue::TimestampMillisecond(_, _)
+            | ScalarValue::TimestampMicrosecond(_, _)
+            | ScalarValue::TimestampNanosecond(_, _)
+            | ScalarValue::Binary(_)
+            | ScalarValue::FixedSizeBinary(_, _)
+            | ScalarValue::LargeBinary(_)
+            | ScalarValue::BinaryView(_) => (),
+            ScalarValue::FixedSizeList(arr) => {
+                let taken = arrow::compute::take(
+                    arr.as_ref(),
+                    &Int32Array::from_iter_values([0]),
+                    None,
+                )
+                .unwrap();
+                *Arc::make_mut(arr) = cast::as_fixed_size_list_array(&taken).clone();
path: datafusion/common/src/scalar/mod.rs
user: login: chenkovsky
body: @alamb  

in take's comments
```
Note that this kernel, similar to other kernels in this crate,
will avoid allocating where not necessary. Consequently
the returned array may share buffers with the inputs
```
If a large buffer is shared, it's still a problem.
so I didnt choose this method. 
created_at: 2025-05-07T00:02:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -3415,6 +3415,100 @@ impl ScalarValue {
                 .map(|sv| sv.size() - size_of_val(sv))
                 .sum::<usize>()
     }
+
+    /// Compacts the allocation referenced by `self` to the minimum, copying the data if
+    /// necessary.
+    ///
+    /// This can be relevant when `self` is a list or contains a list as a nested value, as
+    /// a single list holds an Arc to its entire original array buffer.
+    pub fn compact(&mut self) {
+        match self {
+            ScalarValue::Null
+            | ScalarValue::Boolean(_)
+            | ScalarValue::Float16(_)
+            | ScalarValue::Float32(_)
+            | ScalarValue::Float64(_)
+            | ScalarValue::Decimal128(_, _, _)
+            | ScalarValue::Decimal256(_, _, _)
+            | ScalarValue::Int8(_)
+            | ScalarValue::Int16(_)
+            | ScalarValue::Int32(_)
+            | ScalarValue::Int64(_)
+            | ScalarValue::UInt8(_)
+            | ScalarValue::UInt16(_)
+            | ScalarValue::UInt32(_)
+            | ScalarValue::UInt64(_)
+            | ScalarValue::Date32(_)
+            | ScalarValue::Date64(_)
+            | ScalarValue::Time32Second(_)
+            | ScalarValue::Time32Millisecond(_)
+            | ScalarValue::Time64Microsecond(_)
+            | ScalarValue::Time64Nanosecond(_)
+            | ScalarValue::IntervalYearMonth(_)
+            | ScalarValue::IntervalDayTime(_)
+            | ScalarValue::IntervalMonthDayNano(_)
+            | ScalarValue::DurationSecond(_)
+            | ScalarValue::DurationMillisecond(_)
+            | ScalarValue::DurationMicrosecond(_)
+            | ScalarValue::DurationNanosecond(_)
+            | ScalarValue::Utf8(_)
+            | ScalarValue::LargeUtf8(_)
+            | ScalarValue::Utf8View(_)
+            | ScalarValue::TimestampSecond(_, _)
+            | ScalarValue::TimestampMillisecond(_, _)
+            | ScalarValue::TimestampMicrosecond(_, _)
+            | ScalarValue::TimestampNanosecond(_, _)
+            | ScalarValue::Binary(_)
+            | ScalarValue::FixedSizeBinary(_, _)
+            | ScalarValue::LargeBinary(_)
+            | ScalarValue::BinaryView(_) => (),
+            ScalarValue::FixedSizeList(arr) => {
+                let taken = arrow::compute::take(
+                    arr.as_ref(),
+                    &Int32Array::from_iter_values([0]),
+                    None,
+                )
+                .unwrap();
+                *Arc::make_mut(arr) = cast::as_fixed_size_list_array(&taken).clone();
path: datafusion/common/src/scalar/mod.rs
user: login: ashdnazg
body: @chenkovsky copying is not the goal, though. If the array you're pointing to is already compact, you don't lose anything by holding a single copy of it instead of multiple.

As far as I understand, there's no reason to force clone all the non-array scalars in min/max and first/last.
created_at: 2025-05-07T06:52:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -3415,6 +3415,100 @@ impl ScalarValue {
                 .map(|sv| sv.size() - size_of_val(sv))
                 .sum::<usize>()
     }
+
+    /// Compacts the allocation referenced by `self` to the minimum, copying the data if
+    /// necessary.
+    ///
+    /// This can be relevant when `self` is a list or contains a list as a nested value, as
+    /// a single list holds an Arc to its entire original array buffer.
+    pub fn compact(&mut self) {
+        match self {
+            ScalarValue::Null
+            | ScalarValue::Boolean(_)
+            | ScalarValue::Float16(_)
+            | ScalarValue::Float32(_)
+            | ScalarValue::Float64(_)
+            | ScalarValue::Decimal128(_, _, _)
+            | ScalarValue::Decimal256(_, _, _)
+            | ScalarValue::Int8(_)
+            | ScalarValue::Int16(_)
+            | ScalarValue::Int32(_)
+            | ScalarValue::Int64(_)
+            | ScalarValue::UInt8(_)
+            | ScalarValue::UInt16(_)
+            | ScalarValue::UInt32(_)
+            | ScalarValue::UInt64(_)
+            | ScalarValue::Date32(_)
+            | ScalarValue::Date64(_)
+            | ScalarValue::Time32Second(_)
+            | ScalarValue::Time32Millisecond(_)
+            | ScalarValue::Time64Microsecond(_)
+            | ScalarValue::Time64Nanosecond(_)
+            | ScalarValue::IntervalYearMonth(_)
+            | ScalarValue::IntervalDayTime(_)
+            | ScalarValue::IntervalMonthDayNano(_)
+            | ScalarValue::DurationSecond(_)
+            | ScalarValue::DurationMillisecond(_)
+            | ScalarValue::DurationMicrosecond(_)
+            | ScalarValue::DurationNanosecond(_)
+            | ScalarValue::Utf8(_)
+            | ScalarValue::LargeUtf8(_)
+            | ScalarValue::Utf8View(_)
+            | ScalarValue::TimestampSecond(_, _)
+            | ScalarValue::TimestampMillisecond(_, _)
+            | ScalarValue::TimestampMicrosecond(_, _)
+            | ScalarValue::TimestampNanosecond(_, _)
+            | ScalarValue::Binary(_)
+            | ScalarValue::FixedSizeBinary(_, _)
+            | ScalarValue::LargeBinary(_)
+            | ScalarValue::BinaryView(_) => (),
+            ScalarValue::FixedSizeList(arr) => {
+                let taken = arrow::compute::take(
+                    arr.as_ref(),
+                    &Int32Array::from_iter_values([0]),
+                    None,
+                )
+                .unwrap();
+                *Arc::make_mut(arr) = cast::as_fixed_size_list_array(&taken).clone();
path: datafusion/common/src/scalar/mod.rs
user: login: chenkovsky
body: @ashdnazg

I mean, If the array is already compact,  according to the documents, `take` may still hold the input buffer. but deep copy will always free this buffer. maybe I'm wrong, I haven't taken a deep dive into `take`, feel free to correct me. 


created_at: 2025-05-07T07:49:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -3415,6 +3415,100 @@ impl ScalarValue {
                 .map(|sv| sv.size() - size_of_val(sv))
                 .sum::<usize>()
     }
+
+    /// Compacts the allocation referenced by `self` to the minimum, copying the data if
+    /// necessary.
+    ///
+    /// This can be relevant when `self` is a list or contains a list as a nested value, as
+    /// a single list holds an Arc to its entire original array buffer.
+    pub fn compact(&mut self) {
+        match self {
+            ScalarValue::Null
+            | ScalarValue::Boolean(_)
+            | ScalarValue::Float16(_)
+            | ScalarValue::Float32(_)
+            | ScalarValue::Float64(_)
+            | ScalarValue::Decimal128(_, _, _)
+            | ScalarValue::Decimal256(_, _, _)
+            | ScalarValue::Int8(_)
+            | ScalarValue::Int16(_)
+            | ScalarValue::Int32(_)
+            | ScalarValue::Int64(_)
+            | ScalarValue::UInt8(_)
+            | ScalarValue::UInt16(_)
+            | ScalarValue::UInt32(_)
+            | ScalarValue::UInt64(_)
+            | ScalarValue::Date32(_)
+            | ScalarValue::Date64(_)
+            | ScalarValue::Time32Second(_)
+            | ScalarValue::Time32Millisecond(_)
+            | ScalarValue::Time64Microsecond(_)
+            | ScalarValue::Time64Nanosecond(_)
+            | ScalarValue::IntervalYearMonth(_)
+            | ScalarValue::IntervalDayTime(_)
+            | ScalarValue::IntervalMonthDayNano(_)
+            | ScalarValue::DurationSecond(_)
+            | ScalarValue::DurationMillisecond(_)
+            | ScalarValue::DurationMicrosecond(_)
+            | ScalarValue::DurationNanosecond(_)
+            | ScalarValue::Utf8(_)
+            | ScalarValue::LargeUtf8(_)
+            | ScalarValue::Utf8View(_)
+            | ScalarValue::TimestampSecond(_, _)
+            | ScalarValue::TimestampMillisecond(_, _)
+            | ScalarValue::TimestampMicrosecond(_, _)
+            | ScalarValue::TimestampNanosecond(_, _)
+            | ScalarValue::Binary(_)
+            | ScalarValue::FixedSizeBinary(_, _)
+            | ScalarValue::LargeBinary(_)
+            | ScalarValue::BinaryView(_) => (),
+            ScalarValue::FixedSizeList(arr) => {
+                let taken = arrow::compute::take(
+                    arr.as_ref(),
+                    &Int32Array::from_iter_values([0]),
+                    None,
+                )
+                .unwrap();
+                *Arc::make_mut(arr) = cast::as_fixed_size_list_array(&taken).clone();
path: datafusion/common/src/scalar/mod.rs
user: login: ashdnazg
body: Definitely, I'm not contending that. I'm just saying there's no point in cloning string/byte vectors etc.
Is your goal to save memory or to not hold the input buffer?
Because if the array is already compact, why is it bad to hold the input buffer?

In any case, I am switching to using your `copy_array_data`, but keeping the in-place nature of compact and not cloning what we don't need.
created_at: 2025-05-07T08:57:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -3415,6 +3415,100 @@ impl ScalarValue {
                 .map(|sv| sv.size() - size_of_val(sv))
                 .sum::<usize>()
     }
+
+    /// Compacts the allocation referenced by `self` to the minimum, copying the data if
+    /// necessary.
+    ///
+    /// This can be relevant when `self` is a list or contains a list as a nested value, as
+    /// a single list holds an Arc to its entire original array buffer.
+    pub fn compact(&mut self) {
+        match self {
+            ScalarValue::Null
+            | ScalarValue::Boolean(_)
+            | ScalarValue::Float16(_)
+            | ScalarValue::Float32(_)
+            | ScalarValue::Float64(_)
+            | ScalarValue::Decimal128(_, _, _)
+            | ScalarValue::Decimal256(_, _, _)
+            | ScalarValue::Int8(_)
+            | ScalarValue::Int16(_)
+            | ScalarValue::Int32(_)
+            | ScalarValue::Int64(_)
+            | ScalarValue::UInt8(_)
+            | ScalarValue::UInt16(_)
+            | ScalarValue::UInt32(_)
+            | ScalarValue::UInt64(_)
+            | ScalarValue::Date32(_)
+            | ScalarValue::Date64(_)
+            | ScalarValue::Time32Second(_)
+            | ScalarValue::Time32Millisecond(_)
+            | ScalarValue::Time64Microsecond(_)
+            | ScalarValue::Time64Nanosecond(_)
+            | ScalarValue::IntervalYearMonth(_)
+            | ScalarValue::IntervalDayTime(_)
+            | ScalarValue::IntervalMonthDayNano(_)
+            | ScalarValue::DurationSecond(_)
+            | ScalarValue::DurationMillisecond(_)
+            | ScalarValue::DurationMicrosecond(_)
+            | ScalarValue::DurationNanosecond(_)
+            | ScalarValue::Utf8(_)
+            | ScalarValue::LargeUtf8(_)
+            | ScalarValue::Utf8View(_)
+            | ScalarValue::TimestampSecond(_, _)
+            | ScalarValue::TimestampMillisecond(_, _)
+            | ScalarValue::TimestampMicrosecond(_, _)
+            | ScalarValue::TimestampNanosecond(_, _)
+            | ScalarValue::Binary(_)
+            | ScalarValue::FixedSizeBinary(_, _)
+            | ScalarValue::LargeBinary(_)
+            | ScalarValue::BinaryView(_) => (),
+            ScalarValue::FixedSizeList(arr) => {
+                let taken = arrow::compute::take(
+                    arr.as_ref(),
+                    &Int32Array::from_iter_values([0]),
+                    None,
+                )
+                .unwrap();
+                *Arc::make_mut(arr) = cast::as_fixed_size_list_array(&taken).clone();
path: datafusion/common/src/scalar/mod.rs
user: login: chenkovsky
body: I want to save memory, because I only need one row. but the buffer contains multiple rows.
created_at: 2025-05-07T09:11:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -645,19 +645,29 @@ fn min_max_batch_struct(array: &ArrayRef, ordering: Ordering) -> Result<ScalarVa
             }
         }
     }
-    // use force_clone to free array reference
-    Ok(extreme.force_clone())
+
+    Ok(extreme)
 }
 
 macro_rules! min_max_struct {
     ($VALUE:expr, $DELTA:expr, $OP:ident) => {{
         if $VALUE.is_null() {
-            $DELTA.clone()
+            let mut delta_copy = $DELTA.clone();
+            // When the new value won we want to compact it to
+            // avoid storing the entire input
+            delta_copy.compact();
path: datafusion/functions-aggregate/src/min_max.rs
user: login: ashdnazg
body: I moved compaction here so it's only done if we're definitely going to keep a reference to the new value.
created_at: 2025-05-07T09:12:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -3415,6 +3415,100 @@ impl ScalarValue {
                 .map(|sv| sv.size() - size_of_val(sv))
                 .sum::<usize>()
     }
+
+    /// Compacts the allocation referenced by `self` to the minimum, copying the data if
+    /// necessary.
+    ///
+    /// This can be relevant when `self` is a list or contains a list as a nested value, as
+    /// a single list holds an Arc to its entire original array buffer.
+    pub fn compact(&mut self) {
+        match self {
+            ScalarValue::Null
+            | ScalarValue::Boolean(_)
+            | ScalarValue::Float16(_)
+            | ScalarValue::Float32(_)
+            | ScalarValue::Float64(_)
+            | ScalarValue::Decimal128(_, _, _)
+            | ScalarValue::Decimal256(_, _, _)
+            | ScalarValue::Int8(_)
+            | ScalarValue::Int16(_)
+            | ScalarValue::Int32(_)
+            | ScalarValue::Int64(_)
+            | ScalarValue::UInt8(_)
+            | ScalarValue::UInt16(_)
+            | ScalarValue::UInt32(_)
+            | ScalarValue::UInt64(_)
+            | ScalarValue::Date32(_)
+            | ScalarValue::Date64(_)
+            | ScalarValue::Time32Second(_)
+            | ScalarValue::Time32Millisecond(_)
+            | ScalarValue::Time64Microsecond(_)
+            | ScalarValue::Time64Nanosecond(_)
+            | ScalarValue::IntervalYearMonth(_)
+            | ScalarValue::IntervalDayTime(_)
+            | ScalarValue::IntervalMonthDayNano(_)
+            | ScalarValue::DurationSecond(_)
+            | ScalarValue::DurationMillisecond(_)
+            | ScalarValue::DurationMicrosecond(_)
+            | ScalarValue::DurationNanosecond(_)
+            | ScalarValue::Utf8(_)
+            | ScalarValue::LargeUtf8(_)
+            | ScalarValue::Utf8View(_)
+            | ScalarValue::TimestampSecond(_, _)
+            | ScalarValue::TimestampMillisecond(_, _)
+            | ScalarValue::TimestampMicrosecond(_, _)
+            | ScalarValue::TimestampNanosecond(_, _)
+            | ScalarValue::Binary(_)
+            | ScalarValue::FixedSizeBinary(_, _)
+            | ScalarValue::LargeBinary(_)
+            | ScalarValue::BinaryView(_) => (),
+            ScalarValue::FixedSizeList(arr) => {
+                let taken = arrow::compute::take(
+                    arr.as_ref(),
+                    &Int32Array::from_iter_values([0]),
+                    None,
+                )
+                .unwrap();
+                *Arc::make_mut(arr) = cast::as_fixed_size_list_array(&taken).clone();
path: datafusion/common/src/scalar/mod.rs
user: login: ashdnazg
body: If the buffer has multiple rows, the scalar is definitely not compact since  the scalar is supposed to be just one row.
If the buffer only has a single row, the scalar might be compact (depending on nested values etc.) and we might not need to copy, as copying will actually waste memory rather than save it.

So the goal is not to copy, the goal is to make sure what you hold is as small as possible.
created_at: 2025-05-07T09:15:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -3435,49 +3435,80 @@ impl ScalarValue {
                 .sum::<usize>()
     }
 
-    /// Performs a deep clone of the ScalarValue, creating new copies of all nested data structures.
-    /// This is different from the standard `clone()` which may share data through `Arc`.
-    /// Aggregation functions like `max` will cost a lot of memory if the data is not cloned.
-    pub fn force_clone(&self) -> Self {
+    /// Compacts the allocation referenced by `self` to the minimum, copying the data if
path: datafusion/common/src/scalar/mod.rs
user: login: alamb
body: 👍 
created_at: 2025-05-12T15:20:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -827,9 +827,13 @@ impl FirstValueAccumulator {
     }
 
     // Updates state with the values in the given row.
-    fn update_with_new_row(&mut self, row: &[ScalarValue]) {
-        self.first = row[0].clone();
-        self.orderings = row[1..].to_vec();
+    fn update_with_new_row(&mut self, mut row: Vec<ScalarValue>) {
+        row.iter_mut().for_each(|s| {
path: datafusion/functions-aggregate/src/first_last.rs
user: login: alamb
body: ```suggestion
        // Ensure any Array based scalars hold have a single value to reduce memory pressure
        row.iter_mut().for_each(|s| {
```
created_at: 2025-05-12T15:21:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1226,9 +1232,13 @@ impl LastValueAccumulator {
     }
 
     // Updates state with the values in the given row.
-    fn update_with_new_row(&mut self, row: &[ScalarValue]) {
-        self.last = row[0].clone();
-        self.orderings = row[1..].to_vec();
+    fn update_with_new_row(&mut self, mut row: Vec<ScalarValue>) {
+        row.iter_mut().for_each(|s| {
path: datafusion/functions-aggregate/src/first_last.rs
user: login: alamb
body: ```suggestion
        // Ensure any Array based scalars hold have a single value to reduce memory pressure
        row.iter_mut().for_each(|s| {
```
created_at: 2025-05-12T15:21:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1772,4 +1790,60 @@ mod tests {
 
         Ok(())
     }
+
+    #[test]
+    fn test_first_list_acc_size() -> Result<()> {
+        fn size_after_batch(values: &[ArrayRef]) -> Result<usize> {
+            let mut first_accumulator = FirstValueAccumulator::try_new(
+                &DataType::List(Arc::new(Field::new_list_field(DataType::Int64, false))),
+                &[],
+                LexOrdering::default(),
+                false,
+            )?;
+
+            first_accumulator.update_batch(values)?;
+
+            Ok(first_accumulator.size())
+        }
+
+        let batch1 = ListArray::from_iter_primitive::<Int32Type, _, _>(
+            repeat_with(|| Some(vec![Some(1)])).take(10000),
+        );
+        let batch2 =
+            ListArray::from_iter_primitive::<Int32Type, _, _>([Some(vec![Some(1)])]);
+
+        let size1 = size_after_batch(&[Arc::new(batch1)])?;
+        let size2 = size_after_batch(&[Arc::new(batch2)])?;
+        assert_eq!(size1, size2);
+
+        Ok(())
+    }
+
+    #[test]
+    fn test_last_list_acc_size() -> Result<()> {
+        fn size_after_batch(values: &[ArrayRef]) -> Result<usize> {
+            let mut last_accumulator = FirstValueAccumulator::try_new(
path: datafusion/functions-aggregate/src/first_last.rs
user: login: alamb
body: I think this is the wrong accumulator:

```suggestion
            let mut last_accumulator = LastValueAccumulator::try_new(
```
created_at: 2025-05-12T15:23:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -645,19 +645,29 @@ fn min_max_batch_struct(array: &ArrayRef, ordering: Ordering) -> Result<ScalarVa
             }
         }
     }
-    // use force_clone to free array reference
-    Ok(extreme.force_clone())
+
+    Ok(extreme)
 }
 
 macro_rules! min_max_struct {
     ($VALUE:expr, $DELTA:expr, $OP:ident) => {{
         if $VALUE.is_null() {
-            $DELTA.clone()
+            let mut delta_copy = $DELTA.clone();
+            // When the new value won we want to compact it to
+            // avoid storing the entire input
+            delta_copy.compact();
path: datafusion/functions-aggregate/src/min_max.rs
user: login: alamb
body: it would be great to try and avoid the duplication in these min/max macros anyways (and instead use an Array)  -- no changes are neede din this PR, I am just observing
created_at: 2025-05-12T15:26:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1772,4 +1790,60 @@ mod tests {
 
         Ok(())
     }
+
+    #[test]
+    fn test_first_list_acc_size() -> Result<()> {
+        fn size_after_batch(values: &[ArrayRef]) -> Result<usize> {
+            let mut first_accumulator = FirstValueAccumulator::try_new(
+                &DataType::List(Arc::new(Field::new_list_field(DataType::Int64, false))),
+                &[],
+                LexOrdering::default(),
+                false,
+            )?;
+
+            first_accumulator.update_batch(values)?;
+
+            Ok(first_accumulator.size())
+        }
+
+        let batch1 = ListArray::from_iter_primitive::<Int32Type, _, _>(
+            repeat_with(|| Some(vec![Some(1)])).take(10000),
+        );
+        let batch2 =
+            ListArray::from_iter_primitive::<Int32Type, _, _>([Some(vec![Some(1)])]);
+
+        let size1 = size_after_batch(&[Arc::new(batch1)])?;
+        let size2 = size_after_batch(&[Arc::new(batch2)])?;
+        assert_eq!(size1, size2);
+
+        Ok(())
+    }
+
+    #[test]
+    fn test_last_list_acc_size() -> Result<()> {
+        fn size_after_batch(values: &[ArrayRef]) -> Result<usize> {
+            let mut last_accumulator = FirstValueAccumulator::try_new(
path: datafusion/functions-aggregate/src/first_last.rs
user: login: eshed-flarion
body: :facepalm: 
created_at: 2025-05-12T15:31:12Z
author_association: NONE

PR comments:
user: login: ashdnazg
created_at: 2025-05-07T09:21:40Z
body: @alamb @chenkovsky 
I pushed a version that uses `copy_array_data` to make sure `take` isn't doing anything funny (looking at the code, at the moment it doesn't, but the docs promise nothing). I also replaced the `force_clone()` calls with `compact()`.

One could improve the `compact` function in the future to avoid copying if it's unnecessary, but it's difficult to do this correctly (you need to go recursively into all arrays, offsets and buffers), so I decided it's probably not worth the effort at the moment.
author_association: CONTRIBUTOR
user: login: alamb
created_at: 2025-05-12T15:28:12Z
body: 🤖 `./gh_compare_branch.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch.sh) Running
Linux aal-dev 6.11.0-1013-gcp #13~24.04.1-Ubuntu SMP Wed Apr  2 16:34:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing first-last (db511504d776f2c5369b9e454b801881d83299f3) to f12ba602e3fcb3238229fccf8e419ec86ca74b1b [diff](https://github.com/apache/datafusion/compare/f12ba602e3fcb3238229fccf8e419ec86ca74b1b..db511504d776f2c5369b9e454b801881d83299f3)
Benchmarks: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

author_association: CONTRIBUTOR
user: login: ashdnazg
created_at: 2025-05-12T15:36:04Z
body: @alamb fixed.
author_association: CONTRIBUTOR
user: login: alamb
created_at: 2025-05-12T16:09:59Z
body: 🤖: Benchmark completed

<details><summary>Details</summary>
<p>


```
Comparing HEAD and first-last
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query        ┃       HEAD ┃ first-last ┃    Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 0     │  1963.07ms │  1949.75ms │ no change │
│ QQuery 1     │   744.70ms │   723.55ms │ no change │
│ QQuery 2     │  1495.89ms │  1434.26ms │ no change │
│ QQuery 3     │   708.87ms │   702.80ms │ no change │
│ QQuery 4     │  1495.21ms │  1475.90ms │ no change │
│ QQuery 5     │ 15444.08ms │ 15369.43ms │ no change │
│ QQuery 6     │  2119.75ms │  2059.58ms │ no change │
│ QQuery 7     │  2118.53ms │  2139.73ms │ no change │
└──────────────┴────────────┴────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary         ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)         │ 26090.11ms │
│ Total Time (first-last)   │ 25855.01ms │
│ Average Time (HEAD)       │  3261.26ms │
│ Average Time (first-last) │  3231.88ms │
│ Queries Faster            │          0 │
│ Queries Slower            │          0 │
│ Queries with No Change    │          8 │
└───────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃       HEAD ┃ first-last ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     2.64ms │     2.67ms │     no change │
│ QQuery 1     │    37.81ms │    39.39ms │     no change │
│ QQuery 2     │    92.74ms │    94.23ms │     no change │
│ QQuery 3     │   100.07ms │   101.29ms │     no change │
│ QQuery 4     │   600.54ms │   588.44ms │     no change │
│ QQuery 5     │   861.94ms │   858.08ms │     no change │
│ QQuery 6     │     2.23ms │     2.29ms │     no change │
│ QQuery 7     │    45.52ms │    44.71ms │     no change │
│ QQuery 8     │   924.83ms │   907.51ms │     no change │
│ QQuery 9     │  1205.60ms │  1181.79ms │     no change │
│ QQuery 10    │   267.26ms │   267.05ms │     no change │
│ QQuery 11    │   306.34ms │   302.54ms │     no change │
│ QQuery 12    │   929.03ms │   913.76ms │     no change │
│ QQuery 13    │  1384.16ms │  1252.06ms │ +1.11x faster │
│ QQuery 14    │   856.10ms │   834.82ms │     no change │
│ QQuery 15    │   821.97ms │   823.12ms │     no change │
│ QQuery 16    │  1756.20ms │  1722.45ms │     no change │
│ QQuery 17    │  1630.31ms │  1599.56ms │     no change │
│ QQuery 18    │  3097.86ms │  3093.14ms │     no change │
│ QQuery 19    │    85.78ms │    86.04ms │     no change │
│ QQuery 20    │  1166.88ms │  1135.45ms │     no change │
│ QQuery 21    │  1356.47ms │  1329.31ms │     no change │
│ QQuery 22    │  2297.91ms │  2213.79ms │     no change │
│ QQuery 23    │  8573.79ms │  8443.98ms │     no change │
│ QQuery 24    │   480.54ms │   476.49ms │     no change │
│ QQuery 25    │   401.48ms │   388.76ms │     no change │
│ QQuery 26    │   552.16ms │   535.47ms │     no change │
│ QQuery 27    │  1629.15ms │  1586.57ms │     no change │
│ QQuery 28    │ 12717.20ms │ 12700.24ms │     no change │
│ QQuery 29    │   538.65ms │   536.55ms │     no change │
│ QQuery 30    │   843.33ms │   828.57ms │     no change │
│ QQuery 31    │   885.04ms │   860.23ms │     no change │
│ QQuery 32    │  2665.26ms │  2709.90ms │     no change │
│ QQuery 33    │  3371.18ms │  3385.10ms │     no change │
│ QQuery 34    │  3418.73ms │  3425.83ms │     no change │
│ QQuery 35    │  1285.15ms │  1300.10ms │     no change │
│ QQuery 36    │   128.09ms │   134.00ms │     no change │
│ QQuery 37    │    60.70ms │    57.72ms │     no change │
│ QQuery 38    │   126.54ms │   129.32ms │     no change │
│ QQuery 39    │   203.13ms │   200.80ms │     no change │
│ QQuery 40    │    47.73ms │    50.11ms │     no change │
│ QQuery 41    │    47.36ms │    48.22ms │     no change │
│ QQuery 42    │    41.02ms │    40.05ms │     no change │
└──────────────┴────────────┴────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary         ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)         │ 57846.45ms │
│ Total Time (first-last)   │ 57231.51ms │
│ Average Time (HEAD)       │  1345.27ms │
│ Average Time (first-last) │  1330.97ms │
│ Queries Faster            │          1 │
│ Queries Slower            │          0 │
│ Queries with No Change    │         42 │
└───────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query        ┃     HEAD ┃ first-last ┃       Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1     │ 123.84ms │   125.41ms │    no change │
│ QQuery 2     │  23.86ms │    23.94ms │    no change │
│ QQuery 3     │  35.92ms │    35.52ms │    no change │
│ QQuery 4     │  20.78ms │    20.97ms │    no change │
│ QQuery 5     │  55.93ms │    57.19ms │    no change │
│ QQuery 6     │  12.48ms │    12.34ms │    no change │
│ QQuery 7     │ 104.35ms │    99.48ms │    no change │
│ QQuery 8     │  26.22ms │    27.74ms │ 1.06x slower │
│ QQuery 9     │  64.07ms │    65.57ms │    no change │
│ QQuery 10    │  56.51ms │    58.68ms │    no change │
│ QQuery 11    │  13.63ms │    13.24ms │    no change │
│ QQuery 12    │  44.21ms │    44.90ms │    no change │
│ QQuery 13    │  30.10ms │    31.23ms │    no change │
│ QQuery 14    │  10.31ms │     9.91ms │    no change │
│ QQuery 15    │  24.78ms │    25.52ms │    no change │
│ QQuery 16    │  22.75ms │    23.48ms │    no change │
│ QQuery 17    │ 100.73ms │   101.60ms │    no change │
│ QQuery 18    │ 229.94ms │   237.21ms │    no change │
│ QQuery 19    │  26.85ms │    27.67ms │    no change │
│ QQuery 20    │  38.48ms │    39.33ms │    no change │
│ QQuery 21    │ 170.36ms │   168.65ms │    no change │
│ QQuery 22    │  16.74ms │    16.72ms │    no change │
└──────────────┴──────────┴────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary         ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)         │ 1252.84ms │
│ Total Time (first-last)   │ 1266.27ms │
│ Average Time (HEAD)       │   56.95ms │
│ Average Time (first-last) │   57.56ms │
│ Queries Faster            │         0 │
│ Queries Slower            │         1 │
│ Queries with No Change    │        21 │
└───────────────────────────┴───────────┘
```


</p>
</details>


author_association: CONTRIBUTOR
user: login: alamb
created_at: 2025-05-14T20:53:47Z
body: Thanks again @ashdnazg 
author_association: CONTRIBUTOR