Repository: apache/datafusion
Pull request #15160: Change `flatten` so it does only a level, not recursively
Author: login: velaskuez
Labels: name: api change
description: Changes the API exposed to users of the crate
name: sqllogictest
description: SQL Logic Tests (.slt)
Created: 2025-03-11T14:45:51Z | Merged: 2025-04-17T16:54:02Z

## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->

- Closes #13757

## Rationale for this change
Parity with the `flatten` implementation in duckdb.
<!--
 Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
 Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
-->

## What changes are included in this PR?

<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
Remove the recursion in `flatten_internal` so that only the top level elements are flattened.

## Are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
Existing sqllogictests have been updated.

## Are there any user-facing changes?
Yes, `flatten` no longer recursively flattens the array
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->


Linked issue #123: Implement physical plan for EXISTS subquery
*Note*: migrated from original JIRA: https://issues.apache.org/jira/browse/ARROW-10819

The TPC-H queries include use of the EXISTS which is used to test for the existence of any record in a subquery. For example:

```sql
and *exists* (
    select
        *
    from
        lineitem
    where
        l_orderkey = o_orderkey
        and l_commitdate < l_receiptdate
)
```
Linked issue comments: user: login: alamb
created_at: 2021-04-26T13:23:43Z
body: Comment from Andy Grove(andygrove) @ 2020-12-31T19:42:45.132+0000:
<pre>The example given here is a correlated subquery that can be translated into a join.

Here is a random stackoverflow discussion on this for reference (I have not reviewed it)

https://stackoverflow.com/questions/1772609/procedurally-transform-subquery-into-join</pre>
author_association: CONTRIBUTOR
user: login: alamb
created_at: 2022-10-21T19:21:30Z
body: In case anyone is curious -- we support correlated versions of these queries (via a join) but if there is no correlation (not super useful) we do not

```sql
❯ create table foo as select * from (values (1), (2), (NULL)) as sql
;
0 rows in set. Query took 0.022 seconds.
3 rows in set. Query took 0.007 seconds.
❯ create table bar as select * from (values (1), (2), (NULL)) as sql;
0 rows in set. Query took 0.000 seconds.
❯ select * from foo where exists (select column1 from bar);
NotImplemented("Physical plan does not support logical expression EXISTS (<subquery>)")
❯ select * from foo where exists (select column1 from bar where foo.column1 = bar.column1);
+---------+
| column1 |
+---------+
| 2       |
| 1       |
+---------+
```
author_association: CONTRIBUTOR
user: login: logan-keede
created_at: 2025-02-09T19:53:56Z
body: 
> In case anyone is curious -- we support correlated versions of these queries (via a join) but if there is no correlation (not super useful) we do not
> 
> ❯ create table foo as select * from (values (1), (2), (NULL)) as sql
> ;
> 0 rows in set. Query took 0.022 seconds.
> 3 rows in set. Query took 0.007 seconds.
> ❯ create table bar as select * from (values (1), (2), (NULL)) as sql;
> 0 rows in set. Query took 0.000 seconds.
> ❯ select * from foo where exists (select column1 from bar);
> NotImplemented("Physical plan does not support logical expression EXISTS (<subquery>)")
> ❯ select * from foo where exists (select column1 from bar where foo.column1 = bar.column1);
> +---------+
> | column1 |
> +---------+
> | 2       |
> | 1       |
> +---------+

```sql

> explain select * from foo where exists (select column1 from bar);
+---------------+-----------------------------------------------------+
| plan_type     | plan                                                |
+---------------+-----------------------------------------------------+
| logical_plan  | LeftSemi Join:                                      |
|               |   TableScan: foo projection=[column1]               |
|               |   SubqueryAlias: __correlated_sq_1                  |
|               |     TableScan: bar projection=[]                    |
| physical_plan | NestedLoopJoinExec: join_type=RightSemi             |
|               |   DataSourceExec: partitions=1, partition_sizes=[1] |
|               |   DataSourceExec: partitions=1, partition_sizes=[1] |
|               |                                                     |
+---------------+-----------------------------------------------------+
2 row(s) fetched. 
Elapsed 0.007 seconds.

> select * from foo where exists (select column1 from bar);
+---------+
| column1 |
+---------+
| 1       |
| 2       |
| NULL    |
+---------+
3 row(s) fetched. 
Elapsed 0.006 seconds.
```


I get this on `datafusion 45.0.0`, perhaps this issue can be closed or at least updated if there is a case where we still don't support EXISTS in physical plan.

author_association: CONTRIBUTOR
user: login: alamb
created_at: 2025-02-10T15:04:31Z
body: Looks good -- thanks for checking @logan-keede  -- we can open a new issue if we find another hole
author_association: CONTRIBUTOR

Diff:
diff --git a/datafusion/functions-nested/src/flatten.rs b/datafusion/functions-nested/src/flatten.rs
index f288035948dcb..4279f04e3dc44 100644
--- a/datafusion/functions-nested/src/flatten.rs
+++ b/datafusion/functions-nested/src/flatten.rs
@@ -18,19 +18,18 @@
 //! [`ScalarUDFImpl`] definitions for flatten function.
 
 use crate::utils::make_scalar_function;
-use arrow::array::{ArrayRef, GenericListArray, OffsetSizeTrait};
+use arrow::array::{Array, ArrayRef, GenericListArray, OffsetSizeTrait};
 use arrow::buffer::OffsetBuffer;
 use arrow::datatypes::{
     DataType,
     DataType::{FixedSizeList, LargeList, List, Null},
 };
-use datafusion_common::cast::{
-    as_generic_list_array, as_large_list_array, as_list_array,
-};
+use datafusion_common::cast::{as_large_list_array, as_list_array};
+use datafusion_common::utils::ListCoercion;
 use datafusion_common::{exec_err, utils::take_function_args, Result};
 use datafusion_expr::{
-    ArrayFunctionSignature, ColumnarValue, Documentation, ScalarUDFImpl, Signature,
-    TypeSignature, Volatility,
+    ArrayFunctionArgument, ArrayFunctionSignature, ColumnarValue, Documentation,
+    ScalarUDFImpl, Signature, TypeSignature, Volatility,
 };
 use datafusion_macros::user_doc;
 use std::any::Any;
@@ -77,9 +76,11 @@ impl Flatten {
     pub fn new() -> Self {
         Self {
             signature: Signature {
-                // TODO (https://github.com/apache/datafusion/issues/13757) flatten should be single-step, not recursive
                 type_signature: TypeSignature::ArraySignature(
-                    ArrayFunctionSignature::RecursiveArray,
+                    ArrayFunctionSignature::Array {
+                        arguments: vec![ArrayFunctionArgument::Array],
+                        array_coercion: Some(ListCoercion::FixedSizedListToList),
+                    },
                 ),
                 volatility: Volatility::Immutable,
             },
@@ -102,25 +103,23 @@ impl ScalarUDFImpl for Flatten {
     }
 
     fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
-        fn get_base_type(data_type: &DataType) -> Result<DataType> {
-            match data_type {
-                List(field) | FixedSizeList(field, _)
-                    if matches!(field.data_type(), List(_) | FixedSizeList(_, _)) =>
-                {
-                    get_base_type(field.data_type())
-                }
-                LargeList(field) if matches!(field.data_type(), LargeList(_)) => {
-                    get_base_type(field.data_type())
+        let data_type = match &arg_types[0] {
+            List(field) | FixedSizeList(field, _) => match field.data_type() {
+                List(field) | FixedSizeList(field, _) => List(Arc::clone(field)),
+                _ => arg_types[0].clone(),
+            },
+            LargeList(field) => match field.data_type() {
+                List(field) | LargeList(field) | FixedSizeList(field, _) => {
+                    LargeList(Arc::clone(field))
                 }
-                Null | List(_) | LargeList(_) => Ok(data_type.to_owned()),
-                FixedSizeList(field, _) => Ok(List(Arc::clone(field))),
-                _ => exec_err!(
-                    "Not reachable, data_type should be List, LargeList or FixedSizeList"
-                ),
-            }
-        }
+                _ => arg_types[0].clone(),
+            },
+            Null => Null,
+            _ => exec_err!(
+                "Not reachable, data_type should be List, LargeList or FixedSizeList"
+            )?,
+        };
 
-        let data_type = get_base_type(&arg_types[0])?;
         Ok(data_type)
     }
 
@@ -146,14 +145,62 @@ pub fn flatten_inner(args: &[ArrayRef]) -> Result<ArrayRef> {
 
     match array.data_type() {
         List(_) => {
-            let list_arr = as_list_array(&array)?;
-            let flattened_array = flatten_internal::<i32>(list_arr.clone(), None)?;
-            Ok(Arc::new(flattened_array) as ArrayRef)
+            let (field, offsets, values, nulls) =
+                as_list_array(&array)?.clone().into_parts();
+
+            match field.data_type() {
+                List(_) => {
+                    let (inner_field, inner_offsets, inner_values, _) =
+                        as_list_array(&values)?.clone().into_parts();
+                    let offsets = get_offsets_for_flatten::<i32>(inner_offsets, offsets);
+                    let flattened_array = GenericListArray::<i32>::new(
+                        inner_field,
+                        offsets,
+                        inner_values,
+                        nulls,
+                    );
+
+                    Ok(Arc::new(flattened_array) as ArrayRef)
+                }
+                LargeList(_) => {
+                    exec_err!("flatten does not support type '{:?}'", array.data_type())?
+                }
+                _ => Ok(Arc::clone(array) as ArrayRef),
+            }
         }
         LargeList(_) => {
-            let list_arr = as_large_list_array(&array)?;
-            let flattened_array = flatten_internal::<i64>(list_arr.clone(), None)?;
-            Ok(Arc::new(flattened_array) as ArrayRef)
+            let (field, offsets, values, nulls) =
+                as_large_list_array(&array)?.clone().into_parts();
+
+            match field.data_type() {
+                List(_) => {
+                    let (inner_field, inner_offsets, inner_values, _) =
+                        as_list_array(&values)?.clone().into_parts();
+                    let offsets = get_large_offsets_for_flatten(inner_offsets, offsets);
+                    let flattened_array = GenericListArray::<i64>::new(
+                        inner_field,
+                        offsets,
+                        inner_values,
+                        nulls,
+                    );
+
+                    Ok(Arc::new(flattened_array) as ArrayRef)
+                }
+                LargeList(_) => {
+                    let (inner_field, inner_offsets, inner_values, nulls) =
+                        as_large_list_array(&values)?.clone().into_parts();
+                    let offsets = get_offsets_for_flatten::<i64>(inner_offsets, offsets);
+                    let flattened_array = GenericListArray::<i64>::new(
+                        inner_field,
+                        offsets,
+                        inner_values,
+                        nulls,
+                    );
+
+                    Ok(Arc::new(flattened_array) as ArrayRef)
+                }
+                _ => Ok(Arc::clone(array) as ArrayRef),
+            }
         }
         Null => Ok(Arc::clone(array)),
         _ => {
@@ -162,37 +209,6 @@ pub fn flatten_inner(args: &[ArrayRef]) -> Result<ArrayRef> {
     }
 }
 
-fn flatten_internal<O: OffsetSizeTrait>(
-    list_arr: GenericListArray<O>,
-    indexes: Option<OffsetBuffer<O>>,
-) -> Result<GenericListArray<O>> {
-    let (field, offsets, values, _) = list_arr.clone().into_parts();
-    let data_type = field.data_type();
-
-    match data_type {
-        // Recursively get the base offsets for flattened array
-        List(_) | LargeList(_) => {
-            let sub_list = as_generic_list_array::<O>(&values)?;
-            if let Some(indexes) = indexes {
-                let offsets = get_offsets_for_flatten(offsets, indexes);
-                flatten_internal::<O>(sub_list.clone(), Some(offsets))
-            } else {
-                flatten_internal::<O>(sub_list.clone(), Some(offsets))
-            }
-        }
-        // Reach the base level, create a new list array
-        _ => {
-            if let Some(indexes) = indexes {
-                let offsets = get_offsets_for_flatten(offsets, indexes);
-                let list_arr = GenericListArray::<O>::new(field, offsets, values, None);
-                Ok(list_arr)
-            } else {
-                Ok(list_arr)
-            }
-        }
-    }
-}
-
 // Create new offsets that are equivalent to `flatten` the array.
 fn get_offsets_for_flatten<O: OffsetSizeTrait>(
     offsets: OffsetBuffer<O>,
@@ -205,3 +221,16 @@ fn get_offsets_for_flatten<O: OffsetSizeTrait>(
         .collect();
     OffsetBuffer::new(offsets.into())
 }
+
+// Create new large offsets that are equivalent to `flatten` the array.
+fn get_large_offsets_for_flatten<O: OffsetSizeTrait, P: OffsetSizeTrait>(
+    offsets: OffsetBuffer<O>,
+    indexes: OffsetBuffer<P>,
+) -> OffsetBuffer<i64> {
+    let buffer = offsets.into_inner();
+    let offsets: Vec<i64> = indexes
+        .iter()
+        .map(|i| buffer[i.to_usize().unwrap()].to_i64().unwrap())
+        .collect();
+    OffsetBuffer::new(offsets.into())
+}
diff --git a/datafusion/sqllogictest/test_files/array.slt b/datafusion/sqllogictest/test_files/array.slt
index 3b7f129606818..39e5d5c1fbb9a 100644
--- a/datafusion/sqllogictest/test_files/array.slt
+++ b/datafusion/sqllogictest/test_files/array.slt
@@ -7090,12 +7090,10 @@ select array_concat(column1, [7]) from arrays_values_v2;
 
 # flatten
 
-#TODO: https://github.com/apache/datafusion/issues/7142
-# follow DuckDB
-#query ?
-#select flatten(NULL);
-#----
-#NULL
+query ?
+select flatten(NULL);
+----
+NULL
 
 # flatten with scalar values #1
 query ???
@@ -7103,21 +7101,21 @@ select flatten(make_array(1, 2, 1, 3, 2)),
        flatten(make_array([1], [2, 3], [null], make_array(4, null, 5))),
        flatten(make_array([[1.1]], [[2.2]], [[3.3], [4.4]]));
 ----
-[1, 2, 1, 3, 2] [1, 2, 3, NULL, 4, NULL, 5] [1.1, 2.2, 3.3, 4.4]
+[1, 2, 1, 3, 2] [1, 2, 3, NULL, 4, NULL, 5] [[1.1], [2.2], [3.3], [4.4]]
 
 query ???
 select flatten(arrow_cast(make_array(1, 2, 1, 3, 2), 'LargeList(Int64)')),
        flatten(arrow_cast(make_array([1], [2, 3], [null], make_array(4, null, 5)), 'LargeList(LargeList(Int64))')),
        flatten(arrow_cast(make_array([[1.1]], [[2.2]], [[3.3], [4.4]]), 'LargeList(LargeList(LargeList(Float64)))'));
 ----
-[1, 2, 1, 3, 2] [1, 2, 3, NULL, 4, NULL, 5] [1.1, 2.2, 3.3, 4.4]
+[1, 2, 1, 3, 2] [1, 2, 3, NULL, 4, NULL, 5] [[1.1], [2.2], [3.3], [4.4]]
 
 query ???
 select flatten(arrow_cast(make_array(1, 2, 1, 3, 2), 'FixedSizeList(5, Int64)')),
        flatten(arrow_cast(make_array([1], [2, 3], [null], make_array(4, null, 5)), 'FixedSizeList(4, List(Int64))')),
        flatten(arrow_cast(make_array([[1.1], [2.2]], [[3.3], [4.4]]), 'FixedSizeList(2, List(List(Float64)))'));
 ----
-[1, 2, 1, 3, 2] [1, 2, 3, NULL, 4, NULL, 5] [1.1, 2.2, 3.3, 4.4]
+[1, 2, 1, 3, 2] [1, 2, 3, NULL, 4, NULL, 5] [[1.1], [2.2], [3.3], [4.4]]
 
 # flatten with column values
 query ????
@@ -7127,8 +7125,8 @@ select flatten(column1),
        flatten(column4)
 from flatten_table;
 ----
-[1, 2, 3] [1, 2, 3, 4, 5, 6] [1, 2, 3] [1.0, 2.1, 2.2, 3.2, 3.3, 3.4]
-[1, 2, 3, 4, 5, 6] [8] [1, 2, 3] [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]
+[1, 2, 3] [[1, 2, 3], [4, 5], [6]] [[[1]], [[2, 3]]] [1.0, 2.1, 2.2, 3.2, 3.3, 3.4]
+[1, 2, 3, 4, 5, 6] [[8]] [[[1, 2]], [[3]]] [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]
 
 query ????
 select flatten(column1),
@@ -7137,8 +7135,8 @@ select flatten(column1),
        flatten(column4)
 from large_flatten_table;
 ----
-[1, 2, 3] [1, 2, 3, 4, 5, 6] [1, 2, 3] [1.0, 2.1, 2.2, 3.2, 3.3, 3.4]
-[1, 2, 3, 4, 5, 6] [8] [1, 2, 3] [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]
+[1, 2, 3] [[1, 2, 3], [4, 5], [6]] [[[1]], [[2, 3]]] [1.0, 2.1, 2.2, 3.2, 3.3, 3.4]
+[1, 2, 3, 4, 5, 6] [[8]] [[[1, 2]], [[3]]] [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]
 
 query ????
 select flatten(column1),
@@ -7147,8 +7145,19 @@ select flatten(column1),
        flatten(column4)
 from fixed_size_flatten_table;
 ----
-[1, 2, 3] [1, 2, 3, 4, 5, 6] [1, 2, 3] [1.0, 2.1, 2.2, 3.2, 3.3, 3.4]
-[1, 2, 3, 4, 5, 6] [8, 9, 10, 11, 12, 13] [1, 2, 3] [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]
+[1, 2, 3] [[1, 2, 3], [4, 5], [6]] [[[1]], [[2, 3]]] [1.0, 2.1, 2.2, 3.2, 3.3, 3.4]
+[1, 2, 3, 4, 5, 6] [[8], [9, 10], [11, 12, 13]] [[[1, 2]], [[3]]] [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]
+
+# flatten with different inner list type
+query ??????
+select flatten(arrow_cast(make_array([1, 2], [3, 4]), 'List(FixedSizeList(2, Int64))')),
+       flatten(arrow_cast(make_array([[1, 2]], [[3, 4]]), 'List(FixedSizeList(1, List(Int64)))')),
+       flatten(arrow_cast(make_array([1, 2], [3, 4]), 'LargeList(List(Int64))')),
+       flatten(arrow_cast(make_array([[1, 2]], [[3, 4]]), 'LargeList(List(List(Int64)))')),
+       flatten(arrow_cast(make_array([1, 2], [3, 4]), 'LargeList(FixedSizeList(2, Int64))')),
+       flatten(arrow_cast(make_array([[1, 2]], [[3, 4]]), 'LargeList(FixedSizeList(1, List(Int64)))'))
+----
+[1, 2, 3, 4] [[1, 2], [3, 4]] [1, 2, 3, 4] [[1, 2], [3, 4]] [1, 2, 3, 4] [[1, 2], [3, 4]]
 
 ## empty (aliases: `array_empty`, `list_empty`)
 # empty scalar function #1


Reviews:
user: login: Weijun-H
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-12T05:03:28Z
user: login: velaskuez
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T14:36:14Z
user: login: findepi
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-12T15:50:16Z
user: login: velaskuez
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T12:54:06Z
user: login: Weijun-H
body: Thanks @delamarch3 👍  , I think the pr looks good. Waiting for other comments.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-10T02:14:25Z
user: login: cht42
body: Hey @delamarch3 , noticed that null handling is not managed properly in the current implementation. Might as well fix it here directly
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-11T14:07:46Z

Review comments:
diff_hunk: @@ -102,25 +101,21 @@ impl ScalarUDFImpl for Flatten {
     }
 
     fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
-        fn get_base_type(data_type: &DataType) -> Result<DataType> {
-            match data_type {
-                List(field) | FixedSizeList(field, _)
-                    if matches!(field.data_type(), List(_) | FixedSizeList(_, _)) =>
-                {
-                    get_base_type(field.data_type())
-                }
-                LargeList(field) if matches!(field.data_type(), LargeList(_)) => {
-                    get_base_type(field.data_type())
-                }
-                Null | List(_) | LargeList(_) => Ok(data_type.to_owned()),
-                FixedSizeList(field, _) => Ok(List(Arc::clone(field))),
-                _ => exec_err!(
-                    "Not reachable, data_type should be List, LargeList or FixedSizeList"
-                ),
-            }
-        }
+        let data_type = match &arg_types[0] {
+            List(field) | FixedSizeList(field, _) => match field.data_type() {
+                List(field) | FixedSizeList(field, _) => List(Arc::clone(field)),
+                _ => arg_types[0].clone(),
+            },
+            LargeList(field) => match field.data_type() {
+                LargeList(field) => LargeList(Arc::clone(field)),
path: datafusion/functions-nested/src/flatten.rs
user: login: Weijun-H
body: It deserves to return `LargeList`, if the nested data is `List`
```suggestion
                List(field) | FixedSizeList(field, _) | LargeList(field) => {
                    LargeList(Arc::clone(field))
                }
```
created_at: 2025-03-12T05:03:27Z
author_association: MEMBER
diff_hunk: @@ -102,25 +101,21 @@ impl ScalarUDFImpl for Flatten {
     }
 
     fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
-        fn get_base_type(data_type: &DataType) -> Result<DataType> {
-            match data_type {
-                List(field) | FixedSizeList(field, _)
-                    if matches!(field.data_type(), List(_) | FixedSizeList(_, _)) =>
-                {
-                    get_base_type(field.data_type())
-                }
-                LargeList(field) if matches!(field.data_type(), LargeList(_)) => {
-                    get_base_type(field.data_type())
-                }
-                Null | List(_) | LargeList(_) => Ok(data_type.to_owned()),
-                FixedSizeList(field, _) => Ok(List(Arc::clone(field))),
-                _ => exec_err!(
-                    "Not reachable, data_type should be List, LargeList or FixedSizeList"
-                ),
-            }
-        }
+        let data_type = match &arg_types[0] {
+            List(field) | FixedSizeList(field, _) => match field.data_type() {
+                List(field) | FixedSizeList(field, _) => List(Arc::clone(field)),
+                _ => arg_types[0].clone(),
+            },
+            LargeList(field) => match field.data_type() {
+                LargeList(field) => LargeList(Arc::clone(field)),
path: datafusion/functions-nested/src/flatten.rs
user: login: velaskuez
body: Ah ok, I'll need to make some changes to support this. Currently trying to flatten `LargeList(List)` will fail when casting the inner list to generic list with i64 offset (using the type parameter `O`)
created_at: 2025-03-12T14:36:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -77,7 +77,6 @@ impl Flatten {
     pub fn new() -> Self {
         Self {
             signature: Signature {
-                // TODO (https://github.com/apache/datafusion/issues/13757) flatten should be single-step, not recursive
                 type_signature: TypeSignature::ArraySignature(
                     ArrayFunctionSignature::RecursiveArray,
path: datafusion/functions-nested/src/flatten.rs
user: login: findepi
body: Is this signature still applicable?

Maybe we should switch to Array, and mark RecursiveArray as deprecated?
created_at: 2025-03-12T15:50:16Z
author_association: MEMBER
diff_hunk: @@ -77,7 +77,6 @@ impl Flatten {
     pub fn new() -> Self {
         Self {
             signature: Signature {
-                // TODO (https://github.com/apache/datafusion/issues/13757) flatten should be single-step, not recursive
                 type_signature: TypeSignature::ArraySignature(
                     ArrayFunctionSignature::RecursiveArray,
path: datafusion/functions-nested/src/flatten.rs
user: login: velaskuez
body: What would the deprecated note be since, from what I understand, this was added in specifically for `flatten` to recursively coerce `FixedLengthList` to `List`. I'm wondering if any users would rely on that downstream
created_at: 2025-03-13T12:54:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -146,14 +145,61 @@ pub fn flatten_inner(args: &[ArrayRef]) -> Result<ArrayRef> {
 
     match array.data_type() {
         List(_) => {
-            let list_arr = as_list_array(&array)?;
-            let flattened_array = flatten_internal::<i32>(list_arr.clone(), None)?;
-            Ok(Arc::new(flattened_array) as ArrayRef)
+            let (field, offsets, values, _) = as_list_array(&array)?.clone().into_parts();
path: datafusion/functions-nested/src/flatten.rs
user: login: cht42
body: ```suggestion
            let (field, offsets, values, nulls) = as_list_array(&array)?.clone().into_parts();
```
created_at: 2025-04-11T14:05:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -146,14 +145,61 @@ pub fn flatten_inner(args: &[ArrayRef]) -> Result<ArrayRef> {
 
     match array.data_type() {
         List(_) => {
-            let list_arr = as_list_array(&array)?;
-            let flattened_array = flatten_internal::<i32>(list_arr.clone(), None)?;
-            Ok(Arc::new(flattened_array) as ArrayRef)
+            let (field, offsets, values, _) = as_list_array(&array)?.clone().into_parts();
+
+            match field.data_type() {
+                List(_) => {
+                    let (inner_field, inner_offsets, inner_values, _) =
+                        as_list_array(&values)?.clone().into_parts();
+                    let offsets = get_offsets_for_flatten::<i32>(inner_offsets, offsets);
+                    let flattened_array = GenericListArray::<i32>::new(
+                        inner_field,
+                        offsets,
+                        inner_values,
+                        None,
path: datafusion/functions-nested/src/flatten.rs
user: login: cht42
body: ```suggestion
                        nulls,
```
created_at: 2025-04-11T14:05:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -146,14 +145,61 @@ pub fn flatten_inner(args: &[ArrayRef]) -> Result<ArrayRef> {
 
     match array.data_type() {
         List(_) => {
-            let list_arr = as_list_array(&array)?;
-            let flattened_array = flatten_internal::<i32>(list_arr.clone(), None)?;
-            Ok(Arc::new(flattened_array) as ArrayRef)
+            let (field, offsets, values, _) = as_list_array(&array)?.clone().into_parts();
+
+            match field.data_type() {
+                List(_) => {
+                    let (inner_field, inner_offsets, inner_values, _) =
+                        as_list_array(&values)?.clone().into_parts();
+                    let offsets = get_offsets_for_flatten::<i32>(inner_offsets, offsets);
+                    let flattened_array = GenericListArray::<i32>::new(
+                        inner_field,
+                        offsets,
+                        inner_values,
+                        None,
+                    );
+
+                    Ok(Arc::new(flattened_array) as ArrayRef)
+                }
+                LargeList(_) => {
+                    exec_err!("flatten does not support type '{:?}'", array.data_type())?
+                }
+                _ => Ok(Arc::clone(array) as ArrayRef),
+            }
         }
         LargeList(_) => {
-            let list_arr = as_large_list_array(&array)?;
-            let flattened_array = flatten_internal::<i64>(list_arr.clone(), None)?;
-            Ok(Arc::new(flattened_array) as ArrayRef)
+            let (field, offsets, values, _) =
path: datafusion/functions-nested/src/flatten.rs
user: login: cht42
body: ```suggestion
            let (field, offsets, values, nulls) =
```
created_at: 2025-04-11T14:05:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -146,14 +145,61 @@ pub fn flatten_inner(args: &[ArrayRef]) -> Result<ArrayRef> {
 
     match array.data_type() {
         List(_) => {
-            let list_arr = as_list_array(&array)?;
-            let flattened_array = flatten_internal::<i32>(list_arr.clone(), None)?;
-            Ok(Arc::new(flattened_array) as ArrayRef)
+            let (field, offsets, values, _) = as_list_array(&array)?.clone().into_parts();
+
+            match field.data_type() {
+                List(_) => {
+                    let (inner_field, inner_offsets, inner_values, _) =
+                        as_list_array(&values)?.clone().into_parts();
+                    let offsets = get_offsets_for_flatten::<i32>(inner_offsets, offsets);
+                    let flattened_array = GenericListArray::<i32>::new(
+                        inner_field,
+                        offsets,
+                        inner_values,
+                        None,
+                    );
+
+                    Ok(Arc::new(flattened_array) as ArrayRef)
+                }
+                LargeList(_) => {
+                    exec_err!("flatten does not support type '{:?}'", array.data_type())?
+                }
+                _ => Ok(Arc::clone(array) as ArrayRef),
+            }
         }
         LargeList(_) => {
-            let list_arr = as_large_list_array(&array)?;
-            let flattened_array = flatten_internal::<i64>(list_arr.clone(), None)?;
-            Ok(Arc::new(flattened_array) as ArrayRef)
+            let (field, offsets, values, _) =
+                as_large_list_array(&array)?.clone().into_parts();
+
+            match field.data_type() {
+                List(_) => {
+                    let (inner_field, inner_offsets, inner_values, _) =
+                        as_list_array(&values)?.clone().into_parts();
+                    let offsets = get_large_offsets_for_flatten(inner_offsets, offsets);
+                    let flattened_array = GenericListArray::<i64>::new(
+                        inner_field,
+                        offsets,
+                        inner_values,
+                        None,
path: datafusion/functions-nested/src/flatten.rs
user: login: cht42
body: ```suggestion
                        nulls,
```
created_at: 2025-04-11T14:05:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -146,14 +145,61 @@ pub fn flatten_inner(args: &[ArrayRef]) -> Result<ArrayRef> {
 
     match array.data_type() {
         List(_) => {
-            let list_arr = as_list_array(&array)?;
-            let flattened_array = flatten_internal::<i32>(list_arr.clone(), None)?;
-            Ok(Arc::new(flattened_array) as ArrayRef)
+            let (field, offsets, values, _) = as_list_array(&array)?.clone().into_parts();
+
+            match field.data_type() {
+                List(_) => {
+                    let (inner_field, inner_offsets, inner_values, _) =
+                        as_list_array(&values)?.clone().into_parts();
+                    let offsets = get_offsets_for_flatten::<i32>(inner_offsets, offsets);
+                    let flattened_array = GenericListArray::<i32>::new(
+                        inner_field,
+                        offsets,
+                        inner_values,
+                        None,
+                    );
+
+                    Ok(Arc::new(flattened_array) as ArrayRef)
+                }
+                LargeList(_) => {
+                    exec_err!("flatten does not support type '{:?}'", array.data_type())?
+                }
+                _ => Ok(Arc::clone(array) as ArrayRef),
+            }
         }
         LargeList(_) => {
-            let list_arr = as_large_list_array(&array)?;
-            let flattened_array = flatten_internal::<i64>(list_arr.clone(), None)?;
-            Ok(Arc::new(flattened_array) as ArrayRef)
+            let (field, offsets, values, _) =
+                as_large_list_array(&array)?.clone().into_parts();
+
+            match field.data_type() {
+                List(_) => {
+                    let (inner_field, inner_offsets, inner_values, _) =
+                        as_list_array(&values)?.clone().into_parts();
+                    let offsets = get_large_offsets_for_flatten(inner_offsets, offsets);
+                    let flattened_array = GenericListArray::<i64>::new(
+                        inner_field,
+                        offsets,
+                        inner_values,
+                        None,
+                    );
+
+                    Ok(Arc::new(flattened_array) as ArrayRef)
+                }
+                LargeList(_) => {
+                    let (inner_field, inner_offsets, inner_values, _) =
path: datafusion/functions-nested/src/flatten.rs
user: login: cht42
body: ```suggestion
                    let (inner_field, inner_offsets, inner_values, nulls) =
```
created_at: 2025-04-11T14:05:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -146,14 +145,61 @@ pub fn flatten_inner(args: &[ArrayRef]) -> Result<ArrayRef> {
 
     match array.data_type() {
         List(_) => {
-            let list_arr = as_list_array(&array)?;
-            let flattened_array = flatten_internal::<i32>(list_arr.clone(), None)?;
-            Ok(Arc::new(flattened_array) as ArrayRef)
+            let (field, offsets, values, _) = as_list_array(&array)?.clone().into_parts();
+
+            match field.data_type() {
+                List(_) => {
+                    let (inner_field, inner_offsets, inner_values, _) =
+                        as_list_array(&values)?.clone().into_parts();
+                    let offsets = get_offsets_for_flatten::<i32>(inner_offsets, offsets);
+                    let flattened_array = GenericListArray::<i32>::new(
+                        inner_field,
+                        offsets,
+                        inner_values,
+                        None,
+                    );
+
+                    Ok(Arc::new(flattened_array) as ArrayRef)
+                }
+                LargeList(_) => {
+                    exec_err!("flatten does not support type '{:?}'", array.data_type())?
+                }
+                _ => Ok(Arc::clone(array) as ArrayRef),
+            }
         }
         LargeList(_) => {
-            let list_arr = as_large_list_array(&array)?;
-            let flattened_array = flatten_internal::<i64>(list_arr.clone(), None)?;
-            Ok(Arc::new(flattened_array) as ArrayRef)
+            let (field, offsets, values, _) =
+                as_large_list_array(&array)?.clone().into_parts();
+
+            match field.data_type() {
+                List(_) => {
+                    let (inner_field, inner_offsets, inner_values, _) =
+                        as_list_array(&values)?.clone().into_parts();
+                    let offsets = get_large_offsets_for_flatten(inner_offsets, offsets);
+                    let flattened_array = GenericListArray::<i64>::new(
+                        inner_field,
+                        offsets,
+                        inner_values,
+                        None,
+                    );
+
+                    Ok(Arc::new(flattened_array) as ArrayRef)
+                }
+                LargeList(_) => {
+                    let (inner_field, inner_offsets, inner_values, _) =
+                        as_large_list_array(&values)?.clone().into_parts();
+                    let offsets = get_offsets_for_flatten::<i64>(inner_offsets, offsets);
+                    let flattened_array = GenericListArray::<i64>::new(
+                        inner_field,
+                        offsets,
+                        inner_values,
+                        None,
path: datafusion/functions-nested/src/flatten.rs
user: login: cht42
body: ```suggestion
                        nulls,
```
created_at: 2025-04-11T14:06:05Z
author_association: CONTRIBUTOR

PR comments:
user: login: cht42
created_at: 2025-04-02T06:10:17Z
body: Hey @delamarch3 are you still tracking merging this ? interested in this new behavior
author_association: CONTRIBUTOR
user: login: velaskuez
created_at: 2025-04-02T14:06:24Z
body: Hi @cht42, I'm just waiting for another review.

@findepi could you take another look at this when you get the chance please?
author_association: CONTRIBUTOR
user: login: cht42
created_at: 2025-04-09T09:13:44Z
body: Maybe @alamb or @Weijun-H can review ?
author_association: CONTRIBUTOR
user: login: velaskuez
created_at: 2025-04-12T09:58:48Z
body: Thanks for the reviews!
author_association: CONTRIBUTOR
user: login: cht42
created_at: 2025-04-16T07:53:01Z
body: @alamb can we get this merged please ?
author_association: CONTRIBUTOR
user: login: alamb
created_at: 2025-04-17T16:53:59Z
body: Thanks @cht42 @delamarch3 and @Weijun-H  -- I reviewed this PR and marked it as an API change. I think it looks good and does what is requested

Thanks!
author_association: CONTRIBUTOR