Repository: apache/datafusion
Pull request #15825: Preserve projection for inline scan
Author: login: jayzhan211
Labels: name: logical-expr
description: Logical plan and expressions
name: core
description: Core DataFusion crate
Created: 2025-04-23T11:38:11Z | Merged: 2025-04-24T09:54:30Z

## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->

- Closes #.

## Rationale for this change

<!--
 Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
 Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
-->

## What changes are included in this PR?

<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->

## Are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->


Linked issue #123: Implement physical plan for EXISTS subquery
*Note*: migrated from original JIRA: https://issues.apache.org/jira/browse/ARROW-10819

The TPC-H queries include use of the EXISTS which is used to test for the existence of any record in a subquery. For example:

```sql
and *exists* (
    select
        *
    from
        lineitem
    where
        l_orderkey = o_orderkey
        and l_commitdate < l_receiptdate
)
```
Linked issue comments: user: login: alamb
created_at: 2021-04-26T13:23:43Z
body: Comment from Andy Grove(andygrove) @ 2020-12-31T19:42:45.132+0000:
<pre>The example given here is a correlated subquery that can be translated into a join.

Here is a random stackoverflow discussion on this for reference (I have not reviewed it)

https://stackoverflow.com/questions/1772609/procedurally-transform-subquery-into-join</pre>
author_association: CONTRIBUTOR
user: login: alamb
created_at: 2022-10-21T19:21:30Z
body: In case anyone is curious -- we support correlated versions of these queries (via a join) but if there is no correlation (not super useful) we do not

```sql
❯ create table foo as select * from (values (1), (2), (NULL)) as sql
;
0 rows in set. Query took 0.022 seconds.
3 rows in set. Query took 0.007 seconds.
❯ create table bar as select * from (values (1), (2), (NULL)) as sql;
0 rows in set. Query took 0.000 seconds.
❯ select * from foo where exists (select column1 from bar);
NotImplemented("Physical plan does not support logical expression EXISTS (<subquery>)")
❯ select * from foo where exists (select column1 from bar where foo.column1 = bar.column1);
+---------+
| column1 |
+---------+
| 2       |
| 1       |
+---------+
```
author_association: CONTRIBUTOR
user: login: logan-keede
created_at: 2025-02-09T19:53:56Z
body: 
> In case anyone is curious -- we support correlated versions of these queries (via a join) but if there is no correlation (not super useful) we do not
> 
> ❯ create table foo as select * from (values (1), (2), (NULL)) as sql
> ;
> 0 rows in set. Query took 0.022 seconds.
> 3 rows in set. Query took 0.007 seconds.
> ❯ create table bar as select * from (values (1), (2), (NULL)) as sql;
> 0 rows in set. Query took 0.000 seconds.
> ❯ select * from foo where exists (select column1 from bar);
> NotImplemented("Physical plan does not support logical expression EXISTS (<subquery>)")
> ❯ select * from foo where exists (select column1 from bar where foo.column1 = bar.column1);
> +---------+
> | column1 |
> +---------+
> | 2       |
> | 1       |
> +---------+

```sql

> explain select * from foo where exists (select column1 from bar);
+---------------+-----------------------------------------------------+
| plan_type     | plan                                                |
+---------------+-----------------------------------------------------+
| logical_plan  | LeftSemi Join:                                      |
|               |   TableScan: foo projection=[column1]               |
|               |   SubqueryAlias: __correlated_sq_1                  |
|               |     TableScan: bar projection=[]                    |
| physical_plan | NestedLoopJoinExec: join_type=RightSemi             |
|               |   DataSourceExec: partitions=1, partition_sizes=[1] |
|               |   DataSourceExec: partitions=1, partition_sizes=[1] |
|               |                                                     |
+---------------+-----------------------------------------------------+
2 row(s) fetched. 
Elapsed 0.007 seconds.

> select * from foo where exists (select column1 from bar);
+---------+
| column1 |
+---------+
| 1       |
| 2       |
| NULL    |
+---------+
3 row(s) fetched. 
Elapsed 0.006 seconds.
```


I get this on `datafusion 45.0.0`, perhaps this issue can be closed or at least updated if there is a case where we still don't support EXISTS in physical plan.

author_association: CONTRIBUTOR
user: login: alamb
created_at: 2025-02-10T15:04:31Z
body: Looks good -- thanks for checking @logan-keede  -- we can open a new issue if we find another hole
author_association: CONTRIBUTOR

Diff:
diff --git a/datafusion/core/tests/execution/logical_plan.rs b/datafusion/core/tests/execution/logical_plan.rs
index b30636ddf6a81..fdee6fd5dbbce 100644
--- a/datafusion/core/tests/execution/logical_plan.rs
+++ b/datafusion/core/tests/execution/logical_plan.rs
@@ -19,15 +19,19 @@
 //! create them and depend on them. Test executable semantics of logical plans.
 
 use arrow::array::Int64Array;
-use arrow::datatypes::{DataType, Field};
+use arrow::datatypes::{DataType, Field, Schema};
+use datafusion::datasource::{provider_as_source, ViewTable};
 use datafusion::execution::session_state::SessionStateBuilder;
-use datafusion_common::{Column, DFSchema, Result, ScalarValue, Spans};
+use datafusion_common::{Column, DFSchema, DFSchemaRef, Result, ScalarValue, Spans};
 use datafusion_execution::TaskContext;
 use datafusion_expr::expr::{AggregateFunction, AggregateFunctionParams};
 use datafusion_expr::logical_plan::{LogicalPlan, Values};
-use datafusion_expr::{Aggregate, AggregateUDF, Expr};
+use datafusion_expr::{
+    Aggregate, AggregateUDF, EmptyRelation, Expr, LogicalPlanBuilder, UNNAMED_TABLE,
+};
 use datafusion_functions_aggregate::count::Count;
 use datafusion_physical_plan::collect;
+use insta::assert_snapshot;
 use std::collections::HashMap;
 use std::fmt::Debug;
 use std::ops::Deref;
@@ -96,3 +100,37 @@ where
     };
     element
 }
+
+#[test]
+fn inline_scan_projection_test() -> Result<()> {
+    let name = UNNAMED_TABLE;
+    let column = "a";
+
+    let schema = Schema::new(vec![
+        Field::new("a", DataType::Int32, false),
+        Field::new("b", DataType::Int32, false),
+    ]);
+    let projection = vec![schema.index_of(column)?];
+
+    let provider = ViewTable::new(
+        LogicalPlan::EmptyRelation(EmptyRelation {
+            produce_one_row: false,
+            schema: DFSchemaRef::new(DFSchema::try_from(schema)?),
+        }),
+        None,
+    );
+    let source = provider_as_source(Arc::new(provider));
+
+    let plan = LogicalPlanBuilder::scan(name, source, Some(projection))?.build()?;
+
+    assert_snapshot!(
+        format!("{plan}"),
+        @r"
+    SubqueryAlias: ?table?
+      Projection: a
+        EmptyRelation
+    "
+    );
+
+    Ok(())
+}
diff --git a/datafusion/expr/src/logical_plan/builder.rs b/datafusion/expr/src/logical_plan/builder.rs
index 05a43444d4ae4..24a5c0fe9a211 100644
--- a/datafusion/expr/src/logical_plan/builder.rs
+++ b/datafusion/expr/src/logical_plan/builder.rs
@@ -501,6 +501,21 @@ impl LogicalPlanBuilder {
         if table_scan.filters.is_empty() {
             if let Some(p) = table_scan.source.get_logical_plan() {
                 let sub_plan = p.into_owned();
+
+                if let Some(proj) = table_scan.projection {
+                    let projection_exprs = proj
+                        .into_iter()
+                        .map(|i| {
+                            Expr::Column(Column::from(
+                                sub_plan.schema().qualified_field(i),
+                            ))
+                        })
+                        .collect::<Vec<_>>();
+                    return Self::new(sub_plan)
+                        .project(projection_exprs)?
+                        .alias(table_scan.table_name);
+                }
+
                 // Ensures that the reference to the inlined table remains the
                 // same, meaning we don't have to change any of the parent nodes
                 // that reference this table.


Reviews:
user: login: xudong963
body: Which one do we wanna merge to main?
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-23T14:51:39Z
user: login: vadimpiven
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-23T15:12:18Z
user: login: jayzhan211
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-24T00:40:19Z
user: login: xudong963
body: Thank you
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-24T05:52:13Z
user: login: vadimpiven
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-24T09:07:42Z
user: login: alamb
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-28T20:28:21Z

Review comments:
diff_hunk: @@ -96,3 +100,37 @@ where
     };
     element
 }
+
+#[test]
+fn inline_scan_projection_test() -> Result<()> {
+    let name = UNNAMED_TABLE;
+    let column = "a";
+
+    let schema = Schema::new(vec![
+        Field::new("a", DataType::Int32, false),
+        Field::new("b", DataType::Int32, false),
+    ]);
+    let projection = vec![schema.index_of(column)?];
+
+    let provider = ViewTable::new(
+        LogicalPlan::EmptyRelation(EmptyRelation {
+            produce_one_row: false,
+            schema: DFSchemaRef::new(DFSchema::try_from(schema)?),
+        }),
+        None,
+    );
+    let source = provider_as_source(Arc::new(provider));
+
+    let plan = LogicalPlanBuilder::scan(name, source, Some(projection))?.build()?;
+
+    assert_snapshot!(
+        format!("{plan}"),
+        @r"
path: datafusion/core/tests/execution/logical_plan.rs
user: login: vadimpiven
body: May be it would be more readable to format like here https://github.com/apache/datafusion/blob/03fa3b99f6c87b04321f85ecdc71b4b54ed4d884/datafusion/optimizer/src/eliminate_filter.rs#L142-L145 ?
created_at: 2025-04-23T15:12:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -96,3 +100,37 @@ where
     };
     element
 }
+
+#[test]
+fn inline_scan_projection_test() -> Result<()> {
+    let name = UNNAMED_TABLE;
+    let column = "a";
+
+    let schema = Schema::new(vec![
+        Field::new("a", DataType::Int32, false),
+        Field::new("b", DataType::Int32, false),
+    ]);
+    let projection = vec![schema.index_of(column)?];
+
+    let provider = ViewTable::new(
+        LogicalPlan::EmptyRelation(EmptyRelation {
+            produce_one_row: false,
+            schema: DFSchemaRef::new(DFSchema::try_from(schema)?),
+        }),
+        None,
+    );
+    let source = provider_as_source(Arc::new(provider));
+
+    let plan = LogicalPlanBuilder::scan(name, source, Some(projection))?.build()?;
+
+    assert_snapshot!(
+        format!("{plan}"),
+        @r"
path: datafusion/core/tests/execution/logical_plan.rs
user: login: jayzhan211
body: you mean to move out the string to `expected`?

but for insta test, we keep the string inline to update easily
created_at: 2025-04-24T00:40:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -96,3 +100,37 @@ where
     };
     element
 }
+
+#[test]
+fn inline_scan_projection_test() -> Result<()> {
+    let name = UNNAMED_TABLE;
+    let column = "a";
+
+    let schema = Schema::new(vec![
+        Field::new("a", DataType::Int32, false),
+        Field::new("b", DataType::Int32, false),
+    ]);
+    let projection = vec![schema.index_of(column)?];
+
+    let provider = ViewTable::new(
+        LogicalPlan::EmptyRelation(EmptyRelation {
+            produce_one_row: false,
+            schema: DFSchemaRef::new(DFSchema::try_from(schema)?),
+        }),
+        None,
+    );
+    let source = provider_as_source(Arc::new(provider));
+
+    let plan = LogicalPlanBuilder::scan(name, source, Some(projection))?.build()?;
+
+    assert_snapshot!(
+        format!("{plan}"),
+        @r"
path: datafusion/core/tests/execution/logical_plan.rs
user: login: vadimpiven
body: Actually, I just thought that the correct test would be
```
assert_eq!(
        plan.schema().field_names(),
        vec![format!("{name}.{column}")]
    );
```
because what matters is that projection is applied, not how. Updated the code in my PR.
created_at: 2025-04-24T09:07:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -501,6 +501,21 @@ impl LogicalPlanBuilder {
         if table_scan.filters.is_empty() {
             if let Some(p) = table_scan.source.get_logical_plan() {
                 let sub_plan = p.into_owned();
+
+                if let Some(proj) = table_scan.projection {
path: datafusion/expr/src/logical_plan/builder.rs
user: login: alamb
body: 👍 


created_at: 2025-04-28T20:28:21Z
author_association: CONTRIBUTOR

PR comments:
user: login: jayzhan211
created_at: 2025-04-24T00:41:48Z
body: You can merge this PR, commit from @vadimpiven is included
author_association: CONTRIBUTOR
user: login: jayzhan211
created_at: 2025-04-24T09:54:44Z
body: Thanks @xudong963 @vadimpiven 
author_association: CONTRIBUTOR