Repository: apache/datafusion
Pull request #15801: refactor filter pushdown apis
Author: login: adriangb
Labels: name: optimizer
description: Optimizer rules
name: core
description: Core DataFusion crate
name: sqllogictest
description: SQL Logic Tests (.slt)
name: datasource
description: Changes to the datasource crate
Created: 2025-04-22T07:11:17Z | Merged: 2025-05-05T13:40:15Z

This improves over the existing APIs by:
1. Not requiring downcast matching of specific `ExecutionPlan`s (as discussed previously this could be resolved by adding a new method, but in the end both approaches require two methods).
1. Doing less invasive modification of the plans (e.g. not popping and then re-inserting FilterExecs).
1. Reducing the algorithmic complexity of the optimizer rule (it's now a single pass over the tree for all cases, and avoids cloning unless strictly needed).
1. Allowing plans to decide how any of the leftover filters from parents or that they generated should be handled (I think this will be useful for TopK).

This is in some ways simpler and in some ways more complex than what we have right now. The difference is not large in terms of LOC, I think the rest boils down to personal preference.

I didn't use the TreeNode APIs, maybe there's a way to do that using `transform_down_up` or something but I couldn't figure it out. In any case that seems like an internal refactor that can be done later, as long as the manual implementation passes tests and the ExecutionPlan APIs are the same.


Diff:
diff --git a/datafusion/core/Cargo.toml b/datafusion/core/Cargo.toml
index 3ace3e14ec259..4b6d8f274932f 100644
--- a/datafusion/core/Cargo.toml
+++ b/datafusion/core/Cargo.toml
@@ -179,6 +179,10 @@ name = "csv_load"
 harness = false
 name = "distinct_query_sql"
 
+[[bench]]
+harness = false
+name = "push_down_filter"
+
 [[bench]]
 harness = false
 name = "sort_limit_query_sql"
diff --git a/datafusion/core/benches/push_down_filter.rs b/datafusion/core/benches/push_down_filter.rs
new file mode 100644
index 0000000000000..92de1711a9e8c
--- /dev/null
+++ b/datafusion/core/benches/push_down_filter.rs
@@ -0,0 +1,124 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow::array::RecordBatch;
+use arrow::datatypes::{DataType, Field, Schema};
+use bytes::{BufMut, BytesMut};
+use criterion::{criterion_group, criterion_main, BenchmarkId, Criterion};
+use datafusion::config::ConfigOptions;
+use datafusion::prelude::{ParquetReadOptions, SessionContext};
+use datafusion_execution::object_store::ObjectStoreUrl;
+use datafusion_physical_optimizer::filter_pushdown::FilterPushdown;
+use datafusion_physical_optimizer::PhysicalOptimizerRule;
+use datafusion_physical_plan::ExecutionPlan;
+use object_store::memory::InMemory;
+use object_store::path::Path;
+use object_store::ObjectStore;
+use parquet::arrow::ArrowWriter;
+use std::sync::Arc;
+
+async fn create_plan() -> Arc<dyn ExecutionPlan> {
+    let ctx = SessionContext::new();
+    let schema = Arc::new(Schema::new(vec![
+        Field::new("id", DataType::Int32, true),
+        Field::new("name", DataType::Utf8, true),
+        Field::new("age", DataType::UInt16, true),
+        Field::new("salary", DataType::Float64, true),
+    ]));
+    let batch = RecordBatch::new_empty(schema);
+
+    let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;
+    let mut out = BytesMut::new().writer();
+    {
+        let mut writer = ArrowWriter::try_new(&mut out, batch.schema(), None).unwrap();
+        writer.write(&batch).unwrap();
+        writer.finish().unwrap();
+    }
+    let data = out.into_inner().freeze();
+    store
+        .put(&Path::from("test.parquet"), data.into())
+        .await
+        .unwrap();
+    ctx.register_object_store(
+        ObjectStoreUrl::parse("memory://").unwrap().as_ref(),
+        store,
+    );
+
+    ctx.register_parquet("t", "memory:///", ParquetReadOptions::default())
+        .await
+        .unwrap();
+
+    let df = ctx
+        .sql(
+            r"
+        WITH brackets AS (
+            SELECT age % 10 AS age_bracket
+            FROM t
+            GROUP BY age % 10
+            HAVING COUNT(*) > 10
+        )
+        SELECT id, name, age, salary
+        FROM t
+        JOIN brackets ON t.age % 10 = brackets.age_bracket
+        WHERE age > 20 AND t.salary > 1000
+        ORDER BY t.salary DESC
+        LIMIT 100
+    ",
+        )
+        .await
+        .unwrap();
+
+    df.create_physical_plan().await.unwrap()
+}
+
+#[derive(Clone)]
+struct BenchmarkPlan {
+    plan: Arc<dyn ExecutionPlan>,
+    config: ConfigOptions,
+}
+
+impl std::fmt::Display for BenchmarkPlan {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(f, "BenchmarkPlan")
+    }
+}
+
+fn bench_push_down_filter(c: &mut Criterion) {
+    // Create a relatively complex plan
+    let plan = tokio::runtime::Runtime::new()
+        .unwrap()
+        .block_on(create_plan());
+    let mut config = ConfigOptions::default();
+    config.execution.parquet.pushdown_filters = true;
+    let plan = BenchmarkPlan { plan, config };
+
+    c.bench_with_input(
+        BenchmarkId::new("push_down_filter", plan.clone()),
+        &plan,
+        |b, plan| {
+            b.iter(|| {
+                let optimizer = FilterPushdown::new();
+                optimizer
+                    .optimize(Arc::clone(&plan.plan), &plan.config)
+                    .unwrap();
+            });
+        },
+    );
+}
+
+criterion_group!(benches, bench_push_down_filter);
+criterion_main!(benches);
diff --git a/datafusion/core/tests/physical_optimizer/push_down_filter.rs b/datafusion/core/tests/physical_optimizer/push_down_filter.rs
index b19144f1bcffe..326a7b837e7a2 100644
--- a/datafusion/core/tests/physical_optimizer/push_down_filter.rs
+++ b/datafusion/core/tests/physical_optimizer/push_down_filter.rs
@@ -44,11 +44,10 @@ use datafusion_physical_expr::{
     aggregate::AggregateExprBuilder, conjunction, Partitioning,
 };
 use datafusion_physical_expr_common::physical_expr::fmt_sql;
-use datafusion_physical_optimizer::push_down_filter::PushdownFilter;
+use datafusion_physical_optimizer::filter_pushdown::FilterPushdown;
 use datafusion_physical_optimizer::PhysicalOptimizerRule;
 use datafusion_physical_plan::filter_pushdown::{
-    filter_pushdown_not_supported, FilterDescription, FilterPushdownResult,
-    FilterPushdownSupport,
+    FilterPushdownPropagation, PredicateSupports,
 };
 use datafusion_physical_plan::{
     aggregates::{AggregateExec, AggregateMode, PhysicalGroupBy},
@@ -154,29 +153,24 @@ impl FileSource for TestSource {
 
     fn try_pushdown_filters(
         &self,
-        mut fd: FilterDescription,
+        mut filters: Vec<Arc<dyn PhysicalExpr>>,
         config: &ConfigOptions,
-    ) -> Result<FilterPushdownResult<Arc<dyn FileSource>>> {
+    ) -> Result<FilterPushdownPropagation<Arc<dyn FileSource>>> {
         if self.support && config.execution.parquet.pushdown_filters {
             if let Some(internal) = self.predicate.as_ref() {
-                fd.filters.push(Arc::clone(internal));
+                filters.push(Arc::clone(internal));
             }
-            let all_filters = fd.take_description();
-
-            Ok(FilterPushdownResult {
-                support: FilterPushdownSupport::Supported {
-                    child_descriptions: vec![],
-                    op: Arc::new(TestSource {
-                        support: true,
-                        predicate: Some(conjunction(all_filters)),
-                        statistics: self.statistics.clone(), // should be updated in reality
-                    }),
-                    revisit: false,
-                },
-                remaining_description: FilterDescription::empty(),
+            let new_node = Arc::new(TestSource {
+                support: true,
+                predicate: Some(conjunction(filters.clone())),
+                statistics: self.statistics.clone(), // should be updated in reality
+            });
+            Ok(FilterPushdownPropagation {
+                filters: PredicateSupports::all_supported(filters),
+                updated_node: Some(new_node),
             })
         } else {
-            Ok(filter_pushdown_not_supported(fd))
+            Ok(FilterPushdownPropagation::unsupported(filters))
         }
     }
 }
@@ -201,7 +195,7 @@ fn test_pushdown_into_scan() {
 
     // expect the predicate to be pushed down into the DataSource
     insta::assert_snapshot!(
-        OptimizationTest::new(plan, PushdownFilter{}, true),
+        OptimizationTest::new(plan, FilterPushdown{}, true),
         @r"
     OptimizationTest:
       input:
@@ -225,7 +219,7 @@ fn test_pushdown_into_scan_with_config_options() {
     insta::assert_snapshot!(
         OptimizationTest::new(
             Arc::clone(&plan),
-            PushdownFilter {},
+            FilterPushdown {},
             false
         ),
         @r"
@@ -244,7 +238,7 @@ fn test_pushdown_into_scan_with_config_options() {
     insta::assert_snapshot!(
         OptimizationTest::new(
             plan,
-            PushdownFilter {},
+            FilterPushdown {},
             true
         ),
         @r"
@@ -269,7 +263,7 @@ fn test_filter_collapse() {
     let plan = Arc::new(FilterExec::try_new(predicate2, filter1).unwrap());
 
     insta::assert_snapshot!(
-        OptimizationTest::new(plan, PushdownFilter{}, true),
+        OptimizationTest::new(plan, FilterPushdown{}, true),
         @r"
     OptimizationTest:
       input:
@@ -278,7 +272,7 @@ fn test_filter_collapse() {
         -     DataSourceExec: file_groups={0 groups: []}, projection=[a, b, c], file_type=test, pushdown_supported=true
       output:
         Ok:
-          - DataSourceExec: file_groups={0 groups: []}, projection=[a, b, c], file_type=test, pushdown_supported=true, predicate=b@1 = bar AND a@0 = foo
+          - DataSourceExec: file_groups={0 groups: []}, projection=[a, b, c], file_type=test, pushdown_supported=true, predicate=a@0 = foo AND b@1 = bar
     "
     );
 }
@@ -288,25 +282,28 @@ fn test_filter_with_projection() {
     let scan = test_scan(true);
     let projection = vec![1, 0];
     let predicate = col_lit_predicate("a", "foo", schema());
-    let plan = Arc::new(
-        FilterExec::try_new(predicate, Arc::clone(&scan))
+    let filter = Arc::new(
+        FilterExec::try_new(Arc::clone(&predicate), Arc::clone(&scan))
             .unwrap()
             .with_projection(Some(projection))
             .unwrap(),
     );
+    let predicate = col_lit_predicate("b", "bar", &filter.schema());
+    let plan = Arc::new(FilterExec::try_new(predicate, filter).unwrap());
 
     // expect the predicate to be pushed down into the DataSource but the FilterExec to be converted to ProjectionExec
     insta::assert_snapshot!(
-        OptimizationTest::new(plan, PushdownFilter{}, true),
+        OptimizationTest::new(plan, FilterPushdown{}, true),
         @r"
     OptimizationTest:
       input:
-        - FilterExec: a@0 = foo, projection=[b@1, a@0]
-        -   DataSourceExec: file_groups={0 groups: []}, projection=[a, b, c], file_type=test, pushdown_supported=true
+        - FilterExec: b@0 = bar
+        -   FilterExec: a@0 = foo, projection=[b@1, a@0]
+        -     DataSourceExec: file_groups={0 groups: []}, projection=[a, b, c], file_type=test, pushdown_supported=true
       output:
         Ok:
           - ProjectionExec: expr=[b@1 as b, a@0 as a]
-          -   DataSourceExec: file_groups={0 groups: []}, projection=[a, b, c], file_type=test, pushdown_supported=true, predicate=a@0 = foo
+          -   DataSourceExec: file_groups={0 groups: []}, projection=[a, b, c], file_type=test, pushdown_supported=true, predicate=a@0 = foo AND b@1 = bar
     ",
     );
 
@@ -320,7 +317,7 @@ fn test_filter_with_projection() {
             .unwrap(),
     );
     insta::assert_snapshot!(
-        OptimizationTest::new(plan, PushdownFilter{},true),
+        OptimizationTest::new(plan, FilterPushdown{},true),
         @r"
     OptimizationTest:
       input:
@@ -349,7 +346,7 @@ fn test_push_down_through_transparent_nodes() {
 
     // expect the predicate to be pushed down into the DataSource
     insta::assert_snapshot!(
-        OptimizationTest::new(plan, PushdownFilter{},true),
+        OptimizationTest::new(plan, FilterPushdown{},true),
         @r"
     OptimizationTest:
       input:
@@ -362,7 +359,7 @@ fn test_push_down_through_transparent_nodes() {
         Ok:
           - RepartitionExec: partitioning=RoundRobinBatch(1), input_partitions=0
           -   CoalesceBatchesExec: target_batch_size=1
-          -     DataSourceExec: file_groups={0 groups: []}, projection=[a, b, c], file_type=test, pushdown_supported=true, predicate=b@1 = bar AND a@0 = foo
+          -     DataSourceExec: file_groups={0 groups: []}, projection=[a, b, c], file_type=test, pushdown_supported=true, predicate=a@0 = foo AND b@1 = bar
     "
     );
 }
@@ -413,7 +410,7 @@ fn test_no_pushdown_through_aggregates() {
 
     // expect the predicate to be pushed down into the DataSource
     insta::assert_snapshot!(
-        OptimizationTest::new(plan, PushdownFilter{}, true),
+        OptimizationTest::new(plan, FilterPushdown{}, true),
         @r"
     OptimizationTest:
       input:
diff --git a/datafusion/datasource/src/file.rs b/datafusion/datasource/src/file.rs
index 6bbe9e50d3b06..c9b5c416f0c0e 100644
--- a/datafusion/datasource/src/file.rs
+++ b/datafusion/datasource/src/file.rs
@@ -28,10 +28,8 @@ use crate::file_stream::FileOpener;
 use arrow::datatypes::SchemaRef;
 use datafusion_common::config::ConfigOptions;
 use datafusion_common::{Result, Statistics};
-use datafusion_physical_expr::LexOrdering;
-use datafusion_physical_plan::filter_pushdown::{
-    filter_pushdown_not_supported, FilterDescription, FilterPushdownResult,
-};
+use datafusion_physical_expr::{LexOrdering, PhysicalExpr};
+use datafusion_physical_plan::filter_pushdown::FilterPushdownPropagation;
 use datafusion_physical_plan::metrics::ExecutionPlanMetricsSet;
 use datafusion_physical_plan::DisplayFormatType;
 
@@ -108,14 +106,14 @@ pub trait FileSource: Send + Sync {
     }
 
     /// Try to push down filters into this FileSource.
-    /// See [`ExecutionPlan::try_pushdown_filters`] for more details.
+    /// See [`ExecutionPlan::handle_child_pushdown_result`] for more details.
     ///
-    /// [`ExecutionPlan::try_pushdown_filters`]: datafusion_physical_plan::ExecutionPlan::try_pushdown_filters
+    /// [`ExecutionPlan::handle_child_pushdown_result`]: datafusion_physical_plan::ExecutionPlan::handle_child_pushdown_result
     fn try_pushdown_filters(
         &self,
-        fd: FilterDescription,
+        filters: Vec<Arc<dyn PhysicalExpr>>,
         _config: &ConfigOptions,
-    ) -> Result<FilterPushdownResult<Arc<dyn FileSource>>> {
-        Ok(filter_pushdown_not_supported(fd))
+    ) -> Result<FilterPushdownPropagation<Arc<dyn FileSource>>> {
+        Ok(FilterPushdownPropagation::unsupported(filters))
     }
 }
diff --git a/datafusion/datasource/src/file_scan_config.rs b/datafusion/datasource/src/file_scan_config.rs
index fb756cc11fbbc..ae94af5a7b263 100644
--- a/datafusion/datasource/src/file_scan_config.rs
+++ b/datafusion/datasource/src/file_scan_config.rs
@@ -48,14 +48,12 @@ use datafusion_common::{DataFusionError, ScalarValue};
 use datafusion_execution::{
     object_store::ObjectStoreUrl, SendableRecordBatchStream, TaskContext,
 };
+use datafusion_physical_expr::PhysicalExpr;
 use datafusion_physical_expr::{
     expressions::Column, EquivalenceProperties, LexOrdering, Partitioning,
     PhysicalSortExpr,
 };
-use datafusion_physical_plan::filter_pushdown::{
-    filter_pushdown_not_supported, FilterDescription, FilterPushdownResult,
-    FilterPushdownSupport,
-};
+use datafusion_physical_plan::filter_pushdown::FilterPushdownPropagation;
 use datafusion_physical_plan::{
     display::{display_orderings, ProjectSchemaDisplay},
     metrics::ExecutionPlanMetricsSet,
@@ -596,40 +594,29 @@ impl DataSource for FileScanConfig {
 
     fn try_pushdown_filters(
         &self,
-        fd: FilterDescription,
+        filters: Vec<Arc<dyn PhysicalExpr>>,
         config: &ConfigOptions,
-    ) -> Result<FilterPushdownResult<Arc<dyn DataSource>>> {
-        let FilterPushdownResult {
-            support,
-            remaining_description,
-        } = self.file_source.try_pushdown_filters(fd, config)?;
-
-        match support {
-            FilterPushdownSupport::Supported {
-                child_descriptions,
-                op,
-                revisit,
-            } => {
-                let new_data_source = Arc::new(
-                    FileScanConfigBuilder::from(self.clone())
-                        .with_source(op)
-                        .build(),
-                );
-
-                debug_assert!(child_descriptions.is_empty());
-                debug_assert!(!revisit);
-
-                Ok(FilterPushdownResult {
-                    support: FilterPushdownSupport::Supported {
-                        child_descriptions,
-                        op: new_data_source,
-                        revisit,
-                    },
-                    remaining_description,
+    ) -> Result<FilterPushdownPropagation<Arc<dyn DataSource>>> {
+        let result = self.file_source.try_pushdown_filters(filters, config)?;
+        match result.updated_node {
+            Some(new_file_source) => {
+                let file_scan_config = FileScanConfigBuilder::new(
+                    self.object_store_url.clone(),
+                    Arc::clone(&self.file_schema),
+                    new_file_source,
+                )
+                .build();
+                Ok(FilterPushdownPropagation {
+                    filters: result.filters,
+                    updated_node: Some(Arc::new(file_scan_config) as _),
                 })
             }
-            FilterPushdownSupport::NotSupported => {
-                Ok(filter_pushdown_not_supported(remaining_description))
+            None => {
+                // If the file source does not support filter pushdown, return the original config
+                Ok(FilterPushdownPropagation {
+                    filters: result.filters,
+                    updated_node: None,
+                })
             }
         }
     }
diff --git a/datafusion/datasource/src/source.rs b/datafusion/datasource/src/source.rs
index 022f77f2e4214..cf42347e3abae 100644
--- a/datafusion/datasource/src/source.rs
+++ b/datafusion/datasource/src/source.rs
@@ -33,11 +33,10 @@ use crate::file_scan_config::FileScanConfig;
 use datafusion_common::config::ConfigOptions;
 use datafusion_common::{Constraints, Result, Statistics};
 use datafusion_execution::{SendableRecordBatchStream, TaskContext};
-use datafusion_physical_expr::{EquivalenceProperties, Partitioning};
+use datafusion_physical_expr::{EquivalenceProperties, Partitioning, PhysicalExpr};
 use datafusion_physical_expr_common::sort_expr::LexOrdering;
 use datafusion_physical_plan::filter_pushdown::{
-    filter_pushdown_not_supported, FilterDescription, FilterPushdownResult,
-    FilterPushdownSupport,
+    ChildPushdownResult, FilterPushdownPropagation,
 };
 
 /// A source of data, typically a list of files or memory
@@ -95,13 +94,15 @@ pub trait DataSource: Send + Sync + Debug {
         _projection: &ProjectionExec,
     ) -> Result<Option<Arc<dyn ExecutionPlan>>>;
     /// Try to push down filters into this DataSource.
-    /// See [`ExecutionPlan::try_pushdown_filters`] for more details.
+    /// See [`ExecutionPlan::handle_child_pushdown_result`] for more details.
+    ///
+    /// [`ExecutionPlan::handle_child_pushdown_result`]: datafusion_physical_plan::ExecutionPlan::handle_child_pushdown_result
     fn try_pushdown_filters(
         &self,
-        fd: FilterDescription,
+        filters: Vec<Arc<dyn PhysicalExpr>>,
         _config: &ConfigOptions,
-    ) -> Result<FilterPushdownResult<Arc<dyn DataSource>>> {
-        Ok(filter_pushdown_not_supported(fd))
+    ) -> Result<FilterPushdownPropagation<Arc<dyn DataSource>>> {
+        Ok(FilterPushdownPropagation::unsupported(filters))
     }
 }
 
@@ -237,39 +238,31 @@ impl ExecutionPlan for DataSourceExec {
         self.data_source.try_swapping_with_projection(projection)
     }
 
-    fn try_pushdown_filters(
+    fn handle_child_pushdown_result(
         &self,
-        fd: FilterDescription,
+        child_pushdown_result: ChildPushdownResult,
         config: &ConfigOptions,
-    ) -> Result<FilterPushdownResult<Arc<dyn ExecutionPlan>>> {
-        let FilterPushdownResult {
-            support,
-            remaining_description,
-        } = self.data_source.try_pushdown_filters(fd, config)?;
-
-        match support {
-            FilterPushdownSupport::Supported {
-                child_descriptions,
-                op,
-                revisit,
-            } => {
-                let new_exec = Arc::new(DataSourceExec::new(op));
-
-                debug_assert!(child_descriptions.is_empty());
-                debug_assert!(!revisit);
-
-                Ok(FilterPushdownResult {
-                    support: FilterPushdownSupport::Supported {
-                        child_descriptions,
-                        op: new_exec,
-                        revisit,
-                    },
-                    remaining_description,
+    ) -> Result<FilterPushdownPropagation<Arc<dyn ExecutionPlan>>> {
+        // Push any remaining filters into our data source
+        let res = self.data_source.try_pushdown_filters(
+            child_pushdown_result.parent_filters.collect_all(),
+            config,
+        )?;
+        match res.updated_node {
+            Some(data_source) => {
+                let mut new_node = self.clone();
+                new_node.data_source = data_source;
+                new_node.cache =
+                    Self::compute_properties(Arc::clone(&new_node.data_source));
+                Ok(FilterPushdownPropagation {
+                    filters: res.filters,
+                    updated_node: Some(Arc::new(new_node)),
                 })
             }
-            FilterPushdownSupport::NotSupported => {
-                Ok(filter_pushdown_not_supported(remaining_description))
-            }
+            None => Ok(FilterPushdownPropagation {
+                filters: res.filters,
+                updated_node: None,
+            }),
         }
     }
 }
diff --git a/datafusion/physical-optimizer/src/push_down_filter.rs b/datafusion/physical-optimizer/src/filter_pushdown.rs
similarity index 67%
rename from datafusion/physical-optimizer/src/push_down_filter.rs
rename to datafusion/physical-optimizer/src/filter_pushdown.rs
index 80201454d06d4..6c445458b51bc 100644
--- a/datafusion/physical-optimizer/src/push_down_filter.rs
+++ b/datafusion/physical-optimizer/src/filter_pushdown.rs
@@ -19,25 +19,25 @@ use std::sync::Arc;
 
 use crate::PhysicalOptimizerRule;
 
-use datafusion_common::tree_node::{Transformed, TreeNode, TreeNodeRecursion};
 use datafusion_common::{config::ConfigOptions, Result};
-use datafusion_physical_expr::conjunction;
-use datafusion_physical_plan::filter::FilterExec;
+use datafusion_physical_expr::PhysicalExpr;
 use datafusion_physical_plan::filter_pushdown::{
-    FilterDescription, FilterPushdownResult, FilterPushdownSupport,
+    ChildPushdownResult, FilterPushdownPropagation, PredicateSupport, PredicateSupports,
 };
-use datafusion_physical_plan::tree_node::PlanContext;
-use datafusion_physical_plan::ExecutionPlan;
+use datafusion_physical_plan::{with_new_children_if_necessary, ExecutionPlan};
+
+use itertools::izip;
 
 /// Attempts to recursively push given filters from the top of the tree into leafs.
 ///
 /// # Default Implementation
 ///
-/// The default implementation in [`ExecutionPlan::try_pushdown_filters`] is a no-op
-/// that assumes that:
+/// The default implementation in [`ExecutionPlan::gather_filters_for_pushdown`]
+/// and [`ExecutionPlan::handle_child_pushdown_result`] assumes that:
 ///
-/// * Parent filters can't be passed onto children.
-/// * This node has no filters to contribute.
+/// * Parent filters can't be passed onto children (determined by [`ExecutionPlan::gather_filters_for_pushdown`])
+/// * This node has no filters to contribute (determined by [`ExecutionPlan::gather_filters_for_pushdown`]).
+/// * Any filters that could not be pushed down to the children are marked as unsupported (determined by [`ExecutionPlan::handle_child_pushdown_result`]).
 ///
 /// # Example: Push filter into a `DataSourceExec`
 ///
@@ -240,7 +240,7 @@ use datafusion_physical_plan::ExecutionPlan;
 /// The point here is that:
 /// 1. We cannot push down `sum > 10` through the [`AggregateExec`] node into the `DataSourceExec` node.
 ///    Any filters above the [`AggregateExec`] node are not pushed down.
-///    This is determined by calling [`ExecutionPlan::try_pushdown_filters`] on the [`AggregateExec`] node.
+///    This is determined by calling [`ExecutionPlan::gather_filters_for_pushdown`] on the [`AggregateExec`] node.
 /// 2. We need to keep recursing into the tree so that we can discover the other [`FilterExec`] node and push
 ///    down the `id=1` filter.
 ///
@@ -362,47 +362,29 @@ use datafusion_physical_plan::ExecutionPlan;
 /// [`ProjectionExec`]: datafusion_physical_plan::projection::ProjectionExec
 /// [`AggregateExec`]: datafusion_physical_plan::aggregates::AggregateExec
 #[derive(Debug)]
-pub struct PushdownFilter {}
+pub struct FilterPushdown {}
 
-impl Default for PushdownFilter {
+impl FilterPushdown {
+    pub fn new() -> Self {
+        Self {}
+    }
+}
+
+impl Default for FilterPushdown {
     fn default() -> Self {
         Self::new()
     }
 }
 
-pub type FilterDescriptionContext = PlanContext<FilterDescription>;
-
-impl PhysicalOptimizerRule for PushdownFilter {
+impl PhysicalOptimizerRule for FilterPushdown {
     fn optimize(
         &self,
         plan: Arc<dyn ExecutionPlan>,
         config: &ConfigOptions,
     ) -> Result<Arc<dyn ExecutionPlan>> {
-        let context = FilterDescriptionContext::new_default(plan);
-
-        context
-            .transform_up(|node| {
-                if node.plan.as_any().downcast_ref::<FilterExec>().is_some() {
-                    let initial_plan = Arc::clone(&node.plan);
-                    let mut accept_updated = false;
-                    let updated_node = node.transform_down(|filter_node| {
-                        Self::try_pushdown(filter_node, config, &mut accept_updated)
-                    });
-
-                    if accept_updated {
-                        updated_node
-                    } else {
-                        Ok(Transformed::no(FilterDescriptionContext::new_default(
-                            initial_plan,
-                        )))
-                    }
-                }
-                // Other filter introducing operators extends here
-                else {
-                    Ok(Transformed::no(node))
-                }
-            })
-            .map(|updated| updated.data.plan)
+        Ok(push_down_filters(Arc::clone(&plan), vec![], config)?
+            .updated_node
+            .unwrap_or(plan))
     }
 
     fn name(&self) -> &str {
@@ -414,122 +396,146 @@ impl PhysicalOptimizerRule for PushdownFilter {
     }
 }
 
-impl PushdownFilter {
-    pub fn new() -> Self {
-        Self {}
-    }
+/// Support state of each predicate for the children of the node.
+/// These predicates are coming from the parent node.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum ParentPredicateStates {
+    NoChildren,
+    Unsupported,
+    Supported,
+}
 
-    fn try_pushdown(
-        mut node: FilterDescriptionContext,
-        config: &ConfigOptions,
-        accept_updated: &mut bool,
-    ) -> Result<Transformed<FilterDescriptionContext>> {
-        let initial_description = FilterDescription {
-            filters: node.data.take_description(),
-        };
+fn push_down_filters(
+    node: Arc<dyn ExecutionPlan>,
+    parent_predicates: Vec<Arc<dyn PhysicalExpr>>,
+    config: &ConfigOptions,
+) -> Result<FilterPushdownPropagation<Arc<dyn ExecutionPlan>>> {
+    // If the node has any child, these will be rewritten as supported or unsupported
+    let mut parent_predicates_pushdown_states =
+        vec![ParentPredicateStates::NoChildren; parent_predicates.len()];
+    let mut self_filters_pushdown_supports = vec![];
+    let mut new_children = vec![];
 
-        let FilterPushdownResult {
-            support,
-            remaining_description,
-        } = node
-            .plan
-            .try_pushdown_filters(initial_description, config)?;
+    let children = node.children();
+    let filter_description =
+        node.gather_filters_for_pushdown(parent_predicates.clone(), config)?;
 
-        match support {
-            FilterPushdownSupport::Supported {
-                mut child_descriptions,
-                op,
-                revisit,
-            } => {
-                if revisit {
-                    // This check handles cases where the current operator is entirely removed
-                    // from the plan and replaced with its child. In such cases, to not skip
-                    // over the new node, we need to explicitly re-apply this pushdown logic
-                    // to the new node.
-                    //
-                    // TODO: If TreeNodeRecursion supports a Revisit mechanism in the future,
-                    //       this manual recursion could be removed.
+    for (child, parent_filters, self_filters) in izip!(
+        children,
+        filter_description.parent_filters(),
+        filter_description.self_filters()
+    ) {
+        // Here, `parent_filters` are the predicates which are provided by the parent node of
+        // the current node, and tried to be pushed down over the child which the loop points
+        // currently. `self_filters` are the predicates which are provided by the current node,
+        // and tried to be pushed down over the child similarly.
 
-                    // If the operator is removed, it should not leave any filters as remaining
-                    debug_assert!(remaining_description.filters.is_empty());
-                    // Operators having 2 children cannot be removed
-                    debug_assert_eq!(child_descriptions.len(), 1);
-                    debug_assert_eq!(node.children.len(), 1);
+        let num_self_filters = self_filters.len();
+        let mut parent_supported_predicate_indices = vec![];
+        let mut all_predicates = self_filters;
 
-                    node.plan = op;
-                    node.data = child_descriptions.swap_remove(0);
-                    node.children = node.children.swap_remove(0).children;
-                    Self::try_pushdown(node, config, accept_updated)
-                } else {
-                    if remaining_description.filters.is_empty() {
-                        // Filter can be pushed down safely
-                        node.plan = op;
-                        if node.children.is_empty() {
-                            *accept_updated = true;
-                        } else {
-                            for (child, descr) in
-                                node.children.iter_mut().zip(child_descriptions)
-                            {
-                                child.data = descr;
-                            }
-                        }
-                    } else {
-                        // Filter cannot be pushed down
-                        node = insert_filter_exec(
-                            node,
-                            child_descriptions,
-                            remaining_description,
-                        )?;
+        // Iterate over each predicate coming from the parent
+        for (idx, filter) in parent_filters.into_iter().enumerate() {
+            // Check if we can push this filter down to our child.
+            // These supports are defined in `gather_filters_for_pushdown()`
+            match filter {
+                PredicateSupport::Supported(predicate) => {
+                    // Queue this filter up for pushdown to this child
+                    all_predicates.push(predicate);
+                    parent_supported_predicate_indices.push(idx);
+                    // Mark this filter as supported by our children if no child has marked it as unsupported
+                    if parent_predicates_pushdown_states[idx]
+                        != ParentPredicateStates::Unsupported
+                    {
+                        parent_predicates_pushdown_states[idx] =
+                            ParentPredicateStates::Supported;
                     }
-                    Ok(Transformed::yes(node))
                 }
-            }
-            FilterPushdownSupport::NotSupported => {
-                if remaining_description.filters.is_empty() {
-                    Ok(Transformed {
-                        data: node,
-                        transformed: false,
-                        tnr: TreeNodeRecursion::Stop,
-                    })
-                } else {
-                    node = insert_filter_exec(
-                        node,
-                        vec![FilterDescription::empty(); 1],
-                        remaining_description,
-                    )?;
-                    Ok(Transformed {
-                        data: node,
-                        transformed: true,
-                        tnr: TreeNodeRecursion::Stop,
-                    })
+                PredicateSupport::Unsupported(_) => {
+                    // Mark as unsupported by our children
+                    parent_predicates_pushdown_states[idx] =
+                        ParentPredicateStates::Unsupported;
                 }
             }
         }
-    }
-}
 
-fn insert_filter_exec(
-    node: FilterDescriptionContext,
-    mut child_descriptions: Vec<FilterDescription>,
-    remaining_description: FilterDescription,
-) -> Result<FilterDescriptionContext> {
-    let mut new_child_node = node;
+        // Any filters that could not be pushed down to a child are marked as not-supported to our parents
+        let result = push_down_filters(Arc::clone(child), all_predicates, config)?;
 
-    // Filter has one child
-    if !child_descriptions.is_empty() {
-        debug_assert_eq!(child_descriptions.len(), 1);
-        new_child_node.data = child_descriptions.swap_remove(0);
-    }
-    let new_plan = Arc::new(FilterExec::try_new(
-        conjunction(remaining_description.filters),
-        Arc::clone(&new_child_node.plan),
-    )?);
-    let new_children = vec![new_child_node];
-    let new_data = FilterDescription::empty();
+        if let Some(new_child) = result.updated_node {
+            // If we have a filter pushdown result, we need to update our children
+            new_children.push(new_child);
+        } else {
+            // If we don't have a filter pushdown result, we need to update our children
+            new_children.push(Arc::clone(child));
+        }
+
+        // Our child doesn't know the difference between filters that were passed down
+        // from our parents and filters that the current node injected. We need to de-entangle
+        // this since we do need to distinguish between them.
+        let mut all_filters = result.filters.into_inner();
+        let parent_predicates = all_filters.split_off(num_self_filters);
+        let self_predicates = all_filters;
+        self_filters_pushdown_supports.push(PredicateSupports::new(self_predicates));
 
-    Ok(FilterDescriptionContext::new(
-        new_plan,
-        new_data,
-        new_children,
-    ))
+        for (idx, result) in parent_supported_predicate_indices
+            .iter()
+            .zip(parent_predicates)
+        {
+            let current_node_state = match result {
+                PredicateSupport::Supported(_) => ParentPredicateStates::Supported,
+                PredicateSupport::Unsupported(_) => ParentPredicateStates::Unsupported,
+            };
+            match (current_node_state, parent_predicates_pushdown_states[*idx]) {
+                (r, ParentPredicateStates::NoChildren) => {
+                    // If we have no result, use the current state from this child
+                    parent_predicates_pushdown_states[*idx] = r;
+                }
+                (ParentPredicateStates::Supported, ParentPredicateStates::Supported) => {
+                    // If the current child and all previous children are supported,
+                    // the filter continues to support it
+                    parent_predicates_pushdown_states[*idx] =
+                        ParentPredicateStates::Supported;
+                }
+                _ => {
+                    // Either the current child or a previous child marked this filter as unsupported
+                    parent_predicates_pushdown_states[*idx] =
+                        ParentPredicateStates::Unsupported;
+                }
+            }
+        }
+    }
+    // Re-create this node with new children
+    let updated_node = with_new_children_if_necessary(Arc::clone(&node), new_children)?;
+    // Remap the result onto the parent filters as they were given to us.
+    // Any filters that were not pushed down to any children are marked as unsupported.
+    let parent_pushdown_result = PredicateSupports::new(
+        parent_predicates_pushdown_states
+            .into_iter()
+            .zip(parent_predicates)
+            .map(|(state, filter)| match state {
+                ParentPredicateStates::NoChildren => {
+                    PredicateSupport::Unsupported(filter)
+                }
+                ParentPredicateStates::Unsupported => {
+                    PredicateSupport::Unsupported(filter)
+                }
+                ParentPredicateStates::Supported => PredicateSupport::Supported(filter),
+            })
+            .collect(),
+    );
+    // Check what the current node wants to do given the result of pushdown to it's children
+    let mut res = updated_node.handle_child_pushdown_result(
+        ChildPushdownResult {
+            parent_filters: parent_pushdown_result,
+            self_filters: self_filters_pushdown_supports,
+        },
+        config,
+    )?;
+    // Compare pointers for new_node and node, if they are different we must replace
+    // ourselves because of changes in our children.
+    if res.updated_node.is_none() && !Arc::ptr_eq(&updated_node, &node) {
+        res.updated_node = Some(updated_node)
+    }
+    Ok(res)
 }
diff --git a/datafusion/physical-optimizer/src/lib.rs b/datafusion/physical-optimizer/src/lib.rs
index 57dac21b6eeed..5a43d7118d638 100644
--- a/datafusion/physical-optimizer/src/lib.rs
+++ b/datafusion/physical-optimizer/src/lib.rs
@@ -29,6 +29,7 @@ pub mod coalesce_batches;
 pub mod combine_partial_final_agg;
 pub mod enforce_distribution;
 pub mod enforce_sorting;
+pub mod filter_pushdown;
 pub mod join_selection;
 pub mod limit_pushdown;
 pub mod limited_distinct_aggregation;
@@ -36,7 +37,6 @@ pub mod optimizer;
 pub mod output_requirements;
 pub mod projection_pushdown;
 pub mod pruning;
-pub mod push_down_filter;
 pub mod sanity_checker;
 pub mod topk_aggregation;
 pub mod update_aggr_exprs;
diff --git a/datafusion/physical-optimizer/src/optimizer.rs b/datafusion/physical-optimizer/src/optimizer.rs
index d4ff7d6b9e153..78d3e2ad88731 100644
--- a/datafusion/physical-optimizer/src/optimizer.rs
+++ b/datafusion/physical-optimizer/src/optimizer.rs
@@ -25,12 +25,12 @@ use crate::coalesce_batches::CoalesceBatches;
 use crate::combine_partial_final_agg::CombinePartialFinalAggregate;
 use crate::enforce_distribution::EnforceDistribution;
 use crate::enforce_sorting::EnforceSorting;
+use crate::filter_pushdown::FilterPushdown;
 use crate::join_selection::JoinSelection;
 use crate::limit_pushdown::LimitPushdown;
 use crate::limited_distinct_aggregation::LimitedDistinctAggregation;
 use crate::output_requirements::OutputRequirements;
 use crate::projection_pushdown::ProjectionPushdown;
-use crate::push_down_filter::PushdownFilter;
 use crate::sanity_checker::SanityCheckPlan;
 use crate::topk_aggregation::TopKAggregation;
 use crate::update_aggr_exprs::OptimizeAggregateOrder;
@@ -125,7 +125,7 @@ impl PhysicalOptimizer {
             // The FilterPushdown rule tries to push down filters as far as it can.
             // For example, it will push down filtering from a `FilterExec` to
             // a `DataSourceExec`, or from a `TopK`'s current state to a `DataSourceExec`.
-            Arc::new(PushdownFilter::new()),
+            Arc::new(FilterPushdown::new()),
             // The LimitPushdown rule tries to push limits down as far as possible,
             // replacing operators with fetching variants, or adding limits
             // past operators that support limit pushdown.
diff --git a/datafusion/physical-plan/src/coalesce_batches.rs b/datafusion/physical-plan/src/coalesce_batches.rs
index 34b3f1b0241b3..a0dd7371b4a06 100644
--- a/datafusion/physical-plan/src/coalesce_batches.rs
+++ b/datafusion/physical-plan/src/coalesce_batches.rs
@@ -32,11 +32,12 @@ use arrow::datatypes::SchemaRef;
 use arrow::record_batch::RecordBatch;
 use datafusion_common::Result;
 use datafusion_execution::TaskContext;
+use datafusion_physical_expr::PhysicalExpr;
 
 use crate::coalesce::{BatchCoalescer, CoalescerState};
 use crate::execution_plan::CardinalityEffect;
 use crate::filter_pushdown::{
-    filter_pushdown_transparent, FilterDescription, FilterPushdownResult,
+    ChildPushdownResult, FilterDescription, FilterPushdownPropagation,
 };
 use datafusion_common::config::ConfigOptions;
 use futures::ready;
@@ -226,14 +227,22 @@ impl ExecutionPlan for CoalesceBatchesExec {
         CardinalityEffect::Equal
     }
 
-    fn try_pushdown_filters(
+    fn gather_filters_for_pushdown(
         &self,
-        fd: FilterDescription,
+        parent_filters: Vec<Arc<dyn PhysicalExpr>>,
         _config: &ConfigOptions,
-    ) -> Result<FilterPushdownResult<Arc<dyn ExecutionPlan>>> {
-        Ok(filter_pushdown_transparent::<Arc<dyn ExecutionPlan>>(
-            Arc::new(self.clone()),
-            fd,
+    ) -> Result<FilterDescription> {
+        Ok(FilterDescription::new_with_child_count(1)
+            .all_parent_filters_supported(parent_filters))
+    }
+
+    fn handle_child_pushdown_result(
+        &self,
+        child_pushdown_result: ChildPushdownResult,
+        _config: &ConfigOptions,
+    ) -> Result<FilterPushdownPropagation<Arc<dyn ExecutionPlan>>> {
+        Ok(FilterPushdownPropagation::transparent(
+            child_pushdown_result,
         ))
     }
 }
diff --git a/datafusion/physical-plan/src/execution_plan.rs b/datafusion/physical-plan/src/execution_plan.rs
index 9551c2b1743e1..b81b3c8beeac1 100644
--- a/datafusion/physical-plan/src/execution_plan.rs
+++ b/datafusion/physical-plan/src/execution_plan.rs
@@ -17,7 +17,7 @@
 
 pub use crate::display::{DefaultDisplay, DisplayAs, DisplayFormatType, VerboseDisplay};
 use crate::filter_pushdown::{
-    filter_pushdown_not_supported, FilterDescription, FilterPushdownResult,
+    ChildPushdownResult, FilterDescription, FilterPushdownPropagation,
 };
 pub use crate::metrics::Metric;
 pub use crate::ordering::InputOrderMode;
@@ -491,39 +491,60 @@ pub trait ExecutionPlan: Debug + DisplayAs + Send + Sync {
         Ok(None)
     }
 
-    /// Attempts to recursively push given filters from the top of the tree into leafs.
-    ///
-    /// This is used for various optimizations, such as:
-    ///
-    /// * Pushing down filters into scans in general to minimize the amount of data that needs to be materialzied.
-    /// * Pushing down dynamic filters from operators like TopK and Joins into scans.
-    ///
-    /// Generally the further down (closer to leaf nodes) that filters can be pushed, the better.
-    ///
-    /// Consider the case of a query such as `SELECT * FROM t WHERE a = 1 AND b = 2`.
-    /// With no filter pushdown the scan needs to read and materialize all the data from `t` and then filter based on `a` and `b`.
-    /// With filter pushdown into the scan it can first read only `a`, then `b` and keep track of
-    /// which rows match the filter.
-    /// Then only for rows that match the filter does it have to materialize the rest of the columns.
-    ///
-    /// # Default Implementation
-    ///
-    /// The default implementation assumes:
-    /// * Parent filters can't be passed onto children.
-    /// * This node has no filters to contribute.
-    ///
-    /// # Implementation Notes
-    ///
-    /// Most of the actual logic is implemented as a Physical Optimizer rule.
-    /// See [`PushdownFilter`] for more details.
-    ///
-    /// [`PushdownFilter`]: https://docs.rs/datafusion/latest/datafusion/physical_optimizer/filter_pushdown/struct.PushdownFilter.html
-    fn try_pushdown_filters(
+    /// Collect filters that this node can push down to its children.
+    /// Filters that are being pushed down from parents are passed in,
+    /// and the node may generate additional filters to push down.
+    /// For example, given the plan FilterExec -> HashJoinExec -> DataSourceExec,
+    /// what will happen is that we recurse down the plan calling `ExecutionPlan::gather_filters_for_pushdown`:
+    /// 1. `FilterExec::gather_filters_for_pushdown` is called with no parent
+    ///    filters so it only returns that `FilterExec` wants to push down its own predicate.
+    /// 2. `HashJoinExec::gather_filters_for_pushdown` is called with the filter from
+    ///    `FilterExec`, which it only allows to push down to one side of the join (unless it's on the join key)
+    ///    but it also adds its own filters (e.g. pushing down a bloom filter of the hash table to the scan side of the join).
+    /// 3. `DataSourceExec::gather_filters_for_pushdown` is called with both filters from `HashJoinExec`
+    ///    and `FilterExec`, however `DataSourceExec::gather_filters_for_pushdown` doesn't actually do anything
+    ///    since it has no children and no additional filters to push down.
+    ///    It's only once [`ExecutionPlan::handle_child_pushdown_result`] is called on `DataSourceExec` as we recurse
+    ///    up the plan that `DataSourceExec` can actually bind the filters.
+    ///
+    /// The default implementation bars all parent filters from being pushed down and adds no new filters.
+    /// This is the safest option, making filter pushdown opt-in on a per-node pasis.
+    fn gather_filters_for_pushdown(
+        &self,
+        parent_filters: Vec<Arc<dyn PhysicalExpr>>,
+        _config: &ConfigOptions,
+    ) -> Result<FilterDescription> {
+        Ok(
+            FilterDescription::new_with_child_count(self.children().len())
+                .all_parent_filters_unsupported(parent_filters),
+        )
+    }
+
+    /// Handle the result of a child pushdown.
+    /// This is called as we recurse back up the plan tree after recursing down and calling [`ExecutionPlan::gather_filters_for_pushdown`].
+    /// Once we know what the result of pushing down filters into children is we ask the current node what it wants to do with that result.
+    /// For a `DataSourceExec` that may be absorbing the filters to apply them during the scan phase
+    /// (also known as late materialization).
+    /// A `FilterExec` may absorb any filters its children could not absorb, or if there are no filters left it
+    /// may remove itself from the plan altogether.
+    /// It combines both [`ChildPushdownResult::parent_filters`] and [`ChildPushdownResult::self_filters`] into a single
+    /// predicate and replaces it's own predicate.
+    /// Then it passes [`PredicateSupport::Supported`] for each parent predicate to the parent.
+    /// A `HashJoinExec` may ignore the pushdown result since it needs to apply the filters as part of the join anyhow.
+    /// It passes [`ChildPushdownResult::parent_filters`] back up to it's parents wrapped in [`FilterPushdownPropagation::transparent`]
+    /// and [`ChildPushdownResult::self_filters`] is discarded.
+    ///
+    /// The default implementation is a no-op that passes the result of pushdown from the children to its parent.
+    ///
+    /// [`PredicateSupport::Supported`]: crate::filter_pushdown::PredicateSupport::Supported
+    fn handle_child_pushdown_result(
         &self,
-        fd: FilterDescription,
+        child_pushdown_result: ChildPushdownResult,
         _config: &ConfigOptions,
-    ) -> Result<FilterPushdownResult<Arc<dyn ExecutionPlan>>> {
-        Ok(filter_pushdown_not_supported(fd))
+    ) -> Result<FilterPushdownPropagation<Arc<dyn ExecutionPlan>>> {
+        Ok(FilterPushdownPropagation::transparent(
+            child_pushdown_result,
+        ))
     }
 }
 
diff --git a/datafusion/physical-plan/src/filter.rs b/datafusion/physical-plan/src/filter.rs
index 6df3e236a0ddc..9f5d9dc2984e0 100644
--- a/datafusion/physical-plan/src/filter.rs
+++ b/datafusion/physical-plan/src/filter.rs
@@ -16,6 +16,7 @@
 // under the License.
 
 use std::any::Any;
+use std::collections::HashMap;
 use std::pin::Pin;
 use std::sync::Arc;
 use std::task::{ready, Context, Poll};
@@ -27,7 +28,7 @@ use super::{
 use crate::common::can_project;
 use crate::execution_plan::CardinalityEffect;
 use crate::filter_pushdown::{
-    FilterDescription, FilterPushdownResult, FilterPushdownSupport,
+    ChildPushdownResult, FilterDescription, FilterPushdownPropagation, PredicateSupport,
 };
 use crate::projection::{
     make_with_child, try_embed_projection, update_expr, EmbeddedProjection,
@@ -44,24 +45,29 @@ use arrow::record_batch::RecordBatch;
 use datafusion_common::cast::as_boolean_array;
 use datafusion_common::config::ConfigOptions;
 use datafusion_common::stats::Precision;
+use datafusion_common::tree_node::{
+    Transformed, TransformedResult, TreeNode, TreeNodeRecursion,
+};
 use datafusion_common::{
     internal_err, plan_err, project_schema, DataFusionError, Result, ScalarValue,
 };
 use datafusion_execution::TaskContext;
 use datafusion_expr::Operator;
 use datafusion_physical_expr::equivalence::ProjectionMapping;
-use datafusion_physical_expr::expressions::{BinaryExpr, Column};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, Column};
 use datafusion_physical_expr::intervals::utils::check_support;
 use datafusion_physical_expr::utils::collect_columns;
 use datafusion_physical_expr::{
-    analyze, split_conjunction, AcrossPartitions, AnalysisContext, ConstExpr,
-    ExprBoundaries, PhysicalExpr,
+    analyze, conjunction, split_conjunction, AcrossPartitions, AnalysisContext,
+    ConstExpr, ExprBoundaries, PhysicalExpr,
 };
 
 use datafusion_physical_expr_common::physical_expr::fmt_sql;
 use futures::stream::{Stream, StreamExt};
 use log::trace;
 
+const FILTER_EXEC_DEFAULT_SELECTIVITY: u8 = 20;
+
 /// FilterExec evaluates a boolean predicate against all input batches to determine which rows to
 /// include in its output batches.
 #[derive(Debug, Clone)]
@@ -88,7 +94,7 @@ impl FilterExec {
     ) -> Result<Self> {
         match predicate.data_type(input.schema().as_ref())? {
             DataType::Boolean => {
-                let default_selectivity = 20;
+                let default_selectivity = FILTER_EXEC_DEFAULT_SELECTIVITY;
                 let cache = Self::compute_properties(
                     &input,
                     &predicate,
@@ -448,54 +454,126 @@ impl ExecutionPlan for FilterExec {
         try_embed_projection(projection, self)
     }
 
-    fn try_pushdown_filters(
+    fn gather_filters_for_pushdown(
         &self,
-        mut fd: FilterDescription,
+        parent_filters: Vec<Arc<dyn PhysicalExpr>>,
         _config: &ConfigOptions,
-    ) -> Result<FilterPushdownResult<Arc<dyn ExecutionPlan>>> {
-        // Extend the filter descriptions
-        fd.filters.push(Arc::clone(&self.predicate));
-
-        // Extract the information
-        let child_descriptions = vec![fd];
-        let remaining_description = FilterDescription { filters: vec![] };
-        let filter_input = Arc::clone(self.input());
+    ) -> Result<FilterDescription> {
+        let self_filter = Arc::clone(&self.predicate);
 
-        if let Some(projection_indices) = self.projection.as_ref() {
-            // Push the filters down, but leave a ProjectionExec behind, instead of the FilterExec
-            let filter_child_schema = filter_input.schema();
-            let proj_exprs = projection_indices
+        let parent_filters = if let Some(projection_indices) = self.projection.as_ref() {
+            // We need to invert the projection on any referenced columns in the filter
+            // Create a mapping from the output columns to the input columns (the inverse of the projection)
+            let inverse_projection = projection_indices
                 .iter()
-                .map(|p| {
-                    let field = filter_child_schema.field(*p).clone();
-                    (
-                        Arc::new(Column::new(field.name(), *p)) as Arc<dyn PhysicalExpr>,
-                        field.name().to_string(),
-                    )
+                .enumerate()
+                .map(|(i, &p)| (p, i))
+                .collect::<HashMap<_, _>>();
+            parent_filters
+                .into_iter()
+                .map(|f| {
+                    f.transform_up(|expr| {
+                        let mut res =
+                            if let Some(col) = expr.as_any().downcast_ref::<Column>() {
+                                let index = col.index();
+                                let index_in_input_schema =
+                                    inverse_projection.get(&index).ok_or_else(|| {
+                                        DataFusionError::Internal(format!(
+                                            "Column {} not found in projection",
+                                            index
+                                        ))
+                                    })?;
+                                Transformed::yes(Arc::new(Column::new(
+                                    col.name(),
+                                    *index_in_input_schema,
+                                )) as _)
+                            } else {
+                                Transformed::no(expr)
+                            };
+                        // Columns can only exist in the leaves, no need to try all nodes
+                        res.tnr = TreeNodeRecursion::Jump;
+                        Ok(res)
+                    })
+                    .data()
                 })
-                .collect::<Vec<_>>();
-            let projection_exec =
-                Arc::new(ProjectionExec::try_new(proj_exprs, filter_input)?) as _;
-
-            Ok(FilterPushdownResult {
-                support: FilterPushdownSupport::Supported {
-                    child_descriptions,
-                    op: projection_exec,
-                    revisit: false,
-                },
-                remaining_description,
-            })
+                .collect::<Result<Vec<_>>>()?
         } else {
-            // Pull out the FilterExec, and inform the rule as it should be re-run
-            Ok(FilterPushdownResult {
-                support: FilterPushdownSupport::Supported {
-                    child_descriptions,
-                    op: filter_input,
-                    revisit: true,
-                },
-                remaining_description,
-            })
+            parent_filters
+        };
+
+        Ok(FilterDescription::new_with_child_count(1)
+            .all_parent_filters_supported(parent_filters)
+            .with_self_filter(self_filter))
+    }
+
+    fn handle_child_pushdown_result(
+        &self,
+        mut child_pushdown_result: ChildPushdownResult,
+        _config: &ConfigOptions,
+    ) -> Result<FilterPushdownPropagation<Arc<dyn ExecutionPlan>>> {
+        assert_eq!(
+            child_pushdown_result.self_filters.len(),
+            1,
+            "FilterExec should only have one child"
+        );
+        assert_eq!(
+            child_pushdown_result.self_filters[0].len(),
+            1,
+            "FilterExec produces only one filter"
+        );
+
+        // We absorb any parent filters that were not handled by our children
+        let mut unhandled_filters =
+            child_pushdown_result.parent_filters.collect_unsupported();
+
+        let self_filters = child_pushdown_result
+            .self_filters
+            .swap_remove(0)
+            .into_inner()
+            .swap_remove(0);
+        if let PredicateSupport::Unsupported(expr) = self_filters {
+            unhandled_filters.push(expr);
         }
+
+        // If we have unhandled filters, we need to create a new FilterExec
+        let filter_input = Arc::clone(self.input());
+        let new_predicate = conjunction(unhandled_filters);
+        let new_exec = if new_predicate.eq(&lit(true)) {
+            // FilterExec is no longer needed, but we may need to leave a projection in place
+            match self.projection() {
+                Some(projection_indices) => {
+                    let filter_child_schema = filter_input.schema();
+                    let proj_exprs = projection_indices
+                        .iter()
+                        .map(|p| {
+                            let field = filter_child_schema.field(*p).clone();
+                            (
+                                Arc::new(Column::new(field.name(), *p))
+                                    as Arc<dyn PhysicalExpr>,
+                                field.name().to_string(),
+                            )
+                        })
+                        .collect::<Vec<_>>();
+                    Arc::new(ProjectionExec::try_new(proj_exprs, filter_input)?)
+                        as Arc<dyn ExecutionPlan>
+                }
+                None => {
+                    // No projection needed, just return the input
+                    filter_input
+                }
+            }
+        } else {
+            // Create a new FilterExec with the new predicate
+            Arc::new(
+                FilterExec::try_new(new_predicate, filter_input)?
+                    .with_default_selectivity(self.default_selectivity())?
+                    .with_projection(self.projection().cloned())?,
+            )
+        };
+        Ok(FilterPushdownPropagation {
+            filters: child_pushdown_result.parent_filters.make_supported(),
+            updated_node: Some(new_exec),
+        })
     }
 }
 
diff --git a/datafusion/physical-plan/src/filter_pushdown.rs b/datafusion/physical-plan/src/filter_pushdown.rs
index 38f5aef5923e1..0003fc9d7277f 100644
--- a/datafusion/physical-plan/src/filter_pushdown.rs
+++ b/datafusion/physical-plan/src/filter_pushdown.rs
@@ -16,80 +16,285 @@
 // under the License.
 
 use std::sync::Arc;
+use std::vec::IntoIter;
 
-use crate::ExecutionPlan;
 use datafusion_physical_expr_common::physical_expr::PhysicalExpr;
 
-#[derive(Clone, Debug)]
-pub struct FilterDescription {
-    /// Expressions coming from the parent nodes
-    pub filters: Vec<Arc<dyn PhysicalExpr>>,
+/// The result of a plan for pushing down a filter into a child node.
+/// This contains references to filters so that nodes can mutate a filter
+/// before pushing it down to a child node (e.g. to adjust a projection)
+/// or can directly take ownership of `Unsupported` filters that their children
+/// could not handle.
+#[derive(Debug, Clone)]
+pub enum PredicateSupport {
+    Supported(Arc<dyn PhysicalExpr>),
+    Unsupported(Arc<dyn PhysicalExpr>),
 }
 
-impl Default for FilterDescription {
-    fn default() -> Self {
-        Self::empty()
+/// A thin wrapper around [`PredicateSupport`]s that allows for easy collection of
+/// supported and unsupported filters. Inner vector stores each predicate for one node.
+#[derive(Debug, Clone)]
+pub struct PredicateSupports(Vec<PredicateSupport>);
+
+impl PredicateSupports {
+    /// Create a new FilterPushdowns with the given filters and their pushdown status.
+    pub fn new(pushdowns: Vec<PredicateSupport>) -> Self {
+        Self(pushdowns)
     }
-}
 
-impl FilterDescription {
-    /// Takes the filters out of the struct, leaving an empty vector in its place.
-    pub fn take_description(&mut self) -> Vec<Arc<dyn PhysicalExpr>> {
-        std::mem::take(&mut self.filters)
+    /// Create a new [`PredicateSupports`] with all filters as supported.
+    pub fn all_supported(filters: Vec<Arc<dyn PhysicalExpr>>) -> Self {
+        let pushdowns = filters
+            .into_iter()
+            .map(PredicateSupport::Supported)
+            .collect();
+        Self::new(pushdowns)
+    }
+
+    /// Create a new [`PredicateSupports`] with all filters as unsupported.
+    pub fn all_unsupported(filters: Vec<Arc<dyn PhysicalExpr>>) -> Self {
+        let pushdowns = filters
+            .into_iter()
+            .map(PredicateSupport::Unsupported)
+            .collect();
+        Self::new(pushdowns)
+    }
+
+    /// Transform all filters to supported, returning a new FilterPushdowns.
+    /// This does not modify the original [`PredicateSupports`].
+    pub fn make_supported(self) -> Self {
+        let pushdowns = self
+            .0
+            .into_iter()
+            .map(|f| match f {
+                PredicateSupport::Supported(expr) => PredicateSupport::Supported(expr),
+                PredicateSupport::Unsupported(expr) => PredicateSupport::Supported(expr),
+            })
+            .collect();
+        Self::new(pushdowns)
+    }
+
+    /// Collect unsupported filters into a Vec, without removing them from the original
+    /// [`PredicateSupports`].
+    pub fn collect_unsupported(&self) -> Vec<Arc<dyn PhysicalExpr>> {
+        self.0
+            .iter()
+            .filter_map(|f| match f {
+                PredicateSupport::Unsupported(expr) => Some(Arc::clone(expr)),
+                PredicateSupport::Supported(_) => None,
+            })
+            .collect()
+    }
+
+    /// Collect all filters into a Vec, without removing them from the original
+    /// FilterPushdowns.
+    pub fn collect_all(self) -> Vec<Arc<dyn PhysicalExpr>> {
+        self.0
+            .into_iter()
+            .map(|f| match f {
+                PredicateSupport::Supported(expr)
+                | PredicateSupport::Unsupported(expr) => expr,
+            })
+            .collect()
+    }
+
+    pub fn into_inner(self) -> Vec<PredicateSupport> {
+        self.0
     }
 
-    pub fn empty() -> FilterDescription {
-        Self { filters: vec![] }
+    /// Return an iterator over the inner `Vec<FilterPushdown>`.
+    pub fn iter(&self) -> impl Iterator<Item = &PredicateSupport> {
+        self.0.iter()
+    }
+
+    /// Return the number of filters in the inner `Vec<FilterPushdown>`.
+    pub fn len(&self) -> usize {
+        self.0.len()
+    }
+
+    /// Check if the inner `Vec<FilterPushdown>` is empty.
+    pub fn is_empty(&self) -> bool {
+        self.0.is_empty()
+    }
+}
+
+impl IntoIterator for PredicateSupports {
+    type Item = PredicateSupport;
+    type IntoIter = IntoIter<PredicateSupport>;
+
+    fn into_iter(self) -> Self::IntoIter {
+        self.0.into_iter()
     }
 }
 
-#[derive(Debug)]
-pub enum FilterPushdownSupport<T> {
-    Supported {
-        // Filter predicates which can be pushed down through the operator.
-        // NOTE that these are not placed into any operator.
-        child_descriptions: Vec<FilterDescription>,
-        // Possibly updated new operator
-        op: T,
-        // Whether the node is removed from the plan and the rule should be re-run manually
-        // on the new node.
-        // TODO: If TreeNodeRecursion supports Revisit mechanism, this flag can be removed
-        revisit: bool,
-    },
-    NotSupported,
+/// The result of pushing down filters into a child node.
+/// This is the result provided to nodes in [`ExecutionPlan::handle_child_pushdown_result`].
+/// Nodes process this result and convert it into a [`FilterPushdownPropagation`]
+/// that is returned to their parent.
+///
+/// [`ExecutionPlan::handle_child_pushdown_result`]: crate::ExecutionPlan::handle_child_pushdown_result
+#[derive(Debug, Clone)]
+pub struct ChildPushdownResult {
+    /// The combined result of pushing down each parent filter into each child.
+    /// For example, given the fitlers `[a, b]` and children `[1, 2, 3]` the matrix of responses:
+    ///
+    // | filter | child 1     | child 2   | child 3   | result      |
+    // |--------|-------------|-----------|-----------|-------------|
+    // | a      | Supported   | Supported | Supported | Supported   |
+    // | b      | Unsupported | Supported | Supported | Unsupported |
+    ///
+    /// That is: if any child marks a filter as unsupported or if the filter was not pushed
+    /// down into any child then the result is unsupported.
+    /// If at least one children and all children that received the filter mark it as supported
+    /// then the result is supported.
+    pub parent_filters: PredicateSupports,
+    /// The result of pushing down each filter this node provided into each of it's children.
+    /// This is not combined with the parent filters so that nodes can treat each child independently.
+    pub self_filters: Vec<PredicateSupports>,
 }
 
-#[derive(Debug)]
-pub struct FilterPushdownResult<T> {
-    pub support: FilterPushdownSupport<T>,
-    // Filters which cannot be pushed down through the operator.
-    // NOTE that caller of try_pushdown_filters() should handle these remanining predicates,
-    // possibly introducing a FilterExec on top of this operator.
-    pub remaining_description: FilterDescription,
+/// The result of pushing down filters into a node that it returns to its parent.
+/// This is what nodes return from [`ExecutionPlan::handle_child_pushdown_result`] to communicate
+/// to the optimizer:
+///
+/// 1. What to do with any parent filters that were not completely handled by the children.
+/// 2. If the node needs to be replaced in the execution plan with a new node or not.
+///
+/// [`ExecutionPlan::handle_child_pushdown_result`]: crate::ExecutionPlan::handle_child_pushdown_result
+#[derive(Debug, Clone)]
+pub struct FilterPushdownPropagation<T> {
+    pub filters: PredicateSupports,
+    pub updated_node: Option<T>,
 }
 
-pub fn filter_pushdown_not_supported<T>(
-    remaining_description: FilterDescription,
-) -> FilterPushdownResult<T> {
-    FilterPushdownResult {
-        support: FilterPushdownSupport::NotSupported,
-        remaining_description,
+impl<T> FilterPushdownPropagation<T> {
+    /// Create a new [`FilterPushdownPropagation`] that tells the parent node
+    /// that echoes back up to the parent the result of pushing down the filters
+    /// into the children.
+    pub fn transparent(child_pushdown_result: ChildPushdownResult) -> Self {
+        Self {
+            filters: child_pushdown_result.parent_filters,
+            updated_node: None,
+        }
     }
+
+    /// Create a new [`FilterPushdownPropagation`] that tells the parent node
+    /// that none of the parent filters were not pushed down.
+    pub fn unsupported(parent_filters: Vec<Arc<dyn PhysicalExpr>>) -> Self {
+        let unsupported = PredicateSupports::all_unsupported(parent_filters);
+        Self {
+            filters: unsupported,
+            updated_node: None,
+        }
+    }
+}
+
+#[derive(Debug, Clone)]
+struct ChildFilterDescription {
+    /// Description of which parent filters can be pushed down into this node.
+    /// Since we need to transmit filter pushdown results back to this node's parent
+    /// we need to track each parent filter for each child, even those that are unsupported / won't be pushed down.
+    /// We do this using a [`PredicateSupports`] which simplifies manipulating supported/unsupported filters.
+    parent_filters: PredicateSupports,
+    /// Description of which filters this node is pushing down to its children.
+    /// Since this is not transmitted back to the parents we can have variable sized inner arrays
+    /// instead of having to track supported/unsupported.
+    self_filters: Vec<Arc<dyn PhysicalExpr>>,
 }
 
-pub fn filter_pushdown_transparent<T>(
-    plan: Arc<dyn ExecutionPlan>,
-    fd: FilterDescription,
-) -> FilterPushdownResult<Arc<dyn ExecutionPlan>> {
-    let child_descriptions = vec![fd];
-    let remaining_description = FilterDescription::empty();
-
-    FilterPushdownResult {
-        support: FilterPushdownSupport::Supported {
-            child_descriptions,
-            op: plan,
-            revisit: false,
-        },
-        remaining_description,
+impl ChildFilterDescription {
+    fn new() -> Self {
+        Self {
+            parent_filters: PredicateSupports::new(vec![]),
+            self_filters: vec![],
+        }
+    }
+}
+
+#[derive(Debug, Clone)]
+pub struct FilterDescription {
+    /// A filter description for each child.
+    /// This includes which parent filters and which self filters (from the node in question)
+    /// will get pushed down to each child.
+    child_filter_descriptions: Vec<ChildFilterDescription>,
+}
+
+impl FilterDescription {
+    pub fn new_with_child_count(num_children: usize) -> Self {
+        Self {
+            child_filter_descriptions: vec![ChildFilterDescription::new(); num_children],
+        }
+    }
+
+    pub fn parent_filters(&self) -> Vec<PredicateSupports> {
+        self.child_filter_descriptions
+            .iter()
+            .map(|d| &d.parent_filters)
+            .cloned()
+            .collect()
+    }
+
+    pub fn self_filters(&self) -> Vec<Vec<Arc<dyn PhysicalExpr>>> {
+        self.child_filter_descriptions
+            .iter()
+            .map(|d| &d.self_filters)
+            .cloned()
+            .collect()
+    }
+
+    /// Mark all parent filters as supported for all children.
+    /// This is the case if the node allows filters to be pushed down through it
+    /// without any modification.
+    /// This broadcasts the parent filters to all children.
+    /// If handling of parent filters is different for each child then you should set the
+    /// field direclty.
+    /// For example, nodes like [`RepartitionExec`] that let filters pass through it transparently
+    /// use this to mark all parent filters as supported.
+    ///
+    /// [`RepartitionExec`]: crate::repartition::RepartitionExec
+    pub fn all_parent_filters_supported(
+        mut self,
+        parent_filters: Vec<Arc<dyn PhysicalExpr>>,
+    ) -> Self {
+        let supported = PredicateSupports::all_supported(parent_filters);
+        for child in &mut self.child_filter_descriptions {
+            child.parent_filters = supported.clone();
+        }
+        self
+    }
+
+    /// Mark all parent filters as unsupported for all children.
+    /// This is the case if the node does not allow filters to be pushed down through it.
+    /// This broadcasts the parent filters to all children.
+    /// If handling of parent filters is different for each child then you should set the
+    /// field direclty.
+    /// For example, the default implementation of filter pushdwon in [`ExecutionPlan`]
+    /// assumes that filters cannot be pushed down to children.
+    ///
+    /// [`ExecutionPlan`]: crate::ExecutionPlan
+    pub fn all_parent_filters_unsupported(
+        mut self,
+        parent_filters: Vec<Arc<dyn PhysicalExpr>>,
+    ) -> Self {
+        let unsupported = PredicateSupports::all_unsupported(parent_filters);
+        for child in &mut self.child_filter_descriptions {
+            child.parent_filters = unsupported.clone();
+        }
+        self
+    }
+
+    /// Add a filter generated / owned by the current node to be pushed down to all children.
+    /// This assumes that there is a single filter that that gets pushed down to all children
+    /// equally.
+    /// If there are multiple filters or pushdown to children is not homogeneous then
+    /// you should set the field directly.
+    /// For example:
+    /// - `TopK` uses this to push down a single filter to all children, it can use this method.
+    /// - `HashJoinExec` pushes down a filter only to the probe side, it cannot use this method.
+    pub fn with_self_filter(mut self, predicate: Arc<dyn PhysicalExpr>) -> Self {
+        for child in &mut self.child_filter_descriptions {
+            child.self_filters = vec![Arc::clone(&predicate)];
+        }
+        self
     }
 }
diff --git a/datafusion/physical-plan/src/repartition/mod.rs b/datafusion/physical-plan/src/repartition/mod.rs
index f7c4f7477f121..c86a37697a053 100644
--- a/datafusion/physical-plan/src/repartition/mod.rs
+++ b/datafusion/physical-plan/src/repartition/mod.rs
@@ -54,7 +54,7 @@ use datafusion_physical_expr::{EquivalenceProperties, PhysicalExpr};
 use datafusion_physical_expr_common::sort_expr::LexOrdering;
 
 use crate::filter_pushdown::{
-    filter_pushdown_transparent, FilterDescription, FilterPushdownResult,
+    ChildPushdownResult, FilterDescription, FilterPushdownPropagation,
 };
 use futures::stream::Stream;
 use futures::{FutureExt, StreamExt, TryStreamExt};
@@ -743,14 +743,22 @@ impl ExecutionPlan for RepartitionExec {
         )?)))
     }
 
-    fn try_pushdown_filters(
+    fn gather_filters_for_pushdown(
         &self,
-        fd: FilterDescription,
+        parent_filters: Vec<Arc<dyn PhysicalExpr>>,
         _config: &ConfigOptions,
-    ) -> Result<FilterPushdownResult<Arc<dyn ExecutionPlan>>> {
-        Ok(filter_pushdown_transparent::<Arc<dyn ExecutionPlan>>(
-            Arc::new(self.clone()),
-            fd,
+    ) -> Result<FilterDescription> {
+        Ok(FilterDescription::new_with_child_count(1)
+            .all_parent_filters_supported(parent_filters))
+    }
+
+    fn handle_child_pushdown_result(
+        &self,
+        child_pushdown_result: ChildPushdownResult,
+        _config: &ConfigOptions,
+    ) -> Result<FilterPushdownPropagation<Arc<dyn ExecutionPlan>>> {
+        Ok(FilterPushdownPropagation::transparent(
+            child_pushdown_result,
         ))
     }
 }
diff --git a/datafusion/sqllogictest/test_files/parquet_filter_pushdown.slt b/datafusion/sqllogictest/test_files/parquet_filter_pushdown.slt
index 99fbb713646dd..252704f260b8d 100644
--- a/datafusion/sqllogictest/test_files/parquet_filter_pushdown.slt
+++ b/datafusion/sqllogictest/test_files/parquet_filter_pushdown.slt
@@ -186,14 +186,14 @@ COPY  (
 )  TO 'test_files/scratch/parquet_filter_pushdown/parquet_part_test/part=c/file.parquet'
 STORED AS PARQUET;
 
-query TT rowsort
-select * from t_pushdown where part == val
+query TT
+select * from t_pushdown where part == val order by part, val;
 ----
 a a
 b b
 
 query TT
-select * from t_pushdown where part != val
+select * from t_pushdown where part != val order by part, val;
 ----
 xyz c
 


Reviews:
user: login: adriangb
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-22T18:05:13Z
user: login: alamb
body: Thank you @adriangb and @berkaysynnada  

I am approving this PR because i think the plans that come out are better (filters stay in the same evaluation order), though I do think that could be done more simply. 

I do think it would be really good to avoid splitting the logic into two in-sync methods on `ExecutionPlan` (see comments)

I am mostly +0 on the code reorganization (I don't think it is any better or worse than what is on main)

Thank you both for working on this feature and I apologize for the late review
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T12:50:14Z
user: login: adriangb
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T12:53:24Z
user: login: adriangb
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T13:04:30Z
user: login: adriangb
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T14:31:01Z
user: login: alamb
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T21:07:36Z
user: login: berkaysynnada
body: as @alamb has approved it, I don't think he has a reason to keep this open.

Thank you @adriangb for the detailed explanation. I was just concerned about the complexity, but as you said, as we progress, things will become more familiar and perhaps we will see different angles to make it more simple. No need to stuck in here.

I see a questionable change in your last commit. After your reply, I'll merge this
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T08:32:50Z
user: login: adriangb
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T09:45:12Z

Review comments:
diff_hunk: @@ -438,55 +440,112 @@ impl ExecutionPlan for FilterExec {
         try_embed_projection(projection, self)
     }
 
-    fn try_pushdown_filters(
+    fn gather_filters_for_pushdown(
         &self,
-        mut fd: FilterDescription,
+        parent_filters: &[Arc<dyn PhysicalExpr>],
         _config: &ConfigOptions,
-    ) -> Result<FilterPushdownResult<Arc<dyn ExecutionPlan>>> {
-        // Extend the filter descriptions
-        fd.filters.push(Arc::clone(&self.predicate));
-
-        // Extract the information
-        let child_descriptions = vec![fd];
-        let remaining_description = FilterDescription { filters: vec![] };
-        let filter_input = Arc::clone(self.input());
-
+    ) -> Result<FilterPushdownPlan> {
         if let Some(projection_indices) = self.projection.as_ref() {
-            // Push the filters down, but leave a ProjectionExec behind, instead of the FilterExec
-            let filter_child_schema = filter_input.schema();
-            let proj_exprs = projection_indices
+            // We need to invert the projection on any referenced columns in the filter
+            // Create a mapping from the output columns to the input columns (the inverse of the projection)
+            let inverse_projection = projection_indices
                 .iter()
-                .map(|p| {
-                    let field = filter_child_schema.field(*p).clone();
-                    (
-                        Arc::new(Column::new(field.name(), *p)) as Arc<dyn PhysicalExpr>,
-                        field.name().to_string(),
-                    )
+                .enumerate()
+                .map(|(i, &p)| (i, p))
+                .collect::<HashMap<_, _>>();
+            let predicate = Arc::clone(&self.predicate)
+                .transform_up(|expr| {
+                    if let Some(col) = expr.as_any().downcast_ref::<Column>() {
+                        let index = col.index();
+                        let index_in_input_schema =
+                            inverse_projection.get(&index).ok_or_else(|| {
+                                DataFusionError::Internal(format!(
+                                    "Column {} not found in projection",
+                                    index
+                                ))
+                            })?;
+                        Ok(Transformed::yes(Arc::new(Column::new(
+                            col.name(),
+                            *index_in_input_schema,
+                        )) as _))
+                    } else {
+                        Ok(Transformed::no(expr))
+                    }
path: datafusion/physical-plan/src/filter.rs
user: login: adriangb
body: There's a bug with this inverting the projection logic that I haven't been able to spot... will continue later
created_at: 2025-04-22T18:05:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -154,29 +153,25 @@ impl FileSource for TestSource {
 
     fn try_pushdown_filters(
         &self,
-        mut fd: FilterDescription,
+        filters: &[Arc<dyn PhysicalExpr>],
path: datafusion/core/tests/physical_optimizer/push_down_filter.rs
user: login: alamb
body: I would personally prefer keeping a struct as an argument as it
1. Is easier to document / explain what the structures mean via comments
2. it is clearer in the struct definition what types of operations (e.g. append only vs swap, etc) are done
3. It is easier to add new fields if needed

Basically,  something like

```rust
pub struct FilterDescription {
        filters: &[Arc<dyn PhysicalExpr>],
}
```
...

This is a preference thing and i can easily see alternative opinions
created_at: 2025-05-01T12:36:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -154,29 +153,25 @@ impl FileSource for TestSource {
 
     fn try_pushdown_filters(
         &self,
-        mut fd: FilterDescription,
+        filters: &[Arc<dyn PhysicalExpr>],
         config: &ConfigOptions,
-    ) -> Result<FilterPushdownResult<Arc<dyn FileSource>>> {
+    ) -> Result<FilterPushdownPropagation<Arc<dyn FileSource>>> {
+        let mut all_filters = filters.iter().map(Arc::clone).collect::<Vec<_>>();
path: datafusion/core/tests/physical_optimizer/push_down_filter.rs
user: login: alamb
body: For example, if this was in a struct, we could implement `clone()` for the struct and hide this complexity
created_at: 2025-05-01T12:38:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -278,7 +273,7 @@ fn test_filter_collapse() {
         -     DataSourceExec: file_groups={0 groups: []}, projection=[a, b, c], file_type=test, pushdown_supported=true
       output:
         Ok:
-          - DataSourceExec: file_groups={0 groups: []}, projection=[a, b, c], file_type=test, pushdown_supported=true, predicate=b@1 = bar AND a@0 = foo
+          - DataSourceExec: file_groups={0 groups: []}, projection=[a, b, c], file_type=test, pushdown_supported=true, predicate=a@0 = foo AND b@1 = bar
path: datafusion/core/tests/physical_optimizer/push_down_filter.rs
user: login: alamb
body: This actually looks like an improvement to me as now `a = foo` will be evaluated before `b=bar` as was done in the input plan. This might be important for short circuiting, perhaps

The prior version of this optimization seems to have reordered them
created_at: 2025-05-01T12:40:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -363,46 +362,139 @@ use datafusion_physical_plan::ExecutionPlan;
 /// [`AggregateExec`]: datafusion_physical_plan::aggregates::AggregateExec
 #[derive(Debug)]
 pub struct PushdownFilter {}
-
 impl Default for PushdownFilter {
     fn default() -> Self {
         Self::new()
     }
 }
 
-pub type FilterDescriptionContext = PlanContext<FilterDescription>;
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum FilterPushdownState {
path: datafusion/physical-optimizer/src/push_down_filter.rs
user: login: alamb
body: it would be nice to add some comments here 
created_at: 2025-05-01T12:41:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -363,46 +362,139 @@ use datafusion_physical_plan::ExecutionPlan;
 /// [`AggregateExec`]: datafusion_physical_plan::aggregates::AggregateExec
 #[derive(Debug)]
 pub struct PushdownFilter {}
-
 impl Default for PushdownFilter {
     fn default() -> Self {
         Self::new()
     }
 }
 
-pub type FilterDescriptionContext = PlanContext<FilterDescription>;
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum FilterPushdownState {
+    NoChildren,
+    Unsupported,
+    Supported,
+}
+
+fn push_down_filters(
+    node: &Arc<dyn ExecutionPlan>,
+    parent_filters: Vec<Arc<dyn PhysicalExpr>>,
+    config: &ConfigOptions,
+) -> Result<FilterPushdownPropagation<Arc<dyn ExecutionPlan>>> {
+    let pushdown_plan = node.gather_filters_for_pushdown(&parent_filters, config)?;
+    let children = node
+        .children()
+        .into_iter()
+        .map(Arc::clone)
+        .collect::<Vec<_>>();
+    let mut parent_pushdown_result =
path: datafusion/physical-optimizer/src/push_down_filter.rs
user: login: alamb
body: one idea to make this code easier to follow might be to break it up into multiple functions. Maybe even a struct or something that held the key state

```rust
pub struct SingleNodePushdown {
  pushdown_plan: ....
  new_children: Vec<>
}
...
```
created_at: 2025-05-01T12:44:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -471,39 +471,53 @@ pub trait ExecutionPlan: Debug + DisplayAs + Send + Sync {
         Ok(None)
     }
 
-    /// Attempts to recursively push given filters from the top of the tree into leafs.
-    ///
-    /// This is used for various optimizations, such as:
-    ///
-    /// * Pushing down filters into scans in general to minimize the amount of data that needs to be materialzied.
-    /// * Pushing down dynamic filters from operators like TopK and Joins into scans.
-    ///
-    /// Generally the further down (closer to leaf nodes) that filters can be pushed, the better.
-    ///
-    /// Consider the case of a query such as `SELECT * FROM t WHERE a = 1 AND b = 2`.
-    /// With no filter pushdown the scan needs to read and materialize all the data from `t` and then filter based on `a` and `b`.
-    /// With filter pushdown into the scan it can first read only `a`, then `b` and keep track of
-    /// which rows match the filter.
-    /// Then only for rows that match the filter does it have to materialize the rest of the columns.
-    ///
-    /// # Default Implementation
-    ///
-    /// The default implementation assumes:
-    /// * Parent filters can't be passed onto children.
-    /// * This node has no filters to contribute.
-    ///
-    /// # Implementation Notes
-    ///
-    /// Most of the actual logic is implemented as a Physical Optimizer rule.
-    /// See [`PushdownFilter`] for more details.
-    ///
-    /// [`PushdownFilter`]: https://docs.rs/datafusion/latest/datafusion/physical_optimizer/filter_pushdown/struct.PushdownFilter.html
-    fn try_pushdown_filters(
path: datafusion/physical-plan/src/execution_plan.rs
user: login: alamb
body: to be honest, I prefer the previous implementation with just a single method on ExectionPlan -- I think having two methods whose implementation must remain synchronized is more complicated to reason about and likely be harder to work with
created_at: 2025-05-01T12:47:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -17,79 +17,207 @@
 
 use std::sync::Arc;
 
-use crate::ExecutionPlan;
 use datafusion_physical_expr_common::physical_expr::PhysicalExpr;
 
-#[derive(Clone, Debug)]
-pub struct FilterDescription {
-    /// Expressions coming from the parent nodes
-    pub filters: Vec<Arc<dyn PhysicalExpr>>,
+/// The result of or a plan for pushing down a filter into a child node.
+/// This contains references to filters so that nodes can mutate a filter
+/// before pushing it down to a child node (e.g. to adjust a projection)
+/// or can directly take ownership of `Unsupported` filters that their children
+/// could not handle.
+#[derive(Debug, Clone)]
+pub enum FilterPushdown {
+    Supported(Arc<dyn PhysicalExpr>),
+    Unsupported(Arc<dyn PhysicalExpr>),
 }
 
-impl Default for FilterDescription {
-    fn default() -> Self {
-        Self::empty()
+/// A thin wrapper around [`FilterPushdown`]s that allows for easy collection of
+/// supported and unsupported filters.
+#[derive(Debug, Clone)]
+pub struct FilterPushdowns(Vec<FilterPushdown>);
+
+impl FilterPushdowns {
+    /// Create a new FilterPushdowns with the given filters and their pushdown status.
+    pub fn new(pushdowns: Vec<FilterPushdown>) -> Self {
+        Self(pushdowns)
+    }
+
+    /// Create a new FilterPushdowns with all filters as supported.
+    pub fn all_supported(filters: &[Arc<dyn PhysicalExpr>]) -> Self {
+        let pushdowns = filters
+            .iter()
+            .map(|f| FilterPushdown::Supported(Arc::clone(f)))
+            .collect();
+        Self::new(pushdowns)
+    }
+
+    /// Create a new FilterPushdowns with all filters as unsupported.
+    pub fn all_unsupported(filters: &[Arc<dyn PhysicalExpr>]) -> Self {
+        let pushdowns = filters
+            .iter()
+            .map(|f| FilterPushdown::Unsupported(Arc::clone(f)))
+            .collect();
+        Self::new(pushdowns)
+    }
+
+    /// Transform all filters to supported, returning a new FilterPushdowns.
+    /// This does not modify the original FilterPushdowns.
+    pub fn as_supported(&self) -> Self {
+        let pushdowns = self
+            .0
+            .iter()
+            .map(|f| match f {
+                FilterPushdown::Supported(expr) => {
+                    FilterPushdown::Supported(Arc::clone(expr))
+                }
+                FilterPushdown::Unsupported(expr) => {
+                    FilterPushdown::Supported(Arc::clone(expr))
+                }
+            })
+            .collect();
+        Self::new(pushdowns)
+    }
+
+    /// Collect unsupported filters into a Vec, without removing them from the original
+    /// FilterPushdowns.
+    pub fn collect_unsupported(&self) -> Vec<Arc<dyn PhysicalExpr>> {
+        self.0
+            .iter()
+            .filter_map(|f| match f {
+                FilterPushdown::Unsupported(expr) => Some(Arc::clone(expr)),
+                FilterPushdown::Supported(_) => None,
+            })
+            .collect()
+    }
+
+    /// Collect all filters as PhysicalExprs into a Vec, without removing them from the original
+    /// FilterPushdowns.
+    pub fn into_inner_filters(&self) -> Vec<Arc<dyn PhysicalExpr>> {
+        self.0
+            .iter()
+            .map(|f| match f {
+                FilterPushdown::Supported(expr) => Arc::clone(expr),
+                FilterPushdown::Unsupported(expr) => Arc::clone(expr),
+            })
+            .collect()
     }
-}
 
-impl FilterDescription {
-    /// Takes the filters out of the struct, leaving an empty vector in its place.
-    pub fn take_description(&mut self) -> Vec<Arc<dyn PhysicalExpr>> {
-        std::mem::take(&mut self.filters)
+    /// Return the inner `Vec<FilterPushdown>` without modifying the original FilterPushdowns.
+    pub fn into_inner(&self) -> Vec<FilterPushdown> {
+        self.0.clone()
     }
 
-    pub fn empty() -> FilterDescription {
-        Self { filters: vec![] }
+    /// Return an iterator over the inner `Vec<FilterPushdown>`.
+    pub fn iter(&self) -> impl Iterator<Item = &FilterPushdown> {
+        self.0.iter()
+    }
+
+    /// Return the number of filters in the inner `Vec<FilterPushdown>`.
+    pub fn len(&self) -> usize {
+        self.0.len()
+    }
+
+    /// Check if the inner `Vec<FilterPushdown>` is empty.
+    pub fn is_empty(&self) -> bool {
+        self.0.is_empty()
     }
 }
 
-#[derive(Debug)]
-pub enum FilterPushdownSupport<T> {
-    Supported {
-        // Filter predicates which can be pushed down through the operator.
-        // NOTE that these are not placed into any operator.
-        child_descriptions: Vec<FilterDescription>,
-        // Possibly updated new operator
-        op: T,
-        // Whether the node is removed from the plan and the rule should be re-run manually
-        // on the new node.
-        // TODO: If TreeNodeRecursion supports Revisit mechanism, this flag can be removed
-        revisit: bool,
-    },
-    NotSupported,
+/// The result of pushing down filters into a child node.
+/// This is the result provided to nodes in [`ExecutionPlan::handle_child_pushdown_result`].
+/// Nodes process this result and convert it into a [`FilterPushdownPropagation`]
+/// that is returned to their parent.
+///
+/// [`ExecutionPlan::handle_child_pushdown_result`]: crate::ExecutionPlan::handle_child_pushdown_result
+#[derive(Debug, Clone)]
+pub struct ChildPushdownResult {
+    /// The combined result of pushing down each parent filter into each child.
+    /// For example, given the fitlers `[a, b]` and children `[1, 2, 3]` the matrix of responses:
+    ///
+    // | filter | child 1     | child 2   | child 3   | result      |
+    // |--------|-------------|-----------|-----------|-------------|
+    // | a      | Supported   | Supported | Supported | Supported   |
+    // | b      | Unsupported | Supported | Supported | Unsupported |
+    ///
+    /// That is: if any child marks a filter as unsupported or if the filter was not pushed
+    /// down into any child then the result is unsupported.
+    /// If at least one children and all children that received the filter mark it as supported
+    /// then the result is supported.
+    pub parent_filters: FilterPushdowns,
+    /// The result of pushing down each filter this node provided into each of it's children.
+    /// This is not combined with the parent filters so that nodes can treat each child independently.
+    pub self_filters: Vec<FilterPushdowns>,
 }
 
-#[derive(Debug)]
-pub struct FilterPushdownResult<T> {
-    pub support: FilterPushdownSupport<T>,
-    // Filters which cannot be pushed down through the operator.
-    // NOTE that caller of try_pushdown_filters() should handle these remanining predicates,
-    // possibly introducing a FilterExec on top of this operator.
-    pub remaining_description: FilterDescription,
+/// The result of pushing down filters into a node that it returns to its parent.
path: datafusion/physical-plan/src/filter_pushdown.rs
user: login: alamb
body: this is great comments,
created_at: 2025-05-01T12:47:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -154,29 +153,25 @@ impl FileSource for TestSource {
 
     fn try_pushdown_filters(
         &self,
-        mut fd: FilterDescription,
+        filters: &[Arc<dyn PhysicalExpr>],
path: datafusion/core/tests/physical_optimizer/push_down_filter.rs
user: login: adriangb
body: Happy to do that 👍🏻 
created_at: 2025-05-01T12:53:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -471,39 +471,53 @@ pub trait ExecutionPlan: Debug + DisplayAs + Send + Sync {
         Ok(None)
     }
 
-    /// Attempts to recursively push given filters from the top of the tree into leafs.
-    ///
-    /// This is used for various optimizations, such as:
-    ///
-    /// * Pushing down filters into scans in general to minimize the amount of data that needs to be materialzied.
-    /// * Pushing down dynamic filters from operators like TopK and Joins into scans.
-    ///
-    /// Generally the further down (closer to leaf nodes) that filters can be pushed, the better.
-    ///
-    /// Consider the case of a query such as `SELECT * FROM t WHERE a = 1 AND b = 2`.
-    /// With no filter pushdown the scan needs to read and materialize all the data from `t` and then filter based on `a` and `b`.
-    /// With filter pushdown into the scan it can first read only `a`, then `b` and keep track of
-    /// which rows match the filter.
-    /// Then only for rows that match the filter does it have to materialize the rest of the columns.
-    ///
-    /// # Default Implementation
-    ///
-    /// The default implementation assumes:
-    /// * Parent filters can't be passed onto children.
-    /// * This node has no filters to contribute.
-    ///
-    /// # Implementation Notes
-    ///
-    /// Most of the actual logic is implemented as a Physical Optimizer rule.
-    /// See [`PushdownFilter`] for more details.
-    ///
-    /// [`PushdownFilter`]: https://docs.rs/datafusion/latest/datafusion/physical_optimizer/filter_pushdown/struct.PushdownFilter.html
-    fn try_pushdown_filters(
path: datafusion/physical-plan/src/execution_plan.rs
user: login: adriangb
body: Agreed I'd prefer a single method as well - but as per https://github.com/apache/datafusion/pull/15770#discussion_r2052311257 we probably would have had to add a new method to the existing implementation anyway. I don't see a way to have a single method without making it absurdly convoluted just to avoid 2 methods.

I'll also point out that there are multiple dimensions of filter pushdown for each node:
- does the node allow parent filters to be pushed down to it's children?
- does the node want to add any filters to be passed to it's children?
- does the node need to handle the result of pushdown?

Having two methods makes it easy to have canned implementations for each one of these independently and combine/compose them.

E.g.:
- TopK or HashJoin will override `gather_filters_for_pushdown` but not `handle_child_pushdown_result`
- FilterExec will override both
- DataSourceExec will override only `handle_child_pushdown_result`
- ProjectionExec will override only `gather_filters_for_pushdown`
created_at: 2025-05-01T13:04:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -154,29 +153,25 @@ impl FileSource for TestSource {
 
     fn try_pushdown_filters(
         &self,
-        mut fd: FilterDescription,
+        filters: &[Arc<dyn PhysicalExpr>],
         config: &ConfigOptions,
-    ) -> Result<FilterPushdownResult<Arc<dyn FileSource>>> {
+    ) -> Result<FilterPushdownPropagation<Arc<dyn FileSource>>> {
+        let mut all_filters = filters.iter().map(Arc::clone).collect::<Vec<_>>();
path: datafusion/core/tests/physical_optimizer/push_down_filter.rs
user: login: adriangb
body: Wouldn't the same be true if it was just `Vec<Arc<dyn PhysicalExpr>>`?

I tried implementing a struct with some helper methods but the thing is that for `ExecutionPlan` we have a slightly different flow than `DataSource` and `FileSource` (which are their own traits and hence need their own method) so re-using the struct becomes a bit wonky.
created_at: 2025-05-01T14:31:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -471,39 +471,53 @@ pub trait ExecutionPlan: Debug + DisplayAs + Send + Sync {
         Ok(None)
     }
 
-    /// Attempts to recursively push given filters from the top of the tree into leafs.
-    ///
-    /// This is used for various optimizations, such as:
-    ///
-    /// * Pushing down filters into scans in general to minimize the amount of data that needs to be materialzied.
-    /// * Pushing down dynamic filters from operators like TopK and Joins into scans.
-    ///
-    /// Generally the further down (closer to leaf nodes) that filters can be pushed, the better.
-    ///
-    /// Consider the case of a query such as `SELECT * FROM t WHERE a = 1 AND b = 2`.
-    /// With no filter pushdown the scan needs to read and materialize all the data from `t` and then filter based on `a` and `b`.
-    /// With filter pushdown into the scan it can first read only `a`, then `b` and keep track of
-    /// which rows match the filter.
-    /// Then only for rows that match the filter does it have to materialize the rest of the columns.
-    ///
-    /// # Default Implementation
-    ///
-    /// The default implementation assumes:
-    /// * Parent filters can't be passed onto children.
-    /// * This node has no filters to contribute.
-    ///
-    /// # Implementation Notes
-    ///
-    /// Most of the actual logic is implemented as a Physical Optimizer rule.
-    /// See [`PushdownFilter`] for more details.
-    ///
-    /// [`PushdownFilter`]: https://docs.rs/datafusion/latest/datafusion/physical_optimizer/filter_pushdown/struct.PushdownFilter.html
-    fn try_pushdown_filters(
path: datafusion/physical-plan/src/execution_plan.rs
user: login: alamb
body: I am convinced
created_at: 2025-05-01T21:07:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -191,6 +191,7 @@ impl<T> FilterPushdownPropagation<T> {
 
 #[derive(Debug, Clone)]
 pub struct FilterDescription {
+    num_children: usize,
path: datafusion/physical-plan/src/filter_pushdown.rs
user: login: berkaysynnada
body: the vector sizes give this count, is there a reason to keep this explicitly?
created_at: 2025-05-05T08:29:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -191,6 +191,7 @@ impl<T> FilterPushdownPropagation<T> {
 
 #[derive(Debug, Clone)]
 pub struct FilterDescription {
+    num_children: usize,
path: datafusion/physical-plan/src/filter_pushdown.rs
user: login: adriangb
body: Yes sorry forgot to implement it but my thought was that you pass in the size to `new()` and then if you add filters you're forced to either add the right number (one per child) or use the broadcast helper functions. I want to make it error if you do e.g. `FilterDescription::new(1).with_parent_filters(vec![vec![predicate], vec![predicate]])`. I've implemented an internal refactor.
created_at: 2025-05-05T09:45:12Z
author_association: CONTRIBUTOR

PR comments:
user: login: adriangb
created_at: 2025-04-22T20:54:03Z
body: cc @berkaysynnada 
author_association: CONTRIBUTOR
user: login: berkaysynnada
created_at: 2025-04-24T13:08:41Z
body:  > 1. Not requiring downcast matching of specific `ExecutionPlan`s (as discussed previously this could be resolved by adding a new method, but in the end both approaches require two methods).

We can actually do that with the current version in a very similar way, and I think the API should just look like:
```rust
/// Gets the fetch count for the operator. `None` means there is no fetch.
fn fetch(&self) -> Option<usize> {
    None
}
```
Similarly, we could just define `fn filter(&self) -> Option<Arc<dyn PhysicalExpr>>`

> 2. Doing less invasive modification of the plans (e.g. not popping and then re-inserting FilterExecs).

That’s kind of the nature of transform APIs. I don’t think it’s a big deal. When the algorithm encounters a FilterExec, it either tries to push it down, or takes a conservative path and only removes it when it’s certain that it should be removed. Either way, we're biasing the rule, and if the outcome doesn’t go our way, the cost is just a small overhead.

> 3. Reducing the algorithmic complexity of the optimizer rule (it's now a single pass over the tree for all cases, and avoids cloning unless strictly needed).

The filter() API also enables us to convert the current algorithm to a single-pass one.

> 4. Allowing plans to decide how any of the leftover filters from parents or that they generated should be handled (I think this will be useful for TopK).

That’s still possible with the current approach.

> 
> This is in some ways simpler and in some ways more complex than what we have right now. The difference is not large in terms of LOC, I think the rest boils down to personal preference.

What you’ve done here works very well, but it seems a bit more complicated, and I think folks who aren’t deep in this context might find it harder to follow. As I mentioned, I prefer to make things more explicit and easier to understand in these early implementation stages. Once the requirements and implementations are stable, we can shift focus to optimizations.

I'm planning to open a PR which brings in the revisit recursion variant and introducing the filter() API. With those, I hope to address your concerns. If I don't get something wrong, these issues don't block any implementation, but you are only worried about some design nuances and performance?
author_association: CONTRIBUTOR
user: login: adriangb
created_at: 2025-04-24T14:19:25Z
body: > As I mentioned, I prefer to make things more explicit and easier to understand in these early implementation stages. Once the requirements and implementations are stable, we can shift focus to optimizations.

I totally agree, we are 100% aligned on this. I think where we may disagree is what "simpler" means. It's obviously somewhat subjective but the objective measures I'm trying to look at are:
- Lines of code: both implementations seem to be about the same. If anything this implementations is less: this PR is currently +160 LOC, but I think that can be brought down with some cleanup, and it is introducing 1 new method that the current implementation would also have to contend with.
- Touch points: the current implementation requires knowledge of the helpers being used, e.g. to set the `revisit` parameter. Maybe some of that can be solved but I still feel like the public APIs are more inextricably linked to the implementation of the optimizer pass than the implementation in this PR is.
- Functionality: this PR solves the problems stated in the description that I think we both agree are real. Now as you say there may be other ways to solve those with the current implementation but I'd fear that makes the points above worse (more lines of code, more methods in the public APIs, more linkage between them). Especially if it's introducing both `fetch()` and ` filter()`.

> We can actually do that with the current version in a very similar way, and I think the API should just look like:
> Similarly, we could just define fn filter(&self) -> Option<Arc<dyn PhysicalExpr>>

I don't understand where these new methods would go or how they would help. It would be helpful to me if when methods were proposed the trait or struct they are intended to go on was included and maybe an example of them being called was as well.

> > Doing less invasive modification of the plans (e.g. not popping and then re-inserting FilterExecs).

> That’s kind of the nature of transform APIs. I don’t think it’s a big deal. When the algorithm encounters a FilterExec, it either tries to push it down, or takes a conservative path and only removes it when it’s certain that it should be removed. Either way, we're biasing the rule, and if the outcome doesn’t go our way, the cost is just a small overhead.

There have had to be efforts in the past to minimize the amount of cloning / copying going on during optimizer passes, especially since DataFusion does not have prepared statements they can end up being a non trivial portion of query time for systems with optimized caching, etc. And I'm not sure that I would call removing a node optimistically then putting it back in place more conservative than only removing it if we know that will be the final decision.


>> Allowing plans to decide how any of the leftover filters from parents or that they generated should be handled (I think this will be useful for TopK).

> That’s still possible with the current approach.

It's not clear to me how. It seems like the current approach hardcodes the behavior for FilterExec - we'd have to either hardcode it for `SortExec` (and later `HashJoin`, etc.) or add more methods to `ExecutionPlan` which will make the current implementation even more complex.

> If I don't get something wrong, these issues don't block any implementation, but you are only worried about some design nuances and performance?

This is blocking work to some extent in the sense that I've had to work around the limitations of the current implementation in the TopK filter pushdown and other open PRs.


Ultimately this is a working implementation that covers all cases and solves the current limitations. I do not understand the need to delay progress so we can shoehorn the implementation into using specific helper APIs for recursion that put limitations on how we structure our user facing APIs.
author_association: CONTRIBUTOR
user: login: berkaysynnada
created_at: 2025-04-24T16:31:43Z
body: I understand your concerns and motivations. Since you're leading this work, I'm happy to follow your direction. I'm sure we both aim to do what's right, even if our priorities and perspectives naturally differ.

I'd like to take one more day to push a commit to this PR, mainly focusing on stylistic and idiomatic improvements as well as some comments (so we can be confident that at least two people stand behind it 😄). If you disagree with any of the changes, please feel free to iterate on those points. Thanks for the work
author_association: CONTRIBUTOR
user: login: adriangb
created_at: 2025-04-24T16:49:56Z
body: Sounds good happy to wait for your suggestions. And we can always iterate again!
author_association: CONTRIBUTOR
user: login: berkaysynnada
created_at: 2025-05-01T14:10:02Z
body: Hi @alamb. I've a WIP commit for this PR. I'm trying to both address @adriangb concerns and needs, and at the same time trying to keep complexity at minimum and trying to make things similar to other DF patterns. I think I can finish it at the end of today. Thank you for the patience
author_association: CONTRIBUTOR
user: login: alamb
created_at: 2025-05-01T21:06:54Z
body: > Hi @alamb. I've a WIP commit for this PR. I'm trying to both address @adriangb concerns and needs, and at the same time trying to keep complexity at minimum and trying to make things similar to other DF patterns. I think I can finish it at the end of today. Thank you for the patience

Thank you for your work and help. I will not merge this PR until I hear from you
author_association: CONTRIBUTOR
user: login: berkaysynnada
created_at: 2025-05-02T11:11:13Z
body: Hi again @adriangb. I've sent the commit, but unfortunately I can't say I fully achieved what I had in mind. Most of the work are idiomatic changes, comments and naming -- I'd still love to hear your thoughts on them.

There are a few points I'm not completely comfortable with:

1) New ExecutionPlan APIs: We ended up introducing two new APIs, and I know we can't merge them into one. However, the existing ExecutionPlan APIs are general-purpose, self-explanatory, and simple, whereas these new ones feel harder to understand and are quite specific to the current rule.
I initially tried to add something like `filter(&self) -> Arc<dyn PhysicalExpr>` and `try_filter_pushdown(&self, filter: Arc<dyn PhysicalExpr>) -> Self`, but couldn't get it passing all the tests. It seems theoretically possible, but I prefer not spending more time on it at this stage.
2) Related to the first item, I'm also a bit concerned that the two new APIs -- `gather_filters_for_pushdown` and `handle_child_pushdown_result` -- are both doing parts of the pushdown work. The first one analyzes, the second applies changes. This feels a bit confusing, as the responsibilities aren't clearly separated (because I was always thinking the API's like 1-> information of having filter 2-> how do you treat these filters). At least, we should write in detail what each method and parameter means (I couldn't put them into words :/ )
3) Another thing I couldn't fully understand is why we need to separate the analysis & pushdown work of parent filters vs. self filters. Could you give an example where keeping them separate is necessary?
If this distinction isn't critical, it could simplify `handle_child_pushdown_result()`, since we could avoid passing `parent_filters: Vec<Arc<dyn PhysicalExpr>>`, helping address item-1 as well.

I don't want to hold up this PR any longer and block your further work, but I wanted to write down my concerns so we can figure out cleaner, simpler ways of achieving the same goal. This is a great feature, and it will be a shame if it ends up being understandable, extendable, or usable by only a few of us 😄
author_association: CONTRIBUTOR
user: login: adriangb
created_at: 2025-05-03T22:40:38Z
body: > However, the existing ExecutionPlan APIs are general-purpose, self-explanatory, and simple, whereas these new ones feel harder to understand and are quite specific to the current rule.
I initially tried to add something like filter(&self) -> Arc<dyn PhysicalExpr> and try_filter_pushdown(&self, filter: Arc<dyn PhysicalExpr>) -> Self, but couldn't get it passing all the tests. It seems theoretically possible, but I prefer not spending more time on it at this stage.

I don't know that these new APIs are all that more specialized than something like `try_swapping_with_projection`, `supports_limit_pushdown` or `cardinality_effect`.
Yes they're somewhat complex APIs but unless someone sees a way to simplify them and still address all of the use cases I don't think there's much to be done about that. In other words, the use case may warrant this level of complexity.
For example: HashJoinExec needing to direct a different set of filters to each child, possibly modifying the projections and other bits, and also wanting to push down the own filters it generated - and needing to communicate the result of pushdown to it's parents - is just a _complex_ thing to describe. And because of that I don't even think it's theoretically possible to implement via `filter(&self) -> Arc<dyn PhysicalExpr>` + `try_filter_pushdown(&self, filter: Arc<dyn PhysicalExpr>) -> Self`. In my opinion our best bet is to craft good helper structs and functions to operate on the state, which I've attempted to do here but certainly could use further refinement.

> Related to the first item, I'm also a bit concerned that the two new APIs -- gather_filters_for_pushdown and handle_child_pushdown_result -- are both doing parts of the pushdown work. The first one analyzes, the second applies changes. This feels a bit confusing, as the responsibilities aren't clearly separated (because I was always thinking the API's like 1-> information of having filter 2-> how do you treat these filters). At least, we should write in detail what each method and parameter means (I couldn't put them into words :/ )

I don't follow what the confusing part is: I think of it the same way you are describing.
One method takes in the parent filters and spits out a description of how to push those + any additional filters down to each child.
The second method handles the result of pushdown to the children.
I do agree more / better docstrings would help.

> Another thing I couldn't fully understand is why we need to separate the analysis & pushdown work of parent filters vs. self filters. Could you give an example where keeping them separate is necessary?

I think it helps the invariant of `filters should only be pushed down, not up`.
Consider the case of a TopK node or FilterExec:
1. It receives N filters from it's parents.
2. It adds its own filter and pushes everything down to it's children.
3. When `handle_child_pushdown_result` gets called it needs to split them back up: it needs to tell its parent the result of pushing down each one of the parent filters, but it should *not* also include the result of or a copy of its own filter, that would result in confusion at the least if not bugs with filters flowing up the execution plan.
I think keeping them seperate makes it obvious which are the filters you injected into your children vs. the ones that came from your parents.

author_association: CONTRIBUTOR
user: login: adriangb
created_at: 2025-05-04T01:21:33Z
body: @alamb there's several in-flight PRs now. They all interact and they'll be things we want to tweak from the resultant merged change, but could we start merging them, starting with this one which I believe has no concrete blockers, and I'll work on integrating the changes?
author_association: CONTRIBUTOR
user: login: adriangb
created_at: 2025-05-05T09:46:27Z
body: Thanks @berkaysynnada! Could you take a look at 3b4c4fa? I think it should resolve the concern in https://github.com/apache/datafusion/pull/15801#discussion_r2073043171
author_association: CONTRIBUTOR
user: login: berkaysynnada
created_at: 2025-05-05T09:52:20Z
body: waiting to actions pass
author_association: CONTRIBUTOR
user: login: adriangb
created_at: 2025-05-05T10:59:47Z
body: I want to fix the benchmark I added. One sec...
author_association: CONTRIBUTOR
user: login: adriangb
created_at: 2025-05-05T13:30:59Z
body: @berkaysynnada let's merge this first; I have some work in https://github.com/apache/datafusion/pull/15769 that will be easier once we do / we can wait for Andy and others to take a look
author_association: CONTRIBUTOR
user: login: alamb
created_at: 2025-05-05T18:55:22Z
body: Woohoo -- it is moving!
author_association: CONTRIBUTOR
user: login: berkaysynnada
created_at: 2025-05-05T21:34:36Z
body: @adriangb I forgot to add a note: we have converted the design here, but there wasn't any test change or addition. I think we should add some tests to protect these behaviors, especially for the ones added in this PR.
author_association: CONTRIBUTOR
user: login: adriangb
created_at: 2025-05-06T07:38:52Z
body: @berkaysynnada I added some: https://github.com/apache/datafusion/pull/15955
author_association: CONTRIBUTOR