Repository: pola-rs/polars
Pull request #22672: fix: Fix deadlock joining scanned tables with low thread count
Author: login: nameexhaustion
Labels: name: python
description: Related to Python Polars
name: rust
description: Related to Rust Polars
name: fix
description: Bug fix
Created: 2025-05-08T13:22:33Z | Merged: 2025-05-12T18:18:21Z

* Fixes https://github.com/pola-rs/polars/issues/22641

> sum_horizontal is calling into rayon here from the new-streaming parquet statistics evaluation, but the single rayon thread is blocked on the StreamingQueryExecutor.

Possible solutions I could think of:

* [ ] (1) Check all expressions for use of rayon, and add propagation of a `parallel` argument to all expressions that use rayon.
* [x] (2) Hide the scans inside a `CacheExec`, this gets called by `CachePrefiller` which guarantees the scans are never called (blocked on) from a rayon thread.

(2) is very much a hack, but it's a hack that's nicely confined to the in-mem engine (which is also the only place where it's needed). In the future after everything runs on new-streaming, it may no longer be needed.



Diff:
diff --git a/crates/polars-lazy/src/frame/mod.rs b/crates/polars-lazy/src/frame/mod.rs
index bbdc9311ad76..dad9c41ae8d9 100644
--- a/crates/polars-lazy/src/frame/mod.rs
+++ b/crates/polars-lazy/src/frame/mod.rs
@@ -2593,6 +2593,7 @@ pub use streaming_dispatch::build_streaming_query_executor;
 mod streaming_dispatch {
     use std::sync::{Arc, Mutex};
 
+    use polars_core::POOL;
     use polars_core::error::PolarsResult;
     use polars_core::frame::DataFrame;
     use polars_expr::state::ExecutionState;
@@ -2637,6 +2638,9 @@ mod streaming_dispatch {
 
     impl Executor for StreamingQueryExecutor {
         fn execute(&mut self, _cache: &mut ExecutionState) -> PolarsResult<DataFrame> {
+            // Must not block rayon thread on pending new-streaming future.
+            assert!(POOL.current_thread_index().is_none());
+
             let mut df = { self.executor.try_lock().unwrap().take() }
                 .expect("unhandled: execute() more than once")
                 .execute()
diff --git a/crates/polars-mem-engine/src/executors/cache.rs b/crates/polars-mem-engine/src/executors/cache.rs
index ad4220cc686f..4689883ff193 100644
--- a/crates/polars-mem-engine/src/executors/cache.rs
+++ b/crates/polars-mem-engine/src/executors/cache.rs
@@ -1,11 +1,16 @@
 use std::sync::atomic::Ordering;
 
+#[cfg(feature = "async")]
+use polars_io::pl_async;
+
 use super::*;
 
 pub struct CacheExec {
     pub input: Option<Box<dyn Executor>>,
     pub id: usize,
+    /// `(cache_hits_before_drop - 1)`
     pub count: u32,
+    pub is_new_streaming_scan: bool,
 }
 
 impl Executor for CacheExec {
@@ -20,6 +25,9 @@ impl Executor for CacheExec {
                 let out = cache.1.get().expect("prefilled").clone();
                 let previous = cache.0.fetch_sub(1, Ordering::Relaxed);
                 if previous == 0 {
+                    if state.verbose() {
+                        eprintln!("CACHE DROP: cache id: {:x}", self.id);
+                    }
                     state.remove_df_cache(self.id);
                 }
 
@@ -40,7 +48,7 @@ impl Executor for CacheExec {
 }
 
 pub struct CachePrefiller {
-    pub caches: PlIndexMap<usize, Box<dyn Executor>>,
+    pub caches: PlIndexMap<usize, Box<CacheExec>>,
     pub phys_plan: Box<dyn Executor>,
 }
 
@@ -49,16 +57,87 @@ impl Executor for CachePrefiller {
         if state.verbose() {
             eprintln!("PREFILL CACHES")
         }
+
+        #[cfg(feature = "async")]
+        let parallel_scan_exec_limit = {
+            // Note, this needs to be less than the size of the tokio blocking threadpool (which
+            // defaults to 512).
+            let parallel_scan_exec_limit = POOL.current_num_threads().min(128);
+
+            if state.verbose() {
+                eprintln!(
+                    "CachePrefiller: concurrent streaming scan exec limit: {}",
+                    parallel_scan_exec_limit
+                )
+            }
+
+            Arc::new(tokio::sync::Semaphore::new(parallel_scan_exec_limit))
+        };
+
+        #[cfg(feature = "async")]
+        let mut scan_handles: Vec<tokio::task::JoinHandle<PolarsResult<()>>> = vec![];
+
         // Ensure we traverse in discovery order. This will ensure that caches aren't dependent on each
         // other.
-        for cache in self.caches.values_mut() {
+        for (_, mut cache_exec) in self.caches.drain(..) {
             let mut state = state.split();
             state.branch_idx += 1;
-            let _df = cache.execute(&mut state)?;
+
+            #[cfg(feature = "async")]
+            if cache_exec.is_new_streaming_scan {
+                let parallel_scan_exec_limit = parallel_scan_exec_limit.clone();
+
+                scan_handles.push(pl_async::get_runtime().spawn(async move {
+                    let _permit = parallel_scan_exec_limit.acquire().await.unwrap();
+
+                    tokio::task::spawn_blocking(move || {
+                        cache_exec.execute(&mut state)?;
+
+                        Ok(())
+                    })
+                    .await
+                    .unwrap()
+                }));
+
+                continue;
+            }
+
+            // This cache node may have dependency on the in-progress scan nodes,
+            // ensure all of them complete here.
+
+            #[cfg(feature = "async")]
+            if state.verbose() && !scan_handles.is_empty() {
+                eprintln!(
+                    "CachePrefiller: wait for {} scans executors",
+                    scan_handles.len()
+                )
+            }
+
+            #[cfg(feature = "async")]
+            for handle in scan_handles.drain(..) {
+                pl_async::get_runtime().block_on(handle).unwrap()?;
+            }
+
+            let _df = cache_exec.execute(&mut state)?;
+        }
+
+        #[cfg(feature = "async")]
+        if state.verbose() && !scan_handles.is_empty() {
+            eprintln!(
+                "CachePrefiller: wait for {} scans executors",
+                scan_handles.len()
+            )
+        }
+
+        #[cfg(feature = "async")]
+        for handle in scan_handles {
+            pl_async::get_runtime().block_on(handle).unwrap()?;
         }
+
         if state.verbose() {
             eprintln!("EXECUTE PHYS PLAN")
         }
+
         self.phys_plan.execute(state)
     }
 }
diff --git a/crates/polars-mem-engine/src/planner/lp.rs b/crates/polars-mem-engine/src/planner/lp.rs
index 628a9829ad1c..223104179d09 100644
--- a/crates/polars-mem-engine/src/planner/lp.rs
+++ b/crates/polars-mem-engine/src/planner/lp.rs
@@ -208,7 +208,7 @@ fn create_physical_plan_impl(
     expr_arena: &mut Arena<AExpr>,
     state: &mut ConversionState,
     // Cache nodes in order of discovery
-    cache_nodes: &mut PlIndexMap<usize, Box<dyn Executor>>,
+    cache_nodes: &mut PlIndexMap<usize, Box<executors::CacheExec>>,
     build_streaming_executor: Option<StreamingExecutorBuilder>,
 ) -> PolarsResult<Box<dyn Executor>> {
     use IR::*;
@@ -438,6 +438,7 @@ fn create_physical_plan_impl(
             scan_type,
             predicate,
             mut unified_scan_args,
+            id: scan_mem_id,
         } => {
             unified_scan_args.pre_slice = if let Some(mut slice) = unified_scan_args.pre_slice {
                 *slice.len_mut() = _set_n_rows_for_scan(Some(slice.len())).unwrap();
@@ -447,7 +448,7 @@ fn create_physical_plan_impl(
                     .map(|len| polars_utils::slice_enum::Slice::Positive { offset: 0, len })
             };
 
-            let mut state = ExpressionConversionState::new(true);
+            let mut expr_conversion_state = ExpressionConversionState::new(true);
 
             let mut create_skip_batch_predicate = false;
             #[cfg(feature = "parquet")]
@@ -470,7 +471,7 @@ fn create_physical_plan_impl(
                         &predicate,
                         expr_arena,
                         output_schema.as_ref().unwrap_or(&file_info.schema),
-                        &mut state,
+                        &mut expr_conversion_state,
                         create_skip_batch_predicate,
                         false,
                     )
@@ -485,14 +486,54 @@ fn create_physical_plan_impl(
                         unified_scan_args,
                         file_info,
                         output_schema,
-                        predicate_has_windows: state.has_windows,
+                        predicate_has_windows: expr_conversion_state.has_windows,
                     }))
                 },
                 #[allow(unreachable_patterns)]
                 _ => {
-                    let build_func = build_streaming_executor
-                        .expect("invalid build. Missing feature new-streaming");
-                    return build_func(root, lp_arena, expr_arena);
+                    // We wrap in a CacheExec so that the new-streaming scan gets called from the
+                    // CachePrefiller. This ensures it is called from outside of rayon to avoid
+                    // deadlocks.
+                    //
+                    // Note that we don't actually want it to be kept in memory after being used,
+                    // so we set the count to have it be dropped after a single use (or however
+                    // many times it is referenced after CSE (subplan)).
+                    state.has_cache_parent = true;
+                    state.has_cache_child = true;
+
+                    let scan_mem_id: usize = scan_mem_id.to_usize();
+
+                    if !cache_nodes.contains_key(&scan_mem_id) {
+                        let build_func = build_streaming_executor
+                            .expect("invalid build. Missing feature new-streaming");
+
+                        let executor = build_func(root, lp_arena, expr_arena)?;
+
+                        cache_nodes.insert(
+                            scan_mem_id,
+                            Box::new(executors::CacheExec {
+                                input: Some(executor),
+                                id: scan_mem_id,
+                                // This is (n_hits - 1), because the drop logic is `fetch_sub(1) == 0`.
+                                count: 0,
+                                is_new_streaming_scan: true,
+                            }),
+                        );
+                    } else {
+                        // Already exists - this scan IR is under a CSE (subplan). We need to
+                        // increment the cache hit count here.
+                        let cache_exec = cache_nodes.get_mut(&scan_mem_id).unwrap();
+                        cache_exec.count = cache_exec.count.saturating_add(1);
+                    }
+
+                    Ok(Box::new(executors::CacheExec {
+                        id: scan_mem_id,
+                        // Rest of the fields don't matter - the actual node was inserted into
+                        // `cache_nodes`.
+                        input: None,
+                        count: Default::default(),
+                        is_new_streaming_scan: true,
+                    }))
                 },
                 #[allow(unreachable_patterns)]
                 _ => unreachable!(),
@@ -577,6 +618,7 @@ fn create_physical_plan_impl(
                     id,
                     input: Some(input),
                     count: cache_hits,
+                    is_new_streaming_scan: false,
                 });
 
                 cache_nodes.insert(id, cache);
@@ -586,6 +628,7 @@ fn create_physical_plan_impl(
                 id,
                 input: None,
                 count: cache_hits,
+                is_new_streaming_scan: false,
             }))
         },
         Distinct { input, options } => {
diff --git a/crates/polars-pipe/src/pipeline/convert.rs b/crates/polars-pipe/src/pipeline/convert.rs
index 8c7f2c587790..ab80da423920 100644
--- a/crates/polars-pipe/src/pipeline/convert.rs
+++ b/crates/polars-pipe/src/pipeline/convert.rs
@@ -71,6 +71,7 @@ where
             predicate,
             output_schema,
             scan_type,
+            id: _,
         } => {
             let paths = sources.into_paths();
             let schema = output_schema.as_ref().unwrap_or(&file_info.schema);
diff --git a/crates/polars-plan/src/plans/conversion/dsl_to_ir.rs b/crates/polars-plan/src/plans/conversion/dsl_to_ir.rs
index 9b9ffa4a297b..9ef18217f71d 100644
--- a/crates/polars-plan/src/plans/conversion/dsl_to_ir.rs
+++ b/crates/polars-plan/src/plans/conversion/dsl_to_ir.rs
@@ -374,6 +374,7 @@ pub fn to_alp_impl(lp: DslPlan, ctxt: &mut DslConversionContext) -> PolarsResult
                         scan_type,
                         output_schema: None,
                         unified_scan_args,
+                        id: Default::default(),
                     }
                 };
 
diff --git a/crates/polars-plan/src/plans/conversion/mod.rs b/crates/polars-plan/src/plans/conversion/mod.rs
index 3725c8a1e031..a9e3cb50de5e 100644
--- a/crates/polars-plan/src/plans/conversion/mod.rs
+++ b/crates/polars-plan/src/plans/conversion/mod.rs
@@ -70,6 +70,7 @@ impl IR {
                     scan_type,
                     output_schema: _,
                     unified_scan_args,
+                    id: _,
                 } = ir.clone()
                 else {
                     unreachable!()
diff --git a/crates/polars-plan/src/plans/ir/dot.rs b/crates/polars-plan/src/plans/ir/dot.rs
index 9c6c572cf0a0..0b34e0ff5b5c 100644
--- a/crates/polars-plan/src/plans/ir/dot.rs
+++ b/crates/polars-plan/src/plans/ir/dot.rs
@@ -246,6 +246,7 @@ impl<'a> IRDotDisplay<'a> {
                 scan_type,
                 unified_scan_args,
                 output_schema: _,
+                id: _,
             } => {
                 let name: &str = (&**scan_type).into();
                 let path = ScanSourcesDisplay(sources);
diff --git a/crates/polars-plan/src/plans/ir/format.rs b/crates/polars-plan/src/plans/ir/format.rs
index d4e6745487c1..7511b8936c34 100644
--- a/crates/polars-plan/src/plans/ir/format.rs
+++ b/crates/polars-plan/src/plans/ir/format.rs
@@ -73,12 +73,14 @@ fn write_scan(
     predicate: &Option<ExprIRDisplay<'_>>,
     pre_slice: Option<Slice>,
     row_index: Option<&RowIndex>,
+    scan_mem_id: Option<usize>,
 ) -> fmt::Result {
     write!(
         f,
-        "{:indent$}{name} SCAN {}",
+        "{:indent$}{name} SCAN {} [id: {:?}]",
         "",
-        ScanSourcesDisplay(sources)
+        ScanSourcesDisplay(sources),
+        scan_mem_id,
     )?;
 
     let total_columns = total_columns - usize::from(row_index.is_some());
@@ -678,6 +680,7 @@ pub fn write_ir_non_recursive(
                     .n_rows
                     .map(|len| polars_utils::slice_enum::Slice::Positive { offset: 0, len }),
                 None,
+                None,
             )
         },
         IR::Slice {
@@ -704,6 +707,7 @@ pub fn write_ir_non_recursive(
             unified_scan_args,
             hive_parts: _,
             output_schema: _,
+            id: scan_mem_id,
         } => {
             let n_columns = unified_scan_args
                 .projection
@@ -723,6 +727,7 @@ pub fn write_ir_non_recursive(
                 &predicate,
                 unified_scan_args.pre_slice.clone(),
                 unified_scan_args.row_index.as_ref(),
+                Some(scan_mem_id.to_usize()),
             )
         },
         IR::DataFrameScan {
diff --git a/crates/polars-plan/src/plans/ir/inputs.rs b/crates/polars-plan/src/plans/ir/inputs.rs
index 174658e9ed31..e3a4bb1f7ec2 100644
--- a/crates/polars-plan/src/plans/ir/inputs.rs
+++ b/crates/polars-plan/src/plans/ir/inputs.rs
@@ -103,6 +103,7 @@ impl IR {
                 predicate,
                 unified_scan_args,
                 scan_type,
+                id: _,
             } => {
                 let mut new_predicate = None;
                 if predicate.is_some() {
@@ -117,6 +118,7 @@ impl IR {
                     unified_scan_args: unified_scan_args.clone(),
                     predicate: new_predicate,
                     scan_type: scan_type.clone(),
+                    id: Default::default(),
                 }
             },
             DataFrameScan {
diff --git a/crates/polars-plan/src/plans/ir/mod.rs b/crates/polars-plan/src/plans/ir/mod.rs
index 6c657ac8b715..ba8979ab5479 100644
--- a/crates/polars-plan/src/plans/ir/mod.rs
+++ b/crates/polars-plan/src/plans/ir/mod.rs
@@ -11,6 +11,7 @@ pub use dot::{EscapeLabel, IRDotDisplay, PathsDisplay, ScanSourcesDisplay};
 pub use format::{ExprIRDisplay, IRDisplay, write_group_by, write_ir_non_recursive};
 use polars_core::prelude::*;
 use polars_utils::idx_vec::UnitVec;
+use polars_utils::unique_id::UniqueId;
 use polars_utils::unitvec;
 #[cfg(feature = "ir_serde")]
 use serde::{Deserialize, Serialize};
@@ -62,6 +63,16 @@ pub enum IR {
         scan_type: Box<FileScan>,
         /// generic options that can be used for all file types.
         unified_scan_args: Box<UnifiedScanArgs>,
+        /// This used as part of a hack to prevent deadlocks when we run the in-memory engine with
+        /// scans dispatched to new-streaming. This ID is used as the ID of the CacheExec that
+        /// wraps this scan. It will not be needed once everything runs in new-streaming.
+        ///
+        /// We use this instead of the Arc-address of the ScanSources as it's possible to pass the
+        /// same set of ScanSources with different scan options.
+        ///
+        /// NOTE: This must be reset to a new Arc during e.g. predicate / slice pushdown.
+        #[cfg_attr(feature = "serde", serde(skip, default))]
+        id: UniqueId,
     },
     DataFrameScan {
         df: Arc<DataFrame>,
diff --git a/crates/polars-plan/src/plans/optimizer/expand_datasets.rs b/crates/polars-plan/src/plans/optimizer/expand_datasets.rs
index 780350c1f1cb..397746444fd3 100644
--- a/crates/polars-plan/src/plans/optimizer/expand_datasets.rs
+++ b/crates/polars-plan/src/plans/optimizer/expand_datasets.rs
@@ -107,6 +107,7 @@ impl OptimizationRule for ExpandDatasets {
                                 hive_parts: _,
                                 predicate: _,
                                 output_schema: _,
+                                id: _,
                             } = &mut ir
                             else {
                                 unreachable!()
diff --git a/crates/polars-plan/src/plans/optimizer/predicate_pushdown/mod.rs b/crates/polars-plan/src/plans/optimizer/predicate_pushdown/mod.rs
index a2abde0e1e6d..c667f94e2d8b 100644
--- a/crates/polars-plan/src/plans/optimizer/predicate_pushdown/mod.rs
+++ b/crates/polars-plan/src/plans/optimizer/predicate_pushdown/mod.rs
@@ -362,6 +362,7 @@ impl PredicatePushDown<'_> {
                 scan_type,
                 unified_scan_args,
                 output_schema,
+                id: _,
             } => {
                 let mut blocked_names = Vec::with_capacity(2);
 
@@ -416,6 +417,7 @@ impl PredicatePushDown<'_> {
                         unified_scan_args,
                         output_schema,
                         scan_type,
+                        id: Default::default(),
                     }
                 } else {
                     let lp = Scan {
@@ -426,6 +428,7 @@ impl PredicatePushDown<'_> {
                         unified_scan_args,
                         output_schema,
                         scan_type,
+                        id: Default::default(),
                     };
                     if let Some(predicate) = predicate {
                         let input = lp_arena.add(lp);
diff --git a/crates/polars-plan/src/plans/optimizer/projection_pushdown/mod.rs b/crates/polars-plan/src/plans/optimizer/projection_pushdown/mod.rs
index 7a989f7ae37c..97ba749d39d7 100644
--- a/crates/polars-plan/src/plans/optimizer/projection_pushdown/mod.rs
+++ b/crates/polars-plan/src/plans/optimizer/projection_pushdown/mod.rs
@@ -435,6 +435,7 @@ impl ProjectionPushDown {
                 predicate,
                 mut unified_scan_args,
                 mut output_schema,
+                id: _,
             } => {
                 let do_optimization = match &*scan_type {
                     FileScan::Anonymous { function, .. } => function.allows_projection_pushdown(),
@@ -561,6 +562,7 @@ impl ProjectionPushDown {
                     scan_type,
                     predicate,
                     unified_scan_args,
+                    id: Default::default(),
                 };
 
                 Ok(lp)
diff --git a/crates/polars-plan/src/plans/optimizer/slice_pushdown_lp.rs b/crates/polars-plan/src/plans/optimizer/slice_pushdown_lp.rs
index ff46aa386169..4b450e7172a7 100644
--- a/crates/polars-plan/src/plans/optimizer/slice_pushdown_lp.rs
+++ b/crates/polars-plan/src/plans/optimizer/slice_pushdown_lp.rs
@@ -224,6 +224,7 @@ impl SlicePushDown {
                 mut unified_scan_args,
                 predicate,
                 scan_type,
+                id: _,
             }, Some(state)) if predicate.is_none() && match &*scan_type {
                 #[cfg(feature = "parquet")]
                 FileScan::Parquet { .. } => true,
@@ -253,6 +254,7 @@ impl SlicePushDown {
                     scan_type,
                     unified_scan_args,
                     predicate,
+                    id: Default::default(),
                 };
 
                 Ok(lp)
diff --git a/crates/polars-plan/src/plans/visitor/hash.rs b/crates/polars-plan/src/plans/visitor/hash.rs
index b10d03352844..8abaf88a153c 100644
--- a/crates/polars-plan/src/plans/visitor/hash.rs
+++ b/crates/polars-plan/src/plans/visitor/hash.rs
@@ -81,6 +81,7 @@ impl Hash for HashableEqLP<'_> {
                 output_schema: _,
                 scan_type,
                 unified_scan_args,
+                id: _,
             } => {
                 // We don't have to traverse the schema, hive partitions etc. as they are derivative from the paths.
                 scan_type.hash(state);
@@ -262,6 +263,7 @@ impl HashableEqLP<'_> {
                     output_schema: _,
                     scan_type: stl,
                     unified_scan_args: ol,
+                    id: _,
                 },
                 IR::Scan {
                     sources: pr,
@@ -271,6 +273,7 @@ impl HashableEqLP<'_> {
                     output_schema: _,
                     scan_type: str,
                     unified_scan_args: or,
+                    id: _,
                 },
             ) => {
                 pl == pr
diff --git a/crates/polars-python/src/lazyframe/visitor/nodes.rs b/crates/polars-python/src/lazyframe/visitor/nodes.rs
index d4b4b639bdc4..2b3e85541e6b 100644
--- a/crates/polars-python/src/lazyframe/visitor/nodes.rs
+++ b/crates/polars-python/src/lazyframe/visitor/nodes.rs
@@ -371,6 +371,7 @@ pub(crate) fn into_py(py: Python<'_>, plan: &IR) -> PyResult<PyObject> {
             output_schema: _,
             scan_type,
             unified_scan_args,
+            id: _,
         } => Scan {
             paths: sources
                 .into_paths()
diff --git a/crates/polars-stream/src/physical_plan/lower_ir.rs b/crates/polars-stream/src/physical_plan/lower_ir.rs
index be8dd31480d4..51ce51c9c509 100644
--- a/crates/polars-stream/src/physical_plan/lower_ir.rs
+++ b/crates/polars-stream/src/physical_plan/lower_ir.rs
@@ -472,6 +472,7 @@ pub fn lower_ir(
                 scan_type,
                 predicate,
                 unified_scan_args,
+                id: _,
             } = v.clone()
             else {
                 unreachable!();
diff --git a/crates/polars-stream/src/utils/late_materialized_df.rs b/crates/polars-stream/src/utils/late_materialized_df.rs
index 4d1a0d8acd74..9094a7d8ed43 100644
--- a/crates/polars-stream/src/utils/late_materialized_df.rs
+++ b/crates/polars-stream/src/utils/late_materialized_df.rs
@@ -36,6 +36,7 @@ impl LateMaterializedDataFrame {
                 function: self,
             }),
             unified_scan_args: Box::new(UnifiedScanArgs::default()),
+            id: Default::default(),
         }
     }
 }
diff --git a/crates/polars-utils/src/lib.rs b/crates/polars-utils/src/lib.rs
index 70fbebffbe74..d25bdfad1032 100644
--- a/crates/polars-utils/src/lib.rs
+++ b/crates/polars-utils/src/lib.rs
@@ -38,6 +38,7 @@ pub mod sync;
 #[cfg(feature = "sysinfo")]
 pub mod sys;
 pub mod total_ord;
+pub mod unique_id;
 
 pub use functions::*;
 pub mod file;
diff --git a/crates/polars-utils/src/unique_id.rs b/crates/polars-utils/src/unique_id.rs
new file mode 100644
index 000000000000..0e118e62d07a
--- /dev/null
+++ b/crates/polars-utils/src/unique_id.rs
@@ -0,0 +1,23 @@
+use std::sync::Arc;
+
+/// Unique ref-counted identifier.
+#[derive(Debug, Clone, Hash, Eq, PartialEq)]
+pub struct UniqueId(
+    // We use an Arc to reserve a memory address rather than a static atomic counter, as the latter
+    // may cause duplicate IDs on wrap-around.
+    //
+    // Note, this inner repr is a private implementation detail.
+    Arc<()>,
+);
+
+impl UniqueId {
+    pub fn to_usize(&self) -> usize {
+        Arc::as_ptr(&self.0) as usize
+    }
+}
+
+impl Default for UniqueId {
+    fn default() -> Self {
+        Self(Arc::new(()))
+    }
+}
diff --git a/py-polars/tests/unit/io/test_lazy_parquet.py b/py-polars/tests/unit/io/test_lazy_parquet.py
index d209534e2a38..54ce8bf6cb55 100644
--- a/py-polars/tests/unit/io/test_lazy_parquet.py
+++ b/py-polars/tests/unit/io/test_lazy_parquet.py
@@ -907,6 +907,8 @@ def test_scan_parquet_prefilter_panic_22452() -> None:
 import polars as pl
 from polars.testing import assert_frame_equal
 
+assert pl.thread_pool_size() == 2
+
 f = io.BytesIO()
 
 df = pl.DataFrame({x: 1 for x in ["a", "b", "c", "d", "e"]})
@@ -926,3 +928,94 @@ def test_scan_parquet_prefilter_panic_22452() -> None:
     )
 
     assert out == b"OK"
+
+
+def test_scan_parquet_in_mem_to_streaming_dispatch_deadlock_22641() -> None:
+    out = subprocess.check_output(
+        [
+            sys.executable,
+            "-c",
+            """\
+import os
+
+os.environ["POLARS_MAX_THREADS"] = "1"
+os.environ["POLARS_VERBOSE"] = "1"
+
+import io
+import sys
+from threading import Thread
+
+import polars as pl
+
+assert pl.thread_pool_size() == 1
+
+f = io.BytesIO()
+pl.DataFrame({"x": 1}).write_parquet(f)
+
+q = (
+    pl.scan_parquet(f)
+    .filter(pl.sum_horizontal(pl.col("x"), pl.col("x"), pl.col("x")) >= 0)
+    .join(pl.scan_parquet(f), on="x", how="left")
+)
+
+results = [pl.DataFrame(), pl.DataFrame(), pl.DataFrame(), pl.DataFrame()]
+
+
+def run():
+    # Also test just a single scan
+    pl.scan_parquet(f).collect()
+
+    print("QUERY-FENCE", file=sys.stderr)
+
+    results[0] = q.collect()
+
+    print("QUERY-FENCE", file=sys.stderr)
+
+    results[1] = pl.concat([q, q, q]).collect().head(1)
+
+    print("QUERY-FENCE", file=sys.stderr)
+
+    results[2] = pl.collect_all([q, q, q])[0]
+
+    print("QUERY-FENCE", file=sys.stderr)
+
+    results[3] = pl.collect_all(3 * [pl.concat(3 * [q])])[0].head(1)
+
+
+t = Thread(target=run, daemon=True)
+t.start()
+t.join(5)
+
+assert [x.equals(pl.DataFrame({"x": 1})) for x in results] == [True, True, True, True]
+
+print("OK", end="", file=sys.stderr)
+""",
+        ],
+        stderr=subprocess.STDOUT,
+    )
+
+    assert out.endswith(b"OK")
+
+    def ensure_caches_dropped(verbose_log: str) -> None:
+        cache_hit_prefix = "CACHE HIT: cache id: "
+
+        ids_hit = {
+            x[len(cache_hit_prefix) :]
+            for x in verbose_log.splitlines()
+            if x.startswith(cache_hit_prefix)
+        }
+
+        cache_drop_prefix = "CACHE DROP: cache id: "
+
+        ids_dropped = {
+            x[len(cache_drop_prefix) :]
+            for x in verbose_log.splitlines()
+            if x.startswith(cache_drop_prefix)
+        }
+
+        assert ids_hit == ids_dropped
+
+    out_str = out.decode()
+
+    for logs in out_str.split("QUERY-FENCE"):
+        ensure_caches_dropped(logs)


Reviews:
user: login: nameexhaustion
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-11T16:28:08Z
user: login: nameexhaustion
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-11T16:30:02Z
user: login: nameexhaustion
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-12T14:20:09Z

Review comments:
diff_hunk: @@ -20,6 +21,9 @@ impl Executor for CacheExec {
                 let out = cache.1.get().expect("prefilled").clone();
                 let previous = cache.0.fetch_sub(1, Ordering::Relaxed);
                 if previous == 0 {
+                    if state.verbose() {
+                        eprintln!("CACHE DROP: cache id: {:x}", self.id);
path: crates/polars-mem-engine/src/executors/cache.rs
user: login: nameexhaustion
body: Added `CACHE DROP` verbose print here that can be used by tests.

created_at: 2025-05-11T16:28:08Z
author_association: COLLABORATOR
diff_hunk: @@ -926,3 +928,94 @@ def test_scan_parquet_prefilter_panic_22452() -> None:
     )
 
     assert out == b"OK"
+
+
+def test_scan_parquet_in_mem_to_streaming_dispatch_deadlock_22641() -> None:
+    out = subprocess.check_output(
+        [
+            sys.executable,
+            "-c",
+            """\
+import os
+
+os.environ["POLARS_MAX_THREADS"] = "1"
+os.environ["POLARS_VERBOSE"] = "1"
+
+import io
+import sys
+from threading import Thread
+
+import polars as pl
+
+assert pl.thread_pool_size() == 1
+
+f = io.BytesIO()
+pl.DataFrame({"x": 1}).write_parquet(f)
+
+q = (
+    pl.scan_parquet(f)
+    .filter(pl.sum_horizontal(pl.col("x"), pl.col("x"), pl.col("x")) >= 0)
+    .join(pl.scan_parquet(f), on="x", how="left")
+)
+
+results = [pl.DataFrame(), pl.DataFrame(), pl.DataFrame(), pl.DataFrame()]
+
+
+def run():
+    # Also test just a single scan
+    pl.scan_parquet(f).collect()
+
+    print("QUERY-FENCE", file=sys.stderr)
+
+    results[0] = q.collect()
+
+    print("QUERY-FENCE", file=sys.stderr)
+
+    results[1] = pl.concat([q, q, q]).collect().head(1)
+
+    print("QUERY-FENCE", file=sys.stderr)
+
+    results[2] = pl.collect_all([q, q, q])[0]
+
+    print("QUERY-FENCE", file=sys.stderr)
+
+    results[3] = pl.collect_all(3 * [pl.concat(3 * [q])])[0].head(1)
+
+
+t = Thread(target=run, daemon=True)
+t.start()
+t.join(5)
+
+assert [x.equals(pl.DataFrame({"x": 1})) for x in results] == [True, True, True, True]
+
+print("OK", end="", file=sys.stderr)
+""",
+        ],
+        stderr=subprocess.STDOUT,
+    )
+
+    assert out.endswith(b"OK")
+
+    def ensure_caches_dropped(verbose_log: str) -> None:
+        cache_hit_prefix = "CACHE HIT: cache id: "
+
+        ids_hit = {
+            x[len(cache_hit_prefix) :]
+            for x in verbose_log.splitlines()
+            if x.startswith(cache_hit_prefix)
+        }
+
+        cache_drop_prefix = "CACHE DROP: cache id: "
+
+        ids_dropped = {
+            x[len(cache_drop_prefix) :]
+            for x in verbose_log.splitlines()
+            if x.startswith(cache_drop_prefix)
+        }
+
+        assert ids_hit == ids_dropped
path: py-polars/tests/unit/io/test_lazy_parquet.py
user: login: nameexhaustion
body: Test checks for and ensures all cache ids are dropped - we use the newly added print message to determine that all cache IDs that are hit are also dropped.

created_at: 2025-05-11T16:30:02Z
author_association: COLLABORATOR
diff_hunk: @@ -49,16 +57,87 @@ impl Executor for CachePrefiller {
         if state.verbose() {
             eprintln!("PREFILL CACHES")
         }
+
+        #[cfg(feature = "async")]
+        let parallel_scan_exec_limit = {
+            // Note, this needs to be less than the size of the tokio blocking threadpool (which
+            // defaults to 512).
+            let parallel_scan_exec_limit = POOL.current_num_threads().min(128);
+
+            if state.verbose() {
+                eprintln!(
+                    "CachePrefiller: concurrent streaming scan exec limit: {}",
+                    parallel_scan_exec_limit
+                )
+            }
+
+            Arc::new(tokio::sync::Semaphore::new(parallel_scan_exec_limit))
+        };
+
+        #[cfg(feature = "async")]
+        let mut scan_handles: Vec<tokio::task::JoinHandle<PolarsResult<()>>> = vec![];
+
         // Ensure we traverse in discovery order. This will ensure that caches aren't dependent on each
         // other.
-        for cache in self.caches.values_mut() {
+        for (_, mut cache_exec) in self.caches.drain(..) {
             let mut state = state.split();
             state.branch_idx += 1;
-            let _df = cache.execute(&mut state)?;
+
+            #[cfg(feature = "async")]
+            if cache_exec.is_new_streaming_scan {
+                let parallel_scan_exec_limit = parallel_scan_exec_limit.clone();
+
+                scan_handles.push(pl_async::get_runtime().spawn(async move {
+                    let _permit = parallel_scan_exec_limit.acquire().await.unwrap();
path: crates/polars-mem-engine/src/executors/cache.rs
user: login: nameexhaustion
body: Added logic to execute sequences of scan executors in parallel (called via `spawn_blocking()`):

```
 1 2 3 4 5 6
[X S S S X S]
* X: Generic executor
* S: Scan node
```
Scan nodes 2-4 will be run in parallel (we wait for them at 5 due to cache dependency).

created_at: 2025-05-12T14:20:09Z
author_association: COLLABORATOR

PR comments:
user: login: codecov[bot]
created_at: 2025-05-08T14:02:24Z
body: ## [Codecov](https://app.codecov.io/gh/pola-rs/polars/pull/22672?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pola-rs) Report
Attention: Patch coverage is `94.87179%` with `4 lines` in your changes missing coverage. Please review.
> Project coverage is 80.94%. Comparing base [(`2338ab4`)](https://app.codecov.io/gh/pola-rs/polars/commit/2338ab48d0a6e67bd25be447a0219d64e3ce154f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pola-rs) to head [(`2bf1c5e`)](https://app.codecov.io/gh/pola-rs/polars/commit/2bf1c5e202d50d867d493ab54ee29f850c7b3270?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pola-rs).
> Report is 6 commits behind head on main.

| [Files with missing lines](https://app.codecov.io/gh/pola-rs/polars/pull/22672?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pola-rs) | Patch % | Lines |
|---|---|---|
| [crates/polars-plan/src/plans/ir/dot.rs](https://app.codecov.io/gh/pola-rs/polars/pull/22672?src=pr&el=tree&filepath=crates%2Fpolars-plan%2Fsrc%2Fplans%2Fir%2Fdot.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pola-rs#diff-Y3JhdGVzL3BvbGFycy1wbGFuL3NyYy9wbGFucy9pci9kb3QucnM=) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/pola-rs/polars/pull/22672?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pola-rs) |
| [crates/polars-plan/src/plans/ir/format.rs](https://app.codecov.io/gh/pola-rs/polars/pull/22672?src=pr&el=tree&filepath=crates%2Fpolars-plan%2Fsrc%2Fplans%2Fir%2Fformat.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pola-rs#diff-Y3JhdGVzL3BvbGFycy1wbGFuL3NyYy9wbGFucy9pci9mb3JtYXQucnM=) | 83.33% | [1 Missing :warning: ](https://app.codecov.io/gh/pola-rs/polars/pull/22672?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pola-rs) |
| [...polars-plan/src/plans/optimizer/expand\_datasets.rs](https://app.codecov.io/gh/pola-rs/polars/pull/22672?src=pr&el=tree&filepath=crates%2Fpolars-plan%2Fsrc%2Fplans%2Foptimizer%2Fexpand_datasets.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pola-rs#diff-Y3JhdGVzL3BvbGFycy1wbGFuL3NyYy9wbGFucy9vcHRpbWl6ZXIvZXhwYW5kX2RhdGFzZXRzLnJz) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/pola-rs/polars/pull/22672?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pola-rs) |
| [...rates/polars-python/src/lazyframe/visitor/nodes.rs](https://app.codecov.io/gh/pola-rs/polars/pull/22672?src=pr&el=tree&filepath=crates%2Fpolars-python%2Fsrc%2Flazyframe%2Fvisitor%2Fnodes.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pola-rs#diff-Y3JhdGVzL3BvbGFycy1weXRob24vc3JjL2xhenlmcmFtZS92aXNpdG9yL25vZGVzLnJz) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/pola-rs/polars/pull/22672?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pola-rs) |

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff            @@
##             main   #22672    +/-   ##
========================================
  Coverage   80.93%   80.94%            
========================================
  Files        1666     1672     +6     
  Lines      235856   236291   +435     
  Branches     2789     2790     +1     
========================================
+ Hits       190891   191266   +375     
- Misses      44295    44355    +60     
  Partials      670      670            
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/pola-rs/polars/pull/22672?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pola-rs).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pola-rs).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: ritchie46
created_at: 2025-05-11T14:15:45Z
body: < (2) is very much a hack, but it's a hack that's nicely confined to the in-mem engine (which is also the only place where it's needed). In the future after everything runs on new-streaming, it may no longer be needed.

Yes, it indeed is. Do we properly clear the cache after the first node? 
author_association: MEMBER
user: login: nameexhaustion
created_at: 2025-05-11T16:32:53Z
body: > Do we properly clear the cache after the first node?

It should, there is also a test to ensure the caches are dropped - have left comments pointing out where.

author_association: COLLABORATOR
user: login: ritchie46
created_at: 2025-05-12T06:31:57Z
body: 1 side effect from this hack is that if we join a lot of tables, we first load them all in memory. Maybe we should just turn of bushy parallelism instead.
author_association: MEMBER
user: login: nameexhaustion
created_at: 2025-05-12T07:40:51Z
body: > 1 side effect from this hack is that if we join a lot of tables, we first load them all in memory. Maybe we should just turn of bushy parallelism instead.

I don't think this should add too much memory overhead - the old scan nodes for in-mem engine were also fully eagerly evaluated (once they were hit).

author_association: COLLABORATOR