Repository: rapidsai/cudf
Pull request #18516: Refactor polars configuration
Author: login: TomAugspurger
Labels: name: Python
description: Affects Python cuDF API.
name: improvement
description: Improvement / enhancement to an existing function
name: non-breaking
description: Non-breaking change
name: cudf-polars
description: Issues specific to cudf-polars
Created: 2025-04-17T17:47:56Z | Merged: 2025-04-25T15:31:19Z

## Description

This updates our internal handling of the user-provided configuration for our polars GPUEngine. We use a set of dataclasses to manage the configuration options.

There aren't any user-facing changes.

## Checklist
- [x] I am familiar with the [Contributing Guidelines](https://github.com/rapidsai/cudf/blob/HEAD/CONTRIBUTING.md).
- [ ] New or existing tests cover these changes.
- [ ] The documentation is up to date with these changes.



Diff:
diff --git a/python/cudf_polars/cudf_polars/callback.py b/python/cudf_polars/cudf_polars/callback.py
index 73e2f204d81..7ba8c24f15f 100644
--- a/python/cudf_polars/cudf_polars/callback.py
+++ b/python/cudf_polars/cudf_polars/callback.py
@@ -13,6 +13,7 @@
 from typing import TYPE_CHECKING, Literal, overload
 
 import nvtx
+from typing_extensions import assert_never
 
 from polars.exceptions import ComputeError, PerformanceWarning
 
@@ -185,9 +186,7 @@ def _callback(
     n_rows: int | None,
     should_time: Literal[False],
     *,
-    device: int | None,
-    memory_resource: int | None,
-    executor: Literal["in-memory", "streaming"] | None,
+    memory_resource: rmm.mr.DeviceMemoryResource | None,
     config_options: ConfigOptions,
     timer: Timer | None,
 ) -> pl.DataFrame: ...
@@ -201,9 +200,7 @@ def _callback(
     n_rows: int | None,
     should_time: Literal[True],
     *,
-    device: int | None,
-    memory_resource: int | None,
-    executor: Literal["in-memory", "streaming"] | None,
+    memory_resource: rmm.mr.DeviceMemoryResource | None,
     config_options: ConfigOptions,
     timer: Timer | None,
 ) -> tuple[pl.DataFrame, list[tuple[int, int, str]]]: ...
@@ -216,9 +213,7 @@ def _callback(
     n_rows: int | None,
     should_time: bool,  # noqa: FBT001
     *,
-    device: int | None,
-    memory_resource: int | None,
-    executor: Literal["in-memory", "streaming"] | None,
+    memory_resource: rmm.mr.DeviceMemoryResource | None,
     config_options: ConfigOptions,
     timer: Timer | None,
 ) -> pl.DataFrame | tuple[pl.DataFrame, list[tuple[int, int, str]]]:
@@ -230,21 +225,20 @@ def _callback(
     with (
         nvtx.annotate(message="ExecuteIR", domain="cudf_polars"),
         # Device must be set before memory resource is obtained.
-        set_device(device),
+        set_device(config_options.device),
         set_memory_resource(memory_resource),
     ):
-        if executor is None or executor == "in-memory":
+        if config_options.executor.name == "in-memory":
             df = ir.evaluate(cache={}, timer=timer).to_polars()
             if timer is None:
                 return df
             else:
                 return df, timer.timings
-        elif executor == "streaming":
+        elif config_options.executor.name == "streaming":
             from cudf_polars.experimental.parallel import evaluate_streaming
 
             return evaluate_streaming(ir, config_options).to_polars()
-        else:
-            raise ValueError(f"Unknown executor '{executor}'")
+        assert_never(f"Unknown executor '{config_options.executor}'")
 
 
 def execute_with_cudf(
@@ -263,7 +257,7 @@ def execute_with_cudf(
         profiling should occur).
 
     config
-        GPUEngine configuration object
+        GPUEngine object. Configuration is available as ``engine.config``.
 
     Raises
     ------
@@ -281,21 +275,20 @@ def execute_with_cudf(
     else:
         start = time.monotonic_ns()
         timer = Timer(start - duration_since_start)
-    device = config.device
+
     memory_resource = config.memory_resource
-    raise_on_fail = config.config.get("raise_on_fail", False)
-    executor = config.config.get("executor", None)
+
     with nvtx.annotate(message="ConvertIR", domain="cudf_polars"):
         translator = Translator(nt, config)
         ir = translator.translate_ir()
         ir_translation_errors = translator.errors
         if timer is not None:
             timer.store(start, time.monotonic_ns(), "gpu-ir-translation")
+
         if (
             memory_resource is None
-            and executor == "streaming"
-            and translator.config_options.get("executor_options.scheduler")
-            == "distributed"
+            and translator.config_options.executor.name == "streaming"
+            and translator.config_options.executor.scheduler == "distributed"
         ):  # pragma: no cover; Requires distributed cluster
             memory_resource = rmm.mr.get_current_device_resource()
         if len(ir_translation_errors):
@@ -312,7 +305,7 @@ def execute_with_cudf(
             exception = NotImplementedError(error_message, unique_errors)
             if bool(int(os.environ.get("POLARS_VERBOSE", 0))):
                 warnings.warn(error_message, PerformanceWarning, stacklevel=2)
-            if raise_on_fail:
+            if translator.config_options.raise_on_fail:
                 raise exception
         else:
             if POLARS_VERSION_LT_125:  # pragma: no cover
@@ -321,9 +314,7 @@ def execute_with_cudf(
                         _callback,
                         ir,
                         should_time=False,
-                        device=device,
                         memory_resource=memory_resource,
-                        executor=executor,
                         config_options=translator.config_options,
                         timer=None,
                     )
@@ -333,9 +324,7 @@ def execute_with_cudf(
                     partial(
                         _callback,
                         ir,
-                        device=device,
                         memory_resource=memory_resource,
-                        executor=executor,
                         config_options=translator.config_options,
                         timer=timer,
                     )
diff --git a/python/cudf_polars/cudf_polars/dsl/ir.py b/python/cudf_polars/cudf_polars/dsl/ir.py
index eb94e2460fa..4c970ef4899 100644
--- a/python/cudf_polars/cudf_polars/dsl/ir.py
+++ b/python/cudf_polars/cudf_polars/dsl/ir.py
@@ -579,7 +579,7 @@ def do_evaluate(
                 options.set_columns(with_columns)
             if filters is not None:
                 options.set_filter(filters)
-            if config_options.get("parquet_options.chunked", default=True):
+            if config_options.parquet_options.chunked:
                 # We handle skip_rows != 0 by reading from the
                 # up to n_rows + skip_rows and slicing off the
                 # first skip_rows entries.
@@ -591,14 +591,8 @@ def do_evaluate(
                     options.set_num_rows(nrows)
                 reader = plc.io.parquet.ChunkedParquetReader(
                     options,
-                    chunk_read_limit=config_options.get(
-                        "parquet_options.chunk_read_limit",
-                        default=cls.PARQUET_DEFAULT_CHUNK_SIZE,
-                    ),
-                    pass_read_limit=config_options.get(
-                        "parquet_options.pass_read_limit",
-                        default=cls.PARQUET_DEFAULT_PASS_LIMIT,
-                    ),
+                    chunk_read_limit=config_options.parquet_options.chunk_read_limit,
+                    pass_read_limit=config_options.parquet_options.pass_read_limit,
                 )
                 chunk = reader.read_chunk()
                 rows_left_to_skip = skip_rows
diff --git a/python/cudf_polars/cudf_polars/dsl/translate.py b/python/cudf_polars/cudf_polars/dsl/translate.py
index d235de93e45..ed6fcd8ad45 100644
--- a/python/cudf_polars/cudf_polars/dsl/translate.py
+++ b/python/cudf_polars/cudf_polars/dsl/translate.py
@@ -5,7 +5,6 @@
 
 from __future__ import annotations
 
-import copy
 import functools
 import json
 from contextlib import AbstractContextManager, nullcontext
@@ -48,7 +47,7 @@ class Translator:
 
     def __init__(self, visitor: NodeTraverser, engine: GPUEngine):
         self.visitor = visitor
-        self.config_options = config.ConfigOptions(copy.deepcopy(engine.config))
+        self.config_options = config.ConfigOptions.from_polars_engine(engine)
         self.errors: list[Exception] = []
 
     def translate_ir(self, *, n: int | None = None) -> ir.IR:
diff --git a/python/cudf_polars/cudf_polars/experimental/groupby.py b/python/cudf_polars/cudf_polars/experimental/groupby.py
index a133a3047b9..fe9da9abff0 100644
--- a/python/cudf_polars/cudf_polars/experimental/groupby.py
+++ b/python/cudf_polars/cudf_polars/experimental/groupby.py
@@ -212,9 +212,14 @@ def _(
     # Check if we are dealing with any high-cardinality columns
     post_aggregation_count = 1  # Default tree reduction
     groupby_key_columns = [ne.name for ne in ir.keys]
+
+    assert ir.config_options.executor.name == "streaming", (
+        "'in-memory' executor not supported in 'generate_ir_tasks'"
+    )
+
     cardinality_factor = {
         c: min(f, 1.0)
-        for c, f in ir.config_options.get("executor_options.cardinality_factor").items()
+        for c, f in ir.config_options.executor.cardinality_factor.items()
         if c in groupby_key_columns
     }
     if cardinality_factor:
@@ -278,7 +283,11 @@ def _(
         partition_info[gb_inter] = PartitionInfo(count=post_aggregation_count)
     else:
         # N-ary tree reduction
-        n_ary = ir.config_options.get("executor_options.groupby_n_ary", default=32)
+        assert ir.config_options.executor.name == "streaming", (
+            "'in-memory' executor not supported in 'generate_ir_tasks'"
+        )
+
+        n_ary = ir.config_options.executor.groupby_n_ary
         count = child_count
         gb_inter = gb_pwise
         while count > 1:
diff --git a/python/cudf_polars/cudf_polars/experimental/io.py b/python/cudf_polars/cudf_polars/experimental/io.py
index 190f8dac2b3..35094205b08 100644
--- a/python/cudf_polars/cudf_polars/experimental/io.py
+++ b/python/cudf_polars/cudf_polars/experimental/io.py
@@ -4,6 +4,7 @@
 
 from __future__ import annotations
 
+import dataclasses
 import enum
 import math
 import random
@@ -35,10 +36,12 @@
 def _(
     ir: DataFrameScan, rec: LowerIRTransformer
 ) -> tuple[IR, MutableMapping[IR, PartitionInfo]]:
-    rows_per_partition = ir.config_options.get(
-        "executor_options.max_rows_per_partition"
+    assert ir.config_options.executor.name == "streaming", (
+        "'in-memory' executor not supported in 'generate_ir_tasks'"
     )
 
+    rows_per_partition = ir.config_options.executor.max_rows_per_partition
+
     nrows = max(ir.df.shape()[0], 1)
     count = math.ceil(nrows / rows_per_partition)
 
@@ -98,7 +101,11 @@ def from_scan(ir: Scan) -> ScanPartitionPlan:
         """Extract the partitioning plan of a Scan operation."""
         if ir.typ == "parquet":
             # TODO: Use system info to set default blocksize
-            blocksize: int = ir.config_options.get("executor_options.parquet_blocksize")
+            assert ir.config_options.executor.name == "streaming", (
+                "'in-memory' executor not supported in 'generate_ir_tasks'"
+            )
+
+            blocksize: int = ir.config_options.executor.parquet_blocksize
             # _sample_pq_statistics is generic over the bit-width of the array
             # We don't care about that here, so we ignore it.
             stats = _sample_pq_statistics(ir)  # type: ignore[var-annotated]
@@ -283,8 +290,11 @@ def _(
         paths = list(ir.paths)
         if plan.flavor == ScanPartitionFlavor.SPLIT_FILES:
             # Disable chunked reader when splitting files
-            config_options = ir.config_options.set(
-                name="parquet_options.chunked", value=False
+            config_options = dataclasses.replace(
+                ir.config_options,
+                parquet_options=dataclasses.replace(
+                    ir.config_options.parquet_options, chunked=False
+                ),
             )
 
             slices: list[SplitScan] = []
diff --git a/python/cudf_polars/cudf_polars/experimental/join.py b/python/cudf_polars/cudf_polars/experimental/join.py
index 53ead408e61..46994f00217 100644
--- a/python/cudf_polars/cudf_polars/experimental/join.py
+++ b/python/cudf_polars/cudf_polars/experimental/join.py
@@ -122,13 +122,13 @@ def _should_bcast_join(
     #    TODO: Make this value/heuristic configurable).
     #    We may want to account for the number of workers.
     # 3. The "kind" of join is compatible with a broadcast join
+    assert ir.config_options.executor.name == "streaming", (
+        "'in-memory' executor not supported in 'generate_ir_tasks'"
+    )
+
     return (
         not large_shuffled
-        and small_count
-        <= ir.config_options.get(
-            # Maximum number of "small"-table partitions to bcast
-            "executor_options.broadcast_join_limit"
-        )
+        and small_count <= ir.config_options.executor.broadcast_join_limit
         and (
             ir.options[0] == "Inner"
             or (ir.options[0] in ("Left", "Semi", "Anti") and large == left)
diff --git a/python/cudf_polars/cudf_polars/experimental/parallel.py b/python/cudf_polars/cudf_polars/experimental/parallel.py
index cd3081c83d2..607bdca83ff 100644
--- a/python/cudf_polars/cudf_polars/experimental/parallel.py
+++ b/python/cudf_polars/cudf_polars/experimental/parallel.py
@@ -138,7 +138,12 @@ def task_graph(
 
 def get_scheduler(config_options: ConfigOptions) -> Any:
     """Get appropriate task scheduler."""
-    scheduler = config_options.get("executor_options.scheduler")
+    assert config_options.executor.name == "streaming", (
+        "'in-memory' executor not supported in 'generate_ir_tasks'"
+    )
+
+    scheduler = config_options.executor.scheduler
+
     if (
         scheduler == "distributed"
     ):  # pragma: no cover; block depends on executor type and Distributed cluster
diff --git a/python/cudf_polars/cudf_polars/experimental/shuffle.py b/python/cudf_polars/cudf_polars/experimental/shuffle.py
index 80045586812..1251768c65a 100644
--- a/python/cudf_polars/cudf_polars/experimental/shuffle.py
+++ b/python/cudf_polars/cudf_polars/experimental/shuffle.py
@@ -233,7 +233,11 @@ def _(
     ir: Shuffle, partition_info: MutableMapping[IR, PartitionInfo]
 ) -> MutableMapping[Any, Any]:
     # Extract "shuffle_method" configuration
-    shuffle_method = ir.config_options.get("executor_options.shuffle_method")
+    assert ir.config_options.executor.name == "streaming", (
+        "'in-memory' executor not supported in 'generate_ir_tasks'"
+    )
+
+    shuffle_method = ir.config_options.executor.shuffle_method
 
     # Try using rapidsmpf shuffler if we have "simple" shuffle
     # keys, and the "shuffle_method" config is set to "rapidsmpf"
diff --git a/python/cudf_polars/cudf_polars/experimental/utils.py b/python/cudf_polars/cudf_polars/experimental/utils.py
index 149f46be982..36bffbe9bff 100644
--- a/python/cudf_polars/cudf_polars/experimental/utils.py
+++ b/python/cudf_polars/cudf_polars/experimental/utils.py
@@ -55,9 +55,7 @@ def _lower_ir_fallback(
     if fallback and msg:
         # Warn/raise the user if any children were collapsed
         # and the "fallback_mode" configuration is not "silent"
-        match fallback_mode := rec.state["config_options"].get(
-            "executor_options.fallback_mode"
-        ):
+        match fallback_mode := rec.state["config_options"].executor.fallback_mode:
             case "warn":
                 warnings.warn(msg, stacklevel=2)
             case "raise":
diff --git a/python/cudf_polars/cudf_polars/utils/config.py b/python/cudf_polars/cudf_polars/utils/config.py
index 49d27ef9b73..1814649cf65 100644
--- a/python/cudf_polars/cudf_polars/utils/config.py
+++ b/python/cudf_polars/cudf_polars/utils/config.py
@@ -5,209 +5,286 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class StreamingFallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How the streaming executor handles operations that don't support multiple partitions.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    Upon encountering an unsupported operation, the streaming executor will fall
+    back to using a single partition, which might use a large amount of memory.
+
+    * ``StreamingFallbackMode.WARN`` : Emit a warning and fall back to a single partition.
+    * ``StreamingFallbackMode.SILENT``: Silently fall back to a single partition.
+    * ``StreamingFallbackMode.RAISE`` : Raise an exception.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-            >>> options = options.set("parquet_options.chunked", False)
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers with the streaming executor.
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMPF`` : Use the rapidsmpf scheduler.
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
+    In :class:`StreamingExecutor`, the default of ``None`` will attempt to use
+    ``ShuffleMethod.RAPIDSMPF``, but will fall back to ``ShuffleMethod.TASKS``
+    if rapidsmpf is not installed.
+    """
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+    TASKS = "tasks"
+    RAPIDSMPF = "rapidsmpf"
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """
+    Configuration for the cudf-polars Parquet engine.
+
+    Parameters
+    ----------
+    chunked
+        Whether to use libcudf's ``ChunkedParquetReader`` to read the parquet
+        dataset in chunks. This is useful when reading very large parquet
+        files.
+    chunk_read_limit
+        Limit on total number of bytes to be returned per read, or 0 if
+        there is no limit.
+    pass_read_limit
+        Limit on the amount of memory used for reading and decompressing data
+        or 0 if there is no limit.
+    """
+
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-        Returns
-        -------
-        Valid config-option dictionary.
+    def __post_init__(self) -> None:
+        if not isinstance(self.chunked, bool):
+            raise TypeError("chunked must be a bool")
+        if not isinstance(self.chunk_read_limit, int):
+            raise TypeError("chunk_read_limit must be an int")
+        if not isinstance(self.pass_read_limit, int):
+            raise TypeError("pass_read_limit must be an int")
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
+
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor. ``Scheduler.SYNCHRONOUS``
+        by default.
+
+        Note ``scheduler="distributed"`` requires a Dask cluster to be running.
+    fallback_mode
+        How to handle errors when the GPU engine fails to execute a query.
+        ``StreamingFallbackMode.WARN`` by default.
+    max_rows_per_partition
+        The maximum number of rows to process per partition. 1_000_000 by default.
+        When the number of rows exceeds this value, the query will be split into
+        multiple partitions and executed in parallel.
+    cardinality_factor
+        A dictionary mapping column names to floats between 0 and 1 (inclusive
+        on the right).
+
+        Each factor estimates the fractional number of unique values in the
+        column. By default, ``1.0`` is used for any column not included in
+        ``cardinality_factor``.
+    parquet_blocksize
+        Controls how large parquet files are split into multiple partitions.
+        Files larger than ``parquet_blocksize`` bytes are split into multiple
+        partitions.
+    groupby_n_ary
+        The factor by which the number of partitions is decreased when performing
+        a groupby on a partitioned column. For example, if a column has 64 partitions,
+        it will first be reduced to ``ceil(64 / 32) = 2`` partitions.
+
+        This is useful when the absolute number of partitions is large.
+    broadcast_join_limit
+        The maximum number of partitions to allow for the smaller table in
+        a broadcast join.
+    shuffle_method
+        The method to use for shuffling data between workers. ``None``
+        by default, which will use 'rapidsmpf' if installed and fall back to
+        'tasks' if not.
+    """
+
+    name: Literal["streaming"] = dataclasses.field(default="streaming", init=False)
+    scheduler: Scheduler = Scheduler.SYNCHRONOUS
+    fallback_mode: StreamingFallbackMode = StreamingFallbackMode.WARN
+    max_rows_per_partition: int = 1_000_000
+    cardinality_factor: dict[str, float] = dataclasses.field(default_factory=dict)
+    parquet_blocksize: int = 1_000_000_000
+    groupby_n_ary: int = 32
+    broadcast_join_limit: int = 4
+    shuffle_method: ShuffleMethod | None = None
+
+    def __post_init__(self) -> None:
+        if self.scheduler == "synchronous" and self.shuffle_method == "rapidsmpf":
             raise ValueError(
-                f"Engine configuration contains unsupported settings: {unsupported}"
+                "rapidsmpf shuffle method is not supported for synchronous scheduler"
             )
-        assert {"chunked", "chunk_read_limit", "pass_read_limit"}.issuperset(
-            config.get("parquet_options", {})
-        )
 
-        # Validate executor_options
-        executor = config.get("executor", "in-memory")
-        if executor == "streaming":
-            executor_options, unsupported = _valid_streaming_options(
-                config["executor_options"]
-            )
-            config["executor_options"] = executor_options
-        else:
-            unsupported = config.get("executor_options", {}).keys()
-        if unsupported:
-            raise ValueError(
-                f"Unsupported executor_options for {executor}: {unsupported}"
+        # frozen dataclass, so use object.__setattr__
+        object.__setattr__(
+            self, "fallback_mode", StreamingFallbackMode(self.fallback_mode)
+        )
+        object.__setattr__(self, "scheduler", Scheduler(self.scheduler))
+        if self.shuffle_method is not None:
+            object.__setattr__(
+                self, "shuffle_method", ShuffleMethod(self.shuffle_method)
             )
 
-        return config
-
-
-def _valid_option(
-    options: MutableMapping[str, Any],
-    key: str,
-    *,
-    default: Any,
-    choices: Sequence[Any] | None = None,
-    astype: type | None = None,
-) -> MutableMapping[str, Any]:
-    """Return a valid configuration option."""
-    value = options.get(key, default)
-    if astype is not None:
-        value = astype(value)
-    if choices is not None and value not in choices:
-        raise ValueError(f"Unsupported {key} option: {value}. Choices are: {choices}")
-    options[key] = value
-    return options
-
-
-def _valid_streaming_options(
-    options: MutableMapping[str, Any],
-) -> tuple[MutableMapping[str, Any], set[str]]:
-    """Return valid options for the streaming executor."""
-    default: Any
-    choices: Sequence[Any]
-    for key in (
-        _STREAMING_OPTIONS := (
-            "scheduler",  # Validate scheduler first
-            "fallback_mode",
-            "max_rows_per_partition",
-            "parquet_blocksize",
-            "cardinality_factor",
-            "groupby_n_ary",
-            "broadcast_join_limit",
-            "shuffle_method",
+        # Type / value check everything else
+        if not isinstance(self.max_rows_per_partition, int):
+            raise TypeError("max_rows_per_partition must be an int")
+        if not isinstance(self.cardinality_factor, dict):
+            raise TypeError("cardinality_factor must be a dict of column name to float")
+        if not isinstance(self.parquet_blocksize, int):
+            raise TypeError("parquet_blocksize must be an int")
+        if not isinstance(self.groupby_n_ary, int):
+            raise TypeError("groupby_n_ary must be an int")
+        if not isinstance(self.broadcast_join_limit, int):
+            raise TypeError("broadcast_join_limit must be an int")
+
+    def __hash__(self) -> int:
+        # cardinality factory, a dict, isn't natively hashable. We'll dump it
+        # to json and hash that.
+        d = dataclasses.asdict(self)
+        d["cardinality_factor"] = json.dumps(d["cardinality_factor"])
+        return hash(tuple(sorted(d.items())))
+
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class InMemoryExecutor:
+    """
+    Configuration for the cudf-polars in-memory executor.
+
+    Parameters
+    ----------
+    scheduler:
+        The scheduler to use for the in-memory executor. Currently
+        only ``Scheduler.SYNCHRONOUS`` is supported for the in-memory executor.
+    """
+
+    name: Literal["in-memory"] = dataclasses.field(default="in-memory", init=False)
+
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class ConfigOptions:
+    """
+    Configuration for the polars GPUEngine.
+
+    Parameters
+    ----------
+    raise_on_fail
+        Whether to raise an exception when the GPU engine cannot execute a
+        query. ``False`` by default.
+    parquet_options
+        Options controlling parquet file reading and writing. See
+        :class:`ParquetOptions` for more.
+    executor
+        The executor to use for the GPU engine. See :class:`StreamingExecutor`
+        and :class:`InMemoryExecutor` for more.
+    device
+        The GPU used to run the query. If not provided, the
+        query uses the current CUDA device.
+    """
+
+    raise_on_fail: bool = False
+    parquet_options: ParquetOptions = dataclasses.field(default_factory=ParquetOptions)
+    executor: StreamingExecutor | InMemoryExecutor = dataclasses.field(
+        default_factory=InMemoryExecutor
+    )
+    device: int | None = None
+
+    @classmethod
+    def from_polars_engine(cls, engine: pl.GPUEngine) -> Self:
+        """
+        Create a `ConfigOptions` object from a `pl.GPUEngine` object.
+
+        This creates our internal, typed, configuration object from the
+        user-provided `polars.GPUEngine` object.
+        """
+        # these are the valid top-level keys in the engine.config that
+        # the user passes as **kwargs to GPUEngine.
+        valid_options = {
+            "executor",
+            "executor_options",
+            "parquet_options",
+            "raise_on_fail",
+        }
+
+        extra_options = set(engine.config.keys()) - valid_options
+        if extra_options:
+            raise TypeError(f"Unsupported executor_options: {extra_options}")
+
+        user_executor = engine.config.get("executor", "in-memory")
+        if user_executor is None:
+            user_executor = "in-memory"
+        user_executor_options = engine.config.get("executor_options", {})
+        user_parquet_options = engine.config.get("parquet_options", {})
+        user_raise_on_fail = engine.config.get("raise_on_fail", False)
+
+        # These are user-provided options, so we need to actually validate
+        # them.
+
+        if user_executor not in {"in-memory", "streaming"}:
+            raise ValueError(f"Unknown executor '{user_executor}'")
+
+        if not isinstance(user_raise_on_fail, bool):
+            raise TypeError("GPUEngine option 'raise_on_fail' must be a boolean.")
+
+        executor: InMemoryExecutor | StreamingExecutor
+
+        match user_executor:
+            case "in-memory":
+                executor = InMemoryExecutor(**user_executor_options)
+            case "streaming":
+                executor = StreamingExecutor(**user_executor_options)
+                # Update with the streaming defaults, but user options take precedence.
+
+            case _:  # pragma: no cover; Unreachable
+                raise ValueError(f"Unsupported executor: {user_executor}")
+
+        return cls(
+            raise_on_fail=user_raise_on_fail,
+            parquet_options=ParquetOptions(**user_parquet_options),
+            executor=executor,
+            device=engine.device,
         )
-    ):
-        match key:
-            case "scheduler":
-                options = _valid_option(
-                    options,
-                    key,
-                    default="synchronous",
-                    choices=("synchronous", "distributed"),
-                )
-            case "fallback_mode":
-                options = _valid_option(
-                    options, key, default="warn", choices=("warn", "raise", "silent")
-                )
-            case "max_rows_per_partition":
-                options = _valid_option(options, key, default=1_000_000, astype=int)
-            case "parquet_blocksize":
-                options = _valid_option(options, key, default=1_000_000_000, astype=int)
-            case "cardinality_factor":
-                options = _valid_option(options, key, default={}, astype=dict)
-            case "groupby_n_ary":
-                options = _valid_option(options, key, default=32, astype=int)
-            case "broadcast_join_limit":
-                default = 4 if options["scheduler"] == "distributed" else 32
-                options = _valid_option(options, key, default=default, astype=int)
-            case "shuffle_method":
-                if options["scheduler"] != "distributed":
-                    default = "tasks"
-                    choices = (None, "tasks")
-                else:  # pragma: no cover; Requires distributed testing.
-                    default = None
-                    choices = (None, "tasks", "rapidsmpf")
-                options = _valid_option(options, key, default=default, choices=choices)
-
-    return options, options.keys() - set(_STREAMING_OPTIONS)
diff --git a/python/cudf_polars/docs/overview.md b/python/cudf_polars/docs/overview.md
index 39bff4a36a6..d090a95f6a0 100644
--- a/python/cudf_polars/docs/overview.md
+++ b/python/cudf_polars/docs/overview.md
@@ -467,10 +467,24 @@ q = ...
 ir = Translator(q._ldf.visit(), pl.GPUEngine()).translate_ir()
 
 # DataFrame living on the device
-result = ir.evaluate(cache={})
+result = ir.evaluate(cache={}, timer=None)
 
 # Polars dataframe
 host_result = result.to_polars()
 ```
 
 If we get any exceptions, we can then debug as normal in Python.
+
+# Configuration
+
+Polars users can configure various options about how the plan is executed
+through the `pl.GPUEngine()`. This includes some configuration options
+defined in polars itself: https://docs.pola.rs/api/python/dev/reference/lazyframe/api/polars.lazyframe.engine_config.GPUEngine.html
+
+All additional keyword arguments are made available to cudf-polars through
+`engine.config`.
+
+To centralize validation and keep things well-typed internally, we model our
+additional configuration as a set of dataclasses defined in
+`cudf_polars/utils/config.py`. To transition from user-provided options to our
+(validated) internal options, use `ConfigOptions.from_polars_engine`.
diff --git a/python/cudf_polars/tests/experimental/test_groupby.py b/python/cudf_polars/tests/experimental/test_groupby.py
index 2618540cc2c..7ab5802263b 100644
--- a/python/cudf_polars/tests/experimental/test_groupby.py
+++ b/python/cudf_polars/tests/experimental/test_groupby.py
@@ -52,7 +52,7 @@ def test_groupby_single_partitions(df, op, keys):
             raise_on_fail=True,
             executor="streaming",
             executor_options={
-                "max_rows_per_partition": 1e9,
+                "max_rows_per_partition": int(1e9),
                 "scheduler": DEFAULT_SCHEDULER,
             },
         ),
@@ -111,7 +111,8 @@ def test_groupby_fallback(df, engine, fallback_mode):
         ctx = pytest.raises(pl.exceptions.ComputeError, match=match)
     elif fallback_mode == "foo":
         ctx = pytest.raises(
-            pl.exceptions.ComputeError, match="Unsupported fallback_mode option"
+            pl.exceptions.ComputeError,
+            match="'foo' is not a valid StreamingFallbackMode",
         )
     else:
         ctx = pytest.warns(UserWarning, match=match)
diff --git a/python/cudf_polars/tests/experimental/test_select.py b/python/cudf_polars/tests/experimental/test_select.py
index 55b8c61d581..97169976590 100644
--- a/python/cudf_polars/tests/experimental/test_select.py
+++ b/python/cudf_polars/tests/experimental/test_select.py
@@ -62,7 +62,8 @@ def test_select_reduce_fallback(df, fallback_mode):
         ctx = pytest.raises(pl.exceptions.ComputeError, match=match)
     elif fallback_mode == "foo":
         ctx = pytest.raises(
-            pl.exceptions.ComputeError, match="Unsupported fallback_mode option"
+            pl.exceptions.ComputeError,
+            match="'foo' is not a valid StreamingFallbackMode",
         )
     else:
         ctx = pytest.warns(UserWarning, match=match)
diff --git a/python/cudf_polars/tests/experimental/test_shuffle.py b/python/cudf_polars/tests/experimental/test_shuffle.py
index 6b2b8b3d2b9..849a4c6cabc 100644
--- a/python/cudf_polars/tests/experimental/test_shuffle.py
+++ b/python/cudf_polars/tests/experimental/test_shuffle.py
@@ -40,13 +40,13 @@ def df():
     )
 
 
-def test_hash_shuffle(df, engine):
+def test_hash_shuffle(df: pl.LazyFrame, engine: pl.GPUEngine) -> None:
     # Extract translated IR
     qir = Translator(df._ldf.visit(), engine).translate_ir()
 
     # Add first Shuffle node
     keys = (NamedExpr("x", Col(qir.schema["x"], "x")),)
-    options = ConfigOptions(engine.config)
+    options = ConfigOptions.from_polars_engine(engine)
     qir1 = Shuffle(qir.schema, keys, options, qir)
 
     # Add second Shuffle node (on the same keys)
diff --git a/python/cudf_polars/tests/test_config.py b/python/cudf_polars/tests/test_config.py
index 52c5c9894fe..155d004785f 100644
--- a/python/cudf_polars/tests/test_config.py
+++ b/python/cudf_polars/tests/test_config.py
@@ -1,4 +1,4 @@
-# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES.
+# SPDX-FileCopyrightText: Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES.
 # SPDX-License-Identifier: Apache-2.0
 
 from __future__ import annotations
@@ -16,6 +16,7 @@
     assert_gpu_result_equal,
     assert_ir_translation_raises,
 )
+from cudf_polars.utils.config import ConfigOptions
 
 
 def test_polars_verbose_warns(monkeypatch):
@@ -100,3 +101,160 @@ def allocate(bytes, stream):
     result = q.collect(engine=pl.GPUEngine(memory_resource=mr))
     assert_frame_equal(q.collect(), result)
     assert n_allocations > 0
+
+
+@pytest.mark.parametrize("executor", ["streaming", "in-memory"])
+def test_parquet_options(executor: str) -> None:
+    config = ConfigOptions.from_polars_engine(
+        pl.GPUEngine(
+            executor=executor,
+        )
+    )
+    assert config.parquet_options.chunked is True
+
+    config = ConfigOptions.from_polars_engine(
+        pl.GPUEngine(
+            executor=executor,
+            parquet_options={"chunked": False},
+        )
+    )
+    assert config.parquet_options.chunked is False
+
+
+def test_validate_streaming_executor_shuffle_method() -> None:
+    config = ConfigOptions.from_polars_engine(
+        pl.GPUEngine(
+            executor="streaming",
+            executor_options={"shuffle_method": "tasks"},
+        )
+    )
+    assert config.executor.shuffle_method == "tasks"
+
+    config = ConfigOptions.from_polars_engine(
+        pl.GPUEngine(
+            executor="streaming",
+            executor_options={
+                "shuffle_method": "rapidsmpf",
+                "scheduler": "distributed",
+            },
+        )
+    )
+    assert config.executor.shuffle_method == "rapidsmpf"
+
+    # rapidsmpf with sync is not allowed
+
+    with pytest.raises(ValueError, match="rapidsmpf shuffle method"):
+        ConfigOptions.from_polars_engine(
+            pl.GPUEngine(
+                executor="streaming",
+                executor_options={
+                    "shuffle_method": "rapidsmpf",
+                    "scheduler": "synchronous",
+                },
+            )
+        )
+
+
+@pytest.mark.parametrize("executor", ["in-memory", "streaming"])
+def test_hashable(executor: str) -> None:
+    config = ConfigOptions.from_polars_engine(
+        pl.GPUEngine(
+            executor=executor,
+        )
+    )
+    assert hash(config) == hash(config)
+
+
+def test_validate_fallback_mode() -> None:
+    config = ConfigOptions.from_polars_engine(
+        pl.GPUEngine(
+            executor="streaming",
+        )
+    )
+    assert config.executor.name == "streaming"
+    assert config.executor.fallback_mode == "warn"
+
+    with pytest.raises(ValueError, match="'foo' is not a valid StreamingFallbackMode"):
+        ConfigOptions.from_polars_engine(
+            pl.GPUEngine(
+                executor="streaming",
+                executor_options={"fallback_mode": "foo"},
+            )
+        )
+
+
+def test_validate_scheduler() -> None:
+    config = ConfigOptions.from_polars_engine(
+        pl.GPUEngine(
+            executor="streaming",
+        )
+    )
+    assert config.executor.scheduler == "synchronous"
+
+    with pytest.raises(ValueError, match="'foo' is not a valid Scheduler"):
+        ConfigOptions.from_polars_engine(
+            pl.GPUEngine(
+                executor="streaming",
+                executor_options={"scheduler": "foo"},
+            )
+        )
+
+
+def test_validate_shuffle_method() -> None:
+    config = ConfigOptions.from_polars_engine(
+        pl.GPUEngine(
+            executor="streaming",
+        )
+    )
+    assert config.executor.shuffle_method is None
+
+    with pytest.raises(ValueError, match="'foo' is not a valid ShuffleMethod"):
+        ConfigOptions.from_polars_engine(
+            pl.GPUEngine(
+                executor="streaming",
+                executor_options={"shuffle_method": "foo"},
+            )
+        )
+
+
+@pytest.mark.parametrize(
+    "option",
+    [
+        "max_rows_per_partition",
+        "cardinality_factor",
+        "parquet_blocksize",
+        "groupby_n_ary",
+        "broadcast_join_limit",
+    ],
+)
+def test_validate_max_rows_per_partition(option: str) -> None:
+    with pytest.raises(TypeError, match=f"{option} must be"):
+        ConfigOptions.from_polars_engine(
+            pl.GPUEngine(
+                executor="streaming",
+                executor_options={option: object()},
+            )
+        )
+
+
+@pytest.mark.parametrize("option", ["chunked", "chunk_read_limit", "pass_read_limit"])
+def test_validate_parquet_options(option: str) -> None:
+    with pytest.raises(TypeError, match=f"{option} must be"):
+        ConfigOptions.from_polars_engine(
+            pl.GPUEngine(
+                executor="streaming",
+                parquet_options={option: object()},
+            )
+        )
+
+
+def test_validate_raise_on_fail() -> None:
+    with pytest.raises(TypeError, match="'raise_on_fail' must be"):
+        ConfigOptions.from_polars_engine(
+            pl.GPUEngine(executor="streaming", raise_on_fail=object())
+        )
+
+
+def test_validate_executor() -> None:
+    with pytest.raises(ValueError, match="Unknown executor 'foo'"):
+        ConfigOptions.from_polars_engine(pl.GPUEngine(executor="foo"))
diff --git a/python/cudf_polars/tests/test_executors.py b/python/cudf_polars/tests/test_executors.py
index 3b3db73bd3a..9ccbd913869 100644
--- a/python/cudf_polars/tests/test_executors.py
+++ b/python/cudf_polars/tests/test_executors.py
@@ -74,7 +74,7 @@ def test_unknown_executor_options(executor):
 
     with pytest.raises(
         pl.exceptions.ComputeError,
-        match="Unsupported executor_options",
+        match="unexpected keyword argument 'foo'",
     ):
         df.collect(
             engine=pl.GPUEngine(


Reviews:
user: login: TomAugspurger
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T17:56:25Z
user: login: rjzamora
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-17T19:01:11Z
user: login: TomAugspurger
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T19:38:01Z
user: login: TomAugspurger
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T19:55:56Z
user: login: rjzamora
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-18T20:22:11Z
user: login: TomAugspurger
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T20:53:41Z
user: login: TomAugspurger
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T12:56:53Z
user: login: TomAugspurger
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T12:58:52Z
user: login: Matt711
body: Thanks @TomAugspurger, this is a nice change to make! I left some suggestions
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T18:19:50Z
user: login: TomAugspurger
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T18:24:45Z
user: login: rjzamora
body: Took a first pass at this. I really like the overall design!

I pointed out a few details that need to be adjusted/resolved somehow.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-21T18:43:21Z
user: login: TomAugspurger
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T18:45:43Z
user: login: TomAugspurger
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T18:58:08Z
user: login: TomAugspurger
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T19:00:07Z
user: login: TomAugspurger
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T19:02:40Z
user: login: rjzamora
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-21T19:13:47Z
user: login: Matt711
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T19:14:47Z
user: login: TomAugspurger
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T19:57:50Z
user: login: TomAugspurger
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T20:08:25Z
user: login: TomAugspurger
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T20:25:26Z
user: login: rjzamora
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-21T20:59:36Z
user: login: TomAugspurger
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T21:26:24Z
user: login: rjzamora
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-21T21:45:00Z
user: login: TomAugspurger
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-22T11:49:47Z
user: login: rjzamora
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-22T14:12:40Z
user: login: rjzamora
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-22T14:26:29Z
user: login: TomAugspurger
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-22T14:37:32Z
user: login: rjzamora
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-22T14:42:13Z
user: login: rjzamora
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-23T14:28:50Z
user: login: rjzamora
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-23T22:19:20Z
user: login: wence-
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-24T10:38:01Z
user: login: rjzamora
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-24T11:56:23Z
user: login: rjzamora
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-24T11:59:08Z
user: login: TomAugspurger
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-24T12:37:31Z
user: login: TomAugspurger
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-24T13:04:08Z
user: login: TomAugspurger
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-24T13:16:28Z
user: login: TomAugspurger
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-24T13:18:59Z
user: login: rjzamora
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-24T13:59:05Z
user: login: rjzamora
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-24T14:10:21Z
user: login: rjzamora
body: Thanks @TomAugspurger - I definitely think this is an improvement over what we have now.  My (non-blocking) preference is to remove the `parquet_blocksize` change, since I think it's outside the scope of this PR (and I'd probably prefer a different "fix" anyway).
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-24T14:33:00Z

Review comments:
diff_hunk: @@ -211,9 +211,13 @@ def _(
     # Check if we are dealing with any high-cardinality columns
     post_aggregation_count = 1  # Default tree reduction
     groupby_key_columns = [ne.name for ne in ir.keys]
+
+    # TODO: determine if InMemoryExecutor can reach here
+    assert ir.config_options.executor.name == "streaming"
path: python/cudf_polars/cudf_polars/experimental/groupby.py
user: login: TomAugspurger
body: @rjzamora I confused myself between "executors" and "schedulers", and which are possible to have where.

Is it ever possible for the `in-memory` executor to reach this code (in `experimental/groupby.py`)?

This matters because `ir.config_options.executor` has a union type: `InMemoryExecutor | StreamingExecutor`. At the moment, I haven't defined `InMemoryExecutor.cardinality_factory`, because I'd like to avoid defining configuration options that don't *actually* have an effect.

Similar question for the 3 other places where I've left "TODO: determine if InMemoryExecutor".

created_at: 2025-04-17T17:51:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -283,8 +286,11 @@ def _(
         paths = list(ir.paths)
         if plan.flavor == ScanPartitionFlavor.SPLIT_FILES:
             # Disable chunked reader when splitting files
-            config_options = ir.config_options.set(
-                name="parquet_options.chunked", value=False
+            config_options = dataclasses.replace(
path: python/cudf_polars/cudf_polars/experimental/io.py
user: login: TomAugspurger
body: `dataclasses.replace` doesn't handle "nested" replacements like `config_options.parquet_options.chunked`. This was the only place using `ConfigOptions.set` so I've just done it manually. If this becomes a pain we can define a helper to do it dynamically.
created_at: 2025-04-17T17:52:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,122 @@
 
 from __future__ import annotations
 
-import copy
-import json
-from typing import TYPE_CHECKING, Any
+import dataclasses
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
-    """
-    GPUEngine configuration-option manager.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """Configuration for the cudf-polars Parquet engine."""
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
-    """
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
+FALLBACK_MODE = Literal["warn", "raise"]
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
 
-            >>> options = options.set("parquet_options.chunked", False)
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor.
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+        "distributed" requires a Dask cluster to be running.
+    """
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
+    name: Literal["streaming"] = "streaming"
+    scheduler: Literal["synchronous", "distributed"] = "synchronous"
+    fallback_mode: FALLBACK_MODE = "warn"
+    max_rows_per_partition: int = 1_000_000
+    cardinality_factor: dict[str, float] = dataclasses.field(default_factory=dict)
+    parquet_blocksize: int = 1_000_000_000  # why isn't this a ParquetOption?
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: TomAugspurger
body: I was a little confused why this shows up as an executor option, rather than a parquet option. I guess because the blocksize we want by default depends on the executor?
created_at: 2025-04-17T17:54:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,122 @@
 
 from __future__ import annotations
 
-import copy
-import json
-from typing import TYPE_CHECKING, Any
+import dataclasses
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
-    """
-    GPUEngine configuration-option manager.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """Configuration for the cudf-polars Parquet engine."""
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
-    """
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
+FALLBACK_MODE = Literal["warn", "raise"]
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
 
-            >>> options = options.set("parquet_options.chunked", False)
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor.
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+        "distributed" requires a Dask cluster to be running.
+    """
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
+    name: Literal["streaming"] = "streaming"
+    scheduler: Literal["synchronous", "distributed"] = "synchronous"
+    fallback_mode: FALLBACK_MODE = "warn"
+    max_rows_per_partition: int = 1_000_000
+    cardinality_factor: dict[str, float] = dataclasses.field(default_factory=dict)
+    parquet_blocksize: int = 1_000_000_000  # why isn't this a ParquetOption?
+    groupby_n_ary: int = 32
+    broadcast_join_limit: int = 4
+    shuffle_method: Literal["tasks"] | None = None
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
-
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+@dataclasses.dataclass(frozen=True, eq=True)
+class InMemoryExecutor:
+    """Configuration for the cudf-polars in-memory executor."""
 
-        Returns
-        -------
-        Valid config-option dictionary.
+    name: Literal["in-memory"] = "in-memory"
+    scheduler: Literal["synchronous"] = "synchronous"
+    shuffle_method: Literal["tasks", "rapidsmpf"] | None = None
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: TomAugspurger
body: I think I mixed this up while adapting https://github.com/rapidsai/cudf/blob/76baf5ccad27f0858347e1ed8d2b6acd5f114b64/python/cudf_polars/cudf_polars/utils/config.py#L204-L210. The InMemoryExecutor should only allow `tasks` IIUC.
created_at: 2025-04-17T17:55:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,122 @@
 
 from __future__ import annotations
 
-import copy
-import json
-from typing import TYPE_CHECKING, Any
+import dataclasses
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
-    """
-    GPUEngine configuration-option manager.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """Configuration for the cudf-polars Parquet engine."""
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
-    """
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
+FALLBACK_MODE = Literal["warn", "raise"]
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
 
-            >>> options = options.set("parquet_options.chunked", False)
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor.
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+        "distributed" requires a Dask cluster to be running.
+    """
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
+    name: Literal["streaming"] = "streaming"
+    scheduler: Literal["synchronous", "distributed"] = "synchronous"
+    fallback_mode: FALLBACK_MODE = "warn"
+    max_rows_per_partition: int = 1_000_000
+    cardinality_factor: dict[str, float] = dataclasses.field(default_factory=dict)
+    parquet_blocksize: int = 1_000_000_000  # why isn't this a ParquetOption?
+    groupby_n_ary: int = 32
+    broadcast_join_limit: int = 4
+    shuffle_method: Literal["tasks"] | None = None
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
-
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+@dataclasses.dataclass(frozen=True, eq=True)
+class InMemoryExecutor:
+    """Configuration for the cudf-polars in-memory executor."""
 
-        Returns
-        -------
-        Valid config-option dictionary.
+    name: Literal["in-memory"] = "in-memory"
+    scheduler: Literal["synchronous"] = "synchronous"
+    shuffle_method: Literal["tasks", "rapidsmpf"] | None = None
+    broadcast_join_limit: int = 32
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class ConfigOptions:
+    """Configuration for the polars GPUEngine."""
+
+    raise_on_fail: bool = False
+    # device?
+    # memory resource?
+    parquet_options: ParquetOptions = dataclasses.field(default_factory=ParquetOptions)
+    executor: StreamingExecutor | InMemoryExecutor = dataclasses.field(
+        default_factory=InMemoryExecutor
+    )
+
+    @classmethod
+    def from_polars_engine(cls, engine: pl.GPUEngine) -> Self:
         """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
-            raise ValueError(
-                f"Engine configuration contains unsupported settings: {unsupported}"
-            )
-        assert {"chunked", "chunk_read_limit", "pass_read_limit"}.issuperset(
-            config.get("parquet_options", {})
-        )
+        Create a `ConfigOptions` object from a `pl.GPUEngine` object.
 
-        # Validate executor_options
-        executor = config.get("executor", "in-memory")
-        if executor == "streaming":
-            executor_options, unsupported = _valid_streaming_options(
-                config["executor_options"]
-            )
-            config["executor_options"] = executor_options
-        else:
-            unsupported = config.get("executor_options", {}).keys()
-        if unsupported:
-            raise ValueError(
-                f"Unsupported executor_options for {executor}: {unsupported}"
-            )
-
-        return config
-
-
-def _valid_option(
-    options: MutableMapping[str, Any],
-    key: str,
-    *,
-    default: Any,
-    choices: Sequence[Any] | None = None,
-    astype: type | None = None,
-) -> MutableMapping[str, Any]:
-    """Return a valid configuration option."""
-    value = options.get(key, default)
-    if astype is not None:
-        value = astype(value)
-    if choices is not None and value not in choices:
-        raise ValueError(f"Unsupported {key} option: {value}. Choices are: {choices}")
-    options[key] = value
-    return options
-
-
-def _valid_streaming_options(
-    options: MutableMapping[str, Any],
-) -> tuple[MutableMapping[str, Any], set[str]]:
-    """Return valid options for the streaming executor."""
-    default: Any
-    choices: Sequence[Any]
-    for key in (
-        _STREAMING_OPTIONS := (
-            "scheduler",  # Validate scheduler first
-            "fallback_mode",
-            "max_rows_per_partition",
-            "parquet_blocksize",
-            "cardinality_factor",
-            "groupby_n_ary",
-            "broadcast_join_limit",
-            "shuffle_method",
+        This creates our internal, typed, configuration object from the
+        user-provided `polars.GPUEngine` object.
+        """
+        # these are the valid top-level keys in the engine.config that
+        # the user passes as **kwargs to GPUEngine.
+        valid_options = {
+            "executor",
+            "executor_options",
+            "parquet_options",
+        }
+
+        extra_options = set(engine.config.keys()) - valid_options
+        if extra_options:
+            raise ValueError(f"Unsupported options: {extra_options}")
+
+        user_executor = engine.config.get("executor", "in-memory")
+        user_executor_options = engine.config.get("executor_options", {})
+        user_parquet_options = engine.config.get("parquet_options", {})
+
+        executor: InMemoryExecutor | StreamingExecutor
+        match user_executor:
+            case "in-memory":
+                executor = InMemoryExecutor(**user_executor_options)
+            case "streaming":
+                executor = StreamingExecutor(**user_executor_options)
+                # Update with the streaming defaults, but user options take precedence.
+                # TODO: test this!
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: TomAugspurger
body: Make sure to add some tests for this directly.
created_at: 2025-04-17T17:56:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -211,9 +211,13 @@ def _(
     # Check if we are dealing with any high-cardinality columns
     post_aggregation_count = 1  # Default tree reduction
     groupby_key_columns = [ne.name for ne in ir.keys]
+
+    # TODO: determine if InMemoryExecutor can reach here
+    assert ir.config_options.executor.name == "streaming"
path: python/cudf_polars/cudf_polars/experimental/groupby.py
user: login: rjzamora
body: > Is it ever possible for the in-memory executor to reach this code (in experimental/groupby.py)?

No, that would be a bug. You can think of the `cudf_polars.experimental` module as being the `cudf_polars.streaming` module - This directory only contains code that is used by the streaming/experimental executor.
created_at: 2025-04-17T19:01:11Z
author_association: MEMBER
diff_hunk: @@ -211,9 +211,13 @@ def _(
     # Check if we are dealing with any high-cardinality columns
     post_aggregation_count = 1  # Default tree reduction
     groupby_key_columns = [ne.name for ne in ir.keys]
+
+    # TODO: determine if InMemoryExecutor can reach here
+    assert ir.config_options.executor.name == "streaming"
path: python/cudf_polars/cudf_polars/experimental/groupby.py
user: login: TomAugspurger
body: Great, thanks for confirming.

In that case, I think I'll just keep the `asserts` like I have have here. Catching this a type-checking time would be possible if we made `IR` generic over the type of `config_options.executor`. IMO that's not worth it at the moment.
created_at: 2025-04-17T19:38:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,148 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
 import json
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
-    """
-    GPUEngine configuration-option manager.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """Configuration for the cudf-polars Parquet engine."""
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
-    """
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
+FALLBACK_MODE = Literal["warn", "raise"]
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
 
-            >>> options = options.set("parquet_options.chunked", False)
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
-
-        Nested dictionary keys should be separated by periods.
-        For example::
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor.
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
+        "distributed" requires a Dask cluster to be running.
+    """
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+    name: Literal["streaming"] = "streaming"
+    scheduler: Literal["synchronous", "distributed"] = "synchronous"
+    fallback_mode: FALLBACK_MODE = "warn"
+    max_rows_per_partition: int = 1_000_000
+    cardinality_factor: dict[str, float] = dataclasses.field(default_factory=dict)
+    parquet_blocksize: int = 1_000_000_000  # why isn't this a ParquetOption?
+    groupby_n_ary: int = 32
+    broadcast_join_limit: int = 4
+    shuffle_method: Literal["tasks", "rapidsmpf"] | None = None
+
+    def __post_init__(self) -> None:
+        if self.scheduler == "synchronous" and self.shuffle_method == "rapidsmpf":
+            raise ValueError(
+                "rapidsmpf shuffle method is not supported for synchronous scheduler"
+            )
 
     def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
+        # cardinatlity factory, a dict, isn't natively hashable. We'll dump it
+        # to json and hash that.
+        d = dataclasses.asdict(self)
+        d["cardinality_factor"] = json.dumps(d["cardinality_factor"])
+        return hash(tuple(sorted(d.items())))
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
 
-        Returns
-        -------
-        Valid config-option dictionary.
+@dataclasses.dataclass(frozen=True, eq=True)
+class InMemoryExecutor:
+    """Configuration for the cudf-polars in-memory executor."""
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
-            raise ValueError(
-                f"Engine configuration contains unsupported settings: {unsupported}"
-            )
-        assert {"chunked", "chunk_read_limit", "pass_read_limit"}.issuperset(
-            config.get("parquet_options", {})
-        )
+    name: Literal["in-memory"] = "in-memory"
+    scheduler: Literal["synchronous"] = "synchronous"
+    shuffle_method: Literal["tasks"] | None = None
+    broadcast_join_limit: int = 32
 
-        # Validate executor_options
-        executor = config.get("executor", "in-memory")
-        if executor == "streaming":
-            executor_options, unsupported = _valid_streaming_options(
-                config["executor_options"]
-            )
-            config["executor_options"] = executor_options
-        else:
-            unsupported = config.get("executor_options", {}).keys()
-        if unsupported:
-            raise ValueError(
-                f"Unsupported executor_options for {executor}: {unsupported}"
-            )
 
-        return config
-
-
-def _valid_option(
-    options: MutableMapping[str, Any],
-    key: str,
-    *,
-    default: Any,
-    choices: Sequence[Any] | None = None,
-    astype: type | None = None,
-) -> MutableMapping[str, Any]:
-    """Return a valid configuration option."""
-    value = options.get(key, default)
-    if astype is not None:
-        value = astype(value)
-    if choices is not None and value not in choices:
-        raise ValueError(f"Unsupported {key} option: {value}. Choices are: {choices}")
-    options[key] = value
-    return options
-
-
-def _valid_streaming_options(
-    options: MutableMapping[str, Any],
-) -> tuple[MutableMapping[str, Any], set[str]]:
-    """Return valid options for the streaming executor."""
-    default: Any
-    choices: Sequence[Any]
-    for key in (
-        _STREAMING_OPTIONS := (
-            "scheduler",  # Validate scheduler first
-            "fallback_mode",
-            "max_rows_per_partition",
-            "parquet_blocksize",
-            "cardinality_factor",
-            "groupby_n_ary",
-            "broadcast_join_limit",
-            "shuffle_method",
+@dataclasses.dataclass(frozen=True, eq=True)
+class ConfigOptions:
+    """Configuration for the polars GPUEngine."""
+
+    raise_on_fail: bool = False
+    # device?
+    # memory resource?
+    parquet_options: ParquetOptions = dataclasses.field(default_factory=ParquetOptions)
+    executor: StreamingExecutor | InMemoryExecutor = dataclasses.field(
+        default_factory=InMemoryExecutor
+    )
+
+    @classmethod
+    def from_polars_engine(cls, engine: pl.GPUEngine) -> Self:
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: TomAugspurger
body: Note to self: see if we can handle https://github.com/rapidsai/cudf/pull/18531/files#diff-2f8f881e6eb5299e80fc0808df3d453617d6618fa7a51f1e8fd76cb52f2250b0R294 here.

We can check if `engine.memory_resource is None and executor == "streaming" and scheduler == "distributed"`.

That does mean we'd maybe have two `memory_resource` objects floating around: the user-provided `memory_resource` on `pl.GPUEngine(..., memory_resource=...)` and this other one, which would presumably be a new attribute of `ConfigOptions`.
created_at: 2025-04-18T19:55:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,122 @@
 
 from __future__ import annotations
 
-import copy
-import json
-from typing import TYPE_CHECKING, Any
+import dataclasses
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
-    """
-    GPUEngine configuration-option manager.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """Configuration for the cudf-polars Parquet engine."""
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
-    """
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
+FALLBACK_MODE = Literal["warn", "raise"]
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
 
-            >>> options = options.set("parquet_options.chunked", False)
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor.
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+        "distributed" requires a Dask cluster to be running.
+    """
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
+    name: Literal["streaming"] = "streaming"
+    scheduler: Literal["synchronous", "distributed"] = "synchronous"
+    fallback_mode: FALLBACK_MODE = "warn"
+    max_rows_per_partition: int = 1_000_000
+    cardinality_factor: dict[str, float] = dataclasses.field(default_factory=dict)
+    parquet_blocksize: int = 1_000_000_000  # why isn't this a ParquetOption?
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: rjzamora
body: It's actually because the `"in-memory"` executor doesn't support this option at all.
created_at: 2025-04-18T20:22:11Z
author_association: MEMBER
diff_hunk: @@ -55,9 +55,7 @@ def _lower_ir_fallback(
     if fallback and msg:
         # Warn/raise the user if any children were collapsed
         # and the "fallback_mode" configuration is not "silent"
-        match fallback_mode := rec.state["config_options"].get(
-            "executor_options.fallback_mode"
-        ):
+        match fallback_mode := rec.state["config_options"].executor.fallback_mode:
path: python/cudf_polars/cudf_polars/experimental/utils.py
user: login: TomAugspurger
body: I missed this in the refactor earlier, because `rec.state` is just typed as `Mapping[str, Any]` so mypy didn't see that `rec.state["config_options"].get` would be an `AttributeError`.

If we think the set of things going into `rec.state` is pretty stable, we might consider making it a `typing.TypedDict`. If it's pretty dynamic, then we'll just need to be careful.
created_at: 2025-04-18T20:49:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,205 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
-    """
-    GPUEngine configuration-option manager.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """Configuration for the cudf-polars Parquet engine."""
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
-    """
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    def __post_init__(self) -> None:
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: TomAugspurger
body: I've added some basic validation to these dataclasses, since they are coming from arbitrary user input.

That's also why I've used `Enums` for some things that I'd normally just use `Literal` for: by putting them through an enum we'll validate that the user-provided value is in the set of expected values.
created_at: 2025-04-18T20:50:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,122 @@
 
 from __future__ import annotations
 
-import copy
-import json
-from typing import TYPE_CHECKING, Any
+import dataclasses
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
-    """
-    GPUEngine configuration-option manager.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """Configuration for the cudf-polars Parquet engine."""
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
-    """
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
+FALLBACK_MODE = Literal["warn", "raise"]
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
 
-            >>> options = options.set("parquet_options.chunked", False)
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor.
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+        "distributed" requires a Dask cluster to be running.
+    """
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
+    name: Literal["streaming"] = "streaming"
+    scheduler: Literal["synchronous", "distributed"] = "synchronous"
+    fallback_mode: FALLBACK_MODE = "warn"
+    max_rows_per_partition: int = 1_000_000
+    cardinality_factor: dict[str, float] = dataclasses.field(default_factory=dict)
+    parquet_blocksize: int = 1_000_000_000  # why isn't this a ParquetOption?
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: TomAugspurger
body: Ah, makes sense then.
created_at: 2025-04-18T20:53:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -185,9 +185,6 @@ def _callback(
     n_rows: int | None,
     should_time: Literal[False],
     *,
-    device: int | None,
path: python/cudf_polars/cudf_polars/callback.py
user: login: TomAugspurger
body: I've moved all of our configuration usage on to `ConfigOptions`. The user-facing API will still have the split of some things specified at the top-level of `GPUEngine(...)` and other things ending up in `GPUEngine.config`.
created_at: 2025-04-21T12:53:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -281,23 +273,15 @@ def execute_with_cudf(
     else:
         start = time.monotonic_ns()
         timer = Timer(start - duration_since_start)
-    device = config.device
-    memory_resource = config.memory_resource
-    raise_on_fail = config.config.get("raise_on_fail", False)
-    executor = config.config.get("executor", None)
+
+    # config_options = ConfigOptions.from_polars_engine(config)
path: python/cudf_polars/cudf_polars/callback.py
user: login: TomAugspurger
body: Remove this.
created_at: 2025-04-21T12:53:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -281,23 +273,15 @@ def execute_with_cudf(
     else:
         start = time.monotonic_ns()
         timer = Timer(start - duration_since_start)
-    device = config.device
-    memory_resource = config.memory_resource
-    raise_on_fail = config.config.get("raise_on_fail", False)
-    executor = config.config.get("executor", None)
+
+    # config_options = ConfigOptions.from_polars_engine(config)
+
     with nvtx.annotate(message="ConvertIR", domain="cudf_polars"):
         translator = Translator(nt, config)
         ir = translator.translate_ir()
         ir_translation_errors = translator.errors
         if timer is not None:
             timer.store(start, time.monotonic_ns(), "gpu-ir-translation")
-        if (
path: python/cudf_polars/cudf_polars/callback.py
user: login: TomAugspurger
body: This is handled in `ConfigOptions.from_polars_engine`.
created_at: 2025-04-21T12:54:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,215 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Literal
 
-if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
+import rmm.mr
 
+if TYPE_CHECKING:
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
-    """
-    GPUEngine configuration-option manager.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """Configuration for the cudf-polars Parquet engine."""
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
-    """
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    def __post_init__(self) -> None:
+        if not isinstance(self.chunked, bool):
+            raise TypeError("chunked must be a bool")
+        if not isinstance(self.chunk_read_limit, int):
+            raise TypeError("chunk_read_limit must be an int")
+        if not isinstance(self.pass_read_limit, int):
+            raise TypeError("pass_read_limit must be an int")
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+# TODO: Use enum.StrEnum when we drop Python 3.10
 
-        Nested dictionary keys should be separated by periods.
-        For example::
 
-            >>> options = options.set("parquet_options.chunked", False)
+class FallbackMode(str, enum.Enum):
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+class Scheduler(str, enum.Enum):
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-            >>> chunked = config_options.get("parquet_options.chunked")
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
+class ShuffleMethod(str, enum.Enum):
+    TASKS = "tasks"
+    RAPIDSMFP = "rapidsmpf"
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
 
-        Returns
-        -------
-        Valid config-option dictionary.
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
+    Parameters
+    ----------
+    scheduler
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: TomAugspurger
body: Finish documenting these. Also the enums and `ParquetOptions` above, 
created_at: 2025-04-21T12:56:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -185,9 +185,6 @@ def _callback(
     n_rows: int | None,
     should_time: Literal[False],
     *,
-    device: int | None,
-    memory_resource: int | None,
path: python/cudf_polars/cudf_polars/callback.py
user: login: TomAugspurger
body: I think this type signature was incorrect. https://github.com/pola-rs/polars/blame/6259441a9994a4c1f33f03c26a1a1dccd2455fce/py-polars/polars/lazyframe/engine_config.py#L39 claims that `memory_resource` should be a `DeviceMemoryResource | None`. That's what we have on `ConfigOptions` now.
created_at: 2025-04-21T12:58:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,306 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How to handle errors when the GPU engine fails to execute a query.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to the CPU engine.
+    * ``FallbackMode.RAISE`` : Raise an exception.
+    * ``FallbackMode.SILENT``: Silently fall back to the CPU engine.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
+
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-            >>> options = options.set("parquet_options.chunked", False)
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMFP`` : Use the rapidsmpf scheduler.
+    """
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    TASKS = "tasks"
+    RAPIDSMFP = "rapidsmpf"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """
+    Configuration for the cudf-polars Parquet engine.
+
+    Parameters
+    ----------
+    chunked
+        Whether to use cudf's ``ChunkedParquetReader`` to read the parquet
+        dataset in chunks. This is useful when reading very large parquet
+        files.
+    chunk_read_limit
+        Limit on total number of bytes to be returned per read, or 0 if
+        there is no limit.
+    pass_read_limit
+        Limit on the amount of memory used for reading and decompressing data
+        or 0 if there is no limit.
+    """
+
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-        Returns
-        -------
-        Valid config-option dictionary.
+    def __post_init__(self) -> None:
+        if not isinstance(self.chunked, bool):
+            raise TypeError("chunked must be a bool")
+        if not isinstance(self.chunk_read_limit, int):
+            raise TypeError("chunk_read_limit must be an int")
+        if not isinstance(self.pass_read_limit, int):
+            raise TypeError("pass_read_limit must be an int")
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
+
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor. ``Scheduler.SYNCHRONOUS``
+        by default.
+
+        Note taht the "distributed" requires a Dask cluster to be running.
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: Matt711
body: ```suggestion
        Note that `scheduler="distributed"` requires a Dask cluster to be running.
```
created_at: 2025-04-21T17:57:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,306 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How to handle errors when the GPU engine fails to execute a query.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to the CPU engine.
+    * ``FallbackMode.RAISE`` : Raise an exception.
+    * ``FallbackMode.SILENT``: Silently fall back to the CPU engine.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
+
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-            >>> options = options.set("parquet_options.chunked", False)
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMFP`` : Use the rapidsmpf scheduler.
+    """
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    TASKS = "tasks"
+    RAPIDSMFP = "rapidsmpf"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """
+    Configuration for the cudf-polars Parquet engine.
+
+    Parameters
+    ----------
+    chunked
+        Whether to use cudf's ``ChunkedParquetReader`` to read the parquet
+        dataset in chunks. This is useful when reading very large parquet
+        files.
+    chunk_read_limit
+        Limit on total number of bytes to be returned per read, or 0 if
+        there is no limit.
+    pass_read_limit
+        Limit on the amount of memory used for reading and decompressing data
+        or 0 if there is no limit.
+    """
+
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-        Returns
-        -------
-        Valid config-option dictionary.
+    def __post_init__(self) -> None:
+        if not isinstance(self.chunked, bool):
+            raise TypeError("chunked must be a bool")
+        if not isinstance(self.chunk_read_limit, int):
+            raise TypeError("chunk_read_limit must be an int")
+        if not isinstance(self.pass_read_limit, int):
+            raise TypeError("pass_read_limit must be an int")
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
+
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor. ``Scheduler.SYNCHRONOUS``
+        by default.
+
+        Note taht the "distributed" requires a Dask cluster to be running.
+
+    fallback_mode
+        How to handle errors when the GPU engine fails to execute a query.
+        ``FallbackMode.WARN`` by default.
+    max_rows_per_partition
+        The maximum number of rows to process per partition. 1,000,000 by default.
+        When the number of rows exceeds this value, the query will be split into
+        multiple partitions and executed in parallel.
+    cardinality_factor
+        A dictionary mapping column names to a float strictly greater than 0 and
+        less than or equal to 1. Each factor estimates the fractional number of
+        unique values in the column. By default, a value of ``1.0`` is used
+        for any column not found in ``cardinality_factor``.
+    parquet_blocksize
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: Matt711
body: `parquet_blocksize` is related to partitioning, so it makes sense that it is in the `StreamingExecutor`. Can we name it something like `partition_blocksize` to avoid confusion since there's a `ParquetOptions` dataclass too?
created_at: 2025-04-21T17:58:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,306 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How to handle errors when the GPU engine fails to execute a query.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to the CPU engine.
+    * ``FallbackMode.RAISE`` : Raise an exception.
+    * ``FallbackMode.SILENT``: Silently fall back to the CPU engine.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
+
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-            >>> options = options.set("parquet_options.chunked", False)
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMFP`` : Use the rapidsmpf scheduler.
+    """
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    TASKS = "tasks"
+    RAPIDSMFP = "rapidsmpf"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """
+    Configuration for the cudf-polars Parquet engine.
+
+    Parameters
+    ----------
+    chunked
+        Whether to use cudf's ``ChunkedParquetReader`` to read the parquet
+        dataset in chunks. This is useful when reading very large parquet
+        files.
+    chunk_read_limit
+        Limit on total number of bytes to be returned per read, or 0 if
+        there is no limit.
+    pass_read_limit
+        Limit on the amount of memory used for reading and decompressing data
+        or 0 if there is no limit.
+    """
+
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-        Returns
-        -------
-        Valid config-option dictionary.
+    def __post_init__(self) -> None:
+        if not isinstance(self.chunked, bool):
+            raise TypeError("chunked must be a bool")
+        if not isinstance(self.chunk_read_limit, int):
+            raise TypeError("chunk_read_limit must be an int")
+        if not isinstance(self.pass_read_limit, int):
+            raise TypeError("pass_read_limit must be an int")
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
+
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor. ``Scheduler.SYNCHRONOUS``
+        by default.
+
+        Note taht the "distributed" requires a Dask cluster to be running.
+
+    fallback_mode
+        How to handle errors when the GPU engine fails to execute a query.
+        ``FallbackMode.WARN`` by default.
+    max_rows_per_partition
+        The maximum number of rows to process per partition. 1,000,000 by default.
+        When the number of rows exceeds this value, the query will be split into
+        multiple partitions and executed in parallel.
+    cardinality_factor
+        A dictionary mapping column names to a float strictly greater than 0 and
+        less than or equal to 1. Each factor estimates the fractional number of
+        unique values in the column. By default, a value of ``1.0`` is used
+        for any column not found in ``cardinality_factor``.
+    parquet_blocksize
+        Controls how large parquet files are split into multiple partitions.
+        Files larger than ``parquet_blocksize`` bytes are split into multiple
+        partitions.
+    groupby_n_ary
+        The factor by which the number of partitions is decreased when performing
+        a groupby on a partitioned column. For example, if a column has 64 partitions,
+        it will first be reduced to ``ceil(64 / 32) = 2`` partitions.
+
+        This is useful when the absolute number of partitions is large.
+
+    broadcast_join_limit
+        The maximum number of partitions to allow for the smaller table in
+        a broadcast join.
+    shuffle_method
+        The method to use for shuffling data between workers. ``ShuffleMethod.TASKS``
+        by default.
+    """
+
+    name: Literal["streaming"] = dataclasses.field(default="streaming", init=False)
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: Matt711
body: Can we simplify?
```suggestion
    name: Literal["streaming"] = "streaming"
```
created_at: 2025-04-21T17:58:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,306 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How to handle errors when the GPU engine fails to execute a query.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to the CPU engine.
+    * ``FallbackMode.RAISE`` : Raise an exception.
+    * ``FallbackMode.SILENT``: Silently fall back to the CPU engine.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
+
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-            >>> options = options.set("parquet_options.chunked", False)
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMFP`` : Use the rapidsmpf scheduler.
+    """
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    TASKS = "tasks"
+    RAPIDSMFP = "rapidsmpf"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """
+    Configuration for the cudf-polars Parquet engine.
+
+    Parameters
+    ----------
+    chunked
+        Whether to use cudf's ``ChunkedParquetReader`` to read the parquet
+        dataset in chunks. This is useful when reading very large parquet
+        files.
+    chunk_read_limit
+        Limit on total number of bytes to be returned per read, or 0 if
+        there is no limit.
+    pass_read_limit
+        Limit on the amount of memory used for reading and decompressing data
+        or 0 if there is no limit.
+    """
+
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-        Returns
-        -------
-        Valid config-option dictionary.
+    def __post_init__(self) -> None:
+        if not isinstance(self.chunked, bool):
+            raise TypeError("chunked must be a bool")
+        if not isinstance(self.chunk_read_limit, int):
+            raise TypeError("chunk_read_limit must be an int")
+        if not isinstance(self.pass_read_limit, int):
+            raise TypeError("pass_read_limit must be an int")
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
+
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor. ``Scheduler.SYNCHRONOUS``
+        by default.
+
+        Note taht the "distributed" requires a Dask cluster to be running.
+
+    fallback_mode
+        How to handle errors when the GPU engine fails to execute a query.
+        ``FallbackMode.WARN`` by default.
+    max_rows_per_partition
+        The maximum number of rows to process per partition. 1,000,000 by default.
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: Matt711
body: nitpick: Avoids ambiguity I think
```suggestion
        The maximum number of rows to process per partition. 1_000_000 by default.
```
created_at: 2025-04-21T18:02:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,306 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How to handle errors when the GPU engine fails to execute a query.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to the CPU engine.
+    * ``FallbackMode.RAISE`` : Raise an exception.
+    * ``FallbackMode.SILENT``: Silently fall back to the CPU engine.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
+
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-            >>> options = options.set("parquet_options.chunked", False)
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMFP`` : Use the rapidsmpf scheduler.
+    """
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    TASKS = "tasks"
+    RAPIDSMFP = "rapidsmpf"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """
+    Configuration for the cudf-polars Parquet engine.
+
+    Parameters
+    ----------
+    chunked
+        Whether to use cudf's ``ChunkedParquetReader`` to read the parquet
+        dataset in chunks. This is useful when reading very large parquet
+        files.
+    chunk_read_limit
+        Limit on total number of bytes to be returned per read, or 0 if
+        there is no limit.
+    pass_read_limit
+        Limit on the amount of memory used for reading and decompressing data
+        or 0 if there is no limit.
+    """
+
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-        Returns
-        -------
-        Valid config-option dictionary.
+    def __post_init__(self) -> None:
+        if not isinstance(self.chunked, bool):
+            raise TypeError("chunked must be a bool")
+        if not isinstance(self.chunk_read_limit, int):
+            raise TypeError("chunk_read_limit must be an int")
+        if not isinstance(self.pass_read_limit, int):
+            raise TypeError("pass_read_limit must be an int")
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
+
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor. ``Scheduler.SYNCHRONOUS``
+        by default.
+
+        Note taht the "distributed" requires a Dask cluster to be running.
+
+    fallback_mode
+        How to handle errors when the GPU engine fails to execute a query.
+        ``FallbackMode.WARN`` by default.
+    max_rows_per_partition
+        The maximum number of rows to process per partition. 1,000,000 by default.
+        When the number of rows exceeds this value, the query will be split into
+        multiple partitions and executed in parallel.
+    cardinality_factor
+        A dictionary mapping column names to a float strictly greater than 0 and
+        less than or equal to 1. Each factor estimates the fractional number of
+        unique values in the column. By default, a value of ``1.0`` is used
+        for any column not found in ``cardinality_factor``.
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: Matt711
body: ```suggestion
        A dictionary mapping column names to floats strictly greater than 0 and less
        than or equal to 1. Each factor estimates the fractional number of unique values
        in the column. By default, a value of ``1.0`` is used for any column not
        included in ``cardinality_factor``.
```
created_at: 2025-04-21T18:07:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -212,9 +212,14 @@ def _(
     # Check if we are dealing with any high-cardinality columns
     post_aggregation_count = 1  # Default tree reduction
     groupby_key_columns = [ne.name for ne in ir.keys]
+
+    assert ir.config_options.executor.name == "streaming", (
+        "'in-memory' executor not supported in 'generate_ir_tasks'"
+    )
path: python/cudf_polars/cudf_polars/experimental/groupby.py
user: login: rjzamora
body: I don't think these kinds of assertions are necessary. Is there a specific reason you think the "in-memory" executor might end up here?
created_at: 2025-04-21T18:09:10Z
author_association: MEMBER
diff_hunk: @@ -283,8 +286,11 @@ def _(
         paths = list(ir.paths)
         if plan.flavor == ScanPartitionFlavor.SPLIT_FILES:
             # Disable chunked reader when splitting files
-            config_options = ir.config_options.set(
-                name="parquet_options.chunked", value=False
+            config_options = dataclasses.replace(
path: python/cudf_polars/cudf_polars/experimental/io.py
user: login: rjzamora
body: I'm hoping this pattern isn't common.
It may make sense to just disable the chunked parquet reader for the `"streaming"` executor in all cases.
created_at: 2025-04-21T18:12:33Z
author_association: MEMBER
diff_hunk: @@ -5,209 +5,306 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How to handle errors when the GPU engine fails to execute a query.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to the CPU engine.
+    * ``FallbackMode.RAISE`` : Raise an exception.
+    * ``FallbackMode.SILENT``: Silently fall back to the CPU engine.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
+
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-            >>> options = options.set("parquet_options.chunked", False)
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMFP`` : Use the rapidsmpf scheduler.
+    """
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    TASKS = "tasks"
+    RAPIDSMFP = "rapidsmpf"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """
+    Configuration for the cudf-polars Parquet engine.
+
+    Parameters
+    ----------
+    chunked
+        Whether to use cudf's ``ChunkedParquetReader`` to read the parquet
+        dataset in chunks. This is useful when reading very large parquet
+        files.
+    chunk_read_limit
+        Limit on total number of bytes to be returned per read, or 0 if
+        there is no limit.
+    pass_read_limit
+        Limit on the amount of memory used for reading and decompressing data
+        or 0 if there is no limit.
+    """
+
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-        Returns
-        -------
-        Valid config-option dictionary.
+    def __post_init__(self) -> None:
+        if not isinstance(self.chunked, bool):
+            raise TypeError("chunked must be a bool")
+        if not isinstance(self.chunk_read_limit, int):
+            raise TypeError("chunk_read_limit must be an int")
+        if not isinstance(self.pass_read_limit, int):
+            raise TypeError("pass_read_limit must be an int")
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
+
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor. ``Scheduler.SYNCHRONOUS``
+        by default.
+
+        Note taht the "distributed" requires a Dask cluster to be running.
+
+    fallback_mode
+        How to handle errors when the GPU engine fails to execute a query.
+        ``FallbackMode.WARN`` by default.
+    max_rows_per_partition
+        The maximum number of rows to process per partition. 1,000,000 by default.
+        When the number of rows exceeds this value, the query will be split into
+        multiple partitions and executed in parallel.
+    cardinality_factor
+        A dictionary mapping column names to a float strictly greater than 0 and
+        less than or equal to 1. Each factor estimates the fractional number of
+        unique values in the column. By default, a value of ``1.0`` is used
+        for any column not found in ``cardinality_factor``.
+    parquet_blocksize
+        Controls how large parquet files are split into multiple partitions.
+        Files larger than ``parquet_blocksize`` bytes are split into multiple
+        partitions.
+    groupby_n_ary
+        The factor by which the number of partitions is decreased when performing
+        a groupby on a partitioned column. For example, if a column has 64 partitions,
+        it will first be reduced to ``ceil(64 / 32) = 2`` partitions.
+
+        This is useful when the absolute number of partitions is large.
+
+    broadcast_join_limit
+        The maximum number of partitions to allow for the smaller table in
+        a broadcast join.
+    shuffle_method
+        The method to use for shuffling data between workers. ``ShuffleMethod.TASKS``
+        by default.
+    """
+
+    name: Literal["streaming"] = dataclasses.field(default="streaming", init=False)
+    scheduler: Scheduler = Scheduler.SYNCHRONOUS
+    fallback_mode: FallbackMode = FallbackMode.WARN
+    max_rows_per_partition: int = 1_000_000
+    cardinality_factor: dict[str, float] = dataclasses.field(default_factory=dict)
+    parquet_blocksize: int = 1_000_000_000  # why isn't this a ParquetOption?
+    groupby_n_ary: int = 32
+    broadcast_join_limit: int = 4
+    shuffle_method: ShuffleMethod | None = None
+
+    def __post_init__(self) -> None:
+        if self.scheduler == "synchronous" and self.shuffle_method == "rapidsmpf":
             raise ValueError(
-                f"Engine configuration contains unsupported settings: {unsupported}"
+                "rapidsmpf shuffle method is not supported for synchronous scheduler"
             )
-        assert {"chunked", "chunk_read_limit", "pass_read_limit"}.issuperset(
-            config.get("parquet_options", {})
-        )
 
-        # Validate executor_options
-        executor = config.get("executor", "in-memory")
-        if executor == "streaming":
-            executor_options, unsupported = _valid_streaming_options(
-                config["executor_options"]
+        # frozen dataclass, so use object.__setattr__
+        object.__setattr__(self, "fallback_mode", FallbackMode(self.fallback_mode))
+        object.__setattr__(self, "scheduler", Scheduler(self.scheduler))
+        if self.shuffle_method is not None:
+            object.__setattr__(
+                self, "shuffle_method", ShuffleMethod(self.shuffle_method)
             )
-            config["executor_options"] = executor_options
-        else:
-            unsupported = config.get("executor_options", {}).keys()
-        if unsupported:
+
+        # Type / value check everything else
+        if not isinstance(self.max_rows_per_partition, int):
+            raise TypeError("max_rows_per_partition must be an int")
+        if not isinstance(self.cardinality_factor, dict):
+            raise TypeError("cardinality_factor must be a dict of column name to float")
+        if not isinstance(self.parquet_blocksize, int):
+            raise TypeError("parquet_blocksize must be an int")
+        if not isinstance(self.groupby_n_ary, int):
+            raise TypeError("groupby_n_ary must be an int")
+        if not isinstance(self.broadcast_join_limit, int):
+            raise TypeError("broadcast_join_limit must be an int")
+
+    def __hash__(self) -> int:
+        # cardinatlity factory, a dict, isn't natively hashable. We'll dump it
+        # to json and hash that.
+        d = dataclasses.asdict(self)
+        d["cardinality_factor"] = json.dumps(d["cardinality_factor"])
+        return hash(tuple(sorted(d.items())))
+
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class InMemoryExecutor:
+    """
+    Configuration for the cudf-polars in-memory executor.
+
+    Parameters
+    ----------
+    scheduler:
+        The scheduler to use for the in-memory executor. Currently
+        only ``Scheduler.SYNCHRONOUS`` is supported for the in-memory executor.
+    shuffle_method
+        The method to use for shuffling data between workers. Currently only
+        ``ShuffleMethod.TASKS`` is supported for the in-memory executor.
+    broadcast_join_limit
+        The maximum number of partitions to allow for the smaller table in
+        a broadcast join.
+    """
+
+    name: Literal["in-memory"] = dataclasses.field(default="in-memory", init=False)
+    scheduler: Literal["synchronous"] = "synchronous"
+    shuffle_method: Literal["tasks"] | None = None
+    broadcast_join_limit: int = 32
+
+    def __post_init__(self) -> None:
+        if self.scheduler != "synchronous":
             raise ValueError(
-                f"Unsupported executor_options for {executor}: {unsupported}"
+                "'synchronous' is the only valid scheduler for the in-memory executor"
             )
+        if self.shuffle_method is not None and self.shuffle_method != "tasks":
+            raise ValueError(
+                "'tasks' is the only valid shuffle method for the in-memory executor"
+            )
+
+        if not isinstance(self.broadcast_join_limit, int):
+            raise TypeError("broadcast_join_limit must be an int")
+
 
-        return config
-
-
-def _valid_option(
-    options: MutableMapping[str, Any],
-    key: str,
-    *,
-    default: Any,
-    choices: Sequence[Any] | None = None,
-    astype: type | None = None,
-) -> MutableMapping[str, Any]:
-    """Return a valid configuration option."""
-    value = options.get(key, default)
-    if astype is not None:
-        value = astype(value)
-    if choices is not None and value not in choices:
-        raise ValueError(f"Unsupported {key} option: {value}. Choices are: {choices}")
-    options[key] = value
-    return options
-
-
-def _valid_streaming_options(
-    options: MutableMapping[str, Any],
-) -> tuple[MutableMapping[str, Any], set[str]]:
-    """Return valid options for the streaming executor."""
-    default: Any
-    choices: Sequence[Any]
-    for key in (
-        _STREAMING_OPTIONS := (
-            "scheduler",  # Validate scheduler first
-            "fallback_mode",
-            "max_rows_per_partition",
-            "parquet_blocksize",
-            "cardinality_factor",
-            "groupby_n_ary",
-            "broadcast_join_limit",
-            "shuffle_method",
+@dataclasses.dataclass(frozen=True, eq=True)
+class ConfigOptions:
+    """
+    Configuration for the polars GPUEngine.
+
+    Parameters
+    ----------
+    raise_on_fail
+        Whether to raise an exception when the GPU engine fails to execute a
+        query. ``False`` by default.
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: Matt711
body: ```suggestion
        Whether to raise an exception when the GPU engine cannot execute a
        query. ``False`` by default.
```
created_at: 2025-04-21T18:13:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,306 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How to handle errors when the GPU engine fails to execute a query.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to the CPU engine.
+    * ``FallbackMode.RAISE`` : Raise an exception.
+    * ``FallbackMode.SILENT``: Silently fall back to the CPU engine.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
+
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-            >>> options = options.set("parquet_options.chunked", False)
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMFP`` : Use the rapidsmpf scheduler.
+    """
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    TASKS = "tasks"
+    RAPIDSMFP = "rapidsmpf"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """
+    Configuration for the cudf-polars Parquet engine.
+
+    Parameters
+    ----------
+    chunked
+        Whether to use cudf's ``ChunkedParquetReader`` to read the parquet
+        dataset in chunks. This is useful when reading very large parquet
+        files.
+    chunk_read_limit
+        Limit on total number of bytes to be returned per read, or 0 if
+        there is no limit.
+    pass_read_limit
+        Limit on the amount of memory used for reading and decompressing data
+        or 0 if there is no limit.
+    """
+
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-        Returns
-        -------
-        Valid config-option dictionary.
+    def __post_init__(self) -> None:
+        if not isinstance(self.chunked, bool):
+            raise TypeError("chunked must be a bool")
+        if not isinstance(self.chunk_read_limit, int):
+            raise TypeError("chunk_read_limit must be an int")
+        if not isinstance(self.pass_read_limit, int):
+            raise TypeError("pass_read_limit must be an int")
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
+
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor. ``Scheduler.SYNCHRONOUS``
+        by default.
+
+        Note taht the "distributed" requires a Dask cluster to be running.
+
+    fallback_mode
+        How to handle errors when the GPU engine fails to execute a query.
+        ``FallbackMode.WARN`` by default.
+    max_rows_per_partition
+        The maximum number of rows to process per partition. 1,000,000 by default.
+        When the number of rows exceeds this value, the query will be split into
+        multiple partitions and executed in parallel.
+    cardinality_factor
+        A dictionary mapping column names to a float strictly greater than 0 and
+        less than or equal to 1. Each factor estimates the fractional number of
+        unique values in the column. By default, a value of ``1.0`` is used
+        for any column not found in ``cardinality_factor``.
+    parquet_blocksize
+        Controls how large parquet files are split into multiple partitions.
+        Files larger than ``parquet_blocksize`` bytes are split into multiple
+        partitions.
+    groupby_n_ary
+        The factor by which the number of partitions is decreased when performing
+        a groupby on a partitioned column. For example, if a column has 64 partitions,
+        it will first be reduced to ``ceil(64 / 32) = 2`` partitions.
+
+        This is useful when the absolute number of partitions is large.
+
+    broadcast_join_limit
+        The maximum number of partitions to allow for the smaller table in
+        a broadcast join.
+    shuffle_method
+        The method to use for shuffling data between workers. ``ShuffleMethod.TASKS``
+        by default.
+    """
+
+    name: Literal["streaming"] = dataclasses.field(default="streaming", init=False)
+    scheduler: Scheduler = Scheduler.SYNCHRONOUS
+    fallback_mode: FallbackMode = FallbackMode.WARN
+    max_rows_per_partition: int = 1_000_000
+    cardinality_factor: dict[str, float] = dataclasses.field(default_factory=dict)
+    parquet_blocksize: int = 1_000_000_000  # why isn't this a ParquetOption?
+    groupby_n_ary: int = 32
+    broadcast_join_limit: int = 4
+    shuffle_method: ShuffleMethod | None = None
+
+    def __post_init__(self) -> None:
+        if self.scheduler == "synchronous" and self.shuffle_method == "rapidsmpf":
             raise ValueError(
-                f"Engine configuration contains unsupported settings: {unsupported}"
+                "rapidsmpf shuffle method is not supported for synchronous scheduler"
             )
-        assert {"chunked", "chunk_read_limit", "pass_read_limit"}.issuperset(
-            config.get("parquet_options", {})
-        )
 
-        # Validate executor_options
-        executor = config.get("executor", "in-memory")
-        if executor == "streaming":
-            executor_options, unsupported = _valid_streaming_options(
-                config["executor_options"]
+        # frozen dataclass, so use object.__setattr__
+        object.__setattr__(self, "fallback_mode", FallbackMode(self.fallback_mode))
+        object.__setattr__(self, "scheduler", Scheduler(self.scheduler))
+        if self.shuffle_method is not None:
+            object.__setattr__(
+                self, "shuffle_method", ShuffleMethod(self.shuffle_method)
             )
-            config["executor_options"] = executor_options
-        else:
-            unsupported = config.get("executor_options", {}).keys()
-        if unsupported:
+
+        # Type / value check everything else
+        if not isinstance(self.max_rows_per_partition, int):
+            raise TypeError("max_rows_per_partition must be an int")
+        if not isinstance(self.cardinality_factor, dict):
+            raise TypeError("cardinality_factor must be a dict of column name to float")
+        if not isinstance(self.parquet_blocksize, int):
+            raise TypeError("parquet_blocksize must be an int")
+        if not isinstance(self.groupby_n_ary, int):
+            raise TypeError("groupby_n_ary must be an int")
+        if not isinstance(self.broadcast_join_limit, int):
+            raise TypeError("broadcast_join_limit must be an int")
+
+    def __hash__(self) -> int:
+        # cardinatlity factory, a dict, isn't natively hashable. We'll dump it
+        # to json and hash that.
+        d = dataclasses.asdict(self)
+        d["cardinality_factor"] = json.dumps(d["cardinality_factor"])
+        return hash(tuple(sorted(d.items())))
+
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class InMemoryExecutor:
+    """
+    Configuration for the cudf-polars in-memory executor.
+
+    Parameters
+    ----------
+    scheduler:
+        The scheduler to use for the in-memory executor. Currently
+        only ``Scheduler.SYNCHRONOUS`` is supported for the in-memory executor.
+    shuffle_method
+        The method to use for shuffling data between workers. Currently only
+        ``ShuffleMethod.TASKS`` is supported for the in-memory executor.
+    broadcast_join_limit
+        The maximum number of partitions to allow for the smaller table in
+        a broadcast join.
+    """
+
+    name: Literal["in-memory"] = dataclasses.field(default="in-memory", init=False)
+    scheduler: Literal["synchronous"] = "synchronous"
+    shuffle_method: Literal["tasks"] | None = None
+    broadcast_join_limit: int = 32
+
+    def __post_init__(self) -> None:
+        if self.scheduler != "synchronous":
             raise ValueError(
-                f"Unsupported executor_options for {executor}: {unsupported}"
+                "'synchronous' is the only valid scheduler for the in-memory executor"
             )
+        if self.shuffle_method is not None and self.shuffle_method != "tasks":
+            raise ValueError(
+                "'tasks' is the only valid shuffle method for the in-memory executor"
+            )
+
+        if not isinstance(self.broadcast_join_limit, int):
+            raise TypeError("broadcast_join_limit must be an int")
+
 
-        return config
-
-
-def _valid_option(
-    options: MutableMapping[str, Any],
-    key: str,
-    *,
-    default: Any,
-    choices: Sequence[Any] | None = None,
-    astype: type | None = None,
-) -> MutableMapping[str, Any]:
-    """Return a valid configuration option."""
-    value = options.get(key, default)
-    if astype is not None:
-        value = astype(value)
-    if choices is not None and value not in choices:
-        raise ValueError(f"Unsupported {key} option: {value}. Choices are: {choices}")
-    options[key] = value
-    return options
-
-
-def _valid_streaming_options(
-    options: MutableMapping[str, Any],
-) -> tuple[MutableMapping[str, Any], set[str]]:
-    """Return valid options for the streaming executor."""
-    default: Any
-    choices: Sequence[Any]
-    for key in (
-        _STREAMING_OPTIONS := (
-            "scheduler",  # Validate scheduler first
-            "fallback_mode",
-            "max_rows_per_partition",
-            "parquet_blocksize",
-            "cardinality_factor",
-            "groupby_n_ary",
-            "broadcast_join_limit",
-            "shuffle_method",
+@dataclasses.dataclass(frozen=True, eq=True)
+class ConfigOptions:
+    """
+    Configuration for the polars GPUEngine.
+
+    Parameters
+    ----------
+    raise_on_fail
+        Whether to raise an exception when the GPU engine fails to execute a
+        query. ``False`` by default.
+    parquet_options
+        Options controlling parquet file reading and writing. See
+        :class:`ParquetOptions` for more.
+    executor
+        The executor to use for the GPU engine. See :class:`StreamingExecutor`
+        and :class:`InMemoryExecutor` for more.
+    device
+        The device to use for the GPU engine.
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: Matt711
body: nitpick: From the polars docs
```suggestion
        The GPU used to run the query. If not provided, the
        query uses the current CUDA device.
```
created_at: 2025-04-21T18:14:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -467,10 +467,28 @@ q = ...
 ir = Translator(q._ldf.visit(), pl.GPUEngine()).translate_ir()
 
 # DataFrame living on the device
-result = ir.evaluate(cache={})
+result = ir.evaluate(cache={}, timer=None)
 
 # Polars dataframe
 host_result = result.to_polars()
 ```
 
 If we get any exceptions, we can then debug as normal in Python.
+
+# Configuration
+
+polars users can configure various options about how the plan is executed
+through the `pl.GPUEngine()`. This includes some configuration options
+defined in polars itself, currently:
+
+- `device`
+- `memory_resource`
+- `raise_on_fail`
path: python/cudf_polars/docs/overview.md
user: login: Matt711
body: Maybe link to the docs https://docs.pola.rs/api/python/dev/reference/lazyframe/api/polars.lazyframe.engine_config.GPUEngine.html instead of listing them?
created_at: 2025-04-21T18:17:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -100,3 +101,192 @@ def allocate(bytes, stream):
     result = q.collect(engine=pl.GPUEngine(memory_resource=mr))
     assert_frame_equal(q.collect(), result)
     assert n_allocations > 0
+
+
+def test_user_streaming_options() -> None:
+    config = ConfigOptions.from_polars_engine(
+        pl.GPUEngine(
+            executor="streaming",
+        )
+    )
+    assert config.parquet_options.chunked is False
+
+    config = ConfigOptions.from_polars_engine(
+        pl.GPUEngine(
+            executor="streaming",
+            parquet_options={"chunked": True},
+        )
+    )
+    assert config.parquet_options.chunked is True
+
+    config = ConfigOptions.from_polars_engine(
+        pl.GPUEngine(
+            executor="in-memory",
+        )
+    )
+    assert config.parquet_options.chunked is True
+
+
+def test_validate_streaming_executor_shuffle_method() -> None:
+    config = ConfigOptions.from_polars_engine(
+        pl.GPUEngine(
+            executor="streaming",
+            executor_options={"shuffle_method": "tasks"},
+        )
+    )
+    assert config.executor.shuffle_method == "tasks"
+
+    config = ConfigOptions.from_polars_engine(
+        pl.GPUEngine(
+            executor="streaming",
+            executor_options={
+                "shuffle_method": "rapidsmpf",
+                "scheduler": "distributed",
+            },
+        )
+    )
+    assert config.executor.shuffle_method == "rapidsmpf"
+
+    # rapidsmpf with sync is not allowed
+
+    with pytest.raises(ValueError, match="rapidsmpf shuffle method"):
+        ConfigOptions.from_polars_engine(
+            pl.GPUEngine(
+                executor="streaming",
+                executor_options={
+                    "shuffle_method": "rapidsmpf",
+                    "scheduler": "synchronous",
+                },
+            )
+        )
+
+
+@pytest.mark.parametrize("executor", ["in-memory", "streaming"])
+def test_hashable(executor: str) -> None:
+    config = ConfigOptions.from_polars_engine(
+        pl.GPUEngine(
+            executor=executor,
+        )
+    )
+    assert hash(config) == hash(config)
+
+
+def test_validate_fallback_mode() -> None:
+    config = ConfigOptions.from_polars_engine(
+        pl.GPUEngine(
+            executor="streaming",
+        )
+    )
+    assert config.executor.name == "streaming"
+    assert config.executor.fallback_mode == "warn"
+
+    with pytest.raises(ValueError, match="'foo' is not a valid FallbackMode"):
+        ConfigOptions.from_polars_engine(
+            pl.GPUEngine(
+                executor="streaming",
+                executor_options={"fallback_mode": "foo"},
+            )
+        )
+
+
+def test_validate_scheduler() -> None:
+    config = ConfigOptions.from_polars_engine(
+        pl.GPUEngine(
+            executor="streaming",
+        )
+    )
+    assert config.executor.scheduler == "synchronous"
+
+    with pytest.raises(ValueError, match="'foo' is not a valid Scheduler"):
+        ConfigOptions.from_polars_engine(
+            pl.GPUEngine(
+                executor="streaming",
+                executor_options={"scheduler": "foo"},
+            )
+        )
+
+
+def test_validate_shuffle_method() -> None:
+    config = ConfigOptions.from_polars_engine(
+        pl.GPUEngine(
+            executor="streaming",
+        )
+    )
+    assert config.executor.shuffle_method is None
+
+    with pytest.raises(ValueError, match="'foo' is not a valid ShuffleMethod"):
+        ConfigOptions.from_polars_engine(
+            pl.GPUEngine(
+                executor="streaming",
+                executor_options={"shuffle_method": "foo"},
+            )
+        )
+
+
+@pytest.mark.parametrize(
+    "option",
+    [
+        "max_rows_per_partition",
+        "cardinality_factor",
+        "parquet_blocksize",
+        "groupby_n_ary",
+        "broadcast_join_limit",
+    ],
+)
+def test_validate_max_rows_per_partition(option: str) -> None:
+    with pytest.raises(TypeError, match=f"{option} must be"):
+        ConfigOptions.from_polars_engine(
+            pl.GPUEngine(
+                executor="streaming",
+                executor_options={option: object()},
+            )
+        )
+
+
+def test_validate_in_memory_executor() -> None:
path: python/cudf_polars/tests/test_config.py
user: login: rjzamora
body: I don't think the "in-memory" executor should accept any of the "scheduler", "shuffle_method", **or** "broadcast_join_limit" options. Those options are only used by the streaming executor.
created_at: 2025-04-21T18:18:08Z
author_association: MEMBER
diff_hunk: @@ -100,3 +101,192 @@ def allocate(bytes, stream):
     result = q.collect(engine=pl.GPUEngine(memory_resource=mr))
     assert_frame_equal(q.collect(), result)
     assert n_allocations > 0
+
+
+def test_user_streaming_options() -> None:
path: python/cudf_polars/tests/test_config.py
user: login: rjzamora
body: Not sure about this name - Seems like this is testing the chunked-parquet default?
created_at: 2025-04-21T18:20:25Z
author_association: MEMBER
diff_hunk: @@ -185,9 +185,6 @@ def _callback(
     n_rows: int | None,
     should_time: Literal[False],
     *,
-    device: int | None,
path: python/cudf_polars/cudf_polars/callback.py
user: login: TomAugspurger
body: I had to partially revert this: `ConfigOptions` gets serialized by dask in the streaming executor with the distributed scheduler. Having a `rmm.mr.DeviceMemoryResource` instance on the ConfigOptions made it not serializable.
created_at: 2025-04-21T18:24:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -467,10 +467,28 @@ q = ...
 ir = Translator(q._ldf.visit(), pl.GPUEngine()).translate_ir()
 
 # DataFrame living on the device
-result = ir.evaluate(cache={})
+result = ir.evaluate(cache={}, timer=None)
 
 # Polars dataframe
 host_result = result.to_polars()
 ```
 
 If we get any exceptions, we can then debug as normal in Python.
+
+# Configuration
+
+polars users can configure various options about how the plan is executed
path: python/cudf_polars/docs/overview.md
user: login: rjzamora
body: ```suggestion
Polars users can configure various options about how the plan is executed
```
I think?
created_at: 2025-04-21T18:33:00Z
author_association: MEMBER
diff_hunk: @@ -5,209 +5,306 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: rjzamora
body: The `"fallback_mode"` config is currently used by the `"streaming"` executor only, and it is unrelated to CPU fallback. Rather, this tells the `"streaming"` engine how to handle fallback to single-partition execution (still on GPU).
created_at: 2025-04-21T18:35:47Z
author_association: MEMBER
diff_hunk: @@ -5,209 +5,306 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How to handle errors when the GPU engine fails to execute a query.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to the CPU engine.
+    * ``FallbackMode.RAISE`` : Raise an exception.
+    * ``FallbackMode.SILENT``: Silently fall back to the CPU engine.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
+
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-            >>> options = options.set("parquet_options.chunked", False)
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMFP`` : Use the rapidsmpf scheduler.
+    """
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    TASKS = "tasks"
+    RAPIDSMFP = "rapidsmpf"
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: rjzamora
body: We probably need an "auto" option to emulate the `None` default we had previously (i.e. we should try using "rapidsmp" when we have a distributed schedule and rapidsmpf is available, otherwise use `"tasks"`)
created_at: 2025-04-21T18:38:58Z
author_association: MEMBER
diff_hunk: @@ -5,209 +5,306 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How to handle errors when the GPU engine fails to execute a query.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to the CPU engine.
+    * ``FallbackMode.RAISE`` : Raise an exception.
+    * ``FallbackMode.SILENT``: Silently fall back to the CPU engine.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
+
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-            >>> options = options.set("parquet_options.chunked", False)
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMFP`` : Use the rapidsmpf scheduler.
+    """
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    TASKS = "tasks"
+    RAPIDSMFP = "rapidsmpf"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """
+    Configuration for the cudf-polars Parquet engine.
+
+    Parameters
+    ----------
+    chunked
+        Whether to use cudf's ``ChunkedParquetReader`` to read the parquet
+        dataset in chunks. This is useful when reading very large parquet
+        files.
+    chunk_read_limit
+        Limit on total number of bytes to be returned per read, or 0 if
+        there is no limit.
+    pass_read_limit
+        Limit on the amount of memory used for reading and decompressing data
+        or 0 if there is no limit.
+    """
+
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-        Returns
-        -------
-        Valid config-option dictionary.
+    def __post_init__(self) -> None:
+        if not isinstance(self.chunked, bool):
+            raise TypeError("chunked must be a bool")
+        if not isinstance(self.chunk_read_limit, int):
+            raise TypeError("chunk_read_limit must be an int")
+        if not isinstance(self.pass_read_limit, int):
+            raise TypeError("pass_read_limit must be an int")
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
+
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor. ``Scheduler.SYNCHRONOUS``
+        by default.
+
+        Note taht the "distributed" requires a Dask cluster to be running.
+
+    fallback_mode
+        How to handle errors when the GPU engine fails to execute a query.
+        ``FallbackMode.WARN`` by default.
+    max_rows_per_partition
+        The maximum number of rows to process per partition. 1,000,000 by default.
+        When the number of rows exceeds this value, the query will be split into
+        multiple partitions and executed in parallel.
+    cardinality_factor
+        A dictionary mapping column names to a float strictly greater than 0 and
+        less than or equal to 1. Each factor estimates the fractional number of
+        unique values in the column. By default, a value of ``1.0`` is used
+        for any column not found in ``cardinality_factor``.
+    parquet_blocksize
+        Controls how large parquet files are split into multiple partitions.
+        Files larger than ``parquet_blocksize`` bytes are split into multiple
+        partitions.
+    groupby_n_ary
+        The factor by which the number of partitions is decreased when performing
+        a groupby on a partitioned column. For example, if a column has 64 partitions,
+        it will first be reduced to ``ceil(64 / 32) = 2`` partitions.
+
+        This is useful when the absolute number of partitions is large.
+
+    broadcast_join_limit
+        The maximum number of partitions to allow for the smaller table in
+        a broadcast join.
+    shuffle_method
+        The method to use for shuffling data between workers. ``ShuffleMethod.TASKS``
+        by default.
+    """
+
+    name: Literal["streaming"] = dataclasses.field(default="streaming", init=False)
+    scheduler: Scheduler = Scheduler.SYNCHRONOUS
+    fallback_mode: FallbackMode = FallbackMode.WARN
+    max_rows_per_partition: int = 1_000_000
+    cardinality_factor: dict[str, float] = dataclasses.field(default_factory=dict)
+    parquet_blocksize: int = 1_000_000_000  # why isn't this a ParquetOption?
+    groupby_n_ary: int = 32
+    broadcast_join_limit: int = 4
+    shuffle_method: ShuffleMethod | None = None
+
+    def __post_init__(self) -> None:
+        if self.scheduler == "synchronous" and self.shuffle_method == "rapidsmpf":
             raise ValueError(
-                f"Engine configuration contains unsupported settings: {unsupported}"
+                "rapidsmpf shuffle method is not supported for synchronous scheduler"
             )
-        assert {"chunked", "chunk_read_limit", "pass_read_limit"}.issuperset(
-            config.get("parquet_options", {})
-        )
 
-        # Validate executor_options
-        executor = config.get("executor", "in-memory")
-        if executor == "streaming":
-            executor_options, unsupported = _valid_streaming_options(
-                config["executor_options"]
+        # frozen dataclass, so use object.__setattr__
+        object.__setattr__(self, "fallback_mode", FallbackMode(self.fallback_mode))
+        object.__setattr__(self, "scheduler", Scheduler(self.scheduler))
+        if self.shuffle_method is not None:
+            object.__setattr__(
+                self, "shuffle_method", ShuffleMethod(self.shuffle_method)
             )
-            config["executor_options"] = executor_options
-        else:
-            unsupported = config.get("executor_options", {}).keys()
-        if unsupported:
+
+        # Type / value check everything else
+        if not isinstance(self.max_rows_per_partition, int):
+            raise TypeError("max_rows_per_partition must be an int")
+        if not isinstance(self.cardinality_factor, dict):
+            raise TypeError("cardinality_factor must be a dict of column name to float")
+        if not isinstance(self.parquet_blocksize, int):
+            raise TypeError("parquet_blocksize must be an int")
+        if not isinstance(self.groupby_n_ary, int):
+            raise TypeError("groupby_n_ary must be an int")
+        if not isinstance(self.broadcast_join_limit, int):
+            raise TypeError("broadcast_join_limit must be an int")
+
+    def __hash__(self) -> int:
+        # cardinatlity factory, a dict, isn't natively hashable. We'll dump it
+        # to json and hash that.
+        d = dataclasses.asdict(self)
+        d["cardinality_factor"] = json.dumps(d["cardinality_factor"])
+        return hash(tuple(sorted(d.items())))
+
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class InMemoryExecutor:
+    """
+    Configuration for the cudf-polars in-memory executor.
+
+    Parameters
+    ----------
+    scheduler:
+        The scheduler to use for the in-memory executor. Currently
+        only ``Scheduler.SYNCHRONOUS`` is supported for the in-memory executor.
+    shuffle_method
+        The method to use for shuffling data between workers. Currently only
+        ``ShuffleMethod.TASKS`` is supported for the in-memory executor.
+    broadcast_join_limit
+        The maximum number of partitions to allow for the smaller table in
+        a broadcast join.
+    """
+
+    name: Literal["in-memory"] = dataclasses.field(default="in-memory", init=False)
+    scheduler: Literal["synchronous"] = "synchronous"
+    shuffle_method: Literal["tasks"] | None = None
+    broadcast_join_limit: int = 32
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: rjzamora
body: None of these are used by the in-memory executor.
created_at: 2025-04-21T18:41:29Z
author_association: MEMBER
diff_hunk: @@ -5,209 +5,306 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How to handle errors when the GPU engine fails to execute a query.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to the CPU engine.
+    * ``FallbackMode.RAISE`` : Raise an exception.
+    * ``FallbackMode.SILENT``: Silently fall back to the CPU engine.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
+
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-            >>> options = options.set("parquet_options.chunked", False)
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMFP`` : Use the rapidsmpf scheduler.
+    """
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    TASKS = "tasks"
+    RAPIDSMFP = "rapidsmpf"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """
+    Configuration for the cudf-polars Parquet engine.
+
+    Parameters
+    ----------
+    chunked
+        Whether to use cudf's ``ChunkedParquetReader`` to read the parquet
+        dataset in chunks. This is useful when reading very large parquet
+        files.
+    chunk_read_limit
+        Limit on total number of bytes to be returned per read, or 0 if
+        there is no limit.
+    pass_read_limit
+        Limit on the amount of memory used for reading and decompressing data
+        or 0 if there is no limit.
+    """
+
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-        Returns
-        -------
-        Valid config-option dictionary.
+    def __post_init__(self) -> None:
+        if not isinstance(self.chunked, bool):
+            raise TypeError("chunked must be a bool")
+        if not isinstance(self.chunk_read_limit, int):
+            raise TypeError("chunk_read_limit must be an int")
+        if not isinstance(self.pass_read_limit, int):
+            raise TypeError("pass_read_limit must be an int")
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
+
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor. ``Scheduler.SYNCHRONOUS``
+        by default.
+
+        Note taht the "distributed" requires a Dask cluster to be running.
+
+    fallback_mode
+        How to handle errors when the GPU engine fails to execute a query.
+        ``FallbackMode.WARN`` by default.
+    max_rows_per_partition
+        The maximum number of rows to process per partition. 1,000,000 by default.
+        When the number of rows exceeds this value, the query will be split into
+        multiple partitions and executed in parallel.
+    cardinality_factor
+        A dictionary mapping column names to a float strictly greater than 0 and
+        less than or equal to 1. Each factor estimates the fractional number of
+        unique values in the column. By default, a value of ``1.0`` is used
+        for any column not found in ``cardinality_factor``.
+    parquet_blocksize
+        Controls how large parquet files are split into multiple partitions.
+        Files larger than ``parquet_blocksize`` bytes are split into multiple
+        partitions.
+    groupby_n_ary
+        The factor by which the number of partitions is decreased when performing
+        a groupby on a partitioned column. For example, if a column has 64 partitions,
+        it will first be reduced to ``ceil(64 / 32) = 2`` partitions.
+
+        This is useful when the absolute number of partitions is large.
+
+    broadcast_join_limit
+        The maximum number of partitions to allow for the smaller table in
+        a broadcast join.
+    shuffle_method
+        The method to use for shuffling data between workers. ``ShuffleMethod.TASKS``
+        by default.
+    """
+
+    name: Literal["streaming"] = dataclasses.field(default="streaming", init=False)
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: TomAugspurger
body: Thanks: that's what I had initially, but moved to this and forgot to mention why: I wanted an easy way to check which executor we had without having to import the executor classes and doing an `isinstance` check, so I added this `name` field.

However, because it wouldn't be sensible to do something like `InMemoryExecutor(name="streaming")`, I changed the field to`init=False` to keep it out of the signature for the executors.

So with that in mind, LMK whether you prefer `Literal["streaming"]` vs. `Literal["streaming"] = dataclasses.field(default="streaming", init=False)`. Either works for me.
created_at: 2025-04-21T18:45:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -212,9 +212,14 @@ def _(
     # Check if we are dealing with any high-cardinality columns
     post_aggregation_count = 1  # Default tree reduction
     groupby_key_columns = [ne.name for ne in ir.keys]
+
+    assert ir.config_options.executor.name == "streaming", (
+        "'in-memory' executor not supported in 'generate_ir_tasks'"
+    )
path: python/cudf_polars/cudf_polars/experimental/groupby.py
user: login: TomAugspurger
body: The type signature of `ir.config_options.executor` is `StreamingExecutor | InMemoryExecutor`, so without this we get:

```
$ mypy python/cudf_polars/cudf_polars
python/cudf_polars/cudf_polars/experimental/groupby.py:222: error: Item "InMemoryExecutor" of "StreamingExecutor | InMemoryExecutor" has no attribute "cardinality_factor"  [union-attr]
python/cudf_polars/cudf_polars/experimental/groupby.py:286: error: Item "InMemoryExecutor" of "StreamingExecutor | InMemoryExecutor" has no attribute "groupby_n_ary"  [union-attr]
Found 2 errors in 1 file (checked 50 source files)
```

So as far as mypy knows, it *is* possible to get here with an `InMemoryExecutor`.

I think that if we wanted to avoid the `assert`s, we would need to make `IR` generic over `config_options` and `config_options` generic over `executor`. I can attempt that (though IR is subclassing the generic `Node`, so it might get a little tricky. We'll see). 
created_at: 2025-04-21T18:58:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -100,3 +101,192 @@ def allocate(bytes, stream):
     result = q.collect(engine=pl.GPUEngine(memory_resource=mr))
     assert_frame_equal(q.collect(), result)
     assert n_allocations > 0
+
+
+def test_user_streaming_options() -> None:
+    config = ConfigOptions.from_polars_engine(
+        pl.GPUEngine(
+            executor="streaming",
+        )
+    )
+    assert config.parquet_options.chunked is False
+
+    config = ConfigOptions.from_polars_engine(
+        pl.GPUEngine(
+            executor="streaming",
+            parquet_options={"chunked": True},
+        )
+    )
+    assert config.parquet_options.chunked is True
+
+    config = ConfigOptions.from_polars_engine(
+        pl.GPUEngine(
+            executor="in-memory",
+        )
+    )
+    assert config.parquet_options.chunked is True
+
+
+def test_validate_streaming_executor_shuffle_method() -> None:
+    config = ConfigOptions.from_polars_engine(
+        pl.GPUEngine(
+            executor="streaming",
+            executor_options={"shuffle_method": "tasks"},
+        )
+    )
+    assert config.executor.shuffle_method == "tasks"
+
+    config = ConfigOptions.from_polars_engine(
+        pl.GPUEngine(
+            executor="streaming",
+            executor_options={
+                "shuffle_method": "rapidsmpf",
+                "scheduler": "distributed",
+            },
+        )
+    )
+    assert config.executor.shuffle_method == "rapidsmpf"
+
+    # rapidsmpf with sync is not allowed
+
+    with pytest.raises(ValueError, match="rapidsmpf shuffle method"):
+        ConfigOptions.from_polars_engine(
+            pl.GPUEngine(
+                executor="streaming",
+                executor_options={
+                    "shuffle_method": "rapidsmpf",
+                    "scheduler": "synchronous",
+                },
+            )
+        )
+
+
+@pytest.mark.parametrize("executor", ["in-memory", "streaming"])
+def test_hashable(executor: str) -> None:
+    config = ConfigOptions.from_polars_engine(
+        pl.GPUEngine(
+            executor=executor,
+        )
+    )
+    assert hash(config) == hash(config)
+
+
+def test_validate_fallback_mode() -> None:
+    config = ConfigOptions.from_polars_engine(
+        pl.GPUEngine(
+            executor="streaming",
+        )
+    )
+    assert config.executor.name == "streaming"
+    assert config.executor.fallback_mode == "warn"
+
+    with pytest.raises(ValueError, match="'foo' is not a valid FallbackMode"):
+        ConfigOptions.from_polars_engine(
+            pl.GPUEngine(
+                executor="streaming",
+                executor_options={"fallback_mode": "foo"},
+            )
+        )
+
+
+def test_validate_scheduler() -> None:
+    config = ConfigOptions.from_polars_engine(
+        pl.GPUEngine(
+            executor="streaming",
+        )
+    )
+    assert config.executor.scheduler == "synchronous"
+
+    with pytest.raises(ValueError, match="'foo' is not a valid Scheduler"):
+        ConfigOptions.from_polars_engine(
+            pl.GPUEngine(
+                executor="streaming",
+                executor_options={"scheduler": "foo"},
+            )
+        )
+
+
+def test_validate_shuffle_method() -> None:
+    config = ConfigOptions.from_polars_engine(
+        pl.GPUEngine(
+            executor="streaming",
+        )
+    )
+    assert config.executor.shuffle_method is None
+
+    with pytest.raises(ValueError, match="'foo' is not a valid ShuffleMethod"):
+        ConfigOptions.from_polars_engine(
+            pl.GPUEngine(
+                executor="streaming",
+                executor_options={"shuffle_method": "foo"},
+            )
+        )
+
+
+@pytest.mark.parametrize(
+    "option",
+    [
+        "max_rows_per_partition",
+        "cardinality_factor",
+        "parquet_blocksize",
+        "groupby_n_ary",
+        "broadcast_join_limit",
+    ],
+)
+def test_validate_max_rows_per_partition(option: str) -> None:
+    with pytest.raises(TypeError, match=f"{option} must be"):
+        ConfigOptions.from_polars_engine(
+            pl.GPUEngine(
+                executor="streaming",
+                executor_options={option: object()},
+            )
+        )
+
+
+def test_validate_in_memory_executor() -> None:
path: python/cudf_polars/tests/test_config.py
user: login: TomAugspurger
body: Confirmed, thanks. I think I mixed up the executor and scheduler.
created_at: 2025-04-21T19:00:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -100,3 +101,192 @@ def allocate(bytes, stream):
     result = q.collect(engine=pl.GPUEngine(memory_resource=mr))
     assert_frame_equal(q.collect(), result)
     assert n_allocations > 0
+
+
+def test_user_streaming_options() -> None:
path: python/cudf_polars/tests/test_config.py
user: login: TomAugspurger
body: Yeah, this should be something like `test_parquet_options` or `test_parquet_options_defaults`
created_at: 2025-04-21T19:02:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -212,9 +212,14 @@ def _(
     # Check if we are dealing with any high-cardinality columns
     post_aggregation_count = 1  # Default tree reduction
     groupby_key_columns = [ne.name for ne in ir.keys]
+
+    assert ir.config_options.executor.name == "streaming", (
+        "'in-memory' executor not supported in 'generate_ir_tasks'"
+    )
path: python/cudf_polars/cudf_polars/experimental/groupby.py
user: login: rjzamora
body: Ah, okay - No need to avid the assertions if it's a pain.
created_at: 2025-04-21T19:13:46Z
author_association: MEMBER
diff_hunk: @@ -5,209 +5,306 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How to handle errors when the GPU engine fails to execute a query.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to the CPU engine.
+    * ``FallbackMode.RAISE`` : Raise an exception.
+    * ``FallbackMode.SILENT``: Silently fall back to the CPU engine.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
+
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-            >>> options = options.set("parquet_options.chunked", False)
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMFP`` : Use the rapidsmpf scheduler.
+    """
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    TASKS = "tasks"
+    RAPIDSMFP = "rapidsmpf"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """
+    Configuration for the cudf-polars Parquet engine.
+
+    Parameters
+    ----------
+    chunked
+        Whether to use cudf's ``ChunkedParquetReader`` to read the parquet
+        dataset in chunks. This is useful when reading very large parquet
+        files.
+    chunk_read_limit
+        Limit on total number of bytes to be returned per read, or 0 if
+        there is no limit.
+    pass_read_limit
+        Limit on the amount of memory used for reading and decompressing data
+        or 0 if there is no limit.
+    """
+
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-        Returns
-        -------
-        Valid config-option dictionary.
+    def __post_init__(self) -> None:
+        if not isinstance(self.chunked, bool):
+            raise TypeError("chunked must be a bool")
+        if not isinstance(self.chunk_read_limit, int):
+            raise TypeError("chunk_read_limit must be an int")
+        if not isinstance(self.pass_read_limit, int):
+            raise TypeError("pass_read_limit must be an int")
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
+
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor. ``Scheduler.SYNCHRONOUS``
+        by default.
+
+        Note taht the "distributed" requires a Dask cluster to be running.
+
+    fallback_mode
+        How to handle errors when the GPU engine fails to execute a query.
+        ``FallbackMode.WARN`` by default.
+    max_rows_per_partition
+        The maximum number of rows to process per partition. 1,000,000 by default.
+        When the number of rows exceeds this value, the query will be split into
+        multiple partitions and executed in parallel.
+    cardinality_factor
+        A dictionary mapping column names to a float strictly greater than 0 and
+        less than or equal to 1. Each factor estimates the fractional number of
+        unique values in the column. By default, a value of ``1.0`` is used
+        for any column not found in ``cardinality_factor``.
+    parquet_blocksize
+        Controls how large parquet files are split into multiple partitions.
+        Files larger than ``parquet_blocksize`` bytes are split into multiple
+        partitions.
+    groupby_n_ary
+        The factor by which the number of partitions is decreased when performing
+        a groupby on a partitioned column. For example, if a column has 64 partitions,
+        it will first be reduced to ``ceil(64 / 32) = 2`` partitions.
+
+        This is useful when the absolute number of partitions is large.
+
+    broadcast_join_limit
+        The maximum number of partitions to allow for the smaller table in
+        a broadcast join.
+    shuffle_method
+        The method to use for shuffling data between workers. ``ShuffleMethod.TASKS``
+        by default.
+    """
+
+    name: Literal["streaming"] = dataclasses.field(default="streaming", init=False)
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: Matt711
body: Ok we can keep it as-is
created_at: 2025-04-21T19:14:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -212,9 +212,14 @@ def _(
     # Check if we are dealing with any high-cardinality columns
     post_aggregation_count = 1  # Default tree reduction
     groupby_key_columns = [ne.name for ne in ir.keys]
+
+    assert ir.config_options.executor.name == "streaming", (
+        "'in-memory' executor not supported in 'generate_ir_tasks'"
+    )
path: python/cudf_polars/cudf_polars/experimental/groupby.py
user: login: TomAugspurger
body: Someone more familiar with mypy and generics could probably make this work, but I got stuck on the first part: making `ConfigOptions` generic over the executor. Here's my test program that failed:

<details>

```python
from __future__ import annotations

import dataclasses
import typing


@dataclasses.dataclass
class A:
    name: typing.Literal["a"] = dataclasses.field(default="a", init=False)


@dataclasses.dataclass
class B:
    name: typing.Literal["b"] = dataclasses.field(default="b", init=False)


E = typing.TypeVar("E", A, B)


@dataclasses.dataclass
class C(typing.Generic[E]):
    value: E = dataclasses.field(default_factory=A)

    @classmethod
    def from_value(cls, value: str) -> C[E]:
        match value:
            case "a":
                return cls(value=A())
            case "b":
                return cls(value=B())
            case _:
                raise ValueError(f"Invalid value: {value}")


reveal_type(C.from_value("a"))
reveal_type(C.from_value("b"))
```

</details>

which gives

```
$ mypy t.py
t.py:22: error: Incompatible types in assignment (expression has type "A", variable has type "E")  [assignment]
t.py:28: error: Argument "value" to "C" has incompatible type "A"; expected "B"  [arg-type]
t.py:30: error: Argument "value" to "C" has incompatible type "B"; expected "A"  [arg-type]
t.py:35: note: Revealed type is "t.C[t.A]"
t.py:36: note: Revealed type is "t.C[t.A]"
Found 3 errors in 1 file (checked 1 source file)
```
created_at: 2025-04-21T19:57:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,306 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How to handle errors when the GPU engine fails to execute a query.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to the CPU engine.
+    * ``FallbackMode.RAISE`` : Raise an exception.
+    * ``FallbackMode.SILENT``: Silently fall back to the CPU engine.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
+
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-            >>> options = options.set("parquet_options.chunked", False)
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMFP`` : Use the rapidsmpf scheduler.
+    """
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    TASKS = "tasks"
+    RAPIDSMFP = "rapidsmpf"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """
+    Configuration for the cudf-polars Parquet engine.
+
+    Parameters
+    ----------
+    chunked
+        Whether to use cudf's ``ChunkedParquetReader`` to read the parquet
+        dataset in chunks. This is useful when reading very large parquet
+        files.
+    chunk_read_limit
+        Limit on total number of bytes to be returned per read, or 0 if
+        there is no limit.
+    pass_read_limit
+        Limit on the amount of memory used for reading and decompressing data
+        or 0 if there is no limit.
+    """
+
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-        Returns
-        -------
-        Valid config-option dictionary.
+    def __post_init__(self) -> None:
+        if not isinstance(self.chunked, bool):
+            raise TypeError("chunked must be a bool")
+        if not isinstance(self.chunk_read_limit, int):
+            raise TypeError("chunk_read_limit must be an int")
+        if not isinstance(self.pass_read_limit, int):
+            raise TypeError("pass_read_limit must be an int")
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
+
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor. ``Scheduler.SYNCHRONOUS``
+        by default.
+
+        Note taht the "distributed" requires a Dask cluster to be running.
+
+    fallback_mode
+        How to handle errors when the GPU engine fails to execute a query.
+        ``FallbackMode.WARN`` by default.
+    max_rows_per_partition
+        The maximum number of rows to process per partition. 1,000,000 by default.
+        When the number of rows exceeds this value, the query will be split into
+        multiple partitions and executed in parallel.
+    cardinality_factor
+        A dictionary mapping column names to a float strictly greater than 0 and
+        less than or equal to 1. Each factor estimates the fractional number of
+        unique values in the column. By default, a value of ``1.0`` is used
+        for any column not found in ``cardinality_factor``.
+    parquet_blocksize
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: TomAugspurger
body: Good call. It's a bit verbose, but I think `parquet_partition_blocksize` might be best since this is specific to parquet.  @rjzamora how does that sound? I can put in a bit of compat code to accept either the new or old name.
created_at: 2025-04-21T20:08:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,306 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How to handle errors when the GPU engine fails to execute a query.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to the CPU engine.
+    * ``FallbackMode.RAISE`` : Raise an exception.
+    * ``FallbackMode.SILENT``: Silently fall back to the CPU engine.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
+
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-            >>> options = options.set("parquet_options.chunked", False)
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMFP`` : Use the rapidsmpf scheduler.
+    """
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    TASKS = "tasks"
+    RAPIDSMFP = "rapidsmpf"
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: TomAugspurger
body: I'll document this, but on `ConfigOptions` this is `shuffle_method: ShuffleMethod | None` and the default is to try rapidsmpf and fall back.

We *could* add `"auto"` as an allowed value here, but that would maybe require importing rapidsmp when we initialize the config.
created_at: 2025-04-21T20:25:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,306 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How to handle errors when the GPU engine fails to execute a query.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to the CPU engine.
+    * ``FallbackMode.RAISE`` : Raise an exception.
+    * ``FallbackMode.SILENT``: Silently fall back to the CPU engine.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
+
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-            >>> options = options.set("parquet_options.chunked", False)
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMFP`` : Use the rapidsmpf scheduler.
+    """
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    TASKS = "tasks"
+    RAPIDSMFP = "rapidsmpf"
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: rjzamora
body: >I'll document this, but on ConfigOptions this is shuffle_method: ShuffleMethod | None and the default is to try rapidsmpf and fall back.

Ah, okay.

>We could add "auto" as an allowed value here, but that would maybe require importing rapidsmp when we initialize the config.

I was imagining the case that `"auto"` **is** the shuffle method as far as the `ShuffleMethod` is concerned. We wouldn't check if rapidsmpf was available until we needed to build the graph.
created_at: 2025-04-21T20:59:36Z
author_association: MEMBER
diff_hunk: @@ -5,209 +5,306 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How to handle errors when the GPU engine fails to execute a query.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to the CPU engine.
+    * ``FallbackMode.RAISE`` : Raise an exception.
+    * ``FallbackMode.SILENT``: Silently fall back to the CPU engine.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
+
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-            >>> options = options.set("parquet_options.chunked", False)
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMFP`` : Use the rapidsmpf scheduler.
+    """
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    TASKS = "tasks"
+    RAPIDSMFP = "rapidsmpf"
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: TomAugspurger
body: Cool, that's what we should have now, it's just via `executor.shuffle_method is None`.
created_at: 2025-04-21T21:26:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,313 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+import warnings
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How the streaming executor handles operations that don't support multiple partitions.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    Upon encountering an unsupported operation, the streaming executor will fall
+    back to using a single-partition, which might use a large amount of memory.
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: rjzamora
body: ```suggestion
    back to using a single partition, which might use a large amount of memory.
```
Nit.
created_at: 2025-04-21T21:37:01Z
author_association: MEMBER
diff_hunk: @@ -5,209 +5,313 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+import warnings
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How the streaming executor handles operations that don't support multiple partitions.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    Upon encountering an unsupported operation, the streaming executor will fall
+    back to using a single-partition, which might use a large amount of memory.
+
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to the CPU engine.
+    * ``FallbackMode.SILENT``: Silently fall back to the CPU engine.
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: rjzamora
body: ```suggestion
    * ``FallbackMode.WARN`` : Emit a warning and fall back to a single partition.
    * ``FallbackMode.SILENT``: Silently fall back to a single partition.
```
created_at: 2025-04-21T21:38:44Z
author_association: MEMBER
diff_hunk: @@ -5,209 +5,313 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+import warnings
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How the streaming executor handles operations that don't support multiple partitions.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    Upon encountering an unsupported operation, the streaming executor will fall
+    back to using a single-partition, which might use a large amount of memory.
+
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to the CPU engine.
+    * ``FallbackMode.SILENT``: Silently fall back to the CPU engine.
+    * ``FallbackMode.RAISE`` : Raise an exception.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-            >>> options = options.set("parquet_options.chunked", False)
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers with the streaming executor.
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMPF`` : Use the rapidsmpf scheduler.
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
+    In :class:`StreamingExecutor`, the default of ``None`` will attempt to use
+    ``ShuffleMethod.RAPIDSMPF``, but will fall back to ``ShuffleMethod.TASKS``
+    if rapidsmpf is not installed.
+    """
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+    TASKS = "tasks"
+    RAPIDSMPF = "rapidsmpf"
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
 
-        Returns
-        -------
-        Valid config-option dictionary.
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """
+    Configuration for the cudf-polars Parquet engine.
+
+    Parameters
+    ----------
+    chunked
+        Whether to use cudf's ``ChunkedParquetReader`` to read the parquet
+        dataset in chunks. This is useful when reading very large parquet
+        files.
+    chunk_read_limit
+        Limit on total number of bytes to be returned per read, or 0 if
+        there is no limit.
+    pass_read_limit
+        Limit on the amount of memory used for reading and decompressing data
+        or 0 if there is no limit.
+    """
+
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
+
+    def __post_init__(self) -> None:
+        if not isinstance(self.chunked, bool):
+            raise TypeError("chunked must be a bool")
+        if not isinstance(self.chunk_read_limit, int):
+            raise TypeError("chunk_read_limit must be an int")
+        if not isinstance(self.pass_read_limit, int):
+            raise TypeError("pass_read_limit must be an int")
+
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
+
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor. ``Scheduler.SYNCHRONOUS``
+        by default.
+
+        Note ``scheduler="distributed"`` requires a Dask cluster to be running.
+
+    fallback_mode
+        How to handle errors when the GPU engine fails to execute a query.
+        ``FallbackMode.WARN`` by default.
+    max_rows_per_partition
+        The maximum number of rows to process per partition. 1_000_000 by default.
+        When the number of rows exceeds this value, the query will be split into
+        multiple partitions and executed in parallel.
+    cardinality_factor
+        A dictionary mapping column names floats strictly greater than 0 and less
+        than or equal to 1. Each factor estimates the fractional number of unique values
+        in the column. By default, a value of ``1.0`` is used for any column not
+        included in ``cardinality_factor``.
+    parquet_partition_blocksize
+        Controls how large parquet files are split into multiple partitions.
+        Files larger than ``parquet_partition_blocksize`` bytes are split into multiple
+        partitions.
+    groupby_n_ary
+        The factor by which the number of partitions is decreased when performing
+        a groupby on a partitioned column. For example, if a column has 64 partitions,
+        it will first be reduced to ``ceil(64 / 32) = 2`` partitions.
+
+        This is useful when the absolute number of partitions is large.
+
+    broadcast_join_limit
+        The maximum number of partitions to allow for the smaller table in
+        a broadcast join.
+    shuffle_method
+        The method to use for shuffling data between workers. ``None``
+        by default, which will use 'rapidsmpf' if installed and fall back to
+        'tasks' if not.
+    """
+
+    name: Literal["streaming"] = dataclasses.field(default="streaming", init=False)
+    scheduler: Scheduler = Scheduler.SYNCHRONOUS
+    fallback_mode: FallbackMode = FallbackMode.WARN
+    max_rows_per_partition: int = 1_000_000
+    cardinality_factor: dict[str, float] = dataclasses.field(default_factory=dict)
+    parquet_partition_blocksize: int = 1_000_000_000
+    groupby_n_ary: int = 32
+    broadcast_join_limit: int = 4
+    shuffle_method: ShuffleMethod | None = None
+
+    def __post_init__(self) -> None:
+        if self.scheduler == "synchronous" and self.shuffle_method == "rapidsmpf":
             raise ValueError(
-                f"Engine configuration contains unsupported settings: {unsupported}"
+                "rapidsmpf shuffle method is not supported for synchronous scheduler"
             )
-        assert {"chunked", "chunk_read_limit", "pass_read_limit"}.issuperset(
-            config.get("parquet_options", {})
-        )
 
-        # Validate executor_options
-        executor = config.get("executor", "in-memory")
-        if executor == "streaming":
-            executor_options, unsupported = _valid_streaming_options(
-                config["executor_options"]
-            )
-            config["executor_options"] = executor_options
-        else:
-            unsupported = config.get("executor_options", {}).keys()
-        if unsupported:
-            raise ValueError(
-                f"Unsupported executor_options for {executor}: {unsupported}"
+        # frozen dataclass, so use object.__setattr__
+        object.__setattr__(self, "fallback_mode", FallbackMode(self.fallback_mode))
+        object.__setattr__(self, "scheduler", Scheduler(self.scheduler))
+        if self.shuffle_method is not None:
+            object.__setattr__(
+                self, "shuffle_method", ShuffleMethod(self.shuffle_method)
             )
 
-        return config
-
-
-def _valid_option(
-    options: MutableMapping[str, Any],
-    key: str,
-    *,
-    default: Any,
-    choices: Sequence[Any] | None = None,
-    astype: type | None = None,
-) -> MutableMapping[str, Any]:
-    """Return a valid configuration option."""
-    value = options.get(key, default)
-    if astype is not None:
-        value = astype(value)
-    if choices is not None and value not in choices:
-        raise ValueError(f"Unsupported {key} option: {value}. Choices are: {choices}")
-    options[key] = value
-    return options
-
-
-def _valid_streaming_options(
-    options: MutableMapping[str, Any],
-) -> tuple[MutableMapping[str, Any], set[str]]:
-    """Return valid options for the streaming executor."""
-    default: Any
-    choices: Sequence[Any]
-    for key in (
-        _STREAMING_OPTIONS := (
-            "scheduler",  # Validate scheduler first
-            "fallback_mode",
-            "max_rows_per_partition",
-            "parquet_blocksize",
-            "cardinality_factor",
-            "groupby_n_ary",
-            "broadcast_join_limit",
-            "shuffle_method",
+        # Type / value check everything else
+        if not isinstance(self.max_rows_per_partition, int):
+            raise TypeError("max_rows_per_partition must be an int")
+        if not isinstance(self.cardinality_factor, dict):
+            raise TypeError("cardinality_factor must be a dict of column name to float")
+        if not isinstance(self.parquet_partition_blocksize, int):
+            raise TypeError("parquet_partition_blocksize must be an int")
+        if not isinstance(self.groupby_n_ary, int):
+            raise TypeError("groupby_n_ary must be an int")
+        if not isinstance(self.broadcast_join_limit, int):
+            raise TypeError("broadcast_join_limit must be an int")
+
+    def __hash__(self) -> int:
+        # cardinatlity factory, a dict, isn't natively hashable. We'll dump it
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: rjzamora
body: ```suggestion
        # cardinality factory, a dict, isn't natively hashable. We'll dump it
```
created_at: 2025-04-21T21:44:00Z
author_association: MEMBER
diff_hunk: @@ -5,209 +5,313 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+import warnings
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How the streaming executor handles operations that don't support multiple partitions.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    Upon encountering an unsupported operation, the streaming executor will fall
+    back to using a single-partition, which might use a large amount of memory.
+
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to the CPU engine.
+    * ``FallbackMode.SILENT``: Silently fall back to the CPU engine.
+    * ``FallbackMode.RAISE`` : Raise an exception.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-            >>> options = options.set("parquet_options.chunked", False)
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers with the streaming executor.
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMPF`` : Use the rapidsmpf scheduler.
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
+    In :class:`StreamingExecutor`, the default of ``None`` will attempt to use
+    ``ShuffleMethod.RAPIDSMPF``, but will fall back to ``ShuffleMethod.TASKS``
+    if rapidsmpf is not installed.
+    """
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+    TASKS = "tasks"
+    RAPIDSMPF = "rapidsmpf"
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: TomAugspurger
body: @rjzamora strange question: but do you remember why I added this?

This is what's causing us to miss covering [these lines](https://github.com/rapidsai/cudf/blob/79b23be786420824ea6dcaefe3b8956ca3fc33f9/python/cudf_polars/cudf_polars/dsl/ir.py#L632-L634) in `ir.py`. On `branch-25.06`, `config_options.get("parquet_options.chunked", True)` resolves to `True` because `parquet_options.chunked` is None. That seems to be the only test covering this, and so we fail.

This block defines the per-executor defaults for `ParquetOptions`. I'm probably misremembering, but I thought you mentioned `chunked=False` being desirable for the streaming executor.
created_at: 2025-04-22T11:49:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,313 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+import warnings
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How the streaming executor handles operations that don't support multiple partitions.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    Upon encountering an unsupported operation, the streaming executor will fall
+    back to using a single-partition, which might use a large amount of memory.
+
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to the CPU engine.
+    * ``FallbackMode.SILENT``: Silently fall back to the CPU engine.
+    * ``FallbackMode.RAISE`` : Raise an exception.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-            >>> options = options.set("parquet_options.chunked", False)
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers with the streaming executor.
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMPF`` : Use the rapidsmpf scheduler.
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
+    In :class:`StreamingExecutor`, the default of ``None`` will attempt to use
+    ``ShuffleMethod.RAPIDSMPF``, but will fall back to ``ShuffleMethod.TASKS``
+    if rapidsmpf is not installed.
+    """
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+    TASKS = "tasks"
+    RAPIDSMPF = "rapidsmpf"
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: rjzamora
body: On branch-25.06 we change the `"parquet_options.chunked"` config to `False` when we construct a `SplitScan` node (i.e. when we decide to create new IR nodes to read  _partial_ files). When we don't run into large files, the streaming executor doesn't bother changing any defaults, so we still use/cover this chunked-reader code path.
created_at: 2025-04-22T14:12:40Z
author_association: MEMBER
diff_hunk: @@ -5,209 +5,313 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+import warnings
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How the streaming executor handles operations that don't support multiple partitions.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    Upon encountering an unsupported operation, the streaming executor will fall
+    back to using a single-partition, which might use a large amount of memory.
+
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to the CPU engine.
+    * ``FallbackMode.SILENT``: Silently fall back to the CPU engine.
+    * ``FallbackMode.RAISE`` : Raise an exception.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-            >>> options = options.set("parquet_options.chunked", False)
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers with the streaming executor.
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMPF`` : Use the rapidsmpf scheduler.
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
+    In :class:`StreamingExecutor`, the default of ``None`` will attempt to use
+    ``ShuffleMethod.RAPIDSMPF``, but will fall back to ``ShuffleMethod.TASKS``
+    if rapidsmpf is not installed.
+    """
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+    TASKS = "tasks"
+    RAPIDSMPF = "rapidsmpf"
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: rjzamora
body: I think it makes sense to remove this default for the streaming executor. I'm pretty sure you are using `replace` to disable the chunked reader in the split-file case.
created_at: 2025-04-22T14:26:29Z
author_association: MEMBER
diff_hunk: @@ -5,209 +5,313 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+import warnings
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How the streaming executor handles operations that don't support multiple partitions.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    Upon encountering an unsupported operation, the streaming executor will fall
+    back to using a single-partition, which might use a large amount of memory.
+
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to the CPU engine.
+    * ``FallbackMode.SILENT``: Silently fall back to the CPU engine.
+    * ``FallbackMode.RAISE`` : Raise an exception.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-            >>> options = options.set("parquet_options.chunked", False)
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers with the streaming executor.
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMPF`` : Use the rapidsmpf scheduler.
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
+    In :class:`StreamingExecutor`, the default of ``None`` will attempt to use
+    ``ShuffleMethod.RAPIDSMPF``, but will fall back to ``ShuffleMethod.TASKS``
+    if rapidsmpf is not installed.
+    """
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+    TASKS = "tasks"
+    RAPIDSMPF = "rapidsmpf"
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: TomAugspurger
body: Thanks for clearing that up. Sounds like I mistook the specific guidance to use `chunked=False` in that specific case to mean use `chunked=False` always with `executor="streaming"`.
created_at: 2025-04-22T14:37:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,313 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+import warnings
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How the streaming executor handles operations that don't support multiple partitions.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    Upon encountering an unsupported operation, the streaming executor will fall
+    back to using a single-partition, which might use a large amount of memory.
+
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to the CPU engine.
+    * ``FallbackMode.SILENT``: Silently fall back to the CPU engine.
+    * ``FallbackMode.RAISE`` : Raise an exception.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-            >>> options = options.set("parquet_options.chunked", False)
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers with the streaming executor.
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMPF`` : Use the rapidsmpf scheduler.
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
+    In :class:`StreamingExecutor`, the default of ``None`` will attempt to use
+    ``ShuffleMethod.RAPIDSMPF``, but will fall back to ``ShuffleMethod.TASKS``
+    if rapidsmpf is not installed.
+    """
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+    TASKS = "tasks"
+    RAPIDSMPF = "rapidsmpf"
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: rjzamora
body: >Sounds like I mistook the specific guidance to use chunked=False in that specific case to mean use chunked=False always with executor="streaming".

I probably did say that we could disallow chunked=True with the streaming executor if it makes things "easier." In this case, it looks like it just complicates things a bit.
created_at: 2025-04-22T14:42:13Z
author_association: MEMBER
diff_hunk: @@ -212,9 +212,14 @@ def _(
     # Check if we are dealing with any high-cardinality columns
     post_aggregation_count = 1  # Default tree reduction
     groupby_key_columns = [ne.name for ne in ir.keys]
+
+    assert ir.config_options.executor.name == "streaming", (
+        "'in-memory' executor not supported in 'generate_ir_tasks'"
+    )
path: python/cudf_polars/cudf_polars/experimental/groupby.py
user: login: rjzamora
body: I think this is probably the only sticking point I see with this PR (though I know it's also a bit nit-picky).

@wence- - You have worked through some pretty-painful typing situations recently (IMO). How difficult would it be to avoid the need for these kinds of assertions?
created_at: 2025-04-23T14:28:50Z
author_association: MEMBER
diff_hunk: @@ -5,209 +5,306 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How to handle errors when the GPU engine fails to execute a query.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to the CPU engine.
+    * ``FallbackMode.RAISE`` : Raise an exception.
+    * ``FallbackMode.SILENT``: Silently fall back to the CPU engine.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
+
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-            >>> options = options.set("parquet_options.chunked", False)
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMFP`` : Use the rapidsmpf scheduler.
+    """
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    TASKS = "tasks"
+    RAPIDSMFP = "rapidsmpf"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """
+    Configuration for the cudf-polars Parquet engine.
+
+    Parameters
+    ----------
+    chunked
+        Whether to use cudf's ``ChunkedParquetReader`` to read the parquet
+        dataset in chunks. This is useful when reading very large parquet
+        files.
+    chunk_read_limit
+        Limit on total number of bytes to be returned per read, or 0 if
+        there is no limit.
+    pass_read_limit
+        Limit on the amount of memory used for reading and decompressing data
+        or 0 if there is no limit.
+    """
+
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-        Returns
-        -------
-        Valid config-option dictionary.
+    def __post_init__(self) -> None:
+        if not isinstance(self.chunked, bool):
+            raise TypeError("chunked must be a bool")
+        if not isinstance(self.chunk_read_limit, int):
+            raise TypeError("chunk_read_limit must be an int")
+        if not isinstance(self.pass_read_limit, int):
+            raise TypeError("pass_read_limit must be an int")
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
+
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor. ``Scheduler.SYNCHRONOUS``
+        by default.
+
+        Note taht the "distributed" requires a Dask cluster to be running.
+
+    fallback_mode
+        How to handle errors when the GPU engine fails to execute a query.
+        ``FallbackMode.WARN`` by default.
+    max_rows_per_partition
+        The maximum number of rows to process per partition. 1,000,000 by default.
+        When the number of rows exceeds this value, the query will be split into
+        multiple partitions and executed in parallel.
+    cardinality_factor
+        A dictionary mapping column names to a float strictly greater than 0 and
+        less than or equal to 1. Each factor estimates the fractional number of
+        unique values in the column. By default, a value of ``1.0`` is used
+        for any column not found in ``cardinality_factor``.
+    parquet_blocksize
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: rjzamora
body: Sorry - I missed this ping earlier. What about adding `config_options.parquet_options.streaming_blocksize`? 

Something about `parquet_partition_blocksize` feels a bit unfortunate to me. This will most-likely be one of the most *used* options for the streaming executor. It may even make sense to generalize the option to simply `executor_options.blocksize` (and refer to this setting for all forms of IO).
created_at: 2025-04-23T22:19:19Z
author_association: MEMBER
diff_hunk: @@ -5,209 +5,304 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+import warnings
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: wence-
body: Can we call this `StreamingFallbackMode` ?
created_at: 2025-04-24T10:17:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,304 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+import warnings
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How the streaming executor handles operations that don't support multiple partitions.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    Upon encountering an unsupported operation, the streaming executor will fall
+    back to using a single partition, which might use a large amount of memory.
+
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to a single partition.
+    * ``FallbackMode.SILENT``: Silently fall back to a single partition.
+    * ``FallbackMode.RAISE`` : Raise an exception.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-            >>> options = options.set("parquet_options.chunked", False)
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers with the streaming executor.
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMPF`` : Use the rapidsmpf scheduler.
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
+    In :class:`StreamingExecutor`, the default of ``None`` will attempt to use
+    ``ShuffleMethod.RAPIDSMPF``, but will fall back to ``ShuffleMethod.TASKS``
+    if rapidsmpf is not installed.
+    """
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+    TASKS = "tasks"
+    RAPIDSMPF = "rapidsmpf"
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """
+    Configuration for the cudf-polars Parquet engine.
+
+    Parameters
+    ----------
+    chunked
+        Whether to use cudf's ``ChunkedParquetReader`` to read the parquet
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: wence-
body: ```suggestion
        Whether to use libcudf's ``ChunkedParquetReader`` to read the parquet
```
created_at: 2025-04-24T10:18:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,304 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+import warnings
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How the streaming executor handles operations that don't support multiple partitions.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    Upon encountering an unsupported operation, the streaming executor will fall
+    back to using a single partition, which might use a large amount of memory.
+
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to a single partition.
+    * ``FallbackMode.SILENT``: Silently fall back to a single partition.
+    * ``FallbackMode.RAISE`` : Raise an exception.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-            >>> options = options.set("parquet_options.chunked", False)
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers with the streaming executor.
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMPF`` : Use the rapidsmpf scheduler.
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
+    In :class:`StreamingExecutor`, the default of ``None`` will attempt to use
+    ``ShuffleMethod.RAPIDSMPF``, but will fall back to ``ShuffleMethod.TASKS``
+    if rapidsmpf is not installed.
+    """
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+    TASKS = "tasks"
+    RAPIDSMPF = "rapidsmpf"
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """
+    Configuration for the cudf-polars Parquet engine.
+
+    Parameters
+    ----------
+    chunked
+        Whether to use cudf's ``ChunkedParquetReader`` to read the parquet
+        dataset in chunks. This is useful when reading very large parquet
+        files.
+    chunk_read_limit
+        Limit on total number of bytes to be returned per read, or 0 if
+        there is no limit.
+    pass_read_limit
+        Limit on the amount of memory used for reading and decompressing data
+        or 0 if there is no limit.
+    """
 
-        Returns
-        -------
-        Valid config-option dictionary.
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
+    def __post_init__(self) -> None:
+        if not isinstance(self.chunked, bool):
+            raise TypeError("chunked must be a bool")
+        if not isinstance(self.chunk_read_limit, int):
+            raise TypeError("chunk_read_limit must be an int")
+        if not isinstance(self.pass_read_limit, int):
+            raise TypeError("pass_read_limit must be an int")
+
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
+
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor. ``Scheduler.SYNCHRONOUS``
+        by default.
+
+        Note ``scheduler="distributed"`` requires a Dask cluster to be running.
+
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: wence-
body: ```suggestion
```
created_at: 2025-04-24T10:19:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,304 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+import warnings
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How the streaming executor handles operations that don't support multiple partitions.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    Upon encountering an unsupported operation, the streaming executor will fall
+    back to using a single partition, which might use a large amount of memory.
+
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to a single partition.
+    * ``FallbackMode.SILENT``: Silently fall back to a single partition.
+    * ``FallbackMode.RAISE`` : Raise an exception.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-            >>> options = options.set("parquet_options.chunked", False)
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers with the streaming executor.
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMPF`` : Use the rapidsmpf scheduler.
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
+    In :class:`StreamingExecutor`, the default of ``None`` will attempt to use
+    ``ShuffleMethod.RAPIDSMPF``, but will fall back to ``ShuffleMethod.TASKS``
+    if rapidsmpf is not installed.
+    """
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+    TASKS = "tasks"
+    RAPIDSMPF = "rapidsmpf"
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """
+    Configuration for the cudf-polars Parquet engine.
+
+    Parameters
+    ----------
+    chunked
+        Whether to use cudf's ``ChunkedParquetReader`` to read the parquet
+        dataset in chunks. This is useful when reading very large parquet
+        files.
+    chunk_read_limit
+        Limit on total number of bytes to be returned per read, or 0 if
+        there is no limit.
+    pass_read_limit
+        Limit on the amount of memory used for reading and decompressing data
+        or 0 if there is no limit.
+    """
 
-        Returns
-        -------
-        Valid config-option dictionary.
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
+    def __post_init__(self) -> None:
+        if not isinstance(self.chunked, bool):
+            raise TypeError("chunked must be a bool")
+        if not isinstance(self.chunk_read_limit, int):
+            raise TypeError("chunk_read_limit must be an int")
+        if not isinstance(self.pass_read_limit, int):
+            raise TypeError("pass_read_limit must be an int")
+
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
+
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor. ``Scheduler.SYNCHRONOUS``
+        by default.
+
+        Note ``scheduler="distributed"`` requires a Dask cluster to be running.
+
+    fallback_mode
+        How to handle errors when the GPU engine fails to execute a query.
+        ``FallbackMode.WARN`` by default.
+    max_rows_per_partition
+        The maximum number of rows to process per partition. 1_000_000 by default.
+        When the number of rows exceeds this value, the query will be split into
+        multiple partitions and executed in parallel.
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: wence-
body: *question*: Is this a hard limit or a guideline?
created_at: 2025-04-24T10:19:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,304 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+import warnings
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How the streaming executor handles operations that don't support multiple partitions.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    Upon encountering an unsupported operation, the streaming executor will fall
+    back to using a single partition, which might use a large amount of memory.
+
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to a single partition.
+    * ``FallbackMode.SILENT``: Silently fall back to a single partition.
+    * ``FallbackMode.RAISE`` : Raise an exception.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-            >>> options = options.set("parquet_options.chunked", False)
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers with the streaming executor.
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMPF`` : Use the rapidsmpf scheduler.
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
+    In :class:`StreamingExecutor`, the default of ``None`` will attempt to use
+    ``ShuffleMethod.RAPIDSMPF``, but will fall back to ``ShuffleMethod.TASKS``
+    if rapidsmpf is not installed.
+    """
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+    TASKS = "tasks"
+    RAPIDSMPF = "rapidsmpf"
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """
+    Configuration for the cudf-polars Parquet engine.
+
+    Parameters
+    ----------
+    chunked
+        Whether to use cudf's ``ChunkedParquetReader`` to read the parquet
+        dataset in chunks. This is useful when reading very large parquet
+        files.
+    chunk_read_limit
+        Limit on total number of bytes to be returned per read, or 0 if
+        there is no limit.
+    pass_read_limit
+        Limit on the amount of memory used for reading and decompressing data
+        or 0 if there is no limit.
+    """
 
-        Returns
-        -------
-        Valid config-option dictionary.
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
+    def __post_init__(self) -> None:
+        if not isinstance(self.chunked, bool):
+            raise TypeError("chunked must be a bool")
+        if not isinstance(self.chunk_read_limit, int):
+            raise TypeError("chunk_read_limit must be an int")
+        if not isinstance(self.pass_read_limit, int):
+            raise TypeError("pass_read_limit must be an int")
+
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
+
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor. ``Scheduler.SYNCHRONOUS``
+        by default.
+
+        Note ``scheduler="distributed"`` requires a Dask cluster to be running.
+
+    fallback_mode
+        How to handle errors when the GPU engine fails to execute a query.
+        ``FallbackMode.WARN`` by default.
+    max_rows_per_partition
+        The maximum number of rows to process per partition. 1_000_000 by default.
+        When the number of rows exceeds this value, the query will be split into
+        multiple partitions and executed in parallel.
+    cardinality_factor
+        A dictionary mapping column names floats strictly greater than 0 and less
+        than or equal to 1. Each factor estimates the fractional number of unique values
+        in the column. By default, a value of ``1.0`` is used for any column not
+        included in ``cardinality_factor``.
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: wence-
body: `-ENOGRAMMAR`

BTW I would still love to not need this but it needs a big rethink of how we execute partitioned tasks, so...
created_at: 2025-04-24T10:23:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,304 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+import warnings
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How the streaming executor handles operations that don't support multiple partitions.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    Upon encountering an unsupported operation, the streaming executor will fall
+    back to using a single partition, which might use a large amount of memory.
+
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to a single partition.
+    * ``FallbackMode.SILENT``: Silently fall back to a single partition.
+    * ``FallbackMode.RAISE`` : Raise an exception.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-            >>> options = options.set("parquet_options.chunked", False)
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers with the streaming executor.
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMPF`` : Use the rapidsmpf scheduler.
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
+    In :class:`StreamingExecutor`, the default of ``None`` will attempt to use
+    ``ShuffleMethod.RAPIDSMPF``, but will fall back to ``ShuffleMethod.TASKS``
+    if rapidsmpf is not installed.
+    """
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+    TASKS = "tasks"
+    RAPIDSMPF = "rapidsmpf"
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """
+    Configuration for the cudf-polars Parquet engine.
+
+    Parameters
+    ----------
+    chunked
+        Whether to use cudf's ``ChunkedParquetReader`` to read the parquet
+        dataset in chunks. This is useful when reading very large parquet
+        files.
+    chunk_read_limit
+        Limit on total number of bytes to be returned per read, or 0 if
+        there is no limit.
+    pass_read_limit
+        Limit on the amount of memory used for reading and decompressing data
+        or 0 if there is no limit.
+    """
 
-        Returns
-        -------
-        Valid config-option dictionary.
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
+    def __post_init__(self) -> None:
+        if not isinstance(self.chunked, bool):
+            raise TypeError("chunked must be a bool")
+        if not isinstance(self.chunk_read_limit, int):
+            raise TypeError("chunk_read_limit must be an int")
+        if not isinstance(self.pass_read_limit, int):
+            raise TypeError("pass_read_limit must be an int")
+
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
+
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor. ``Scheduler.SYNCHRONOUS``
+        by default.
+
+        Note ``scheduler="distributed"`` requires a Dask cluster to be running.
+
+    fallback_mode
+        How to handle errors when the GPU engine fails to execute a query.
+        ``FallbackMode.WARN`` by default.
+    max_rows_per_partition
+        The maximum number of rows to process per partition. 1_000_000 by default.
+        When the number of rows exceeds this value, the query will be split into
+        multiple partitions and executed in parallel.
+    cardinality_factor
+        A dictionary mapping column names floats strictly greater than 0 and less
+        than or equal to 1. Each factor estimates the fractional number of unique values
+        in the column. By default, a value of ``1.0`` is used for any column not
+        included in ``cardinality_factor``.
+    parquet_partition_blocksize
+        Controls how large parquet files are split into multiple partitions.
+        Files larger than ``parquet_partition_blocksize`` bytes are split into multiple
+        partitions.
+    groupby_n_ary
+        The factor by which the number of partitions is decreased when performing
+        a groupby on a partitioned column. For example, if a column has 64 partitions,
+        it will first be reduced to ``ceil(64 / 32) = 2`` partitions.
+
+        This is useful when the absolute number of partitions is large.
+
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: wence-
body: ```suggestion
```
created_at: 2025-04-24T10:31:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,148 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
 import json
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
-    """
-    GPUEngine configuration-option manager.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """Configuration for the cudf-polars Parquet engine."""
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
-    """
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
+FALLBACK_MODE = Literal["warn", "raise"]
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
 
-            >>> options = options.set("parquet_options.chunked", False)
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
-
-        Nested dictionary keys should be separated by periods.
-        For example::
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor.
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
+        "distributed" requires a Dask cluster to be running.
+    """
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+    name: Literal["streaming"] = "streaming"
+    scheduler: Literal["synchronous", "distributed"] = "synchronous"
+    fallback_mode: FALLBACK_MODE = "warn"
+    max_rows_per_partition: int = 1_000_000
+    cardinality_factor: dict[str, float] = dataclasses.field(default_factory=dict)
+    parquet_blocksize: int = 1_000_000_000  # why isn't this a ParquetOption?
+    groupby_n_ary: int = 32
+    broadcast_join_limit: int = 4
+    shuffle_method: Literal["tasks", "rapidsmpf"] | None = None
+
+    def __post_init__(self) -> None:
+        if self.scheduler == "synchronous" and self.shuffle_method == "rapidsmpf":
+            raise ValueError(
+                "rapidsmpf shuffle method is not supported for synchronous scheduler"
+            )
 
     def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
+        # cardinatlity factory, a dict, isn't natively hashable. We'll dump it
+        # to json and hash that.
+        d = dataclasses.asdict(self)
+        d["cardinality_factor"] = json.dumps(d["cardinality_factor"])
+        return hash(tuple(sorted(d.items())))
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
 
-        Returns
-        -------
-        Valid config-option dictionary.
+@dataclasses.dataclass(frozen=True, eq=True)
+class InMemoryExecutor:
+    """Configuration for the cudf-polars in-memory executor."""
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
-            raise ValueError(
-                f"Engine configuration contains unsupported settings: {unsupported}"
-            )
-        assert {"chunked", "chunk_read_limit", "pass_read_limit"}.issuperset(
-            config.get("parquet_options", {})
-        )
+    name: Literal["in-memory"] = "in-memory"
+    scheduler: Literal["synchronous"] = "synchronous"
+    shuffle_method: Literal["tasks"] | None = None
+    broadcast_join_limit: int = 32
 
-        # Validate executor_options
-        executor = config.get("executor", "in-memory")
-        if executor == "streaming":
-            executor_options, unsupported = _valid_streaming_options(
-                config["executor_options"]
-            )
-            config["executor_options"] = executor_options
-        else:
-            unsupported = config.get("executor_options", {}).keys()
-        if unsupported:
-            raise ValueError(
-                f"Unsupported executor_options for {executor}: {unsupported}"
-            )
 
-        return config
-
-
-def _valid_option(
-    options: MutableMapping[str, Any],
-    key: str,
-    *,
-    default: Any,
-    choices: Sequence[Any] | None = None,
-    astype: type | None = None,
-) -> MutableMapping[str, Any]:
-    """Return a valid configuration option."""
-    value = options.get(key, default)
-    if astype is not None:
-        value = astype(value)
-    if choices is not None and value not in choices:
-        raise ValueError(f"Unsupported {key} option: {value}. Choices are: {choices}")
-    options[key] = value
-    return options
-
-
-def _valid_streaming_options(
-    options: MutableMapping[str, Any],
-) -> tuple[MutableMapping[str, Any], set[str]]:
-    """Return valid options for the streaming executor."""
-    default: Any
-    choices: Sequence[Any]
-    for key in (
-        _STREAMING_OPTIONS := (
-            "scheduler",  # Validate scheduler first
-            "fallback_mode",
-            "max_rows_per_partition",
-            "parquet_blocksize",
-            "cardinality_factor",
-            "groupby_n_ary",
-            "broadcast_join_limit",
-            "shuffle_method",
+@dataclasses.dataclass(frozen=True, eq=True)
+class ConfigOptions:
+    """Configuration for the polars GPUEngine."""
+
+    raise_on_fail: bool = False
+    # device?
+    # memory resource?
+    parquet_options: ParquetOptions = dataclasses.field(default_factory=ParquetOptions)
+    executor: StreamingExecutor | InMemoryExecutor = dataclasses.field(
+        default_factory=InMemoryExecutor
+    )
+
+    @classmethod
+    def from_polars_engine(cls, engine: pl.GPUEngine) -> Self:
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: wence-
body: We need to think about how to handle memory resources in the distributed setting. We need a way for the user to specify the mr to be used on the workers, but we can't get them to pass an already initialised mr because you can't ship that around. We could ask them to provide a callable that would construct an mr though.
created_at: 2025-04-24T10:36:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,304 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+import warnings
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How the streaming executor handles operations that don't support multiple partitions.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    Upon encountering an unsupported operation, the streaming executor will fall
+    back to using a single partition, which might use a large amount of memory.
+
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to a single partition.
+    * ``FallbackMode.SILENT``: Silently fall back to a single partition.
+    * ``FallbackMode.RAISE`` : Raise an exception.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-            >>> options = options.set("parquet_options.chunked", False)
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers with the streaming executor.
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMPF`` : Use the rapidsmpf scheduler.
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
+    In :class:`StreamingExecutor`, the default of ``None`` will attempt to use
+    ``ShuffleMethod.RAPIDSMPF``, but will fall back to ``ShuffleMethod.TASKS``
+    if rapidsmpf is not installed.
+    """
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+    TASKS = "tasks"
+    RAPIDSMPF = "rapidsmpf"
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """
+    Configuration for the cudf-polars Parquet engine.
+
+    Parameters
+    ----------
+    chunked
+        Whether to use cudf's ``ChunkedParquetReader`` to read the parquet
+        dataset in chunks. This is useful when reading very large parquet
+        files.
+    chunk_read_limit
+        Limit on total number of bytes to be returned per read, or 0 if
+        there is no limit.
+    pass_read_limit
+        Limit on the amount of memory used for reading and decompressing data
+        or 0 if there is no limit.
+    """
 
-        Returns
-        -------
-        Valid config-option dictionary.
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
+    def __post_init__(self) -> None:
+        if not isinstance(self.chunked, bool):
+            raise TypeError("chunked must be a bool")
+        if not isinstance(self.chunk_read_limit, int):
+            raise TypeError("chunk_read_limit must be an int")
+        if not isinstance(self.pass_read_limit, int):
+            raise TypeError("pass_read_limit must be an int")
+
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
+
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor. ``Scheduler.SYNCHRONOUS``
+        by default.
+
+        Note ``scheduler="distributed"`` requires a Dask cluster to be running.
+
+    fallback_mode
+        How to handle errors when the GPU engine fails to execute a query.
+        ``FallbackMode.WARN`` by default.
+    max_rows_per_partition
+        The maximum number of rows to process per partition. 1_000_000 by default.
+        When the number of rows exceeds this value, the query will be split into
+        multiple partitions and executed in parallel.
+    cardinality_factor
+        A dictionary mapping column names floats strictly greater than 0 and less
+        than or equal to 1. Each factor estimates the fractional number of unique values
+        in the column. By default, a value of ``1.0`` is used for any column not
+        included in ``cardinality_factor``.
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: rjzamora
body: This is definitely a hack until we (hopefully) work out a better way to switch between tree- and shuffles-based algorithms.
created_at: 2025-04-24T11:56:23Z
author_association: MEMBER
diff_hunk: @@ -5,209 +5,304 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+import warnings
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How the streaming executor handles operations that don't support multiple partitions.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    Upon encountering an unsupported operation, the streaming executor will fall
+    back to using a single partition, which might use a large amount of memory.
+
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to a single partition.
+    * ``FallbackMode.SILENT``: Silently fall back to a single partition.
+    * ``FallbackMode.RAISE`` : Raise an exception.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-            >>> options = options.set("parquet_options.chunked", False)
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers with the streaming executor.
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMPF`` : Use the rapidsmpf scheduler.
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
+    In :class:`StreamingExecutor`, the default of ``None`` will attempt to use
+    ``ShuffleMethod.RAPIDSMPF``, but will fall back to ``ShuffleMethod.TASKS``
+    if rapidsmpf is not installed.
+    """
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+    TASKS = "tasks"
+    RAPIDSMPF = "rapidsmpf"
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """
+    Configuration for the cudf-polars Parquet engine.
+
+    Parameters
+    ----------
+    chunked
+        Whether to use cudf's ``ChunkedParquetReader`` to read the parquet
+        dataset in chunks. This is useful when reading very large parquet
+        files.
+    chunk_read_limit
+        Limit on total number of bytes to be returned per read, or 0 if
+        there is no limit.
+    pass_read_limit
+        Limit on the amount of memory used for reading and decompressing data
+        or 0 if there is no limit.
+    """
 
-        Returns
-        -------
-        Valid config-option dictionary.
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
+    def __post_init__(self) -> None:
+        if not isinstance(self.chunked, bool):
+            raise TypeError("chunked must be a bool")
+        if not isinstance(self.chunk_read_limit, int):
+            raise TypeError("chunk_read_limit must be an int")
+        if not isinstance(self.pass_read_limit, int):
+            raise TypeError("pass_read_limit must be an int")
+
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
+
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor. ``Scheduler.SYNCHRONOUS``
+        by default.
+
+        Note ``scheduler="distributed"`` requires a Dask cluster to be running.
+
+    fallback_mode
+        How to handle errors when the GPU engine fails to execute a query.
+        ``FallbackMode.WARN`` by default.
+    max_rows_per_partition
+        The maximum number of rows to process per partition. 1_000_000 by default.
+        When the number of rows exceeds this value, the query will be split into
+        multiple partitions and executed in parallel.
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: rjzamora
body: This is only used for DataFrameScan, and is therefore mostly for testing/demonstration. I was thinking yesterday that we could try to replace `executor_options.partquet_blocksize` with `executor_options.blocksize` and use that number to determine the number of partitions instead. However, it's really nice to a have a specific row count for testing purposes.
created_at: 2025-04-24T11:59:08Z
author_association: MEMBER
diff_hunk: @@ -5,209 +5,304 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+import warnings
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How the streaming executor handles operations that don't support multiple partitions.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    Upon encountering an unsupported operation, the streaming executor will fall
+    back to using a single partition, which might use a large amount of memory.
+
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to a single partition.
+    * ``FallbackMode.SILENT``: Silently fall back to a single partition.
+    * ``FallbackMode.RAISE`` : Raise an exception.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-            >>> options = options.set("parquet_options.chunked", False)
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers with the streaming executor.
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMPF`` : Use the rapidsmpf scheduler.
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
+    In :class:`StreamingExecutor`, the default of ``None`` will attempt to use
+    ``ShuffleMethod.RAPIDSMPF``, but will fall back to ``ShuffleMethod.TASKS``
+    if rapidsmpf is not installed.
+    """
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+    TASKS = "tasks"
+    RAPIDSMPF = "rapidsmpf"
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """
+    Configuration for the cudf-polars Parquet engine.
+
+    Parameters
+    ----------
+    chunked
+        Whether to use cudf's ``ChunkedParquetReader`` to read the parquet
+        dataset in chunks. This is useful when reading very large parquet
+        files.
+    chunk_read_limit
+        Limit on total number of bytes to be returned per read, or 0 if
+        there is no limit.
+    pass_read_limit
+        Limit on the amount of memory used for reading and decompressing data
+        or 0 if there is no limit.
+    """
 
-        Returns
-        -------
-        Valid config-option dictionary.
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
+    def __post_init__(self) -> None:
+        if not isinstance(self.chunked, bool):
+            raise TypeError("chunked must be a bool")
+        if not isinstance(self.chunk_read_limit, int):
+            raise TypeError("chunk_read_limit must be an int")
+        if not isinstance(self.pass_read_limit, int):
+            raise TypeError("pass_read_limit must be an int")
+
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
+
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor. ``Scheduler.SYNCHRONOUS``
+        by default.
+
+        Note ``scheduler="distributed"`` requires a Dask cluster to be running.
+
+    fallback_mode
+        How to handle errors when the GPU engine fails to execute a query.
+        ``FallbackMode.WARN`` by default.
+    max_rows_per_partition
+        The maximum number of rows to process per partition. 1_000_000 by default.
+        When the number of rows exceeds this value, the query will be split into
+        multiple partitions and executed in parallel.
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: TomAugspurger
body: There shouldn't be any behavior change here, so it should be the same as on branch-25.06.  Looking at `lower_ir_node.register(DataFrameScan)` I see

```
    rows_per_partition = ir.config_options.executor.max_rows_per_partition

    nrows = max(ir.df.shape()[0], 1)
    count = math.ceil(nrows / rows_per_partition)

    if count > 1:
        ....
```

so it seems to be a hard limit there, and that's the only current usage.

However, I'm a little hesitant to document that it's a hard limit, in case we start using this value elsewhere and want some fuzziness. So I'd recommend either leaving it ambiguous or documenting it as a guideline. Do you have a preference?
created_at: 2025-04-24T12:37:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,148 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
 import json
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
-    """
-    GPUEngine configuration-option manager.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """Configuration for the cudf-polars Parquet engine."""
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
-    """
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
+FALLBACK_MODE = Literal["warn", "raise"]
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
 
-            >>> options = options.set("parquet_options.chunked", False)
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
-
-        Nested dictionary keys should be separated by periods.
-        For example::
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor.
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
+        "distributed" requires a Dask cluster to be running.
+    """
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+    name: Literal["streaming"] = "streaming"
+    scheduler: Literal["synchronous", "distributed"] = "synchronous"
+    fallback_mode: FALLBACK_MODE = "warn"
+    max_rows_per_partition: int = 1_000_000
+    cardinality_factor: dict[str, float] = dataclasses.field(default_factory=dict)
+    parquet_blocksize: int = 1_000_000_000  # why isn't this a ParquetOption?
+    groupby_n_ary: int = 32
+    broadcast_join_limit: int = 4
+    shuffle_method: Literal["tasks", "rapidsmpf"] | None = None
+
+    def __post_init__(self) -> None:
+        if self.scheduler == "synchronous" and self.shuffle_method == "rapidsmpf":
+            raise ValueError(
+                "rapidsmpf shuffle method is not supported for synchronous scheduler"
+            )
 
     def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
+        # cardinatlity factory, a dict, isn't natively hashable. We'll dump it
+        # to json and hash that.
+        d = dataclasses.asdict(self)
+        d["cardinality_factor"] = json.dumps(d["cardinality_factor"])
+        return hash(tuple(sorted(d.items())))
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
 
-        Returns
-        -------
-        Valid config-option dictionary.
+@dataclasses.dataclass(frozen=True, eq=True)
+class InMemoryExecutor:
+    """Configuration for the cudf-polars in-memory executor."""
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
-            raise ValueError(
-                f"Engine configuration contains unsupported settings: {unsupported}"
-            )
-        assert {"chunked", "chunk_read_limit", "pass_read_limit"}.issuperset(
-            config.get("parquet_options", {})
-        )
+    name: Literal["in-memory"] = "in-memory"
+    scheduler: Literal["synchronous"] = "synchronous"
+    shuffle_method: Literal["tasks"] | None = None
+    broadcast_join_limit: int = 32
 
-        # Validate executor_options
-        executor = config.get("executor", "in-memory")
-        if executor == "streaming":
-            executor_options, unsupported = _valid_streaming_options(
-                config["executor_options"]
-            )
-            config["executor_options"] = executor_options
-        else:
-            unsupported = config.get("executor_options", {}).keys()
-        if unsupported:
-            raise ValueError(
-                f"Unsupported executor_options for {executor}: {unsupported}"
-            )
 
-        return config
-
-
-def _valid_option(
-    options: MutableMapping[str, Any],
-    key: str,
-    *,
-    default: Any,
-    choices: Sequence[Any] | None = None,
-    astype: type | None = None,
-) -> MutableMapping[str, Any]:
-    """Return a valid configuration option."""
-    value = options.get(key, default)
-    if astype is not None:
-        value = astype(value)
-    if choices is not None and value not in choices:
-        raise ValueError(f"Unsupported {key} option: {value}. Choices are: {choices}")
-    options[key] = value
-    return options
-
-
-def _valid_streaming_options(
-    options: MutableMapping[str, Any],
-) -> tuple[MutableMapping[str, Any], set[str]]:
-    """Return valid options for the streaming executor."""
-    default: Any
-    choices: Sequence[Any]
-    for key in (
-        _STREAMING_OPTIONS := (
-            "scheduler",  # Validate scheduler first
-            "fallback_mode",
-            "max_rows_per_partition",
-            "parquet_blocksize",
-            "cardinality_factor",
-            "groupby_n_ary",
-            "broadcast_join_limit",
-            "shuffle_method",
+@dataclasses.dataclass(frozen=True, eq=True)
+class ConfigOptions:
+    """Configuration for the polars GPUEngine."""
+
+    raise_on_fail: bool = False
+    # device?
+    # memory resource?
+    parquet_options: ParquetOptions = dataclasses.field(default_factory=ParquetOptions)
+    executor: StreamingExecutor | InMemoryExecutor = dataclasses.field(
+        default_factory=InMemoryExecutor
+    )
+
+    @classmethod
+    def from_polars_engine(cls, engine: pl.GPUEngine) -> Self:
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: TomAugspurger
body: A callable that returns a MemoryResource (by either constructing it or getting an existing one) makes sense...

We'll need to coordinate this with the `memory_resource` parameter provided to `polars.GPUEngine`, which I believe is only sensible for the `in-memory` executor. Maybe best as a followup?
created_at: 2025-04-24T13:04:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -212,9 +212,14 @@ def _(
     # Check if we are dealing with any high-cardinality columns
     post_aggregation_count = 1  # Default tree reduction
     groupby_key_columns = [ne.name for ne in ir.keys]
+
+    assert ir.config_options.executor.name == "streaming", (
+        "'in-memory' executor not supported in 'generate_ir_tasks'"
+    )
path: python/cudf_polars/cudf_polars/experimental/groupby.py
user: login: TomAugspurger
body: I'm a little pessimistic about solving this, especially now that I see that not all IR nodes have a `config_options`. I'm not sure that we could make `IR` generic over `ConfigOptions`, though I guess it could maybe be optional...

I briefly looked into using [TypeGuards](https://typing.python.org/en/latest/spec/narrowing.html#typeguard) but didn't get very far.
created_at: 2025-04-24T13:16:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,306 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How to handle errors when the GPU engine fails to execute a query.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to the CPU engine.
+    * ``FallbackMode.RAISE`` : Raise an exception.
+    * ``FallbackMode.SILENT``: Silently fall back to the CPU engine.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
+
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-            >>> options = options.set("parquet_options.chunked", False)
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMFP`` : Use the rapidsmpf scheduler.
+    """
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    TASKS = "tasks"
+    RAPIDSMFP = "rapidsmpf"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """
+    Configuration for the cudf-polars Parquet engine.
+
+    Parameters
+    ----------
+    chunked
+        Whether to use cudf's ``ChunkedParquetReader`` to read the parquet
+        dataset in chunks. This is useful when reading very large parquet
+        files.
+    chunk_read_limit
+        Limit on total number of bytes to be returned per read, or 0 if
+        there is no limit.
+    pass_read_limit
+        Limit on the amount of memory used for reading and decompressing data
+        or 0 if there is no limit.
+    """
+
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-        Returns
-        -------
-        Valid config-option dictionary.
+    def __post_init__(self) -> None:
+        if not isinstance(self.chunked, bool):
+            raise TypeError("chunked must be a bool")
+        if not isinstance(self.chunk_read_limit, int):
+            raise TypeError("chunk_read_limit must be an int")
+        if not isinstance(self.pass_read_limit, int):
+            raise TypeError("pass_read_limit must be an int")
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
+
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor. ``Scheduler.SYNCHRONOUS``
+        by default.
+
+        Note taht the "distributed" requires a Dask cluster to be running.
+
+    fallback_mode
+        How to handle errors when the GPU engine fails to execute a query.
+        ``FallbackMode.WARN`` by default.
+    max_rows_per_partition
+        The maximum number of rows to process per partition. 1,000,000 by default.
+        When the number of rows exceeds this value, the query will be split into
+        multiple partitions and executed in parallel.
+    cardinality_factor
+        A dictionary mapping column names to a float strictly greater than 0 and
+        less than or equal to 1. Each factor estimates the fractional number of
+        unique values in the column. By default, a value of ``1.0`` is used
+        for any column not found in ``cardinality_factor``.
+    parquet_blocksize
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: TomAugspurger
body: I can move it there, as long as we're comfortable with `ParquetOptions` having a configuration value that will only affect one executor.
created_at: 2025-04-24T13:18:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,209 +5,304 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+import warnings
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class FallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How the streaming executor handles operations that don't support multiple partitions.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    Upon encountering an unsupported operation, the streaming executor will fall
+    back to using a single partition, which might use a large amount of memory.
+
+    * ``FallbackMode.WARN`` : Emit a warning and fall back to a single partition.
+    * ``FallbackMode.SILENT``: Silently fall back to a single partition.
+    * ``FallbackMode.RAISE`` : Raise an exception.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-            >>> options = options.set("parquet_options.chunked", False)
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers with the streaming executor.
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMPF`` : Use the rapidsmpf scheduler.
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
+    In :class:`StreamingExecutor`, the default of ``None`` will attempt to use
+    ``ShuffleMethod.RAPIDSMPF``, but will fall back to ``ShuffleMethod.TASKS``
+    if rapidsmpf is not installed.
+    """
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+    TASKS = "tasks"
+    RAPIDSMPF = "rapidsmpf"
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """
+    Configuration for the cudf-polars Parquet engine.
+
+    Parameters
+    ----------
+    chunked
+        Whether to use cudf's ``ChunkedParquetReader`` to read the parquet
+        dataset in chunks. This is useful when reading very large parquet
+        files.
+    chunk_read_limit
+        Limit on total number of bytes to be returned per read, or 0 if
+        there is no limit.
+    pass_read_limit
+        Limit on the amount of memory used for reading and decompressing data
+        or 0 if there is no limit.
+    """
 
-        Returns
-        -------
-        Valid config-option dictionary.
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
+    def __post_init__(self) -> None:
+        if not isinstance(self.chunked, bool):
+            raise TypeError("chunked must be a bool")
+        if not isinstance(self.chunk_read_limit, int):
+            raise TypeError("chunk_read_limit must be an int")
+        if not isinstance(self.pass_read_limit, int):
+            raise TypeError("pass_read_limit must be an int")
+
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
+
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor. ``Scheduler.SYNCHRONOUS``
+        by default.
+
+        Note ``scheduler="distributed"`` requires a Dask cluster to be running.
+
+    fallback_mode
+        How to handle errors when the GPU engine fails to execute a query.
+        ``FallbackMode.WARN`` by default.
+    max_rows_per_partition
+        The maximum number of rows to process per partition. 1_000_000 by default.
+        When the number of rows exceeds this value, the query will be split into
+        multiple partitions and executed in parallel.
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: rjzamora
body: >There shouldn't be any behavior change here, so it should be the same as on branch-25.06.

Yeah, sorry I should have been more explicit. I don't think it's within the scope of this PR to change the actual behavior of any options unless something is "broken". My comments are more-so "wondering out loud" if/how we might want to modify these options in a follow-up.
created_at: 2025-04-24T13:59:05Z
author_association: MEMBER
diff_hunk: @@ -5,209 +5,148 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
 import json
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
-    """
-    GPUEngine configuration-option manager.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """Configuration for the cudf-polars Parquet engine."""
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
-    """
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
+FALLBACK_MODE = Literal["warn", "raise"]
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+STREAMING_EXECUTOR_PARQUET_DEFAULTS = {
+    "chunked": False,
+}
 
-            >>> options = options.set("parquet_options.chunked", False)
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
-
-        Nested dictionary keys should be separated by periods.
-        For example::
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor.
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
+        "distributed" requires a Dask cluster to be running.
+    """
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+    name: Literal["streaming"] = "streaming"
+    scheduler: Literal["synchronous", "distributed"] = "synchronous"
+    fallback_mode: FALLBACK_MODE = "warn"
+    max_rows_per_partition: int = 1_000_000
+    cardinality_factor: dict[str, float] = dataclasses.field(default_factory=dict)
+    parquet_blocksize: int = 1_000_000_000  # why isn't this a ParquetOption?
+    groupby_n_ary: int = 32
+    broadcast_join_limit: int = 4
+    shuffle_method: Literal["tasks", "rapidsmpf"] | None = None
+
+    def __post_init__(self) -> None:
+        if self.scheduler == "synchronous" and self.shuffle_method == "rapidsmpf":
+            raise ValueError(
+                "rapidsmpf shuffle method is not supported for synchronous scheduler"
+            )
 
     def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
+        # cardinatlity factory, a dict, isn't natively hashable. We'll dump it
+        # to json and hash that.
+        d = dataclasses.asdict(self)
+        d["cardinality_factor"] = json.dumps(d["cardinality_factor"])
+        return hash(tuple(sorted(d.items())))
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
 
-        Returns
-        -------
-        Valid config-option dictionary.
+@dataclasses.dataclass(frozen=True, eq=True)
+class InMemoryExecutor:
+    """Configuration for the cudf-polars in-memory executor."""
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
-            raise ValueError(
-                f"Engine configuration contains unsupported settings: {unsupported}"
-            )
-        assert {"chunked", "chunk_read_limit", "pass_read_limit"}.issuperset(
-            config.get("parquet_options", {})
-        )
+    name: Literal["in-memory"] = "in-memory"
+    scheduler: Literal["synchronous"] = "synchronous"
+    shuffle_method: Literal["tasks"] | None = None
+    broadcast_join_limit: int = 32
 
-        # Validate executor_options
-        executor = config.get("executor", "in-memory")
-        if executor == "streaming":
-            executor_options, unsupported = _valid_streaming_options(
-                config["executor_options"]
-            )
-            config["executor_options"] = executor_options
-        else:
-            unsupported = config.get("executor_options", {}).keys()
-        if unsupported:
-            raise ValueError(
-                f"Unsupported executor_options for {executor}: {unsupported}"
-            )
 
-        return config
-
-
-def _valid_option(
-    options: MutableMapping[str, Any],
-    key: str,
-    *,
-    default: Any,
-    choices: Sequence[Any] | None = None,
-    astype: type | None = None,
-) -> MutableMapping[str, Any]:
-    """Return a valid configuration option."""
-    value = options.get(key, default)
-    if astype is not None:
-        value = astype(value)
-    if choices is not None and value not in choices:
-        raise ValueError(f"Unsupported {key} option: {value}. Choices are: {choices}")
-    options[key] = value
-    return options
-
-
-def _valid_streaming_options(
-    options: MutableMapping[str, Any],
-) -> tuple[MutableMapping[str, Any], set[str]]:
-    """Return valid options for the streaming executor."""
-    default: Any
-    choices: Sequence[Any]
-    for key in (
-        _STREAMING_OPTIONS := (
-            "scheduler",  # Validate scheduler first
-            "fallback_mode",
-            "max_rows_per_partition",
-            "parquet_blocksize",
-            "cardinality_factor",
-            "groupby_n_ary",
-            "broadcast_join_limit",
-            "shuffle_method",
+@dataclasses.dataclass(frozen=True, eq=True)
+class ConfigOptions:
+    """Configuration for the polars GPUEngine."""
+
+    raise_on_fail: bool = False
+    # device?
+    # memory resource?
+    parquet_options: ParquetOptions = dataclasses.field(default_factory=ParquetOptions)
+    executor: StreamingExecutor | InMemoryExecutor = dataclasses.field(
+        default_factory=InMemoryExecutor
+    )
+
+    @classmethod
+    def from_polars_engine(cls, engine: pl.GPUEngine) -> Self:
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: rjzamora
body: >We need to think about how to handle memory resources in the distributed setting.

Right now, we definitely don't want cudf-polars to be responsible for defining/creating/managing the memory resource used on the workers. In fact, we probably don't want the user to be allowed to use the `memory_resource` configuration at all. Cudf-polars is essentially a "client" in this case, and the user is responsible for deploying the cluster (with appropriate memory resources on each worker) separately. If/when we decide to make cluster deployment automagical, we will want those memory-resource options to be in some `DistributedOptions` configuration.
created_at: 2025-04-24T14:10:20Z
author_association: MEMBER
diff_hunk: @@ -5,209 +5,306 @@
 
 from __future__ import annotations
 
-import copy
+import dataclasses
+import enum
 import json
-from typing import TYPE_CHECKING, Any
+import warnings
+from typing import TYPE_CHECKING, Literal
 
 if TYPE_CHECKING:
-    from collections.abc import MutableMapping, Sequence
-
     from typing_extensions import Self
 
+    import polars as pl
+
+
 __all__ = ["ConfigOptions"]
 
 
-class ConfigOptions:
+# TODO: Use enum.StrEnum when we drop Python 3.10
+
+
+class StreamingFallbackMode(str, enum.Enum):
     """
-    GPUEngine configuration-option manager.
+    How the streaming executor handles operations that don't support multiple partitions.
 
-    This is a convenience class to help manage the nested
-    dictionary of user-accessible `GPUEngine` options.
+    Upon encountering an unsupported operation, the streaming executor will fall
+    back to using a single partition, which might use a large amount of memory.
+
+    * ``StreamingFallbackMode.WARN`` : Emit a warning and fall back to a single partition.
+    * ``StreamingFallbackMode.SILENT``: Silently fall back to a single partition.
+    * ``StreamingFallbackMode.RAISE`` : Raise an exception.
     """
 
-    __slots__ = ("_hash_value", "config_options")
-    _hash_value: int
-    config_options: dict[str, Any]
-    """The underlying (nested) config-option dictionary."""
+    WARN = "warn"
+    RAISE = "raise"
+    SILENT = "silent"
 
-    def __init__(self, options: dict[str, Any]):
-        self.config_options = self.validate(copy.deepcopy(options))
 
-    def set(self, name: str, value: Any) -> Self:
-        """
-        Set a user config option.
+class Scheduler(str, enum.Enum):
+    """
+    The scheduler to use for the streaming executor.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+    * ``Scheduler.SYNCHRONOUS`` : Use the synchronous scheduler.
+    * ``Scheduler.DISTRIBUTED`` : Use the distributed scheduler.
+    """
 
-            >>> options = options.set("parquet_options.chunked", False)
+    SYNCHRONOUS = "synchronous"
+    DISTRIBUTED = "distributed"
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        value
-            New config value.
-        """
-        options = config_options = copy.deepcopy(self.config_options)
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            if k not in options:
-                options[k] = {}
-            options = options[k]
-        options[keys[-1]] = value
-        return type(self)(config_options)
-
-    def get(self, name: str, *, default: Any = None) -> Any:
-        """
-        Get a user config option.
 
-        Nested dictionary keys should be separated by periods.
-        For example::
+class ShuffleMethod(str, enum.Enum):
+    """
+    The method to use for shuffling data between workers with the streaming executor.
 
-            >>> chunked = config_options.get("parquet_options.chunked")
+    * ``ShuffleMethod.TASKS`` : Use the task-based shuffler.
+    * ``ShuffleMethod.RAPIDSMPF`` : Use the rapidsmpf scheduler.
 
-        Parameters
-        ----------
-        name
-            Period-separated config name.
-        default
-            Default return value.
+    In :class:`StreamingExecutor`, the default of ``None`` will attempt to use
+    ``ShuffleMethod.RAPIDSMPF``, but will fall back to ``ShuffleMethod.TASKS``
+    if rapidsmpf is not installed.
+    """
 
-        Returns
-        -------
-        The user-specified config value, or `default`
-        if the config is not found.
-        """
-        options = self.config_options
-        keys = name.split(".")
-        for k in keys[:-1]:
-            assert isinstance(options, dict)
-            options = options.get(k, {})
-        return options.get(keys[-1], default)
+    TASKS = "tasks"
+    RAPIDSMPF = "rapidsmpf"
 
-    def __hash__(self) -> int:
-        """Hash a ConfigOptions object."""
-        try:
-            return self._hash_value
-        except AttributeError:
-            self._hash_value = hash(json.dumps(self.config_options))
-            return self._hash_value
-
-    @staticmethod
-    def validate(config: dict) -> dict:
-        """
-        Validate a configuration-option dictionary.
 
-        Parameters
-        ----------
-        config
-            GPUEngine configuration options to validate.
+@dataclasses.dataclass(frozen=True)
+class ParquetOptions:
+    """
+    Configuration for the cudf-polars Parquet engine.
+
+    Parameters
+    ----------
+    chunked
+        Whether to use libcudf's ``ChunkedParquetReader`` to read the parquet
+        dataset in chunks. This is useful when reading very large parquet
+        files.
+    chunk_read_limit
+        Limit on total number of bytes to be returned per read, or 0 if
+        there is no limit.
+    pass_read_limit
+        Limit on the amount of memory used for reading and decompressing data
+        or 0 if there is no limit.
+    """
+
+    chunked: bool = True
+    chunk_read_limit: int = 0
+    pass_read_limit: int = 0
 
-        Returns
-        -------
-        Valid config-option dictionary.
+    def __post_init__(self) -> None:
+        if not isinstance(self.chunked, bool):
+            raise TypeError("chunked must be a bool")
+        if not isinstance(self.chunk_read_limit, int):
+            raise TypeError("chunk_read_limit must be an int")
+        if not isinstance(self.pass_read_limit, int):
+            raise TypeError("pass_read_limit must be an int")
 
-        Raises
-        ------
-        ValueError
-            If the configuration contains unsupported options.
-        """
-        if unsupported := (
-            config.keys()
-            - {"raise_on_fail", "parquet_options", "executor", "executor_options"}
-        ):
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class StreamingExecutor:
+    """
+    Configuration for the cudf-polars streaming executor.
+
+    Parameters
+    ----------
+    scheduler
+        The scheduler to use for the streaming executor. ``Scheduler.SYNCHRONOUS``
+        by default.
+
+        Note ``scheduler="distributed"`` requires a Dask cluster to be running.
+    fallback_mode
+        How to handle errors when the GPU engine fails to execute a query.
+        ``StreamingFallbackMode.WARN`` by default.
+    max_rows_per_partition
+        The maximum number of rows to process per partition. 1_000_000 by default.
+        When the number of rows exceeds this value, the query will be split into
+        multiple partitions and executed in parallel.
+    cardinality_factor
+        A dictionary mapping column names to floats between 0 and 1 (inclusive
+        on the right).
+
+        Each factor estimates the fractional number of unique values in the
+        column. By default, ``1.0`` is used for any column not included in
+        ``cardinality_factor``.
+    parquet_partition_blocksize
+        Controls how large parquet files are split into multiple partitions.
+        Files larger than ``parquet_partition_blocksize`` bytes are split into multiple
+        partitions.
+    groupby_n_ary
+        The factor by which the number of partitions is decreased when performing
+        a groupby on a partitioned column. For example, if a column has 64 partitions,
+        it will first be reduced to ``ceil(64 / 32) = 2`` partitions.
+
+        This is useful when the absolute number of partitions is large.
+    broadcast_join_limit
+        The maximum number of partitions to allow for the smaller table in
+        a broadcast join.
+    shuffle_method
+        The method to use for shuffling data between workers. ``None``
+        by default, which will use 'rapidsmpf' if installed and fall back to
+        'tasks' if not.
+    """
+
+    name: Literal["streaming"] = dataclasses.field(default="streaming", init=False)
+    scheduler: Scheduler = Scheduler.SYNCHRONOUS
+    fallback_mode: StreamingFallbackMode = StreamingFallbackMode.WARN
+    max_rows_per_partition: int = 1_000_000
+    cardinality_factor: dict[str, float] = dataclasses.field(default_factory=dict)
+    parquet_partition_blocksize: int = 1_000_000_000
+    groupby_n_ary: int = 32
+    broadcast_join_limit: int = 4
+    shuffle_method: ShuffleMethod | None = None
+
+    def __post_init__(self) -> None:
+        if self.scheduler == "synchronous" and self.shuffle_method == "rapidsmpf":
             raise ValueError(
-                f"Engine configuration contains unsupported settings: {unsupported}"
+                "rapidsmpf shuffle method is not supported for synchronous scheduler"
             )
-        assert {"chunked", "chunk_read_limit", "pass_read_limit"}.issuperset(
-            config.get("parquet_options", {})
-        )
 
-        # Validate executor_options
-        executor = config.get("executor", "in-memory")
-        if executor == "streaming":
-            executor_options, unsupported = _valid_streaming_options(
-                config["executor_options"]
-            )
-            config["executor_options"] = executor_options
-        else:
-            unsupported = config.get("executor_options", {}).keys()
-        if unsupported:
-            raise ValueError(
-                f"Unsupported executor_options for {executor}: {unsupported}"
+        # frozen dataclass, so use object.__setattr__
+        object.__setattr__(
+            self, "fallback_mode", StreamingFallbackMode(self.fallback_mode)
+        )
+        object.__setattr__(self, "scheduler", Scheduler(self.scheduler))
+        if self.shuffle_method is not None:
+            object.__setattr__(
+                self, "shuffle_method", ShuffleMethod(self.shuffle_method)
             )
 
-        return config
-
-
-def _valid_option(
-    options: MutableMapping[str, Any],
-    key: str,
-    *,
-    default: Any,
-    choices: Sequence[Any] | None = None,
-    astype: type | None = None,
-) -> MutableMapping[str, Any]:
-    """Return a valid configuration option."""
-    value = options.get(key, default)
-    if astype is not None:
-        value = astype(value)
-    if choices is not None and value not in choices:
-        raise ValueError(f"Unsupported {key} option: {value}. Choices are: {choices}")
-    options[key] = value
-    return options
-
-
-def _valid_streaming_options(
-    options: MutableMapping[str, Any],
-) -> tuple[MutableMapping[str, Any], set[str]]:
-    """Return valid options for the streaming executor."""
-    default: Any
-    choices: Sequence[Any]
-    for key in (
-        _STREAMING_OPTIONS := (
-            "scheduler",  # Validate scheduler first
-            "fallback_mode",
-            "max_rows_per_partition",
-            "parquet_blocksize",
-            "cardinality_factor",
-            "groupby_n_ary",
-            "broadcast_join_limit",
-            "shuffle_method",
+        # Type / value check everything else
+        if not isinstance(self.max_rows_per_partition, int):
+            raise TypeError("max_rows_per_partition must be an int")
+        if not isinstance(self.cardinality_factor, dict):
+            raise TypeError("cardinality_factor must be a dict of column name to float")
+        if not isinstance(self.parquet_partition_blocksize, int):
+            raise TypeError("parquet_partition_blocksize must be an int")
+        if not isinstance(self.groupby_n_ary, int):
+            raise TypeError("groupby_n_ary must be an int")
+        if not isinstance(self.broadcast_join_limit, int):
+            raise TypeError("broadcast_join_limit must be an int")
+
+    def __hash__(self) -> int:
+        # cardinality factory, a dict, isn't natively hashable. We'll dump it
+        # to json and hash that.
+        d = dataclasses.asdict(self)
+        d["cardinality_factor"] = json.dumps(d["cardinality_factor"])
+        return hash(tuple(sorted(d.items())))
+
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class InMemoryExecutor:
+    """
+    Configuration for the cudf-polars in-memory executor.
+
+    Parameters
+    ----------
+    scheduler:
+        The scheduler to use for the in-memory executor. Currently
+        only ``Scheduler.SYNCHRONOUS`` is supported for the in-memory executor.
+    """
+
+    name: Literal["in-memory"] = dataclasses.field(default="in-memory", init=False)
+
+
+@dataclasses.dataclass(frozen=True, eq=True)
+class ConfigOptions:
+    """
+    Configuration for the polars GPUEngine.
+
+    Parameters
+    ----------
+    raise_on_fail
+        Whether to raise an exception when the GPU engine cannot execute a
+        query. ``False`` by default.
+    parquet_options
+        Options controlling parquet file reading and writing. See
+        :class:`ParquetOptions` for more.
+    executor
+        The executor to use for the GPU engine. See :class:`StreamingExecutor`
+        and :class:`InMemoryExecutor` for more.
+    device
+        The GPU used to run the query. If not provided, the
+        query uses the current CUDA device.
+    """
+
+    raise_on_fail: bool = False
+    parquet_options: ParquetOptions = dataclasses.field(default_factory=ParquetOptions)
+    executor: StreamingExecutor | InMemoryExecutor = dataclasses.field(
+        default_factory=InMemoryExecutor
+    )
+    device: int | None = None
+
+    @classmethod
+    def from_polars_engine(cls, engine: pl.GPUEngine) -> Self:
+        """
+        Create a `ConfigOptions` object from a `pl.GPUEngine` object.
+
+        This creates our internal, typed, configuration object from the
+        user-provided `polars.GPUEngine` object.
+        """
+        # these are the valid top-level keys in the engine.config that
+        # the user passes as **kwargs to GPUEngine.
+        valid_options = {
+            "executor",
+            "executor_options",
+            "parquet_options",
+            "raise_on_fail",
+        }
+
+        extra_options = set(engine.config.keys()) - valid_options
+        if extra_options:
+            raise TypeError(f"Unsupported executor_options: {extra_options}")
+
+        user_executor = engine.config.get("executor", "in-memory")
+        if user_executor is None:
+            user_executor = "in-memory"
+        # we mutate this below
+        user_executor_options = dict(engine.config.get("executor_options", {}))
+        user_parquet_options = engine.config.get("parquet_options", {})
+        user_raise_on_fail = engine.config.get("raise_on_fail", False)
+
+        # These are user-provided options, so we need to actually validate
+        # them.
+
+        if user_executor not in {"in-memory", "streaming"}:
+            raise ValueError(f"Unknown executor '{user_executor}'")
+
+        if not isinstance(user_raise_on_fail, bool):
+            raise TypeError("GPUEngine option 'raise_on_fail' must be a boolean.")
+
+        executor: InMemoryExecutor | StreamingExecutor
+
+        match user_executor:
+            case "in-memory":
+                executor = InMemoryExecutor(**user_executor_options)
+            case "streaming":
+                # backwards compatibility for the old parquet_blocksize option
+                if (
+                    "parquet_blocksize" in user_executor_options
+                    and "parquet_partition_blocksize" in user_executor_options
+                ):
+                    raise ValueError(
+                        "Specify just one of 'parquet_blocksize' or 'parquet_partition_blocksize'."
+                    )
+                elif "parquet_blocksize" in user_executor_options:
+                    warnings.warn(
+                        "The 'parquet_blocksize' option is deprecated. Use 'parquet_partition_blocksize' instead.",
+                        FutureWarning,
+                        stacklevel=3,
+                    )
+                    user_executor_options["parquet_partition_blocksize"] = (
+                        user_executor_options.pop("parquet_blocksize")
+                    )
path: python/cudf_polars/cudf_polars/utils/config.py
user: login: rjzamora
body: There are no actual users of the streaming executor yet, so I don't think there is any great reason to add a deprecation warning.

With that said, I'm personally -1 on the new name anyway. It make the option name longer, and we don't gain much (if anything) from the change. I also think it's outside the scope of this PR to change the specific option names (unless something is fundamentally wrong/broken).

I think the goal of this change is to clarify that this is related to streaming partition size, and to distinguish it from the general `parquet_options`. Can we just remove this option in favor of a general `target_partition_size` in a follow-up. The docs can say that supported IO functions (only parquet at first) will use this as a guideline for partitioning.
created_at: 2025-04-24T14:30:11Z
author_association: MEMBER

PR comments:
user: login: copy-pr-bot[bot]
created_at: 2025-04-17T17:48:00Z
body: Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message [here](https://docs.gha-runners.nvidia.com/cpr/auto-sync).

author_association: NONE
user: login: TomAugspurger
created_at: 2025-04-17T20:08:59Z
body: We need to figure out exactly how much validation to do, and which errors we raise when we encounter an issue. 
author_association: CONTRIBUTOR
user: login: TomAugspurger
created_at: 2025-04-21T14:54:01Z
body: Looking into the [CI failures](https://github.com/rapidsai/cudf/actions/runs/14574542521/job/40879107975?pr=18516#step:9:1975)

<details>

```
_______________________ test_parallel_scan[csv-scan_csv] _______________________
[gw0] linux -- Python 3.10.17 /opt/conda/envs/test/bin/python
Traceback (most recent call last):
  File "/opt/conda/envs/test/lib/python3.10/site-packages/_pytest/runner.py", line 341, in from_call
    result: Optional[TResult] = func()
  File "/opt/conda/envs/test/lib/python3.10/site-packages/_pytest/runner.py", line 262, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/opt/conda/envs/test/lib/python3.10/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/opt/conda/envs/test/lib/python3.10/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/opt/conda/envs/test/lib/python3.10/site-packages/pluggy/_callers.py", line 182, in _multicall
    return outcome.get_result()
  File "/opt/conda/envs/test/lib/python3.10/site-packages/pluggy/_result.py", line 100, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/opt/conda/envs/test/lib/python3.10/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/opt/conda/envs/test/lib/python3.10/site-packages/_pytest/runner.py", line 177, in pytest_runtest_call
    raise e
  File "/opt/conda/envs/test/lib/python3.10/site-packages/_pytest/runner.py", line 169, in pytest_runtest_call
    item.runtest()
  File "/opt/conda/envs/test/lib/python3.10/site-packages/_pytest/python.py", line 1792, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/opt/conda/envs/test/lib/python3.10/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/opt/conda/envs/test/lib/python3.10/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/opt/conda/envs/test/lib/python3.10/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/opt/conda/envs/test/lib/python3.10/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/opt/conda/envs/test/lib/python3.10/site-packages/_pytest/python.py", line 194, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/__w/cudf/cudf/python/cudf_polars/tests/experimental/test_scan.py", line 60, in test_parallel_scan
    assert_gpu_result_equal(q, engine=engine)
  File "/__w/cudf/cudf/python/cudf_polars/cudf_polars/testing/asserts.py", line 107, in assert_gpu_result_equal
    got = lazydf.collect(**final_cudf_collect_kwargs, engine=engine)
  File "/opt/conda/envs/test/lib/python3.10/site-packages/polars/_utils/deprecation.py", line 93, in wrapper
    return function(*args, **kwargs)
  File "/opt/conda/envs/test/lib/python3.10/site-packages/polars/lazyframe/frame.py", line 2188, in collect
    return wrap_df(ldf.collect(engine, callback))
polars.exceptions.ComputeError: TypeError: ('Could not serialize object of type HighLevelGraph', '<ToPickle: HighLevelGraph with 1 layers.\n<dask.highlevelgraph.HighLevelGraph object at 0x7fdec297a500>\n 0. 140594695501376\n>')
------------------------------ Captured log call -------------------------------
ERROR    distributed.protocol.pickle:pickle.py:79 Failed to serialize <ToPickle: HighLevelGraph with 1 layers.
<dask.highlevelgraph.HighLevelGraph object at 0x7fdec297a500>
 0. 140594695501376
>.
Traceback (most recent call last):
  File "/opt/conda/envs/test/lib/python3.10/site-packages/distributed/protocol/pickle.py", line 60, in dumps
    result = pickle.dumps(x, **dump_kwargs)
  File "<stringsource>", line 2, in rmm.pylibrmm.memory_resource.CudaMemoryResource.__reduce_cython__
TypeError: no default __reduce__ due to non-trivial __cinit__

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/envs/test/lib/python3.10/site-packages/distributed/protocol/pickle.py", line 65, in dumps
    pickler.dump(x)
  File "<stringsource>", line 2, in rmm.pylibrmm.memory_resource.CudaMemoryResource.__reduce_cython__
TypeError: no default __reduce__ due to non-trivial __cinit__

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/envs/test/lib/python3.10/site-packages/distributed/protocol/pickle.py", line 77, in dumps
    result = cloudpickle.dumps(x, **dump_kwargs)
  File "/opt/conda/envs/test/lib/python3.10/site-packages/cloudpickle/cloudpickle.py", line 1537, in dumps
    cp.dump(obj)
  File "/opt/conda/envs/test/lib/python3.10/site-packages/cloudpickle/cloudpickle.py", line 1303, in dump
    return super().dump(obj)
  File "<stringsource>", line 2, in rmm.pylibrmm.memory_resource.CudaMemoryResource.__reduce_cython__
TypeError: no default __reduce__ due to non-trivial __cinit__
```

</details>

I wasn't able to immediately reproduce the failure.

Edit: I wasn't passing the `--executor streaming --scheduler distributed` flags. With those I can reproduce it.
author_association: CONTRIBUTOR
user: login: TomAugspurger
created_at: 2025-04-21T15:51:23Z
body: One other thing to figure out: we have a couple other configuration options floating around as environment variables:

- `CUDF_POLARS_NUM_WORKERS`
- `POLARS_GPU_ENABLE_CUDA_MANAGED_MEMORY`
- `POLARS_VERBOSE`

Perhaps not `POLARS_VERBOSE` since it's a general polars setting, but maybe the others. I'd suggest making all our configuration options configurable through environment variables (with a consistent prefix) and through kwargs passed to `GPUEngine`.

But that can probably wait for a separate PR
author_association: CONTRIBUTOR
user: login: TomAugspurger
created_at: 2025-04-21T21:24:55Z
body: Is there an easy way to see which lines aren't covered in the CI failure here https://github.com/rapidsai/cudf/actions/runs/14580875852/job/40898672037?pr=18516#step:10:703?

I have 100% coverage locally when I run `./ci/run_cudf_polars_pytests.sh --cov cudf_polars --cov-fail-under=100 --cov-config=./pyproject.toml --junitxml=junit-cudf-polars.xml`

Edit: nevermind, I *can* reproduce it now. At least locally, it's just from the `if TYPE_CHECKING:` block not being measured.

Edit again: I think my failing one was getting overwritten by a subsquent run in that script. Anyway, the actual failure is `python -m pytest --cov cudf_polars --cov-fail-under=100 --cov-config=pyproject.toml --executor streaming tests` and `coverage report -m` shows that it's  from

```
cudf_polars/dsl/ir.py                          789      1    99%   632
```

That line isn't changed, but clearly I've changed the semantics of some configuration value. I'll investigate.
author_association: CONTRIBUTOR
user: login: TomAugspurger
created_at: 2025-04-24T14:41:45Z
body: I reverted the parquet_blocksize change, so now that's back to a 1:1 translation from the previous config to just using a dataclass. 
author_association: CONTRIBUTOR
user: login: TomAugspurger
created_at: 2025-04-25T15:31:15Z
body: /merge
author_association: CONTRIBUTOR