Repository: Eventual-Inc/Daft
Pull request #4203: feat: defines pushdown terms for python
Author: login: rchowell
Labels: name: feat
Created: 2025-04-17T00:11:14Z | Merged: 2025-04-22T17:15:40Z

## Changes Made

This introduces a simple Term IR ([much like pyiceberg](https://py.iceberg.apache.org/reference/pyiceberg/expressions/#pyiceberg.expressions.Term)) which is a model for pushdown expressions. This is scoped to the python pushdown package, and the Term IR comes with a visitor which can be used to translate to other domains. This enable easy consumption of PyExpr / rust expressions. I have an example visitor which produces s-expressions, and can add one to translate to the PyIceberg Expression IR.

An alternative design is no IR and visitor-only - this is nice because there is no additional IR, however it comes at the tradeoff of possible tighter coupling to the Rust expressions and we may want an IR anyways if we ever want to serde pushdown expressions. Please let me know what you all think is best here. #4210 

## TODOs
- [x] Iceberg transforms
- [x] PyPushdown conversion tests
- [x] Documentation for scan tasks and pushdowns



Diff:
diff --git a/daft/daft/__init__.pyi b/daft/daft/__init__.pyi
index 21a580c0b0..70fb5f5fe4 100644
--- a/daft/daft/__init__.pyi
+++ b/daft/daft/__init__.pyi
@@ -17,6 +17,7 @@ if TYPE_CHECKING:
     from pyiceberg.schema import Schema as IcebergSchema
     from pyiceberg.table import TableProperties as IcebergTableProperties
 
+    from daft.io.pushdowns import Term
     from daft.runners.runner import Runner
 
 class ImageMode(Enum):
@@ -775,10 +776,22 @@ class Pushdowns:
     partition_filters: PyExpr | None
     limit: int | None
 
+    def __init__(
+        self,
+        columns: list[str] | None = None,
+        filters: PyExpr | None = None,
+        partition_filters: PyExpr | None = None,
+        limit: int | None = None,
+    ) -> None: ...
     def filter_required_column_names(self) -> list[str]:
         """List of field names that are required by the filter predicate."""
         ...
 
+    @staticmethod
+    def _to_term(expr: PyExpr, schema: PySchema | None = None) -> Term:
+        """Converts a PyExpr into a pushdown Term, optionally binding to the given schema."""
+        ...
+
 def read_parquet(
     uri: str,
     columns: list[str] | None = None,
diff --git a/daft/io/pushdowns.py b/daft/io/pushdowns.py
new file mode 100644
index 0000000000..324d6b7958
--- /dev/null
+++ b/daft/io/pushdowns.py
@@ -0,0 +1,242 @@
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from dataclasses import dataclass
+from functools import singledispatchmethod
+from typing import Generic, Sequence, TypeVar, Union
+
+Value = Union[str, int, float, bool, None]
+R = TypeVar("R")
+C = TypeVar("C")
+
+
+@dataclass(frozen=True)
+class Pushdowns(ABC):
+    """Base class for pushdown information."""
+
+    pass
+
+
+@dataclass(frozen=True)
+class Term(ABC):
+    """Term is the base class for representing expressions in pushdowns."""
+
+    def __str__(self) -> str:
+        return LispyVisitor().visit(self, None)
+
+
+@dataclass(frozen=True)
+class Reference(Term):
+    """Reference to a field in some schema.
+
+    Attributes:
+        path: Field path in the schema which is typically a column name.
+        index: Field index if this reference is bound to a schema.
+    """
+
+    path: str
+    index: int | None = None
+
+
+@dataclass(frozen=True)
+class Literal(Term):
+    """Literal value which does not currently carry its daft type information.
+
+    The representation of types and the full-fidelity of values complicates both
+    the s-expression encoding and the end user consumption. The goal here is for
+    simple consumption of daft's rust expressions from a python context. We may
+    iterate on this design's complexity as more advanced use-cases emerge.
+
+    Attributes:
+        value: Literal value is some python object.
+
+    Examples:
+        >>> v1 = Literal("hello")
+        >>> v2 = Literal(1)
+        >>> v3 = Literal(2.0)
+        >>> v4 = Literal(False)
+        >>> v5 = Literal(None)
+    """
+
+    value: Value
+
+
+@dataclass(frozen=True)
+class Expr(Term):
+    """Expr is a generic expression represented by a procedure symbol and its args.
+
+    This representation was chosen because it's the simplest while avoiding the complexity
+    of a more 'pure' concatenation style or introducing additional Term variants.
+    This is a trade-off where lower representation complexity requires slightly more creativity
+    in representing the various expression forms, but named arguments ease consumption.
+
+    Exprs follow the common rule that positional come before named.
+    This is enforced on instantiation since we use python args and kwargs.
+
+    Example:
+        >>> Expr("f", 42)  # (f 42)
+        >>> Expr("f", "hello")  # (f "hello")
+        >>> Expr("f", 1, 2.5, "test")  # (f 1 2.50 "test")
+        >>> Expr("f", value=True)  # (f value::true)
+        >>> Expr("f", x=10, y=20.5)  # (f x::10 y::20.50)
+    """
+
+    proc: str
+    args: list[Arg]
+
+    def __init__(self, proc: str, *args: Term | Value, **kwargs: Term | Value):
+        self_proc = proc
+        self_args = []
+        for arg in args:
+            term = arg if isinstance(arg, Term) else Literal(arg)
+            self_args.append(Arg(term))
+        for label, arg in kwargs.items():
+            term = arg if isinstance(arg, Term) else Literal(arg)
+            self_args.append(Arg(term, label))
+        object.__setattr__(self, "proc", self_proc)
+        object.__setattr__(self, "args", self_args)
+
+    def __str__(self) -> str:
+        return LispyVisitor().visit(self, None)
+
+    def __len__(self) -> int:
+        return len(self.args)
+
+    def __getitem__(self, key: int | str | slice) -> Term | Sequence[Term] | None:
+        """Get an argument by index, name, or slice."""
+        if isinstance(key, int):
+            return self.args[key].term
+        elif isinstance(key, str):
+            return next((arg.term for arg in self.args if arg.label == key), None)
+        elif isinstance(key, slice):
+            return [arg.term for arg in self.args[key]]
+        else:
+            raise TypeError("Expected key to be int, str, or slice.")
+
+
+@dataclass(frozen=True)
+class Arg:
+    """Arg is just an s-expression with optional label, notably does not inherit from Term.
+
+    Attributes:
+        term: Term is the actual argument.
+        label: Label is an optional label if this argument was named.
+    """
+
+    term: Term
+    label: str | None = None
+
+
+class TermVisitor(ABC, Generic[C, R]):
+    """TermVisitor uses the @singledispatchmethod for a class-based visitor.
+
+    Note that we are not using the typical "accept" method on an term variant
+    for dispatching because the @singledispatchmethod handles this for us.
+    There is no need to add accept methods to each variant which simplifies
+    both the term tree and the visitor implementations.
+
+    The type `R` represents the return type, and `C` represents the context.
+    The context parameter is useful when passing state which is scoped when
+    performing a visitor traversal (fold). For non-scoped state, you can just
+    add normal instance properties.
+    """
+
+    @singledispatchmethod
+    def visit(self, term: Term, context: C) -> R:
+        raise NotImplementedError(f"No visit method for type {type(term)}")
+
+    @visit.register
+    def _(self, term: Reference, context: C) -> R:
+        return self.visit_reference(term, context)
+
+    @visit.register
+    def _(self, term: Literal, context: C) -> R:
+        return self.visit_literal(term, context)
+
+    @visit.register
+    def _(self, term: Expr, context: C) -> R:
+        return self.visit_expr(term, context)
+
+    @abstractmethod
+    def visit_reference(self, term: Reference, context: C) -> R: ...
+
+    @abstractmethod
+    def visit_literal(self, term: Literal, context: C) -> R: ...
+
+    @abstractmethod
+    def visit_expr(self, term: Expr, context: C) -> R: ...
+
+
+class LispyVisitor(TermVisitor[None, str]):
+    """LispyVisitor is an example visitor implementation for printing s-expressions.
+
+    This can be implemented *much* more concisely directly in the __str__ .. but
+    this is an exercise and tutorial to show off visitor usage and patterns. We
+    use a 'str' return type and there is currently no scoped context. An example
+    of scoped context here might be an indentation level for pretty-printing.
+
+    An interesting exercise would be implementing https://norvig.com/lispy.html.
+
+    Example:
+    >>> visitor = LispyVisitor()
+    >>> term = Expr("+", 1, 2)
+    >>> print(visitor.visit(term))
+    """
+
+    ###
+    # visitor variants
+    ###
+
+    def visit_reference(self, term: Reference, context: None) -> str:
+        """References just use their unquoted path."""
+        return term.path
+
+    def visit_literal(self, term: Literal, context: None) -> str:
+        """Literals uses the underlying value's lisp representation."""
+        return self._value(term.value)
+
+    def visit_expr(self, term: Expr, context: None) -> str:
+        """Expr is represented as procs, so (proc args...)."""
+        proc = term.proc
+        args = ""
+        for arg in term.args:
+            args += " "  # no join, since we may add indentation
+            args += self._arg(arg, context)
+        return f"({proc}{args})" if args else f"({proc})"
+
+    ###
+    # helpers
+    ###
+
+    @singledispatchmethod
+    def _value(self, value: Value | None) -> str:
+        """Lisp value representation variants."""
+        raise NotImplementedError
+
+    @_value.register
+    def _nil(self, _: None) -> str:
+        """Lisp uses nil, but sicp scheme prefers () :shrug: .. using null."""
+        return "null"
+
+    @_value.register
+    def _int(self, value: int) -> str:
+        return str(value)
+
+    @_value.register
+    def _float(self, value: float) -> str:
+        """Use two decimal places since precision doesn't actually matter here."""
+        return f"{value:.2f}"
+
+    @_value.register
+    def _bool(self, value: bool) -> str:
+        return "true" if value else "false"
+
+    @_value.register
+    def _str(self, value: str) -> str:
+        """Lisp uses double-quotes for string literals."""
+        return f'"{value}"'
+
+    def _arg(self, arg: Arg, context: None) -> str:
+        """Use annotation style labels e.g. label::term."""
+        term: str = self.visit(arg.term, context)
+        return f"{arg.label}::{term}" if arg.label else term
diff --git a/daft/io/scan.py b/daft/io/scan.py
index 132bd63d64..97f12b1fbb 100644
--- a/daft/io/scan.py
+++ b/daft/io/scan.py
@@ -1,9 +1,18 @@
 from __future__ import annotations
 
 import abc
+from dataclasses import dataclass
 from typing import TYPE_CHECKING
 
-from daft.daft import PartitionField, PartitionTransform, Pushdowns, ScanTask
+from daft.daft import (
+    PartitionField,
+    PartitionTransform,
+    ScanTask,
+)
+from daft.daft import (
+    Pushdowns as PyPushdowns,
+)
+from daft.io.pushdowns import Pushdowns, Reference, Term
 
 if TYPE_CHECKING:
     from collections.abc import Iterator
@@ -24,32 +33,99 @@ def make_partition_field(
 class ScanOperator(abc.ABC):
     @abc.abstractmethod
     def schema(self) -> Schema:
+        """Returns the schema of the data source."""
         raise NotImplementedError()
 
     @abc.abstractmethod
     def display_name(self) -> str:
+        """Returns a human-readable name for this scan operator."""
         return self.__class__.__name__
 
     @abc.abstractmethod
     def partitioning_keys(self) -> list[PartitionField]:
+        """Returns the partitioning keys for this data source."""
         raise NotImplementedError()
 
     @abc.abstractmethod
     def can_absorb_filter(self) -> bool:
+        """Returns true if this scan can accept predicate pushdowns."""
         raise NotImplementedError()
 
     @abc.abstractmethod
     def can_absorb_limit(self) -> bool:
+        """Returns true if this scan can accept limit pushdowns."""
         raise NotImplementedError()
 
     @abc.abstractmethod
     def can_absorb_select(self) -> bool:
+        """Returns true if this scan can accept projection pushdowns."""
         raise NotImplementedError()
 
     @abc.abstractmethod
     def multiline_display(self) -> list[str]:
+        """Returns a multi-line string representation of this scan operator."""
         raise NotImplementedError()
 
     @abc.abstractmethod
-    def to_scan_tasks(self, pushdowns: Pushdowns) -> Iterator[ScanTask]:
+    def to_scan_tasks(self, pushdowns: PyPushdowns) -> Iterator[ScanTask]:
+        """Converts this scan operator into scan tasks with the given pushdowns."""
         raise NotImplementedError()
+
+
+@dataclass(frozen=True)
+class ScanPushdowns(Pushdowns):
+    """ScanPushdowns is a python-friendly representation of daft's rust Pushdown type.
+
+    The existing Pushdowns class comes from pyo3 which holds references to wrapper
+    classes for daft's expression enum. This is not ammenable for python consumption.
+
+    In Daft 0.5.0 we will change the pushdown parameter from daft.daft.Pushdowns to
+    daft.io.Pushdowns. For now, please use `ScanPushdowns._from_pypushdowns(py_pushdowns)`
+    to convert the rust expressions to this python pushdowns class.
+
+    As part of a migration plan to python-friendly pushdowns, we've introduced
+    a python "ScanPushdowns" object which uses the long-term "Pushdowns" ABC.
+    Additionally, we have renamed Pushdowns to PyPushdowns for consistency with
+    daft's other pyo3 wrapper classes.
+
+    Attributes:
+        projections (list[Term] | None): Optional list of Terms (typically column references) to project.
+        predicate (Term | None): Optional filter predicate to apply to rows.
+        limit (int | None): Optional limit on the number of rows to return.
+    """
+
+    projections: list[Reference] | None = None
+    predicate: Term | None = None
+    limit: int | None = None
+
+    @classmethod
+    def _from_pypushdowns(cls, pypushdowns: PyPushdowns, schema: Schema) -> ScanPushdowns:
+        # need the PySchema to send to rust side
+        _schema = schema._schema
+
+        # use both the 'filters' and 'partition_filters' as the predicate
+        predicate: Term | None = None
+        p1 = pypushdowns.filters
+        p2 = pypushdowns.partition_filters
+
+        # combine the two predicates with 'and' if necessary
+        if p1 is not None and p2 is not None:
+            predicate = PyPushdowns._to_term(p1 & p2, _schema)
+        elif p1 is not None:
+            predicate = PyPushdowns._to_term(p1, _schema)
+        elif p2 is not None:
+            predicate = PyPushdowns._to_term(p2, _schema)
+
+        # field to index map is case-sensitive .. to do it proper we'll need rust-side work.
+        fields = {field.name: index for index, field in enumerate(schema)}
+
+        # convert each column name to a reference term, this requires resolving in the schema.
+        if pypushdowns.columns:
+            projections = [Reference(path=col, index=fields[col]) for col in pypushdowns.columns]
+        else:
+            projections = None
+
+        # limit is just an int, so no conversion necessary
+        limit = pypushdowns.limit
+
+        return ScanPushdowns(projections, predicate, limit)
diff --git a/docs/advanced/pushdowns.md b/docs/advanced/pushdowns.md
new file mode 100644
index 0000000000..e2a9da8108
--- /dev/null
+++ b/docs/advanced/pushdowns.md
@@ -0,0 +1,92 @@
+# Pushdowns
+
+!!! warning "Warning"
+
+    These APIs are considered experimental.
+
+Daft supports predicate, projection, and limit pushdowns in its scan operators.
+
+## Usage
+
+Daft does not *officially* support user-defined scan operators, but it is
+possible today by extending the [`ScanOperator`][daft.io.scan.ScanOperator].
+
+The [`to_scan_tasks`][daft.io.scan.ScanOperator.to_scan_tasks] method receives a
+pushdowns object which contains information the optimizer determined can be
+pushed down into the scan. This is a wrapper over a Rust type, and its members
+cannot be easily interpreted from python. To solve this, we use a python
+[`Pushdowns`][daft.io.pushdowns.Pushdowns] class holding
+[`Term`][daft.io.pushdowns.Term] objects which are a representation of the
+pushed-down expressions. You can construct a
+[`ScanPushdowns`][daft.io.scan.ScanPushdowns] object using `_from_pypushdowns(pushdowns, schema)`.
+
+Once you have the pushdown terms, you will need to apply them based upon
+your own context. One way to operate on a term tree is using the
+[`TermVisitor`][daft.io.pushdowns.TermVisitor].
+
+## Term Reference
+
+!!! warn "Warning!"
+
+    This list is not comprehensive, and it is recommended to either inspect
+    a term tree or test your
+
+### Comparison Predicates
+
+| Procedure | Arguments  | Description              |
+|-----------|------------|--------------------------|
+| `=`       | `lhs, rhs` | Equal to                 |
+| `!=`      | `lhs, rhs` | Not equal to             |
+| `<`       | `lhs, rhs` | Less than                |
+| `<=`      | `lhs, rhs` | Less than or equal to    |
+| `>`       | `lhs, rhs` | Greater than             |
+| `>=`      | `lhs, rhs` | Greater than or equal to |
+
+### Logical Predicates
+
+| Procedure | Arguments  | Description |
+|-----------|------------|-------------|
+| `and`     | `lhs, rhs` | Logical AND |
+| `or`      | `lhs, rhs` | Logical OR  |
+| `not`     | `expr`     | Logical NOT |
+
+### Other Predicates
+
+| Procedure  | Arguments            | Description                                    |
+|------------|----------------------|------------------------------------------------|
+| `is_null`  | `expr`               | Check if expression is NULL                    |
+| `is_nan`   | `expr`               | Check if expression is NaN                     |
+| `not_null` | `expr`               | Check if expression is not NULL                |
+| `not_nan`  | `expr`               | Check if expression is not NaN                 |
+| `between`  | `expr, lower, upper` | Check if expression is between lower and upper |
+| `is_in`    | `item, list`         | Check if item is in list                       |
+
+### Expressions
+
+Terms may represent any arbitrary expression, not just predicates!
+
+| Procedure | Arguments  | Description                 |
+|-----------|------------|-----------------------------|
+| `+`       | `lhs, rhs` | Arithmetic addition         |
+| `-`       | `lhs, rhs` | Arithmetic subtraction      |
+| `*`       | `lhs, rhs` | Arithmetic multiplication   |
+| `/`       | `lhs, rhs` | Arithmetic division         |
+| `mod`     | `lhs, rhs` | Modulus                     |
+| `lshift`  | `lhs, rhs` | Bitshift left               |
+| `rshift`  | `lhs, rhs` | Bitshift right              |
+| `list`    | `items...` | List constructor or literal |
+
+### Functions
+
+Terms for functions are uniformly represented with the function name as the
+procedure name followed by the function's arguments. For example, here is
+how the Iceberg partition transforms are represented as terms.
+
+| Function           | Arguments     | Description                          |
+|--------------------|---------------|--------------------------------------|
+| `years`            | `expr`        | Extract years from a date/timestamp  |
+| `months`           | `expr`        | Extract months from a date/timestamp |
+| `days`             | `expr`        | Extract days from a date/timestamp   |
+| `hours`            | `expr`        | Extract hours from a timestamp       |
+| `iceberg_bucket`   | `expr, n`     | Iceberg bucketing function           |
+| `iceberg_truncate` | `expr, width` | Iceberg truncate function            |
diff --git a/docs/api/io.md b/docs/api/io.md
new file mode 100644
index 0000000000..4da345a0cf
--- /dev/null
+++ b/docs/api/io.md
@@ -0,0 +1,16 @@
+# I/O
+
+Daft supports diverse input sources and output sinks which are covered in [DataFrame Creation](dataframe_creation.md) —
+this page covers lower-level APIs which we are evolving for more advanced usage.
+
+!!! warning "Warning"
+
+    These APIs are considered experimental.
+
+::: daft.io.scan.ScanOperator
+    options:
+        filters: ["!^_"]
+
+::: daft.io.scan.ScanPushdowns
+    options:
+        filters: ["!^_"]
diff --git a/docs/api/pushdowns.md b/docs/api/pushdowns.md
new file mode 100644
index 0000000000..fb3bf4c51f
--- /dev/null
+++ b/docs/api/pushdowns.md
@@ -0,0 +1,31 @@
+# Pushdowns
+
+Daft has predicate, projection, and limit pushdowns with expressions being represented by *Terms*. Learn more about [Pushdowns](../advanced/pushdowns.md) in the Daft User Guide.
+
+::: daft.io.pushdowns.Pushdowns
+    options:
+        filters: ["!^_"]
+
+::: daft.io.pushdowns.Term
+    options:
+        filters: ["!^_"]
+
+::: daft.io.pushdowns.Reference
+    options:
+        filters: ["!^_"]
+
+::: daft.io.pushdowns.Literal
+    options:
+        filters: ["!^_"]
+
+::: daft.io.pushdowns.Expr
+    options:
+        filters: ["!^_"]
+
+::: daft.io.pushdowns.Arg
+    options:
+        filters: ["!^_"]
+
+::: daft.io.pushdowns.TermVisitor
+    options:
+        filters: ["!^_"]
diff --git a/mkdocs.yml b/mkdocs.yml
index c98670ce8b..421ecde406 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -28,6 +28,7 @@ nav:
   - Advanced:
     - Managing Memory Usage: advanced/memory.md
     - Partitioning: advanced/partitioning.md
+    - Pushdowns: advanced/pushdowns.md
   - Integrations:
     - Catalogs & Tables:
       - Apache Hudi: integrations/hudi.md
@@ -63,6 +64,8 @@ nav:
   - Groupby: api/groupby.md
   - User-Defined Functions: api/udf.md
   - Series: api/series.md
+  - I/O: api/io.md
+  - Pushdowns: api/pushdowns.md
   - Configuration: api/config.md
   - Miscellaneous: api/misc.md
 - SQL Reference:
diff --git a/src/common/scan-info/src/python.rs b/src/common/scan-info/src/python.rs
index 8c296f32a9..1dcbca13d3 100644
--- a/src/common/scan-info/src/python.rs
+++ b/src/common/scan-info/src/python.rs
@@ -1,13 +1,23 @@
-use pyo3::prelude::*;
+use daft_dsl::{
+    functions::{partitioning::PartitioningExpr, FunctionExpr},
+    Column, Expr, LiteralValue, Operator, ResolvedColumn,
+};
+use daft_schema::schema::SchemaRef;
+use pyo3::{
+    exceptions::PyValueError,
+    prelude::*,
+    types::{PyList, PyTuple},
+};
 
 pub mod pylib {
     use std::sync::Arc;
 
     use daft_dsl::python::PyExpr;
-    use daft_schema::python::field::PyField;
+    use daft_schema::python::{field::PyField, schema::PySchema};
     use pyo3::{prelude::*, pyclass};
     use serde::{Deserialize, Serialize};
 
+    use super::TermBuilder;
     use crate::{PartitionField, PartitionTransform, Pushdowns};
 
     #[pyclass(module = "daft.daft", name = "PartitionField", frozen)]
@@ -95,11 +105,35 @@ pub mod pylib {
     #[pyclass(module = "daft.daft", name = "Pushdowns", frozen)]
     #[derive(Debug, Clone, Serialize, Deserialize)]
     pub struct PyPushdowns(pub Arc<Pushdowns>);
+
     #[pymethods]
     impl PyPushdowns {
+        #[new]
+        #[pyo3(signature = (
+            filters = None,
+            partition_filters = None,
+            columns = None,
+            limit = None,
+        ))]
+        pub fn new(
+            filters: Option<PyExpr>,
+            partition_filters: Option<PyExpr>,
+            columns: Option<Vec<String>>,
+            limit: Option<usize>,
+        ) -> Self {
+            let pushdowns = Pushdowns::new(
+                filters.map(|f| f.expr),
+                partition_filters.map(|f| f.expr),
+                columns.map(Arc::new),
+                limit,
+            );
+            Self(Arc::new(pushdowns))
+        }
+
         pub fn __repr__(&self) -> PyResult<String> {
             Ok(format!("{:#?}", self.0))
         }
+
         #[getter]
         #[must_use]
         pub fn limit(&self) -> Option<usize> {
@@ -133,6 +167,19 @@ pub mod pylib {
                 .as_ref()
                 .map(daft_dsl::optimization::get_required_columns)
         }
+
+        #[staticmethod]
+        #[pyo3(signature = (expr,schema=None))]
+        pub fn _to_term(
+            py: Python<'_>,
+            expr: PyExpr,
+            schema: Option<PySchema>,
+        ) -> PyResult<PyObject> {
+            let schema = schema.map(|s| s.schema);
+            let tb = TermBuilder::new(py, schema)?;
+            let term = tb.to_term(expr.expr.as_ref())?;
+            Ok(term.unbind())
+        }
     }
 }
 
@@ -142,3 +189,234 @@ pub fn register_modules(parent: &Bound<PyModule>) -> PyResult<()> {
     parent.add_class::<pylib::PyPushdowns>()?;
     Ok(())
 }
+
+//-------------------------------
+// Produce python pushdown terms.
+//-------------------------------
+
+/// Terms during building are bound, then we unbind once done.
+type Term<'py> = Bound<'py, PyAny>;
+
+/// Holds the py GIL and module to make bound term building easy.
+struct TermBuilder<'py> {
+    py: Python<'py>,
+    schema: Option<SchemaRef>,
+    module: Bound<'py, PyModule>,
+}
+
+impl<'py> TermBuilder<'py> {
+    fn new(py: Python<'py>, schema: Option<SchemaRef>) -> PyResult<Self> {
+        Ok(Self {
+            py,
+            schema,
+            module: PyModule::import(py, "daft.io.pushdowns")?,
+        })
+    }
+
+    fn to_term(&self, expr: &Expr) -> PyResult<Term> {
+        if let Expr::Column(col) = expr {
+            return self.to_reference(col);
+        }
+        if let Expr::Literal(lit) = expr {
+            return self.to_literal(lit);
+        }
+        let proc: &str;
+        let args = PyList::empty(self.py);
+        match &expr {
+            Expr::Alias(expr, alias) => {
+                // (alias <alias> <expr>)
+                proc = "alias";
+                args.append(alias.to_string())?;
+                args.append(self.to_term(expr)?)?;
+            }
+            Expr::Not(expr) => {
+                // (not <expr>)
+                proc = "not";
+                args.append(self.to_term(expr)?)?;
+            }
+            Expr::BinaryOp { op, left, right } => {
+                // (operator <lhs> <rhs>)
+                proc = Self::proc_of_predicate(op);
+                args.append(self.to_term(left)?)?;
+                args.append(self.to_term(right)?)?;
+            }
+            Expr::IsNull(expr) => {
+                // (is_null <expr>)
+                proc = "is_null";
+                args.append(self.to_term(expr)?)?;
+            }
+            Expr::NotNull(expr) => {
+                // (not_null <expr>)
+                proc = "not_null";
+                args.append(self.to_term(expr)?)?;
+            }
+            Expr::Between(expr, expr1, expr2) => {
+                // (between <expr> <lower> <upper>)
+                proc = "between";
+                args.append(self.to_term(expr)?)?;
+                args.append(self.to_term(expr1)?)?;
+                args.append(self.to_term(expr2)?)?;
+            }
+            Expr::Function { func, inputs } => {
+                // (<proc> <args>...)
+                proc = Self::proc_of_function(func)?;
+                for arg in inputs {
+                    args.append(self.to_term(arg.as_ref())?)?;
+                }
+                // ineligant .. add args from the two iceberg partitioning functions
+                if let FunctionExpr::Partitioning(pexpr) = func {
+                    match pexpr {
+                        PartitioningExpr::IcebergBucket(buckets) => {
+                            let lit = LiteralValue::Int32(*buckets);
+                            let arg = self.to_literal(&lit)?;
+                            args.append(arg)?;
+                        }
+                        PartitioningExpr::IcebergTruncate(size) => {
+                            let lit = LiteralValue::Int64(*size);
+                            let arg = self.to_literal(&lit)?;
+                            args.append(arg)?;
+                        }
+                        _ => {}
+                    };
+                }
+            }
+            Expr::ScalarFunction(scalar_function) => {
+                // (<proc> <args>...)
+                proc = scalar_function.name();
+                for arg in scalar_function.inputs.iter().as_ref() {
+                    args.append(self.to_term(arg.as_ref())?)?;
+                }
+            }
+            Expr::List(items) => {
+                // (list <items>...)
+                proc = "list";
+                for arg in items.iter().as_ref() {
+                    args.append(self.to_term(arg.as_ref())?)?;
+                }
+            }
+            Expr::IsIn(item, items) => {
+                // (is_in item (list <items>))
+                proc = "is_in";
+                let items = Expr::List(items.clone());
+                args.append(self.to_term(item)?)?;
+                args.append(self.to_term(items.as_ref())?)?;
+            }
+            _ => {
+                return Err(PyValueError::new_err(format!(
+                    "Unsupported pushdown expression: {}",
+                    expr
+                )))
+            }
+        };
+        self.to_expr(proc, args)
+    }
+
+    /// Reference(<path> [, <index>])
+    fn to_reference(&self, column: &Column) -> PyResult<Term> {
+        // we allow unbound columns in construction.
+        let name = match column {
+            Column::Unresolved(col) => col.name.to_string(),
+            Column::Resolved(col) => match col {
+                ResolvedColumn::Basic(col) => col.to_string(),
+                _ => {
+                    return Err(PyValueError::new_err(format!(
+                        "Unexpected or unresolved column found in pushdown: {}",
+                        column
+                    )))
+                }
+            },
+        };
+        // if we have a schema, create a bound reference by setting its index.
+        let index = if let Some(schema) = &self.schema {
+            Some(schema.get_index(&name)?)
+        } else {
+            None
+        };
+        self.module.getattr("Reference")?.call1((name, index))
+    }
+
+    /// Literal(<value>)
+    fn to_literal(&self, literal: &LiteralValue) -> PyResult<Term> {
+        let cls = self.module.getattr("Literal")?;
+        match literal {
+            // null
+            LiteralValue::Null => cls.call1((None::<bool>,)),
+            // bool
+            LiteralValue::Boolean(b) => cls.call1((b,)),
+            // int
+            LiteralValue::Int8(i) => cls.call1((i,)),
+            LiteralValue::UInt8(i) => cls.call1((i,)),
+            LiteralValue::Int16(i) => cls.call1((i,)),
+            LiteralValue::UInt16(i) => cls.call1((i,)),
+            LiteralValue::Int32(i) => cls.call1((i,)),
+            LiteralValue::UInt32(i) => cls.call1((i,)),
+            LiteralValue::Int64(i) => cls.call1((i,)),
+            LiteralValue::UInt64(i) => cls.call1((i,)),
+            // float
+            LiteralValue::Float64(f) => cls.call1((f,)),
+            // str
+            LiteralValue::Utf8(s) => cls.call1((s,)),
+            // unsupported literals
+            _ => Err(PyValueError::new_err(format!(
+                "Unsupported pushdown literal: {}",
+                literal
+            ))),
+        }
+    }
+
+    /// Expr(<proc>, *args, **kwargs)
+    fn to_expr(&self, proc: &str, args: Bound<'py, PyList>) -> PyResult<Term> {
+        args.insert(0, proc.to_string())?;
+        let args = PyTuple::new(self.py, args)?;
+        self.module.getattr("Expr")?.call1(args)
+    }
+
+    /// Term procedure names for binary operator predicates inpushdowns.
+    fn proc_of_predicate(operator: &Operator) -> &'static str {
+        match operator {
+            // only works for logical connectives!
+            Operator::And => "and",
+            Operator::Or => "or",
+            Operator::Xor => "xor",
+            // use scheme-like comparison operators (please don't use display).
+            Operator::Eq => "=",
+            Operator::NotEq => "!=",
+            Operator::Lt => "<",
+            Operator::LtEq => "<=",
+            Operator::Gt => ">",
+            Operator::GtEq => ">=",
+            // use schema-like arithmetic operators.
+            Operator::Plus => "+",
+            Operator::Minus => "-",
+            Operator::Multiply => "*",
+            Operator::TrueDivide => "/",
+            // use scheme-like names (please don't use display).
+            Operator::FloorDivide => "quotient",
+            Operator::Modulus => "mod",
+            Operator::EqNullSafe => "eq_null_safe",
+            // ash is too arcane and more complicated
+            Operator::ShiftLeft => "lshift",
+            Operator::ShiftRight => "rshift",
+        }
+    }
+
+    /// Term procedure names for functions
+    fn proc_of_function(function: &FunctionExpr) -> PyResult<&str> {
+        match function {
+            FunctionExpr::Python(udf) => Ok(&*udf.name),
+            FunctionExpr::Partitioning(pexpr) => match pexpr {
+                PartitioningExpr::Years => Ok("years"),
+                PartitioningExpr::Months => Ok("months"),
+                PartitioningExpr::Days => Ok("days"),
+                PartitioningExpr::Hours => Ok("hours"),
+                // TODO parameters
+                PartitioningExpr::IcebergBucket(_) => Ok("iceberg_bucket"),
+                PartitioningExpr::IcebergTruncate(_) => Ok("iceberg_truncate"),
+            },
+            _ => Err(PyValueError::new_err(format!(
+                "Unsupported pushdown function: {}",
+                function
+            ))),
+        }
+    }
+}
diff --git a/src/daft-dsl/src/expr/mod.rs b/src/daft-dsl/src/expr/mod.rs
index f61088a006..a4aa2d7d0d 100644
--- a/src/daft-dsl/src/expr/mod.rs
+++ b/src/daft-dsl/src/expr/mod.rs
@@ -165,9 +165,9 @@ pub enum ResolvedColumn {
     OuterRef(Field, PlanRef),
 }
 
-impl Display for Column {
-    fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
-        let name = match self {
+impl Column {
+    pub fn name(&self) -> String {
+        match self {
             Self::Unresolved(UnresolvedColumn {
                 name,
                 plan_ref: PlanRef::Alias(plan_alias),
@@ -181,9 +181,13 @@ impl Display for Column {
                 PlanRef::Alias(plan_alias),
             )) => format!("{plan_alias}.{name}"),
             Self::Resolved(ResolvedColumn::OuterRef(Field { name, .. }, _)) => name.to_string(),
-        };
+        }
+    }
+}
 
-        write!(f, "col({name})")
+impl Display for Column {
+    fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+        write!(f, "col({})", self.name())
     }
 }
 
@@ -1492,6 +1496,7 @@ impl Expr {
         }
     }
 
+    /// Returns the expression as SQL using PostgreSQL's dialect.
     pub fn to_sql(&self) -> Option<String> {
         fn to_sql_inner<W: Write>(expr: &Expr, buffer: &mut W) -> io::Result<()> {
             match expr {
diff --git a/src/daft-dsl/src/python.rs b/src/daft-dsl/src/python.rs
index f8fe7da167..dfe47e42e5 100644
--- a/src/daft-dsl/src/python.rs
+++ b/src/daft-dsl/src/python.rs
@@ -23,7 +23,7 @@ use serde::{Deserialize, Serialize};
 
 use crate::{
     expr::{Expr, WindowExpr},
-    ExprRef, LiteralValue,
+    ExprRef, LiteralValue, Operator,
 };
 
 #[pyfunction]
@@ -388,49 +388,47 @@ impl PyExpr {
     }
 
     pub fn __add__(&self, other: &Self) -> PyResult<Self> {
-        Ok(crate::binary_op(crate::Operator::Plus, self.into(), other.expr.clone()).into())
+        Ok(crate::binary_op(Operator::Plus, self.into(), other.expr.clone()).into())
     }
+
     pub fn __sub__(&self, other: &Self) -> PyResult<Self> {
-        Ok(crate::binary_op(crate::Operator::Minus, self.into(), other.expr.clone()).into())
+        Ok(crate::binary_op(Operator::Minus, self.into(), other.expr.clone()).into())
     }
+
     pub fn __mul__(&self, other: &Self) -> PyResult<Self> {
-        Ok(crate::binary_op(crate::Operator::Multiply, self.into(), other.expr.clone()).into())
+        Ok(crate::binary_op(Operator::Multiply, self.into(), other.expr.clone()).into())
     }
+
     pub fn __floordiv__(&self, other: &Self) -> PyResult<Self> {
-        Ok(crate::binary_op(
-            crate::Operator::FloorDivide,
-            self.into(),
-            other.expr.clone(),
-        )
-        .into())
+        Ok(crate::binary_op(Operator::FloorDivide, self.into(), other.expr.clone()).into())
     }
 
     pub fn __truediv__(&self, other: &Self) -> PyResult<Self> {
-        Ok(crate::binary_op(crate::Operator::TrueDivide, self.into(), other.expr.clone()).into())
+        Ok(crate::binary_op(Operator::TrueDivide, self.into(), other.expr.clone()).into())
     }
 
     pub fn __mod__(&self, other: &Self) -> PyResult<Self> {
-        Ok(crate::binary_op(crate::Operator::Modulus, self.into(), other.expr.clone()).into())
+        Ok(crate::binary_op(Operator::Modulus, self.into(), other.expr.clone()).into())
     }
 
     pub fn __and__(&self, other: &Self) -> PyResult<Self> {
-        Ok(crate::binary_op(crate::Operator::And, self.into(), other.expr.clone()).into())
+        Ok(crate::binary_op(Operator::And, self.into(), other.expr.clone()).into())
     }
 
     pub fn __or__(&self, other: &Self) -> PyResult<Self> {
-        Ok(crate::binary_op(crate::Operator::Or, self.into(), other.expr.clone()).into())
+        Ok(crate::binary_op(Operator::Or, self.into(), other.expr.clone()).into())
     }
 
     pub fn __xor__(&self, other: &Self) -> PyResult<Self> {
-        Ok(crate::binary_op(crate::Operator::Xor, self.into(), other.expr.clone()).into())
+        Ok(crate::binary_op(Operator::Xor, self.into(), other.expr.clone()).into())
     }
 
     pub fn __lshift__(&self, other: &Self) -> PyResult<Self> {
-        Ok(crate::binary_op(crate::Operator::ShiftLeft, self.into(), other.expr.clone()).into())
+        Ok(crate::binary_op(Operator::ShiftLeft, self.into(), other.expr.clone()).into())
     }
 
     pub fn __rshift__(&self, other: &Self) -> PyResult<Self> {
-        Ok(crate::binary_op(crate::Operator::ShiftRight, self.into(), other.expr.clone()).into())
+        Ok(crate::binary_op(Operator::ShiftRight, self.into(), other.expr.clone()).into())
     }
 
     pub fn __richcmp__(&self, other: &Self, op: CompareOp) -> PyResult<Self> {
@@ -462,7 +460,7 @@ impl PyExpr {
     }
 
     pub fn eq_null_safe(&self, other: &Self) -> PyResult<Self> {
-        Ok(crate::binary_op(crate::Operator::EqNullSafe, self.into(), other.into()).into())
+        Ok(crate::binary_op(Operator::EqNullSafe, self.into(), other.into()).into())
     }
 
     pub fn is_in(&self, other: Vec<Self>) -> PyResult<Self> {
diff --git a/tests/io/test_iceberg_pushdowns.py b/tests/io/test_iceberg_pushdowns.py
new file mode 100644
index 0000000000..ae3ad282c9
--- /dev/null
+++ b/tests/io/test_iceberg_pushdowns.py
@@ -0,0 +1,65 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+from daft.daft import Pushdowns as PyPushdowns
+from daft.expressions import col, lit
+from daft.io.pushdowns import Expr, Reference, Term
+from daft.logical.schema import Schema
+
+if TYPE_CHECKING:
+    from daft.expressions import Expression
+
+
+def _term(expr: Expression, schema: Schema | None = None) -> Term:
+    return PyPushdowns._to_term(expr._expr, schema._schema if schema else None)
+
+
+def test_iceberg_predicates():
+    """Test we can translate each one of the Iceberg predicates.
+
+    References:
+        - https://iceberg.apache.org/javadoc/1.0.0/index.html?org/apache/iceberg/expressions/Expressions.html
+        - https://py.iceberg.apache.org/reference/pyiceberg/expressions/
+    """
+    assert _term(~col("a")) == Expr("not", Reference("a"))
+
+    assert _term(lit(True) & lit(False)) == Expr("and", True, False)
+    assert _term(lit(True) | lit(False)) == Expr("or", True, False)
+
+    assert _term(lit(1) == lit(2)) == Expr("=", 1, 2)
+    assert _term(lit(1) != lit(2)) == Expr("!=", 1, 2)
+    assert _term(lit(1) < lit(2)) == Expr("<", 1, 2)
+    assert _term(lit(1) <= lit(2)) == Expr("<=", 1, 2)
+    assert _term(lit(1) > lit(2)) == Expr(">", 1, 2)
+    assert _term(lit(1) >= lit(2)) == Expr(">=", 1, 2)
+
+    assert _term(col("a").is_null()) == Expr("is_null", Reference("a"))
+    assert _term(col("a").not_null()) == Expr("not_null", Reference("a"))
+
+    assert _term(col("a").float.is_nan()) == Expr("is_nan", Reference("a"))
+    assert _term(col("a").float.not_nan()) == Expr("not_nan", Reference("a"))
+
+    items = [1, 2, 3]
+    assert _term(col("a").is_in(items)) == Expr("is_in", Reference("a"), Expr("list", 1, 2, 3))
+    assert _term(~(col("a").is_in(items))) == Expr("not", Expr("is_in", Reference("a"), Expr("list", 1, 2, 3)))
+
+    assert _term(col("a").str.startswith("xyz")) == Expr("startswith", Reference("a"), "xyz")
+    assert _term(~(col("a").str.startswith("xyz"))) == Expr("not", Expr("startswith", Reference("a"), "xyz"))
+
+
+def test_iceberg_transforms():
+    """Test we can translate each one of the Iceberg partition expressions.
+
+    References:
+        - https://iceberg.apache.org/javadoc/1.0.0/index.html?org/apache/iceberg/transforms/Transforms.html
+        - https://py.iceberg.apache.org/reference/pyiceberg/transforms/
+    """
+    assert _term(col("a").partitioning.years()) == Expr("years", Reference("a"))
+    assert _term(col("a").partitioning.months()) == Expr("months", Reference("a"))
+    assert _term(col("a").partitioning.days()) == Expr("days", Reference("a"))
+    assert _term(col("a").partitioning.hours()) == Expr("hours", Reference("a"))
+
+    # need args from functions
+    assert _term(col("a").partitioning.iceberg_bucket(16)) == Expr("iceberg_bucket", Reference("a"), 16)
+    assert _term(col("a").partitioning.iceberg_truncate(10)) == Expr("iceberg_truncate", Reference("a"), 10)
diff --git a/tests/io/test_pushdowns.py b/tests/io/test_pushdowns.py
new file mode 100644
index 0000000000..aef8cee050
--- /dev/null
+++ b/tests/io/test_pushdowns.py
@@ -0,0 +1,255 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+from daft.daft import Pushdowns as PyPushdowns
+from daft.expressions import col, lit
+from daft.io.pushdowns import Expr, Literal, Reference, Term
+from daft.io.scan import ScanPushdowns
+from daft.logical.schema import DataType as dt
+from daft.logical.schema import Schema
+
+if TYPE_CHECKING:
+    from daft.expressions import Expression
+
+
+def _term(expr: Expression, schema: Schema | None = None) -> Term:
+    return PyPushdowns._to_term(expr._expr, schema._schema if schema else None)
+
+
+def test_expr_getitem():
+    # positional arguments
+    expr = Expr("f", 1, 2, 3)
+    assert expr[0] == Literal(1)
+    assert expr[1] == Literal(2)
+    assert expr[2] == Literal(3)
+
+    # named arguments
+    expr = Expr("g", x=10, y=20)
+    assert expr["x"] == Literal(10)
+    assert expr["y"] == Literal(20)
+
+    # mixed arguments
+    expr = Expr("h", 100, 200, name="test")
+    assert expr[0] == Literal(100)
+    assert expr[1] == Literal(200)
+    assert expr["name"] == Literal("test")
+
+    # nested expressions
+    expr = Expr("nested", Expr("inner", 5))
+    assert expr[0] == Expr("inner", 5)
+    assert expr[0][0] == Literal(5)
+
+
+###
+# Sanity Translation Tests
+###
+
+
+def test_pyexpr_lit():
+    # null/nil/none
+    assert _term(lit(None)) == Literal(None)
+
+    # bool
+    assert _term(lit(True)) == Literal(True)
+    assert _term(lit(False)) == Literal(False)
+
+    # int
+    assert _term(lit(0)) == Literal(0)
+    assert _term(lit(1)) == Literal(1)
+    assert _term(lit(-1)) == Literal(-1)
+
+    # float
+    assert _term(lit(0.0)) == Literal(0)
+    assert _term(lit(1.0)) == Literal(1.0)
+    assert _term(lit(-1.0)) == Literal(-1.0)
+
+    # string
+    assert _term(lit("hello")) == Literal("hello")
+    assert _term(lit("🤠🤠")) == Literal("🤠🤠")
+
+
+def test_pyexpr_col():
+    assert _term(col("a")) == Reference("a")
+
+
+def test_pyexpr_alias():
+    assert _term(col("a").alias("xyz")) == Expr("alias", "xyz", Reference("a"))
+    assert _term(lit(42).alias("answer")) == Expr("alias", "answer", 42)
+
+
+def test_pyexpr_not():
+    assert _term(~lit(42)) == Expr("not", 42)
+
+
+def test_pyexpr_predicates():
+    # logical operators
+    assert _term(lit(True) & lit(False)) == Expr("and", True, False)
+    assert _term(lit(True) | lit(False)) == Expr("or", True, False)
+
+    # comparisons
+    assert _term(lit(1) == lit(2)) == Expr("=", 1, 2)
+    assert _term(lit(1) != lit(2)) == Expr("!=", 1, 2)
+    assert _term(lit(1) < lit(2)) == Expr("<", 1, 2)
+    assert _term(lit(1) <= lit(2)) == Expr("<=", 1, 2)
+    assert _term(lit(1) > lit(2)) == Expr(">", 1, 2)
+    assert _term(lit(1) >= lit(2)) == Expr(">=", 1, 2)
+
+
+###
+# Pushdowns Translations Tests
+###
+
+
+def test_column_pushdown_binding():
+    schema = Schema._from_pydict(
+        {
+            "a": dt.bool(),  # 0
+            "b": dt.bool(),  # 1
+            "c": dt.bool(),  # 2
+        }
+    )
+    pypushdowns = PyPushdowns(columns=["c", "b", "a"])  # !! reverse order on purpose !!
+    pushdowns = ScanPushdowns._from_pypushdowns(pypushdowns, schema)
+
+    assert pushdowns.projections
+    assert pushdowns.projections[0] == Reference("c", 2)
+    assert pushdowns.projections[1] == Reference("b", 1)
+    assert pushdowns.projections[2] == Reference("a", 0)
+    assert pushdowns.limit is None
+    assert pushdowns.predicate is None
+
+
+def test_simple_predicate_pushdown():
+    schema = Schema._from_pydict(
+        {
+            "a": dt.bool(),  # 0
+        }
+    )
+    predicate = col("a") == lit(1)  # (= a 1)
+    pypushdowns = PyPushdowns(filters=predicate._expr)
+    pushdowns = ScanPushdowns._from_pypushdowns(pypushdowns, schema)
+
+    assert pushdowns.projections is None
+    assert pushdowns.predicate == Expr("=", Reference("a", 0), 1)
+    assert pushdowns.limit is None
+
+
+def test_complex_predicate_pushdown():
+    schema = Schema._from_pydict(
+        {
+            "a": dt.bool(),  # 0
+            "b": dt.bool(),  # 1
+            "c": dt.bool(),  # 2
+        }
+    )
+    predicate = col("a") == (col("b") + col("c"))  # (= a (+ b c))
+    pypushdowns = PyPushdowns(filters=predicate._expr)
+    pushdowns = ScanPushdowns._from_pypushdowns(pypushdowns, schema)
+
+    assert pushdowns.projections is None
+    assert pushdowns.predicate == Expr("=", Reference("a", 0), Expr("+", Reference("b", 1), Reference("c", 2)))
+    assert pushdowns.limit is None
+
+
+def test_limit_pushdown():
+    schema = Schema._from_pydict({"a": dt.bool()})
+    pypushdowns = PyPushdowns(limit=1738)
+    pushdowns = ScanPushdowns._from_pypushdowns(pypushdowns, schema)
+
+    assert pushdowns.projections is None
+    assert pushdowns.predicate is None
+    assert pushdowns.limit == 1738
+
+
+def test_simple_partition_pushdown():
+    schema = Schema._from_pydict(
+        {
+            "a": dt.bool(),  # 0
+        }
+    )
+    predicate = col("a") == lit(1)  # (= a 1)
+    pypushdowns = PyPushdowns(partition_filters=predicate._expr)
+    pushdowns = ScanPushdowns._from_pypushdowns(pypushdowns, schema)
+
+    assert pushdowns.projections is None
+    assert pushdowns.predicate == Expr("=", Reference("a", 0), 1)
+    assert pushdowns.limit is None
+
+
+def test_composite_partition_pushdown():
+    schema = Schema._from_pydict(
+        {
+            "a": dt.bool(),  # 0
+            "b": dt.bool(),  # 1
+        }
+    )
+
+    # rust side has these split
+    filters = col("a") == lit(1)  # (= a 1)
+    partition_filters = col("b") > lit(2)  # (> b 2)
+    pypushdowns = PyPushdowns(filters=filters._expr, partition_filters=partition_filters._expr)
+    pushdowns = ScanPushdowns._from_pypushdowns(pypushdowns, schema)
+
+    # translation should combine them.
+    p1 = Expr("=", Reference("a", 0), 1)
+    p2 = Expr(">", Reference("b", 1), 2)
+
+    assert pushdowns.projections is None
+    assert pushdowns.predicate == Expr("and", p1, p2)
+    assert pushdowns.limit is None
+
+
+###
+# LispyVisitor Tests
+###
+
+
+def test_print_literal():
+    # str
+    assert str(Literal("hello")) == '"hello"'
+    assert str(Literal("🤠🤠")) == '"🤠🤠"'
+
+    # int
+    assert str(Literal(1)) == "1"
+    assert str(Literal(0)) == "0"
+    assert str(Literal(-1)) == "-1"
+
+    # float
+    assert str(Literal(2.0)) == "2.00"
+    assert str(Literal(3.14159)) == "3.14"
+
+    # bool
+    assert str(Literal(True)) == "true"
+    assert str(Literal(False)) == "false"
+
+    # nil/null/none
+    assert str(Literal(None)) == "null"
+
+
+def test_print_reference():
+    assert str(Reference("a")) == "a"
+    assert str(Reference("a")) == "a"
+
+
+def test_print_expr():
+    # zero arguments
+    assert str(Expr("f")) == "(f)"
+
+    # positional arguments
+    assert str(Expr("f", 42)) == "(f 42)"
+    assert str(Expr("f", "hello")) == '(f "hello")'
+    assert str(Expr("f", 1, 2.5, "test")) == '(f 1 2.50 "test")'
+
+    # named arguments
+    assert str(Expr("f", value=True)) == "(f value::true)"
+    assert str(Expr("f", x=10, y=20.5, label="data")) == '(f x::10 y::20.50 label::"data")'
+
+    # mixed arguments
+    assert str(Expr("f", 100, False, name="example", flag=True)) == '(f 100 false name::"example" flag::true)'
+
+    # nested expressions
+    assert str(Expr("f", Expr("g", 5))) == "(f (g 5))"
+
+    # literal objects
+    assert str(Expr("f", Literal(42), keyword=Literal("value"))) == '(f 42 keyword::"value")'


Reviews:
user: login: rchowell
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T00:19:24Z
user: login: mcember
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-18T14:44:27Z
user: login: mcember
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-18T14:46:09Z
user: login: mcember
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-18T14:48:20Z
user: login: mcember
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-18T14:51:18Z
user: login: mcember
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-18T14:52:17Z
user: login: mcember
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-18T14:52:54Z
user: login: mcember
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-18T14:53:33Z
user: login: mcember
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-18T14:54:55Z
user: login: rchowell
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T16:34:28Z
user: login: rchowell
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T16:38:21Z
user: login: rchowell
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T16:45:17Z
user: login: rchowell
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T16:59:30Z
user: login: mcember
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-21T13:33:32Z
user: login: mcember
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-21T13:44:09Z
user: login: desmondcheongzx
body: Very nice! Implementation-wise it makes sense to me, but it'd be nice to have more in the way of examples (and tests) of end-to-end construction and usage of the pushdowns
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-21T22:34:46Z
user: login: rchowell
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T22:38:05Z
user: login: ccmao1130
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-22T01:39:34Z
user: login: desmondcheongzx
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-22T21:34:13Z
user: login: rchowell
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-22T21:35:55Z
user: login: rchowell
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-24T15:16:01Z

Review comments:
diff_hunk: @@ -0,0 +1,182 @@
+from __future__ import annotations
+
+from abc import ABC
+
+
+from dataclasses import dataclass
+from functools import singledispatchmethod
+from typing import Generic, TypeVar
+
+Value = str | int | float | bool
+R = TypeVar("R")
+C = TypeVar("C")
+
+
+@dataclass(frozen=True)
+class Pushdowns(ABC):
+    """Marker interface for pushdown objects."""
+    pass
path: daft/io/sexp.py
user: login: rchowell
body: ignore, came along when I moved out sexp 
created_at: 2025-04-17T00:19:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,230 @@
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from dataclasses import dataclass
+from functools import singledispatchmethod
+from typing import Generic, Sequence, TypeVar, Union
+
+Value = Union[str, int, float, bool]
+R = TypeVar("R")
+C = TypeVar("C")
+
+
+@dataclass(frozen=True)
+class Pushdowns(ABC):
path: daft/io/pushdowns.py
user: login: mcember
body: Out of curiosity, any reason why you're using an ABC instead of a Protocol in this case when it's just used as a marker class?
created_at: 2025-04-18T14:44:27Z
author_association: NONE
diff_hunk: @@ -0,0 +1,230 @@
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from dataclasses import dataclass
+from functools import singledispatchmethod
+from typing import Generic, Sequence, TypeVar, Union
+
+Value = Union[str, int, float, bool]
+R = TypeVar("R")
+C = TypeVar("C")
+
+
+@dataclass(frozen=True)
+class Pushdowns(ABC):
+    """Base class for pushdown information."""
+
+    pass
+
+
+@dataclass(frozen=True)
+class Term(ABC):
+    """Term is the base class for representing expressions in pushdowns."""
+
+    def __str__(self) -> str:
+        return LispyVisitor().visit(self, None)
+
+
+@dataclass(frozen=True)
+class Reference(Term):
+    """Reference to a field in some schema, this is not bound to any schema."""
+
+    name: str
path: daft/io/pushdowns.py
user: login: mcember
body: this would only support top level column, not a nested field reference? I'm assuming nested references wouldn't be a requirement? You might make this more extensible by calling this "path" and for now just taking a non delimited string which must refer to a column
created_at: 2025-04-18T14:46:09Z
author_association: NONE
diff_hunk: @@ -0,0 +1,230 @@
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from dataclasses import dataclass
+from functools import singledispatchmethod
+from typing import Generic, Sequence, TypeVar, Union
+
+Value = Union[str, int, float, bool]
+R = TypeVar("R")
+C = TypeVar("C")
+
+
+@dataclass(frozen=True)
+class Pushdowns(ABC):
+    """Base class for pushdown information."""
+
+    pass
+
+
+@dataclass(frozen=True)
+class Term(ABC):
+    """Term is the base class for representing expressions in pushdowns."""
+
+    def __str__(self) -> str:
+        return LispyVisitor().visit(self, None)
+
+
+@dataclass(frozen=True)
+class Reference(Term):
+    """Reference to a field in some schema, this is not bound to any schema."""
+
+    name: str
+
+
+@dataclass(frozen=True)
+class Literal(Term):
path: daft/io/pushdowns.py
user: login: mcember
body: I think we do, but just sanity checking: do we need the extra level of nesting from Literal into Value? Or could we flatten it so that Value adds `None` to the type union and it is used wherever literal is?
created_at: 2025-04-18T14:48:19Z
author_association: NONE
diff_hunk: @@ -0,0 +1,230 @@
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from dataclasses import dataclass
+from functools import singledispatchmethod
+from typing import Generic, Sequence, TypeVar, Union
+
+Value = Union[str, int, float, bool]
+R = TypeVar("R")
+C = TypeVar("C")
+
+
+@dataclass(frozen=True)
+class Pushdowns(ABC):
+    """Base class for pushdown information."""
+
+    pass
+
+
+@dataclass(frozen=True)
+class Term(ABC):
+    """Term is the base class for representing expressions in pushdowns."""
+
+    def __str__(self) -> str:
+        return LispyVisitor().visit(self, None)
+
+
+@dataclass(frozen=True)
+class Reference(Term):
+    """Reference to a field in some schema, this is not bound to any schema."""
+
+    name: str
+
+
+@dataclass(frozen=True)
+class Literal(Term):
+    """Literal value which does not currently carry its daft type information.
+
+    The representation of types and the full-fidelity of values complicates both
+    the s-expression encoding and the end user consumption. The goal here is for
+    simple consumption of daft's rust expressions from a python context. We may
+    iterate on this design's complexity as more advanced use-cases emerge.
+
+    Example:
+    >>> v1 = Literal("hello")
+    >>> v2 = Literal(1)
+    >>> v3 = Literal(2.0)
+    >>> v4 = Literal(False)
+    """
+
+    value: Value | None
+
+
+@dataclass(frozen=True)
+class Expr(Term):
+    """Expr is a generic expression form which has the procedure symbol and its args.
+
+    This representation was chosen because it's the simplest while avoiding the complexity
+    of a more 'pure' concatenation style or introducing additional Term variants.
+    This is a trade-off where lower representation complexity requires slightly more creativity
+    in representing the various expression forms, but named arguments ease consumption.
+
+    Exprs follow the common rule that positional come before named.
+    This is enforced on instantiation since we use python args and kwargs.
+
+    Example:
+    >>> Expr("f", 42)  # (f 42)
+    >>> Expr("f", "hello")  # (f "hello")
+    >>> Expr("f", 1, 2.5, "test")  # (f 1 2.50 "test")
+    >>> Expr("f", value=True)  # (f value::true)
+    >>> Expr("f", x=10, y=20.5)  # (f x::10 y::20.50)
+    """
+
+    proc: str
+    args: list[Arg]
+
+    def __init__(self, proc: str, *args: Term | Value, **kwargs: Term | Value):
+        self_proc = proc
+        self_args = []
+        for arg in args:
+            term = arg if isinstance(arg, Term) else Literal(arg)
+            self_args.append(Arg(term))
+        for label, arg in kwargs.items():
path: daft/io/pushdowns.py
user: login: mcember
body: Instead of accepting labelled args through kwargs, should you explicitly model that as List[str, Arg] to preserve order?
created_at: 2025-04-18T14:51:17Z
author_association: NONE
diff_hunk: @@ -0,0 +1,230 @@
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from dataclasses import dataclass
+from functools import singledispatchmethod
+from typing import Generic, Sequence, TypeVar, Union
+
+Value = Union[str, int, float, bool]
+R = TypeVar("R")
+C = TypeVar("C")
+
+
+@dataclass(frozen=True)
+class Pushdowns(ABC):
+    """Base class for pushdown information."""
+
+    pass
+
+
+@dataclass(frozen=True)
+class Term(ABC):
+    """Term is the base class for representing expressions in pushdowns."""
+
+    def __str__(self) -> str:
+        return LispyVisitor().visit(self, None)
+
+
+@dataclass(frozen=True)
+class Reference(Term):
+    """Reference to a field in some schema, this is not bound to any schema."""
+
+    name: str
+
+
+@dataclass(frozen=True)
+class Literal(Term):
+    """Literal value which does not currently carry its daft type information.
+
+    The representation of types and the full-fidelity of values complicates both
+    the s-expression encoding and the end user consumption. The goal here is for
+    simple consumption of daft's rust expressions from a python context. We may
+    iterate on this design's complexity as more advanced use-cases emerge.
+
+    Example:
+    >>> v1 = Literal("hello")
+    >>> v2 = Literal(1)
+    >>> v3 = Literal(2.0)
+    >>> v4 = Literal(False)
+    """
+
+    value: Value | None
+
+
+@dataclass(frozen=True)
+class Expr(Term):
+    """Expr is a generic expression form which has the procedure symbol and its args.
+
+    This representation was chosen because it's the simplest while avoiding the complexity
+    of a more 'pure' concatenation style or introducing additional Term variants.
+    This is a trade-off where lower representation complexity requires slightly more creativity
+    in representing the various expression forms, but named arguments ease consumption.
+
+    Exprs follow the common rule that positional come before named.
+    This is enforced on instantiation since we use python args and kwargs.
+
+    Example:
+    >>> Expr("f", 42)  # (f 42)
+    >>> Expr("f", "hello")  # (f "hello")
+    >>> Expr("f", 1, 2.5, "test")  # (f 1 2.50 "test")
+    >>> Expr("f", value=True)  # (f value::true)
+    >>> Expr("f", x=10, y=20.5)  # (f x::10 y::20.50)
+    """
+
+    proc: str
+    args: list[Arg]
+
+    def __init__(self, proc: str, *args: Term | Value, **kwargs: Term | Value):
+        self_proc = proc
+        self_args = []
+        for arg in args:
+            term = arg if isinstance(arg, Term) else Literal(arg)
+            self_args.append(Arg(term))
+        for label, arg in kwargs.items():
+            term = arg if isinstance(arg, Term) else Literal(arg)
+            self_args.append(Arg(term, label))
+        object.__setattr__(self, "proc", self_proc)
+        object.__setattr__(self, "args", self_args)
+
+    def __str__(self) -> str:
+        return LispyVisitor().visit(self, None)
+
+    def __len__(self) -> int:
+        return len(self.args)
+
+    def __getitem__(self, key: int | str | slice) -> Term | Sequence[Term] | None:
+        """Get an argument by index, name, or slice."""
+        if isinstance(key, int):
+            return self.args[key].term
+        elif isinstance(key, str):
+            return next((arg.term for arg in self.args if arg.label == key), None)
+        elif isinstance(key, slice):
+            return [arg.term for arg in self.args[key]]
+        else:
+            raise TypeError("Expected key to be int, str, or slice.")
+
+
+@dataclass(frozen=True)
+class Arg:
+    """Arg is just an s-expression with optional label, notably does not inherit from Term."""
+
+    term: Term
+    label: str | None = None
+
+    def __str__(self) -> str:
+        return LispyVisitor()._arg(self, None)
+
+
+class TermVisitor(ABC, Generic[C, R]):
+    """TermrVisitor uses the @singledispatchmethod for a class-based visitor.
path: daft/io/pushdowns.py
user: login: mcember
body: s/Termr/Term
created_at: 2025-04-18T14:52:17Z
author_association: NONE
diff_hunk: @@ -0,0 +1,230 @@
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from dataclasses import dataclass
+from functools import singledispatchmethod
+from typing import Generic, Sequence, TypeVar, Union
+
+Value = Union[str, int, float, bool]
+R = TypeVar("R")
+C = TypeVar("C")
+
+
+@dataclass(frozen=True)
+class Pushdowns(ABC):
+    """Base class for pushdown information."""
+
+    pass
+
+
+@dataclass(frozen=True)
+class Term(ABC):
+    """Term is the base class for representing expressions in pushdowns."""
+
+    def __str__(self) -> str:
+        return LispyVisitor().visit(self, None)
+
+
+@dataclass(frozen=True)
+class Reference(Term):
+    """Reference to a field in some schema, this is not bound to any schema."""
+
+    name: str
+
+
+@dataclass(frozen=True)
+class Literal(Term):
+    """Literal value which does not currently carry its daft type information.
+
+    The representation of types and the full-fidelity of values complicates both
+    the s-expression encoding and the end user consumption. The goal here is for
+    simple consumption of daft's rust expressions from a python context. We may
+    iterate on this design's complexity as more advanced use-cases emerge.
+
+    Example:
+    >>> v1 = Literal("hello")
+    >>> v2 = Literal(1)
+    >>> v3 = Literal(2.0)
+    >>> v4 = Literal(False)
+    """
+
+    value: Value | None
+
+
+@dataclass(frozen=True)
+class Expr(Term):
+    """Expr is a generic expression form which has the procedure symbol and its args.
+
+    This representation was chosen because it's the simplest while avoiding the complexity
+    of a more 'pure' concatenation style or introducing additional Term variants.
+    This is a trade-off where lower representation complexity requires slightly more creativity
+    in representing the various expression forms, but named arguments ease consumption.
+
+    Exprs follow the common rule that positional come before named.
+    This is enforced on instantiation since we use python args and kwargs.
+
+    Example:
+    >>> Expr("f", 42)  # (f 42)
+    >>> Expr("f", "hello")  # (f "hello")
+    >>> Expr("f", 1, 2.5, "test")  # (f 1 2.50 "test")
+    >>> Expr("f", value=True)  # (f value::true)
+    >>> Expr("f", x=10, y=20.5)  # (f x::10 y::20.50)
+    """
+
+    proc: str
+    args: list[Arg]
+
+    def __init__(self, proc: str, *args: Term | Value, **kwargs: Term | Value):
+        self_proc = proc
+        self_args = []
+        for arg in args:
+            term = arg if isinstance(arg, Term) else Literal(arg)
+            self_args.append(Arg(term))
+        for label, arg in kwargs.items():
+            term = arg if isinstance(arg, Term) else Literal(arg)
+            self_args.append(Arg(term, label))
+        object.__setattr__(self, "proc", self_proc)
+        object.__setattr__(self, "args", self_args)
+
+    def __str__(self) -> str:
+        return LispyVisitor().visit(self, None)
+
+    def __len__(self) -> int:
+        return len(self.args)
+
+    def __getitem__(self, key: int | str | slice) -> Term | Sequence[Term] | None:
+        """Get an argument by index, name, or slice."""
+        if isinstance(key, int):
+            return self.args[key].term
+        elif isinstance(key, str):
+            return next((arg.term for arg in self.args if arg.label == key), None)
+        elif isinstance(key, slice):
+            return [arg.term for arg in self.args[key]]
+        else:
+            raise TypeError("Expected key to be int, str, or slice.")
+
+
+@dataclass(frozen=True)
+class Arg:
+    """Arg is just an s-expression with optional label, notably does not inherit from Term."""
+
+    term: Term
+    label: str | None = None
+
+    def __str__(self) -> str:
+        return LispyVisitor()._arg(self, None)
+
+
+class TermVisitor(ABC, Generic[C, R]):
+    """TermrVisitor uses the @singledispatchmethod for a class-based visitor.
+
+    Note that we are not using the typical "accept" method on an term variant
+    for dispatching because the @singledispatchmethod handles this for us.
+    There is no need to add accept methods to each variant which simplifies
+    both the term tree and the visitor implementations.
+
+    The type `R` represents the return type, and `C` represents the context.
+    The context parameter is useful when passing state which is scoped when
+    performing a visitor traversal (fold). For non-scoped state, you can just
+    add normal instance properties.
+    """
+
+    @singledispatchmethod
+    def visit(self, term: Term, context: C) -> R:
+        raise NotImplementedError(f"No visit method for type {type(term)}")
+
+    @visit.register
+    def _(self, term: Reference, context: C) -> R:
+        return self.visit_reference(term, context)
+
+    @visit.register
+    def _(self, term: Literal, context: C) -> R:
+        return self.visit_literal(term, context)
+
+    @visit.register
+    def _(self, term: Expr, context: C) -> R:
+        return self.visit_expr(term, context)
+
+    @abstractmethod
+    def visit_reference(self, term: Reference, context: C) -> R: ...
+
+    @abstractmethod
+    def visit_literal(self, term: Literal, context: C) -> R: ...
+
+    @abstractmethod
+    def visit_expr(self, term: Expr, context: C) -> R: ...
+
+
+class LispyVisitor(TermVisitor[None, str]):
+    """LispyVisitor is an example visitor implementation for printing s-expressions.
+
+    This can be implemented *much* more concisely directly in the __str__ .. but
+    this is an exercise and tutorial to show off visitor usage and atterns. We
path: daft/io/pushdowns.py
user: login: mcember
body: s/atterns/patterns
created_at: 2025-04-18T14:52:54Z
author_association: NONE
diff_hunk: @@ -0,0 +1,230 @@
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from dataclasses import dataclass
+from functools import singledispatchmethod
+from typing import Generic, Sequence, TypeVar, Union
+
+Value = Union[str, int, float, bool]
+R = TypeVar("R")
+C = TypeVar("C")
+
+
+@dataclass(frozen=True)
+class Pushdowns(ABC):
+    """Base class for pushdown information."""
+
+    pass
+
+
+@dataclass(frozen=True)
+class Term(ABC):
+    """Term is the base class for representing expressions in pushdowns."""
+
+    def __str__(self) -> str:
+        return LispyVisitor().visit(self, None)
+
+
+@dataclass(frozen=True)
+class Reference(Term):
+    """Reference to a field in some schema, this is not bound to any schema."""
+
+    name: str
+
+
+@dataclass(frozen=True)
+class Literal(Term):
+    """Literal value which does not currently carry its daft type information.
+
+    The representation of types and the full-fidelity of values complicates both
+    the s-expression encoding and the end user consumption. The goal here is for
+    simple consumption of daft's rust expressions from a python context. We may
+    iterate on this design's complexity as more advanced use-cases emerge.
+
+    Example:
+    >>> v1 = Literal("hello")
+    >>> v2 = Literal(1)
+    >>> v3 = Literal(2.0)
+    >>> v4 = Literal(False)
+    """
+
+    value: Value | None
+
+
+@dataclass(frozen=True)
+class Expr(Term):
+    """Expr is a generic expression form which has the procedure symbol and its args.
+
+    This representation was chosen because it's the simplest while avoiding the complexity
+    of a more 'pure' concatenation style or introducing additional Term variants.
+    This is a trade-off where lower representation complexity requires slightly more creativity
+    in representing the various expression forms, but named arguments ease consumption.
+
+    Exprs follow the common rule that positional come before named.
+    This is enforced on instantiation since we use python args and kwargs.
+
+    Example:
+    >>> Expr("f", 42)  # (f 42)
+    >>> Expr("f", "hello")  # (f "hello")
+    >>> Expr("f", 1, 2.5, "test")  # (f 1 2.50 "test")
+    >>> Expr("f", value=True)  # (f value::true)
+    >>> Expr("f", x=10, y=20.5)  # (f x::10 y::20.50)
+    """
+
+    proc: str
+    args: list[Arg]
+
+    def __init__(self, proc: str, *args: Term | Value, **kwargs: Term | Value):
+        self_proc = proc
+        self_args = []
+        for arg in args:
+            term = arg if isinstance(arg, Term) else Literal(arg)
+            self_args.append(Arg(term))
+        for label, arg in kwargs.items():
+            term = arg if isinstance(arg, Term) else Literal(arg)
+            self_args.append(Arg(term, label))
+        object.__setattr__(self, "proc", self_proc)
+        object.__setattr__(self, "args", self_args)
+
+    def __str__(self) -> str:
+        return LispyVisitor().visit(self, None)
+
+    def __len__(self) -> int:
+        return len(self.args)
+
+    def __getitem__(self, key: int | str | slice) -> Term | Sequence[Term] | None:
+        """Get an argument by index, name, or slice."""
+        if isinstance(key, int):
+            return self.args[key].term
+        elif isinstance(key, str):
+            return next((arg.term for arg in self.args if arg.label == key), None)
+        elif isinstance(key, slice):
+            return [arg.term for arg in self.args[key]]
+        else:
+            raise TypeError("Expected key to be int, str, or slice.")
+
+
+@dataclass(frozen=True)
+class Arg:
+    """Arg is just an s-expression with optional label, notably does not inherit from Term."""
+
+    term: Term
+    label: str | None = None
+
+    def __str__(self) -> str:
+        return LispyVisitor()._arg(self, None)
+
+
+class TermVisitor(ABC, Generic[C, R]):
+    """TermrVisitor uses the @singledispatchmethod for a class-based visitor.
+
+    Note that we are not using the typical "accept" method on an term variant
+    for dispatching because the @singledispatchmethod handles this for us.
+    There is no need to add accept methods to each variant which simplifies
+    both the term tree and the visitor implementations.
+
+    The type `R` represents the return type, and `C` represents the context.
+    The context parameter is useful when passing state which is scoped when
+    performing a visitor traversal (fold). For non-scoped state, you can just
+    add normal instance properties.
+    """
+
+    @singledispatchmethod
+    def visit(self, term: Term, context: C) -> R:
+        raise NotImplementedError(f"No visit method for type {type(term)}")
+
+    @visit.register
+    def _(self, term: Reference, context: C) -> R:
+        return self.visit_reference(term, context)
+
+    @visit.register
+    def _(self, term: Literal, context: C) -> R:
+        return self.visit_literal(term, context)
+
+    @visit.register
+    def _(self, term: Expr, context: C) -> R:
+        return self.visit_expr(term, context)
+
+    @abstractmethod
+    def visit_reference(self, term: Reference, context: C) -> R: ...
+
+    @abstractmethod
+    def visit_literal(self, term: Literal, context: C) -> R: ...
+
+    @abstractmethod
+    def visit_expr(self, term: Expr, context: C) -> R: ...
+
+
+class LispyVisitor(TermVisitor[None, str]):
path: daft/io/pushdowns.py
user: login: mcember
body: +1 to including example visitor as reference implementation
created_at: 2025-04-18T14:53:33Z
author_association: NONE
diff_hunk: @@ -0,0 +1,230 @@
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from dataclasses import dataclass
+from functools import singledispatchmethod
+from typing import Generic, Sequence, TypeVar, Union
+
+Value = Union[str, int, float, bool]
+R = TypeVar("R")
+C = TypeVar("C")
+
+
+@dataclass(frozen=True)
+class Pushdowns(ABC):
+    """Base class for pushdown information."""
+
+    pass
+
+
+@dataclass(frozen=True)
+class Term(ABC):
+    """Term is the base class for representing expressions in pushdowns."""
+
+    def __str__(self) -> str:
+        return LispyVisitor().visit(self, None)
+
+
+@dataclass(frozen=True)
+class Reference(Term):
+    """Reference to a field in some schema, this is not bound to any schema."""
+
+    name: str
+
+
+@dataclass(frozen=True)
+class Literal(Term):
+    """Literal value which does not currently carry its daft type information.
+
+    The representation of types and the full-fidelity of values complicates both
+    the s-expression encoding and the end user consumption. The goal here is for
+    simple consumption of daft's rust expressions from a python context. We may
+    iterate on this design's complexity as more advanced use-cases emerge.
+
+    Example:
+    >>> v1 = Literal("hello")
+    >>> v2 = Literal(1)
+    >>> v3 = Literal(2.0)
+    >>> v4 = Literal(False)
+    """
+
+    value: Value | None
+
+
+@dataclass(frozen=True)
+class Expr(Term):
+    """Expr is a generic expression form which has the procedure symbol and its args.
+
+    This representation was chosen because it's the simplest while avoiding the complexity
+    of a more 'pure' concatenation style or introducing additional Term variants.
+    This is a trade-off where lower representation complexity requires slightly more creativity
+    in representing the various expression forms, but named arguments ease consumption.
+
+    Exprs follow the common rule that positional come before named.
+    This is enforced on instantiation since we use python args and kwargs.
+
+    Example:
+    >>> Expr("f", 42)  # (f 42)
+    >>> Expr("f", "hello")  # (f "hello")
+    >>> Expr("f", 1, 2.5, "test")  # (f 1 2.50 "test")
+    >>> Expr("f", value=True)  # (f value::true)
+    >>> Expr("f", x=10, y=20.5)  # (f x::10 y::20.50)
+    """
+
+    proc: str
path: daft/io/pushdowns.py
user: login: mcember
body: Does it make sense to have an enum of all the possible proc types in expressions? Otherwise, how do users know the full set and ensure that any visitors they write are covering all cases? Note with enum you will still have the problem of additions to the enum potentially hitting reaching exception cases for code written before the addition
created_at: 2025-04-18T14:54:54Z
author_association: NONE
diff_hunk: @@ -0,0 +1,230 @@
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from dataclasses import dataclass
+from functools import singledispatchmethod
+from typing import Generic, Sequence, TypeVar, Union
+
+Value = Union[str, int, float, bool]
+R = TypeVar("R")
+C = TypeVar("C")
+
+
+@dataclass(frozen=True)
+class Pushdowns(ABC):
path: daft/io/pushdowns.py
user: login: rchowell
body: I originally was just calling this a marker, but I learned that combining `@dataclass(frozen=True)` on this base class defines the behavior that all subclasses must also be frozen dataclasses. I quite like this since pushdowns are sent between application layers, rust optimizer to python scan operators, so keeping them [simple](https://en.wikipedia.org/wiki/Value_object) felt right here. I don't foresee pushdown classes ever containing methods, just data holders.
created_at: 2025-04-18T16:34:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,230 @@
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from dataclasses import dataclass
+from functools import singledispatchmethod
+from typing import Generic, Sequence, TypeVar, Union
+
+Value = Union[str, int, float, bool]
+R = TypeVar("R")
+C = TypeVar("C")
+
+
+@dataclass(frozen=True)
+class Pushdowns(ABC):
+    """Base class for pushdown information."""
+
+    pass
+
+
+@dataclass(frozen=True)
+class Term(ABC):
+    """Term is the base class for representing expressions in pushdowns."""
+
+    def __str__(self) -> str:
+        return LispyVisitor().visit(self, None)
+
+
+@dataclass(frozen=True)
+class Reference(Term):
+    """Reference to a field in some schema, this is not bound to any schema."""
+
+    name: str
+
+
+@dataclass(frozen=True)
+class Literal(Term):
path: daft/io/pushdowns.py
user: login: rchowell
body: Good point, the extra layer leaves the door open for more complex literals, as well as including the Daft datatype. The original impl was more like,

```python
@dataclass(frozen=True)
class Literal(Term):
    value: Value
    dtype: DataType
```

But I wanted to avoid dealing with datatypes at the moment. I can see how this would be useful though, so if you think the datatype addition would help you at present, I will include it. 
created_at: 2025-04-18T16:38:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,230 @@
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from dataclasses import dataclass
+from functools import singledispatchmethod
+from typing import Generic, Sequence, TypeVar, Union
+
+Value = Union[str, int, float, bool]
+R = TypeVar("R")
+C = TypeVar("C")
+
+
+@dataclass(frozen=True)
+class Pushdowns(ABC):
+    """Base class for pushdown information."""
+
+    pass
+
+
+@dataclass(frozen=True)
+class Term(ABC):
+    """Term is the base class for representing expressions in pushdowns."""
+
+    def __str__(self) -> str:
+        return LispyVisitor().visit(self, None)
+
+
+@dataclass(frozen=True)
+class Reference(Term):
+    """Reference to a field in some schema, this is not bound to any schema."""
+
+    name: str
+
+
+@dataclass(frozen=True)
+class Literal(Term):
+    """Literal value which does not currently carry its daft type information.
+
+    The representation of types and the full-fidelity of values complicates both
+    the s-expression encoding and the end user consumption. The goal here is for
+    simple consumption of daft's rust expressions from a python context. We may
+    iterate on this design's complexity as more advanced use-cases emerge.
+
+    Example:
+    >>> v1 = Literal("hello")
+    >>> v2 = Literal(1)
+    >>> v3 = Literal(2.0)
+    >>> v4 = Literal(False)
+    """
+
+    value: Value | None
+
+
+@dataclass(frozen=True)
+class Expr(Term):
+    """Expr is a generic expression form which has the procedure symbol and its args.
+
+    This representation was chosen because it's the simplest while avoiding the complexity
+    of a more 'pure' concatenation style or introducing additional Term variants.
+    This is a trade-off where lower representation complexity requires slightly more creativity
+    in representing the various expression forms, but named arguments ease consumption.
+
+    Exprs follow the common rule that positional come before named.
+    This is enforced on instantiation since we use python args and kwargs.
+
+    Example:
+    >>> Expr("f", 42)  # (f 42)
+    >>> Expr("f", "hello")  # (f "hello")
+    >>> Expr("f", 1, 2.5, "test")  # (f 1 2.50 "test")
+    >>> Expr("f", value=True)  # (f value::true)
+    >>> Expr("f", x=10, y=20.5)  # (f x::10 y::20.50)
+    """
+
+    proc: str
+    args: list[Arg]
+
+    def __init__(self, proc: str, *args: Term | Value, **kwargs: Term | Value):
+        self_proc = proc
+        self_args = []
+        for arg in args:
+            term = arg if isinstance(arg, Term) else Literal(arg)
+            self_args.append(Arg(term))
+        for label, arg in kwargs.items():
path: daft/io/pushdowns.py
user: login: rchowell
body: The [kwargs preserve order](https://peps.python.org/pep-0468/) so I just add them to the arg list in the order they come in.
created_at: 2025-04-18T16:45:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,230 @@
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from dataclasses import dataclass
+from functools import singledispatchmethod
+from typing import Generic, Sequence, TypeVar, Union
+
+Value = Union[str, int, float, bool]
+R = TypeVar("R")
+C = TypeVar("C")
+
+
+@dataclass(frozen=True)
+class Pushdowns(ABC):
+    """Base class for pushdown information."""
+
+    pass
+
+
+@dataclass(frozen=True)
+class Term(ABC):
+    """Term is the base class for representing expressions in pushdowns."""
+
+    def __str__(self) -> str:
+        return LispyVisitor().visit(self, None)
+
+
+@dataclass(frozen=True)
+class Reference(Term):
+    """Reference to a field in some schema, this is not bound to any schema."""
+
+    name: str
+
+
+@dataclass(frozen=True)
+class Literal(Term):
+    """Literal value which does not currently carry its daft type information.
+
+    The representation of types and the full-fidelity of values complicates both
+    the s-expression encoding and the end user consumption. The goal here is for
+    simple consumption of daft's rust expressions from a python context. We may
+    iterate on this design's complexity as more advanced use-cases emerge.
+
+    Example:
+    >>> v1 = Literal("hello")
+    >>> v2 = Literal(1)
+    >>> v3 = Literal(2.0)
+    >>> v4 = Literal(False)
+    """
+
+    value: Value | None
+
+
+@dataclass(frozen=True)
+class Expr(Term):
+    """Expr is a generic expression form which has the procedure symbol and its args.
+
+    This representation was chosen because it's the simplest while avoiding the complexity
+    of a more 'pure' concatenation style or introducing additional Term variants.
+    This is a trade-off where lower representation complexity requires slightly more creativity
+    in representing the various expression forms, but named arguments ease consumption.
+
+    Exprs follow the common rule that positional come before named.
+    This is enforced on instantiation since we use python args and kwargs.
+
+    Example:
+    >>> Expr("f", 42)  # (f 42)
+    >>> Expr("f", "hello")  # (f "hello")
+    >>> Expr("f", 1, 2.5, "test")  # (f 1 2.50 "test")
+    >>> Expr("f", value=True)  # (f value::true)
+    >>> Expr("f", x=10, y=20.5)  # (f x::10 y::20.50)
+    """
+
+    proc: str
path: daft/io/pushdowns.py
user: login: rchowell
body: This is the primary pain point of such a simple representation. One one hand, the strictness of introducing an enum can result in many breaking changes for consumers as the enum evolves. On the other hand, something as open-ended as a procedure tag (proc string or proc tag) is more of a challenge in documentation and can be considered a flexibility-fragility trade-off.

I did not want to introduce a [type hierarchy like PyIceberg](https://py.iceberg.apache.org/reference/pyiceberg/expressions/) since Daft already has Expression and this would be confusing. However Expression is a wrapper over a Rust enum so we have no python type hierarchy. IMO the most sane way was with these "terms" and the procedure tag creates a documentation responsibility for Daft.

I don't know if we can avoid the enum (sealed) vs. tagged (open) trade-off since there will always be additions to operators and functions. 

Even if we have the visitor only solution, this problem still remains,

```python
# op is an enum (sealed), will break if new operators are added
def visit_binary_operator(op: BinaryOperator, lhs: Expr, rhs: Expr) -> R

# op is a string tag (open), still need to switch
def visit_binary_operator(op: str, lhs: Expr, rhs: Expr) -> R
```

In my PartiQL experience, sealed members as part of the public API was painful for evolution. Rust has very strict semver rules with enums for this reason, where unless marked as "open", it's a MAJOR version breaking change. Java/Kotlin doesn't even have this capability. 

We could use python int enums and document these should be used non-exhaustively, but I feel that, while it's somewhat self-documenting, it will increase the size of this IR.
created_at: 2025-04-18T16:59:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,230 @@
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from dataclasses import dataclass
+from functools import singledispatchmethod
+from typing import Generic, Sequence, TypeVar, Union
+
+Value = Union[str, int, float, bool]
+R = TypeVar("R")
+C = TypeVar("C")
+
+
+@dataclass(frozen=True)
+class Pushdowns(ABC):
+    """Base class for pushdown information."""
+
+    pass
+
+
+@dataclass(frozen=True)
+class Term(ABC):
+    """Term is the base class for representing expressions in pushdowns."""
+
+    def __str__(self) -> str:
+        return LispyVisitor().visit(self, None)
+
+
+@dataclass(frozen=True)
+class Reference(Term):
+    """Reference to a field in some schema, this is not bound to any schema."""
+
+    name: str
+
+
+@dataclass(frozen=True)
+class Literal(Term):
path: daft/io/pushdowns.py
user: login: mcember
body: IMO it's OK to omit it for now and add it later if we need it
created_at: 2025-04-21T13:33:32Z
author_association: NONE
diff_hunk: @@ -0,0 +1,230 @@
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from dataclasses import dataclass
+from functools import singledispatchmethod
+from typing import Generic, Sequence, TypeVar, Union
+
+Value = Union[str, int, float, bool]
+R = TypeVar("R")
+C = TypeVar("C")
+
+
+@dataclass(frozen=True)
+class Pushdowns(ABC):
+    """Base class for pushdown information."""
+
+    pass
+
+
+@dataclass(frozen=True)
+class Term(ABC):
+    """Term is the base class for representing expressions in pushdowns."""
+
+    def __str__(self) -> str:
+        return LispyVisitor().visit(self, None)
+
+
+@dataclass(frozen=True)
+class Reference(Term):
+    """Reference to a field in some schema, this is not bound to any schema."""
+
+    name: str
+
+
+@dataclass(frozen=True)
+class Literal(Term):
+    """Literal value which does not currently carry its daft type information.
+
+    The representation of types and the full-fidelity of values complicates both
+    the s-expression encoding and the end user consumption. The goal here is for
+    simple consumption of daft's rust expressions from a python context. We may
+    iterate on this design's complexity as more advanced use-cases emerge.
+
+    Example:
+    >>> v1 = Literal("hello")
+    >>> v2 = Literal(1)
+    >>> v3 = Literal(2.0)
+    >>> v4 = Literal(False)
+    """
+
+    value: Value | None
+
+
+@dataclass(frozen=True)
+class Expr(Term):
+    """Expr is a generic expression form which has the procedure symbol and its args.
+
+    This representation was chosen because it's the simplest while avoiding the complexity
+    of a more 'pure' concatenation style or introducing additional Term variants.
+    This is a trade-off where lower representation complexity requires slightly more creativity
+    in representing the various expression forms, but named arguments ease consumption.
+
+    Exprs follow the common rule that positional come before named.
+    This is enforced on instantiation since we use python args and kwargs.
+
+    Example:
+    >>> Expr("f", 42)  # (f 42)
+    >>> Expr("f", "hello")  # (f "hello")
+    >>> Expr("f", 1, 2.5, "test")  # (f 1 2.50 "test")
+    >>> Expr("f", value=True)  # (f value::true)
+    >>> Expr("f", x=10, y=20.5)  # (f x::10 y::20.50)
+    """
+
+    proc: str
path: daft/io/pushdowns.py
user: login: mcember
body: Fair enough. It's not practical to make this a sealed enum and at that point I buy the argument that having a type hierarchy isn't substantially better than a string name
created_at: 2025-04-21T13:44:09Z
author_association: NONE
path: tests/io/test_pushdowns.py
user: login: desmondcheongzx
body: I would love to see an end-to-end test where we create the pushdown and check the result from a scan with the pushdown.
created_at: 2025-04-21T22:19:19Z
author_association: COLLABORATOR
path: tests/io/test_pushdowns.py
user: login: rchowell
body: For sure, working on the end-to-end example now before merging so Mingshi and Matt have a solid example! Otherwise totally agreed this is quite difficult to grok from source only.
created_at: 2025-04-21T22:38:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,16 @@
+# I/O
+
+Daft supports diverse input sources and output sinks which are covered in [DataFrame Creation](./dataframe_creation.md) —
path: docs/api/io.md
user: login: ccmao1130
body: `[DataFrame Creation](dataframe_creation.md)`
created_at: 2025-04-22T01:36:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,31 @@
+# Pushdowns
+
+Daft has predicate, projection, and limit pushdowns with expressions being represented by *Terms*.
path: docs/api/pushdowns.md
user: login: ccmao1130
body: Add something like:
`Learn more about [Pushdowns](../advanced/pushdowns.md) in Daft User Guide`
created_at: 2025-04-22T01:38:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,92 @@
+# Pushdowns
+
+!!! warning "Warning"
+
+    These APIs are considered experimental.
+
+Daft supports predicate, projection, and limit pushdowns in its scan operators.
+
+## Usage
+
+Daft does not *officially* support user-defined scan operators, but it is
+possible today by extending the [`ScanOperator`][daft.io.scan.ScanOperator].
+
+The [`to_scan_tasks`][daft.io.scan.ScanOperator.to_scan_tasks] method receives a
+pushdowns object which contains information the optimizer determined can be
+pushed down into the scan. This is a wrapper over a Rust type, and its members
+cannot be easily interpreted from python. To solve this, we use a python
+[`Pushdowns`][daft.io.pushdowns.Pushdowns] class holding
+[`Term`][daft.io.pushdowns.Term] objects which are a representation of the
+pushed-down expressions. You can construct a
+[`ScanPushdowns`][daft.io.scan.ScanPushdowns] object using `_from_pypushdowns(pushdowns, schema)`.
+
+Once you have the pushdown terms, you will need to apply them based upon
+your own context. One way to operate on a term tree is using the
+[`TermVisitor`][daft.io.pushdowns.TermVisitor].
+
+## Term Reference
+
+!!! warn "Warning!"
+
+    This list is not comprehensive, and it is recommended to either inspect
+    a term tree or test your
path: docs/advanced/pushdowns.md
user: login: desmondcheongzx
body: Nit: this cuts off
created_at: 2025-04-22T21:34:13Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,92 @@
+# Pushdowns
+
+!!! warning "Warning"
+
+    These APIs are considered experimental.
+
+Daft supports predicate, projection, and limit pushdowns in its scan operators.
+
+## Usage
+
+Daft does not *officially* support user-defined scan operators, but it is
+possible today by extending the [`ScanOperator`][daft.io.scan.ScanOperator].
+
+The [`to_scan_tasks`][daft.io.scan.ScanOperator.to_scan_tasks] method receives a
+pushdowns object which contains information the optimizer determined can be
+pushed down into the scan. This is a wrapper over a Rust type, and its members
+cannot be easily interpreted from python. To solve this, we use a python
+[`Pushdowns`][daft.io.pushdowns.Pushdowns] class holding
+[`Term`][daft.io.pushdowns.Term] objects which are a representation of the
+pushed-down expressions. You can construct a
+[`ScanPushdowns`][daft.io.scan.ScanPushdowns] object using `_from_pypushdowns(pushdowns, schema)`.
+
+Once you have the pushdown terms, you will need to apply them based upon
+your own context. One way to operate on a term tree is using the
+[`TermVisitor`][daft.io.pushdowns.TermVisitor].
+
+## Term Reference
+
+!!! warn "Warning!"
+
+    This list is not comprehensive, and it is recommended to either inspect
+    a term tree or test your
path: docs/advanced/pushdowns.md
user: login: rchowell
body: oops, fixing in the example follow-up.
created_at: 2025-04-22T21:35:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,92 @@
+# Pushdowns
+
+!!! warning "Warning"
+
+    These APIs are considered experimental.
+
+Daft supports predicate, projection, and limit pushdowns in its scan operators.
+
+## Usage
+
+Daft does not *officially* support user-defined scan operators, but it is
+possible today by extending the [`ScanOperator`][daft.io.scan.ScanOperator].
+
+The [`to_scan_tasks`][daft.io.scan.ScanOperator.to_scan_tasks] method receives a
+pushdowns object which contains information the optimizer determined can be
+pushed down into the scan. This is a wrapper over a Rust type, and its members
+cannot be easily interpreted from python. To solve this, we use a python
+[`Pushdowns`][daft.io.pushdowns.Pushdowns] class holding
+[`Term`][daft.io.pushdowns.Term] objects which are a representation of the
+pushed-down expressions. You can construct a
+[`ScanPushdowns`][daft.io.scan.ScanPushdowns] object using `_from_pypushdowns(pushdowns, schema)`.
+
+Once you have the pushdown terms, you will need to apply them based upon
+your own context. One way to operate on a term tree is using the
+[`TermVisitor`][daft.io.pushdowns.TermVisitor].
+
+## Term Reference
+
+!!! warn "Warning!"
+
+    This list is not comprehensive, and it is recommended to either inspect
+    a term tree or test your
path: docs/advanced/pushdowns.md
user: login: rchowell
body: https://github.com/Eventual-Inc/Daft/pull/4243/files#diff-f040a7795b039311f0f3416427bc5cec87373141fac4235dd89505947af2c030R31-R32
created_at: 2025-04-24T15:16:01Z
author_association: CONTRIBUTOR

PR comments:
user: login: codecov[bot]
created_at: 2025-04-17T23:36:09Z
body: ## [Codecov](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4203?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) Report
Attention: Patch coverage is `85.84337%` with `47 lines` in your changes missing coverage. Please review.
> Project coverage is 78.45%. Comparing base [(`89949f1`)](https://app.codecov.io/gh/Eventual-Inc/Daft/commit/89949f107405ad5e5e3ebec9068a9f2a92533802?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) to head [(`3758429`)](https://app.codecov.io/gh/Eventual-Inc/Daft/commit/37584294d670b639b63c7d423c67e24596ea5892?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc).
> Report is 5 commits behind head on main.

| [Files with missing lines](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4203?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) | Patch % | Lines |
|---|---|---|
| [src/common/scan-info/src/python.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4203?src=pr&el=tree&filepath=src%2Fcommon%2Fscan-info%2Fsrc%2Fpython.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2NvbW1vbi9zY2FuLWluZm8vc3JjL3B5dGhvbi5ycw==) | 77.47% | [41 Missing :warning: ](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4203?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) |
| [daft/io/pushdowns.py](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4203?src=pr&el=tree&filepath=daft%2Fio%2Fpushdowns.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-ZGFmdC9pby9wdXNoZG93bnMucHk=) | 94.28% | [6 Missing :warning: ](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4203?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) |

<details><summary>Additional details and impacted files</summary>


[![Impacted file tree graph](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4203/graphs/tree.svg?width=650&height=150&src=pr&token=J430QVFE89&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc)](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4203?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc)

```diff
@@            Coverage Diff             @@
##             main    #4203      +/-   ##
==========================================
+ Coverage   78.44%   78.45%   +0.01%     
==========================================
  Files         798      799       +1     
  Lines      104706   105116     +410     
==========================================
+ Hits        82133    82473     +340     
- Misses      22573    22643      +70     
```

| [Files with missing lines](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4203?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) | Coverage Δ | |
|---|---|---|
| [daft/io/scan.py](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4203?src=pr&el=tree&filepath=daft%2Fio%2Fscan.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-ZGFmdC9pby9zY2FuLnB5) | `83.33% <100.00%> (+12.74%)` | :arrow_up: |
| [src/daft-dsl/src/expr/mod.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4203?src=pr&el=tree&filepath=src%2Fdaft-dsl%2Fsrc%2Fexpr%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtZHNsL3NyYy9leHByL21vZC5ycw==) | `78.69% <100.00%> (+0.30%)` | :arrow_up: |
| [src/daft-dsl/src/python.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4203?src=pr&el=tree&filepath=src%2Fdaft-dsl%2Fsrc%2Fpython.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtZHNsL3NyYy9weXRob24ucnM=) | `91.35% <100.00%> (-0.11%)` | :arrow_down: |
| [daft/io/pushdowns.py](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4203?src=pr&el=tree&filepath=daft%2Fio%2Fpushdowns.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-ZGFmdC9pby9wdXNoZG93bnMucHk=) | `94.28% <94.28%> (ø)` | |
| [src/common/scan-info/src/python.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4203?src=pr&el=tree&filepath=src%2Fcommon%2Fscan-info%2Fsrc%2Fpython.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2NvbW1vbi9zY2FuLWluZm8vc3JjL3B5dGhvbi5ycw==) | `79.84% <77.47%> (-5.34%)` | :arrow_down: |

... and [10 files with indirect coverage changes](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/4203/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc)

</details>
<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
author_association: NONE
user: login: MingshiPeng
created_at: 2025-04-18T19:55:03Z
body: Overall LTGM - it would be nice to have an example implementation using the proposed new interface. 
author_association: NONE