Repository: apache/iceberg-python
Pull request #1830: Use a balanced tree instead of unbalanced one to prevent recursion error in create_match_filter
Author: login: koenvo
Created: 2025-03-22T14:36:16Z | Merged: 2025-03-31T12:24:48Z

**Use a balanced tree instead of an unbalanced one to prevent recursion error in `create_match_filter`**

<!-- Closes #1776 -->

## Rationale for this change

In the `create_match_filter` function, the previous implementation used `functools.reduce(operator.or_, filters)` to combine expressions. This approach constructed a right-heavy, unbalanced tree, which could lead to a `RecursionError` when dealing with a large number of expressions (e.g., over 1,000).

To address this, we've introduced the `_build_balanced_tree` function. This utility constructs a balanced binary tree of expressions, reducing the maximum depth to O(log n) and thereby preventing potential recursion errors. This makes expression construction more stable and scalable, especially when working with large datasets.

```python

# Past behavior

Or(*[A, B, C, D]) = Or(A, Or(B, Or(C, D))

# New behavior
Or(*[A, B, C, D]) = Or(Or(A, B), Or(C, D))

```

## Are these changes tested?

Yes, existing tests cover the functionality of `Or`. Additional testing was done with large expression sets (e.g., 10,000 items) to ensure that balanced tree construction avoids recursion errors.

## Are there any user-facing changes?

No, there are no user-facing changes. This is an internal implementation improvement that does not affect the public API.

Closes #1759
Closes #1785

<!-- In the case of user-facing changes, please add the changelog label. -->

Linked issue #1776: Transactions do not support Upsert
### Feature Request / Improvement

Given the issue #1759 

I tried instead writing smaller amounts of rows but also wanted a guarantee that all rows would be upserted.

Tried wrapping `upsert` within a Transaction but noticed Transaction does not support upserts


Using 0.9.0 release
Linked issue comments: user: login: Fokko
created_at: 2025-03-10T20:20:52Z
body: Thanks @suarez-agu for raising this. The transaction should also support upsert. Are you interested in contributing this?
author_association: CONTRIBUTOR
user: login: koenvo
created_at: 2025-03-11T15:57:31Z
body: I would like to take a look at this one.
author_association: CONTRIBUTOR
user: login: potatochipcoconut
created_at: 2025-05-13T04:45:17Z
body: @koenvo how's it looking? I'm also interested in this feature.
author_association: NONE
user: login: koenvo
created_at: 2025-05-13T04:56:13Z
body: Thanks for the reminder. Let me finish this
author_association: CONTRIBUTOR
user: login: potatochipcoconut
created_at: 2025-05-13T15:35:27Z
body: thanks @koenvo looking forward to it!
author_association: NONE
user: login: potatochipcoconut
created_at: 2025-05-14T21:30:49Z
body: @Fokko just curious, any eta on when this would land with next release?
author_association: NONE
user: login: potatochipcoconut
created_at: 2025-05-14T22:34:42Z
body: @koenvo just tried this out but wondering if it'd be possible to have upsert available while updating [snapshot](https://github.com/apache/iceberg-python/blob/main/pyiceberg/table/update/snapshot.py#L604)?

I'm hoping to implement upsert() for a workflow that uses the distributed write pattern as described [here](https://github.com/apache/iceberg-python/issues/1751)

Where the data is written into the table like such:
```
        with table.transaction() as trx:
            with trx.update_snapshot().fast_append() as update_snapshot:
                update_snapshot.append_data_file(data_file)
```

Thanks!
author_association: NONE
user: login: koenvo
created_at: 2025-05-15T04:34:03Z
body: The pattern you describing only does appends, right? In that case it’s easier to have distributed workers and a central worker doing the commits.

With an upsert you probably also have to delete data. When a delete is involved it’s harder as upserts get dependent on each other.

@Fokko what do you think?
author_association: CONTRIBUTOR
user: login: potatochipcoconut
created_at: 2025-05-15T15:14:30Z
body: I see, hmm, dang yeah I have a concern because I'm using a FIFO queue to process the data, but those have a limitation of 10 items per batch, so I want to try a regular queue with higher throughput, but that introduces possibility of duplicate events being processed (exactly-once vs at-least-once delivery). My thought was that using upsert would make it safe and wouldn't end up with duplicate data.
author_association: NONE
user: login: potatochipcoconut
created_at: 2025-05-20T18:37:07Z
body: FYI tried this out in distributed write scenario where data files are serialized, then deserialized as described in link above, this is result when trying to do `txn.upsert()`

```
[ERROR] AttributeError: 'DataFile' object has no attribute 'select'
Traceback (most recent call last):
  File "/var/task/write/lambda_function.py", line 26, in handler
    write_to_iceberg(data_files)
  File "/var/task/ocr_experiments/utils.py", line 317, in write_to_iceberg
    write_data(table, data)
  File "/var/task/ocr_experiments/utils.py", line 329, in write_data
    txn.upsert(data_file, join_cols=['id'])
  File "/var/lang/lib/python3.12/site-packages/pyiceberg/table/__init__.py", line 763, in upsert
    if upsert_util.has_duplicate_rows(df, join_cols):
  File "/var/lang/lib/python3.12/site-packages/pyiceberg/table/upsert_util.py", line 53, in has_duplicate_rows
    return len(df.select(join_cols).group_by(join_cols).aggregate([([], "count_all")]).filter(pc.field("count_all") > 1)) > 0
```
author_association: NONE

Diff:
diff --git a/pyiceberg/expressions/__init__.py b/pyiceberg/expressions/__init__.py
index 8b006a28f1..2adf898fea 100644
--- a/pyiceberg/expressions/__init__.py
+++ b/pyiceberg/expressions/__init__.py
@@ -18,11 +18,13 @@
 from __future__ import annotations
 
 from abc import ABC, abstractmethod
-from functools import cached_property, reduce
+from functools import cached_property
 from typing import (
     Any,
+    Callable,
     Generic,
     Iterable,
+    Sequence,
     Set,
     Tuple,
     Type,
@@ -79,6 +81,45 @@ def __or__(self, other: BooleanExpression) -> BooleanExpression:
         return Or(self, other)
 
 
+def _build_balanced_tree(
+    operator_: Callable[[BooleanExpression, BooleanExpression], BooleanExpression], items: Sequence[BooleanExpression]
+) -> BooleanExpression:
+    """
+    Recursively constructs a balanced binary tree of BooleanExpressions using the provided binary operator.
+
+    This function is a safer and more scalable alternative to:
+        reduce(operator_, items)
+
+    Using `reduce` creates a deeply nested, unbalanced tree (e.g., operator_(a, operator_(b, operator_(c, ...)))),
+    which grows linearly with the number of items. This can lead to RecursionError exceptions in Python
+    when the number of expressions is large (e.g., >1000).
+
+    In contrast, this function builds a balanced binary tree with logarithmic depth (O(log n)),
+    helping avoid recursion issues and ensuring that expression trees remain stable, predictable,
+    and safe to traverse — especially in tools like PyIceberg that operate on large logical trees.
+
+    Parameters:
+        operator_ (Callable): A binary operator function (e.g., pyiceberg.expressions.Or, And) that takes two
+            BooleanExpressions and returns a combined BooleanExpression.
+        items (Sequence[BooleanExpression]): A sequence of BooleanExpression objects to combine.
+
+    Returns:
+        BooleanExpression: The balanced combination of all input BooleanExpressions.
+
+    Raises:
+        ValueError: If the input sequence is empty.
+    """
+    if not items:
+        raise ValueError("No expressions to combine")
+    if len(items) == 1:
+        return items[0]
+    mid = len(items) // 2
+
+    left = _build_balanced_tree(operator_, items[:mid])
+    right = _build_balanced_tree(operator_, items[mid:])
+    return operator_(left, right)
+
+
 class Term(Generic[L], ABC):
     """A simple expression that evaluates to a value."""
 
@@ -214,7 +255,7 @@ class And(BooleanExpression):
 
     def __new__(cls, left: BooleanExpression, right: BooleanExpression, *rest: BooleanExpression) -> BooleanExpression:  # type: ignore
         if rest:
-            return reduce(And, (left, right, *rest))
+            return _build_balanced_tree(And, (left, right, *rest))
         if left is AlwaysFalse() or right is AlwaysFalse():
             return AlwaysFalse()
         elif left is AlwaysTrue():
@@ -257,7 +298,7 @@ class Or(BooleanExpression):
 
     def __new__(cls, left: BooleanExpression, right: BooleanExpression, *rest: BooleanExpression) -> BooleanExpression:  # type: ignore
         if rest:
-            return reduce(Or, (left, right, *rest))
+            return _build_balanced_tree(Or, (left, right, *rest))
         if left is AlwaysTrue() or right is AlwaysTrue():
             return AlwaysTrue()
         elif left is AlwaysFalse():
diff --git a/pyiceberg/table/upsert_util.py b/pyiceberg/table/upsert_util.py
index d2bd48bc99..e67f6c0232 100644
--- a/pyiceberg/table/upsert_util.py
+++ b/pyiceberg/table/upsert_util.py
@@ -26,6 +26,7 @@
     BooleanExpression,
     EqualTo,
     In,
+    Or,
 )
 
 
@@ -39,7 +40,12 @@ def create_match_filter(df: pyarrow_table, join_cols: list[str]) -> BooleanExpre
             functools.reduce(operator.and_, [EqualTo(col, row[col]) for col in join_cols]) for row in unique_keys.to_pylist()
         ]
 
-        return AlwaysFalse() if len(filters) == 0 else functools.reduce(operator.or_, filters)
+        if len(filters) == 0:
+            return AlwaysFalse()
+        elif len(filters) == 1:
+            return filters[0]
+        else:
+            return Or(*filters)
 
 
 def has_duplicate_rows(df: pyarrow_table, join_cols: list[str]) -> bool:
diff --git a/tests/expressions/test_expressions.py b/tests/expressions/test_expressions.py
index 12d9ff95a9..858a9ff852 100644
--- a/tests/expressions/test_expressions.py
+++ b/tests/expressions/test_expressions.py
@@ -591,11 +591,11 @@ def test_negate(lhs: BooleanExpression, rhs: BooleanExpression) -> None:
     [
         (
             And(ExpressionA(), ExpressionB(), ExpressionA()),
-            And(And(ExpressionA(), ExpressionB()), ExpressionA()),
+            And(ExpressionA(), And(ExpressionB(), ExpressionA())),
         ),
         (
             Or(ExpressionA(), ExpressionB(), ExpressionA()),
-            Or(Or(ExpressionA(), ExpressionB()), ExpressionA()),
+            Or(ExpressionA(), Or(ExpressionB(), ExpressionA())),
         ),
         (Not(Not(ExpressionA())), ExpressionA()),
     ],
diff --git a/tests/expressions/test_visitors.py b/tests/expressions/test_visitors.py
index 94bfcf076c..586ba9f5d4 100644
--- a/tests/expressions/test_visitors.py
+++ b/tests/expressions/test_visitors.py
@@ -230,14 +230,14 @@ def test_boolean_expression_visitor() -> None:
         "NOT",
         "OR",
         "EQUALTO",
-        "OR",
         "NOTEQUALTO",
         "OR",
+        "OR",
         "EQUALTO",
         "NOT",
-        "AND",
         "NOTEQUALTO",
         "AND",
+        "AND",
     ]
 
 
@@ -335,14 +335,14 @@ def test_always_false_or_always_true_expression_binding(table_schema_simple: Sch
                 ),
             ),
             And(
-                And(
-                    BoundIn(
-                        BoundReference(
-                            field=NestedField(field_id=1, name="foo", field_type=StringType(), required=False),
-                            accessor=Accessor(position=0, inner=None),
-                        ),
-                        {literal("bar"), literal("baz")},
+                BoundIn(
+                    BoundReference(
+                        field=NestedField(field_id=1, name="foo", field_type=StringType(), required=False),
+                        accessor=Accessor(position=0, inner=None),
                     ),
+                    {literal("bar"), literal("baz")},
+                ),
+                And(
                     BoundEqualTo[int](
                         BoundReference(
                             field=NestedField(field_id=2, name="bar", field_type=IntegerType(), required=True),
@@ -350,13 +350,13 @@ def test_always_false_or_always_true_expression_binding(table_schema_simple: Sch
                         ),
                         literal(1),
                     ),
-                ),
-                BoundEqualTo(
-                    BoundReference(
-                        field=NestedField(field_id=1, name="foo", field_type=StringType(), required=False),
-                        accessor=Accessor(position=0, inner=None),
+                    BoundEqualTo(
+                        BoundReference(
+                            field=NestedField(field_id=1, name="foo", field_type=StringType(), required=False),
+                            accessor=Accessor(position=0, inner=None),
+                        ),
+                        literal("baz"),
                     ),
-                    literal("baz"),
                 ),
             ),
         ),
@@ -408,28 +408,28 @@ def test_and_expression_binding(
                 ),
             ),
             Or(
+                BoundIn(
+                    BoundReference(
+                        field=NestedField(field_id=1, name="foo", field_type=StringType(), required=False),
+                        accessor=Accessor(position=0, inner=None),
+                    ),
+                    {literal("bar"), literal("baz")},
+                ),
                 Or(
                     BoundIn(
                         BoundReference(
                             field=NestedField(field_id=1, name="foo", field_type=StringType(), required=False),
                             accessor=Accessor(position=0, inner=None),
                         ),
-                        {literal("bar"), literal("baz")},
+                        {literal("bar")},
                     ),
                     BoundIn(
                         BoundReference(
                             field=NestedField(field_id=1, name="foo", field_type=StringType(), required=False),
                             accessor=Accessor(position=0, inner=None),
                         ),
-                        {literal("bar")},
-                    ),
-                ),
-                BoundIn(
-                    BoundReference(
-                        field=NestedField(field_id=1, name="foo", field_type=StringType(), required=False),
-                        accessor=Accessor(position=0, inner=None),
+                        {literal("baz")},
                     ),
-                    {literal("baz")},
                 ),
             ),
         ),
diff --git a/tests/io/test_pyarrow_visitor.py b/tests/io/test_pyarrow_visitor.py
index 9f5aff3f70..9d5772d01c 100644
--- a/tests/io/test_pyarrow_visitor.py
+++ b/tests/io/test_pyarrow_visitor.py
@@ -836,5 +836,5 @@ def test_expression_to_complementary_pyarrow(
     # Notice an isNan predicate on a str column is automatically converted to always false and removed from Or and thus will not appear in the pc.expr.
     assert (
         repr(result)
-        == """<pyarrow.compute.Expression (((invert((((((string_field == "hello") and (float_field > 100)) or (is_nan(float_field) and (double_field == 0))) or (float_field > 100)) and invert(is_null(double_field, {nan_is_null=false})))) or is_null(float_field, {nan_is_null=false})) or is_null(string_field, {nan_is_null=false})) or is_nan(double_field))>"""
+        == """<pyarrow.compute.Expression (((invert(((((string_field == "hello") and (float_field > 100)) or ((is_nan(float_field) and (double_field == 0)) or (float_field > 100))) and invert(is_null(double_field, {nan_is_null=false})))) or is_null(float_field, {nan_is_null=false})) or is_null(string_field, {nan_is_null=false})) or is_nan(double_field))>"""
     )


Reviews:
user: login: Fokko
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T09:20:27Z
user: login: Fokko
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T09:21:37Z
user: login: Fokko
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T09:23:16Z
user: login: koenvo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T10:09:56Z
user: login: koenvo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T10:10:16Z
user: login: Fokko
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T15:24:12Z
user: login: Fokko
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T08:13:37Z

Review comments:
diff_hunk: @@ -28,6 +29,43 @@
     In,
 )
 
+T = TypeVar("T")
+
+
+def build_balanced_tree(operator_: Callable[[T, T], T], items: List[T]) -> T:
path: pyiceberg/table/upsert_util.py
user: login: Fokko
body: I think we could replace `T` with `BooleanExpression`.
created_at: 2025-03-24T09:20:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -28,6 +29,43 @@
     In,
 )
 
+T = TypeVar("T")
+
+
+def build_balanced_tree(operator_: Callable[[T, T], T], items: List[T]) -> T:
path: pyiceberg/table/upsert_util.py
user: login: Fokko
body: ```suggestion
def _build_balanced_tree(operator_: Callable[[T, T], T], items: List[T]) -> T:
```
created_at: 2025-03-24T09:21:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -39,7 +77,7 @@ def create_match_filter(df: pyarrow_table, join_cols: list[str]) -> BooleanExpre
             functools.reduce(operator.and_, [EqualTo(col, row[col]) for col in join_cols]) for row in unique_keys.to_pylist()
         ]
 
-        return AlwaysFalse() if len(filters) == 0 else functools.reduce(operator.or_, filters)
+        return AlwaysFalse() if len(filters) == 0 else build_balanced_tree(operator.or_, filters)
path: pyiceberg/table/upsert_util.py
user: login: Fokko
body: I'm thinking out loud here. Maybe bring this logic into the `Or` operation:

https://github.com/apache/iceberg-python/blob/71cb2473bee8161372b057207c82f73d42ae3092/pyiceberg/expressions/__init__.py#L260

We could pass it in like:
```suggestion
        return AlwaysFalse() if len(filters) == 0 else Or(*filters)
```

By bringing this into the `Op` operation, then this is much more structured. Thoughts/opinions?
created_at: 2025-03-24T09:23:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -39,7 +77,7 @@ def create_match_filter(df: pyarrow_table, join_cols: list[str]) -> BooleanExpre
             functools.reduce(operator.and_, [EqualTo(col, row[col]) for col in join_cols]) for row in unique_keys.to_pylist()
         ]
 
-        return AlwaysFalse() if len(filters) == 0 else functools.reduce(operator.or_, filters)
+        return AlwaysFalse() if len(filters) == 0 else build_balanced_tree(operator.or_, filters)
path: pyiceberg/table/upsert_util.py
user: login: koenvo
body: Yes! That's a great change. And this change will only affect when you pass more than 2 expressions, so you can still keep full control if you want to build an unbalanced tree yourself.

Let me change this.
created_at: 2025-03-24T10:09:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -28,6 +29,43 @@
     In,
 )
 
+T = TypeVar("T")
+
+
+def build_balanced_tree(operator_: Callable[[T, T], T], items: List[T]) -> T:
path: pyiceberg/table/upsert_util.py
user: login: koenvo
body: I believe I tried that, but the linting was complaining. Let me doublecheck.
created_at: 2025-03-24T10:10:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -257,7 +298,7 @@ class Or(BooleanExpression):
 
     def __new__(cls, left: BooleanExpression, right: BooleanExpression, *rest: BooleanExpression) -> BooleanExpression:  # type: ignore
         if rest:
-            return reduce(Or, (left, right, *rest))
path: pyiceberg/expressions/__init__.py
user: login: Fokko
body: I think we also want to apply this in the `And` situation :)
created_at: 2025-03-24T15:24:11Z
author_association: CONTRIBUTOR

PR comments:
user: login: Fokko
created_at: 2025-03-24T15:28:17Z
body: I think this is pretty elegant, thoughts @kevinjqliu?

I expressed some concerns with in https://github.com/apache/iceberg-python/pull/1783 around performance. I have played around with the `singledispatch` stuff in the past, and it was always crazy fast because the type dispatching is pushed down into the C-layer of Python.
author_association: CONTRIBUTOR
user: login: Fokko
created_at: 2025-03-31T12:24:31Z
body: Moving this forward since I think it is a great idea in general :)
author_association: CONTRIBUTOR
user: login: Fokko
created_at: 2025-03-31T12:24:52Z
body: Thanks @koenvo 
author_association: CONTRIBUTOR