Repository: voxel51/fiftyone
Pull request #5705: Optimize frame lookups to be as late as possible in aggregations
Author: login: brimoor
Labels: name: enhancement
description: Code enhancement
Created: 2025-04-10T02:03:13Z | Merged: 2025-04-21T18:24:40Z

## Change log

- Optimizes frame lookups to be as late as possible in aggregation pipelines

cc @ehofesmann this seemingly small change results in big performance improvements in certain cases:
1. Significantly improves the performance of frame-level aggregations on views where `len(view) << len(dataset)`
2. Resolves many (but *not yet* all) instances of "16MB errors", aka [#1607](https://github.com/voxel51/fiftyone/issues/1607)

See below for examples of these cases.

## Example optimizations

```py
import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F

dataset = foz.load_zoo_dataset("quickstart-video")
dataset.compute_metadata()
```

### Case 1: a view that doesn't filter on frame-level information

```py
view = dataset.match(F("metadata.size_bytes") > 2671165)
_ = view.count("frames")
```

#### Previous pipeline ❌ ❌ ❌ ❌ 

`$match` is between `$lookup` and `$unwind`, which means that this pipeline will fail with "16MB errors" on datasets with many frames.

```
[
    {
        '$lookup': {
            'from': 'frames.samples.67f7211b32b0fedf4532de6f',
            'let': {'sample_id': '$_id'},
            'pipeline': [
                {'$match': {'$expr': {'$eq': ['$$sample_id', '$_sample_id']}}},
                {'$sort': {'frame_number': 1}},
            ],
            'as': 'frames',
        },
    },
    {'$match': {'$expr': {'$gt': ['$metadata.size_bytes', 2671165]}}},
    {'$unwind': '$frames'},
    {'$count': 'count'},
]
```

#### New pipeline ✅ ✅ ✅ ✅   

`$match` comes first _and then_ `$lookup + $unwind` in immediate succession, which means no "16MB error" issues!

Additionally, `$lookup` is now only performed on samples that match the sample-level filter, which could result in *substantially faster* aggregations when `len(view) << len(dataset)`.

```
[
    {'$match': {'$expr': {'$gt': ['$metadata.size_bytes', 2671165]}}},
    {
        '$lookup': {
            'from': 'frames.samples.67f720ab45345727feab06bd',
            'let': {'sample_id': '$_id'},
            'pipeline': [
                {'$match': {'$expr': {'$eq': ['$$sample_id', '$_sample_id']}}},
                {'$sort': {'frame_number': 1}},
            ],
            'as': 'frames',
        },
    },
    {'$unwind': '$frames'},
    {'$count': 'count'},
]
```

### Case 2: an aggregation that involves explicit frame unwinding

```py
_ = dataset.values("frames[].frame_number")
```

#### Previous pipeline ❌ ❌ ❌ ❌ 

`$project` is between `$lookup` and `$unwind`, which means that this pipeline will fail with "16MB errors" on datasets with many frames.

```
[
    {
        '$lookup': {
            'from': 'frames.samples.67f7211b32b0fedf4532de6f',
            'let': {'sample_id': '$_id'},
            'pipeline': [
                {'$match': {'$expr': {'$eq': ['$$sample_id', '$_sample_id']}}},
                {'$sort': {'frame_number': 1}},
            ],
            'as': 'frames',
        },
    },
    {'$project': {'frames.frame_number': True}},
    {'$unwind': '$frames'},
    {
        '$project': {
            'values': {
                '$cond': {
                    'if': {'$gt': ['$frames.frame_number', None]},
                    'then': '$frames.frame_number',
                    'else': None,
                },
            },
        },
    },
]
```

#### New pipeline ✅ ✅ ✅ ✅   

`$lookup + $unwind` in immediate succession _and then_ `$project`, which means no "16MB error" issues!

```
[
    {
        '$lookup': {
            'from': 'frames.samples.67f720ab45345727feab06bd',
            'let': {'sample_id': '$_id'},
            'pipeline': [
                {'$match': {'$expr': {'$eq': ['$$sample_id', '$_sample_id']}}},
                {'$sort': {'frame_number': 1}},
            ],
            'as': 'frames',
        },
    },
    {'$unwind': '$frames'},
    {'$project': {'frames.frame_number': True}},
    {
        '$project': {
            'values': {
                '$cond': {
                    'if': {'$gt': ['$frames.frame_number', None]},
                    'then': '$frames.frame_number',
                    'else': None,
                },
            },
        },
    },
]
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

## Summary by CodeRabbit

- **New Features**
	- Introduced an advanced option in aggregations that allows direct retrieval of underlying processing pipelines.
  
- **Refactor**
	- Enhanced video sample processing to ensure consistent handling of frame data.
	- Streamlined data view construction for more efficient integration of frame and group information.

- **Bug Fixes**
	- Improved frame label selection logic to correctly include frame-level data when no specific fields are chosen.
	- Adjusted aggregation pipeline construction to properly unwind video frames when needed.

- **Tests**
	- Added tests verifying correct aggregation behavior and pipeline structure for video frames and nested detections.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->


Diff:
diff --git a/fiftyone/core/aggregations.py b/fiftyone/core/aggregations.py
index 4a09ecd0530..f5e341efdef 100644
--- a/fiftyone/core/aggregations.py
+++ b/fiftyone/core/aggregations.py
@@ -3114,6 +3114,10 @@ def _parse_field_and_expr(
         elif not context and not optimize:
             pipeline.append({"$project": {path: True}})
     elif unwind_list_fields:
+        if is_frame_field and unwind_list_fields[0] == "frames":
+            unwind_list_fields.pop(0)
+            pipeline.append({"$unwind": "$frames"})
+
         pipeline.append({"$project": {path: True}})
 
     (
diff --git a/fiftyone/core/collections.py b/fiftyone/core/collections.py
index 432591e0ceb..3c2e970c2a1 100644
--- a/fiftyone/core/collections.py
+++ b/fiftyone/core/collections.py
@@ -10244,7 +10244,7 @@ def _add_view_stage(self, stage):
         """
         raise NotImplementedError("Subclass must implement _add_view_stage()")
 
-    def aggregate(self, aggregations):
+    def aggregate(self, aggregations, _mongo=False):
         """Aggregates one or more
         :class:`fiftyone.core.aggregations.Aggregation` instances.
 
@@ -10297,6 +10297,9 @@ def aggregate(self, aggregations):
             idx_map[idx] = len(pipelines)
             pipelines.append(pipeline)
 
+        if _mongo:
+            return pipelines[0] if scalar_result else pipelines
+
         # Run all aggregations
         _results = foo.aggregate(self._dataset._sample_collection, pipelines)
 
diff --git a/fiftyone/core/stages.py b/fiftyone/core/stages.py
index 021bd16a98a..58e05fba7bb 100644
--- a/fiftyone/core/stages.py
+++ b/fiftyone/core/stages.py
@@ -6539,7 +6539,7 @@ def _needs_frames(self, sample_collection):
             return False
 
         if self._field_names is None:
-            return False
+            return True
 
         # @todo consider `meta_filter` here too?
         return any(
diff --git a/fiftyone/core/view.py b/fiftyone/core/view.py
index 80b9dafe452..6e2d43a46da 100644
--- a/fiftyone/core/view.py
+++ b/fiftyone/core/view.py
@@ -1655,66 +1655,39 @@ def _pipeline(
         if _attach_frames_idx is None and (attach_frames or frames_only):
             _attach_frames_idx = len(_pipelines)
 
-        #######################################################################
-        # Insert frame lookup pipeline(s) if needed
-        #######################################################################
-
         if _attach_frames_idx1 is not None and _attach_frames_idx is not None:
             _attach_frames_idx = _attach_frames_idx1
 
-        if _attach_frames_idx0 is not None and _attach_frames_idx is not None:
-            # Two lookups are required; manually do the **last** one and rely
-            # on dataset._pipeline() to do the first one
-            attach_frames = True
-            _pipeline = self._dataset._attach_frames_pipeline(
-                limit=limit_frames, support=support
-            )
-            _pipelines.insert(_attach_frames_idx, _pipeline)
-        elif _found_flattened_videos and _attach_frames_idx is not None:
-            # Must manually attach frames after the group $lookup
-            attach_frames = None  # special syntax: frames already attached
-            _pipeline = self._dataset._attach_frames_pipeline(
-                limit=limit_frames, support=support
-            )
-            _pipelines.insert(_attach_frames_idx, _pipeline)
-        elif _attach_frames_idx0 is not None or _attach_frames_idx is not None:
-            # Exactly one lookup is required; rely on dataset._pipeline() to
-            # do it
-            attach_frames = True
+        _inserts = []
 
-        # @todo use the optimization below instead, which injects frames as
-        # late as possible in the pipeline. We can't currently use it because
-        # there's some issue with poster frames in the App if the frames are
-        # not attached first...
+        # Handles incrementing idx values in case of multiple inserts
+        def _adjust(idx):
+            idx += sum(i < idx for i in _inserts)
+            _inserts.append(idx)
+            return idx
 
-        """
+        # Insert frame lookup pipeline(s) if needed
         if _attach_frames_idx0 is not None or _attach_frames_idx is not None:
             attach_frames = None  # special syntax: frames already attached
 
             if _attach_frames_idx0 is not None:
                 _pipeline = self._dataset._attach_frames_pipeline(
-                    support=support
+                    limit=limit_frames, support=support
                 )
-                _pipelines.insert(_attach_frames_idx0, _pipeline)
+                _pipelines.insert(_adjust(_attach_frames_idx0), _pipeline)
 
             if _attach_frames_idx is not None:
-                if _attach_frames_idx0 is not None:
-                    _attach_frames_idx += 1
-
                 _pipeline = self._dataset._attach_frames_pipeline(
-                    support=support
+                    limit=limit_frames, support=support
                 )
-                _pipelines.insert(_attach_frames_idx, _pipeline)
-        """
-
-        #######################################################################
+                _pipelines.insert(_adjust(_attach_frames_idx), _pipeline)
 
         # Insert group lookup pipeline if needed
         if _attach_groups_idx is not None:
             _pipeline = self._dataset._attach_groups_pipeline(
                 group_slices=_group_slices
             )
-            _pipelines.insert(_attach_groups_idx, _pipeline)
+            _pipelines.insert(_adjust(_attach_groups_idx), _pipeline)
 
         if pipeline is not None:
             _pipelines.append(pipeline)
diff --git a/tests/unittests/aggregation_tests.py b/tests/unittests/aggregation_tests.py
index 1b88809e256..8a9e9495d22 100644
--- a/tests/unittests/aggregation_tests.py
+++ b/tests/unittests/aggregation_tests.py
@@ -1196,6 +1196,142 @@ def test_needs_frames(self):
         values = dataset.values(expr1 & expr2)
         self.assertListEqual(values, [False, False])
 
+    @drop_datasets
+    def test_frames_unwind(self):
+        #
+        # These tests ensure that:
+        # 1. $lookup + $unwind are adjacent when possible
+        # 2. $lookup occurs as late as possible in pipelines
+        #
+
+        sample1 = fo.Sample(filepath="video1.mp4", int=1)
+        sample1.frames[1] = fo.Frame(
+            detections=fo.Detections(
+                detections=[
+                    fo.Detection(label="rabbit"),
+                    fo.Detection(label="squirrel"),
+                ]
+            )
+        )
+        sample1.frames[7] = fo.Frame()
+
+        sample2 = fo.Sample(filepath="video2.mp4", int=2)
+        sample2.frames[1] = fo.Frame(
+            detections=fo.Detections(
+                detections=[
+                    fo.Detection(label="cat"),
+                    fo.Detection(label="dog"),
+                ]
+            )
+        )
+        sample2.frames[3] = fo.Frame()
+        sample2.frames[5] = fo.Frame(
+            detections=fo.Detections(
+                detections=[
+                    fo.Detection(label="dog"),
+                    fo.Detection(label="fox"),
+                ]
+            )
+        )
+
+        dataset = fo.Dataset()
+        dataset.add_samples([sample1, sample2])
+
+        # Full dataset
+
+        agg = fo.Count("frames")
+        value = dataset.aggregate(agg)
+        pipeline = dataset.aggregate(agg, _mongo=True)
+
+        self.assertEqual(value, 5)
+        self.assertTrue("$lookup" in pipeline[0])
+        self.assertTrue("$unwind" in pipeline[1])
+
+        agg = fo.Max("frames.frame_number")
+        value = dataset.aggregate(agg)
+        pipeline = dataset.aggregate(agg, _mongo=True)
+
+        self.assertEqual(value, 7)
+        self.assertTrue("$lookup" in pipeline[0])
+        self.assertTrue("$unwind" in pipeline[1])
+
+        agg = fo.Distinct("frames.detections.detections.label")
+        value = dataset.aggregate(agg)
+        pipeline = dataset.aggregate(agg, _mongo=True)
+
+        self.assertSetEqual(
+            set(value), {"cat", "dog", "fox", "rabbit", "squirrel"}
+        )
+        self.assertTrue("$lookup" in pipeline[0])
+        self.assertTrue("$unwind" in pipeline[1])
+
+        agg = fo.Values("frames[].detections")
+        value = dataset.aggregate(agg)
+        pipeline = dataset.aggregate(agg, _mongo=True)
+
+        self.assertEqual(len(value), 5)
+        self.assertTrue("$lookup" in pipeline[0])
+        self.assertTrue("$unwind" in pipeline[1])
+
+        agg = fo.Values("frames.detections.detections.label", unwind=True)
+        value = dataset.aggregate(agg)
+        pipeline = dataset.aggregate(agg, _mongo=True)
+
+        self.assertListEqual(
+            value, ["rabbit", "squirrel", "cat", "dog", "dog", "fox"]
+        )
+        self.assertTrue("$lookup" in pipeline[0])
+        self.assertTrue("$unwind" in pipeline[1])
+
+        # View that only involves sample-level filtering
+
+        view = dataset.match(F("int") > 1)
+
+        agg = fo.Count("frames")
+        value = view.aggregate(agg)
+        pipeline = view.aggregate(agg, _mongo=True)
+
+        self.assertEqual(value, 3)
+        self.assertTrue("$match" in pipeline[0])
+        self.assertTrue("$lookup" in pipeline[1])
+        self.assertTrue("$unwind" in pipeline[2])
+
+        agg = fo.Max("frames.frame_number")
+        value = view.aggregate(agg)
+        pipeline = view.aggregate(agg, _mongo=True)
+
+        self.assertEqual(value, 5)
+        self.assertTrue("$match" in pipeline[0])
+        self.assertTrue("$lookup" in pipeline[1])
+        self.assertTrue("$unwind" in pipeline[2])
+
+        agg = fo.Distinct("frames.detections.detections.label")
+        value = view.aggregate(agg)
+        pipeline = view.aggregate(agg, _mongo=True)
+
+        self.assertSetEqual(set(value), {"cat", "dog", "fox"})
+        self.assertTrue("$match" in pipeline[0])
+        self.assertTrue("$lookup" in pipeline[1])
+        self.assertTrue("$unwind" in pipeline[2])
+
+        agg = fo.Values("frames[].detections")
+        value = view.aggregate(agg)
+        pipeline = view.aggregate(agg, _mongo=True)
+
+        self.assertEqual(len(value), 3)
+        self.assertTrue("$match" in pipeline[0])
+        self.assertTrue("$lookup" in pipeline[1])
+        self.assertTrue("$unwind" in pipeline[2])
+
+        agg = fo.Values("frames.detections.detections.label", unwind=True)
+        value = view.aggregate(agg)
+        pipeline = view.aggregate(agg, _mongo=True)
+
+        self.assertListEqual(value, ["cat", "dog", "dog", "fox"])
+        self.assertTrue("$match" in pipeline[0])
+        self.assertTrue("$lookup" in pipeline[1])
+        self.assertTrue("$unwind" in pipeline[2])
+
     @drop_datasets
     def test_serialize(self):
         bbox_area = F("bounding_box")[2] * F("bounding_box")[3]
diff --git a/tests/unittests/server_samples_tests.py b/tests/unittests/server_samples_tests.py
index aff00d8bdac..b87e8567c77 100644
--- a/tests/unittests/server_samples_tests.py
+++ b/tests/unittests/server_samples_tests.py
@@ -106,7 +106,9 @@ async def test_full_frames_lookup(self, dataset: fo.Dataset):
 
 async def _resolve_lookup_stage(view: fo.DatasetView):
     pipeline = await get_samples_pipeline(view, None)
-    return pipeline[0]
+    for stage in pipeline:
+        if "$lookup" in stage:
+            return stage
 
 
 def _get_expected_lookup_stage(view: fo.DatasetView, clips=False, limit=False):


Reviews:
user: login: coderabbitai[bot]
body: **Actionable comments posted: 0**

<details>
<summary>🧹 Nitpick comments (1)</summary><blockquote>

<details>
<summary>fiftyone/core/view.py (1)</summary><blockquote>

`1645-1649`: **Ensure robust handling of multiple insertions**

The `_adjust(idx)` function correctly adjusts new insertion indices based on previous insertions. However, consider documenting how it behaves if insertion indices are out of ascending order, as that could affect insertion positions in more complex scenarios.

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: .coderabbit.yaml**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between 7ec992862705ae753de87d8b09f0476710c62e72 and d6298068f36feff4491124b1ebc08bd1cc1a1497.

</details>

<details>
<summary>📒 Files selected for processing (5)</summary>

* `fiftyone/core/aggregations.py` (1 hunks)
* `fiftyone/core/collections.py` (2 hunks)
* `fiftyone/core/stages.py` (1 hunks)
* `fiftyone/core/view.py` (1 hunks)
* `tests/unittests/aggregation_tests.py` (1 hunks)

</details>

<details>
<summary>🧰 Additional context used</summary>

<details>
<summary>🧬 Code Graph Analysis (1)</summary>

<details>
<summary>tests/unittests/aggregation_tests.py (2)</summary><blockquote>

<details>
<summary>fiftyone/core/aggregations.py (4)</summary>

* `Count` (448-597)
* `Max` (1619-1738)
* `Distinct` (844-988)
* `Values` (2693-2914)

</details>
<details>
<summary>fiftyone/core/collections.py (1)</summary>

* `aggregate` (10213-10296)

</details>

</blockquote></details>

</details>

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (12)</summary>

* GitHub Check: test / test-python (ubuntu-latest-m, 3.12)
* GitHub Check: test / test-python (ubuntu-latest-m, 3.11)
* GitHub Check: test-windows / test-python (windows-latest, 3.12)
* GitHub Check: test / test-python (ubuntu-latest-m, 3.10)
* GitHub Check: test-windows / test-python (windows-latest, 3.11)
* GitHub Check: test-windows / test-python (windows-latest, 3.10)
* GitHub Check: test-windows / test-python (windows-latest, 3.9)
* GitHub Check: test / test-app
* GitHub Check: lint / eslint
* GitHub Check: e2e / test-e2e
* GitHub Check: build / build
* GitHub Check: build

</details>

<details>
<summary>🔇 Additional comments (14)</summary><blockquote>

<details>
<summary>fiftyone/core/collections.py (2)</summary>

`10266-10268`: **Implementation correctly handles both scalar and list results.**

The conditional block properly implements the return of raw MongoDB pipelines when `_mongo=True`, maintaining the same scalar/list result pattern as the actual aggregation results.

---

`10213-10213`: <details>
<summary>✅ Verification successful</summary>

**Good addition of the `_mongo` parameter for debugging and optimization.**

The new `_mongo` parameter allows returning raw MongoDB aggregation pipelines instead of executing them, which aligns well with the PR's optimization goals. This provides a helpful way to inspect pipeline structures and understand how frame lookups are being processed.


---

🏁 Script executed:

```shell
#!/bin/bash
# Check if there are any existing usages of this parameter in the codebase
rg -A 1 "\.aggregate\(.+_mongo\s*=\s*True" --type py
```

Length of output: 1366

---



</details>

**Approval: The `_mongo` Parameter Implementation is Verified and Effective**  
The addition of the `_mongo` parameter in the `aggregate` method is a welcome enhancement that aligns with the PR's optimization and debugging goals. Testing confirms that passing `_mongo=True` now reliably returns the raw MongoDB aggregation pipelines, as evidenced by its consistent usage in `tests/unittests/aggregation_tests.py`. There are no outstanding issues with this change.

</details>
<details>
<summary>fiftyone/core/stages.py (1)</summary>

`542-542`: **Changed behavior in `_needs_frames` method to optimize frame lookups.**

The method now returns `True` when `self._field_names` is `None` instead of the previous behavior returning `False`. This change supports the PR's objective of optimizing frame lookups in aggregation pipelines by ensuring frames are always attached when no specific fields are specified in a SelectFields stage.

</details>
<details>
<summary>fiftyone/core/aggregations.py (1)</summary>

`3117-3120`: **Optimized frame field handling improves aggregation pipeline efficiency**

This change adds specialized handling for frame fields during aggregation pipeline construction. By checking if the first element of `unwind_list_fields` is "frames" and immediately adding an `$unwind` operation rather than deferring it, the pipeline can optimize the ordering of operations.


This change helps ensure that `$lookup` and `$unwind` stages are kept adjacent in the pipeline, which is more efficient for MongoDB execution and reduces the chance of encountering 16MB document size limits referenced in the PR objectives.

</details>
<details>
<summary>tests/unittests/aggregation_tests.py (4)</summary>

`1199-1206`: **Well-documented test for frame unwinding optimization**

The test header clearly states what's being tested - specifically validating that `$lookup` and `$unwind` operations are adjacent when possible, and that `$lookup` occurs as late as possible in the pipeline.


This documentation makes the test's purpose very clear and aligns with the optimization made in the aggregations.py file.

---

`1207-1238`: **Good test data setup with representative video samples**

The test creates two video samples with different frame numbers and detection content, providing a realistic dataset for testing frame-based aggregations.


The test data includes frames with varying positions and content, which is important for thoroughly testing the unwinding optimization across different scenarios.

---

`1239-1285`: **Comprehensive validation of full dataset aggregations**

This section thoroughly tests multiple aggregation operations on the full dataset, validating both the results and the structure of the generated MongoDB pipeline.


The tests cover a good variety of aggregation types (Count, Max, Distinct, Values) and verify both the returned values and the pipeline structure, ensuring that `$lookup` and `$unwind` operations are correctly ordered.

---

`1286-1334`: **Thorough testing of view-based aggregations**

This section tests the same aggregations but on a filtered view, ensuring the optimization works correctly when sample-level filtering is applied.


The tests validate that operations are correctly ordered (`$match` followed by `$lookup` and `$unwind`) in the filtered view scenario, which is important for ensuring query efficiency on larger datasets.

</details>
<details>
<summary>fiftyone/core/view.py (6)</summary>

`1643-1643`: **Initialize insertion index tracking**

Introducing `_inserts = []` to track insertion points is straightforward and sets the stage for orderly pipeline insertions.

---

`1651-1651`: **Clear and meaningful comment**

The added comment enhances clarity about upcoming frame lookups. No issues noted.

---

`1657-1657`: **Verifying `_attach_frames_pipeline` parameters**

The `limit=limit_frames, support=support` parameters align with variable naming and appear intended. Looks good.

---

`1663-1663`: **Repeat of consistent parameter usage**

Again, passing `limit=limit_frames, support=support` is coherent with prior usage and consistent with method signatures.

---

`1665-1665`: **Correct insertion index application**

Inserting the pipeline at `_adjust(_attach_frames_idx)` is a valid approach to account for previously inserted indices. No concerns.

---

`1672-1672`: **Proper group pipeline insertion**

Using `_adjust(_attach_groups_idx)` ensures the group lookup pipeline is slotted in the correct location. Implementation looks good.

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T02:06:17Z
user: login: brimoor
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-10T02:07:44Z
user: login: benjaminpkane
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-11T14:40:07Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 1**

<details>
<summary>📜 Review details</summary>

**Configuration used: .coderabbit.yaml**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between d6298068f36feff4491124b1ebc08bd1cc1a1497 and 2c866b6b3bf25cf4d4c207bba7e9cf3526366df5.

</details>

<details>
<summary>📒 Files selected for processing (6)</summary>

* `fiftyone/core/aggregations.py` (1 hunks)
* `fiftyone/core/collections.py` (2 hunks)
* `fiftyone/core/stages.py` (1 hunks)
* `fiftyone/core/view.py` (1 hunks)
* `tests/unittests/aggregation_tests.py` (1 hunks)
* `tests/unittests/server_samples_tests.py` (1 hunks)

</details>

<details>
<summary>🚧 Files skipped from review as they are similar to previous changes (5)</summary>

* fiftyone/core/aggregations.py
* tests/unittests/aggregation_tests.py
* fiftyone/core/view.py
* fiftyone/core/stages.py
* fiftyone/core/collections.py

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (10)</summary>

* GitHub Check: test-windows / test-python (windows-latest, 3.12)
* GitHub Check: test-windows / test-python (windows-latest, 3.11)
* GitHub Check: test-windows / test-python (windows-latest, 3.10)
* GitHub Check: test-windows / test-python (windows-latest, 3.9)
* GitHub Check: test / test-python (ubuntu-latest-m, 3.12)
* GitHub Check: test / test-python (ubuntu-latest-m, 3.11)
* GitHub Check: test / test-python (ubuntu-latest-m, 3.10)
* GitHub Check: test / test-python (ubuntu-latest-m, 3.9)
* GitHub Check: e2e / test-e2e
* GitHub Check: build

</details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-19T18:35:52Z
user: login: brimoor
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-19T18:46:00Z
user: login: benjaminpkane
body: LGTM 🐎 
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-21T15:40:24Z

Review comments:
diff_hunk: @@ -1637,66 +1637,39 @@ def _pipeline(
         if _attach_frames_idx is None and (attach_frames or frames_only):
             _attach_frames_idx = len(_pipelines)
 
-        #######################################################################
-        # Insert frame lookup pipeline(s) if needed
-        #######################################################################
-
         if _attach_frames_idx1 is not None and _attach_frames_idx is not None:
             _attach_frames_idx = _attach_frames_idx1
 
-        if _attach_frames_idx0 is not None and _attach_frames_idx is not None:
-            # Two lookups are required; manually do the **last** one and rely
-            # on dataset._pipeline() to do the first one
-            attach_frames = True
-            _pipeline = self._dataset._attach_frames_pipeline(
-                limit=limit_frames, support=support
-            )
-            _pipelines.insert(_attach_frames_idx, _pipeline)
-        elif _found_flattened_videos and _attach_frames_idx is not None:
-            # Must manually attach frames after the group $lookup
-            attach_frames = None  # special syntax: frames already attached
-            _pipeline = self._dataset._attach_frames_pipeline(
-                limit=limit_frames, support=support
-            )
-            _pipelines.insert(_attach_frames_idx, _pipeline)
-        elif _attach_frames_idx0 is not None or _attach_frames_idx is not None:
-            # Exactly one lookup is required; rely on dataset._pipeline() to
-            # do it
-            attach_frames = True
+        _inserts = []
 
-        # @todo use the optimization below instead, which injects frames as
path: fiftyone/core/view.py
user: login: brimoor
body: @benjaminpkane I first tried to make this optimization awhile ago, but, as documented in the deleted code here, when I initially tried it, there was some App rendering issue where poster frames would not show labels.

However, I am not seeing any issues with first-frame label rendering now, so it seems like whatever the previous limitation was has now been lifted! 🤞🤞🤞

A couple questions for you:
- Do you know what might have changed to resolve the symptom reported in this `@todo`?
- Do you foresee any issues with `$lookup` potentially occurring later in pipelines now? 
created_at: 2025-04-10T02:07:43Z
author_association: MEMBER
diff_hunk: @@ -1637,66 +1637,39 @@ def _pipeline(
         if _attach_frames_idx is None and (attach_frames or frames_only):
             _attach_frames_idx = len(_pipelines)
 
-        #######################################################################
-        # Insert frame lookup pipeline(s) if needed
-        #######################################################################
-
         if _attach_frames_idx1 is not None and _attach_frames_idx is not None:
             _attach_frames_idx = _attach_frames_idx1
 
-        if _attach_frames_idx0 is not None and _attach_frames_idx is not None:
-            # Two lookups are required; manually do the **last** one and rely
-            # on dataset._pipeline() to do the first one
-            attach_frames = True
-            _pipeline = self._dataset._attach_frames_pipeline(
-                limit=limit_frames, support=support
-            )
-            _pipelines.insert(_attach_frames_idx, _pipeline)
-        elif _found_flattened_videos and _attach_frames_idx is not None:
-            # Must manually attach frames after the group $lookup
-            attach_frames = None  # special syntax: frames already attached
-            _pipeline = self._dataset._attach_frames_pipeline(
-                limit=limit_frames, support=support
-            )
-            _pipelines.insert(_attach_frames_idx, _pipeline)
-        elif _attach_frames_idx0 is not None or _attach_frames_idx is not None:
-            # Exactly one lookup is required; rely on dataset._pipeline() to
-            # do it
-            attach_frames = True
+        _inserts = []
 
-        # @todo use the optimization below instead, which injects frames as
path: fiftyone/core/view.py
user: login: benjaminpkane
body: It is likely due to https://github.com/voxel51/fiftyone/pull/5450. I don't see any issue with this PR. Though it looks like some tests need to be updated
created_at: 2025-04-11T14:40:07Z
author_association: MEMBER
diff_hunk: @@ -106,7 +106,9 @@ async def test_full_frames_lookup(self, dataset: fo.Dataset):
 
 async def _resolve_lookup_stage(view: fo.DatasetView):
     pipeline = await get_samples_pipeline(view, None)
-    return pipeline[0]
+    for stage in pipeline:
+        if "$lookup" in stage:
+            return stage
path: tests/unittests/server_samples_tests.py
user: login: coderabbitai[bot]
body: _💡 Verification agent_

<details>
<summary>🧩 Analysis chain</summary>

**Good optimization for proper lookup stage identification.**

The function now correctly searches for the first `$lookup` stage in the pipeline rather than assuming it's always the first stage. This change aligns well with the PR's objective of optimizing frame lookups in aggregation pipelines.


Consider handling the edge case where no lookup stage exists in the pipeline:

---

🏁 Script executed:

```shell
#!/bin/bash
# Check if there are test cases or scenarios where no lookup stage exists in the pipeline
rg -A 5 -B 5 "_resolve_lookup_stage" tests/
```

Length of output: 6106

---



</details>

**Ensure a clear fallback when no `$lookup` stage is found**

The helper `_resolve_lookup_stage` currently returns `None` if no `$lookup` stage exists in the pipeline. To avoid silent failures, add an explicit fallback (e.g. raise an error or return a sentinel) and cover this edge case with a unit test.

Locations to update:
- tests/unittests/server_samples_tests.py: `_resolve_lookup_stage` definition (around lines 108–111)

Suggested changes:

```diff
 async def _resolve_lookup_stage(view: fo.DatasetView):
     pipeline = await get_samples_pipeline(view, None)
     for stage in pipeline:
         if "$lookup" in stage:
             return stage
-    # implicit None return
+    raise LookupError("no $lookup stage found in pipeline")
```

Add a new test in the same file:

```python
async def test_no_lookup_stage(self, dataset: fo.Dataset):
    # force a pipeline without any $lookup stage
    view = dataset.match({"_id": {"$exists": True}})
    # monkey‑patch get_samples_pipeline to return a pipeline without lookups
    monkeypatch.setattr(
        server_samples, "get_samples_pipeline", lambda _view, _: [{"$match": {}}]
    )
    with pytest.raises(LookupError):
        await _resolve_lookup_stage(view)
```

<!-- This is an auto-generated comment by CodeRabbit -->
created_at: 2025-04-19T18:35:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1637,66 +1637,39 @@ def _pipeline(
         if _attach_frames_idx is None and (attach_frames or frames_only):
             _attach_frames_idx = len(_pipelines)
 
-        #######################################################################
-        # Insert frame lookup pipeline(s) if needed
-        #######################################################################
-
         if _attach_frames_idx1 is not None and _attach_frames_idx is not None:
             _attach_frames_idx = _attach_frames_idx1
 
-        if _attach_frames_idx0 is not None and _attach_frames_idx is not None:
-            # Two lookups are required; manually do the **last** one and rely
-            # on dataset._pipeline() to do the first one
-            attach_frames = True
-            _pipeline = self._dataset._attach_frames_pipeline(
-                limit=limit_frames, support=support
-            )
-            _pipelines.insert(_attach_frames_idx, _pipeline)
-        elif _found_flattened_videos and _attach_frames_idx is not None:
-            # Must manually attach frames after the group $lookup
-            attach_frames = None  # special syntax: frames already attached
-            _pipeline = self._dataset._attach_frames_pipeline(
-                limit=limit_frames, support=support
-            )
-            _pipelines.insert(_attach_frames_idx, _pipeline)
-        elif _attach_frames_idx0 is not None or _attach_frames_idx is not None:
-            # Exactly one lookup is required; rely on dataset._pipeline() to
-            # do it
-            attach_frames = True
+        _inserts = []
 
-        # @todo use the optimization below instead, which injects frames as
path: fiftyone/core/view.py
user: login: brimoor
body: Good catch, test has been updated!
created_at: 2025-04-19T18:46:00Z
author_association: MEMBER

PR comments:
user: login: coderabbitai[bot]
created_at: 2025-04-10T02:03:19Z
body: <!-- This is an auto-generated comment: summarize by coderabbit.ai -->
<!-- walkthrough_start -->

## Walkthrough
This pull request introduces specific handling for frame fields within aggregation pipelines. It adds a conditional check in the field parsing function to manage unwinding of frame lists, allows returning raw MongoDB pipelines via a new parameter in the aggregation method, updates frame requirement logic in stage processing, and restructures the view pipeline construction. Additionally, a new test method validates the correct ordering of pipeline stages for video sample aggregations.

## Changes
| File(s) | Summary of Changes |
|---------|--------------------|
| `fiftyone/core/aggregations.py` | Added a conditional branch in `_parse_field_and_expr` to check for `"frames"` in `unwind_list_fields`, removing it and appending a `$unwind` stage specifically for frame fields. |
| `fiftyone/core/collections.py` | Modified `SampleCollection.aggregate()` to accept an optional `_mongo` parameter that, when `True`, returns raw MongoDB aggregation pipelines instead of executing them. |
| `fiftyone/core/stages.py` | Changed `SelectFields._needs_frames()` to return `True` when `_field_names` is `None`, indicating frame data is required if no fields are specified. |
| `fiftyone/core/view.py` | Refactored `DatasetView._pipeline()` to unify and simplify insertion of frame and group lookup pipelines using a dynamic index adjustment mechanism instead of multiple conditional blocks. |
| `tests/unittests/aggregation_tests.py` | Added `test_frames_unwind()` method to test aggregation pipelines on video samples with frames and nested detections, verifying correct `$lookup` and `$unwind` stage placement and aggregation results. |
| `tests/unittests/server_samples_tests.py` | Modified `_resolve_lookup_stage()` to return the first pipeline stage containing `$lookup` instead of always returning the first stage. |

## Sequence Diagram(s)
```mermaid
sequenceDiagram
    participant U as User
    participant SC as SampleCollection.aggregate
    participant AP as Aggregation Pipeline Builder
    participant VD as DatasetView._pipeline
    participant VS as ViewStage._needs_frames

    U->>SC: Call aggregate(aggregations, _mongo)
    SC->>AP: Process aggregation pipeline (_parse_field_and_expr)
    AP->>AP: Check if "frames" is first in unwind_list_fields
    AP-->>AP: Remove "frames" and append $unwind stage
    SC->>VD: Construct view pipeline with dynamic insertion
    VD->>VS: Check frame requirements
    VS-->>VD: Return True (frames needed)
    VD-->>SC: Return constructed pipeline
    SC-->>U: Return raw pipeline (if _mongo=True)
```

## Possibly related PRs
- **voxel51/fiftyone#5538**: Modifications to `_parse_field_and_expr` in aggregations align with changes in the Count class’s constructor and `to_mongo` method.
- **voxel51/fiftyone#5450**: Adjustments in frame processing and pipeline parameter handling are closely related to the frame-handling modifications in this PR.
- **voxel51/fiftyone#5569**: Related changes to `_parse_field_and_expr` for aggregation pipeline optimization and parameter handling.

## Suggested reviewers
- benjaminpkane

## Poem
> I’m a little rabbit with hops so bright,  
> Unwinding frames by day and night.  
> Pipelines adjust with a magical spin,  
> Bugs hop away as smooth workflows begin.  
> With code and carrots, my spirit sings—  
> A joyful dance in our realm of strings! 🐇

<!-- walkthrough_end -->

<!-- announcements_start -->

> [!TIP]
> <details>
> <summary>⚡💬 Agentic Chat (Pro Plan, General Availability)</summary>
> 
> - We're introducing multi-step agentic chat in review comments and issue comments, within and outside of PR's. This feature enhances review and issue discussions with the CodeRabbit agentic chat by enabling advanced interactions, including the ability to create pull requests directly from comments and add commits to existing pull requests.
> 
> </details>

<!-- announcements_end -->
<!-- internal state start -->


<!-- DwQgtGAEAqAWCWBnSTIEMB26CuAXA9mAOYCmGJATmriQCaQDG+Ats2bgFyQAOFk+AIwBWJBrngA3EsgEBPRvlqU0AgfFwA6NPEgQAfACgjoCEYDEZyAAUASpETZWaCrKNwSPbABsvkCiQBHbGlcSHFcLzpIACIAeW5xZngALw8AMyo2SC98fABrbG5kAkgBDzRkL2py5G58RER4AUiULDQiIn8ianh8DERoyAAKW0gzAFYAdgAGcYBKUvlsREpSinhmXL40/AZl6X4sMlhMBngMIjDYD25KHYpmU498NMgMtDYwSKlfds6SbriPrIADu6gQWFw10gADF4GlcLJYuQ3pkSCD8BQ8hoYNDeBtnPIGCcLh5zhJ8F4pMh8AkNilzpcoR5Eoz+K93lkcvlCshzugOl0en0ePBbl5ztIADSQEEIYnZEhoWjFfD2eBEDDw+AMTChW4Ue6PDAMMnMXj4KRsDC4RAy7jOcR7KoULzyfmIU0YZy9UHXfxXDwYRxlPgvVEfA78tCQCTwdEoZDMbAKxCPHyrKGYQNhfC4NC+YPMUPsiNsPmQ6G0agVEiaIwAaRI7vNFEtJGtttaDC82CUHCMAEYcQBRDAk030A1Gp5vTFlkhfEg/AX/QG9fqHWPxkF+ygs6GRC5Q0vM7cJ1CNTXa3U2t2KhpXbNZ0Lhs/V/MresAJhxNmklJ4BupbGu6/T5iaBzhtEg4AGwALIAEKQJQbYUHagxyjqsDoAGtC7I47BRPySAOB4ZhwdMkxbgA4uoAAS2ACBobh4t4vj+EEIQ8G2cZKMgJAAB4fNwkTIEomzgVQ4gXDmtKsskwr9DiACSlbpPA6GhLqKwyjGcYJi+kD4QcGB5m88BeDQYZYJyi7fCQvjnDOQJYLwy69Ms95pNokT0LQwS5jmoloKanantCAAGAAkjy4MSkX2PmpClHWIIkGQkAxdyBTcIlmD0DF2AYGCGC0JFOLuJA5AgqK4qSklFAprg2ABiFprIDFcUJRZWl6T4+ClZc9z2CmHWIGk3goYJohASKYJQvgeAoSaS02pQbIwQhyGoZiiAaIMACCtC0OoG4Fm6MpQqgxKYClZAOP4xQnKE2W5LliWoH097TpibD0CKiAiWJkDdRCTLQkD5qREuK5pJZ1n2nm7DwBd8hPd4MmXB6TGIBB4ho28FTWauQquftRhqTmKxMGVjC1npbSCgCSmtBSVJskJok6uoC6QMVpWnbJBaIGqZTkPDND0BkLByXSSSKa5lV4v4cZLbUYqOQ1eNNWIrUeH5QW7ehxmBSUZ6YkoYavLSyjkzKWEtGeNWNc1+srY9BxGW9PJ5egdNFSV5zlbhHhCbNUutCgrB0KjNCjQw43AWU9zlBg8gWiIYgbldsDLOgFLwELlzG8gBXRxacayb9DxPCxBixFIVA+Ln0geLdpJlwGXOiJHJSRMqQU+Xjqx/GTG5l3TMabN3aTw2c7CQCStASrJ4Z2bDjnGTWUdnnCCJIiidkYliys3OxfiBMEeP01gZToIn0grPQ1CQAA2tdlTnHkAC6QywLgXARQOAAHoQFEHBExDQTBmAgIpDNLw4xBwgPhgfPoJAQHcHYiAqYsw5j130MYcAUAyAA1eGgPAhBSDkGklEGBnYuC8H4MIXukgDhyAUFbFQahNDaF0GAQwJgoBwFQKgbMFCCDEDIHbOhLAGF+DQLVBwTgXCLE4coVQ6gtA6EIUQ0wBhUGInQSApg/gQFjxZuTDQ3BZADmiPYgwFhICHRUlImh1R6DKMeKo8MHdSCIFYu3EkKVlQqnQAoMqZ0+gFlKDkBgeRWiNCUDmSKAB9B06ESCpPho5WgqSCqpK5hQRKk0TSuSCsvJ20IdIeDlJlSKAtg6pIlHjbJ8YvAqk+sgMyGAwAdgSPIcuZ4ckdMTOEuyFlcmynBDmeGWl+ZBzpi00IIz6BXxiSUSK0Q7IDAqpAKmn96a6RQKEfwmxqRZW2WiXZqJZZngaYsvJyy2m5MQIlZZ/sX7cFuGVMuWUADe0RoqNLKtELgQKdnRAAL6JTxu0FkaozwWPXCKbgmtV4eBTpico3zSFsjPA4AQKwuI2gBUCzOvcwWQH+Q6KEXBoBNRIFCmFSV4XK1QJsU6OTnrtz6LgNsvg0g5FqhwrmEozgRCWI8/FUUrmRhuas2U1wsC8zQLi5wZceW9RviC+gHzFUjQmasu0HtWoypuOihq5wex9i9tCXV/ADSs0Ndc3eCBkCejID6fA9cnGHSsnbCeQV3yiBdEpGkrwuaYkjvOLBzQdQrXCPGAJBgoAADk1R+K9oiqsobnDhtLFGigMa+BxvFYms6Bx0oBkeMk/khyuVpBYvY6IRgIBgCMIY2QxjTEYKYBmbOwJrG2IMC2xxlgXFuJkZ4xw3j5C+OCdIQJWVkXVESmwRaZCUkAGVgYkAAMKUkiIOjAiUewVFBBUUGihtRRBKKFU0CR/boBOlE70vh5LnV8AIXIA83L5o3asNJkkiD4ESkMJQPlMb4rVJFGEIsSCRXwZAAA6sqrKqSQNgbGV+IKkUGXBEiq3UGdZYCKEvi1Cgm4zxUFqvBPooGAAiyFV3lLRfVcgQxEALFpjrZqUQRpnggVIJma4lJcYWOcEeg9wzhz2FjQMzBPkUdalqWSSLmYoqwBjKy+1IDbtuGceeaMZTwnCY0C4LRPQFmcKTSxwFUCVyLnQUzpy6yqa1XMm+7GtbkAANz8GZBQMExzeb+Eo9Rw8SBXyvB8xivTKlXjAYY9h1AcGENEZzBusjU42ymjoJq1KqcZRyaAupvElryCTynBq81mlL4OF0w/Uxxc3S+onQG6SQbza5vPV14EhbBJ1GLVEWNTFy0o3EEu1NkB4KkfI1eb0lGPCFA/HQLgkVIN2cBCQLjjk0h6U0+GuYiVABJhFlLbrHdsrC8Ad7b4aZSYZSwAXng14FYJ2o7nsfJFXd0MD1HtYX0T6WBIpdp7dikxgOT37RsZFUdDjU36PB+QKHZi4X+OHXYhxfrXHUOnaNFR87XhZpTVVLN16uXJpSak8gBXsnXPXfN+g9aorbscr3OEryz1VAaDiKwqtPKIEukqzK9yXkdNp/K/KgD1gCDwLUq9kUM3kEy2ebL5HwuqaiOl97iHTORN1ApoyEyqhlHe7Kfc1VzJ06ULQM+/NuBrfoEwZJWuqOdQI4ht1N0Gbqk1AWGVr9TcqEcl3DwnFsCaSiHUrAZl7CGZvJM0Z3pyyhx4paZz9vcQR+4m+aEGv6BnO0JuYqWbaAyjxpZDi7mqNsnw4yz65D04RQ8IgRPOTpbtPoKng4pinp1DpiUI13e9MiOQBTtAc9e5apyBAhgT5QhKGskkKrougsLk3r4D809lihHvtQfMxIBPznuez49uAucdLeay0g7XnGdYLT1jwSg+sFtk0N6No3S3jYTZN5Nbas2zOGGtuiADO8qe2t2lee6qS/aF+G4n2/I32nU5+nOo+IOWUKOfakOGO0gw6Z6i69AJQ7uoOXuiUseWUN2TaEueSveN+aWKu3uUmNAMmSWb2KwFUCOraSOHaBi8IRiqOvacCO4WOXB46zieO0itCM6ROpYpOy6jal8PkYge0OYUmlA5Sc+Ca68aIymnQS03A2Q70hQdUvmVa4IrOHgaScWkoTOm6reWUjGNYX4AAajuDzheqpG5ucgcO5GrF5PICsBktUBEqdK5DEs0LsHkM9K/JUgJroTlCYTYSiOocWkGgILWADKDvkoAqFLAOAeWKkkXIJNMJlmkofnkQUdIEUbQIJJluXGkjsMVHkkKoftInknxCQPUHsmpNJhXicq0PyooCmAcDGB8mkqkbaIlObFQPEokhoQ5obqMVPEvI5AaG8GXuUuUbQEIHvkMMUZ9kZMqLsXjBWKdB1EFA+mtCsrGusPOJMRPCVv0GarJP3r3PMWkSKJbKsBQcmFZGKC0MkaMQGJMXQPXFVKbsYYYUCeHtbrVKCSzpGuCEBjkUfvkTsjUSUYlPOOUbkcSFUWAcUV0vYHWA7DMpFBUfiTsrCnWHhowdMWqMHDqCETsungWP4MqIMnidcNnjCPOEqAqD0mAIwa0EoIJMRm8e3n0MXHzIkdCZVmSBPhyZHAtDhJFBKEkLgM9hqeoASflAHA4N8tGs9oacNrgPqSzv0BoVEK/EijsXvsRGVEJA7n4ULgoDHOtLQGAEtNpIoAeLEQVIbH8eIKJO3LWAJENuKuoPeKgGcu2PbkYNuvSC6CLkJm2CYXKaYRih8eUvsFqtsSceaRsWUsBCUJMegKEGeG8WIKKUJKLtkZSfkfobyJicSV+OCe6owIuowOwM3CkMsZ4hsNzNymoVaZ8VgDoZGHoemYYZmTCWosaO0GyA8SKEyRcbQLIKnsyT4LIDKP4O1GyMGQCbypEuERbqqeEsVLepaSsOOSRh3EgMwPfv6tZE/jmi/nmv1puB/mad/p4PGgvv/tNumpmouqqNTBqEtvrBGtNL+QDD/gBRWlNogAFt9GBNZO+kYfPg4dYQqXYTll9rzp1E4Z+HWG4eiBgWDvwd2oIZDgZCCPgbKFepytec2oju2kYDQKcSAleYAiEIgOYkdq5KklxbaKIWOrjlOtIYTnOnIWBUYIdNVIZNxIXllKJQSakrqolCcDIBlG0CdHeu+Y4c4XWNAPxR4XzriBysAU3LelqqxsBECalCcGrDZNvKRV2LTPmOcGyJ0WqFDKGaCDMqyeXFVpHMvkDspPsqEO3qIDeITKJTEZWdCPRhcPgMxvduUrgZ1NFHKRaVlMCo8vlAGMcaFIvBQXUA0E0C0EMi9AVXlWMu1DaZUCEVepVY0M0GSOpFmZKA7qJYwMqV7BiNuEoP5XukFSeKyV5SXr5d6vnGbhzv5HWJFdVjwHcH9EmNBqGZlcBLbF+R6imDhFekwMVApjsjKPDJEuVh4I8IJBsI4HzEWKGIzEXnWOsB5LJKdFXmUsZMtSetkKHu9jiHyXwAKUdUJTnAMRehoVqtWbgFVjBTRrXuQPQBIAWNfMprTHMswFqu5CsJBMpt8dbPVVCflYHILLCslFGN1fjtJUCQ7qpSLIivxYGCsDtQNiKDGPDJ1lEPRQ4Tvl+E8Y9EHqEDGF1NQD1LgTxKIHQHalYblaTcpuTcHJTfCiauXGjRKGtvZRDSKDpl2OXE5XxnrCVYnJbIyG1kYOYB1q+eTMGr1mGnbT+V/vBf+RNjaJWimlAHNvYaEuthdiQK8OpRibqpAWkIgVgMgcZR5WZacZRYlTxVqHxdxQ5X0CJfxfgWITwZxfxYneoAnbeU3KkgFWJOnaceJTjhOpIe4pHF4oSHJXdNNhCZscBGkk9JSFIM0lCcXVTeQcxTep3kFPHjkKSHwCQbMppDfFLfnhahxitqtKeV+m1vsuBEqP0bzGZPCbbXaumUQEdT4LftTcFAqcpiQZ5pPTFVTYvhEt5ZuGLQrb7IlHkM2OyhPt2QWNZLPvgNhSULFUZtufeEJPyqFMlQbFCYfbckpmeE5dJP6E+G0A0I4MuaEGIl4CCGgLIOffMrgfXNbQ/rbd1kZa/o7UGs7SNq7WWn/h7chYARml2Y3eBQSpBdQNBYNnBfwAhe7UmlWlbrWiQGxdwbokIitFuhIlQlIR4u6fIrRjJfXRwq7hojwtovwoIsQlI7qUXGAYLuiHQD3Y6Co3olALQLBN+AAJwAAc0wsE5jaQAAzLBGkIHWkAACzOOmODiDjfjOMCCDgkACAMDTDmMCC0CDiJyDhoCDhuPUSELCPfgMDmOwSwQCBJO2MCBpDfjjAMAuO0DOMMDfhUSqBoCTAkCmNZO2PjDfiwT2OwS0BpDjAGMGDCP0IaMqipLaPpR5KkIqNAA=== -->

<!-- internal state end -->
<!-- finishing_touch_checkbox_start -->

<details>
<summary>✨ Finishing Touches</summary>

- [ ] <!-- {"checkboxId": "7962f53c-55bc-4827-bfbf-6a18da830691"} --> 📝 Generate Docstrings

</details>

<!-- finishing_touch_checkbox_end -->
<!-- tips_start -->

---



<details>
<summary>🪧 Tips</summary>

### Chat

There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai?utm_source=oss&utm_medium=github&utm_campaign=voxel51/fiftyone&utm_content=5705):

- Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  - `I pushed a fix in commit <commit_id>, please review it.`
  - `Generate unit testing code for this file.`
  - `Open a follow-up GitHub issue for this discussion.`
- Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples:
  - `@coderabbitai generate unit testing code for this file.`
  -	`@coderabbitai modularize this function.`
- PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  - `@coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.`
  - `@coderabbitai read src/utils.ts and generate unit testing code.`
  - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.`
  - `@coderabbitai help me debug CodeRabbit configuration file.`

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

### CodeRabbit Commands (Invoked using PR comments)

- `@coderabbitai pause` to pause the reviews on a PR.
- `@coderabbitai resume` to resume the paused reviews.
- `@coderabbitai review` to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
- `@coderabbitai full review` to do a full review from scratch and review all the files again.
- `@coderabbitai summary` to regenerate the summary of the PR.
- `@coderabbitai generate docstrings` to [generate docstrings](https://docs.coderabbit.ai/finishing-touches/docstrings) for this PR.
- `@coderabbitai resolve` resolve all the CodeRabbit review comments.
- `@coderabbitai configuration` to show the current CodeRabbit configuration for the repository.
- `@coderabbitai help` to get help.

### Other keywords and placeholders

- Add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed.
- Add `@coderabbitai summary` to generate the high-level summary at a specific location in the PR description.
- Add `@coderabbitai` anywhere in the PR title to generate the title automatically.

### Documentation and Community

- Visit our [Documentation](https://docs.coderabbit.ai) for detailed information on how to use CodeRabbit.
- Join our [Discord Community](http://discord.gg/coderabbit) to get help, request features, and share feedback.
- Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements.

</details>

<!-- tips_end -->
author_association: CONTRIBUTOR
user: login: kaixi-wang
created_at: 2025-04-21T04:50:21Z
body: I remember looking into frame aggregations a long time ago and found that swapping the base collection from the sample to the frame collection overcomes nearly all 16mb issues and avoids the costly lookup+ unwind. Wondering why that approach wouldn't work instead?
author_association: MEMBER
user: login: brimoor
created_at: 2025-04-21T13:33:57Z
body: I agree we should find a way to minimize unnecessary lookups, but its not quite as simple as just swapping the base collections; it will take more work than that.
author_association: MEMBER
user: login: swheaton
created_at: 2025-04-21T13:52:31Z
body: Ben has more weight on this one but lgtm also 👍🏼 
author_association: CONTRIBUTOR