Repository: voxel51/fiftyone

Issue:
[BUG] Grouped video datasets: labels shown only in default group slice
### System information

-   **OS Platform and Distribution** Linux Ubuntu 20.04
-   **Python version** (`python --version`): 3.10
-   **FiftyOne version** (`fiftyone --version`): 0.21.2
-   **FiftyOne installed from** (pip or source): pip

### Commands to reproduce

```
import fiftyone as fo
import fiftyone.zoo as foz

ds = foz.load_zoo_dataset('quickstart-video')

fps = ds.values('filepath')
fp_pairs = zip(fps[:2],fps[2:4])

ds2 = fo.Dataset(name='qsv-test')
ds2.add_group_field('group',default='left')

samples = []
for fpl,fpr in fp_pairs:
    g = fo.Group()    
    s1 = fo.Sample(filepath=fpl, group=g.element('left'))
    s2 = fo.Sample(filepath=fpr, group=g.element('rigt'))
    samples.append(s1)
    samples.append(s2)

ds2.add_samples(samples)

ds2.group_slice = 'left'
ds2.ensure_frames()
samp = ds2.first()
for f,frame in samp.frames.items():
    d1 = fo.Detection(bounding_box=[0.25,0.25,0.5,0.5])
    frame['dets'] = fo.Detections(detections=[d1])
    frame.save()
samp.save()

ds2.group_slice = 'rigt'
ds2.ensure_frames()
samp = ds2.first()
for f,frame in samp.frames.items():
    d2 = fo.Detection(bounding_box=[0.25,0.25,0.25,0.25])
    frame['dets2'] = fo.Detections(detections=[d2])
    frame.save()
samp.save()

session = fo.launch_app(ds2,auto=False)

# With default_group_slice = 'left', detection shows only in left slice (detection "flashes" in right slice)

# With default_group_slice = 'right', detection shows only in right slice (detection "flashes" in left slice)
ds2.default_group_slice = 'rigt'
```

### Describe the problem

Frame-level labels (eg obj detections) are shown only for the default group slice within the group modal, for grouped video datasets.

Non-default-slices flash labels in eg the first frame of playback, but then labels are not shown for remaining frames.

Data(set) structure in SDK appears as expected.

See OSS Slack: https://fiftyone-users.slack.com/archives/C0189A476EQ/p1687962462949109


### Other info/logs

### What areas of FiftyOne does this bug affect?

-   [x] `App`: FiftyOne application issue
-   [ ] `Core`: Core Python library issue
-   [ ] `Server`: FiftyOne server issue




Discussion:
Grouped video datasets are undocumented and should be considered unstable for the time being
Hello, I am having the same issue.

Grouped video datasets work fine in version `0.19.1`, however, I need the CVAT >=2.4 integration which is only working properly starting from version `0.21.0`

I hope this can be resolved soon.


Resolving patch:
diff --git a/fiftyone/server/routes/frames.py b/fiftyone/server/routes/frames.py
--- a/fiftyone/server/routes/frames.py
+++ b/fiftyone/server/routes/frames.py
@@ -6,6 +6,7 @@
 |
 """
 from starlette.endpoints import HTTPEndpoint
+from starlette.responses import JSONResponse
 from starlette.requests import Request
 
 from fiftyone.core.expressions import ViewField as F
@@ -27,10 +28,14 @@ async def post(self, request: Request, data: dict):
         dataset = data.get("dataset")
         stages = data.get("view")
         sample_id = data.get("sampleId")
+        group_slice = data.get("slice", None)
 
         view = fosv.get_view(dataset, stages=stages, extended_stages=extended)
         view = fov.make_optimized_select_view(view, sample_id)
 
+        if group_slice is not None:
+            view.group_slice = group_slice
+
         end_frame = min(num_frames + start_frame, frame_count)
         support = None if stages else [start_frame, end_frame]
         if not support:
@@ -47,7 +52,9 @@ async def post(self, request: Request, data: dict):
             view._pipeline(frames_only=True, support=support),
         ).to_list(end_frame - start_frame + 1)
 
-        return {
-            "frames": foj.stringify(frames),
-            "range": [start_frame, end_frame],
-        }
+        return JSONResponse(
+            {
+                "frames": foj.stringify(frames),
+                "range": [start_frame, end_frame],
+            }
+        )


Test patch: