Mitigating Surgical Data Imbalance with Dual-Prediction Video Diffusion Model

## Abstract

Abstract Surgical video datasets are essential for scene understanding, enabling procedural modeling and intra-operative support. However, these datasets are often heavily imbalanced, with rare actions and tools under-represented, which limits the robustness of downstream models. We address this challenge with SurgiFlowVid , a sparse and controllable video diffusion framework for generating surgical videos of under-represented classes. Our approach introduces a dual-prediction diffusion module that jointly denoises RGB frames and optical flow, providing temporal inductive biases to improve motion modeling from limited samples. In addition, a sparse visual encoder conditions the generation process on lightweight signals (e.g., sparse segmentation masks or RGB frames), enabling controllability without dense annotations. We validate our approach on three surgical datasets across tasks including action recognition, tool presence detection, and laparoscope motion prediction. Synthetic data generated by our method yields consistent gains of 10 10 – 20 % 20\% over competitive baselines, establishing SurgiFlowVid as a promising strategy to mitigate data imbalance and advance surgical video understanding methods.

## 1 Introduction

Robotic-assisted minimally invasive surgery (RAMIS) has become a cornerstone of modern surgical practice, offering patients reduced trauma, faster recovery, and fewer complications (Haidegger et al., 2022, Taylor et al., 2016). However, operating using an endoscopic video feed rather than direct vision introduces challenges such as limited depth perception, reduced haptic feedback, and altered hand–eye coordination. These limitations increase both the cognitive and technical demands placed on surgeons during procedures (Sørensen et al., 2016, Dagnino and Kundrat, 2024).
(^1^11Work done during an internship at Intuitive Surgical Inc.)(^2^22Corresponding author: Adam Schmidt, Adam.Schmidt@intusurg.com)

The emerging field of *Surgical Data Science* seeks to address these challenges by developing computational methods that leverage the video data generated during surgery. In particular, deep learning (DL) methods could be utilized to understand the surgical scene, thereby supporting intraoperative decisions and reducing the burden on surgeons. Surgical video datasets, therefore, play a central role in enabling tasks, including surgical phase and gesture recognition (Padoy et al., 2012, Funke et al., 2025, 2019a), instrument detection and segmentation (Nwoye et al., 2022b, Kolbinger et al., 2023), tool tracking (Schmidt et al., 2024), and skill assessment (Funke et al., 2019b, Hoffmann et al., 2024). However, despite recent efforts to release annotated datasets (Nasirihaghighi et al., 2025, Ayobi et al., 2024, Psychogyios et al., 2023, Wang et al., 2022), these resources remain heavily imbalanced, with rare actions, steps, or tool usages under-represented (see Fig. [1](https://arxiv.org/html/2510.07345v1#S1.F1)). Such skewed distributions limit the generalization of DL models. Common approaches such as class-sampling and augmentation can increase the frequency of these samples but do not contribute to the diversity of the dataset.

Figure: Figure 1: Data challenge in the surgical domain. During a laparoscopic procedure, the surgeon operates via the endoscopic video feed (video on the monitor). ML models can leverage these videos for providing guidance through surgical scene understanding. However, the datasets are *skewed* as shown in the bar plots. We aim to mitigate data imbalance with synthetic samples. The right plot shows improvements from adding samples generated from our approach (SurgiFlowVid).
Refer to caption: x1.png

The data imbalance challenge in surgical datasets have motivated increasing interest in synthetic data generation. With the advent of diffusion models (DMs) (Ho et al., 2020, Dhariwal and Nichol, 2021a), synthetic surgical images have been successfully utilized to augment real datasets, thereby reducing imbalance and enhancing downstream performance Venkatesh et al. (2025b), Frisch et al. (2023), Nwoye et al. (2025). However, extending DMs to surgical video generation remains underexplored due to the substantial demands in data and compute. While recent progress in video synthesis is promising, controllability is especially critical in the surgical domain, where specific tools and anatomical structures must appear in procedure-dependent contexts (e.g., laparoscopy vs. robotic surgery). Prior work often relies on dense per-frame segmentation masks to control video generation (Biagini et al., 2025, Sivakumar et al., 2025, Yeganeh et al., 2024, Iliash et al., 2024, Cho et al., 2024), but these require costly expert annotations that are rarely available. In practice, surgical datasets typically contain only sparse segmentation masks—or none at all—while under-represented classes are particularly scarce. This raises a critical question: how can generative models improve learning for under-represented classes when only sparse or no conditional signals are available?

To address this challenge, we propose *SurgiFlowVid* (Surgical Flow-inducted Video generation), a diffusion-based framework designed to synthesize spatially and temporally coherent surgical videos of under-represented classes.
We introduce a dual-prediction approach that jointly denoises RGB frames and optical flow maps, providing inductive biases to improve motion modeling from limited data. Beyond text prompts, SurgiFlowVid can condition directly on RGB frames or sparse segmentation masks, when available, via a visual encoder. While video DMs typically rely on heavy compute, our approach is tailored to the constrained settings common in healthcare, ensuring practical applicability.
SurgiFlowVid generates diverse and coherent videos of under-represented classes, which we use to augment real datasets and evaluate the models across multiple datasets and downstream tasks.
By tackling the challenges of data imbalance, our approach advances robust DL methods for surgical video understanding, contributing to the broader goal of improving surgical healthcare.
We summarize our contributions as follows:

- 1.
We address the critical challenge of data imbalance in surgical datasets by synthesizing video samples of under-represented classes with diffusion models, providing a principled way to augment real world datasets.
- 2.
We introduce *SurgiFlowVid*, a surgical video diffusion framework equipped with a dual-prediction diffusion U-Net that leverages both RGB frames and optical flow to capture spatio-temporal relationships, even in the minimal available video samples of under-represented classes. In addition, a visual encoder enables conditioning on sparse conditional frames when available, removing the need for costly dense annotations.
- 3.
We extensively evaluate the proposed framework, starting with an analysis of synthetic data attributes and extending to three surgical datasets across diverse surgical downstream tasks: action recognition, tool presence detection, and laparoscope motion prediction. The results show consistent performance gains of $10$–$20\%$ over strong baselines, highlighting the effectiveness of our approach in advancing robust surgical video understanding models.

## 2 Related Work

#### Synthetic data in surgery

$2$D synthetic laparoscopic surgical images generated using
GANs (Goodfellow et al., 2014) and diffusion models (DMs) (Dhariwal and Nichol, 2021b, Sohl-Dickstein et al., 2015) have been shown to enhance downstream tasks (Venkatesh et al., 2024, 2025b, Frisch et al., 2023, Nwoye et al., 2025, Allmendinger et al., 2024, Martyniak et al., 2025, Pfeiffer et al., 2019).
However, these approaches remain limited to static image generation and fail to capture the temporal context essential for surgical videos, which are the primary data source in real-time procedures. While diffusion models have also shown success in medical imaging domains such as MRI and CT (Dorjsembe et al., 2022, Khader et al., 2023, Zhao et al., 2025a), these modalities differ fundamentally from surgical video data.

#### Surgical Video Synthesis

Although laparoscopic video synthesis has attracted increasing attention in recent years, its potential for addressing data imbalance in surgical tasks remains underexplored. Endora (Li et al., 2024) introduced unconditional video generation by incorporating semantic features from a DINO (Caron et al., 2021) backbone, while
MedSora (Wang et al., 2024) proposed a framework based on a Mamba diffusion model. However, both approaches lacked controllability, which is crucial for generating task-specific videos that can mitigate data imbalance.  Iliash et al. (2024) and SurGen (Cho et al., 2024) extended video generation by conditioning on pre-defined instrument masks to synthesize coherent surgical phases. Yet, these methods requires vast quantities of labeled real data ($\approx 200$K videos), which restricts its applicability to well-studied procedures, such as cholecystectomy (Nwoye et al., 2022a, Twinanda et al., 2016), and prevents its generalization to less documented surgeries.

Other works, such as VISAGE (Yeganeh et al., 2024) and SG2VID Sivakumar et al., 2025, condition generation on action graphs
which require curated datasets with detailed annotations and they are often unavailable for many surgical procedures. SurgSora (Chen et al., 2024a) instead conditions video synthesis on user-defined instrument trajectories, whereas Bora (Sun et al., 2024) leverages large language models (LLMs) to generate instruction prompts for controlling video generation. More recently, SurV-Gen (Venkatesh et al., 2025a) was proposed as a video diffusion framework for generating samples of rare classes. This method
employs a rejection sampling strategy to filter out degenerate cases (poor consistency) of synthetic videos from a large candidate pool. Although there exists plethora of state-of-the-art video diffusion models for the natural domain (Rombach et al., 2022b, Yang et al., 2024a, Agarwal et al., 2025, Polyak et al., 2024), adapting them for the surgical domain is challenging due to the large amounts of curated video data and compute needed to train them. Additional related work is in the appendix ([A](https://arxiv.org/html/2510.07345v1#A1)).

Our approach, although closely related to SurV-Gen, introduces notable advantages: by incorporating optical flow as an inductive bias, we generate temporally coherent and plausible videos without the need for rejection sampling. Additionally, by conditioning on sparse segmentation masks or RGB frames, we achieve greater controllability and diversity in generating under-represented classes.

## 3 SurgiFlowVid

Our goal is to alleviate data imbalance by generating spatially and temporally coherent surgical videos of under-represented classes, a task that is made difficult by the limited data available to model spatial and temporal dynamics accurately. To address this, we introduce *SurgiFlowVid*, which includes a multi-stage conditioninal training process built upon the SurV-Gen framework (Venkatesh et al., 2025a) with the following core modifications:

(i) Dual-prediction diffusion U-Net: we introduce a U-Net module that jointly predicts RGB frames and optical flow maps during training, enabling the model to capture temporal motion alongside spatial appearance which cannot be reliably inferred from RGB appearance alone.

Figure: Figure 2: SurgiFlowVid approach.The dual-prediction diffusion U-Net module reconstructs both RGB and optical flow frames from noised inputs to capture spatio-temporal dynamics from limited data. Sparse visual encoder is trained with segmentation masks (if available) or RGB frames for conditional generation; optical flow is used only during training.
Refer to caption: x2.png

(ii) Sparse conditional guidance: dense segmentation masks are rarely available in surgical datasets, and relying solely on text or label conditioning provides weak guidance. Instead, we design a sparse visual guidance encoder that conditions the diffusion process on either the available sparse segmentation masks or the RGB frames from the input video. Our model supports both text-based unconditional generation and conditional generation with sparse masks (if available), generating under-represented class samples with spatio-temporal consistency.

We first review SurV-Gen and follow it with explaining our approach. The overview of our approach is shown in Fig. [2](https://arxiv.org/html/2510.07345v1#S3.F2).

#### (i) Surgical Video Generation

We build our framework on top of the SurV-Gen model, which follows a two-stage training strategy. In the first stage, Stable Diffusion (SD) (Rombach et al., 2022a) is adopted as the base text-to-image model, where the diffusion process is performed in the latent space. An image $x_{0}$ is first encoded into $z_{0}$ via an encoder $E(x_{0})$, and during the forward diffusion process $z_{0}$ is iteratively perturbed as $z_{t}=\sqrt{\bar{\alpha}_{t}}\,z_{0}+\sqrt{1-\bar{\alpha}_{t}}\,\epsilon,$ with $\epsilon\sim\mathcal{N}(0,I)$, $\alpha_{t}=1-\beta_{t}$, and $\bar{\alpha}_{t}=\prod_{s=0}^{t}\alpha_{s}$, where $\beta_{t}$ determines the noise strength. A denoising network $\epsilon_{\theta}(\cdot)$ is trained to reverse this process by minimizing the reconstruction loss

$$ $\mathcal{L}=\mathbb{E}_{E(x_{0}),y,\epsilon\sim\mathcal{N}(0,I),t}\big[\lVert\epsilon-\epsilon_{\theta}(z_{t},t,y)\rVert_{2}^{2}\big],$ (1) $$

where $y$ denotes the text prompt associated with $x_{0}$. In the second stage, the fine-tuned spatial layers of the SD model is extended to operate directly on surgical video sequences. Temporal transformer blocks (Vaswani et al., 2017) are inserted after each spatial block while keeping spatial layers frozen, thereby focusing the training on temporal dynamics. Given a video tensor $v\in\mathbb{R}^{b\times c\times f\times h\times w}$, where $b$ is the batch size, $f$ the number of frames, $h$, $w$ and $c$ are the height, width and channel dimensions respectively, the temporal layers reshape $v$ to $(bhw)\times f\times c$ and apply self-attention: $v_{\text{out}}=\text{Softmax}\left(\frac{QK^{T}}{\sqrt{c}}\right)V$
with $Q=v_{\text{in}}W_{Q}$, $K=v_{\text{in}}W_{K}$, and $V=v_{\text{in}}W_{V}$ as the query, key, and value projections. Cross-frame attention captures motion dynamics, but relying solely on it—or on text and label conditioning—is insufficient to model tool and tissue motion.

#### (ii) Dual-prediction module

In our approach, we modify the U-net such that optical flow, $p$, is taken as an input along with input tensor, $v$. Given two consecutive frames $v_{1},v_{2}\in\mathbb{R}^{3\times H\times W}$, the optical flow is computed as $D_{t}(v_{1},v_{2})=(d_{1},d_{2})$, which encodes the pixel displacement at location $(v_{1},v_{2})$. We convert $D_{t}$ into an RGB image by computing a normalized magnitude $r(v_{1},v_{2})$ and angle $\theta$:

$$ $r(v_{1},v_{2})\;=\;\frac{\sqrt{\widehat{d}_{1}^{2}+\widehat{d}_{2}^{2}}}{\|D_{t}\|_{\max}+\varepsilon},\qquad\theta(v_{1},v_{2})\;=\;\frac{1}{\pi}\,\mathrm{atan2}\!\big(\widehat{d}_{2},\,\widehat{d}_{1}\big),$ $$

where $\widehat{d}_{1},\widehat{d}_{2}$ denote the normalized flow components and $\varepsilon>0$ ensures numerical stability. The angle $\theta$ is mapped to a color, while the magnitude $r$ attenuates this color to produce the RGB encoding resulting in the flow tensor $p^{c\times(f-1)\times h\times w}$. We define the *dual-prediction* diffusion U-Net by modifying its input and output layers to process RGB frames and optical flow jointly. Specifically, the first layer is adapted to accommodate both tensors, $v$ and $p$, while the final layer is modified to predict both RGB and flow frames. These layers are trained together with the temporal attention layers using the loss function ($L$) defined as,

$$ $\mathcal{L}=\mathbb{E}_{E(x_{0}),y,\epsilon\sim\mathcal{N}(0,I),t}\Big[\lVert\epsilon-\epsilon_{\theta}(z_{t},t,y)\rVert_{2}^{2}+\lambda_{p}\,\lVert\epsilon-\epsilon_{\theta}(z_{p},t,y)\rVert_{2}^{2}\Big],$ (2) $$

where $z_{p}$ is the noised optical flow frames and $\lambda_{p}$ is a weighting parameter. The model jointly denoises each chunk of RGB and flow frames. We freeze the spatial layers in this stage and optical flow is used solely as a training-time signal with text prompts being used during sampling.

#### (iii) Sparse visual guidance

To incorporate conditional guidance, we extend the sparse condition encoder proposed in SparseCtrl (Guo et al., 2024), which propagates sparse signals (e.g., frames) across time using spatial and temporal layers to improve consistency between conditioned and unconditioned frames.
In our framework, we integrate the dual-prediction U-net and redefine the *sparse visual encoder* as a lightweight module that encodes only the sparse conditional frames. The U-Net backbone is frozen, and only the visual encoder is optimized using the loss in Eq. [2](https://arxiv.org/html/2510.07345v1#S3.E2). By incorporating optical flow into the loss, we explicitly supervise both motion and structure, allowing the model to move beyond appearance propagation alone, thereby reducing data requirements and improving robustness. Formally, given sparse conditional signals $s_{s}\in\mathbb{R}^{3\times H\times W}$ (e.g., RGB frame or segmentation mask) and a binary mask $m\in\{0,1\}^{1\times H\times W}$ indicating whether a frame is conditioned, the sparse encoder input is constructed as $\hat{c}=[\,s_{s}\,\|\,m\,]$ where $\|$ denotes channel-wise concatenation. This design offers flexibility by enabling diverse conditional inputs to guide the generation process. At inference, we sample sparse frames from the real dataset and reassign them to different temporal positions to synthesize videos.

## 4 Experiments

In this section, we outline our experimental setup, evaluation schemes and the downstream tasks we evaluate the generated synthetic datasets.
We focus particularly on the under-represented classes, and generate videos of such classes to match their instances to the well represented ones.

#### Datasets

(i) SAR-RARP50
consists of radical prostatectomy (robotic) videos from $50$ patients, with a split of $35$, $5$, and $10$ patients for training, validation, and test sets, respectively (Psychogyios et al., 2023). The annotated surgical actions include: picking up the needle (A$1$), positioning the needle (A$2$), pushing the needle through tissue (A$3$), pulling the needle (A$4$), cutting the suture (A$5$), tying the knot (A$6$), and returning the needle (A$7$). Since action A$6$ occurs only once in the test set, it is omitted from evaluation. The under-represented classes in this dataset are A$1$, A$5$, and A$7$.
The primary task involves recognizing the surgical action at time $t$ given a video clip. In addition, segmentation masks are available for nine classes collected at $1$fps.
Using these masks, we construct the task of surgical tool presence detection, where the objective is to identify which instruments are present in a given surgical video.
(ii) GraSP
includes robotic prostatectomy procedures (Ayobi et al., 2024). It consists of $13$ patients with a two-fold cross-validation setup, where five patients are held out for testing. The dataset contains annotations for $20$ different surgical actions. For this study, we focus on a subset of five actions: pulling the suture (G$1$), tying the suture (G$2$), cutting the suture (G$3$), cutting between the prostate and bladder neck (G$4$), and identifying the iliac artery (G$5$). All classes except G$5$ are under-represented. Instrument annotations are also provided for six classes at every $35$ secs making them sparse in nature.
We use this dataset for both surgical action recognition and tool presence detection tasks.
(iii) AutoLaparo
contains laparoscopic hysterectomy videos from 21 patients, with annotations describing the movements of the laparoscope (Wang et al., 2022). In total, it contains approximately $300$ clips, each lasting $10$ seconds, covering six motion types: up, down, left, right, zoom-in, and zoom-out. The laparoscope motion occurs precisely at the $5$th second of each clip, enabling the formulation of two tasks. In the online recognition setting, only the first $5$ seconds are provided to the model to predict the upcoming motion, which is particularly relevant for real-time applications. In the offline setting, the entire clip is available, and the task is to classify the laparoscope motion using full temporal context. These annotations can be used for developing automatic field-of-view control systems. Owing to the limited dataset size, all movement classes are considered under-represented.

#### Baselines

For comparison, we evaluate against recent surgical video diffusion models. Endora (Li et al., 2024) is a fully transformer-based unconditional diffusion model, which we train separately on each minor class due to its lack of controllability. SurV-Gen (Venkatesh et al., 2025a) serves as a conditional baseline with both text and label guidance. We also include its rejection sampling (RS) strategy, which filters out degenerate generations and thus represents a strong reference baseline. In addition, we adapt the SparseCtrl (Guo et al., 2024) model, an effective conditional video diffusion approach that generates videos conditioned on text and sparse conditional masks. We train the SurgiFlowVid model with only text conditioning and follow it by sparse segmentation and RGB frames. These serve as both baselines and ablations of our approach. We maintain a patient specific test split and train the model only on the train split.
Videos of $16$-frames are generated at four frames-per-second aligning with the requirements of the downstream task.
Together, these baselines span unconditional, conditional, and sparse conditional video diffusion approaches, providing a comprehensive reference for evaluating our method. Additional details are in the appendix ( [B.6](https://arxiv.org/html/2510.07345v1#A2.SS6)).

#### Evaluation scheme

We systematically structure our experimental design into three parts to evaluate the role of synthetic data in addressing class imbalance.

(i) Synthetic data attributes: We analyze which attributes of synthetic data are essential for improving downstream performance. To this end, we conduct controlled experiments on the surgical action recognition task.
First, we *duplicate* the training set and train for the same number of epochs to evaluate whether performance gains arise from true data diversity rather than simple repetition.
Second, to assess the effects of *spatial* and *temporal* consistency, we simulate degraded data by applying elastic deformations and noise to video frames (disrupting spatial structure) and by shuffling frames (disrupting temporal order). Third, we evaluate the effect of *sparse conditioning* by constructing videos from only sparse frames and examining their impact on downstream performance.

(ii) Class modeling: We investigate whether synthetic data is more effective when all under-represented classes are modeled jointly or when each class is modeled separately.

(iii) Downstream tasks: We evaluate the effect of synthetic data on three surgical downstream applications: surgical action recognition, surgical tool presence detection, and laparoscope motion prediction.

#### Downstream models

For surgical action (step) recognition, we employ the MViT-v2 (Li et al., 2022) model, which has shown strong performance on the SAR-RARP$50$ dataset and we report the averaged video-wise Jaccard index per class. The TAPIS model was used for the GraSP dataset, which incorporates an MViT backbone, and evaluate performance using mean average precision (mAP) averaged across videos, as described in Ayobi et al. (2024). For surgical tool presence detection, the Swin Transformer (base) (Liu et al., 2021) was opted in a multi-label classification setting, reporting the Dice score as the evaluation metric. Finally, for laparoscope motion recognition, we utilize a ResNet$3$D (Hara et al., 2017) model to classify motion categories from input clips, with mean F$1$ score as the metric. We apply inverse frequency balancing with video frame augmentation only on the real datasets during training. Especially, we add synthetic videos of under-represented to the real dataset and leave the well balanced classes undisturbed. Each model is run with three different seeds, and we report the mean and standard deviation across videos. These model choices ensure fair and robust state-of-the-art baselines for video understanding tasks. Please refer to the appendix for details on model training( [D](https://arxiv.org/html/2510.07345v1#A4)), additional experiments and evaluations( [B](https://arxiv.org/html/2510.07345v1#A2)) and qualitative results ([E](https://arxiv.org/html/2510.07345v1#A5)).

## 5 Results & Discussion

#### Synthetic data attributes

Our evaluation of different synthetic data attributes for under-represented classes in the SAR-RARP$50$ dataset is in Table [1](https://arxiv.org/html/2510.07345v1#S5.T1).

**Table 1: Attributes of synthetic data experiment on the under-represented classes of the SAR-RARP$50$ dataset.**
| Method | A$1$ | A$5$ | A$7$ |
| --- | --- | --- | --- |
| Real | $0.32_{\pm{0.19}}$ | $0.10_{\pm{0.04}}$ | $0.32_{\pm{0.15}}$ |
| Data duplicate | $0.32_{\pm{0.17}}$ | $0.11_{\pm{0.02}}$ | $0.32_{\pm{0.13}}$ |
| Frame shuffle | $0.30_{\pm{0.14}}$ | $0.06_{\pm{0.09}}$ | $0.30_{\pm{0.17}}$ |
| Sparse frame | $0.28_{\pm{0.14}}$ | $0.05_{\pm{0.05}}$ | $0.29_{\pm{0.10}}$ |
| Noisy frame | $0.29_{\pm{0.14}}$ | $0.04_{\pm{0.05}}$ | $0.29_{\pm{0.10}}$ |

Readers can refer to the suppl. for additional results (Sec. [B.1](https://arxiv.org/html/2510.07345v1#A2.SS1)).
Merely duplicating the training set does not improve performance, as it fails to introduce additional sample diversity. Frame shuffling causes a slight decline in performance, underscoring the importance of temporal consistency in video-based tasks. Similarly, injecting noise into frames or conditioning only on sparse frames results in a more substantial drop of about $3$–$5\%$. Together, these findings reveal three key aspects: (i) synthetic data must not simply replicate the training set, but rather provide *data diversity*, (ii) maintaining *temporal consistency* is critical, and (iii) preserving *spatial structure* is essential to sustain downstream performance. Overall, this analysis underlines that downstream tasks inherently require both spatial and temporal consistency, and synthetic data must therefore satisfy both to be effective.

#### Surgical action recognition.

(i) *SAR-RARP$50$*:
The results of surgical action recognition task is reported in Tab. [2](https://arxiv.org/html/2510.07345v1#S5.T2). The SurV-Gen model with rejection sampling achieves better performance on under-represented classes compared to using synthetic samples directly, suggesting that its gains stem primarily from the sampling strategy rather than the generative model itself.
Synthetic samples from SparseCtrl improves scores across all three underrepresented classes. Our approach, *SurgiFlowVid*, even with text-only conditioning, yields performance improvements in two out of the three under-represented classes, with gains in the range of $3$–$11\%$. Adding conditional masks further enhances performance across all classes, with SurgiFlowVid conditioned on segmentation masks achieving improvements of $12\%$, $8\%$, and $10\%$ were noticed with for the under-represented classes. Performance gains are also observed in well-balanced classes, which we attribute to the mutual dependencies among actions. For instance, augmenting data for the “picking the needle” class may indirectly benefit “positioning the needle” class, as the latter can often follow in the surgical workflow. Another noteworthy observation is that modeling each class individually produces a substantial improvement in mean performance, reaching $0.53$ compared to $0.46$ with real data alone. Particularly notable is the nearly $20\%$ gain for A$7$, obtained with synthetic samples from SurgiFlowVid (RGB-frame) combined with individual-class training.

**Table 2: Surgical action recognition on the SAR-RARP$50$ dataset. Under-represented classes are highlighted, and Jaccard index is reported. Ic denotes individual class modeling, and RS indicates rejection sampling. Addition of synthetic samples from SurgiFlowVid indicates comprehensive gains for the under-represented classes.**
| Training data | Cond. type | Pick<br>the needle | Position<br>the needle | Push<br>the needle | Pull<br>the needle | Cut<br>the suture | Return<br>the needle | Mean. |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Text | Sparse mask |  |  |  |  |  |  |  |  |
| Real | – | – | $0.32_{\pm 0.19}$ | $0.66_{\pm 0.09}$ | $0.78_{\pm 0.10}$ | $0.61_{\pm 0.09}$ | $0.10_{\pm 0.04}$ | $0.32_{\pm 0.15}$ | $0.46_{\pm 0.08}$ |
| Real $+$ Endora | – | – | $0.32_{\pm 0.14}$ | $0.63_{\pm 0.05}$ | $0.76_{\pm 0.07}$ | $0.61_{\pm 0.11}$ | $0.08_{\pm 0.04}$ | $0.33_{\pm 0.10}$ | $0.45_{\pm 0.05}$ |
| Real $+$ SurV-Gen (w/o RS) | ✓ | – | $0.31_{\pm 0.19}$ | $0.64_{\pm 0.07}$ | $0.77_{\pm 0.06}$ | $0.60_{\pm 0.10}$ | $0.13_{\pm 0.10}$ | $0.37_{\pm 0.18}$ | $0.46_{\pm 0.03}$ |
| Real $+$ SurV-Gen (RS) | ✓ | – | $0.35_{\pm 0.12}$ | $0.63_{\pm 0.02}$ | $0.77_{\pm 0.03}$ | $0.61_{\pm 0.08}$ | $0.14_{\pm 0.09}$ | $0.39_{\pm 0.15}$ | $0.48_{\pm 0.06}$ |
| Real $+$ SparseCtrl | ✓ | RGB | $0.36_{\pm 0.17}$ | $0.65_{\pm 0.06}$ | $0.78_{\pm 0.07}$ | $0.64_{\pm 0.11}$ | $0.09_{\pm 0.07}$ | $0.40_{\pm 0.12}$ | $0.48_{\pm 0.04}$ |
| Real $+$ SparseCtrl | ✓ | Seg. | $0.36_{\pm 0.14}$ | $0.61_{\pm 0.12}$ | $0.77_{\pm 0.07}$ | $0.63_{\pm 0.11}$ | $0.16_{\pm 0.11}$ | $0.38_{\pm 0.17}$ | $0.49_{\pm 0.04}$ |
| Real $+$ SurgFlowVid | ✓ | – | $0.43_{\pm 0.12}$ | $0.65_{\pm 0.07}$ | $0.77_{\pm 0.07}$ | $0.63_{\pm 0.11}$ | $0.11_{\pm 0.03}$ | $0.35_{\pm 0.12}$ | $0.49_{\pm 0.04}$ |
| Real $+$ SurgFlowvid | ✓ | RGB | $0.36_{\pm 0.17}$ | $\mathbf{0.67_{\pm 0.06}}$ | $0.78_{\pm 0.08}$ | $\mathbf{0.65_{\pm 0.12}}$ | $0.17_{\pm 0.10}$ | $0.42_{\pm 0.12}$ | $0.51_{\pm 0.04}$ |
| Real $+$ SurgFlowVid | ✓ | Seg. | $\mathbf{0.44_{\pm 0.18}}$ | $0.66_{\pm 0.07}$ | $\mathbf{0.79_{\pm 0.08}}$ | $0.64_{\pm 0.04}$ | $0.18_{\pm 0.09}$ | $0.42_{\pm 0.15}$ | $0.52_{\pm 0.04}$ |
| Real $+$ SurgFlowVid (Ic) | ✓ | – | $0.37_{\pm 0.16}$ | $0.65_{\pm 0.04}$ | $0.77_{\pm 0.07}$ | $0.61_{\pm 0.10}$ | $0.14_{\pm 0.03}$ | $0.42_{\pm 0.18}$ | $0.49_{\pm 0.06}$ |
| Real $+$ SurgFlowvid (Ic) | ✓ | RGB | $0.36_{\pm 0.14}$ | $0.65_{\pm 0.03}$ | $\mathbf{0.79_{\pm 0.15}}$ | $0.64_{\pm 0.08}$ | $\mathbf{0.20_{\pm 0.09}}$ | $\mathbf{0.52_{\pm 0.12}}$ | $\mathbf{0.53_{\pm 0.02}}$ |
| Real $+$ SurgFlowVid (Ic) | ✓ | Seg. | $0.41_{\pm 0.19}$ | $0.63_{\pm 0.06}$ | $0.77_{\pm 0.03}$ | $0.62_{\pm 0.12}$ | $0.10_{\pm 0.05}$ | $0.38_{\pm 0.16}$ | $0.48_{\pm 0.06}$ |

**Table 3: Surgical step recognition on the GraSP dataset. The best scores are in bold and the mAP scores are reported. Considerable performance gains are noticed for our approach with the sparse RGB frames in comparison to solely using the real dataset.**
| Training data | Cond. type | Pull<br>the suture | Tie<br>the suture | Cut<br>the suture | Cut<br>btw. the prostate | Identify the<br>iliac artery | Mean. |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Text | Sparse mask |  |  |  |  |  |  |  |
| Real | – | – | $0.26_{\pm 0.03}$ | $0.44_{\pm 0.01}$ | $0.43_{\pm 0.06}$ | $0.72_{\pm 0.07}$ | $0.52_{\pm 0.08}$ | $0.47_{\pm 0.03}$ |
| Real $+$ Endora | – | – | $0.26_{\pm 0.02}$ | $0.39_{\pm 0.02}$ | $0.40_{\pm 0.05}$ | $0.70_{\pm 0.01}$ | $0.51_{\pm 0.03}$ | $0.45_{\pm 0.04}$ |
| Real $+$ SurV-Gen (w/o RS) | ✓ | – | $0.30_{\pm 0.01}$ | $0.43_{\pm 0.02}$ | $0.41_{\pm 0.09}$ | $0.71_{\pm 0.04}$ | $0.57_{\pm 0.07}$ | $0.48_{\pm 0.03}$ |
| Real $+$ SurV-Gen (RS) | ✓ | – | $0.30_{\pm 0.02}$ | $0.44_{\pm 0.03}$ | $0.42_{\pm 0.09}$ | $0.73_{\pm 0.02}$ | $0.58_{\pm 0.04}$ | $0.49_{\pm 0.02}$ |
| Real $+$ SparseCtrl | ✓ | RGB | $0.27_{\pm 0.01}$ | $0.43_{\pm 0.01}$ | $0.40_{\pm 0.09}$ | $0.71_{\pm 0.04}$ | $0.55_{\pm 0.04}$ | $0.46_{\pm 0.04}$ |
| Real $+$ SurgFlowVid | ✓ | – | $0.30_{\pm 0.01}$ | $0.43_{\pm 0.03}$ | $\underline{0.44_{\pm 0.09}}$ | $0.69_{\pm 0.04}$ | $\underline{0.60_{\pm 0.07}}$ | $0.49_{\pm 0.04}$ |
| Real $+$ SurgFlowvid | ✓ | RGB | $\mathbf{0.33_{\pm 0.01}}$ | $\mathbf{0.48_{\pm 0.02}}$ | $\mathbf{0.47_{\pm 0.01}}$ | $\mathbf{0.74_{\pm 0.02}}$ | $\underline{0.60_{\pm 0.05}}$ | $\mathbf{0.52_{\pm 0.04}}$ |
| Real $+$ SurgFlowVid (Ic) | ✓ | – | $\underline{0.31_{\pm 0.04}}$ | $0.41_{\pm 0.03}$ | $0.42_{\pm 0.04}$ | $0.72_{\pm 0.04}$ | $\mathbf{0.61_{\pm 0.05}}$ | $0.49_{\pm 0.03}$ |
| Real $+$ SurgFlowvid (Ic) | ✓ | RGB | $\underline{0.31_{\pm 0.01}}$ | $\underline{0.45_{\pm 0.01}}$ | $0.43_{\pm 0.03}$ | $0.72_{\pm 0.02}$ | $0.55_{\pm 0.05}$ | $\underline{0.50_{\pm 0.02}}$ |

Figure: Figure 3: Qualitative results of the action “tie the suture.” Purple boxes denote the sparse RGB conditioning frames. Suprious tools are generated in SurV-Gen (white box, row $1$), while SparseCtrl alters tool types compared to the conditioning frames (white box, row $2$), reflecting limited spatial consistency. SurgiFlowVid preserves both spatial and temporal structure, with consistent tools maintained across generated frames (yellow boxes, row $3$).
Refer to caption: images/qual_comp.png

(ii) *GraSP*: The effect of adding synthetic samples on the GraSP dataset is shown in Tab. [3](https://arxiv.org/html/2510.07345v1#S5.T3) and the qualitative results are shown in Fig. [3](https://arxiv.org/html/2510.07345v1#S5.F3). Incorporating samples from SurV-Gen yields small performance gains, whereas adding data generated by Endora (the unconditional baseline) or SparseCtrl with RGB-frame conditioning results in a decline in mAP score.
By contrast, our method, SurgiFlowVid, achieves improvements in two of the four underrepresented classes even with text conditioning. Furthermore, with sparse segmentation masks SurgiFlowVid achieves performance gains across all under-represented classes. These results highlight the combined value of our dual-prediction and spar encoder modules, which enables the model to learn spatio-temporal relationships from limited data more effectively.

#### Surgical tool presence detection

**Table 4: Surgical tool presence detection on SAR-RARP$50$ dataset. Our approach with seg. conditioning outperforms the baseline across all seven tool categories.**
| Training data | Tool<br>clasper | Tool<br>wrist | Tool<br>shaft | Suturing<br>needle | Thread | Suction<br>tool | Needle<br>holder | Clamps | Catheter | Mean |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Real | $0.85_{\pm 0.10}$ | $0.84_{\pm 0.09}$ | $0.88_{\pm 0.07}$ | $0.70_{\pm 0.15}$ | $0.75_{\pm 0.12}$ | $0.69_{\pm 0.11}$ | $0.66_{\pm 0.07}$ | $0.44_{\pm 0.11}$ | $0.46_{\pm 0.08}$ | $0.69_{\pm 0.06}$ |
| Real<br>$+$ SparseCtrl(Seg) | $0.87_{\pm 0.11}$ | $0.83_{\pm 0.05}$ | $\mathbf{0.89_{\pm 0.06}}$ | $0.73_{\pm 0.12}$ | $0.80_{\pm 0.13}$ | $\mathbf{0.79_{\pm 0.10}}$ | $0.74_{\pm 0.09}$ | $0.69_{\pm 0.08}$ | $0.50_{\pm 0.12}$ | $0.74_{\pm 0.03}$ |
| Real<br>$+$ SurgFlowVid(Seg) | $\mathbf{0.88_{\pm 0.09}}$ | $\mathbf{0.85_{\pm 0.07}}$ | $0.88_{\pm 0.10}$ | $\mathbf{0.75_{\pm 0.11}}$ | $\mathbf{0.81_{\pm 0.09}}$ | $0.78_{\pm 0.15}$ | $\mathbf{0.75_{\pm 0.04}}$ | $\mathbf{0.73_{\pm 0.10}}$ | $\mathbf{0.59_{\pm 0.05}}$ | $\mathbf{0.79_{\pm 0.04}}$ |

**Table 5: Surgical tool presence detection on GraSP dataset. Combining synthetic data from SurgiFlowVid yields marked improvements in dice scores.**
| Training data | Bipolar<br>forceps | L.needle<br>driver | Mono<br>curved scissors | Prograsp<br>forceps | Suction<br>inst. | Clip<br>applier | Laparoscopic<br>inst. | Mean |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Real | $0.94_{\pm 0.01}$ | $0.56_{\pm 0.03}$ | $0.95_{\pm 0.02}$ | $0.72_{\pm 0.02}$ | $0.71_{\pm 0.03}$ | $0.34_{\pm 0.09}$ | $0.56_{\pm 0.04}$ | $0.68_{\pm 0.10}$ |
| Real<br>$+$ SparseCtrl(Seg) | $\mathbf{0.95_{\pm 0.02}}$ | $0.56_{\pm 0.02}$ | $0.97_{\pm 0.01}$ | $0.75_{\pm 0.03}$ | $\mathbf{0.74_{\pm 0.07}}$ | $0.35_{\pm 0.02}$ | $\mathbf{0.60_{\pm 0.05}}$ | $0.70_{\pm 0.04}$ |
| Real<br>$+$ SurgFlowVid(Seg) | $0.94_{\pm 0.01}$ | $\mathbf{0.58_{\pm 0.02}}$ | $\mathbf{0.98_{\pm 0.01}}$ | $\mathbf{0.78_{\pm 0.01}}$ | $0.73_{\pm 0.04}$ | $\mathbf{0.37_{\pm 0.03}}$ | $\mathbf{0.60_{\pm 0.02}}$ | $\mathbf{0.72_{\pm 0.02}}$ |

The results of the surgical tool presence detection task on the GraSP and SAR-RARP$50$ datasets are shown in Tab. [4](https://arxiv.org/html/2510.07345v1#S5.T4) and Tab.[5](https://arxiv.org/html/2510.07345v1#S5.T5), respectively. Overall, the addition of synthetic samples from generative models leads to consistent performance improvements. This trend can be explained by the fact that the generated surgical videos naturally increase the occurrence of individual tools within the training set. On SAR-RARP$50$, our approach, SurgiFlowVid, achieves a $10$-point improvement over using only real data, compared to a $5$-point gain from SparseCtrl. Notably, SparseCtrl’s reliance on sparse conditioning yields only limited benefits, improving performance for a single under-represented class out of four. These findings further underscore the importance of generating videos with coherent spatio-temporal context for downstream tool detection models to perform effectively. For the GraSP dataset, improvements with synthetic samples are more subtle. SparseCtrl yields modest gains, while SurgiFlowVid achieves a $6\%$ improvement for the prograsp forceps class and an overall $4\%$ improvement across the dataset. Together, these results highlight that SurgiFlowVid not only improves rare-class detection but also strengthens overall tool recognition performance.

#### Laparoscope motion prediction

Fig. [5](https://arxiv.org/html/2510.07345v1#S5.F5) presents the results of laparoscope motion detection on the AutoLaparo dataset. Among the baselines, SurV-Gen (RS) achieves better performance than Endora, while SparseCtrl with RGB-frame conditioning performs best on the online recognition task.
Our approach, SurgiFlowVid, already outperforms SurV-Gen with text-only conditioning, and the RGB-mask conditioned version surpasses all baselines. Similar trends are observed for the offline recognition task, where both F$1$ scores are higher compared to the online setting. This suggests that providing a longer temporal context enables the downstream model to classify laparoscope motion more accurately. Overall, these findings demonstrate that SurgiFlowVid can effectively adapt to smaller datasets while offering substantial benefits for developing automatic field-of-view control systems. This highlights the practical utility of our method in developing real-time surgical assistance systems.

Figure: Figure 4: Laparoscope motion prediction in *online* (left) and *offline* (right) fashion on the AutoLapro dataset.
Refer to caption: x3.png

#### Limitations

While our approach demonstrates performance gains for underrepresented classes, it also has certain limitations. Currently, we generate only short video clips of about four seconds. Extending it with autoregressive generation could enable longer sequences, that are important for tasks such as surgical phase recognition. Moreover, the sparsity of segmentation frames lead to incorrect tool position generation (see Fig. [5](https://arxiv.org/html/2510.07345v1#S5.F5)), which could be mitigated by richer conditional signals such as tool kinematics or feature-level injections—directions we leave for future investigation.

## 6 Conclusion

In this work, we addressed the critical challenge of data imbalance in surgical datasets by generating synthetic video samples of under-represented classes with our proposed framework, *SurgiFlowVid*. The framework generates spatially and temporally coherent videos through a dual-prediction diffusion U-Net that jointly models RGB frames and optical flow, while a sparse visual encoder enables controllable generation using only the limited conditional signals typically available in surgical datasets. Extensive experiments across three datasets and downstream tasks—surgical action recognition, tool presence detection, and laparoscope motion prediction—demonstrate consistent improvements over strong baselines. By bridging advances in machine learning with the needs of surgical data science, this work helps address the scarcity of data on rare events and moves toward more robust surgical video understanding models.

## 7 Reproducibility Statement

All the information such as models, hyper-parameters and datasets needed to reproduce this work has been included in the appendix.

## References

- pik (2025)
Pika, 2025.
URL [https://pika.art/login](https://pika.art/login).
- run (2025)
Runway ml, 2025.
URL [https://runwayml.com/](https://runwayml.com/).
- sor (2025)
Sora, 2025.
URL [https://openai.com/sora/](https://openai.com/sora/).
- veo (2025)
Veo-3, 2025.
URL [https://deepmind.google/models/veo/](https://deepmind.google/models/veo/).
- Agarwal et al. (2025)
N. Agarwal, A. Ali, M. Bala, Y. Balaji, E. Barker, T. Cai, P. Chattopadhyay,
Y. Chen, Y. Cui, Y. Ding, et al.
Cosmos world foundation model platform for physical ai.
*arXiv preprint arXiv:2501.03575*, 2025.
- Allmendinger et al. (2024)
S. Allmendinger, P. Hemmer, M. Queisner, I. Sauer, L. Müller, J. Jakubik,
M. Vössing, and N. Kühl.
Navigating the synthetic realm: Harnessing diffusion-based models for
laparoscopic text-to-image generation.
In *AI for Health Equity and Fairness: Leveraging AI to Address
Social Determinants of Health*, pages 31–46. Springer, 2024.
- Ayobi et al. (2024)
N. Ayobi, S. Rodríguez, A. Pérez, I. Hernández, N. Aparicio,
E. Dessevres, S. Peña, J. Santander, J. I. Caicedo, N. Fernández,
et al.
Pixel-wise recognition for holistic surgical scene understanding.
*arXiv preprint arXiv:2401.11174*, 2024.
- Belhaouari et al. (2024)
S. B. Belhaouari, A. Islam, K. Kassoul, A. Al-Fuqaha, and A. Bouzerdoum.
Oversampling techniques for imbalanced data in regression.
*Expert systems with applications*, 252:124118, 2024.
- Biagini et al. (2025)
D. Biagini, N. Navab, and A. Farshad.
Hierasurg: Hierarchy-aware diffusion model for surgical video
generation.
*arXiv preprint arXiv:2506.21287*, 2025.
- Blattmann et al. (2023)
A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz,
Y. Levi, Z. English, V. Voleti, A. Letts, et al.
Stable video diffusion: Scaling latent video diffusion models to
large datasets.
*arXiv preprint arXiv:2311.15127*, 2023.
- Caron et al. (2021)
M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and
A. Joulin.
Emerging properties in self-supervised vision transformers.
In *Proceedings of the IEEE/CVF international conference on
computer vision*, pages 9650–9660, 2021.
- Chefer et al. (2025)
H. Chefer, U. Singer, A. Zohar, Y. Kirstain, A. Polyak, Y. Taigman, L. Wolf,
and S. Sheynin.
Videojam: Joint appearance-motion representations for enhanced motion
generation in video models.
*arXiv preprint arXiv:2502.02492*, 2025.
- Chen et al. (2024a)
T. Chen, S. Yang, J. Wang, L. Bai, H. Ren, and L. Zhou.
Surgsora: Decoupled rgbd-flow diffusion model for controllable
surgical video generation.
*arXiv preprint arXiv:2412.14018*, 2024a.
- Chen et al. (2024b)
W. Chen, K. Yang, Z. Yu, Y. Shi, and C. P. Chen.
A survey on imbalanced learning: latest research, applications and
future directions.
*Artificial Intelligence Review*, 57(6):137,
2024b.
- Cho et al. (2024)
J. Cho, S. Schmidgall, C. Zakka, M. Mathur, D. Kaur, R. Shad, and W. Hiesinger.
Surgen: Text-guided diffusion model for surgical video generation.
*arXiv preprint arXiv:2408.14028*, 2024.
- Colleoni and Stoyanov (2021)
E. Colleoni and D. Stoyanov.
Robotic instrument segmentation with image-to-image translation.
*IEEE Robotics and Automation Letters*, 6(2):935–942, 2021.
- Colleoni et al. (2022)
E. Colleoni, D. Psychogyios, B. Van Amsterdam, F. Vasconcelos, and D. Stoyanov.
Ssis-seg: Simulation-supervised image synthesis for surgical
instrument segmentation.
*IEEE Transactions on Medical Imaging*, 41(11):3074–3086, 2022.
- Cubuk et al. (2019a)
E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le.
Autoaugment: Learning augmentation strategies from data.
In *Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition*, pages 113–123, 2019a.
- Cubuk et al. (2019b)
E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le.
Randaugment: Practical data augmentation with no separate search.
*arXiv preprint arXiv:1909.13719*, 2(4):7,
2019b.
- Dagnino and Kundrat (2024)
G. Dagnino and D. Kundrat.
Robot-assistive minimally invasive surgery: trends and future
directions.
*International Journal of Intelligent Robotics and
Applications*, 8(4):812–826, 2024.
- Dhariwal and Nichol (2021a)
P. Dhariwal and A. Nichol.
Diffusion models beat gans on image synthesis.
*Advances in neural information processing systems*,
34:8780–8794, 2021a.
- Dhariwal and Nichol (2021b)
P. Dhariwal and A. Nichol.
Diffusion models beat gans on image synthesis.
*Advances in neural information processing systems*,
34:8780–8794, 2021b.
- Dorjsembe et al. (2022)
Z. Dorjsembe, S. Odonchimed, and F. Xiao.
Three-dimensional medical image synthesis with denoising diffusion
probabilistic models.
In *Medical imaging with deep learning*, 2022.
- Feichtenhofer (2020)
C. Feichtenhofer.
X3d: Expanding architectures for efficient video recognition.
In *Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition*, pages 203–213, 2020.
- Frisch et al. (2023)
Y. Frisch, M. Fuchs, A. Sanner, F. A. Ucar, M. Frenzel, J. Wasielica-Poslednik,
A. Gericke, F. M. Wagner, T. Dratsch, and A. Mukhopadhyay.
Synthesising rare cataract surgery samples with guided diffusion
models.
In *International Conference on Medical Image Computing and
Computer-Assisted Intervention*, pages 354–364. Springer, 2023.
- Funke et al. (2019a)
I. Funke, S. Bodenstedt, F. Oehme, F. von Bechtolsheim, J. Weitz, and
S. Speidel.
Using 3d convolutional neural networks to learn spatiotemporal
features for automatic surgical gesture recognition in video.
In *International conference on medical image computing and
computer-assisted intervention*, pages 467–475. Springer,
2019a.
- Funke et al. (2019b)
I. Funke, S. T. Mees, J. Weitz, and S. Speidel.
Video-based surgical skill assessment using 3d convolutional neural
networks.
*International journal of computer assisted radiology and
surgery*, 14(7):1217–1225, 2019b.
- Funke et al. (2025)
I. Funke, D. Rivoir, S. Krell, and S. Speidel.
Tunes: A temporal u-net with self-attention for video-based surgical
phase recognition.
*IEEE Transactions on Biomedical Engineering*, 2025.
- Goodfellow et al. (2014)
I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair,
A. Courville, and Y. Bengio.
Generative adversarial nets.
*Advances in neural information processing systems*, 27, 2014.
- Guo et al. (2023)
Y. Guo, C. Yang, A. Rao, Z. Liang, Y. Wang, Y. Qiao, M. Agrawala, D. Lin, and
B. Dai.
Animatediff: Animate your personalized text-to-image diffusion models
without specific tuning.
*arXiv preprint arXiv:2307.04725*, 2023.
- Guo et al. (2024)
Y. Guo, C. Yang, A. Rao, M. Agrawala, D. Lin, and B. Dai.
Sparsectrl: Adding sparse controls to text-to-video diffusion models.
In *European Conference on Computer Vision*, pages 330–348.
Springer, 2024.
- Haidegger et al. (2022)
T. Haidegger, S. Speidel, D. Stoyanov, and R. M. Satava.
Robot-assisted minimally invasive surgery—surgical robotics in the
data age.
*Proceedings of the IEEE*, 110(7):835–846,
2022.
- Hara et al. (2017)
K. Hara, H. Kataoka, and Y. Satoh.
Learning spatio-temporal features with 3d residual networks for
action recognition.
In *Proceedings of the IEEE international conference on computer
vision workshops*, pages 3154–3160, 2017.
- Harvey et al. (2022)
W. Harvey, S. Naderiparizi, V. Masrani, C. Weilbach, and F. Wood.
Flexible diffusion modeling of long videos.
*Advances in neural information processing systems*,
35:27953–27965, 2022.
- Hessel et al. (2021)
J. Hessel, A. Holtzman, M. Forbes, R. L. Bras, and Y. Choi.
Clipscore: A reference-free evaluation metric for image captioning.
*arXiv preprint arXiv:2104.08718*, 2021.
- Ho et al. (2020)
J. Ho, A. Jain, and P. Abbeel.
Denoising diffusion probabilistic models.
*Advances in neural information processing systems*,
33:6840–6851, 2020.
- Ho et al. (2022a)
J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma,
B. Poole, M. Norouzi, D. J. Fleet, et al.
Imagen video: High definition video generation with diffusion models.
*arXiv preprint arXiv:2210.02303*, 2022a.
- Ho et al. (2022b)
J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet.
Video diffusion models.
*Advances in neural information processing systems*,
35:8633–8646, 2022b.
- Hoffmann et al. (2024)
H. Hoffmann, I. Funke, P. Peters, D. K. Venkatesh, J. Egger, D. Rivoir,
R. Röhrig, F. Hölzle, S. Bodenstedt, M.-C. Willemer, et al.
Aixsuture: vision-based assessment of open suturing skills.
*International Journal of Computer Assisted Radiology and
Surgery*, 19(6):1045–1052, 2024.
- Hong et al. (2022)
W. Hong, M. Ding, W. Zheng, X. Liu, and J. Tang.
Cogvideo: Large-scale pretraining for text-to-video generation via
transformers.
*arXiv preprint arXiv:2205.15868*, 2022.
- Iliash et al. (2024)
I. Iliash, S. Allmendinger, F. Meissen, N. Kühl, and D. Rückert.
Interactive generation of laparoscopic videos with diffusion models.
In *MICCAI Workshop on Deep Generative Models*, pages 109–118.
Springer, 2024.
- Khader et al. (2023)
F. Khader, G. Müller-Franzes, S. Tayebi Arasteh, T. Han, C. Haarburger,
M. Schulze-Hagen, P. Schad, S. Engelhardt, B. Baeßler, S. Foersch, et al.
Denoising diffusion probabilistic models for 3d medical image
generation.
*Scientific Reports*, 13(1):7303, 2023.
- Kolbinger et al. (2023)
F. R. Kolbinger, F. M. Rinner, A. C. Jenke, M. Carstens, S. Krell, S. Leger,
M. Distler, J. Weitz, S. Speidel, and S. Bodenstedt.
Anatomy segmentation in laparoscopic surgery: comparison of machine
learning and human expertise–an experimental study.
*International Journal of Surgery*, 109(10):2962–2974, 2023.
- Li et al. (2024)
C. Li, H. Liu, Y. Liu, B. Y. Feng, W. Li, X. Liu, Z. Chen, J. Shao, and
Y. Yuan.
Endora: Video generation models as endoscopy simulators.
In *International conference on medical image computing and
computer-assisted intervention*, pages 230–240. Springer, 2024.
- Li et al. (2022)
Y. Li, C.-Y. Wu, H. Fan, K. Mangalam, B. Xiong, J. Malik, and C. Feichtenhofer.
Mvitv2: Improved multiscale vision transformers for classification
and detection.
In *Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition*, pages 4804–4814, 2022.
- Liang et al. (2024)
F. Liang, B. Wu, J. Wang, L. Yu, K. Li, Y. Zhao, I. Misra, J.-B. Huang,
P. Zhang, P. Vajda, et al.
Flowvid: Taming imperfect optical flows for consistent video-to-video
synthesis.
In *Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition*, pages 8207–8216, 2024.
- Liu et al. (2021)
Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo.
Swin transformer: Hierarchical vision transformer using shifted
windows.
In *Proceedings of the IEEE/CVF international conference on
computer vision*, pages 10012–10022, 2021.
- Loshchilov and Hutter (2017)
I. Loshchilov and F. Hutter.
Decoupled weight decay regularization.
*arXiv preprint arXiv:1711.05101*, 2017.
- Maier-Hein et al. (2017)
L. Maier-Hein, S. S. Vedula, S. Speidel, N. Navab, R. Kikinis, A. Park,
M. Eisenmann, H. Feussner, G. Forestier, S. Giannarou, et al.
Surgical data science for next-generation interventions.
*Nature Biomedical Engineering*, 1(9):691–696, 2017.
- Martyniak et al. (2025)
S. Martyniak, J. Kaleta, D. Dall Alba, M. Naskrket, S. Plotka, and
P. Korzeniowski.
Simuscope: Realistic endoscopic synthetic dataset generation through
surgical simulation and diffusion models.
In *2025 IEEE/CVF Winter Conference on Applications of Computer
Vision (WACV)*, pages 4268–4278. IEEE, 2025.
- Nasirihaghighi et al. (2025)
S. Nasirihaghighi, N. Ghamsarian, L. Peschek, M. Munari, H. Husslein,
R. Sznitman, and K. Schoeffmann.
Gynsurg: A comprehensive gynecology laparoscopic surgery dataset.
*arXiv preprint arXiv:2506.11356*, 2025.
- Nwoye et al. (2022a)
C. I. Nwoye, T. Yu, C. Gonzalez, B. Seeliger, P. Mascagni, D. Mutter,
J. Marescaux, and N. Padoy.
Rendezvous: Attention mechanisms for the recognition of surgical
action triplets in endoscopic videos.
*Medical Image Analysis*, 78:102433,
2022a.
- Nwoye et al. (2022b)
C. I. Nwoye, T. Yu, C. Gonzalez, B. Seeliger, P. Mascagni, D. Mutter,
J. Marescaux, and N. Padoy.
Rendezvous: Attention mechanisms for the recognition of surgical
action triplets in endoscopic videos.
*Medical Image Analysis*, 78:102433,
2022b.
- Nwoye et al. (2025)
C. I. Nwoye, R. Bose, K. Elgohary, L. Arboit, G. Carlino, J. L. Lavanchy,
P. Mascagni, and N. Padoy.
Surgical text-to-image generation.
*Pattern Recognition Letters*, 190:73–80, 2025.
- Padoy et al. (2012)
N. Padoy, T. Blum, S.-A. Ahmadi, H. Feussner, M.-O. Berger, and N. Navab.
Statistical modeling and recognition of surgical workflow.
*Medical image analysis*, 16(3):632–641,
2012.
- Peebles and Xie (2023)
W. Peebles and S. Xie.
Scalable diffusion models with transformers.
In *Proceedings of the IEEE/CVF international conference on
computer vision*, pages 4195–4205, 2023.
- Pfeiffer et al. (2019)
M. Pfeiffer, I. Funke, M. R. Robu, S. Bodenstedt, L. Strenger, S. Engelhardt,
T. Roß, M. J. Clarkson, K. Gurusamy, B. R. Davidson, et al.
Generating large labeled data sets for laparoscopic image processing
tasks using unpaired image-to-image translation.
In *International Conference on Medical Image Computing and
Computer-Assisted Intervention*, pages 119–127. Springer, 2019.
- Polyak et al. (2024)
A. Polyak, A. Zohar, A. Brown, A. Tjandra, A. Sinha, A. Lee, A. Vyas, B. Shi,
C.-Y. Ma, C.-Y. Chuang, et al.
Movie gen: A cast of media foundation models.
*arXiv preprint arXiv:2410.13720*, 2024.
- Psychogyios et al. (2023)
D. Psychogyios, E. Colleoni, B. Van Amsterdam, C.-Y. Li, S.-Y. Huang, Y. Li,
F. Jia, B. Zou, G. Wang, Y. Liu, et al.
Sar-rarp50: Segmentation of surgical instrumentation and action
recognition on robot-assisted radical prostatectomy challenge.
*arXiv preprint arXiv:2401.00496*, 2023.
- Radford et al. (2021)
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry,
A. Askell, P. Mishkin, J. Clark, et al.
Learning transferable visual models from natural language
supervision.
In *International conference on machine learning*, pages
8748–8763. PmLR, 2021.
- Ramesh et al. (2022)
A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen.
Hierarchical text-conditional image generation with clip latents.
*arXiv preprint arXiv:2204.06125*, 1(2):3,
2022.
- Rombach et al. (2022a)
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer.
High-resolution image synthesis with latent diffusion models.
In *Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition*, pages 10684–10695, 2022a.
- Rombach et al. (2022b)
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer.
High-resolution image synthesis with latent diffusion models.
In *Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition*, pages 10684–10695, 2022b.
- Saharia et al. (2022)
C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour,
R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, et al.
Photorealistic text-to-image diffusion models with deep language
understanding.
*Advances in neural information processing systems*,
35:36479–36494, 2022.
- Salmi et al. (2024)
M. Salmi, D. Atif, D. Oliva, A. Abraham, and S. Ventura.
Handling imbalanced medical datasets: review of a decade of research.
*Artificial intelligence review*, 57(10):273, 2024.
- Schmidt et al. (2024)
A. Schmidt, O. Mohareri, S. DiMaio, M. C. Yip, and S. E. Salcudean.
Tracking and mapping in medical computer vision: A review.
*Medical Image Analysis*, 94:103131, 2024.
- Schuhmann et al. (2022)
C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti,
T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al.
Laion-5b: An open large-scale dataset for training next generation
image-text models.
*Advances in neural information processing systems*,
35:25278–25294, 2022.
- Singer et al. (2022)
U. Singer, A. Polyak, T. Hayes, X. Yin, J. An, S. Zhang, Q. Hu, H. Yang,
O. Ashual, O. Gafni, et al.
Make-a-video: Text-to-video generation without text-video data.
*arXiv preprint arXiv:2209.14792*, 2022.
- Sivakumar et al. (2025)
S. K. Sivakumar, Y. Frisch, G. Ghazaei, and A. Mukhopadhyay.
Sg2vid: Scene graphs enable fine-grained control for video synthesis.
*arXiv preprint arXiv:2506.03082*, 2025.
- Sohl-Dickstein et al. (2015)
J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli.
Deep unsupervised learning using nonequilibrium thermodynamics.
In *International conference on machine learning*, pages
2256–2265. pmlr, 2015.
- Sørensen et al. (2016)
S. M. D. Sørensen, M. M. Savran, L. Konge, and F. Bjerrum.
Three-dimensional versus two-dimensional vision in laparoscopy: a
systematic review.
*Surgical endoscopy*, 30(1):11–23, 2016.
- Sun et al. (2024)
W. Sun, X. You, R. Zheng, Z. Yuan, X. Li, L. He, Q. Li, and L. Sun.
Bora: Biomedical generalist video generation model.
*arXiv preprint arXiv:2407.08944*, 2024.
- Taylor et al. (2016)
R. H. Taylor, A. Menciassi, G. Fichtinger, P. Fiorini, and P. Dario.
Medical robotics and computer-integrated surgery.
In *Springer handbook of robotics*, pages 1657–1684. Springer,
2016.
- Teed and Deng (2020)
Z. Teed and J. Deng.
Raft: Recurrent all-pairs field transforms for optical flow.
In *European conference on computer vision*, pages 402–419.
Springer, 2020.
- Twinanda et al. (2016)
A. P. Twinanda, S. Shehata, D. Mutter, J. Marescaux, M. De Mathelin, and
N. Padoy.
Endonet: a deep architecture for recognition tasks on laparoscopic
videos.
*IEEE transactions on medical imaging*, 36(1):86–97, 2016.
- Vaswani et al. (2017)
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,
Ł. Kaiser, and I. Polosukhin.
Attention is all you need.
*Advances in neural information processing systems*, 30, 2017.
- Venkatesh et al. (2024)
D. K. Venkatesh, D. Rivoir, M. Pfeiffer, F. Kolbinger, M. Distler, J. Weitz,
and S. Speidel.
Exploring semantic consistency in unpaired image translation to
generate data for surgical applications.
*International journal of computer assisted radiology and
surgery*, 19(6):985–993, 2024.
- Venkatesh et al. (2025a)
D. K. Venkatesh, I. Funke, M. Pfeiffer, F. Kolbinger, H. M. Schmeiser,
M. Distler, J. Weitz, and S. Speidel.
Mission Balance: Generating Under-represented Class Samples using
Video Diffusion Models .
In *proceedings of Medical Image Computing and Computer Assisted
Intervention – MICCAI 2025*, volume LNCS 15970. Springer Nature Switzerland,
September 2025a.
- Venkatesh et al. (2025b)
D. K. Venkatesh, D. Rivoir, M. Pfeiffer, F. Kolbinger, and S. Speidel.
Data augmentation for surgical scene segmentation with anatomy-aware
diffusion models.
In *2025 IEEE/CVF Winter Conference on Applications of Computer
Vision (WACV)*, pages 2280–2290. IEEE, 2025b.
- Wang et al. (2022)
Z. Wang, B. Lu, Y. Long, F. Zhong, T.-H. Cheung, Q. Dou, and Y. Liu.
Autolaparo: A new dataset of integrated multi-tasks for image-guided
surgical automation in laparoscopic hysterectomy.
In *International Conference on Medical Image Computing and
Computer-Assisted Intervention*, pages 486–496. Springer, 2022.
- Wang et al. (2024)
Z. Wang, L. Zhang, L. Wang, M. Zhu, and Z. Zhang.
Optical flow representation alignment mamba diffusion model for
medical video generation.
*arXiv preprint arXiv:2411.01647*, 2024.
- Yang et al. (2024a)
Z. Yang, J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y. Yang, W. Hong,
X. Zhang, G. Feng, et al.
Cogvideox: Text-to-video diffusion models with an expert transformer.
*arXiv preprint arXiv:2408.06072*, 2024a.
- Yang et al. (2024b)
Z. Yang, J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y. Yang, W. Hong,
X. Zhang, G. Feng, et al.
Cogvideox: Text-to-video diffusion models with an expert transformer.
*arXiv preprint arXiv:2408.06072*, 2024b.
- Yeganeh et al. (2024)
Y. Yeganeh, R. Lazuardi, A. Shamseddin, E. Dari, Y. Thirani, N. Navab, and
A. Farshad.
Visage: Video synthesis using action graphs for surgery.
In *International Conference on Medical Image Computing and
Computer-Assisted Intervention*, pages 146–156. Springer, 2024.
- Yun et al. (2019)
S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo.
Cutmix: Regularization strategy to train strong classifiers with
localizable features.
In *Proceedings of the IEEE/CVF international conference on
computer vision*, pages 6023–6032, 2019.
- Zhang et al. (2017)
H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz.
mixup: Beyond empirical risk minimization.
*arXiv preprint arXiv:1710.09412*, 2017.
- Zhang et al. (2018)
R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang.
The unreasonable effectiveness of deep features as a perceptual
metric.
In *Proceedings of the IEEE conference on computer vision and
pattern recognition*, pages 586–595, 2018.
- Zhao et al. (2025a)
C. Zhao, P. Guo, D. Yang, Y. Tang, Y. He, B. Simon, M. Belue, S. Harmon,
B. Turkbey, and D. Xu.
Maisi-v2: Accelerated 3d high-resolution medical image synthesis with
rectified flow and region-specific contrastive loss.
*arXiv preprint arXiv:2508.05772*, 2025a.
- Zhao et al. (2025b)
S. Zhao, L. Bai, K. Yuan, F. Li, J. Yu, W. Dong, G. Wang, M. Islam, N. Padoy,
N. Navab, et al.
Rethinking data imbalance in class incremental surgical instrument
segmentation.
*Medical Image Analysis*, page 103728, 2025b.
- Zheng et al. (2024)
Z. Zheng, X. Peng, T. Yang, C. Shen, S. Li, H. Liu, Y. Zhou, T. Li, and Y. You.
Open-sora: Democratizing efficient video production for all.
*arXiv preprint arXiv:2412.20404*, 2024.
- Zhong et al. (2020)
Z. Zhong, L. Zheng, G. Kang, S. Li, and Y. Yang.
Random erasing data augmentation.
In *Proceedings of the AAAI conference on artificial
intelligence*, volume 34, pages 13001–13008, 2020.

## Appendix A Extended Related Work

#### Video diffusion models

Diffusion-based video generation methods have recently demonstrated strong efficiency and scalability by operating in continuous latent spaces (Ho et al., 2020, Rombach et al., 2022a). Early work by (Ho et al., 2022b) extended pixel-space diffusion to videos using probabilistic DMs, while (Harvey et al., 2022) proposed generating sparse frames with interpolation, though limited to low-resolution synthetic datasets. Large-scale efforts such as Make-A-Video (Singer et al., 2022) and Imagen Video (Ho et al., 2022a) employ cascaded super-resolution pipelines built on DALLE-2 (Ramesh et al., 2022) and Imagen (Saharia et al., 2022), respectively, but require billions of parameters and massive compute resources. Stable Video Diffusion (Blattmann et al., 2023) has been widely adopted in the natural image/video community, while several closed-source systems—such as MovieGen (Polyak et al., 2024), Pika (pik, 2025), and Gen (Runway) (run, 2025), Veo (veo, 2025)—achieve high-quality generation conditioned on diverse modalities ranging from text to depth maps.

On the open-source side, AnimateDiff (Guo et al., 2023) and SparseCtrl (Guo et al., 2024) extend image diffusion models to videos, while OpenSora (Zheng et al., 2024) represents a large community-driven effort to replicate Sora (sor, 2025). The CogVideo family (Yang et al., 2024b, Hong et al., 2022) introduces expert transformer architectures for video synthesis and has been adopted in prior surgical applications (Biagini et al., 2025, Iliash et al., 2024). However, CogVideo is a $5$B parameter model requiring vast datasets and heavy compute, making it impractical for limited surgical data where overfitting is a risk. We inspire our approach from the more recently proposed methods such as FlowVid (Liang et al., 2024) and VideoJam (Chefer et al., 2025). FlowVid proposed a flow warped video-to-video generation framework, wherein optical flow was used to maintain the structure of objects between frames during translation. This framework trained on a corpus of $10$M videos. The primary application of this work differs from ours such that we intend to generate new videos with conditional signals. Secondly, VideoJam explored video prediction with a DiT-based architecture (Peebles and Xie, 2023), but its $30$B parameter model was trained on $100$M videos, produces only $256\times 256$ outputs, and lacks controllability—an essential requirement for surgical applications.

In contrast, our work targets the surgical domain under constrained compute budgets, focusing on the critical issue of data imbalance. We build upon small-scale surgical video diffusion models and introduce a sparse, controllable framework tailored to generate under-represented surgical classes. To the best of our knowledge, we are the first to introduce a conditional video diffusion framework to mitigate the data imbalance issue for surgical application. While future work could explore scaling to larger models, our approach demonstrates a practical pathway toward improving surgical video understanding in realistic healthcare settings.

#### Data imbalance

The presence of rare classes is a common challenge in real-world datasets. In classification, oversampling is frequently used to mitigate this issue by sampling under-represented classes more often during training (Belhaouari et al., 2024). Standard augmentation methods such as horizontal flipping, random resizing, and cropping are widely used, while regional dropout methods (Zhong et al., 2020) randomly remove image regions to improve robustness and generalization. More advanced strategies, including RandAugment (Cubuk et al., 2019b) and AutoAugment (Cubuk et al., 2019a), apply diverse pixel-level operations (e.g., rotation, shear, translation, color jitter) through either random selection or learned policies. Other approaches combine multiple images, such as Mixup (Zhang et al., 2017), which blends both pixel values and labels, and CutMix (Yun et al., 2019), which replaces patches from one image with regions from another, maximizing pixel efficiency while mixing labels. These augmentation strategies have been specifically proposed for image classification tasks. Readers can refer to (Chen et al., 2024b) for a detailed survey.

Within the surgical domain, class imbalance is particularly prevalent due to challenges in data collection (e.g., reliance on single-center data), the rarity of specific surgical events, and ethical or legal restrictions on data sharing (Salmi et al., 2024, Maier-Hein et al., 2017). Such imbalance often degrades the performance of downstream models. While augmentation and re-sampling strategies have been shown to improve medical imaging tasks (Salmi et al., 2024), surgical video understanding tasks lacks dedicated augmentation approaches. Prior attempts have used synthetic data, for instance via image-to-image translation, to complement real datasets for only surgical instrument segmentation tasks (Colleoni et al., 2022, Colleoni and Stoyanov, 2021, Zhao et al., 2025b). In this work, we establish a strong baseline for real datasets by combining curated image-level augmentation techniques with inverse frequency balancing, which up-weights under-represented classes. We use this strategy only during the training of real datasets. To directly assess the utility of synthetic data as a complementary augmentation strategy, we merge generated videos with real data without applying further augmentations.

## Appendix B Additional results

### B.1 Synthetic data attributes

The results on different aspects of synthetic data for the SAR-RARP$50$ dataset are presented in Tab. [6](https://arxiv.org/html/2510.07345v1#A2.T6). Performance remains unchanged when the training data is merely duplicated, a trend consistent across most classes. In contrast, perturbations to either the spatial or temporal structure of the videos result in clear performance degradation. This behavior aligns with the role of the downstream model, which relies on both spatial structure (e.g., the arrangement of organs) and temporal dynamics (e.g., tissue motion and single or multi-tool interactions) to classify an action. Notably, the action class “cutting the suture,” which is already highly imbalanced, suffers a substantial drop in performance when frame-level noise is introduced. Similar results were noticed for the GraSP dataset (Tab. [7](https://arxiv.org/html/2510.07345v1#A2.T7)). Interestingly we noticed for shuffling the frames lead to a small improvement in scores for two of the under-represented classes. This results could also be attributed to the downstream model architecture difference between the TAPIS model and the plain MViT model. However, overall these findings highlight that synthetic data cannot simply replicate training samples, nor can it exhibit spatial or temporal inconsistencies, if it is to provide meaningful benefits for downstream tasks.

**Table 6: Attributes of synthetic data experiment on the SAR-RARP$50$ dataset. Merely replicating the training data does not lead to any improvement in performance. The degradation of the spatial or temporal structure leads to a decline in downstream model performance.**
| Training data | Pick<br>the needle | Position<br>the needle | Push<br>the needle | Pull<br>the needle | Cut<br>the suture | Return<br>the needle | Mean. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Real | $0.32_{\pm 0.19}$ | $0.66_{\pm 0.09}$ | $0.78_{\pm 0.10}$ | $0.61_{\pm 0.09}$ | $0.10_{\pm 0.04}$ | $0.32_{\pm 0.15}$ | $0.46_{\pm 0.08}$ |
| Data duplication | $0.32_{\pm 0.17}$ | $0.60_{\pm 0.03}$ | $0.78_{\pm 0.08}$ | $0.61_{\pm 0.10}$ | $0.10_{\pm 0.03}$ | $0.31_{\pm 0.11}$ | $0.45_{\pm 0.06}$ |
| Frame shuffle | $0.30_{\pm 0.19}$ | $0.63_{\pm 0.08}$ | $0.74_{\pm 0.11}$ | $0.60_{\pm 0.08}$ | $0.06_{\pm 0.09}$ | $0.30_{\pm 0.17}$ | $0.43_{\pm 0.04}$ |
| Sparse frame | $0.28_{\pm 0.14}$ | $0.60_{\pm 0.07}$ | $0.70_{\pm 0.04}$ | $0.59_{\pm 0.09}$ | $0.05_{\pm 0.05}$ | $0.29_{\pm 0.10}$ | $0.42_{\pm 0.03}$ |
| Noisy frame | $0.29_{\pm 0.14}$ | $0.62_{\pm 0.07}$ | $0.76_{\pm 0.04}$ | $0.60_{\pm 0.09}$ | $0.04_{\pm 0.05}$ | $0.29_{\pm 0.10}$ | $0.43_{\pm 0.02}$ |

**Table 7: Attributes of synthetic data experiment on the GraSP dataset.**
| Training data | Pull<br>the suture | Tie<br>the suture | Cut<br>the suture | Cut<br>btw.the prostate | Identify<br>iliac artery | Mean. |
| --- | --- | --- | --- | --- | --- | --- |
| Real | $0.26_{\pm 0.03}$ | $0.44_{\pm 0.01}$ | $0.43_{\pm 0.06}$ | $0.72_{\pm 0.07}$ | $0.52_{\pm 0.08}$ | $0.46_{\pm 0.08}$ |
| Data duplication | $0.25_{\pm 0.02}$ | $0.44_{\pm 0.02}$ | $0.43_{\pm 0.05}$ | $0.71_{\pm 0.06}$ | $0.52_{\pm 0.04}$ | $0.46_{\pm 0.04}$ |
| Frame shuffle | $0.27_{\pm 0.04}$ | $0.40_{\pm 0.02}$ | $0.42_{\pm 0.01}$ | $0.69_{\pm 0.03}$ | $0.53_{\pm 0.04}$ | $0.46_{\pm 0.02}$ |
| Sparse frame | $0.24_{\pm 0.02}$ | $0.38_{\pm 0.03}$ | $0.40_{\pm 0.02}$ | $0.68_{\pm 0.02}$ | $0.48_{\pm 0.01}$ | $0.43_{\pm 0.02}$ |
| Noisy frame | $0.20_{\pm 0.04}$ | $0.35_{\pm 0.05}$ | $0.34_{\pm 0.06}$ | $0.66_{\pm 0.03}$ | $0.46_{\pm 0.05}$ | $0.40_{\pm 0.04}$ |

### B.2 Additional Surgical action dataset

We further evaluated surgical action recognition on the GynSurg dataset (Nasirihaghighi et al., 2025), which consists of laparoscopic gynecological procedures with four annotated actions: coagulation (P$1$), needle passing (P$2$), suction/irrigation (P$3$), and transection (P$4$). The classes P$3$ and P$4$ are under-represented. Each action is provided as short $3$-second video clips, making the dataset well-suited for action recognition. Importantly, this dataset differs substantially from SAR-RARP$50$ and GraSP in terms of anatomy, environment, tool usage, and camera motion, allowing us to demonstrate the generalizability of our approach across diverse surgical settings. We adopt the MViTv$2$ model as the downstream architecture.

Results are reported in Fig. [6](https://arxiv.org/html/2510.07345v1#A2.F6). Synthetic samples from SparseCtrl improve performance by $8$–$9\%$ for the under-represented classes. In contrast, our method with text conditioning achieves consistent gains across all four classes, raising the average Jaccard score to $0.72$ compared to $0.66$ with real data only. Conditioning with RGB frames yields further improvements of nearly $20$ points for P$3$ and P$4$. These results highlight the advantage of combining dual-prediction with sparse visual encoding to generate synthetic videos that preserve both spatial and temporal consistency.

Figure: Figure 6: Surgical action recognition results on the GynSurg dataset, reported using the F$1$ score. The under-represented classes are “Suction” and “Transection”. The addition of synthetic samples for both the balanced classes shows smaller improvements. However, the synthetic video samples from our approach (SurgiFlowVid) with text conditioning improves performance for both under-represented classes, while sparse RGB frame conditioning yields gains of up to $20$ points in comparison to using only the real dataset.
Refer to caption: x4.png

### B.3 Model architecture

We further analyzed the impact of synthetic data using a different architecture for action recognition on SAR-RARP$50$. Since the MViT model is purely transformer-based, we tested whether synthetic samples introduce any architectural bias by comparing against X3D (Feichtenhofer, 2020), a lightweight $3$D convolutional model with only $3$M parameters (vs. $30$M for MViT). The evaluation setup remained identical to previous experiments. The results are shown in Tab. [8](https://arxiv.org/html/2510.07345v1#A2.T8). Compared to Tab. [2](https://arxiv.org/html/2510.07345v1#S5.T2), the mean Jaccard score with real data dropped to $0.38$ for X3D (vs. $0.46$ for MViT), as expected given the smaller capacity of X3D.

**Table 8: Influence of model architecture. The surgical action recognition task on the SAR-RARP$50$ dataset using X$3$D model. The Jaccard index is reported. Best and second-best scores are highlighted in blue and green, respectively. Under-represented classes are indicated with shade. We notice similar trends to Tab. [2](https://arxiv.org/html/2510.07345v1#S5.T2), where the addition of samples from our approach leads to performance gains for all the under-represented classes.**
| Training data | Cond. type | Pick<br>the needle | Position<br>the needle | Push<br>the needle | Pull<br>the needle | Cut<br>the suture | Return<br>the needle | Mean. |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Text | Sparse mask |  |  |  |  |  |  |  |  |
| Real | – | – | $0.22_{\pm 0.01}$ | $0.54_{\pm 0.08}$ | $0.75_{\pm 0.07}$ | $0.51_{\pm 0.13}$ | $0.10_{\pm 0.02}$ | $0.20_{\pm 0.12}$ | $0.38_{\pm 0.06}$ |
| Real $+$ Endora | – | – | $0.19_{\pm 0.04}$ | $0.53_{\pm 0.02}$ | $0.75_{\pm 0.05}$ | $0.50_{\pm 0.10}$ | $0.09_{\pm 0.05}$ | $0.18_{\pm 0.04}$ | $0.38_{\pm 0.06}$ |
| Real $+$ SurV-Gen (w/o RS) | ✓ | – | $0.22_{\pm 0.10}$ | $0.54_{\pm 0.04}$ | $0.75_{\pm 0.02}$ | $0.51_{\pm 0.08}$ | $0.11_{\pm 0.09}$ | $0.19_{\pm 0.08}$ | $0.39_{\pm 0.07}$ |
| Real $+$ SurV-Gen (RS) | ✓ | – | $0.23_{\pm 0.11}$ | $0.54_{\pm 0.06}$ | $0.74_{\pm 0.07}$ | $0.52_{\pm 0.11}$ | $0.10_{\pm 0.09}$ | $0.23_{\pm 0.16}$ | $0.39_{\pm 0.06}$ |
| Real $+$ SparseCtrl | ✓ | RGB | $0.34_{\pm 0.17}$ | $0.60_{\pm 0.07}$ | $0.77_{\pm 0.08}$ | $0.58_{\pm 0.09}$ | $0.08_{\pm 0.05}$ | $0.23_{\pm 0.16}$ | $0.43_{\pm 0.03}$ |
| Real $+$ SparseCtrl | ✓ | Seg. | $0.33_{\pm 0.14}$ | $0.58_{\pm 0.06}$ | $0.75_{\pm 0.07}$ | $0.57_{\pm 0.13}$ | $0.09_{\pm 0.03}$ | $0.28_{\pm 0.17}$ | $0.43_{\pm 0.04}$ |
| Real $+$ SurgFlowVid | ✓ | – | $0.34_{\pm 0.13}$ | $0.58_{\pm 0.06}$ | $0.75_{\pm 0.05}$ | $0.55_{\pm 0.13}$ | $0.18_{\pm 0.09}$ | $0.29_{\pm 0.12}$ | $0.45_{\pm 0.04}$ |
| Real $+$ SurgFlowvid | ✓ | RGB | $0.30_{\pm 0.19}$ | $0.58_{\pm 0.06}$ | $0.74_{\pm 0.07}$ | $0.58_{\pm 0.10}$ | $0.10_{\pm 0.08}$ | $0.26_{\pm 0.17}$ | $0.43_{\pm 0.02}$ |
| Real $+$ SurgFlowVid | ✓ | Seg. | $0.39_{\pm 0.12}$ | $0.60_{\pm 0.05}$ | $0.76_{\pm 0.08}$ | $0.56_{\pm 0.12}$ | $0.13_{\pm 0.05}$ | $0.35_{\pm 0.12}$ | $0.47_{\pm 0.02}$ |

Synthetic data from SparseCtrl led to modest improvements, while SurgiFlowVid with text conditioning provided only subtle gains. However, consistent with trends in Tab. [2](https://arxiv.org/html/2510.07345v1#S5.T2), adding sparse RGB or segmentation masks as conditional signals in SurgiFlowVid yielded considerable improvements across the under-represented classes. Similar trends were noticed when we performed individual class modelling with the results shown in Tab. [9](https://arxiv.org/html/2510.07345v1#A2.T9). These findings suggest that performance gains from synthetic data are not biased toward a specific architecture; instead, both transformer- and convolution-based models benefit from the spatial and temporal consistency encoded in synthetic videos. For the GraSP dataset, we opted to use the TAPIS model as proposed in (Ayobi et al., 2024) as this model performed in par with other convolutional architectues.

Using the features extracted from downstream models, temporal models are trained to enhance action recognition further. However, the reported performance improvements were minimal (Funke et al., 2025), and we therefore did not pursue such experiments in this study. Future work could explore this direction in greater depth, focusing on identifying which features from synthetic data are most beneficial for improving the generation process. Additionally, incorporating temporal learning strategies on top of these features may provide further gains for surgical action recognition tasks.

**Table 9: Influence of model architecture. The surgical action recognition task on the SAR-RARP$50$ dataset using X$3$D model with *individual class modelling*. The Jaccard index is reported. We notice smaller gains for the action “cut the suture” (see Tab. [8](https://arxiv.org/html/2510.07345v1#A2.T8)) by modeling each of the under-represented classes separately.**
| Training data | Cond. type | Pick<br>the needle | Position<br>the needle | Push<br>the needle | Pull<br>the needle | Cut<br>the suture | Return<br>the needle | Mean. |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Text | Sparse mask |  |  |  |  |  |  |  |  |
| Real | – | – | $0.22_{\pm 0.01}$ | $0.54_{\pm 0.08}$ | $0.75_{\pm 0.07}$ | $0.51_{\pm 0.13}$ | $0.10_{\pm 0.02}$ | $0.20_{\pm 0.12}$ | $0.38_{\pm 0.06}$ |
| Real $+$ SurV-Gen (RS) | ✓ | – | $0.25_{\pm 0.12}$ | $0.54_{\pm 0.03}$ | $0.76_{\pm 0.09}$ | $0.51_{\pm 0.09}$ | $0.10_{\pm 0.13}$ | $0.24_{\pm 0.18}$ | $0.40_{\pm 0.05}$ |
| Real $+$ SparseCtrl | ✓ | RGB | $0.30_{\pm 0.16}$ | $0.59_{\pm 0.07}$ | $0.75_{\pm 0.06}$ | $0.57_{\pm 0.11}$ | $0.10_{\pm 0.09}$ | $0.21_{\pm 0.12}$ | $0.42_{\pm 0.03}$ |
| Real $+$ SparseCtrl | ✓ | Seg. | $0.30_{\pm 0.17}$ | $0.57_{\pm 0.04}$ | $0.76_{\pm 0.07}$ | $0.57_{\pm 0.09}$ | $0.20_{\pm 0.05}$ | $0.37_{\pm 0.10}$ | $0.46_{\pm 0.01}$ |
| Real $+$ SurgFlowvid | ✓ | RGB | $0.40_{\pm 0.16}$ | $0.56_{\pm 0.02}$ | $0.75_{\pm 0.04}$ | $0.56_{\pm 0.16}$ | $0.23_{\pm 0.13}$ | $0.35_{\pm 0.15}$ | $0.48_{\pm 0.02}$ |
| Real $+$ SurgFlowVid | ✓ | Seg. | $0.39_{\pm 0.11}$ | $0.59_{\pm 0.04}$ | $0.77_{\pm 0.03}$ | $0.55_{\pm 0.10}$ | $0.15_{\pm 0.06}$ | $0.40_{\pm 0.10}$ | $0.48_{\pm 0.05}$ |

### B.4 Video metrics

We assess the temporal performance of the model using Segmental F1@K score. This metric penalizes
both out-of-order predictions and over-segmentation. Segmental F1@K quantifies the temporal overlap between predicted and ground-truth segments, while being less sensitive to small boundary shifts caused by
annotation noise. The metric is defined as,

$$ $\text{SegmentalF1@K}=\frac{2\times(\text{Pr}\times\text{Rc})}{(\text{Pr}+\text{Rc})},$ (3) $$

where Pr and Rc denotes precision and recall. A prediction is considered a true positive (TP) if the IoU exceeds the threshold $T=K/100$; otherwise, it is counted as a false positive (FP). The results of the recognition task are shown in Tab.[10](https://arxiv.org/html/2510.07345v1#A2.T10) and Tab.[11](https://arxiv.org/html/2510.07345v1#A2.T11). Compared to using only the real dataset, the addition of synthetic samples leads to smaller improvements in overall performance. The addition of either RGB or segmentation conditioning lead to a similar scores of $0.37$ and $0.36$ respectively. Overall, the synthetic samples from SurgiFlowVid prove very beneficial for both the balanced and the under-represented classes.

**Table 10: Surgical action recognition on the SAR-RARP$50$ dataset. Segmental F1 scores are reported.**
| Training data | Cond. type | Pick<br>the needle | Position<br>the needle | Push<br>the needle | Pull<br>the needle | Cut<br>the suture | Return<br>the needle | Mean. |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Text | Sparse mask |  |  |  |  |  |  |  |  |
| Real | – | – | $0.28_{\pm 0.17}$ | ${0.40_{\pm 0.16}}$ | $0.62_{\pm 0.18}$ | $0.41_{\pm 0.14}$ | $0.09_{\pm 0.08}$ | $0.22_{\pm 0.18}$ | $0.32_{\pm 0.06}$ |
| Real $+$ Endora | – | – | $0.23_{\pm 0.13}$ | $0.38_{\pm 0.06}$ | $0.55_{\pm 0.09}$ | $0.41_{\pm 0.10}$ | $0.09_{\pm 0.09}$ | $0.21_{\pm 0.08}$ | $0.31_{\pm 0.08}$ |
| Real $+$ SurV-Gen (w/o RS) | ✓ | – | $0.26_{\pm 0.12}$ | $0.40_{\pm 0.04}$ | $0.55_{\pm 0.08}$ | $0.41_{\pm 0.06}$ | $0.12_{\pm 0.09}$ | $0.23_{\pm 0.12}$ | $0.33_{\pm 0.04}$ |
| Real $+$ SurV-Gen (RS) | ✓ | – | $0.27_{\pm 0.14}$ | $0.40_{\pm 0.15}$ | $0.58_{\pm 0.19}$ | $0.42_{\pm 0.18}$ | $0.20_{\pm 0.13}$ | $0.23_{\pm 0.18}$ | $0.35_{\pm 0.07}$ |
| Real $+$ SparseCtrl | ✓ | RGB | $0.32_{\pm 0.20}$ | $0.41_{\pm 0.16}$ | $0.57_{\pm 0.17}$ | $0.44_{\pm 0.15}$ | $0.10_{\pm 0.09}$ | $0.25_{\pm 0.11}$ | $0.35_{\pm 0.03}$ |
| Real $+$ SurgFlowVid | ✓ | – | $0.27_{\pm 0.14}$ | $0.40_{\pm 0.16}$ | $0.57_{\pm 0.16}$ | $0.43_{\pm 0.13}$ | $0.13_{\pm 0.08}$ | $0.16_{\pm 0.07}$ | $0.33_{\pm 0.04}$ |
| Real $+$ SurgFlowvid | ✓ | RGB | $0.31_{\pm 0.17}$ | $0.43_{\pm 0.17}$ | $0.59_{\pm 0.16}$ | $0.45_{\pm 0.10}$ | $0.15_{\pm 0.04}$ | $0.31_{\pm 0.12}$ | $0.37_{\pm 0.03}$ |

**Table 11: Surgical action recognition on the SAR-RARP$50$ dataset. Segmental F1 scores are reported. for seg. frame conditioning.**
| Training data | Cond. type | Pick<br>the needle | Position<br>the needle | Push<br>the needle | Pull<br>the needle | Cut<br>the suture | Return<br>the needle | Mean. |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Text | Sparse mask |  |  |  |  |  |  |  |  |
| Real | – | – | $0.28_{\pm 0.17}$ | ${0.40_{\pm 0.16}}$ | $0.62_{\pm 0.18}$ | $0.41_{\pm 0.14}$ | $0.09_{\pm 0.08}$ | $0.22_{\pm 0.18}$ | $0.32_{\pm 0.06}$ |
| Real $+$ SparseCtrl | ✓ | Seg | $0.33_{\pm 0.19}$ | $0.43_{\pm 0.14}$ | $0.60_{\pm 0.19}$ | $0.44_{\pm 0.15}$ | $0.12_{\pm 0.10}$ | $0.20_{\pm 0.10}$ | $0.35_{\pm 0.05}$ |
| Real $+$ SurgFlowvid | ✓ | Seg | $0.30_{\pm 0.14}$ | $0.42_{\pm 0.16}$ | $0.58_{\pm 0.14}$ | $0.43_{\pm 0.13}$ | $0.13_{\pm 0.08}$ | $0.32_{\pm 0.11}$ | $0.36_{\pm 0.02}$ |

### B.5 Ablation on sparse frames

We conducted an ablation study to examine the effect of the number of sparse RGB frames used during generation. We hypothesized that too few frames would provide insufficient controllability, while too many would replicate training data, reducing diversity. To test this, we varied the number of conditioning frames ($1,3,5,10,12$) and generated videos, comparing their performance against models trained solely on real data. Results are shown in Fig. [7](https://arxiv.org/html/2510.07345v1#A2.F7) (all minor classes modeled jointly) and Fig. [8](https://arxiv.org/html/2510.07345v1#A2.F8) (each class modeled separately). A consistent trend across both settings is that using only one frame yields performance similar to the real-only baseline, indicating limited consistency and, in some cases, degenerate generations. Conversely, conditioning on 12 of the 16 frames produced results close to the real dataset baseline, as little additional diversity was introduced. Based on these findings, we adopted a strategy of sampling 3–5 random frames from the real dataset as conditional inputs. These experiments were initially conducted with the X3D model, and the same frame distribution was subsequently applied across all experiments, including the SparseCtrl baseline.

Figure: Figure 7: Frame ablation. The ablation on the number of sparse RGB frames on the SAR-RARP$50$ dataset. The results consists of using a X$3$D model with all the minor classes modeled together.
Refer to caption: x5.png

Figure: Figure 8: Frame ablation. The ablation on the number of sparse RGB frames on the SAR-RARP$50$ dataset. The results consists of using a X$3$D model with all the minor classes modeled separately.
Refer to caption: x6.png

### B.6 Model Analysis

In this section, we analyze the model in terms of the video generation cost. The results are shown in Tab. [12](https://arxiv.org/html/2510.07345v1#A2.T12). In comparison to Endora, both SurgV-Gen and our approach have lesser number of training parameters as the training is conducted in different stages. Our method, SurgiFlowVid is capable of generating videos at the resolution of $512\times 512$ pixels whereas the baselines, SurV-Gen and SparseCtrl generates videos at $256\times 256$ pixels and Endora at $128\times 128$ pixels. We also train our approach at $512\times 512$ pixels. Our framework is capable of training at lower resolutions but we opted to train them at higher resolutions as it could be helpful for the downstream task. There exists certain organs or auxillary tool structures which appears to be very small in shape. Generating videos at higher resolution can benefit these downstream models to learn these spatial structures effectively. We noticed the benefits for the classification of *catheter* and *clamps* in SAR-RARP$50$ dataset with synthetic videos from SurgiFlowVid (see Tab. [4](https://arxiv.org/html/2510.07345v1#S5.T4)). However, an analysis on the video resolution for the downstream task could shed more insights and we leave that for future work. As we generate videos at higher resolution, our approach requires a small overhead in terms of training and sampling times. We believe with the innovations in high performant GPUs these costs could be lowered drastically.

**Table 12: Model analysis. The various parameters of the different baselines. SVE denotes the sparse visual encoder in our approach. The inference time was measured on a A100-40GB GPU.**
| Method | Trainable<br>params. (M) | Video<br>resolution | Sampling<br>steps | Inf.<br>time(sec) |
| --- | --- | --- | --- | --- |
| Endora | $675$ | $128\times 128$ | $50$ | $7.85$s |
| SurV-Gen | $435$ | $256\times 256$ | $50$ | $6.55$s |
| SurgiFlowVid | $437$ | $512\times 512$ | $50$ | $7.53$s |
| SparseCtrl | $453$ | $256\times 256$ | $30$ | $10.20$s |
| SurgiFlowVid + SVE | $456$ | $512\times 512$ | $30$ | $10.45$s |

**Table 13: Image quality metrics. The CLIP image score of different methods are reported here. Higher is better.**
| Method | SAR-RARP$50$ | GynSurg | GraSP |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | A$1$ | A$5$ | A$7$ | P$3$ | P$4$ | G$1$ | G$2$ | G$3$ | G$4$ |
| Endora | $70.30$ | $66.85$ | $73.65$ | $69.43$ | $70.12$ | $57.09$ | $68.10$ | $74.41$ | $60.72$ |
| SurV-Gen | $75.30$ | $70.22$ | $78.85$ | $71.30$ | $75.83$ | $62.15$ | $73.10$ | $68.32$ | $62.15$ |
| SurgiFlowVid | $74.46$ | $76.08$ | $78.25$ | $72.95$ | $66.76$ | $68.20$ | $70.10$ | $72.15$ | $65.27$ |

### B.7 Image Quality Analysis

As our goal is to mitigate data imbalance, we focused primarily on generating videos of under-represented classes and evaluating them on the downstream task. We consider this approach as an effective way to directly measure the effectiveness and the usefulness of the synthetic videos. In this section, we evaluate the quality of the generated videos with the CLIP (Hessel et al., 2021) image and the LPIPS (Zhang et al., 2018) score. Both these metrics evaluate the quality of the generated frames using features from pre-trained models on large-scale natural images. The results are shown in Tab. [13](https://arxiv.org/html/2510.07345v1#A2.T13) and Tab. [14](https://arxiv.org/html/2510.07345v1#A2.T14). We compare our approach, SurgiFlowVid with text conditioning against Endora and SurV-Gen. We do not compute these scores for SparseCtrl or sparse visual encoder using our approach, as there already exists frames from the real dataset. The image quality varied between different classes and we did not notice a co-relation between these scores to the downstream model performance. Hence, these values should be interpreted with caution given that they are computed with pre-trained weights from models not trained on surgical images/videos.

**Table 14: Image quality metrics. The LPIPS score of different methods are reported here. Lower is better.**
| Method | SAR-RARP$50$ | GynSurg | GraSP |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | A$1$ | A$5$ | A$7$ | P$3$ | P$4$ | G$1$ | G$2$ | G$3$ | G$4$ |
| Endora | $0.70$ | $0.53$ | $0.59$ | $0.54$ | $0.49$ | $0.63$ | $0.66$ | $0.65$ | $0.63$ |
| SurV-Gen | $0.68$ | $0.54$ | $0.57$ | $0.51$ | $0.56$ | $0.57$ | $0.67$ | $0.71$ | $0.74$ |
| SurgiFlowVid | $0.66$ | $0.56$ | $0.52$ | $0.49$ | $0.50$ | $0.51$ | $0.60$ | $0.74$ | $0.72$ |

### B.8 Laparoscope motion

In addition to the F$1$ score, we also computed the balanced accuracy as an additional metric. Fig. [9](https://arxiv.org/html/2510.07345v1#A2.F9) shows the results on the laparoscope motion prediction task. Similar to the results seen in Fig. [5](https://arxiv.org/html/2510.07345v1#S5.F5), the overall scores are higher for the the offline recognition.

Figure: Figure 9: Laparoscope motion prediction on the Autolaparo dataset. Bars show mean score with standard deviation (error bars).
Refer to caption: x7.png

## Appendix C Dataset

SAR-RARP$50$: The dataset consists actions annotated at $10$ fps. Our initial experiments indicated this temporal frame to be very fine and hence we chose to sample the frames at $5$ fps. The annotations for the surgical tools were available at $1$ fps making it sparse in nature. For the sparse conditional generation, we randomly samples video frames in the range $3-5$ and place them in a different temporal order than the real dataset, so as to create the synthetic data as diverse as possible. For the sparse segmentation conditioning, we opted to include a minimum of $4$ frames in the $16$ frames video clips during training and sampling time.
GraSP: This dataset consists of annotations at both $30$ and $1$ fps temporal windows. As $1$fps was very coarse in nature, we opted to sample frames at $5$ fps from the $30$ fps annotations. The segmentation annotations were available at every $35$ seconds making them very sparse in nature. Based on dataset analysis, we noticed that creating video clips with at least one segmentation frame as conditioning for the under-represented samples were very challenging. Hence, we opted out of segmentation frames conditioning for the sparse visual encoder in our experiments. However, for the surgical tool presence detection task, we sampled a minimum of $4$ frames around the available segmentation frame and used it as the conditioning to generate videos for this task.

The details on the addition of synthetic samples are shown in Tab. [15](https://arxiv.org/html/2510.07345v1#A3.T15).

**Table 15: Dataset details. The values in the table include the total number of video clips from the training set. We add only synthetic samples to the under-represented classes to match and balance the instances with the well balanced classes.**
| Dataset | Step/action class | Data points<br>in real dataset | Added<br>syn. samples |
| --- | --- | --- | --- |
| SAR-RARP$50$ | Pick the needle | $332$ | $900$ |
| Position the needle | $1329$ | - |  |
| Push the needle | $1395$ | - |  |
| Pull the needle | $1208$ | - |  |
| Cut the suture | $115$ | $1100$ |  |
| Return the needle | $168$ | $1100$ |  |
| GraSP | Pull the suture | $992$ | $1600$ |
| Tie the suture | $712$ | $1800$ |  |
| Cut the suture | $1213$ | $1300$ |  |
| Cut btw.<br>the prostate | $1616$ | $1000$ |  |
| Identify iliac artery | $2800$ | - |  |
| GynSurg | Coagulation | $690$ | - |
| Needle passing | $869$ | - |  |
| Suction/Irrigation | $267$ | $550$ |  |
| Transection | $168$ | $650$ |  |

## Appendix D model training

### D.1 Diffusion Image pre-training

We build upon the SurV-Gen model (Venkatesh et al., 2025a), which was initially proposed to generate synthetic samples of under-represented classes to mitigate data imbalance in surgical datasets. The framework adopts a multi-stage training procedure. In the first stage, frames are extracted from the training split of surgical videos and a $2$D Stable Diffusion (SD) model (Rombach et al., 2022a) is trained. We follow the same pipeline with several modifications. Training the spatial SD directly on the limited frames from the downstream task datasets can result in overfitting, reduced diversity of generated frames, or potential data leakage. This phenomenon was observed in SurV-Gen, where synthetic augmentation yielded only marginal improvements without rejection sampling.

To address this issue, we curated an in-house dataset comprising video recordings from different surgical procedures. The dataset consists of approximately $7000$ clips, each ranging from $6$ to $8$ minutes in length. From this collection, we extracted $\sim 4000$ frames to train the $2$D component of the model. We initialized training from the SD-v$1.5$ checkpoint, pre-trained on the large-scale LAION-5B dataset (Schuhmann et al., 2022), which provided a strong initialization compared to training from scratch. The model was fine-tuned for $3000$ steps using the AdamW optimizer (Loshchilov and Hutter, 2017) with a learning rate of $1\text{e}^{-4}$, a batch size of $2$, and gradient checkpointing enabled. Due to computational constraints, frames were resized from their original resolution of $1048\times 2048$ to $512\times 512$. For text conditioning, we employed simple prompts such as “An image of a surgical procedure”, with embeddings generated using the CLIP text encoder (Radford et al., 2021). This fine-tuned SD model served as the base $2$D diffusion prior for any subsquent $2$D diffusion models. We fine-tune this model on the downstream datasets before video diffusion training. The spatial priors are learnt during this stage.

### D.2 Diffusion Video pre-training

Next, we focus on the video training stage. In the SurV-Gen approach, the spatial layers are frozen and only the temporal attention layers are trained during the second stage. In contrast, our framework trains the temporal layers jointly with both RGB and optical flow frames. To further improve temporal modeling, we investigated a video pre-training strategy inspired by previous works on video diffusion models (Rombach et al., 2022b, Polyak et al., 2024). Our hypothesis is that temporal motion priors, such as the movement of tools, tissue motions andpartially tool tissue interactions can be better learned by training on the unconditional internally curated dataset, which contains diverse anatomical structures, varying illumination conditions, different endoscope motions, and a wide range of surgical tools and tool interactions. This dataset introduces substantial variability that more closely reflects real-world surgical scenarios.

To test this, we extended SurV-Gen and trained it in two ways, keeping the training recipe unchanged (i.e., only the temporal attention layers are updated). First, we trained SurV-Gen directly on the SAR-RARP$50$ dataset, where the $2$D SD backbone was also trained on frames extracted from the same dataset. Second, we replaced the $2$D SD backbone with our fine-tuned $2$D model and pre-trained the temporal layers on the curated dataset of $\sim 7000$ videos. For this, we created overlapping subsets of $3000$, $5000$, and $7000$ videos, each containing at least $1500$ new clips. The pre-trained temporal layers were then fine-tuned on SAR-RARP$50$.

This pre-training strategy is expected to accelerate learning of spatio-temporal representations from the limited SAR-RARP$50$ data. We then generated synthetic samples of under-represented classes using label guidance, following SurV-Gen, and evaluated their impact on downstream action recognition performance. The results are shown in Fig. [10](https://arxiv.org/html/2510.07345v1#A4.F10).

Figure: Figure 10: The results on video pre-training.
Refer to caption: x8.png

We analyzed only the three under-represented classes and report the weighted average Jaccard index of these classes. We notice that the pre-training strategy leads to higher recognition scores in comparison to using only the real dataset for the same number of training steps. We noticed smaller dips in performance for the $5$k and $7$k samples, which could be attributed to a distributional shift to the SAR-RARP$50$ dataset. On the other hand, we noticed a continuous improvement in jaccard scores for the $3$k samples. Overall, these results indicate that the pre-training strategy leads to learning the spatio-temporal relationships better, such that when minimal data is available, the model can learn faster. Based on these results, we used the $2$D spatial SD model and temporal attention layers pre-trained on our internal dataset as the starting checkpoints for the SurgiFlowVid training scheme.

### D.3 SurgiFlowVid training

Based on these results we opted to use the temporal layers trained on our internal dataset as the pre-trained model. This offers the advantage that, the SurgiFlowVid training time reduces and also we can avoid the over-fitting of the dataset given the fact that there exists only limited training data from the downstream datasets. We fine-tune the pre-trained temporal attention layers using our proposed dual-prediction U-net module. The optical flow frames are extracted using the RAFT model (Teed and Deng, 2020). For SurgiFlowVid training, we extract clips of $16$ frames at a frame rate of $5$ for all the datasets. The hyperparameter details are mentioned in Tab. [16](https://arxiv.org/html/2510.07345v1#A4.T16).

### D.4 Downstream model training

For the action recognition task (SAR-RARP$50$), we used the MviT-v2 model from the SlowFast library(^3^33[https://github.com/facebookresearch/SlowFast](https://github.com/facebookresearch/SlowFast)). We downsampled the videos to $224\times 384$ pixels for training with a temporal resolution of $5$ fps. Image augmentations such as PCA jitter, RGB scale shift, brightness and contrast shift, random flipping with scale cropping was used along with inverse frequency balancing during the training on the real data. For additional details on the model, readers can refer to SlowFast repo. We followed the similar recipe for the GynSurg dataset. The model were trained for $150$ epochs with a learning rate of $1e^{4}$ with the best model being chosen using a validation dataset.

For the GraSP dataset, we used the similar settings from the TAPIS model(^4^44[https://github.com/BCV-Uniandes/GraSP/tree/main/TAPIS](https://github.com/BCV-Uniandes/GraSP/tree/main/TAPIS)). It is to be noted that we do not compare the values directly to the work from (Ayobi et al., 2024) on the GraSP dataset. This is due to the fact that the results reported from the TAPIS model have been obtained directly using the test set as the selection criteria during training. We create a separate validation set from the training set which we use as the selection criteria of the trained model. The test set is clearly separated during the training of both diffusion and downstream models to avoid any data leakage. For the combined training of real and synthetic videos, we opted for a simple and easier strategy than rejection sampling as proposed in (Venkatesh et al., 2025a). We sampled a batch of data points such that $25\%$ of this batch consists of synthetic videos. We chose this method as it works on the fly during training and the time and effort in rejecting synthetic samples are drastically reduced.

For the surgical tool presence detection task, we used the Swin transformer model. The videos were resized to a resolution of $384\times 384$ during training with augmentations such as RGB channel shift, scaled cropping and temporal shift. We trained the models using binary cross entropy loss with weighted sampling to include the imbalance in the surgical tools.

**Table 16: Hyperparameters for training the $2$D and the temporal attention layers of the diffusion model. SVE denotes *Sparse visual encoder* used for conditional generation.**
| Hyperparameter | Image fine-tuning | Video-pretraining | SurgiFlowVid training |
| --- | --- | --- | --- |
| Datatset |  |  |  |
| No. of samples | $4000$ | $7000$ | Train split of the dataset |
| Resolution | $512\times 512$ | $256\times 256$ & $512\times 512$ | $512\times 512$ |
| Video length | - | $16$ frames | $16$ frames |
| Sample rate | - | $5$ | $4$-$5$ |
| Context length | - | $16$ | $16$ |
| Model params |  |  |  |
| Pre-trained model | SDv-$1.5$ | Pre-trained on<br>internal | Pre-trained on<br>internal |
| Params frozen | - | Spatial layers | Spatial layers |
| Temporal layers |  |  |  |
| Depth | - | $2$ | $2$ |
| Temporal resolution | - | $[1,2,4,8]$ | $[1,2,4,8]$ |
| Head channels | - | $16$ | $16$ |
| No. of heads | - | $8$ | $8$ |
| Position encoding | - | sinusoidal | sinusoidal |
| PE dim | - | $24$ | $24$ |
| Cross attention dim | - | $32$ | $32$ |
| Act.function | - | GeLU | GeLU |
| Training params |  |  |  |
| Optimizer | AdamW | AdamW | AdamW |
| Learning rate | $1e^{-4}$ | $1e^{-5}$ | $1e^{-5}$ |
| Lr warm steps | $500$ | $5000$ | $5000$ |
| Lr scheduler | cosine | cosine | cosine |
| $\beta_{1}$ | $0.9$ | $0.9$ | $0.94$ |
| $\beta_{2}$ | $0.999$ | $0.999$ | $0.995$ |
| Weight decay $\omega$ | $1e^{-2}$ | - | - |
| Train steps | $3000$ | $125$k | $75-125$k |
| Train timestep |  |  |  |
| Diffusion step | $1000$ | $1000$ | $1000$ |
| Noise schedule | linear | linear | linear |
| $\beta_{0}$ | $1e-^{4}$ | $0.00085$ | $0.00085$ |
| $\beta_{T}$ | $0.02$ | $0.012$ | $0.012$ |
| Sampling params |  |  |  |
| Sampler | DDPM | DDIM | DDIM |
| Steps | - | 50 | 50 (30 for SVE) |
| CFG scale | $6.5$ | $5.5$ | $5.0$ |
| Device requirements |  |  |  |
| GPU-type | A100-40GB | H200-80GB | H200-140GB |
| No. of gpus | 1 | 1 | 1 |

## Appendix E Qualitative Results

Figure: Figure 11: Results from SurgiFlowVid with text conditioning on GraSP dataset.
Refer to caption: x9.png

Figure: Figure 12: Results from SurgiFlowVid with text conditioning on GraSP dataset.
Refer to caption: x10.png

Figure: Figure 13: Results from SurgiFlowVid with text conditioning on GraSP dataset. In the $2$nd row, we notice the presence of smoke as the tissue is cauterized using the tool.
Refer to caption: x11.png

Figure: Figure 14: Results from SurgiFlowVid with text conditioning on GraSP dataset.
Refer to caption: x12.png

Figure: Figure 15: Results from SurgiFlowVid with RGB conditioning on GraSP dataset.The frames on the left indicate the sparse conditioning frames and the left frames indicate the generated video frames. The coloured boxes show the position of the corresponding condition frame. The dotted arrow indicates the next subsequent frames. The action corresponds to *pull the suture*.
Refer to caption: x13.png

Figure: Figure 16: Results from SurgiFlowVid with RGB frame conditioning on GraSP dataset. The action corresponds to *cut the tissue*.
Refer to caption: x14.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 4
presentation: 2
questions: I noticed that some of the references include URL links. Could you clarify whether citing websites directly is appropriate, or if this might affect the scientific rigor of the paper?
soundness: 2
strengths: 1. The research question is well positioned and made clear.
2. The experiments are extensive and well analyzed.
summary: The authors address this challenge with SurgiFlowVid, a sparse and controllable video diffusion framework for generating surgical videos
of under-represented classes. Their approach introduces a dual-prediction diffusion module that jointly denoises RGB frames and optical flow, providing temporal inductive biases to improve motion modeling from limited samples.
weaknesses: It is not entirely clear whether this paper includes a comparison with other state-of-the-art methods. Were there no existing studies or similar approaches available for comparison in this line of research? If comparable methods do exist, the authors are encouraged to include a quantitative comparison and perform appropriate statistical significance testing to strengthen the scientific rigor and credibility of the results.

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 2
questions: On the Role of Optical Flow: Optical flow is used only as a training-time signal. Have you explored or considered using optical flow as an additional conditioning signal during inference to potentially further improve generation quality or control?

On "Individual Class Modeling": The results show that modeling each under-represented class separately (IC) sometimes leads to significant gains (e.g., for class A7 in SAR-RARP50). Could you elaborate more on the advantages and ideal scenarios for this strategy? Does it increase the risk of overfitting or training complexity?

On the Realism-Diversity Trade-off: The paper emphasizes both spatial-temporal consistency and diversity. Are there more quantitative metrics (e.g., feature diversity extracted using a pre-trained model) to evaluate the diversity of the generated videos and compare it against the baselines?

On Future Work Directions: Beyond autoregressive generation, for mitigating "tool position shifts," the paper suggests "richer conditional signals such as tool kinematics." Could you propose one or two concrete technical pathways to implement this?
soundness: 3
strengths: Originality: The integration of optical flow as an explicit supervisory signal within a video diffusion model to enhance temporal coherence is a novel and clever idea. The sparse conditioning approach is aptly designed for the constraints of surgical data annotation.

Quality: The experimental work is thorough and compelling. The method is validated across three distinct datasets and three different downstream tasks, supported by comprehensive ablations and baseline comparisons.

Clarity: The methodology, particularly the dual-prediction module and sparse encoder, is described clearly. The experimental section is well-structured, facilitating reproducibility.

Significance: It directly addresses a key bottleneck in deploying surgical AI models. The approach has high practical utility and can be directly applied to improve the performance of existing systems.
summary: This paper addresses the critical challenge of class imbalance in surgical video datasets by proposing SurgiFlowVid, a sparse and controllable video diffusion framework for generating videos of under-represented classes. Its core contributions are:

Dual-Prediction Diffusion U-Net: A novel module that jointly denoises RGB frames and optical flow maps, providing explicit temporal inductive biases to improve motion modeling from limited data.

Sparse Visual Encoder: Enables controllable generation using lightweight conditional signals (e.g., sparse segmentation masks or RGB frames), reducing the reliance on costly, dense annotations that are rarely available in surgical datasets.

Extensive Validation: The method is rigorously evaluated on three surgical datasets (SAR-RARP50, GraSP, AutoLaparo) across three downstream tasks (action recognition, tool presence detection, laparoscope motion prediction). Results demonstrate that augmenting real datasets with synthetic data from SurgiFlowVid yields consistent and significant performance gains of 10-20% over strong, competitive baselines.

This work presents a effective and practical strategy to mitigate data imbalance, thereby advancing the development of robust surgical video understanding models.
weaknesses: Limited Generation Length: As noted in the limitations, the method currently generates short clips (~4 seconds). This is a constraint for tasks requiring long-term temporal modeling (e.g., surgical phase recognition). While autoregressive generation is mentioned as future work, it is not explored here.

Limitations of Sparse Conditioning: When using very sparse segmentation masks, the model can generate tools with incorrect positions (Figure 5). This indicates room for improvement in understanding complex spatial relationships from sparse guidance alone.

Insufficient Computational Cost Analysis: While Appendix B.6 provides a comparison of parameters and inference time, a deeper discussion of the training/inference cost versus performance trade-off, especially in the context of resource-constrained healthcare environments, would strengthen the practical deployment analysis.

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: Long-horizon behavior. What breaks when generating ≥16 frames? Is an autoregressive strategy (as in FlowVid) feasible with your frozen-spatial setup? 

Will you release the synthetic videos per class and the visual encoder checkpoints to enable reproducibility on SAR-RARP50/GraSP/AutoLaparo? 

How do you prevent label leakage and ensure synthetic samples do not overfit to specific sites/instrument brands? Any filtering for tool hallucinations?
soundness: 3
strengths: Well-motivated objective (rare-class augmentation) with domain relevance; aligns with documented imbalance in SAR-RARP50/GraSP.

Temporal inductive bias via flow: predicting motion jointly with appearance is principled and consistent with general-domain findings that explicit motion priors improve coherence.

Sparse masks/RGB keyframes are realistic for surgical datasets and extend SparseCtrl-style conditioning to this domain

Three datasets (SAR-RARP50, GraSP, AutoLaparo) and three downstream tasks show consistent, practically relevant gains; class-wise reporting is appreciated.

Demonstrate that duplication ≠ diversity; shuffling/noise degrade performance; sparse-only is insufficient—supporting the method’s design choices.
summary: The paper proposes SurgiFlowVid, a controllable video diffusion framework to mitigate severe class imbalance in surgical video datasets by synthesizing short clips for under-represented actions/tools. The two main technical ideas are: (i) a dual-prediction diffusion U-Net that jointly denoises RGB frames and optical flow, injecting a temporal motion prior during training; and (ii) a sparse visual encoder that conditions generation on lightweight signals (text, sparse segmentation masks, or RGB keyframes) without requiring dense per-frame labels. The method builds on a two-stage video DM pipeline (spatial SD + temporal transformers) and freezes spatial layers while training temporal layers and the dual-head I/O. Experiments on SAR-RARP50, GraSP, and AutoLaparo show consistent boosts (typically ~10–20%) for rare classes across action recognition, tool presence detection, and laparoscope motion prediction versus Endora, SurV-Gen (± rejection sampling), and SparseCtrl variants; ablations indicate that temporal consistency and spatial structure matter, whereas naive duplication/shuffling hurts. Limitations include short (≈4 s) clips and occasional tool misplacement under sparse conditioning.
weaknesses: The paper encodes optical flow to RGB and uses it only at training time, but does not quantify sensitivity to (a) the flow estimator (e.g., RAFT vs. alternatives), (b) flow noise, or (c) the weighting 𝜆𝑝. Motion quality metrics (e.g., warping error, flow-consistency) are absent.

Results are for ~4-second clips; impact on longer sequences (phase recognition) remains untested. An autoregressive extension is suggested but not validated.

Baselines include Endora, SurV-Gen, SparseCtrl, but SurgSora (RGBD+flow controllable) and other recent controllable methods are only cited; direct side-by-side comparisons would strengthen claims.

Qualitative failure (tool position drift) is shown; quantitative analyses of conditioning faithfulness (mask overlap over time, keyframe similarity) are missing.

All datasets are from related laparoscopic/robotic procedures; cross-procedure transfer (e.g., cholecystectomy) is not studied.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: Optimization and efficiency details are missing. What is the computational overhead introduced by optical flow in inference stage? And others refer to weakness.
soundness: 3
strengths: 1) Addresses an important issue of data imbalance in surgical video datasets.
2) Incorporates optical flow as an additional modality, which reasonably enhances downstream performance.
3) Shows consistent quantitative improvement across tasks, demonstrating the potential of data synthesis for surgical video analysis.
summary: This paper presents SurgiFlowVid, a controllable video diffusion model that generates synthetic surgical videos to address data imbalance. It jointly denoises RGB frames and optical flow for better motion modeling and uses sparse visual conditioning for controllable generation. Experiments on three datasets show 10–20% gains across surgical video understanding tasks.
weaknesses: 1) Two-branch design (RGB + optical flow) is computationally heavy and lacks efficiency analysis.
2) Technical novelty is limited; using multi-branch or multi-task setups to boost performance is a common approach.
3) Insufficient technical details—unclear how the control branch conditions generation or how intermediate frames are produced from sparse inputs.
4) Unclear definition of “under-represented” frames or classes.
5) Table 1 results not convincing, simple data augmentation baselines could achieve comparable gains.
6) Missing ablation studies (e.g., one vs. two branches, with vs. without sparse guidance) to isolate the source of improvement.
7) No computational efficiency comparison, especially considering the added cost of optical flow processing.


Decision: Reject