<figure><img src="image_1.png" /></figure>

HAL
open science

# Text-driven Adaptation of Foundation Models for Few-shot Surgical Workflow Analysis

Tingxuan Chen, Kun Yuan, Vinkle Srivastav, Nassir Navab, Nicolas Padoy

► To cite this version:

Tingxuan Chen, Kun Yuan, Vinkle Srivastav, Nassir Navab, Nicolas Padoy. Text-driven Adaptation of Foundation Models for Few-shot Surgical Workflow Analysis. International Journal of Computer Assisted Radiology and Surgery, 2025, 20, pp.1175-1183. 10.1007/s11548-025-03341-0. hal-04981199

HAL Id: hal-04981199

https://hal.science/hal-04981199v1

Submitted on 15 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Text-driven adaptation of foundation models for few-shot surgical workflow analysis

Tingxuan Chen<fnref n="3" /> ID · Kun Yuan<fnref n="1" />,<fnref n="2" />,<fnref n="3" /> · Vinkle Srivastav<fnref n="1" />,<fnref n="2" /> · Nassir Navab<fnref n="3" /> · Nicolas Padoy<fnref n="1" />,<fnref n="2" />

Received: 20 January 2025 / Accepted: 28 February 2025 / Published online: 17 April 2025

© The Author(s) 2025

## Abstract

**Purpose** Surgical workflow analysis is crucial for improving surgical efficiency and safety. However, previous studies rely heavily on large-scale annotated datasets, posing challenges in cost, scalability, and reliance on expert annotations. To address this, we propose Surg-FTDA (Few-shot Text-driven Adaptation), designed to handle various surgical workflow analysis tasks with minimal paired image–label data.

**Methods** Our approach has two key components. First, few-shot selection-based modality alignment selects a small subset of images and aligns their embeddings with text embeddings from the downstream task, bridging the modality gap. Second, text-driven adaptation leverages only text data to train a decoder, eliminating the need for paired image–text data. This decoder is then applied to aligned image embeddings, enabling image-related tasks without explicit image–text pairs.

**Results** We evaluate our approach on generative tasks (image captioning) and discriminative tasks (triplet recognition and phase recognition). Results show that Surg-FTDA outperforms baselines and generalizes well across downstream tasks.

**Conclusion** We propose a text-driven adaptation approach that mitigates the modality gap and handles multiple downstream tasks in surgical workflow analysis, with minimal reliance on large annotated datasets. The code and dataset will be released in https://github.com/CAMMApublic/Surg-FTDA.

**Keywords** Foundation model · Surgical data science · Multi-modality learning · Surgical workflow analysis

## Introduction

Surgical workflow analysis is crucial for computer-assisted interventions [1, 2]. It requires precise surgical scene under-

standing and human intent anticipation to provide peripheral assistance feedback to surgeons. Early approaches, due to the lack of sufficient data, rely heavily on feature engineering, incorporating semantic features [3], surgical scene interaction information [4, 5], and other domain-specific knowledge to improve scene understanding. With the rise of deep neural networks and the availability of annotated datasets, recent methods have shifted toward designing architectures that better capture temporal dependencies [6, 7] and other complex features. While these architectures have led to notable improvements, they often exhibit limited generalizability and require large amounts of paired image–label data for training, limiting their scalability.

Surgical foundation models are an emerging field leveraging multimodal representation learning to enable systems to interpret visual concepts using natural language [8–10]. These models have started to demonstrate good generalizability, allowing adaptation to various surgical procedures and tasks, such as surgical workflow recognition, triplet recognition [11], and visual question answering [12, 13].

Tingxuan Chen and Kun Yuan have contributed equally to this work.

<figure><img src="image_3.png" /></figure>

Tingxuan Chen
tingxuan.chen@tum.de

<figure><img src="image_4.png" /></figure>

Kun Yuan
kyuan@unistra.fr

Vinkle Srivastav
srivastav@unistra.fr

Nassir Navab
nassir.navab@tum.de

Nicolas Padoy
npadoy@unistra.fr

<footnote n="1">¹ University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France</footnote>

<footnote n="2">² IHU Strasbourg, Strasbourg, France</footnote>

<footnote n="3">³ CAMP, Technische Universität München, Munich, Germany</footnote>
<figure><img src="image_5.png" /><figcaption>**Fig. 1** a Conventional adaptation of multimodal foundation model requires paired image–label data for training; b Our text-driven adaptation of the foundation model does not require a large number of image–label pairs to achieve the surgical workflow analysis</figcaption></figure>

The zero-shot adaptation of these methods, however, still shows a considerable performance gap w.r.t fully supervised baselines. The main challenge in adapting the surgical multimodal foundational models to the downstream task is the gap between the vision and the text embeddings on the downstream dataset w.r.t the pre-training dataset. This gap arises when vision and text embeddings are clustered separately, leading to situations where semantically dissimilar images are closer to each other than their corresponding texts. One way to adapt these models to various surgical downstream tasks is through prompt tuning [14, 15] or multimodal fine-tuning [16] by utilizing the pre-trained visual and textual encoders as illustrated in Fig. 1a. However, these adaptation approaches rely heavily on the annotated image–label data pair, which is expensive and limited to specific vocabulary. Moreover, since the pre-trained visual and textual models only consist of encoders, it prevents open-vocabulary generative tasks like image captioning [17].

We address these challenges by proposing a novel adaptation strategy, called Surg-FTDA, to model both discriminative and generative tasks as text generation problems. Our goal is to transfer a pre-trained surgical multimodal foundation model to various surgical workflow analysis tasks using minimal paired image–label data. The Surg-FTDA approach operates in two stages. In the first stage, we implement a strategy to select a few data anchors, consisting of image–label pairs from the downstream dataset. This is achieved by performing KMeans clustering on the visual embeddings and selecting data anchors that are maximally distant from each other within the embedding space. These anchors are then used for modality alignment between the visual and textual encoders by projecting both visual and textual embeddings through an MLP layer, followed by minimizing the L2 loss between them.

In the second stage, we perform text-only training inspired by CapDec [18], CLOSE [19], and SurgVLP [8]. Here, we solely rely on the textual data and use the label text from the downstream dataset for the training. A trainable text decoder

is appended to the frozen pre-trained text encoder after the modality alignment from stage one. During training, this operates as an encoder–decoder model, where the frozen text encoder generates textual embeddings from the label text, and the trainable text decoder reconstructs the label texts from these embeddings. During inference, the visual embeddings are generated by the frozen pre-trained visual encoder (after modality alignment from the first stage) and passed through the trained text decoder to generate the corresponding textual labels as the output. Our proposed two-stage training leverages the multimodal knowledge learned during pre-training and the modality alignment, enabling the effective transfer of visual embeddings to the text decoder for accurate label predictions. We evaluate our approach on both generative tasks, such as image captioning, and discriminative tasks, including triplet recognition and phase recognition. The experimental results demonstrate that Surg-FTDA outperforms baseline methods and exhibits a good generalization across diverse downstream tasks in surgical workflow analysis. We summarize our main contributions as follows:

* We propose a text-driven adaptation approach to transfer surgical multimodal foundation models to various surgical workflow analysis tasks with minimal paired image–label data.

* We address the modality gap by performing few-shot data selection and modality alignment, which selects a small number of data anchors to learn a transformation matrix between visual and textual modalities.

* We show that our method generalizes well across both discriminative and generative tasks in surgical workflow analysis, reducing the reliance on large-scale annotated datasets.
# Methodology

In this section, we outline the process of few-shot data anchor selection, text-driven adaptation, and application of our model to both generative and classification tasks.

## Few-shot data anchor selection

Few-shot data anchor selection minimizes the need for large-scale annotated image–label pairs during the adaptation of the surgical foundation model. These selected data anchors play a key role in the subsequent text-driven adaptation process, as they help address the modality gap. It is crucial to choose data anchors that are evenly distributed and representative of the unlabeled dataset to ensure effective modality alignment and to enhance the performance of the text-driven adaptation method.

As shown in Fig. 2a, for a downstream dataset with large-scale unlabeled images, we first project these images into the embedding space using the surgical foundation model’s image encoder, obtaining the image embedding vectors $V_{image}^i \in \mathbb{R}^d$, where $i$ represents each image and $d$ is the dimensionality of the embedding space. To identify the data anchors, we apply either KMeans clustering [20] or farthest point sampling (FPS) [21].

For KMeans, the image embeddings are clustered into $K$ clusters, and the embedding vectors closest to the centroids are selected. Alternatively, FPS selects $K$ image embeddings by maximizing the minimum distance between sampled points, ensuring a diverse selection. These methods preserve the structure of the embedding space by selecting a set of diverse data anchors, referred to as $V_{image}^{sampled}$. Once the anchor image embeddings are selected, we retrieve their corresponding textual label embeddings, denoted as $V_{text}^{sampled}$.

Based on the selected data anchors, we train a multilayer perceptron (MLP) to learn a transformation that aligns the image and text modalities. The MLP, denoted as $f_{MLP}$, takes an image embedding as input and outputs an aligned image embedding. Let $\theta$ represent the parameters of MLP. The MLP is trained by minimizing the MSE loss between the aligned image embeddings $\hat{v}_{image}^i = f_{MLP}(v_{image}^i; \theta)$ and the corresponding text embeddings. The MSE loss is defined as:

$$ \frac{1}{K} \sum_{i=1}^{K} \| \hat{v}_{image}^{i} - v_{text}^{i} \|_{2}^{2} \quad (1) $$

where $K$ is the number of sampled image–label pairs. This process helps reduce the modality gap by aligning the image and text embeddings more effectively.

## Text-driven adaptation

After aligning the vision and language modalities, we propose a text-driven approach to adapt the foundation model for various surgical workflow analysis tasks, treating both discriminative and generative tasks as a text generation problem. We train a text decoder to generate class labels or captions, as depicted in Fig. 2b. Specifically, we extract possible target texts from downstream datasets, such as class labels or captions, and fine-tune the decoder to reconstruct these texts based on the extracted text embeddings.

During the training process, we first extract text embeddings $V_{text}$ using the frozen text encoder from the foundation model. The decoder is then fine-tuned to reconstruct the original texts from these embeddings: $\hat{T}^i = f_{decoder}(v_{text}^i; \theta)$, where $f_{decoder}$ represents the decoder with parameters $\theta$. The learning objective during the fine-tuning is to minimize the auto-regressive cross-entropy loss $\ell$ over all tokens in the target text $T$:

$$ \mathcal{L}_{reconstruction} = \sum_{T \in \mathcal{T}} \ell(\hat{T}, T) \quad (2) $$

where $\hat{T}$ is the predicted text and $T$ is the original text as ground truth from the dataset $\mathcal{T}$.

During inference, we adapt the model to downstream visual tasks by swapping the pre-trained text encoder with the image encoder, allowing the model to accept images as input. We use the frozen image encoder from the foundation model to generate image embeddings $V_{image}$. However, due to the modality gap between vision and text modalities, we need to align these image embeddings with the text embeddings before feeding them into the decoder. To achieve this, we use the previously trained modality alignment function $f_{MLP}$ to transform the image embeddings into aligned embeddings, denoted as $\hat{v}_{image'}$. Then, we pass the aligned image embeddings to the fine-tuned text decoder: $\hat{T} = f_{decoder}(\hat{v}_{image'}; \theta)$, where the decoder outputs the task-specific text output, such as class labels or captions. Since the aligned image embeddings are now closer to the text embeddings in the semantic space, the decoder can generate more accurate and contextually relevant text outputs for the task.

## Multi-task text decoder

In our model, the text decoder behaves differently across different types of tasks. In discriminative tasks, such as triplet recognition and phase recognition, the decoder is trained as a classifier, identifying the decision boundary of the embeddings. In generative tasks, such as image captioning, the decoder functions as a generator, autoregressively producing descriptive text for surgical images.
<figure><img src="image_6.png" /><figcaption>**Fig. 2** a Few-shot data anchor selection based on the visual embedding space using KMeans or FPS; b The text-driven training and inference process of Surg-FTDA, demonstrating how the model applies text-based training to various tasks with minimal paired data</figcaption></figure>

We also adapt the foundation model to handle multiple tasks jointly by training a single text decoder capable of performing two types of discriminative tasks: triplet recognition and phase recognition. For each image, there are associated labels for both phase and triplet tasks. During training, the input to the model consists of the text embeddings derived from phase labels and triplet labels. The decoder is trained to reconstruct the corresponding textual labels based on these embeddings.

To enable the model to generate task-specific outputs based on tasks (either triplet recognition or phase recognition), we train separate MLP's for image embeddings to phase text embeddings and triplet text embeddings. During inference, the task-specific output is controlled by aligning the image embeddings to the corresponding task embeddings using a task-specific MLP. This alignment ensures that the input to the decoder is tailored to the specific task, allowing the model to generate task-relevant outputs effectively.

## Experiments

In this section, we first visualize the effect of our modality alignment function on reducing the modality gap. Second, we evaluate the model's performance on individual downstream tasks. Third, we conduct ablation studies to assess the contributions of various model components. Last, we test the performance of a multi-task decoder trained with mixed input

for different tasks. In these experiments, both the CLIP [22] and SurgVLP [8] foundation models are used to assess the generalizability of our proposed pipeline. We use GPT-2 [23] as the text decoder.

The architecture of the MLP for modality alignment consists of an input layer with dimensions matching the input features, two hidden layers with 128 neurons each and ReLU activation, and an output layer with dimensions matching the target features. The model is optimized using the Adam optimizer with a learning rate of 0.001, trained for 15 epochs with a batch size of 16, and the loss function used is mean squared error (MSELoss).

For text-driven adaptation, the GPT-2 decoder [23] is trained over 10 epochs using the AdamW optimizer with a learning rate of 2e-5. The batch size is set to 34 by default but can be adjusted based on GPU memory and dataset size.

### Modality gap

To better demonstrate the effectiveness of our few-shot anchor selection-based modality alignment, we visualize the original distributions of the two modalities and the distributions after modality alignment using various sampling methods on the SVL-Caption validation set, which contains 1351 image-text pairs. These visualizations provide an intuitive understanding of how our alignment approach reduces the modality gap and improves the coherence between the embeddings.
<figure><img src="image_7.png" /><figcaption>**Fig. 3** Visualization of modality gap across foundation models using few-shot data anchor selection and modality alignment. Yellow points represent the image embedding vectors. Blue points represent the text embedding vectors</figcaption></figure>

<table><caption>**Table 1** Results of triplet recognition</caption><thead><tr><th rowspan="2">Metric</th><th colspan="3">SurgVLP</th><th colspan="3">CLIP</th></tr><tr><th>Precision</th><th>Recall</th><th>F1</th><th>Precision</th><th>Recall</th><th>F1</th></tr></thead><tbody><tr><td>Fully supervised approaches</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Image-Text Training<br>(8331)</td><td>33.18</td><td>30.45</td><td>31.75</td><td>29.20</td><td>31.63</td><td>30.37</td></tr><tr><td>Weakly supervised approaches</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Surg-FTDA (500)</td><td><strong>23.81</strong></td><td><strong>23.94</strong></td><td><strong>23.87</strong></td><td><strong>21.08</strong></td><td><strong>19.64</strong></td><td><strong>20.33</strong></td></tr><tr><td>CapDec (1351)</td><td>20.23</td><td>21.63</td><td>20.90</td><td>10.90</td><td>16.49</td><td>13.12</td></tr></tbody></table>

The numbers in the bracket indicate the number of image-text pairs used by the model
The bolded values indicate the highest performance achieved under weakly supervised approaches for each metric. Specifically, we bold the metric values of the best-performing method among the weakly supervised methods (e.g., Surg-FTDA and CapDec), under both SurgVLP and CLIP models

<table><caption>**Table 2** Results of phase recognition</caption><thead><tr><th rowspan="2">Metric</th><th colspan="4">SurgVLP</th><th colspan="4">CLIP</th></tr><tr><th>ACC</th><th>Precision</th><th>Recall</th><th>F1</th><th>ACC</th><th>Precision</th><th>Recall</th><th>F1</th></tr></thead><tbody><tr><td>Fully supervised approaches</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Image-Text Training<br>(8331)</td><td>57.79</td><td>41.35</td><td>42.27</td><td>38.37</td><td>54.04</td><td>33.61</td><td>31.65</td><td>30.37</td></tr><tr><td>Weakly supervised approaches</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Surg-FTDA (500)</td><td><strong>55.33</strong></td><td><strong>48.10</strong></td><td><strong>37.21</strong></td><td><strong>39.02</strong></td><td><strong>51.20</strong></td><td><strong>27.62</strong></td><td><strong>31.59</strong></td><td><strong>28.93</strong></td></tr><tr><td>CapDec (1351)</td><td>10.46</td><td>18.31</td><td>20.77</td><td>8.92</td><td>31.63</td><td>8.64</td><td>19.92</td><td>11.01</td></tr></tbody></table>

The numbers in the bracket indicate the number of image-text pairs used by the model
The bolded values indicate the highest performance achieved under weakly supervised approaches for each metric. Specifically, we bold the metric values of the best-performing method among the weakly supervised methods (e.g., Surg-FTDA and CapDec), under both SurgVLP and CLIP models

Figure 3 demonstrates a significant reduction in the modality gap after aligning vision and text embeddings. Notably, embeddings from different modalities are more closely aligned when using 500 data anchors compared with using 100 anchors. Additionally, SurgVLP [8] provides better initialization for alignment than CLIP [22], with a smaller initial modality gap. These results confirm that our proposed

modality alignment function based on few-shot data anchor selection effectively bridges the modality gap.

## Downstream tasks

In this work, we investigate two discriminative tasks, phase and action triplet recognition, and generative task, image
<table><caption>**Table 3** Results of image caption</caption><thead><tr><th rowspan="2">Metric</th><th colspan="4">SurgVLP</th><th colspan="4">CLIP</th></tr><tr><th>B@1</th><th>B@4</th><th>METEOR</th><th>CIDEr</th><th>B@1</th><th>B@4</th><th>METEOR</th><th>CIDEr</th></tr></thead><tbody><tr><td colspan="9">Fully supervised approaches</td></tr><tr><td>Image-Text Training (5404)</td><td>25.76</td><td>3.21</td><td>17.41</td><td>15.02</td><td>23.23</td><td>1.92</td><td>15.59</td><td>12.84</td></tr><tr><td colspan="9">Weakly supervised approaches</td></tr><tr><td>Surg-FTDA (500)</td><td><strong>24.60</strong></td><td><strong>2.98</strong></td><td><strong>18.66</strong></td><td><strong>15.65</strong></td><td><strong>25.32</strong></td><td><strong>2.43</strong></td><td><strong>17.06</strong></td><td><strong>11.56</strong></td></tr><tr><td>CapDec (1351)</td><td>23.21</td><td>1.79</td><td>15.21</td><td>11.15</td><td>21.99</td><td>1.11</td><td>14.08</td><td>7.75</td></tr></tbody></table>

The numbers in the bracket indicate the number of image-label pairs used by the model

The bolded values indicate the highest performance achieved under weakly supervised approaches for each metric. Specifically, we bold the metric values of the best-performing method among the weakly supervised methods (e.g., Surg-FTDA and CapDec), under both SurgVLP and CLIP models

captioning. We compare our text-driven adaptation method with the fully supervised models that are fine-tuned on large-scale image-label pairs, and with the weakly supervised CapDec [18] approach, which optimizes noise parameters using image-label pairs.

## Discriminative tasks

**Triplet recognition** We use triplet text samples from the CholecT50 [11] dataset to train the decoder through our text-driven adaptation and test the model on images from the test split. The model's performance is evaluated using precision, recall, and F1 score with macro-averaging. As shown in Table 1, our model consistently outperforms CapDec across various metrics.

**Phase recognition** We report accuracy, precision, recall, and F1 score on the test split of Cholec80 [3] dataset, with macro-averaging applied. As shown in Table 2, our model's performance on these metrics closely approaches that of fully supervised approaches and significantly outperforms CapDec. This demonstrates that our text-driven adaptation trains the text decoder to become a semantic classifier. Additionally, it highlights the capability of the pre-trained foundation model to comprehend both coarse and fine-grained semantics, such as phases and triplets.

## Generative tasks

**Image caption** We train the model using caption text in the SVL-Caption dataset. We evaluate the models using standard captioning metrics: BLEU [24] (B@1, B@4), METEOR [25], and CIDEr [26]. As shown in Table 3, our model outperforms CapDec [18] across various metrics on both foundation models, demonstrating its effectiveness for generative tasks.

## Ablation study

In this section, we evaluate the impact of our few-shot selection techniques and foundation model choice on both

discriminative and generative tasks. We compare KMeans and FPS sampling strategies using 100 and 500 sampled anchors, and no-anchor (without modality alignment) on SurgVLP and CLIP models. We further compare Surg-FTDA with fully supervised models trained using 10%, 30%, 50%, and 100% of the image-text pairs from the dataset.

Tables 4, 5, and 6 demonstrate that using KMeans for sampling generally outperforms FPS across most tasks. This is likely because KMeans selects more representative data anchors, leading to better alignment between the visual and textual modalities. Also, increasing the number of sampling points improves model's performance by learning a robust modality alignment function. SurgVLP consistently outperforms CLIP, demonstrating the benefit of surgical domain-specific pre-training.

As shown in Table 7, for the phase recognition task, our Surg-FTDA significantly outperforms fully supervised models trained with 10% (833 image-text pairs), 30% (2499 pairs), and 50% (4165 pairs) of the dataset. It performs only slightly below the fully supervised model trained on the entire dataset (100%, 8331 pairs). Remarkably, Surg-FTDA achieves this using only 500 image-text pairs.

The fully supervised model trained on 10% of the data fails to generate phase outputs in the correct format under the same experimental settings (e.g., epochs, learning rate, and hyperparameters), resulting in all metrics being zero. This highlights the limitations of fully supervised models in low-data scenarios and underscores the robust performance of Surg-FTDA even with limited annotated data.

For the triplet recognition task, the results in Table 8 show a similar trend. Surg-FTDA significantly outperforms the fully supervised models trained on 10% (833 image-text pairs), 30% (2499 pairs), and 50% (4165 pairs) of the dataset, while performing slightly below the fully supervised model trained on the full dataset (100%, 8331 pairs). Similarly, the fully supervised model trained on 10% of the data still failed to generate valid outputs, resulting in zero metrics, further highlighting the robustness and data efficiency of Surg-FTDA.
<table><caption>**Table 4** Ablation experiments of image caption</caption><thead><tr><th rowspan="2">Metric</th><th colspan="4">SurgVLP</th><th colspan="4">CLIP</th></tr><tr><th>B@1</th><th>B@4</th><th>METEOR</th><th>CIDEr</th><th>B@1</th><th>B@4</th><th>METEOR</th><th>CIDEr</th></tr></thead><tbody><tr><td>No-Anchor</td><td>23.26</td><td>2.01</td><td>16.36</td><td>10.83</td><td>15.22</td><td>1.36</td><td>10.96</td><td>8.03</td></tr><tr><td>KMeans 100</td><td><strong>24.81</strong></td><td>2.84</td><td>18.64</td><td>15.35</td><td>23.62</td><td>1.34</td><td>16.33</td><td>10.00</td></tr><tr><td>KMeans 500</td><td>24.60</td><td>2.98</td><td>18.66</td><td>15.65</td><td><strong>25.32</strong></td><td>2.43</td><td>17.06</td><td>11.56</td></tr><tr><td>FPS 100</td><td>22.77</td><td>2.15</td><td>16.71</td><td>10.52</td><td>24.03</td><td>1.56</td><td>15.90</td><td>7.75</td></tr><tr><td>FPS 500</td><td>24.69</td><td><strong>3.26</strong></td><td><strong>18.85</strong></td><td><strong>15.97</strong></td><td>24.30</td><td><strong>2.64</strong></td><td><strong>17.83</strong></td><td><strong>12.42</strong></td></tr></tbody></table>

The performance of the model improves with an increasing number of selected data anchors

The bolded values represent the best results obtained across different ablation experiment settings, including different sampling strategies (KMeans, FPS) and numbers of selected anchors (100 vs. 500), as well as the no-anchor baseline. These values highlight the most effective combination of sampling strategy and number of anchors

<table><caption>**Table 5** Ablation experiments of triplet recognition</caption><thead><tr><th rowspan="2">Metric</th><th colspan="4">SurgVLP</th><th colspan="4">CLIP</th></tr><tr><th>Acc</th><th>Precision</th><th>Recall</th><th>F1</th><th>Acc</th><th>Precision</th><th>Recall</th><th>F1</th></tr></thead><tbody><tr><td>No-Anchor</td><td>1.34</td><td>9.76</td><td>15.40</td><td>11.95</td><td>0.35</td><td>6.05</td><td>7.21</td><td>6.58</td></tr><tr><td>KMeans 100</td><td>3.01</td><td>13.09</td><td>17.84</td><td>15.10</td><td>19.89</td><td>14.03</td><td>8.53</td><td>10.61</td></tr><tr><td>KMeans 500</td><td><strong>27.97</strong></td><td><strong>23.81</strong></td><td><strong>23.94</strong></td><td><strong>23.87</strong></td><td><strong>26.49</strong></td><td>21.08</td><td><strong>19.64</strong></td><td><strong>20.33</strong></td></tr><tr><td>FPS 100</td><td>10.28</td><td>14.68</td><td>16.97</td><td>15.74</td><td>22.30</td><td>15.04</td><td>8.58</td><td>10.93</td></tr><tr><td>FPS 500</td><td>25.08</td><td>20.46</td><td>21.58</td><td>21.00</td><td>25.94</td><td><strong>23.04</strong></td><td>15.53</td><td>18.55</td></tr></tbody></table>

The performance of the model improves with an increasing number of selected data anchors, and KMeans proves to be more suitable for this task compared to FPS

The bolded values represent the best results obtained across different ablation experiment settings, including different sampling strategies (KMeans, FPS) and numbers of selected anchors (100 vs. 500), as well as the no-anchor baseline. These values highlight the most effective combination of sampling strategy and number of anchors

<table><caption>**Table 6** Ablation experiments of phase recognition</caption><thead><tr><th rowspan="2">Metric</th><th colspan="4">SurgVLP</th><th colspan="4">CLIP</th></tr><tr><th>ACC</th><th>Precision</th><th>Recall</th><th>F1</th><th>ACC</th><th>Precision</th><th>Recall</th><th>F1</th></tr></thead><tbody><tr><td>No-Anchor</td><td>8.12</td><td>12.89</td><td>16.67</td><td>4.79</td><td>3.91</td><td>4.01</td><td>14.28</td><td>1.08</td></tr><tr><td>KMeans 100</td><td>50.63</td><td>29.29</td><td>26.49</td><td>24.77</td><td>44.64</td><td>13.69</td><td>17.42</td><td>14.49</td></tr><tr><td>KMeans 500</td><td><strong>55.33</strong></td><td>48.10</td><td>37.21</td><td>39.02</td><td><strong>51.20</strong></td><td><strong>27.62</strong></td><td>31.59</td><td><strong>28.93</strong></td></tr><tr><td>FPS 100</td><td>41.36</td><td>17.09</td><td>16.88</td><td>14.96</td><td>30.44</td><td>7.33</td><td>21.73</td><td>9.96</td></tr><tr><td>FPS 500</td><td>54.92</td><td><strong>48.42</strong></td><td><strong>38.58</strong></td><td><strong>40.26</strong></td><td>37.00</td><td>18.85</td><td><strong>32.10</strong></td><td>21.76</td></tr></tbody></table>

The performance of the model improves with an increasing number of data anchors

The bolded values represent the best results obtained across different ablation experiment settings, including different sampling strategies (KMeans, FPS) and numbers of selected anchors (100 vs. 500), as well as the no-anchor baseline. These values highlight the most effective combination of sampling strategy and number of anchors

<table><caption>**Table 7** Phase recognition performance of Surg-FTDA and fully supervised models</caption><thead><tr><th>Metrics</th><th>ACC</th><th>Precision</th><th>Recall</th><th>F1</th></tr></thead><tbody><tr><td>10% (833)</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>30% (2499)</td><td>43.37</td><td>29.29</td><td>26.49</td><td>24.77</td></tr><tr><td>50% (4165)</td><td>52.47</td><td>40.66</td><td>39.48</td><td>34.44</td></tr><tr><td>100% (8331)</td><td>57.79</td><td>41.35</td><td>42.27</td><td>38.37</td></tr><tr><td>Surg-FTDA (500)</td><td>55.33</td><td>48.10</td><td>37.21</td><td>39.02</td></tr></tbody></table>

<table><caption>**Table 8** Triplet recognition performance of Surg-FTDA and fully supervised models</caption><thead><tr><th>Metrics</th><th>ACC</th><th>Precision</th><th>Recall</th><th>F1</th></tr></thead><tbody><tr><td>10% (833)</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>30% (2499)</td><td>12.02</td><td>17.30</td><td>22.79</td><td>20.38</td></tr><tr><td>50% (4165)</td><td>17.83</td><td>15.99</td><td>20.75</td><td>18.07</td></tr><tr><td>100% (8331)</td><td>30.26</td><td>33.18</td><td>30.45</td><td>31.75</td></tr><tr><td>Surg-FTDA (500)</td><td>27.97</td><td>23.81</td><td>23.94</td><td>23.87</td></tr></tbody></table>
<table><caption>**Table 10** Results of phase recognition using multi-task trained decoder</caption><thead><tr><th>Metric</th><th>ACC</th><th>Precision</th><th>Recall</th><th>F1</th></tr></thead><tbody><tr><td>Surg-FTDA: Task-Specific Trained</td><td>55.33</td><td>48.10</td><td>37.21</td><td>39.02</td></tr><tr><td>Surg-FTDA: Multi-Task Trained</td><td><strong>56.04</strong></td><td><strong>48.13</strong></td><td><strong>37.79</strong></td><td><strong>40.41</strong></td></tr></tbody></table>

The multi-task trained decoder performs better than the task-specific trained decoder
The bolded values refer to the higher metric between the task-specific trained decoder and the multi-task trained decoder for each metric. This highlights which training strategy performs better for each task

<table><caption>**Table 9** Results of triplet recognition using multi-task trained decoder</caption><thead><tr><th>Metric</th><th>Precision</th><th>Recall</th><th>F1</th></tr></thead><tbody><tr><td>Surg-FTDA: Task-Specific Trained</td><td>23.81</td><td>23.94</td><td>23.87</td></tr><tr><td>Surg-FTDA: Multi-Task Trained</td><td><strong>24.55</strong></td><td><strong>24.23</strong></td><td><strong>24.39</strong></td></tr></tbody></table>

The multi-task trained decoder performs better than the task-specific trained decoder
The bolded values refer to the higher metric between the task-specific trained decoder and the multi-task trained decoder for each metric. This highlights which training strategy performs better for each task

## Multi-task text decoder for better decision boundary

In this section, we evaluate the effectiveness of training a single decoder with mixed input for multiple tasks, i.e., phase recognition and triplet recognition. As shown in Tables 9 and 10, the multi-task decoder consistently

outperforms task-specific decoders trained on single-task texts, demonstrating that the multi-task learning of text-driven adaptation enhances performance on individual tasks. This suggests that the text decoder learns more robust decision boundaries, enabling it to better distinguish complex semantic features.

## Conclusion

In this work, we introduced Surg-FTDA (Few-shot Text-driven Adaptation), a novel approach designed to address the challenges of surgical workflow analysis with minimal reliance on large-scale annotated datasets. By leveraging few-shot data selection and text-driven adaptation, Surg-FTDA bridges the modality gap between vision and text, allowing the model to handle various downstream tasks such as phase recognition, triplet recognition, and image captioning. Our method demonstrates that by selecting a small, diverse subset of image-label pairs and aligning visual and textual embeddings, a text-trained decoder can generalize effectively to visual tasks without requiring large amounts of paired image-label data. The results show that Surg-FTDA outperforms existing baselines, contributing to the field of surgical workflow analysis by enabling more scalable, data-efficient, and versatile models. This few-shot, text-driven adaptation opens up new possibilities for applying foundation models to a broader range of tasks, where limited annotations and multi-modality learning are key challenges.

**Acknowledgements** The authors would like to acknowledge the High-Performance Computing Center of the University of Strasbourg for supporting this work by providing scientific support and access to computing resources. Part of the computing resources were funded by the Equipex Equip@Meso project (Programme Investissements d'Avenir) and the CPER Alsac calcul/Big Data.

**Funding** Open Access funding enabled and organized by Projekt DEAL. This work has received funding from the European Union (ERC, CompSURG, 101088553). Views and opinions expressed are, however, those of the authors only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. This work was also partially supported by French state funds managed by the ANR under Grants ANR-20-CHIA-0029-01 and ANR-10-IAHU-02. This work was granted access to the HPC resources of IDRIS under the allocations AD011013704R1, AD011011631R2, and AD011011631R4 made by GENCI.

## Declarations

**Conflict of interest** The authors have no relevant financial or non-financial interests to disclose.

**Ethical Approval** This study does not involve human participants or animal studies requiring ethical approval.

**Informed Consent** This study uses publicly available datasets that do not contain personally identifiable information.

**Consent for Publication** The authors consent to the publication of this article in IJCARs.

**Open Access** This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

## References

1. Maier-Hein L, Eisenmann M, Sarikaya D, März K, Collins T, Malpani A, Fallert J, Feussner H, Giannarou S, Mascagni P et al (2022) Surgical data science-from concepts toward clinical translation. Media 76:102306
2. Padoy N (2019) Machine and deep learning for workflow recognition during surgery. Minim Invasive Ther Allied Technol 28(2):82–90

3. Twinanda AP, Shehata S, Mutter D, Marescaux J, De Mathelin M, Padoy N (2016) Endonet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans Med Imaging 36(1):86–97

4. Yuan K, Holden M, Gao S, Lee W (2022) Anticipation for surgical workflow through instrument interaction and recognized signals. Med Image Anal 82:102611

5. Murali A, Alapatt D, Mascagni P, Vardazaryan A, Garcia A, Okamoto N, Mutter D, Padoy N (2023) Latent graph representations for critical view of safety assessment. IEEE Trans Med Imaging

6. Czempiel T, Paschali M, Keicher M, Simson W, Feussner H, Kim ST, Navab N (2020) Tecno: Surgical phase recognition with multi-stage temporal convolutional networks. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part III 23, pp. 343–352. Springer

7. Gao X, Jin Y, Long Y, Dou Q, Heng P-A (2021) Trans-svnet: Accurate phase recognition from surgical videos via hybrid embedding aggregation transformer. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part IV 24, pp. 593–603. Springer

8. Yuan K, Srivastav V, Yu T, Lavanchy J, Mascagni P, Navab N, Padoy N (2023) Learning multi-modal representations by watching hundreds of surgical video lectures. arXiv preprint arXiv:2307.15220

9. Yuan K, Srivastav V, Navab N, Padoy N (2024) Hecvl: Hierarchical video-language pretraining for zero-shot surgical phase recognition. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 306–316. Springer

10. Yuan K, Srivastav V, Navab N, Padoy N (2024) Procedure-aware surgical video-language pretraining with hierarchical knowledge augmentation. arXiv preprint arXiv:2410.00263

11. Nwoye CI, Yu T, Gonzalez C, Seeliger B, Mascagni P, Mutter D, Marescaux J, Padoy N (2022) Rendezvous: attention mechanisms for the recognition of surgical action triplets in endoscopic videos. Media 78:102433

12. Seenivasan L, Islam M, Krishna AK, Ren H (2022) Surgical-vqa: visual question answering in surgical scenes using transformer. In: MICCAI, pp. 33–43

13. Yuan K, Kattel M, Lavanchy JL, Navab N, Srivastav V, Padoy N (2024) Advancing surgical vqa with scene graph knowledge. Int J Comput Assist Radiol Surg 19(7):1409–1417

14. Zhou K, Yang J, Loy CC, Liu Z (2022) Conditional prompt learning for vision-language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 16816–16825

15. Yao H, Zhang R, Xu C (2023) Visual-language prompt tuning with knowledge-guided context optimization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 6757–6767

16. Wang M, Xing J, Liu Y (2021) Actionclip: a new paradigm for video action recognition. arXiv preprint arXiv:2109.08472

17. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M et al (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115:211–252

18. Nukrai D, Mokady R, Globerson A (2022) Text-only training for image captioning using noise-injected CLIP. In: Goldberg Y, Kozareva Z, Zhang Y. (eds.) Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 4055–4063. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates. https://doi.org/10.18653/v1/2022.findings-emnlp.299

19. Gu S, Clark C, Kembhavi A (2023) I can't believe there's no images! learning visual tasks using only language supervision. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 2672–2683

20. Hartigan JA, Wong MA (1979) A k-means clustering algorithm. Appl Stat 28(1):100–108

21. Qi CR, Yi L, Su H, Guibas LJ (2017) Pointnet++: deep hierarchical feature learning on point sets in a metric space. arXiv preprint arXiv:1706.02413

22. Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, Sastry G, Askell A, Mishkin P, Clark J, et al. (2021) Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning, pp 8748–8763. PMLR

23. Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I et al (2019) Language models are unsupervised multitask learners. OpenAI blog 1(8):9

24. Papineni K, Roukos S, Ward T, Zhu W-J (2002) Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp 311–318

25. Banerjee S, Lavie A (2005) Meteor: an automatic metric for mt evaluation with improved correlation with human judgments. In: Proceedings of the Acl Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation And/or Summarization, pp 65–72

26. Vedantam R, Lawrence Zitnick C, Parikh D (2015) Cider: consensus-based image description evaluation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 4566–4575

**Publisher's Note** Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.