Video Polyp Segmentation using Implicit Networks

Authors: aviad  dahan
tal  shaharabany
raja  giryes
lior  wolf
Venue: MIDL_io_2024_Conference
Published: 2024-01-27 13:19:20

Abstract: Polyp segmentation in endoscopic videos is an essential task in medical image and video analysis, requiring pixel-level accuracy to accurately identify and localize polyps within the video sequences. Addressing this task unveils the intricate interplay of dynamic changes in the video and the complexities involved in tracking polyps across frames. Our research presents an innovative approach to effectively meet these challenges that integrates, at test time, a pre-trained image (2D) model with a new form of implicit representation. By leveraging the temporal understanding provided by implicit networks and enhancing it with optical flow-based temporal losses, we significantly enhance the precision and consistency of polyp segmentation across sequential frames. Our proposed framework demonstrates excellent performance across various medical benchmarks and datasets, setting a new standard in video polyp segmentation with high spatial and temporal consistency. Our code is publicly available at https://github.com/AviadDahan/VPS-implicit.

Main paper: 
 introduction: 
 The segmentation of polyps in videos plays an important challenge in medical imaging, aiming to precisely identify and localize polyps with pixel-level accuracy within video sequences. This task is crucial for applications such as endoscopic surgery planning. Addressing it involves tackling the complexity of tracking these unique biological structures through various frames, often amidst partial occlusions or changes in viewpoint.The primary difficulty in polyp segmentation lies in the diverse transformations that polyps undergo, such as changes in size, rotation, occlusion, and lighting conditions within the gastrointestinal tract. These variations challenge the consistency of segmentation predictions, particularly in endoscopic videos where the camera's movement may cause significant changes in the background and foreground, potentially leading to inaccurate segmentation.In this paper, we introduce a novel framework for video polyp segmentation, harnessing the capabilities of implicit networks to enhance temporal coherence and segmentation precision throughout video sequences. Building on a 2D pre-trained network, referred to as g, our approach integrates an implicit network, h, augmented with optical flow-based temporal loss functions. This integration is done at test-time, by optimizing network h and without training g or making additional inferences with it.At the heart of our innovation is the effective integration of 2D and implicit networks, which incorporates temporal information into the segmentation process, thereby enhancing the accuracy and consistency of polyp detection over time. By applying optical flow-based losses, our framework ensures more reliable tracking of polyps across frames, addressing the challenges of temporal coherence with greater efficiency.While the framework is general and can be easily adapted to different segmentation tasks and base segmentation networks, we focus on the task of video polyp segmentation using a pretrained image-based AutoSAM (Shaharabany et al., 2023) network or NanoNet (Jha et al., 2021) . The polyp segmentation task provides a well-researched medical imaging domain, with well-established benchmarks, and, in the case of AutoSAM, the opportunity to apply a powerful foundation model derived from the Segment Anything (SAM) network (Kirillov et al., 2023) , for which this domain is out of distribution.We demonstrate the effectiveness of our methodology through extensive testing on various datasets of video polyp segmentation tasks, including capsule endoscopy and colonoscopy. Despite its conceptual simplicity, our approach sets new benchmarks in these areas, proving its potential and efficiency in advancing video polyp segmentation.The use of optical flow to enhance video segmentation has been explored in the past and we discuss three representative examples. Tsai et al. (2016) note that while optical flow can be used to propagate segmentation masks, it can be inaccurate, so they propose to refine the optical flow by using segmentation boundaries. Unlike our work, they use a Conditional Random Field (CRF) for the spatiotemporal segmentation model. Cheng et al. (2017) consider a different mechanism, in which information is shared between a segmentation network and a two-frame optical flow network. Integration occurs through connections between the upsampling layers of the two networks, which are independent up to this point and train with different loss terms. In our method, optical flow is not computed; instead, it is used to add a spatial constraint to the segmentation maps. This constraint is expressed as a loss term, not by using network activations.Another strategy to improve object delineation using optical flow is presented by Fan et al. (2021) who use pre-computed optical flow to create an integrated image that combines multiple frames. They present evidence that performing video object-detection on the resulting image boosts detection accuracy.In some of the experiments, our work employs the AutoSAM architecture introduced by Shaharabany et al. (2023) , a modification of the Segment Anything (SAM) network (Kirillov et al., 2023) , which replaces SAM's prompt encoder with an initial segmentation network.SAM (Kirillov et al., 2023) , similarly to other contributions (Strudel et al., 2021) , employs a visual-transformer (Strudel et al., 2021) . It is trained on a massive supervised segmentation dataset, which contains over 1 billion masks for objects in 11 million natural images. The model quickly became an effective foundation model for other segmentation solutions and downstream tasks (Wu et al., 2023; Liu et al., 2023; Huang et al., 2023) .The task of polyp segmentation involves the precise delineation of polyps from surrounding tissues in images or video frames captured during endoscopic examinations. The work by Jha et al. (2019) , ResUNet++, which builds on the Re-sUNet architecture, showcases superior performance in pixel-wise polyp segmentation across standard datasets. Following this, the authors introduced NanoNet (Jha et al., 2021) , a novel, lightweight architecture designed for real-time segmentation of capsule endoscopy and colonoscopy. In the realm of video polyp segmentation, Ji et al. (2021a) present a pioneering study, introducing the SUN-SEG dataset, as well as a new baseline model, PNS+. Shaharabany et al. (2023) also demonstrate remarkable results in the field of polyp segmentation and video polyp segmentation, despite being image-based.Representing a function by a neural network has been found useful in various applications. A prominent example is neural radiance fields (NeRF) (Mildenhall et al., 2020) , which depict a 3D scene via a neural-based function (Wang et al., 2022; Mildenhall et al., 2022; Gu et al., 2021; Park et al., 2021; Barron et al., 2022; Verbin et al., 2022; Tewari et al., 2022) .In the realm of 2D, neural textures based on rasterization serve a similar purpose, facilitating the generation of lifelike images (Thies et al., 2019; Rivoir et al., 2021) . With an emphasis on representing surface textures through continuous parametric functions without pixel constraints, Burgert et al. (2022) present neural-neural textures, leveraging the potential of Fourier features (Tancik et al., 2020) . Burgert et al. (2023) show the potential of representing a segmentation mask using a combination of Fourier features and implicit models. Building on this foundation and incorporating temporal data in the form of optical flow, our model effectively captures consistent alpha-channels, which play a pivotal role in our step-by-step segmentation scheme. 
method: 
 Our proposed approach for video polyp segmentation builds upon a 2D object extraction network g, e.g., a segmentation network, and incorporates an implicit network h that is trained to predict the output of g for all frames, but with the addition of optical flow losses that ensure temporal consistency between consecutive frames. As the network g one can employ the relevant SOTA algorithm for the associated 2D task. Next, we discuss the training of h for a given pre-trained g.For each frame J ∈ R W ×H×3 the segmentation network g predicts a mask M ∈ R W ×H indicating the pixels that depict the object of interest inside the frame, where W and H denote the width and height of each frame, respectively. Notably, these masks are independent and lack temporal coherence across successive frames, leading to suboptimal accuracy.The implicit network h : x, y, t → α is trained for each input RGB video with N frames V ∈ R N ×W ×H×3 . h aims to refine these frame-wise predictions by introducing temporal information via optical flow. h receives coordinates (x, y, t) as input and returns the probability α of the coordinate belonging to either the foreground or background.The training of h is done during test time, running in an unsupervised manner on the desired test video. During the training of the implicit network h, a composite loss function is employed, comprised of a combination of a labeling loss that compares the output of h to that of g and is based on the Binary Cross-Entropy (BCE) loss, and an optical flow loss. Loss terms. The overall loss function can be expressed as:EQUATIONThe first loss L label aligns the output of h with the prediction of g. Simultaneously, the second loss L f low with the coefficient λ enforces temporal consistency across frames. Neither g nor the optical flow extractor go through any further training at that stage, i.e., the optical flow data and g's output mask are being used as is. The first term in (1), the label loss, is defined asEQUATIONwhere (x, y, t) run over all spatial and temporal locations, and M t = g(V (t, •, •, •)) is the segmentation mask that is obtained by applying g to frame t of the video V .We calculate the optical flow mapping ∆X t , ∆Y t ∈ R W ×H between every frame t and its next frame t + 1 and use it to warp the predicted segmentation mask of one frame to the next frame. We then compute the mean squared error between the warped segmentation mask of a frame and that of the next frame. This can be written as:L f low (h, {(∆X t , ∆Y t )} t ) = (3) 1 N W H (x,y,t) ∥h(x, y, t) -h(x + ∆X t , y + ∆Y t , t + 1)∥ ,where, as in (2), (x, y, t) cover the entire range [1..N ] × [1..W ] × [1..H].Our solution presents a significant advantage in its versatility, as it is not dependent on the specific task that the 2D object extraction network g is addressing. The same procedure described above may be applied to different tasks, where each has its own g function that might be significantly different from the other ones. Fourier Features. To enhance the network's leaning capability, following Burgert et al. (2023) , we have incorporated a Fourier feature transformation to the input. Each coordinate (x,y,t) is normalized in the range [-1,1] and then converted to a vector of N=16 Fourier  
experiments: 
 In this study, we evaluate our proposed method on multiple video datasets. We compare our results with state-of-the-art methods and also present a number of exploratory results in video polyp segmentation.Datasets Our method was tested on three datasets:The first two datasets are SUN-SEG-Easy (unseen) and SUN-SEG-Hard (unseen), both from the SUN-SEG Video-Polyp-Segmentation database (Misawa et al., 2021; Ji et al., 2022) , which contains 1, 106 short video clips, with 158, 690 video frames in total. We used the official train-test split. We compare our results with both image-based methods, namely, UNet (Ronneberger et al., 2015) , UNet++ (Zhou et al., 2019) , ACSNet (Zhang et al., 2020) , PraNet (Fan et al., 2020) , SANet (Wei et al., 2021) and AutoSAM (Shaharabany et al., 2023) as well as video-based methods, including COSNet (Lu et al., 2019) , MAT (Zhou et al., 2020) , PCSA (Gu et al., 2020) , 2/3D (Puyal et al., 2020) , AMD (Liu et al., 2021) , DCF (Zhang et al., 2021) , FSNet (Ji et al., 2021b) and PNSNet (Ji et al., 2021a) . 2021) and contains 55 pillcam endoscopy frames, their segmentation ground truth, and bounding box information. Here, we used a custom 80-20 train-test split, taking the last 20% frames from the dataset as test data, in order to retain the order of the frames. We compare our results to AutoSAM (Shaharabany et al., 2023) and three versions of NanoNet (Jha et al., 2021) , namely A, B and C, where A is the largest architecture and C is the smallest.The architecture of the implicit network h is inspired by the work of Burgert et al. (2023) , expanded to video by including the time dimension. It is described in Fig. 1 . The input to the network consists of spatial coordinates (x, y) and the temporal dimension t. The model h(x, y, t) consists of a positional encoding layer with 16 positional embeddings for each of the three dimensions in the form of Fourier features, followed by an 8-layer MLP with a hidden dimension set to 400. This continuous representation ensures high-resolution predictions alongside smooth transitions, making the model capable of producing detailed saliency masks without the need for upscaling.The optical flow data is extracted using RAFT (Teed and Deng, 2020) .During inference, h is being trained on each test video separately, yielding a set of masks for this specific video for benchmarking. At each step in the training input coordinates (x, y, t) are randomly selected from a uniform distribution in the range [-1,1]. We employ the ADAM optimizer with an initial learning rate of 0.001, and a weight decay regularization parameter set to 1 • 10 -5 . A batch size of 1 is utilized -and a set of 50k points is trained at each step. We conduct training on NVIDIA A6000 with 48GB GPU RAM. The optical flow loss weight, λ of , is set to 1.For the SUN-SEG dataset, we used AutoSAM (Shaharabany et al., 2023) as g, which generates the image-based masks without using temporal information. Once the base masks and the optical flow have been extracted, each video is refined individually using h. The network h trains for 1000 steps. In each step, a subset of the (x, y, t) coordinates is used.For KvasirCapsule-SEG, in order to explore the capabilites of our method, we use NanoNet (Jha et al., 2021) alongside AutoSAM as g and train h for 500 steps. For SUN-SEG, following (Ji et al., 2022) , we use six different metrics for model evaluation between prediction P s and ground-truth G s at timestamp s. These metrics are as follows: (a) Dice coefficient (b) Pixel-wise sensitivity (c) F-measure (d) Weighted F-measure (e) Structure measure (f) Enhanced-alignment measure. For KvasirCapsule-SEG we follow Jha et al. (2021) , using the six metrics: DSC, mIoU, Recall, Precision, F2, and Accuracy.Table 1 shows the results for the SUN-SEG dataset. For the SUN-SEG-Hard (unseen) dataset our method outperforms the state-of-the-art (Shaharabany et al., 2023) in the following metrics: S λ f , F w β , F mn β & Dice with a margin of 2, 2, 2, 3, respectively, with a negative impact of 1% in Sensitivity. For the SUN-SEG-Easy (unseen) dataset our method outperforms the state-of-the-art methods in Dice Score and Sensitivity, with a margin of 2 and 1, respectively, and matches them in S λ f & F mn β , with a negative impact of 1% in E mn ϕ & F w β . To our understanding, the minimal negative impact is a result of the intricate interplay between optical flow and grounding to the existing segmentation mask. We investigate this behaviour in the ablation study for this work. Fig. 2 illustrates the advantage of using the implicit network on SUN-SEG data. Note that applying per-frame object segmentation leads to inconsistent results, which are mitigated by adding the implicit network h. For the KvasirCapsule-SEG, Table 2 shows the results. In both cases, AutoSAM and NanoNet, our method improves the current methods on every metric. Notably, for NanoNet we demonstrate an improvement of 1-2% percent in mIOU and 0.5-1% in DSC and Accuracy.In order to investigate the merits of the optical flow loss, we conducted a number of experiments, with varying levels of λ of while the rest of the parameters are set as mentioned in the training details section, as shown in Figure 3 . Evidently, the introduction of optical flow loss improves performance. We also notice that increasing λ of too much has a negative impact on performance. In such a case the model is less grounded to the segmentation masks provided by g and hence more prone to drifts in optical flow. 
limitations: 
 While the use of h improves the video performance of pre-trained 2D networks, the main limitation of our proposed approach is that the network h is trained at test time. While efficient to train, this is a divergence from the feed-forward nature of other solutions, especially image-based ones. As future work, we may employ meta-learning to make h feed forward, i.e., train, on the training set, a network that changes its behavior based on the input video, using, for example, hypernetworks (Ha et al., 2017) . 
conclusions: 
 We present an innovative approach for video polyp segmentation that lifts 2D segmentation networks to video solutions by employing implicit networks and enforcing temporal consistency through optical flow. Our experimental evaluations have demonstrated the effectiveness of the proposed method. The results consistently showcase improved performance on a variety of metrics compared to 2D segmentation-based approaches. Future work may investigate the expansion of our technique to 3D scans, such as CT or MRI, as well as multi-object segmentation, thereby further solidifying the applicability and impact of our proposed approach across different modalities and tasks in the medical field. 
 
 Appendix:  


Decision: The authors answered the reviewers' questions, sometimes too briefly and without any evidence that this will be incorporated into future articles. It would have been possible to update the manuscript and highlight the changes. However, I trust the authors to do this before final submission. I urge them to do so as soon as possible. I therefore suggest accepting the paper for publication in MIDL.

Reviews: review: paper_summary: paper_summary: In this study, the authors present a method for the segmentation of Polyp on RGB videos. The authors propose the use of a segmentation model, combined with an implicit network to add time coherence between segmented frames. This method has been trained using a new Optical Flow-based loss.
main_review: main_review: I do not seem to find the author's answers on Open review as i do with other revisions. Unless it is an IT issue I am unable to better my grading of the paper nor accept it. Please address comments as soon as possible.
strength_weakness: strength_weakness: As I read the paper, I found It hard to conclude with absolute certainty that the proposed method outperforms an Image-Based AutoSAM as no confidence intervals of the metrics are reported and values remain very close with AutoSAM obtaining higher score averages on the first 4 evaluation metrics (Table 1). The addition of the implicit network seems to improve average scores on the KvasirCapsule-SEG, but with improvements in the order of 10e-3 in some metrics, please make sure to verify whether this difference is significant. 
On a more conceptual level I wonder if such a method is applicable in a real-time segmentation task, as it takes as input the whole video it seems, as polyp video segmentation would prove mostly useful during medical interventions.
questions: questions: Cf weaknesses and detailed comments
paper_summary: paper_summary: This work introduces a postprocessing method for frame-based video segmentation results that improves temporal consistency. The method is based on implicit networks, so a new network h is trained based on the frame-wise results of a segmentation network g to optimize a label loss (based on g's prediction) as well as an optical flow-based loss (based on the label consistency between consecutive frames, warped using vector fields from RAFT). The results are compared against many frame- and video-based methods.
main_review: main_review: I gave a very positive preliminary grade and still believe this to be an interesting contribution, but in light of the missing replies to our reviews (in particular to the more critical ones), I will go one step down to "weak accept".
strength_weakness: strength_weakness: The method requires training h on each new video at inference time, much like using CRFs for consistency postprocessing.

It is not stated how long the training of h takes and how much GPU RAM is necessary (of the 48GB available on the NVIDIA A6000 used).

The results of the method kind of depends on good optical flow results (but the employed RAFT algorithm seems to be a good choice).
questions: questions: Does "taking the last 20% of the dataset as test data" refer to the last 20% of frames in each video?  Or the last 20% of frames, and if so does that split a video?

Which split is used for SUN-SEG?  (I guess an official one, but that should be mentioned in the paper then.)

Why is the value of $\lambda_of=1$ (more or less) optimal?

Why do you use 1000 training iterations for SUN-SEG, but 500 for KvasirCapsule-SEG?
paper_summary: paper_summary: Dahan et al., propose a pipeline for video segmentation that consists of a single 2D image segmentation and an additional unsupervised network to curate the results of the segmentation to preserve the temporal consistency across frames. The underlying idea is that the first network (g) could be pre-trained or a foundation model that computes a preliminary mask, and that the second one (h) can easily curate these segmentations without relying on annotated datasets.
main_review: main_review: The authors suggested a sophisticated and combined approach to perform video segmentation, while given the results and the figures, such a method or equivalent complex approaches are not justified. Just using AutoSAM, without any further training, the task could be performed obtaining equivalent results.
strength_weakness: strength_weakness: - From the description of the second network, it is not clear how much the performance relies on the preliminary 2D segmentation. When looking at the results in Figure 2, it seems that the segmentations of g and h are quite similar and that the proposed improvement could have been achieved by removing the over-segmentation in t = n with simple connected components. Indeed, when looking at the results in Table1, this idea is reinforced as the results from AutoSAM are quite similar if not the same, and are actually the ones with the highest values. 
- The dataset chosen does not show the potential of the method to address time consistency. Therefore, the contribution of the proposed approach is completely diluted. 
- Details about the proposed h and how the training is performed should be clarified (see the detailed comments).
- Poor benchmark: details about the quantitative assessment should be clarified (see the detailed comments).
questions: questions: All the questions have been included in the previous section

Comments: comment: General Response:

We thank the reviewers for their valuable feedback and suggestions. 
We deeply regret any inconvenience caused by the technical issues that prevented some of our responses from being visible. We gratitude to the reviewers for initially providing highly positive scores and feedback. We deeply regret any downgrade caused by the technical issues that affected the visibility of some of our responses.

We assure the reviewers that their initial positive evaluation has not gone unnoticed, and we are committed to addressing their concerns and incorporating their valuable feedback into the revised version of our submission. We apologize for any inconvenience caused and express our sincere appreciation for their understanding and continued support.

Here's a summary of the changes we will make based on the reviewers' comments:

* In response to reviewer woeH and reviewer j1t2, we will provide an explanation on the utilization of Fourier Features.
* In consideration of reviewer j1t2's feedback, we will elaborate on the test-time training process of h.
* As per reviewer PaRD's recommendation, we will include information regarding the SUN-SEG dataset split.
* Responding to reviewer j1t2's input, we will clarify the input of h and the normalization of the xyt coordinates.
* Following reviewer woeH's suggestion, we will relocate the listing of tested models from the results to the experiments section.

Once again, we extend our gratitude to the reviewers for their valuable insights, and we are committed to incorporating their feedback to enhance the quality of our submission.
We thank the reviewer for the valuable feedback. We kindly want to address all of the questions:

>Please remove the listing of tested models from the results and move it to the experiments section.

Thank you for the feedback. The listing will be moved in the revised version.

>Please detail the procedure used to transform the vector (x, y, t) to Fourier features, although simple, from what distribution did you sample them?

Each coordinate (x,y,t) is converted to a vector of N=16 Fourier Features represented by an exponential distribution between base frequency and the $2^N$ harmony. This scale allows us to emphasize the representation of the lower frequencies where most of the data is present. Each frequency is represented by a pair {$\{\sin(n\pi x), \cos(n\pi x)\}$} (in this example the x coordinate is presented). This information will be added to the revision of the manuscript.


We wish to thank the reviewer again for their constructive feedback. With regards to the pointed weaknesses, we wish to highlight the Dice improvement in the SUN-SEG benchmark which is more than 2% over the state-of-the-art method AutoSAM. Also it is worth mentioning the major improvements our method provides to NanoNet which is a much smaller model than AutoSAM. By leveraging h (which is also small), NanoNet was able to drastically narrow the performance gap from AutoSAM.
We thank the reviewer for providing constructive feedback. We are happy to provide additional information regarding the paper.

>Does "taking the last 20% of the dataset as test data" refer to the last 20% of frames in each video? Or the last 20% of frames, and if so does that split a video? Which split is used for SUN-SEG? (I guess an official one, but that should be mentioned in the paper then.)

We take the last 20% of the frames in the video. By doing so we split the video but keep the continuity in each subset of frames. Indeed, the split used in SUN-SEG is the official one. We will add this information in the revised version of the manuscript.


>Why is the value of ${\lambda_{of}=1}$ (more or less) optimal?

To our understanding, the reason the optimum is achieved around ${\lambda_{of}=1}$ might be by chance. But we can confirm that in this setting, ${L_{flow}}$ is about an order of magnitude smaller than ${L_{label}}$ which means that only a small refinement of the original mask is needed. 

>Why do you use 1000 training iterations for SUN-SEG, but 500 for KvasirCapsule-SEG?

The videos provided by SUN-SEG are longer than KvasirCapsule-SEG. In each step of the training process, we sample at random a fixed amount of indices (x,y,t) which is determined by our compute constraint (VRAM).Therefore for longer videos we need more steps to get each pixel to be sampled approximately the same amount of times as in the shorter videos.
We thank the reviewer for the valuable feedback. We wish to note that our method has demonstrated improvements in all metrics on the KvasirCapsule-SEG dataset and showed an improvement of more than 2% in Dice score over the state-of-the-art AutoSAM on both SUN-SEG Easy and Hard datasets.

We would like to highlight the major improvements our method provides to NanoNet which is a much smaller model than AutoSAM. By leveraging h (which is also small), NanoNet was able to drastically narrow the performance gap from AutoSAM.

Lastly, We want to emphasize that our method does not require training the base model thus it is time and cost-efficient while providing a high degree of modularity.

We are happy to provide additional information regarding the paper.


>Input of the h-network: is it the raw values of the image at (x,y,t) or the x,y,t coordinates in the video? i.e. is it (x,y,t) or V(x,y,t)? This information is not clear in the text and in Figure 1 it seems it is the coordinates. If so, how are these coordinates normalised?

The input to the network is (x,y,t). The implicit model provides an output in the form of h(x,y,t). (x,y,t) are normalized in the range of (-1,1). We will add this clarification in the next version of the manuscript.


>How do the Fourier features relate to these coordinates? Are they local? I would suggest the authors to further explain these details in the text.

Each coordinate (x,y,t) is converted to a vector of N=16 Fourier Features represented by an exponential distribution between base frequency and the $2^N$ harmony. This scale allows us to emphasize the representation of the lower frequencies where most of the data is present. Each frequency is represented by a pair {$\{\sin(n\pi x), \cos(n\pi x)\}$} (for example here the x coordinate is presented). This information alongside further clarifications will be added to the revision of the manuscript.
 

>Optical flow: the approach used to compute the optical flow was not trained for this domain, so how much does it contribute to improving the performance? Also, are the authors using the directed optical flow or just the magnitude of Delta X and Delta Y?

Correct, RAFT was not fine-tuned for this specific task. To investigate the contribution of the optical-flow mechanism we have conducted a series of tests, as demonstrated in the ablation subsection (last part of section 4). This study shows the contribution of the Opitcal-Flow mechanism to our method. Although running the method without it $(\lambda_{\text{of}} = 0)$ yields some improvement over the baseline, the major benefit is demonstrated with the optical flow loss being incorporated into the method. 
We are using directed Optical-Flow to estimate the motion vector of each pixel from one frame to the following frame. This means  $\Delta X^t,\Delta Y^t \in \mathbb{R}^{W\times H}$ ,as mentioned in section 3, can have both positive and negative values.


>h-training: it is meant to happen in inference with test images, then is it trained on each video or for the entire test set? Is there any data class balance for the selection of (x,y,t)?
 
During inference, h is being trained on each test video separately, yielding a set of masks for this specific video for benchmarking. At each step in the training, (x,y,t) are randomly selected from a uniform distribution in the range (-1,1). We will add this clarification to the next version of the manuscript.
 
>Benchmark:
>- Were the approaches chosen for comparison retrained for this specific data domain? If not, are any of those supposed to be generalisable such as the foundation model AutoSAM? If so, to provide a fair comparison, it would be recommendable to indicate which ones were retrained and which ones are meant to be generalisable across domains.
>- If possible, I would recommend the authors cite each of the approaches used for comparison.
>- The table only contains mean values and many of them are quite similar, which impedes any conclusion regarding the performance of the method. A solution is to provide plots showing the distribution of these accuracies or the standard deviations of each of these metrics.
>- Were the accuracy metrics for the image-based and video-based approaches computed in the same way? That is, where the accuracy metrics compute for each video and then averaged, or were all the metrics averaged for all the frames?

- All Methods are retrained for the specific task.
- All Methods are cited at the start of the results section.
- The values shown in Table 1. are taken as is from the literature. Since only mean values were published, further statistical analysis is impossible.
- Both image-based and video-based results were computed in the same way. Results were computed for each video frame, then averaged per video, and later averaged across all videos. This follows the evaluation script provided by VPS: https://github.com/GewelsJI/VPS/tree/main/eval (Ji et al. 2021)