HFSTI-Net: Hierarchical Frequency-spatial-temporal Interactions for Video Polyp Segmentation
Authors: Yuanqin He
Guilian Chen
Yuhua Zhang
Huisi Wu
Jing Qin
Venue: ICLR 2026 Poster (2026)
Keywords: Frequency Learning
Video Segmentation
Medical Segmentation
Video Polyp Segmentation
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Automatic video polyp segmentation (VPS) is crucial for preventing and treating colorectal cancer by ensuring accurate identification of polyps in colonoscopy examinations. However, its clinical application is hampered by two key challenges: shape collapse, which compromises structural integrity, and episodic amnesia, which causes instability in challenging video sequences. To address these challenges, we present a novel video segmentation network, \emph{HFSTI-Net}, which integrates global perception with spatiotemporal consistency in spatial, temporal, and frequency domains. Specifically, to address shape collapse under low contrast or visual ambiguity, we design a Hierarchical Frequency-spatial Interaction (HFSI) module that fuses spatial and frequency cues for fine-grained boundary localization. Furthermore,  we propose a recurrent mask-guided propagation (RMP) module that introduces a dual enhancement mechanism based on feature memory and mask alignment, effectively incorporating spatiotemporal information to alleviate inter-frame inconsistencies and ensuring long-term segmentation stability. Extensive experiments on the SUN-SEG and CVC-612 datasets demonstrate that our method achieves real-time inference and outperforms other state-of-the-art approaches. Codes are available at \url{https://github.com/Yuanqin-He/HFSTI-Net}.

Published as a conference paper at ICLR 2026
HFSTI-NET: HIERARCHICAL FREQUENCY-SPATIAL- TEMPORAL INTERACTIONS FOR VIDEO POLYP SEG- MENTATION
Yuanqin He', Guilian Chen', Yuhua Zhang', Huisi Wu' Jing Qin²
'College of Computer Science and Software Engineering, Shenzhen University Centtee for Smart Health, School of Nursing, The Hong Kong Polytechnic University 240010 640mai 1s. szu. w...uu.u....u............u......
edu. hswuêszu edu.cn
ABSTRACT
Automatic video polyp segmentation (VPS) is crucial for preventing and treat- ing colorectal cancer by ensuring accurate identification of polyps colonoscopy examinations. However, its clinical application is hampered by two key chal- lenges: shape collapse, which compromises structural integrity, and episodic am- nesia, which causes instability in challenging video sequences. To address these challenges, we present novel video segmentation network, HFSTI-Net, which integrates global perception with spatiotemporal consistency in spatial, temporal, and frequency domains. Specifically, to address shape collapse under low con- trast or visual ambiguity. design Hierarchical Frequency- (HFSI) module that fuses spatial and frequency cues for fine-grained boundary lo- calization. Furthermore, we propose recurrent mask-guided propagation (RMP) module that introduces a dual enhancement mechanism based on feature memory and mask alignment, effectively incorporating spatiotemporal information to al- leviate inter-frame inconsistencies and ensuring long-term segmentation stability. Extensive experiments on the SUN-SEG and CVC-612 datasets demonstrate that our method achieves real-time inference and outperforms other state-of tarte- approaches. Codes are available https:/ / #FSTI-Net.
Interaction
com/ Yuanqi n- He/
INTRODUCTION
Colorectal cancer (CRC), prevalent gastrointestinal malignancy and the third most common cancer globally, can be effectively prevented Shaukat & Levin (2022) through timely screening and removal of precursor polyps via colonoscopy Wu et al. (2024). However, the diagnostic process heavily relies on endoscopists' expertise, and hence inexperienced practitioners risk missing the detection of precancerous lesions. Therefore, accurate and real-time automated polyp segmentation is crucial for enhancing early CRC diagnosis and supporting timely clinical decision- making.
In recent years, numerous deep learning based methods have been proposed for image polyp seg- mentation (IPS) and achieved remarkable successes Wei et al. (2021); Dong et al. (2021); Zhou al. (2023). However, these approaches still face two inherent limitations in real-world clinical settings. First, as illustrated in Figure 1 (b), the low contrast between polyps and the surrounding mucosa makes it challenging accurately distinguish the target from the background only based on static image information alone Fan et al. (2020b); Wu et al. (2023). This often leads to a phenomenon that usually call shape collapse in the segmentation results. Second, these static image- based methods overlook a critical fact that real-world clinical screening is conducted based on a continu- video stream Puyal et al. (2020); Ji al. (2021); Li et al. (2022). In the video, the appearance of polyps can undergo drastic changes due to variations in viewpoint, intestinal peristalsis, and cam- era motion (as shown in Figure 1 (c)), where appearance not only refers visual texture but also encompasses substantial variations in polyp size, position, and shape. While these temporal varia- tions pose a new challenge for accurate segmentation, they are also probably helpful to address the
"Corresponding Author

---

Published as a conference paper at ICLR 2026
Frames
GT
PNS+
SALI
VPSAM
Ours
(a) Shape collapse
(b) Low contrast
Time
(c) Significant variations
(d) Consecutive low-quality frames
Figure 1: Challenges of VPS including (a) shape collapse, (b) low contrast between polyps and background, (c) significant variations between adjacent frames, and (d) consecutive low-quality frames.
first limitation, different frames can provide complementary views disambiguate low-contrast regions. Existing IPS methods, however, are inherently not able to sufficiently leverage temporal in- formation, and hence cannot achieve satisfactory segmentation results in clinical practice. To more closely align with real-world clinical scenarios, the task of video polyp segmentation (VPS) has al. (2022); Hu al. (2024). VPS methods aim to enhance segmentation robustness and accuracy by leveraging temporal coherence in videos. Current mainstream VPS approaches, either employing hybrid 2D/3D architectures Puyal et al. (2020); Bhattacharya al. (2024); Xu et al. (2024) or harnessing self-attention mechanisms Vaswani (2017); Ji et al. (2021); Chen et al. (2024), predominantly rely on implicit modeling of dense, pixel-level features to propagate tempo- ral information. However, these methods highly sensitive to appearance variations pixel-level features largely lack high-level semantic abstraction. Consequently, these methods cannot address challenges caused by large temporal gaps among frames, such as drastic deformations and sequences of low-contrast and blurry frames (as shown in Figure (d)), which lead to the episodic amnesia. To address this shortcoming, some approaches propose employ global-to-loee learning to capture more temporal information, but these methods are usually unstable as the temporal information they extracted is quite limited.
emerged
Although VPS improves robustness with temporal coherence, it still fails under low contrast or oc- clusion, leading structural loss and missed polyps. Recently, integrating frequency information into deep learning models has shown considerable promise for many computer vision tasks, such as camouflaged object detection (COD) Fan al. (2020a). Existing spatial-domain methods, such as those focusing on enhancing pixel-level discriminability Ji et al. (2021); Cheng et al. (2022); Wang et al. (2022), often struggle with capturing comprehensive contextual information due their inherent limitations in local feature representation. Accordingly, for dealing with 'camouflaged' polyps colon structures, leveraging global frequency perception and harnessing multi-domain representations may enable the model to perceive spatial details within features. While conventional spatial-domain methods Chi et al. (2020); Wang et al. (2023b) struggle to effectively identify tar- gets from the background due their high visual similarity, recent works have begun exploring frequency-domain representations to enhance identification capacity. To address this issue, recent works Li al. (2024); Wang al. (2023a) explore the frequency-domain representations en- hance the identification capacity. However, these methods ignore the modeling of frequency-spatiar interactions, limiting their ability to adaptively fuse cross-domain features and ultimately hindering spatial-de -ontial-dominn feature learning. Besides, these methods fail to model the dependencies between different frequency components and their spatial counterparts, leading incomprehensive feature representations for precise target localization.
In this paper, we propose novel network, HFSTI-Net, for VPS, addressing challenges by jointly modeling frequency, spatial, and temporal features. The network consists of two key components: a hicrarchical frequency-spatial interaction (HFSI) module and a recurrent mask-guided propaga- tion (RMP) module. The HFSI module uses dual-path design to combine local spatial cues with global frequency representations, effectively capturing fine-grained boundary details and prevent- ing shape collapse, even in low-contrast ambiguous conditions, The RMP module introduces a memory-based dual enhancement mechanism, storing historical embeddings and predictions to model temporal dependencies and mitigate inconsistencies, improving spatiotemporal consistency
2

---

Published as a conference paper at ICLR 2026
and reducing episodic amnesia. Extensive experiments SUN-SEG and CVC-612 demonstrate that HFSTI-Net outperforms SOTA methods. Our contributions are summarized as follows:
We propose a novel video polyp segmentation network, HFSTI-Net, that jointly models spatial, frequency, and temporal information to address challenges such background interference, low contrast, and rapid endoscope movements.
We design a HFSI module to enhance boundary localization and structural integrity by integrating local spatial cues with global frequency representations through dual-path Fourier-based interaction. Furthermore, we propose a RMP module that leverages feature memory and mask alignment to capture long-term spatiotemporal dependencies, effectively alleviating episodic amnesia and reducing tracking errors.
We conduct extensive experiments on the SUN-SEG and CVC-612 dataset, which demon- strates that our method achieves superior performance compared to other SOTA methods but also maintains real-time efficiency, making it more suitable for clinical deployment.
2 RELATED WORKS
2.1 POLYP SEGMENTATION
With the development of deep learning, remarkable progress has been made in IPS. Although IPS primarily leverages CNN architectures Cheng et al. (2021); Wu et al. (2022) for local feature ex- traction, their limited global perspective leads to blurred boundaries and low-contrast issues. To address this, Transformers Vaswani (2017) or hybrid CNN-Transformer architectures Zhang et al. (2021); Li et al. (2021) were introduced to enhance global context awareness, alongside boundary constraint methods Fan et al. (2020a); Cheng al. (2021) for edge refinement. However, VPS requires modeling temporal dynamics, an inherent limitation of static IPS methods. Early meth- ods used hybrid 2D/3D convolutions to fuse spatial and short-term temporal features Puyal et al. (2020), but their limited receptive field struggles with long-range dependencies. To address this, attention-based methods emerged Ji et (2021), with models like PNS+ Ji et al. (2022) captur- ing global temporal relationships across entire sequences. While effective, global attention is often computationally expensive and sensitive to noise. improve efficiency, later works introduced key frame- frame-guided guided strategies Xu et al. (2022); Hu et al. (2024), using high-quality frames as anchors. This highlights that most temporal models implicitly learn inter-frame relationships via end-to-end training. To address this limitation and mitigate episodic amnesia in challenging video sequences, we propose the RMP module. Leveraging high-level features and predicted masks from a memory bank, RMP captures rich spatiotemporal cues, allowing the model to recall dynamic changes and historical context across frames. This enhances polyp recognition and localization while ensuring long-term segmentation stability.
2.2 FREQUENCY LEARNING
The frequuecyy domain plays important role in signal analysis and has been increasingly applied to computer vision tasks. Several methods He et al. (2023); Wang et al. (2023a) leverage frequency information for enhanced feature representation. FeaNet Qin et al. (2021) views channel attention as frequency-domat compression problem and introduces multi-spectral attention to preserve in- formative frequency features with added complexity. Wang et al. (2023b) investigate frequency learning in segmentation, showing that networks tend to focus on class-specific frequency compo- nents, which may lead frequency shortcuts and hinder generalization. To address the camouflage challenge, FAGF-Net Li et al. (2024) incorporates frequency- aware attention and graph-based fu- methods often treat
sion, outperforming spatial-only methods. However, existing frequency-bai frequency and spatial information somewhat independently or lack a deep integration. This can lead to an incomplete capture of intricate relationships between different frequency components and their precise spatial counterparts, potentially limiting its effectiveness in the task of VPS. Therefore, we propose a novel HFSI module designed to counteract shape collapse so enhance the structural integrity of segmented polyps. It allows for exploring fine-grinedd structural details by learning and enhancing frequency- interactions from the frequency domain to distinguish foreground polyp targets from surrounding tissues.
3

---

Published as a conference paper at ICLR 2026
HIST
FFR
IFB
Feature Extraction Emcodler
Frequency
Fealure
SRIR
Pr
Spatital
1--1
Fealure
P
Hicrarcbical Frequency-spate Interaction
Fneadar
Mark
functional conectivity
Retrieve
skip connections
TAM
PEmbeddings
store
Reshape
Element- wiise Addition
Recurrent Musk-guided Prepagation
Multiplication
Figure 2: Overview the proposed network for VPS. Our model addresses shape collapse and episodic amnesia to ensure structural integrity and long-term segmentation stability. The HFSI mod- ule processes features both spatial and frequency domains for fine rre-graaned boundary localization, counteracting shape collapse. The RMP module utilizes memory bank and cross-attention for con- sistent tracking, mitigating episodic amnesia. MFE blocks, with parallel convolutions (1x1, 3x3, 5x5), extract rich spatial features for complementary enhancement.
METHOD
3.1 OVERVIEW
Figure 2 presents the overall framework of our method, which addresses two key challenges in VPS: maintaining structural integrity in segmented polyps and ensuring long-term segmentation stability across challenging video sequences. To address shape collapse and preserve structural integrity, we introduce Hierarchical Frequency-spatial Interaction (HFSI) module. By jointly processing features in the spatial and frequency domains and fusing them through an interwoven dual-path design, it combines local detail cues with global context, enabling lin-grained boundary localization and robust shape preservation, segmentation stability and mitigate episodic amnesia in challenging video sequences, we propose a Recurrent Mask-guided Propagation (RMP) module. It stores previous frame features and masks in a memory bank and retrieves relevant information via cross-attention. A mask affinity mechanism further aligns the current prediction with historical context, ensuring consistent and stable tracking of polyps across frames, even through occlusions or significant appearance changes.
under visual ambiguity and low contrast. To enforee long- long-term
As shown in Figure 2, given an input video sequence {4) with I R. #x we first extract multi-level features F = {F using backbone encoder, where each Fý has spatial resolution 2 T Then, the top-level feature F is enhanced by MFE module, To model temporal coher- ence, the current feature FJ and historical context F(-1_ Pt-1 passed into the RMP module. The refined temporal feature is further processed by the HFSI module, which integrates local spatial and global frequency cues through interwoven fusion, yielding enriched representations X 14 Finally, decoder aggregates these features and progressively refines them to generate the prediction P [P:)4
H W
3.2 HIERARCHICAL FREQUENCY-SPATIO INTERACTION
Video polyp segmentation (VPS) is challenging due the high visual similarity between polyps and surrounding tissues in color, texture, and motion blur. While spatial-domain methods focus on local details, they often miss global context. In contrast, frequency-based approaches capture global semantics but are susceptible to noise and may underutilize informative frequency cues in complex scenes. To address these issues, propose the HFSI module with dual-ppthh structure
4

---

Published as a conference paper at ICLR 2026
(a) Frequency Filter Block (FFB)
(b) Spatial Refinement Block (SRB)
(c) Interwoven Fusion Block (IFB)
Figure 3: The key details module of the hierarchical frequency-spatial interaction (HFSI) module. The module mainly consists of frequency filter block (FFB), spatial refinement block (SRB) and interwoven fusion block (IFB).
that extracts both spatial and frequency representations. Central HFSI is interwoven fusion mechanism, which facilitates fine-grained interaction between the two domains, enabling adaptive enhancement of meaningful features. As shown in Figure 3, the HFSI module consists three components: the Frequency Filter Block (FFB), which extracts global contextual patterns through trequency-domanl self-attention to enhance boundary localization and suppress background noise; the Spatial Refinement Block (SRB), which emphasizes edge-aware details to preserve local struc- tural precision; and the Interwoven Fusion Block (IFB), which adaptively blends frequency- and spatial-domain features via learnable attention mechanism for semantic alignment. By integrating spatial and frequency cues across multiple levels, HFSI effectively improves both local detail and global context, enabling more accurate segmentation of camouflaged polyps in complex scenes.
3.2. FREQUENCY FILTER BLOCK
To extract global contextual patterns in the frequency domain, the FFB applies frequencyy-dmain self-attention mechanism that models channel-wise dependencies across spectral components. By emphasizing salient frequency responses and suppressing redundancy, it sharpens polyp boundaries and reduces background interference, especially in low-contrast frames. Given an input feature X e RCxH*W composed of current-layer and high-level features, first apply layer normalization obtain X LN(X). The normalized feature is then transformed into the frequency domain via Fast Fourier Transform (FFT) to compute the query, key, and value: Q5. Ky. V, (&), respectively. We compute the frequency attention map via matrix multiplication and apply it to reweight Vy, followed by inverse FFT to restore the spatial representation. In parallel, we introduce frequency residual branch that enriches spectral responses through lightweight attention filter (.)) containing convolution, normalization, and activation layers. Finally, the outputs from both branches concatenated yield the enhanced rreuuency-wwan feature:
Xi Cat (F- (Ay Vy), F F (a((()))
where A, Q3 K denotes the attention map and @ represents matrix multiplication.
This design ensures that rich global frequency cues are preserved and fused with spatial cues in later modules, improving the network's robustness to boundary ambiguity and structural complexity in polyp regions.
3.2.2 SPATIAL REFINEMENT BLOCK
To accurately segment polyps with varying sizes and complex shapes, we design the Spatial Refine- ment Block (SRB) to capture fine-gr -raiigr structural details and local context. Unlike frequency- based modeling, SRB operates entirely in the spatial domain and emphasizes edge-sensitive features through spatial self-attention. As illustrated in Figure 3 (b), the input feature X' is first passed through 1 x convolution to encode positional information. To effectively capture multi-scale spatial dependencies. we employ two depthwise separable convolutions with kernel sizes 3 and 5 x 5, respectively. These are used to compute the query, key, and value representations:
Q. Ks, V, Cat(DCg k. (x)), DCg.k." (X)),
The spatial attention map is computed A, Softmax Q2 K2), which highlights salient spatial structures. We then use this attention to reweight V., yielding a refined attention output. In parallel, residual spatial branch X (DC,()() DCs (X)) introduced to preserve original spatial
5

---

Published as a conference paper at ICLR 2026
Temporal Alignment Module
Mask Affinity Modale
Q+
QN
MLP
Ky
K
MLP
m7p
Figure 4: Structure of the recurrent mask-guided propagation module (RMP). The module consists of temporal alignment module (TAM) and mask affinity module (MAM).
details. The final spatial- feature formed by concatenating both branches:
X = Cat @ Va, X7).
(3)
3.2.3 INTERWOVEN FUSION BLOCK
Frequency and spatial features encode complementary information: frequency features emphasize global semantic distributions, while spatial features highlight fine-grained structural details. To fully leverage the strengths of both representations, we design the Interwoven Fusion Block (IFB) learnable bridge that enables dynamic feature entanglement across the two domains.
Given frequency-domin -omainn features x,, spatial-domait features X,, and an earlier-layer feature we first compute a composite representation via residual fusion: X = XX +. We then apply layer normalization obtain the normalized feature X. LN((X), which serves input stage inter-domain interaction process. We separately enhance frequency and spatial representations by projecting X. into each domain and applying gated multiplicative attention:
X? = GeI.U(((F(X.)) @ (F(()),
(4)
X (eL)( DC2(8.),
(5)
where F(.) and (-3(-) denote the fast Fourier transform and depthwise convolution, respectively, and o-)) is lightweight channel-wise attention filter. We begin by combining the intermediate frequency and spatial features via element-wis multiplication:X/: the input both the frequency and spatial refinement branches. The frequency branch performs gated attention in the Fourier domain, while the spatial branch focuses on localized filtering. The outputs of both branches are then aggregated:
(X3 & X?), which serves
= (()
(6)
DC,(( x - Ca((3, X3) X
Through interwoven, cross-domain interaction with gated fusion, IFB aligns global and local fea- tures, reducing misalignment and background ambiguity for more precise segmentation.
3.3 RECURRENT MASK-GUIDED PROPAGATION MODULE
Although HFSI improves structural integrity, VPS still suffers from temporal inconsistency caused by blur, motion, occlusion, and appearance changes. To address this, we propose the RMP mod- ule, which explicitly models spatiotemporal dependencies to ensure long-term consistency. RMP maintains a memory bank of high-level features and masks from past frames. For each incoming frame, current features act query, while memory features and masks serve key-value pairs. Temporal alignment module integrates temporal cues, and the result is refined with MLP and nor- malization. To further enhance alignment, mask affinity module fuses the predicted mask with spatial features and performs another cross-attentio step with memory, enabling motion-consistent polyp localization over time.
To model spatiotemporal dependencies, we employ a cross-aud attennon mechanism within the temporal alignment module. Given the current frame feature Qr as the query and the memory bank features

---

Published as a conference paper at ICLR 2026
Table 1: Quantitative comparison with different state-of-the-arr methods on SUN-SEG and CVC- 612 test sets. The highest value is indicated in bold, while the second highest value is underlined.
SUN SIN-SEG-Fary Fasy
SUN SPG-
CVC-612
Model
Backhone
Class
S,
Fe'
Diec
S,
Ernn
Fj
Dice
5
Eg
Fg
Dioc
ZoomNeet
PVT-82
NVS
88.33
90.48
80.66
85,49
87.66
90.84
80.25
83.51
94,66
97.83
92.45
93.17
PVT-B2
NVS
83.09
85.91
87.04
97.37
93.62
AutoSAM
VIE-B
IPS
86,28
91.67
78.36
81.28
83.57
89.93
73.59
77.37
91.52
95.38
87.47
88.73
WeakPolyp
PVT-B2
IPS
89.04
92.77
83.83
85.27
88.41
92.57
82.93
84.59
91.44
95.78
88.54
88.79
PNS+
Res-50
VPS
86,20
86. 17
76.28
82.23
84.29
86.13
72.98
79:60
94.81
96.75
89.
93.06
MAST
PVT-B2
VPS
84,53
89.81
77.04
78.43
86. 17
91.42 .42
77.76
80.32 32
92.03
95. 38
87.41 $7.47
90.84
VPSAM
VIT-B
VPS
89.31
92.34
82.86
85.62
88.93
92.13
82.98
85.28
93.26
95.75
89.65 99.63
92.33
SALI
PVT-B2
VPS
89,54
93.07
83.68
86.17
87,58
91.93
80,56
83.87
91.73
95.
86.54
88.77
PVT-B2
VPS
94.86
85.82
88.03
89.63
93.92
95.02
98.46
93.58
94.31
Table 2:
Efficiency comparison with SOTA
Table 3:
The performance of different frame rates on the RMP module.
methods on SUN-SEG.
SUN-SEGA SAN-SEG-Hard
SUN-SEG-Hard
Method
Dice
GFlops
(M)
FPS
label
Dice
loU
GFlops
Param.(M)
FPS
SLTNet
83.36
32.47
25.79
12.36
1-frame
86.27
75.86
46.77
28.53
31.27
ZoomNext
83.51
42.95
28.18
9.69
2-frame
86.41
76.07
46.91
28.85
29.43
PNS+ SALI
79.60
45.99
9.79
76.08
3-frame
86.55
76.29
47.06
29.16
28.77
83.87
21.19
26.14
18.07
4-frame
86.66
76.46
47.19
29.48
26.94
Ours
86.27
46.77
28.53
31.27
KT and Vr key and value, we compute the attention-enhancee representation as:
Z L (LQ)))
where L represents linear projection. The output Z is then refined via multi-layer perceptron (MLP), followed by layer normalization with residual connections:
Qu LN(MLP(Z) + 2) + Qr, KM Kr + Vz.
Next, the temporally-awee aware feature Q. is fused with the current frame's spatial information and projected to form query pairs (qx, ). Simultaneously, the combined memory KM is projected to obtain key-value pairs (m;; m.). These are fed into the mask affinity module, where cross-attention
is computed, and the final spatiotemporal representation is obtained by:
output = qe @ Attenttion (9k, my, m.).
(10)
3.4 LOss FUNCTION
We apply multi-level supervision using a hybrid loss that combines weighted binary cross-entropy (BCE) Ji et al. (2022) and weighted intersection over union (IoU) Rahman & Wang (2016). The total loss is defined as:
Call Σ
- ( (Pi, G) C.O. (Pi, G)),
(11)
i=1
where Pi the prediction at the i-th decoder stage, and the ground truth.
EXPERIMENTS
4.1 DATASETS AND IMPLEMENTATION DETAILS
Datasets. We evaluate our proposed HFSTI-Net on the two polyp datasets, SUN-SEG Ji et al. (2022) and CVC-612 Bernal et al. (2015). (1) The SUN-SEG dataset contains a total of 49, frames from 285 sequences, divided into three subsets: the training set with 19,544 frames from 112 sequences; the SUN-SEG-Easy test set with 17,070 frames from 119 sequences; and the SUN-SEG- Hard test set with 12,522 frames from 54 sequences. (2) The CVC-612 dataset consists of 612 frames from

---

Published as a conference paper at ICLR 2026
Frame
GT
Neom
SL.Teet
AutoSAM
WeakPolyp
PNS+
MAST
VPSAM
SALI
Ours
Figure 5: Visual comparison with SOTA methods on SUN-SEG. Red, green, and yellow represent ground truth, prediction, and their overlapping regions, respectively.
Table 4: Data visualization of module ablation on the SUN-SEG.
Table 5: Sub-component ablation of HFSI SUN-SEG.
HEST
RMP
SUN-SEG-EON BaGy-Eay
SUN-SEG Hard
FEB
SRB
IFB
Eay--Eay
F7
89.71
93.88
94.15
87.04
88.90
02.91
$1.26
$4.05
93.08
MM.91
95.76
86.10
87
19.17
95.26
63.16
81
$1.50
90.51
93.87
94.00
84.43
87.28
10.92
95.18
$2.14
85.27
31 colonoscopy sequences. For SUN-SEG, set aside 20% of the training set the validation set. For CVC-612, the dataset was split into training, validation, and test sets with a 6:2:2 ratio.
Evaluation metrics. For thorough comparison, we utilize four evaluation metries: Dice, structure- measure (Sx) Fan et al. (2017), enhanced- -aligmeell measure (Eg")) Fan et al. (2021), and weighted F-measure (FFg) Margolin et al. (2014), following previous studies Ji et al. (2022); Hu et al. (2024).
Implementation details. The proposed network was implemented on the PyTorch Paszke et al. (2019) platform and trained on one NVIDIA 3090 GPU. Pvtv2.b2 Wang al. (2022) used the backbone for all experiments. We trained our model for 30 epochs with batch size of 8. A video clip of 2 frames with a patch size of 352 x 352 was fed into the network. We adopted the Adam optimizer with an initial learning rate of le and weight decay of 0.1 for 10 epochs.
4.2 COMPARISON WITH EXISTING METHODS
To demonstrate the superiority of our proposed method, we compare it with popular image- and video-level object/polyp segmentation methods on SUN-SEG-Easy, SUN-SEG-Hard, and CVC- 612. The compared methods include ZoomNext Pang et al. (2024), SLTNet Cheng et al. (2022), AutoSAM Shaharabany et al. (2023), WeaklyPolyp Wei al. (2023), PNS+ Ji al. (2022), MAST Chen et al. (2024), VPSAM Fang et al. (2024) and SALI Hu et al. (2024). These methods can be categorized into four groups: (1) natural video segmentation (NVS), (2) IPS, and (3) VPS. All training parameters are controlled and set identically for consistency.
Comparison with SOTA methods. The comparison results between our method and above state-of- the-art methods the SUN-SEG and CVC-612 shown in Table 1. Under identical experimental settings, our method outperforms
soite-of-the--ee art approaches across every metric.
Visual comparison with SOTA. We present visual comparison of our approach with state-of-the- art methods in Figure 5. The figure showcases segmentation results on three challenging cases: (1) similar foreground and background (lower case), (2) sequence of consecutive low-quality frames
8

---

Published as a conference paper at ICLR 2026
RMP both Figure 6: Visualization of module ablation on SUN-SEG. Red, green, and yellow denote GT, prediction, and overlap, respectively.
Frame
GT
both HFSI
wio both
FFB
SRB
IPB
Figure 7: Visual comparison of image embed- ding HFSI components in the SUN-SEG dataset.
Only Frequency Figure 8: T-SNE visualization of different mod- eling strategies, where purple represents polyps and pink represents background.
Ours
Only Space
(d)
(e)
Figure Failure cases. Red, green and yellow represent the GT, prediction and their overlap- ping regions, respectively.
(middle case), and (3) variations between consecutive frames (upper case). As shown Figure 5, our method consistently achieves precise, stable segmentation where others struggle, demonstrating its effectiveness in challenging conditions.
Performance efficiency comparison with SOTA. The focus of this paper is to improve accuracy. To comprehensively analyze the strengths of our method, we also examine the trade-off between accuracy and efficiency. Our method achieves real-time inference speed, with FPS of 31.27, while surpassing the accuracy of other SOTA methods, as shown in Table 2.
Table Ablation study on the SUN-SEG-Hard Dataset.
Interaction Method
Dice (%)
Params (M)
Linear (1 Conv)
84.15 2.12)
28.42
Spatial Attention Ours
85.38 (1 0.89)
29.80
86.27
28.53
4.3 ABLATION STUDIES
Effectiveness of RMP. To assess the contribution of the recurrent mask-guided propagation (RMP) module, we performed ablation studies. Removing RMP leads to a clear drop in performance Ta- ble 4, as temporal cues essential for consistent segmentation in video sequences. Visual results in Figure 6 show that RMP mitigates polyp fragmentation and improves boundary continuity across frames. We further evaluated the impact of varying the number of memory frames during training (1-4), as shown in Table 3. While using more frames improves accuracy, it slightly reduces FPS. Notably, even a single memory frame provides competitive performance with real-time inference, offering practical trade-off between accuracy and efficiency.
Effectiveness of HFSI. As shown in Table 4 and Figure 6, HFSI improves both accuracy and bound- ary quality. To assess its components, we ablated FFB, SRB, and IFB individually. Results in Figure and Table show that removing IFB causes notable drop, confirming the value of its terwoven fusion. Unlike simple merging, IFB enables bidirectional interaction between spatial and frequency domains for better global-local alignment. Further degradation occurs when disabling both IFB and SRB, and the worst performance appears when both FFB and IFB are removed, due to the lack of spectral filtering and fusion. These results validate the tightly integrated design of HFSI. As shown in Figure 8, shows that combining both frequency and spatial domains (Ours) leads to better separation of polyp and background classes compared to using only spatial or frequency information individually.
Effectiveness of FFT/IFFT. To verify the necessity of FFT for frequency space interaction, we perform ablation comparing FFT with linear 1 x convolution block and spatial-attention

---

Published as a conference paper at ICLR 2026
block (see Table 6). FFT naturally provides global context and isolates high-frequency components important for preventing "shape collapse," whereas linear and attention- saattoo--baed operators remain lim- ited to spatial-domain processing. Results SUN-SEG-Hard show that replacing FFT with linear block leads to clear performance drop (-2.1 2.2 Dice), and spatial attention also underperforms while using
parameters. These findings confirm that spectral interaction offers complementary structural cues that spatial operators cannot replicate.
Impact of memory size N. To determine the optimal number of historical frames stored in memory, we conduct an ablation study on N. As shown in Figure 11, the performance peaks when using N 8, where the model effectively leverages temporal context without accumulating excessive noise. Increasing N further (e.g., to 12) leads to accuracy degradation (Dice 84.31 %) due to error accumulation. Thus, reliability.
achieves the best trade-off between temporal richness and memory
Impact of input clip. As shown in Figure 10 (a-b), we also explore the impact of different clip length L. The performance improves greatly when L increases from to 2 because more spatio-temporal information is obtained. However, larger L values (e.g., and 7) cause performance degradation. Longer clips can theoretically bring more spatio-temporal information, which is effective for clips composed of frames with high boundary discrimination. However, for colonoscopic videos with low boundary discrimination, information between frames with a long temporal distance may bring redundant information that interferes with effective spatio-temporal information.
analyze the possible reason is that establishing spatio-tenne temporal
5, Score
Ep" Score
Score (95)
C5.14
Soone 1%53
O7.A2
92
F3 Cost
Dice Score
63.26
Score 195
Score 1%
01.42
SEG-Hard
CVC-612
Mumber of frue Clo
Rumber if Op
Memory
Memory
Figure 10: Ablation study input clip.
Figure 11: Ablation study memory size.
4.4 DISCUSSIONS AND LIMITATIONS
While we only conduct experiments colonoscopy video datasets, believe that our HFSTI-Net general enough to analyze other medical videos with similar challenges. Moreover, our method- ology still exhibits certain limitations that warrant further investigation. As shown in Figure 9, spot interference (a-b), small polyps with very low contrast (c-d), and dramatic shapes (e-f) may limit method.
CONCLUSION
In this paper, we propose the (HFSTI-Net), novel network tackle the critical challenges of shape collapse and episodic amnesia in vidco polyp segmentation. Its architecture effectively integrates three domains: the Hierarchical Frequency-spatial Interaction (HFSI) module leverages interwo- ven fusion of frequency and spatial cues to ensure fine-grained boundary localization and prevent shape collapse, while the Recurrent Mask-guided Propagation (RMP) module forms stable tempo- ral memory to maintain long-term consistency. These pathways synergistically fused, yielding segmentations that are both precise and temporally coherent. Paired with efficient inference ca- pabilities, method achieves superior balance between performance and real-time application. Extensive experimental results on SUN-SEG and CVC-612 demonstrate the effectiveness our proposed method.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported partly by National Natural Science Foundation of China (No. 62273241), Natural Science Foundation of Guangdong Province, China (No. 2024 19160 the Shenzhen Research Foundation for Basic Research, China (No. 1YY12025060441114000444 and grant under Hong Kong RGC Collaborative Research Fund (project no C5055-24G).
REFERENCES
Jorge Bernal, F Javier Sánchez, Gloria Fernández-Esparree Debora Gil, Cristina Rodríguez, and Fernando Vilariño. Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation saliency maps from physicians. Computerized medical imaging and graphics, 43:99-111, 2015.
Debayan Bhattacharya, Konrad Reuter, Finn Behrendt, Lennart Maack, Sarah Grube, and Alexander Schlaefer. Polypnextistm: a lightweight and fast polyp video segmentation network using con- vnext and convlstm. International journal of computer assisted radiology and surgery, 19(10): 2111-2119, 2024.
Geng Chen, Junqing Yang, Xiaozhou Pu, Ge-Peng Ji, Huan Xiong, Yongsheng Pan, Hengfei Cui, and Yong Xia. Mast: Video polyp segmentation with mixture-attention siamese transformer. arXiv preprint arXiv:2401. 2ate:2201.229999 2024.
Mengjun Cheng, Zishang Kong, Guoli Song, Yonghong Tian, Yongsheng Liang, and Jie Chen. Learnable oriented-derivative network for polyp segmentation. In Medical Image Computing and Computer Assisted Intervention-MICCC. 2021: 24th International Conference, Strasbourg, France, September 27-October 1, 2021, Proceedings, Part I 14, pp. 720-730. Springer, 2021.
Xuclian Cheng, Huan Xiong, Deng-Ping ping Fan, Yiran Zhong, Mchrtash Harandi, Tom Drummond, and Zongyuan Ge. Implicit motion handling for video camouflaged object detection. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13864- 13873, 2022.
Lu Chi, Borui Jiang, and Yadong Mu. Fast fourier convolution. Advances in Neural Information Processing Systems, 33:4479 44888 2020.
Bo Dong, Wenhai Wang, Deng-Ping Fan, Jinpeng Li, Huazhu Fu, and Ling Shao. Polyp-pvt: Polyp segmentation with pyramid vision transformers. arXiv preprint arXiv:2108. 06932, 2021.
Deng-Ping Fan, Ming-Ming Cheng, Yun Liu, Tao Li, and Ali Borji. Stracture-measure: A new way to evaluate foreground maps. In Proceedings of the IEFE international conference on computer vision, pp. 4548 4557, 2017.
Deng- Ping Fan, Ge-Peng Ji, Guolei Sun, Ming-Ming Cheng, Jianbing Shen, and Ling Shao. Cam- ouflaged object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2777-2787, 2020a.
Deng-Ping Fan, Ge-Peng Ji. Tao Zhou, Geng Chen, Huazhu Fu, Jianbing Shen, and Ling Shao. Pranet: Parallel reverse attention network for polyp segmentation. International conference on medical image computing and computer-assisted intervention, pp. 263-273. Springer, 2020b.
Deng-Ping Fan, Ge-Peng Ji, Xuebin Qin, and Ming-Ming Cheng. Cognitive vision inspired object segmentation metric and loss function. Scientia Sinica Informationis, 6(6):5, 2021.
Zhixue Fang, Yuzhi Liu, Huisi Wu, and Jin Qin. Vp-sam: Taming segment anything model for video polyp segmentation via disentanglement and spatio-temporal side network. In European Conference on Computer Vision, pp. 367-383. Springer, 2024.
Chunming He, Kai Li, Yachao Zhang, Longxiang Tang, Yulun Zhang, Zhenhua Guo, and Xiu Li. Camouflaged object detection with feature decomposition and edge reconstruction. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22046-22222 22055, 2023.
11

---

Published as a conference paper at ICLR 2026
Qiang Hu, Zhenyu Yi, Ying Zhou, Fang Peng, Mei Liu, Qiang Li, and Zhiwei Wang. Sali: Short- term aligmment and long-term interaction network for colonoscopy video polyp segmentation. In International Conference Medical Image Computing and Computer-Asiisted Intervention, pp- 531-541. Springer, 2024.
Ge-Peng Ji, Yu-Cheng Chou, Deng-Ping Fan, Geng Chen, Huazhu Fu, Debesh Jha, and Ling Shao. Progressively normalized self-attention network for video polyp segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 142-152. Springer, 2021.
Ge-Peng Ji, Guobao Xiao, Yu-Cheng Chou, Deng-Ping Fan, Kai Zhao, Geng Chen, and Luc Van Gool. Video polyp segmentation: A deep learning perspective. Machine Intelligence Re- search, 19())531-554, 2022.
Shaohua Li, Xiuchao Sui, Xiangde Luo, Xinxing Xu, Yong Liu, and Rick Goh. Medical image seg- mentation using squeeze-anddeepaasion transformers. arXiv preprint arXiv:2105.09511, 2021.
Xiaotong Li, Jilan Xu, Yuejie Zhang, Rui Feng, Rui-Wet Zhao, Tao Zhang, Xuequan Lu, and Shang Gao. Tecnet: Temporally consistent context-free network for semi-supervised video polyp seg- mentation. In IJCAI, pp. 1109-1115, 2022.
Yan Li, Zhuoran Zheng, Wenqi Ren, Yunfeng Nie, Jingang Zhang. and Xiuyi Jia. Frequency aware and graph fusion network for polyp segmentation. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1586-1590, IEEE, 2024.
Yu Liu, Haihang Li, Juan Cheng, and Xun Chen. Mscaf-net: A general framework for camouflaged object detection via learning multi-scale contexr-aware features. IEEE Transactions on Circuits and Systems for Video Technology, 33(9):4934-494 2023.
Ran Margolin, Lihi Zelnik-Mann Manor, and Ayellet Tal. How to evaluate foreground maps? Proceed- ings of the IEEE conference on computer vision and pattern recognition, pp. 248-255, 2014.
Youwei Pang, Xiaoqi Zhao, Tian-Zhu Xiang, Lihe Zhang, and Huchuan Lu. Zoomnext: A unified collaborative pyramid network for camouflaged object detection. JEEE transactions on pattern analysis and machine intelligence, 2024.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Juana González-Buene Puyal, Kanwal K Bhatia, Patrick Brandao, Omer F Ahmad, Daniel Toth, Rawen Kader, Laurence Lovat, Peter Mountney, and Danail Stoyanov. Endoscopic polyp seg- mentation using a hybrid 2d/3d cnn. In Medical Image Computing and Computer Assisted Intervention-MICCA/ 2020: 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part VI 23, pp. 295-305, Springer, 2020.
Zequn Qin, Pengyi Zhang, Fei Wu, and Xi Li. Fcanet: Frequency channel attention networks. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 783-792, 2021.
Md Atiqur Rahman and Yang Wang. Optimizing intersection-over-uiiinn in deep neural networks for image segmentation. In International symposium on visual computing, pp. 234-244. Springer, 2016.
Tal Shaharabany, Aviad Dahan, Raja Giryes, and Lior Wolf. Autosam: Adapting sam to medical images by overloading the prompt encoder. arXiv preprint arXiv:2306.06370. 2023.
Aasma Shaukat and Theodore R Levin. Current and future colorectal cancer screening strategies. Nature reviews Gastroenterology & hepatology, 19())5211-531 2022.
Yanguang Sun, Chunyan Xu, Jian Yang, Hanyu Xuan, and Lei Luo. Frequency-spatial entanglement learning for camouflaged object detection, In European Conference on Computer Vision, pp. 343- 360. Springer, 2024.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
12

---

Published as a conference paper at ICLR 2026
Chenyang Wang. Junjun Jiang, Zhiwei Zhong, and Xianming Liu. Spatial-frequenn mutual learning for face super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13356-22366, 2023a.
Shunxin Wang, Raymond Veldhuis, Christoph Brune, and Nicola Strisciuglio. What do neural networks learn in image classification? frequency shortcut perspective. In Proceedings of the IEEE/CVP International Conference on Computer Vision. pp. 1433-1442, 2023b.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 833)415-424, 2022.
Jun Wei, Yiwen Hu, Ruimao Zhang. Zhen Li, S Kevin Zhou, and Shuguang Cui. Shallow at- tention network for polyp segmentation. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2021: 24th International Conference, Strasbourg, France, September 27- October 1, 2021, Proceedings, Part 24, pp. 699-708. Springer, 2021.
Jun Wei, Yiwen Hu, Shuguang Cui, S Kevin Zhou, and Zhen Li. Weakpolyp: You only look bound- ing box for polyp segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 757-766. Springer, 2023.
Huisi Wu, Zebin Zhao, Jiafu Zhong, Wei Wang. Zhenkun Wen, and Jing Qin. Polypseg+: A lightweight context-aware network for real-time polyp segmentation. IEEE Transactions Cy- bernetics, 53(4):2610-262) 2022.
Huisi Wu, Zebin Zhao, and Zhaoze Wang. Meta-unet: Multi-scale efficient transformer attention unet for fast and high-accuracy polyp segmentation. IEEE Transactions on Automation Science and Engineering. 2023.
Zhenyu Wu, Fengmao Lv,, Chenglizhao Chen, Aimin Hao, and Shuo Li. Colorectal polyp segmen- tation in the deep learning era: A comprehensive survey, arXiv preprint arXiv:2401.1173, 2024.
Ziang Xu, Jens Rittscher, and Sharib Ali. Sstfb: Leveraging self-supervised pretext learning and temporal self-attention with feature branching for real-time video polyp segmentation. arXiv preprint arXiv:2406. /0200, 2024.
Ziheng Xu, Dehui Qiu, Senlin Lin. Xinyue Zhang. Sheng Shi, Shengtao Zhu, Fa Zhang, and Xiaohua Wan. Temporal correlation network for video polyp segmentation. In 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 1317-1322. IEEE, 2022.
Yundong Zhang, Huiye Liu, and Qiang Hu. Transfuse: Fusing transformers and cnns for medical image segmentation. In Medical image computing and computer assisted Ciovvettio-MMCCAN 2021: 24th international conference, Strasbourg. France, September 27 27-October 1. 2021, pro- ceedings, Part 24, pp. 14-24. Springer, 2021.
Tao Zhou, Yi Zhou, Kelei He, Chen Gong, Jian Yang, Huazhu Fu, and Dinggang Shen. Cross-level feature aggregation network for polyp segmentation. Pattern Recognition, 140:109555, 2023.
13

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.1 MORE ABLATION STUDIES ON SUN-SEG
More comparisons on efficiency. To provide more comprehensive comparison of efficiency and aceuracy, as shown in Figure 12, we further evaluate our method against other approaches. The results demonstrate that our method not only achieves real-time inference but also reaches state-of- the-art accuracy.
87
86
+0.99
+1.68
Ours
85
VPSAM
Dice (%)
84
SALI
Weakpolyp
83
SLTNet
ZoomNext
82
81
AutoSAM
80
MAST
PNS+
79
10
20
30
40
50
60 70
80
Inference speed (FPS) Figure 12: Cerformance-effcieme comparison with other State-of-the-an methods on SUN-SEG.
Component ablation of CVC-612. We validated the effectiveness of our module design through ablation experiments on the CVC-612 dataset. As shown in Table 7, the results demonstrate that each module contributes individually, and the integrated framework outperforms existing methods, confirming the component effectiveness within our architecture. This validates the modules' gener- alizability across heterogeneous datasets.
Performaace-effciinnyyy of subcomponent ablation. In addition to comparing the efficiency and performance with other SOTA methods, as shown Table 8, we conducted a module-wise ablation study to analyze the trade-off between segmentation accuracy (Dice) and processing speed (FPS). The table's s indicators further confirm that in automatic video polyp segmentation, HFSI and RMP address different challenges: HFSI decouples frequency-spatial features, while RMP ensures spa- tiotemporal consistency across frames. Their synergy enhances segmentation accuracy but impacts real-time performance. In summary, HFSI enhances feature representation, while RMP stabilizes temporal consistency, balancing global perception and local adjustment. Although this multidi- mensional interaction reduces processing speed, the decrease in efficiency is acceptable given the improved performance.
Table 7: Component ablation experiments on the CVC-612 Dataset.
Table 8: Subcomponent ablation performance- efficiency comparison on SUN-SEG-Easy Hard set with 352 352 resolution.
CVC-612
HESI
RMP
Fmmn
Diec
SUN-SEG-Hard
Method
93.87
96.73
90.92
92.43
Dice
GFlops
Param.(M)
FPS
94.01
97.55
91.14
93.63
w/o both
83,67
31.96
28.16
37.39
94.35
97.95
92.64
93.74
w/o IIFSI
84.03
45.86
28.23
35.27
95.02
98.86
93.58
94.31
w/o RMP
85.27
32.87
28.47
32.73
86.27
46.77
28.53
31.27
A.2 MORE VISUAL COMPARISON RESULTS
To demonstrate the superiority of our proposed method, we conduct visual comparisons with eight state-of-thh-aan methods on CVC-612, including FSEL Sun et al. (2024), MSCAF Liu et al. (2023), ZoomNext Pang et al. (2024). SLTNet Cheng al. (2022), AutoSAM Shaharabany al. (2023), WeaklyPolyp Wei et al. (2023), PNS+ Ji et al. (2022), and SALI Hu et al. (2024). As shown in Figure 13, the visual results demonstrate that method outperforms previous state-of-the-ar ap-
14

---

Published as a conference paper at ICLR 2026
proaches in polyp boundary segmentation, integrity preservation, and localization accuracy. This demonstrates the method's s effectiveness on both SUN-SEG and CVC-612 datasets.
To illustrate the segmentation capability continuous video streams, sample four consecutive frames at varying scales from the SUN-SEG dataset Ji et al. (2022) and compare our results with several state-of-the-art methods, shown in Figure 14.The SUN-SEG comparison clearly shows that our method delivers more stable and consistently accurate segmentation across consecutive frames than existing state-of-the-art approaches. Despite the dataset's wide variations in lighting, texture, and object shapes, the red, green, and yellow regions in the figures show our predictions closely match the ground truth, demonstrating our approach robustness.
Fremn
GT
FSE
MSCAF
SLTnet
ZoomNext
AutoSAM
Weskpolpp
PNS+
SALI
Outs
Visualization of module ablation on CVC-612 test set. Red, green and yellow represent the GT, prediction and their overlapping regions, respectively.
Figure 13:
15

---

Published as a conference paper at ICLR 2026
Frame
GT
Zoom/Nes
SLTTutt
AutoSAM
WeakPPlyp
PNS+
MAST
VPSAM
SALI
Ours
Figure 14: Visual comparison with SOTA methods on SUN-SEG, Red, green, and yellow represent ground truth, prediction, and their overlapping regions, respectively.
16


Reviews:
[{"review_id": "UUTGAON229", "reviewer": "Reviewer_EtG3", "summary": "This work presents HFSTI-Net, a framework for Video Polyp Segmentation (VPS). The paper aims to solve two specific  problems: \"shape collapse\" (poor boundary integrity) and \"episodic amnesia\" (temporal instability). To this end, the authors propose a dual-path architecture. The first component is the Hierarchical Frequency-spatial Interaction (HFSI) module, which attempts to improve spatial localization by fusing frequency-domain and spatial-domain features. The second component is the Recurrent Mask-guided Propagation (RMP) module, which is intended to enforce temporal consistency by propagating information via a memory bank.\nFrom a completeness perspective, the paper presents a full pipeline, including the model architecture, training details, and extensive experiments on the SUN-SEG and CVC-612 benchmarks. The reported results suggest that this approach achieves competitive performance against current state-of-the-art methods.", "questions": "1.The RMP module's dynamic memory links temporal information across long sequences, but this introduces a significant potential risk of error propagation. If the model generates an incorrect prediction (a \"bad case\") and stores the associated features and mask, this error could pollute subsequent frames and lead to cascading failures. Have the authors conducted specific experimental analyses to investigate this limitation?\n2.The architecture of the \"Mask Encoder\" shown in Figure 2 is not detailed. Could the authors please clarify its design, specifically how it encodes the 2D mask into a feature representation, and how the resulting \"PEmbeddings\" are subsequently fused into the model?\n3.We request the authors to comment on the transferability of the RMP module's memory design. How effective is this mechanism expected to be when generalized to different datasets, which may feature distinct temporal dynamics or target characteristics?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.The paper identifies and articulates two significant and practical challenges in clinical VPS (shape collapse and episodic amnesia) as the core motivation for the work.\n2.Based on the provided tables, the method achieves strong performance, reportedly outperforming existing state-of-the-art methods on the SUN-SEG and CVC-612 datasets across standard metrics.\n3.Based on the provided tables, the method achieves strong performance, reportedly outperforming existing state-of-the-art methods on the SUN-SEG and CVC-612 datasets across standard metrics.", "weaknesses": "1.On the Novelty of the HFSI Module: The core concept of the HFSI module, an \"interwoven dual-path design\" for frequency and spatial features, appears not to be entirely new. The innovation seems to rest on the unique Interwoven Fusion Block (IFB), which is primarily realized by the cross-concatenation of convolutional features and frequency-domain features and fused with a gating network. This concept has been proposed and implemented in considerable prior work. As such, the innovation is incremental rather than disruptive. The authors must define their claim to novelty more precisely and contrast it with a broader range of related work.\n2.On the Rationale of the Interwoven Fusion Block (IFB) Design: In Figure 3(c), the IFB design is perplexing. Frequency information (from an initial FFT) is concatenated with spatial information, and subsequently, another FFT operation is applied. The rationale for applying an FFT to features that are already (at least partially) in the frequency domain is perplexing and counter-intuitive. If this \"FFT-on-FFT\" operation is a deliberate and innovative component of the design, it requires substantial justification. The authors must provide a detailed motivation and theoretical principle for this specific operation. However, a thorough search of both the main body and the appendix reveals no such explanation , leaving the reader confused regarding the block's fundamental working principles.\n3.On the Clarity of the RMP Module: The paper claims the RMP module is \"mask-guided\" and that the mask $P^{t-1}$ is passed into it. However, Figures 2 and 4, along with the provided formulas, fail to show how the mask $P^{t-1}$ is stored into the Memory (e.g., via a temporal FIFO). The paper also lacks a clear description of the \"Retrieve\" step. This ambiguity in the RMP's core mechanics is likely to cause confusion for the reader.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0qRzIEgFMY", "reviewer": "Reviewer_qtCK", "summary": "This paper proposes HFSTI-Net, a novel hierarchical model that integrates frequency, spatial, and temporal cues through two key components, i.e., Hierarchical Frequency-Spatial Interaction (HFSI) module and Recurrent Mask-Guided Propagation (RMP) module. Extensive experiments demonstrate the effectiveness of each proposed module on two polyp segmentation datasets.", "questions": "1. In Table 3, it is not very clear why increasing the number of frames would lead to the increase of parameters. \n\n2. Computing feature interaction in the frequency domain is not very convincing, the authors are encouraged to show that using FFT/IFFT is better than other approaches (e.g. simple linear transformation).\n\n3. Can you explain how query/key/value is obtained via FFT? (L237-238)", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Despite the model's hierarchical complexity, HFSTI-Net achieves >30 FPS on a single GPU, making it viable for clinical and real-time processing.\n\n2. Detailed ablation and component studies validate the necessity of each sub-module (e.g. FFB, SRB, IFB).", "weaknesses": "1. The authors proposed the Recurrent Mask-Guided Propagation Module to ensure long-term consistency of the segmentation. However, the entire input sequence only contains two frames (L376-377), making the claim less convincing. \n\n2. The number of HFSI layers is not fully studied in the experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9EKNi96UvX", "reviewer": "Reviewer_ytTz", "summary": "This paper presents the HFSTI model for video polyp segmentation to address two major challenges in current video polyp segmentation: shape collapse and episodic amnesia. The HFSTI model consists of two main modules: HFSI and RMP. The HFSI module is responsible for extracting and fusing spatial and frequency features, while the RMP module focuses on capturing long-term spatiotemporal dependencies in polyp videos.\nThe HFSI module comprises three components: the FFB (Frequency Filter Block), SRB (Spatial Refinement Block), and IFB (Interwoven Fusion Block). The FFB uses a frequency-domain self-attention mechanism to extract enhanced frequency-aware features, while the SRB applies spatial self-attention to capture spatial features. The IFB utilizes gated multiplicative attention to effectively fuse the enhanced frequency-aware features, spatial features, and earlier-layer features.\nThe RMP module captures long-term spatiotemporal dependencies by storing and retrieving historical frame features and masks from a memory bank. It consists of two submodules: TAM (Temporal Alignment Module), which generates temporally-aware features, and MAM (Mask Affinity Module), which aligns the current prediction with historical context through cross-attention.\nExtensive experiments were conducted comparing the proposed method with ZoomNext, SLTnet, AutoSAM, WeakPolyp, PNS+, MAST, VPSAM, and SALI on the SUN-SEG and CVC-Clinic-612 datasets. Additionally, performance analysis was carried out on SUN-SEG with different frame rates. Visualizations were provided for the SUN-SEG dataset, and ablation studies were conducted on the HFSI, RMP modules, and submodules of HFSI.", "questions": "In Figure 2, does Memory = N refer to the number of historical frames stored in the memory? If so, what value is N set to? Was an ablation study conducted on N to determine the most suitable value?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "One of the main innovations of this paper is the introduction of frequency features into the field of video polyp segmentation (VPS). Traditional image segmentation methods typically focus on extracting spatial features, often neglecting the potential of frequency-domain information. The authors propose the Hierarchical Frequency-Spatial Interaction (HFSI) module, which effectively integrates frequency features with spatial features. This not only improves segmentation accuracy in low-contrast regions but also addresses the common issue of shape collapse in complex backgrounds. The method innovatively combines global frequency information and local spatial details, enhancing the model's robustness in dynamic scenarios, particularly when polyps are visually similar to surrounding tissues, thus improving target localization accuracy. Additionally, the paper introduces the RMP module, which uses a memory mechanism and mask-guided propagation to effectively capture long-range spatiotemporal dependencies, solving the common problem of episodic amnesia in video sequences. This demonstrates the model's strong innovation.\nThe paper thoroughly validates the proposed method through comprehensive experimental design. First, the authors conduct a quantitative comparison with current state-of-the-art methods on datasets like SUN-SEG and CVC-612, showcasing the superior performance of HFSTI-Net in terms of accuracy and robustness. Moreover, the authors perform a visual analysis to demonstrate the differences in performance across various methods in complex scenarios, such as low-quality frames and background interference, followed by a performance analysis to further confirm the stability and accuracy of the proposed method. Additionally, ablation studies on the HFSI and RMP modules, as well as the submodules of HFSI, are conducted on SUN-SEG, with both quantitative and visual analysis. This indicates the model's strong reliability.\nThe paper also offers a candid discussion of the model's limitations, particularly in handling certain types of polyps, such as those with low contrast or drastic shape variations, where segmentation performance is somewhat limited. Furthermore, the paper suggests the potential application of this model in other medical imaging fields.", "weaknesses": "The images of the modules in this paper are not sufficiently complete and clear, and some parts are prone to misunderstandings. For example, in Figure 2, the segmentation results are denoted as Pt in the image, where t is a subscript, while in the paper, “Pt−1 are passed into the RMP module”on line 200 uses a superscript. Additionally, in line 203, P = {Pi}4i=1 is used to represent the internal components of each P, This easily causes Pi to be mistaken for Pt.In Figure 3, for (a), the layer normalization step applied to the input is not shown in the image. For (b), the paper on line 256 states, “the input feature X is first passed through a 1 × 1 convolution to encode positional information,”but the corresponding operation is not depicted in the image. For (c), after a series of operations on the input, it is passed into both the frequency and spatial branches, but this step is also not shown in the image.\nAlthough the episodic amnesia issue is one of the core problems addressed in this study, the paper's visualizations only show two frames from a single video, which may not fully demonstrate the effectiveness of the solution. Displaying multi-frame video sequences would better validate the model's ability to capture long-term temporal dependencies and show how the RMP module alleviates episodic amnesia. A comparison with other methods would more clearly highlight the model's advantage in solving this issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)