Multi-view Masked Contrastive Representation Learning for Endoscopic Video Analysis
Authors: Kai Hu
Ye Xiao
Yuan Zhang
Xieping Gao
Venue: NeurIPS 2024 poster (2024)
Keywords: Endoscopic Video Analysis
Self-supervised Learning
Multi-view Mask Modeling
Contrastive Learning
Primary area: machine_vision

Abstract: Endoscopic video analysis can effectively assist clinicians in disease diagnosis and treatment, and has played an indispensable role in clinical medicine. Unlike regular videos, endoscopic video analysis presents unique challenges, including complex camera movements, uneven distribution of lesions, and concealment, and it typically relies on contrastive learning in self-supervised pretraining as its mainstream technique. However, representations obtained from contrastive learning enhance the discriminability of the model but often lack fine-grained information, which is suboptimal in the pixel-level prediction tasks. In this paper, we develop a Multi-view Masked Contrastive Representation Learning (M$^2$CRL) framework for endoscopic video pre-training. Specifically, we propose a multi-view mask strategy for addressing the challenges of endoscopic videos. We utilize the frame-aggregated attention guided tube mask to capture global-level spatiotemporal sensitive representation from the global views, while the random tube mask is employed to focus on local variations from the local views. Subsequently, we combine multi-view mask modeling with contrastive learning to obtain endoscopic video representations that possess fine-grained perception and holistic discriminative capabilities simultaneously. The proposed M$^2$CRL is pre-trained on 7 publicly available endoscopic video datasets and fine-tuned on 3 endoscopic video datasets for 3 downstream tasks. Notably, our M$^2$CRL significantly outperforms the current state-of-the-art self-supervised endoscopic pre-training methods, e.g., Endo-FM (3.5% F1 for classification, 7.5% Dice for segmentation, and 2.2% F1 for detection) and other self-supervised methods, e.g., VideoMAE V2 (4.6% F1 for classification, 0.4% Dice for segmentation, and 2.1% F1 for detection).

Multi-view Masked Contrastive Representation Learning for Endoscopic Video Analysis
Kai Hu
Ye Xiao Xiangtan University yxiao@smail xtu. edu. en
Xiangtan University kaihu@xtu, edu. cn
Yuan Zhang
Xieping Gao*
Xiangtan University yuanzäxtu. edu.
Hunan Normal University xpgaoChunnu. edu.
Abstract
Endoscopic video analysis effectively assist clinicians in disease diagnosis and treatment, and has played an indispensable role in clinical medicine. Unlike regular videos, endoscopic video analysis presents unique challenges, including complex camera movements, uneven distribution of lesions, and concealment, and it typically relies on contrastive learning self-supervised pretraining as main- stream technique. However, representations obtained from contrastive learning enhance the discriminability of the model but often lack fine- ---ggrrinedd information, which is suboptimal in the pixel-level prediction tasks. In this paper, we develop a Multi-view Masked Contrastive Representation Learning (MCRL)) framework for endoscopic video pre-training. Specifically, we propose multi-view mask- ing strategy for addressing the challenges endoscopic videos. We utilize the rrame-aggregated attention guided tube mask capture global-level spatiotemporal sensitive representation from the global views, while the random tube mask em- combine
ployed to focus local variations from the local views. Subsequently, multi-view mask modeling with contrastive learning to obtain endoscopic video representations that possess fine fine-grained perception and holistic discriminative capabilities simultancously. The proposed M is pre-trained on publicly available endoscopic video datasets and fine-tuned on 3 endoscopic video datasets for 3 downstream tasks. Notably, our MCCRI significany outperforms the current state-of-the-ar self-supervised endoscopic pre-training methods, e-g., Endo-FM (3.5% Fl for classification, 7.5% Dice for segmentation, and 2.2% Fl for detection) and other self-supervised methods, e.g., VideoMAE V2 (4, 6% Fl for classification, 0.4% Dice for segmentation, and 2. % Fl for detection).
Introduction
Video endoscopy is crucial medical examination and diagnostic tool widely used for inspecting various tissues and structures (the digestive tract, respiratory tract, and abdominal cavity, etc.) [[]. In clinical practice, endoscopic video analysis usually relies on the experience and expertise of physicians, which is not only time-consuming and labor-intensive but also prone to subjective emror.. Computer-aided medical analysis 2 3 can automatically and efficiently identify and classify
*Corresponding authors.
"Code is publicly available at: http://giihub.ccom/MMMMMMPMMMM
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

lesions, thereby assisting physicians in making more accurate diagnoses. In this paper, we focus on endoocopii videos with the aim of developing a robust pre-trained model for endoscopic video analysis to facilitate downstream tasks (i.e., classification, segmentation, and detection).
Yann LeCun has mentioned "the revolution will not be supervised [5] in multiple talks, emphasizing that the future development of artificial intelligence will increasingly rely on un-/self-supervised learning. Among them, self-supervised learning (SSL) aims to learn scalable visual representations from large amounts of unlabelled data for downstream tasks with limited annotated data. To learn
meaningful representations for SSL, researchers crafted visual pretext tasks 6 7 g
13 14 15 I6], which are summarized into two main categories: contrastive and generative [7]. Contrastive methods, also known as discriminative methods, employ a straightforward discriminative idea that pulling closer representations from the same image and pushing away different images, i.e., contrastive learning (CI.) 6 X By utilizing image-level prediction with global features, CL. can naturally endow pre-trained models with strong instance discriminability, which has been proven to be effective in classification tasks. However, CL also presents the challenge that downstream dense prediction tasks, such segmentation and detection, are not fully considered.
Generative methods aim reconstruct the input data itself by encoding the data into features and then decoding it, including AE [18), VAE [9) GAN (20), etc. Recently, masked image modeling (MIM)
L3
14 16 has demonstrated the strong potential in self-supervised learning. MIM masks a
substantial portion of image patches during training and utilizes an autoencoder [18] to reconstruct the original signal of the image, which. unlike CL, can enhance the ability to capture the pixel-level information. Following the success of MIM, some works have tried to extend this new pre-training
paradigm to the video domain for self-supervised video pre-training (21 [2 23). Actually, mask techniques 24 25 2d are crucial for the success of mask modeling. Endoscopic videos, in particular,
have higher dimensions and redundancy compared to static images. They also exhibit unstable inter- frame variations due to the manual manipulation by the doctor. Additionally, lesions in endoscopic videos often have low contrast and appear in obscured regions or with subtle variations. Therefore, simply applying random mask not only requires extensive pre-training time but also easily leads to poor performance.
To address the aforementioned issues, in this paper, we develop a Multi-view Masked Contrastive Representation Learning framework named MCCRI.. First, considering the characteristics of inter- frame instability and small inter-class differences in endoscopic videos, we propose multi-view masking strategy. Specifically, we introduce a -agreeageee attention guided tube masking strategy for the global views, which aggregates features from multiple frames to capture global spatiotemporal information. Simultaneously, a random tube masking strategy is employed from the local views, enabling the model to focus on local features. Second, to address the inadequacy of capturing pixel-level details contrastive learning, we integrate multi-view masked modeling into contrastive approach, which not only encourages the model to learn discriminative representations but also forces it to capture more refined pixel-level features. Extensive experiments have verified that our M²CRL significantly enhances the quality of endoscopic video representation learning and exhibits excellent generalization capabilities in multiple downstream tasks (i.e., classification, segmentation and detection). Overall, our contributions are summarized as follows:
We propose a novel multi-view masking strategy aimed at enhancing the capture of fine- grained representations in endoscopic videos by performing mask modeling both global and local views. This strategy involves utilizing the frame-aggregated attention guided tube mask to capture global-level spatiotemporal contextual relationships from the global views, while employing the random tube mask to focus on local variations from the local views.
We propose a multi-view masked contrastive representation learning framework that com- bines multi-view mask modeling with contrastive method to train endoscopic videos, which effectively addresses the limitation of contrastive method in capturing dense pixel dependen- cies by predicting the intensity of each pixel within masked patches.
We conduct extensive experiments on 10 endoscopic video datasets to evaluate the perfor- mance of MZCRL in comparison to other methods. MCCRL achieves 94.2% top-1 accuracy on PolypDiag 227, 81 81.4% on CVC-12k 28 and 86.3% on KUMC (29. outperforming the state-of-the--rt methods, iee, Endo-FM 30 by +3.5%, +7.5% and +2.2%, and VideoMAE V2 by +4.6%, +0.4%, and +2.1 12, respectively.

---

(a) MCCL Framework
Global Views
Teacher
Softmass
FAGTM
Soltree
Student
Lincar Headd
Vxx
C
Local Views
RTM
Waight
Lincar leedd
2/
Student
Softraa
(b) Frame- aggregated Attention Guided Tube Mask (FAGTM)
Random Tube Musk (RTM)
+ D
Mean & Seloct
Sampling
Guid:
Unnass
Shutfile
Misk
Punchify
The pipeline of the proposed MCCRL. For the generated global and local views with different frame rates and spatial sizes, we adopt two different mask strategies: the frame-aggregated attention guided tube mask and the random tube mask. These strategies are integrated with mask reconstruction and contrastive method, enabling the model simultaneously learn both the pixel-level and discriminative features of the video.
Figure 1:
Related Work
2.1 Self-supervised video representation learning
Sulf-super learning is a machine learning technique that mitigates the reliance on manual labeling by exploiting the inherent structure or properties of the data itself, and its common goal in computer vision is to learn scalable and generalisable visual representations. In SSL, the key lies in designing an appropriate pretext task, which involves generating suitable supervision using only visual signals. Pretext tasks in images [9] II [2] have been widely explore, and motivated by images, there has been great interest in exploring SSL pretext tasks for videos. Some early focused on extending image SSL methods for video, however video has specific temporal dimension. The temporal information can be leveraged for representation learning, including future prediction
10
works 31 322 33
34
35
36
37
38 39 40), temporal ordering [41 42
43 44 45).
48 49 and temporal consistency (1]]1]. In the last few years, contrastive 57 has made great progress in SSL. However, SSL based contrastive methods typically focuses on discriminative features of holistic information, while lacking the ability
object motion [46
47
learning [52 53
54
55
56
focus on fine-grained features.
2.2
Masked visual modeling
Masked visual modeling is proven to be simple and effective self-supervised learning paradigm 13 through straightforward pipeline based on masking and reconstruction. This paradigm quickly expanded the video domain, e.g. VideoMAE [22] and ST-MAE [2I). which extend image MAE to video. As with images, the choice of masking strategy affects self- supervised representation learning 2 25 26 59]. VideoMAE (22) employs random tube mask to model the same locations across frames to ensure spatial discontinuity while maintain temporal continuity. In contrast, ST-MAE (2]) generates a random mask for each frame independently, which are discontinuous in both space and time. Nevertheless, mask video modeling using randomly masked tokens to reconstruct is inefficient because the tokens embedded
14
[5]
16
58
21 22
video frame are not equally important. Several studies 60 61 62 63 have proposed various motion-based video mask strategies. However, our focus on endoscopic videos, which lack a strong motion counterpart compare the above works.

---

2.3 Self-supervised learning for endoscopy
In recent years, SSL has received increasing attention in the field of medical analysis, including endoscopy. Endo-FM [30] employs contrastive method to minimize the disparity in feature repre- sentations between different spatiotemporal views of the same video. Intrator [64] has proposed the use of contrastive learning to adapt video inputs for appearance-based object tracking. Hirsch [65] applies the SSL. framework masked siamese networks (MSNs) to analyze endoscopic videos. While MSNs use masking concept, it primarily serves data augmentation technique, with its essence still rooted in contrastive learning. Currently, most self-supervised pre-training for endoscopic videos relies contrastive methods. While these methods have shown promise for endoscopic video pre-training, relying solely on them may not fully capture the fine feature expressions of endoscopic videos.
3 Method
To address the limitations of contrastive methods in fine gre-ggrinndd information perception in SSL for endoscopic videos, we propose Multi-view Masked Contrastive Representation Learning (MCCLL framework for endoscopic video pre-training, as shown Fig. [ Here, we first review masked prediction in Section 3.1 Then, we present our proposed multi-view masking strategy in Section 3.2 and introduce our M-CRL framework in Section 3.3
3.1 Preliminary
Masked prediction is prevalent representation learning technique in natural language processing (NLP) [66
67 68
691. and many researchers have explore its application to images 14 70 72
22 23 MIM endeavors to learn image representations by solving a regression the pixel values randomly masked patches of R Xxx xC is reshaped into N = HW / p2 flattened patches as (x where (H, W) is the resolution of image, C' is the number of channels and P is the patch size. Each patch is represented with token embedding. MIM constructs a random mask m E {0, 1 to indicate the masked tokens that correspond to m, 1. In MAE [4, only the visible tokens {os [m = 0 are fed into the vision transformer obtain the latent feature, and then the decoder uses the latent feature and the masked tokens as inputs to predict ý. SimMIM [16], visible and invisible tokens are fed into the encoder. The prediction loss is calculated as the loss between the normalized masked tokens and the reconstructed ones in the pixel space by:
and videos 21
problem, where the model is tasked with XC the image. Specifically, an image
L L= 1 Σ
11 Im
(1)
where N,, is the number of masked tokens, I'm is the masked token, p is norm and its value is 2.
3.2 Multi-view masking strategy
Masked video modeling (MVM) 21 employs random mask strategies (i.e., random, spatial- only, temporal-oni to capture meaningful representations from pre-training videos. Although these strategies are effective for general video datasets with well-curated and stable distributions, they do not account for the unique characteristics of medical data. We have summarized two key characteristics of endoscopic videos: (1) Instability of inter-frame variations a prominent feature of endoscopic videos. These variations are driven by camera movement, instrument manipulation, and the uneven distribution of lesion areas, e.g., variations can range from drastic to minor, as the camera navigates from the intestinal wall to specific lesion sites. (2) Endoscopic video exhibits characteristics of small inter-class differences. The lesion tissues typically resemble surrounding the normal tissues in color, texture, shape, which complicates the model's 's ability to accurately identify the lesion area. Therefore, we propose a multi-view masking strategy by considering the above two points, the details are follows.
3.2.1 Frame-aggregated Attention Guided Tube Mask
To address the challenge of instability between video frames, we propose a frame-aggregated attention guided tube masking strategy. We aggregate the attention of all frames of the vidco along the frame

---

dimension to generate a frame- sume--ggregated attention map, which then dynamically guides the masking process. This way can capture the overall scene information in a video sequence from the global spatiotemporal information and ignores irrelevant spatiotemporal noise to some extent.
Semantic information extraction Our architecture consists of the teacher network fi and the student network fs- Our network employs a self-attention mechanism known as divided space-time attention mechanism 77,, which enhances the learning ability of the network while reducing the computational complexity. Specifically, we take an endoscopic video, from which we sample the global views {ug R x3x x3x W, = where T the number of frames in the sampled view. Each frame is then divided into N = H p/2² patches, which are mapped into patch tokens and fed into the transformer blocks of the teacher network. Thus, each encoder block processes N patch (spatial) and T temporal tokens. The network includes a learnable class token, [cls], which represents the global features learned by the network along spatial and temporal dimensions. Given the intermediate token ex E (N++)XD from block u, the token in the next block is computed as follows:
a-1
= M. (LN (e")) +
time
a-1 space
MSAspoce (LN (elttle + Ctime!
u+1
e=1 MLP (LN (espacc) + epptt
Espace
where MSA, LN and MLP denote the multi-head self-attention, layer normalization, and multi- layer perceptron, respectively. Each block utilizes MSA layer to project and divide e into ny parts. Each part contains the query Qr, key K, and value V,, for r 1, 2, ... Ri, where n, denotes the number of heads. We can get the attention map of the last layer of blocks by calculating the correlation between the query embedding of class token Qule and key embeddings of all other patches K. It is averaged for all heads as follows:
Vhg ThR Σ Softmar (Qela
Kr VD,
A
where Dú, = D/m,, and A is T, spatial attention maps. Although single-frame spatial attention integrates temporal information, still only considers the spatial information of the current frame, neglecting the global spatiotemporal dependencies in the video sequence. Thus, we aggregate the attention of T. frames to the mean value obtain a simplified and holistic attention distribution by:
To
Aagg
1
Σ A
T t=1
This attention mechanism is capable of obtaining an approximation of the critical region of the video that are being attended to from both the temporal and spatial dimensions, while reducing the impact of excessive variations in individual frames or regions. We will further exploit this attention dynamic to guide the generation of tube masking to help the model perform the reconstruction task more appropriately.
Visible tokens sampling and masking The traditional random masking strategy treats critical and non-critical areas of the video equally in each iteration, which may lead to excessive masking of key video regions at high masking ratio, thereby affecting the learning ability of the model. Therefore, we utilize a raame-aggeegated attention mechanism to guide the generation of tube mask. By sampling some reasonably visible tokens from the model-focused areas and masking the rest, it allows our method to efficiently perform the reconstruction task even at a high masking ratio, while improving pre-training efficiency. Specifically, we begin by ranking tokens in descending order of attention scores and subsequently select a proportion of high-aut high-attention tokens based on the threshold 7. We randomly sample visible tokens to form a binary mask from selected tokens of attention, depicted in Fig. [(b.. The number of sampled visible tokens N, is determined by the predefined mask ratio 1). Followed by SimMIM [16), we fill the mask tokens with learnable mask embeddings, which are subsequently fed together into the student network for feature mapping and finally into the prediction head for reconstruction.
In comparison to random sampling, which is inefficient token allocation, our method selects visible tokens based on the global spatiotemporal information of the video sequence. This approach
5

---

Table 1: Comparison with other latest SOTA methods on downstream tasks. We report Fl score (%) for PolypDiag, Dice (%) for CVC-12k, and F1 score (%) for KUMC, respectively.
Pretrain
PolypDiag
CVC-12k
KUMC
Method
Venue
Year
Time(h)
(Classificaton)
(Segmentation)
(Detection)
Scratch iRand.init.)
N/A
83.5 + 1.3
53.2
73.5 # 4.3
TimeSformer 73
ICML
2021
104.0
84.2
56.3
75.8 2.1
CORP 74
ICCV
2021
65,4
87.1 0.6
68.4 1.1
78.2 1.4
FAME 75
CVPR
2022
48.9
85.4 0.8
67.2 1.3
76.9 = 1.2
ProViCo 76
CVPR
2022
71.2
86.9
69.0 1.5
78.6 1.7
VCL 77
ECCV
2022
74,9
0.6
69.1 1.2
78.1 = 1.9
ST-Adapter
78
NeurIPS
2022
8.1
84.8
64.3
74.9 = 2.9
VidcoMAE 22
NeurlPS
2022
25.3
0.8
80.9 1.0
82.8 1.9
Endo-FM 30
MICCAI
2023
20.4
90.7
73.9 = 1.2
84.1 = 1.3
DropMAF
79
CVPR
2023
37.9
0.8
80.9 = 0.3
81.7 2.6
VideoMAE V2
23
CVPR
2023
17.3
1.4
81.0 + 0.4
84.2 1.0
M'CRL
Ours
24.3
94.2
81.4 = 0.8
86.3 + 0.8
minimizes redundant background area sampling, these backgrounds have minimal impact on the significance of mask reconstruction. Furthermore, the attention-guided mask generation is dynamic and allows for adjustments during model training. This adaptability enables the model to continuously optimize its focus, adapting to complex or changing data characteristics.
3.2.2 Random Tube Mask
Global view mask modeling primarily employed for comprehensive understanding and spa- tiotemporal contextual awareness of the entire endoscopic video frame, which mitigates the effects of inter-frame variability instability by aggregating frame attention. However, due to the low contrast between lesions and normal tissues in endoscopic videos, global views may struggle to accurately capture local details. Hence, we apply random tube mask reconstruction on local views to Iearn more granular detail information, shown in Fig. [c). Specifically,
obtain the local views {uf R x3x H. 1- (Ti Tg) by random cropping and uniform sampling at different frame rates. In local views, we also implement a high masking ratio م 90% to reduce information leakage during the mask modeling process. By local view mask modeling with different frame rates and spatial cropping, it is possible to make the model proficient in capturing variations in time scale and spatial detail. Moreover, local view mask modeling focuses on specific regions in the video without interference from the global background, enables more targeted learning. This allows the model to finely capture local information within the video, enhancing its ability to recognize subtle differences between abnormalities and normal tissues.
3.3 Multi-view Masked Contrastive Representation Learning
The pipeline of our proposed MCCRL is shown in Fig. aa), which introduces a multi-view masking strategy and combines multi-view mask modeling with contrastive learning to learn representations that possess fine-grained and discriminative capabilities simultaneously.
In DINO [55]. self-distillation is proposed not from the posterior distribution but by teacher-student scheme that extracts knowledge from the model's own past iterations. This self-distillation method of self-supervision is also considered form of contrastive learning [80]. The contrastive learning part of our C2RLL follows Endo-FM representation learning. Given an endoscopic video, two types of views are created under random data augmentation (G global views )= and L local views {w] }/21) The model is encoded by two encoders, a teacher network fa and student network fs, which are respectively parameterized by 0, and 0,. It should be noted that the two student networks depicted in Fig. [ a) actually represent
30, which also employs
distilllaaioo method achieve

---

Ortene
Groual Traf
Eade-PAI
Deap
Figure 2:
Qualitative results of segmentation and detection tasks. The segmentation results on the left from the CVC-12K dataset, while the detection results on the right are from the KUMC dataset.
single student network. We illustrate two student networks in the figure to more clearly convey the data flow. During pre-training, the global views are input into both the teacher and student networks, while the local views are only imput into the student network. The network output f is normalized by a softmax function with a temperature T to obtain the probability distribution p softmax( T). Subsequently, the cross-entropy loss function is used to compute the losses between the teacher's ' global views and the student's s global views, as well as between the teacher's s global views and the student's local views. The specific loss functions are as follows:
Σ
P
log
kzi
(S)
G
1
C6
= Σ
Σ
P
log p
1=
Contrastive learning utilizes class tokens [55] from global and local views to calculate matching losses, which offers strong capabilities for overall discriminative representation. However, this method overlooks the dense pixel dependencies that are crucial for dense prediction tasks like segmentation and detection. Therefore, we integrate the multi-view mask reconstruction pre-training task into the contrastive learning. In the multi-view mask modeling task, the video clips processed through masking are fed into the encoder to be mapped into the feature space, Subsequently, the prediction head utilizes the context of unmasked patches regress the dense pixel intensities within masked patches. The losses for global and local view reconstruction are as follows:
N
N
MCCRL exploits both contrastive loss and reconstruction loss in optimization, and the total loss Liotal L + CeL - C + L MM The student network updates the student parameters A, by minimizing Ltotat, and the teacher network 0g is updated using the exponential moving average (EMA) of the student weights, and 02 + X0, + (1 - A) 0,. Here À denotes the momentum coefficient. By combining contrastive learning and masked video modeling, the model encouraged learn the holistic discriminative representation and detailed pixel information, effectively improving the learning ability of the model in complex visual data of endoscopic videos.
4 Experiments
4.1 Datasets and Experimental Settings
We conduct experiments on 10 publicly available endoscopic video datasets: Colonoscopic [8]
SUN-SEG [82). LDPolypVideo (83), Hlyper-Kvaste [84). EEvasir-Cpssule [85]. Cholec Triplet [86].
Renji-Hospital 30, PolypDiag (27, CVC-12K 288 and KUMC 29] These datasets have a total of 33,231 videos with approximately 5,500,000 frames, covering 3 types of endoscopy examination
protocols and 10+ different diseases. The videos are processed into 30fps clips with an average
duration of 5 seconds. The first 7 datascts are used for pre-training and we sample G 2 global views
7

---

and L = 8 local views, where T, E [8. 16] 7i E [2, 4.8. 8, 16] and the spatial size set to 224 x 224 and 96 96, respectively. We take VIT-B/16 87 the backbone and perform 30 epochs of pre-training. In downstream tasks, we perform classification task on PolypDiag, segmentation task on CVC-12K, and detection task on KUMC, respectively. Our implementation is based on Endo-FM 30 and more experimental details can be found in 8 A
4.2 Comparison with Prior Work
We compare our method with the recent tfue-oof-thearr (SOTA) endoscopic video pre- pre-training model, Endo-FM [30], which is the first pre-training model on a large scale across various endoscopic videos. The results of other methods are taken from the comparative method of Endo-FM, including TimeS- former 73 CORP 74). FAME 75 VroViCo (76.. VCL [77]. and ST-Adapter (78). Additionally,
we also compare the latest video self-supervised methods: VideoMAE 222 VideoMAE V2 (33 and
DropMAE [79]. For a fair comparison, all methods are pretrained on the same union of 7 datasets as
our MRRL, All experimental settings are referred to those documented the original papers or in the released code.
Quantitative evaluation We observe that our method outperforms existing state-of-the-art methods, as shown in Table 1 Particularly, compared to Endo-FM [30] our M achieves improvements of 3.5% Fl in classification (PolypDiag). 7.5% Dice in segmentation (C)C-12k), and 2.2% Fl in detection (KUMC) tasks. These improvements are attributed to our pre-training approach, which inte- grates multi-view mask modeling with contrastive method, substantially enhancing the model's 's ability for representation learning. Clearly, compared to other methods (TimeSformer (73), CORP (74), FAME (75), etc.), M achieves considerable advantages across three downstream tasks. Al- though the performance gains extremely dense task (ie.. segmentation) is modest compared to the latest video supervupe methods (i.e., VideoMAE [22), VideoMAE V2 (31, DropMAE (79), our MºCRL makes great progress on classification task and reaches 94.2%. This shows that M2CRL not only focuses on pixel details but also enhances discriminative capabilities.
Qualitative evaluation We visualize segmentation and detection results in Fig. 2 Compared to other methods, our MCRL demonstrates superior visual results in segmenting both large and small polyp regions st and 2nd rows on the left in Fig. 2 Despite potential issues such as blurry boundaries or lens glare caused by camera movement, M²CRL is still capable of accurately segmenting the target regions (3rd row on the left in Fig. 2 We attribute these results to our multi- view mask modeling, which encourages the model to learn more precise detail information from videos. Similarly, McCRL also exhibits good performance in detection tasks, especially in detecting small target regions (2nd row on the right in Fig. 2). Although there are some differences between the predictions of our method and the ground truth, we achieve high degree of overlap with the ground truth. This demonstrates that our MCRRL significantly improves the pre- pre-training capability for endoscopic videos. See S D foo more segmentation and detection visual comparison.
4.3 Ablation Studies
Multi-view mask Table SableZilluee illlusrates the impact of single-view mask and multi-view mask. All experiments use the same masking ratio. For the single-view, different masking strategies are employed for global and local views, respectively. The random tube mask (22) randomly samples masked tokens the 2D spatial domain and then extends these tokens along the temporal axis. The random mask 21 randomly masks tokens in the spatiotemporal domain. Our approach samples visible tokens by leveraging trgreeagg attention from global views, resulting in better results than above two strategies, particularly achieving 80.6% in segmentation task. Similarly, the random tube mask demonstrates some superiority on local views. However, from Table it can be observed that the performance of solely conducting mask modeling on the single-view is inferior to that on multi-views. In multi-view mask, is evident that employing the aggreagg attention guided tube mask on global views and the random tube mask on local views results in significant performance gains. These two complementary mask methods work together to learn richer details of video features.
Hyper-parameters of the FAGTM We conduct an experiment to investigate the impact of the hyperparameters of the framo-aggregated attention guided tube mask (FAGTM) for global views,
8

---

Table 2: Multi-view mask. We compare multiple different masking strategies on different views.
FAGTM = Frame-aggregan Attention Guided Tube Mask. RTM = Random Tube Mask.
mask strategies
views
cla.
seg.
det.
gloabl
local
random
90.2 1.5
78.6 1.6
83.8 1.9
RTM
93.0 0.8
7.1
84.0 1.0
single-view
FAGTM
92.7 0.4
0.5
84.4 1.4
random
91.1 + 0.7
76.1
83.7 0.7
RTM
91.1 0.5
0.6
0.4
random
random
91.3 + 0.3
77.7
84.9 1.0
multi-view
RTM
RTM
93.2 0.4
80.2
85.2 1.3
FAGTM
RTM
94.2 0.7
81.4 0.8
86.3 0.8
Table 3: Hyper-parammttrs of the FAGTM.
Table 4:
The teacher's block of the FAGTM.
7
cla.
seg.
det.
blocks
cla.
seg.
det.
0.5
0.5
80.7 0.7
84.8 0.4
4
91.8 0.7
76.6 1.5
83.6 1.1
0.6
94.2 0.7
81.4 0.8
86.3 0.8
8
92.5 = 0.4
79.7 -9.7
84.8 1.0
0.7
93.7
8..0
85.9 2.6
10
93.9 1.0
80.6 0.9
85.9 1.7
0.8
92.9
79.0 0.3
85.4 0.8
12
94.2 # 0.7
81.4 0.8
86.3 0.8
the results are shown in Table 3 We perform experiments by sorting each patch of the obtained frame-aggregated attention maps in descending order and selecting the top y proportion of patches as candidate mask patches. Subsequently, visible tokens random sampled in candidate mask patches. From Table 3] we can observe that the model performs best when Y is set to 0.6. A lower value indicates selecting visible patches from smaller high-attention regions, leading excessive attention on non-critical areas during reconstruction, contradicting the setup of the self-supervised pre-text task. On the other hand, higher values of adversely affecting the learning efficacy of the model.
The teacher's block used for the FAGTM In our study, use the last layer block of the teacher ViT-B for the FAGTM. The higher layer block incorporates lower-level features and object-level semantic information, offering comprehensive and abstract features that are essential for the model. Thus, it effectively guides the student network in masking. Table(4shoows that it is most beneficial for the FAGTM to utilize the last layer block of the teacher network.
Masking ratio The impact of different masking ratios is illustrated in Table It can be observed that there an improvement in results across three downstream tasks when the masking ratio increases from 75% 90%. Due to the reundancy in videos, it shows that the 75% masking ratio utilized in ImageMAE is not suitable for videos. When the masking ratio in videos reaches 90%, the task becomes challenging due the limited number of patches available for learning temporal correspondence. thus enhancing the learning capacity of the model. This observation is also validated in VideoMAE [22] Compared to the optimal masking ratio, higher masking ratio increases the difficulty of pre- trainingg hindering the model's 's ability to learn effective representations. Although our ablation experiments have shown that a masking ratio 95% can achieve comparable performance, its effectiveness in three downstream tasks is lower than that of the 90% masking ratio. This suggests that masking ratio of 95%, the model is placed relatively unfavorable learning situation, resulting in suboptimal results.
Analysis of components As see from the first row in Table 6, although the performance of the contrastive learning framework on classification tasks is acceptable, its performance on pixel-Ievel tasks, especially the segmentation task, is not very good. Similarly, within the single mask modeling

---

Table 5:
Masking ratio.
We choose masking ratio of 90% for the FAGTM and RTM.
FAGTM (global)
RTM (local)
cla.
seg.
det.
95%
94.0 0.3
81.3 + 0.4
85.1 1.1
90%
93.8 0.9
80.7 + 0.7
86.2 1.3
95%
85%
0.7
78.5 + 0.6
84.9
75%
92.6 0.4
77.4 1.7
2.1
95%
93.8 + 1.4
80.5 + 0.7
85.8 # 0.9
90%
94.2 0.7
81.4 0.8
86.3 # 0.8
90%
85%
13.80..
81.4 1.7
85.6 + 2.2
75%
9.9
78.5 1.9
84,8 + 1.3
95%
93.2
79.9 0.4
83.1 + 1.5
90%
9
0.2
83.8 + 0.9
85%
85%
94.0 0.4
80.5 1.0
85.1 1.8
75%
92.5 1.2
7.76
8.5
95%
91.7
76.8 + 1.8
84.2 2.1
90%
0.3
7.2
84.0 1.3
75%
85%
9.2
1.9
83.8 0.4
75%
91.2
1.4
85.0 0.8
self-supervised framework, there is an improvement in performance for the pixel-level task (e.g., from 73.9% 80.5% for segmentation), while there is significant drop in performance for the discriminative task (e.g., from 90.7% to 85.7% for classification). These interesting phenomena once again highlights the strong instance discrimination ability of contrastive learning and the robust ability of mask modeling to acquire local pixel-level information. Our approach integrates both methods and demonstrates consistently strong performance across both structural and pixel-level tasks.
Table 6: Components analysis. Our proposed MCCRL combines masked video modeling with contrastive learning and has the best performance.
contrastive learning masked video modeling
cla.
seg.
det.
90.7 = 0.4
73.9 1.2
84.1 + 1.3
85.7 = 0.4
80.5 # 1.2
83.5 # 3.7
94.2 + 0.7
81.4 + 0.8
86.3 + 0.8
5 Conclusion
In this study, we present a novel SSL approach called MCCRL, which integrates multi-view mask modeling with contrastive learning for endoscopic video analysis. Our method aims to address the lack of pixel-level information extraction existing SSL methods for endoscopic videos. We leverage the unique characteristics of endoscopy to propose frame-aggregated attention guided tube mask that captures pixel-level relationships across spatial-temporal dimensions for global views. Additionally, utilize random tube mask to complement the details of local features from local views. Notably, the attention guidance is derived from multi-head self-attention maps extracted from teacher model, without incurring additional computational costs. By integrating contrastive learning, our method not only maintains performance in dense prediction tasks but also ensures effectiveness in discriminative tasks. The experimental results on 10 publicly avaiable datasets demonstrate that our MCCRL outperforms the state-of-the-at methods across multiple downstream vision tasks (i.e., classification, segmentation, and detection).
10

---

Acknowledgments
This work was supported in part by the National Natural Science Foundation of China under Grants 62272404 and 62372170, in part by the Natural Science Foundation of Hunan Province of China under Grants 2022JJ30571 and 2023JJ40638, and in part by the Research Foundation of Education Department of Hunan Province of China under Grant 23A0146.
References
[1] Mon Thiri Myaing, Daniel MacDonald, and Xingde Li. Fiber-optic scanning two-photon fluorescence endoscope. Optics Letters, 31(8):1076-100 1078, 2006.
[2] Yuan Zhang, Yanglin Huang, and Kai Hu. Multi-scale object equalization learning network for intracerebral hemorrhage region segmentation. Neural Networks, 109:006,77 2024.
[3] Kai Hu, Xiang Zhang, Dongjin Lee, Dapeng Xiong, Yuan Zhang, and Xieping Gao. Boundary- guided and region-aware network with global scale-adaptive for accurate segmentation of breast tumors in ultrasound images. IEEE Journal of Biomedical and Health Informatics, 27(9):4421-4422, 2023.
[4] Tongtong Liu, Xiongjun Ye, Kai Hu, Dapeng Xiong, Yuan Zhang, Xuanya Li, and Xieping Gao. Polyp segmentation with distraction separation. Expert Systems with Applications, 228: 120434, 2023.
[5] Atharva Tendle. "the revolution will not be supervised": An investigation of the efficacy and reasoning process self-supervised representations. Revolution, 2021.
[6] Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey on contrastive self-supervised learning. Technologies, 9(1):2, 2020.
[7] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in Neural Information Pprocessing Systems, 32, 2019.
[8] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1007.00 2018.
[9] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, pages 1422-1430, 2015.
[10] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pages 69-84. Springer, 2016.
[11] Richard Zhang. Phillip Isola, and Alexei A Efros. Colorful image colorization. In Computer Vision-ECCH 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14, pages 649-666. Springer, 2016.
[12] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint I 880307778, 2018.
[13] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image trans- formers. arXiv preprint 1.Xiv:210600825:: 2021.
[14] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,, pages 16000-16009, 2022.
[15] Lang Huang, Shan You, Mingkai Zheng, Fei Wang, Chen Qian, and Toshihiko Yamasaki. Green hierarchical vision transformer for masked image modeling. Advances in Neural Information Processing Systems, 35:19997-20010 2022.

---

[16] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao. Qi Dai, and Han Hu. Simmim: simple framework for masked image modeling. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9653-9665 2022.
[17] Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self-supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering, 35(1) ):857-876, 2021.
[18] Geoffrey Hinton and Richard Zemel. Autoencoders, minimum description length and helmholtz free energy. Advances in Neural Information Processing Systems, 6. 1993.
[19] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.61 14, 2013.
[20] Jie Gui, Zhenan Sun, Yonggang Wen, Dacheng Tao, and Jieping Ye. A review on generative adversarial networks: Algorithms, theory, and applications. IEEE Transactions on Kowledge and Data Engineering, 35(4):3313-3333 2021.
[21] Christoph Feichtenhofer, Yanghao Li, Kaiming He, al. Masked autoencoders spatiotempo- ral learners. Advances in Neural Information Processing Systems, 35:35946-35958, 2022.
[22] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in Neural Information Processing Systems, 35:10078
2022.
[23] Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14549-14560, 2023.
[24] Zhaowen Li, Zhiyang Chen, Fan Yang, Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng, Liwei Wu, Rui Zhao, Ming Tang, et al. Mst: Masked self-supervised transformer for visual representation. Advances in Neural Information Processing Systems, 34:13165-133 2021.
[25] Haochen Wang, Kaiyou Song, Junsong Fan, Yuxi Wang, Jin Xie, and Zhaoxiang Zhang. Hard patches mining for masked image modeling, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10375-10385, 2023.
[26] Zhengqi Liu, Jie Gui, and Hao Luo. Good helper is around you: Attention masked image modeling. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 7799,1807, 2023.
[27] Yu Tian, Guansong Pang, Fengbei Liu, Yuyuan Liu, Chong Wang, Yuanhong Chen, Johan Verjans, and Gustavo Carneiro. Contrastive transformer-bast based multiple instance learning for weakly supervised polyp frame detection. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 88 39-9 Springer, 2022.
[28] Jorge Bernal, Javier Sánchez, Gloria Gernández-Esparradee Debora Gil, Cristina Rodríguez, and Fernando Vilariño. Wm-dova maps for accurate polyp highlighting in colonoscopy: Val- idation vs. saliency maps from physicians. Computerized Medical Imaging and Graphies, 43:99-111, 2015
[29] Kaidong Li, Mohammad I Fathan, Krushi Patel, Tianxiao Zhang, Cuncong Zhong, Ajay Bansal, Amit Rastogi, Jean S Wang, and Guanghui Wang. Colonoscopy polyp detection and classification: Dataset creation and comparative evaluations. Plos One, 16(8)=00255809 2021.
[30] Zhao Wang, Chang Liu, Shaoting Zhang, and Qi Dou. Foundation model for endoscopy video analysis via large-scale self-supervised pre-train. In International Conference Medical Image Computing and Computer-Assisted Intervention, pages 101-111. Spriner, 2023.
[31] Unaiza Ahsan, Rishi Madhok, and Irfan Essa. Video jigsaw: Unsupervised learning of spatiotem- poral context for video action recognition. 2019 IEEE Winter Conference on Applications of Computer Vision, pages 179-189. IEEE, 2019.
12

---

[32] Yuqi Huo, Mingyu Ding. Haoyu Lu, Zhiwu Lu, Tao Xiang, Ji-Rong Wen, Ziyuan Huang, Jianwen Jiang, Shiwei Zhang, Mingqian Tang, al. Self-supervised video representation learning with constrained spatiotemporal jigsaw. In International Joint Conference Artificial Intelligence, pages 751-757, 2020.
[33] Dahun Kim, Donghyeon Cho, and In So Kweon. Self-supervised video representation learning with space-time cubic puzzles. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8545-8552, 2019.
[34] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using Istms. In International Conference on Muchine Learning, pages 843-852. PMLR, 2015.
[35] Jacob Walker, Carl Doersch, Abhinav Gupta, and Martial Hebert. An uncertain future: Fore- casting from static images using variational autoencoders. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11 1-14, 2016, Proceedings, Part VII 14, pages 835-851. Springer, 2016.
[36] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Anticipating visual representations from unlabeled video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 98-106, 2016.
[37] Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. arXiv preprint
2015.
[38] William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. arXiv preprint arXiv: IrXiv:1605... 00000, 2016.
[39] Agrim Gupta, Stephen Tian, Yunzhi Zhang, Jiajun Wu, Roberto Martín-Martin, and Li Fei-Fei. Maskvit: Masked visual pre-training for video prediction. arXiv preprint arXiv:2206.11 11894 2022.
[40] Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy, Tracking emerges by colorizing videos. In Proceedings of the European Conference on Com- puter Vision, pages 391-408, 2018.
[41] Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14, pages 527-544. Springer, 2016.
[42] Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould. Self-supervised video representation learning with odd-one-out networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3636-3645 2017.
[43] Hsin- Hsin-Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Unsupervised represen- tation learning by sorting sequences. In Proceedings of the IEEE International Conference on Computer Vision, pages 667-676,77 2017.
[44] Donglai Wei, Joseph J Lim, Andrew Zisserman, and William T Freeman. Learning and using the arrow of time. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages
2018
[45] Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-supervised spatiotemporal learning via video clip order prediction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10334-10343, 2019.
[46] Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan. Learning features by watching objects move. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2701-2710. 2017.
[47] Pulkit Agrawal, Joao Carreira, and Jitendra Malik. Learning to see by moving. Proceedings of the IEEE International Conference on Computer Vision, Pases 37-45, 2015.
13

---

[48] Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In Proceedings of the IEEE International Conference on Computer Vision, pages 2794-2802, 2015.
[49] Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle- consistency of time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2566-2576, 2019.
[50] Laurenz Wiskott and Terrence J Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural Computation, 14(4):715-770, 2002.
[51] Ross Goroshin, Joan Bruna, Jonathan Tompson, David Eigen, and Yann LeCun. Unsupervised learning of spatiotemporally coherent metrics. In Proceedings of the IEEE International Conference on Computer Vision, pages 4086-4093, 2015.
[52] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3733-3742, 2018.
[53] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9738, 2020.
[54] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pages 1597-1607. PMLR, 2020.
[55] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650-9660, 2021.
[56] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758, 2021.
[57] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 33:21271-2111111 2020.
[58] Chen Wei, Haoqi Fan, Saining Xie, Chao- Yuan Wu, Alan Yuille, and Christoph Feichten- hofer. Masked feature prediction for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14668-14678, 2022.
[59] Ioannis Kakogeorgiou, Spyros Gidaris, Bill Psomas, Yannis Avrithis, Andrei Bursuc, Konstanti- nos Karantzalos, and Nikos Komodakis. What to hide from your students: ovention-guides masked image modeling. In European Conference on Computer Vision, pages 300-318. Springer, 2022.
[60] David Fan, Jue Wang, Shuai Liao, Yi Zhu, Vimal Bhat, Hector Santos- Villalobos, Rohith MV, and Xinyu Li. Motion-guided masking for spatiotemporal representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5619-5629, 2023.
[61] Wele Gedara Chaminda Bandara, Naman Patel, Ali Gholami, Mehdi Nikkhah, Motilal Agrawal, and Vishal M Patel. Adamae: Adaptive masking for efficient spatiotemporal learning with masked autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14507-14517. 2023.
[62] Sunil Hwang, Jaehong Yoon, Youngwan Lee, and Sung Hwang. Efficient video representation learning via motion-aware token selection. arXiv preprint arXiv:2211.10636, 2022.
14

---

[63] Bingkun Huang, Zhiyu Zhao. Guozhen Zhang, Yu Qiao, and Limin Wang. Mgmae: Motion guided masking for video masked autoencoding. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 13493-13504, 2023.
[64] Yotam Intrator, Natalie Aizenberg, Amir Livne, Ehud Rivlin, and Roman Goldenberg. Self- supervised polyp re-identification in colonoscopy. In International Conference on Medical Image Computing and Compuler-Assisted Intervention, pages 590-600. Springer, 2023.
[65] Roy Hirsch, Mathilde Caron, Regev Cohen, Amir Livne, Ron Shapiro, Tomer Golany, Roman Goldenberg, Daniel Freedman, and Ehud Rivlin. Self-supervised learning for endoscopic video analysis. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 569-578. Springer, 2023.
[66] Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810004805. 2018.
[67] Kelvin Guu, Kenton Lec, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International Conference on Machine Learning, pages 3929-3938. PMLR, 2020.
[68] Kaitao Song. Xu Tan, Tao Qin, Jianfeng Lu, and Tic- Yan Liu. Mass: Masked sequence to sequence pre- pre-training for language generation. arXiv preprint PrXiv:1905.024555 2019.
[69] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. Advances in Neural Information Processing Systems, 33:16857-16867, 2020.
[70] Yuan Liu, Songyang Zhang, Jiacheng Chen, Kai Chen, and Dahua Lin. Pixmim: Rethinking pixel reconstruction in masked image modeling. arXiv preprint arXiv:2303.02416, 2023.
[71] Xue H, Gao P, and other. Stare at what you see: Masked image modeling without reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22732-22741 2023.
[72] Yunjie Tian, Lingxi Xie, Zhaozhi Wang, Longhui Wei, Xiaopeng Zhang, Jianbin Jiao, Yaowei Wang, Qi Tian, and Qixiang Ye. Integrally pre-trained transformer pyramid networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18610-18620, 2023.
[73] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In International Conference on Machine Learning, volume 2, page 4, 2021.
[74] Kai Hu, Jie Shao, Yuan Liu, Bhiksha Raj, Marios Savvides, and Zhiqiang Shen. Contrast and order representations for video self-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7939-7949, 2021.
[75] Shuangrui Ding, Maomao Li, Tianyu Yang, Rui Qian, Haohang Xu, Qingyi Chen, Jue Wang, and Hongkai Xiong. Motion-aware contrastive video representation learning via foreground- background merging. In Proceedings of the IEEE/CVE Conference on Computer Vision and 2022.
Pattern Recognition, pages
[76] Jungin Park, Jiyoung Lee, Ig-Jae Kim, and Kwanghoon Sohn. Probabilistic representations for video contrastive learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14711-1477 2022.
[77] Rui Qian, Shuangrui Ding, Xian Liu, and Dahua Lin. Static and dynamic concepts for self- supervised video representation learning. In European Conference on Computer Vision, pages 164. Springer, 2022.
[78] Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li. St-adapter: Parameter- efficient image-to-video transfer learning. Advances in Neural Information Processing Systems, 25:26462-26477,, 2022.
15

---

[79] Qiangqiang Wu, Tianyu Yang. Ziquan Liu, Baoyuan Wu, Ying Shan, and Antoni B Chan. Dropmae: Masked autoencoders with spatial-attention dropout for tracking tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 14561-14571, 2023.
[80] Xiaoyi Dong, Jianmin Bao, Yinglin Zheng, Ting Zhang, Dongdong Chen, Hao Yang. Ming Zeng, Weiming Zhang, Lu Yuan, Dong Chen, al. Maskelip: Masked self- ceff-distillation advances contrastive language-imsn pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10995-11005, 2023.
[81] Pablo Mesejo, Daniel Pizarro, Armand Abergel, Olivier Rouquette, Sylvain Beorchia, I Laurent Poincloux, and Adrien Bartoli. Computer-aided classification of gastrointestinal lesions in regular colonoscopy. IEEE Transactions on Medical Imaging, 35(9):2051-2063 2016.
[82] Ge-Peng Ji, Guobao Xiao, Yu-Cheng Chou, Deng-Ping Fan, Kai Zhao, Geng Chen, and Luc Van Gool. Video polyp segmentation: A deep learning perspective. Machine Intelligence Research, 19(6)531-5549 2022.
[83] Yiting Ma, Xuejin Chen, Kai Cheng, Yang Li, and Bin Sun. Ldpolypvideo benchmark: a large-scale colonoscopy video dataset of diverse polyps. In Medical Image Computing and Computer Assisted Intervention-MICCAl 2021: 24th International Conference, Strasbourg. France, September 27-October 1, 2021, Proceedings, Part V 24, pages 387-396. Springer, 2021.
[84] Hanna Borgli, Vajira Thambawita, Pia H Smedsrud, Steven Hicks, Debesh Jha, Sigrun L Eskeland, Kristin Ranheim Randel, Konstantin Pogorelov, Mathias Lux, Due Tien Dang Nguyen, al. Hyperkvasir, comprehensive multi-class image and video dataset for gastrointestinal endoscopy. Scientific Data, 7(1):283, 2020.
[85] Pia H Smedsrud, Vajira Thambawita, Steven A Hicks, Henrik Gjestang, Oda Olsen Nedrejord, Espen Næss, Hanna Borgli, Debesh Jha, Tor Jan Derek Berstad, Sigrun L Eskeland, et al. K Kvasir-capsule, a video capsule endoscopy dataset. Scientific Data, 8(1):142, 2021.
[86] Chinedu Innocent Nwoye, Tong Yu, Cristians Gonzalez, Barbara Seeliger, Pietro Mascagni, Didier Mutter, Jacques Marescaux, and Nicolas Padoy. Rendezvous: Attention mechanisms for the recognition of surgical action triplets in endoscopic videos. Medical Image Analysis, 78: 102433, 2022.
[87] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
[88] Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang. Huisheng Wang. Serge Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2021.
[89] Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.051001 2017.
[90] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.088 033983 2016.
[91] Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan Yuille, and Yuyin Zhou. Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.0433 100.0046.. 2021.
[92] Lingyun Wu, Zhiqiang Hu, Yuanfeng Ji, Ping Luo, and Shaoting Zhang. Multi-frame collabora- tion for effective endoscopic video polyp detection via spatial-temports feature transformation. In Medical Image Computing and Computer Assisted Antervention-MICCA 2021: 24th Inter- national Conference, Strashourg, France, September 27-October 1, 2021, Proceedings, Part V 24, pages 302-312. Springer, 2021.
16

---

[93] Andru P Twinanda, Sherif Shehata, Didier Mutter, Jacques Marescaux, Michel De Mathelin, and Nicolas Padoy. Endonet: deep architecture for recognition tasks on laparoscopic videos. IEEF Transactions on Medical Imaging, 36(1):86-977, 2016.
[94] Sanat Ramesh, Vinkle Srivastav, Deepak Alapatt, Tong Yu, Aditya Murali, Luca Sestini, Chinedu Innocent Nwoye, Idris Hamoud, Saurav Sharma, Antoine Fleurentin, et al. Dissect- ing self-supervised learning methods for surgical computer vision. Medical Image Analysis, 88:102844, 2023.
[95] Xueying Shi, Yueming Jin, Qi Dou, and Pheng-Ann Heng. Semi-supervised learning with progressive unlabeled data excavation for label-efficient surgical workflow recognition. Medical Image Analysis, 73: 102158, 2021.
17

---

Supplementary Material
In this supplementary material, we first provide model implementation details for reproducibility in Section A Next, we introduce additional ablation experiments Section In Section C
we
conduct another experiment surgical phase recognition in downstream tasks. In Section
visualize more segmentation and detection results in downstream tasks and perform qualitative analysis. Subsequently, provide the case study of frame- aagee-gggeeated attention guided tube mask in
Section E Finally, we analyze the limitations and broader impacts of our work in Section F
A Implementation Details
Pre-training We employ data augmentation techniques such as random horizontal flipping, color jitter, gaussian blur, and exposure adjustment, and also utilize temporally consistent spatial enhance- ments (88) to all frames within a single view. The decoder is a lightweight one-layer head [16] and the model performs learning using simple & loss. The FLOPs of our model for a single execution are approximately 190G, with a parameter count of 121M. Pre-training are conducted on 4 Tesla A 1100 GPUs. The training parameters shown in Table 7
Table 7: Pre-training settings.
config
value
optimizer
AdamW [89
optimizer momentum
31,82 = 0.9, 0.999
weight decay
4e
base learning rate
2e
learning rate schedule warmup epochs pretraining epochs batch size temperature parameters mask rate attention areas threshold momentum coefficient
cosine schedule 99)
10
30
12
Te = 0.04, 0.07
P = 0.9
7 = 0.6
0.996
Evaluation methodology For downstream fine-tuning, the procedure is as follows: (1) Classifi- cation: We use the PolypDiag (27) dataset, which includes 253 videos and 485,561 frames. Each video is annotated to indicate the presence or absence of lesion. The dataset divided into 71 normal videos without polyps and 102 abnormal videos with polyps for training, and 20 normal videos and 60 abnormal videos for testing. We sample 8 frames at resolution 224 x 224 from each video input, utilizing pre-trained model to initialize the backbone and appending randomly initialized linear layers for training 20 epochs. The SGD optimizer is employed, with the learning rate set 1e-3, momentum 0.9, and batch size 4. (2) Segmentation: We use the CVC-12k (28 dataset, which includes 29 videos and 612 frames, with 20 videos allocated for training and videos for testing. Each frame in the videos is annotated with ground truth masks (with single class) to identify the regions covered by polyps. We employ TnansUnet [91] as the segmentation decoder following the code of (9I]. The AdamW optimizer used optimize the overall parameters by setting the learning rate as le-4, weight deacy as 5e-2 and the batchsize as 1. We resize the spatial size as 224 x 224 and fine-tune for 150 epochs. (3) Detection: We use the KUMC (29 dataset, which includes 53 videos and 19,832 frames. Each frame in each video is annotated with bounding boxes and polyp categories, with 36 videos allocated for training and 17 videos for testing. We employ STFT 192 to implement the detection task, fine-tuning for 24k iterations at resolution of 640 640. The SGD optimizer used optimize the overall parameters by setting the learning rate as 2.5e-3, weight deacy le-4 and momentum 0.9. See the STFT [92] for more training details.
18

---

B Additional Ablations
Prediction target From Table it is observed that M achieves better performance when using RGB pixel values as the reconstruction target. The model, which utilizes feature distillation the prediction target, demonstrates decent results in classification tasks but shows decline in performance in segmentation and detection tasks. This once again highlights the significant advantage of incorporating reconstruction task involving masked patches for pixel-level tasks.
Table 8: Prediction target.
The effect of pixel regression is better. cla.
prediction target
seg.
det.
feature distillation
94.2 # 0.4
77.9 # 1.6
84.5 + 1.0
pixel regression
94.2 0.7
81.4 0.8
86.3 0.8
Ablations on loss The mask modeling component of our model follows SimMIM [16], which employs the & loss function. The study demonstrates that different loss functions have minimal impact. To maintain consistency, we use the same loss function as SimMIM. Furthermore, we conduct ablation studies demonstrate that different loss functions have negligible effect our results, as shown in Table 9
Table 9:
Ablations on loss.
loss
cla.
seg.
det.
94.2 = 0.7
81.4 = 0.8
86.3 + 0.8
93.8
0.7
82.0 = 0.7
85.9 + 1.5
Ablation on different architectures For fair comparison, we use the weights for initialization as Endo-FM did. However, since this work does not provide weights for ViT variants, are unable to conduct ablation experiments on different architectures with weight initialization. Consequently, conduct set of ablation experiments without weight initialization for the backbone, shown in Table [O] We observe that the performance improvement is more pronounced with larger models due to their increased parameters and more complex structures, enabling them to capture more intricate features. In contrast, smaller models have limited feature extraction capabilities and cannot fully extract visual features. Although larger models exhibit stronger learning abilities, they are more prone to overfitting during training. Additionally, larger models require higher computational resources and longer training times. In conclusion, choosing ViT-B the pre-trained backbone is suitable compromise.
Table 10:
Ablation on different architectures.
backbone
cla.
seg.
det.
ViT-T/16
93.4
76.8 1.2
76.3 + 2.4
VYT-S/16
1.5 9.4
79.4 0.7
VIT-B/16
83.4 0.5 0.94
VIT-L/16
9.9
8.8
84.2 + 2.0
Table 11: Ablation on initialization status.
initialization status
cla.
seg.
det.
random
93.4 + 0.9
80.5 + 0.5
83.4 + 2.8
kinetics weights
94.2 + 0.7
81.4 + 0.8
86.3 + 0.8
We perform ablation experiments on the model initialization status. As shown in Table [1] the results indicate that M without weight initialization performs slightly worse under the same pre-training conditions. This is because weight initialization accelerates model convergence and enhances model stability. However, for fair comparison, we follow Endo-FM and use initialized weights.
Ablation on initialization status
19

---

C Surgical phase recognition
In the downstream tasks, conduct surgical phase recognition experiments using the Cholec80 93 dataset, which contains 80 complete laparoscopic cholecystectomy vidcos. This dataset is specifically designed evaluate the performance of the model in automatic surgical phase recognition. To ensure a fair comparison with existing methods, we follow the data split and evaluation protocol described by [94], using 40 videos for training and 40 for testing. For evaluation, boundary F1 score proposed by [95]. Since our backbone is based on Transformers, we do not use the CNN-based TCN (94] as feature extractor during fine-tuning, so our result is based on single frames. As shown in Table 22 our method achieves superior performance. This experiment further verifies the robustness and effectiveness of our method across various endoscopic video tasks.
adopt the relaxed
Table 12: Surgical phase recognition.
Methods
F1
DINO
77.6
MoCo v2
81.7
SimCLR
84.5
SwAV
86.1
Endo-FM 87.5
MºCRL 88.7
D Qualitative Evaluation
Fig. illustrates the segmentation results of our method and other self-supervised pre-training methods the CVC-126 12k dataset. Polyps are characterized by their blurry boundaries and significant shape variations, which undoubtedly add complexity to the segmentation task. Nevertheless, our method consistently outperforms other state-of-the-art art self-supervised methods. Specifically, when polyps are very close and overlapping (3rd row of Fig. 3). our method produces smoother and more continuous edges, whereas other methods often result in blurred or fragmented edges. Additionally, some small polyps t4th row of Fig. 3) occupy very few pixels and are easily missed, our method successfully segments these small polyps without omission. Fig. shows the detection results of our method and other self-super pre- pre-training methods on the KUMC dataset. As shown in the figure, method performs well in boundary identification and localization for small polyps (1*t and ndd rows of Fig. 4p. Furthermore, 3rd and 4th rows of Fig. 4 that our method can also accurately identity and locate polyps with surrounding tissues. Notably, compared to Endo-FM, discriminative approach, our method significantly enhances the performance on pixel-level tasks.
when they have low contrast and indistinct boundaries
Original
Ground Truth
MÉCRL
Endo-FF
VideOMAE
VileoMAE V2
DropMAE
Figure 3: Qualitative results for segmentation task the CVC-12k dataset.
20

---

Original
Truth
MÉCRL
DropMAR
Figure 4: Qualitative results for detection task on the KUMC dataset.
T
U'ame-angregtte light
Sumpling
Figure 5: Illustration of our frame-aggregated attention guided tube masking strategy. We visualize spatial attention map with temporal information for each frame (2nd column), then aggregate attention maps for all frames and select area of high attention. We sample visible patches in this area (3rd column).
E
Case Study
In Fig. 5 we visualize the attention maps of the ViT-B/16 model employing "the divided space-time attention" mechanism. We utilize the [cls] token as query and employ attention from the last transformer block. From the attention map of individual frames, we observe that the model does not distinctly learn the concept endoscope object boundaries, attributed to the separate computations of attention in time and space. To enhance object concepts within endoscope video sequences, we aggregate attention across all frames of the video and then select certain proportion of high- -igh-attention regions. This approach ensures that even in complex endoscopic video scenarios, key information within the video sequence is retained. Random masking critical regions, thereby impeding the ability of the model to learn video representations. To address this issue, we select regions of high attention from the aggregated attention map for sampling visible tokens.
high ratio in video can obscure
21

---

Limitations and Broader Impacts
Limitations Our work presents multi-view masked contrastive representation learning (M²CRL) framework for endoscopic video analysis. However, our current self-supervised pre-training method only utilizes RGB video streams and does not incorporate additional audio and text streams, In the future, we expect that audio and text data can provide more information for self-supervised pre-training. Furthermore, our study requires extensive pre-training, leading to significant energy consumption and reliance high-performance computing hardware (GPU). These negative impacts underscore the necessity of considering environmental protection and resource conservation, In future work, will adopt more efficient training methods and optimization strategies address these issues.
Broader impacts Our approach demonstrates the potential of SSL in endoscopic video analysis. By utilizing large amount of unlabeled endoscopic video data for pre-training, we can reduce the dependence costly annotated medical data, thus lowering healthcare expenses. Furthermore, pre-trained models can be easily applied to various tasks such as classification, segmentation, and detection, thereby making valuable contributions to medical applications and enhancing the quality and efficiency of clinical disease diagnosis and healthcare services.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims presented in abstract and introduction accurately reflect the contributions and scope the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of paper in
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
23

---

Justification:
We introduce our method in detail in Section
B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We present the information needed reproduce the main experimental results of the paper in Section 4 and
A
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification: We open-source the code. We do not need to release any data, as we use publicly available datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
wttgggiddde/eeSSuuunnnnn ssicnpolicy) for more details. hetpolliicggiddeeCoCoooo
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section and 5 A of our paper present all the training and test details necessary to understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: The reported result is in mean + std format, upon 3 independent runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
25

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We show the time required to run the experiment in Table and the computing resources in AA
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted our work conforms all respects to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts of our paper in S
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
There are no such risks in our paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited all the relevant assets used in paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We use publicly available datasets, so this issue does not arise.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We use publicly available datasets, so this issue does not arise.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "V1veVqLxtX", "reviewer": "Reviewer_wGJR", "summary": "The paper proposes a pre-training approach called Multi-view Masked Contrastive Representation Learning for endoscopic videos. The approach combines self-distillation and masked video modeling under multi-view setting. To consider the characteristics of inter-frame instability and small inter-class differences of endoscopic videos, the paper introduces a frame aggregated attention guided tube masking strategy to capture global spatio-temporal representation and employs random tube masking on local views to capture local representations. The approach is pre-trained on seven endoscopic datasets and fine-tuned on three additional datasets. Experiments show that it outperforms the baselines on classification, segmentation and detection tasks.", "questions": "Please see the weakness section for the questions and suggestions.", "limitations": "Yes, the authors have adequately addressed the limitations.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is easy to read.\n- The paper shows the combination of self-distillation and masked video modeling for pre-training ViT-B model using endoscopic videos.\n- Frame-aggregated attention guided tube masking (FAGTM) to learn global spatio-temporal representation learning.\n- Experiments on multiple tasks to show the efficacy of the approach.", "weaknesses": "- Although the pre-training approach is proposed for endoscopic videos, the novelty is limited. It's a combination of self-distillation and masked video modeling. \n- FAGTM is also an extension of either [1] and [2] which propose attention guide masking strategy for image based pre-training. The paper merely aggregates the attention for all frames and use the mean of that to guide the masking.\n- The paper mentions in section 3.3, \"This self-distillation method of self-supervision is also considered a form of contrastive learning\". Can the authors please give remark on why self-distillation is a form of contrastive learning?\n- Which block of the teacher ViT-B is used for FAGTM? Ablation study would be great.\n- Did the author pre-train ViT-B from the scratch? or initialized from some weights?\n- Ablation on number of epochs during pre-training is missing\n- Only ViT-B is used in the experiment? Different architecture should be studied too.\n- Did the authors also pre-train VideoMAE and other baselines using your dataset? Can the authors show some results using kinetics pre-trained SSL weights? \n- More baselines should be compared with. For example, MME[3], AdaMAE[4], and other masked video modeling approach.\n- What did the author use as an evaluation or metric to stop pre-training?\n- Can FAGTM be used on local views?\n- The approach looks very sensitive to $\\gamma$. Table 3 shows the impact of it on all the tasks.\n- Is the masking ratio used for FAGTM and random tube masking same? Ablation study on the impact of different masking ratio for each of the masking strategy would be useful.\n- The paper only uses linear layer for the reconstruction objective. Study of different decoders would be helpful. In masked video modeling, most of the approaches used asymmetric encoder and decoder design. It would be great to pre-train baselines like VideoMAE with linear layer decoder for a better comparison.\n- Most of the recent approaches use L2 loss for the reconstruction objective. Comparison of L1 and L2 loss is missing and can the authors please give a remark on why L1 loss is preferred?\n- Given the approach is mostly empirical with limited novelty, the performance on other datasets like Cholec80, a surgical phase recognition benchmark dataset, would be great. \n\n- There are some typos in the paper: section 3.2.1 'spatiotempora' -> temporal, table2 'gloabl' -> global\n\n[1] What to Hide from Your Students: Attention-Guided Masked Image Modeling, ECCV 2022\n\n[2] Good helper is around you: Attention-driven Masked Image Modeling, AAAI 2023.\n\n[3] Masked Motion Encoding for Self-Supervised Video Representation Learning, CVPR 2023\n\n[4] AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoders, CVPR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your constructive and insightful comments. We address all weaknesses and questions below.\n\n**W1. The novelty.**\n\nExisting self-supervised pre-training methods for endoscopic videos predominantly rely on contrastive learning. However, using contrast learning alone is not sufficient to capture the fine-grained feature representations required for endoscopic videos. \n\n* To address the limitations of current methods, we have integrated contrastive learning with masked modeling to effectively acquire endoscopic video representations that possess both comprehensive discriminative capability and fine-grained perceptive ability.\n\n* Unlike the general approach of combining self-distillation and masking modeling, our method specifically designed a novel multi-view masking strategy for endoscopic video features, which significantly improved the model performance. This strategy involves utilizing the frame-aggregated attention guided tube mask to capture global-level spatiotemporal contextual relationships from the global views, while also employing the random tube mask to focus on local variations from the local views. \n\n* We conducted extensive experiments on 10 endoscopic video datasets to evaluate the performance of M$^2$CRL in comparison to other methods. Experimental results demonstrate the superiority of our method.\n\nThis work holds inherent value in clinical practice. The development of robust pretrained models for endoscopic video analysis through self-supervised learning can effectively support various downstream tasks, ultimately enhancing clinical workflow efficiency.\n\n**W2. Innovation in FAGTM.**\n\nUnlike [1] and [2], our work focuses on video rather than static images. Videos have a specific temporal dimension that should be considered in the mask design process to fully leverage spatiotemporal information. Furthermore, endoscopic videos present unique challenges due to their domain-specific characteristics. In this study, we propose a multi-view masking strategy tailored to endoscopic video features, which enhances the model’s robustness and effectiveness. \n\nSpecifically, to address the instability of inter-frame variations in endoscopic videos, we designed the FAGTM to capture global spatiotemporal representations. Traditional image-based attention mechanisms only focus on the spatial information of the current frame, neglecting dependencies across the entire video sequence. In contrast, our FAGTM aggregates the attention of frames to the mean value to obtain a simplified and holistic attention distribution. This attention distribution is capable of obtaining an approximation of the critical region of the video that are being attended to from both the temporal and spatial dimensions, while reducing the impact of excessive variations in individual frames or regions. Thus, our FAGTM provides more comprehensive and context-aware guidance for the masking process.\n\n**W9. Comparison with more methods.**\n\nAs you suggested, we have included comparisons with MME, AdaMAE, and UTM methods, as shown in Table 5. The results indicate that our method achieves the best performance in classification and detection tasks and performs moderately well in the segmentation task. This is primarily due to the fact that these mask modeling methods excel in capturing rich pixel-level information, making them particularly effective for dense pixel-level tasks such as segmentation. However, their performance tends to underperform in structural tasks that focus on global features and the holistic understanding of objects in images or videos. Our method integrates masked modeling with contrastive learning, which not only encourages the model to capture fine-grained pixel-level features but also compels it to learn comprehensive discriminative representations. This is friendly for both pixel-level and structured tasks, but it also increases the complexity of model pre-training. During the pre-training process, the model needs to carefully consider feature selection tradeoffs across multiple pre-tasks. This consideration may result in some downstream tasks exhibiting less prominent performance compared to single pre-task training. Overall, our method achieves robust performance across both structural and pixel-level tasks, highlighting its versatility and effectiveness in learning both detailed and global features.\n\nTable 5. Comparison with more methods\n\n| **Methods** |   **Cla.**   |   **Seg.**   |   **Det.**   |\n| :---------: | :----------: | :----------: | :----------: |\n|  MME [10]   |   92.3±1.0   |   81.9±0.7   |   84.8±0.9   |\n| AdaMAE [11] |   92.0±0.3   |   **82.3±0.4**   |   83.4±1.6   |\n|  UTM [12]   |   93.2±0.3   |   80.8±0.5   |   84.0±1.1   |\n|  M$^2$CRL   | **94.2±0.7** | 81.4±0.8 | **86.3±0.8** |\n\n**W16. The performance on other datasets like Cholec80.**\n\nBased on your suggestion, we have added the Cholec80 dataset in our downstream tasks. We compared our approach with the classical endoscopic foundation model Endo-FM [4] and the masked video modeling method VideoMAE [5] using F1-score in Table 9. The experimental results demonstrate that our approach achieves superior performance.\n\nTable 9. Results of surgical phase recognition\n\n| **Methods**  |  **Recog.**  |\n| :----------: | :----------: |\n| Endo-FM [4]  |   82.2±0.8   |\n| VideoMAE [5] |   73.7±1.4   |\n|   M$^2$CRL   | **85.0±0.4** |\n\n**Additional comment:** We would have liked to include the rest of answers to the questions mentioned by Reviewer wGJR (marked as W3, W4, etc..). Unfortunately, we did not have enough space in this rebuttal box. As soon as the discussion phase will begin, we will include the mentioned answers in an additional comment for the reviewer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HA3g9T0Qy7", "reviewer": "Reviewer_R5Z9", "summary": "The paper proposes a representation learning framework that combines masked pretraining strategies with contrastive learning methods. Particularly, this framework aims to generate a representation learning approach that can work with downstream tasks requiring dense pixel-level representations (for image segmentation) or discriminative features (for image classification). The masking strategy is guided by the aggregation of the attention layers of a teacher model over different frames of the endoscopic video, while the contrastive learning framework follows a self-distillation approach. Testing is performed on colonoscopic datasets and on three downstream tasks, including classification, detection, and segmentation.", "questions": "Most of the datasets, including the downstream tasks, are related to colonoscopy. Given the intention to develop a method to operate with endoscopy in general, it would be interesting to see the performance in downstream tasks that involve other endoscopic modalities. \n\nIt appears that the combination of masking and contrastive learning is more beneficial for the classification and detection tasks, while the improvements for the segmentation tasks are still at a similar level as previous works. Would it be possible to elaborate on why the segmentation tasks might not take significant benefits compared with the other tasks?", "limitations": "Limitations are discussed in the supplementary material of the paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* Representation learning in the medical domain can have an impact on the development of models for medical image analysis. \n\n* The model is validated against different works and on three downstream tasks. \n\n* Ablation experiments show the contribution of the components.", "weaknesses": "* Downstream tasks are evaluated on a single endoscopic modality\n\n* Similarly, a cross-validation like approach that keep the downstream tasks but change the testing dataset, could help to support the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your constructive and insightful comments. We address all weaknesses and questions below.\n\n**W1 and Q1. Downstream tasks are evaluated on a single endoscopic modality.**\n\nIn our study, we used multiple publicly available endoscopic video datasets. These datasets cover 3 types of endoscopic procedures (colonoscopy, gastroscopy, and laparoscopy) and 10+ different diseases. We believe that this comprehensive and large-scale dataset is valuable for endoscopic research (as detailed in Table 1). In downstream tasks, classification is for gastroscopy, while segmentation and detection are for colonoscopy.\n\nBased on your suggestion, we have added the Cholec80 dataset in downstream tasks. This dataset contains 80 complete laparoscopic cholecystectomy videos and is specifically designed for laparoscopic surgical recognition. Followed by previous works [1, 2], we used 40, 8, and 32 videos from Cholec80 as our training, validation, and test sets, respectively. We compared our approach with the classical endoscopic foundation model Endo-FM [3] and the masked video modeling method VideoMAE [4] using F1-score in Table 2. The experimental results demonstrate that our approach achieves superior performance. This experiment further validates the robustness and effectiveness of our approach across diverse endoscopic video tasks.\n\nTable 1. Details of pre-train and downstream datasets\n\n| **Phase**  |  **Dataset**   |  **Provider**  | **Videos** |      **Protocol**       |\n| :--------: | :------------: | :------------: | :--------: | :---------------------: |\n| Pre-train  |  Colonoscopic  |      CNRS      |    210     |       colonoscope       |\n|            |    SUN-SEG     |      ANU       |    1018    |       colonoscope       |\n|            |  LDPolypVideo  |      USTC      |    237     |       colonoscope       |\n|            |  Hyper-Kvasir  |     Simula     |    5704    |       gastroscope       |\n|            | Kvasir-Capsule |     Simula     |    1000    |       gastroscope       |\n|            | CholecTriplet  |     BIDMC      |    580     |       laparoscope       |\n|            | Renji-Hospital | Renji Hospital | 16494/7653 | colonoscope/gastroscope |\n| Downstream |   PolypDiag    |    Adelaide    |    253     |       gastroscope       |\n|            |    CVC-12k     |      UAB       |     29     |       colonoscope       |\n|            |      KUMC      |     Kansas     |     53     |       colonoscope       |\n|            |    Cholec80    |     CAMMA      |     80     |       laparoscope       |\n\nTable 2. Results of surgical phase recognition\n\n| **Methods**  |  **Recog.**  |\n| :----------: | :----------: |\n| Endo-FM [3]  |   82.2±0.8   |\n| VideoMAE [4] |   73.7±1.4   |\n|    M$^2$CRL     | **85.0±0.4** |\n\n**W2. A cross-validation like approach.**\n\nWe conducted five-fold cross-validation on the 3 downstream tasks, and the experimental results are presented in Table 3. We observe that the results of cross-validation are lower than those recorded in our original manuscript. We think that this difference is caused by the different ways of data division. For a fair comparison, we strictly followed the public partitioning of the datasets for conducting experiments in our study. Considering that there is no existing work to cross-validate this dataset, we also cross-validated the classical endoscopy foundation model Endo-FM [3] for comparison. The results show that our method outperforms Endo-FM in all three tasks.\n\nTable 3. Results of cross validation for 3 downstream tasks\n\n| **Methods** | **Cla.** | **Seg.** | **Det.** |\n| :---------: | :------: | :------: | :------: |\n|   Endo-FM   | 84.5±4.5 | 55.5±7.0 | 80.5±4.2 |\n|    M$^2$CRL    | 87.5±4.9 | 58.6±8.5 | 82.0±4.2 |\n\n**Q2. Why the segmentation tasks might not take significant benefits compared with the other tasks?**\n\nDue to varying requirements in different downstream tasks, the features acquired through different pre-training methods may exhibit distinct performances on specific downstream tasks. Previous studies, such as VideoMAE, DropMAE, and VideoMAE V2, have shown impressive performance in segmentation tasks due to their reliance on masked modeling. The strength of masked modeling lies in capturing rich pixel-level information, making these single-task pretraining approaches particularly effective for dense pixel-level tasks like segmentation. However, their performance tends to underperform in structural tasks that focus on global features and the holistic understanding of objects in images or videos. \n\nOur method integrates masked modeling with contrastive learning, which not only encourages the model to capture fine-grained pixel-level features but also compels it to learn comprehensive discriminative representations. During the pre-training process, the model needs to carefully consider feature selection tradeoffs across multiple pre-tasks. This consideration may result in some downstream tasks exhibiting less prominent performance compared to single pre-task training. Although the improvement of our method in segmentation tasks is not as significant as that using mask modeling techniques alone, our method still achieves a significant improvement in segmentation tasks compared to the method using only contrast learning. Overall, our method achieves robust performance across both structural and pixel-level tasks, highlighting its versatility and effectiveness in learning both detailed and global features.\n\n[1] Ramesh S, et al. Dissecting self-supervised learning methods for surgical computer vision. Medical Image Analysis, 2023.\n\n[2] Czempiel T, et al. Surgical phase recognition with multi-stage temporal convolutional networks. MICCAI 2020.\n\n[3] Wang Z, et al. Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-train. MICCAI 2023.\n\n[4] Tong Z, et al. VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. NeurIPS 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KRovkIDNdf", "reviewer": "Reviewer_BGWe", "summary": "This work presents M$^2$CRL, a self-supervised learning method for representation learning of endoscopic videos. The method leverages a multi-view masking technique with attention-guided masking of global features and random spatiotemporal tube masking of local features. Both contrastive learning and masked autoencoding pretraining objectives are employed for stronger representation learning. The model is pretrained on 7 publicly available endoscopy video datasets and fine-tuned on 3 other datasets on a variety of tasks, outperforming other relevant baselines.", "questions": "- Were all methods (perhaps except EndoFM) pretrained on the same union of 7 datasets as M$^2$CRL? Please clarify this in Section 4.2.\n- It appears that the same hyperparameters were used for all methods and datasets – is this true? If so, how were hyperparameters selected? Please include full fine-tuning hyperparmaters in Appendix A alongside the pretraining hyperparameters.\n- Several instances of awkward/confusing writing:\n  - The passage from L225-233 is confusing, e.g.: “It is worth noting that the two students in Fig. 1(a) share weights, but are actually one student network. Both encoders share the same structure.” Further, perhaps explain why only local views are fed to the student network.\n  - L267: “The results of other methods are taken from the comparative method of Endo-FM”. Is this saying these were the baselines considered in the EndoFM paper?\n\nMinor comments:\n- I would replace “mask strategy” with “masking strategy” throughout the paper.\n- It might be useful to explain in words what Equation 5 represents.\n- L275: I would refrain from saying “significant” without a statistical significance test.", "limitations": "Limitations are adequately addressed in Appendix E.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The organization of the paper is excellent, with helpful use of bold text, logical flow from one passage to the next, and high-quality, information-dense illustrations and tables.\n- The motivation is clearly laid out and the reference to prior related work is thorough.\n- The experiments are very thorough and appear to be soundly conducted. Results demonstrate notable improvement upon existing competitive baselines, and ablation studies help showcase which elements of M$^2$CRL are most impactful.", "weaknesses": "- Experimental details could be clarified. Were all methods pretrained on the same union of 7 datasets as M$^2$CRL? Was Endo-FM used as is for downstream fine-tuning or retrained on this pretraining dataset? Section 4.2 could clarify these details. Additionally, how were hyperparameters selected and were they the same for each model and task? Appendix A should additionally include fine-tuning hyperparmeters. \n- There are a few instances of confusing passages of writing. See examples in the section below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your constructive and insightful comments. We address all weaknesses and questions below.\n\n**Q1. Were all methods pretrained on the same union of 7 datasets as M$^2$CRL?**\n\nYes, for a fair comparison, all methods were pretrained on the same union of 7 datasets as our M$^2$CRL. We apologize for this and we will make careful revisions in subsequent versions.\n\n**Q2. Was Endo-FM used as is for downstream fine-tuning or retrained on this pretraining dataset?**\n\nEndo-FM [1] was originally pre-trained on this pretraining dataset, and its backbone was subsequently used for fine-tuning in downstream tasks. Endo-FM serves as our baseline, and the datasets we use are also provided and processed by it.\n\n**Q3. How were hyperparameters selected and were they the same for each model and task? Section 4.2 could clarify these details.**\n\nFor the other methods, we used the hyperparameters as documented in their original papers or provided in their code. Similarly, for the hyperparameters of downstream tasks, we followed the guidelines outlined by Endo-FM [1], which are comprehensively explained in their paper and open-source code. We pretrained all methods on the same union of 7 datasets and then loaded them into the backbone of downstream tasks for fine-tuning. We apologize for this unclear presentation and we will make careful revisions in subsequent versions.\n\n**Q4. Appendix A should additionally include fine-tuning hyperparameters.**\n\nThank you for your valuable suggestions. We will update Appendix A to include the fine-tuning hyperparameters as requested.\n\n**Q5. The passage from L225-233 is confusing.**\n\nWe propose a Multi-view Masked Contrastive Representation Learning framework. In the manuscript, L225-233 primarily introduce the contrastive representation learning component of our framework.\n\n**L225-229** describe self-distillation and explain the relationship between contrastive learning and self-distillation. It is clarified that the contrastive representation learning component in our framework is achieved through self-distillation.\n\n**L229-233** provide a description of the specific workflow of our framework, including data generation, model composition, and data flow within the model. Our model employs a teacher-student architecture, consisting of a student network and a teacher network, which consists of a student network and a teacher network sharing the same structure. It should be noted that the two student networks depicted in Fig. 1(a) actually represent a single student network. We illustrated two student networks in the figure to more clearly convey the data flow.\n\n**Q6. Explain why only local views are fed to the student network.**\n\nWe create two spatiotemporal views (global and local views) of the input endoscopic videos with varying spatial sizes and frame rates. The global views are fed into both the teacher and student networks, while the local views are only fed into the student network. Both the teacher and student networks process these views and predict one view from another in the latent feature space. This approach enables the model to learn spatiotemporal invariant features that can be transferred across different endoscopic domains and disease types. \n\n* **Cross-View Matching**: The global views are fed into the teacher network primarily because it provides stable and reliable outputs as references for the student network. These global views contain comprehensive information about the video, which assists the teacher network in generating accurate feature representations. The student network processes both the global and local views. By aligning the global features of the student network with those of the teacher network, the student network can learn rich global information. Furthermore, by aligning the local views of the student network with the global features of the teacher network, it can acquire more fine-grained feature representations. This cross-view matching approach involving global and local views facilitates a more comprehensive learning process for a rich feature representation within this model.\n\n* **Increased Learning Challenge**: Not feeding the local views to the teacher network introduces an information imbalance between the teacher and student networks. This design introduces a learning challenge for the model, as it requires the student network to exert more effort in extracting and integrating information from the local views to compensate for the lack of global information. Consequently, this approach enhances the contrastive learning process by necessitating that the student network maximize agreement between representations of different views despite their partial and incomplete nature. Ultimately, this design encourages the model to develop robust and discriminative features.\n\n**Q7. L267 Is this saying these were the baselines considered in the Endo-FM paper?**\n\nEndo-FM [1] compares several methods, including TimeSformer, CROP, FAME, ProViCo, VCL, and ST-Adapter. These methods are used for comparison rather than serving as baselines. The authors of Endo-FM pre-trained these methods on the same union of 7 datasets and then fine-tuned them on downstream tasks. Since we followed the same experimental procedure, we have directly utilized the recorded results from Endo-FM instead of re-running the experiments.\n\n**Q8. Minor comments:**\n\nThank you for your comments. We will replace the term “mask strategy” with “masking strategy” throughout the entire manuscript in subsequent versions. We will also provide a detailed textual explanation of what Equation (5) represents. Furthermore, we will remove “significant” from Line 275. We sincerely appreciate once again your valuable suggestions, which have greatly contributed to the improvement of our manuscript.\n\n[1] Wang Z, et al. Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-train. MICCAI 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rHA8BpPTXO", "reviewer": "Reviewer_u6Br", "summary": "The authors propose a self-supervised learning regime for spatio-temporal data called multi-view masked contrastive learning which combines a frame-aggregated attention guided tube mask and a multi-view mask strategy using a student-teacher framework. The learnt representations are evaluated on multiple downstream tasks on publicly available datasets.", "questions": "- How many classes do the datasets used for evaluations have in the case of classification and segmentation? What are they?\n- Are any modifications needed to employ this method to other spatio-temporal data?", "limitations": "-\tThe potential negative societal impact is not sufficiently discussed. In the broad impact statement, the author discuss the need and benefits for their method one more time. I urge them to consider potential *negative* impact on society.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tThe manuscript is well and clearly written, different aspects are well motivated and explained.\n-\tThe background section summarizes a lot of related literature and is easy to read. It gives a good overview of related work, particular challenges associated with endoscopy images and motivates the proposed approach.\n-\tThe conducted experiments are exhaustive. The learnt representations are tested in multiple downstream tasks for their usefulness (classification, segmentation, detection) on three different publicly available datasets, comparing to a large number of recent competing methods. The results are averaged over three runs and show consistent improvements in all three tasks. The study includes an ablation study to report on the effect of different components of the proposed methods.\n-\tThe method has been developed for application in endoscopy, but are relevant for spatio-temporal imaging in general and the type of pretraining is relevant for other medical imaging modalities with a temporal component.", "weaknesses": "-\tPlease provide some more information on the downstream tasks (how many classes, what kind of labels, how are they obtained,…)\n-\tSource code is not (yet) released.\n\nMinor Remarks:\n\n-\t“temporal” is missing the “l” in a few places (e.g. l. 162,208) \n-\t“reach” -> “reaches” in l.281\n-\tI would suggest to remove the particular results from the abstract", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your constructive and insightful comments. We address all weaknesses and questions below.\n\n**W1 and Q1. Detail information about downstream tasks.**\n\nOur downstream tasks include classification, segmentation, and detection, each addressing different aspects of endoscopic video analysis.\n\n* **Classification Task:** The classification task is a binary classification problem aimed at diagnosing polyps. We use the PolypDiag [1] dataset, which follows the gastroscope examination protocol and includes 253 videos and 485,561 frames. Each video is annotated to indicate the presence or absence of a lesion. The dataset is divided into 71 normal videos without polyps and 102 abnormal videos with polyps for training, and 20 normal videos and 60 abnormal videos for testing. \n\n* **Segmentation Task:** The segmentation task aims to perform polyp segmentation. We use the CVC-12k [2] dataset, which follows the colonoscopy examination protocol and serves as the official training dataset for the Automatic Polyp Detection sub-challenge of the MICCAI 2015 competition. This dataset includes 29 videos and 612 frames, with 20 videos allocated for training and 9 videos for testing. Each frame in the videos is annotated with ground truth masks (with a single class) to identify the regions covered by polyps.\n\n* **Detection Task:** The detection task aims to perform polyp detection. We use the KUMC [3] dataset, which follows the colonoscopy examination protocol and is sourced from the Kansas University Medical Center. This dataset includes 53 videos and 19,832 frames. Each frame in each video is annotated with bounding boxes and polyp categories, with 36 videos allocated for training and 17 videos for testing, including hyperplastic polyps and adenomatous polyps. \n\nFor a fair comparison, all datasets are sourced from Endo-FM, and the data partitioning for all datasets follows Endo-FM guidelines. For more details on the experimental reproduction of downstream tasks, we will provide more detailed relevant hyperparameters in the appendix.\n\n**W2. Source code is not (yet) released.**\n\nWe will release the code and pre-trained checkpoints upon acceptance. For reproducibility we build on the open source implementation of Endo-FM and provide all the relevant hyperparameters in the appendix.\n\n**W3. Minor Remarks.**\n\nThank you for your valuable suggestions. We will remove the particular results from the Abstract and make careful writing revisions in subsequent revisions.\n\n**Q2. Are any modifications needed to employ this method to other spatio-temporal data?**\n\nAlthough our work is based on an endoscopic video dataset, it is important to note that the video format of endoscopic videos is the same as that of natural videos (Kinetics, SSv2). The basic techniques for processing and analyzing video data, such as frame extraction and video decoding, are essentially the same for both types. Therefore, applying our method to other spatio-temporal data does not necessitate specific modifications; rather, it only requires adherence to standard techniques for processing video data.\n\n**Limitations. The potential negative societal impact is not sufficiently discussed.**\n\nOur study involves self-supervised learning, which requires extensive pre-training leading to significant energy consumption. Additionally, large-scale model training requires high-performance computing hardware (GPUs), leading not only to increased hardware costs but also resource overconsumption and electronic waste. These negative impacts underscore the necessity of considering environmental protection and resource conservation. In future work, we will adopt more efficient training methods and optimization strategies to address these issues.\n\n[1] Tian Y, et al. Contrastive transformer-based multiple instance learning for weakly supervised polyp frame detection. MICCAI 2022.\n\n[2] Bernal J, et al. WM-DOVA maps for accurate polyp highlighting in colonoscopy. Computerized medical imaging and graphics 2015.\n\n[3] Li K, et al. Colonoscopy polyp detection and classification: Dataset creation and comparative evaluations. Plos one 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers and AC,

 

We sincerely appreciate your valuable time and effort spent reviewing our manuscript. We thank all reviewers for their useful comments, positive consideration and relevant feedback on our paper. It seems that the reviews are positive in general and acknowledges our main contributions and soundness of our work. Our submission has received four ratings, including one accept (7), one weak accept (6), one borderline accept (5), and one borderline reject (4). 

 

The valuable feedback from the reviewers has significantly contributed to enhancing the quality of our manuscript. We have supplemented all ablation experiments according to the reviewers’ comments and suggestions. We have addressed each comment and question individually below and we would be glad to engage in discussion in case of more questions or concerns exist. We will also update the main paper in the future with the main requested changes for improvement. Furthermore, we kindly request Reviewer wGJR to reconsider our work after reviewing our response. Your reconsideration will be highly valued.



Based on the comments from the reviewers, we have summarized the strengths of our paper as follows:

* **Motivation: [Reviewer BGWe, u6Br]** The motivation is clearly laid out and the reference to prior related work is thorough. Different aspects are well motivated and explained. It gives a good overview of related work, particular challenges associated with endoscopy images and motivates the proposed approach.

* **Method: [Reviewer wGJR, R5Z9, BGWe, u6Br]** The paper shows the combination of self-distillation and masked video modeling for pre-training ViT-B model using endoscopic videos. Frame-aggregated attention guided tube mask (FAGTM) to learn global spatio-temporal representation learning. Ablation experiments show the contribution of the components. Ablation studies help showcase which elements of M$^2$CRL are most impactful. The study includes an ablation study to report on the effect of different components of the proposed methods.

* **Experiment: [Reviewer wGJR, R5Z9, BGWe, u6Br]** Experiments on multiple tasks to show the efficacy of the approach. The model is validated against different works and on three downstream tasks. The experiments are very thorough and soundly conducted. The conducted experiments are exhaustive. The results are averaged over three runs and show consistent improvements in all three tasks.

* **Expression: [Reviewer wGJR, BGWe, u6Br]** The paper is easy to read. The organization of the paper is excellent, with helpful use of bold text, logical flow from one passage to the next, and high-quality, information-dense illustrations and tables. The manuscript is well and clearly written. The background section summarizes a lot of related literature.

* **Impact: [Reviewer R5Z9, BGWe, u6Br]** Representation learning in the medical domain can have an impact on the development of models for medical image analysis. Results demonstrate notable improvement upon existing competitive baselines. The method has been developed for application in endoscopy, but are relevant for spatio-temporal imaging in general and the type of pretraining is relevant for other medical imaging modalities with a temporal component. 



Here we also present the response to two questions below.

_**Concern about novelty of the model**_

We have summarized our novelty as follows:

* Existing self-supervised pre-training methods for endoscopic videos predominantly rely on contrastive learning. However, using contrastive learning alone is not sufficient to capture the fine-grained feature representations required for endoscopic videos. To address the limitations of current methods, we have integrated contrastive learning with masked modeling to effectively acquire endoscopic video representations that possess both comprehensive discriminative capability and fine-grained perceptive ability.

* Given the characteristics of inter-frame instability and small inter-class differences in endoscopic videos, we propose a multi-view mask strategy. Specifically, we introduce a frame-aggregated attention guided tube mask strategy for the global views, which aggregates features from multiple frames to capture global spatiotemporal information. Simultaneously, a random tube mask strategy is employed from the local views, enabling the model to focus on local features.

* Extensive experiments have verified that our M$^2$CRL significantly enhances the quality of endoscopic video representation learning and exhibits excellent generalization capabilities in multiple downstream tasks.

_**Concern about experiment to other datasets or endoscopy modalities**_

* **[Reviewer wGJR]:** The performance on other datasets like Cholec80, a surgical phase recognition benchmark dataset, would be great.

- **[Reviewer R5Z9]:** It would be interesting to see the performance in downstream tasks that involve other endoscopic modalities.

In our study, we used multiple publicly available endoscopic video datasets, provided by research groups worldwide and previous EndoVis challenges. These datasets cover 3 types of endoscopic procedures (colonoscopy, gastroscopy, and laparoscopy) and 10+ different diseases. We believe that this comprehensive and large-scale dataset is valuable for endoscopic research. In downstream tasks, classification is for gastroscopy, while segmentation and detection are for colonoscopy. Based on reviewer’s suggestion, we have added the Cholec80 dataset in downstream tasks. The results demonstrate that our approach achieves superior performance. This experiment further validates the robustness and effectiveness of our approach across diverse endoscopic video tasks.


We strongly believe that M$^2$CRL can be a useful addition to the NeurIPS community, in particular, due to the enhanced manuscript by reviewers’ comments helping us better deliver the effectiveness of our method.


Thank you very much!

Best regards,

Authors

Meta-review: The authors proposed a pre-training scheme for endoscopic video data by adapting contrastive learning and masked modeling together. A novel masking strategy via frame-aggregated attention is also proposed to enhance the global spatiotemporal contextual relationships. The manuscript is well-written overall, with clarified technical novelty (after the rebuttal and discussion) and solid experimental results on ten public datasets (superior performance compared to prior SSL methods). In the rebuttal, the authors thoroughly addressed the reviewers' questions and supplemented the relevant results. Overall, the paper contributes novel insights and advancements to the pre-training task in medical imaging.
Decision: Accept (poster)