Multi-view Masked Contrastive Representation Learning for Endoscopic Video Analysis

Authors: kai  hu
ye  xiao
yuan  zhang
xieping  gao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 11:21:08

Abstract: Endoscopic video analysis can effectively assist clinicians in disease diagnosis and treatment, and has played an indispensable role in clinical medicine. Unlike regular videos, endoscopic video analysis presents unique challenges, including complex camera movements, uneven distribution of lesions, and concealment, and it typically relies on contrastive learning in self-supervised pretraining as its mainstream technique. However, representations obtained from contrastive learning enhance the discriminability of the model but often lack fine-grained information, which is suboptimal in the pixel-level prediction tasks. In this paper, we develop a Multi-view Masked Contrastive Representation Learning (M$^2$CRL) framework for endoscopic video pre-training. Specifically, we propose a multi-view mask strategy for addressing the challenges of endoscopic videos. We utilize the frame-aggregated attention guided tube mask to capture global-level spatiotemporal sensitive representation from the global views, while the random tube mask is employed to focus on local variations from the local views. Subsequently, we combine multi-view mask modeling with contrastive learning to obtain endoscopic video representations that possess fine-grained perception and holistic discriminative capabilities simultaneously. The proposed M$^2$CRL is pre-trained on 7 publicly available endoscopic video datasets and fine-tuned on 3 endoscopic video datasets for 3 downstream tasks. Notably, our M$^2$CRL significantly outperforms the current state-of-the-art self-supervised endoscopic pre-training methods, e.g., Endo-FM (3.5% F1 for classification, 7.5% Dice for segmentation, and 2.2% F1 for detection) and other self-supervised methods, e.g., VideoMAE V2 (4.6% F1 for classification, 0.4% Dice for segmentation, and 2.1% F1 for detection).

Main paper: 
 introduction: 
 Video endoscopy is a crucial medical examination and diagnostic tool widely used for inspecting various tissues and structures (the digestive tract, respiratory tract, and abdominal cavity, etc.) [1] . In clinical practice, endoscopic video analysis usually relies on the experience and expertise of physicians, which is not only time-consuming and labor-intensive but also prone to subjective errors. Computer-aided medical analysis [2, 3, 4] can automatically and efficiently identify and classify lesions, thereby assisting physicians in making more accurate diagnoses. In this paper, we focus on endoscopic videos with the aim of developing a robust pre-trained model for endoscopic video analysis to facilitate downstream tasks (i.e., classification, segmentation, and detection).Yann LeCun has mentioned "the revolution will not be supervised [5] " in multiple talks, emphasizing that the future development of artificial intelligence will increasingly rely on un-/self-supervised learning. Among them, self-supervised learning (SSL) aims to learn scalable visual representations from large amounts of unlabelled data for downstream tasks with limited annotated data. To learn meaningful representations for SSL, researchers crafted visual pretext tasks [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16] , which are summarized into two main categories: contrastive and generative [17] . Contrastive methods, also known as discriminative methods, employ a straightforward discriminative idea that pulling closer representations from the same image and pushing away different images, i.e., contrastive learning (CL) [6, 7, 8] . By utilizing image-level prediction with global features, CL can naturally endow pre-trained models with strong instance discriminability, which has been proven to be effective in classification tasks. However, CL also presents the challenge that downstream dense prediction tasks, such as segmentation and detection, are not fully considered.Generative methods aim to reconstruct the input data itself by encoding the data into features and then decoding it, including AE [18] , VAE [19] , GAN [20] , etc. Recently, masked image modeling (MIM) [13, 14, 16 ] has demonstrated the strong potential in self-supervised learning. MIM masks a substantial portion of image patches during training and utilizes an autoencoder [18] to reconstruct the original signal of the image, which, unlike CL, can enhance the ability to capture the pixel-level information. Following the success of MIM, some works have tried to extend this new pre-training paradigm to the video domain for self-supervised video pre-training [21, 22, 23] . Actually, mask techniques [24, 25, 26] are crucial for the success of mask modeling. Endoscopic videos, in particular, have higher dimensions and redundancy compared to static images. They also exhibit unstable interframe variations due to the manual manipulation by the doctor. Additionally, lesions in endoscopic videos often have low contrast and appear in obscured regions or with subtle variations. Therefore, simply applying random mask not only requires extensive pre-training time but also easily leads to poor performance.To address the aforementioned issues, in this paper, we develop a Multi-view Masked Contrastive Representation Learning framework named M 2 CRL. First, considering the characteristics of interframe instability and small inter-class differences in endoscopic videos, we propose a multi-view masking strategy. Specifically, we introduce a frame-aggregated attention guided tube masking strategy for the global views, which aggregates features from multiple frames to capture global spatiotemporal information. Simultaneously, a random tube masking strategy is employed from the local views, enabling the model to focus on local features. Second, to address the inadequacy of capturing pixel-level details in contrastive learning, we integrate multi-view masked modeling into contrastive approach, which not only encourages the model to learn discriminative representations but also forces it to capture more refined pixel-level features. Extensive experiments have verified that our M 2 CRL significantly enhances the quality of endoscopic video representation learning and exhibits excellent generalization capabilities in multiple downstream tasks (i.e., classification, segmentation and detection). Overall, our contributions are summarized as follows:• We propose a novel multi-view masking strategy aimed at enhancing the capture of finegrained representations in endoscopic videos by performing mask modeling on both global and local views. This strategy involves utilizing the frame-aggregated attention guided tube mask to capture global-level spatiotemporal contextual relationships from the global views, while employing the random tube mask to focus on local variations from the local views.• We propose a multi-view masked contrastive representation learning framework that combines multi-view mask modeling with contrastive method to train endoscopic videos, which effectively addresses the limitation of contrastive method in capturing dense pixel dependencies by predicting the intensity of each pixel within masked patches.• We conduct extensive experiments on 10 endoscopic video datasets to evaluate the performance of M 2 CRL in comparison to other methods. M 2 CRL achieves 94.2% top-1 accuracy on PolypDiag [27] , 81.4% on CVC-12k [28] , and 86.3% on KUMC [29] , outperforming the state-of-the-art methods, i.e., Endo-FM [30] by +3.5%, +7.5%, and +2.2%, and VideoMAE V2 by +4.6%, +0.4%, and +2.1%, respectively. 2 Related WorkQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The main claims presented in our abstract and introduction accurately reflect the contributions and scope of the paper. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
self-supervised video representation learning: 
 Self-supervised learning is a machine learning technique that mitigates the reliance on manual labeling by exploiting the inherent structure or properties of the data itself, and its common goal in computer vision is to learn scalable and generalisable visual representations. In SSL, the key lies in designing an appropriate pretext task, which involves generating suitable supervision using only visual signals. Pretext tasks in images [9, 10, 11, 12] have been widely explore, and motivated by images, there has been great interest in exploring SSL pretext tasks for videos. Some early works [31, 32, 33] focused on extending image SSL methods for video, however video has a specific temporal dimension. The temporal information can be leveraged for representation learning, including future prediction [34, 35, 36, 37, 38, 39, 40] , temporal ordering [41, 42, 43, 44, 45] , object motion [46, 47, 48, 49] , and temporal consistency [50, 51] . In the last few years, contrastive learning [52, 53, 54, 55, 56, 57] has made great progress in SSL. However, SSL based on contrastive methods typically focuses on discriminative features of holistic information, while lacking the ability to focus on fine-grained features.Masked visual modeling is proven to be a simple and effective self-supervised learning paradigm [13, 14, 15, 16, 58, 21, 22] through a straightforward pipeline based on masking and reconstruction. This paradigm quickly expanded to the video domain, e.g., VideoMAE [22] and ST-MAE [21] , which extend image MAE to video. As with images, the choice of masking strategy affects selfsupervised representation learning [24, 25, 26, 59] . VideoMAE [22] employs random tube mask to model the same locations across frames to ensure spatial discontinuity while maintain temporal continuity. In contrast, ST-MAE [21] generates a random mask for each frame independently, which are discontinuous in both space and time. Nevertheless, mask video modeling using randomly masked tokens to reconstruct is inefficient because the tokens embedded in a video frame are not equally important. Several studies [60, 61, 62, 63] have proposed various motion-based video mask strategies. However, our focus is on endoscopic videos, which lack a strong motion counterpart compare to the above works.In recent years, SSL has received increasing attention in the field of medical analysis, including endoscopy. Endo-FM [30] employs contrastive method to minimize the disparity in feature representations between different spatiotemporal views of the same video. Intrator [64] has proposed the use of contrastive learning to adapt video inputs for appearance-based object tracking. Hirsch [65] applies the SSL framework masked siamese networks (MSNs) to analyze endoscopic videos. While MSNs use a masking concept, it primarily serves as a data augmentation technique, with its essence still rooted in contrastive learning. Currently, most self-supervised pre-training for endoscopic videos relies on contrastive methods. While these methods have shown promise for endoscopic video pre-training, relying solely on them may not fully capture the fine feature expressions of endoscopic videos.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss the limitations of our paper in § F. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
method: 
 To address the limitations of contrastive methods in fine-grained information perception in SSL for endoscopic videos, we propose a Multi-view Masked Contrastive Representation Learning (M 2 CRL) framework for endoscopic video pre-training, as shown in Fig. 1 . Here, we first review masked prediction in Section 3.1. Then, we present our proposed multi-view masking strategy in Section 3.2 and introduce our M 2 CRL framework in Section 3.3.Masked prediction is a prevalent representation learning technique in natural language processing (NLP) [66, 67, 68, 69] , and many researchers have explore its application to images [14, 16, 70, 71, 72] and videos [21, 22, 23] . MIM endeavors to learn image representations by solving a regression problem, where the model is tasked with predicting the pixel values in randomly masked patches of the image. Specifically, an image x ∈ R H×W ×C is reshaped into N = HW P 2 flattened patches as {x i } N i=1 , where (H, W ) is the resolution of image, C is the number of channels and P is the patch size. Each patch is represented with token embedding. MIM constructs a random mask m ∈ {0, 1} N to indicate the masked tokens that correspond to m i = 1. In MAE [14] , only the visible tokens {x i |m i = 0} N i=1 are fed into the vision transformer to obtain the latent feature, and then the decoder uses the latent feature and the masked tokens as inputs to predict ŷ. In SimMIM [16] , visible and invisible tokens are fed into the encoder. The prediction loss is calculated as the loss between the normalized masked tokens and the reconstructed ones in the pixel space by:EQUATIONwhere N m is the number of masked tokens, x m is the masked token, p is norm and its value is 1 or 2.Masked video modeling (MVM) [21, 22, 23] employs random mask strategies (i.e., random, spatialonly, temporal-only) to capture meaningful representations from pre-training videos. Although these strategies are effective for general video datasets with well-curated and stable distributions, they do not account for the unique characteristics of medical data. We have summarized two key characteristics of endoscopic videos: (1) Instability of inter-frame variations is a prominent feature of endoscopic videos. These variations are driven by camera movement, instrument manipulation, and the uneven distribution of lesion areas, e.g., variations can range from drastic to minor, as the camera navigates from the intestinal wall to specific lesion sites. (2) Endoscopic video exhibits characteristics of small inter-class differences. The lesion tissues typically resemble surrounding the normal tissues in color, texture, or shape, which complicates the model's ability to accurately identify the lesion area. Therefore, we propose a multi-view masking strategy by considering the above two points, the details are as follows.To address the challenge of instability between video frames, we propose a frame-aggregated attention guided tube masking strategy. We aggregate the attention of all frames of the video along the frame dimension to generate a frame-aggregated attention map, which then dynamically guides the masking process. This way can capture the overall scene information in a video sequence from the global spatiotemporal information and ignores irrelevant spatiotemporal noise to some extent.Semantic information extraction Our architecture consists of the teacher network f t and the student network f s . Our network employs a self-attention mechanism known as divided space-time attention mechanism [73] , which enhances the learning ability of the network while reducing the computational complexity. Specifically, we take an endoscopic video, from which we sample the global views{v i g ∈ R T i g ×3×Hg×Wg } G i=1, where T g is the number of frames in the sampled view. Each frame is then divided into N = H g W g P 2 patches, which are mapped into patch tokens and fed into the transformer blocks of the teacher network. Thus, each encoder block processes N patch (spatial) and T g temporal tokens. The network includes a learnable class token, [cls], which represents the global features learned by the network along spatial and temporal dimensions. Given the intermediate token e u ∈ R (N +1)×D from block u, the token in the next block is computed as follows: where M SA, LN and M LP denote the multi-head self-attention, layer normalization, and multilayer perceptron, respectively. Each block utilizes M SA layer to project and divide e into n h parts. Each part contains the query Q r , key K r and value V r for r = 1, 2, ..., n h , where n h denotes the number of heads. We can get the attention map of the last layer of blocks by calculating the correlation between the query embedding of class token Q cls and key embeddings of all other patches K. It is averaged for all heads as follows:    EQUATIONwhere D h = D/n h , and A is T g spatial attention maps. Although single-frame spatial attention integrates temporal information, it still only considers the spatial information of the current frame, neglecting the global spatiotemporal dependencies in the video sequence. Thus, we aggregate the attention of T g frames to the mean value to obtain a simplified and holistic attention distribution by:EQUATIONThis attention mechanism is capable of obtaining an approximation of the critical region of the video that are being attended to from both the temporal and spatial dimensions, while reducing the impact of excessive variations in individual frames or regions. We will further exploit this attention dynamic to guide the generation of tube masking to help the model perform the reconstruction task more appropriately.Visible tokens sampling and masking The traditional random masking strategy treats critical and non-critical areas of the video equally in each iteration, which may lead to excessive masking of key video regions at a high masking ratio, thereby affecting the learning ability of the model. Therefore, we utilize a frame-aggregated attention mechanism to guide the generation of tube mask. By sampling some reasonably visible tokens from the model-focused areas and masking the rest, it allows our method to efficiently perform the reconstruction task even at a high masking ratio, while improving pre-training efficiency. Specifically, we begin by ranking tokens in descending order of attention scores and subsequently select a proportion of high-attention tokens based on the threshold γ. We randomly sample visible tokens to form a binary mask from selected tokens of attention, as depicted in Fig. 1(b ). The number of sampled visible tokens N v is determined by the predefined mask ratio ρ ∈ (0, 1). Followed by SimMIM [16] , we fill the mask tokens with learnable mask embeddings, which are subsequently fed together into the student network for feature mapping and finally into the prediction head for reconstruction.In comparison to random sampling, which is inefficient in token allocation, our method selects visible tokens based on the global spatiotemporal information of the video sequence. This approach minimizes redundant background area sampling, as these backgrounds have minimal impact on the significance of mask reconstruction. Furthermore, the attention-guided mask generation is dynamic and allows for adjustments during model training. This adaptability enables the model to continuously optimize its focus, adapting to complex or changing data characteristics.Global view mask modeling is primarily employed for a comprehensive understanding and spatiotemporal contextual awareness of the entire endoscopic video frame, which mitigates the effects of inter-frame variability instability by aggregating frame attention. However, due to the low contrast between lesions and normal tissues in endoscopic videos, global views may struggle to accurately capture local details. Hence, we apply random tube mask reconstruction on local views to learn more granular detail information, as shown in Fig. 1(c ). Specifically, we obtain the local views{v j l = R T i l ×3×H l ×W l } L j=1 (T l < T g) by random cropping and uniform sampling at different frame rates. In local views, we also implement a high masking ratio ρ = 90% to reduce information leakage during the mask modeling process. By local view mask modeling with different frame rates and spatial cropping, it is possible to make the model proficient in capturing variations in time scale and spatial detail. Moreover, local view mask modeling focuses on specific regions in the video without interference from the global background, enables more targeted learning. This allows the model to finely capture local information within the video, enhancing its ability to recognize subtle differences between abnormalities and normal tissues.The pipeline of our proposed M 2 CRL is shown in Fig. 1(a) , which introduces a multi-view masking strategy and combines multi-view mask modeling with contrastive learning to learn representations that possess fine-grained and discriminative capabilities simultaneously.In DINO [55] , self-distillation is proposed not from the posterior distribution but by a teacher-student scheme that extracts knowledge from the model's own past iterations. This self-distillation method of self-supervision is also considered a form of contrastive learning [80] . The contrastive learning part of our M 2 CRL follows Endo-FM [30] , which also employs self-distillation method to achieve representation learning. Given an endoscopic video, two types of views are created under random data augmentation (G global views {v i g } G i=1 and L local views {v j l } L j=1 ). The model is encoded by two encoders, a teacher network f t and a student network f s , which are respectively parameterized by θ t and θ s . It should be noted that the two student networks depicted in Fig. 1 (a) actually represent EQUATIONContrastive learning utilizes class tokens [55] from global and local views to calculate matching losses, which offers strong capabilities for overall discriminative representation. However, this method overlooks the dense pixel dependencies that are crucial for dense prediction tasks like segmentation and detection. Therefore, we integrate the multi-view mask reconstruction pre-training task into the contrastive learning. In the multi-view mask modeling task, the video clips processed through masking are fed into the encoder to be mapped into the feature space. Subsequently, the prediction head utilizes the context of unmasked patches to regress the dense pixel intensities within masked patches. The losses for global and local view reconstruction are as follows:EQUATIONM 2 CRL exploits both contrastive loss and reconstruction loss in optimization, and the total loss isL total = L gg CL + L gl CL + L g M V M + L l M V M .The student network updates the student parameters θ s by minimizing L total , and the teacher network θ t is updated using the exponential moving average (EMA) of the student weights, and θ t ← λθ t + (1 -λ) θ s . Here λ denotes the momentum coefficient. By combining contrastive learning and masked video modeling, the model is encouraged to learn the holistic discriminative representation and detailed pixel information, effectively improving the learning ability of the model in complex visual data of endoscopic videos. 
datasets and experimental settings: 
 We conduct experiments on 10 publicly available endoscopic video datasets: Colonoscopic [81] , SUN-SEG [82] , LDPolypVideo [83] , Hyper-Kvasir [84] , Kvasir-Capsule [85] , CholecTriplet [86] , Renji-Hospital [30] , PolypDiag [27] , CVC-12k [28] , and KUMC [29] . These datasets have a total of 33,231 videos with approximately 5,500,000 frames, covering 3 types of endoscopy examination protocols and 10+ different diseases. The videos are processed into 30fps clips with an average duration of 5 seconds. The first 7 datasets are used for pre-training and we sample G = 2 global views and L = 8 local views, where T g ∈ [8, 16] , T l ∈ [2, 4, 8, 16] and the spatial size set to 224 × 224 and 96 × 96, respectively. We take ViT-B/16 [87] as the backbone and perform 30 epochs of pre-training. In downstream tasks, we perform classification task on PolypDiag, segmentation task on CVC-12k, and detection task on KUMC, respectively. Our implementation is based on Endo-FM [30] , and more experimental details can be found in § A.We compare our method with the recent state-of-the-art (SOTA) endoscopic video pre-training model, Endo-FM [30] , which is the first pre-training model on a large scale across various endoscopic videos. The results of other methods are taken from the comparative method of Endo-FM, including TimeSformer [73] , CORP [74] , FAME [75] , ProViCo [76] , VCL [77] , and ST-Adapter [78] . Additionally, we also compare the latest video self-supervised methods: VideoMAE [22] , VideoMAE V2 [23] and DropMAE [79] . For a fair comparison, all methods are pretrained on the same union of 7 datasets as our M 2 CRL. All experimental settings are referred to those documented in the original papers or in the released code.Quantitative evaluation We observe that our method outperforms existing state-of-the-art methods, as shown in Table 1 . Particularly, compared to Endo-FM [30] , our M 2 CRL achieves improvements of 3.5% F1 in classification (PolypDiag), 7.5% Dice in segmentation (CVC-12k), and 2.2% F1 in detection (KUMC) tasks. These improvements are attributed to our pre-training approach, which integrates multi-view mask modeling with contrastive method, substantially enhancing the model's ability for representation learning. Clearly, compared to other methods (TimeSformer [73] , CORP [74] , FAME [75] , etc.), M 2 CRL achieves considerable advantages across three downstream tasks. Although the performance gains on extremely dense task (i.e., segmentation) is modest compared to the latest video self-supervised methods (i.e., VideoMAE [22] , VideoMAE V2 [23] , DropMAE [79] ), our M 2 CRL makes great progress on classification task and reaches 94.2%. This shows that our M 2 CRL not only focuses on pixel details but also enhances discriminative capabilities.Qualitative evaluation We visualize segmentation and detection results in Fig. 2 . Compared to other methods, our M 2 CRL demonstrates superior visual results in segmenting both large and small polyp regions (1 st and 2 nd rows on the left in Fig. 2 ). Despite potential issues such as blurry boundaries or lens glare caused by camera movement, M 2 CRL is still capable of accurately segmenting the target regions (3 rd row on the left in Fig. 2 ). We attribute these results to our multiview mask modeling, which encourages the model to learn more precise detail information from videos. Similarly, M 2 CRL also exhibits good performance in detection tasks, especially in detecting small target regions (2 nd row on the right in Fig. 2 ). Although there are some differences between the predictions of our method and the ground truth, we achieve a high degree of overlap with the ground truth. This demonstrates that our M 2 CRL significantly improves the pre-training capability for endoscopic videos. See § D for more segmentation and detection visual comparison.Multi-view mask Table 2 illustrates the impact of single-view mask and multi-view mask. All experiments use the same masking ratio. For the single-view, different masking strategies are employed for global and local views, respectively. The random tube mask [22] randomly samples masked tokens in the 2D spatial domain and then extends these tokens along the temporal axis. The random mask [21] randomly masks tokens in the spatiotemporal domain. Our approach samples visible tokens by leveraging frame-aggregated attention from global views, resulting in better results than above two strategies, particularly achieving 80.6% in segmentation task. Similarly, the random tube mask demonstrates some superiority on local views. However, from Table 2 , it can be observed that the performance of solely conducting mask modeling on the single-view is inferior to that on multi-views. In multi-view mask, it is evident that employing the frame-aggregated attention guided tube mask on global views and the random tube mask on local views results in significant performance gains. These two complementary mask methods work together to learn richer details of video features.Hyper-parameters of the FAGTM We conduct an experiment to investigate the impact of the hyperparameters of the frame-aggregated attention guided tube mask (FAGTM) for global views, the results are shown in Table 3 . We perform experiments by sorting each patch of the obtained frame-aggregated attention maps in descending order and selecting the top γ proportion of patches as candidate mask patches. Subsequently, visible tokens are random sampled in candidate mask patches.From Table 3 , we can observe that the model performs best when γ is set to 0.6. A lower value indicates selecting visible patches from smaller high-attention regions, leading to excessive attention on non-critical areas during reconstruction, contradicting the setup of the self-supervised pre-text task. On the other hand, higher values of γ adversely affecting the learning efficacy of the model.The teacher's block used for the FAGTM In our study, we use the last layer block of the teacher ViT-B for the FAGTM. The higher layer block incorporates lower-level features and object-level semantic information, offering comprehensive and abstract features that are essential for the model. Thus, it effectively guides the student network in masking. Table 4 shows that it is most beneficial for the FAGTM to utilize the last layer block of the teacher network.Masking ratio The impact of different masking ratios is illustrated in Table 5 . It can be observed that there is an improvement in results across three downstream tasks when the masking ratio increases from 75% to 90%. Due to the reundancy in videos, it shows that the 75% masking ratio utilized in ImageMAE is not suitable for videos. When the masking ratio in videos reaches 90%, the task becomes challenging due to the limited number of patches available for learning temporal correspondence, thus enhancing the learning capacity of the model. This observation is also validated in VideoMAE [22] . Compared to the optimal masking ratio, a higher masking ratio increases the difficulty of pre-training, hindering the model's ability to learn effective representations. Although our ablation experiments have shown that a masking ratio of 95% can achieve comparable performance, its effectiveness in three downstream tasks is lower than that of the 90% masking ratio. This suggests that at a masking ratio of 95%, the model is placed in a relatively unfavorable learning situation, resulting in suboptimal results.As see from the first row in Table 6 , although the performance of the contrastive learning framework on classification tasks is acceptable, its performance on pixel-level tasks, especially the segmentation task, is not very good. Similarly, within the single mask modeling self-supervised framework, there is an improvement in performance for the pixel-level task (e.g., from 73.9% to 80.5% for segmentation), while there is a significant drop in performance for the discriminative task (e.g., from 90.7% to 85.7% for classification). These interesting phenomena once again highlights the strong instance discrimination ability of contrastive learning and the robust ability of mask modeling to acquire local pixel-level information. Our approach integrates both methods and demonstrates consistently strong performance across both structural and pixel-level tasks.  
conclusion: 
 In this study, we present a novel SSL approach called M 2 CRL, which integrates multi-view mask modeling with contrastive learning for endoscopic video analysis. Our method aims to address the lack of pixel-level information extraction in existing SSL methods for endoscopic videos. We leverage the unique characteristics of endoscopy to propose a frame-aggregated attention guided tube mask that captures pixel-level relationships across spatial-temporal dimensions for global views. Additionally, we utilize random tube mask to complement the details of local features from local views. Notably, the attention guidance is derived from multi-head self-attention maps extracted from a teacher model, without incurring additional computational costs. By integrating contrastive learning, our method not only maintains performance in dense prediction tasks but also ensures effectiveness in discriminative tasks. The experimental results on 10 publicly avaiable datasets demonstrate that our M 2 CRL outperforms the state-of-the-art methods across multiple downstream vision tasks (i.e., classification, segmentation, and detection). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? 
 
 Appendix: In the downstream tasks, we conduct surgical phase recognition experiments using the Cholec80 [93] dataset, which contains 80 complete laparoscopic cholecystectomy videos. This dataset is specifically designed to evaluate the performance of the model in automatic surgical phase recognition. To ensure a fair comparison with existing methods, we follow the data split and evaluation protocol described by [94] , using 40 videos for training and 40 for testing. For evaluation, we adopt the relaxed boundary F1 score proposed by [95] . Since our backbone is based on Transformers, we do not use the CNN-based TCN [94] as a feature extractor during fine-tuning, so our result is based on single frames. As shown in Table our method achieves superior performance. This experiment further verifies the robustness and effectiveness of our method across various endoscopic video tasks. Polyps are characterized by their blurry boundaries and significant shape variations, which undoubtedly add complexity to the segmentation task. Nevertheless, our method consistently outperforms other state-of-the-art self-supervised methods. Specifically, when polyps are very close and overlapping (3 rd row of Fig. 3 ), our method produces smoother and more continuous edges, whereas other methods often result in blurred or fragmented edges. Additionally, some small polyps (4 th row of Fig. 3 ) occupy very few pixels and are easily missed, our method successfully segments these small polyps without omission. Fig. 4 shows the detection results of our method and other self-supervised pre-training methods on the KUMC dataset. As shown in the figure, our method performs well in boundary identification and localization for small polyps (1 st and 2 nd rows of Fig. 4 ). Furthermore, 3 rd and 4 th rows of Fig. 4 demonstrate that our method can also accurately identify and locate polyps even when they have low contrast and indistinct boundaries with surrounding tissues. Notably, compared to Endo-FM, a discriminative approach, our method significantly enhances the performance on pixel-level tasks. Figure 5 : Illustration of our frame-aggregated attention guided tube masking strategy. We visualize spatial attention map with temporal information for each frame (2 nd column), then aggregate attention maps for all frames and select area of high attention. We sample visible patches in this area (3 rd column).In Fig. 5 , we visualize the attention maps of the ViT-B/16 model employing "the divided space-time attention" mechanism. We utilize the [cls] token as a query and employ attention from the last transformer block. From the attention map of individual frames, we observe that the model does not distinctly learn the concept of endoscope object boundaries, attributed to the separate computations of attention in time and space. To enhance object concepts within endoscope video sequences, we aggregate attention across all frames of the video and then select a certain proportion of high-attention regions. This approach ensures that even in complex endoscopic video scenarios, key information within the video sequence is retained. Random masking at a high ratio in the video can obscure critical regions, thereby impeding the ability of the model to learn video representations. To address this issue, we select regions of high attention from the aggregated attention map for sampling visible tokens.Limitations Our work presents a multi-view masked contrastive representation learning (M 2 CRL) framework for endoscopic video analysis. However, our current self-supervised pre-training method only utilizes RGB video streams and does not incorporate additional audio and text streams. In the future, we expect that audio and text data can provide more information for self-supervised pre-training. Furthermore, our study requires extensive pre-training, leading to significant energy consumption and reliance on high-performance computing hardware (GPU). These negative impacts underscore the necessity of considering environmental protection and resource conservation. In future work, we will adopt more efficient training methods and optimization strategies to address these issues.Broader impacts Our approach demonstrates the potential of SSL in endoscopic video analysis. By utilizing a large amount of unlabeled endoscopic video data for pre-training, we can reduce the dependence on costly annotated medical data, thus lowering healthcare expenses. Furthermore, our pre-trained models can be easily applied to various tasks such as classification, segmentation, and detection, thereby making valuable contributions to medical applications and enhancing the quality and efficiency of clinical disease diagnosis and healthcare services.NeurIPS Paper ChecklistJustification: There are no such risks in our paper.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have cited all the relevant assets used in our paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.Code is publicly available at: https://github.com/MLMIP/MMCRL. 


Decision: The authors proposed a pre-training scheme for endoscopic video data by adapting contrastive learning and masked modeling together. A novel masking strategy via frame-aggregated attention is also proposed to enhance the global spatiotemporal contextual relationships. The manuscript is well-written overall, with clarified technical novelty (after the rebuttal and discussion) and solid experimental results on ten public datasets (superior performance compared to prior SSL methods). In the rebuttal, the authors thoroughly addressed the reviewers' questions and supplemented the relevant results. Overall, the paper contributes novel insights and advancements to the pre-training task in medical imaging.

Reviews: review: paper_summary: paper_summary: The paper proposes a pre-training approach called Multi-view Masked Contrastive Representation Learning for endoscopic videos. The approach combines self-distillation and masked video modeling under multi-view setting. To consider the characteristics of inter-frame instability and small inter-class differences of endoscopic videos, the paper introduces a frame aggregated attention guided tube masking strategy to capture global spatio-temporal representation and employs random tube masking on local views to capture local representations. The approach is pre-trained on seven endoscopic datasets and fine-tuned on three additional datasets. Experiments show that it outperforms the baselines on classification, segmentation and detection tasks.
strength_weakness: strength_weakness: - Although the pre-training approach is proposed for endoscopic videos, the novelty is limited. It's a combination of self-distillation and masked video modeling. 
- FAGTM is also an extension of either [1] and [2] which propose attention guide masking strategy for image based pre-training. The paper merely aggregates the attention for all frames and use the mean of that to guide the masking.
- The paper mentions in section 3.3, "This self-distillation method of self-supervision is also considered a form of contrastive learning". Can the authors please give remark on why self-distillation is a form of contrastive learning?
- Which block of the teacher ViT-B is used for FAGTM? Ablation study would be great.
- Did the author pre-train ViT-B from the scratch? or initialized from some weights?
- Ablation on number of epochs during pre-training is missing
- Only ViT-B is used in the experiment? Different architecture should be studied too.
- Did the authors also pre-train VideoMAE and other baselines using your dataset? Can the authors show some results using kinetics pre-trained SSL weights? 
- More baselines should be compared with. For example, MME[3], AdaMAE[4], and other masked video modeling approach.
- What did the author use as an evaluation or metric to stop pre-training?
- Can FAGTM be used on local views?
- The approach looks very sensitive to $\gamma$. Table 3 shows the impact of it on all the tasks.
- Is the masking ratio used for FAGTM and random tube masking same? Ablation study on the impact of different masking ratio for each of the masking strategy would be useful.
- The paper only uses linear layer for the reconstruction objective. Study of different decoders would be helpful. In masked video modeling, most of the approaches used asymmetric encoder and decoder design. It would be great to pre-train baselines like VideoMAE with linear layer decoder for a better comparison.
- Most of the recent approaches use L2 loss for the reconstruction objective. Comparison of L1 and L2 loss is missing and can the authors please give a remark on why L1 loss is preferred?
- Given the approach is mostly empirical with limited novelty, the performance on other datasets like Cholec80, a surgical phase recognition benchmark dataset, would be great. 

- There are some typos in the paper: section 3.2.1 'spatiotempora' -> temporal, table2 'gloabl' -> global

[1] What to Hide from Your Students: Attention-Guided Masked Image Modeling, ECCV 2022

[2] Good helper is around you: Attention-driven Masked Image Modeling, AAAI 2023.

[3] Masked Motion Encoding for Self-Supervised Video Representation Learning, CVPR 2023

[4] AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoders, CVPR 2023
questions: questions: Please see the weakness section for the questions and suggestions.
limitations: limitations: Yes, the authors have adequately addressed the limitations.
paper_summary: paper_summary: The paper proposes a representation learning framework that combines masked pretraining strategies with contrastive learning methods. Particularly, this framework aims to generate a representation learning approach that can work with downstream tasks requiring dense pixel-level representations (for image segmentation) or discriminative features (for image classification). The masking strategy is guided by the aggregation of the attention layers of a teacher model over different frames of the endoscopic video, while the contrastive learning framework follows a self-distillation approach. Testing is performed on colonoscopic datasets and on three downstream tasks, including classification, detection, and segmentation.
strength_weakness: strength_weakness: * Downstream tasks are evaluated on a single endoscopic modality

* Similarly, a cross-validation like approach that keep the downstream tasks but change the testing dataset, could help to support the results.
questions: questions: Most of the datasets, including the downstream tasks, are related to colonoscopy. Given the intention to develop a method to operate with endoscopy in general, it would be interesting to see the performance in downstream tasks that involve other endoscopic modalities. 

It appears that the combination of masking and contrastive learning is more beneficial for the classification and detection tasks, while the improvements for the segmentation tasks are still at a similar level as previous works. Would it be possible to elaborate on why the segmentation tasks might not take significant benefits compared with the other tasks?
limitations: limitations: Limitations are discussed in the supplementary material of the paper.
paper_summary: paper_summary: This work presents M$^2$CRL, a self-supervised learning method for representation learning of endoscopic videos. The method leverages a multi-view masking technique with attention-guided masking of global features and random spatiotemporal tube masking of local features. Both contrastive learning and masked autoencoding pretraining objectives are employed for stronger representation learning. The model is pretrained on 7 publicly available endoscopy video datasets and fine-tuned on 3 other datasets on a variety of tasks, outperforming other relevant baselines.
strength_weakness: strength_weakness: - Experimental details could be clarified. Were all methods pretrained on the same union of 7 datasets as M$^2$CRL? Was Endo-FM used as is for downstream fine-tuning or retrained on this pretraining dataset? Section 4.2 could clarify these details. Additionally, how were hyperparameters selected and were they the same for each model and task? Appendix A should additionally include fine-tuning hyperparmeters. 
- There are a few instances of confusing passages of writing. See examples in the section below.
questions: questions: - Were all methods (perhaps except EndoFM) pretrained on the same union of 7 datasets as M$^2$CRL? Please clarify this in Section 4.2.
- It appears that the same hyperparameters were used for all methods and datasets – is this true? If so, how were hyperparameters selected? Please include full fine-tuning hyperparmaters in Appendix A alongside the pretraining hyperparameters.
- Several instances of awkward/confusing writing:
  - The passage from L225-233 is confusing, e.g.: “It is worth noting that the two students in Fig. 1(a) share weights, but are actually one student network. Both encoders share the same structure.” Further, perhaps explain why only local views are fed to the student network.
  - L267: “The results of other methods are taken from the comparative method of Endo-FM”. Is this saying these were the baselines considered in the EndoFM paper?

Minor comments:
- I would replace “mask strategy” with “masking strategy” throughout the paper.
- It might be useful to explain in words what Equation 5 represents.
- L275: I would refrain from saying “significant” without a statistical significance test.
limitations: limitations: Limitations are adequately addressed in Appendix E.
paper_summary: paper_summary: The authors propose a self-supervised learning regime for spatio-temporal data called multi-view masked contrastive learning which combines a frame-aggregated attention guided tube mask and a multi-view mask strategy using a student-teacher framework. The learnt representations are evaluated on multiple downstream tasks on publicly available datasets.
strength_weakness: strength_weakness: -	Please provide some more information on the downstream tasks (how many classes, what kind of labels, how are they obtained,…)
-	Source code is not (yet) released.

Minor Remarks:

-	“temporal” is missing the “l” in a few places (e.g. l. 162,208) 
-	“reach” -> “reaches” in l.281
-	I would suggest to remove the particular results from the abstract
questions: questions: - How many classes do the datasets used for evaluations have in the case of classification and segmentation? What are they?
- Are any modifications needed to employ this method to other spatio-temporal data?
limitations: limitations: -	The potential negative societal impact is not sufficiently discussed. In the broad impact statement, the author discuss the need and benefits for their method one more time. I urge them to consider potential *negative* impact on society.

Comments: title: Re: Response to Authors (Part 3/3)
Re: Response to Authors (Part 2/3) (2)
Re: Response to Authors (Part 2/3) (1)
Re: Response to the authors (Part 1/3)
Thanks for the feedback!
Response to Authors (Part 3/3)
Response to Authors (Part 2/3)
Response to the authors (Part 1/3)
Renspose to rebuttal
Thanks for the feedback!
Thanks for the feedback!
Response to Rebuttal by Authors
Response to additional questions (Part 3/3)
Response to additional questions (Part 2/3)
Response to additional questions (Part 1/3)
comment: **Ablation on masking ratio**

We agree your point that a masking rate of 0.95 can also yield comparable performance results. Our statement that excessively high and low masking rates are unfavorable for endoscopic representation learning is a relative observation. Compared to the optimal masking rate, a higher masking rate increases the difficulty of pre-training, hindering the model’s ability to learn effective representations. Conversely, a lower masking rate reduces the challenge for the model, preventing it from fully capitalizing on the benefits of masked reconstruction for extracting latent features from the video.

Although our ablation experiments have shown that a masking rate of 0.95 can achieve comparable performance, its effectiveness in three downstream tasks is lower than that of a 0.9 masking rate. This suggests that at a masking rate of 0.95, the model is placed in a relatively unfavorable learning situation, resulting in suboptimal results.



**Overall**.
We have re-summarized the innovation of our methods with a special emphasis on our research motivation and method design. At the same time, for the ablation experiments, we have addressed each comment point-to-point. If our response has resolved your concerns on our paper, we will greatly appreciate it if you could re-evaluate our paper. We are also willing and ready to engage in discussions, if you have any further questions.
**Approach sensitive to $\gamma$**

The parameter $\gamma$ is the threshold for FAGTM, used to guide the model in sampling visible tokens from high-attention regions. Due to the feature of endoscopic videos, where the camera moves within the body and inter-frame variations are unstable, different regions of the video frames exhibit significant changes. When $\gamma$ = 0.5, the model considers half of the area as high-attention regions and samples visible tokens from them. In this case, the model samples within a more concentrated attention area, which can lead to insufficient capture of the extensive variations in endoscopic videos during reconstruction.
Conversely, selecting a relatively larger high-attention region to sample visible tokens enables the model to better adapt to the significant inter-frame variations in endoscopic videos and facilitates the capture of important content across frames. However, it is important to note that the $\gamma$ value should not be too large, as a higher $\gamma$ value implies sampling visible tokens from a wider area, which could potentially dilute the focus on high-attention regions and consequently impact the model’s learning efficiency. Through ablation experiments, we ultimately chose $\gamma$ = 0.6.
**Initialize ViT-B?**

We can pre-train M$^2$CRL from scratch. As shown in the ablation study Table 10, the results indicate that M$^2$CRL without weight initialization performs slightly worse under the same pre-training conditions. This is because weight initialization accelerates model convergence and enhances model stability. However, for a fair comparison, we followed Endo-FM and used initialized weights.

Table 10. Initialization status.

| **Initialization status** |  **Cla.**  |  **Seg.**  |  **Det.**  |
| :-----------------------: | :--------: | :--------: | :--------: |
|          Random           | 93.4 ± 0.9 | 80.5 ± 0.5 | 83.4 ± 2.8 |
|     Kinetics weights      | 94.2 ± 0.7 | 81.4 ± 0.8 | 86.3 ± 0.8 |



**Pre-training epoch?**

We acknowledge your concerns regarding the pre-training epochs. we aligned with Endo-FM by using initialized weights and the same number of epochs when pre-training M$^2$CRL. Our model achieved excellent performance under these settings, thus demonstrating the comparability and effectiveness of our method.
During the pre-training process, if the training loss does not significantly decrease over multiple epochs or if the loss curve flattens, it suggests that the model may have been adequately trained. We select checkpoints from different epochs based on the convergence of the loss and determine the appropriate number of pre-training epochs by balancing computational time and performance.



**Alation on different architecture?**

We agree with your perspective that pre-training a larger M$^2$CRL model using random initialization or Kinetics weights can achieve great performance. Larger models have more parameters, which enhance their learning capacity, allowing them to capture more complex patterns and subtle differences in the data. To ensure the fairness of our experimental comparisons, we followed the Endo-FM by using weight initialization and achieved superior performance under the same training epochs, which also demonstrates the effectiveness of our method.
However, since Endo-FM does not provide ViT variant weights, we were unable to conduct ablation experiments using Kinetics weights for different architectures. Consequently, we conducted a set of ablation experiments without weight initialization for the backbone, as shown in Table 3. Additionally, in Table 10, we compared our method with random initialization and weight loading. The results indicate that M$^2$CRL with weight loading performs better under the same pre-training conditions, as weight loading can accelerate model convergence and enhance model stability.



**VideoMAE and other baseline?**

We did not use initialized SSL weights for pre-training; but refer to the original setup of the comparison methods.

The results from Table 4 demonstrate that single masked modeling pre-training methods, such as VideoMAE and DropMAE, exhibit strong performance in fine-tuning pixel-level tasks for endoscopic downstream tasks following pre-training on the Kinetics dataset. This can be attributed to the extensive size of the Kinetics dataset, which enables the model to acquire a more diverse range of features and patterns through pre-training. Furthermore, both endoscopic data and Kinetics data are comprised of color images with shared fundamental visual characteristics. As a result, the weights pre-trained on Kinetics can be effectively transferred to endoscopic tasks.

The advantage of masked modeling lies in its ability to capture rich pixel-level information, making it particularly effective for tasks requiring dense pixel-level processing. In contrast, the SVT method in Table 4, which is based on contrastive learning, does not perform as well as masked modeling methods in pixel-level tasks. Our method combines contrastive learning and masked modeling, enabling the model to capture both fine-grained pixel-level features and comprehensive discriminative features. This hybrid pre-training strategy may result in some downstream tasks not performing as prominently as single pre-training tasks. 

Compared to Table 4, our method demonstrates a significant improvement in classification tasks. This is due to the fact that classification is a relatively straightforward structured task. Our model was pre-trained on an endoscopic dataset, which enabled it to effectively learn the features and patterns of endoscopic images. However, segmentation and detection tasks, being more complex in nature, may not exhibit as substantial an improvement under the same training conditions.

Overall, our method achieves robust performance in both structured and pixel-level tasks, demonstrating its versatility and effectiveness in learning both detailed and global features.
Thank you for your thoughtful feedback on our submission. These valuable suggestions have improved the clarity and quality of our work. We have further addressed all comments and questions below.

**Novelty**

We summarized our novelty as follows:

* **Why did we apply masked modeling?** As pointed out by the reviewer, existing self-supervised pre-training methods for endoscopic videos primarily rely on contrastive learning. Contrastive learning naturally endows the pre-trained model with strong instance discrimination capabilities. However, relying solely on contrastive learning is insufficient to capture the fine-grained feature representations required for endoscopic videos. Therefore, we combine contrastive learning with masked modeling to acquire endoscopic video representations that possess both comprehensive discriminative capability and fine-grained perceptive ability, effectively addressing the limitations of contrastive learning in capturing dense pixel dependencies. ***This is our first major contribution.***
* **What is unique about our masking strategy?** We have developed a novel multi-view masking strategy specifically tailored to the characteristics of endoscopic video data to address two key challenges inherent in endoscopic videos. First, the instability of inter-frame variations is a prominent issue in endoscopic videos. These variations are caused by factors such as camera movement, instrument manipulation, and the uneven distribution of the lesion area. However, traditional attention-guided masking strategies fail to adequately address the global spatiotemporal nature of videos, and therefore cannot effectively consider the instability between frames in endoscopic videos. To address this, we designed the FAGTM strategy from global views, which aggregates features across multiple frames to capture comprehensive spatiotemporal information. Second, endoscopic videos often exhibit minimal inter-class differences, where lesion characteristics closely resemble those of the surrounding normal tissue. This requires the model to capture fine-grained pixel-level details to distinguish these subtle differences. Thus, we employed a random tube masking strategy from local views to learn finer local details. The effectiveness of our multi-view masking strategy is demonstrated through ablation experiments. ***This is our second major contribution.***
* **Clinical significance.**  We conducted extensive experiments on 10 endoscopic video datasets to evaluate the performance of M$^2$CRL in comparison to other methods. The experimental results demonstrate the superiority of our method, which holds inherent value in clinical practice. The development of robust pretrained models for endoscopic video analysis through self-supervised learning can effectively support various downstream tasks, ultimately enhancing clinical workflow efficiency. 



**Comparison with more methods**

We pre-trained MME and AdaMAE from scratch before fine-tuning them for the endoscopic downstream task.



**Cholec80**

Regarding the supplementary experiments in Table 6 of Paper [1]: The differences in experimental results arise from the use of different data partitioning methods. The results you mentioned in Table 6 are supplementary experiments for comparison with external methods, following the approach in reference [2], which uses 40 videos as the training set and 40 videos as the test set. In contrast, our method follows the data partitioning method in Table 4 of Paper [1], using 40 videos as the training set, 8 videos as the validation set, and 32 videos as the test set. As shown in the table below, our method achieved optimal performance under this setup. Furthermore, we will include additional experimental results based on the data partitioning outlined in Table 6 in the revised manuscript.

Regarding the baselines of TeCNO and LoViT: Our data partitioning and evaluation metrics followed the methods outlined in Paper [1]. However, due to different evaluation metrics (we use F1 score, while TeCNO uses accuracy), direct comparison with TeCNO is not feasible. Additionally, LoViT utilizes different data partitioning and evaluation metrics from ours, which also prevents direct comparison. We appreciate your reminder. In future research, we will conduct additional experiments, including adding TeCNO and LoViT methods to the comparisons presented in Table 6 of Paper [1], to provide more comprehensive results.

Table 9. Result of surgical phase recognition.

| **Methods** |  **F1**  |
| :---------: | :------: |
|    DINO     |   81.6   |
|   MoCo v2   |   79.6   |
|   SimCLR    |   81.1   |
|    SwAV     |   79.5   |
|   Endo-FM   | 82.2±0.8 |
|  VideoMAE   | 73.7±1.4 |
|  M$^2$CRL   | 85.0±0.4 |

[1].Dissecting Self-Supervised Learning Methods for Surgical Computer Vision. MedIA 2023.

[2].Semi-supervised learning with progressive unlabeled data excavation for label-efficient surgical workflow recognition.  MedIA 2023.
Thank you for recognizing our efforts in the rebuttal. We sincerely appreciate your consideration in raising our paper's score to 7: Accept. Your feedback has been invaluable to our work. Thank you for your support.
$\textbf{Ablation on masking ratio}$

It looks like with 0.95 making ratio for both global and local views yield comparable performance which doesn't correlate when the authors say both excessively high and low masking rates are unfavorable for endoscopic representation learning. 

Given all my above mentioned concerns regarding novelty and experiments, I will maintain my original score of 4.
$\textbf{initialize ViT-B?}$

What would be the performance gap if ViT-B is not initialized with weights? Can we pre-train M$^{2}$CRL from the scratch?

$\textbf{Pre-training epochs?}$

The pre-training epochs is too low considering the we are pre-training a transformer using some proxy tasks. I think merely looking at the loss doesn't really justify to stop the pre-training.

$\textbf{Ablation on different architecture?}$

I think it would be great to pre-train a larger model using M$^{2}$CRL and compare it random initialization or imagenet/kinetics weight. The current set of experiments don't really tell the whole story. The only thing I can infer is the performance of M$^{2}$CRL can be achieved without weight initialization or pre-training.

$\textbf{VideoMAE and other baselines?}$

Did you pre-train them from initialized SSL weights? From the Table 4, these SSL methods achieve the same performance without pre-training on pixel-level tasks like segmentation and detection. M$^{2}$CRL only achieves significant performance on classification task. 

$\textbf{Approach sensitive to $\gamma$}$

There's a drastic difference in performance when using 0.5 and 0.6 values. I understand the performance variability is using smaller or higher value, but performance gap in 0.5 and 0.6 don't correlate with the authors justifications.
I sincerely thanks authors for clarifications and experiments. However, I still have some questions regarding novelty and experiments. I would also suggest the authors to add all new experiments in the revision.

$\textbf{Novelty}$

I agree that for surgical videos, most of the papers have been using contrastive learning for pre-training, but recently, there are a few papers that have studied self-distillation and contrastive leanring. EndoFM uses global and local views and apply self-distillation. [1] comprehensively studies the self-distillation and contrastive learning approaches for endoscopic videos. The only component that differentiates this approach with other methods is applying masked modeling on both global and local views. FAGTM module is mere extension of attention guided masking for images. 

[1] Dissecting Self-Supervised Learning Methods for Surgical Computer Vision. Medical Image Analysis 2023

$\textbf{Comparison with more methods.}$

Thanks for adding more experiments. Did you pre-train MME and AdaMAE from the scratch or did you fine-tune these models for endoscopic downstream task?

$\textbf{Cholec80}$

I see some missing baselines like TeCNO[2], LoViT[3]

[2] Tecno: Surgical phase recognition with multi-stage temporal convolutional networks

[3] LoViT: Long Video Transformer for Surgical Phase Recognition

In the above paper [1], table 6 shows F1 score on Cholec-80 dataset. SimCLR, when fine-tuned with 10 labeled videos, can achieve 85.0 and can go upto 93.6 when fine-tuned will all labeled videos. Same goes for other self-supervised approaches like DINO, MoCo v2 and SwaV with temporal model TCN.
I thank the authors for the response. I will update my score to accept. 
Thanks.
We sincerely appreciate your positive feedback on our manuscript. We're glad our efforts to address your questions were satisfactory. Thank you for your support.
Thank you for recognizing our efforts in rebuttal. Your comments have greatly improved the quality and clarity of our paper. We appreciate your support!
I acknowledge having read the other reviewers' reviews as well as responses by the authors. I thank the authors for the detailed responses addressing everyone's questions and concerns. I will keep my original score.
I acknowledge that I have read the authors' rebuttal and thank them for the clarifications. I will maintain my score.
**W13. Ablation on different masking ratio for each of the masking strategy.**

We conducted ablation experiments on different masking rates for FAGTM (global views) and RTM (local views). As our work is video-related, we aligned our masking rates with those used in other video masking modeling studies. As shown in Table 6, both excessively high and low masking rates are unfavorable for endoscopic representation learning. A high masking rate increases the difficulty of pre-training and hinders the ability of the model to learn effective representations, while a low masking rate reduces the challenge for the model and fails to fully utilize the advantages of masked learning to extract potential features from the video.

Table 6. Masking ratio

| **FAGTM (Global)** | **RTM (Local)** | **Cla.** | **Seg.** | **Det.** |
| :----------------: | :-------------: | :------: | :------: | :------: |
|        95%         |       95%       | 94.0±0.3 | 81.3±0.4 | 85.1±1.1 |
|                    |       90%       | 93.8±0.9 | 80.7±0.7 | 86.2±1.3 |
|                    |       85%       | 93.2±0.7 | 78.5±0.6 | 84.9±2.3 |
|                    |       75%       | 92.6±0.4 | 77.4±1.7 | 85.2±2.1 |
|        90%         |       95%       | 93.8±1.4 | 80.5±0.7 | 85.8±0.9 |
|                    |       90%       | 94.2±0.7 | 81.4±0.8 | 86.3±0.8 |
|                    |       85%       | 93.8±0.8 | 81.4±1.7 | 85.6±2.2 |
|                    |       75%       | 93.2±0.9 | 78.5±1.9 | 84.8±1.3 |
|        85%         |       95%       | 93.2±0.8 | 79.9±0.4 | 83.1±1.5 |
|                    |       90%       | 93.8±0.2 | 81.2±0.2 | 83.8±0.9 |
|                    |       85%       | 94.0±0.4 | 80.5±1.0 | 85.1±1.8 |
|                    |       75%       | 92.5±1.2 | 79.6±0.7 | 83.8±2.5 |
|        75%         |       95%       | 91.7±1.3 | 76.8±1.8 | 84.2±2.1 |
|                    |       90%       | 91.3±0.3 | 79.0±2.2 | 84.0±1.3 |
|                    |       85%       | 91.8±0.2 | 77.5±1.9 | 83.8±0.4 |
|                    |       75%       | 91.2±0.7 | 74.6±1.4 | 85.0±0.8 |



[1] Dong X, et al. Maskclip: Masked self-distillation advances contrastive language-image pretraining. CVPR 2023.

[2] Gupta A, et al. Siamese masked autoencoders. NeurIPS 2023.

[3] Chen X, et al. Context autoencoder for self-supervised representation learning. IJCV 2024.

[4] Wang Z, et al. Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-train. MICCAI 2023.

[5] Tong Z, et al. VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. NeurIPS 2022.

[6] Caron M, et al. Emerging properties in self-supervised vision transformers. ICCV 2021.

[7] Ranasinghe K, et al. Self-supervised video transformer. CVPR 2022.

[8] Xie Z, et al. SimMIM: A simple framework for masked image modeling. CVPR 2022.

[9] Wu Q, et al. Dropmae: Masked autoencoders with spatial-attention dropout for tracking tasks. CVPR 2023.

[10] Sun X, et al. Masked motion encoding for self-supervised video representation learning. CVPR 2023.

[11] Bandara, et al. Adamae: Adaptive masking for efficient spatiotemporal learning with masked autoencoders. CVPR 2023.

[12] Li K, et al. Unmasked teacher: Towards training-efficient video foundation models. ICCV 2023.
**W8. Pre-train VideoMAE and other baselines using your dataset? Some results using kinetics pre-trained SSL weights.**

Yes, to ensure the fairness of experiments, all compared methods were pretrained on the same union of 7 datasets as our M$^2$CRL. As you suggested, Table 4 presented results using SSL weights pretrained on Kinetics for 3 downstream tasks. We observed that the performance of these methods pretrained on Kinetics is comparable to those pretrained on the endoscopic datasets. This can be attributed to the significantly larger size of the Kinetics dataset compared to the endoscopic datasets. Pretraining models on such a large dataset allows them to learn more generalized features and patterns. Consequently, when these models are transferred to endoscopic downstream tasks, their robust feature extraction capabilities allow them to effectively fine-tune on the endoscopic datasets, thereby adapting efficiently to the new tasks. Moreover, despite the substantial differences in content between endoscopic datasets and Kinetics, both consist of color images, which means they share some similar fundamental visual features such as edges and colors. Therefore, models pretrained on Kinetics can capture these common features to some extent, enabling them to perform well when transferred to endoscopic downstream tasks.

Table 4. Results of using Kinetics pre-trained SSL weights on 3 downstream tasks

| **Methods**  | **Cla.** | **Seg.** | **Det.** |
| :----------: | :------: | :------: | :------: |
|   SVT [7]    | 88.7±0.7 | 74.8±1.1 | 84.8±0.9 |
| VideoMAE [5] | 90.9±0.6 | 81.1±0.3 | 85.5±0.9 |
| DropMAE [9]  | 85.8±0.9 | 81.2±0.4 | 83.2±0.3 |

**W11. Can FAGTM be used on local views?**

FAGTM cannot be used on local views. In our method, the global views are fed into both the teacher and student networks, allowing the teacher to generate attention corresponding to the global views to guide the student model in masking. However, the local views are only fed into the student networks, thus the teacher does not have attention for the local view to guide masking. This approach is inspired by self-supervised visual transformers [6, 7], where a teacher-student framework is used for contrastive pre-training. In this framework, different views of the video are processed by the teacher and student networks, and predictions are made between views in the latent feature space, enabling the model to learn spatiotemporally invariant features.

**W12. The approach looks very sensitive to $\gamma$.**

The parameter $\gamma$ serves as the threshold for FAGTM, which is designed to allow the model to sample visible tokens from its attention regions and mask the rest in a reasonable manner. This ensures that our method can effectively perform reconstruction tasks even at a high masking rate. A lower value of $\gamma$ implies selecting visible patches from a smaller high-attention region, potentially leading to an overemphasis on non-critical areas during reconstruction, thus contradicting the objectives of the self-supervised pretraining task. Conversely, a higher value of $\gamma$ means selecting visible patches from a broader region, which may dilute the focus on high-attention areas and adversely affect the model’s learning efficiency.

**W14. Ablation of VideoMAE using linear layer decoder.**

As you suggested, we evaluated VideoMAE on downstream tasks after pre-training with a linear layer decoder. As shown in Table 7, the impact of the decoder on the experimental results is minimal and almost negligible. The prediction head can be of arbitrary form and capacity, as long as its input conforms with the encoder output and its output accomplishes the prediction target. This has already been validated in SimMIM [8].

Table 7. Ablations on decoder

| **Decoder**  | **Cla.** | **Seg.** | **Det.** |
| :----------: | :------: | :------: | :------: |
| Linear layer | 91.2±0.8 | 81.2±0.3 | 82.6±1.4 |
|  Asymmetric  | 91.4±0.8 | 80.9±1.0 | 82.8±1.9 |

**W15. Why use L1 loss for reconstruction?**

The mask modeling component of our model follows SimMIM [8], which employs the L1 loss function. The study demonstrates that different loss functions have minimal impact. To maintain consistency, we used the same loss function as SimMIM. Furthermore, we conducted ablation studies to demonstrate that different loss functions have a negligible effect on our results.

Table 8. Ablations on loss

| **Loss** | **Cla.** | **Seg.** | **Det.** |
| :------: | :------: | :------: | :------: |
|    L1    | 94.2±0.7 | 81.4±0.8 | 86.3±0.8 |
|    L2    | 93.8±0.7 | 82.0±0.7 | 85.9±1.5 |



**W17. Some typos.**

We apologize for any typos. We will proofread the whole manuscript carefully and make revisions in subsequent versions.
As we mentioned in the main rebuttal, we include the answers to the additional questions of Reviewer wGJR. We hope that this helps to address all remaining concerns and we thank again for taking the time to review our work.

**W3. Why self-distillation is a form of contrastive learning？**

The pretraining tasks for self-supervised learning are summarized into two categories: contrastive and generative. Contrastive methods focus on maximizing the similarity between different views of the same image after augmentation, while also potentially minimizing the similarity between views of different images after augmentation. Self-distillation is optimized by comparing the feature representations of the teacher and student models to extract a closer representation from the same image. In our work, the self-distillation component also serves to align features across different views, which follows the paradigm of contrastive learning. Therefore, self-distillation can be considered a form of contrastive learning. Furthermore, several other studies [1-3] have also classified self-distillation in self-supervised learning as a type of contrastive learning.

**W4. Ablation on teacher’s block used for FAGTM.**

In our study, we used the last layer block of the teacher ViT-B for FAGTM. The higher layer block incorporates lower-level features and object-level semantic information, offering comprehensive and abstract features that are essential for the model. Thus, it effectively guides the student network in masking. Table 1 shows that it is most beneficial for FAGTM to utilize the last layer block of the teacher network.

Table 1. Ablations on blocks

| **Blocks** |   **Cla.**   |   **Seg.**   |   **Det.**   |
| :--------: | :----------: | :----------: | :----------: |
|     4      |   91.8±0.7   |   76.6±1.5   |   83.6±1.1   |
|     8      |   92.5±0.4   |   79.7±2.2   |   84.8±1.0   |
|     10     |   93.9±1.0   |   80.6±0.9   |   85.9±1.7   |
|     12     | **94.2±0.7** | **81.4±0.8** | **86.3±0.8** |

**W5. Whether to initialize ViT-B?**

We pre-trained ViT-B with weight initialization to accelerate convergence and enhance training stability. This way also ensures consistency with the baseline (Endo-FM), which similarly employed weight initialization for pre-training ViT-B.

**W6 and W10. Ablation on pre-training epochs. Evaluation or metric to stop pre-training.**

For a fair comparison, we used the same number of epochs as Endo-FM. As you suggested, we performed ablation studies on different pre-training epochs, as shown in Table 2. While increasing the number of epochs generally enhances performance, excessive training may result in diminishing returns or overfitting, particularly when dealing with a smaller endoscopic dataset compared to Kinetics. This overfitting can impair the model’s generalization. 

During the pre-training process, we stopped pre-training based on monitoring the loss on the training set. If there is no significant decrease in training loss over multiple epochs or if the loss curve becomes flat, it suggests that the model has likely acquired most of the features. At this point, we consider stopping pre-training.

Table 2. Ablations on epochs

| **Epochs** |   **Cla.**   |   **Seg.**   |   **Det.**   |
| :--------: | :----------: | :----------: | :----------: |
|     10     |   84.6±3.1   |   73.8±1.7   |   83.6±1.3   |
|     20     |   92.7±0.4   |   78.2±1.5   |   85.8±2.5   |
|     30     | **94.2±0.7** | **81.4±0.8** | **86.3±0.8** |
|     40     |   93.7±0.6   |   81.0±0.4   |   86.0±1.5   |
|     50     |   94.7±0.9   |   80.9±0.6   |   85.5±1.0   |

**W7. Ablation on different architecture.**

For a fair comparison, we used the weights for initialization as Endo-FM did. However, since this work did not provide weights for ViT variants, we were unable to conduct ablation experiments on different architectures with weight initialization. Consequently, we conducted a set of ablation experiments without weight initialization for the backbone, as shown in Table 3. It was observed that the performance improvement is more pronounced with larger models due to their increased parameters and more complex structures, enabling them to capture more intricate features. In contrast, smaller models have limited feature extraction capabilities and cannot fully extract visual features. Although larger models exhibit stronger learning abilities, they are more prone to overfitting during training. Additionally, larger models require higher computational resources and longer training times. In conclusion, choosing ViT-B as the pre-trained backbone is a suitable compromise.

Table 3. Ablations on architecture

| **Backbone** | **Cla.** | **Seg.** | **Det.** |
| :----------: | :------: | :------: | :------: |
|   ViT-T/16   | 93.4±0.9 | 76.8±1.2 | 76.3±2.4 |
|   ViT-S/16   | 93.8±0.4 | 78.2±1.5 | 79.4±0.7 |
|   ViT-B/16   | 93.4±0.9 | 80.5±0.5 | 83.4±2.8 |
|   ViT-L/16   | 94.0±0.9 | 83.2±0.8 | 84.2±2.0 |