Laparoflow-SSL: Image Analysis From a Tiny Dataset Through Self-Supervised Transformers Leveraging Unlabeled Surgical Video

Authors: karel  moens
jonas  de vylder
matthew b blaschko
Venue: MIDL_io_2024_Conference
Published: 2024-01-29 14:35:13

Abstract: During minimally invasive surgery, surgeons monitor their actions and the relevant tissue through a camera. This provides an ideal environment for artificial intelligence (AI) assisted surgery. For the development of such AI components, the need for expert annotations remains a key bottleneck. In this paper, we study the application of self-supervised learning (SSL) on surgical data. In a self-supervised setting, a representation backbone is trained on information that is inherently present in the data. There is no need for annotations, leaving the backbone free to train on all recordings, not just labeled ones. We leveraged optical flow for weighting pairs in a view-contrastive self-supervised learning loss. Constructed as an Info Noise-Contrastive Estimation (InfoNCE) loss, it contrasted the pixel representations of two differently, photometrically and geometrically transformed views. The importance of each contrasted pixel pair is determined by computing the difference between the optical flows of the respective pixels. In this way, the optical flow guided the representations of pixels that move together to similar vectors. We tested the usefulness of the representation vectors by training simple networks for semantic segmentation or robotic instrument key point detection. These networks showed competitive performance, even when using over 92% fewer annotated samples than other works. For semantic segmentation, we used as little as 99.73% fewer samples for training, originating from the m2caiSeg dataset, and remained competitive even when testing on the unseen cholecSeg8k dataset.

Main paper: 
 introduction: 
 Minimally invasive surgery reduces incisions on a patient, resulting in a shorter recovery time. As an added benefit, it also requires the use of a camera and therefore digitizes the visual inputs to the surgeon. Increasingly relevant in this age, is that the digitized videos allow computers to learn from the surgery. However, deep learning is data hungry, which is problematic because annotation of medical structures is very expensive. With video data always available during minimally invasive surgery, we propose, therefore, to utilize all video data, even unlabeled, for pretraining of deep learning models through two contrastive self-supervised learning signals. First, we leverage photometric and geometric invariance for all video frames. Second, we extract motion information to infer object boundaries and coherence. In what follows, we show that by integrating a representation backbone that was pretrained in such a way, it is possible to construct competitive downstream task specific models with a fraction of the annotated data. Our results indicate that this holds even when these models are tested on images from different medical institutes.Previous works directly combined optical flow in surgical video with semantic masks (Zhao et al., 2020; Sestini et al., 2023; Alapatt et al., 2021) . This strongly couples the training phases and is more restrictive to the task of segmentation of the chosen classes. Others leave motion unused (Ross et al., 2018) . Mahendran et al. (2019) work on natural videos (Van Gansbeke et al., 2021; Du et al., 2022) and regress their representations to a kernel applied to the optical flow, strongly enforcing distances and leaving little room for other learning signals. A typical use of optical flow is for tracking. Takahashi et al. (2023) , Sharma et al. (2022) , Xiong et al. (2021) and Wang and Gupta (2015) construct positive pairs from pixels tracked through time for self-supervision, as opposed to finding pairs within a video frame. Many works have explored weak-or self-supervision for medical imaging (Chaitanya et al., 2020; Hu et al., 2021; der Sluijs et al., 2023; Wang et al., 2022) . To our knowledge, the use of optical flow during self-supervision to contrast parts of images has remained unstudied. Our contributions are the following:• We provide further exploration of self-supervised learning on laparoscopic video.• We propose novel, dense pixel representation contrastive loss functions that take advantage of the availability of surgical video. The optical flow derived from video is transformed into weights that enable mining of positive and negative pixel pairs within video frames. This mining is theoretically compatible with many contemporary methods.• We describe network architectures for downstream tasks that integrate large representation backbones, irrespective of the backbone architecture, and, as a result, only require a tiny annotated dataset for fine-tuning while demonstrating resilience to overfitting.The Method section provides a top-down explanation of our proposed loss function and the chosen network architectures. The Experiments section details the datasets, hyperparameters and measured performance. 
weighted sum of all pixel triplets.: 
 For our loss function, we adapt the InfoNCE formula (Oord et al., 2018) for representations of pixels instead of entire images, by assigning each pixel in an image a positive and negative weight with respect to every other pixel as anchor. These weights quantify how well the pair of pixels fits as a positive pair or a negative pair. The network is rewarded when representations of pairs from the positive set P i,a for anchor pixel a in image x i are similar and when the representations of pairs from the negative set N i,a are dissimilar. As shown in Equation (1), the entire loss term is the mean of InfoNCE over all images x i in the dataset D, applied to all pixel triplets (anchor, positive, negative) multiplied by both the positive and the negative weights. As proposed by Pinheiro et al. (2020) , we give negative similarities in the denominator a factor of 10 more weight to avoid the failure mode of outputting the same representation vector for all input pixels. The next paragraphs provide more details on the similarity function s(u,v)i,a,p and the weights.EQUATIONA common technique in self-supervised representation learning is view contrasting. When applied, two views of a dataset sample are generated so that both views retain their task specific semantics. This is similar to the use of augmentations during training or at testtime. The representations of the two views are contrasted and a loss is calculated to penalize differences, since the task semantics are the same. Inspired by the equivariance described by Cho et al. (2021) , we use photometric and geometric transformations to generate the views. Photometric transformations stimulate invariance to endoscope properties and lighting. Geometric transformations stimulate invariance to viewing angle and instrument pose.Figure 2 (appendix) gives a visualization and more information on the composition of these transformations. Equations ( 2)-( 4) describe the pixel representations of the different views.EQUATIONEQUATIONEQUATIONwith x i an input image, f θ the representation backbone, P and G random photometric and geometric transformations, and k and l indices of pixels in the image. In the implementation of these composite functions, we enforce all ||z|| 2 = 1 to reduce the cosine similarity to a dot product. Notice that the same composite geometric transformation is applied to z(2) i and z(3) i but that the view that passes through the representation backbone f θ is different. The representations are compared through the similarity function from Equation ( 5) with τ 0 = 0.07, as is common for visual InfoNCE (Pinheiro et al., 2020; He et al., 2020) .EQUATION2.3. Optical flow guided pixel representation learning.When perceiving motion, things that move similarly are likely to be correlated, perhaps even part of the same object. This is the reasoning behind the optical flow guided pixel representation learning. Given the optical flow for every pixel in an image, we calculate the Euclidean distances between these vectors as an approximation of belonging to the same object. When the distance between flow vectors is small, it is likely that they are part of the same object. Conversely, when the distance is large, the pair of pixels should be treated as negatives for each other. This hypothesis is visualized in Figures 9 and 10 in the appendix.In our implementation, we use half of the pixels in every frame as positives for a given anchor, and half as negatives, with the median distance as a threshold, Equations ( 6) and ( 7). Of course, within these halves, some pairs of pixels will be more fitting as positives or negatives than others. We therefore calculate the weight of each pair based on the optical flow distance. Simply relying on Euclidean distance for the calculation of positive and negative weights can be problematic. Because some objects move more than others, like instruments in a surgical setting, the distances to the pixels that comprise these objects will be many times larger than between pixels of objects with more subtle motion. As a result, pair weights based on Euclidean distance of flows can collapse into merely contrasting objects with potentially strong motion against everything else. We propose to mitigate this by calculating the relative distance between motion flows. As the distance is then relative to either of the norms of the involved vectors, this distance is no longer symmetric. As described in Equation ( 8), we normalize the Euclidean distances with the norm of the flow vector f i of the anchor pixel in each anchor-positive-negative triplet. A consequence of this decision is that relative distances to anchor pixels that barely move can become very large.That is why a threshold is introduced for ignoring anchor pixels with too little motion. These quasi-stationary pixels still appear as positives and negatives with respect to other anchors, therefore their representations will still be shaped by the optical flow guided loss function.In our experiments, we set this threshold to 1 3 pixel s . Figure 8 in the appendix shows some examples of pixels in the positive and negative set. To bound the weights between [0, 1], we apply the exponential function to the negative of the relative distance. Finally, we introduce temperature parameters τ P and τ N for the positive and negative weight calculations and set these, in line with the median heuristic (Gretton et al., 2012) , to the 25% and 75% quantiles of the relative distances in the unlabeled pretraining set, respectively. As the only certain positive pair, we assign extra weight to the corresponding pixel in the other view. For all reported results of Laparoflow-SSL, this cross-view exact positive match weight is set to 4. Our final loss function is then given by Equation ( 1) with similarity function s (2,3) (Equations ( 5), ( 3) and ( 4)), and w p and w n as in Equation ( 9). In the following equations, i indicates an image and k, l, a, p and n refer to pixels in that image.EQUATIONEQUATIONEQUATIONEQUATION2.4. Network architecture.The proposed self-supervised learning approach is independent of the network architecture.For the construction of pixel representations, the architecture of any semantic segmentation model can function as a starting point. For our experiments, we chose the smallest SegFormer architecture (Xie et al., 2021) . SegFormer is a hybrid, lightweight Transformer architecture that was designed for real-time semantic segmentation. After removing the softmax over the classes in the final layer, the dimensions of the outputs of that layer can be modified to the desired representation dimensions. We use representation of size 256.In our experiments, we freeze the parameters of the representation backbone. When training on a downstream semantic segmentation task, a 2 layer concatenating side-tune network (Zhang et al., 2020) is placed parallel to the chosen backbone architecture (Figure 1 ). For key point detection, a quick exploration indicated that this task benefits more from narrow heads rather than shallow ones. It is outside the scope of our study to implement components included by Du et al. (2018) , such as edge probability maps and bipartite matching. These components are complementary, however, and we expect them to bring further improvements. For additional details or reasons behind the downstream architectures, we refer the reader to Appendix D. 
experiments: 
 3.1. Datasets.3.1.1. Self-supervised representation backbone pretraining.For the training of the representation backbone to extract meaningful pixel representations, we compose a dataset of laparoscopic and robotic surgery videos. The laparoscopic videos originate from the M2CAI 2016 challenge (Twinanda et al., 2016; Stauder et al., 2016) . The RARP-45 dataset (Van Amsterdam et al., 2022) serves as the source for the robotic surgery videos. Since the latter dataset is smaller, we repeat frames extracted from RARP-45 videos 5 times to have more or less equal occurrences of laparoscopic and robotic instruments in our pretraining dataset.Since every medical institute has some differences in optical and hardware stack, we chose to recreate this setting by using one dataset for pretraining and downstream fine-tuning, and another for testing. To train the segmentation heads and models, we used the m2caiSeg dataset (Maqbool et al., 2020) , based on M2CAI. Cholec80 (Twinanda et al., 2016) was chosen as the test domain for semantic segmentation, a popular laparoscopic surgery dataset that includes 80 video recordings. CholecSeg8k (Hong et al., 2020) is a dataset that builds on Cholec80 by adding semantic mask annotations of 13 classes to 8080 video frames of 101 clips extracted from 17 of those videos. Among the 15 videos in M2CAI, only some are shared with Cholec80. More importantly, M2CAI does not contain data from all contributing institutes of Cholec80. m2caiSeg contains 245 annotated images in its training set and 62 in its test set, all extracted from 2 videos. These 2 videos are not included in the pretraining nor are any of their frames annotated in CholecSeg8k.The sets of annotated classes overlap between CholecSeg8k and m2caiSeg, allowing transfer of models trained on this intersection. Important for this study, the intersection includes tissue classes to verify that the self-supervision based on optical flow not only benefits the more noticeably moving instrument classes. We aggregate all instruments into one class and chose to include the gall-bladder, as primary manipulated object of the cholecystectomy, and fat, a tissue type that can appear in several shapes and places in video frames. An additional consequence of the transfer between datasets is that the annotators for the training and test set are most likely different.As a second downstream task, different from semantic segmentation, we chose pose estimation of robotic instruments. During our experiments, we evaluate on a refined version of the EndoVis15 dataset (Du et al., 2018; EndoVis, 2015) . From its 4 videos for training and 6 videos for testing on robotic instruments, we get 940 and 904 annotated frames respectively. (Hatamizadeh et al., 2022) .88 .71 .42Tables 1 and 2 list semantic segmentation and key point detection results respectively. Segmentation models built on Laparoflow-SSL are competitive, despite using annotated datasets of less than 0.27% the size, from different institutes than the test set. As proven by the entry with Cholec80 pretraining, even better performance can be achieved when one can also collect unlabeled videos from other domains than the annotation source. Stronger augmentations during pretraining can partially imitate this, though this is more effective when there is no domain transfer between train and test set, as can be seen when comparing the entries from Table 1 and Table 4 in the appendix.For fat tissue, a gap still remains between the tiny annotated dataset and the full dataset setting. As patches of fat can be scattered and relatively small, the image resolution of 256 × 256 for these experiments leading to 64 × 64 pixel representations might be too rough. Another explanation is that this tissue type is not as coherent or rigid as instruments and organs. This is supported by the performance of pretrained models that did not use any optical flow. The larger drop in instrument or gall-bladder Dice compared to fat tissue, indicates that the latter benefits less from the learning signal derived from optical flow. Furthermore, it is remarkable that the performance degrades when all 245 images from m2caiSeg are used, even for the models without SSL pretraining. At the time of writing, we were unable to collect conclusive evidence for any explanation of this phenomenon.When evaluating on robotic instrument pose estimation, simple networks trained on top of the representations are competitive with previous work. The advantage of pretraining is most apparent in a setting with few annotated images. Some access to the pixel values, such as through a side-tune network, seems to allow the networks to place their outputs closer to the intended ground truth. 
conclusion: 
 We have shown that it is possible to pretrain pixel representation backbones for laparoscopic surgery in a fully self-supervised manner leveraging the availability of video. By building downstream models on top of these representation backbones, we achieve competitive results while requiring an order of magnitude less data. For semantic segmentation, this applies to both the instrument class and the tissue classes, even with only 16 annotated images for fine-tuning. This proves that self-supervised pretraining on laparoscopic video is a viable strategy to reduce the cost of collecting an annotated dataset. Our approach can perform inference in real-time and is architecture independent, which makes it complementary to future improvements on segmentation models and fine-tune protocols. (2)i ) and two geometric transformations (G(1) i and G(2) i ). The geometric transformations are also applied to the optical flow frame for easy indexing. Consecutively applying two geometric transformations, instead of one entirely independent transformation, leads to every representation z (2) (bottom path) having a counterpart z (1) (middle path). These counterparts are easily accessed by applying G(2) i on the tensor of z (1) representations, resulting in z (3) representations. Note how the color of the outline of the points indicates the last applied geometric transformation. For less reliance on optical flow, the distance between pixels can also be used to choose negative pairs. See Section C.2 in the appendix.the case in our experiments, this makes it viable to forgo augmentations during downstream training and simply precompute the representations of the annotated training data. The result is experimental runs that take up very little GPU memory and converge in a matter of minutes. Developers and researchers can quickly test several ideas in parallel and get feedback for an iterative approach. As an extension, these lightweight experimental runs can make grid search for hyperparameters or network architectures more affordable in time and hardware. 
 
 Appendix: B.1. Optical Flow.For pretraining, we compute optical flows and reduce redundancy by only keeping frames at 5 fps, resulting in an unlabeled dataset of 157450 images. The optical flows are computed from the videos by the NVIDIA DeepStream element Gst-nvof (NVIDIA, 2023) at the highest quality preset. To test for advantages when, in practice, one has access to a wider range of unannotated video sources, we also pretrained a backbone on the 13 M2CAI videos as well as 13 randomly selected videos from Cholec80 that were not annotated in CholecSeg8k ('LSSL Cholec80 pretrain' entry in Table 1 ), resulting in a total of 327100 images in the pretraining set.B.2. Augmentations during pretraining.(a) (b) (c) (d ) (e) (f ) (g) (h) (i ) (j ) (k ) (l ) (m) (n) (o) (p) (q) (r ) (s) (t) (u) (v ) (w ) (x )Figure 3 : Some examples of views that were constructed through our augmentation pipeline.The rows correspond to the paths in Figure 2 . The middle row (i -p) shows the view resulting in representations z (1) . The top row (a -h) shows the corresponding geometrically transformed optical flow frames. With the flow colors encoded in HSV, the hue represents the flow direction and the brightness the flow magnitude. The bottom row (q -x) shows the view resulting in representations z (2) . Because of the composition of geometric transformations, every point in the bottom row can also be found in the middle row.In contrastive self-supervised learning, the usefulness of the representations is heavily dependent on the transformations leading to the two contrasted views. As shown in Figure 2 , when training our representation backbones, the transformations were applied in two steps. First, two different sets of photometric transformations are applied on two copies of an image from the dataset. These transformations are applied on pixel color triplets with values between [0, 255] and include a random brightness shift between -64 and 64 points in RGB space, a contrast change by multiplying RGB values by a random number between 0.25 and 1.75, a saturation change by multiplying the relevant dimension in HSV space by a random number between 0.25 and 1.75, and a hue shift by adding a value between -24 and 24 degrees to the hue channel. Second, both views undergo a different geometric transformation. These transformations can include a flip, a rotation between ] -180, 180] degrees, and a crop up to 150% zoom. The second crop is always taken within the first crop. This ensures a maximum amount of pixels with representations in both views, namely all pixels in the second view. Some examples of these transformations are shown in Figure 3 . The resulting images are 256 × 256 in size.For easy indexing, the optical flow frame and a mask of valid pixels are geometrically transformed along with the video frames. The pixel optical flow vectors themselves do not need to be modified in the current implementation as the training is only influenced by the relative difference between vectors. In an extension, pixels of nearby frames could be contrasted. Properly transforming the flow vectors could make the sampling of such pixel pairs more efficient. Because of the endoscope, pixels are usually captured within a circular mask. Rotations can also introduce pixels with no value. To keep track of these and exclude them in the loss calculation, each video in the datasets has an associated valid pixel mask that undergoes the same geometric transformations.Appendix C. Variations on the loss function.To disentangle the influence of view-contrastive and optical flow guided learning, one can ablate the former in our proposed loss function for representation backbone pretraining. This can be done by using similarity function s (1,1) (Equations ( 5) and ( 2)) instead of s (2,3) in Equation ( 1). Naturally, it then becomes redundant to add the pixel itself to the positive set. The representation of the anchor point is trivially the same and does not need any attracting.A view-contrastive InfoNCE loss term can be constructed to focus only on obtaining invariant representation to endoscope pose, optical settings and lighting conditions. By applying two random pairs of transformations on the same image we obtain two views of that image. The pixel pair consisting of a pixel and its counterpart in the other view is assigned a positive weight of 1. For the negative pairs, one would prefer to use pixels that belong to different semantic classes. This is unknown, however, because we do not make any assumptions about the availability of annotations. To improve the odds of using semantically different negative pairs, we set a radius in the transformed image space. All pixels within the radius of each anchor are too close. while all pixels beyond the radius are assigned a negative weight of 1 (Equations ( 10) and ( 11)). For our experiments, we set a radius of R = 1/3 of the image width. The pixel representations and weights are then combined into a loss term as described in Equation ( 1) with similarity function s (2,3) (Equation ( 5)). In Table 1 , the pretraining for 'View-contrast, no flow' was done using only this loss.EQUATIONEQUATIONC.3. Balancing contrastive loss terms.Although both loss terms can use the same pixel representations, the weights assigned to each triplet of pixels are very different. This affects the scale of the loss terms and the importance of the learning signals when updating the parameters of the network with a sum of loss terms. To balance this, we estimate the difference in scale of the distributions of the weights during training. This is done by summing the total weight of all pixel triplets in the entire batch for each loss term separately. Each loss term is then multiplied by a factor equal to the total of the other loss term divided by the sum of both totals. Mathematically, this is equivalent to a weighted average of the loss terms, weighted by the reverse of their totals. Balancing allows for more interpretable control over the influence of each loss term.Table 3 : Supplemental experiment runs to those listed in Table 1 . Most notably, we list the results here for a run with a DeepLab (Chen et al., 2017) backbone architecture instead of the SegFormer (Xie et al., 2021) architecture that is used in all other instances of Laparoflow-SSL. 'No SSL, with same augmentations' starts with randomly initialized weights and also fine-tunes the weights of the backbone.During its training, the same, relatively heavy augmentations were applied as for the pretraining of the backbones (Figure 3 ). 'Stronger augment LSSL' has the same architecture as 'Laparoflow-SSL' but was built on top of a backbone that was pretrained with even more severe augmentations.# (Zhou et al., 2018) .91 .61 .63 UNETR (Hatamizadeh et al., 2022) .88 .71 .42 DeepLabV3+ (Chen et al., 2017) .86 .62 .60 Appendix F. Visualizations. Figure 9 : These are scatter plots of all anchor pixels in the cholecSeg8k dataset of the 'Instrument' and 'Gallbladder' classes. The position of a dot is determined by the mean values of distances or weights obtained when comparing that anchor pixel to all other pixels in its video frame, with the mean over pixels of other categories on the horizontal axis, and the mean over pixels of the same category on the vertical axis. None of the anchors in the cholecSeg8k dataset fell beneath the motionless threshold. The plots illustrate how using simple Euclidean distance between optical flow vectors would lead to a higher ratio of distractor pixels, i.e. those pixels would cause more attraction between representations of different semantic classes than between representations of the same class. 


Decision: The concerns raised during initial review primarily pertained to over-claiming, insufficient clarity of the writing in various parts, insufficient clarity of the contributions, as well as the lack of strong baselines and competing approaches. 

While the response seems to have satisfactorily addressed the first concerns, the latter - which from my perspective is also the most concerning one, namely the lack of baselines - has not been addressed. One may argue that requesting additional experiments is undesirable (e.g., according to the TPAMI motion in other conferences), and thus, since the reviewers are satisfied this could be acceptable. On the other hand, one may arguet hat without such baselines, the manuscript simply cannot demonstrate the utility of the proposed approach compared to what is already known. 

Since all reviewers now recommend acceptance (2 of whom recommend weak ratings), I am not contradiciting the reviewer verdict but will mention that the above mentioned limitation is, in fact, quite major making this manuscript a lot more borderline than the scores may suggest.

Reviews: review: paper_summary: paper_summary: The work studies the application of self-supervised learning (SSL) in laparoscopic videos. It employs a view-contrastive (photometrically and geometrically transformed views) self-supervised contrastive loss in which the pairs are weighted using optical flow. The SSL-trained model achieves competing performance in downstream tasks (segmentation and key point detection) when fine-tuned on very little annotated data.
main_review: main_review: This work addresses one of the pressing challenges in surgical data science (limited annotated data). While the technical novelty and dataset section was unclear in the initial draft, the revised manuscript has improved the clarity on the technical novelty and dataset usage (for training backbone, segmentation and pose estimation), showcasing the significance of this work: (i) use of optical flow information to mine positive and negative pixel pairs within video frames and (ii) achieving competitive results even when trained on a very small dataset. As such, I am convinced to improve my final rating. Limited SOTA-SSL comparison and ablation studies make it difficult for me to comprehensively benchmark this work. As such, I recommend weak acceptance.
strength_weakness: strength_weakness: 1.	The technical novelty seems limited. It is mostly presented as summations of prior techniques. In addition to prior works on InfoNCE loss (Oord et al., 2018, Pinheiro et al., 2020) and unsupervised learning using invariance (Cho et al., 2021), the use of cross-pixel optical flow for SSL has also been been explored [a]: 

	[a] Mahendran, Aravindh, James Thewlis, and Andrea Vedaldi. "Cross pixel optical-flow similarity for self-supervised learning." Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers, Part V 14. Springer International Publishing, 2019.
2.	Lacks ablation studies on using absolute vs relative distance and w/wo thresholding to show its significance in model improvement.
3.	Lacks benchmarking against any of the other SSL techniques.
questions: questions: 1. Addressing points 1-3 in the detailed comments will significantly improve the clarity of the paper and highlight its advantages. 
2. Ablation studies on using (i) absolute vs relative distance and (ii) w/wo thresholding may highlight the key incremental contributions of this work.
paper_summary: paper_summary: The authors propose a new Self-Supervised Learning method for surgical video. Assuming that pixels/things that move similarly are likely to be correlated or even part of the same object, they propose a novel dense pixel representation driven contrastive loss that leverages optical flow information of different views. Specifically, given an anchor pixel in a view, it is compared to all other pixels in another view by computing the differences between the optical flow of the respective pixels. From these optical flow differences, they can extract a set of positive and negative pixels for the given anchor and finally compute the proposed dense contrastive loss.
main_review: main_review: The authors have adressed most of my concerns about the related works issues and performed the suggested experiment. The revised version is also clearer and better organized. I am increasing my grade to weak accept.
strength_weakness: strength_weakness: * The authors did not respect the mandatory 8 pages limit.
* Even if I am not very familiar with the literature on self-supervised video representation learning, the claim that the authors are the first to study "the use of optical flow during self-supervision to contrast parts of images" seems a bit too strong. For instance, papers [1, 2, 3] seem related to the self-supervised video representation learning based on optical flow. Therefore, I think a more complete related works would be necessary.
* It is more a discussion than a weakness, it has been shown that including more negative pairs when contrasting results in better representations. In the authors' loss formulation, it seems that they use triplets of pixel representations (1 anchor, 1 positive, 1 negative). Therefore, my question is why have authors chosen this kind of contrastive loss instead of contrastive losses that include more negative pairs such as NT-Xent loss [4]?
* It would have been interesting to see a visualization to confirm the main hypothesis that the loss formulation relies on (i.e.: "things that move similarly are likely to be correlated, perhaps even part of the same object"). For instance, for annotated frames with semantic masks, it would have been possible to measure for each anchor pixel to which extent the set of positive pixels contains pixels of the same class (purity metric). I think it would strengthen the paper and would demonstrate the soundness of this hypothesis. 
* The experiments section is a bit hard to follow. I think it would make this section clearer if the authors would make a clear separation between comparisons of the final model + concurrent works and the ablation studies.

References:
* [1] Takahashi, T., Yashima, S., Ishikawa, K., Sato, I., & Yokota, R. (2023). Pixel-Level Contrastive Learning of Driving Videos With Optical Flow. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3179-3186).
* [2] Sharma, Y., Zhu, Y., Russell, C., & Brox, T. (2022). Pixel-level Correspondence for Self-Supervised Learning from Video. arXiv preprint arXiv:2207.03866.
* [3] Xiong, Y., Ren, M., Zeng, W., & Urtasun, R. (2021). Self-supervised representation learning from flow equivariance. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 10191-10200).
* [4] Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020, November). A simple framework for contrastive learning of visual representations. In International conference on machine learning (pp. 1597-1607). PMLR.
questions: questions: Please see the weaknesses section
paper_summary: paper_summary: The authors leveraged optical flow and Info Noise-Contrastive Estimation (InfoNCE) loss to pre-train pixel representation backbones for laparoscopic surgery in a fully self-supervised manner. They evaluate their design by solving downstream tasks on top of these representation backbones for semantic segmentation and robotic instrument key point detection.
main_review: main_review: I would like to thank the authors for their revision and response. The writing issues were fixed in the revision. The authors utilize several techniques for medical images as I shared in my previous review, conduct enough experiments, and share great details in the appendix, I vote Strong accept.
strength_weakness: strength_weakness: (1) The authors did not compare with other state-of-the-art self-supervised training methods in the paper. 

(2) There exists some small writing issues in the paper, please refer to my detailed comments.
questions: questions: It would be great if the authors could compare their method with other state-of-the-art self-supervised training methods. Please fix the small writing issues for the paper.

Comments: comment: Summary:

This manuscript introduces a self-supervised learning (SSL) framework for laparoscopic video analysis, leveraging optical flow and InfoNCE loss to pre-train pixel representation backbones. The primary innovation claimed is the application of a dense pixel representation-driven contrastive loss for better model performance on downstream tasks such as semantic segmentation and key point detection with minimal annotated data.

Strengths:

    The approach to utilize optical flow in SSL for medical image analysis and the proposed contrastive loss methodology are conceptually novel and show promise in addressing the challenge of limited annotated datasets in the surgical domain.
    Evaluation on downstream tasks offers practical insights into the effectiveness of the SSL-trained backbones, showcasing potential real-world applicability.
    The supplementary materials provided, including extensive visualizations and experiment results, enrich the paper's contributions and offer valuable resources for further research.

Weaknesses:

    The paper's novelty is questioned due to its perceived reliance on amalgamating existing techniques rather than introducing groundbreaking methods. A clearer articulation of the unique contributions beyond the combination of established methodologies would be beneficial.
    The absence of direct comparisons with other SSL methods, especially those that inspired the current approach, hampers the ability to accurately assess the proposed method's impact and advancement over the state-of-the-art.

Detailed Comments:

    Technical Novelty and Contribution Clarification: There is a need for a more detailed discussion on how the proposed method diverges from and improves upon existing SSL approaches, specifically in leveraging optical flow for dense pixel representations. Highlighting distinct advantages or innovations could strengthen the novelty aspect.
    Comparative Analysis: Including a comprehensive comparison with other state-of-the-art SSL methods, particularly those from which the proposed method draws inspiration, is crucial. This comparison should aim to clearly demonstrate the proposed method's unique value proposition and its incremental or substantial advancements in the field.
    Writing and Organization Improvements: As previously noted, addressing minor grammatical issues and optimizing the organization of content, such as the sequencing of equations and the presentation of results in tables, would significantly enhance readability and comprehension.
    Addressing Reviewer Concerns: Incorporating detailed responses to the concerns raised regarding the paper's novelty, comparison with existing literature, and minor writing and organizational issues in the rebuttal could provide clarity and potentially sway the preliminary rating.

Preliminary Rating: Borderline

Given the conceptual promise of applying SSL with optical flow in laparoscopic video analysis, the paper holds potential. However, concerns regarding the clear articulation of novelty and the lack of comparative analysis with existing methods suggest that significant revisions are necessary to fully realize and communicate the paper's contributions and impact.

Presentation Recommendation: Poster

Recommended for MELBA Special Issue: No
The authors have adressed most of my concerns about the related works issues and performed the suggested experiment. The revised version is also clearer and better organized. I am increasing my grade to weak accept.
I deleted the last blank page. Thank you for quickly pointing this out.
One small issue to address for the final version, please delete the last blank page (page 25).
## Visualization for the main hypothesis
Thank you for sharing your idea for a kind of purity metric. We measured the optical flow distances and our positive weights for anchor pixels of different classes in the cholecSeg8k dataset. The visualizations were added to the appendix (green text in the temporary pdf). Indeed, they demonstrate the soundness of our main hypothesis. Through optical flow guidance, in the vast majority of cases, pixel pairs with higher positive weights turn out to be part of the same semantic class. Our transformation from Euclidean distances to positive weights further solidifies this characteristic.

## Related work and contributions
We would also like to thank you for bringing these related works to our attention. We included them in our related work paragraph. It's true that optical flow has been used for self-supervised learning before, for example in Wang and Gupta (2015) [1]. All the mentioned works use optical flow to track pixels across video frames. This tracking then produces variations of the same point on an object through time. Forming positive pairs in this manner is relevant and complementary to our work. In fact, we consider it future work to include a term with such positive pairs in the loss function.
The focus of this work is our method that explicitly discovers relationships between pixels within a video frame. We argue that this leads to novel and perhaps more visually distinct positive pairs.

## Discussion on negative pairs
The choice of negative pairs is indeed one more point of further research. Another option is not to include negative pairs at all, for example by regularizing like in VICReg [2].
For this work, we feel that a triplet-loss as a starting point is conceptually easy to grasp. 

Finally, we were able to make changes to fit the text within 8 pages and the tables within the margins.

References:
* [1] Xiaolong Wang and Abhinav Gupta (December 2015). Unsupervised learning of visual representations using
videos. In Proceedings of (ICCV) International Conference on Computer Vision, pages
2794 – 2802.
* [2] Bardes, A., Ponce, J., & LeCun, Y. (2022). Vicregl: Self-supervised learning of local visual features. Advances in Neural Information Processing Systems, 35, 8799-8810.
## Technical novelty
Thank you for pointing out weaknesses in our writing and clarity. We've made changes to the description of the contributions and the related work (green text in the temporary pdf). This should indeed help clarify the novelty and the different approach that needed to be taken to reach our results.
We agree with your advice for the dataset section and have modified it accordingly.

## Effects of side-tune network \& inference performance
Indeed, we view the study of the effects of the side-tune network and other fine-tuning protocols as an interesting follow-up. For this work, however, we feel the focus is on the self-supervised training of the representation backbone. Incidentally, an important advantage of the side-tune network is that the backbone weights remain frozen, just like with visual prompt tuning. Unlike visual prompt tuning, the representation that is fed to the task specific head can be shared among several parallel tasks. To aid readers in considerations related to downstream tasks, we added a section on model size and inference time, as you suggested.

## Comparison between absolute vs relative distances
It's good that you brought this gap in the explanation of the utility of optical flow for contrastive triplet weighting to our attention. To cover the gap, we gathered some statistics on the cholecSeg8k dataset. We added the visualizing scatter plots to the appendix. They show that our weights based on relative distance are more in line with the annotated semantic classes. In this much smaller dataset, none of the pixels had estimated optical flow magnitudes below the threshold. The threshold was necessary, though, to avoid overflow during the processing of the videos for self-supervised pretraining.

We agree, of course, that more ablations and comparisons against other SSL techniques would have been interesting. Our work discovers novel positive pairs for self-supervision. In that sense, it's compatible with many related works. For now, we leave the study of the integration in other methods to future work.
Thank you for acknowledging the novelty of our work. We intend to further clarify this novelty compared to other uses of optical flow to better convey our contributions (green text in the temporary pdf). To support the clarification, we added plots to the appendix that also illustrate the advantage over simply using the Euclidean distance between optical flows in a video frame as a proxy for semantics.

We have fixed the writing issues and grammar errors. Thank you for pointing some out.

Of course, we fully agree that it would be great to show a comparison with other state-of-the-art self-supervised methods. Unfortunately, time or resources are now too limited to apply these methods on our datasets and downstream tasks. We do see it as an advantage that our method is complementary to many other related self-supervised works. We propose a method for discovering positive and negative pixels pairs and this can be integrated in whatever approach that processes these pairs. Naturally, it would be interesting to study the effects of such integration.