DDA: Dimensionality Driven Augmentation Search for Contrastive Learning in Laparoscopic Surgery

Authors: yuning  zhou
henry  badgery
matthew  read
james  bailey
catherine e davey
h  zhou
m  badgery
j  read
c  bailey
  davey
badgery  zhou
bailey  read
Venue: MIDL_io_2024_Conference
Published: 2024-01-14 02:09:36

Abstract: Self-supervised learning (SSL) has the potential for effective representation learning in medical imaging, but the choice of data augmentation is critical and domain-specific. It remains uncertain if general augmentation policies suit surgical applications. In this work, we automate the search for suitable augmentation policies through a new method called Dimensionality Driven Augmentation Search (DDA). DDA leverages the local dimensionality of deep representations as a proxy target, and differentiably searches for suitable data augmentation policies in contrastive learning. We demonstrate the effectiveness and efficiency of DDA in navigating a large search space and successfully identifying an appropriate data augmentation policy for laparoscopic surgery. We systematically evaluate DDA  across three laparoscopic image classification and segmentation tasks, where it significantly improves over existing baselines. Furthermore, DDA's optimised set of augmentations provides insight into domain-specific dependencies when applying contrastive learning in medical applications. For example, while hue is an effective augmentation for natural images, it is not advantageous for laparoscopic images.

Main paper: 
 introduction: 
 Self-supervised learning (SSL) has recently shown its potential for generating representations from large-scale datasets without human supervision (Chen et al., 2020a; Grill et al., 2020; Bardes et al., 2022; He et al., 2022) . In this approach, a model typically conducts representation learning using a data-generated objective on unlabeled datasets. The learned representations can subsequently be transferred to downstream tasks with limited annotations. In medical domains such as endoscopic or laparoscopic surgery, data can be readily generated from surgical recordings. This contrasts with obtaining high-quality annotations, which can be prohibitively expensive in terms of human expert time, particularly for applications like segmentation (Ward et al., 2021) . In such cases, SSL is highly attractive as it allows effective utilisation of unlabeled data, thereby reducing the demand for annotations.Contrastive learning is a type of SSL, where the model minimizes the distance between feature representations of augmented views from the same image and maximizes the distance to different images. This can be achieved either explicitly (Chen et al., 2020a) in the loss Figure 1 : Illustration of differentiable augmentation policy design and an example application of the contrast operation (left), and the comparison of grid search and our DDA framework for contrastive learning (right). function or implicitly (Grill et al., 2020; Bardes et al., 2022) . The augmentation policy that generates augmented views significantly influences the representation learning quality, and consequently the transferred performance on downstream tasks (Wagner et al., 2022; Huang et al., 2023) . Existing augmentation search methods that automatically select the "optimal" policy are usually conducted with supervised learning and require access to annotations (Cubuk et al., 2019; Lim et al., 2019; Ho et al., 2019; Hataya et al., 2020) . Reed et al. (2021) proposed SelfAugment for augmentation policy search in contrastive learning. It requires training additional projectors with proxy SSL tasks for each candidate and uses Bayesian optimisation for selection. It is very time-consuming if the search space is large.Given that the most effective augmentation policy is domain dependent (Xiao et al., 2021; Bendidi et al., 2023) , the transferability of policies that have been developed for natural images (Deng et al., 2009; Chen et al., 2020a; Grill et al., 2020; Garrido et al., 2023) to medical datasets remains uncertain. Van der Sluijs et al. (2023) explored contrastive learning augmentation policies on Chest X-ray datasets by conducting a grid search on a predefined space of 8 augmentations with varying strengths. The 8 2 search space poses challenges for replication due to the exponential growth in computational resources required for SSL (Chen et al., 2020b) . This challenge is further pronounced when extending findings to other medical applications.To address these limitations, we propose Dimensionality Driven Augmentation (DDA) as illustrated in Figure 1 . to streamline augmentation policy selection in contrastive learning. Based on the differentiable augmentation search framework (Hataya et al., 2020) , DDA incorporates the augmentation pool, the number of augmentations in a policy, their probabilities and strengths of application as differentiable parameters. Instead of optimizing performance on downstream tasks, which is often infeasible, we optimise a proxy objective function. DDA considers the optimisation of geometric characteristics of the deep representation in contrastive learning (the intrinsic dimension). As a result, DDA does not require access to an annotated dataset or additional training. We demonstrate that DDA can explore a large search space, encompassing up to 10 8 different choices, with an optimizable augmentation strength.To summarize, the contributions of this paper are:• We propose a novel approach, DDA, to search for optimal augmentation policy in contrastive learning without any additional supervised evaluation (finetuning) on an-notated datasets. DDA effectively identifies a suitable augmentation policy for laparoscopic surgery across various tasks.• Notably, DDA significantly reduces the time required for augmentation search to constant time complexity. For example, navigating a 10 5 search space with DDA only takes 48 hours, which is 10 5 times faster compared to a grid search.• We show that commonly used augmentations for natural images are sub-optimal when used for laparoscopic images. Our selected augmentations provide valuable insights into which techniques are most effective for medical applications. 
method: 
 In this section, we first provide background regarding contrastive learning and its augmentation policy in Section 2.1. Then, we describe the learnable augmentation policy design and an overview of the DDA framework in Section 2.2, followed by a detailed discussion on our proxy objective function in Section 2.3.Contrastive Learning. We consider a typical two-stage semi-supervised setting, a popular SSL application. We are given a dataset, D, comprised of an unlabelled subset D u for contrastive pretraining, and a labelled subset D l for finetuning, whereD = D u ∪ D l .In the first stage, for each x ∈ X in a batch of M images from D u , we generate two positive samples x, x + from the augmentation policy T ( x), and 2(M -1) independent negative samples {x - m } 2(M -1) m=1= {T (x)|x ̸ = x} from the other M -1 images. An encoder f (•) maps input images to the representation space X → R d with the representation z = f (x), and projector g(•) obtains embedding e = g(z) ∈ R e . A classical optimisation objective for contrastive learning (Chen et al., 2020a) is the following,EQUATIONwhere sim(•) is the cosine similarity, and η > 0 is the temperature that controls the smoothness of distance distribution.In the second stage, after the encoder is trained, an additional classifier or segmentation model h(•) can be attached to the encoder as h • f , for simplicity, we denote this as f ′ . The f ′ learns to map the input image to the label space X → Y using {(x, y)} ∈ D l following a typical supervised learning setup by minimizing the following objective,EQUATION)where L is the objective function for supervised downstream tasks (e.g., cross-entropy). Augmentation Policy. The optimal policy for contrastive learning is defined as,EQUATIONwhere D V al is the unseen validation data. Note that the above objective is defined with respect to a specific downstream task. However, in contrastive learning, we are really interested in obtaining the optimal T * that is suitable for a wide range of tasks.Each round of a two-stage learning framework mentioned in the previous section is time consuming. This challenge makes it impossible to apply a thorough grid search, especially for a large search space with a diverse number of operations and choices in each augmentation policy. To tackle these challenges, we propose a streamlined search framework, DDA. Specifically, we setup the augmentation search in a differentiable fashion following existing works (Liu et al., 2019; Hataya et al., 2020) . Let O be a set of K image augmentation operations O k ∈ O : X → X , and X is input space. Each operation O k ( • ; p k , λ k ) has two parameters: the probability p k , and the augmentation magnitude λ k , for applying the operation. For an input image, x ∈ X , the output of applying an augmentation from the set of possible operations O, also known as a sub-policy, depends on weighted sampling. This is defined as, Ō(x; p, λ) ∈ {O k (x; p k , λ k )|k = 1, . . . K}.Let τ be the set of augmentation sub-policies as τ = { Ō1 , . . . ŌNτ } contains N τ consecutive sub-policies. The augmented image is obtained byx ′ = τ (x) = ( ŌNτ • • • • • Ō1 )(x; p n , λ n ), n = 1, . . . N τ .During searching, the output of the n th sub-policy Ōn (x, p n , λ n ) is the weighted sum of all possible operation choices,EQUATIONThe probability p k is convert from learnable real-number parameter, w k , using the softmax function as p nk = σ η (w nk ) = exp(w nk /η) K exp(w nk /η) , with temperature η > 0. Low temperature will generate the distribution of p k as a one-hot like vector.The components of the augmentation policy thus become learnable parameters and can be updated by backpropagation. In this case, our augmentation policy is designed as a block of plug-in-and-play operation layers as T with parameters w, λ. To learn the optimal augmentation policy T * , we further propose a suitable proxy objective in Section 2.3 for the augmentation search, which optimises the contrastive representation on a fixed encoder.Here we provide an overview of the three-step DDA search framework as illustrated in Figure 1 . Firstly, we obtain a contrastive learning encoder f that is trained with the most basic augmentation (e.g. image cropping) with a contrastive projector g. Then, we apply our differentiable augmentation search with the proxy objective explained in Section 2.3 to optimise the parameters of the policy only on the fixed f . Lastly, the optimal policy T * will be applied to re-conduct contrastive learning to obtain g * • f * . We provide a sketch of the pseudocode of the DDA framework in Algorithm 1. in Appendix A.Equation (3) can't be directly optimised due to lacking validation data, and the two-stage training is hard to optimise differentially. We propose a novel objective as a proxy so that the differentiable search only involves contrastive pretraining.In contrastive learning, the deep representation z ∈ R d extracted from a surgical image needs to be distinguishable from the representations of other images. Our focus lies in understanding the local distribution of deep representations in vicinty of a query image. We define a local neighbourhood-of-interest as a d-dimensional sphere with a small radius r centered at the query. Any data within the neighbourhood has a smaller distance than r from the query, and likely has similar visual content as the query. An illustration of the deep representation for a local neighbourhood is shown in Figure 4 , in Appendix A.To evaluate the properties of a query image's neighborhood, one can estimate the growth rate in the number of nearby data points encountered as the distance from the query increases. This growth rate provides an estimation of the local intrinsic dimensionality (LID) of the query data located in the high-dimensional subspace (Houle, 2017) . Intuitively, the LID assesses the effective number of dimensions (the intrinsic dimension) needed to characterize the local neighbourhood of a query point. If the LID equals 2, the local neighbourhood surrounding the query point behaves like it has two dimensions. LID is thus like a complexity measure and assesses the space-filling properties around a query point. In our scenario, we will assess the LID of each point in the deep representation produced by SSL.Let z be the query point, and s denotes any nearby representation of a different image within its vicinity. The distance between z and s is denoted as r s = l(z, s), where l represents a distance measurement. We define the cumulative distribution function (CDF), F (r), of the number of data points concerning the local distance distribution with respect to z as the probability of the sample distance lying within a threshold r, F (r) ≜ Pr[r s ≤ r]. Theorem 1 (Houle (2017)) If F is continuously differentiable at r, then LID F (r) ≜ r • F ′ (r) F (r) .The local intrinsic dimension (LID) at z defined as the limit, when the radius r tends to zero as, LID * F ≜ limr→0 + LID F (r) .In practice, LID * F needs to be estimated, and it is not an integer. For simplicity, for the rest of the paper, we denote 'LID' as the quantity of LID * F .In our scenario, we will wish to optimise the LID of each data sample, encouraging it to be large. Work by Huang et al. (2024) has shown that it is theoretically desirable to optimise the log transform of the LID in deep learning, rather than the LID. Intuitively, if a query point has a large (log) LID, then its neighbourhood requires more dimensions to characterise. This is a desirable property for SSL, where a key objective is to avoid what is known as dimensional collapse (Jing et al., 2022) .This makes LID a suitable proxy objective for the augmentation search. Since 1) it does not require finetuning on downstream tasks, and 2) optimising the LID distribution of samples in the deep space can easily be integrated into a gradient descent framework. Following Huang et al. (2024) , to obtain a representation with higher LIDs, we can optimise the Fisher-Rao distance between the LID of the representations z and a uniform distance distribution (with LID of 1) using the following loss function as,EQUATION)where M is the number of samples in the batch, and LID * F i can be estimated using any popular estimator for LID in the encoder's output representation z.Different from existing work (Huang et al., 2024) that investigated optimisation of model parameters to produce a better contrastive representation, in our work we optimise the augmentation policy with a fixed encoder f . We find this simple but important shift in perspective is highly significant. More concretely, we apply differentiable augmentation search to optimise the parameters of the policy with Equation ( 5). as the objective as shown in Algorithm 2 in the Appendix A. Although it is also possible to apply other proxy functions instead of using LID for DDA, we empirically find that using LID is highly suitable for the augmentation search. To our knowledge, ours is the first work to consider the use of LID as a measure for optimising augmentations. 
experiments: 
 We evaluate the performance of the augmentation policy found by DDA in terms of representation quality following standard evaluation protocols such as linear evaluations and finetuning. We use ResNet-50 (He et al., 2016) and SimCLR (Chen et al., 2020a) as the contrastive learning framework. All hyperparameters closely follow the original papers. We perform the pretraining on two datasets: our private dataset SVHM and the public dataset Cholec80 (Twinanda et al., 2016) . For downstream evaluations, we use the Cholec80 Tool (Twinanda et al., 2016) , CholecSeg8K (Hong et al., 2020) and our annotated private dataset (denoted as SVHM Seg). For the augmentation search, we set N = 5 by default. We search across the following augmentation operations: Identical, Brightness, Contrast, Hue, Saturation, Solarize, GaussianBlur, Posterize, Gray, Sharpness, with further details in Appendix B.2. We set the temperature η in Equation (4). to 0.1. For LID estimation, we use the method of moments estimator (Amsaleg et al., 2018) , with the neighbourhood size 16. We compare with the original augmentation policy used by SimCLR, a Base policy (only with crop and horizontal flip), a randomly generated one using our search space (Random), a manually selected one based on domain experts (Manual ), and SelfAugment (Reed et al., 2021) with differentiable adaptations. For SelfAugment, we use its min-max strategy as it showed the best performance. We use categorical sampling of the final policy as default. Additional results for the Argmax sampling can be found in Appendix B.4. A description of our private dataset, technical details, computing infrastructure and sample source code is provided in Appendix B. The augmentation search using DDA takes 8 hours for our private dataset and 3 hours for Cholec80.As shown in Table 1 , compared against the original SimCLR augmentation, a large 6% improvement can be observed for our DDA on the linear evaluations. The original SimCLR policy performs similarly to the Base policy (crop and resize) on laparoscopic cholecystectomy datasets. For finetuning on segmentation tasks, augmentations found by our method can outperform the original SimCLR policy by 1-3%. This can be considered as a significant improvement in the context of SSL evaluation (He et al., 2020; Bardes et al., 2022) . Without the augmentation search, using randomly selected operations with our search space results in a complete collapse of the representations, where the model outputs a constant vector (Jing et al., 2022) . It has been shown in existing work (Jing et al., 2022 ) that excessive augmentations could cause dimension or complete collapse. It is worth noting that the augmentation policy found by DDA avoids selecting an excessive number of operations by selecting Identical, as shown in Figure 2 (c) and 2(d ). Compared with SelfAugment, our method also demonstrated consistently superior performance. On our private SVHM dataset, the augmentation policy found by SelfAugment caused a complete collpase. For pretraining with Cholec80, DDA outperforms SelfAugment by 13% in the linear evaluations. In this subsection, we investigate the effect of the different numbers of operations in the policy for the representation quality. We also examine the policy found by DDA. As shown in Figure 2 (a) and 2(b), the representation quality evaluated by the linear probing shows a consistent improvement over the augmentation used by SimCLR and the Base policy. Similar results for finetuning can be found in Appendix B.4. This consistent improvement indicates that the commonly used augmentations on natural images appear suboptimal for a medical dataset such as laparoscopic cholecystectomy (LC).To further investigate the difference between the optimal policy for natural images and policies found by DDA, which are suitable for the medical datasets, we plotted the distributions of selected operations in Figure 2 (c) and 2(d ). It can be observed that despite the different pretraining datasets (our private dataset SVHM and Cholec80), the resulting policy is similar. Detailed results for each policy are in Appendix B.3. The policy found by DDA prefers selecting Gaussian Blur, Saturation, Posterize, and Sharpness. One common characteristic regarding these operations is that they do not change the colour profile of the image. In the existing literature, it has been shown that an augmentation resulting in an overlapping view of two different images and semantically similar to each other is beneficial for contrastive learning (Cai et al., 2020; Wang et al., 2022; Huang et al., 2023; Joshi and Mirzasoleiman, 2023) . In the context of LC images, randomly changing the colour profile to other random colours is not ideal for creating such an overlapping view, since the majority of the contents in the images are red. On the other hand, the Gaussian Blur and Posterize could create a blurring effect or decrease the image quality of the image that could easily create an overlapping view. This is because in LC surgery, motion movement of the camera and fog generated during anatomy dissection by diathermy hook can degrade image quality. Similarly, the effect of Sharpness and Saturation could also inherently appear in the dataset. In Figure 3 , we provide a visualisation of augmented images using DDA and SimCLR augmentation, where DDA can create semantically similar views while SimCLR augmentation produces different and unrealistic views. Additional augmented images can be found in Appendix C. In summary, the augmentation policy found by DDA is more effective (verified by linear probing and finetuning) and well suited to the LC dataset. 
conclusion: 
 In this study we introduced DDA, an automatic augmentation search method explicitly tailored for contrastive learning which considers dimensionality characteristics of the deep representation. DDA showcases both efficiency and effectiveness in identifying inherently optimised augmentation policies for laparoscopic images. Beyond its application to laparoscopic images, DDA has the potential to be used in other contexts, particularly in representation learning for medical images. and second (NN-2) nearest data points, respectively. The third nearest neighbour (NN-3) lies on the sphere at the same distance from the query as r.Algorithm 1: Using DDA to obtain a pre-trained model (higher level pipeline)Input: Encoder f , projector g, unlabeled dataset D u , basic augmentation T basic , augmentation policy T w,λ Output: Optimal policy T * w * ,λ * , final encoder pretrained with optimal policy f *Step 1: Conduct contrastive learning for g • f on D u using T basicStep 2: Optimise T w,λ on fixed pre-trained f to find T * w * ,λ * through DDA search (see Algorithm 2)Step 3: Redo contrastive learning for randomly initialised model g • f on D u using T * w * ,λ * to obtain final f *In Figure 4 , we provide an intuitive example of LID, which describes the relative rate at which its cumulative distance function (CDF), the F (r) increases as the distance r increases from 0. We use the representation learned by SimCLR and project it into a 2D space using t-SNE as an example. Considering a radius r for the query point (the red dot), the LID severity and anatomical variability (Madni et al., 2018) , over three major teaching hospitals in Australia, the Epworth HealthCare, St Vincent's Hospital Melbourne public and private sectors. Procedures followed the same supine approach (Carroll, and we only considered data from the initial grasping of the gallbladder up to the cutting of the cystic duct in 1, 960 × 1, 080 pixels.We perform the pretraining with public datasets Cholec80 (Twinanda et al., 2016) , and our private SVHM dataset. Cholec80 has 400,000 frames from 80 videos at 2 fps. We use 70 videos for the pretraining and resevering the rest 10 vidoes for downstream evaluations. For our private SVHM dataset, we use 300,000 images dissected at 4 frame-per-second (fps) from 50 videos. We use image size of 480 × 854 pixels for pretraining with Cholec80 and 432 × 784 for our private dataset. Linear Probing and Finetuning. For downstream evaluations, we use the Cholec80 tool classification dataset (Twinanda et al., 2016) for linear probing, CholecSeg8k (Hong et al., 2020) and our private labelled dataset (SVHM Seg).Cholec80 tool dataset (Twinanda et al., 2016 ) is a multi-class classification task to determine if the sugical tool is presence in the image. We use 70 videos for training and the rest 10 videos for evaluations. There is no overlap between the videos for evaluations and training/pretraining.CholecSeg8k (Hong et al., 2020) is a labeled subset of Cholec80 where 8,080 frames of 480 × 854 at 25 fps are extracted from 17 videos in Cholec80. We removed frames from training set that are presence in the test set to ensure no leakage.Our SVHM Seg dataset is collected from 20 videos and yielded 1,975 frames with a training dataset of 1,583 frames from 16 videos and a test set of 392 frames from 4 videos unseen in the training set. The dataset was annotated with 20 classes and validated by our surgeons, details are in Table 2 . There is no overlap between the SVHM Seg test set, the SVHM Seg training set, or the pretraining SVHM dataset since they are split based on videos (distinctive operating case). For linear probing, we follow the standard protocol that adds a linear classification layer on top of the frozen encoder. Following Ramesh et al. (2023) , for the Cholec80 tool we use weighted binary cross entropy loss. We use SGD as an optimiser with a learning rate of 0.1, weight decay of 1.0 × 10 -4 , and batch size of 256 and 80 epochs.For finetuning on the segmentation task, DeepLabV3+ (Chen et al., 2017) is used for the segmentation head. For all datasets, we use AdamW (Loshchilov and Hutter, 2019) as the optimiser with a learning rate of 0.005, weight decay of 0.05, and batch size of 32 and 100 epochs. We use the same image size as used in pretraining. 
 
 Appendix: We summarized all selected operations for the search space in Table 3 . The augmentation policy used by SimCLR (Chen et al., 2020a) on ImageNet (Deng et al., 2009) is summarized in Table 4 . For easy comparison, we converted the strength used by SimCLR into the same scale as our search space. We perform the search for 10 epochs with a learning rate of 0.01, and Adam (Kingma and Ba, 2014) as the optimiser. This takes around 8 hours for our private dataset and 3 hours for Cholec80.For comparison with the Manual, it is manually selected augmentation based on domain expert. In supervised learning, existing works (Tokuyasu et al.; Silva et al., 2022; Scheikl et al., 2020; Owen et al., 2022) have shown that rotation by 30 degrees, contrast, Gaussian noise, and Gaussian blur are commonly used for supervised segmentation tasks. Based on this domain knowledge, we constructed an additional manual selection policy using these popular augmentations from the literature. We set the probability of applying each augmentation to 0.8. The strength for rotation is 30 degrees, the strength for contrast and Gaussian noise is randomly sampled, and the sigma is set to 0.1 to 2.0 for Gaussian blur based on the settings from the above-mentioned literature.For comparison with the baseline method, the SelfAugment (Reed et al., 2021), for fair comparison and efficiency, we use our differentiable framework instead of the original Bayesian optimisation. This is more efficient because only one additional proxy linear layer is required. We adopt the original proxy SSL task, rotation for the proxy linear layer, and Min-Max (minimize L SS and maximize L N T Xent ) loss objective defined as the following:EQUATIONwhere the L SS is the rotation objective function, and L N T Xent is following Equation (1). More simply, instead of applying Equation ( 5) of the DDA, we apply Equation ( 6) for the SelfAugment to compare within our experiment. All other hyperparameters are kept the same.We summarize the augmentation policy found by SelfAugment in Tables 5 and 6 .Table 3 : List of all image augmentations that the policy can choose from during the search.No augmentation N/A Brightness Adjust the brightness of the image. magnitude of 0 does not modify the input image, whereas magnitude of 1 gives the white image.[0.0, 1.0]Control the contrast of the image. A magnitude of 0 generates a completely black image, 1 does not modify the input image, while any other non-negative number modifies the brightness by this factor.[0.0, 1.0]The image hue is adjusted by converting the image to HSV and cyclically shifting the intensities in the hue channel (H). A magnitude of π and -π give complete reversal of hue channel in HSV space in positive and negative directions, respectively. 0 means no shift.[-π, π]Adjust the saturation of the image. A magnitude of 0 will give a black-and-white image, 1 will give the original image, and 2 will enhance the saturation by a factor of 2. (1) Identical ( 2) Brightness (3) Contrast ( 4) Saturation ( 5) Hue ( 6) Gray ( 7) GaussianBlur ( 8) Posterize ( 9) Solarize (10) Sharpness In this section, we summarize the augmentation policy found by our DDA. For our default choice N = 5, the found augmentation policy is summarized in Tables 7 and 8 . The augmentation policies corresponding to Figure 2 our DDA. Note that the SimCLR augmentation policy also uses sampling from categorical distributions (see Table 4 ).In Figure 11 , we plotted additional results for finetuning on CholecSeg8k and our Private Seg dataset. All details are the same as in Figure 2 . It can be observed that it can either outperform or be on par with the SimCLR policy. We believe that the result in Figure 11 (b) that slightly under-perform to the Base policy is due to the data distribution difference between our SVHM dataset and Cholec80. It is worth noting that slightly under-performing in a few finetuning tasks is common in SSL evaluations, and the main evaluation metric is the linear evaluations (Bardes et al., 2022; Huang et al., 2024) . The goal of the SSL is learning general representations; on average, DDA demonstrates solid improvement over existing methods.For the initial encoder, we used image cropping due to its importance and effectiveness in contrastive learning. This has been studied in existing works on both natural images (Chen et al., 2020a ) and X-ray images (Van der Sluijs et al., 2023) . Other augmentations are also plausible. We performed an experiment with rotation as the initial augmentation choice. The experiment is conducted with the public dataset Cholec80, and the downstream evaluations are the same as the main paper. Results are in Table 11 . It can be observed that using rotation can also outperform the baseline methods. However, image cropping as base for DDA is indeed more effective compared to rotation.Although in this paper, we mainly focused on the laparoscopic images, DDA can also be applied in other domains. In this subsection, we perform an experiment with an X-ray image dataset, CheXpert (Irvin et al., 2019) and the natural image dataset, CIFAR10 (Krizhevsky and Hinton, 2009) . All experimental settings are the same as our main paper. For CheXpert, we removed the operation converting to gray scale from the search space since X-ray images are already gray scale. Results reported using macro area under the ROC Curve (AUROC) with linear probing. For CIFAR10, the search space is the same as our main paper, and results are reported as classification accuacy. Results are in Table 12 Table 12 : For CIFAR10, the results are reported with classification accuracy. For CheXpert, the results are reported as macro AUROC. It can be observed that DDA performs similarly with SimCLR on X-ray images. This does not indicate that DDA is not effective. This is because SimCLR policy is already performing in the optimal range. This dataset is also used by Van der Sluijs et al. (2023) . They conducted a grid search for the optimal augmentation policy on X-ray images. By conducting grid-search, they tried different combinations to find the best one. They reported Macro AUROC with optimal augmentation operations on this dataset is in the range of 68.8 to 73.6. This indicates that the SimCLR augmentation policy behaves differently with Xray images compared to laparoscopic images. This result also indicates that DDA can find suitable augmentation for X-ray images. In the SimCLR paper (Chen et al., 2020a) , the authors also conducted a grid search on CIFAR10 to find the optimal policy. It can be observed that DDA can also find suitable augmentation for natural images.In existing works (Chen et al., 2020a; Van der Sluijs et al., 2023) that conduct a grid search, one needs to perform the pretraining on every possible combination of the augmentation. For DDA, it just needs the pretraining once, regardless of the number of combinations. Although the DDA result might not be the best one, it is very close to the best-performing one. The efficiency of DDA makes it very suitable for other medical images, which often have different characteristics. https://github.com/kornia/kornia 


Decision: After revision, there is clear reviewer agreement (reviewer wQKj updated through private channels) that the strengths of the manuscript now clearly outweigh the weaknesses and that the paper can be accepted at MIDL. 

However, the response to reviewers included several additional experiments and results, and so it is mandatory that these new findings be included in the final manuscript.

Reviews: review: paper_summary: paper_summary: This paper proposes a learnable framework for finding optimal augmentations, to be later used for self-supervised contrastive learning. This learnable framework enables to cover and test optimal set of augmentations (with their probabilities and parameters). This is done by minimizing over the augmentation set the _local intrisic dimension_ of the embedding,



(Note: this was an emergency review.)
main_review: main_review: I thank the authors for their thorough response. Due to big changes in schedule on my end I could not participate in the discussion with the authors, I apologize for that.

I think that the manuscript has quite improved in readability, and the results and method's motivation are quite compelling.

This paper could make an interesting talk, and in any case, I would encourage the authors to explore the other applications (as mentioned in the conclusion) in a journal extension (possibly, the Melba special issue).

Good job, see you in Paris hopefully! :-)
strength_weakness: strength_weakness: - Overall the writing of the paper should be improved, especially in Section 2 (describing the method). See the detailed comments and questions for rebuttal, but the many small imprecision in notation do stack-up and by the end of the section many grey area remains.
questions: questions: - The final framework is not clear to me. If I understood correctly, first an initial encoder $f_\theta$ is trained with only cropping used as augmentation, then the DDA algorithm is used to find an optimal set of augmentations $\mathcal T$, and then a final encoder $f_\theta$ (2?) is trained from scratch, using this time $\mathcal T$. Is this it?
- I'd like to see a discussion on the final "cost" (in time, computational ressources, other) of this proposed method and others. Notably, the LID estimator does not seem trivial either, so is it adding extra computational cost? The authors mention 4 NVidia A100 in the Appendix, which is pretty significant.
- In Algorithm 1, for $\mathcal L$ computation, an `abs` appear in the pseudo code (contrary to Eq. (5)). Why is that?
- It is not fully clear to me why the probability $p$ parameter in replaced by some weight $\boldsymbol w$, which is then put into a softmax (softmax over what? this is not clear). Cannot $p$ be learned directly?
- What is the difference between $f$ and $f_\theta$? 
- What is $g$ (the projector)? Is it to be trained? Where is it defined? Where does it map to? $g: \mathcal R^d \rightarrow ?$
paper_summary: paper_summary: The paper proposes optimizing an augmentation policy from among a set of manually selected possible augmentations, to improve the representation of contrastive self-supervised pretraining. The authors study their proposed method, Dimensionality Driven Augmentation (DDA) with SimCLR method in two surgical datasets, and show that the method mostly improves downstream classification and segmentation compared to randomly sampled augmentations and original SimCLR augmentation.
main_review: main_review: The authors have adequately addressed the comments; the paper is valuable and interesting for the community working in endoscopic datasets, and those working in developing better ways to automatically identify best augmentation policies in contrastive SSL training. I've upgraded the rating.
strength_weakness: strength_weakness: - The paper’s experiments with only SimCLR and only photographic medical image is limiting assessment of how the method’s performance (and assumptions) hold for other popular contrastive learning methods and for radiology images such as X-ray or ultrasound.
- The baseline’s performance depends on the author’s choice of set of possible augmentation. A better baseline would be a set of reasonable augmentations known in the literature for the target dataset (or types of images) coming from domain knowledge (e.g. choosing from semantics-preserving set of augmentations)? Or previous work where optimal performance has been achieved in classification and segmentation through manual selection of sensible augmentations during pretraining of the target dataset.
questions: questions: Comments in the Weaknesses.
paper_summary: paper_summary: An important consideration in SSL is the choice of data augmentation, which often depends on the specific domain. It is unclear whether commonly used augmentation policies for general image types are suitable for surgical applications. Authors propose to automate the search for a high performing combination of augmentations to achieve good contrastive learning results. They name their method as Dimensionality Driven Augmentation Search (DDA). DDA utilizes the local dimensionality of deep representations as a proxy target and explores appropriate data augmentation strategies within the framework of contrastive learning.
main_review: main_review: The paper is well written and the proposed methodology has good theoretical support.

Extensive experiments are presented to help the readers understand and evaluate the presented method.

Some comparisons and results require further explanation to evaluate the performance
strength_weakness: strength_weakness: Supervised baseline performance if not presented. 

Looking at the Figures 5-9 in appendix, it seems like DDA can pick the same augmentation multiple times. This is an advantage of the DDA, but also makes the comparisons against the other methods a bit unfair as they are forced to use harming augmentations. 

Behavior of DDA illustrated in Fig 8-9 in appendix B is confusing for some experiments
questions: questions: Please present the supervised baseline model performance to see where the self-supervised model stands wrt that baseline. 

Is it possible for the authors to also show statistics on which augmentations are picked by the SelfAugment. 

Can DDA work effectively without these initial training on basic augmentations. 

Can the authors conduct an experiment with only using a subset augmentations that they found to be most beneficial. This may beat the purpose of the study a bit as the selection is done manually, but can be a good experiment to understand of the cause of the "Collapses".

Comments: title: Keen request for discussion
Thanks for your feedback
A follow up message
Any additional questions?
Response to Reviewer cd1Y part 2
Response to Reviewer cd1Y part 1
Response to Reviewer wQKj part 4
Response to Reviewer wQKj part 3
Response to Reviewer wQKj part 2
Response to Reviewer wQKj part 1
Response to Reviewer U9nn part 4
Response to Reviewer U9nn part 3
Response to Reviewer U9nn part 2
Response to Reviewer U9nn part 1
Summary of changes in the revision
comment: Dear AC,

We have provided a point-to-point response to each reviewer's concerns. We have yet to receive any feedback from reviewers in the discussion period. We would appreciate it if you could remind reviewers to participate in the discussion. 

Thank you for your attention to this matter.


Authors.
Dear reviewer wQKj,

Thank you so much for recognising our paper and providing us with such a great opportunity to improve it further. Please kindly let us know if we have addressed your questions, and we would be grateful for any further feedback.
Dear reviewer U9nn,

We appreciate your initial comments on our paper. In response to your review, we have conducted additional experiments and incorporated new results in our replies. Hopefully, they have addressed your concerns. Should you have any further questions or require clarification, please kindly let us know. We are more than willing to address them before the discussion concludes. We sincerely appreciate any opportunities you could provide us to improve our work.
Dear reviewer cd1Y,

Thanks again for reviewing our paper. Please kindly let us know if you have any additional questions or require further clarification. We are happy to address them before the discussion ends.
**Q7:** Notation for Section 2.2

**A7:** There is no clash in $\bar{\mathcal{O}}$, $\mathcal{O}$ and $\mathbb{O}$. The $\bar{\mathcal{O}}$ is an augmentation policy, which is the result of applying the augmentation operation $\mathcal{O}$ with probability $p$ (at the bottom of page 3). The $\mathbb{O}$ denotes the set of all possible augmentation operations. We follow existing works [6, 7] that use these notations. The definition at the bottom of page 3 is a simplified case with the augmentation operation $\mathcal{O}_k$ being applied with $p_k$, otherwise the other operations from $\mathbb{O}$.


For the search, we are interested in multiple different augmentations. Eq (4) is the mixing of different augmented images with their corresponding probability into a single output used during the differentiable search. Eq (4) is an approximation of selecting multiple augmentation operations. Each operation $\mathcal{O}_k$ with their probability $p_k$ and strength $\lambda_k$, the final output is $\bar{\mathcal{O}}$, which is defined as their weighted sum (the Eq(4)). This is a standard approximation technique in the differentiable search frameworks [7, 8]. 

During the SSL training, we sample an augmentation operation per layer according to the optimized parameter (using softmax to convert to probability). In other words, the Eq (4) is only used during the search. 

In the revision, we have updated the draft to make this clearer.


---


**Q8:** Theorem 1 and LID definition

**A8:** The $\mathrm{IntrDimF(r)}$ refers to the general definition of LID [9], and it applies to the more general cases. In our case, following existing work [3], we are only interested in a function $F$ that satisfies the conditions of a cumulative distribution function(CDF), which is Theorem 1. We have removed the $\mathrm{IntrDimF(r)}$ in the revision to avoid confusion. 


--- 
**Q9:** Notation for function composition, Eq (5), iterators, and Equation

**A9:** We have updated these typos in the updated revision.

---
[1] Chen, Ting, et al. "A simple framework for contrastive learning of visual representations." ICML 2020.\
[2] Van der Sluijs, Rogier, et al. "Exploring image augmentations for siamese representation learning with chest x-rays." MIDL, 2023.\
[3] Huang, Hanxun, et al. "LDReg: Local Dimensionality Regularized Self-Supervised Learning." ICLR 2024.\
[4] Grill, Jean-Bastien, et al. "Bootstrap your own latent-a new approach to self-supervised learning." NeurIPS 2020.\
[5] Bardes, Adrien, Jean Ponce, and Yann LeCun. "VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning." ICLR 2021.\
[6] Lim, Sungbin, et al. "Fast autoaugment." NeurIPS (2019).\
[7] Hataya, Ryuichiro, et al. "Faster autoaugment: Learning augmentation strategies using backpropagation." ECCV, 2020.\
[8] Liu, Hanxiao, Karen Simonyan, and Yiming Yang. "DARTS: Differentiable Architecture Search." ICLR. 2019.\
[9] Houle, Michael E. "Local intrinsic dimensionality I: an extreme-value-theoretic foundation for similarity applications." SISAP 2017.
Thank you very much for reviewing our paper and the valuable comments. We have prepared a clarification for each of your questions, please kindly let us know if anything is still unclear.

---

**Q1:** The final framework unclear?

**A1:** Yes, your understanding is correct. It's important to note that the initial encoder is essential for extracting representations used in the search process. This component is also integral to the SelfAugment method, which necessitates an additional layer on top of the initial encoder. In contrast, DDA does not require any additional model or layer.

---

**Q2:** Discussion on cost (time, computational resources..)

**A2:** Using our SVHM dataset as an example. For DDA, each round of pretraining takes 40 hours, the search takes 8 hours, and applying the found augmentation for final pretraining takes 40 hours, totaling 88 hours for the augmentation search and contrastive learning pretraining. 

If we consider the total number of possible choices of augmentation policy as $N$, using grid search would cost $N \times (40+2)$, (40 hours for pretraining and 2 hours for finetuning evaluation). For reference, the search space we used is $N=10^5$ without considering the strength parameter.
Within our search space encompassing 10 augmentation operations, identifying a policy with 5 operations on SVHM would demand **4,200,000 hours** using grid search, whereas DDA accomplishes this task in only **48 hours**, speeding up roughly by $10^5$.

Long pretraining time is necessary for SSL rather than DDA. Please refer to the SimCLR [1] paper for more details. DDA significantly speeds up the searching process to a constant 40+8 hours, rather than $N \times 42$, which depends on the search space size.

The LID estimation can be performed with a single matrix multiplication to estimate the pairwise distance. It has been reported in existing work [3] that estimating LID does not introduce computation overheads. Additionally, note that LID estimation only requires use of the data within a batch, rather than use of the whole dataset.


We have described our computational infrastructure to adhere to the MIDL reproducibility guidelines. A100 GPUs are not necessary for running DDA. There are no extra compute requirements beyond SSL pretraining. If a machine is capable of training with SSL, it can also utilize DDA.


---

**Q3:** What is the absolute operation in the loss function in Algorithm 1?

**A3:** It is not necessary to include the absolute in the implementation. We have removed the `abs` from the pseudo-code. 

---

**Q4:** Why the probability p  is replaced by w and need to go through a softmax, rather than being learned directly 

**A4:** The softmax function  $\sigma_\eta(\cdot)$ is to convert the real-number parameters $\boldsymbol{w}$ into a probability distribution. It ensures that the sum of probabilities is equal to 1. The $\boldsymbol{w}$ are learnable parameters, and $\boldsymbol{p}=\sigma_\eta(\boldsymbol{w})$. Softmax is a common procedure for  converting a real numbered value to a probability. For example, the commonly used cross-entropy loss function also uses softmax to transform logits into probabilities. We have refined the description in Section 2.2 for Equation (4).

---

**Q5:** What is the difference between $f$ and $f_{\theta}$

**A5:** Both terms refer to the encoder. $\theta$ is the parameters of the $f$, we omitted the $\theta$ for readability. We have updated the draft to make this simpler. 

---

**Q6:** What is the projector $g$? 

**A6:** The projector $g(\cdot)$ maps the representations $z$ to embeddings $e$. It is a multilayer perceptron (MLP) model. The contrastive learning loss is calculated with embeddings. It is trained during the pretraining. It will be disregarded after pretraining. This is the standard model architecture for SSL [1, 4, 5]. There is an excellent illustration here, a published open-source project for SimCLR: [https://github.com/google-research/simclr](https://github.com/google-research/simclr). In the Figure, CNN is the encoder, and MLP is the projector. 

---
**A5** (continued):

Below tables show the policies found by DDA and SelfAugment in S2. 

The augmentation policy found by DDA (with N = 5) using Cholec80 as the pretraining dataset is shown below.

|                | Augmentations                                                          | Strengths                    |
|----------------|------------------------------------------------------------------------|------------------------------|
| Operation No.1 | GaussianBlur (48%), Identical (23%), Saturation (21%), Posterize (8%) | [0.15,0.16], N/A, 1.23, 1.00 |
| Operation No.2 | Identical (100%)                                                       | N/A                          |
| Operation No.3 | Identical (100%)                                                       | N/A                          |
| Operation No.4 | Identical (100%)                                                       | N/A                          |
| Operation No.5 | GaussianBlur (84%), Sharpness (16%)                                    | [0.84, 1.45], 0.97        |

The augmentation policy found by SelfAugment (with N = 5) using Cholec80 as the pretraining dataset is shown below.

|                | Augmentations                                                       | Strengths                      |
|----------------|---------------------------------------------------------------------|--------------------------------|
| Operation No.1 | Sharpness (96%), Saturation (2%), GaussianBlur (1%), Posterize (1%) | 0.75, 1.25, [0.23, 0.59], 0.68 |
| Operation No.2 | Posterize (68%), Brightness (29%), GaussianBlur (3%)                | 0.78, 0.21, [0.63, 1.01]       |
| Operation No.3 | Contrast (98%), Sharpness (2%)                                      | 0.42, 0.70                     |
| Operation No.4 | Brightness (82%), Contrast (17%), GaussianBlur (1%)                 | 0.39, 0.52, [0.11, 0.12]       |
| Operation No.5 | Posterize (100%)                                                    | 1.00                           |

---

**Q6:** Why the "Random" augmentation model always collapsed. Can the reason be a limited search space? 

**A6:** We speculate that this phenomenon arises from the variance induced by augmentation being greater than the variance stemming from the data distribution itself. A similar observation has been noted in a previous study [5]. In laparoscopic surgery, images typically contain anatomical tissue and organs, where red hues are prevalent while other colors are considerably less common. This suggests a limited variance in the data distribution within the pixel space. Excessive augmentation may lead to an increased variance, potentially triggering collapse. 

To further illustrate this concept, we have included visualizations of randomly selected batches of images from the SVHM dataset with various augmentation policies applied. These visualizations can be found in Appendix C, Figures 11-15, for reference.

---

**Q7:** Behavior of DDA illustrated in Fig 8-9 in appendix B is confusing for some experiments. Why "sharpness" augmentation is picked to be used by DDA after 2 "identical" augmentation steps?

**A7:** It is indeed what DDA chose. An explanation could be the optimizations and initializations of parameters. Different parameter initializations may lead to varying operations in different sequences. However, we have found that DDA consistently tends to select the same set of operations, irrespective of parameter initializations or the number of operations to be selected.

Figures 5-9 were intended to depict the exact augmentation choice found by DDA for varying numbers of augmentation operations ($N$). The statistics of each particular operation are provided in Figure 3 (c-d). These Figures (5-9 and 3) are intended to show which operations are more likely to be selected by DDA; the exact augmentation choices may vary based on the parameter initializations. Statistically, DDA demonstrates a preference for the augmentations depicted in Figure 3 (c-d).

In the revised version, we have updated the captions for Figures 5-9. The x-axis represents the operation, the y-axis represents the index in the sequence of the policy, and the number in each cell indicates the probability, with darker shades (blue) indicating higher probabilities.

--------------------------------------------------------------------

[1] Chen, Ting, et al. "A simple framework for contrastive learning of visual representations." ICML 2020.\
[2] Grill, Jean-Bastien, et al. "Bootstrap your own latent-a new approach to self-supervised learning." NeurIPS 2020.\
[3] Bardes, Adrien, Jean Ponce, and Yann LeCun. "VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning." ICLR 2021.\
[4] Van der Sluijs, Rogier, et al. "Exploring image augmentations for siamese representation learning with chest x-rays." MIDL, 2023.\
[5] Jing, Li, et al. "Understanding Dimensional Collapse in Contrastive Self-supervised Learning." ICLR 2022.
**Q5:** Experiments with the most beneficial subset augmentations

**A5:** Based on our experimental findings with DDA, we have conducted an additional experiment where we exclusively utilize augmentation operations that DDA found to be suitable for laparoscopic images. These augmentations constitute the initial pool for the search and include Identical, Brightness, Contrast, Saturation, Gaussian Blur, Posterize, and Sharpness. We removed the Hue, Solarize, and Gray Scale since they alter color profiles. It's worth noting that the selection of these operations would not have been possible without the utilization of DDA. We refer to the original search space as **S1** and this refined search space as **S2**. 

The results are in the following Table. 

| Pretraining Dataset | Search Space | Augmentation | Linear Prob (Classification) |  Finetune |   Finetune  |
|:-------------------:|:------------:|:------------:|:----------------------------:|:---------:|:-----------:|
|                     |              |              |         Cholec80 Tool        |  SVHM Seg | CholecSeg8K |
|         SVHM        |       -      |     Base     |             60.00            |   57.28   |    56.18    |
|         SVHM        |       -      |    SimCLR    |             59.01            |   57.15   |  **58.71**  |
|         SVHM        |      S1      |    Random    |           Collapse           |    N/A    |     N/A     |
|         SVHM        |      S1      |  SelfAugment |           Collapse           |    N/A    |     N/A     |
|         SVHM        |      S1      |      DDA     |           **65.95**          | **58.29** |    57.86    |
|         SVHM        |      S2      |    Random    |           Collapse           |    N/A    |     N/A     |
|         SVHM        |      S2      |  SelfAugment |           Collapse           |    N/A    |     N/A     |
|         SVHM        |      S2      |      DDA     |             62.23            |   57.57   |    58.51    |
|         ---         |      ---     |      ---     |              ---             |    ---    |     ---     |
|       Cholec80      |       -      |     Base     |             67.59            |   58.29   |    56.02    |
|       Cholec80      |       -      |    SimCLR    |             67.78            |   58.36   |    57.04    |
|       Cholec80      |      S1      |    Random    |           Collapse           |    N/A    |     N/A     |
|       Cholec80      |      S1      |  SelfAugment |             60.24            |   58.41   |    55.86    |
|       Cholec80      |      S1      |      DDA     |           **73.59**          | **59.31** |    59.40    |
|       Cholec80      |      S2      |    Random    |           Collapse           |    N/A    |     N/A     |
|       Cholec80      |      S2      |  SelfAugment |           Collapse           |    N/A    |     N/A     |
|       Cholec80      |      S2      |      DDA     |             72.08            |   58.13   |  **60.51**  |


It can be observed that changing the search space does not affect the performance of DDA. DDA consistently demonstrates significant improvements in linear probing, which serves as the primary evaluation metric for existing SSL works. Furthermore, we investigated the policies selected within search space S2. In S2, DDA demonstrates a preference for selecting Identical, Gaussian Blur, and Saturation augmentations. This choice remains consistent with the selection observed in search space S1, albeit with slight variations in the strength and sequence of operations. Conversely, both Random policies and SelfAugment once again result in complete collapse.
**Q3:** Show statistics on which augmentations are picked by the SelfAugment

**A3:**  The augmentation policy found by SelfAugment (with N = 5) using our private SVHM dataset as the pretraining dataset is shown below.

|                |                          Augmentations                         |        Strengths       |
|:--------------:|:--------------------------------------------------------------:|:----------------------:|
| Operation No.1 |                            Hue (2%)                           |          1.11          |
| Operation No.2 |          Contrast (97%), Hue (2%), Saturation (1%)          |    0.01, 1.06, 1.98    |
| Operation No.3 |                  Brightness (95%), Hue (5%)                  |       0.32, 0.73       |
| Operation No.4 |                           Hue (100%)                          |          1.66          |
| Operation No.5 | Saturation (91%), Hue (7%), Brightness (\%), Contrast (1%) | 2.00, 1.11, 0.71, 0.25 |

The augmentation policy found by SelfAugment (with N = 5) using Cholec80 as the pretraining dataset is shown below.

|                |                         Augmentations                         |        Strengths       |
|:--------------:|:-------------------------------------------------------------:|:----------------------:|
| Operation No.1 |                       Saturation(100%)                       |          2.00          |
| Operation No.2 |                       Saturation(100%)                       |          2.00          |
| Operation No.3 |                          Hue (100%)                          |          1.54          |
| Operation No.4 |               Saturation (89%), Solarize (11%)              |       2.00, 0.03       |
| Operation No.5 | Saturation (54%), Hue (25%), Solarize (14%) Contrast (%) | 2.00, 2.57, 0.79, 0.09 |

We have added these statistics in the paper Appendix B.2. Tables 5 and 6.

---

**Q4:** DDA can pick the same augmentation multiple times, which could be unfair to other methods

**A4:** This is not the case. In our comparison with SelfAugment, we utilized the exact same search space (initial pool of augmentations for selection). However, the outcome of SelfAugment's selected policy resulted in dimensional collapse. This observation also extends to randomly selected augmentations (Random Policy), which are chosen from the same search space.

Both SelfAugment and Random policies have the ability to select identical operations multiple times. However, SelfAug neither selects beneficial augmentations nor identical ones. For the random policy, the repetition of identical operations is naturally a rare occurrence.

For SimCLR, the augmentation is the one selected by the authors of the original paper, who conducted a grid search using natural images. They determined their optimal augmentation policy through supervised evaluation with labeled data. As discussed in Section 2.2, conducting a two-stage grid search can be exceedingly time-consuming and requires additional labeled data. To provide context, within our search space comprising 10 augmentation operations, identifying a policy with 5 operations on SVHM would demand $10^5 \times 42 = 4,200,000 $ hours (42 hours for contrastive pretraining and finetuning) using grid search. Given the constraints of time and computational resources, it is impractical for us to replicate the grid search on our datasets.
 
In summary, we did not force any baseline to select harmful augmentations. 

---
Thanks for your insightful reviews. Please find our response to your questions below:

---

**Q1:** Supervised baseline performance

**A1:** We have revised the draft to incorporate the results of supervised learning (i.e., without self-supervised learning pretraining). In the Cholec80 Tool classification results, for the supervised baseline, we employed end-to-end supervised training instead of linear probing. It's worth noting that linear probing is the standard evaluation protocol typically reserved for self-supervised learning [1, 2, 3]. Please find the results below:

| Pretraining Dataset | Augmentation | Linear Prob (Classification) | Finetune | Finetune |
|:-------------------:|:------------:|:-------------------------------:|:-----------:|:-----------:|
|                     |              |          Cholec80 Tool          |   SVHM Seg  | CholecSeg8K |
|  None (Supervised)  |  Supervised  |              49.95              |    55.77    |    57.79    |
|         SVHM        |    SimCLR    |              60.00              |    57.28    |    56.18    |
|         SVHM        |     Base     |              59.01              |    57.15    |  **58.71**  |
|         SVHM        |    Random    |             Collapse            |     N/A     |     N/A     |
|         SVHM        |  SelfAugment |             Collapse            |     N/A     |     N/A     |
|         SVHM        |      DDA     |            **65.95**            |  **58.29**  |    57.86    |
|       Cholec80      |    SimCLR    |              67.59              |    58.29    |    56.02    |
|       Cholec80      |     Base     |              67.78              |    58.36    |    57.04    |
|       Cholec80      |    Random    |             Collapse            |     N/A     |     N/A     |
|       Cholec80      |  SelfAugment |              60.24              |    58.41    |    57.04    |
|       Cholec80      |      DDA     |            **73.59**            |  **59.31**  |  **59.40**  |

---

**Q2:**  Basic augmentation lacks clarification 

**A2:** The initial encoder plays a crucial role in extracting representations utilized for the search process. This component is also integral to the SelfAugment method, which necessitates an additional layer atop the initial encoder. Notably, DDA does not require any supplementary layer.

Image cropping is selected for its significance and efficacy in contrastive learning, which has been extensively explored in studies on both natural images [1] and X-ray images [4]. Other augmentations are also plausible. We performed an experiment with rotation as the initial augmentation choice. The experiment was conducted with the public dataset Cholec80, with downstream evaluations consistent with those outlined in the main paper. It can be observed that rotation also performs similarly to image cropping as the basic augmentation for the initial encoder, and DDA can outperform the baseline regardless of the initial choice of the augmentation. 

Results are in table below:

| Initial Augmentation | Augmentation | Linear Prob (Classification) | Finetune | Finetune |
|:--------------------:|:------------:|:-------------------------------:|:-----------:|:-----------:|
|                      |              |          Cholec80 Tool          |   SVHM Seg  | CholecSeg8K |
|          N/A         |    SimCLR    |              67.59              |    58.29    |    56.02    |
|          N/A         |     Base     |              67.78              |    58.36    |    57.04    |
|    Image Cropping    |      DDA     |            **73.59**            |  **59.31**  |  **59.40**  |
|       Rotation       |      DDA     |              72.38              |    58.02    |    59.26    |
**Q5:** Ablation on initial condition (base augmentation)

**A5:** We employed image cropping due to its recognized importance and effectiveness in contrastive learning, as evidenced by existing studies on both natural images [4] and X-ray images [3]. Other augmentations are also plausible. We performed an additional experiment utilizing rotation as the initial augmentation choice. The experiment was carried out using the public dataset Cholec80, with downstream evaluations consistent with those outlined in the main paper. 

The results indicate that utilizing rotation can also surpass baseline methods. However, image cropping as base for DDA is indeed more effective compared to rotation. Please refer to the table below for the detailed results:

| Initial Augmentation | Augmentation | Linear Prob (Classification) | Finetune | Finetune |
|:--------------------:|:------------:|:-------------------------------:|:-----------:|:-----------:|
|                      |              |          Cholec80 Tool          |   SVHM Seg  | CholecSeg8K |
|          N/A         |    SimCLR    |              67.59              |    58.29    |    56.02    |
|          N/A         |     Base     |              67.78              |    58.36    |    57.04    |
|    Image Cropping    |      DDA     |            **73.59**            |  **59.31**  |  **59.40**  |
|       Rotation       |      DDA     |              72.38              |    58.02    |    59.26    |

---

[1] He, Kaiming, et al. "Momentum contrast for unsupervised visual representation learning." CVPR 2020.\
[2] Irvin, Jeremy, et al. "Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison." AAAI 2019.\
[3] Van der Sluijs, Rogier, et al. "Exploring image augmentations for siamese representation learning with chest x-rays." MIDL, 2023.\
[4] Chen, Ting, et al. "A simple framework for contrastive learning of visual representations." ICML 2020.\
[5] Tokuyasu, Tatsushi, et al. "Development of an artificial intelligence system using deep learning to indicate anatomical landmarks during laparoscopic cholecystectomy." Surgical endoscopy 35 (2021): 1651-1658.\
[6] Silva, Bruno, et al. "Analysis of current deep learning networks for semantic segmentation of anatomical structures in laparoscopic surgery." 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE, 2022.\
[7] Scheikl, Paul Maria, et al. "Deep learning for semantic segmentation of organs and tissues in laparoscopic surgery." Current Directions in Biomedical Engineering. Vol. 6. No. 1. De Gruyter, 2020.\
[8] Owen, David, et al. "Automated identification of critical structures in laparoscopic cholecystectomy." International Journal of Computer Assisted Radiology and Surgery 17.12 (2022): 2173-2181.
**Q3:** Better baseline of augmentation policies (with domain knowledge)?

**A3:** To our knowledge, there are currently no existing augmentation studies specifically tailored for self-supervised learning of laparoscopic images. Regarding supervised learning, previous works [5, 6, 7, 8] have demonstrated that rotation by 30 degrees, contrast adjustment, Gaussian noise, and Gaussian blurring are commonly used for segmentation tasks. Leveraging this domain knowledge, we constructed an additional manual selection policy using these popular augmentations from the literature. Each augmentation has a probability of 0.8 for application. Specifically, rotation strength is set at 30 degrees, while contrast and Gaussian noise strengths are randomly sampled. For Gaussian blur, sigma values range from 0.1 to 2.0, as per settings outlined in the aforementioned literature.

Please refer to the results below, which also included outcomes from the main paper for reference. From the findings, it is evident that incorporating manually selected augmentations based on domain knowledge does not establish a strong baseline for contrastive learning.

| Pretraining Dataset | Augmentation | Linear Prob (Classification) | Finetune |  Finetune |
|:-------------------:|:------------:|:-------------------------------:|:----------:|:-----------:|
|                     |              |          Cholec80 Tool          |  SVHM Seg  | CholecSeg8K |
|         SVHM        |    SimCLR    |              60.00              |    57.28   |    56.18    |
|         SVHM        |     Base     |              59.01              |    57.15   |    **58.71**    |
|         SVHM        |    Manual    |              47.97              |    54.09   |  57.11  |
|         SVHM        |      DDA     |            **65.95**            |  **58.29** |    57.86    |
|       Cholec80      |    SimCLR    |              67.59              |    58.29   |    56.02    |
|       Cholec80      |     Base     |              67.78              |    58.36   |    57.04    |
|       Cholec80      |    Manual    |              59.00              |    55.12   |    59.01    |
|       Cholec80      |      DDA     |            **73.59**            |  **59.31** |  **59.40**  |


---

**Q4**: Sec 2.3 explain the terms in the context of surgical data

**A4**: We have edited Section 2.3 accordingly, and provided a visualization of the local neighbourhood of a query data projected into a 2 dimensional space (Appendix A, Figure 4.). In the context of contrastive learning, one might view LID as the minimal number of features (dimensions) required to encapsulate the neighborhood of a single medical image in comparison to other images within the representation space. Given that laparoscopic surgery images frequently feature similar organs, instruments, and viewpoints, distinguishing between them becomes challenging. In contrastive learning, negative pairs (augmented views from different images) should exhibit significant dissimilarity. To effectively discriminate between representations of different laparoscopic images, it is intuitive to expect that a greater number of features (dimensions) are necessary, thereby implying that a higher LID is desirable.
**Q2:** Other medical dataset (radiology images such as X-ray or ultrasound)

**A2:** 
Although X-ray data is outside the scope of our study (our focus is on RGB data in the context of laparoscopic surgery), we conducted an additional experiment using an X-ray dataset, the CheXpert [2]. All experimental settings remained consistent with those outlined in our main paper, with the exception of removing the operation converting RGB inputs to grayscale versions from the search space, as X-ray images are inherently grayscale. Please find the results in the table below, which are reported using Macro AUROC (area under the ROC Curve) with linear probing.

| Augmentation | Linear Prob |
|:------------:|:-------------------------------:|
|    SimCLR    |               72.4              |
|      DDA     |               71.5              |

It can be observed that DDA performs similarly (but with a little lower accuracy) to SimCLR on X-ray images. However, this similarity does not imply ineffectiveness of DDA. Rather, it likely suggests that the SimCLR policy is already operating within an optimal range for this type of dataset. Van der Sluijs, Rogier, et al. [3], who also utilized this dataset, conducted a grid search to determine the optimal augmentation policy for X-ray images. Through exhaustive exploration, they identified augmentation policies yielding Macro AUROC scores ranging from 68.8 to 73.6. This variance indicates that the SimCLR augmentation policy behaves differently with X-ray images compared to laparoscopic images. Moreover, these findings suggest that DDA is capable of discovering suitable augmentations for X-ray images.


Additionally, we extended our experimentation to the CIFAR10 dataset, commonly employed for natural images. In the SimCLR paper [4], the authors conducted a grid search to find the optimal policy for natural images. Please find our results below, results are reported with linear probing classification accuracy.

| Augmentation | Linear Prob |
|:------------:|:-------------------------------:|
|    SimCLR    |               92.2              |
|      DDA     |               91.6              |

Again, it can be observed that DDA can also find suitable augmentation (but with a little lower accuracy) for natural images.    Note that SimCLR requires supervised validation data to be used in grid search, whilst DDA does not.

Please note that in conducting a grid search, one needs to perform the pretraining on every possible combination of the augmentation and subsequent supervised finetuning for evaluation. In contrast, DDA needs pretraining only once, regardless of the number of combinations, and doesn’t require any labeled datasets for selection. Although the DDA result might not always achieve the best performance, it consistently approaches the optimal outcome closely. The efficiency of DDA makes it highly suitable for various medical images, which often exhibit diverse characteristics.
Thank you very much for your valuable comments. We hope the following clarifications can help address the raised issues. 

---

**Q1:** Other contrastive learning methods

**A1:** We have added an experiment including an additional popular contrastive learning framework, MoCo [1]. We compare the augmentation policy found by our DDA with the augmentation policy used by MoCo. Please see the results in the following table. We also included the results from our main paper as a reference. 

We use the policy found by DDA in the main paper for MoCo evaluations. It can be observed that the policy transfers well to MoCo on the SVHM dataset by substantially boosting classification accuracy and giving small increases in finetuning performance. On Cholec80, it is marginally less effective for the linear probing. We believe this might be due to the initial encoder used for the search being trained with SimCLR. In the future, we will add another experiment that uses MoCo with basic augmentation for the initial encoder. 

| Pretraining Dataset | Loss Objective | Augmentation | Linear Prob (Classification) |  Finetune |   Finetune  |
|:-------------------:|:--------------:|:------------:|:----------------------------:|:---------:|:-----------:|
|                     |                |              |         Cholec80 Tool        |  SVHM Seg | CholecSeg8K |
|         SVHM        |     SimCLR     |    SimCLR    |             60.00            |   57.28   |    56.18    |
|         SVHM        |     SimCLR     |      DDA     |           **65.95**          | **58.29** |  **57.86**  |
|         SVHM        |      MoCo      |     MoCo     |             53.94            |   57.03   |    58.60    |
|         SVHM        |      MoCo      |      DDA     |           **57.35**          | **57.81** |  **58.91**  |
|       Cholec80      |     SimCLR     |    SimCLR    |             67.59            |   58.29   |    56.02    |
|       Cholec80      |     SimCLR     |      DDA     |           **73.59**          | **59.31** |  **59.40**  |
|       Cholec80      |      MoCo      |     MoCo     |             61.75            |   57.89   |    55.89    |
|       Cholec80      |      MoCo      |      DDA     |           **61.83**          | **58.29** |  **57.20**  |
We appreciate the insightful reviews we have received and have implemented the following changes to the draft based on reviewers' suggestions. These modifications are also reflected in the responses to the reviewers below.

- Results of supervised learning (without pretraining) were added to Table 1, to address concerns raised by wQKj. It clearly shows the benefits of using SSL to enhance performance. 
- Results of manually selected augmentations, based on literature settings [3, 4, 5, 6], were added to Table 1 as a baseline method denoted as *Manual*. This is to address reviewer U9nn’s concern that the baseline should include a set of reasonable augmentations known in the literature. It can be observed that augmentations effective in supervised segmentation for laparoscopic images do not perform well in contrastive learning.
- To address common concerns raised by reviewers U9nn and wQKj, an ablation study on the augmentation choice for the initial encoder was added in Appendix B.5. It is observed that the choice of basic augmentation does not significantly affect the effectiveness of DDA. 
- In Appendix B.6, an experiment was conducted performing the search with an X-ray dataset (CheXpert [7]) and a natural image dataset (CIFAR10 [8]). Results indicate that DDA can find appropriate augmentations. This change is to address the concerns of reviewer U9nn. Although the final performance is marginally lower than with grid search (exhaustive search over all possible combinations), DDA can identify an augmentation policy that closely matches the best one for each dataset without requiring supervised evaluation. Moreover, DDA has considerably lower computational complexity than grid search.
- The augmentation policy found by SelfAugment for the private SVHM dataset was added into Table 5, and those for Cholec80 presented in Table 6, as requested by reviewer wQKj. 
- We provided the visualization of a randomly selected batch of images from the SVHM dataset, showcasing the application of various augmentation policies. These visualizations are available in Figure 12 through 16. This is to address the concern by reviewer wQKj. 
- Captions and axis titles were added to the heatmaps in Figure 6 through 10. This is to address the concern by reviewer wQKj. 
- We updated the typos and suggested notations from the reviewer cd1Y and U9nn.

Finally, we wish to emphasize the efficiency of DDA, distinguishing it from existing methods. Traditional approaches, such as grid search, as employed in studies concerning natural images [1] and X-ray datasets [2], require exhaustive exploration of all possible combinations alongside pretraining. Moreover, selecting the optimal policy entails finetuning the pretrained model on labeled datasets. This process escalates exponentially in duration based on the search space and the number of operations in the policy. 

In contrast, DDA requires only one pretraining session alongside the search, without any prerequisites for labeled dataset evaluation. This results in constant searching time, regardless of the number of potential combinations. To illustrate, within our search space encompassing 10 augmentation operations, identifying a policy with 5 operations on SVHM would demand roughly **4,200,000 hours** using grid search, whereas DDA accomplishes this task in only **48 hours**, speeding up by $10^5$. This underscores DDA's suitability for diverse medical datasets with unique characteristics, as it circumvents the computationally exhaustive nature of grid search.

--- 

[1] He, Kaiming, et al. "Momentum contrast for unsupervised visual representation learning." CVPR 2020.\
[2] Van der Sluijs, Rogier, et al. "Exploring image augmentations for siamese representation learning with chest x-rays." MIDL, 2023.\
[3] Tokuyasu, Tatsushi, et al. "Development of an artificial intelligence system using deep learning to indicate anatomical landmarks during laparoscopic cholecystectomy." Surgical endoscopy 35 (2021): 1651-1658.\
[4] Silva, Bruno, et al. "Analysis of current deep learning networks for semantic segmentation of anatomical structures in laparoscopic surgery." 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE, 2022.\
[5] Scheikl, Paul Maria, et al. "Deep learning for semantic segmentation of organs and tissues in laparoscopic surgery." Current Directions in Biomedical Engineering. Vol. 6. No. 1. De Gruyter, 2020.\
[6] Owen, David, et al. "Automated identification of critical structures in laparoscopic cholecystectomy." International Journal of Computer Assisted Radiology and Surgery 17.12 (2022): 2173-2181.\
[7] Irvin, Jeremy, et al. "Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison." AAAI 2019.\
[8] Krizhevsky, Alex, and Geoffrey Hinton. "Learning multiple layers of features from tiny images." 2009.