Efficient Anatomy Segmentation in Laparoscopic Surgery using Multi-Teacher Knowledge Distillation

Authors: lennart  maack
finn  behrendt
debayan  bhattacharya
sarah  latus
alexander  schlaefer
Venue: MIDL_io_2024_Conference
Published: 2024-02-01 10:33:45

Abstract: Automatic segmentation of anatomical structures in laparoscopic images or videos is an important prerequisite for visual assistance tools which are designed to increase efficiency and safety during an intervention.
In order to be used in a realistic clinical scenario, both high accuracy and real-time capability are required.
Current deep learning networks for anatomy segmentation show high accuracy, but are not suitable for real-time clinical application due to their large size.
As smaller, real-time capable deep learning networks show lower segmentation performance, we propose a multi-teacher knowledge distillation approach applicable to partially labeled datasets.
We leverage the knowledge of multiple anatomy-specific, high-accuracy teacher networks to improve the segmentation performance of a single and efficient student network capable of segmenting multiple anatomies simultaneously.
To do so, we minimize the Kullback-Leibler divergence between the normalized anatomy-specific teacher logits and the respective normalized logits of the student.
We conduct experiments on the Dresden Surgical Anatomy Dataset, which provides multiple subsets of binary segmented anatomical structures.
Results show that our approach can increase the overall Dice score for different real-time capable network architectures for anatomy segmentation.

Main paper: 
 introduction: 
 Postoperative complications remain a major problem for both the healthcare system and the individual patient and are associated with higher healthcare costs and poorer outcomes (Dencker et al., 2021) . An important factor in decreasing postoperative complications is the reduction of technical errors, which are defined as adverse events directly related to manual errors of the surgeon (Fecso et al., 2017; Suliburk et al., 2019) . The increasing adoption of minimally invasive procedures, which rely on the visualization by endoscopic cameras, leads to an increasing amount of available surgical video data. This enables the data-driven analysis of surgical video data using computer vision techniques, such as deep learning, to visually assist surgeons (Mascagni et al., 2022 ). An essential prerequisite for computer vision-based surgical assistance applications is the accurate and real-time perception of the intraoperative environment, e.g. by segmenting the surgical instruments and anatomical structures. To evaluate the accuracy of computer vision-based methods for anatomy segmentation, various data sets with pixel-wise annotations of anatomical structures were introduced (Bamba et al., 2021; Madani et al., 2020; Allan et al., 2020) . As these data sets have little diversity due to their size or are based solely on simpler porcine tissue data, the applicability of deep learning models trained with such data sets in a real clinical setting is limited. Therefore, Carstens et al. (Carstens et al., 2023) published the largest public data set of laparoscopic images to date, namely the Dresden Surgical Anatomy Dataset (DSAD). It is divided into partially annotated sub-datasets, containing overall 13, 195 laparoscopic images with pixel-wise annotations of eleven anatomical structures. As only binary annotations for one anatomy are provided in a sub-dataset, although several other anatomical structures without annotation are visible, complete information about the background remains unknown. To tackle the problem of training networks with partially labeled datasets, the usage of annotation adaptive loss functions has been proposed (Vu et al., 2021; Ulrich et al., 2023) . In their recent work, Kolbinger et al. (Kolbinger et al., 2023) trained a combined network with a shared encoder and multiple decoders for each of the sub-datasets in the DSAD. In addition, they made use of mutual-exclusivity by incorporating the information of a positive annotation of one class as a negative annotation for all other classes. Despite achieving segmentation accuracy comparable to human experts, there are two limitations. First, the binary segmentation accuracy of the combined network is inferior compared to anatomy-specific single-encoder, single-decoder networks (Kolbinger et al., 2023) . Second, due to the large size of the encoder and decoder networks, achieving a sufficiently high frame rate for delay-free segmentation of anatomical structures on dedicated hardware in the operating room can not be guaranteed. Methods for real-time capable segmentation in surgical videos include the development of lightweight convolution-based architectures for faster segmentation of surgical instruments in laparoscopy (Tomasini et al., 2022; Pakhomov and Navab, 2020; Jha et al., 2021) . While networks with a low amount of parameters can provide good segmentation performance for simpler tasks such as surgical instrument segmentation, they are ineffective at learning complex features required for accurate segmentation of anatomies in surgical videos. Knowledge Distillation has drawn attention to overcome the dilemma of decreasing performance when using smaller models capable of faster inference speed. Xie et al. (Xie et al., 2018) transfer the zero-and firstorder knowledge from a strong teacher network to guide the fast student network. Qin et al. (Qin et al., 2021) improve the segmentation performance on public CT datasets by proposing a novel module that encodes regional knowledge for a student network. To overcome the dependency of student networks on a single teacher network, Amirkhani et al. (Amirkhani et al., 2021) include multiple teacher networks trained with the same input but different style transfers and data augmentations. In this work, we propose a multi-teacher knowledge distillation (MT-KD) approach that leverages the knowledge of multiple anatomy-specific, high-accuracy teacher networks to tackle the problem of training a single network with partially labeled datasets. Specifically, we use MT-KD to improve the segmentation performance of a real-time capable student network with a small number of parameters. In a first step, multiple teacher networks are trained to obtain high anatomy-specific accuracy. In a second step, the Kullback-Leibler (KL) divergence is minimized between the normalized output logits of the individ-ual anatomy-specific teacher models and the normalized output logits of the corresponding anatomy-specific decoder of the student model. This way, the teacher networks guide the student to pay more attention to the most salient regions in order to accurately segment the anatomies. By improving the segmentation accuracy of small student networks, capable of segmenting multiple anatomies simultaneously, we aim to increase the applicability of computer vision methods in realistic surgical scenarios. The comprehensive evaluation of our MT-KD approach shows increased segmentation performance across various network architectures with a small number of parameters, as shown in Figure 1 .  
method: 
 Our multi-teacher knowledge distillation approach with its two stages is schematically illustrated in Figure 2 and explained in more detail in the following.During the first stage, D = {A i=1 , A i=2 , .., A i=N } corresponds to the overall data set, whereA i denotes the anatomy-specific data set of the i th anatomy of N anatomies. In the following, only one of the anatomy-specific data sets is considered.Let A be denoted as A = {(x j , y j )} M j=1 , where x j ∈ R 3×H×W and y j ∈ R 1×H×W correspond to the input image and binary segmentation mask, respectively. M denotes the overall number of images in A. Further, a teacher segmentation model T , consisting of one encoder F enc and one decoder F dec , is trained using the standard pixel-wise binary cross entropy loss function formulated as:L CE = - M j=1 H×W k=1 [y j,k log(T (x j,k )) + (1 -y j,k ) log(1 -T (x j,k ))](1)Stage 2:In this stage, the same data set D is utilized as in stage 1. This time, a student segmentation model S that consists of one encoder F enc and N anatomy-specific decoders {F dec 1 , F dec 2 , .., F dec N } is optimized using two different objective functions. For the first objective function, we follow the description from Kolbinger et al. (Kolbinger et al., 2023) . There, the pixel-wise binary cross entropy loss according to Equation 1 is calculated separately using the output probabilities of each of the N anatomy-specific decoders. In detail, the loss is calculated for each pixel, only if the annotated anatomy i in the input image corresponds to the respective anatomy-specific decoder F dec i of S. For all other decoders F dec ̸ =i , only the pixels in x j that belong to the anatomy i are considered for the loss calculation as the false positive class, as shown in Figure 2 . The remaining pixels are not considered for the loss calculation, as only binary segmentation masks are used in DSAD and several anatomies can appear per image and it cannot be ruled out that all other pixels do not contain the anatomy i. The second objective function utilizes the anatomy-specific knowledge of the various teacher models with frozen parameters from stage 1. Similar as in work by Shu et al. (Shu et al., 2021) , where the normalized activations of corresponding channels between the teacher and student network are aligned using the KL divergence, we utilize the normalized output logits of each of the anatomy-specific teacher models and minimize the discrepancy to the normalized output logits of the corresponding anatomy-specific decoder of the student model. As the logits of a well-trained, anatomy-specific teacher model generally show salient anatomical regions, the student model, capable of segmenting multiple anatomies simultaneously, can be guided. This results in overall higher segmentation performance of the student. Let z T i,C and z S i,C be the output logits of the anatomy-specific teacher model T i and the student decoder F dec i of anatomy i, with C being either the anatomy or false positive/background class. First, the output logits z T i,C and z S i,C are divided by a temperature value T and then normalized using the softmax function σ(z) = e z (e z ) . The temperature value T is used to control the softness of the probability distribution. Second, to evaluate the discrepancy between the two probability distributions p T i,C = σ(z T i,C T ) and p S i,C = σ( z S i,CT ), we utilize the KL divergence (Kullback and Leibler, 1951) . The second objective function during stage 2 can therefore be denoted as:L KL i (p S i,C , p T i,C ) = p T i,C • log( p T i,C p S i,C ) (2)With λ as a weighting parameter, the overall objective function of stage 2 can be formulated as: EQUATION 
data: 
 To evaluate our MT-KD approach, we use the DSAD that consists of 13, 195 high-quality laparoscopic images with pixel-wise annotations of eleven intra-abdominal anatomical structures, i.e., abdominal wall, colon, inferior mesenteric artery, intestinal veins, liver, pancreas, small intestine, spleen, stomach, ureter and vesicular glands (Carstens et al., 2023) . We follow the training, validation and test splits as well as the pre-processing steps and augmentations as in the original work from (Kolbinger et al., 2023) .For all experiments with the proposed approach, we utilize the Adam optimizer (Kingma and Ba, 2014), using a learning rate of 5e-4 during stage 1 and a learning rate of 1e-3 for stage 2. Additionally, an exponential learning rate scheduler is used. We train our models for 100 epochs and 60 epochs in stage 1 and 2 and end up with a final learning rate of 1.5e-6 and 2.5e-5, respectively. During the teacher-student knowledge distillation, we follow the implementation details from (Shu et al., 2021) and use the temperature value T = 4.0 for the calculation of the KL divergence and the weighting parameter λ = 3. During all training experiments, a batch size of 8 and an input image size of 640 × 512 is utilized. The utilized convolutional encoder architectures are pretrained on either the COCO or Cityscapes data set (Lin et al., 2014; Cordts et al., 2016) . For the transformer-based segmentation networks, i.e., SegFormer, we use pretrained weights from the Cityscape data set. For the evaluation on the test data set, the model that obtained the best results in terms of the Dice score on the validation set is used. We implement all models in Pytorch1 . 
results: 
 To evaluate the proposed approach in this work, the mean Dice score among all eleven anatomies in the DSAD is used. Furthermore, the number of parameters of each segmentation model as well as the inference time in form of time for computing the segmentation mask of one input image of size 1280 × 720 (High Definition) is determined. In order to assess the improvements of our the approach, we evaluate previous deep learning models applied to the DSAD regarding segmentation accuracy and inference time, as well as another state-of-the-art network for efficient surgical segmentation, as shown in 1 . Both anatomy-specific models with one encoder and one decoder (■), as well as models with one common encoder and multiple anatomy-specific decoders (⋆) are evaluated. From the results in Table 1 , we observe that segmentation models trained on each anatomy separately outperform networks with the same architecture but using a common encoder and eleven anatomy-specific decoders in terms of Dice score. However, the number of parameters increases significantly when using multiple anatomy-specific models which leads to low inference speed. Furthermore, the results show superior performance in terms of Dice score when DeepLabv3 is used with EfficientNet as an encoder compared to ResNet encoders. Small network architectures such as MiniNetv2 enable real-time capabilities, but show significantly lower segmentation performance. The transformer-based architectures SegFormer achieves higher segmentation accuracy in comparison to convolutional-based DeepLabv3 networks. Our proposed MT-KD approach increases the segmentation performance of both convolutional-based segmentation networks and transformer-based segmentation networks. The most significant increase due to our MT-KD approach can be shown for small models, i.e. MiniNetv2. In this case, the Dice score increases from 36.1% to 60.5%. Qualitative results in Figure 3 show more accurate segmentation of anatomies for MiniNetv2 when trained with our MT-KD. Especially for smaller details, the segmentation accuracy can be increased by using large and accurate teacher models. A uniform increase in segmentation performance can be observed across all eleven anatomies. The specific segmentation results in terms of Dice score for individual anatomies can be found in the Appendix. 
discussion and conclusion: 
 Current deep learning networks for anatomy segmentation suffer from two problems. Either they show good segmentation performance, similar to human experts, but are too large for real-time applications or they are efficient enough for real-time applications but do not show sufficient performance for high-accuracy segmentation. In this work, we propose a multiteacher knowledge distillation (MT-KD) approach that leverages the knowledge of multiple anatomy-specific, high-accuracy teacher networks to tackle the problem of training a single and efficient network with partially labeled datasets. By minimizing the discrepancy between the normalized logits of anatomy-specific, high-accuracy teacher networks and a single and efficient student network, the segmentation accuracy of various small, real-time capable network architectures is improved while retaining high inference speed. Our results demonstrate highest segmentation performance with 66.3% and 69.7% Dice score for anatomy-specific, high capacity teacher networks such as DeepLabv3/Eff.Netb3 ■ and SegFormerB3 ■. We assume higher capacity to learn complex, anatomy-specific features. The results further demonstrate that anatomy-specific, low capacity networks such as MiniNetv3 ■ only achieve an overall Dice Score of 43.2%, failing to learn valuable features in the data. For application in realistic surgical scenarios, anatomy-specific models need to be operated sequentially in order to segment several anatomies. A combined architecture with one common encoder and anatomy-specifc decoders enables simultaneous anatomy segmentation. Although the segmentation performance of the combined architecture decreases only by 2.1% and 2.8%, for DeepLabv3/EfficientNetb3 ⋆ and SegFormerB3 ⋆, respectively, models with large encoder and decoder architectures still do not achieve a sufficiently high frame rate (6 FPS and 15 FPS). With our approach, the segmentation performance of models with smaller and thus faster architectures increases by up to 24.4%. Especially, the accurate segmentation of smaller and more complex anatomies can be improved for smaller segmentation networks when guided by accurate teacher models with our MT-KD approach. Overall, the segmentation accuracy of high-capacity, anatomy-specific networks remains at least 1.8% higher, however, almost two orders of magnitude fewer parameters are required. We evaluated our approach on the recently published DSAD. In order to evaluate whether the generalization ability of smaller student models changes similarly to that of larger teacher models when applied to other laparoscopic data sets, it is of interest to perform further comprehensive studies on different laparoscopic data sets. 
 
 Appendix: Code available at: https://github.com/lennart-maack/Efficient-Anatomy-Segmentation-w-Multi-Teacher-KD 


Decision: Based on the improvements made by the authors in response to the reviewers' comments, the manuscript shows promising progress in addressing the concerns raised. The authors have taken steps to improve the clarity, readability and novelty of their work, and have conducted additional experiments to further validate their approach. Some concerns remain, particularly with regard to further validation of the proposed approach, but I still suggest accepting the paper for a poster presentation in MIDL.

Reviews: review: paper_summary: paper_summary: This paper introduces a multi-teacher knowledge distillation approach to improve the performance of small and capable neural network which targets for real-time automatic anatomy segmentation task in laparoscopic surgery. This approach inlcudes training anatomy specific binary segmentataion models and using these models as teachers to guide the training of multi-anatomy student segmentation model. The authors perform experiment on the largest public dataset of laparoscopic image and show that their approch can improve the segmentation performance.
main_review: main_review: * First of all, the novelty and contribution of this paper seem limited.
* Some related works on multi-teacher knowledge distillation and its state-of-the-art application in segmentation are missing.
* The experiment results from current settings are not sufficient to support the generality of the proposed method.
strength_weakness: strength_weakness: * The novelty and the contribution of the paper seem limited. The models this paper used are directly from related papers. The Multi-Teacher knowledge distillation approach in general is also not novel. And the objective functions for training are very basic. From the experiment results, the proposed approach only improved the disc score compared with previous models. The model size, and inference time are the same, which make what the authors claimed in the title as “efficient” questionable.

* The paper content is not structured well. The authors use quite a lot of text to introduce the DEAD dataset, e.g section 2.1 and the 3rd paragraph in Introduction section. It is quite redundant since this dataset is not their work. The introduction section mainly focus on the motivation of the proposed approach and missing related works on multi-teacher knowledge distillation which is very relevant.

* The authors highlight the improvement of models based on segmentation performance.  From the results in Table 1, MiniNetv2 is small the inference time is the best, and has a potential for a big performance improvement with the proposed method. Why the author did not try their approach on MiniNetv2 to check how much the performance can improve with their proposed method?
questions: questions: * What are the results on DeepLabv3 with other backbones such as ResNet18 and MiniNetv2 ? Do they all have significant performance improvement with the proposed knowledge distillation approach?
paper_summary: paper_summary: The authors propose a teacher-student learning approach on the DSAD dataset, which enables learning from a data with imbalanced annotations. Their method allows for real-time applications with better performance than models with similar inference times.
main_review: main_review: I would like to thank the authors for thoroughly addressing all my questions and the concerns of the other reviewers. Acknowledging other possible design choices, such as a shared decoder with multiple outputs and loss functions tackling the same issue is certainly helpful, however it would be more appropriate (but also time consuming) to compare against these alternate methods. The clarifications and the more comprehensive evaluations are much appreciated, so I am slightly increasing my rating.
strength_weakness: strength_weakness: The significance of the improvements is difficult to assess without tests of significance. The introduction lacks important aspects of the field that should be adressed to fully understand the problem the authors wish to tackle.
questions: questions: - The authors describe two approaches for multi-anatomy segmentation, the first being that every anatomy has a separate encoder and decoder, while the other is that all anatomies have the same encoder and different decoders. The authors fail to address the arguably most common segmentation approach where all anatomies have the same encoder and decoder, and they are returned in a multi-channel feature map (eg. in the example of the UNet). Can the authors explain why this was not mentioned in the manuscript? 
- The authors claim that the DSAD dataset cannot be used to train a combined segmentation model, since its segmentations are imbalanced (some images contain anatomies that are not segmented). However these can be addressed with loss functions customized for imbalanced datasets, eg. the data-adaptive loss function proposed in Vu et al. 2022 "A Data-Adaptive Loss Function for Incomplete Data and Incremental Learning in Semantic Image Segmentation". Can the authors discuss why the teacher models are required to train the light-weight model to handle all anatomies?
- The authors claim that while the teacher models were trained for 100 epochs, the student models required only 60. I assume these numbers were found empirically? Please explain how so. If the performance did not improve over 100 and 60 epochs respectively, can the authors address why the student models are faster to train than the teacher models?
- In the results section the authors say that "A uniform increase in segmentation performance can be observed across all eleven anatomies". However these detailed results are not presented in the manuscript. Perhaps the authors could include a more detailed evaluation table in the appendix to show anatomy-based evaluations?
- The performance of the proposed model is questionable, based on the results in Table 1. Looking at the values, a reader can say, that models that have small number of trainable parameters (MiniNetv2) perform poorly, however they have a short inference time. Whereas models that have a large number of parameters (SegFormerB3) perform well, however they have a slow inference time. What the authors propose is a model that follows this trend, it has more parameters than the MiniNet, and it performs better, and is slower, however it has fewer parameters than SegFormer, therefore it performs worse and is faster. It is difficult to see what the model adds to the list of solutions in the field. Models trained with the KD-approach doesn't stand out in any of the metrics individually, only when looking at the metrics collectively. What do the authors think of collecting the results instead only for a certain model architecture, showing how the KD-approach and the other two learning methods impact performance and inference time? What is the benefit of a collective overall table such as Table 1 for all model performances?
paper_summary: paper_summary: The study aims to enhance a specific dataset released by Carstens et al. using distillation learning. This dataset has a significant problem as all images contain one category while potentially including other categories without labels. The proposed method effectively addresses this issue by integrating information into the feature space through distillation learning, allowing the model to extract features from all categories within a single model. Although this approach is successful for datasets with similar challenges, its application is limited to such datasets, and the distillation learning method it employs may lack novelty in the field.
main_review: main_review: This paper addresses the specific problem in the public dataset, but it might not be general in many medical datasets. Although it does have an improvement in efficiency and effectiveness with the proposed distill learning, it has limited contribution to medical analysis considering the novelty of the method and its benefit to the community.
strength_weakness: strength_weakness: 1. The paper primarily introduces a method utilizing distillation learning, which is considered a common approach in the field. Moreover, it lacks comparisons with other distillation learning methods and is highly specific to a particular dataset. Therefore it might have a limited contribution.
2. The experimental section would benefit from the inclusion of visualization analysis to gain a deeper understanding of how the method performs in various scenarios, particularly those that are more complex and challenging.
3. The authors conducted experiments on two recent methods and only one dataset to evaluate the proposed method, so it might lacked persuasiveness.
4. In the figures, Fig. 1 could be improved by adding color to the legends, and the readability of Fig. 2 could be improved by adding more explanation in the title.
questions: questions: 1. Highlight the novelty and contribution.
2. Improve the readability of the method, figure, and the relative description.
3. Analyse the experiment more in focus and present visualization analysis.
4. Consider the comparison methods and justify the design of the article.

Comments: title: Update 22.03.
comment: Update 22.03.

We updated the abstract in the manuscript to highlight the generality of our proposed approach and its applicability to partially labelled datasets.
Additionally, we added Figure 3 to show qualitative segmentation results for different anatomies. We adapted the results section and discussed the qualitative results. 
All changes in the revised manuscript are highlighted in orange. We thank the reviewers for their assessment of the revised manuscript.
"1. Highlight the novelty and contribution"

- We thank the reviewer for the important request. Our main contribution consists of using multiple teachers to increase segmentation performance of real-time capable student networks. To the best of our knowledge, we firstly propose this approach for partially labeled datasets. We conduct a comprehensive evaluation of our approach for multiple network architecturs and and show the advantages of our approach. We revised the introduction section to highlight the novelty and contribution more in depth.

"2. Improve the readability of the method, figure, and the relative description."

- We updated the Figures as well as relative description to improve readability.

"3. Analyse the experiment more in focus and present visualization analysis."
- We conducted additional experiments with DeepLabv3/ResNet18 and MiniNetv2. Additionally, we added anatomy-specific results in the appendix. We added qualitative results in the appendix and aim to add more qualitative results in the future.

"4. Consider the comparison methods and justify the design of the article."
- We considered comparison methods in the Introduction and aim to add associated experiments in the future.
"The authors describe two approaches for multi-anatomy segmentation, the first being that every anatomy has a separate encoder and decoder, while the other is that all anatomies have the same encoder and different decoders. The authors fail to address the arguably most common segmentation approach where all anatomies have the same encoder and decoder, and they are returned in a multi-channel feature map (eg. in the example of the UNet). Can the authors explain why this was not mentioned in the manuscript?"

- We agree with the reviewer that the arguably most common segmentation approach is a single encoder, single decoder architecture with multi-channel output. However, we claim that because of the partially labeled sub-datasets and the missing information about the background, a single decoder would show inferior performance compared to a multi-decoder architecture. This must be empirically proven in future work. Previous work on annotation adaptive loss functions as one way to tackle the problem of partially labeled datasets has been added to the Introduction Section.

"The authors claim that the DSAD dataset cannot be used to train a combined segmentation model, since its segmentations are imbalanced (some images contain anatomies that are not segmented). However these can be addressed with loss functions customized for imbalanced datasets, eg. the data-adaptive loss function proposed in Vu et al. 2022 "A Data-Adaptive Loss Function for Incomplete Data and Incremental Learning in Semantic Image Segmentation". Can the authors discuss why the teacher models are required to train the light-weight model to handle all anatomies?"

- We agree with the reviewer that the benefit of data-adaptive loss is one way to train networks with partially labeled datasets. This has been added to the Introduction section and we consider adding experiments to compare with such methods. In our work, we build upon previous work from Kolbinger et al. [1] and use  one network with a shared encoder and separate decoders for each sub-dataset

"The authors claim that while the teacher models were trained for 100 epochs, the student models required only 60. I assume these numbers were found empirically? Please explain how so. If the performance did not improve over 100 and 60 epochs respectively, can the authors address why the student models are faster to train than the teacher models?"

- Our experiments have shown that training with an additional training signal through the Kullback Leibler divergence causes the student networks to diverge more quickly. We have therefore reduced the number of epochs from 100 to 60.


"In the results section the authors say that "A uniform increase in segmentation performance can be observed across all eleven anatomies". However these detailed results are not presented in the manuscript. Perhaps the authors could include a more detailed evaluation table in the appendix to show anatomy-based evaluations?"

- We thank the reviewer to highlight this important aspect. We added a more detailed evaluation table in the appendix.

"The performance of the proposed model is questionable, based on the results in Table 1. Looking at the values, a reader can say, that models that have small number of trainable parameters (MiniNetv2) perform poorly, however they have a short inference time. [..] It is difficult to see what the model adds to the list of solutions in the field. Models trained with the KD-approach doesn't stand out in any of the metrics individually, only when looking at the metrics collectively. What do the authors think of collecting the results instead only for a certain model architecture, showing how the KD-approach and the other two learning methods impact performance and inference time? What is the benefit of a collective overall table such as Table 1 for all model performances?"

- With our Multi-Teacher Knowledge Distillation approach we aim to increase segmentation performance across various network architectures. We added a more comprehensive evaluation by adding results from MiniNetv2 and ResNet18 to Figure 1 and Table 1.




[1] Fiona R Kolbinger, Franziska M Rinner, Alexander C Jenke, Matthias Carstens, Stefanie
Krell, Stefan Leger, Marius Distler, J ̈urgen Weitz, Stefanie Speidel, and Sebastian Boden-
stedt. Anatomy segmentation in laparoscopic surgery: comparison of machine learning
and human expertise–an experimental study. International Journal of Surgery, 109(10):
2962–2974, 2023.
"The paper content is not structured well. The authors use quite a lot of text to introduce the DEAD dataset, e.g section 2.1 and the 3rd paragraph in Introduction section. It is quite redundant since this dataset is not their work."

- The reviewer is right that the DSAD information are partially redundant across section 2.1 and 3rd paragraph in Introduction Section. We updated the Introduction Section and merely moved important and supplementary information from the section 2 (Method) to the section 3 („ Experimantal Setup“)

"Please clearly state the novelty of this work from the previous work. What has been done before and what are not. / Please add more related works in teacher-student learning in segmentation such as "Improving Fast Segmentation With Teacher-student Learning" and differentiate your work from the others."

- We agree with the reviewer, that the novelty of our work has not been stated clearly enough and especially a comparison with previous work could be explained in more detail.
This has been added to the introduction section of our work. 

"What are the results on DeepLabv3 with other backbones such as ResNet18 and MiniNetv2 ? Do they all have significant performance improvement with the proposed knowledge distillation approach?"

- We thank the reviewer to highlight this important aspect. We added further results to Table 1 and can show highest performance improvements for the MiniNetv2 when using our MT-KD approach. This strengthens the hypothesis that our proposed approach shows high generality.
Dear reviewers, 
first of all, we want to thank you for the detailed and constructive reviews on our work. We are happy to see that most reviewers appreciate the important application scenario, our aim to make the code repository publicly available and the results of our learning approach.
However, the proposed directions to improve our work are especially helpful. This includes, among others, a more comprehensive evaluation of our approach to show its generality, as well as a comparison with existing work in the field of knowledge distiallation and especially a focus on the novelty of our method. In the following comments, we aim to answer your questions. All changes in the revised manuscript are highlighted in orange. Overall, we are very grateful for the detailed reviews of our work and hope that this will improve our future research.