PAAN: Pyramid Attention Augmented Network for polyp segmentation

Authors: sida  yi
hao-cheng  kan
guibo  luo
Venue: MIDL_io_2024_Conference
Published: 2024-01-22 10:15:34

Abstract: Polyp segmentation is a task of segmenting polyp lesion regions from normal tissues in medical images, which is crucial for medical diagnosis and treatment planning. However, existing methods still suffer from low accuracy in polyp boundary delineation and insufficient suppression of irrelevant background due to the blur boundaries and textures of polyps. To overcome these limitations, in this paper a Pyramid Attention Augmented Network (PAAN) is proposed, in which a pyramid feature diversion structure with spatial attention mechanism is developed so that good feature representation with low information loss can be achieved by conducting channel attention-based feature diversion and inter-layer fusion, while reducing computational complexity. Also, our framework includes an Enhanced Spatial Attention module (ESA), which can improve the quality of initial polyp segmentation predictions through spatial self-attention mechanism and multi-scale feature fusion. Our approach is evaluated on five challenging polyp datasets— Kvasir, CVC-ClinicDB, CVC-300, ETIS, and CVC-colonDB and achieves excellent results. In particular, we achieve 94.2% Dice and 89.7% IoU on Kvasir, outperforming other state-of-the-art methods.

Main paper: 
 introduction: 
 Polyp segmentation is a crucial task in medical image analysis, as it plays a vital role in the early detection and diagnosis of colorectal cancer (Hsu et al., 2021) . Colorectal cancer is the third most common type of cancer worldwide, accounting for 9.4% of all cancerrelated deaths (Liu et al., 2021) . Colonoscopy is an effective technique for colorectal cancer screening and prevention. The accurate and effective segmentation of polyp regions from endoscopic images directly influences medical decisions and patient treatment outcomes. However, polyps acquired from colonoscopy images are subject to background interference such as specular reflections and tissue occlusions, while the polyps exhibit variable shapes and textures, along with unclear boundaries with normal tissues (Sun et al., 2023) . Therefore, achieving accurate polyp segmentation from medical images is a challenging task.Early polyp segmentation methods rely on manually annotated features (Mamonov et al., 2014) , such as color, texture, and shape combinations. These methods have high false positive rates due to the limited representation of manually extracted features. The emergence of deep learning provides promising approaches for accurate polyp segmentation. U-Net (Ronneberger et al., 2015) captures complex patterns and restores detailed features through a symmetrical encoder-decoder U-shaped structure. Building upon this, PraNet (Fan et al., 2020) generates initial predictions using parallel decoders and applies reverse attention to generate more precise polyp boundaries. TGANet (Tomar et al., 2022) improves feature representation by introducing text-guided attention. Poly-SAM (Li et al., 2023) is applied to the polyp segmentation by fine-tuning the SAM (Kirillov et al., 2023) .Although the above methods have made certain progress in polyp segmentation, they still have the following limitations: insufficient extraction of features from different hierarchical levels and fusion of interlayer information in the encoder stage, which impacts the segmentation accuracy of blurry boundaries. In the initial decoder stage, there is inadequate attention to the spatial relationship and multi-scale features of polyps, resulting in insufficient suppression of irrelevant background in the initial predicted map and affecting the quality of subsequent decoder generation. Insufficient attention to uncertain regions leads to inaccurate boundary delineation.To address identified challenges, we present the Pyramid Attention Augmented Network (PAAN), aimed at precisely delineating polyp boundaries and reducing background noise. PAAN features a pyramid feature diversion structure with channel attention to enhance feature extraction across different layers, ensuring minimal information loss and improved feature significance. An Enhanced Spatial Attention (ESA) module is introduced early in the decoding process to refine segmentation accuracy through spatial attention and multiscale information fusion. In addition, we refer to and improve the framework of UACANet (Kim et al., 2021) to identify and learn from uncertain regions, aligning predictions more closely with ground truth. Our contributions are as follows:• We propose the Pyramid Feature Diversion module (PFD), which effectively captures important features and reduces information loss by sequentially splitting and merging the channel attention at different levels while reducing computational complexity. • We introduce the Enhanced Spatial Attention (ESA) in the decoder, enhancing the suppression of irrelevant background while improving multi-scale feature extraction. • Extensive experiments on multiple datasets demonstrate that our method outperforms existing methods both qualitatively and quantitatively.Due to space limitations, more related work is relegated to the Appendix, which can be found in Appendix B. 
methodology: 
 Learning more powerful feature representations can improve the discriminative ability of models (Su et al., 2022) . In this paper, we propose the Pyramid Attention Augmentation Network (PAAN), which is an enhanced model based on channel attention and spatial attention mechanisms. The overall framework of PAAN is shown in Figure 1 . The loss function is described in Appendix A.1.PAAN adopts an encoder-decoder structure, where the encoder captures the context and the decoder restores detailed features. The parallel PFD-e encoder aggregates features to The original image is first input into the Res2Net50 (Gao et al., 2019) backbone for feature extraction. By gradually reducing the size of the feature maps and increasing the number of channels, deep feature maps contain more abstract semantic information compared to shallow ones (Wu et al., 2019) , while also reducing computation. Therefore, we only consider the deep feature maps f 3 , f 4 , and f 5 at layers i=3, 4, and 5 as inputs. Each level of the input feature maps has a size of { h 2 i-1 , w 2 i-1 }. The PFD-e module serves two purposes: 1. Concatenating its output with the input of the next decoder module at a deeper level. 2. Allowing the three different levels of PFD modules to collectively input the decoder ESA-d to generate the initial prediction map.In the PFD-e module, the feature maps from the backbone are input into the Pyramid Feature diversion module (PFD in Figure 2 ) through pathways with different receptive fields (1×3,5,7) and then concatenated as output. The PFD consists of two key components: the Feature Diversion Attention module (FDA) and the Hierarchical Interactive Attention module (HIA). In the FDA, only half of the channels are fed into the next FDA module, while the other half is directly fused at the HIA, which helps reduce computational complexity and improve attention effectiveness in the network. In the decoder ESA-d that generates the initial prediction map, we introduce the Enhanced Spatial Attention module (ESA), which can adaptively generate masks based on the input features, thereby improving the accuracy of polyp segmentation. The initial prediction map is progressively enhanced through several UACA modules and is processed with the Sigmoid function and 4x bilinear upsampling to generate the final segmentation map. UACA is the Uncertainty Augmented Decoder in UACANet (Kim et al., 2021) , which can work well with the proposed module. In PFD-e, multi-scale polyp features are captured through the convolutional receptive fields of different kernel sizes. Deep feature maps from the backbone at levels i=3, 4, 5 enter four pathways with distinct receptive fields. The pathway with receptive fields (1×3,5,7) further extracts features through the PFD. The features undergo three levels of distribution:(1) different depths in the backbone, (2) different receptive fields in PFD-e, (3) different levels in the PFD. This hierarchical feature extraction process ensures precise separation and extraction of different features at each stage of the decoder, thereby creating a strong feature representation for polyp segmentation.Multilevel feature fusion is crucial in dense prediction tasks (Su et al., 2022) as low-level and high-level features are complementary. In the pyramid structure of the PFD shown in Figure 2 , the feature extractor separates features into different abstraction levels in the upward phase and gradually fuses them in the downward phase, providing a method to extract higher-level feature representations.In the PFD, the feature flows through a feature distribution and fusion network with a pyramid shape. The pyramid consists of 3 FDA modules and 2 HIA modules. Both modules utilize channel attention mechanisms (Hu et al., 2018) (Implemented by the CA module) to achieve adaptive channel weights and feature selection. Each FDA module takes the output of the lower-level FDA as input and enhances feature selection through cascaded connections. Each HIA fuses the output from this layer and the high-level FDA and achieves feature compression and extraction through feature fusion. The PFD achieves the purpose of better extraction of feature representation while reducing computation.(1) Feature Diversion Attention module (FDA): In the PFD, the FDA is used to divide the feature map m into two branches x 1 and x 2 with equal channel numbers and apply different processing to the two branches. In the first stage, x 1 is subjected to a series of convolutions and ReLU activation for feature extraction, while no operations are performed on x 2 . In the second stage, x 1 and x 2 are input into the CA module to obtain feature maps x 2 ⊙ CA(x 1 ) and x 2 ⊙ CA(x 2 ) with adaptive channel weights, which are then concatenated with the original inputs x 1 and x 2 . The feature map after concatenation is finally fused and compressed by a 1 × 1 convolution layer to obtain the final output.In summary, the FDA module adaptively adjusts the importance of different channels in the feature map through channel feature diversion and channel attention mechanisms, thereby generating more informative and expressive output feature maps.(2) Hierarchical Interactive Attention module (HIA): The HIA module aims to fuse the FDA outputs x 1 and x 2 from the current layer and the previous layer, and enhance the channel attention and compress the channels. Similar to the FDA module, x 1 and x 2 are fed into the CA module for channel attention enhancement, and the outputs x 2 ⊙ CA(x 1 ) and x 2 ⊙ CA(x 2 ) are obtained. Then, they are concatenated and compressed to obtain the output. In summary, the HIA module performs inter-layer fusion using the channel attention mechanism, focusing on the feature representations of different layers, and generating more informative and expressive output feature maps.The decoder ESA-d is designed to generate initial saliency maps for prediction, which is crucial for subsequent decoding processes. The ESA-d receives output feature maps f 1 , f 2 , and f 3 from different levels of the backbone network PAA. To match the feature map sizes, f 1 and f 2 are upsampled by 4 times and 2 times respectively, and f 3 is the original size feature map. These feature maps are concatenated to generate the input feature map f in . The feature shuffling structure is applied to f in to separate the feature representations of different levels after concatenation, and then input into the ESA module.The ESA module aims to generate a feature mask to enhance the important spatial regions of the input feature maps. The module first applies a 1x1 convolutional layer to compress the number of channels to 1/4, and further extracts and compresses spatial information through downsampling. The maximum pooling operation selects the maximum value in each region as the salient feature, providing the basis for generating spatial attention maps. Meanwhile, a series of convolutional layers are applied for feature processing, which can extract features at different spatial scales and help understand different parts of the scene. Finally, the feature map size is adjusted back to the size of the input feature map using bilinear interpolation, and the adjusted feature map is fused with the original map, resulting in pixel-wise enhancement. The spatial mask is generated by Sigmoid, effectively combining deep spatial features and shallow spatial features. The specific formula is as follows:EQUATIONwhere f i and f o are the input and output of ESA, B represents the spatial feature extraction which includes pooling and upsampling, and σ and cls denote Sigmoid and 1×1 convolution. In summary, the ESA enhances the network's attention ability to specific regions through the self-attention mechanism and feature transformation operations, extracts more accurate and region-specific feature representations, and generates better output feature maps. 
experiments: 
 In this section, we present the experimental setup and results. The results obtained by PAAN are compared quantitatively and qualitatively with six state-of-the-art polyp segmentation methods. Appendix A.2-A.4 provides details about the experimental setup, datasets, and evaluation metrics. The ablation study is described in Appendix C.PAAN is trained on NVIDIA TITAN 3090 GPU in Pytorch. We used Adam for optimization, with a learning rate of 0.0001, a batch size of 16, and 1000 epochs. We use the following 5 datasets for experiments and evaluations: CVC-ClinicDB (Bernal et al., 2015) , CVC-ColonDB (Tajbakhsh et al., 2015) , ETIS (Silva et al., 2014) , Kvasir (Jha et al., 2020) , and CVC-300 (Vázquez et al., 2017) . 1450 of these images were used for training and 800 for testing. The evaluation metrics used in this paper are IoU, Dice, MAE, F w β , S α , and E mean ϕ . These metrics provide multifaceted insight into both accuracy and robustness. More details are in Appendix A.2-A.4.The quantitative results of PAAN on five polyp datasets compared with the state-of-the-art methods are presented in Table 1 . PAAN demonstrates outstanding performance in terms of IoU, Dice, MAE, and surpasses the existing methods on all datasets. In particular, compared to the previous state-of-the-art model (i.e., UACANet-L), our average Dice scores on Kvasir and EITS have improved by 3.0% and 4.7% respectively. Furthermore, our network has also achieved significant improvements on the other four metrics: MAE, F w β , S α , and E mean ϕ . Figure 3 presents the visual results of our network and contrast methods on multiple datasets. We intentionally selected challenging polyp images with smaller and more blurred regions compared to typical polyp images. Additionally, there is significant interference from lighting and irrelevant features. The results show that 6 comparative methods exhibit inaccurate judgments of blurry boundaries or mistakenly identify reflective irrelevant areas as polyp regions. In the 3rd row, other methods are affected by background interference, resulting in the mislabeling of irrelevant regions. In the 4th row, the under-segmentation occurs due to the small size of the polyp area. In contrast, the PAAN can accurately identify polyps, effectively capturing their boundaries and texture changes, while excluding interference from irrelevant background information. This highlights the advantages of our attention-enhanced network in detail recognition and background suppression. By utilizing visual methods, researchers can have a clearer observation and understanding of the segmentation results of PAAN, as well as its relative advantages in perceiving fine details, handling fuzzy boundaries, and suppressing irrelevant information.By comparing the qualitative and quantitative analysis results of other networks, it can be seen that the method proposed in this paper has the following advantages:(1) Accuracy of results: In terms of quantitative results, it can be seen that the networks in this paper outperform competitors in major indicators such as mDice, mIoU, and MAE. In terms of qualitative results, even in cases with fuzzy polyp boundaries and smaller polyp sizes, the networks in this paper can still accurately depict the polyp boundaries, which demonstrates the good resolution ability of pyramid feature diversion and spatial attention to fuzzy boundary details. The uncertainty analysis can be found in Appendix E.(2) Ability to suppress background interference: By the qualitative results, it can be observed that our method has accuracy in capturing polyp boundaries and the ability to suppress background interference. Even in complex polyp scenes with lots of fuzzy features, PAAN can still accurately extract effective features and suppress irrelevant features. The ESA module improves the accuracy of segmentation results by better capturing the spatial dependencies in the input image. By enhancing spatial attention, the focus on polyp and the irrelevant background suppression are improved, thus improving the accuracy.(3) Lower computational complexity: Our network improves segmentation with less complexity than previous methods. Detailed can be found in Table 2 and Appendix D. 
ablation study: 
 We conducted complete ablation experiments under all five datasets The quantitative experiments employed mDice, mIoU, and MAE as evaluation metrics, with the results shown in Table 3 and Figure 4 . Ablation results prove the effectiveness of each module in this paper, and the complete PAAN network has better segmentation results under all datasets. Detailed discussion is provided in Appendix C.  
conclusion: 
 In this paper, we propose the Pyramid Attention Augmented Network (PAAN) for accurate polyp segmentation. Our network combines spatial attention mechanism and pyramid feature diversion structure to effectively capture important features and reduce information loss. The FDA and HIA module in the pyramid feature diversion structure reduce computational complexity and improve the effectiveness of attention. The Enhanced Spatial Attention module utilizes spatial self-attention and multi-scale feature fusion to further improve the accuracy of polyp segmentation. Experiments on five challenging polyp datasets demonstrate that our network achieves excellent quantitative and qualitative results in terms of accuracy and robustness, outperforming existing methods.separable convolution, enhances the recognition performance of electron microscopy images. These studies have demonstrated positive outcomes in various medical segmentation areas. However, specific feature extraction methods tailored to address challenges such as indistinct boundaries and diverse textures in polyp images are neglected in these methods. 
 
 Appendix: In the context of polyp segmentation, CANet (Zhang et al., 2019) To validate the effectiveness of the mentioned modules in the paper, we conducted comprehensive qualitative and quantitative experiments on five datasets. The quantitative experiments employed mDice, mIoU, and MAE as evaluation metrics, with the results shown in Table 4 , while the qualitative experimental results are presented in Figure 5 .To validate the effectiveness of PFD, we trained and evaluated PAAN without the PFD module. One group of experiments completely removed PFD-e, while the other group of experiments retained the multi-scale framework of PFD-e while removing the PFD, specifically focusing on investigating the role of attention in the pyramid feature diversion structure of PFD. We conducted tests on all five test datasets, and the results are shown in Table 4 . The performance gap between "w/o PFD" and "PAAN" is significantly larger than the gap between "w/o PFD" and "w/o PFD-e". This indicates that the effect of a simple multi-scale receptive field is not ideal, and the advantages of enhanced feature extraction and better feature representation brought by the pyramid attention structure are crucial. Additionally, as can be seen in Figure 5 , the poor handling of blurry boundary features without PFD results in scattered boundary regions detached from the main subject, demonstrating the effectiveness of the feature fusion structure employed in this study.Ablation Study for ESA: To validate the effectiveness of ESA, we removed the ESA from the ESA-d decoder while retaining the decoding module based on multiple convolutional operations. This allowed us to specifically investigate the role of spatial attention in the initial decoder. We conducted tests on the same evaluation metrics on five datasets, and the results are shown in Table 4 . The ESA has shown improvements in metrics such as mDice, mIoU, and MAE. Furthermore, as seen in Figure 5 , the lack of ESA leads to 


Decision: All reviewers found the proposed method to be novel and the results promising. 2 weak accept and 2 strong accept recommendations from reviewers.

Reviews: review: paper_summary: paper_summary: The author proposed a Pyramid Attention Augmented Network (PAAN), for segmenting polyp lesions, overcoming the current suffering of low accuracy in polyp boundary delineation and insufficient suppression of irrelevant background. Key modules (PFD, ESA) were proposed to 1) reduce information loss and reduce computational complexity 2) suppressbackground noise. However, some strong weaknesses should be paying attention.
main_review: main_review: The author partially solved my questions, and the major limitation is the really limited contribution/edition onto the existing work UACA-Net, and they just modified part of the modules. There are some arbitrarily designed network architecture design with not enough ablation study added. In addition, the authors' utilization justification the differences between 1x3 & 3x1 conv kernels and 3x3 conv kernels seem to be not convincing to me.
strength_weakness: strength_weakness: 1) There are too many arbitrary hyperparameter selections in module designs. Some of the proposed modules have figure illustration, but lack of details and reasons why the authors proposed to design like these. For example, why there are only 1x1, 1x3, 1x5, 1x7 were selected? Why not including 1x9? In addition, why there is always a 3x3 conv kernel following 1x3,5,7, but now a conv 1x3 and conv 3x1 if the authors really wanted to reduce calculation complexity?
2) Compared with the work UACANet, the author seems like only changing the PAA module in the work UACANet to be PFD & ESA modules proposed here. The overall structure, and even the naming are pretty much the same. For this specific task, with these small amount of modification, I would say the contribution is limited.
3) In this similarity level between this paper and the paper UACANet, the authors should clearly state your limited contribution on just modifying one of the module inside the previous UACANet. However, the introduction doesn’t mention this, and it seems like the authors intended to claim the whole architecture was newly invented, which might not be appropriate.
4) One of the contributions claimed by the authors is “rrelevant background suppression”. However, how was the “irrelevant background suppression” achieved by the proposed PFD&ESA module is not clearly explained and not well co-related with the proposed module designs. Moreover, here is no quantitative measurements for this conclusion.
5) One of the contributions claimed by the authors is “reducing computational complexity”. However, there is no quantitative measurement and comparisons for computational complexities among different models.
questions: questions: 1) Page 2 Paragraph 2: “Although the above methods have made certain progress …” Are there any references support all of your assumptions? For example, why “insufficient extraction of features from different hierarchical levels and fusion of interlayer information in the encoder stage”?

2) Although it is good to compare with existing SOTA methods with same research objective, it would be better to include newer Unet-like methods. The general segmentation models the author chose were published before 2019, which could be a bit old. Some choices could be but not limited to: ViT-based models and nnUNet, etc.

3) Please refer to the five points in Weaknesses.
paper_summary: paper_summary: The authors tackle the difficult task of polyp detection in endoscopic images.
This is a very difficult task on which even SOTA models have had troubles (we know from experience).

The authors build a new convolutional neural network architecture that introduces several modules that are focused on aggregating information over larger ranges in the image, and applying more adaptive normalizations/attention to deal with the larger noise and image-intensity differences in endoscopy images.

The authors then test the architecture on 5 standard benchmarks where they compare SOTA (at time of submission presumably), and also do an ablation study to see if the various added modules perform a function (which is only in the appendix).
main_review: main_review: Authors clarified the issues I had. The introduced method is achieving at the time SOTA on a difficult and relevant task, and lessons from the architecture can be learned and will probably improve future models as well. Also the validation done is quite thorough. I would say the paper is in the top 20% of papers (from rough estimation) so I would give it a 4 (I reserve 5 for remarkable papers).
The main 'weakness' is that there are many papers that change architectures and it can seem less novel where many things have been tried in different papers, but to bring it together and actually achieve results on a real task is not trivial and valuable to the field (this method for example can be directly used in real applications).
strength_weakness: strength_weakness: - The model performed state of the art on the tested dataset according to this paper. However the field moves quickly, and on several of the dataset tested there are now better performing models. This is not perse a negative; the performance of this method is still close and the new methods use orthogonal improvements. So this work is obviously still relevant, since the innovations in the model can still be used. Also the newer methods use Transformers which can be computationally problematic, especially as in medical applications one needs real-time segmentation on relatively slower hardware.
- Also one could see this as more of an engineering paper, and not a fundamental new invention, thus scoring less on the innovation scoring. However I also see the perspective of tackling this difficult problem and getting real results and as such I see this as very valuable.
questions: questions: Clarify the questions, especially how the backbone is trained.
paper_summary: paper_summary: The authors propose a network architecture with Pyramid Feature Divesion and Enhanced Spatial Attention modules aimed to enhance polyp segmentation. They evaluate their trained model on a series of datasets and compare it to various already established methods.
main_review: main_review: I would like to thank the authors for actively participating in the rebuttal phase, and addressing all the concerns me and the other reviewers have raised. I believe the quality of the manuscript improved during the review process, therefore I am adjusting my rating.
strength_weakness: strength_weakness: The implementations of the experiments are not described well enough, to fully understand the other baseline models used for comparison. Furthermore the results section lacks important information for the conclusions to be well-founded and thorough.
questions: questions: - The authors should address what kinds of statistical tests of significance they used to determine the improvements "significant". They should also report the uncertainties (eg. standard deviations) of their results to show better comparison between the models.
- The experimental setup and a short description of the datasets are essential to include in the main body of the paper, so the experiments are more detailed. Perhaps the related works section could be placed in the appendix instead?
- How did the authors implement the other segmentation methods? A proper description is needed for the reader to understand the comparisons. Did the authors train the models on the same datasets with the same data spits and hyperparameters?
- What is the complexity (eg. number of trainable parameters) of the models? Could it be that the other compared models are simpler, and therefore perform worse?
- What is the inference time of the models? Could it be that the proposed model works well, however it is slower? The starting point for the authors were the limitations of other works in the field, however they do not comment on any limitations of their approach. A slow inference time would not make the results less impressive, however it would paint a better picture on how the model works so well.
paper_summary: paper_summary: This paper introduces PAAN (Pyramid Attention Augmented Network) for polyp segmentation detection tasks, aiming to address the following challenges: 
Despite progress in polyp segmentation, previous methods suffer from insufficient feature extraction across hierarchical levels and inadequate integration of interlayer information, impacting segmentation accuracy, especially in areas with blurry boundaries. 
Additionally, the initial decoder stage lacks attention to spatial relationships and multi-scale features of polyps, leading to ineffective background suppression and inaccurate boundary delineation in uncertain regions.
main_review: main_review: Once again, the most significant issue with this paper is that the so-called pyramid-structured PAAN network is not actually a novel concept as presented by the authors. Readers, unaware of this fact, would naturally assume that the pyramid structure network is a novel contribution by the authors. However, this seems not to be the case. 

Besides, I can confidently state that the PAAN network in this paper shares a fundamentally similar idea with a much earlier paper titled "Pyramid Attention Network for Semantic Segmentation," as well as with the referenced paper "Fapn: feature augmented pyramid network for polyp segmentation." Both of these sources indicate that the pyramid structure was not originally proposed by the authors. 

Based on this analysis and understanding, it appears that this work merely makes minor adjustments to the network architecture and adapts it for polyp segmentation tasks. Consequently, the research innovation of this paper is deemed to be quite low. Therefore, overall, I do not recommend the publication of this paper in the top-tier conference MIDL.
strength_weakness: strength_weakness: The most significant issue with this paper is that the so-called pyramid-structured PAAN network is not actually a novel concept as presented by the authors. Readers, unaware of this fact, would naturally assume that the pyramid structure network is a novel contribution by the authors. However, this seems not to be the case. 

Besides, I can confidently state that the PAAN network in this paper shares a fundamentally similar idea with a much earlier paper titled "Pyramid Attention Network for Semantic Segmentation," as well as with the referenced paper "Fapn: feature augmented pyramid network for polyp segmentation." Both of these sources indicate that the pyramid structure was not originally proposed by the authors. 

Based on this analysis and understanding, it appears that this work merely makes minor adjustments to the network architecture and adapts it for polyp segmentation tasks. Consequently, the research innovation of this paper is deemed to be quite low. Therefore, overall, I do not recommend the publication of this paper in the top-tier conference MIDL.
questions: questions: I have totally said in "Weaknesses" part. So I do not suggest authors to rebuttal.
paper_summary: paper_summary: A pyramid attention method is presented for polyp segmentation. There is an enhanced spatial attention mechanism, multiple datasets were used for experiments, and outperformed the SOTA results. This spatial attention mechanism is introduced in decoder side in early phase to minimize the information loss and diversify the feature space. There is also an uncertainty-augmented module, which further increase the results (which authors failed to mention in abstract).  Overall this is a fair conference article at MIDL, I believe.
main_review: main_review: I change my scoring to weak accept after rebuttal period, 
the questions were more or less answered, although they are not fully convincing the authors have a point for their design.
I think the paper can be a fair conference paper.
strength_weakness: strength_weakness: - polyp segmentation literature is not covered completely. Although I dont expect to see all related words, but at least some recent ones and known ones can be mentioned and compared. 
- Pyramid approaches are often used for multi-level feature extraction, authors presented this as a new approach, it is not. Not in medical image segmentation in particular, even in polyp segmentation, there are numerous papers on this subject. 
- figure 3 is hard to interpret. Why not just overlay those segmentation results on the images? it is not standard to show like this.
- ablation experiments are missing.
questions: questions: - if dilated conv are used, do you still need pyramid approach?
- why not compare with other pyramid based methods ?
- figure 3 is hard to interpret. Why not just overlay those segmentation results on the images? it is not standard to show like this.
- ablation experiments are missing. 
- Not clear how uncertainty guidance is used.
paper_summary: paper_summary: The authors propose to modify UACANet with the proposed Pyramid Feature Diversion module and Enhanced Spatial Attention for better performance in polyp segmentation. They named their proposed method as Pyramid Attention Augmented Network. The authors evaluate their methods on five polyp segmentation datasets and compare the results with some previous studies.
main_review: main_review: I would like to thank the authors for their revision. All the weak points are well addressed. Although there are new innovative methods that outperform the proposed method in some tasks, the paper presents interesting modification ideas for UACANet. The newly added comparison experiments with more recent state-of-the-art methods also improve the quality of the paper. I vote Strong accept.
strength_weakness: strength_weakness: (1) Lack of comparison with more recent state-of-the-art methods that also utilize multi-scale feature extraction and attention.

(2) Please conduct a grammar error check for the paper and fix grammar errors.
questions: questions: Please compare with more recent state-of-the-art methods that utilize multi-scale feature extraction and attention or recent state-of-the-art methods networks that modified UACANet.

Comments: title: Response to the authors
comment: Thank you very much for your valuable feedback. We have submitted a new revised version of the paper. In this revised paper, we have added an Appendix section on uncertainty analysis and provided the mean and standard deviation for different metrics across all five datasets. We aim to provide readers with a clearer understanding of the actual performance of our network under different initial training conditions. Thank you for the suggestions you have provided for enhancing this paper.
Q: If one of your contribution is the time complexity, then the table regarding the time complexity should be appeared in the main manuscript but not the appendix.

A: Thank you for your valuable comment. In the revised version of the paper, we have further adjusted the chapter structure. Additionally, we have incorporated a comparative table of computational complexity and parameter complexity into the main body to facilitate readers in quickly obtaining detailed information on the network's computational complexity.

Q: Would you mind showing me where did you emphasize your sentences about "you only modifed partial module designs of the UACANet"? I think this should be more obvious, and should not be stated as "significant augmentation"

A: Thank you for your valuable suggestions. We have submitted a new version of the paper. In this revised version, we have meticulously refined the articulation of the article to provide readers with a clearer understanding of our innovative contributions and correlation with the UACANet. This will assist readers in better comprehending the proactive innovations and enhancements we have made to the entire network.
Follow-ups:
Q5: Would you mind showing me where did you emphasize your sentences about "you only modifed partial module designs of the UACANet"? I think this should be more obvious, and should not be stated as "significant augmentation"

Q6: If one of your contribution is the time complexity, then the table regarding the time complexity should be appeared in the main manuscript but not the appendix.
Thanks for clarifying my questions!
I would like to thank the authors for thoroughly responding to my questions, and the questions of all the other reviewers.

Q2. I'm afraid my concerns might not have been described well enough in my review. In a scientific setting, when claiming that an improvement  is significant ("our network has also achieved significal improvements...") this claim has a statistical meaning, and it has to be shown though proper evaluations. Perhaps, for example an improvement of the Dice score by 3% is purely by noise, and it has nothing to do with the performance of the model. To ensure that the improvements are significant, t-tests are required, which really don't fit in the future works section, instead they should be part of the current manuscript to show that the model indeed works significantly better than the baselines. If t-tests are not possible, the uncertainties, such as standard deviations must be reported to get a sense of the significance. If none of these are possible, I suggest the authors to refrain from using the phrase "significantly" when comparing the results. I only recommend this, because of the time limitation, however I would encourage performing a simple t-test instead. Once the standard deviations are available, performing these tests is quite straightforward and quick. In that line, including more comparative tables and figures does not paint a clearer picture about uncertainty. Instead, reported standard deviation values would.

Thank you for including information about complexity, and inference times, I believe it really strengthens the power of the proposed model.
Q1: Page 2 Paragraph 2: “Although the above methods have made certain progress …” Are there any references support all of your assumptions? ...

A1: Thank you for this valuable suggestion. Regarding the assumptions mentioned in the second paragraph on page 2, there indeed exists literature supporting our viewpoints. For instance, in addressing the issue of "insufficient feature extraction at different levels and inadequate inter-layer information fusion during the encoding phase," the significance of introducing feature extraction at various levels and information fusion for the quality of medical segmentation was already manifested in early attention-based medical segmentation articles like Attention U-Net (Ozan Oktay, et al.). Additionally, the paper “Dual Attention Network for Scene Segmentation” (CVPR2019) also emphasizes the importance of using the attention mechanism to effective integration of multi-level information during the encoding phase for enhancing model performance. These studies provide solid theoretical backing for our hypotheses and guide the direction of our research. 

Q2: Although it is good to compare with existing SOTA methods with same research objective...

A2: Thank you for your valuable comments. In the revised version of the paper, we have added updated comparative experiments to make our experimental data more convincing, including the two articles in 2022, and added them to Table 1.

Q3: There are too many arbitrary hyperparameter selections in module designs...

A3: Thank you for this valuable comment. Regarding the issue of hyperparameter selection in our module design, our choices were based on a series of preliminary experiments. In our research, the use of 1x1, 1x3, 1x5, and 1x7 convolution kernels was experimentally validated, demonstrating a good balance of performance and efficiency in our model. The decision not to include the 1x9 convolution kernel was based on our experimental data, where we observed that the 1x9 kernel did not offer additional performance improvements, but increased computational burden. As for why we chose to use a 3x3 convolution kernel after the 1x3, 1x5, and 1x7 convolutions, instead of a combination of 1x3 and 3x1, it is because in our model architecture, the 3x3 kernel was more effective in capturing spatial features, while also maintaining good performance with relatively low computational complexity.

Q4: Compared with the work UACANet, the author seems like...

A4: Thank you for your insightful comments. I appreciate the opportunity to clarify the unique contributions of our work. While it is true that the overall structure of our model bears resemblance to UACANet, the introduction of the PFD and ESA modules represents a significant enhancement over the existing PAA module. These modules were specifically designed to address limitations observed in UACANet, bringing in advanced feature processing capabilities and more effective spatial attention mechanisms. Our experimental results, which we can elaborate on in the revised manuscript, show notable improvements in performance on the specific task, attributed to these modifications.The decision to retain a similar structural framework as UACANet was intentional, as this architecture has proven effective for this class of tasks. Our contribution, therefore, lies in refining and augmenting this framework where we identified opportunities for improvement. The PFD and ESA modules are not mere replacements but are substantial upgrades that address specific challenges and enhance the model's capabilities.

Q5: In this similarity level between this paper and the paper UACANet, the authors should clearly state...

A5: Thank you for your attention to our work and for raising this important issue. Indeed, our work substantially draws upon the base architecture of UACANet, but our enhancements and optimizations in module design are a significant augmentation to this framework. We will make this explicit in the revised manuscript and more clearly delineate the relationship between our contributions and UACANet. It was not our intention to claim the entire architecture as a novel invention, we will make appropriate modifications to ensure our statements are more accurate and transparent. Our primary contribution is in building upon the framework by introducing innovative PFD and ESA modules and the corresponding pyramid and spatial attention mechanisms, which significantly improve model performance and are optimized for specific tasks.

Q6: One of the contributions claimed by the authors is “reducing computational complexity”...

A6: Thank you for this valuable comment. In response to one of our contributions of reducing computational complexity, we conducted additional experiments comparing the computational complexity and model parameter count of our method with previous state-of-the-art approaches in the revised version of our paper. The results are presented in a new table.
Q1: Lack of comparison with more recent state-of-the-art methods that also utilize multi-scale feature extraction and attention. The authors only compared with methods developed in 2021. It would be great if the authors were able to search through the papers that cited UACANet or recent literature to find more recent studies that utilize multi-scale feature extraction and attention. Please compare with more recent state-of-the-art methods that utilize multi-scale feature extraction and attention or recent state-of-the-art methods networks that modified UACANet.

A1: Thank you for your valuable comment. In the revised version of our paper, we have introduced more recent studies for comparison, including the methods based on UACANet and attention mechanism like FAPN. We have replicated these studies on the existing dataset and included a comparison table from the original papers to facilitate further comparison for the readers.

Q2: Please conduct a grammar error check for the paper and fix grammar errors. On the 1st page, for example: the blur boundaries: the blurred boundaries.

A2: Thank you for your valuable suggestions. In the new version of the paper, we have meticulously reviewed and rectified grammatical issues to ensure the precision and accuracy of the article.
Q1: if dilated conv are used, do you still need pyramid approach?

A1: Thank you for this valuable comment. Using dilated convolutions can indeed increase the receptive field of the network, similar to what a pyramid approach does. However, the pyramid approach offers a structured way to integrate features at multiple scales, which is beneficial for capturing both local and global context. While dilated convolutions are effective for increasing spatial resolution in the feature maps, they might not fully replace the hierarchical, multi-scale integration offered by pyramid structures. Therefore, depending on the specific requirements of the task, combining dilated convolutions with a pyramid approach could potentially yield better results by leveraging the strengths of both techniques.

Q2: why not compare with other pyramid based methods?

A2: Thank you very much for your valuable suggestion. In the new version of the paper, we have incorporated additional updated comparative experiments, including the use of the pyramid approach FAPN, to offer readers more detailed comparative experimental results.

Q3: figure 3 is hard to interpret. Why not just overlay those segmentation results on the images? it is not standard to show like this.

A3: Thank you for your valuable feedback. In the revised version of the paper, we superimposed the segmentation regions of each network onto the original images according to your requirements. Regarding the issue of image superimposition, we drew inspiration from multiple articles to achieve a superior presentation effect. The updated comparative images can convey the detailed information of the segmentation results more clearly to the readers.

Q4: ablation experiments are missing.

A4: Thanks for your valuable comment, we have conducted complete ablation experiments on all the modules proposed in this paper on all five datasets, which were previously placed in the appendix section due to article length limitations, and we have now added the ablation experiments to the main text in the revised version of the paper.

Q5: Not clear how uncertainty guidance is used.

A5: Thank you for your insightful comments. You may be referring to the section of the article related to Uncertainty Augmented Decoder. In this section, we have used the same modules as UACANet, so we did not elaborate on it extensively. This is because the primary innovation of our paper lies in the pyramid attention encoder and the initial decoder for generating initial prediction maps.
Q1: The most significant issue with this paper is that the so-called pyramid-structured PAAN network is not actually a novel concept as presented by the authors. Readers, unaware of this fact, would naturally assume that the pyramid structure network is a novel contribution by the authors. However, this seems not to be the case.

A1: Thank you very much for your valuable input. The primary innovation of this article lies in the introduction of pyramid feature attention into various decoding stages with different receptive field structures. This, in essence, represents a novel feature enhancement design concept, offering a method to extract and integrate feature representations at different abstract levels, enabling the network to learn more robust feature representations.

Q2: PAAN network in this paper shares a fundamentally similar idea with a much earlier paper titled "Pyramid Attention Network for Semantic Segmentation," as well as with the referenced paper "Fapn: feature augmented pyramid network for polyp segmentation." Both of these sources indicate that the pyramid structure was not originally proposed by the authors.Based on this analysis and understanding, it appears that this work merely makes minor adjustments to the network architecture and adapts it for polyp segmentation tasks. Consequently, the research innovation of this paper is deemed to be quite low. 

A2: Thank you for this valuable comment. In addressing the comparison between our PAAN network and earlier works on pyramid structures, it's crucial to underline the distinctive advancements we've introduced. While pyramid structures are not entirely novel in themselves, our application to polyp segmentation is marked by significant modifications that go beyond superficial adjustments. These enhancements, particularly in feature extraction and attention mechanisms, are tailored to the unique demands of polyp segmentation, demonstrating a clear leap in efficiency and accuracy. Innovation often builds upon existing frameworks, and our work exemplifies this by adapting a familiar concept to meet specific, complex challenges in medical imaging. Therefore, the innovative value of the PAAN network should be recognized as a meaningful contribution to the field. In addition, in the revised version of this paper, we have included comparative experiments with FAPN in the quantitative experimental section. The experiments indicate that we have a certain advantage over other pyramid methods on multiple datasets. Thank you very much for your constructive feedback.

Q3: the author surprisingly did not compare their proposed methodological and detailed aspects with the two aforementioned network modules proposed by others, which is essential to us (future readers). Therefore, I believe that this paper must incorporate this discussion to let readers know which part of their innovation is truly novel, rather than just the pyramid structure (of which I am certain is not originally proposed by the authors).

A3: Thank you for this valuable comment. In the newly revised paper, we extend and compare the network of this paper with the modules proposed by others on all five data sets. At the same time, we adjust the structure of the paper to better show the novelty of our proposed method. In the previous question, we provided detailed explanations of the proposed module's functionality and our innovative contributions. Thank you again for your valuable comment.
Q1: The implementations of the experiments are not described well enough, to fully understand the other baseline models used for comparison. Furthermore the results section lacks important information for the conclusions to be well-founded and thorough.

A1: Thank you for this valuable comment. In the revised version of the paper, we have restructured the brief descriptions of other baseline models and datasets into the experimental section of the main text, while providing detailed descriptions in the appendix. To draw more thorough and comprehensive conclusions, we have included additional comparative experiments and result analyses to further bolster our findings. Moreover, we have provided more detailed explanations in the discussion section.

Q2: The authors should address what kinds of statistical tests of significance they used to determine the improvements "significant". They should also report the uncertainties (eg. standard deviations) of their results to show better comparison between the models.

A2: Regarding the issues of statistical testing and reporting uncertainty, we have referenced a lot of papers in the field of polyp segmentation. Typically, significance prediction and standard deviation are used in statistical survey methods. In the field of polyps, the more commonly utilized metrics are the six indicators provided in the paper. Significance is more reflected in our detailed comparisons with multiple state-of-the-art methods on various datasets. In the revised version of our paper, we have incorporated more comprehensive comparative experiments and performance investigations, along with a thorough analysis to validate the importance of our improvements. The suggestion for comparing significance and uncertainty is indeed valuable. In future work, we plan to employ tests like paired t-tests to compare the performance metrics of our model with existing benchmarks, aiming for better statistical enhancements. Furthermore, in our improved paper, we have included more comparative tables and images to highlight the differences between our approach and existing methods. This not only offers a clearer picture of the variations in our model's performance but also allows for more precise and reliable comparisons between different models. Once again, we appreciate your valuable insights. 
 
Q3: The experimental setup and a short description of the datasets are essential to include in the main body of the paper, so the experiments are more detailed. Perhaps the related works section could be placed in the appendix instead?

A3:  As you suggested, we have added the experimental setup and dataset section to the main text to provide readers with more complete details of the experiment.

Q4: How did the authors implement the other segmentation methods? A proper description is needed for the reader to understand the comparisons. Did the authors train the models on the same datasets with the same data spits and hyperparameters?

A4: For the comparative methods, we used the code provided by the authors for testing, with all settings configured according to the parameters given in their original papers. The dataset setup was also consistent, using the same division method as outlined in these studies. At the same time, in order to make a more fair comparison, all the experiments are carried out on the same experimental machine.

Q5: What is the complexity (eg. number of trainable parameters) of the models? Could it be that the other compared models are simpler, and therefore perform worse?

A5: Thank you for this valuable suggestion. In the experimental section, we have supplemented our report with a comparison experiment on model complexity. Our model achieved better performance with lower computational complexity and fewer parameters.

Q6: What is the inference time of the models? Could it be that the proposed model works well, however it is slower? The starting point for the authors were the limitations of other works in the field, however they do not comment on any limitations of their approach. A slow inference time would not make the results less impressive, however it would paint a better picture on how the model works so well.

A6: In response to your query about inference time, we've added a comparison in the experimental section to the revised version of the paper. Since inference time is closely related to the machine used, we added Macs, a metric for the number of accumulations and floating-point operations required by the model ,indicating that our model demonstrates impressive performance. In fact, the inference time is only 15-20ms per image on the same machine as the experiments in this paper, which is faster compared to other models we benchmarked. While we acknowledge every approach has limitations, our model effectively addresses key challenges in the field while maintaining a swift inference time, thereby providing a comprehensive understanding of its high-performance capabilities.
Q1: The model performed state of the art on the tested dataset according to this paper. However the field moves quickly, and on several of the dataset tested there are now better performing models. This is not perse a negative; the performance of this method is still close and the new methods use orthogonal improvements. So this work is obviously still relevant, since the innovations in the model can still be used. Also the newer methods use Transformers which can be computationally problematic, especially as in medical applications one needs real-time segmentation on relatively slower hardware.

A1: Thank you for your insightful feedback. We recognize the dynamic nature of our field and the emergence of new models that may show superior performance on some datasets. We add comparative experiments with more novel methods in the revised version of the paper, including two papers from 2022, to show the effectiveness of the proposed method. Your point about the practical relevance of our model, especially in scenarios requiring real-time segmentation on less powerful hardware, is particularly encouraging. Regarding the segmentation performance of the model under computationally constrained environments, we have included discussions on model complexity and parameter count in the revised version of the paper, and have validated the model's computational efficiency advantages. We appreciate your recognition of our work's ongoing relevance and are inspired to further refine our approach in light of these developments.

Q2: Also one could see this as more of an engineering paper, and not a fundamental new invention, thus scoring less on the innovation scoring. However I also see the perspective of tackling this difficult problem and getting real results and as such I see this as very valuable.

A2: Thank you very much for your valuable feedback. In fact, this article presents numerous innovative design concepts. For instance, the introduction of a pyramid attention mechanism structure at different levels of the main network hierarchy and receptive fields proposes a novel feature enhancement design concept. By incorporating attention mechanisms sensibly across various abstract and perceptual levels, a scalable and versatile design philosophy is introduced. Moreover, the integration of spatial attention during the generation of initial prediction maps represents a commendable new design approach.

Q3: The figures use (c) for concatenation and (+) for element-wise addition, make sure there is some kind of legend (there is no standard for these things)

A3: Thank you for your valuable response. We have added a legend in the bottom left corner of Figure 1 to indicate the meanings of the various symbols used in the image, ensuring that readers do not experience confusion or misunderstanding.

Q4: In Figure 1, what part is the Resnet. There are 'empty' blocks right after the input which presumably are the feature outputs of resnet? But it's not clear.

A4: Thank you for your valuable response. In our revised version of the paper, we have included new wireframes and annotations to indicate the position of the ResNet backbone network in the document, with detailed explanations provided in the legend.

Q5: The FDA module applies two 1x1 operations with a concat followed by another 1x1. Since these are linear operations, it would seem that you could simple concatenate directly and then do one 1x1 operation?

A5: Thank you for your valuable comments. The main purpose of the 1×1 convolution operation in the FDA module is to reduce the dimension of the channels so that the number of channels after concatenation is the same as before. Since the number of channels doubles after each splicing operation in FDA, a 1×1 convolution operation after splicing can keep the number of channels and the input consistent.

Q6: The training parameters, like optimizer used should probably be in the main paper and not appendix. I know it's for space saving, but such things belong in the main paper.

A6: Thanks for your valuable comments, we have moved the optimizer and training parameters used in the revised paper to the main paper so that the reader can get more details quickly. At the same time, detailed descriptions have been retained in the appendix.

Q7: It is not clear if full backprob is done through the backbone, or only until the feature maps. Also if a pre-trained resnet is uses, which one is it.

A7: Thank you for this valuable comment. The entire backbone network underwent complete backpropagation during training, with feature maps being visualizations of the output from intermediate layers. In this process, the feature extraction part of the backbone network utilized a pre-trained ResNet50 to extract features.