Segmentation using efficient residual networks with attention-fusion modules

Authors: Serdar Erisen
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Segmentation, Attention mechanisms, Efficient residual networks

Abstract:
Fusing global and local semantic information in segmentation networks remains
challenging due to computational costs and the need for effective long-range
recognition. Based on the recent success of transformers and attention mechanisms,
this research applies attention-based methods of attention-boosting modules
and attention-fusion networks in enhancing the performance of state-of-the-art
segmentation networks, such as InternImage and SERNet-Former, addressing
these challenges. Integrating attention-boosting modules into residual networks
generates baseline architectures like Efficient-ResNet, enabling them to extract
global context feature maps in the encoder while minimizing computational costs.
Attention-based algorithms can also be applied to networks utilizing vision transformers
and convolutional layers, such as InternImage, to improve the existing
results of state-of-the-art networks. In this research, SERNet-Former is deployed
on the challenging benchmarking datasets such as ADE20K, BDD100K, CamVid,
and Cityscapes by depending on the attention-based methods with new implementations
of the network, SERNet-Former v2. Our methods have also been implemented
for InternImage-XL and improved the test performance of the network on
the Cityscapes dataset (85.1 % mean IoU). Respectively, the results of the selected
networks developed by our methods on the challenging benchmarking datasets are
found worth considering: 85.1 % mean IoU on the Cityscapes test dataset, 59.35
% mean IoU on ADE20K validation dataset, 67.42 % mean IoU on BDD100K
validation dataset, and 84.62 % mean IoU on the CamVid dataset.

Decision: reject
Reviewer scores: 3, 5, 1, 5

Reviews and discussion:
<thread>
creation date:
1730273879015

thread_type:
official_review

<thread_object>
creator: reviewer_sf1b

<content>
summary:
This paper proposes AbGs,AfNs to fuse global and local semantic information in segmentation. Attention-fusion networks are desined in the decoder part to improve the efficiency.

strengths:
1.This paper has made useful explorations in the fusion of global and local information, bringing some inspiration to this field.
2.Experimental results show that this method has some advantages

weaknesses:
1.The writing of this article is poor. Many sentences are not clear and not easy to understand. Some sentences are too long and difficult to understand. e.g. line 125: The multi-scale problem in computer vision can be described as the discrepancy in integrating the different sizes of spatial and channel-based semantic information of an object acquired from the global and local contexts of segmentation networks.
line 203：It is aimed at developing an encoder-decoder architecture with additional attention mechanisms to get efficient segmentation networks fusing semantic information from different contexts by regarding the multi-scale problem.

2.The method in this article lacks insight, and many designs are tricky.  e.g. Why are there two consecutive layers (AbM4, AbM5) in H/8 and W/8 resolutions? For another question, please refer to Question 2.

3.The experimental analysis is not enough, and the ablation experiment is not very sufficient. More details can refer to Question 4.

4.This paper seems to have multiple submissions，which was accepted by IEEE CVMI previously.

questions:
1. Why is the sigmoid function used as the activation function in the AbG module? Will it aggravate the gradient vanishing problem during training? Have you tried other activation functions?
2. Dilation-based convolution is used in DbN module, why not use dilation convolution in encoder and decoder part?
3. During upsampling, the image size changes from H/4, W/4 to H, W. Why not use progressive upsampling?
4. From Table 3 and Table 4, your performances are not as good as InternImage and VitAdapter-L(test mIoU). What are the parameters and inference speed(e.g millisecond) of these two methods? How do they compare to yours?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730609548988

thread_type:
official_review

<thread_object>
creator: reviewer_ueku

<content>
summary:
-

strengths:
-

weaknesses:
Considering that this paper has already been accepted by IEEE CVMI, I think it should be rejected.

questions:
-
</content>
</thread_object>

</thread>
<thread>
creation date:
1730730896072

thread_type:
official_review

<thread_object>
creator: reviewer_hfav

<content>
summary:
The paper "SERNet-Former: Segmentation by Efficient-ResNet with Attention-Boosting Gates and Attention-Fusion Networks" introduces an innovative segmentation framework that leverages advanced attention mechanisms within a robust network architecture. The authors provide comprehensive experimental results that validate their approach against existing methods, demonstrating its effectiveness across multiple datasets. However, the manuscript could be improved by addressing the limitations of the method, expanding the discussion of the ablation studies, and including a broader range of comparative benchmarks. Overall, this work represents a significant contribution to the field of image segmentation. 

**However**, I noticed that this paper has already been accepted by IEEE CVMI 2024, titled "SERNet-Former: Segmentation by Efficient-ResNet with Attention-Boosting Gates and Attention-Fusion Networks." You can view the acceptance list for the conference at https://cvmi2024.iiita.ac.in/AcceptedPapers.php. After comparing the version in the GitHub repository (https://github.com/serdarch/SERNet-Former) with the version submitted by the authors to ICLR, it appears that there are only minimal differences between the two.

strengths:
- **Innovative Approach**: The paper presents a novel method, SERNet-Former, which combines Efficient-ResNet with attention mechanisms, demonstrating a promising advancement in segmentation tasks.
- **Comprehensive Experiments**: The authors conduct extensive experiments across various datasets, showcasing the effectiveness of their approach and providing a thorough comparison with existing methods.
- **Clear Presentation**: The manuscript is well-organized, with a logical flow that makes the methodology and results easy to follow, enhancing the overall readability.

weaknesses:
- **Limited Discussion on Limitations**: The paper could benefit from a more in-depth discussion of the limitations of the proposed method, particularly in relation to different types of data or specific segmentation challenges.
- **Insufficient Detail in Ablation Studies**: While the authors present some ablation studies, additional detail on the impact of each component in the network would strengthen the understanding of their contributions.
- **Comparative Analysis**: The comparison with state-of-the-art methods could be more robust, particularly by including more recent benchmarks to provide a clearer context for the performance claims.

questions:
I noticed that this paper has already been accepted by IEEE CVMI 2024, titled "SERNet-Former: Segmentation by Efficient-ResNet with Attention-Boosting Gates and Attention-Fusion Networks."

Additionally, I have a question regarding:

1. **What specific metrics were used to evaluate the performance of SERNet-Former compared to existing segmentation methods, and how do these metrics support the claims made by the authors regarding its effectiveness?**

2. **Can the authors provide more details on the design choices behind the attention mechanisms used in SERNet-Former and how they contribute to the model's overall performance in segmentation tasks?**
</content>
</thread_object>

</thread>
<thread>
creation date:
1731062541340

thread_type:
official_review

<thread_object>
creator: reviewer_unau

<content>
summary:
This paper introduces a transformer with an encoder-decoder structure to fuse the global and local information from the image for semantic segmentation. The proposed method improves the semantic segmentation on multiple datasets, demonstrating its effectiveness.

strengths:
The proposed method improves the segmentation results based on the transformer architecture, which is lightweight.

weaknesses:
1. It should be noted that IEEE CVMI has accepted the manuscript "SERNet-Former: Segmentation by Efficient-ResNet with Attention-Boosting Gates and Attention-Fusion Networks." Although CVMI has not yet provided the official version of the accepted paper, the author has provided a GitHub repository, which indicates that CVMI has accepted the paper. Furthermore, the figures and experimental results in the GitHub repository with arXiv and CVPRW versions are the same as those in the paper submitted to ICLR. The author should clarify this.

2. Apart from the above point, I find that this paper's presentation is of low quality. It lacks the motivation to propose a new method of fusing local and global semantics, which has been well-known for improving semantic segmentation performance. I suppose this motivation is presented in the introduction, which is missed in every part of the paper. Though the performances have been compared in the experimental section, I still cannot figure out why the proposed method yields better results. Furthermore, the presentation of the method lacks the necessary information. The critical Figure 2 fails to provide a clear illustration of the method. The relationship between Figure 2 and the equations is also unclear. This fact further disallows the reader to understand the insight behind the method.

3. Though the proposed method improves the segmentation results, it still lags behind other methods on important datasets (see test set on Cityscapes in Tab 4).

Based on the above points, I believe this submission fails to meet the ICLR standard and recommend its rejection.

questions:
See the "Weaknesses" above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732019444865

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Official Comments to the Reviewers about the Revision_1

comment:
Dear reviewers, 

Thanks for your comments and evaluations.

We would like to present the last condition of the revised draft that is improved according to your comments and questions.
First, let us clarify that this paper has been completely revised to report the efficiency of our attention-based methods applied to some networks and new datasets, yet not to acclaim a specific network or work. We have ensured that this work is distinct from the mentioned papers accepted at CVMI 2024 or CVPRW 2024, and apparently has not been published elsewhere. However, we have adopted many details from the state-of-the-art works that the reviewers mention by properly citing them throughout the whole text, completely rewritten according to this revision's changing aim and motivation. Moreover, we worked on new models to assess the efficiency of our methods applied to encoder-decoder architectures using CNNs and attention mechanisms such as SERNet-Former_v2 and InternImage-XL using datasets ADE20K, BDD100K, Cityscapes, CamVid, Mapillary Vistas. Accordingly, we shared many new results in Tables and we hope that you find this revision improved.

Briefly, we have revised the title, abstract, introduction, highlights, methods, experiments, tables and figures, ablations work, conclusions, and references sections regarding your comments to improve the article. We also add new details about the implementations of the networks and datasets in the appendices that we newly worked on. 

Following the comments and questions, the authors recognized some missing details in Figure 2 and revised them as well as the narration of the motivation and methods. On the other hand, it has been decided to keep the SERNet-Former architecture, which is still the most compatible network, showing the influence of the applied methods on different parts of a network. We also recompile this architecture in PyTorch, called SERNet-Former_v2, which is not published elsewhere, and we present its implementation details and results in this revision. As discussed, this revision reports new datasets and application details. We also conducted new ablation works on SERNet-Former_v2 based on the official Cityscapes test dataset, and we will present the results and explain them. We discuss the limitations in the revision based on earlier works as well as new experiments on new models conducted during this time.
We also apply attention-boosting methods to state-of-the-art InternImage-XL architecture and present the implementation details and results in this revision. Hence, we revised the tables and results accordingly, including the results of new models that were not published elsewhere. Some of the supplementary details are moved to the appendix to make room for the new explanations of the scope, method, and implementation details and results. The sentence structures and paragraphs have been completely revised regarding your questions, including the design tricks, performance parameters, and evaluations of the paper's weaknesses.

We also answer your questions in the text where necessary. You may also find details about the questions in the official comments to the reviewers.

Hence, we hope that the efforts we put into this revision will help to change your decisions in a positive direction and may help to get this research what it deserves.

We would like to improve the text further based upon your questions and comments.

We use notations in official comments to reviewers for== 
W: Weaknesses responded; Q: Questions responded
</content>
</thread_object>

</thread>
<thread>
creation date:
1732176560751

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Additions to previous revisions

comment:
Dear reviewers and readers,

We have made new revisions to the text based on our previous revisions.
We added new figures to the paper illustrating the predicition results on the Cityscapes test dataset,
ADE20K and BDD100K datasets.

We hope that you find the paper improved from the earlier revisions.

We kindly ask your suggestions and evaluations for 
the improvement of the article further.

Regards
</content>
</thread_object>

</thread>
<thread>
creation date:
1732357669295

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Official comment on the revised version of the manuscript on 22.11.2024

comment:
Based on our earlier arguments and two revisions, we have revised the rebuttal submission further. Accordingly:

- We added a new subsection on **Attention-Boosting Modules** in the Methods section and revised the narration of this section.
- We moved the subsection **Dilation-Based Networks** to the Appendix, as it does not include attention mechanisms.
- We revised and updated the highlights and the related text in the article accordingly.
- In this regard, we also added details to the main text and Appendix about the design tricks and implementation details of the new networks based on the applied methods.
- Moreover, we added paragraphs of **Future Works and Limitations** to the Conclusion.
- Finally, we corrected sentence structures and improved the narration throughout the text.
- The proofreading stage has been completed.

We hope that you find the article improved.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732360023976

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
The Brief about our revisions

comment:
- **The title, Abstract, Introduction** and **Related Works** sections are revised.

- **Methods** section and its subsections are revised. A new subsection about **Attention-boosting Modules** is arranged.

- New results and implementation details in the methods and experiments about **SERNet-Former_v2** and **InternImage-XL**, selected to apply our methods, are provided in many places in the text.

- The details about new datasets **ADE20K** and **BDD100K** are provided and the text about existing datasets are revised.

- **Tables** are revised and the related references, where necessary, are provided.	


Table 1.

| Reference | Method | Building | Tree | Sky | Car | Sign | Road | Pedestrian | Fence | Pole | Sidewalk | Bicycle | mIoU |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CVMI 2024 |	SERNet-Former	| 93.0  |  88.8  | 95.1 |  91.9 |  73.9 |  97.7 |  76.4 |  83.4 |  57.3 |  90.3 |  83.1 |  84.6 | 

…


Table 2.
| Methods | Road | Sidewalk | Building | Wall | Fence | Pole | Traffic Light | Traffic Sign | Vegetation | Terrain | Sky  | Person   | Rider    | Car | Truck | Bus  | Train         | Motorcycle   | Bicycle    |   mIoU      | 
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| **SERNet-Former_v2††** (ours) | 98.8 | 88.3 | 94.6 | 72.6 | 69.5 | 73.3 | 78.7 | 83.2 | 94.2 | 74.7 | 96.2 | 88.9 | 76.6 | 96.5 | 84.3 | 95.3 | 92.7 | 77.0 | 79.8 | 85.0 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| InternImage-XL | 98.9 | 88.7 | 94.7 | 72.1 | 70.3 | 73.4 | 79.1 | 83.5 | 94.3 | 74.5 | 96.1 | 88.9 | 76.1 | 96.7 | 84.2 | 94.7 | 91.1 | 75.0 | 79.8 | 84.8 |
| **InternImage-XL†††** (ours) | 98.9 | 88.7 | 94.7 | 72.8 | 70.2 | 73.4 | 79.1 | 83.5 | 94.3 | 74.5 | 96.2 | 88.9 | 76.2 | 96.7 | 85.0 | 95.2 | 92.4 | 76.2 | 79.9 | 85.1 |

…


Table 3.

|  | **SERNet-Former_v2** (ours) | Efficient-ResNet v2 (ours) | 85.02 | 86.5 |
| --- | --- | --- | --- | --- |
| CVPR 2023 | InternImage-XL | UperNet | 84.85 | 86.2 |
|  | **InternImage-XL** (ours) | UperNet | 85.1 | 86.5 |

…
- New table is provided including the new models together with **inference time and parameters**.


Table 4: Performance results of models developed and tested in PyTorch

| Dataset | Model | mIoU | inference time (s/task) | parameters (M) |
| --- | --- | --- | --- | --- |
| ADE20K (2K validation) |  |  |  |  |
|  | SERNet-Former v2 (ours) | 59.35 | 0.75 | 245 |
| BDD100K (10K validation) |  |  |  |  |
|  | SERNet-Former v2 (ours) | 67.42 | 0.75 | 245 |
| Cityscapes (test) |  |  |  |  |
|  | SERNet-Former v2 (ours) | 85.02 | 0.75 | 245 |
|  | InternImage-XL (ours) | 85.10 | 0.76 | 368 |

…


- The details about the **new ablation studies** are provided in Table 5.


Table 5: Ablation studies on the Cityscapes testset

| AbM5 | Af N1 | Af N2 | DbN | mIoU |
| --- | --- | --- | --- | --- |
|  |  |  |  | 68.71 (-15.72) |
| ✓ | ✓ |  | ✓ | 77.04 (-7.39) |
| ✓ |  | ✓ | ✓ | 80.2 (-4.23) |
| ✓ | ✓ | ✓ |  | 82.6 (-1.83) |
|  | ✓ | ✓ | ✓ | 84.04 (-0.39) |
| ✓ | ✓ | ✓ | ✓ | 84.43 |

- We made some corrections in the tables and figures, including **Figure 2**.
- Thus, we also revised **the highlights** and the text of introduction and methods according to our changes.
- We add **future work** and **limitations** in the Conclusion section.

- In appendix, a new section **“Implementation details of SERNet-Former_v2 and InternImage-XL in Pytorch”** is provided.

- The existing section about the results and illustrations on the selected datasets in the Appendix is expanded.
For instance, illustrations from SERNet-Former_v2 test results are added in the Cityscapes testset illustrations. 
The illustrations from ADE20K & BDD100K are added including the prediction results of SERNet-Former_v2.

- Some methods and supplementary results are moved to Appendix.


- Finally, we corrected **the sentence structures** and **the narration of the text**. The proofreading stage is completed by getting professional help.

We hope that the revisions meet your expectations and you find the article improved.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732649345150

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Little corrections in the text, 26th November, 19:26 GMT

comment:
Dear reviewers,

We have made little corrections in the text by finding some erroneous sentences in technical means, corrected.

Best regards
</content>
</thread_object>

</thread>


Metareview:
metareview:
In this work, most reviewers point out that this paper has a minimal difference from the "SERNet-Former: Segmentation by Efficient-ResNet with Attention-Boosting Gates and Attention-Fusion Networks" accepted by IEEE CVMI. After comparing the two papers, the AC agrees and thus votes for rejection.

Overall, the summary of this paper is as follows:

- **Strengths**:
  - Comprehensive revisions with new datasets, methods, and benchmarking results.
  - Improved clarity and detailed responses to reviewer concerns.
  - Significant effort in addressing presentation and originality issues.

- **Weaknesses**:
  - Persistent concerns about duplicate submission and minimal novelty.
  - Incremental improvements in methods compared to prior works.
  - Questions about scalability and broader applicability remain unresolved.