ViT-UWA: Vision Transformer Underwater-Adapter for Dense Predictions Beneath the Water Surface

Authors: Qirui LIN, Hua Li, Yuheng Jia, Yutong Li, Shijie Lian, Huazhong Liu, Sam Kwong, Runmin Cong
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Underwater Image Dense Prediction, Adapted ViT Backbone

Abstract:
Vision Transformer (ViT) and its variants have witnessed a significant success in computer vision. However, they do not perform well in underwater dense prediction tasks due to challenges like complex underwater environments, quality degradation, and light scattering in underwater images. To solve this problem, we propose the  Vision Transformer Underwater-Adapter (ViT-UWA), the first detail-focused and adapted ViT backbone for underwater dense prediction tasks, without requiring task-specific pretraining. In ViT-UWA, we first introduce High-frequency Components Prior (HFCP) to add high-frequency information of underwater images to the plain ViT, which can help recover and capture lost high-frequency information of underwater images. Then, we propose an Detail Aware Module (DAM) to obtain a detail-focused multi-scale convolutional feature pyramid, which can be used in kinds of dense prediction tasks. Through the ViT-CNN Interaction Module (VCIM), we achieve bidirectional feature fusion between ViT and CNN. We evaluate ViT-UWA on multiple underwater dense prediction tasks, including semantic segmentation, instance segmentation, and object detection. Notably, with only ImageNet-22K pretraining, our ViT-UWA-B yields state-of-the-art 46.4 box AP and 44.2 mask AP on USIS10K dataset. We hope ViT-UWA could provide a new backbone for future research on underwater dense prediction tasks.

Decision: reject
Reviewer scores: 8, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1729412016730

thread_type:
official_review

<thread_object>
creator: reviewer_jkrf

<content>
summary:
This paper addresses the dense prediction tasks (semantic segmentation, instance segmentation and object detection) from underwater images via an adapted Vision Transformer (ViT) model. The work (named ViT-UWA) adds the High-Frequency Component Prior (HFCP) into the tokens for the plain ViT, and designs two modules of Detail Aware Module (DAM) and ViT-CNN Interaction Module (VCIM), where DAM is devised to extract detail-focused multi-scale features for VCIM and VCIM interacts with the plain ViT to fuse and refine the features for dense predictions. The experiments were conducted on three underwater image datasets, covering semantic segmentation, instance segmentation and object detection, compared with existing general ViT adapters and underwater dense prediction methods.

strengths:
1. The effectiveness and efficiency of the proposed method seems good compared to the related works (Figure 2).
2. The proposed method ViT-UWA has advantages to adapt the underwater images with specific degradation on the details (Figure 3).

weaknesses:
1. While the proposed method achieves good performance, it seems that the claimed contributions, i.e., HFCP, DAM and VCIM, are designed by considering some related works, thus it is somewhat questionable about the novelty of the design.
2. For efficiency evaluation, it's better to consider the aspects of Parameters, FLOPs, as well as training and inference time, especially the FLOPs and inference time for real-word applications.
3. The design seems to only consider the detail loss (HFCB and DAM) on underwater images, while the other degradations caused by the light absorption and scattering on underwater images are not considered, so how does the model learn these different degradations for dense prediction tasks.

questions:
1. What are the specific differences on each contribution compared to existing similar works?
2. Please consider to evaluate the efficiency towards real-world applications.
3. How does the proposed model learn different degradations on underwater images for dense prediction tasks?
4. Please provide the discussion about the SAM 2 and proposed ViT-UWA.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729514545507

thread_type:
official_review

<thread_object>
creator: reviewer_nqpn

<content>
summary:
This paper introduces a new backbone, ViT-UWA, specifically designed for underwater vision tasks. ViT-UWA consists of two branches: a ViT branch and a convolutional branch, which leverage the ViT-CNN Interaction Module (VCIM) to facilitate information sharing between them. Moreover, the ViT branch incorporates high-frequency components as additional input, while the CNN branch utilizes High-frequency Detail Convolution (HFDConv) to focus details. The model has achieved good results in several underwater tasks, such as object detection and instance segmentation.

strengths:
1. The paper is well written, and the methods are introduced very clearly.
2. The paper conducted experiments on many tasks, demonstrating the performance advantages of ViT-UWA.

weaknesses:
1. The biggest drawback of this paper is the lack of originality; the proposed ViT-UWA is composed of existing module combinations, such as the adaptive DC in HFDConv and the Deform Attention in VCIM. The so-called HFCP is merely the concatenation of the high-pass filtered result of the original image with the original image input to the model. Therefore, I believe the contribution of this paper to the community is insufficient.
2. Although ViT-UWA is specifically designed for underwater tasks, some unique designs, such as high-frequency components and detail information, are important in any scenario. However, ViT-UWA performs mediocrely on the Coco dataset in Table 10, while it performs well in underwater scenes. The authors' motivation for the model design does not convince me, and the reason for ViT-UWA's better performance in underwater scenarios remains unclear.

questions:
1. Should the phrase "... improve by 2.3 and 0.6 AP" in line 504 be changed to "decrease by 2.3 and 0.6 AP"?
2. What would happen if HFCP were directly removed from Table 7 instead of being replaced by USUIR? Additionally, have any better supervised underwater image enhancement methods been tried instead of the unsupervised USUIR method, which often performs worse than supervised methods in many scenarios?
3. Why does VCIM use Deform Attention instead of ordinary cross attention?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730543893938

thread_type:
official_review

<thread_object>
creator: reviewer_zvdc

<content>
summary:
In this paper, a novel underwater dense prediction backbone by combining the plain ViT. Besides, the high-frequency components prior and the detail aware module are proposed. The proposed method was trained and tested on multiple datasets and achieved comparable results. Finally, multiple ablation experiments illustrate the effectiveness of high-frequency components prior and the detail aware module in this method.

strengths:
1. In this paper, multiple experiments were conducted on different datasets and compared quantitatively and qualitatively with multiple methods.
2. This paper is generally smooth, and the paper is easy to understand.
3. The visualization results of the method proposed in the paper look better than other compared methods.

weaknesses:
1.	In ablation experiments, only indicators show whether the high-frequency module is useful or not, but there is a lack of visuals to show that the module can capture high-frequency information.
2.	In the ablation experiment, only the DAM was removed, but other simple CNN structures or Transformer structures were not considered to replace the DAM, which did not well illustrate the uniqueness and effectiveness of the module

questions:
1. Can you show some visuals to illustrate that HFCP can capture high-frequency information?
2. Can you replace DAM with other simple structures to illustrate the uniqueness and effectiveness of your DAM?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730734531824

thread_type:
official_review

<thread_object>
creator: reviewer_pmdp

<content>
summary:
The paper proposes adapted backbone for the Vision Transformer (ViT) architecture termed as Vision Transformer Underwater Adapter (ViT-UWA) specifically designed for dense prediction tasks in images acquired underwater. The two salient features of the proposed ViT-UWA are (a)  its ability to recover and capture high-frequency information in underwater images via the introduction of a high-frequency components prior (HFCP) and (b) it ability to capture features at multiple scales of detail for dense prediction tasks via a detail aware module (DAM) that is based on a multi-scale detail-focused convolutional feature pyramid.

strengths:
The paper proposes a useful enhancement of the standard ViT architecture. The experimental results are encouraging when compared to other state-of-the-art (SOTA) methods for dense prediction in underwater imagery.

weaknesses:
The high frequency signal attenuation observed in underwater imagery is very dependent on the environmental conditions. The proposed method should have incorporated underwater image signal correction that is based on the physics of light scattering under water such as the Sea-Thru algorithm [1].  

[1] D. Akkaynak and T. Treibitz, "Sea-Thru: A Method for Removing Water From Underwater Images," 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019, pp. 1682-1691, doi: 10.1109/CVPR.2019.00178.

questions:
The authors should clarify how they have addressed or would address the underlying physics of underwater signal correction.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732630062557

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We sincerely thank all the reviewers for their efforts in reviewing our paper and for providing valuable and insightful feedback.

We are pleased to see that the reviewers found our work to be a **useful enhancement** of the standard ViT architecture, specifically noting its applicability to underwater imagery (`pmDp`, `JkrF`). We also appreciate the recognition of the **extensive experiments** conducted on different datasets, both quantitatively and qualitatively (`pmDp`, `ZvDc`, `nQpN`, `JkrF`), and the **superior visualization results** compared to other methods (`ZvDc`). Furthermore, we are encouraged by the feedback that the paper is **well-written**, the methods are clearly introduced (`nQpN`, `ZvDc`). 

In the revised paper and our responses to each reviewer, we have included additional insights, visual results, and ablation experiments to further clarify and strengthen the understanding of our method. We welcome any further questions or comments from the reviewers and are happy to provide additional clarifications or explanations as needed.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper has received mixed reviews with widely varying scores. Reviewers generally commend the authors for the comprehensive quantitative and qualitative results obtained across a large number of datasets, as well as for the clear and well-organized presentation of the paper.

The primary concerns stem from Reviewer nQpN and JkrF, who argue that the design of HFCP, DAM, and VCIM lacks sufficient novelty. These reviewers suggest that the proposed modules appear to be designed with consideration of existing work, which raises questions about the originality of the approach.

In light of the feedback, while the paper presents valuable insights and demonstrates solid results, the concerns about the novelty and innovation of the proposed methods cannot be overlooked. Thus, based on the current evaluation, the paper does not meet the threshold for acceptance.