Dual-Stream Adapters for Anomaly Segmentation

Authors: Shyam Nandan Rai, Massimiliano Mancini, Marco Ciccone, Iuliana Georgescu, Zeynep Akata, Barbara Caputo, Carlo Masone
Primary area: applications to robotics, autonomy, planning
Keywords: Adapters, Anomaly Segmentation

Abstract:
Anomaly segmentation aims to identify pixels of objects not present during the model’s training. Recent approaches address this task using mask-based architectures, but these methods have high training costs due to the large transformer backbones involved. While vision adapters can help reduce training costs, they are not specialized for this task, leading to inferior performance. In this work, we propose Dual-Stream Adapters (DSA), a vision adapter tailored for anomaly segmentation. DSA extracts both in-distribution and out-of-distribution features via (i) an anomaly prior module that produces separate initial embeddings for the two streams; and (ii) a dual-stream feature refinement that implicitly guides the separation of in-distribution from out-of-distribution features. We train DSA using a novel hyperbolic loss function that provides supervised guidance for differentiating in-distribution and out-of-distribution features. Experiments on various benchmarks show that dual-stream adapters achieve the best results while reducing training parameters by 38\% w.r.t. the previous state-of-the-art.

Decision: reject
Reviewer scores: 6, 5, 3

Reviews and discussion:
<thread>
creation date:
1729655136118

thread_type:
official_review

<thread_object>
creator: reviewer_wr5a

<content>
summary:
This paper introduces a novel adapter architecture for anomaly segmentation. The architecture incorporates two key architectural components: anomaly prior modules and dual-stream feature refinement tailored to improve anomaly segmentation. The anomaly prior
module learns to extract the initial ID and OOD features that are refined and improved by passing through a set of dual-stream feature refinement blocks. It introduces an uncertainty-based hyperbolic loss to explicitly learn the ID and OOD features. The results on several datasets in terms of some metrics show the effectiveness.

strengths:
+ a novel adapter architecture for anomaly segmentation. It designs anomaly prior modules and dual-stream feature refinement tailored to improve anomaly segmentation. 
+ The results on several datasets in terms of some metrics are the SOTA.

weaknesses:
- The definaton of this task is unclear. How to define the Anomaly regions. For example, I put the first input of Fig. 5 into the ChatGPT and ask it to indicate the abnormal regions. It responses: "
From the image, it appears to show a traffic checkpoint or roadblock scene with a few police officers, some cones marking the road, and vehicles either being stopped or passing through the checkpoint. To analyze any potential abnormal regions, I would need specific criteria for identifying abnormalities, such as:
Traffic flow: Are there any unusual movements or traffic behavior?
Objects: Are there any misplaced cones, vehicles, or people?
Security concerns: Is there anything out of place, like suspicious behavior or unmarked vehicles?
If you're looking for more detailed insights, could you clarify what type of abnormality you're focused on—whether it's safety, traffic irregularities, or something else?"
This answer is more reasonable, since the anomaly regions can be different in different settings.

-The above example gives another question. The current multimodal large language model already achieves better responses.

- The preposed method lacks novelty, and it is mainly combined by Vit adaptors with cross-attention.

questions:
Based on the above the weaknesses, I have the following questions: 

- What are the key criteria or benchmarks used to evaluate anomaly detection in your scenario?
Defining the task clearly with precise criteria is critical. How do you plan to measure the model's success in detecting abnormal regions?

- What are the distinguishing features of your method compared to existing multimodal models?

- How does the model handle the contextual differences in anomaly definitions?
Since anomalies are highly context-dependent, is your model trained to recognize such context? How does it deal with shifting or ambiguous definitions of normalcy versus anomaly?

- Are there plans for improving the generalization and adaptability of the model?
For instance, using domain adaptation techniques or fine-tuning the model on specific datasets related to the task could improve its performance in different scenarios.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730363402465

thread_type:
official_review

<thread_object>
creator: reviewer_r2z4

<content>
summary:
The paper aims to reduce the high training cost of the transformer-based anomaly segmentation model. The authors introduce the Dual-Stream Adapter (DSA) to tackle this issue, specifically designed for anomaly segmentation tasks. The DSA comprises two main components: an anomaly prior module and a dual-stream feature refinement module. The anomaly prior module is responsible for learning both in-distribution and out-of-distribution features through separate feature-level encodings. Meanwhile, the dual-stream feature refinement module enhances and integrates these features with the robust representations derived from a frozen Vision Transformer backbone. Additionally, the architecture employs a hyperbolic loss function to guide the learning process effectively. The authors carried out extensive experiments across five datasets, demonstrating their proposed model's effectiveness.

strengths:
+ The paper is clearly structured, featuring well-organized paragraphs and accessible figures that enhance comprehension.
+ The design of the proposed Dual-Stream Adapter is sound and well-motivated.

weaknesses:
- The experiments conducted were compared to several competitors; however, the values reported in Table 3 do not align with those in the original publications. This inconsistency raises questions about the validity of the experimental outcomes. It is highly advisable for the authors to verify the accuracy of the reported experiments to ensure reliable results.

- It appears that there is an inconsistency between Figure 3 and equations (6) and (7). It would be beneficial to verify their accuracy.

- It would be interesting to analyze the amount of training time that can be saved due to the reduction in trainable parameters.

- There are some related papers [A-B] that have not yet been cited in the current version.

    [A]  Yuanpeng Tu, Yuxi Li, Boshen Zhang, Liang Liu, Jiangning Zhang, Yabiao Wang, Cairong Zhao: Self-Supervised Likelihood Estimation with Energy Guidance for Anomaly Segmentation in Urban Scenes. AAAI 2024: 21637-21645

    [B] Tomás Vojír, Jirí Matas: Image-Consistent Detection of Road Anomalies as Unpredictable Patches. WACV 2023: 5480-5489

questions:
This paper is well-structured and has a clear motivation. However, a significant concern arises from the experiments reported, as the values presented appear inconsistent with those in the original publications. At this stage, I prefer to take a cautious approach to the rating and await the author's response before reaching a final conclusion.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730711008164

thread_type:
official_review

<thread_object>
creator: reviewer_wmg8

<content>
summary:
This paper introduces a ViT adapter designed specifically for anomaly detection in semantic segmentation. The approach begins by extracting multi-scale features using a ResNet convolutional stem. Two distinct sets of learnable encodings are added to these features to specialize them for in-distribution (ID) and out-of-distribution (OOD) learning. These initial features are then fused with ViT backbone activations through a series of cross-attention layers and feed-forward networks, maintaining separate streams for ID and OOD features throughout the entire downstream path. The model is trained using an uncertainty-based hyperbolic loss function, where features are projected into hyperbolic space: OOD features are attracted towards the origin of the Poincaré ball, while ID features are repelled away from it.

The authors evaluate their dual-stream adapter on road-driving anomaly detection benchmarks, comparing its performance to other ViT adapters and alternative outlier detection methods. Additionally, they conduct an ablation study to assess the impact of individual components of their proposed method.

strengths:
1) Method is simple and more parameter efficient than existing outlier detection methods.
2) The method preserves ID performance well.

weaknesses:
1) Mixed results in terms of OOD detection on different datasets, both in comparison to fully fine-tuned models, but also different adapter types.
2) I do not agree that using void/background class makes this training self-supervised, as there is explicit supervision in terms of labelling pixels into this additional class. Furthermore, the method could not be used on datasets that do not have ignore regions. It would maybe be better to follow the convention of using the term "auxiliary data". It  would be useful if this was reflected in the organization of Table 3. 
3) Missing methods in ood-detection tables. e.g. [a] is the current leader on SMIYC-RA
4) Some things about the method are still not clear -> see questions.

[a] Delic et al., Outlier detection by ensembling uncertainty with negative objectness, BMVC 2024

questions:
1) How are the encoder and adapter features integrated before being passed to the decoder as described in Equation 3?
2) Is the background class utilized in the computation of L_{mask} and L_{class}​?
3) Is the convolutional module within the anomaly prior module also fine-tuned? Was it pretrained, and if so, how? What specific ResNet variant is used?
4) Can you provide detailed specifications of the Cross-Attention and Feed-Forward Network (FFN) modules used in the adapters?
5) How is the void/background class incorporated into L_{ubhl}? At what stage is the ground truth label applied?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733216840278

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank all the reviewers for their suggestions and comments. In our responses, we tried to address all the concerns. Please, let us know if any questions should remain, we are available for further clarification.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a ViT adapter for anomaly detection in semantic segmentation. The proposal uses two streams for in-distribution and out-of-distribution, respectively. The model is trained using an uncertainty-based hyperbolic loss.

Strengths:
- Simple method
- Uses fewer parameters than existing outlier detection methods
- Clear and well-organized paper

Weaknesses:
- Mixed results for OOD detection
- The baselines used for comparison are questionable since they do not align with reports in the literature
- The setup is not self-supervised as there is explicit supervision by labeling pixels into the additional class
- The experimental setup is unclear, and the differences between the presented results and those in the literature are not clearly explained.

While the paper received a wide range of recommendations and the understanding of the problem varies among reviewers, there are major concerns regarding the validity of the experimental results. Despite the authors' efforts during the rebuttal, there are still concerns about the validity of the experiments, and it is unclear why the setups differ from the literature, leading to different results. Moreover, the strengths raised do not outweigh these concerns. The proposal is borderline, but due to the lack of stronger reasons to accept the paper, I lean towards rejecting it.