Unconstrained Salient and Camouflaged Object Detection

Authors: Zhangjun Zhou, Yiping Li, Chunlin Zhong, Jianuo Huang, Jialun Pei, He Tang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Salient object detection, Camouflaged object detection, SAM, Benchmark

Abstract:
Visual Salient Object Detection (SOD) and Camouflaged Object Detection (COD) are two interrelated yet distinct tasks. Both tasks model the human visual system's ability to perceive the presence of objects. The traditional SOD datasets and methods are designed for scenes where only salient objects are present, similarly, COD datasets and methods are designed for scenes where only camouflaged objects are present. Scenes where both salient and camouflaged objects coexist, or where neither is present, are not considered. This simplifies the existing research on SOD and COD. In this paper, to explore a more generalized approach to SOD and COD, we introduce a benchmark called Unconstrained Salient and Camouflaged Object Detection \textbf{(USCOD)}, which supports the simultaneous detection of salient and camouflaged objects in unconstrained scenes, regardless of their presence. Towards this, we construct a large-scale dataset, \textbf{CS12K}, that encompasses a variety of scenes, including four distinct types: scenes containing only salient objects, scenes with only camouflaged objects, scenes where both salient and camouflaged objects coexist, and scenes without any objects. In our benchmark experiments, we find that a major challenge in USCOD is distinguishing salient objects from camouflaged objects within the same model. To address this, we propose a USCOD baseline called \textbf{\ourmodel}, which freezes the SAM mask decoder for mask reconstruction, allowing the model to focus on distinguishing between salient and camouflaged objects. Furthermore, to evaluate models’ ability to distinguish between salient and camouflaged objects, we design a metric called Camouflage-Saliency Confusion Score (\textbf{CSCS}). The proposed method achieves state-of-the-art performance on the newly introduced USCOD task. The code and dataset will be publicly available.

Decision: reject
Reviewer scores: 3, 3, 5, 8

Reviews and discussion:
<thread>
creation date:
1730534488255

thread_type:
official_review

<thread_object>
creator: reviewer_luqd

<content>
summary:
This paper introduces a new benchmark: USCOD, where both salient and camouflaged objects are considered during detection. Towards this, the authors construct a new dataset CS12K, containing four types of occasions, and proposes a new baseline for this benchmark. Furthermore, they design a new metric called CSCS to evaluate the performance of this task.

strengths:
- The authors point out the current limitation of SOD and COD tasks, and introduce a new challenging task USCOD, combining SOD and COD together, which can improve the generalizability of both tasks.
- The authors carefully construct a new dataset for USCOD, which can contribute to this certain area.
- The competitive experiments demonstrate the essential to specially design datasets and networks for USCOD.

weaknesses:
- More details about the CS12K are expected. For example, when manually annotating, how is the object judged to be as salient or camouflaged?
- The loss function of the proposed method is not stated clearly. In equation (4), is there some difference when calculating loss for salient and camouflaged objects? If not, the different difficulties of COD and SOD may make the optimization less effective.
- Also, when evaluating, there can be some factors influencing the actual performance. For example, the different difficulty of the two tasks: if the model performs well at SOD but poor at COD, will it achieve the same score on CSCS with another model which performs averagely at both tasks? Or will the size [1] and number [2] of objects influence the evaluation? The authors are expected to conduct some discussion on the evaluation.

[1] Li F, Xu Q, Bao S, et al. Size-invariance Matters: Rethinking Metrics and Losses for Imbalanced Multi-object Salient Object Detection. Proceedings of the 41st International Conference on Machine Learning, PMLR 235:28989-29021, 2024.

[2] Mingchen Zhuge, Deng-Ping Fan, Nian Liu, Dingwen Zhang, Dong Xu, and Ling Shao. Salient object detection via integrity learning. IEEE TPAMI, 45(3):3738–3752, 2022.

questions:
1. How do you judge whether an object is salient or camouflaged? Does this process align with popular datasets?

2. When evaluating and optimizing, do you give equal treatment to salient and camouflaged objects?  If so, why not give different weights to the two types of objects considering their difference in difficulty?

3. In the metric CSCS, I did not see the $P_{CB}$ and $P_{SB}$, which reflects the false positive rate to some extent. Can you explain the reason why $P_{CB}$ and $P_{SB}$ are not considered?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730636633516

thread_type:
official_review

<thread_object>
creator: reviewer_fwyu

<content>
summary:
This paper addresses both salient object detection (SOD) and camouflaged object detection (COD). Specifically, the authors first introduce a benchmark called Unconstrained Salient and Camouflaged Object Detection (USCOD) and construct a large database, CS12K. They then propose a SAM-based baseline to tackle USCOD.

strengths:
1. The construction of the CS12K dataset has contributed to the further development of this task.
2. This paper proposes a new metric for simultaneously evaluating saliency and camouflage detection.
3. The writing is good and easy to read.

weaknesses:
1. Is the comparison with other models fair? Were the other models compared using the same training data as this model?
2. The model design in this paper does not look very novel. Specifically, this paper proposes an adapter prompting strategy based on a cross-attention mechanism, which is very similar to [1]. 
3. The authors repeatedly state that distinguishing salient and camouflaged objects within the same image is challenging. However, theoretically, as salient and camouflaged are antonyms, it should not be excessively difficult to differentiate between them.

[1] Sun, Yanpeng, et al. "VRP-SAM: SAM with visual reference prompt." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

questions:
1. Is the comparison with other models fair? Were the other models compared using the same training data as this model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730697895635

thread_type:
official_review

<thread_object>
creator: reviewer_224k

<content>
summary:
This work aims to solve the unconstrained salient and camouflaged object detection problem. It is a challenging task, that needs to jointly detect the potential salient and/or camouflaged objects in the image. To this end, they propose the first joint salient and camouflaged object detection dataset, CS12K. They also propose a solution, USCNet, for the task. The experimental results shown their effectiveness, in particularly in scenes containing both salient and camouflaged objects.

strengths:
1. They propose a great USCOD dataset, CS12K. Different from previous SOD/COD dataset, CS12K includes images containing both salient and camouflaged objects jointly. It is more challenging and may be practical in some scenes.

2. They conduct sufficient experiments to proof the effectiveness of their approach.

weaknesses:
1. The motivation of this work is unclear. I am curious that why we need to jointly detect salient and camouflaged objects without any pre-defined prompts? I cannot see critical problems if using the pre-defined prompts. May the authors add more explanations regarding this?

2. The novelty of this work is very limited. I cannot see the technique novelty. One APG is proposed, but the APG design is not novel, since using different type of prompts for different task setting is usual. The idea of the proposed method is not novel as well. May the author highlight it? 

3. The contribution of this work is insufficient. While it proposes a great dataset, the motivation is unclear, and the idea and method are not novel to me. I believe the contribution is not enough.

questions:
1. In L088-L091, "The visual prompt of EVP, as part of the model, needs to be retrained according to the datasets of different tasks and requires pre-defining the category of the detection task. Similarly, the task prompts in VSCode and Spider also need to be pre-defined. Similarly, the task prompts in VSCode and Spider also need to be pre-defined." The "Similarly" here is confusing. Do VSCode and Spider need to be retrained? Or only require pre-defined prompts? Thanks for classify this!

2. If VSCode and Spider do not need to be retrained, I am curious about their evaluation protocols, as well as that of EVP (Liu 2023). How are the prompts set for scenes A, B, C, and D? I cannot find any details regarding this, but I believe it is critical since it directly relates to the fairness and necessity of using the same type of prompt for the USCOD task. Thus, I hope the authors can make the evaluation protocols clearer (this is important).

3. Regarding the CSCS metrics, based on the formulation, it seems they only consider the confusion between salient and camouflaged pixels, while the false positives and false negatives between salient/camouflaged pixels and the background are not considered. Therefore, we still need to look at mIOU to understand the quality of the method. However, mIOU can also reveal aspects such as the confusion between salient and camouflaged pixels. Thus, I wonder why we still need CSCS metrics?

4. If the paper is accepted, will the dataset be released in its “full version”? By this, I mean the version containing the entire training set (8,400 images and the corresponding masks) and the testing set (3,600 images and the corresponding masks), along with the scene types.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730811654897

thread_type:
official_review

<thread_object>
creator: reviewer_htmz

<content>
summary:
This work proposes a USCOD benchmark to bridge the difference between salient and camouflaged object detection. The authors explains that, although there are many works highlighting COD and SOD individually, USCOD tries to generalize the scenario by detecting both camouflaged and salient objects. To achieve that, the authors propose a new dataset CS12K, a new evaluation metric to understand the semantic difference between COD and SOD and present a baseline method USCNet. In USCNet, the authors considers the SAM com-
ponents including SAM encoder and mask decoder. Along with it, the authors proposes attribute specific prompt generation for saliency, camouflaged, and background prompts.

strengths:
+The authors propose a new dataset CS12K which combines both salient and
camouflaged scenarios.
+The authors present a thorough experimental validation under all the
mentioned scenarios and outperform in all.
+The authors also present a new metric to distinguish between the salient
and camouflaged regions in the image

weaknesses:
- This work tackles only extremes, salient or camouflaged objects. What about intermediate scenarios, for e.g., non-salient or less-salient objects. What about objects with complex backgrounds? If we are trying to generalize, it should cover the entire range, not just the two extremes.  
 
- The scenario of multiple objects seems missing from the dialogue. Although the authors have proposed the dataset by collecting most of the images from previous benchmark datasets, the presence of multiple camouflaged/salient objects (as per real-world scenario) seems to be missing from the dataset or model itself.

- Salient and camouflaged objects belong to different distributions based on their visual appearance and structural complexity. However, the loss function or the network does not cater to that. Is there a way to analyze this difference in the distribution of salient and camouflaged objects so that it does not impact the final prediction?

- Although the authors leverage the SAM-adapter idea, it's strange that they haven't compared with it, which does provide COD results. Also, please provide results with other recent COD and SOD works. Some of them are listed below:
1) CamoFocus [1]
2) CamoDiffusion [2]
3) CamoFormer [3]
4) PGT [4]

- It is unclear how the static prompt queries (SPQ) are generated. Moreover, the operation to predict the coarse operation is also poorly explained. It is hard to see the role of combining dynamic prompt queries (DPQ) and static prompt queries. Moreover, the motivation for the DPQ method is also unclear; authors should explain the reason why a learning component is required for DPQ calculation. Additionally, authors should explain the motivation and how they employ self-attention with salient, camouflaged, and background queries and further generate the prompts for the same.

- In line 355, the authors state that they employ self-attention, while in line 352, they mention cross-attention. This statement is contradictory and unclear.

- In table 5, there seems to be some ambiguity in the results. Specially for ICEG, the results in the manuscript provides different results than what the paper has provided (in all combined settings). Some light on this issue is required.

References:
[1] Abbas Khan, Mustaqeem Khan, Wail Gueaieb, Abdulmotaleb El-Saddik,
Giulia De Masi, and Fakhri Karray. Camofocus: Enhancing camouflage ob-
ject detection with split-feature focal modulation and context refinement. In
IEEE/CVF Winter Conference on Applications of Computer Vision, WACV
2024, Waikoloa, HI, USA, January 3-8, 2024, pages 1423–1432. IEEE, 2024.
[2] Zhongxi Chen, Ke Sun, and Xianming Lin. Camodiffusion: Camouflaged
object detection via conditional diffusion models. In Michael J. Wooldridge,
Jennifer G. Dy, and Sriraam Natarajan, editors, Thirty-Eighth AAAI Confer-
ence on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innova-
tive Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium
on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27,
2024, Vancouver, Canada, pages 1272–1280. AAAI Press, 2024.
[3] Bowen Yin, Xuying Zhang, Deng-Ping Fan, Shaohui Jiao, Ming-Ming
Cheng, Luc Van Gool, and Qibin Hou. Camoformer: Masked separable atten-
tion for camouflaged object detection. IEEE Transactions on Pattern Analysis
and Machine Intelligence, pages 1–14, 2024.
[4] Rui Wang, Caijuan Shi, Changyu Duan, Weixiang Gao, Hongli Zhu,
Yunchao Wei, and Meiqin Liu. Camouflaged object segmentation with prior via
two-stage training. Comput. Vis. Image Underst., 246:104061, 2024

questions:
Could you please address the issues raised in the weakness section?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732900986488

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Gratitude for Reviewers' Feedback and Summary of Contributions

comment:
Dear AC and Reviewers,

We would like to express our sincere gratitude for your valuable feedback and insightful comments on our paper. Below, we provide a summary of our contributions and a brief acknowledgment of the reviewers’ recognition.

**Our Contributions:**
1. **New Benchmark: USCOD**  
   We propose a new benchmark called USCOD, designed to detect both salient and camouflaged objects in unconstrained scenes. This benchmark addresses the limitations of traditional Salient Object Detection (SOD) and Camouflaged Object Detection (COD) tasks, which generally focus on specific types of scenes.

2. **New Dataset: CS12K**  
   We introduce CS12K, a large-scale dataset for USCOD, consisting of four diverse scenes with both mask-level and attribute annotations. To our knowledge, CS12K is the first dataset to cover a variety of scenes without restricting the presence of salient or camouflaged objects, offering a more comprehensive challenge for current detection methods.

3. **New Evaluation Metric: CSCS**  
   We propose the Camouflage-Saliency Confusion Score (CSCS), a new metric that evaluates the confusion between camouflaged and salient objects. This fills a crucial gap in existing evaluation methods, which typically do not address the unique challenges presented by scenes containing both object types.

4. **Proposed Baseline for USCOD: USCNet**  
   We introduce USCNet, a baseline method for USCOD that combines Static Prompt Query (SPQ) and Dynamic Prompt Query (DPQ) to query both generic and sample-specific features. This allows USCNet to generate discriminative attribute-specific prompts for detecting both salient and camouflaged objects, which achieves the new state-of-the-art performance on USCOD task.

5. **Extensive Experiments and Comparative Analysis**  
   We conduct extensive experiments comparing USCNet with 19 existing SOD and COD methods on the CS12K dataset across four different scenes, providing a thorough analysis of performance gaps and challenges within USCOD. 

**Summary:**

Our work introduces a novel benchmark, dataset, evaluation metric, and baseline method for USCOD. We believe these contributions can advance the field of salient and camouflaged object detection, paving the way for **future research** in these areas. Additionally, this work holds significant potential for **application** in special scenario analysis, where it plays a crucial role in improving operational efficiency and enhancing safety.

**Acknowledgments:**

- **Contribution of CS12K Dataset:** We sincerely thank **all reviewers** for recognizing the significant contribution of the CS12K dataset to the field of USCOD research.

- **Value of the CSCS Metric:** We are grateful to **Reviewer Htmz and Reviewer FwYu** for acknowledging the value of the CSCS as an essential evaluation metric in handling scenes with both types of objects.

- **Validation of Our Approach:** Thanks to **Reviewer Htmz, Reviewer 224k, and Reviewer LUqD** for acknowledging the validity of our approach, which has been thoroughly verified through extensive experiments. We also appreciate their recognition of the necessity for designing specialized datasets and networks for USCOD.

We deeply appreciate your support and constructive feedback, which has greatly contributed to the improvement of our work. 

Thank you for your time and consideration.

Best regards,

Authors of #38
</content>
</thread_object>

</thread>


Metareview:
metareview:
In this paper, the authors presented a new task -- unconstrained salient and camouflaged object detection (USCOD), that linked the conventional tasks of visual salient object detection (SOD) and camouflaged object detection (COD). Specifically, a USCOD benchmark was introduced, aiming at the simultaneous detection of salient and camouflaged objects. A large-scale dataset (CS12K) was constructed to support this. A baseline model was presented to address this problem. Experimental results show the effectiveness of the proposed method on this new USCOD task. The main strengths of this paper are:
- A new USCOD benchmark was introduced, with a supported dataset CS12K and a corresponding new evaluation metric. The large-scale dataset could be a good contribution to either the salient object detection community, the camouflaged object detection community, or the newly introduced USCOD task.
- A thorough experimental evaluation was presented, showing the effectiveness of the proposed method.
- The paper is generally well-written and easy to follow.

The main weaknesses of this paper include:
The definition, motivation, and rationale of the introduced USCOD task are questionable. This is a major concern shared by the reviewers and AC. The designed USCOD task was considered to tackle only extremes, salient or camouflaged objects, with an unclear practical application scenario. The corresponding technical design was also unclear.
- The novelty and contribution of the proposed method were found to be limited. The technical design of the components proposed was not that novel, given similar designs have been proposed in prior works (more details please see the review comments). Whereas the authors failed to provide a detailed convincing clarification and comparison for this.
- Some of the technical designs and experimental settings lack information, and some statements are unclear without convincing evidence to support them. For more details, please refer to the review comments.

Although some minor concerns were addressed by the authors through the rebuttal and discussions, the major concerns (especially the first two mentioned above) remain. Even after reading the authors' explanation, the reviewers and AC are still not convinced by the designed new USCOD task, there is a considerable disagreement over this. This new task is more like an artificially designed task rather than a real challenging scenario/problem that was well motivated. Before clearing this key issue, the paper in its current form is considered to be not ready to present at ICLR. Apart from this, the novelty concern also stands. The AC (and reviewers) understand and appreciate the good contribution of the new large-scale dataset this paper brings to the community, but considering the major concern about the key problem this paper studied, the AC agreed with the reviewers that this paper is not ready to be presented at ICLR, and encourage the authors to carefully consider the reviewers' comments (especially the key concern mentioned above) to revise their paper.