Text-prompt Camouflaged Instance Segmentation with Graduated Camouflage Learning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-09 12:44:39

Abstract: Camouflaged instance segmentation (CIS) aims to seamlessly detect and segment objects blending with their surroundings. While existing CIS methods rely heavily on fully-supervised training with massive precisely annotated data, consuming considerable annotation efforts yet struggling to segment highly camouflaged objects accurately. 
Despite their visual similarity to the background, camouflaged objects differ semantically. Since text associated with images offers explicit semantic cues to underscore this difference, in this paper we propose a novel approach: the first \textbf{T}ext-\textbf{P}rompt based weakly-supervised camouflaged instance segmentation method named TPNet, leveraging semantic distinctions for effective segmentation. 
Specifically, TPNet operates in two stages: initiating with the generation of pseudo masks followed by a self-training process. 
In the pseudo mask generation stage, we innovatively align text prompts with images using a pre-training language-image model to obtain region proposals containing camouflaged instances and specific text prompt. Additionally, a Semantic-Spatial Iterative Fusion module is ingeniously designed to assimilate spatial information with semantic insights, iteratively refining pseudo mask.
In the following stage, we employ Graduated Camouflage Learning, a straightforward self-training optimization strategy that evaluates camouflage levels to sequence training from simple to complex images, facilitating for an effective learning gradient.
Through the collaboration of the dual phases, our method offers a comprehensive experiment on two common benchmark and demonstrates a significant advancement, delivering a novel solution that bridges the gap between weak-supervised and high camouflaged instance segmentation.

Main paper: 
 relate work 2.1 camouflaged instance segmentation: 
 Although Camouflaged Instance Segmentation (CIS) has garnered increasing attention in recent years, it remains a challenging task due to the intricate nature of identifying camouflaged objects in complex backgrounds. Le et al. [20] first introduced the concept of the CIS task and conducted instance-level annotations on the available CAMO dataset, laying the groundwork for CIS research. Pei et al. [31] utilized transformer to address the CIS problem, leveraging its powerful global information processing capabilities to segment camouflaged objects. Building on the work of predecessors, Luo et al. [26] introduced the concept of frequency analysis from the task of camouflaged object detection [9, 46] into CIS, utilizing fourier transform to de-camouflaging. Inspired by query-based transformers [5, 51] , Dong et al. [11] propose a unified query-based multi-task learning framework for camouflaged instance segmentation. This framework, by integrating learning objectives from different tasks, further enhance the model's segmentation performance.Despite the significant achievements of fully-supervised methods in camouflaged instance segmentation (CIS), these methods rely on time-consuming fully-supervised annotations. In particular, annotating camouflaged images becomes notably harder when the foreground and background share striking similarities. However, there has been no exploration of CIS beyond fully-supervised methods. To tackle this challenge, our work is the first to explore accurately segmenting camouflaged images under a non-fully-supervised framework, effectively reducing the exorbitant costs associated with annotating camouflaged images.Although there hasn't been any non-fully supervised work on CIS tasks yet, the field of general instance segmentation has seen considerable research. Especially since the advent of deep learning, significant progress has been made. In the unsupervised instance segmentation, DINO [6] has demonstrated that salient features can be acquired through self-supervised learning. Building upon this notion, recent methods such as LOST [36] and TokenCut [44] utilize self-supervised ViT [12] features to segment individual salient objects within images, employing graph-based techniques that leverage DINO's patch features. In weakly-supervised instance segmentation, various types of supervision exist. For instance, some methods [8, 37] employ box supervision, where specialized loss functions are designed to achieve end-to-end instance segmentation. Others [7] utilize point supervision, which, in addition to augment bounding box guidance by introducing a selection of random points, for instance, 3, 5, or 10 points, to refine the prediction of the final mask. Although these methods [38, 41, 42] have been successful in general instance segmentation tasks, their performance in camouflaged instance segmentation tasks is disappointing because of the high similarity between camouflaged objects and their backgrounds.Recently, some research [18, 49] has delved into unsupervised camouflaged object detection, making significant achievements within the field of camouflaged object detection . However, there has been a lack of research focusing on weakly-supervised tasks in CIS. Due to the fundamental differences and distinct requirements between CIS and COD tasks, the methods developed for COD cannot be directly applied to CIS tasks.In response to these challenges, we introduce the first textprompt based camouflaged instance segmentation model. Our model does not depend on time-consuming, pixel-level annotations; Instead, it utilizes text prompts from language model as weak supervision. This unique approach leverages rich semantic information to guide camouflaged instance segmentation, overcoming limitations often encountered by general unsupervised methods when dealing with complex scenes. 
method: 
 We introduce TPNet, a novel text-prompt based weakly-supervised framework for camouflaged instance segmentation. The framework primarily consists of two stages: pseudo mask generation and graduated camouflage learning. Firstly, we detail the method's overview (Sec 3.1). Furthermore, we specifically introduce the Semantic-Spatial Iterative Fusion module, a novel component of the first stage that effectively integrates semantic and spatial information to generate refined pseudo masks (Sec 3.2). Additionally, we provide an in-depth discussion on the Graduated Camouflage Learning method, which is in the second stage and designed to optimize the self-training phase of our model, enabling it to handle the complexity of camouflaged images (Sec 3.3).Our proposed framework is illustrated in Figure . 2. It primarily consists of two stages: pseudo mask generation and graduated camouflage learning.In the first stage, given a camouflaged image 𝐼 ∈ R 𝐻 ×𝑊 ×3 and a specific text-prompt 𝑃 𝑠 : "a photo of camouflaged objects", our framework aims to generate pseudo mask 𝑀 ∈ R 𝐻 ×𝑊 ×1 . In the image branch, we discover camouflaged regions with a object detection model based on DINO [2] to obtain region proposals R = {𝑟 1 , 𝑟 2 , ..., 𝑟 𝑛 }. To capture more camouflaged instances, we also treat the entire image as a region proposal for prediction. In the text branch, we firstly feed a singular, predefined prompt 𝑃 𝑠 into GPT [1] , which produces a spectrum of prompts 𝑃 covering diverse camouflaged categories. Then we employ CLIP [32] to align text-prompts and image regions by encoding the regions and text-prompts and compute the cosine similarity metric to filter the region-text pairs.After obtaining region-text pairs, we innovatively design a Semantic-Spatial Iterative Fusion module to generate the final pesudo mask. Specifically, we employ a Semantic Mask Generator and a Spatial Mask Generator to produce masks that correspond to semantic and spatial features, respectively. Then we iteratively integrate these two kinds of masks to adequately fuse semantic and spatial information and progressively improve the quality of the pseudo mask.In the second stage, our framework propose a Graduated Camouflage Learning mechanism to train an instance segmentation model based on camouflage level. To clarify, we firstly design a Camouflage Measurer to encode camouflaged images and predefined camouflage level text-prompt, and match each image with camouflage level. After that, we construct a Camouflage Scheduler for graduated training, which gradually increases the camouflage level of the training samples and decreases the gradient of the training loss. Under the collaborative effort of these two stages, our framework leads to the generation of refined CIS results.In previous studies on unsupervised or weakly-supervised learning, researchers [24, 44] often relied on methods such as spectral decomposition [35] or class activation mapping (CAM) [50] for image segmentation. However, these methods have limitations. For example, spectral decomposition primarily uses spatial information and may not fully utilize semantic cues for segmentation. Similarly, while CAM generates heatmap results to identify relevant image regions, it may not capture fine spatial details effectively. Although a vanilla idea is to simply combine the two kinds of masks using a weighted approach, this method may compromise the quality of generated masks. We will compare this straightforward integration approach in the experiment section.To this end, we propose a Semantic-Spatial Iterative Fusion (SSIF) module to adequately integrate generated semantic and spatial masks wit CAM and spectral decomposition respectively. As shown in the Figure . 2, SSIF consists of three major components: Semantic Mask Generator, Spatial Mask Generator and Iterative Mask Fusion. SSIF begins by using the Semantic Mask Generator and Spatial Mask Generator to create masks from the image. These masks are then iteratively refined by the Iterative Mask Fusion component to produce accurate pseudo labels. Semantic Mask Generator. In order to utilize the text prompt, Semantic Mask Generator builds upon the foundation of Class Activation Mapping (CAM) [50] , a technique widely used in such tasks to generate heatmaps that highlight regions of interest in an image. It employs feature maps from the final convolutional layer to generate a heatmap. GradCAM [34] , on the other hand, improves upon CAM by replacing the weights of the fully connected layer after the Global Average Pooling with gradients to weight the activation map, resulting in a more refined class activation map. Therefore, GradCAM can be effectively employed within the CLIP to generate class activation maps for camouflage image prompts, allowing for the precise localization of object categories within the Vision Transformer model [12] . Given camouflaged region 𝑟 ∈ R ℎ×𝑤×3 within single instance and foreground class text prompt 𝑓 , we adopt GradCAM to distinguish foreground class and background class, which can be formulated as:EQUATIONwhere 𝑤 𝑓 𝑘 denotes the weight of foreground class for k-th feature map, 𝐴 𝑘 𝑖 𝑗 denotes the activation value for the k-th feature map and 𝑀 𝑠𝑒 denotes the semantic mask and RELU is an activation function that equal to 𝑚𝑎𝑥 (0, 𝑥). Spatial Mask Generator. We employ DINO and NCut to facilitate the generation of spatial masks. Ncut [35] is a graph-based image segmentation algorithm that utilizes the spectral decomposition of a graph to partition an image into regions of similarity. This is achieved by solving a generalized eigenvalue system to find the eigenvector 𝑥 corresponding to the second smallest eigenvalue 𝜆EQUATIONwhere 𝑊 ∈ 𝑁 × 𝑁 is a symmetrical matrix and 𝐷 ∈ 𝑁 × 𝑁 is a diagonal matrix derived from 𝑊 . Firstly, we obtain patches yielded by DINO for all regions. Next, we utilize the key feature of each patch to calculate the inter-patch affinity 𝑊 across all patches within the DINO feature space. This is achieved by solving Eq. 2 to find the second smallest eigenvector 𝑥. With a threshold value, we obtain the spatial mask 𝑀 𝑠𝑝 using 𝑀 𝑠𝑝 𝑖 𝑗 = max(0, sign(𝑀 𝑠𝑝 𝑖 𝑗 -𝑥)).Iterative Mask Fusion. After obtaining semantic and spatial masks, we iteratively fuse these two kinds of masks and refine the generated pesudo mask based on the evaluation of fused masks, rather than simply adding them together. Firstly, we evaluate the fused masks on semantic and spatial aspects.In the semantic aspect, we employ the CLIP model to evaluate the similarity between foreground and background images with the camouflage category text prompt individually. Then, we calculate the ratio between these similarity scores. The entire process can be formulated as:EQUATIONwhere 𝑥 𝑓 𝐼 and 𝑥 𝑏 𝐼 is the feature of the foreground and background mask from image encoder, 𝑥 𝑇 denotes the feature of camouflaged category text-prompt from text encoder, and 𝑆𝐼𝑀 denotes the cosine similarity function.In the spatial aspect, we exploit frequency features to evaluate the spatial quality of fused masks. Despite the strong resemblance between camouflaged objects and the background, disparities in energy distribution exist in the frequency domain.To visualize the energy distribution across different frequency domains, we conducted an analysis and present the results in Fig. 3 . Each point corresponds to a pair of values representing the variance of energy distribution along various frequency axes, with lower values indicating a more balanced distribution. The scatter plot reveals clear distinctions between the ground truth and coarse masks . As shown in Figure . 3 (c), the ground truth, labeled with smaller values on the plot, demonstrate a uniform energy spread, while the coarse masks in Figure . 3 (a) exhibit higher values, pointing to a less balanced distribution. This leads us to conjecture that fully segmented instances present a balanced energy distribution, while incomplete segmentation of camouflaged instances may result in uneven energy distribution, especially where partially disguised regions intersect with the background. Therefore, leveraging frequency features may enable the recognition of subtle differences in spatial quality. In this paper, we introduce Directional Energy Distribution metric 𝐸 𝐷𝐸 ∈ [0, 1] and Concentric Circle Energy metric 𝐸 𝐶𝐶𝐸 ∈ [0, 1]. These metric respectively calculate the ratio of energy distribution along angular and radial directions on polar coordinates in the frequency domain of the original image 𝑜 and the foreground 𝑓 as follows:EQUATIONwhere 𝑀 and 𝑁 denote the total number of energy values measured along the radial and angle direction respectively, 𝐸 𝑗 and 𝐸 𝑖 denote the energy values along angular and radial directions on polar coordinates in the frequency domain, and Ē represents the average energy value. The mask evaluation in spatial aspect is composed of two evaluations as follows:EQUATIONwhere 𝛽 is utilized to balance the contributions of the two evaluation metrics. Guided by mask evaluation results in semantic and spatial aspects, we iteratively fuse two kinds of masks and generate the final pseudo masks. Secondly, the evaluation metrics 𝐸 𝑠𝑒 and 𝐸 𝑠𝑝 directly influence the weight of semantic mask and spatial mask during the fusion,. Higher metrics indicate better quality in the corresponding masks, resulting in increased weights. Specifically, we first merge the initial masks generated by spectral decomposition and CAM with fixed weight. In each iteration, we use momentum weighted iterative fusion to update the pseudo mask guided by 𝐸 𝑠𝑒 and 𝐸 𝑠𝑝 . The whole updating process can be formulated as:EQUATIONEQUATIONEQUATIONwhere 𝑊 𝑡 +1 𝑠𝑒 and 𝑊 𝑡 +1 𝑠𝑝 denote weight for the semantic and spatial mask respectively in the next iteration. 𝛼 denotes momentum parameter controlling the influence of previous iterations' results on the current iteration. 𝑀 𝑡 +1 mean the pseudo mask generated in the iteration 𝑡 + 1, and this iterative process is carried out three times to achieve a significantly refined mask.It is crucial to note that images with varying level of camouflage within the same dataset exhibit different levels of camouflage. Even for the same semantic instance, the level of camouflage can significantly differ as a function of scene complexity. This variation presents a challenge for the first stage of pseudo mask generation, where the quality of the masks is directly impacted by the degree of camouflage. As illustrated in Fig. 4 , image (a) shows a highly camouflaged image where the cat is extremely difficult to discern, resulting in a pseudo mask that poorly captures the object. In contrast, image (f) shows a cat that is not well camouflaged, resulting in a pseudo mask that is comparatively more accurate and easier to generate. In other words, as the camouflage level increases, the difficulty of achieving precise segmentation rises, and the quality of the pseudo masks correspondingly declines.If these less refined pseudo masks from the first stage were to be directly utilized in the typical self-supervised training setup, they would not fully exploit the potential of the data, potentially leading to suboptimal model performance. This insight has been supported by previous research [3] , highlighting the need for a more sophisticated training approach.Inspired by the human learning process and curriculum learning [3, 40] , we propose a graduated camouflage learning mechanism to train an instance segmentation model based on camouflage level, which can acquire the ability to segment and learn from highly camouflaged samples after learning from simple samples. The core of our proposed mechanism is to gradually introduce more challenging camouflage samples for training once the model is proficient in handling simple samples. This ensures that more difficult samples are introduced to the model only after it has acquired sufficient foundational knowledge. This graduated learning mechanism can help the model better adapt to various degrees of camouflage and improve its performance in different camouflaged situations. To this end, we design two modules: Camouflage Measurer and Camouflage Scheduler for Graduated Camouflage Learning. Camouflage Measurer. While prior works on camouflaged object detection [19, 21] have conducted analyses on the degree of camouflage, they all rely on pixel-level annotations. To quantify the level of image camouflage without mask, we ingeniously propose the Camouflage Measurer, which use CLIP [32] to measure the level of camouflage. We empirically design 6 different formats of CLIP prompts 𝑃 𝑐 with GPT [1] , each representing a different level of quantization. For each image in the camouflaged datasets, we employ CLIP to embed it and compute its similarity scores with each prompt. Subsequently, by identifying the prompt associated with the highest similarity score, we confirm the camouflage level of the image. Formally, this process is represented as:EQUATIONwhere 𝜔 𝑖 denotes the weight associated with the similarity prompt, and 𝑆𝐼 𝑀 (𝐼, 𝑃 𝑖 𝑐 ) denotes the similarity between the image 𝐼 and the prompt 𝑃 𝑖 𝑐 . and then the samples are sorted in descending order of camouflage level. In the initial stages of training, priority is given to selecting samples with lower camouflage levels, specifically the "easily camo subset and pseudo mask" and "slightly camo subset and pseudo mask", allowing the model to begin learning from easier camouflage instances. For each training batch, the training gradients are adjusted based on the camouflage level of the samples in the batch. Lower camouflage instances are emphasized by receiving higher gradients by increasing learning rate, highlighting the learning of their features. Conversely, higher camouflage instances are allocated lower gradients to mitigate the potential negative impact of inaccuracies in the masks. The loss function can be formulated as:EQUATIONwhere 𝐼 denotes the camouflaged image, 𝐿 denotes the level evaluated by Camouflage Measurer and L 𝑜𝑟𝑖 denotes original loss result.As training progresses, the camouflage scheduler gradually adjusts the camouflage levels of samples in the training batches. More complex and highly camouflaged samples, such as those in the "highly camouflaged subset and pseudo mask", are progressively introduced, enabling the model to adapt to increasingly complex camouflage instance. The systematic increase in training with more challenging samples, ensures that the complexity of the learning task gradually escalates, thereby enabling the model to effectively adapt to a broader range of camouflage scenarios. 
experiments 4.1 implementation details: 
 We utilize four NVIDIA RTX 2080ti GPUs for all experiments and implemented our model using PyTorch [30] . For pseudo mask generation stage, we employ the ViT-B-8 DINO [6] in spatial mask generator and CLIP pre-trained models ViT-B-16 [32] in sematic mask generator. The images are resized to 360 × 360 pixels for mask generation. In the self-training process of second stage, we employ Cascade R-CNN [4] for all experiments. To ensure fairness in comparisons, we utilize a ResNet-50 backbone [17] initialized with pre-trained weights from ImageNet [10] . The self-training models are trained for 30,000 iterations with batch size of 8, while the maximum learning rate was set to 2.5 × 10 -4 and then decays by the step strategy.Dataset. Since CIS is an emerging task and labeling data for it is challenging, datasets for this task are relatively limited. Currently, there are only two publicly available benchmark datasets, COD10K [13] and NC4K [28] , which offer instance-level annotations for CIS training. COD10K comprises 5066 images, while NC4K provides instance-level annotations for 4121 images. In this research, we follow the methods and procedures detailed in OSFormer [31] and utilize the COD10K training data as unlabeled images to generate pseudo masks. Additionally, we evaluate our model's performance on both the COD10K and NC4K test sets. Evaluation metrics. We adopt the evaluation metrics of AP50, AP75, and AP scores [23] , employing the same settings as used in OSFormer [31] to evaluate the segmentation results. These metrics is used to evaluate the performance of a model at different Intersection over Union (IoU) thresholds.To validate the effectiveness of TPNet, we compare various supervision settings for camouflaged instance segmentation, including unsupervised, point supervision, and full supervision, as depicted in Table . 1. The considered methods encompass general unsupervised approaches utilizing Tokencut [44] , FreeSolo [42] and Cutler [41] ; general point-supervised method employing PointSup [7] ; fullysupervised methods such as Mask R-CNN [16] , cascade R-CNN [4] , OSFormer [31] , UQFormer [11] and DCNet [26] ; and our proposed weakly-supervised approach TPNet. All methods or results are obtained from official downloads. Based on the experimental results presented in Table . 1, our method, TPNet, comprehensively outperforms existing unsupervised and point-supervised methods in terms of segmentation performance on both datasets. Specifically, our results demonstrate a significant improvement over the best-performing unsupervised instance segmentation model Freesolo [42] , with a 41% increase in AP on the COD10K dataset and a 31.3% increase in AP on the NC4K dataset. Furthermore, when compared to point supervision methods, our approach achieves a 19.6% increase in AP. While there is still a gap when compared to fully-supervised methods, TPNet's performance is notably close to that of Mask R-CNN [16] . This demonstrates that TPNet effectively bridges the gap between weak supervision and the high accuracy required for camouflaged instance segmentation.We conduct ablation studies to validate the respective roles of proposed components in the overall framework.Effect of text prompt. To validate the effectiveness of text prompts, we first establish the DINO encoder and cascade R-CNN in selftraining as our baseline, denoted as ①. Building upon this foundation, we incorporate the text branch with text prompt, setting this configuration as ②. Compared to baseline ①, it is evident that the introduction of text prompt in ② leads to a significant performance improvement, achieving a 10% increase in AP. This further demonstrates the effectiveness of semantic information in CIS. Effect of Semantic-Spatial Iterative Fusion. Firstly, we conduct a comparison between a segmentation method that excludes semantic information and a basic vanilla weighted fusion approach, as shown in Table 3 . The results indicate that the simple weighted method falls short in accurately segmenting camouflaged instances. This indicates that in camouflaged instance segmentation, direct fusion may not fully leverage semantic information.To better integrate spatial information with semantic insights, then we employ our proposed method SSIF based on the setting of ②. The results are shown in the rows of ④. Compared results of the setting of ④ with the setting of ②, we can find significant improvements with an increase of 11.16% and 12% on COD10K in terms of AP and AP50, respectively. The significant improvement in quality observed with SSIF demonstrates its effectiveness in identifying and leveraging rich semantic and spatial features to enhance mask results.We also verify the effectiveness of iterative fusion technique in SSIF. As shown in Fig. 5 , we conduct experiments with one, two, and three iterations, and observe a significant improvement in the quality of camouflaged instance segmentation results as the number of iterations increased. Particularly, with three iterations, we find that the details and edges of the images were segmented more clearly, confirming the effectiveness of the iterative fusion method in enhancing image quality. Therefore, we choose three iterations as our final approach. Effect of Graduated Camouflage Learning. To assess the impact of GCL on model performance, we implement GCL as a modification to the original self-training process based on ② and ④. The Effect of camouflage level. To assess the influence of varying degrees of camouflage on segmentation accuracy, we conduct a validation experiment by sampling three subsets from the COD10K test dataset based on different levels of camouflage: highly camouflaged dataset, moderately camouflaged dataset, and lowly camouflaged dataset. In Table . 4, as the level of camouflage increases, all metrics decline accordingly. This validation experiment not only validates the effectiveness of our Camouflage Measurer but also emphasizes that images with higher levels of camouflage yield worse segmentation results.While TPNet shows promising results, challenges remain in the quality of image prompt generated and current unsupervised object region detection. Firstly, the quality of image propmt generated presents a significant challenge, directly impacting the efficacy of text prompt-supervised learning. Improving the generation quality of image descriptions is imperative for advancing the performance of our approach. Secondly, the limitations of current unsupervised object region detection frameworks, particularly the lack of fine-tuning, pose challenges in accurately identifying camouflaged objects. This limitation adversely affects the overall performance of our model. Overcoming these bottlenecks necessitates innovative approaches to enhance image description generation and fine-tuning strategies for unsupervised object region detection frameworks. We will do more in-depth research in the future to meet these challenges. 
conclusion: 
 In this study, we proposed TPNet, the first text prompt based framework for camouflaged instance segmentation, aiming to leverage both visual and semantic information from images and text streams for camouflaged instances mask. In the pseudo mask generation and self-training stages, we introduce the Semantic-Spatial Iterative Fusion (SSIF) and Graduated Camouflage Learning (GCL) modules respectively. SSIF assimilate spatial information with semantic insights, iteratively refining pseudo mask guided by the mask evaluator. Additionally, we introduce GCL, a self-training strategy that uses images of different camouflage levels to establish a gradient affected by the level of camouflage to overcome the accuracy issues caused by camouflage images. Experimental results demonstrate that our proposed network achieves outstanding performance on two common benchmark. 
: 
 (c) and fully-supervised method Mask R-CNN [16] (d) encounter difficulties in distinguishing foreground and background in camouflaged images with similar appearances. To overcome this challenge, we introduce TPNet, a novel approach that leverages text prompts to integrate semantic information and employs graduated camouflage learning for accurate weakly-supervised camouflaged instance segmentation.to their surroundings renders the task significantly more difficult compared to generic instance segmentation tasks. Stimulating significant interest within the computer vision community, CIS proves valuable across numerous domains, such as wildlife protection [29, 45] , medical image segmentation [14, 15, 22] and industrial defect detection [27] .Recently, benefiting from the strong perceptual capability of deep neural networks, especially the Transformer architecture [39] , CIS [11, 26, 31] has made significant progress. However, these methods are all based on fully-supervised learning, which poses challenges due to the difficulty and labor-intensive nature of annotating camouflage images with high precision. In fact, the adoption of weaklysupervised methods to avoid the high costs associated with massive precise annotation required by full supervision has become a trend. This way has been applied in various tasks, such as object detection [43, 48] , image segmentation [33] , video object segmentation [47] , achieving significant progress.This leads us to consider: Can we explore CIS tasks under weak supervision? Such an approach could significantly alleviate the challenge of accurately labeling camouflage data, which would be of profound significance. However, even fully-supervised trained with precisely labeled data still cannot accurately segment camouflaged objects in CIS. For example, the bird concealed within the forest in the first row of Figure . 1 is not effectively segmented by Mask R-CNN [16] , as its color closely resembles that of the surrounding leaves. Not to mention the disappointing performance of unsupervised methods like Cutler [41] . Due to the lack of prior works conducted under weakly-supervised setting in CIS, therefore it is a significant challenge to introduce appropriate weak supervision to accurately differentiate foreground and background and thus achieve accurate camouflaged instance segmentation.To this end, we carefully observe the characteristics of the camouflage image itself and then realize that despite the strong resemblance between foreground and background, they differ semantically. As shown in the second row of Figure . 1, although the tail of the snail bears a remarkable resemblance in color and texture to the background beach, it remains clearly distinguishable as a separate semantic instance. Notably, text-prompt have been effectively employed as semantic weak supervision in tasks like camouflaged object detection [18] and open-vocabulary object detection [25] , demonstrating remarkable capabilities. These tasks further confirm the feasibility and effectiveness of incorporating semantic weak supervision. Hence, in this paper text-prompt is utilized as weak supervison for the first time to explore CIS.Besides, some weakly-supervised works in other tasks often utilize self-training methods to refine initial pseudo-labels and obtain final results. These self-training approaches often treat the training data as if it is unordered, implying that samples are introduced into the training process without regard to their difficulty or complexity. However, related researchs [3, 40] on curriculum learning suggest that training with unordered samples does not fully exploit the training potential, while ordered training dataset may enhance model capacity and endow it with stronger generalization ability. Due to the unique characteristics of camouflage images compared to other conventional images, it is feasible to evaluate the camouflage level for each image. Inspired by this, optimizing self-training methods based on graduating the camouflage levels is another key exploration in this paper.In response to the aforementioned explorations, we propose the first text-prompt based weakly-supervised camouflage instance segmentation framework named TPNet, aimed at alleviating the demand for instance-level pixel annotation. TPNet employs camouflaged text prompts together with images to simultaneously utilize semantic and spatial information. This framework adopts a two-stage approach, initially generating pseudo mask, followed by self-training.Firstly, during the pseudo mask generation stage, we aim to generate pseudo masks for camouflaged images based on a given text-prompt. We identify camouflaged regions in the image using a existing object detection model and create a series of prompts covering camouflaged categories. Additionally, we align the text-prompt with these instance regions, filtering and pairing them based on cosine similarity. Following this, a Semantic-Spatial Iterative Fusion (SSIF) module is innovatively employed to iteratively refine the pseudo mask. This module integrates semantic and spatial information guided by a carefully designed mask evaluator. In the second stage, we introduce a novel self-training strategy named Graduated Camouflage Learning (GCL). Comprising Camouflage Measurer and Camouflage Scheduler, GCL is devised to utilize the distinctive features of camouflage images to enhance the model's capabilities. Initially, the Camouflage Measurer evaluates the level of camouflage in images and utilizes this evaluation as the criterion for sorting training samples. Subsequently, the Camouflage Scheduler prioritizes images with lower levels of camouflage during training. As training progresses, more complex images with higher levels of camouflage are gradually incorporated into the training. Through this optimized training strategy, the model can acquire the ability to segment and learn from highly camouflaged samples with low gradient after simple samples with high gradient.Our major contributions can be summarized as follows:• We introduce TPNet, the first text-prompt based weaklysupervised framework for camouflaged instance segmentation, which significantly reduces the annotation burden of image data by leveraging semantic understanding for effective segmentation.  
 
 Appendix:  


Decision: This paper has received two WAs, one BA, and one WR as initial scores. 

Pros:
All reviewers recognize the novelty about using the text prompt to improve the segmentation results of camouflaged objects. The experimental results demonstrate TPNet's superior performance over existing unsupervised and point-supervised methods.

Cons:
Reviewers xhGN and U7oG have concerns about the quality of the prompts that may affect the performance of the proposed TPNet. 
Reviewers shNc and 9FbE raise questions about the model performance as shown in Table 1 and Table 2. 

The authors provide a rebuttal which addresses the reviewers' concerns. The AC agrees with the majority of reviewers that the paper should be accepted.

Reviews: review: paper_summary: paper_summary: The authors propose a new method called TPNet for camouflaged instance segmentation (CIS). TPNet uses text associated with images to help find hidden objects more effectively. 

TPNet works in two stages: first, it generates rough outlines of objects based on text cues and images, and then it refines these outlines using a clever method that combines spatial and semantic information. Additionally, TPNet employs a self-training process that gradually learns to recognize more difficult camouflage patterns, improving its performance over time. 

Experiments show that TPNet performs better than existing point-supervised or unsupervised methods, making it a promising approach for weakly-supervised camouflaged instance segmentation.
strength_weakness: strength_weakness: The idea of using text to improve the segmentation results of camouflaged objects is interesting and promising.
TPNet innovatively integrates text cues with images, improving segmentation accuracy. The Semantic-Spatial Fusion refines outlines iteratively, enhancing segmentation quality. Graduated Camouflage Learning optimizes self-training, adapting to varying camouflage complexities. Extensive benchmarking shows TPNet outperforms existing methods reliably.
limitations: limitations: As the authors already pointed out, TPNet's performance may be limited without high-quality prompts, highlighting the need for improvements in image description generation. 
The manuscript should be revised to correct spelling or grammatical errors.
paper_summary: paper_summary: -The paper addresses camouflaged instance segmentation by proposing TPNet, a weakly-supervised framework utilizing text prompts and self-learning method. The text-prompt information is claimed to enhance the camouflaged features. The graduated camouflage learning method benefits the learning process of the model by gradually increasing the level of camouflaged instances.
- Despite the contributions, the performance of the proposed method is limited.
strength_weakness: strength_weakness: - The paper explores the text prompt to enhance camouflaged features, which provide more information to distinguish camouflaged instances and background.
- The paper is among the first works addressing CIS in a non-fully-supervised manner.
- The paper is well-organized and easy to follow.
limitations: limitations: - The storytelling in the introduction lacks comprehension, it is recommended that the authors provide further detailed explanation behind the utilization of text-prompt and self-learning in camouflage research.

- It is recommended that the authors provide further discussion on the general text-prompt-based and self-training approach in the Related Work.

- The proposed methods adopt a combination of existing models to address the task, thus the contributions appear marginal. The limitation is that the authors did not provide the analyzed reasons behind each of their proposed adoptions, e.g. CAM/GradCAM in Semantic Mask Generator or DINO/NCut in Spatial Mask Generator.
- The authors decide to iteratively fuse the two semantic and spatial information to form the pseudo mask due to the failure of weighted fusion. Please provide further explanation on how they are weighted fused.


- As indicated in Table 2, the proposed GCL seems to contribute minor improvement in comparison with the other components, i.e. (4) vs (5). Please clarify the contribution of the GCL in the framework. 
- In GCL - Camouflage Measurer, the authors select CLIP [32] to measure the level of camouflage. Why do the authors believe that the CLIP model can well extract the camouflage features?
- How many levels of camouflage are effective in the proposed method?
 
Misc.:
- Please proofread the manuscript to correct existing typos or paragraph re-arrangements (e.g. paragraph in L493, L569).
paper_summary: paper_summary: The paper "Text-prompt Camouflaged Instance Segmentation with Graduated Camouflage Learning" introduces TPNet, a novel text-prompt based weakly-supervised framework for camouflaged instance segmentation (CIS). The primary aim is to detect and segment objects that blend into their surroundings using minimal annotations. TPNet operates in two stages: pseudo mask generation and graduated camouflage learning. The pseudo mask generation stage aligns text prompts with images using a pre-trained language-image model, creating pseudo masks iteratively refined through a Semantic-Spatial Iterative Fusion (SSIF) module. In the self-training stage, Graduated Camouflage Learning (GCL) sorts training images by their camouflage level, facilitating an effective learning gradient from simpler to more complex images. The method significantly reduces the annotation burden while achieving superior segmentation performance on benchmark datasets compared to existing unsupervised and point-supervised methods.
strength_weakness: strength_weakness: •	Novelty: The paper presents the first text-prompt based weakly-supervised framework for CIS, addressing the significant challenge of annotating camouflaged images.

•	Methodological Innovation: TPNet combines semantic and spatial information through a Semantic-Spatial Iterative Fusion (SSIF) module, enhancing the refinement of pseudo masks.

•	Effective Self-Training: The Graduated Camouflage Learning (GCL) strategy optimizes self-training by progressively introducing images with higher camouflage levels, improving the model's robustness and accuracy.

•	Practical Applications: The approach is applicable across various domains, including wildlife protection, medical image segmentation, and industrial defect detection.

•	Evaluation: The experimental results demonstrate TPNet's superior performance over existing unsupervised and point-supervised methods, achieving results comparable to fully-supervised approaches.
limitations: limitations: •	Dependency on Text Prompts: The method's reliance on text prompts requires the creation of diverse and accurate prompts, which may be challenging and could impact the model's performance if not done correctly.

•	Complexity of Implementation: Integrating SSIF and GCL adds complexity to the implementation, potentially making it difficult for practitioners to adopt without extensive understanding.

•	Generalization to Diverse Contexts: While the method shows promising results on benchmark datasets, its performance in more diverse and unstructured real-world scenarios remains to be extensively validated.

•	Evaluation Metrics: The paper primarily compares TPNet with unsupervised and point-supervised methods. Additional comparisons with other weakly-supervised approaches could provide a more comprehensive evaluation of its effectiveness.
paper_summary: paper_summary: The authors introduce a framework TPNet for the text-prompted weak-supervised camouflage instance segmentation. Only class labels are utilized during training for supervision. Text prompts together with camouflaged regions detected by DINO are used to generated pseudo masks with CLIP. They also designed a camouflage measurer to predict the camouflage scores and for the Graduated Camouflage Learning process.
strength_weakness: strength_weakness: It is the first Text-Prompt based weakly-supervised camouflaged instance segmentation method. Authors combine the text prompts and camouflage region discovery cleverly for Text-Prompt based weakly-supervised CIS. 
The camouflage evaluation method and the Energy metric for evaluation are interesting.
limitations: limitations: 1. In Figure 2, both camouflage regions and text prompts are fed into the Semantic Mask Generator for heatmap generation. It is not clearly explained in the 'Semantic Mask Generator' section how they are used. The authors mentioned 'effectively employed within the CLIP'. It's confusing as the process is not well illustrated. 
2. What is the author's rationale for designing the different formats of CLIP prompts in 6, and what is the criteria for grading the different levels of quantization? These key questions are not clearly stated in Section 3.3.
3. In table 2, the results with/without Graduated Camouflage Learning are almost the same. I doubt whether this module is useful or not.
4. The authors mention that they employ the ViT-B-8 DINO in the spatial mask generator and CLIP pre-trained models ViT-B-16 in the semantic mask generator. Both are heavy models, which raises the concern about the computational complexity of the method. Could the authors add comparisons of inference time and GFLOPS with previous models?
5. The model performance in Table 1 is not good enough, which raises concerns about model effectiveness. If possible I would suggest that the author could make the code public after the final revision. Also, in Table 1, the AP scores for Tokencut, a method accepted by TPAMI 23', are below than 5. I doubt whether the authors have implemented this method correctly or not. Could the authors provide possible reasons for this?
6. A lot of writing errors in the manuscript, for example:
Implementation Details: ..., while the maximum learning rate was set to 2.5 × 10−4 and then decays by the step strategy. --> decay?
Evaluation metrics: These metrics is used to ... --> are used to?
Experiment Results and Analysis: Table.1 or Table 1? Fig. X or Figure. X?
Table 1: ECCV’2023-->ECCV’2022.