Few-shot Semantic Segmentation via Perceptual Attention and Spatial Control

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 15:49:27

Abstract: Few-shot semantic segmentation (FSS) aims to locate pixels of unseen classes with clues from a few labeled samples. Recently, thanks to profound prior knowledge, diffusion models have been expanded to achieve FSS tasks. However, due to probabilistic noising and denoising processes, it is difficult for them to maintain spatial relationships between inputs and outputs, leading to inaccurate segmentation masks. To address this issue, we propose a Diffusion-based Segmentation network (DiffSeg), which decouples probabilistic denoising and segmentation processes. Specifically, DiffSeg leverages attention maps extracted from a pretrained diffusion model as support-query interaction information to guide segmentation, which mitigates the impact of probabilistic processes while benefiting from rich prior knowledge of diffusion models. In the segmentation stage, we present a Perceptual Attention Module (PAM), where two cross-attention mechanisms capture semantic information of support-query interaction and spatial information produced by the pretrained diffusion model. Furthermore, a self-attention mechanism within PAM ensures a balanced dependence for segmentation, thus preventing inconsistencies between the aforementioned semantic and spatial information. Additionally, considering the uncertainty inherent in the generation process of diffusion models, we equip DiffSeg with a Spatial Control Module (SCM), which models spatial structural information of query images to control boundaries of attention maps, thus aligning the spatial location between knowledge representation and query images. Experiments on PASCAL-5$^i$ and COCO datasets show that DiffSeg achieves new state-of-the-art performance with remarkable advantages.

Main paper: 
 few-shot semantic segmentation: 
 Few-shot semantic segmentation extends the ability of segmentation to novel category with a few labeled samples. Many researchers regard few-shot segmentation as a guided segmentation task, thus following a two-branch framework [24, 41] . For instance, Shaban et al. [28] apply few-shot learning on semantic segmentation using a two-branch framework, where the support branch generates parameters in the last layer of the query branch for segmentation. Following their idea, Michaelis et al. [21] combine embeddings with a U-net to find unknown objects in a complex scene guided by only one sample. To better utilize information of the support set, Wang et al. [34] learn prototype representations based on a few support images in an embedding space, which could match pixels to the learned prototypes, thus performing segmentation in few-shot settings.Recently, Gu et al. [5] propose DRCNet to achieve sufficient support-query interaction for accurate FSS, where a dynamic context module is presented to capture spatial details in query images by building dynamic convolutions in local views. For better feature fusion, MFNet [45] utilizes an attention mechanism to achieve support feature modulation and multi-scale combination. Facing heavy computational operations of high-dimensional vectors, QCLNet [47] explores latent interaction between images through the utilization of operations grounded in well-established quaternion algebra.. However, those methods are challenging in digging internal relationship between query and support images, due to limited available semantic information with a few samples. Unlike former methods, DiffSeg introduces a pretrained diffusion model to provide sufficient prior knowledge for guiding segmentation, thus reducing the dependence on training data scale.Diffusion models have recently gained significant attention from the research community due to their ability to generate high-fidelity contents, which are gradually extended to FSS tasks [2, 33] . For instance, to effectively train segmentation models using generated images, Roy et al. [27] propose diffusion-based DiffAlign, which could align the synthetic images to the real images and minimize the domain gap.In contrast to those methods that synthesize samples, LEDM [2] directly use the generative model to perform segmentation, which extracts the intermediate activations from the reverse diffusion process as excellent pixel-level representations for the segmentation problem. In order to enhance the step-wise regional attention in diffusion probabilistic model for the image segmentation, MedSegDiff [36] proposes dynamic conditional encoding, which establishes the state-adaptive conditions for each sampling step.Different from previous methods, our method decouples probabilistic processes and segmentation phase, which could reduce the impact of probabilistic processes while benefiting from rich prior knowledge of pretrained diffusion models. 
task description: 
 Given only one or a few images with pixel-level annotations, fewshot segmentation aims to accurately locate foreground pixels in test images. Specifically, we divide the dataset into training set 𝐶 𝑡𝑟𝑎𝑖𝑛 and test set 𝐶 𝑡𝑒𝑠𝑡 according to category of images, where 𝐶 𝑡𝑟𝑎𝑖𝑛 and 𝐶 𝑡𝑒𝑠𝑡 don't contain same-category images, which can be represented as 𝐶 𝑡𝑟𝑎𝑖𝑛 ∩ 𝐶 𝑡𝑒𝑠𝑡 = ∅.For each 𝑘-shot segmentation task, we firstly define a target class 𝑐, and then sample 𝑘 + 1 images with the class label 𝑐 from 𝐶 𝑡𝑒𝑠𝑡 . Defining the first 𝑘 labeled images as support set 𝑆 and the last image as 𝑥 𝑞 , few-shot segmentation model 𝑀 𝜃 aims to compute the segmentation mask of the query image ŷ𝑞 with:EQUATIONwhere 𝜃 are parameters of the model. Assuming that 𝑦 𝑞 is the ground-truth label for the query image 𝑥 𝑞 , task goal of few-shot segmentation is to minimize the loss 𝐿( ŷ𝑞 , 𝑦 𝑞 ) by updating 𝜃 , where such process could be represented as:EQUATIONSince an image may contain objects of different categories, the ground-truth query mask could vary with different assigned labels. 
method 4.1 overview: 
 The framework of DiffSeg is shown in Fig. 2 , where a Perceptual Attention Module (PAM) performs perceptual attention mechanisms among support latent images, query latent images and attention maps generated by a knowledgeable diffusion model, which helps locate target areas with compact boundaries. Besides, a Spatial Control Module (SCM) inserts conditional spatial information into the denoising process of diffusion UNet, which aligns edges between query images and attention maps, thus keeping spatial structures of query images during probabilistic diffusion processes. Specifically, we first multiply a support image 𝐼 𝑠 and a binary support mask 𝑀 𝑠 to remove background. Then, the output is processed by a Variational Auto-Encoder (VAE) encoder and a Clip [23] to generate the support latent image 𝐿 𝑠 and embedding 𝐸 𝑚 , respectively. The process can be expressed as:EQUATIONEQUATION) where ⊗ is element-wise multiplication. The embedding 𝐸 𝑚 generated by Clip is the conditional information of diffusion models, which is interacted with query latent images during denoising processes. Then encoder module transforms the query image 𝐼 𝑞 to the query latent image 𝐿 𝑞 , which is input into a diffusion model prompted by 𝐸 𝑚 to obtain attention maps 𝐴 𝑚 . Noted that 𝐼 𝑠 is processed by HDE algorithm [38] to generate an edge map 𝐼 𝑙 , which is used by SCM to control attention maps of diffusion model, thus aligning spatial structures between attention maps and query images. The process can be expressed as:EQUATIONEQUATION) where 𝐷𝑖 𝑓 𝑓 (•) and 𝑆𝐶𝑀 (•) represent the process of diffusion model and spatial control module, respectively. Then, the attention maps 𝐴 𝑚 is processed to obtain an attention score map 𝑀 𝑎 :EQUATIONwhere 𝑃𝑟𝑜𝑐𝑒𝑠𝑠 (•) presents the operation that transforms 𝐴 𝑚 to 𝑀 𝑎 , which details are shown in section 4.2. Afterwards, PAM performs perceptual attention mechanisms among attention maps 𝑀 𝑎 , the support latent image 𝐿 𝑠 and the query latent image 𝐿 𝑞 to locate target areas with compact boundaries, which outputs the predicted latent image 𝐿 𝑝𝑟𝑒 .EQUATIONFinally, a VAE decoder transforms 𝐿 𝑝𝑟𝑒 to prediction image 𝐼 𝑝𝑟𝑒 , which can be formulated as:EQUATIONIn training phase, we only update parameters of SCM and PAM. Therefore, we supervise the attention score map 𝑀 𝑎 and the predicted latent image 𝐿 𝑝𝑟𝑒 , where both two supervision use the MSE loss function. For computing loss of 𝐿 𝑝𝑟𝑒 , we use a VAE [10] encoder to obtain the latent image of ground truth. The whole loss L can be computed as:EQUATIONwhere 𝐿(•) presents MSE loss function and 𝐼 𝑔𝑡 refers to ground truth. 𝛼 and 𝛽 are weights of each loss. Based on experimental study shown in supplementary, we set them to 2 and 1, respectively. For 𝑘-shot segmentation, both 𝑘 embeddings and 𝑘 support latent images are fused in an average manner. Thanks to the ability of Clip to align image with text and rich prior knowledge of pretrained diffusion models, there is little gap in segmentation performance between different fusion methods. Experiments in supplementary can prove it. Therefore, we select simple but effective average fusion as our 𝑘-shot solution.To incorporate profound prior knowledge into the FSS task, we equip DiffSeg with a pretrained diffusion model named Kandinsky [25] , which helps understand semantic contents of query images and achieve support-query interaction. Specifically, we input support images as prompt and query images as latent into Kandinsky, and then extract attention maps from the model, i.e., cross-attention maps and self-attention maps. The former contain the response information of query images to support images, and the latter helps fully extract semantic information from query images.Kandinsky diffusion contains attention maps of 3 different scales (i.e., 8×8, 16×16, 32×32), which are interpolated to the same size for fusion. By averaging self-attention maps and cross-attention maps, respectively, we can obtain a fused self-attention map 𝑀 𝑠𝑒𝑙 𝑓 ∈ R 𝐻𝑊 ×𝐻𝑊 and a fused cross-attention map 𝑀 𝑐𝑟𝑜𝑠𝑠 ∈ R 𝐻 ×𝑊 .In fact, cross-attention maps serve as a vital component in the conditional generation process, providing valuable insights into the conditional probability distribution. However, the produced crossattention score maps often lack clear object boundaries and may exhibit internal holes. Fortunately, self-attention maps are able to establish the correlations between different pixels, which provides the ability to perform region completion, thus compensating for the incomplete activation regions in cross-attention. Therefore, the final attention score map 𝐴 𝑚 can be obtained by multiplying the cross-attention maps with pixel affinity weights obtained from self-attention maps:EQUATIONwhere 𝑛𝑜𝑟𝑚(•) is min-max normalization to ensure the segmentation score maps are appropriately scaled, 𝑣𝑒𝑐 (𝑀 𝑐𝑟𝑜𝑠𝑠 ) ∈ R 𝐻𝑊 ×1 , and 𝑣𝑒𝑐 (•) is a vectorization operation of a matrix. Briefly, the attention score map 𝐴 𝑚 is simply produced during the denoising inference process of the pretrained diffusion model, which locates target areas with prior knowledge of the diffusion model, thus providing useful guidance for segmentation phases. The design of perceptual attention module. Two cross-attention operations provide segmentation task for semantic information of support-query interaction and spatial information produced by prior information of pretrained diffusion models, respectively. Considering confliction between them, we utilize a self-attention to find a balance of dependence for segmentation.Different from most previous methods which performs dense comparisons within feature maps of images, PAM performs perceptual attention mechanisms in latent image level. A latent image is a compression transformation that removes high frequency information of a natural image, which not only reduces computational burdens in comparison, but also keeps visual features of original images. Moreover, PAM considers extracted attention maps to perceptual attention, thus introducing prior knowledge of diffusion models into segmentation phase. The design of PAM is shown in Fig. 3 . Firstly, we perform crossattention for a query latent image 𝐿 𝑞 with a support latent image 𝐿 𝑠 and an attention score map 𝑀 𝑎 , respectively. In this process, we set 𝐿 𝑞 as Query (𝑄) and set 𝐿 𝑠 or 𝑀 𝑎 as Key (𝐾) and Value (𝑉 ). The cross-attention can be computed as:EQUATIONwhere 𝐿𝑖𝑛𝑒𝑟 (•) function refers to linear transformation and 𝑑 presents the dimension of linear features. In the same manner, cross-attention is performed between 𝐿 𝑞 and 𝑀 𝑎 to obtain 𝐿 2 .Essentially, 𝐿 1 contains semantic information of support-query interaction, while 𝐿 2 contains spatial information produced by the pretrained diffusion model. In most cases, they have the same interested areas, which could provide positive guidance for segmentation. Unfortunately, their interested areas are completely inconsistent on occasion.Considering confliction between the prior information and the semantic interaction information, we utilize a self-attention to find a balance of dependence for segmentation. Specifically, we fuse 𝐿 𝑞 , 𝐿 1 and 𝐿 2 in an average manner to achieve 𝐿 3 , which is set to 𝐾, 𝑄 and 𝑉 in the self-attention operation. Then, the output of self-attention is incorporated to 𝐿 3 in a residual form. Finally, we use a convolution and ReLU block to achieve the latent image 𝐿 𝑝𝑟𝑒 of predicted mask. Due to probabilistic noising and denoising processes, diffusion models are unable to maintain spatial relationships between inputs and outputs, leading to inaccurate segmentation masks. To solve this problem, we present an SCM for DiffSeg to align semantic boundaries between extracted attention maps and query images, which is shown in Fig. 4 . SCM treats the pretrained diffusion model as a strong backbone for learning conditional controls. Firstly, an edge map is processed by convolution and ReLU blocks to obtain spatial information of a query image. Then it merges with features resized to 64×64 of diffusion UNet and goes through a linear layer. The output of the linear layer not only goes through a normalized layer and a Multi-Layer Perception (MLP), but is also added to output of MLP in a residual form to avoid excessive transforms. Finally, after the process of a linear layer, we insert spatial controls into diffusion UNet in a residual form, which helps align spatial structures between attention maps and query images. On the one hand, the spatial information is regarded as conditional information to control spatial structures of attention maps in up-sample processes. On the other hand, direct connections between corresponding down-sample and up-sample processes could reduce changes in spatial structures. Noted that SCM extracts features from each scale level of UNet and inserts spatial controls into a corresponding level, as shown in Fig. 4 , where arrows of different color represent different paths.Differing from ControlNet [44] , which employs latent images for control, SCM utilizes conditional information to govern attention maps Additionally, while ControlNet operates at each encoder level of a diffusion U-net with cascaded multiscale encoder blocks, SCM handles state information from various levels by parameter sharing, thus achieving control with reduced computational overhead. 
experiments 5.1 dataset and evaluation metric: 
 PASCAL-5 𝑖 [28] is a public dataset of few-shot semantic segmentation, which is made up of PASCAL VOC 2012 [3] with extra SDS [7] annotations. The dataset includes 20 categories, which are divided into 4 splits, and each split contains 5 categories. COCO 2014 [15] is a challenging large-scale dataset containing 80 categories. The purpose of COCO is scene understanding, which is mainly acquired from complex daily scenes. The target objects in images are annotated in precise pixel-level masks.According to the introduction of few-shot segmentation task, a cross-validation experiment is performed. On PASCAL-5 𝑖 , three splits are utilized for training, and the final split is used to evaluate models. On COCO 2014, according to [41] , we select 40 classes for training, 20 classes for validation and 20 classes for test.The mean Intersection-over-Union (mIoU) of all classes and the average of the foreground & background IoU (FB-IOU) are two main evaluation metrics of few-shot segmentation. For fair comparison with current methods, we use mIoU and FB-IOU to measure our model performance, but mIOU is regarded as the main metric due to its higher evaluation ability.To evaluate the performance of our method, we implement DiffSeg using the PyTorch library, and train it for 200 epochs on 4 Nvidia V100 GPUs. We set the learning rate to 0.01 and use the StepLR scheduler in PyTorch, thus reducing the learning rate to 0.9 times for every 20 epochs. Diffusion and Clip are pretrained models, which parameters are frozen in training phase. We only update parameters of PAM and SCM with a gradient descent algorithm. To reduce the effect of the random selection of support images, we run all tests 10 times and report the mean results.PASCAL-5 𝑖 . We compare DiffSeg with current methods on the PASCAL-5 𝑖 dataset, where Table 1 shows results with mIOU and FB-IOU metrics. By comparing results, DiffSeg outperforms current methods and reaches new state-of-the-art performance in most cases. Under mIOU metric, DiffSeg is superior to current works in both 1-shot and 5-shot segmentation. We notice that DPCN [17] exceeds DiffSeg on split-3 of PASCAL-5 𝑖 , which is caused by nonuniform distribution of categories in splits. In fact, the average size of objects on split-3 is relatively large, where DPCN can better find the pixel-wise dense correlation between query and support images. Besides, considering that our approach is transformer-based, we compare DiffSeg with transformer-based methods (e.g., AAFormer [35] and VAT [8] ) for fair comparison. The comparison results show that our method outperforms existing transformer-based methods. Essentially, our method benefits from rich prior knowledge rather than solely relying on transformer architectures, which is not present in other transformer-based methods.Under FB-IOU metric, DiffSeg still achieves outstanding performance. Since mIOU is regarded as the main evaluation metric, we only report the average FB-IOU of 4 splits. More detailed results are reported in the Supplementary.MS COCO. The results of DiffSeg and other methods on the MS COCO dataset are shown in Qualitative results. Some qualitative results of 1-shot segmentation are shown in Fig. 5 , where each column represents the support set, query set, attention map of the pretrained diffusion model, prediction, and ground-truth, respectively.Noted that given the same query image, DiffSeg is able to segment different targets when different objects are labeled in support images. For instance, in the 7th and 8th examples, given different support images with the category "dog" or "sofa", DiffSeg can segment different objects in the same query image. Essentially, the pretrained diffusion model can generate different attention score maps of a query image, which provides a strong guidance for segmenting different objects. To validate the effectiveness of each module, we perform ablation experiments on PASCAL-5 𝑖 , where we still use the cross-validation method and report the average mIOU of 4 splits. Network Design. To prove effectiveness of the proposed modules, we compare the performance of DiffSeg when certain modules are removed. When UNet with SCM is removed, PAM only compares support latent images and query latent images, without considering attention maps. When PAM is removed, a convolutional layer is used to generate predicted latent image 𝐿 𝑝𝑟𝑒 .Table 3 shows the elimination of certain module would reduce the performance of DiffSeg, which means that each module plays a positive role in segmentation. When diffusion UNet is removed, segmentation performance drops significantly, which proves that prior knowledge in the pretrained diffusion model is powerful to interact semantic information between support-query images and provide a strong guidance for segmentation. Knowledge representation of diffusion models. To validate the significance of extracted prior knowledge, we compare results of DiffSeg which extracts feature maps at different locations as knowledge representation. First, we explore the performance impact of removing self-attention or cross-attention. Second, we study the impact of extracting features at other locations in diffusion UNet as prior knowledge, e.g., Features of Resnet Blcok (FRB) and Features of Spatial Transformer (FST).The ablation results are as shown in Table 4 , where SA and CA refer to self-attention and cross-attention, respectively. The results show that extracting only self-attention or cross-attention will result in a performance decrease. Self-attention maps effectively capture the similarity relationship between pixels of query images, and cross-attention maps contain the response information of query images to support images, which could contribute to extract semantic features and locate target areas. Noted that the result of DiffSeg with cross-attention is better than its with self-attention, which shows that information interaction between support images and query images is more important than semantic perception in query images. Besides, when using features of resnet block or spatial transformer as prior knowledge, the performance of DiffSeg decreases, which shows that attention could provide the fullest prior knowledge.Attention in PAM. To validate the effectiveness of 3 attention operations in PAM, we compare the performance of DiffSeg where certain attention operations are removed or changed to convolutional operations. The comparison results are shown in Table 5 , where Att_1 and Att_2 present to cross-attention of query-support Query image 88 1616 3232 pairs and query-attention pairs, respectively. Att_3 presents a selfattention operation in PAM. From Table 5 we can see, removing certain attention operation could reduce the performance of Diff-Seg, which signifies all attention operations in PAM play positive roles in segmentation. When Att_3 (self-attention) is removed, the model encounters difficulties in preserving a balanced segmentation dependency between the guidance from diffusion and support images. In cases these two sources of guidance are incongruent, accurate segmentation cannot be attained. It's worth noting that DiffSeg with Att_2 performs better than it with Att_1, indicating that spatial information from diffusion is more crucial than semantic information from support images for segmentation. Besides, the result of DiffSeg with attention operations is better than its with convolutional operations, which shows that attention mechanisms have more strong ability to capture interest areas for segmentation. Design of SCM. To validate advantages of SCM design, we insert SCM into different levels of diffusion UNet and compare the performance whether SCMs from different levels share parameters. The comparison results are shown in Table 5 .From the Table we can see, removing SCM from each level will lead to drop in performance, which shows SCM could work well in each level of diffusion UNet. Specifically, when SCM is deactivated in 16×16 scale level, performance of DiffSeg significantly decreases. Through visualizing attention maps of each level as shown in Fig. 6 , we find 16×16 attention maps are the most complete. Therefore, controls of 16×16 attention maps by SCM directly affect the performance of segmentation. Besides, training a separate SCM for each layer doesn't improve performance, while introducing additional parameters and increasing computational burden. It shows sharing parameters of SCM is a better strategy for lightweight computation. 
conclusion: 
 In this paper, we present a DiffSeg for FSS, which decouples probabilistic denoising and segmentation processes, thereby mitigating the impact of probabilistic processes while benefiting from rich prior knowledge of the model. We propose a PAM for segmentation, where two cross-attention capture semantic information of support-query interaction and spatial information produced by the pretrained diffusion model. Moreover, considering the probabilistic generation of diffusion models, we present an SCM to align semantic boundaries between extracted attention maps and query images. Comprehensive experiments show that DiffSeg achieves new state-of-the-art performance. 
: 
 algorithms require large amounts of dataset to learn knowledge, where labeling is extremely time-consuming and annoying. Essentially, humans could understand a new category with a few samples, which inspires researchers to transfer knowledge from known to unknown. By properly encoding humans' knowledge, few-shot learning successfully relieves the high demand of collecting large-scale dataset, thus expanding its usage into variant scenarios [1, 6, 14] .In this paper, we focus on the task of Few-shot Semantic Segmentation (FSS), which aims to segment new categories with a few samples. Recently, metric-based methods [5, 17] have significantly contributed to the FSS task, which locate targets by finding co-occurrence features between support images and query images, as shown in Fig. 1 (a). However, these methods face challenges in fully leveraging intrinsic features of categories with limited samples, resulting in compromises in target integrity and boundary accuracy. Thanks to profound prior knowledge, diffusion models [26] can thoroughly express intrinsic features of new categories.Nevertheless, extended to FSS, those methods [27] are merely used to generate samples for training segmentation models, as shown in Fig. 1 (b ). It essentially reverts FSS tasks back to conventional segmentation tasks, which fails to meet the requirements of low time consumption and computational cost for training. To avoid this issue, researchers propose diffusion segmenters [12? ], which directly perform the segmentation task during denoising processes, as shown in Fig. 1 (c ). Due to probabilistic noising and denoising processes, those methods struggle to maintain spatial relationships between inputs and outputs, thus generating results with inaccurate boundaries.Facing the aforementioned challenges, we propose a novel FSS paradigm named DiffSeg, as shown in Fig. 1 (d) . DiffSeg utilizes prior knowledge of a pretrained diffusion model to assist in segmentation, thus mining intrinsic features of categories with limited samples and avoiding generating probabilistic results. Specifically, we feed support images as prompts and query images as latent images into a diffusion model, where we extract multiscale attention maps, i.e., self-attention maps and cross-attention maps. The former effectively capture the similarity relationship among pixels, which helps understand semantic contents of query images. The latter contain response information of support-query interaction, which could help the model locate target areas. In fact, the attention maps are able to locate the general position of target areas through simply processing, which provide a strong guidance for segmentation.In the segmentation phase, we propose a Perceptual Attention Module (PAM) to highlight significant regions within query images based on support images and attention maps of a pretrained diffusion model, respectively. Firstly, we employ two cross-attention mechanisms to handle data of different branches. The cross-attention between query-attention pairs could capture spatial information produced by pretrained diffusion models, thus introducing prior knowledge of pretrained diffusion models into segmentation phase. Simultaneously, the cross-attention between query-support images could capture semantic interaction information, which could find co-occurrence objects between support images and query images. Secondly, considering confliction between the spatial information and the semantic information, we utilize a self-attention to establish a balanced dependence for segmentation, which improves the robustness of DiffSeg. Noted that those computations are conducted at the latent image level. A latent image is a compression transformation that attenuates high frequency information of a natural image, which not only reduces computational burdens, but also keeps visual features of the original natural image.Considering the generation uncertainty of diffusion models, we propose a Spatial Control Module (SCM) for DiffSeg to align semantic boundaries between extracted attention maps and query images. Specifically, we integrate edge maps of query images with the hidden states of down-sample processes in diffusion models. Subsequently, this combined information is transformed into spatial structural information and inserted into corresponding scale levels of up-sample processes. On the one hand, the spatial structural information serves as conditional input to control spatial structural information of attention maps during up-sample processes. On the other hand, direct connections between corresponding downsample and up-sample processes help mitigate changes in spatial structures of attention maps. Differing from ControlNet [44] , which employs latent images for control, SCM utilizes conditional information to govern attention maps.The main contributions of this paper are as follows:• We introduce a novel FSS paradigm named DiffSeg, which effectively decouples probabilistic processes of diffusion models with segmentation phases, thus avoiding uncertain results while utilizing profound prior knowledge. • We propose a Perceptual Attention Module for segmentation, where two cross-attention capture semantic information of support-query interaction and spatial information produced by the pretrained diffusion model, respectively, Additionally, a self-attention mechanism is employed to find a balance in dependency for segmentation tasks. [13, 18, 43] utilize convolution operations to extract semantic information from feature maps for pixel-level label prediction. Considering dependence of context information, RNN-based methods [9, 22, 32] use recurrent layers to capture local and global spatial structure information of images. Using topological structure of graphs, GNN-based methods [19, 20, 30] transform task of image segmentation into the classification task of graph nodes. Recently, transformer-based methods [37, 39, 46] have received more popularity. They regard patches of images as a sequence, then utilize an encoder-decoder structure with attention mechanisms to achieve segmentation. Among them, transformer-based methods are most relevant to our work. To comprehensively understand image contents, In-cepFormer [4] introduces an efficient inception transformer that integrates global context, fine localization information and multiscale features to segment images. Considering the background incompleteness issue, Liu et al. [16] propose WegFormer, where the depth-taylor decomposition principle and soft erasure module are incorporated to generate more complete pseudo-labels. Using a pretrained transformer in a data collection loop, Kirillov et al. [11] propose a Segment Anything (SA) project: a new task, model, and dataset for image segmentation, which shows the advantages of transformer for image segmentation tasks. 
 
 Appendix:  


Decision: The work is of reasonable quality --there are some concerns about the limited novelty, yet is has merits.

Reviews: review: paper_summary: paper_summary: The paper proposes a framework to leverage pre-trained diffusion models in the segmentation problem.
DiffSeg is proposed for Few-shot Segmentation (FSS) problem to tackle the issues of probabilistic processes within current diffusion-based segmentation models and further utilize the prior knowledge from existing diffusion models for segmentation

+ The authors propose a Perceptual Attention Module (PAM) for segmentation, which comprises a cross-attention module for capturing the semantic and spatial information obtained from the supporting query and the diffusion model, respectively. The output is post-processed by a self-attention mechanism to balance out the weight of each cross-attention module previously. 
+ The authors propose a Spatial Control Module (SCM) for aligning the semantic boundaries between the input query images and the extracted attention maps, which bridges the gap between the spatial structures of the input and output features obtained from the diffusion model.

The model's performance is reported to be the current SOTA for the FSS task, as it achieves 69.3% for 1-shot segmentation and 72.1% for 5-shot segmentation.
strength_weakness: strength_weakness: The proposed idea of segmentation as a conditioned generation with diffusion models is creative.

The architecture mostly relies on pre-trained models and only introduces a few additional modules, which require fewer computational resources to train and preserve the utility of the pre-trained models.

The modules have clear functionalities. The SCM module is well-motivated and easy to understand.

The methods of the authors have been experimented and compared under various settings, with the proof of efficiency and accuracy of the
model by stress-testing at the end of the Experiments section.
limitations: limitations: The presentation is somewhat confusing. In Figure 2, trying to include the training losses makes the pipeline cluttered. 

Details regarding the VAE seem not mentioned anywhere. Where are their parameters coming from?

The authors mentioned that their PAM module is different from previous works. However, they do not justify the differences in detail enough. Given that the authors' idea is interesting, its implementation warrants a more in-depth discussion and possibly more experiments to justify the design choices, rather than traditional ablation studies.

To formally support the claim of the paper's main purposes, the authors should describe the problems of uncertainties in a more formal fashion and prove or support that the model can help segregate the process from the segmentation phase.
paper_summary: paper_summary: This article applies diffusion models to FSS tasks, in which it is difficult to maintain spatial relationships between inputs and outputs due to probabilistic noising and denoising processes. To address this issue, a diffusion-based Segmentation network is proposed.
strength_weakness: strength_weakness: The novelty is to decouple probabilistic denoising and segmentation processes to maintain spatial relationships between inputs and outputs. And the experiments on PASCAL-5𝑖 and COCO datasets show that the method proposed achieves the state-of-the-art performance.
limitations: limitations: 1) The effect has not been significantly improved.
2) The problem is that it is difficult to maintain spatial relationships between inputs and outputs due to probabilistic noising and denoising processes. The experimental results do not indicate whether the problem has been solved.
paper_summary: paper_summary: This paper presents a model called DiffSeg, which leverages prior knowledge from diffusion models to assist in few-shot semantic segmentation tasks. To address the issues of semantic and spatial information inconsistency and the uncertainty in the generation process, the paper proposes the Perceptual Attention Module (PAM) and the Spatial Control Module (SCM) respectively. Experiments have demonstrated that DiffSeg achieves competitive results in few-shot semantic segmentation tasks.
strength_weakness: strength_weakness: 1. The paper proposes utilizing edge information to enhance the diffusion model's ability to preserve the original shape, thereby increasing the certainty of the generated segmentation masks.
2. The number of modules that need to be trained is relatively small. Although the model uses CLIP and diffusion models, only PAM and SCM need to be trained to adapt these features, alleviating the difficulty of fine-tuning the pre-trained models.
3. The paper is well-written, and the experiments are thorough, demonstrating the validity of the model.
limitations: limitations: 1. Compared to the original features, what does the Spatial Control Module learn? The model keeps the parameters of the diffusion model completely frozen, yet the features change after passing through the Spatial Control Module, leading to domain shift. Explaining the changes that the features undergo can better elucidate the role of edge information and SCM.
2. Compare the model size and efficiency with other related works. In semantic segmentation tasks, it is crucial to balance performance and efficiency. DiffSeg employs both diffusion models and CLIP models, making it necessary to compare metrics such as model size and inference time.
paper_summary: paper_summary: This paper presents a new way to apply diffusion model in the task of Few-shot semantic segmentation (FSS). The spacial control module and the perceptual module are designed to encode the semantic information via the diffusion model and decode the semantic classification, respectively. Experimental results show that the proposed method can outperform previous SOTA methods on multiple metrics.
strength_weakness: strength_weakness: + The presentation of this paper is good.

+ The way of how to apply diffusion models into FSS is novel, although there exist many works about diffusion-model-based semantic segmentation.

+ The visualization is clear and enough. The good and failure cases show in the main paper and the supplementary materials can provide enough information for the researchers in this community.
limitations: limitations: - One of our main concerns is what the role CLIP plays in the proposed framework? Intuitively, we think the semantic information from support images are important. According to the pipeline, the CLIP seems to help to encode the semantic information of support images. After checking the related works, we find that CLIP is rarely used. So, using CLIP may be a contribution? But the discussion about CLIP is missing. We want to see more discussions about the role of CLIP. If there is an ablation study about CLIP, this concern can be clearly solved.

- Although we find that previous SOTAs rarely provide the information about computational cost, we still want to know the inference cost (time and FLOPS) of the proposed method. Because the proposed framework includes both the CLIP and UNet of diffusion model, we think the computational cost may be large.

- Why to choose edge maps for SCM? Can other conditions help? Discussions about this choice may be expected.

- Citation information should be updated: Citation 12 has been accepted by AAAI 24.