Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation
Authors: Muzhi Zhu
Yang Liu
Zekai Luo
Chenchen Jing
Hao Chen
Guangkai Xu
Xinlong Wang
Chunhua Shen
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion Model; Few-shot Semantic Segmentation
Primary area: diffusion_based_models

Abstract: The Diffusion Model has not only garnered noteworthy achievements in the realm of image generation 
but has also demonstrated its potential as an effective pretraining method utilizing unlabeled data. 
Drawing from the extensive potential unveiled by the Diffusion Model in both semantic correspondence and open vocabulary segmentation, our work initiates an investigation into employing the Latent Diffusion Model for Few-shot Semantic Segmentation.
Recently, inspired by the in-context learning ability of large language models, Few-shot Semantic Segmentation has evolved into In-context Segmentation tasks, morphing into a crucial element in assessing generalist segmentation models.
In this context, we concentrate 
on Few-shot Semantic Segmentation, 
establishing a solid foundation for the future development of a Diffusion-based generalist model for segmentation. Our initial focus lies in understanding how to facilitate interaction between the query image and the support image, resulting in the proposal of a KV fusion method within the self-attention framework.
Subsequently, we delve deeper into optimizing the infusion of information from the support mask and simultaneously re-evaluating how to provide reasonable supervision from the query mask.
Based on our analysis, we establish a simple and effective framework named DiffewS, maximally retaining the original Latent Diffusion Model's generative framework and effectively utilizing the pre-training prior. Experimental results demonstrate that our method significantly outperforms the previous SOTA models in multiple settings.

Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation
Muzhi Zhu
Yang Liu
Zekai Luo
Chenchen Jing' Chunhua Shen
Hao Chen
Guangkai Xu
Xinlong Wang"
Zhejiangg University
Beijing Academy Artificial Intelligence
Abstract
The Diffusion Model has not only garnered noteworthy achievements in the realm of image generation but has also demonstrated its potential as an effective pre- training method utilizing unlabeled data. Drawing from the extensive potential unveiled by the Diffusion Model in both semantic correspondence and open vo- cabulary segmentation, our work initiates an investigation into employing the Latent Diffusion Model for Few-shot Semantic Segmentation. Recently, inspired by the in-context learning ability of large language models, Few-shot Semantic Segmentation has evolved into In-context Segmentation tasks, morphing into a crucial element in assessing generalist segmentation models. In this context, we concentrate Few-shot Semantic Segmentation, establishing solid foundation for the future development of a Diffusion-basee generalist model for segmenta- tion. Our initial focus lies in understanding how to facilitate interaction between the query image and the support image, resulting in the proposal of KV fusion method within the self-attention framework. Subsequently, optimizing the infusion of information from the support mask and simultaneously re-evaluating how provide reasonable supervision from the query mask. Based on our analysis, we establish a simple and effective framework named DiffewS, maximally retaining the original Latent Diffusion Model's generative framework and effectively utilizing the pre-training prior. Experimental results demonstrate that our method significantly outperforms the previous SOTA models in multiple settings. Our code is released at: atttp://githin com/ aim-uofa/Diffen
delve deeper into
Introduction
The Diffusion Model (DM) has demonstrated powerful capabilities in multiple visual generation tasks, including image generation 212. image editing [3] 4). video generation (5/7), etc. At the same time, DM has also been proven to be a powerful method for self-supervised pre-training [8] [ employing unlabelled data. To exploit the representation ability of DM, there are currently two emerging topics in vision research: improving the learning paradigm [10 [1 and downstream task adaptation 12) The latter often focuses on the Latent Diffusion Model (LDM). By compressing images into latent space, it significantly decreases computational expenses and emerges as the first open- poen-source Text-to- Image Diffusion Model scaled up the LAION-5B (15) level. For example, ODISE DVV (12). DatasetDM [17] adapt LDM to multiple tasks such as depth estimation, semantic segmentation, but they all require training additional decoder heads, which increases training costs and may undermine the generalization ability and generation quality. Therefore, some works [3] 14] have emerged that attempt to repurpose the Diffusion Model's s generative framework and apply it to visual perception tasks without adding extra decoder heads. Nonetheless, these paradigms still cannot uniformly adapt to all tasks.
MZ, and ZL contributed equally. YL the project lead. HC the corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Let's reconsider the most fundamental question in using generative models for visual perception: how to design a fine-tuning framework that can guarantee both generalization ability and precise prediction of details? ? Unfortunately, existing methods do not sufficiently address this challenge. The demands of the FSS task for open-set generalization and high-quality segmentation results precisely align with this challenge. Thus, our first motivation is to further address the fundamental question posed above by exploring the Diffusion Model on the FSS task.
FSS aims to segment query images given support samples. Traditional FSS methods 18-20 rely on a pre-trained backbone, achieving semantic matching and pixel-level prediction tasks through designing complex modules and long-term training. Recently, with the emergence of SAM 22], some works are based on foundation models to complete FSS, such as Matcher [22]. It employs DINO combine SAM
for semantic matching and SAM for segmentation. Similarly, other works 24 25
with CLIPP or MILLM to complete other open-set segmentation tasks. The current methods deal with matching( (semantic) and segmentation as two distinct tasks through different modules. The Diffusion Model itself, however, exhibits significant potential in fine- ene-grrined pixel prediction tasks [3] 14 [6 and semantic correspondence tasks (26 Hence, we seek to maximize the reuse of the generative framework by taking advantage of the innate priors within the Diffusion Model to accomplish the FSS task.
Recently, inspired by the in-context learning ability of large language models, Few-shot Semantic Segmentation has further evolved into the In-contex: Segmentation [29] 30 task (see Section 2 In-context Segmentation requires the model have in-context learning ability for few-shot samples. posing new challenges to model's generalization capabilities. Consequently, it's now recognized as a erucial component in the evaluation process for generalist segmentation models. Therefore, the second motivation of our work is to lay the groundwork for the development of diffusion- based generalist segmentation models.
As foundational work of Diffusion-based methods in the FSS field, we strive to achieve optimal performance with a simple and efficient design, while maximally preserving the generative framework of the Latent Diffusion Model. This minimal disruption to the original UNet structure allows us to better make use of pre-trained priors. We embark on a systematic exploration around the following four questions: 1) How to implement the interaction between the query image and the support image? 2) How to effectively inject information from the support mask? 3) What is a reasonable form of supervision from the query mask? 4) How to design effective generation process to transfer the pre-trained diffusion models to mask prediction task? Based on our observations, we ultimately establish the DiffewS framework and validate it in multiple settings, demonstrating the effectiveness of our method. Our main contributions include:
We systematically study four crucial elements of applying the Diffusion Model Few-shot Semantic Segmentation. For each of these aspects, we propose several reasonable solutions and validate them through comprehensive experiments.
Building upon our observations, we establish the DiffewS framework, which maximally retains the generative framework and effectively utilizes the pre-training prior. Notably, we introduce the first diffusion-bare model dedicated to Few-shot Semantic Segmentation, setting the groundwork for a diffusion-based generalist segmentation model.
We validate the effectiveness of the DiffewS framework under several experimental settings, demonstrating that our method not only achieves performance comparable with the state- of-the-art (SOTA) model in a strict Few-shot Semantic Segmentation setting, but also significantly outperforms the current SOTA model in 'in-context learning' setting,
2
Related Work
Diffusion models have shown impressive performance on visual generation tasks such as text-based image generation [22] 2]. image editing B 4 and video generation 75 Current research on leveraging Diffusion models to enhance visual perception tasks mainly focuses on two directions: one is the direct use of diffusion models to generate images, aiming to address the issue of insufficient data, such as instance segmentation 31 semantic segmentation R4] few-shot segmentation (35] and so on. Another direction to transfer features from Diffusion models to other visual tasks, which aligns with the research direction of this paper.

---

ODISE 16 uses frozen diffusion models for panoptic segmentation of any category in the wild. DVP 12 DatasetDM [17], GenPercept [36], Geowizard [37] adapt LDM to multiple tasks such as depth estimation, semantic segmentation, and surface normal. Marigold [3] fine-tunes diffusion models on synthetic data for affin-iivariint monocular depth estimation and achieves impressive performance. Different from the above methods, we focus on using diffusion models to model the visual correlations of multiple reference images and a target image for few-shot segmentation. The most related work to this paper is concurrent study [38), which focuses utilizing diffusion models for in-context segmentation. However, it disrupts the original U-Net structure and the priors of the diffusion model to some extent. In contrast, our work offers more comprehensive and systematic analysis of applying diffusion models to Few-Shot Semantic Segmentation tasks.
Few-shot semantic segmentation 39 40 aims to segment target objects in an input image given a few annotated support images. Traditional FSS methods either explore prototype learning
[1 of
support images to predict query images' masks or use pixel-level information 44
45
I8
to exploit
the support information. For example, some works (2 30 demonstrate powerful generalization
ability by unifying various segmentation tasks in an in-context learning framework. SegGPT 330 can exactly segment any semantic conception by using one or a few support images, which motivates us to explore the potential of the diffusion model for the FSS task under the in-context setting 300
3 Preliminary
We first review the Latent Diffusion Model [2 used in our paper. It consists of an auto-encoder (VAE) and UNet. The auto-encoder facilitates two-way transformation between the RGB image I E R xx x and the latent space z E Axxxx Both the forward and backward processes of diffusion are carried out in the latent space, and we denote the noisy latent code at time / as
(t) = Va.2 + vi açe, where a₁ = II [I - B&) the noise schedule. B3 is the variance sampled from variance schedule B, (0, 1); The UNet can be considered as a series of equally weighted denoiser eg (z(()),))
.). The training objective & can be simplified as:
||| ee Eo (z(),
C = E N.E.e.
Furthermore, to simplify comprehension and narration, we can reparametrize the output of UNet eg as the form of v-prediciton Ug. The training objective can be further elaborated as:
[||-- uo (2())
(z())+)l
L = -=E,,o-x00.,,eeuur
(2)
This implies that the goal of every training round is denoise (t) to z for any time step t.
Secondly, we present our task definition, using one-shot segmentation as an illustration. Given a data triplet (I,, M,, I,), here I, and I, denote the support image and query image respectively, both sharing an overlapping category c. M, is the mask category e in the support image. Our task is to predict the mask corresponding category in L,. In the strict one-shot segmentation setting, the category sets of the training set and the test set are disjoint.
Our objective is to fully utilize the priors in the Latent Diffusion Model and equip it with Few-shot Semantic Segmentation capabilities. This leads us to reuse the original VAE to convert I,, I, and M, into latent variables z., Zy and Zmq- Thus, our task is further simplified to explore how to improve the structure of UNet to vg so that can accept z,, Z, and M, as inputs, and use Zmq as supervision.
This supervised approach in the latent space has been certified effective in tasks such as depth esti- mation [ and semantic segmentation [4]. Concretely, our training objective Lyss is transformed into:
Lass E(e,, 2, M, IDD ||zmg vg (Za, Zeg M.
where D represents the constructed training dataset. In addition, we omitted the input of time I. Our early experiments revealed that performing multiple steps of noise addition and denoising during training did not bring performance improvement.
3

---

Latent Encoder
Lafent
concat
Encoder
M
Encoder
Rec
(8) Support Image Enooding.
(b) Query Image Encoding.
Training Objective
ta value
Re Rece
K
K
<
Ryg
query
WAME
tokens
tokens X
Res
(c) DiffewS fine-tuning protocol.
Self-Attention (FSA).
Figure Overview of DiffewS framework. (a)(b) display that query image I query mask Mg. support image I, and support mask M, are all encoded by VAE into latent variables Zg, Zmg. 2s, Zme, respectively, where 2, and Zong concatenated to input into UNet. (c) demonstrates the DiffewS fintuning protocol (d) elucidates the detailed implementation of FSA, acquiring information from support images by concatenating query and key features.
Method
Our investigation into model design primarily adheres to two criteria: 1. Strive for the design to be as simple and efficient as possible, while optimizing performance in Few-shot Semantic Segmentation. Maximize the preservation of the Latent Diffusion Model's generative schema, minimizing alteration the original UNet structure, so as better utilize the pre-training prior.
Specifically, four key issues need to be addressed: 1) How to facilitate interaction between the query image and support image? How effectively incorporate information from the support mask? 3) What form of supervision from the query mask would be most reasonable? 4) How to design an effective generation process transfer the pre-trained diffusion models mask prediction task? In this section, we discuss the four issues mentioned above in detail. We engage in fair comparison tests and analysis on several feasible strategies. Drawing on our observations, we eventually settle on our framework, DiffewS (see Figure
4.1 Interaction between query and support images
We first decompose the block of the 1-th layer in UNet into three components: self-attention layer SelfAttn, cross-attention layer CrossAtin, and feedforward layer FFN. Given the feature map X! of the I-th image and the textual input t (which is empty character in our task), obtain:
X
FFN (CrossAttn (SelfAttn (X') CLIP (ez)(t)))
where CLIP text represents CLIP text encoder, and we have skipped over skip cxip-connection in the formula.
Before considering the incorporation of the support mask, two straightforward and intuitive methods can be leveraged to facilitate interaction between the query image and support image. One approach entails interaction within the self-attention module, while the other involves interaction within the cross-attention module.
KV Fusion Self-Attention. We first propose a KV fusion method in self-attention layer to achieve interaction between query image and support image. For the input image feature X, the standard self-attention layer first maps it to query Q key K and value V with a linear projection layer. Therefore, SelfAttn (X) can be further represented as:
QKT d
X SeeffAttn(X) Attention Q Q,K.V Softmax
JV
(5)

---

where d is the dimension of query and key, while X is the feature updated by self-attention. Back to our task, we can also map the features of the support image and query image X, and X, to Q.- K. V. and Qq. gg V, through the linear projection layer. We hope that the features of the query image can effectively utilize the information of the support image, so we need to let Q2 access K, and Va *. To achieve this, we can concatenate K, and K, to form gs = [K K,]- Similarly, we can get V qs = aV V.J. Finally, our KV Fusion Self-Attention layer can be represented as: ; FusionALtni Xg. x = At Aenttion Q2 Kgs, Vgs) Since we only replaced K and V, we can fully reuse the weights of the original self-attention.
The second alternative is inject information originating from the support image via cross-attention. This strategy has been widely used in Customized Text-to-lmage Generation 49] In particular, the initial cross-atiention employed to introduce the text information, encoded using CLIP text encoder. We can encode the support image into series of tokens using the CLIP image encoder and utilize as the cross-attention input. At this point, the process can be represented as:
Tokenized Interaction
Xg" CrossAttn(Xx Flatten (CLIP ing (I.)))
(7)
where Flatten means flattening the token sequence after image encoding. CLIP smg represents the CLIP image encoder corresponding to the CLIP text encoder used in the original UNet.
4.2 Injection of support mask information
Building upon the Self-attention kv fusion approach, we investigate methodologies for incorporating support mask information. We categorize the injection methods into four types:
a. Concatenation The support mask M, can be converted into an RGB image, then directly encoded into latent variable Zma using VAE, which is then concatenated with in the channel dimension. Due to the resulting mismatch dimensionality from the concatenation, we adopt the approach of Marigold [13] where the first layer weight tensor is duplicated and its values are halved.
b. Multiplication We can directly multiply M, the image I, to form the image I; = I,-M,. and finally encode I into a latent variable 2 using VAE as the input of UNet.
Attention Mask M, can serve as attention mask to control self-attention that only K, in the masked region can be accessed by Q2, Since the feature map sizes of different layers are different, we need to resize M, to fit the dimensions of each layer.
d. Addition Alternatively, M, can be directly added to the image I,, generating the image I; 0.51, + 0.5M, Following that, I is encoded into latent variable z using VAE, which is then used as the UNet input.
For cross-attention tokenized interaction, information of the support mask can also be injected in the same four ways. There are just some slight differences in the implementation details (see the Appendix A.3)..
CA
We carry out a comparison of two interaction methods (Sec- tion [4.1) paired with four injection methods (Section 4.2) these eight combinations are then verified experimentally. and the results are presented in Figure 2 Overall, we observe that KV Fusion Self-AAttentinn(FSAA outperforms Tokenized Inter- action Cross-Attention((TAA) We attribute this mainly the preservation and flexible utilization of information from the support image by FSA. Conversely, TCA, which only com- presses support image to tokens via the CLIP image encoder, leads to some information loss. Notably, within the FSA, the Concatenation method surpassed the other three. It offered more free- frre-form handling of RGB images and MASK informa- tion via subsequent learnable convolutional layers, compared to other hard injection methods. In the case of TCA, the Attention Mask method seems more apt as other operations are actually constrained by the CLIP image encoder. The CLIP image encoder itself not good at dealing with mask information. Of course, we believe that there still room for further exploration here, referring to FGVP [50].
PUU 190
10-
concalenation
mutepiccation
attenton mask
addition
Figure 2 Exploring the Interaction and Injection Methods
5

---

211
26.1
10-
(a) White foreground Black background
(b) foreground black background
(c) Black foreground real background
(d) Adding mask on roal image
Comparison of different superviaion.
Figure 3 Illustrations and comparisons of differrnt forms of supervision from query mask.
4.3 Supervision from query mask
In Section 3 we mentioned that we encode the query mask M, into a latent variable Zmnq and directly supervise in the latent space. However, M3 E [0. 1| is vo-dimensional mask, while the input of VAE needs be an RGB image. Consequently, conversion of M, into an RGB image becomes necessary, but it's unclear which form of conversion would yield optimal results as no research has delved into this as yet. reasonable conversion method should satisfy the following two conditions:1. 1, It is easier for UNet to learn 2. It is more convenient to get the final segmentation result through post-pres possssrocessing. In this section, we explore the following four forms of conversion.
a. White foreground black background Visualizing the segmentation annotation with white mask and black background is a common way in the academic community. Specifi- cally, we only need replicate M, three thrice along the channel dimension to form the corresponding RGB image denoted by the mask. We employed this conversion approach as a default in Section 4.2
b. Real foreground black background Considering LDM's original pre-training on real images, forcing the model to output purely black-and-white images that do not fit within real-image distribution might amplify the model's ' learning difficulty. Therefore, we also attempted to retain the real pixels of the foreground, while setting the background to black
Black foreground real background Following the same logic, also try preserving the pixels of the real background but render the foreground pixel black.
d. Adding mask on real image We also consider overlaying M, on the real image to form the mask on the real image, which the Addition method mentioned in Section 4.2 This approach makes the output space of UNet closer to the distribution of real images, but it requires more complex post-processing to get the final segmentation results. That is, we need to subtract the original image from the model output to get the final segmentation result.
As shown in Figure 3 assess the performance of the four forms of supervision, among which (a) method achieved the best performance in all experiments. Although (b) (c) (d) methods being closer the real image distribution, the performance is lower. On the one hand, it difficult obtain the mask through simple post-pros pess-processing, and on the other hand. it may increase the learning difficulty because the model needs retain the ability generate the original image. In conclusion, our results demonstrate that UNetUNet can effortlessly learn to output in forms such as 'white foreground + black background', Therefore, we eventually chose this approach for Diffews.
4.4 Exploration of generation process
In this section, we further discuss how to design an effective generation process to transfer the pre-trained diffusion models to mask prediction tasks. Inspired by the success of transferring pre- trained diffusion models depth estimation task [3 51, explore three different mask generation processes. The illustration of different mask generation processes is shown in Figure
* Multi-step noise-to-mask generation (MN2M) MN2M follows the denoise pipeline of original diffusion models. The training and inference schemes of MN2M are similar to Marigold [3]. Figure 4b1) shows the illustration of inference process. The image latent 24

---

Support
Lakeri Encoder
Lalani Decocer
Prediction
(4) Pipeline of diffusion for mask generation.
reploce
suppprt KV
(02) Muli-step mage to mask generation (M2M).
23.0
auppoet
x1
20-
0.
MEM
replaoe
R
R
(c) Comparisons of different pipelines mask generation process.
(b1) Mulli-stpp no metee---mmkk generation (MN2M).
imago-tomask (DI2M).
(b) Illustaations of different pipelines mask generation process latent space.
Figure 4 Illustrations and comparisons of different mask generation processes.
concatenates with the mask latent mm- The UNet takes it as input and predicts the mask latent ((1-1), After T steps, the final mask latent 2 is decoded into mask prediction. The mask latent ((T) is initialized as random noise. We also use the annealed multi-resolution
noise and test-time ensemble tricks (13] proposed in Marigold.
Multi-step image-to-mask generation (MI2M) MI2M formulates the diffusion denoising process deterministic multi-step conversion process from image to prediction, similar to DMP [5I]. Figure (b2) shows the illustration of inference process. The mask latent (T) is initialized as image latent z.. Then similar to MN2M, the UNet takes 2 as input and predicts <((-1) After T steps, the final mask latent ( is decoded into mask prediction.
One-step image-to-mask generation (OI2M) O12M further transforms MI2M's multi-step prediction into a one-step prediction, i.e., UNet takes Zy as input and outputs the prediction
2ug directly.
We explore the mask generation pipeline starting from MN2M. As shown in Figure Te) MN2N achieves 15.2% mloU. Then, change MN2M into MI2M keeping same variance 8² (0.00085, 0.012), respectively representing the initial and final values of 3 in the DDIM sched- uler. The performance has improved by 4.7% mloU. However, despite the improvement, both methods exhibit suboptimal performance. We hypothesize that this is because adding very small noise or image to the binary mask during the training process and then predicting does not lead to challenging task compared with diffusion pre- pre-training.
We hypothesize that the suboptimal performance is due the minimal noise image added the binary mask during training, which results in an insufficiently challenging task compared to diffusion pre-training. The binary mask is inherently simpler than natural images, and even after adding noise, the latent mask can still easily distinguish between the foreground and background. This simplicity causes significant information leakage during UNet training, ultimately leading poor performance.
To verify this hypothesis, we increase the variance of MI2M from 8 = (0.00085, 0.012) to B² (0.0272, 0.384). The performance has significantly improved by 23.3% mloU. To fully increase the challenge of training. we convert MI2M into O12M, which does not introduce any ground-truth information into the input of the UNet during training. Additionally, O12M reduces the number of iterations to one, significantly boosting the network's 's predictive efficiency. As shown Figure O12M achieves the best performance, making it the preferred choice for the mask generation pipeline.
4.5 1-shot to N-shot
So far, we have primarily explored the training and inference processes specifically designed for 1-shot scenarios. A natural question arises: can this framework be extended to n-shot settings? To address this, we first present the simplest and most straightforward method for adaptation, which requires only minor modifications during the inference phase to accommodate n-shot tasks.
7

---

we introduced how to inject the information of the support image into the features of the query image using the KV Fusion Self-Attention method. In inference, our support set S may contain more than one image, S = { s2, ..., sn J. We encode each image into the features X Correspondingly, after mapping, we can obtain a series of Qsi. K... si and Qgi. K Vgi. We can concatenate Kg and K to form g = [K qi: 1 2 K Km and similarly we can obtain Va= = [Vgi, V1l, V V22 ..., V.n Finally, our kv fusion self attention layer can be represented as:
In the Section 4.1
x KVF X. X.) = Attention (qrii Kges Vas)
(8)
While the aforementioned solutions enable N-shot inference, their performance does not match that of ttate-of-the-ar (SOTA) models. This discrepancy primarily arises because the model receives only a single support image during the training phase, which leads to inconsistencies when transitioning to the inference phase with 5-shot or 10-shot configurations.
To address this issue, we explore improvements from both the inference and training perspectives. From the perspective of inference, transitioning from I-shot to N-shot involves concatenating the keys and values of additional support samples, which significantly increases the number of keys and values processed during inference. To address this, we implement random sampling of the keys and values from the support samples during inference, ensuring that their quantity matches that of the training phase (see Table 6). Another more straightforward idea is to introduce multiple support samples during the training phase. In this way, the model can learn how to utilize multiple support images during training. we randomly select 1 to N support samples as input using KV Fusion in Equation (8) during single training iteration (see Table
Our experiments demonstrate that improvements during the training phase are more effective than those during the inference phase. Therefore, we include the results of the model with training phase improvements in Table 2
5 Experiment
Datasets We test our method in two settings: 1. Strict few shot setting: Following the few-shot setting on COCO-20' [52]. we organize 80 classes from COCO2014 (53] into 4 folds. Each trial consists of 60 classes allocated for training and 20 classes designated for testing. For evaluation, we randomly sample 1000 reference-target pairs each fold with the same seed used in HSNet [18).
2. In-context setting: Following the setting in SegGPT 300. COCO, ADE 54 and PASCAL VOC [55] serve as the training set. In-domain testing is conducted on COCO-20' and PASCAL-5' [39] to evaluate our model. In line with Matcher [22). LVIS-92 92' function as the out-of-domain test set.
The Adam optimizer is used with a weight decay set at 0.01 and a learning rate of le-5, coupled with a linear schedule. In terms of data augmentation, our methodology only involves resizing the input image directly to 512x512. No additional data augmentation occurs. Under the strict few-shot setting, the model undergoes training on four V100 GPUs With the gradient accumulation set at 4, the total batch size comes to 16. Training carries out for 10,000 iterations, typically requiring six hours. For in-context setting, since the training set is larger, we keep other hyperparameters consistent with the strict few-shot setting, and adjust the total training iterations to 30000 iterations. Lastly, our ablation experiments are validated on Foldo of COCO-20' (52). The training took place on a single 4090 GPU, with gradient accumulation set at 4, which brought the total batch size to 4. The training, which consisted of 10,000 iterations, took roughly 11 hours.
Implementation details We initialize our model with Stable Diffusion 2.1 2
5.1 In-context setting
We first compare DiffewS with other generalist models such as Painter
29
SegGPT
29
PerSAM-
F(58), and Matcher 22 as well as specialist models like HSNet 18 VAT (56, FPTTrans
37
Regarding the specialist models, we directly refer to the results presented within the SegGPT
30
and Matcher [22] research papers. These specialist models are also trained on the test categories
from COCO 53 and PASCAL VOC (55). We employ COCO-204 (52 and PASCAL-5' 39 to
validate the in-domain performance of DiffewS. Remarkably, on COCO, DiffewS achieves a 1 1-shot score of 71.3, considerably exceeding the generalist model SegGPT (+15.2) and specialist model
FPTrans (+ 14.8), both trained with in-domain data. DiffewS furthermore significantly outperforms
8

---

Table Results few-shot semantic segmentation on COCO-20', PASCAL-S", and LVIS-92', under
in-context setting.
COCO-20*
PASCAL-5
LVIS 92'
Methods
Venue
one-shol
few-shot
one-shot
few-shol
one-shot
few-shot
HSNet
ICCV'21
41.7
50.7
68.7
73.8
17.4
22.9
VAT 156
ECCV'22
42.9
49.4
72.4
76.3
18.5
22.7
HPTrans
[57
NeurlPS'22
56.5
65.5
77.7
83.2
Painter
29
CVPR'23
32.8
32.6
64.5
64.6
10.5
10.9
SegGPT
30
ICCV'23
56.1
67.9
83.2
89.8
18.6
25.4
PerSAM
58
23.0
15.6
ICLR'24
PerSAM-F
588
23.5
18.4
Matcher
22
ICLR'24
52.7
60.7
67.9
75.6
33.0
40.0
DiffewS
this work
71.3
72.2
88.3
87.8
31.4
35,4
Table 2 Results of strict few-shot semantic segmentation on COCO-20', DiffewS-n represents using training time improvements for N-shot.
I-shot
5-shot
Methods
Venue
20
20
20
20
20
20
20
20
HSNet 18
ICCV'21
37.2
44.1
42.4
41.3
41.2
45.9
53.0
51.8
47.1
49.5
CyCTR [59
21
38.9
43.0
39.6
39.8
40.3
41.1
48.9
45.2
47.0
45.6
VAT [56
ECCV'22
39.0
43,8
42.6
39.7
41.3
44.1
51.1
50.2
46.1
47.9
BAM 60
CVPR 22
43.4
50.6
47.5
43.4
46.2
49.3
54.2
51.6
49.6
51.2
DCAMA
19
ECCV'22
49.5
52.7
52.8
48.7
50.9
55.4
60.3
59.9
57.5
58.3
HDMNet (20
CVPR'23 23
43.8
55.3
51.6
49.4
50.0
50.6
61.6
55.7
56.0
56.0
DiffewS
47.7
56.4
51.9
48.7
51.2
52.0
63.0
54.5
54.3
56.0
this work
S-
47.1
56.6
53.8
48.3
52.2
57.3
66.5
60.3
58.8
60.7
SAM-based models PerSAM-F (+47.8) and Matcher (+18.6). On PASCAL-5', DiffewS records 88.3 in 1-shot, clearly surpassing SegGPT (+5.1) and Matcher (+20.4). These results evidence that DiffewS effectively utilizes the prior of Stable Diffusion, unlocking the full potential of Stable Diffusion in segmentation. Furthermore, the generalization ability of DiffewS. In this setting, DiffewS registers 31.4 in 1-shot and 35.4 in 5-shot, markedly outperforming other generalist models, aside from Matcher. It worth mentioning that Matcher simultaneously utilizes two Foundation models (SAM (2] and DINO V2 (23) ), and SAM itself is pre-trained on an exhaustive, finely annotated segmentation dataset. On the other hand, DiffewS undergoes fine-tuning iterations, still delivering performance that rivals Matcher. This indicates that using the paradigm of DiffewS, there is potential to achieve significant breakthroughs in the segmentation field if further trained on larger-scale segmentation data. should be noted that the improvement of DiffewS in 5-shot is not significant, with a 4.0 distinct improvement only on LVIS-92'. This might be due to the presence of many small objects in the support images LVIS, so increasing the number of support images can alleviate this problem. Conversely, the DiffewS 5-shot performance on PASCAL-51 (39 slightly deficient compared to the 1-shot. This could be ascribed to the presence relatively larger and more simplistic objects within PASCAL VOC's support images, inputting more images might interfere with the original architecture of the model. In this case, we do not apply the improvement strategies discussed in Section 4.5 therefore, the relatively weaker performance in the 5-shot scenario reasonable.
examination LVIS 92-
22 underpins
relatively smaller quantity of segmentation data for limited
5.2 Strict few-shot setting
We also undertake validation of DiffewS under the standard few-shot setting, comparing with other specialist models such as HSNet [18), CyCTR (59), VAT (56). BAM (60], HDMNet [20], and DCAMA [19]. For the one-shot setting, the average performance of DiffewS across all four folds (SOTA) model DCAMA, scoring 50.9 mloU.
attains 51.2, surpassing the current state-of-ther Worth mentioning is that DCAMA relies on a highly complex additional block, whereas DiffewS entirely utilizes the generative framework of UNet. In terms of the efficiency of convergence, DiffewS necessitates just a 30000-iteration training. in contrast to both DCAMA and HSNet which require training spanning hundreds of epochs. typically costing several days. This demonstrates the successful employment of Stable Diffusion priors by DiffewS, thereby securing impressive performance without requiring extended periods of fine-tuning. In the five-shot setting, the average performance across four

---

Reference
GT
Prediction
Reference
GT
Prediction
Reference
GT
Prediction
Figure Qualitative results of one-shot semantic segmentation on LVIS-92'. The blue color denotes the support mask while the red represents the query mask.
folds reaches 56.0, higher than all other models aside from DCAMA. Currently, DiffewS primarily focuses the 1-shot situation lacking specific optimizations for the 5-shot scenario in its training and inference systems. This explains why DiffewS at present marginally inferior to DCAMA. Furthermore, when employing our proposed training improvement strategy, DiffewS-n outperforms other models in both the 1-shot and 5-shot settings.
5.3 Visualization
As shown in Figure DiffewS effectively segments categories not in the training set, such as slippers and aprons. It also accurately segments objects of different styles and smaller items, demonstrating strong generalization capabilities. In some cases, DiffewS even achieves more accurate results than GT.
In addition, DiffewS demonstrates impressive results various cross-style segmentation tasks and small object segmentation cases (see Figure (). We hypothesize that DiffewS's exceptional generalization ability stems from its extensive utilization prior knowledge from diffusion models. However, DiffewS also struggles with certain challenging cases, also present several failure cases in Figure 7 and categorize the reasons for these failures.
Conclusion
In this work, we have presented DiffewS,, simple and efficient framework for few-shot semantic segmentation. By directly generating the target mask, DiffewS capable of retaining the original latent diffusion models' generative framework and effectively utilizing the visual prior of pre-trained diffusion models. By introducing several designs about multi-image interaction, information injection, and supervision signals, DiffewS outperforms SOTA models in the in-context learning setting, and reaches comparable performance to specialist models in the strict few-shot setting.
Limitation and more Discussions are provided in Appendix A.
Acknowledgement
This work is partially supported by the National Key R&D Program and the National Natural Science Foundation of China (No. 62206244).
10

---

References
[1] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Proc. Adv. Neural Inf. Process. Syst., 34:8780-887777 2021.
[2] Robin Rombach, Andreas Blatmmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In IEEE Conf. Comput. Vis. Pattern Recog.. 2022.
[3] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. Int. Conf. Learn. Represent., 2022.
[4] Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pix: Learning to follow image editing instructions. IEEE Conf. Comput. Vis. Pattern Recog., 2023.
[5] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proc. IEEE/CVF Int. Conf. Computer Vision, pages 7623-7633. 2023.
[6] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Proc. Adv. Neural Information Processing Systems, 35: 1633-8646, 2022.
[7] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022.
[8] Korbinian Abstreiter, Sarthak Mittal, Stefan Bauer, Bernhard Schölkopf, and Arash Mehrjou. Diffusion- based representation learning. arXiv preprint 205.v:2205./22..... 2021.
[9] Weilai Xiang. Hongyu Yang, Di Huang, and Yunhong Wang. Denoising diffusion autoencoders are unified self-supervised learners. In Int. Conf. Comput. Vis., pages 15802-15812, 2023.
[10] Drew A Hudson, Daniel Zoran, Mateusz Malinowski, Andrew Lampinen, Andrew Jaegle, James I. McClelland, Loie Matthey, Felix Hill, and Alexander Lerehner. Soda: Bottleneck diffusion models for representation learning. arXiv preprint arXiv:2311.17901, 2023.
[11] Xinlei Chen, Zhuang Liu, Saining Xie, and Kaiming He. Deconstructing denoising diffusion models for self-supervised learning. arXiv preprint 2...:::401..14004 2024.
[12] Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion models for visual perception. In Int. Conf. Comput. Vis., 2023.
[13] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Kon- rad Schindler. Repurposing diffusion-barn based image generators for monocular depth estimation. arXiv preprint arXiv:2312.02 2023.
[14] Hsin- Hsin-Ying Lee, Hung- Yu Tseng, Hsin- Hsin-Ying Lee, and Ming- Yang. Exploiting diffusion prior for generalizable dense prediction. In IEEE Conf. Comput. Vis. Pattern Recog., 2024.
[15] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in 2022.
Neural Information Processing Systems, 35:25278-2551
[16] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In IEEE Conf. Comput. Vis. Pattern Recog., 2023.
[17] Weijia Wu, Yuzhong Zhao, Hao Chen, Yuchao Gu, Rui Zhao, Yefei He, Hong Zhou, Mike Zheng Shou, and Chunhua Shen. Datasetdm: Synthesizing data with perception annotations using diffusion models. In Proc. Adv. Neural Inf. Process. Syst. 2023.
11

---

[18] Juhong Min, Dahyun Kang, and Minsu Cho. Hypercorrelation squeeze for few-shot segmenta- tion. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
[19] Xinyu Shi, Dong Wei, Yu Zhang, Donghuan Lu, Munan Ning, Jiashun Chen, Kai Ma, and Yefeng Zheng. Dense cross-query-and -unpor attention weighted mask aggregation for few-shot segmentation. In Eur. Conf. Comput. Vis., 2022.
[20] Bohao Peng, Zhuotao Tian, Xiaoyang Wu, Chengyao Wang, Shu Liu, Jingyong Su, and Jiaya Jia. Hicrarchical dense correlation distillation for few-shot segmentation. In IEEE Conf. Comput. Vis. Pattern Recog., 2023.
[21] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg. Wan-Yen Lo, al. Segment anything. Int. Conf. Comput. Vis., 2023.
[22] Yang Liu, Muzhi Zhu, Hengtao Li, Hao Chen, Xinlong Wang, and Chunhua Shen. Matcher: Segment anything with one shot using all all-purpose feature matching. Int. Conf. Learn. Represent. 2024.
[23] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07/ 2023.
[24] Haoxiang Wang, Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Mehrdad Farajtabar, Sachin Mehta, Mohammad Rastegari, Oncel Tuzel, and Hadi Pouransari. Sam-clip: Merging vision foundation models towards semantic and spatial understanding. arXiv preprint 10X.:2330.113305 2023.
[25] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv preprint arXiv:2308.0000 2023.
[26] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emer- gent correspondence from image diffusion. In Proc. Adv. Neural Inf. Process. Syst., 2023.
[27] Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. In Proc. Adv. Neural Inf. Process. Syst., 2024.
[28] Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hae Hsuan Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. In Proc. Adv. Neural Inf. Process. Syst., 2024.
[29] Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. In IEEE Conf. Comput. Vis. Pattern Recog., 2023.
[30] Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting everything in context. In Int. Conf. Comput. Vis., 2023.
[31] Hanqing Zhao, Dianmo Sheng, Jianmin Bao, Dongdong Chen, Dong Chen, Fang Wen, Lu Yuan, Ce Liu, Wenbo Zhou, Qi Chu, Weiming Zhang, and Nenghai Yu. X-paste: Revisiting scalable copy-paste for instance segmentation using clip and stablediffusion. In International Conference on Machine Learning, 2023.
[32] Chengxiang Fan, Muzhi Zhu, Hao Chen, Yang Liu, Weijia Wu, Huaqi Zhang, and Chunhua Shen. Divergen: Improving instance segmentation by learning wider data distribution with more diverse generative data. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3986-3995, 2024.
[33] Muzhi Zhu, Chengxiang Fan, Hao Chen, Yang Liu, Weian Mao, Xiaogang Xu, and Chun- hua Shen. Generative active learning for long-tailed instance segmentation. In Forty-first International Conference Machine Learning.
12

---

[34] Lihe Yang, Xiaogang Xu, Bingyi Kang, Yinghuan Shi, and Hengshuang Zhao. Freemask: Synthetic images with dense annotations make stronger segmentation models. Advances in Neural Information Processing Systems, 36, 2024.
[35] Weimin Tan, Siyuan Chen, and Bo Yan. Diffss: Diffusion model for few-shot semantic segmentation. arXiv preprint arXiv:2307.00773, 2023.
[36] Guangkai Xu, Yongtao Ge, Mingyu Liu, Chengxiang Fan, Kangyang Xie, Zhiyue Zhao, Hao Chen, and Chunhua Shen. Diffusion models trained with large data are transferable visual models. arXiv preprint ar arXhv:2403.06000, 2024.
[37] Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image. In ECCV, 2024.
[38] Chaoyang Wang. Xiangtai Li, Henghui Ding, Lu Qi, Jiangning Zhang, Yunhai Tong, Chen Change Loy, and Shuicheng Yan. Explore in-context segmentation via latent diffusion models. arXiv preprini arXiv:2403.09616 2024.
[39] Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. In Brit. Mach. Vis. Conf., 2017.
[40] Kate Rakelly, Evan Shelhamer, Trevor Darrell, Alyosha Efros, and Sergey Levine. Conditional networks for few-shot semantic segmentation. Proc. Int. Conf. Learning Representations Workshop, 2018.
[41] Gen Li, Varun Jampani, Laura Sevilla-Lara, Deqing Sun, Jonghyun Kim, and Joongkyu Kim. Adaptive prototype learning and allocation for few-shot segmentation. In IEEE Conf. Comput. Vis. Pattern Recog., pages 8334 8343, 2021.
[42] Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy. MeViS: large-scale benchmark for video segmentation with motion expressions. In Int. Conf. Comput. Vis., pages 2694-2703, 2023.
[43] Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, Philip HS Torr, and Song Bai. MOSE: A new dataset for video object segmentation in complex scenes. In Int. Conf. Comput. Vis., pages 20224-20234, 2023.
[44] Chi Zhang, Guosheng Lin, Fayao Liu, Jiushuang Guo, Qingyao Wu, and Rui Yao. Pyramid graph networks with connection attentions for region- one-shot semantic segmentation. In Int. Conf. Comput. Vis., 2019.
[45] Haochen Wang, Xudong Zhang, Yutao Hu, Yandan Yang, Xianbin Cao, and Xiantong Zhen. Few-shot semantic segmentation with democratic attention networks. In Eur. Conf. Comput. Vis., 2020.
[46] Yang Liu, Chenchen Jing, Hengtao Li, Muzhi Zhu, Hao Chen, Xinlong Wang, and Chunhua Shen. A simple image segmentation framework via in-context examples. arXiv preprint 24X1v:2410.04842. 2024.
[47] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. In Int. Conf. Comput. Vis., 2023.
[48] Zhen Li, Mingdeng Cao. Xintao Wang, Zhongang Qi, Ming-Ming Cheng, and Ying Shan. Photomaker: Customizing realistic human photos via stacked id embedding. arXiv preprint arXiv:2312.04461, 2023.
[49] Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen. Instantid: Zero-shot identity-preserving generation in seconds. arXiv preprint axXiv:2401.05119 2024.
[50] Lingfeng Yang. Yueze Wang, Xiang Li, Xinlong Wang, and Jian Yang. Fine-grained visual prompting. In Proc. Adv. Neural Inf. Process. Syst., 2024.
13

---

[51] Hsin- Ying Lee, Hung- Yu Tseng, and Ming- Haunn Yang. Exploiting diffusion prior for general- izable pixel-level semantic prediction. arXiv preprint arXiv:2311.18822 2023.
[52] Khoi Nguyen and Sinisa Todorovic. Feature weighting and boosting for few-shot segmentation. In Int. Conf. Comput. Vis., 2019.
[53] Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Eur: Conf. Comput. Vis., 2014.
[54] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In IEEE Conf. Comput. Vis. Pattern Recog., 2017.
[55] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. Inl. J. Comput. Vis., 2010.
[56] Sunghwan Hong, Seokju Cho, Jisu Nam, Stephen Lin, and Seungryong Kim. Cost aggregation with 4d convolutional swin transformer for few-shot segmentation. Eur. Conf. Comput. Vis., 2022.
[57] Jian-Wei Zhang, Yifan Sun, Yi Yang, and Wei Chen. Feature-proxy transformer for few-shot segmentation. In Proc. Adv. Neural Inf. Process. Syst. 2022.
[58] Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junting Pan, Hao Dong, Peng Gao, and Hongsheng Li. Personalize segment anything model with one shot. arXiv preprint arXiv:2305.03048, 2023.
[59] Gengwei Zhang, Guoliang Kang, Yi Yang, and Yunchao Wei. Few-shot segmentation via cycle-consistent transformer. Proc. Adv. Neural Inf. Process. Syst., 2021.
[60] Chunbo Lang, Gong Cheng, Binfei Tu, and Junwei Han. Learning what not to segment: new perspective on few-shot segmentation. In IEEE Conf. Comput. Vis. Patlern Recog., 2022.
[61] Lu Qi, Lehan Yang, Weidong Guo, Yu Xu, Bo Du, Varun Jampani, and Ming-Hsuan Yang. Unigs: Unified representation for image generation and segmentation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6305-6311 2024.
[62] Muzhi Zhu, Hengtao Li, Hao Chen, Chengxiang Fan, Weian Mao, Chenchen Jing, Yifan Liu, and Chunhua Shen. Segprompt: Boosting open-world segmentation via category-level prompt learning. In Int. Conf. Comput. Vis., 2023.
[63] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang. Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. Advances in Neural Information Processing Systems, 36, 2024.
A Appendix
A.1 Discussion
Broader Impacts We do not foresee any obvious undesirable ethical or social impacts now.
Limitations Our method, as the first diffusion- FSS model, proposes a simple and intuitive design, which maximizes the retention of the generative framework of LDM. There is still lot of room for improvement in performance (especially the n-shot setting), including more sophisticated model design and more optimized training strategies. We hope that our method can serve diffusion-basse FSS baseline to inspire more researchers to invest in this field.
On the other hand, we believe that our method is not limited to FSS. Our framework has the potential to unify few-shot segmentation and open vocabulary segmentation by leveraging prompts from different modalities, some work (61-63) has already proven the possibility.
14

---

A.2 More details on generation process
In the above section we have discussed the generation process of DiffewS. addition the final choice of O12M, we also tried MN2M and MI2M. Here we detail the training objectives of these three generation processes.
O12M We directly input the image and let the UNet output the mask. This process can be described as:
L02M = E(, Time Bmy |lzmq - vi (Ze, Zy. Zmo)
(9)
MN2M We add noise query mask Zmns 2mg
(1)
VāzZmn + vI aee, and during inference
use Zing as the mask prediction. The supervised form is as follows:
|imq 10 (2/10g Zy Zgg 2mss t) 1)11]
LMNMM Eeiz. Rms D,,.)),,,e((,,)eel.
(10)
MI2M We add image(as noise) to the query mask Zmg Zing = Zarzmg + V Crlly. form is as follows:
The supervised
(2(0) Imma r (2mq Bor Zup zmes
MI2M = Eig,, Shy Bmme R
(11)
A.3 Cross-attention tokenized interaction
the Section 4.2 we only discussed how to inject information from support mask based on Self-attention kv fusion method. Here we discuss how to inject information from the support mask based on the Tokenized Interaction Cross-Attention method. There are also the following four ways.
Concatenation We convert the support mask M, into RGB image, encode /, and M, into token sequences using CLIP image encoder respectively, concatenate them on the sequence, and finally use them the input of cross-attention.
b. Multiplication We can directly multiply M, on the image I, to form the image I I, and finally encode I into a token sequence using CLIP image encoder as the input of cross- attention.
c. Addition We can also directly add M, to the image I, to form the image I 0.5I, 0.5M,. Similarly, encode I into token sequence using CLIP image encoder as the input of cross-attention.
d. Attention Mask We can use M. as an attention mask to control self-attention, so that only K2 in the masked can be accessed by Q,-
A.4 Post processing
The original prediction of the model is an RGB three-channel image. We first average the channel dimension obtain single-channel M, [0,1] Then tried thresholding methods, absolute threshold T, and relative threshold Ty, The absolute threshold is a fixed value, and the final binary mask M,, can be represented as:
M, = {
1. if M, Ta
(12)
0, otherwise
Using relative threshold, we have:
{
if M, > t, max(M.)
M, =
(13)
0. otherwise
Our experiments (see Table 3) have shown that the relative threshold method achieved better results on COCO-20' [52] fold0. The optimal 7, is 0.25.
15

---

Table 3 Comparison of different thresholding methods
0.2
0.25
0.3
0.35
0,4
mloU
47.56
47.69
47.48
47.4
47.11
To
0.1
0.15
0.2
0.25
0.3
mloU
46.64
47.21
46.91
46.53
46
Table 4 Comparison of different Multiplication methods
Multiplication
mloU
latent RGB
32.14
33.12
A.5 More ablation studies
Multiplication We found in the experiment that Multiplication be directly applied to RGB images, and another choice is to apply to the latent space.
As shown in the Multiplication method directly applied to RGB images achieved better results. However, the overall disparity is not significant.
Self-Attention fusion In previous sections, we mentioned that we use a KV fusion strategy. An alternative to QKV fusion strategy, in which also concatenate Q and Q, form Q2s Q2, Q+).
This strategies means the support image can also access the query image information. As shown in
Table 5 Comparison of different Self-Attention fusion strategies
strategy
mloU
KV fusion
46.64
QKV fusion
46.61
the Table 5 KV fusion is slightly better than QKV fusion, and KV fusion has lower computational complexity, which can effectively reduce memory usage and inference time. Therefore, we choose KV fusion as our default strategy.
A.6 Other visualization
To better explore the capabilities of DiffewS, we visualize its performance COCO-20' 12 LVIS-92 [22] and several cases from Internet. Figure & shows the remarkable results of DiffewS COCO-20' Figure demonstrates the impressive generalization capabilities of DiffewS. For some categories not present in the training set, such as apron and violin, DiffewS able to perform accurate segmentation. In addition, DiffewS is demonstrated effective results in some cross-style segmentation and small object segmentation cases. For abstract concepts, such as Western dragons and Chinese dragons, DiffewS links them together to achieve accurate results. We speculate that the impressive generalization ability of DiffewS stems from its effective utilization of prior knowledge from the diffusion model. As shown Figure 77] DiffewS also fails segment some challenging cases. When there is significant appearance disparity between the reference image and the target image (Appearance disparity), DiffewS may encounter segmentation errors. Additionally, if there are other objects with similar appearances in the target image (Look-alike interference) or if the objects the image are severely occluded (Occlusion interference), DiffewS struggles to produce accurate results.
16

---

Reference
Prediction
Reference
Prediction
Reference
Prediction
Reference
Prediction
Figure 6 Visualization of one-shot semantic segmentation on various Internet cases. The blue color denotes the support mask while the red represents the query mask. DiffewS also performs impressively on cases with cross-styles and significant appearance differences, well abstract concepts has never encountered before.
Appearance disparity
Look-alike interference
Occlusion Inter infemfeencc
Reference
GT
Prediction
Reference
GT
Prediction
Reference
GT
Prediction
Figure 7 Three types of failed cases in one-shot semantic segmentation LVIS-92' and COCO-20'
A.7
N-shot studies
As mentioned in Section 4.5 conduct Inference Time Improvement COCO and PASCAL,
Training Time Improvement on COCO fold-0, see Table and Table 7
Table 6 Performance of DiffewS with Inference Time Improvement
1-shot
5-shot
10-shot
COCO
Diffews (ori)
71.3
72.2
70.1
Diffews (sample)
71.3
74.7
73.4
PASCAL
Diffews (ori)
88.3
87.8
87.2
Diffews (sample)
88.3
89.4
89.6
17

---

Reference
GT
Prediction
Reference
GT
Prediction
Reference
GT
Prediction
Figure Qualitative results of one-shot semantic segmentation on COCO-20'. blue color denotes support mask while the red represents the query mask. DiffewS has an impressive performance on COCO-20',
Table 7 Performance of DiffewS with Training Time Improvement
1-shot
5-shot
10-shot
Diffews (ori, train shot)
47.7
52.0
49.1
Diffews (train 1-5 shot)
46.4
57.6
55.9
Diffews (train 1-7 shot)
47.1
57.3
58.7
18

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
In the abstract and the end of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be analyzed.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
the paper does not include theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The dataset, model, and training procedures are clearly described in this paper and we will release code upon acceptance.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [No]
Justification: Our work is based on the open-source source code. The dataset, model, and training procedures are clearly described in this paper and we will release our code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cmm cc/
public/guides httple/eeeooeeeeeeeesssssooooona
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https wetgguddddeeeuuubbueeee ssiniioiccy for more details.
//nips.co
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We clearly described the training and test details in the setup section and in the training details of appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Like the previous works we follow, we do not report error bars. The overhead of retraining with different random seeds is significant. If computational resources permit, we will supplement this in the future.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In training details of appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We follow the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, we discuss these in the discussion.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the thing in this paper credited and are the license and terms use explicitly mentioned and properly respected
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
We do not release assets
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "o52PjZhJd3", "reviewer": "Reviewer_WLLp", "summary": "This paper introduces DiffewS, a novel Diffusion-based generalist model designed for few-shot segmentation. It systematically examines four key components involved in applying Diffusion Models to Few-shot Semantic Segmentation. For each component, the work proposes several viable solutions, which are validated through extensive experiments. Notably, the work introduces Key-Value Fusion Self-Attention (FSA) to facilitate the interaction between the query and support images.", "questions": "I am highly intrigued by the possibility of whether DiffewS can perform open-vocabulary segmentation by substituting the support mask with text.", "limitations": "The limitation is discussed in the paper supplementary material.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The idea is clear and readily comprehensible\n2. The writing is of high quality and the structure is coherent.\n3. The experiments are adequate and demonstrate the eff ectiveness of the framework.", "weaknesses": "The method of incorporating support masks, adapted from Marigold, may have limitations in a few-shot setting. The experiments detailed in Table 1 indicate that DiffewS achieves only marginal improvements in the COCO few-shot scenario compared to the one-shot scenario and even exhibits a decline in performance on the PASCAL dataset. Further evaluation of DiffewS’s capabilities with additional support samples, such as through experiments in a ten-shot setting, would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for acknowledging our motivation and approach. The questions and weaknesses you've pointed out are incredibly helpful to us, and we will do our utmost to address them below.\n\n**Weaknesses: N-shot setting**\n\nThanks again for pointing out this issue. As we mentioned in the Experiment and Limitation sections of our paper, our current work primarily focuses on unleashing the potential of Diffusion Models in Few-shot Semantic Segmentation. Consequently, our experiments and optimizations are mainly concentrated on the One-shot setting. Since the model receives only one support image at a time during the training phase, using 5-shot or 10-shot during the inference phase can indeed cause inconsistencies. The model might not learn how to effectively utilize multiple support images during training, which could result in interference from lower-quality samples. However, we fully agree with your point that we should further explore the performance of DiffewS with additional support samples. Reviewer zc26 raised a similar request, asking, \"can this be mitigated by making adjustments during training?\"\n\nTherefore, we have tried two approaches to address the N-shot setting issue.\n\n**Inference Time Improvement**\n\nThe first approach is an improvement in the inference phase. Considering the transition from 1-shot to n-shot, where additional support samples' Keys and Values are concatenated (see Appendix A.6), this leads to a significantly larger number of Keys and Values for the self-attention layer to process during inference compared to the training phase. An intuitive solution is to randomly sample the Keys and Values of the support samples at each layer during inference, ensuring the number of samples matches the number of Keys and Values during training. We found that this approach indeed improves the model's performance in 5-shot and 10-shot settings. The experimental results are shown in the table below.\n\n\n\nTable: Performance of DiffewS with Inference Time Improvement\n|                     | 1-shot | 5-shot | 10-shot |\n|---------------------|--------|--------|---------|\n| **COCO**            |        |        |         |\n| Diffews (ori)       | 71.3   | 72.2   | 70.1    |\n| Diffews (sample)    | 71.3   | 74.7   | 73.4    |\n| **PASCAL**          |        |        |         |\n| Diffews (ori)       | 88.3   | 87.8   | 87.2 |\n| Diffews (sample)    | 88.3   | 89.4   | 89.6 |\n\n\nFrom the table, we can see that following the original DiffewS inference method, the 10-shot performance is even lower than the 1-shot result. This might be due to the reasons we analyzed earlier. However, after adopting the random sampling strategy during inference, there is a significant improvement in the model's performance for both 5-shot and 10-shot scenarios\n\n**Training Time Improvement**\n\nAnother more straightforward idea is to introduce multiple support samples during the training phase. This way, the model can learn how to select among multiple support images during training. To save computational resources, we conducted experiments under the ablation study setting on the COCO fold0. The experimental results are shown in the table below.\n\nTable: Performance of DiffewS with Training Time Improvement\n|                           | 1-shot | 5-shot | 10-shot |\n|---------------------------|--------|--------|---------|\n| Diffews (ori, train 1 shot)| 47.7   | 52.0   | 49.1    |\n| Diffews (train 1-5 shot)   | 46.4   | 57.6   | 55.9    |\n| Diffews (train 1-7 shot)   | 47.1   | 57.3   | 58.7    |\n\n(train 1-5 shot) indicates that we randomly select 1 to 5 support samples as input during a single training iteration. From the table, we can see that when multiple support samples are introduced during the training phase, the model's performance significantly improves in both 5-shot and 10-shot scenarios. Additionally, as the number of support samples increases during training, the model's performance in the 10-shot scenario gradually improves. We also found that when multiple support samples are introduced during the training phase, the 1-shot performance may decrease. This could be due to the inconsistency between training and inference.\n\nIn summary, both approaches effectively alleviate the issues associated with the N-shot setting, demonstrating the potential of DiffewS in Few-shot Semantic Segmentation. We also hope that our work can provide more inspiration for researchers in the fields of diffusion-based models and Few-shot Semantic Segmentation.\n\n**Questions:open-vocabulary segmentation**\n\nWe believe it is feasible; however, it would be more appropriate to inject text information through cross-attention, as discussed in (Sec 4.1 Tokenized Interaction Cross-Attention). A similar idea has already been validated in the Referring Segmentation task (UniGS [1]). We will include a discussion on this in the Related Work section. Our work focuses more on how to effectively achieve interaction between the features of two images, which is why we ultimately chose the KV Fusion Self-Attention approach. However, these two methods are not mutually exclusive and can coexist. Nevertheless, to achieve open-vocabulary segmentation, more segmentation datasets with richly annotated text are needed for training, as merely using a few-shot dataset is insufficient. Therefore, we have not conducted experimental verification.\n\n[1] Qi L, Yang L, Guo W, et al. Unigs: Unified representation for image generation and segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 6305-6315.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UY8RTdEWLf", "reviewer": "Reviewer_uveh", "summary": "This paper explores the potential of diffusion model in Few-Shot Segmentation (FSS) tasks. For achieve better performance, it examines four critical elements of applying the diffusion model to FSS. Building on this research, the paper introduces the DiffewS framework, with experimental results validating its effectiveness. The authors assert that this is the first diffusion-based model specifically designed for Few-shot Semantic Segmentation.", "questions": "1. Did the pretrained Stable Diffusion (SD) weights utilized in the model training? How much parameters were updated? \n2. What are the experimental settings of the comparing results in Fig2/3/4?\n3. In Section 4.4, this paper chose OI2M as the generation pipeline. Since it is a one-step process, what is the timestep setting in this pipeline during training? And did any accelerated generation schedule (such as DDIM) is utilized?\n4. Did the few-shot support images being processed as the same as one-shot support image in support image encoding branch? What is the difference of model settings between using few-shot support images and one-shot support image? How about the comparison of computational cost?\n5. In Section 5.2, the “In-context” setting means “these specialist models are also trained on the test categories from COCO and PASCAL VOC”. Is it a general experimental dataset setting? And is it a fair comparison for the methods in Table 1?", "limitations": "No", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This article comprehensively explores four key aspects of adapting the Diffusion Model to FSS tasks: the interaction between query and support images, injection of support mask information, supervision from query masks, and exploration of the generation process. Multiple implementation approaches regarding the above four aspects are studied and compared through experiments. And the relevant experimental results, presented in figures 2, 3, and 4, are convincing.", "weaknesses": "The weakness of this article is as follows:\n\n1. Why should the diffusion model be applied to FSS tasks? What advantages does DM offer over Multi-modal fundamental models (like SAM or LLaVA) for FSS tasks? This article's research motivation requires stronger support through theoretical explanations or experimental evidence.\n2. The default generation process (OI2M) transforms \"gradually approaching ground truth in denoising\" to \"single-step prediction in segmentation\", yielding optimal performance. Does this suggest that DM's multi-step denoising process is unsuitable for FSS? Does this paper not unleash the potential of DM, instead transforming it into a segmentation model? Section 4.4 should include more theoretical explanations.\n3. The comparison of inference speed and training cost is important and should be supplemented to illustrate the advantages of DiffewS.\n4. Other studies that apply the Diffusion Model to Few-Shot Segmentation (e.g., DifFSS[1], Ref LDM-Seg[2]) need to be compared. \n\n[1] DifFSS: Diffusion Model for Few-Shot Semantic Segmentation. Github: https://github.com/donglongzi/DifFSS-BAM/tree/main\n\n[2] Explore In-Context Segmentation via Latent Diffusion Models. Project Page: https://wang-chaoyang.github.io/project/refldmseg/", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your very careful and detailed review. We are delighted that you found our work to be comprehensive and convincing. The questions and weaknesses you've pointed out are incredibly helpful to us, and we will do our utmost to address them below.\n\n**W1: Why should the diffusion model be applied to FSS tasks?**\nIn L43-L53, we briefly introduce the motivation for applying the diffusion model to FSS tasks. We will make this clearer in the revised version.  The  pretrained diffusion model exhibits significant potential in fine-grained pixel prediction ability[1,2] \nand semantic correspondence ability [3-5]. Both of these capabilities are essential for Few-Shot Segmentation (FSS) tasks, making diffusion models particularly suitable for this application.\n\nFor example, SAM has strong fine-grained pixel prediction capabilities but lacks semantic abilities, leading to a series of works attempting to address this issue[6,7]. Therefore, it is difficult to use SAM in FSS. For example, PerSAM[8] is based solely on SAM's capabilities but does not perform well in FSS.\n\nOn the other hand, LLaVA has strong semantic abilities but lacks fine-grained pixel prediction capabilities due to its reliance on only image-text pairs for pretraining(lacks fine-grained supervision). Therefore, LISA[7] needs to rely on SAM to achieve segmentation prediction.\n\n\n\n**W2: Multi-step denoising process**\nWe believe that the supervised objective of the multi-step denoising process is not suitable for FSS, but this does not mean that single-step prediction fails to unleash the potential of the diffusion model (DM). The difference between OI2M and MI2M/MN2M lies only in their training objectives (see Appendix A.2). Compared to OI2M, MI2M and MN2M use noisy target masks as input, which could potentially lead to information leakage in segmentation tasks, especially when the noise level is low, thus hindering the model's learning. The experimental results in Figure 3 also confirm this, showing that setting $\\beta_{2}$ is more effective than $\\beta_{1}$. Intuitively, $\\beta_{2}$ adds more noise at smaller time steps compared to $\\beta_{1}$, and OI2M can be understood as an extreme case where the input contains no information about the target mask at all (or the target mask has become pure noise). We will provide more theoretical explanations in Section 4.4.\n\n**W3: Inference speed and training cost**\nThank you for your suggestions. We can compare the advantages of DiffewS in two aspects:\n\n1.Compared to traditional FSS methods, DiffewS converges faster and has lower training overhead. For instance, taking DCAMA[9] as an example, both our model and DCAMA were trained on 4 V100 GPUs. The comparison of training time is as follows:\n\nTable: Training Time Comparison\n| Method  | Training Time |\n|---------|---------------|\n| DCAMA   | 36h           |\n| DiffewS | 6h            |\n\n2. Compared to other generative methods based on multi-step denoising, DiffewS has a faster inference speed. For instance, considering MI2M and MN2M mentioned in this paper, the inference speed comparison is as follows:\n\nTable: Inference Speed Comparison\n| Method  | Forward times  |\n|---------|-----------------|\n| OI2M    | x1              |\n| MI2M    | x10             |\n| MN2M    | x10             |\n\nIn our implementation, MI2M and MN2M use DDIM sampling, requiring the model to forward 10 times, while OI2M only needs to forward once. Thus, OI2M should be 10 times faster than MI2M/MN2M.\"\n\n**W4:Other studies**\nThank you for your valuable suggestions. We will include discussions of DifFSS and Ref LDM-Seg in the Related Work section, as well as comparisons with our work. DifFSS uses generated images as auxiliary support images, which can improve the performance of existing FSS methods to some extent. This approach enhances FSS performance from the data level, making it orthogonal to our method. Ref LDM-Seg is a contemporaneous work to ours. Unlike DiffewS, which utilizes self-attention to achieve interaction between the query and support images, Ref LDM-Seg uses cross-attention for this purpose. Additionally, Ref LDM-Seg introduces multiple linear projection layers, increasing the number of parameters in the diffusion model, which might also disrupt the original priors of the diffusion model. Our work provides a more comprehensive and systematic analysis of applying diffusion models to Few-shot Semantic Segmentation tasks. Considering that Ref LDM-Seg has not been open-sourced, and the training data and experimental setups cannot be strictly aligned, we currently cannot conduct a direct and fair experimental comparison.\n\n**References**\n\n[1]Ke B, Obukhov A, Huang S, et al. Repurposing diffusion-based image generators for monocular depth estimation. CVPR2024\n\n[2]Lee H Y, Tseng H Y, Yang M H. Exploiting Diffusion Prior for Generalizable Dense Prediction. CVPR2024\n\n[3]Tang, Luming, et al. Emergent correspondence from image diffusion. NeurIPS2023\n\n[4]Luo, Grace, et al. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. NeurIPS2023\n\n[5]Zhang, Junyi, et al. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. NeurIPS2023\n\n[6]Li F, Zhang H, Sun P, et al. Semantic-sam: Segment and recognize anything at any granularity. ECCV 2024\n\n[7]Lai X, Tian Z, Chen Y, et al. Lisa: Reasoning segmentation via large language model. CVPR 2024\n\n[8]Zhang R, Jiang Z, Guo Z, et al. Personalize segment anything model with one shot. ICLR2024\n\n[9]Shi, Xinyu, et al. Dense cross-query-and-support attention weighted mask aggregation for few-shot segmentation. European Conference on Computer Vision.  ECCV2022\n\n\n**Due to space constraints, we will respond to your valuable questions in the global response.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oOV9x0GQMI", "reviewer": "Reviewer_zc26", "summary": "This paper explores the potential of the Latent Diffusion Model for Few-shot Semantic Segmentation. The authors study four crucial elements of applying the Diffusion Model to Few-shot Semantic Segmentation and propose several reasonable solutions for each aspect. \nBased on their observations, the authors establish the DiffewS framework, which maximally retains the generative framework and effectively utilizes the pre-training prior. \nExperimental results demonstrate that the proposed method outperforms the previous SOTA models in multiple settings.", "questions": "I've noticed that the current method doesn't perform well enough on 5-shot, can this be mitigated by making adjustments during training?", "limitations": "NaN", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The topic is interesting and the proposed method is novel. How to better utilize the diffusion model for perception tasks is currently a promising direction. \nAnd this paper takes a first step to apply diffusion model to few-shot semantic segmentation.\n- The writing is clear and easy to follow.   The author presents clear and logical thinking, starting from four key elements of implementing FSS, proposing a series of reasonable solutions, and validating the effectiveness of these solutions through experiments. The KV-fusion strategy, the supervisory form of the query mask, or the single-step generation strategy, all of these provide certain insights for subsequent research on FSS and visual perception.\n- The Diffews method has demonstrated strong performance in in-context settings, especially in in-domain scenarios. It also shows excellent performance in strict few-shot settings and converges faster compared to traditional methods.", "weaknesses": "- Some places lack proper references. For example, in L110, \"v-prediction,\" we typically cite Tim Salimans and Jonathan Ho's work: \"Progressive distillation for fast sampling of diffusion models\" (ICLR, 2022).\nIn L241, I noticed that the appendix contains an analysis of the training objectives for different generation processes, and a reference should be added here.\n\n- In Section 4.4, L257, the variances $\\beta_1$ and $\\beta_2$  are not clearly defined. Since $\\beta_t$ in L106 represents the variance at time t, it would be better to use $\\beta^1$ and $\\beta^2$  to denote these two different settings here. Additionally, the meaning of the two values within the tuple is not explained. In my view,  $\\beta^1 = (\\beta^{start},\\beta^{end})$ should represent the initial and final values of the DDIM scheduler's $\\beta$ .\nFurthermore, it is necessary to explain the intuitive differences between these two settings.\n\n- Other typos:\nL273,  \"we tested\" -> \"we test\"\nL151, \"linear projection layer. .\" ->\"linear projection layer\"\nL279, \"function as\"-> \"functions as\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate your acknowledgment of our motivation and approach. We will try our best to address the questions and weaknesses you have raised.\n\n**W1: Lack of proper references**\nThank you for your feedback. We will add appropriate references in the next version.\n\n**W2: Definition of $\\beta_{1}$ and $\\beta_{2}$**\nYes, you are correct. We will clearly define $\\beta^{1}$ and $\\beta^{2}$ in Section 4.4 and explain the intuitive meanings of the two values. $\\beta^{1}=(\\beta_{start},\\beta_{end})$, where $\\beta_{start}$ and $\\beta_{end}$ respectively represent the initial and final values of $\\beta$ in the DDIM scheduler. We also mentioned this in our discussion with Reviewer uveh. Intuitively, $\\beta^{2}$ adds more noise at smaller time steps compared to $\\beta^{1}$. Experimental results show that adding more noise at smaller time steps improves the model's performance, which is why we ultimately chose OI2M. .\n\n**W3: Other typos**\nThank you once again. We will fix these errors in the next version.\n\n**Questions: 5-shot performance**\n\nWe are grateful for your question. We have also mentioned this issue in our response to Reviewer zc26. We have tried two approaches to address the N-shot setting issue.\n\n**Inference Time Improvement**\n\nThe first approach is an improvement in the inference phase. Considering the transition from 1-shot to n-shot, where additional support samples' Keys and Values are concatenated (see Appendix A.6), this leads to a significantly larger number of Keys and Values for the self-attention layer to process during inference compared to the training phase. An intuitive solution is to randomly sample the Keys and Values of the support samples at each layer during inference, ensuring the number of samples matches the number of Keys and Values during training. We found that this approach indeed improves the model's performance in 5-shot and 10-shot settings. The experimental results are shown in the table below.\n\nTable: Performance of DiffewS with Inference Time Improvement\n|                     | 1-shot | 5-shot | 10-shot |\n|---------------------|--------|--------|---------|\n| **COCO**            |        |        |         |\n| Diffews (ori)       | 71.3   | 72.2   | 70.1    |\n| Diffews (sample)    | 71.3   | 74.7   | 73.4    |\n| **PASCAL**          |        |        |         |\n| Diffews (ori)       | 88.3   | 87.8   | 87.2 |\n| Diffews (sample)    | 88.3   | 89.4   | 89.6 |\n\nAnother more straightforward idea is to introduce multiple support samples during the training phase. This way, the model can learn how to select among multiple support images during training. To save computational resources, we conducted experiments under the ablation study setting on the COCO fold0. The experimental results are shown in the table below.\n\nTable: Performance of DiffewS with Training Time Improvement\n|                           | 1-shot | 5-shot | 10-shot |\n|---------------------------|--------|--------|---------|\n| Diffews (ori, train 1 shot)| 47.7   | 52.0   | 49.1    |\n| Diffews (train 1-5 shot)   | 46.4   | 57.6   | 55.9    |\n| Diffews (train 1-7 shot)   | 47.1   | 57.3   | 58.7    |\n\n(train 1-5 shot) indicates that we randomly select 1 to 5 support samples as input during a single training iteration. From the table, we can see that when multiple support samples are introduced during the training phase, the model's performance significantly improves in both 5-shot and 10-shot scenarios. Additionally, as the number of support samples increases during training, the model's performance in the 10-shot scenario gradually improves. We also found that when multiple support samples are introduced during the training phase, the 1-shot performance may decrease. This could be due to the inconsistency between training and inference.\n\nIn summary, both approaches effectively alleviate the issues associated with the N-shot setting, demonstrating the potential of DiffewS in Few-shot Semantic Segmentation. We also hope that our work can provide some inspiration for researchers in the fields of diffusion-based models and Few-shot Semantic Segmentation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We would like to extend our heartfelt thanks to all of you for taking the time and demonstrating professionalism in thoroughly evaluating our work. Your constructive feedback has been immensely helpful in further improving the quality of our work and refining our research. We are also greatly encouraged by your recognition of our efforts.

We will carefully address the issues and suggestions raised by the reviewers and will make further revisions and improvements to our paper. Below each Official Review, we have provided responses to the questions and suggestions made by the reviewers, and we hope these responses adequately address the issues and concerns raised.

Due to the character limit for each review's rebuttal, we have moved some of the responses to Reviewer **uveh**'s questions here.

**Q1: Pretrained Stable Diffusion (SD) weights**
Yes, we utilized the pretrained Stable Diffusion (SD) weights in our model training. We keep the VAE fixed and only fine-tune the Unet. 

**Q2: Experimental settings of Fig2/3/4**
The experimental settings for the results in Fig2/3/4 are our ablation study settings. We will provide more detailed experimental settings in the revised version. Specifically, We use one 4090 GPU to train the model with a batch size of 4, other settings are strictly aligned with the Table 2 settings. There are small differences between Fig3 and Fig4(46.64 vs 47.7).
This is because we did not use an optimized threshold in Fig. 3. In Fig. 3, we used different forms of mask supervision, each maybe corresponding to a different optimal threshold. See Appendix A.4 for more details.

**Q3: OI2M timestep setting**
We set the timestep to 1 in the OI2M pipeline during training. We don't use any ccelerated generation schedule like DDIM, because it is a one-step process. We also add an additional ablation study to compare different timestep settings(the input time embbeding of unet) during training, the results are shown in Table below.

Tabel: Performance of DiffewS with different timestep settings
| Timestep | mIoU  |
|----------|-------|
| 1        | 47.7  |
| 10       | 47.8  |
| 50       | 47.2  |
| 100      | 43.14 |

It can be observed that the performance is better with smaller timesteps. This is because the UNet in the original diffusion model receives noiseless real images as input at smaller timesteps, and our current model also receives noiseless real images as input.

**Q4: Few-shot support images**
Yes, the few-shot support images are processed in the same way as the one-shot support image in the support image encoding branch. We use the same model weights for both one-shot and few-shot settings. The only difference is that we concatenate the Key and Value of the support images during self-attention(see Appendix A.6). 
Since our model can process multiple support images in batches, the inference time doesn't increase significantly. Specifically, when evaluating on COCO fold0, the inference time for 5-shot is approximately twice that of 1-shot, and the GPU memory usage is also about twice that of 1-shot.


**Q5: In-context setting**
Yes, the "In-context" setting is a general experimental dataset setting. 
This setting was first proposed by SegGPT[A] and has been adopted by works such as Matcher[B] and PerSAM[C]. The comparison is fair, as these specialist models were also trained on COCO and PASCAL VOC test categories. The specific results can be found in Table 1 of the SegGPT paper. We will provide a more detailed description in the experimental section.

[A] SegGPT: Segmenting Everything In Context ICCV2023

[B] Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching ICLR2024

[C] Personalize Segment Anything Model with One Shot ICLR2024

Meta-review: The paper introduces a latent diffusion model (DM)-based method for few-shot segmentation (FSS). It develops several effective solutions from four key aspects of applying DM to FSS and achieves strong performance on several benchmarks. 

The paper received mostly positive reviews. The three reviewers considered the proposed approach novel and effective (WLLP, zc26), the paper well-written (WLLP, zc26), the study of its solutions comprehensive (WLLP, uveh) and the experimental results strong/convincing (WLLP, uveh, zc26). However, they initially also raised several important concerns regarding its mixed performance under multi-shot settings (WLLP, uveh, zc26), insufficient motivation (uveh), and the lack of clarity in technical details and settings (uveh, zc26). The author's rebuttal provided thorough clarifications on its motivation and technical details, and additional results on 5 and 10-shot settings, which largely addressed the reviewers' concerns. During the rebuttal discussion, the three reviewers reached a consensus with positive recommendations on this work.  

After considering the paper, the accompanying reviews, and the rebuttal discussion, despite some limitations in the multi-shot setting, the AC concurs with the reviewers regarding the substantive contributions of this work and therefore recommends it for acceptance. The author should take into account the rebuttal and the reviewers' feedback, especially regarding the refined 5 and 10-shot results, when preparing the final version.
Decision: Accept (poster)