Better to Teach than to Give: Domain Generalized Semantic Segmentation via Agent Queries with Diffusion Model Guidance
Authors: Fan Li
Xuan Wang
Min Qi
Zhaoxiang Zhang
Yuelei Xu
Venue: ICML 2025 spotlightposter (2025)
Keywords: semantic segmentation
domain generalization
diffusion model
Primary area: applications->computer_vision

Abstract: Domain Generalized Semantic Segmentation (DGSS) trains a model on a labeled source domain to generalize to unseen target domains with consistent contextual distribution and varying visual appearance.
Most existing methods rely on domain randomization or data generation but struggle to capture the underlying scene distribution, resulting in the loss of useful semantic information. 
Inspired by the diffusion model's capability to generate diverse variations within a given scene context, we consider harnessing its rich prior knowledge of scene distribution to tackle the challenging DGSS task.
In this paper, we propose a novel agent \textbf{Query}-driven learning framework based on \textbf{Diff}usion model guidance for DGSS, named QueryDiff. 
Our recipe comprises three key ingredients: (1) generating agent queries from segmentation features to aggregate semantic information about instances within the scene; 
(2) learning the inherent semantic distribution of the scene through agent queries guided by diffusion features; 
(3) refining segmentation features using optimized agent queries for robust mask predictions.
Extensive experiments across various settings demonstrate that our method significantly outperforms previous state-of-the-art methods. 
Notably, it enhances the model's ability to generalize effectively to extreme domains, such as cubist art styles. Code is available at https://github.com/FanLiHub/QueryDiff.

method Better to Teach than to Give: Domain Generalized Semantic Segmentation via Agent Queries with Diffusion Model Guidance
Fan Li Xuan Wang'
Min Qi"
Zhaoxiang Zhang
Yuelei Xu
Abstract
(a) Predicted results in the extreme domains
Image
Rein
QueryDiff (Ours)
Domain Generalized Semantic Segmentation (DGSS) trains model on labeled source do- main generalize unseen target domains with consistent contextual distribution and varying vi- sual appearance. Most existing methods rely on domain randomization or data generation but struggle to capture the underlying scene distri- bution, resulting in the loss useful semantic information. Inspired by the diffusion model's s capability to generate diverse variations within given scene context, consider harnessing its rich prior knowledge of scene distribution to tackle the challenging DGSS task. In this paper. we propose a novel agent Query-driven learning framework based on Diffusion model guidance for DGSS, named QueryDiff. Our recipe com- prises three key ingredients: (I) generating agent queries from segmentation features to aggregate semantic information about instances within the scene; (2) learning the inherent semantic distri- bution of the scene through agent queries guided by diffusion features; (3) refining segmentation features using optimized agent queries for robust mask predictions. Extensive experiments across various settings demonstrate that our method sig- nificantly outperforms previous state-of-thes thet--rrrt methods. Notably, it enhances the model's 's ability to generalize effectively to extreme domains, such as cubist art styles. Code available at https: //gi /ggithub com/F anLi Hub/ QueryDiff.
(b) Architecture comparison
(i) Diffusion models as data generators
Perception
Seg.
Model
Diffusion
Giving More
(ii) Our method
Pereeption Model
Seg.
Agont
Queries
Diflusiin
DCL
Teaching Distribution
Genented Data
Different
Figure 1. (a) The segmentation results predicted and previous SOTA approach in extreme domains, including cubist and pest-impressionist styles generated by ChatGPT. (b) Pre- vious methods use the diffusion model as a data generator, which struggles cover all variations the target domain, resulting limited performance. Our method employs agent queries to learn scene distribution knowledge from the diffusion model, capitaliz- ing the inherent consistency of this distribution across domains to improve segmentation model generalization.
1. Introduction
Semantic segmentation, a fundamental task in computer vision, has seen significant progress in recent years (Long al., 2015; Chen et al., 2017; Fu al., 2019; Zhong al.,
2020; Cheng et al., 2022). However, it experiences consider- able performance degradation when there is a gap between the training data and the test data. This has sparked inter- est in Domain Generalized Semantic Segmentation (DGSS) (Zhao et al., 2022; Jia et al., 2023; Li et al., 2023). DGSS aims to train a model to learn generalizable features from the source domain, enabling it to perform well on a va- ricty of unseen target domains with consistent contextual distribution but significantly varying visual appearances.
Northwester Polytechnical University, China. Correspon- dence to: Yuelei Xu sruyyuleiinnwuu.cuu...
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Better to Teach than to Give: Domain Generalized Semantic Segmentation via Agent Queries with Diffusion Model Guidance
The majority of DGSS methods are built upon Domain Randomization (DR) (Wang et al., 2020; Zhao al., 2022; Zhong et al., 2022; Jiang et al., 2023), which diversifies source domain images through photometric and geomet- ric transformations. Although somewhat effective, such transformations are constrained by the structure and content of the source data, failing to create new contextual depen- dencies between instances within the scene. This results in limited diversity that fails to encompass broader scene distributions, i.e., the spatial arrangement and contextual de- pendencies implicit in the inter-instance relationships (e.g.. cars on roads, buildings aligned along streets) that enable the model to move beyond isolated instances and grasp uni- fied objective laws across different scenario domains. As a result, the model remains robust only to specific patterns of variation, resulting in poor generalization performance.
and even detrimental for perception tasks.
To address all these challenges, in this work, we propose QueryDiff, an agent queries-driven framework guided by dif- fusion models for DGSS, which utilizes the powerful scene distribution priors embedded in diffusion models to enhance semantic segmentation generalization. Specifically, for the first challenge, QueryDiff employs learnable queries to in- teract with multi-scale image features from the segmentation backbone, aggregating instance-level semantic information to form agent queries that serve as the interface throughout the pipeline, instead of directly relying on diffusion features. Next, guided by the diffusion features, the agent queries learn to construct the semantic distribution of the scene. To address the second challenge, we propose diffusion con- sistency loss (DCL) that aims to eliminate intricate visual appearance details from the diffusion features, allowing the agent queries to focus on comprehending scene distribution for generalized semantic representations. Ultimately, we use these agent queries to refine the features in the segmentation decoder, integrating the prior knowledge of scene distribu- tion from the diffusion features into the segmentation model, and thereby producing more robust predictions. Extensive experiments across various DGSS settings demonstrate that our method surpasses existing approaches, achieving state- th-hheart performance. Moreover, as shown in Figure (a), our method generalizes effectively to extreme domains, such as cubist and impressionist paintings. The contributions are summarized as follows:
Recently, diffusion models have opened up new avenues for DGSS with remarkable capabilities in capturing com- plex scene distributions to generate high-quality, realistic samples (Dhariwal & Nichol, 2021; Ho et al., 2020; Rom- bach et al., 2022). Building on this, several studies have utilized the scene distribution priors of diffusion models to expand the semantic distribution of source domain data. For instance, DIDEX (Niemeijer et al., 2024) employs diffu- sion model to generate pseudo-target domain for domain extension. DatasetDM (Wu et al., 2023a) presents a generic dataset generation model capable of producing diverse im- ages and corresponding annotations using diffusion models. DGInStyle (Jia et al., 2025) crafts diverse task- sask-specific im- ages by sampling the rich prior of a pretrained diffusion model while maintaining strict adherence to semantic layout conditions. While these methods have achieved notable im- provements in model performance, expanded data remains inadequate to support the almost infinite distributional varia- tions in target scenarios, as shown in Figure 1(b). Moreover, expanding the training data substantially incurs significant computational costs and time consumption for the model. Adhering to the principle that teaching man to fish is better than giving him a fish, natural curiosity has been raised: Can the powerful scene distribution priors embedded in diffusion models be directly leveraged to enhance seg- mentation generalization?
We propose a novel agent queries-driven learning framework named QueryDiff, which employs agent queries as the interface to mine the scene distributions embedded in diffusion features, thereby improving the robustness of the model with respect to domain shifts.
We propose diffusion consistency loss (DCL. avoid intricate visual details in diffusion features from inter- fering with agent queries, enabling them to better focus on learning the semantic distribution of the scene.
QueryDiff is a concise framework with high perfor- mance. It outperforms wide variety of baselines and reaches a new state-of-the-art performance on exten- sive DGSS benchmarks.
straightforward solution is to harness the diffusion model as general knowledge extractor to enhance DGSS perfor- mance by extracting diffusion features and mapping them to segmentation results. However, this comes with two new challenges: 1) The iterative sample process in diffu- sion models involves multiple steps of denoising, which is computationally expensive and consumin,,, making it inefficient for perception tasks. 2) Diffusion features en- compass not only high-level semantic information but also low-level visual appearance details (e.g., color and texture), which, though crucial for generation tasks, are unnecessary
2. Related Work
2.1. Diffuusion Model
Diffusion models (Ho et al., 2020; Dhariwal & Nichol, 2021; Rombach et al., 2022) have recently demonstrated state- of-the-art image generation quality. With the success of diffusion models in generative tasks, some pioneer works have explored the application of diffusion models to var- ious visual perception tasks (Xu et al., 2023; Tang et al.,
2

---

Better to Teach than to Give: Domain Generalized Semantic Segmentation via Agent Queries with Diffusion Model Guidance
2023: Ji et al., 2023; Lee et al., 2024; Ke al., 2024). For example, Wu et al. (2023b) exploit cross-attention of the dif- fusion model to localize class-specific regions and generate a high-resolution segmentation mask. Xu et al. (2023) intro- duce ODISE, which integrates the internal representations of pre-trained diffusion models and discriminative models to effectively perform panoptic segmentation across any cat- egory the wild. Ji et al. (2023) propose DDP, an extension of denoising diffusion process for semantic segmentation and depth estimation, transforming noise samples into de- sired predictions iteratively guided by input images. These methods rely on the diffusion model as the backbone, but its iterative sampling process, involving multiple denoising steps, incurs significant computational overhead and time consumption, rendering it inefficient for perception tasks. Thus, our method leverages agent queries as an interface to mine the underlying scene distribution embedded in dif- fusion features and refine the features of the segmentation model to improve its robustness, avoiding the direct involve- ment of diffusion features in the segmentation process.
ing the generalization capability of semantic segmentation.
2.3. Learnable Query Design
Recently, series of learnable query-based frameworks in- spired by DETR (Carion et al., 2020) have been proposed. For instance, Panoptic SegFormer (Li et al., 2022) intro- duces query decoupling strategy to prevent mutual interfer- ence between thing queries and stuff queries. MaskFormer (Cheng et al., 2021) and Mask2Former (Cheng et al., 2022) solve both semantic- and instance-Ievel segmentation tasks in a unified framework by employing object queries to ag- gregate pixels within the same semantic region. MP-Former (Zhang al., 2023) accelerates training by feeding noisy ground truth masks and learnable queries into the Trans- former decoder to reconstruct the originals. Rein (Wei et al., 2024) fine-tunes vision foundation models by incorporating a set of randomly initialized queries as learnable parame- ters. While these studies have shown that specific learnable queries improve performance and functionality in percep- tion tasks, research on learnable queries for DGSS remains unexplored. Our work aims address DGSS by designing agent queries, constructed through learnable queries, to de- velop a concise, general, and efficient framework to improve the generalization of the model.
2.2. Domain Generalized Semantic Segmentation
Domain generalized semantic segmentation (DGSS) aims to enhance model robustness against domain shifts. The current approaches fall into two categories: normaliza- tion/whitening and domain randomization. Normaliza- tion/whitening methods learn domain invariant features by mader-made modules to remove domain- -peaaiff features (Pan et al., 201 8; Choi et al., 2021; Peng et al., 2022; Xu et al., 2022), but they struggle with complex domain shifts. Domain randomization seeks to broaden the distribution of source domain by either diversifying the data style through data augmentation (Lee al., 2022; Zhao et al., 2022; Kim et al., 2023) or enriching the data content through generating new data (Benigmim et al., 2024), offering improved gen- eralization performance. Recent studies have investigated the use of visual foundation models in DGSS. Benigmim et al. (2024) integrate various foundation models, including employing the diffusion model as a data generator, leverag- ing the robust features of CLIP, and utilizing SAM to refine pseudo labels for self training.. Jia et al. (2023) propose DGInStyle, controlled pipeline for generating task-specific images with widely varying appearance data that samples the rich prior of a pre-trained diffusion model. Niemeijer et al. (2024) introduce a novel domain extension method uti- lizing a diffusion model to generate a pseudo-targe domain with diverse text prompts and then training a generalizing model by adapting to this pseudo-target domain. Despite these advancements, current approaches leverage diffusion primarily as an auxiliary tool of data generation within train- ing pipelines. In contrast, this paper introduces an agent pucries-drivens approach to fully mine domain-invarianl se- mantic information of the diffusion features, thereby enhanc-
3. Preliminary
Problem Definition. The goal of Domain Generalized Se- mantic Segmentation (DGSS) is to train a segmentation model yp = g v, where g and v denote the backbone and decoder, respectively, on a labeled source domain S, en- abling it to generalize effectively to an unseen target domain T. Both domains share the same set of K categories.
Stable Diffusion. Stable diffusion involves forward diffu- sion process that adds noise the data and a reverse process that converts the noisy samples into raw data. The forward diffusion process is defined as:
9 (z{ | 20) = N (ze Vāizo, (1-a,) I.)
(I)
where 2g = E(x) and E is pre-trained encoder that encodes image x E Hf XWxxWW> x3 into a latent representation space. Or = II: a2 = II' (1 - B.) and B. represents the pre-defined noise schedule (Ho et al., 2020). This process transforms data sample 20 to a latent noisy sample 2,, and is the diffusion step. Generally, larger t corresponds to larger noise weights. The reverse process is a denoising procedure that gradually transforms the noisy samples into raw data. A step in the reverse process can be defined as:
(2) where the mean 14g is predicted by noise predictor c0(-) and the covariance Eg is generally fixed as a predefined value.
Po (2t-1 2)) := N (z-11 | M4 (2 t) Eg (zt,
3

---

Better to Teach than to Give: Domain Generalized Semantic Segmentation via Agent Queries with Diffusion Model Guidance
(a) Agent Queries Generation
(c) Feature Refinement and Mask Prediction
Image
Backbone
Refined Features
Ground Truth
Pixel Decoder
Refinement Module
gating
Share Weight
Hierarchical Aggregating
Transformer
Seg. Loss Cov
Learnable Queries
Overlap
Decoder
Layer-wise
Queries
Mask
Mask
Progressive Merging
Decoder
Progressive Merging
-
Agent Queries
Class
Prediction
Learnable Queries
AverPool Mapping
Linear
Linear
Encoder
Pre terairaae
Add Noise
MLP
Dist. Loss
Diffusion c Model
Sup. Loss Cuve
Latent 2g
strong
(b) Diffusion-Guiin Guiided Agent Queries Optimization
Figure brief illustration of our proposed framework, First, we use learnable queries to aggregate hierarchical instance features from the segmentation backbone, progressively merging them form agent queries. Next, utilize agent queries to learn the scene distribution information embedded in the diffusion features, optimizing their semantic representations through diffusion consistency loss (DCL) that removes visual appearance information irrelevant the perceptual task from the diffusion features. Finally, we use the optimized agent queries to refine the instance features of the segmentation decoder and output the prediction mask.
4. Methodology
Aggregating Hierarchical Visual Features. We establish multiple groups of learnable queries, each corresponding to a feature layer in the segmentation backbone. For the features Seeg produced by the I-th layer of the segmentation backbone g. we first construct layer-wise queries follows:
With their powerful ability generate high-quality sam- ples, diffusion models have been successfully applied to improve segmentation generalization by expanding source domain data, drawing on their rich priors of scene distribu- tion. Guided by the principle that teaching a man to fish is better than giving him fish, a natural question arises: Can the scene distribution priors of diffusion models be directly leveraged enhance segmentation generalization? As illus- trated in Figure 2, we propose novel agent query-driver DGSS framework QueryDiff based on diffusion models, which consists of three key steps: (I) aggregating seman- tic information within a scene to generate agent queries (Section 4. 4). . (2) leveraging agent queries to mine scene dis- tribution knowledge encoded in diffusion features (Section 4.2), and (3) reinforeing scene distribution information in the segmentation features via agent queries (Section 4.3).
Blayer MLP(SSflmmax(
((mix)) feeg)
(3)
with Glayer (dayer Txe | SSIS L.}
where Rayer are layer-wise queries for 1-th feature fleg Lnit E Rex* are randomly initialized learnable queries. c represents the dimension of each feature layer, r denotes the sequence length of dinie represents the total number of feature layers in the segmentation backbone. All layers share the same MLP weights.
Progressive Merging for Unified Representation. Each layer-wise query corresponds to at least one instance of in- terest in the scene, thus there is information overlap among the layer-wise queries. To ensure that each query ultimately retrieves the features of the same instance in different layers, we merge the layer-wise queries into a whole using a pro- gressive merging strategy. Specifically, for each stage i, we employ learnable queries Lmaa map them queries Q' and the outputs of the previous stage, lstager are respectively projected into the keys K and values V
4.1. Agent Queries Generation
Considering the substantial inefficiency resulting from the computationally intensive iterative sampling process of dif- fusion models, our approach employs agent queries as the interface throughout the framework, instead of directly in- corporating diffusion features into the segmentation process. Specifically, we aggregate hierarchical visual features from the segmentation backbone and progressively merge them into a unified representation, referred agent queries.
Q' = Vinst W K' Lon W V g W
(4)
;-1 Dinie E xe Astage E -ini-Iixee 15is M

---

Better to Teach than to Give: Domain Generalized Semantic Segmentation via Agent Queries with Diffusion Model Guidance
where W W and W are linear projections. Ginit randomly initialized learnable queries, n, denotes the number of the queries produced at the i-th merge stage, M represents the total number of merging stages and Leage = Rayer L Next, we merge the smaller groups into larger ones based on the similarity matrix in the embedding space:
where weak noise added timestep lw results in diffusion features f fffw.)) with finer- meer-grained details, while strong noise at timestep t, leads to features f(ford) with coarser-grained semantic information.
The Diffusion Consistency Loss. Our goal is to enable agent queries to comprchensively understand the scene se- mantic distribution, guided by diffusion features, to cap- ture the complex contextual relationships between instances within the scenario. To achieve this, we firstly perform a dot product operation on the agent queries lagent with the weak noise diffusion features fa ((fw))) to obtain a similarity map:
exp (st)
(
I Estage = FFN
X V : s' = vd
hw j= exp (st)
(5) where Estage E TRueee are the output of the current stage, d' is a scaling factor, and FFN consists of linear mapping followed by an activation layer. h and w denote the height and width of the similarity matrix, respectively. Notably, the number of queries generated at each merge stage decreases progressively, i.e. n4+1 <ngg Next, Estge are fed to self- attention and FFN to enrich the representation of cach query and obtain the output Tsages which is the next stage of inputs. After the final stage, M, we average their outputs to obtain the final global agent queries:
5 = Softmax (fl x Tinear(gagent )7)
(8)
where Lincar is two-layer MLP with layer normalization. The dot product operation effectively associates the instance semantics of the agent queries with the scene distribution knowledge in the weak-noise diffusion features. As a re- sult, 5 inherently contains the scene distribution prior embedded the weak-noise diffusion features. However, the visual detail information inherent in the weak-noise dif- fusion features f f((=d)) inevitably propagates into ST. To mitigate the interference of these details, it is necessary to strip away the potential visual detail within Sp Con- sidering that the primary distinction between strong-noise diffusion features f(((-:)) and weak-noise diffusion features f ((tw))) lies in the low-level visual detail, while both re- main essentially identical in representing the scene semantic distribution, we leverage 5 and Gagent to reorganize the weak features, with the original strong-nois diffusion features f/ ((tnd) supervising the reorganization process:
Jogent AvgPool (gMlage) x Wa + bas Gagent R
(6) where Wa and by signify the weights and biases, respec- tively. As result, we embed semantic information from the segmentation backbone into agent queries and establish an implicit linkage between agent queries and instances within the scene. These agent queries are then guided by the diffusion features learn contextual relationships be- tween different instances, enabling more comprehensive understanding of the scene distribution.
4.2. Difiusion-Guided Agent Queries Optimization
A w f f (h, u) log fll (h, w)
Caist
>>>
The diffusion model encapsulates not only prior knowledge of scene distribution but also various visual appearances (e.g., colors and textures), which are irrelevant or even detri- mental to perceptual tasks. To guide agent queries in fo- cusing on the scene distribution, we propose a diffusion consistency loss (DCL) that decouples scene distribution knowledge from intricate visual details in diffusion features.
f2 w)
(9)
h=1
Pd
Sp 5tW x Linear
where A and W represent the height and width dimensions of the feature map f((,'), respectively. By enforcing a consistency constraint, visual details in S are effectively minimized. This is because if 5 includes excessive visual details, it would result in a significant discrepancy between the reorganized result få and the strong-noise diffusion features fl ((( The consistency loss minimizes this dis- crepancy, ensuring that during the construction of S only the semantic distribution of the scene is preserved while visual details are suppressed.
Extraction of Diffusion Features. Before that, we need to extract diffusion features. To do this, we first transform the image v to the latent space using the pre-trained encoder E to obtain the latent code 20 = E(x), then use Equation I to get the noise samples 24, and finally use the noise predictor c-(-) to perform the denoising process to get diffusion features:
((t)))) = eg ls T(e)), p(l = (Ztu: : L T(e))
Ultimately, we utilize 5 to optimize the instance semantic representation in the agent queries, resulting in updated feature representations:
(7)
where T is the text encoder, e is the empty text, 1 J < K and K represents the total number of diffusion features. Dif- ferent timesteps/ correspond to different denoising stages,
Lope = -1((r/)) x S
(10)
5

---

Better to Teach than to Give: Domain Generalized Semantic Segmentation via Agent Queries with Diffusion Model Guidance
To guide the agent queries Gagent in constructing the scene distribution, we supervise them using Copt=
where a controls the weight of the diffusion consistency loss. Notably, during inference, the step involving diffusion features is omitted, ensuring our method remains concise, efficient, and general.
K
Sup
La (9ggens, (Popt) )²)
j=1
5. Experiments
0.5(x y)²
if x yl <
with Lo(z, v)
x - y1 0.5
otherwise
5.1. Experimental Setups
Datasets. We evaluate the performance of QueryDiff fol- lowing the domain generalization standard setting (Hümmer et al., 2023; Benigmim et al., 2024; Wei et al., 2024). As synthetic datasets, GTAS (Richter et al., 2016) provides 24,966 images at a resolution of 1914x1052. As real-world datasets, Cityscapes (Cordts et al., 2016) includes 2,975 images for training and 500 images for validation, with im- ages at 2048x 1024 resolution. BDD100K (Yu et al., 2020) consists of 7,000 training images and 1,000 validation im- ages, each at 1280x720 resolution. Mapillary (Neuhold et al., 2017) offers 18,000 training images and 2,000 vali- dation images, with resolutions varying across the dataset.
The diffusion consistency loss is:
Lair Sup + Laist
(12)
This process enables the agent queries to actively estab- lish contextual relationships between instances within the scene during their generation, fostering comprehensive understanding of the underlying scene distribution.
4.3. Feature Refinement and Mask Prediction
The final step involves leveraging the optimized agent queries to refine the features in the segmentation decoder and predicting the segmentation mask. The typical decoder can be divided into three components: a pixel decoder that extracts features, a transformer decoder that outputs mask embedding, and mask decoder that outputs class proba- bilities (Cheng et al., 2022). In our pipeline, we insert the refinement module after the pixel decoder, which takes the multi-scale features output by the pixel decoder and the optimized agent queries as inputs and outputs the refined features. For the refinement module, we use cross-attention and multiple parallel 3x3 depthwise separable convolu- tions (Chollet, 2017) with different dilation rates to refine feature representations. In particular, we start by feeding the multi-scale features and optimized agent queries into cross-attention to get new feature representations. Next, we concatenate these with the original multi-scale features, perform fusion using depth-separable convolution, and then align dimensions with 1x1 1 convolution to obtain refined pixel features. The transformer decoder produces a set of mask embeddings, which take the refined pixel features as input. These mask embeddings are then classified by the mask decoder to produce set of class probabilities. Finally, we use the segmentation loss Leeg as follows:
As adverse weather datasets, ACDC (Sakaridis et al., 2021) consists of 406 validated images at 1920x 1080 resolution, including night, snow, rain and fog. For simplicity, we ab- breviate GTA5, Cityscapes, BDD and Mapillary as G, C, B, and M, respectively; AN, AS, AR and AF denote night, snow, rain and fog subsets of ACDC, respectively.
Implementation Details. Following previous works (Be- nigmim et al., 2024; Wei et al., 2024), we use the decoder from mask2former (Cheng et al., 2022), a widely-used seg- mentation head compatible with various backbones, includ- ing ResNer50 (He al., 2016), MiT-B5 (Xie et al., 2021), and DINOv2 (Oquab et al., 2023). For the training phase, the AdamW optimizer (Loshchilov & Hutter, 2017) is em- ployed, setting the learning rate at le-5 for the backbone and le-4 for both the decoder and the learnable queries. We uti- lize a configuration of 60,000 iterations with a batch size of 4, and crop images to a resolution of 512 x 512. We employ Stable Diffusion v2-1 (Rombach et al., 2022) as our dif- fusion model, which is trained on LAIONSB (Schuhmann et al., 2022) and remains frozen throughout.
5.2. Comparison with Previous Methods
Cxeg
Aboe Lose >daie Laice + Xuts Less
(13)
We comprehensively compare our method with existing DGSS methods. We conduct experiments in three general- ization settings: synthetic-to-red (G-C, B, M), real-to-real (C-B, M) and normal-to-adverse (C-AF, AR, AS, AN).
where the predicted mask is optimized using binary cross entropy loss Lice and dice loss Ldice while the predicted classes of the mask embedding are optimized using binary cross entropy loss La. The loss weights Ance. A and Adt are set to the same values.
Synthetic-to-real generalization. Table I presents a com- parison between our proposed method and existing DGSS approaches under the G-+C, B, M scenario. Compared to the previous SOTA method Rein, our approach achieves substantial improvement of 2.4 points and consistently surpasses all DGSS methods across various backbone archi-
Full Objective. The full training objective consists of the segmentation loss and the diffusion consistency loss:
Liotal = Lreg + a.Cai
(14)
6

---

Better to Teach than to Give: Domain Generalized Semantic Segmentation via Agent Queries with Diffusion Model Guidance
Table
Performance comparison between the proposed QueryDiff and existing DGSS methods. Top three results are highlighted
best
second and third respectively. VFM refers the vision foundation model. Our results are an average of times.
Utilizing Diffusion
symthctic-to-rcal
rcal-to-real C- M
Method
Proc. Year
Backbone
G CC
G BB
G-M
Avg.
C-+B
Avg.
ResNet bosed:
WildNet (Lec et ul., 2022)
CVPR2022
RN101
X
44.6
38.4
46.1
43.0
50.9
58.8
54.9
SHADE (Zhao et al., 2022)
ECCV2022
RN101
X
46.7
43.7
45.5
45.3
51.0
60.7
55.8
SAW (Peng et ul., 2022)
CVPR2022
RN50
X
39.8
37.3
41.9
39.7
53.0
59.8
56.4
TLDR (Kim et al., 2023)
ICCV2023
RNIO1
X
47.6
44.9
48.8
47.1
DIDEX (Niemeijer al., 2024)
WACV2024
RNI01
52.4
40.9
49.2
47.5
BlindNet (Ahn et al., 2024)
CVPR2024
RN50
45.7
41.3
47.1
44.7
51.8
60.2
56.0
FAMix (Fahes et al., 2024)
CVPR2024
RN101
49.5
46.4
52.0
49.3
QueryDiff (Ours)
RN50
52.5
49.2
53.3
51.7
53.7
64.1
58.9
Trunsformer based:
HRDA (Hoyer et al., 2022)
ECCV2022
MiT-BS
57.4
61.1
55.9
58.5
68.3
63.4
HGFormer (Ding al., 2023)
CVPR2023
Swin-T
53.4
66.9
60.2
DGInStyle (Jia et al., 2023)
ECCV2024
MiT-B5
58.6
52.3
62.5
57.8
58.8
68.0
63.4
QueryDiff (Ours)
MiT-B5
61.9
55.4
63.2
60.2
61.9
70.7
66.3
VFM based:
CLOUDS (Benigmim al., 2024)
CVPR2024 CVPR2024
ConvNext-L DINOv2-L DINOv2-I.
60.2
57.4
67.0
61.5
Rein (Wei al., 2024)
66.4
60.4
66.1
64.3
63.5
74.0
68.8
QueryDiff (Ours)
69.1
62.3
68.6
66.7
66.1
75.9
71.0
datasets generalization, Comparison with state-of-thh-an methods for DGSS on C AF, AN, AR, AS.
Table
Table 4. Generalization ability test of the proposed QueryDiff across multiple VFMs backhones. denotes trainable parame- ters in backbones.
Models are tested on the validation set of ACDC. VFM refers to the vision foundation model. Our results are an average times.
Training Method
Trainable
mloU
Backbone
Params
G-C
G-BB
G-M
Avg.
Backbone
C-AAF
C-
C-AAR
Full
304.15M
51.3
47.6
54,3
51.1
ReoNer Iosel:
Freeze
0.00M
53.7
48.7
55.0
52.5
Mask2Hommer (Cheng 2022)
RN50 RN50
54.1
36.5
53.1
506
89.8
CLIP
JJGFummr (Ding 2023)
36.3
25.8
57.7
56.2
5L.
Rein
2.99M
57.1
54.7
60.5
57.4
QueryDiff
RN50
QueryDiff
5.73M
58.9
56.0
61.9
58.9
Prauforsee Baseed
ISSA (Li et al., 2023)
MiT B3
67.5
33.2
55.9
53.2
52.5
Full
632.18M
57.6
51.7
61.5
56.9
Mark2Fermer (Cheng 2022)
HGForner (Ding 2023)
Swin-L L
Swin-T,
69.1
53.1
68.3
65.2
63.9
SAM
Freeze
Rein
0.00M
4.51M
$7.0
59.6
47.1
52.0
62.1
58.4
57.9
54.2
VVM Janndd
QueryDiff (Ours)
MiT-85
73.1
53.9
QueryDiff
7.25M
61.4
53.7
64.2
59.8
Rein (Wei ol., 2014)
DINOv2LL
76.4
70.6
75.2
79.5
77.6
Full 304.20M
63.7
57.4
64.2
61.8
QutryDiff (Ours)
DINOv2
78.5
72.5
82.3
79,9
Freeze
0.00M
63.3
56.1
63.9
61.1
DINOV2
Rein 2.99M
66.4
60.4
66.1
64.3
QueryDiff
5.73M
69.1
62.3
68.6
66.7
Table Ablation study primary components. The experiments
are conducted using Rein with DINOv2-L backbone, trained on
the GTAV dataset. AQ denotes agent queries.
Bise 92.4 59.1 90.7 58.3 53.7 51.8 582 46.4 89.8 49.4 90.8 73.9 43.5 92.3 64.3 81.6 70.9 40.4 54.0
AQ
Lup
Las
G-5B
G-M
Avg.
AmloU
Ours 92.5 60.0 90.6 54.6 53.5 51.3
$3.7
89.6
$2.6
90.7
75.5
50.7.99
93.1
83.9
79.1
53.26533
66.4
60.4
66.1
64.3
Road
Swalt
Buld
Well
Rewe
Pole
T.light
T:5kgn
Vega
Termin
Sky
Possnn
Ruder
Bus
Trin
Makk
Dike
68.0
60.9
67.4
65.4
+1.1
4
68.8
61.7
67.9
66.1
+1.8
69.1
62.3
68.6
66.7
+2.4
Figure 3. Comparison of the class-wise IoU on GTA -> using Rein with and without our method. The red colors visualize the differences in class-wise IoU between the baseline and our method.
Cityscapes
tectures, including ResNet, MiT-B5, and DINOV2. In partic- ular, when using ResNet50, our method achieves 51.7%, i.e. +7% compared to BlindNet, which also uses ResNet50, and +2.4% compared to FAMix using the stronger 10..
setup, all the models are trained on Cityscapes and tested on ACDC. Table 2 highlights that our method achieves consistently better performance than competing approaches across all three backbone variants. When using ResNet50, our method improves performance from 51.6 to 53.8 com- pared with HGFormer. As for the backbone of MiT-B5 and DINOV2-L, our method respectively improves the mloU by 1.9 and 1.6 compared with the previous SOTA method.
Real-to-real generalization. Under this evaluation setup, models are trained on Cityscapes and tested on BDDI00K and Mapillary. The results, starting from the ninth column of Table 1, confirm that our method consistently delivers better domain generalization performance across various datasets and backbone configurations, underscoring its effectiveness in adapting to diverse real-world scenarios.
Normal-to-adverse generalization. Under this evaluation
Comparison the class-wise IoU. We gauge the impact
7

---

Better to Teach than to Give: Domain Generalized Semantic Segmentation via Agent Queries with Diffusion Model Guidance
Image
Rein
Ours
Ground truth
B
Table 5. Ablation study on different sets of timesteps.
tx
M
Avg.
Citys
50
68.2
62.0
68.4
66.2
50
100
68.7
62.1
68.6
66.5
100
69.1
62.3
68.6
66.7
75
100
68.3
62.0
68.2
66.
BDD
100
200
68.5
62.1
67.1
65.9
0
200
67.6
61.9
67.4
65.6
Table 6. Quantitative comparison of different difffusion models.
Model
Rein
+SD 1,4
+SD 1.5
+SD
mloU 64.3
66.4
66.6
66.7
Figure 4. Qualitative comparison under G-C, B, M generalization setting. From left to right: target image, the visual results predicted by Rein, Ours, and Ground Truth. We deploy the white dash boxes to highlight different prediction parts.
over the baseline), highlighting their synergy in enhancing cross- domsind segmentation.
69.2
69
Investigating various VFMs. We evaluate QueryDiff across multiple Vision Foundation Models (VFMs), in- cluding CLIP (Radford al., 2021), SAM (Kirillov et al., 2023), and DINOv2 (Oquab al., 2023), under full fine- tuning, lightweight fine-tuning (Rein), and frozen backbone schemes (Wei et al., 2024) in the G - C. B, M setting. As shown in Table 4, QueryDiff achieves the highest mloU, significantly outperforming other methods.
Citsccapes
69.0
69.0
68.9
Mapillary
68.8
mloU (%)
68.6
68.6
60.5
6844
6
68.3
68.0
67
67.7
50
100
125
Longth
Study on agent queries length r. The core component of our method is a set of agent queries. We explore various lengths for the agent query sequence, ranging from 50 to 150. As demonstrated in Figure 5, models with r 100 achieve a strong mloU of 69.1% and 68.6% on Cityscapes and Mapillary datasets, respectively.
Figure 5. Ablation study on agent queries length r.
of our method on class-wise IoU scores, as shown in Figure 3, which demonstrates notable performance gains across multiple classes, especially in rarer ones like rider, truck, train, and motorbike. The heatmap affirms the capability of our method across a wide range of classes
The choice of tw and t.. Noise intensity increases with the timestep value, with higher values introducing stronger perturbations. Previous studies (Xu et al., 2023; Baranchuk et al., 2022) reveal that semantically meaningful diffusion features primarily generated within the range of 0 to 200, and that neighboring timesteps often produce highly similar representations. As shown in Table 5, we empirically adapt 4yp = 0 and 4, = 100 to ensure that agent queries can capture high-dimensions semantics while avoiding being influenced by trivial information.
Qualitative results. We visually compare the segmentation results with the previous SOTA Rein under G-+,, B, M setting in Figure 4. The results highlighted by white dash boxes show that our method creates more reasonable class predictions as well as more complete spatial distributions. We think it is because the proposed method learns the correct knowledge of the scenario, which is crucial for improving the generalization of the model.
Comparing different stable diffusion. In Table 6, Rein the baseline model and conduct the comparative experiment with three currently dominant stable diffusion models. The final results are not sensitive to the choice of different diffusion models, and our proposed method, when combined with different diffusion models, consistently outperforms the baseline model.
5.3. Ablation Studies
In this section, we present comprehensive experiments to validate the effectiveness of our method.
Effect of components. We evaluate the effectiveness of the primary components under G-+C, B, M settings, focusing on agent queries, Suu and Laist in the proposed method. The baseline model is based on Rein. As shown in Table 3, we can observe: (1) Each module positively contributes to the model's 's performance, demonstrating the individual effectiveness of these components. (2) The combination of all three modules yields the highest mloU of 66.7 (+2.4
6. Conclusion
In this work, we propose QueryDiff, a novel agent query- driven learning framework based on diffusion model guid- ance for DGSS. QueryDiff leverages agent queries as an interface to mine scene distribution priors embedded in
8

---

Better to Teach than to Give: Domain Generalized Semantic Segmentation via Agent Queries with Diffusion Model Guidance
diffusion model. Also, to avoid the interference of visual de- tails, propose diffusion consistency loss to enable agent queries to focus on domain-invariant semantic information. Extensive experiments demonstrate that QueryDiff signif- icantly surpasses previous SOTA methods across various benchmarks, highlighting its effectivvness. This work not only bridges a critical research gap but also establishes a new standard for domain generalized semantic segmentation.
conference on computer vision and paltern recognition, pp. 1290-12999 2022.
Choi, S., Jung, S., Yun, H.,, Kim, T., Kim, S., and Choo, J. Robustmet: Improving domain generalization in urban- scene segmentation via instance selective whitening. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 11580-11590, 2021.
Chollet, F. Xception: Deep learning with depthwise separa- ble convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251- 1258, 2017.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must specifically highlighted here.
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B. The cityscapes dataset for semantic urban scene un- derstanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
References
Ahn, W.-., Yang, G.-Y., Choi, H.-D., and Lim, M.-T. Style blind domain generalized semantic segmentation via co- variance alignment and semantic consistence contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3616 3626, 2024.
Dhariwal, P. and Nichol, A. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Ding, J., Xue, N., Xia, G.-S., Schiele, B., and Dai, D. Hg- former: Hicrarchical grouping transformer for domain generalized semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15413-15423, 2023.
Baranchuk, D., Voynov, A., Rubachev, I., Khrulkov, V., and Babenko, A. Label-efficient semantic segmenta- tion with diffusion models. In International Confer- ence on Learning Representations, 2022. URL https: //openreview, net http://www..eeeeeooomd////d----
Fahes, M., Vu, T.-H., Bursuc, A., Pérez, P., and de Charette, R. A simple recipe for pooguage-guidee domain gener- alized segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2-428-234223 2024.
Benigmim, Y., Roy, S., Essid, S., Kalogeiton, V., and Lath- uilière, S. Collaborating foundation models for domain generalized semantic segmentation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pallern Recognition, pp. 3108-3119, 2024.
Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z.,, and Lu, H. Dual attention network for scene segmentation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 3146-3154, 2019.
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual leam- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 700-77077777 778, 2016.
Chen, L.-C.,, Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A. L. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834 848, 2017.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Cheng, B., Schwing, A., and Kirillov, A. Per-pixel clas- sification is not all you need for semantic segmentation. Advances in neural information processing systems, 34: 17864-17875. 2021.
Hoyer, I.., Dai, D., and Van Gool, L. Hrda: Context-aware high-resolution domain-adaptive semantic segmentation. In European conference on computer vision, pp. 372-391. Springer, 2022.
Cheng, B., Misra, I,, Schwing, A. G., Kirillov, A., and Gird- har, R. Masked- -tteentio mask transformer for universal image segmentation. In Proceedings of the IEEF/CVF
Hümmer, C., Schwonberg, M., Zhong, L., Cao, H., Knoll, A., and Gottschalk, H. Vltseg: Simple transfer of

---

Better to Teach than to Give: Domain Generalized Semantic Segmentation via Agent Queries with Diffusion Model Guidance
clip-based vision-language representations for domain generalized semantic segmentation. arXiv preprint arXiv:2312.022211 2023.
Li, Z., Wang, W., Xie, E., Yu, Z., Anandkumar, A., Alvarez, J. M., Luo, P., and Lu, T. Panoptic segformer: Delving deeper into panoptic segmentation with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1280-1289, 2022.
Ji, Y., Chen, Z., Xie, E., Hong, L., Liu, X., Liu, Lu, T., Li, Z., and Luo, P. Ddp: Diffusion model for dense visual prediction. In Proceedings the IEEE/CVF International Conference on Computer Vision, pp. 21741-21752 2023.
Long, J., Shelhamer, E., and Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
Jia, Y., Hoyer, L., Huang, S., Wang, T., Van Gool, L.., Schindler, K., and Obukhov, A. Dginstyle: Domain- generalizable semantic segmentation with image diffu- sion models and stylized semantic control. In Synthetic Data for Computer Vision @orkshop@ CVPR 2024, 2023.
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. arXiv preprint arXiv: 2017.
Neuhold, G., Ollmann, T., Rota Bulo, S., and Kontschieder, P. The mapillary vistas dataset for semantic understanding of street scenes. In Proceedings of the IEEE international conference on computer vision, pp. 4990 4900004999,400 2017.
Jia, Y., Hoyer, L.,, Huang, S., Wang, T., Van Gool, I.., Schindler, K., and Obukhov, A. Dginstyle: Domain- generalizable semantic segmentation with image diffu- sion models and stylized semantic control. In European Conference on Computer Vision, pp. 91-109. Springer, 2025.
Niemeijer, J., Schwonberg, M., Termöhlen, J.-A., Schmidt, N. M., and Fingscheidt, T. Generalization by adap- tation: Diffusion-based domain extension for domain- generalized semantic segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Com- puter Vision, pp. 2830-2840, 2024.
Jiang. X., Huang, J., Jin, S., and Lu, S. Domain generaliza- tion via balancing training difficulty and model capability. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 18993-19003, 2023.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F,, El- Nouby, A., et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07195 2023.
Ke, B., Obukhov, A., Huang, S., Metzger, N., Daudt, R. C., and Schindler, K. Repurposing diffusion- based image generators for monocular depth estimation. In Proceed- ings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9492-9502, 2024.
Pan, X., Luo, P., Shi, J., and Tang, X. Two once: En- hancing learning and generalization capacities via ibn-net. In Proceedings of the european conference on computer vision (ECCV), pp. 464 479, 2018
Kim, S., Kim, D.-h., and Kim, H. Texture learning domain randomization for domain generalized segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 677-687, 2023.
Peng, D., Lei, Y., Hayat, M., Guo, Y., and Li, W. Semantic- aware domain generalized segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2594-2605, 2022.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C.,, Gustafsson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Lee, H.-Y., Tseng, H.-Y., and Yang, M.-H. Exploiting diffusion prior for generalizable dense prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7861-7871, 2024.
Richter, S. R., Vineet, V., Roth, S., and Koltun, V. Play- ing for data: Ground truth from computer games. In Computer Vision-ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October l 11-14, 2016, Proceedings, Part II 14, pp. 102-118. Springer, 2016.
Lee, S., Seong, H., Lee, S., and Kim, E. Wildnet: Learning domain generalized semantic segmentation from the wild. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 9936-9946, 2022.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Li, Y.. Zhang, D., Keuper, M., and Khoreva, A. Intra-source style augmentation for improved domain generalization. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 509-519, 2023.
10

---

Better to Teach than to Give: Domain Generalized Semantic Segmentation via Agent Queries with Diffusion Model Guidance
Sakaridis, C.,, Dai, D., and Van Gool, L. Acde: The ad- verse conditions dataset with correspondences for seman- tic driving scene understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10765-10775, 2021.
representation learning for generalizable semantic seg- mentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 2884-2892, 2022.
Yu, F., Chen, H., Wang, X., Xian, W., Chen, Y., Liu, F., Mad- havan, V., and Darrell, T. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceed- ings of the IEEE/CVF conference computer vision and pattern recognition, pp. 2636 2605,2664,,10 2020.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 15278-25299 2022.
Zhang, H., Li, F., Xu, H., Huang, S., Liu, S., Ni, L. M., and Zhang. L. Mp-former: Mask-piloted transformer for image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18074-18083, 2023.
Tang, L., Jia, M., Wang, Q.. Phoo, C. P., and Hariharan, B. Emergent correspondence from image diffusion. Ad- vances in Neural Information Processing Systems, 36: 1363-1389, 2023.
Zhao, Y., Zhong, Z., Zhao, N., Sebe, N., and Lee, G. H. hellccaallee dual consistency learning for domain generalized semantic segmentation. In European confer- ence on computer vision, pp. 535-552. Springer, 2022.
Wang, Y., Li, H., and Kot, A. C. Heterogeneous domain generalization via domain mixup. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 3622-3626. IEEE, 2020.
Zhong, Z., Lin, Z. Q., Bidart, R., Hu, X., Daya, B., Li, Z.,
Zheng, W.-S., Li, and Wong, A. Squeeze-and networks for semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
Wei. Z., Chen, L., Jin, Y., Ma. X., Liu, T., Ling, P., Wang, B., Chen, H., and Zheng, J. Stronger fewer & superior: Har- nessing vision foundation models for domain generalized semantic segmentation. In Proceedings of the IEEESCVF Conference on Computer Vision and Pattern Recognition, 2024.
2020.
Zhong, Z., Zhao, Y., Lee, G. H., and Sebe, N. Adversarial style augmentation for domain generalized urban-scene segmentation. Advances in neural information processing systems, 35:338-355 2022.
pp.
Wu, W., Zhao, Y., Chen, H., Gu, Y., Thao, R., He, Y., Zhou, H., Shou, M. Z., and Shen, C. Datasetdm: Synthesizing data with perception annotations using diffusson models. Advances in Neural Information Processing Systems, 36: 54683-54695, 2023a.
Wu, W., Zhao, Y., Shou, M. Z., Zhou, H., and Shen, C. Dif- fumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1206-1217, 2023b.
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J. M., and Luo, Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems, 34:12077-12090, 2021.
Xu, J.. Liu, S., Vahdat, A., Byeon, W., Wang, X., and De Mello, S. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pp. 2955-2966, 2023.
Xu, Q. Yao, L., Jiang, Z., Jiang, G., Chu, W., Han, W., Zhang, W., Wang, C., and Tai, Y. Dirl: iomaintinvariant


Reviews:
[{"review_id": "zJYH7jEdkf", "reviewer": "Reviewer_t5xc", "summary": "The paper presents a novel framework for utilizing diffusion models for domain-generalized semantic segmentation. While previous works often struggle to generate reasonable scenes for semantic segmentation, this paper introduces agent queries from segmentation features and incorporates additional pretrained knowledge from diffusion models. Extensive experimental results demonstrate significant performance improvements achieved by the proposed method.\n\n## update after rebuttal\n\nI appreciate the additional experiments provided in the rebuttal. Since the results are satisfactory, I will maintain my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q**: Have you conducted experiments in in-domain settings, such as few-shot or fully-supervised learning on the Cityscapes dataset? This approach could potentially achieve state-of-the-art performance even beyond domain-generalized settings.\n\n**A**: Thank you for this valuable suggestion. We conducted additional experiments in a few-shot in-domain setting on the Cityscapes dataset, using only 1/16 of the labeled training data. As shown in the table below, our method surpasses the Rein baseline, demonstrating its effectiveness not only in domain generalization tasks but also in few-shot in-domain semantic segmentation. In future work, we will further explore the performance of our method under a range of few-shot and fully supervised settings to more comprehensively evaluate its potential in broader in-domain scenarios.\n\n| Method   | Source                  | mIoU     |\n| -------- | ----------------------- | -------- |\n| Rein     | +1/16 of Cityscapes     | 82.5     |\n| **Ours** | **+1/16 of Cityscapes** | **83.6** |", "claims_and_evidence": "The authors argue that instead of generating datasets using diffusion models, leveraging intermediate features from diffusion models as a form of loss to train the segmentation model is more effective. They liken this approach to teaching how to fish rather than simply providing fish. The final performance shows a significant improvement over other baselines, making their claim clear and well-supported.", "methods_and_evaluation_criteria": "The proposed methods are all clear and reasonable. Agent Queries Generation seems both reasonable and novel; by aggregating semantic information within a scene, the agent queries effectively represent the scene’s semantic content. While previous approaches relied solely on features from a small noise step, this study introduces an additional step that leverages features from two different noise steps to verify consistency. This allows the model to learn from diffusion features even in strongly noisy images.", "theoretical_claims": "There is no theoretical claim.", "experimental_designs_or_analyses": "The experiments are valid and include state-of-the-art baselines for synthetic dataset generation, such as CLOUDS and Rein, effectively demonstrating the superiority of the proposed method. The results consistently show improved performance across various backbones. The experimental setup comprehensively covers synthetic-to-real, real-to-real, and normal-to-adverse scenarios, maintaining consistently high performance. Additionally, the ablation study is well-conducted, and class-wise performance is thoroughly analyzed.", "relation_to_broader_scientific_literature": "The idea presented in this paper—incorporating diffusion models as a loss rather than generating datasets—is a significant breakthrough in the application of diffusion models for semantic segmentation. This approach has the potential to shift the paradigm in the field.", "essential_references_not_discussed": "I found no essential references that should have been discussed but were missing from the paper.", "other_strengths_and_weaknesses": "I have discussed all strengths and weaknesses in other sections.", "other_comments_or_suggestions": "I have no additional comments.", "questions_for_authors": "Have you conducted experiments in in-domain settings, such as few-shot or fully-supervised learning on the Cityscapes dataset? This approach could potentially achieve state-of-the-art performance even beyond domain-generalized settings.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BDws4b6pxL", "reviewer": "Reviewer_6H1m", "summary": "The authors leverage on refined features of diffusion models to stabilize the features of vision transformers and other backbones when feeding them into the mask2former decoders for semantic segmentation. In this way, the authors achieve considerable domain generalization capabilities for their network. In their method, the authors first construct layer wise queries over the layers of the backbone which are thereafter aggregated over the layers using a kind of a progressive, layer wise cross attenion mechanism. This ultimately leads to trainable \"agent queries\" which are thereafter transformed by a NLP trained for similarity with features of stable diffusion from the encoding space of the VAE encoder of stable diffusion. These features are the noise predictions from the stable diffusion model at a low and a high level of noise. It is trained that the \"agent queries\" remain stable under different noise levels, which enhances dependency on global structures and removes texture bias.\nThe authors then test their network on two domain generalization benchmarks (DG), namely GTA to Cityscapes, BDD and Mapplilary and the real to real benchmark Cityscapes to ACDC. The authors report decent DG performance and claim superiority to the present DG SOTA throughout their tests. The authors also provide extensive testing utilizing various backbones from ResNet over Transformer to pretrained FM models. Also, the single modules of the proposed architecture are tested separately.\n\n# # Due to the results on the official CS-> ACDC benchmark and the authors announcement that they will publish their code I am now convinced that QuaryDiff is really a model that surpasses SOTA. I therefore will raise my score an d favor publication. BUT: The way we got here was somehow strange. I hope that the authors will take a more direct way in the future. If the results are good, as it seems to be in this case after new experiments have been conducted during rebuttal, no maneuvers are needed.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q**:Reported results for Rein differ from those in its original paper.\n\n**A**:The performance difference arises because we reproduced Rein at a 512×512 resolution on the ACDC validation set using the official code—as indicated in Table 2 (line 351)—rather than the original 1024×1024 resolution on the test set reported in the paper. \n\n(1) Resolution setting: 512×512 is one of the most commonly used and widely accepted resolutions in domain generalization segmentation, adopted in the main results of Rein and VLTSeg, and set as default in DGInStyle, making our setup standard and reasonable. (2) Evaluation protocol: The ACDC validation set is widely adopted in recent works like DGInStyle and HGFormer, further supporting the validity of our setup. Under this fair and consistent setting, our method consistently outperforms Rein, as shown in Table 2.\n\nWe also evaluated our method on the ACDC test set at a 768×768 resolution, another mainstream setting adopted in recent methods like CLOUDS and FAMix. As shown in the table below, our performance at 768×768 resolution is already on par with or better than Rein’s results at 1024×1024, highlighting the scalability and robustness of our method across resolutions.\n|Method|C→AF|C→AN|C→AR|C→AS|Avg.|\n|-|-|-|-|-|-|\n|Rein(1024x1024)|76.4|70.6|78.2|79.5|77.6|\n|Ours(768x768)|76.7|70.7|79.3|79.7|77.9|\n\n**Q**:Several strong methods (SoRA [R1], VLTSeg [R2]) were not discussed or compared.\n\n**A**:SoRA is a concurrent work that was not accepted at the time of our submission and has not released its code. VLTSeg uses a different experimental setup (high-resolution 1024×1024) and also lacks public code, making fair and consistent comparison difficult. Nevertheless, under the 768×768 resolution setting, our method achieves 77.9 mIoU  (see table above), which is comparable to VLTSeg, demonstrating the competitiveness of our approach. We will include a discussion of VLTSeg in the final version.\n\n**Q**:QueryDiff is not submitted to the official ACDC benchmark, and several strong leaderboard models are not adequately discussed.\n\n**A**:Due to current computational limitations, we are unable to support high-resolution settings (e.g., 1024×1024) used by several top-performing models, and thus conduct experiments at two widely adopted resolutions: 512×512 and 768×768. We plan to submit to the official benchmark once adequate computational resources become available. Nevertheless, our comparisons already include the most representative and competitive methods. Under the 768×768 setting, our method achieves performance comparable to Rein and VLTSeg, despite their results being reported at 1024×1024—demonstrating the effectiveness and scalability of our method. Furthermore, many benchmark submissions lack implementation details and are not open-sourced, making it difficult to reproduce results and ensure fair and transparent comparisons.\n\n**Q**:The strong GTA→CS performance may be affected by DINOv2’s pretraining on CS and Mapillary.\n\n**A**:DINOv2 is a standard backbone in recent domain generalization research, including Rein and VLTSeg. To demonstrate the effectiveness of our method, we also evaluated it with alternative backbones, including CLIP, SAM, and MiT-B5. Notably, even with SAM, our method still outperforms most existing approaches, showing strong generality across architectures. Additionally, we report results with ConvNext in the table below. Compared to CLOUDS, which also uses ConvNext, our method achieves superior performance, further confirming its effectiveness and generalizability.\n|Method|Backbone|G→C|G→B|G→M|Avg.|\n|-|-|-|-|-|-|\n|CLOUDS|ConvNext-L|60.2|57.4|67.0|61.5|\n|Ours|ConvNext-L|62.1|60.3|67.8|63.4|\n\n**Q**:QueryDiff shows lower performance than the fine-tuning results reported in VLTSeg ([R2]) with CLIP.\n\n**A**:VLTSeg reports results using both CLIP and EVA-02-CLIP backbones. EVA-02-CLIP is a stronger backbone than CLIP, due to its enhanced architecture and more extensive pre-training. To fairly evaluate QueryDiff against VLTSeg, we compare them using the same backbone. As shown in the table below, QueryDiff consistently outperforms VLTSeg when both methods use the same backbone.\n|Method|Backbone|G→C|G→B|G→M|Avg.|\n|-|-|-|-|--|-|\n|VLTSeg|CLIP|55.6|52.5|59.9|56.0|\n|Ours|CLIP|58.9| 56.0|61.9|58.9|\n|VLTSeg|EVA-02-CLIP|65.3|58.3|66.0|63.2|\n|Ours|EVA-02-CLIP|66.9|60.9|67.1|65.0|\n\n**Q**:The Synthia source dataset is not evaluated\n\n**A**:SYNTHIA results are provided in Table 1 of the supplementary material, submitted separately in accordance with ICML guidelines.\n\n**Q**:The code release has not yet been announced.\n\n**A**:We will release all code and weights upon acceptance.\n\n**Q**:Several tables lack metric labels (mIoU).\n\n**A**:Thank you. We will clarify the evaluation metrics in relevant tables.\n\n**Q**:Using Linear(...) to denote MLP is not appropriate.\n\n**A**:Thank you for the suggestion. We will revise the final version to use more standard MLP notation for improved clarity.", "claims_and_evidence": "The evidence for the claims is partially convincing, but there are irritating aspects, see weaknesses. Namely, the numbers reported for the competing models differ from those documented in the original papers and in official benchmarks. Also, several competing models are not discussed. Therefore I find the authors claim that their model constantly outperforms the SOTA weakly supported, despite the reported results certainly document strong DG performance. An official sumbission to the official CS-> ACDC benchmark would further strengthen the author's case.", "methods_and_evaluation_criteria": "The method of evaluation follows standard protocols and is widely shared throughout the community. However, the Synthia source data set is not evaluated, which would then complete the standard evaluation protocol in the context of street scenes. What the authors do more than usual is the semantic interpretation of two pieces of artwork, cf. Figure 1.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "The experimental design follows standard procedures (up to omitting Syntia-> CS, BDD, Mappilary DG). The analyses include ablation studies, which are detailed.", "relation_to_broader_scientific_literature": "none", "essential_references_not_discussed": "[R1] Seokju Yun, Seunghye Chae, Dongheon Lee, Youngmin Ro , SoRA: Singular Value Decomposed Low-Rank Adaptation for Domain Generalizable Representation Learning, https://arxiv.org/pdf/2412.04077 \n[R2] Christoph Hümmer, Manuel Schwonberg, Liangwei Zhou, Hu Cao, Alois Knoll, Hanno Gottschalk; Strong but simple: A Baseline for Domain Generalized Dense Perception by CLIP-based Transfer Learning, Proceedings of the Asian Conference on Computer Vision (ACCV), 2024, pp. 4223-4244\n[R1] submitted on 12/24 was well available on arXiv before the submission deadline.  One might adopt the policy not citing non peer reviewed work, but in theis case were the official CS->ACDC results were available which are 'objective'. Therefore I find this work should be included in the comparison with the SOTA. \n[R2] is the published version of Hümmer et al.", "other_strengths_and_weaknesses": "Strength\n*The authors idea of utilizing stable diffusion features for better domain generalization is compelling. Other FM have demonstrated their performance for enhanced DG, so involving Stable Diffusion is a good idea and the way the authors achieve it is clever. \n*Furthermore, Stable Diffusion is not needed during inference making the model efficient.\n* The ablation studies are extensive. \n* The paper is mostly well written, although the key section 4 could be a little clearer. Figures and table are nice.\n\nWeaknesses\n* My most severe criticism is that SOTA figures are not correctly reported. E.g. Rein achieves 76.48 mIoU on CS-> ACDC in average, but in the present paper is reported with 72.1. How can this happen? Likewise, the VLTSeg results on the same benchmark are 77.01 and are not reported as SOTA. Both models and [R1] with 78.75 mIoU are well above the reported QueryDiff performance with 73.7 mIoU. So I don't understand the authors conclusion that they constantly outperfrom the previous SOTA. Note that all these figures are obtained on private labels, so they are out of question. \n* Even more so, on the basis of the official CS->ACDC benchmark at the time of sumbission there were 7 sumbissions from 5 models stronger than the reported performance of QueryDiff. This should be discussed with greater care. Also, the authors should discuss this properly (although some of these models are undocumented)\n* There is no submission of the QuaryDiff result to the official benchmark\n* There is not yet an annoucement of code publication\n* Part of the strong GTA-> CS performance might be due to the fact that DINOV2 has seen CS and Mapliiary data during training. This should at least be mentioned as it is against the 'puristic' DG spirit. \n* Concerning the CLIP results in Table 4 - the QueryDiff results are below the results obtained by pure fine tuning in [R2]. How doe the authors explain this?", "other_comments_or_suggestions": "*Several tables don't write what they are showing (mIoU-values)\n* I find the notation Linear(...) for a MLP irritating.", "questions_for_authors": "See weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XUXSbclyVx", "reviewer": "Reviewer_otjG", "summary": "This paper proposes an agent Query-driven learning framework based on Diffusion model guidance for DGSS. The method employs agent queries to learn scene distribution knowledge from the diffusion model, capitalizing on the inherent consistency of this distribution across domains to improve segmentation model generalization. Diffusion consistency loss (DCL) is proposed to avoid intricate visual details in diffusion features from interfering with agent queries, enabling them to focus on learning the semantic distribution of the scene. Experiments have proven the effectiveness of the method.\n\n\n## update after rebuttal\nI appreciate the response provided in the rebuttal, which addresses some of my concerns, and I maintain the score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q**: The proposed method should be compared with previous methods in terms of computational complexity, time consumption, and memory usage.\n\n**A**: Thank you for the suggestion. We have included a detailed comparison of computational resources between our proposed method and recent diffusion-based segmentation approaches. First, unlike methods such as CLOUDS and DGInStyle that rely on a two-stage pipeline—image generation followed by segmentation training—our method adopts a single-stage, end-to-end paradigm, eliminating intermediate steps and reducing overall training complexity. Second, as shown in the table below, our method incurs the lowest cost among diffusion-based methods in terms of trainable parameters, training time, and memory usage.\n|Method| Generated Image|Trainable Params|Training Time|GPU Memory|Storage|\n|-|-|-|-|-|-|\n|ODISE|--|28.1M|6 days| 32*V100|4.9G|\n|PTDiffSeg|--|--| 32 h|1*V100 (32GB)|--|\n|CLOUDS|5000| 40.54M|19 h| 4*RTX4090|4.3G|\n|DGInStyle|6000| 85.69M|29 h|1*A100 (26687M)|1.37G|\n|Ours|--| 26.31M (5.73M+20.58M)|10.3 h|1*RTX4090 (16738M)|1.24G|\n\n**Q**: A more detailed explanation is requested for the merging process in Formula 5.\n\n**A**: We introduce a set of smaller initial agent queries ${q} _{\\text{init}} ^i$, which are first projected into query embeddings $Q$ using Equation (4). Meanwhile, the current-stage agent queries ${q} _{\\text{stage}} ^{i-1}$ are similarly projected into key embeddings $K$ and value embeddings $V$. In Equation (5), we compute the attention map $s^i$ between $Q$ and $K$, capturing the similarity in the embedding space between the smaller query groups ${q} _{\\text{init}} ^i$ and the larger groups ${q} _{\\text{stage}} ^{i-1}$. This attention map $s^i$ is then utilized to aggregate the values $V$, generating a new set of merged agent queries $\\hat{q} _{\\text{stage}} ^{i}$ which share the same dimensionality and count as the smaller initial set ${q} _{\\text{init}} ^i$. This merging process effectively reduces the number of agent queries while eliminating potential information overlap. As the merging process progresses across stages, the number of agent queries continuously decreases, leading to fewer but semantically richer and more comprehensive query groups.\n\n**Q**: A detailed explanation is requested of how instance-level semantic representation is optimized in Formula 10.\n\n**A**: Equation (10) is built upon the design and functionality of the matrix $S_j^{t_w}$, which plays a pivotal role in deriving optimized agent queries. Specifically, $S_j^{t_w}$, introduced in Equation (8), captures the associations between object features and agent queries. Subsequently, in Equation (9), a consistency loss is applied to explicitly suppress low-level visual texture details within  $S_j^{t_w}$, encouraging it to emphasize the underlying semantic distribution of the scene. Finally, in Equation (10), the refined semantic matrix $S_j^{t_w}$ is used to reorganize feature map $f_{d}^{(t_s,j)}$ through a semantic-guided re-weighting process. This process results in optimized agent queries that more effectively encode the scene’s semantic distribution, leading to more robust scene understanding.\n\n**Q**: More explanation is needed for Formula 11, including the dual segmentation loss and agent query supervision.\n\n**A**: (1) In Equation (11), we adopt the Huber loss due to its robustness to outliers and its balanced sensitivity to errors. This choice is particularly well-suited to our setting, where the intermediate supervision signal is derived by leveraging optimized agent queries to guide the original learnable agent queries. Such supervision inherently involves uncertainty—especially around object boundaries or ambiguous regions—making it a form of soft or intermediate supervision. To effectively handle this uncertainty, a robust loss function is essential. Unlike a purely quadratic ($L_2$) loss, which can result in instability and overly aggressive updates, the Huber loss provides a smoother optimization landscape, facilitating more stable and efficient convergence in these cases.\n\n(2) In Equation (10), we derive optimized agent queries through a diffusion-guided aggregation process that suppresses low-level visual details and emphasizes higher-level structural semantics. We then leverage these optimized agent queries to supervise the original learnable agent queries, aiming to explicitly guide them to prioritize structural context over superficial visual details during their learning process. This strategy enables the agent queries to actively capture semantic distribution of the scene. As a result, the model no longer requires support from the diffusion model during inference, as the structural semantic knowledge has been effectively embedded into the agent queries through this supervised learning process.\n\n**Q**: Will the code be publicly available?\n\n**A**: We will release all code and pretrained models upon acceptance.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "I checked the proofs for theoretical claims and found no issues.", "experimental_designs_or_analyses": "1. In the introduction, page 2, line 101 introduces the first challenge: previous methods are computationally expensive and time-consuming, making it inefficient for perception tasks. However, this has not been verified.", "relation_to_broader_scientific_literature": "Diffusion models have opened up new avenues for DGSS with remarkable capabilities in capturing complex scene distributions to generate high-quality, realistic samples. Diffusion models can generate diverse samples to enhance segmentation generalization by learning data distribution. In this paper, its scene distribution prior is used to enhance the generalization ability of the segmentation model instead of directly generating data.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n1. The idea of using the Diffusion model to improve generalization is novel. It is interesting to use agent queries to learn scene distribution knowledge from the diffusion model.\n2. A large number of experiments verify the effectiveness of the method.\n\nWeaknesses:\n1. In the introduction, page 2, line 101 introduces the first challenge: previous methods are computationally expensive and time-consuming, making it inefficient for perception tasks. However, this has not been verified. The proposed method should be verified experimentally compared with previous methods in terms of computational complexity, time-consuming, and memory usage.\n2. In Formula 5, it is introduced: \"We merge the smaller groups into larger ones based on the similarity matrix in the embedding\nspace\". Please explain in detail how the merge is implemented.\n3. In Formula 10, how to optimize the instance semantic representation in the agent queries, which is just a multiplication calculation. Please explain its mechanism in detail.\n4. More explanations are needed to explain the loss of Formula 11, including the loss form of dual segmentation and the significance of being supervised by agent queries.\n5. Will the code be publicly available?", "other_comments_or_suggestions": "As above", "questions_for_authors": "As above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XKoZTIYGE8", "reviewer": "Reviewer_ennc", "summary": "This paper proposes QueryDiff, an agent query-driven learning framework based on diffusion model guidance for DGSS, which utilizes the scene distribution priors embedded in diffusion models to enhance semantic segmentation generalization. Various experiments show the model’s effectiveness and reach the sota performance on many benchmarks.\n\n# update after rebuttal\n\nThanks the authors for their detailed response. My questions have been answered and I will maintain my positive score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q**: It is suggested that an ablation experiment of AQ + $L_{dist}$ (without $L_{sup}$) be included in Table 3.\n\n**A**: The purpose of $L_{dist}$ is to suppress the visual texture details within the matrix $S_j^{t_w}$. Subsequently, $S_j^{t_w}$ is utilized in Equation (10) to derive optimized queries $q_{opt}$. These optimized queries $q_{opt}$ are then employed to supervise the agent queries through $L_{sup}$ in Equation (11), encouraging the agent queries to proactively capture the semantic distribution of the scene and eliminating the need for diffusion guidance during inference. Therefore, $L_{dist}$ fundamentally enhances the supervision provided by $L_{sup}$. As a result, applying AQ + $L_{dist}$ alone—without $L_{sup}$—would be functionally equivalent to the AQ-only setting, which is already included in our ablation study.\n\n**Q**: There is inadequate discussion of computational cost and inference time.\n\n**A**: We have provided a comprehensive comparison of computational cost and inference speed between our method and representative previous approaches under the same experimental setting, using ConvNext as the backbone. Although diffusion-based feature extraction is used during training, it is discarded at inference and does not affect inference speed, as noted at the end of Section 4.3. Our method achieves inference speed comparable to strong baseline models like Rein, with only minor differences in training time, GPU memory usage, and storage requirements. Compared with CLOUDS—which also leverages diffusion models—our method is clearly more efficient in terms of trainable parameters, training time, GPU memory consumption, and storage size.\n\n|Method|Generated Image|Trainable Params|Training Time|GPU Memory|Storage|Inference Time|\n|-|-|-|-|-|-|-|\n|CLOUDS|5000|40.54M|19 h|4*RTX4090|4.3G|127.87 ms|\n|Rein|--|28.56M (8.29M+20.27M)|5 h|1*RTX4090 (11616M)|1.20G|126.59 ms|\n|Ours|--|31.30M (11.03M+20.27M)|8 h|1*RTX4090 (16091M)|1.21G| 126.81 ms|\n\n**Q**: The timestep ablation is insufficient, and a clear justification for the selected values is requested.\n\n**A**: Different timesteps correspond to varying levels of noise intensity, with larger timesteps introducing stronger noise. Prior studies [1,2] suggest that effective timestep values typically fall within the range of 0 to 200, and that adjacent timesteps often yield highly similar diffusion features. Based on these works, we selected intervals of 50 or 100 within the range of 0 to 200 to ensure sufficient diversity in feature representations. Through empirical comparisons, we ultimately chose timesteps 0 and 100 to represent weak and strong noise conditions, respectively. To further support this choice, we have included additional ablation experiments, as shown in the table below.\n\n|$t_w$|$t_s$|C|B|M|Avg.|\n|-|-|-|-|-|-|\n|200|300|67.7|61.7|67.2|65.5|\n|300|400|67.0|61.2|66.7|64.9|\n|0|25|67.9|61.8|67.8|65.8|\n|25|50|68.2|61.8|68.1|66.0|\n|75|100|68.3|62.0|68.2|66.1|\n\n[1] Xu, Jiarui, et al. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2955–2966, 2023.\n\n[2] Baranchuk, Dmitry, et al. Label-efficient semantic segmentation with diffusion models. In International Conference on Learning Representations, 2022.\n\n**Q**: It is suggested that intuitive visualizations be provided to better illustrate what the agent learns from the diffusion features.\n\n**A**: We conducted clustering analysis on both the original visual features and those refined via agent queries to intuitively illustrate the changes. As shown in the linked visualization, the optimized features exhibit a more coherent semantic distribution within the scene and improved spatial coverage of target objects.\nhttps://anonymous.4open.science/r/vis_feature-215E\n\n**Q**: An interpretation is requested for the class-wise performance variation.\n\n**A**: Thank you for the insightful question. As shown in Figure 3, the most notable improvements occur for rare classes (e.g., train, traffic sign, truck) and semantically similar categories (e.g., person and rider). This is mainly due to our method's explicit focus on learning the semantic distribution of scenes and modeling inter-instance relationships. Specifically, this structured modeling enables our method to effectively capture contextual dependencies, thus better distinguishing visually or semantically ambiguous classes. Moreover, rare classes typically suffer from limited training examples, making it difficult for standard models to learn discriminative features. By explicitly encoding scene semantic distribution, our method provides additional contextual information that compensates for limited visual evidence, significantly improving recognition of underrepresented categories. Thus, our method is particularly effective in improving recognition accuracy for both semantically overlapping categories and infrequent object classes.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "The evaluation makes sense to me.", "theoretical_claims": "The theoretical claims look good to me.", "experimental_designs_or_analyses": "The experimental designs and analyses (for the ablation study) look good to me. Please see the weakness for other issues.", "relation_to_broader_scientific_literature": "The paper explores how to leverage the scene distribution embedded in diffusion models to enhance segmentation generalization. This goes beyond using diffusion models to generate the data in a more direct way - utilizing its knowledge, which is more direct and helpful for generalization.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "**Strengths**\n- This paper proposes a novel application of diffusion models, leveraging their prior embedded distribution to enhance segmentation generalization rather than simply using them for data generation. The approach is both conceptually elegant and methodologically concise.\n- The comprehensive results (both qualitative and quantitative) results show the effectiveness of the method. The qualitative results on different stylistic domains also further strengthen the soundness of the method.\n\n**Weaknesses**\n- While the paper presents extensive ablation studies, the isolated effect of $L_dist$ is not examined. An experiment evaluating AQ + $L_{dist}$ (without $L_{sup}$) should be included in Table 3.\n- There is limited discussion about the computational cost and inference time compared to previous methods, especially when considering added components like diffusion feature extraction.\n- The ablation study of timesteps is not sufficient. The timestep selection (0, 50, 100) in the ablation study lacks justification.\n- It would be helpful to provide some intuitive interpretation of what the agent learns about the diffusion feature (like visualizations of the learned features and the mitigated visual features).", "other_comments_or_suggestions": "None", "questions_for_authors": "- The class-wise results indicate varying degrees of improvement across different classes, with a relatively large variation. Is there an interpretation for this difference? Could the visual features be influencing the results?\n- How does the timestep (0, 50, 100) selected?\n- Please see the weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Hi,

Draft has received overall positive reviews, with 2 weak accept and 2 accept.  Dear authors, please update the draft per recommendations and comments of the reviewers.
Authors have indicated that "We will release all code and weights upon acceptance.", we request that the link to code and model should be part of the camera ready version. Its also requested that the full code to reproduce the results from scratch will be released.

Congratulations.

regards

AC
Decision: Accept (spotlight poster)