Generation and Comprehension Hand-in-Hand: Vision-guided Expression Diffusion for Boosting Referring Expression Generation and Comprehension
Authors: Jingcheng Ke
Jun-cheng Chen
I-hong Jhuo
Chia-Wen Lin
Yen-Yu Lin
Venue: ICLR 2025 Poster (2025)
Keywords: Referring expression generation
referring expression comprehension
vision-guided expression diffusion
vision-text condition
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: we propose a novel VIsion-guided Expression Diffusion Model (VIE-DM) for the REG task, where diverse synonymous expressions adhering to both image and text contexts of the target object are generated to augment REC datasets.

Abstract: Referring expression generation (REG) and comprehension (REC) are vital and complementary in joint visual and textual reasoning.  Existing REC datasets typically contain insufficient image-expression pairs for training, hindering the generalization of REC models to unseen referring expressions. Moreover, REG methods frequently struggle to bridge the visual and textual domains due to the limited capacity, leading to low-quality and restricted diversity in expression generation. To address these issues, we propose a novel VIsion-guided Expression Diffusion Model (VIE-DM) for the REG task, where diverse synonymous expressions adhering to both image and text contexts of the target object are generated to augment REC datasets. VIE-DM consists of a vision-text condition (VTC) module and a transformer decoder. Our VTC and token selection design effectively addresses the feature discrepancy problem prevalent in existing REG methods. This enables us to generate high-quality, diverse synonymous expressions that can serve as augmented data for REC model learning. Extensive experiments on five datasets demonstrate the high quality and large diversity of our generated expressions. Furthermore, the augmented image-expression pairs consistently enhance the performance of existing REC models, achieving state-of-the-art results.

Published as a conference paper at ICLR 2025
GENERATION AND COMPREHENSION HAND-IN- HAND: VISION-GUIDED EXPRESSION DIFFUSION FOR BOOSTING REFERRING EXPRESSION GENERATION AND COMPREHENSION
Jingcheng Ke!
Jun-Cheng Chen
I-Hong Jhuo'
Chia-Wen Lin'
Yen-Yu Lin'
National Tsing Hua University
Academia Sinica
Microsoft AI
*National Yang Ming Chiao Tung University
ABSTRACT
Referring expression generation (REG) and comprehension (REC) are vital and complementary in joint visual and textual reasoning. Existing REC datasets typi- cally contain insufficient image-exprest expresssn pairs for training, hindering the gener- alization of REC models unseen referring expressions. Moreover, REG meth- ods frequently struggle to bridge the visual and textual domains due to the lim- ited capacity, leading low-quality and restricted diversity in expression gen- eration. To address these issues, we propose novel VIsion-guided Expression Diffusion Model (VIE-DM) for the REG task, where diverse synonymous ex- pressions adhering to both image and text contexts of the target object are gen- erated augment REC datasets. VIE-DM consists of vision-text condition (VTC) module and transformer decoder. Our VTC and token selection de- sign effectively addresses the feature discrepancy problem prevalent in existing REG methods. This enables us to generate high-quality, diverse synonymous ex- pressions that can serve as augmented data for REC model learning. Extensive experiments five datasets demonstrate the high quality and large diversity of our generated expressions. Furthermore, the augmented image-expression pairs consistently enhance the performance of existing REC models, achieving state- of-the-art art results. The source code available at https: /titp:://gthub. com/ freedom6927/ /wt.:////EVEEE -V1.:/VVVED........
INTRODUCTION
Joint vision-language representation learning has become burgeoning research focus in computer
vision, with diverse applications such referring expression comprehension (REC) Su et al.
(202));
Yang et al. (2023): : Su et al. (2024). referring expression generation (REG) et al. (2023) referring
expression segmentation Sun et al. (2023) ; Liang et al. (2022). visual question answering Dancette
et al. (023), image captioning Luninngguuuu Luo et al. (202)). and scene understanding [eng et al. (202). Among
these applications, REG and REC are essential and interconnected components joint visual-textual
reasoning, enabling the identification of target objects specified by given expressions.
RBG aims to generate precise textual description of a specific object within an image, while REC identifies the corresponding object in the image based on a given expression. However, REC mod- els frequently encounter difficulties with novel referring expressions due the scarcity of image- expression training pairs. To mitigate this issue, we explore its complementary REG task by intro- ducing a diffusion- based method that generates multiple diverse referring expressions for target object. These generated expressions can be employed to augment training data for REC models. Meanwhile, existing REG methods such as Mao et al. (2016); Yu et al. (016 2017); Liu et al. 2017 y020;; Tanaka et al. (2019); Niu et al. (2021); ; Sun et al. (2023) are developed based on waanformer-LSTTe or CNN-LSTM frameworks. These methods directly input visual features from transformers or CNNs into the LSTM for expression generation. However, they do not effectively bridge the gap between the visual and textual domains. addition, existing REG methods are deter- ministic and fail to generate diverse expressions needed fulfill our goal, expression augmentation.

---

Published as a conference paper at ICLR 2025
REG
Vision-Text Condition
Denoise
Image and Target Object
X
Rounding
notso
Man in white and standing
A ugmentation
Person in hat and white
REC Dataset
Testing
Training
Synthesized Expressions
Man in bluee
REC Model
Image-Expresee Pair G Generation
theright
REC
Our vision-guided expression diffusion model for referring expression generation. The proposed vision-text condition (VTC) module takes an image and the target object as input. integrates these visual conditions into the diffusion process using cross-attention at each denoising step, and can generate diverse and high-quality referring expressions that tightly align with the visual context of the target object. The generated image-text pairs then used to augment the training data to improve the REC performance.
Figure 1:
The diversity of synthesized samples is vital attribute for generative model Sonn et al. (202 Ho et al. (2020), as determines the spectrum of possibilities the model can encompass and predict. A model exhibiting high diversity can generate wide variety of samples, effectively capturing the full spectrum of features the data. In the past, certain machine translation methods, such as back translation Gu et al. (2018); (Saharia al. (2020), using non-autoregressive language models (LMs) have shown the ability to produce diverse sentences. However, these methods are tailored for speech and translation contexts, where the entropy among valid outputs is typically low. It has been demonstrated that these approaches are ineffective for language modeling Ren et al. (2020). Recently, text diffusion models Gong et al. (2023) He et al. (2033 have gained widespread adoption in sentence generation tasks owing to their interpretability and substantial diversity. These models are used in various applications, such text simplification, question generation, and paraphrasing.
As depicted in Figure 1 building on the success of text diffusion models and addressing the afore- mentioned REG challenges, we introduce novel method, Vision-guided Expression Diffusion Model (VIE-DM), for REG. The diffusion model in VIE-DM incorporates a vision-text condition- ing (VTC) module and a transformer decoder for conditional text generation. The VTC module implements two key strategies: cross-altention and token selection. In each denoising step, the cross-attention strategy is employed to infuse the visual features the image and target object into the noisy text features of the expression. Subsequently, the token selection strategy is applied to concentrate on important features of the image. After processing via the VTC module, the output is fed into the transformer decoder predict the noise and generate denoised features. This iterative process continues until the specified number of denoising steps is reached.
The VTC module facilitates enhanced alignment between the noisy text features and the target ob- ject's context compared to conventional CNN-LSTM or Transformer-LSTM frameworks, which are constrained by their model capacity. Additionally, this advantage enables the proposed method to generate more precise referring expressions for the target object in the REG task, outperforming ran- dom expressions produced by unconditional text diffusion models. Guided by the visual features of the image and the target object, we can generate diverse referring expressions adhering to the target object by randomly sampling group of Gaussian noise vectors followed by iteratively denoising steps using our model. The resulting image-expression pairs subsequently used to augment the training set and enhance REC model training.
This paper makes two primary contributions. First, the best of our knowledge, introduce the diffusion model to the REG task. By incorporating our vision-text condition module, the visual features of the image and target object more effectively aligned with the noisy text throughout the forward and backward diffusion processes. This enables the model to generate more accurate and diverse expressions for the target object in the image. Second, substantiate
the first to
2

---

Published as a conference paper at ICLR 2025
claim with comprehensive evaluation across five datasets, demonstrating that our diffusion model can effectively augment the REC datasets to consistently improve different REC methods.
RELATED WORK
Referring Expression Generation (REG). REG and image captioning share some similarities. REG generates a description for a target object, focusing on a particular region of an image, whereas image captioning produces expressions that deseribe the entire image. For REG, Mao et al. (2016) integrate deep learning into REG. They extract visual features and enable expression generation using LSTM. Additionalyy, they introduce Maximum Mutual Information (MMI) training strat- egy, which has become a standard approach in many REG models. To better bridge the visual and textual domains, some studies include additional annotations that detail the visual and positional differences among objects. For instance, Liu ct al. (2017) 2020) use attribute labels to establish connections between the objects in an image and the corresponding expressions before generation.
Recently, some methods Yu et al. (2017; Sun et al. (202)) consider REC and REG jointly. Yu al. (2017) introduce Seistener-Speaker system, where the Speaker module generates expressions guided by REC model, while the Listener module identifies the object in the image based on the provided expression, [Sun et al. (2023) utilize two types of cross-attention modules to connect REC and REG: One combines the image and expression (REC), while the other integrates the image and target object (REG), Subsequently, the outputs of the cross-attention modules are fed into a fusion module to determine the target object's 's location and generate synonymous expressions.
Existing REG methods commonly employ CNN-LSTM or Seiffrmmrr-STTY, frameworks for ex- pression generation. However, they often struggle to capture complex attributes within expressions due to the deterministic nature and limited model capacity of LSTM, thereby limiting data diversity. While existing REG methods employ the Maximum Mutual Information (MMJ) training strategy to distinguish the target object from other irrelevant objects in the image, MMI is only suitable for deterministic methods and cannot be directly applied diffusion models. In contrast, our method introduces an image-guides text-diffusion model for REG. Leveraging the stochastic advantage of diffusion models, our approach significantly enhances the quality and diversity of generated expres- sions. Furthermore, we introduce a token selection strategy that helps our method better capture essential features, thereby reducing the impact of irrelevant features in the image.
Diffusion Models for Sentence Generation. Several studies have explored conditional diffusion models for sentence generation. For instance, Diffusion-L.M [L et al. (2022) the pioneering work in text diffusion modeling, focused on developing language models with complex and fine-grained behavior controls. Unlike Diffusion-LM., DiffusER Reid et al. (023) integrates Levenshtein oper- ations into its diffusion model and generates more linguistically diverse sentences while preserving the original semantics. Lin et al. (023) further introduce a large-scale pre- pre-trained diffusion lan- guage model comprising encoder and diffusion- based decoder. IZheng et al. (2023) propose a reparameterization strategy that enhances the efficiency of discrete diffusion probabilistic models in natural language generation. [oong al. (2023) propose end-to-end classifier-free condi- tional diffusion model that combines the Seq2Seq framework with the diffusion model for sentence generation. He al. (2023) present DiffusionBERT, which investigates the training of BERT to understand the reverse process of the diffusion model within the discrete domain.
However, existing conditional text diffusion models only use additional text descriptions generate sentences and do not consider using images as condition for the REG task. In this paper, we introduce VIE-DM for the REG task. Guided by both the image and the target object, the proposed VIE-DM generates high-quality and diverse synonymous expressions. To the best of our knowledge, we make the first attempt to implement conditional text diffusion models for the REG task.
Transformers for Referring Expression Comprehension (REC). Recent state-of-the-as REC al- gorithms predominantly rely on transformers. Existing REC models often struggle to generalize to unseen referring expressions, they are trained on fixed image-expression pairs. Although SelfEQ He et al. 2024) attempts to paraphrase the expressions by synonym replacement for aug- mentation, it exhibits limitations in the diversity and complexity of the generated maage-expressioon pairs for REC, as acknowledged in its paper. To address this issue and further improve the perfor- mance of REC methods, we augment the REC training set with the image-expression pairs generated
3

---

Published as a conference paper at ICLR 2025
Training
Crus- Alteation
Tokmn Selection Strategy
Vision Encoder
Tanget Tokens
Cross- Attention
Image and Thrget Objectt
Image Tokens
VTC
Notey Text Tiokens
Gausdan Noise
Prodicted Noise
Test imput: moot"
Text Encoder
dddngg
Sampling
Encader
V Vision
VTC
Transformer Decoder
VTC
Transformer
Decoder
Rounding
Target Objoctt
Noisy Feature
1-st strp denolsing
stop denolsing
Training and sampling phases of our proposed VIE-DM. During training, noisy text tokens of the expression and the visual tokens the image and the target object serve as the input the VTC (vision-text condition) module. The VTC module compiles noisy vision-text features, which are passed to a transformer decoder for noise prediction at each denoising step. During sampling, we generate new expression by sampling random noise conditioned on the visual features of the given image and the target object through VTC followed by the transformer decoder for denoising. The noisy vision-text features undergo iterative denoising. Ultimately, we use the rounding operation Reid et al. (2023) to decode the denoised features and yield new expression.
Figure 2:
by our method. Transformer-based REC methods are then applied the augmented dataset during both training and testing.
METHOD
Figure illustrates our REG framework, detailing both the training and sampling phases. Our pro- posed VIE-DM denoising network mainly consists of (1) vision-text condition (VTC) module to condition the noisy text features on the embedded visual features of the entire image and the patches of the target object and (2) transformer decoder to take the interacted feature for noise prediction at each denoising step. Our VIE-DM is elaborated as follows.
3.1 DIFFUSION MODELS
Diffusson models are latent variable models, involving a forward and reverse diffusion process. Given a sample Zo ~ q (xo), the forward process entails producing a Markov chain of latent vari- ables T1, By by iteratively introducing small increments of Gaussian noise to the sample:
q (21(xr111 =N (zz; /1 B,
(1)
where I is identity matrix, and {3: (0,1)) represents the variance schedule that governs the step size for introducing noise. Following T steps, 5T converges to an isotropic Gaussian distribu- tion. If Bg sufficiently small, the reverse process g (ax-1 (w+) can be approximated as a Gaussian distribution, which can be learned using parametric model:
Po (1:,)((a;)) = N (x (xt, 1): Eg ()),
(2)
where μı and Eg are trainable parameterized models. When conditioning 2o, q (11-1/11,,00 2o) has a closed-fome foredd--oommooo solution, the corresponding objective can be simplified
LDM Ez..s~N(00.. [[l= Eo (Vā,z + /1 ā,e,t) t) 1
(3)
where & E {1, T3 denotes the time index of each denoising step, N (0, I) is Gaussian noise, Eo is the function for noise prediction from It, 0z Br, and I1

---

Published as a conference paper at ICLR 2025
3.2 PRELIMINARY
Given an image of size H x W x 3, an image patch of the target object of size Ho x Wa x 3. and expression comprising L words, feed the entire image and the target object into a vision transformer, resulting in two groups of visual tokens: v {u:)== o4 E Rd for the image and o {o.)==1 E RR for the target object, respectively. The expression undergoes processing in BERT yield text tokens representing the words, denoted by
{ex {- In addition, we employ an embedding function EMB and normal distribution to transform the discrete representations into continuous ones Zg. i.e., Zg
EMB (e), by following Li et al. Li al. (2022).
3.3 FORWARD AND REVERSE PROCESSES FOR EXPRESSION DIFFUSION
Forward Process. Let the visual guidance from the image and the target object be the condition с = J, v]. At the (-th forward step, the noisy text feature Z, is generated by adding noise to z through q (Zr|z:-1, c) = N (zrlc; vI B, |c, B,1), where {3: are the noise schedule.
Reverse Process. The reverse process involves recovering the original Zo by iteratively denoising the noisy text features from 2r ~ ~N (0,1) under the guidance of the entire image and target ob- ject. The reverse process is also defined
Markov chain and its joint distribution is formulated as Do (Zo:rle) := p (ZT) II=- Pe (zz1/ c). 2 is sampled according to Pe (2,-1 c)
N (Ze-1i (Ze, t), Zg (Zz, c, c,)),, where μıl and De parameterize the predicted mean and variance of po (z:-2 Ze c), respectively. tioned earlier and modeled by our denoising network consisting of vision-text condition (VTC) module and a transformer decoder, Eo (VTC o, v). Before elaborating on our approach's 's training and sampling processes, we introduce the proposed VTC module below.
can be further reduced to noise prediction similar to what men-
3.4 VISION-TEXT CONDITION (VTC) MODULE
To integrate the vision guidance into the text diffusion model, we propose the VTC module to fuse the noisy text features with the visual features of the image and the target object before noise pre- diction. VTC consists of two modules: the cross rross-attention module and the token selection module.
First, two noisy lision-language features are obtained at the t-th step: (i) z {2",) i=1 are com-
puted by cross attention between Ze and v to condition on the entire image. (ii) 2 = { are also computed by cross attention between 2, and o to condition on the target object. Next, we introduce token selection module aimed at enhancing the features associated with the target object within z,. Specifically, we calculate the cosine similarity between the features of z and z,. result- ing in the following score matrix S [sty] E R L.x. where the i-th row of S denotes the similarities between the i-th feature in z and all features in zP- We then sum up the scores within each row of
S to obtain set of scores, denoted by S [ .... sT.TT 8f >= E we consider =5, as highly relevant to the target object and extract from z Suppose there are noisy vision-language features extracted from zi", and zH. is among these kt noisy vision-language features. For 25, we also update the corresponding noisy vision- lisoon-language feature for the target object as
<S,, Ela
where 3/z exp(s;;)
Finally, the noisy vision- language features &y,t, resulting from combin- ing both ai,, and zi, can be acquired by
El_ exp(x)))
2 W [35 2f,4]
(5)
where W is trainable parameter matrix. Repeat the process in 4 and (5) until all features ex- tracted from z are updated. The updated features (i.e., 2,,) will form 2,, which then fed into the transformer decoder for noise prediction.
3.5 TRAINING AND SAMPLING
Training. In the training phase, according to the additivity of Gaussian noise, the training process can be simplified follows. Given the embedding feature 2o of the expression, we introduce Gaus-
5

---

Published as a conference paper at ICLR 2025
sian noise .V (0, I) to Zo to acquire the noisy feature z, for the t-th step in the forward process. The noisy feature Lg, along with the visual features of the image v and the target object o. are fed into the VTC module to extract the noisy vision-text features. The noisy features subsequently fed into the transformer decoder for noise prediction. Specifically, the training objective is formulated
24 Va + v1 CgC,
RERG = E., v.e [[]= se (z,, v)
Existing literature Ho & Salimans (2021) shows that classifier- free guidance contributes to improv- ing the balance between diversity and accuracy. This enhancement aids the diffusion model in gen- erating accurate and diverse outputs. Within the realm of classifier-free guidance, the function Eo (VTC (ze, o. v), t) in Eq. can be modified as Eg (VTC 0, = Eo (VTC (2, .) + (eg (VTC (Ze, o, v).t) - -eg (2,, ()),
(7)
where y is the guidance weight. The proportion of samples for unconditional process is set to 0.2. In this case, training objective is formulated as
REG V. [||e E (V.C
Sampling. the sampling phase, we first randomly sample the initial latent Zr ~ N (0, I) and feed Zr together with the visual features, v and into VTC followed by the transformer decoder for noise prediction to sample -1- 1. Then, we repeat the same procedure until reaching 2o as outlined below:
1 00 2 VI- a Va
4-1
Eo (ZT
oen,
where t = T...., 1, oy represents the standard deviation at the t-th denoising step, and n ~ ~N (0.I). I). In classifier-free guidance, 7 is also sampled using Eq. 9. where Eg is replaced by En in Eq
Finally, we adopt the rounding operation in Reid et al. pression. Moreover, we use the Minimum Bayes Risk (MBR) decoding strategy Koehn (2004) to enhance the quality of generation, which generates set of candidate samples by applying different random seeds to our model. The final output sequence is selected based on the minimum expected risk, as determined by relevant loss function (e.g., Meteor).
2023 to decode the denoised Zo into ex-
3.6 DATASET AUGMENTATION FOR REC
Suppose there are M images in the training set of the REC dataset, and our VIE-DM generates T expressions for each image and target object, with each generated expression assigned score (e.g., Meteor, CIDEr, or other metric scores) denoting its semantic similarity to the GT expression. Subsequently, the expression with the highest score is selected for each image and its target object, yielding an image-expression pair. The score of the i-th mmaae-expressionn pair is denoted as 9:- Next, sort the M scores {gi ) and select the top h image-expression pairs to integrate them into the this work. Finally, we employ existing conssormer-bssee REC
training set, where = [0.3M] methods (i.e., Trans VG, M-DGT, VLTVG, QRNet, MDETR and OFA-base) both augmented and unaugmented datasets to evaluate the quality and diversity of our generated expressions.
EXPERIMENTAL RESULTS
In this section, we evaluate the performances of the proposed and existing methods for the REG task. Since the REG task and image captioning share some similarities, we will analyze and present ex- perimental comparisons of these two tasks the supplementary material. In addition, we integrate our generated expressions into different REC datasets for model training and evaluate the perfor- mance gains for the REC task to validate the effectiveness of the proposed approach. The details are described in the following sections.
4.1 DATASETS AND EVALUATION METRICS
Datasets. We evaluate our method on RefCOco Kazemzadeh et al.] (2014). Ref-
COCO+ Kazemzadeh et al. (2014). RefCOCOg Mao al. (0166), Flickr30k enetities Plummer

---

Published as a conference paper at ICLR 2025
Table 1: REG performance comparison among VIE-DM and existing REG methods RefCOco, RefCOCO+, and RefCOCOg. CFG and MBR represent classifier-free guidance and Minimum Bayes Risk, respectively.
RefCOCO
RefCOCO:
RefDOCOO
Model
IestA
Test8
lesil
val-g
Meteor
CIDER
Meteor
CIDEr
Meteor
CIDEr
Meteor
CIDEr
Meteor
CIDER
Speaker-MMI (baseline)
Mao of
X016
VGGi6
0.243
0.615
0.300
1.227
0.199
0.462
0.139
0.679
0.149
0.585
Speakar+visdiI-Ma [+MI
VGG16
0.260
0.679
0.319
1.276
0.202
0.475
0.196
0.683
0,147
0.573
Speaker-listeneerrrfffie MMM dl
2017
VGG16
0.268
0.679
0.329
1.323
0.204
0.494
0.202
0.709
0.154
0.592
SpeakerHalt-MMil] Liu el 2017
VGG16
0.274
0.710
0.313
1,257
0.219
0.512
0,203
0.704
0.157
0.639
VC W/Gen Niu al [04
VGG16
0.188
0.707
0.245
1.356
0.142
0,518
0.146
0.731
0.139
0.625
2020
VGG16
0.312
0.802
0.332
1,301
0.236
0.585
0,206
0.692
0.163
0.645
PFOS al. 2025
VGG16
0.292
0.829
0.319
1.281
0.237
0,661
0.198
0.740
0.157
0.722
VIEDIM w/o MBB
VGGT6
0.358
1,09
0.383
1.762
0,366
1.147
0.393
1.276
0,492
1.127
VIE-DM w/o CPG
VGG16
0.363
1.075
0.577
1.775
0.374
1.152
0.411
1.269
0.503
1.135
VIE-DM
VGG16
0.373
1,782
1.160
Spedkeriiseneeecciiineeebbbb Yu el 2017
ResNet101
0,296
0.801
0.341
1,358
0220
0.579
0.221
0.798
0.153
0.742
Speakcr-scinlloggeensiiliM--M Tannka
ResNet10]
0.307
0.343
0.812
0.738
wwwpriiinncceeeioorr---sii-----i Tasaka et a.
2019
ResNet101
0,310
0.859
0.342
1.375
0.241
0.663
0.225
0.812
0.164
0.763
SFOS/Sun et 2023
ResNet101
0.303
0.330
1.333
0.722
0.210
0.758
0.749
MiniGPT-yZ et all 2023
VIT
0:417
1.038
0.125
1.767
0.405
1.124
0.413
1.317
0.612
1.129
Wang et
0.348
1,042
0.379
1.566
VIE.DM wio MBRR
ResNettol
0,432
1,187
0.448
1.966
0.439
1,516
0,474
1.652
0.694
1,535
VIE-DM w/o CPG
ResNet10]
0,439
1.193
0.46
2,085
0.446
1,338
0,485
1.675
0.712
1.546
VIE-DM
ResNet101
0.445
1.307
0.472
2,014
0.453
1,543
0.491
1.663
0.735
1,558
et al. (013)) and Keefclef[azeezaaate et al. (2014) to assess its effectiveness. Due to space limita- tions, detailed descriptions of these five datasets are provided in the Appendix.
Evaluation Metries. BLEU Papineei et al. (2002), Meteor Lavie & Agarwal (2007). Rouge Lin (2044) and CIDEr Vedantam al. (215)) are four widely used evaluation metrics for the image captioning and REG. Among them, Meteor and CIDEr take better consideration in terms of syn- onym matches and sentence structures than BLEU and Rouge. Thus, we employ Meteor and CIDEr to evaluate the generation diversity and quality of the proposed approach. Additionally, we employ metries Div-1 [Shetty et al. (2017), Div-2 Shetty et al. (017), and mBLEU-4 /hhtty et al. (2117) to assess the diversity of our generated expressions. Note that the value range of Meteor, Div- 1, Div-2, and mBLEU-4 are between 0 and 1. We follow the standard protocol to evaluate the REC task using the mAP@1 1 metric. When the Intersection over Union (IoU) between the detected object and the ground truth is greater than 0.5, this prediction is correct.
Implementation Details. We employs VTC module with two independent cross-attention layers followed by a transformer decoder, where the decoder consists of 12 self-attention layers and 12 attention heads for each layer. The established model comprises 1.2 billion parameters. For REG training on the RefCOCO dataset (120,624 image-expression pairs), the network with classifier- free guidance takes approximately 86 hours on 4 NVIDIA V100 GPUs The inference time using one NVIDIA V100 GPU is 8.74 seconds per image with DDPM and 0.93 seconds per image with DDIM. Like position embedding, time step embedding is incorporated. Furthermore, the maximum sequence length is set to 128, with an embedding dimension d 128. a square-root noise schedule, and 2000 diffusion steps. The batch size and learning rate are set to 8 and 1c-04, respectively. The parameter y is set to 3.0 empirically. The optimizer, vision encoder, and textual encoder employed are A, VIT/B-32, and BERT, respectively. Regarding the accuracy metrics using Minimum Bayes Risk (MBR) Koehn (2004), we utilize set of 10 candidate samples.
4.2 PERFORMANCE EVALUATION RESULTS
We first compare our method with other state-of-the-art REG methods regarding the fidelity of the generated expressions on the ReíCOCo, RefCOCO+, and RefCOCOg datasets in Table Note that "visdif" and "reinforcer" are two modules introduced in Yu et al. (2116) and Yu et al. (2117) respectively, while "attr" and "attn" are the other two modules proposed in Liu et al. (2020). With the VGG16 features, all methods are CNN-LSTM-based frameworks except PFOS. We observe a subpar performance in baseline, especially in Meteor, due to the limited model capacity of the CNN-LSTM models. Incorporating the "attr" and "atten" modules better bridges the gap between the visual and textual domains than the vanilla CNN-LSTM models. PFOS is the sole method based on the transformer-LSTY, framework. However, its potential for improvement is constrained since it doesn't 't introduce additional information to bridge the gap between the vision transformer and LSTM. MiniGPT-v2 stands out all-in-one approach that delivers impressive performance
7

---

Published as a conference paper at ICLR 2025
Table 2: Experimental results of existing REC methods on the original and augmented RefCOCo, RefCOCO+, ReiCOCOg, Flickr30k, and Refclef. Here, CC |Sharma et al. (2018), SBU Ordonez et al. (201T) J. COCO Pont-'Tuset & Van Gool (2013), Krishna et al. (2017) and Flickr30K PPlum- mer et al. (2015) are the datasets used for pre- trainin.. Better results are marked in bold. CO and
F30K represent COCO and Flickr30K, respectively.
RefCOCO
RefOOCO+
RefCOCOp
Flickr
Refelef
Methods
Pre train Dataset
lestA
lestB
val
lestA
testB
val
test
Deng al.
2021
CO
81.02
82.72
78.35
64.82
70,70
59.64
68.67
67.73
79.10
70.73
VGaE-D-
CO
83.75
84,58
81.13
67.09
72.52
62.67
70,93
69.65
81.54
72.37
M-DGT Chen & Li
2022
None
85.37
83.01
85.24
70.02
72.26
68.92
79.21
79.06
79.97
M-DGT-VIE-DD
None
86.67
84.35
86,43
70.28
73.35
70.18
81.48
79.95
80.57
VI.TVG Yang et 2022
CO
84:77
87.24
80.49
74.19
78.93
76.04
74.18
79.84
71.98
VLT VI
CO
86.58
88.83
82.59
76.42
80.93
67.82
78.83
76.08
81,28
73.37
QRNei Ye
2022
CO
84.01
85,85
82.34
72.94
76.17
63.81
73.03
72.52
81.95
74.61
QRNet+VIE-DM
CO
86.82
88,50
84.66
75.35
77.89
66.25
74.86
74.67
83,58
76.28
MDETR Kamath al.
2021
CO. VG, F30K
86.57
89.58
81.41
79.52
84.00
09
70.62
81.64
$0.89
83.80
MDETR&VIE-DM
CO, VG, F30K
90.53
92.15
85.95
82.84
86.83
73.35
83,80
83.56
86.08
hase Wang et al.
2022
VG
88.48
90.67
83.30
81.39
87.15
74.29
82.29
82.31
OFA-bised VIE
VG
90.53
91.75
85.63
89.65
76.20
83.98
Table 3:
Diversity comparisons among VIE-DM and other REG methods. Larger values mean higher diversity in all metrics except for mBleu-4, where lower values are preferred. SM. SLRVM and SLRM represent SpEaker+MMI, seeekeellieenctrcinfor---iiiiiii Senaaerreiiseeerrriiooooe---MM- respectively.
and
Ref0O00
RefCOCO+
RefCOCO+
Model
testB
ISSIA
tostB
val-g
mBI.F
Divl
Div2
mBI
FU-4
Divl
Div2
mBI.F1144
Divl
Div2
mBI.FU14
Divl
Div2
Divil
Div2
SM
0.473
0.357
0.423
0.452
0.374
0.450
0.465
0.343
0.435
0.458
0.363
0.471
0.377
0.412
0.468
SLRM
0.576
0.338
0.393
0.554
0.353
0.407
0.533
0.298
0.355
0.529
0.330
0.408
0.485
0.363
0.392
SLRVM
0.595
0.327
0.407
0.538
0.385
0.395
0.603
0.296
0.363
0.556
0.348
0,383
0.506
0.366
0.409
VIE-DM
0.526
0.276
0.509
0.522
0.247
0.256
0.308
0.567
0.634
across various downstream vision-language tasks. As presented in Table 1 our VIE-DM outper- forms MiniGPT-v2 in a favorable manner. Moreover, MiniGPT-v2 is based on which includes at least 7B parameters, while VIE-DM only includes about 1.2B parameters. In the section Appendix, we will provide an example that demonstrates a comparison between our method and MiniGPT-v2. In addition, comparing the methods using VGG16 features with those using ResNet101 features, the ResNet101-based approaches yield better performances.
The proposed VIE-DM consistently outperforms the state-of-the-arr REG methods utilizing both ResNet101 and VGG16 backbones. Specifically, in testA of RefCOCO, VIE-DM surpasses the method with ResNet101 by 0.135 and 0.348 in the Meteor and CIDEr metrics, respectively. When comparing VIE-DM under different conditions, we find that the performances of VIE-DM generally surpass those of VIE-DM without MBR and CFG in most cases. These results demonstrate the high quality of our VIE-DM.
We utilize generated expressions by VIE-DM to augment five REC datasets and apply six transformer-based REC methods to both augmented and unaugmented datasets, aiming to evaluate the diversity and quality of our generated expressions. As reported in Table 2 our method con- sistently enhances the transformer-base method doesn't 't rely on pre-training (e.g., with M-DGT), the performances across the five augmented datasets remain stable. For the methods pre- pre-trained on additional datasets, there is a significant im- provement in performance using the augmented datasets, particularly for MDETR. In most cases, MDETR using the augmented datasets (i.e., MDETR+VIE-DM) outperforms the state-of-the-art transformer-basee based methods (e.g.. OFA-base). These findings further underscore the enhancement in diversity that our generated expressions bring to the original datasets. In addition, they also validate our method's effectiveness and potential to improve other transformer-bact REC methods.
REC methods on these datasets. Notably, even though the
We compare our method with other REG methods using metrics Div-1, Div-2, and mBLEU-4 to evaluate the diversity of generated expressions. As given in Table 3 when the "listener" and "rein- forcement" modules
added to the baseline Speaker+MMI model, the generated expressions can better match the ground truth but with poor diversity. In contrast, combining these results with those in Table 1 our methods not only produce high-quality expressions but also exhibit strong diversity.
8

---

Published as a conference paper at ICLR 2025
Table 4: VIE-DM for REG without the VTC module or without the token selection strategy.
RefCOCo
RefCOCO+
ReiCOCOg
Model
IostA
testi
lestA
testl
val-g
Meteor
CIDEr
Meteor
CIDEr
Metcor
CIDE:
Meteor
CIDE
Metoor
CID
VIE-DM w/o
1.512
0.383
1,308
0.376
1.382
VIE-DM w/o token selection strallggy
0,426
1,163
0.450
1.907
0.426
1.466
0,503
1.592
0,702
1,502
VIE-DM
0,445
1,207
0.472
2014
0.453
1.343
0.491
1.63
0.735
1.588
Table 5: Experimental results of QRNet on the augmented datasets by including 10%, 30%, 50%, and 70% of the generated image- expression pairs into the training sets.
Table 6: The results of QRNet are shown for two scenarios: randomly selecting 30% of image-expression pairs (RS), and sorting and selecting the top 30% of image-expression pairs (SS)
Amount of augmented pairs ReiCOCO ReCCOCO+ ReiCOCOg
10% 30% 50%
70%
testA testB testA testl
lest
case RefCOCO RefCOCO+ RefCOCOg
77.58
73.35
88.50 84.66 77.89 66.25
74.67
RS SS testA testB testA testB
test
85.53 85.82 75.60 63,78
70.25
V
87.14 83.88 78.03 65.12
73.54
84.98 85,23
77.85 62.93
70.65
88.50 84.66 77.89 66.25
74.67
4.3 ABLATION STUDIES
In the following, we use RefCOCo, RefCOCO+, and RefCOCOg to conduct the ablation studies. We first perform unconditional generation without employing the vision-text condition module by directly feeding the noisy features into the transformer decode for noise prediction and sampling- As shown in the first row of Table 4 the performances of VIE-DM without the VTC module are noticeably lower than those with the VTC module. These results indicate that, with the guidance of images, VIE-DM can generate more accurate and diverse expressions for the target objects.
Then, we validate the effectiveness of the token selection strategy exploited by the VTC module. As shown in the second row of Table when the proposed VIE-DM takes all tokens from the image into consideration without performing this strategy, it results in inferior performances to those using this strategy in most cases. This outcome demonstrates that proposed method can effectively select the tokens from the image containing more relevant information about the target objects.
To evaluate the efficacy of the augmented dataset by varying the number of REG-augmented sam- ples, we randomly sample 10%. 30%, 50%, and 70% of image-expression pairs generated by VIE- DM and incorporate them into the training sets of the original REC dataset. We also employ QRNet as the baseline and evaluate its performance on the four augmented datasets. The results in Ta- indiiiie progressive improvement the performances of QRNet when 10% and 30% of the augmented
pairs are incorporated. The performance hits the plateau when more than 30% of the generated image-expression pairs are included. We attribute this two factors: first, the diversity of expressions already reaches a sufficient level, and second, higher augmentation percentages introduce potentially inaccurate pairs, which negatively impact the method.
We also evaluate the efficacy of the augmented dataset generated by VIE-DM in two scenarios. First, we randomly select 30% of image-expression pairs from VIE-DM and integrate them into the training of the original REC dataset. Second, sort and select the top 30% of pairs with higher Meteor scores and integrate them into the original REC dataset. As depicted in Table 6, we observe that the performances of QRNet on the augmented dataset superior in the second case compared to the first one. These findings not only demonstrate the rich diversity of expressions generated by VIE-DM but also indicate that the expressions it generates, especially those with higher Meteor scores, provide more accurate descriptions of the target objects.
4.4 VISUALIZATION
In addition to presenting quantitative results, we also provide visual illustrations of our method in Figure The first and second rows exhibit the accurate outcomes yielded by our method, while the third row illustrates instances with lower Meteor scores. Speifically, the first two rows illustrate the effectiveness of our method in generating accurate and diverse expressions, where the noun phrases in the original expressions are usually replaced with synonyms while the sentences are sometimes restructured. Among the observed patterns, the faithful image-tex relationships can be attributed to the proposed VTC (vision-text condition) module, which ensures precise alignment between visual and textual tokens. Meanwhile, the transformer decoder and diffusion model contribute to high

---

Published as a conference paper at ICLR 2025
Refermnne Person standing
Refermnee fllu right guy Recowar Blue rightside
Refermmce Yellow guy Recover: Yellow shir:
Refevenoe Cuty with Recwer The Middle player
Refermnce Left girl
Reoaer:: Man standing
Remner: Left female
Referencee green leather
Refexences with bay on the left of man in while Recavers Man cortes bag on
Reference:
Man ridinn
Refermnce
oouch living room
pank dress drinking milk Recaven small garl in pink dress having food
hicycle
the left man in back Reuwven Man in white in the middle
Recovars green desk chair hotel
Reuavnn
Man Erunt
Referenne Girl with phone Recevr:: with
Reffevence motorcycle tider Recover: baby
Refermsce The catcher Recovar: The baby
Referencee Wontat holds wine Recowe:: Man holds the food
Reference lile dog In cage Recowers White toy in bag
Visualization of VIE-DM. The first and second rows show the accurate results of VIE-DM in short and long descriptions, while the third row shows the inaccurate results.
Figure 3:
diversity through synonym replacement and sentence restructuring. Regarding the first sample of the third row, although our VIE-DM results in an expression ("Girl with hat.") different from the reference ("Girl with phone."). does capture the phenomenon that the girl wears something. For the second sample, since the stripe pattern on the shirt worn by the rider shares some resemblance with the zebra, VIE-DM makes an inaccurate expression generation. Lastly, the third sample, due to the small patch size providing limited information, the proposed approach results in erroneous generation. While our method may generate some inaccurate expressions, we have implemented a mechanism to effectively eliminate these inaccuracies by utilizing metrics such as METEOR during the process of augmenting the REC dataset.
4.5 DISCUSSION
In dataset augmentation, some studies such Yuan et al. 2024): Zhang al. 2023) have success- fully employed diffusion models. Nevertheless, using diffusion models in vision-guided expression generation for the REG and REC tasks remains unexplored and non-trivial. Specifically, Yuaa al. 2024); Zhang et al. (2023) and other diffusion-based data augmentation methods do not explic- itly handle the misalignment issues between the vision and language modalities for the REG task. However, our proposed vision-text condition (VTC) with the token selection strategy effectively ad- dresses the critical issue of the REG task. helps the model generate expressions that faithfully capture the attributes of the target object in the image. As shown in Table of of the paper, neglecting the proposed VTC or the token selection strategy in the diffusion model suffers from substantial per- formance drops, which can be observed consistently on all five datasets in both Meteor and CIDEr.
5 CONCLUSION
In this paper, to the best of our knowledge, we make a pioneering contribution by introducing the diffusion models for referring expression generation. Our proposed VIsion-guided Expression Diffusion Model (VIE-DM) demonstrates its effectiveness pressions for target objects within given image. In VIE-DM, by incorporating vision-text con- dition module, we can efficiently extract visual tokens from the entire image that are relevant to target object and fuse them with the noisy text features. This integration aids in generating more accurate expressions for the target object in the image. Additionally, we illustrate that expressions generated by method significantly enhance the REC dataset. This augmented REC dataset, when utilized by state-of-te the-art transformer-bast REC methods, leads to improved performance. With these distinct advantages, method emerges powerful tool for the REG and REC tasks.
generating accurate and diverse ex-
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGEMENT
This work was supported in part by the National Science and Technology Council (NSTC) under grants 112-222-EE-449-000-MM
112-222-E-007--77-MMY,, and 113-2634-T-002-003. This work was funded in part by Academia Sinica under the grant number of AS-CDA-110-MEL and by MediaTek. The authors also thank National Center for nigh-performance Computing (NCHC) of National Applied Research Laboratories (NARLabs) in Taiwan for providing computational and storage resources.
112-222-E-001-001--MY-
REFERENCES
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Kr- ishnamoorthi, Vikas Chandra, Yunyang Xiong. and Mohamed Elhoseiny. Minigpt-v2: large lan- guage model as a unified interface for vision-language multi-task learning. In CoRR, 2023.
Sijia Chen and Baochun Li. Multi-modal dynamic graph transformer for visual grounding. In CVPR, 2022.
Corentin Dancette, Spencer Whitehead, Rishabh Maheshwary, Ramakrishna Vedantam, Stefan Scherer, Xinlei Chen, Matthieu Cord, and Marcus Rohrbach. Improving selective visual ques- tion answering by learning from your peers. CVPR, 2023.
Jiajun Deng, Zhengyuan Yang, Tianlang Chen, Wengang Zhou, and Houqiang Li. Trans VG: End- to-end visual grounding with transformers. In ICCV, 2021.
Shansan Gong, Mukai Li, Jiangtao Feng. Zhiyong Wu, and Lingpeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. In ICLR, 2023.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. Non-autoregressive neural machine translation. In ICLR, 2018
Ruozhen He, Paola Cascante-Bonilla Ziyan Yang, Alexander C. Berg, and Vicente Ordonez. Im- proved visual grounding through self-consistent explanations. CVPR, 2024.
Zhengfu He, Tianxiang Sun, Qiong Tang, Kuamning Wang, Xuanjing Huang, and Xipeng Qiu. Dif- BusionBERT: Improving generative masked language models with diffusion models. In ACL, 2023.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NIPS, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NIPS, 2020.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Car- ion. Mdetr Modulated detection for end-to-end multi-modal understanding. In ICCV, 2021.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. ReferltGame: Referring objects in photographs of natural scenes. In EMNLP, 2014.
Philipp Koehn. Statistical significance tests for machine translation evaluation. In ACL, 2004.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, al. Visual Genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 2017.
Alon Lavie and Abhaya Agarwal. METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. In ACL Workshop, 2007.
Xiang Li, John Thickstun, Ishaan Guirajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion- Im improves controllable text generation. In NIPS, 2022.
Chen Liang, Wenguan Wang, Tianfei Zhou, Jiaxu Miao. Yawci Luo, and Yi Yang. Local-global context aware transformer for language-guided video segmentation. In CoRR, 2022.
11

---

Published as a conference paper at ICLR 2025
Chin-Yevs Yew" Lin. ROUGE: A package for automatic evaluation of summaries. In ACL, 2004.
Zhenghao Lin, Yeyun Gong, Yelong Shen, Tong Wu, Zhihao Fan, Chen Lin, Nan Duan, and Weizhu Chen. Text generation with diffusion language models: A pre-training approach with continuous paragraph denoise. In CoRR, 2023.
Jingyu Liu, Liang Wang, and Ming-Haoe Hssan Yang. Referring expression generation and comprehen- sion via attributes. In ICCV, 2017.
Jingyu Liu, Wei Wang, Liang Wang, and Ming-Hsuai Yang. Attribute-guide attention for referring expression generation and comprehension. TIP, 2020.
Jianjie Luo, Yehao Li, Yingwei Pan, Ting Yao, Jianlin Feng, Hongyang Chao, and Tao Mei. Semantic-conditional diffusion networks for image captioning. In CVPR, 2023.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Mur- phy. Generation and comprehension of unambiguous object descriptions. In CVPR, 2016.
Yulei Niu, Hanwang Zhang, Zhiwu Lu, and Shih-Fu Chang. Variational context: Exploiting visual and textual context for grounding referring expressions. 7PAMI, 2021.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2Text: Describing images using million captioned photographs. In NIPS, 2011.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. In ACL, 2002.
Songyou Peng. Kyle Genova, Chiyu "Max" Jiang. Andrea Tagliasacchi, Mare Pollefeys, and Thomas Funkhouser. Openscene: 3d scene understanding with open vocabularies. In CVPR, 2023.
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In 2015.
Jordi ont--Tusttae and Luc Van Gool. Boosting object proposals: From pascal to coco. In ICCV, 2015.
Machel Reid, Vincent Josua Hellendoom, and Graham Neubig. DiffusER: Diffusion via edit-based reconstruction. In ICLR, 2023.
Yi Ren, Jinglin Liu, Xu Tan, Zhou Zhao, Sheng Zhao, and Tie-Yan Liu. A study of non- autoregressive model for sequence generation. ACL, 2020.
Chitwan Saharia, William Chan, Saurabh Saxena, and Mohammad Norouzi. Non-autoregressive machine translation with latent alignments. In EMNLP, 2020.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACI, 2018.
Rakshith Shetty, Marcus Rohrbach, Lisa Anne Hendricks, Mario Fritz, and Bernt Schiele. Speaking the same language: Matching machine to human captions by adversarial training. In ICCV, 2017.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
Wei Su, Peihan Miao, Huanzhang Dou, Gaoang Wang, Liang Qiao, Zheyang Li, and Xi Li. Lan- guage adaptive weight generation for multi-task visual grounding. In CVPR, 2023.
Wei Su, Peihan Miao, Huanzhang Dou, and Xi Li. Scanformer: Referring expression comprehension by iteratively scanning. In CVPR, 2024.
Mengyang Sun, Wei Suo, Peng Wang, Yanning Zhang, and Qi Wu. A proposal-free one-stage frame- work for referring expression comprehension and generation via dense cross-attention. TMM, 2023.
12

---

Published as a conference paper at ICLR 2025
Mikihiro Tanaka, Takayuki Itamochi, Kenichi Narioka, Ikuro Sato, Yoshitaka Ushiku, and Tatsuya Harada. Generating easy-to-ume to-understand referring expressions for target identifications. In ICCV, 2019.
Ramakrishna Vedantam, C. description evaluation. In CVPR, 2015.
Lawrence Zitmick, and Devi Parikh. Cider:
Consensus-based image
Ning Wang, Jiajun Deng, and Mingbo Jia. Cycle-consistency learning for captioning and grounding. In AAAI, 2024.
Peng Wang. An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma. Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: Unifying architectures, tasks, and modalities through a simple squence-to-sequeee
learning framework. In ICML, 2022.
Li Yang, Yan Xu, Chunfeng Yuan, Wei Liu, Bing Li, and Weiming Hu. Improving visual grounding with visual-linguistic verification and iterative reasoning. In CVPR, 2022.
Ziyan Yang. Kushal Kafle, Franck Dernoncourt, and Vicente Ordonez. Improving visual grounding by encouraging consistent gradient-based explanations. In CVPR, 2023.
Jiabo Ye. Junfeng Tian, Ming Yan, Xiaoshan Yang, Xuwu Wang, Ji Zhang, Liang He, and Xin Lin. Shifting more attention to visual backbone: Query-modulade refinement networks for end-to-end visual grounding. In CVPR, 2022.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expressions. In ECCV, 2016.
Licheng Yu, Hao Tan, Mohit Bansal, and Tamara I.. Berg. A joint speaker-listene--eiiforee model for referring expressions. In CVPR, 2017.
Jianhao Yuan, Jie Zhang, Shuyang Sun, Philip Torr, and Bo Zhao. Real-fake: Effective training data synthesis through distribution matching. In ICLR, 2024.
Manlin Zhang, Jie Wu, Yuxi Ren, Ming Li, Jie Qin, Xuefeng Xiao, Wei Liu, Rui Wang, Min Zheng, and Andy Ma. Diffusionengine: Diffusion model is scalable data engine for object detection. In CoRR, 2023.
Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation. In CoRR, 2023.
A APPENDIX
A.1 DATASETS
In the following, we provide concise overview of the datasets utilized in our studies for both training and evaluation purposes.
RefCOCO, RefCOCO+, and RefCOCOg: The RefCOCO and RefCOCO+ datasets comprise 142,210 and 141,564 expressions about 50,000 and 49,856 objects, across 19,994 and 19,992 im- ages, respectively. In both datasets, expressions are gathered through interactive game interface. Specifically, as described in Kazemzadeh et al. 2014). RefCOCO is divided into four sets: training, validation, testA, and testB. They contain 120,624. 10,834, 5,657, and 5,095 samples, respectively. The testA dataset primarily focuses on images with multiple people, while testB encompasses im- ages with various objects. Similarly, RefCOCO+ follows the same four-split configuration, with 120,191, 10,758, 5,726, and 4,889 samples for the training, validation, testA, and testB, respectively. In contrast to RefCOCO, the expressions RefCOCO+ do not include descriptions of absolute cation. For RefCOCOg, it is collected in non-interactive setting, featuring 95,010 lengthy expres- sions corresponding to 49,822 objects within 25,799 images. This dataset includes 80,512, 4,896, and 9,602 magee-xxpression pairs for the training, validation. and test sets, respectively. Notably, the RefCOCO, RefCOCO+, and RefCOCOg datasets are derived from the MSCOCO dataset Pontt Tuset & Van Gool 2015), which encompasses 80 object categories.
13

---

Published as a conference paper at ICLR 2025
Table 7: Quantitative comparison of REG performances of VIE-DM with DDIM and DDPM, spectively.
RefCOCO
RefCOCO+
RefCOCOg
Model
testA
testB
testA
testB
val-g
Meteor
CIDEr
Meteor
CIDEr
Meteor
CIDEr
Meteor
CIDEr
Meteor
CIDEr
VIF-DM DDIM)
0.435
1.213
0.467
1,984
0.457
1,502
0.485
1.642
0.707
1,533
VIE-DM (w/ DDPM)
0.445
0.472
2.014
0.453
1.543
0.491
1663
0.735
1.558
Man in blue on the right of woman red.
REG
Image
Target
Image Captioning
A group of people
Figure 4: Visualization of REG and image captioning.
The dataset comprises 31,783 images, each accompanied by five descriptive sentences. Each sentence refers to objects belonging to one of seven common object categories and an "other" category for objects not falling under these seven categories. The dataset partitioned into 425,831 training samples, 14,433 validation samples, and 14,481 test samples.
Flickr30K:
RefClef: The dataset is gathered using ReferltGame, featuring 19,997 images sourced from SAIAPR-12 12 dataset. It encompasses 54,127 training, 5,842 validation, and 60,103 test referring expressions.
A.2 ADDITIONAL ABLATION STUDIES OF OUR METHOD
As mentioned in Song et al. 2021 diffusion models working with DDIM typically achieve com- petitive performance while exhibiting higher efficiency with significantly reduced diffusion steps. To evaluate the performances of our diffusion model with DDIM, during the sampling phase, we substitute DDIM for DDPM for the denoising process, configuring the number of diffusion steps in DDIM to 500. The findings presented in jablenddemonstral that the performance of VIE-DM with DDIM is comparable that with DDPM. Considering the number of diffusion steps, it is evident that VIE-DM with DDIM exhibits higher efficiency.
A.3 COMPARISON BETWEEN REG AND IMAGE CAPTIONING
In REG, the input includes image and specific region of interest, while in image captioning, only the image is provided as input. REG generates expressions describing the highlighted region, whereas image captioning generates expressions describing the entire image. The visualization of the difference between the two tasks is shown in Figure
A.4 DIVERSITY COMPARISON BETWEEN OUR METHOD AND EXISTING REG METHODS
To further gauge the data diversity introduced by different REG methods, we compare the REC performances of the proposed method with other REG methods where we employ QRNet as the baseline and evaluate its performance on the four augmented datasets. These four aug- mented datasets are derived from Spealerriisnncceeinnoeee-issssie Yu et al. (2017), and our VIE-DM, respectively. Specifically, we collect 30% of samples from the image- expression pairs these four methods and integrate them into the original datasets for augmentation. The results reported in Table g Combined with the findings from Table and Table the paper, we observe that the performance of QRNet the augmented datasets of Mpe shows slight decrease, while the performances
Tanaka et al. (2019),
seeaker-iistner-reiiioore---MMM
14

---

Visdf+MMI Published as a conference paper at ICLR 2025
Table 8: REC performance comparison of QRNet four kinds of augmented datasets. The best results are marked in bold. SM, SLRVM and SLRM represent Speaker+MMI, senkerr-iseenn--rcfffre--iiiilii
and peeelecr-iistnncceefnfrrr--MMMM respectively.
ReYCOC0
RefCoCo:
RefCocog
Methods
Backbone
val
tostA
tesiB
val
lostA
lod
val
lest
ORNer 2016
ResNer101
12.94
QR NeN44SS Mao
Ro
83.35
83.87
51.56
72.25
76,83
63.53
72.76
71.28
QRNa-SLRY Lamia
2019
ResNec101
$4.46
82.57
73.16
76.84
63.25
72.85
72.84
MRNee+SLRMI Yu
ResNet101
$3.85
86.04
$2.75
72.85
75.82
62.83
73.53
72.08
QRNet-VIE-DM
Ro
86.82
88.50
$4.66
75.35
77.89
66.25
74,86
74.67
Man yellow the left.
blue trousers the right
Reference: The orange the table
Reference: Sheeg the foremost
Reference: skirt besides the wall
SLRVM: billboard
SLRVM: Women sitting with paper
SLRVM: A yallow ball
SLRVM: Black dog
SLRVM: man green
Man the left of woman
Women in blue pants on the right
Ours: Orange on left of dish
Sheep with black face
Green clothes besides the wall
Figure 5: Qualitative comparisons between the proposed VIE-DM (Ours) and SLRVM.
main stable on the other two augmented datasets based on and oppalcctliieeatrrrriffootttttttt However, average, the performances of QRNet using the augmented datasets by our VIE-DM exhibit an improvement of approximately 1.5%. These results demonstrate that our VIE-DM significantly enhances the data diversity while keeping the fidelity of the original dataset.
A.5 COMPARISON BETWEEN OUR REG METHOD AND SLRVM FOR EXPRESSION GENERATION
In the following, present visual comparisons between method and existing REG meth- ods. We choose to benchmark our approach against Speaker=Listener-Reinneee (SLRVM) Tanaka al. (2019), given its superior performance among existing REG methods with the provided source codes. The results are shown in Figure(5] In the first column, our method consis- tently generates more accurate and diverse descriptions for the target objects, compared to SLRVM, which focuses on the man's 's back and ultimately results in an inaccurate description. Similarly, in the fourth column, our approach excels in providing more precise and diverse descriptions for the target objects, while SLRVM crroneously identifies the sheep in front as a dog. Furthermore, in the fifth column, SLRVM incorrectly interprets the prop with a green skirt as the man in green.
A.6 EFFECTIVENESS OF THE DIFFUSION MODEL OF OUR METHOD
To evaluate the importance of the diffusion model in this work, we develop variant of our method. In this simplified variant, the embedded image and target object are directly fed into the token selection strategy, and the outputs passed to the transformer decoder. As shown in Table 9 performance decreases significantly without the diffusion model. This underscores the importance of proposed vision-guided diffusion model for generating high-quality expressions suitable for REC dataset augmentation.
A.7 VISUALIZATION BETWEEN OUR REG METHOD AND MINIGPT-v2 ON EXPRESSION GENERATION
In this subsection, present quality comparison between MiniGPT-v2 and our method for pression generation. In Figure 6 we observe notable differences the generated sentences. On the left side, MiniGPT-v2 produces ambiguous sentence it solely focuses extracting the
15

---

Published as a conference paper at ICLR 2025
Table 9: Comparison between method with and without diffusion model the RefCOCO dataset.
RefCOCO
Model
testi
Metcor CIDEZ
Metcor CIDEr
our methed wio Diffision
0.335
0.901
0.347
1.451
MiniGPT-v2 A person in red clothes.
MiniGPT-v2 Man with bag. VIE-DM: Man carries bag on left.
VIE-DM: Woman in red between two
Figure Results between MiniGPT-v2 and our VIE-DM.
target region for expression, disregarding the relationship between the target object and other ob- jects the image. Conversely, our method takes into account both the target object and the entire image simultaneoussly This approach enables method to generate more precise expressions for target object identification. Similarly, on the right side of Figure [6] we observe that our method outperforms MiniGPT-v2 by generating The expression generated by MiniGPT-v2 appears to be ambiguous, as it fails to clearly indicate which object is referring identify the target object with accuracy.
accurate expressions for target object identification.
contrast, method produces precise descriptions that specifically
16


Reviews:
[{"review_id": "OaMsensixj", "reviewer": "Reviewer_jswi", "summary": "The paper explores the integration of referring expression generation (REG) and comprehension tasks. To address challenges such as the scarcity of image-expression pairs in training data for REC and the limitations of the REG methods in bridging visual and textual domains, the authors propose a novel vision-guided expression diffusion model for REG. Extensive experiments demonstrate that the proposed method produces high-quality and diverse generated data.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper explores the potential of applying diffusion models to the REG task, an area that has been largely underexplored.\n2. The authors introduce a vision-text conditioning module and a token selection strategy, which significantly enhance the alignment between visual and textual information.\n3. Extensive experiments and ablation studies validate the generalization capability and effectiveness of the proposed method’s design choices.", "weaknesses": "1. In the visualization results shown in Figure 3, the response labeled as “recover” in the first sample of the third row appears to be an error, as does the response in the last sample of the same row. These results indicate that while the current method enhances diversity, it still includes some erroneous responses. How do you ensure the quality of the generated responses?\n2. It is intriguing that the ViT backbone of CLIP is considered as a unified vision encoder in MLLM. Could this architecture produce different patterns and further improve performance?\n3. The definition of CFG is missing in Table 1.\n4. While the paper provides extensive interpretation of the experimental results, it lacks an in-depth analysis of the reasons behind the observed patterns in the results.\n5. The writing is somewhat verbose. For instance, in Subsection 3.6, the second sentence is redundant as it repeats the information in the first sentence.\n6. Some equations could be improved; for example, Equations 9 and 10 differ by only one symbol.\n7. There are a few typos, such as a missing period on line 82 and an incorrect number on line 360.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IxUEYC4BP7", "reviewer": "Reviewer_5XtB", "summary": "This paper addresses referring expression generation (REG) and referring expression comprehension (REC). In particular, the paper proposes a method for REG that utilizes a language model with a diffusion model and experimental results for REC that augment the dataset with the REG method. The experiments are performed on five representative datasets, three RefCOCOs, Flickr30k, and Refclef, and show that the accuracy of the proposed method for REG is better than the existing methods and that the augmentation of the dataset by the proposed method contributes to multiple REC methods.", "questions": "The reviewer would like to receive responses from the authors about the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- To the best of the reviewer's knowledge, this is the first study to introduce language models using diffusion models into REG and REC.\n- The proposed method has been evaluated using multiple datasets and multiple ablation studies, and has shown a certain degree of effectiveness.", "weaknesses": "- The proposed method is composed of a straightforward combination of existing methods. The Cross-Attention and Token Selection Strategy that make up the proposed method, Vision-Text Condtion, are known to the community, and the Minimum Bayes Risk (MBR) and classifier-free guidance (CFG) that are ablated in Table 1 are not newly proposed in this paper. (In addition, the REG performance of VIE-DM w/o CFG is reported as an ablation study, but the author forgot to explain what CFG stands for, so it is only the reviewer's guess that CFG means classifier-free guidance.)\n- As mentioned in the Introduction, the existing methods compared in this paper adopt the transformer-LSTM or CNN-LSTM framework. In other words, the proposed method differs from other methods not only in that it formulates a language model using a diffusion model, but also in that it uses a transformer-based decoder. It is not clear how the combination of a transformer-based decoder and diffusion model outperforms only a transformer-based decoder. Without this comparison, it is not possible to show the effect of introducing a diffusion model into REG.\n- In line 416, it is claimed that “These results demonstrate the robust data diversity and quality of our VIE-DM.” However, it is misleading to make such a claim of diversity based on Table 1, which discusses the similarity to the ground truth using Meteor and CIDEr. The argument regarding Table 3 is more convincing, so this claim should have been made elsewhere.\n- As the authors acknowledge, Table 5 shows that augmenting the REC data using VIE-DM only leads to a limited improvement due to the accuracy of the synthesized expressions. Therefore, it is essential to show whether VIE-DM is superior to existing approaches. On the other hand, it is not clear how much accuracy is improved when the data set is augmented using methods other than VIE-DM. The idea of amplifying REC data using REG methods has already existed since [Mao+, CVPR 2016]. If it is not possible to show how much REC accuracy is improved when expressions are augmented using methods other than VIE-DM, it will not be possible to understand the extent to which this paper contributes to REC.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jz5Jpsd5aE", "reviewer": "Reviewer_yjk1", "summary": "This paper introduces the Vision-guided Expression Diffusion Model (VIE-DM) to address limitations in referring expression generation (REG) and comprehension (REC) tasks, particularly the scarcity and low diversity of image-expression pairs in existing datasets. The model includes a vision-text condition (VTC) module and a token selection mechanism to mitigate feature discrepancies between the visual and textual domains.", "questions": "See weakness.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Introducing a diffusion model to REG is innovative. VIE-DM generates diverse, high-quality synonymous expressions that align with both the visual and textual context of target objects, enriching REC datasets.\n2. The experimental design is well-structured, including ablation studies. Extensive experiments on five datasets demonstrate significant improvements in REC and REG model performance, achieving state-of-the-art results.\n3. The paper is clearly written and easy to follow.", "weaknesses": "No obvious disadvantages were seen. \nLike any research work, this paper likely has its own limitations, though they are not explicitly discussed. Including a section on potential limitations would provide a more balanced perspective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IUT5MHiWcE", "reviewer": "Reviewer_XHYa", "summary": "Existing REC datasets often contain insufficient semantic pairs for training, hindering the REC model's generalization to unseen referring expressions. Additionally, REG methods, due to limited capacity, frequently struggle to bridge the visual and textual domains, resulting in low quality and diversity of generated expressions. In this work, the authors introduce diffusion models into the referring expression generation task, aligning visual features of varying granularity with noisy text. The experiemts are conduted on benchmark datasets.", "questions": "1. Why does performance on certain metrics improve after losing CFG in Table 1?\n2. What is the number of samples in each augmented dataset? This needs to be reported.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The method is described in detail, and the motivations are fairly well-founded.", "weaknesses": "1. Some recent representative works, such as CCL[1], are not compared, even though these works use similar ideas to enhance REC performance through REG. \n[1] Cycle-Consistency Learning for Captioning and Grounding. AAAI 2024.\n2. Failure cases are lacking; diffusion data generation is usually unstable, and the authors need to analyze this point.\n3. Statistics on model parameters, training time, and inference time are required.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)