OneRef:  Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling
Authors: Linhui Xiao
Xiaoshan Yang
Fang Peng
Yaowei Wang
Changsheng Xu
Venue: NeurIPS 2024 poster (2024)
Keywords: Visual Grounding
Referring Expression Comprehension
Referring Image Segmentation
Multimodality
Masked Visual Language Modeling
Primary area: machine_vision
TL;DR: OneRef:  Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling

Abstract: Constrained by the separate encoding of vision and language, existing grounding and referring segmentation works heavily rely on bulky Transformer-based fusion en-/decoders and a variety of early-stage interaction technologies. Simultaneously, the current mask visual language modeling (MVLM) fails to capture the nuanced referential relationship between image-text in referring tasks. In this paper, we propose **OneRef**, a minimalist referring framework built on the modality-shared one-tower transformer that unifies the visual and linguistic feature spaces. To modeling the referential relationship, we introduce a novel MVLM paradigm called Mask Referring Modeling (**MRefM**), which encompasses both referring-aware mask image modeling and referring-aware mask language modeling. Both modules not only reconstruct modality-related content but also cross-modal referring content. Within MRefM, we propose a referring-aware dynamic image masking strategy that is aware of the referred region rather than relying on fixed ratios or generic random masking schemes. By leveraging the unified visual language feature space and incorporating MRefM's ability to model the referential relations, our approach enables direct regression of the referring results without resorting to various complex techniques. Our method consistently surpasses existing approaches and achieves SoTA performance on both grounding and segmentation tasks, providing valuable insights for future research. Our code and models are available at https://github.com/linhuixiao/OneRef.

OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling
Linhui 1:23, Xiaoshan Yangl.2 1..33 Fang Pengl.24 1.2.3 Yaowei Wang? Changsheng Xu X21.2.3
MAIS, Institute of Automation, Chinese Academy of Sciences *Pngghhegg Laboratory
School of Artificial Intelligence, University of Chinese Academy of Sciences
Harbin Institute of Technology (Shenzhen)
(xiaolinhuite {xiaoshan, yang, csxu)@nlpr. ia.acc.... ac. en, sangy@pcl.ac.com
(angfaannnmmmmaiia wwa.is.uuas.........mmmmmm ucas www.uucacc...................... ac
Abstract
Constrained by the separate encoding of vision and language, existing grounding and referring segmentation works heavily rely on bulky Transformer-basee fusion en-/decoders and variety of early-stage interaction technologies. Simultaneouslyy, the current mask visual language modeling (MVLM) fails to capture the nuanced referential relationship between image-text in referring tasks. In this paper, we propose OneRef. a minimalist referring framework built on the modality-shane one-tower transformer that unifies the visual and linguistic feature spaces, To modeling the referential relationship. we introduce a novel MVLM paradigm called Mask Referring Modeling (MRefM), which encompasses both referring-aware mask image modeling and referring-aware mask language modeling. Both mod- ules only reconstruct modality-related content but also cross-modal referring content. Within MRefM, we propose a referring-aware dynamic image masking strategy that aware of the referred region rather than relying on fixed ratios or generic random masking schemes. By leveraging the unified visual language feature space and incorporating MReiM's 's ability to model the referential relations, our approach enables direct regression of the referring results without resorting various complex techniques. Our method consistently surpasses existing ap- proaches and achieves SoTA performance on both grounding and segmentation tasks, providing valuable insights for future research. Our code and models are available https: //ti://iita thub. com/linhumii Linhuixiaan /
1 Introduction
Visual Grounding (V (VG) aims to ground a region referred by a expression query text in a specific image. The generalized VG referring tasks include Referring Expression Comprehension (REC) [69, 62, 101, 31, 14, 91, 90, 48), Phrase Grounding (PG) [1, 74], and Referring Expression/Image Segmentation (RES/RIS) [69, 94, 89]. In REC/PG, the grounding region is represented by rectangular boundary box. while in RES/RIS, it is represented by an irregular fine-grained segmented mask of the referred object. Unlike object detection [57, 58] instance segmentation [26], which usually relies close-set of categories to detect or segment multiple regions that satisfy the object label, visual grounding not limited to fixed categories. It requires understanding the semantics of the query text and then grounding or segmenting specific areas. Therefore, visual grounding is a task that strongly relies on the multimodal interaction and alignment of visual and linguistic features.
Since the introduction of BERT [16] and VIT [19, 7], the state-of-the-art (SoTA) grounding works have widely adopted a pre-training and fine-tuning paradigm. As illustrated in Fig. 1, existing studies
Cooresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Rosults
Types:
Vision Encoder
Image-
One Transformer Emonder
Fusion Transformer
Image
Unified Modality
Head
Interaction
and Grounding Head
(i One T Transformer Encoder
One Becoder
shared
Text
Language Encoder
(HI) Regresion with languag guideed vision token
Text
Encoder
(a) Previous modality deependet encoding structure
(b) One-tower oee structure (ours)
Figure 1: Comparison between our proposed approach and the mainstream REC/RES architectures.
employing pre-trained models, either utilizing uni- -odaa traitrained models to separately transfer visual and language knowledge [14, 15, 98, 33, 55] or utilizing multimodal pre-trained models [91, 77, 89, 35], primarily fall into three typical architectures: (i) two modality encoders combined with a cross-modal fusion encoder, exemplified by Trans VG etc. [11, 14, 98, 89, 91, 35, 92]; (ii) additionally incorporating decoder, exemplified by MDETR etc. [33, 45, 97, 85, 54, 53, 77, 55]; (iii) direct regression based on language-guii visual features, such as LAVT, TransVG++ etc. [94, 15, 79, 99]. However, incorporating modality-dependent encoders in these studies presents challenge for seamlessly integrating the two modalities into a unified feature space. Consequently, these works not only require an additional cross-modal Transformer-bant [82] en-/decoder (fi) and (ii)), but also propose a variety of careful-designe
interaction structures for modality-dependem encoders
to facilitate early-stage fine-gree fnnegrained gross-modal alignment [15, 98, 92, 79, 54, 35, 55, 59]. such as adapter [15, 35], cross-modal bridge [92], weight generation [79]. image-text cross-attention [55, 54], etc. Therefore, these methods not only entail a large number of parameters but also involve intricate processes. Considering these critical limitations, we aim to explore simpler modality-shared grounding frameworks that can unify vision and language within a unified feature space, thereby obviating the necessity of the elaborate interaction modules, bulky fusion Transformer en-/decoders, as well as the special grounding tokens.
With the advancement of pre-training [70, 66), several studies have been conducted to explore unified modality-shared multimodal frameworks. YORO [29] implemented shared encoder based ViLT [37]. However. its modeling approach tends to overshadow the uni-modal knowledge and requires the encoder to incorporate additional query anchors, limiting its applicability for transfer with common pre-trained models. ONE-PEACE [86] has designed seven expert branches based on Mix-of- off-xxert (MoE) [5, 76, 22] to construct three-modality foundation model to realize the integration of image, text, and audio modalities. However, their research employed extensive tri-modal data without exploring the potential utilization of MVLM for modeling the referring tasks. BEiT-3 [87] built upon multi multi-way Transformer [5, 80], which adopts three MoE heads (i.e., vision, language, vision- language) and a modality-sharee structure that effectively unifies vision and language within shared feature space. It demonstrates notable advantages across various classification-lie cross-modal fields (e.g., Retrieval, VQA etc. ). However, no prior research has explored the utilization of BEiT-3 for achieving transfer in referring tasks. Consequently, our objective is to explore more concise and efficient referring grounding and segmentation transfer within a unified feature space on the one-tower model of BEiT-3, However, BEiT-3 model is pre- pre-trained utilizing generic Mask Vision Language Modeling (MVLM) approach, and this masking paradigm lacks fine-grained cross-modal referring ability and cannot effectively model the intricate referential relationship between images and text. As a result, there exists a significant gap when applying BETT-3 to the regression-lis -ike referring tasks. Therefore, exploring how to incorporate fine-grained cross-modal referring capability into the mask modeling paradigm becomes an important research issue that has not been addressed yet.
In this paper, we propose a novel paradigm called Mask Referring Modeling (MRefM), as well as unified and extremely concise grounding and referring segmentation framework named OneRef that no longer requires the fusion interaction Transformer structure and the special grounding tokens.
Firstly, we propose MReiM paradigm enhance the referring capability of BEiT-3 in a flexible manner. MRefM consists of two components: Referring- aware Mask Image Modeling (Referring MIM) and Referring- aware Mask Language Modeling (Referring MLM). The conventional MVLM is typically trained alternately or randomly with uni-modal MIM and MLM. In contrast, Referring MIM and Referring MLM are required to reconstruct two distinct types of content: their own modality-related content and cross-modal referring information. Specifically, (i) Referring MIM employs visual tokens after the dot product operation with the aggregated text token for reconstruction purposes. This not only entails reconstructing masked visual features itself but also necessitates

---

reconstructing the visual target-relation score, which indicates the distance between the current token and the grounding region. The score encompasses four dimensions: horizontal and vertical distance to the grounding center, as well as width and height of the grounding region. In order to enhance the model's s understanding capability for referred regions, we propose referring-aware dynamic image masking strategy that replaces traditional ratio-fixed random masking so that referred regions are reconstructed with a relatively high mask ratio. (ii) Referring MLM employs text tokens after the dot product operation with the aggregated visual token for reconstruction purposes. This not only involves reconstructing masked text itself but also requires reconstructing semantic target-relation scores that represent the correlation degrees between current text tokens and referred image regions.
Secondly, existing grounding and segmentation models commonly employ a [Region) on] token and multiple query anchors to regress results. However, embedding the region token in backbone will disrupt the pre-trained model [15], and the query anchor also depends on the decoder [33]. With the unified feature space established by modality-shared encoder, we no longer need additional cross- modal en-/decoders to fuse uni-modal features, enabling us to more effectively leverage the knowledge acquired by pre-trained backbone. Benefiting from MRefM paradigm, the visual token inherently contains referring information. Consequently, we can discard special grounding token/anchors and directly construct lightweight and highly concise grounding and segmentation task heads based on the dot product operation within Referring MIM to unify the referring framework.
Contributions: Our contributions are threefold: (i) We pioneer the application of mask modeling to referring tasks by introducing a novel paradigm called mask referring modeling. This paradigm effectively models the referential relation between visual and language. (ii) Diverging from previous works, we propose remarkably concise one-tower framework for grounding and referring segmenta- tion in unified modality-shared feature space. Our model eliminates the commonly used modality interaction modules, modality fusion en-/decoders, and special grounding tokens. (iii) We extensively validate the effectiveness of MRefM in three referring tasks on five datasets. Our method consistently surpasses existing approaches and achieves SoTA performance across several settings, providing a valuable new insights for future grounding and referring segmentation research.
2 Related work
2.1 Referring expression comprehension (REC) and segmentation (RES)
(i) REC. The recent supervised REC task, also known as visual grounding in a narrow sense, can be broadly categorized into five main approaches: (1) Fine-tuning with a uni-modal pre-trained language model and a closed-set detector. This setting is exemplified by TransVG [14], which builds upon the two-stage [102, 56, 52, 30] and one-stage [96, 95, 106] methods from the CNN era. It is considered the most conventional and extensively studied approach. (2) Fine-tuning with a pre-trained uni-modal language model and an open-set detection model pre-trained on box-level datasets mixed with multiple data sources. MDETR [33] represents this type of setting, where Fig. 1- (a)-(i) plays a dominant role in its model structure. (3) Fine-tuning with multimodal self-supervised pre-trained models. CLIP-VG [91] serves as an example for this category, introduced primarily through the proposal of CLIP [70]. (4) Multimodal and multi-task mix-supervised pre-traines models. These methods typically combine multiple tasks while mixing datasets from each downstream task, employing mixed pre-training that incorporates both self-supervision and fine- -reiggai supervision. UniTAB [97]. OFA[85]. etc., represent such approaches where visual grounding often acts as one of the pre-training tasks. (5) Grounding multimodal large language models (GMLLMs). These methods influenced by works like GPT [6] or LLAMA [81] etc. These models integrate visual backbones into Large Language Models (LLMs) to generate grounding results rather than relying on regression techniques. Our approach mainly falls under type (3). (fi) RES. The development and approach categories of RES [49, 13, 35, 89, 79, 94, 93, 88] are generally similar to those of REC. However, the key distinction lies in the finer granularity of RES's output, which necessitates separate study from RÉC. In terms of model architecture, RES works predominantly employ two modality encoders and decoder to generate the segmentation mask. Our work stands out the first endeavor to explore RES within a unified multimodal feature space under a one-tower structure.
2.2 Mask vision language modeling
Motivated by the success of MLM [82] BERT [16), MAE [27] and BETT [4] have primary shifted their attention to MIM [21, 83, 3J. Subsequently, exemplified by BETT-3 [87], numerous MVLM
3

---

Shared Multi- Head Self- Attention
Pest
Recomstrncte
contow
Vision Language Expert
Imagee
Referring-aware
dynamic Image masking
Referring-
SSEP
Referring MIM bead
X-RIHAA
Visual torget-rclation score
y-dturck w--mook
A-mash
Mask Image Modeling
Recontrrnted lmguuge contont "he giratte walking while another
graile
the juj walking
EELS
walking while
cpg inother
another gratle
Sollows it.
gindfe einq it."
masting
[wr"
Referring
One-tower Encoder
another follows
Mask Language Modeling
Referring MLM head
Samontic lorget-relation
Illustration of our multimodal Mask Referring Modeling (MRefM) paradigm, which includes Referring-awant mask image modeling and Referring-awee aware mask language modeling.
Figure 2:
works [61, 47, 36, 104, 2] have emerged, with most of these works implementing randomly alternating uni-modal MIM and MLM. Most relevant to our work are mask region modeling (known MRM) [64, 83], which can be either unimodal MIM (e.g., R-MAE [64]] or employ more fine-grained regional data and contrastive learning reconstruct the alignment between regions and object labels (e.g., .In [61], VLT [17, 18] etc. ). However, our work focuses on modeling the fine-grained referential relationship within image and text, which is significantly different from these works.
enhance the cross-modal referring capability,
3 Methodology
In this section, we propose our multimodal Mask Referring Modeling (MRefM) paradigm, which includes Referring MIM and Referring MLM, as well as feature space unified grounding and segmentation framework OneRef. We will introduce these methods in the following sections.
Following BEiT-3 [87], we employ a multimodal modality-sharee Transformer [5] as the underlying backbone network. Initially, we perform mask-ther -hen--redicc
predict MRefM pre-training, and followed by transfer fine-tuning on the referring tasks. As shown in Fig. 2, the MRefM pre- mre-tranning stage consists of two components: Referring- aware Mask Image Modeling (Referring MIM) and Referring-awe awwaee Mask Language Modeling (Referring MLM). Both modules aim reconstruct two types of content: modality-related content within each modality and cross-modal fine-grainer referring content.
3.1 Preliminaries
BETT-3 [87] utilizes MIM, MLM, and MVLM for processing image, text, and image-text pairs respectively to facilitate the acquisition of general representations through MoE heads and shared muat self-attention. Notably, MVLM involves alternate training of MIM and MLM. Specifically:
(i) Vanilla mask image modeling. We denote
# x3 the input image, and it is tokenized by convolution projection to N, HW / p2 patches {æ=" Ji=1: where x. E N.N.DD DD H. W are the image size, and P is the patch size, D the hidden dimension of the unified feature space. Then, we leverage specific masking strategy to mask a specific number of image patches. The masked position is termed as M,. Thus, shared learnable embedding EIM] is used to replace the masked image patch embeddings n! if E M, Subsequently, we prepend learnable [CLS] token to the input, ie., [cass, (x.) ]- and feed them to the one-tower Transformer. Next, we utilize MIM head which consists of linear projection and softmax classifier to predict the visual tokens of the masked positions based on the corrupted image a.M, The visual tokens are obtained by the image tokenizer VQ-KDeLIIP proposed in BEiT v2 [67], which provides supervisions for the MIM self-supervised N. learning procedure. The visual tokens of the original image denote {z, I and I denotes the pre-training images. Then, the training loss of MIM is defined as:
MMMM = - Σ Σ
log log p(z.la)))
(I)
REZIEM

---

(ii) Vanilla mask language modeling. The input text is tokenized and projected to the word embeddings {w: by SentencePiece tokenizer [41] with vocabulary size of 64010, where w E RDMDD M is the length of tokenized text sequence. Then, following BEÎT-3 [87]. we randomly mask the text tokens with a fixed masking ratio 6. The masked position termed as My- Thus, a shared learnable embedding W|M is used to replace the masked word tokens w, if i MM- We prepend a Iearnable special tokens [SEP] and an end-of-sequence token [EOS] to the sequence, i.e. (SEP: {w; R=1+ Ggos); and feed them the one-tower Transformer. Similarly, we utilize MLM head which consists of a linear projection to predict the text tokens of masked positions based on the corrupted text data w The original textual tokens are denoted as (t,) and 'T denotes the pre-training text sequences. Then, the training loss of MLM is defined as:
MMMM =- E Σ
log p(t.luo/M)
OETIEME
3.2 Referring-aware mask image modeling
After concatenating the visual and text tokens and feeding them into the modality-shared encoder, the vanilla MVLM is commonly implemented through the alternating use of MIM and MLM [87]. Despite the multimodal features are interact within the modality shared encoder, it fundamentally remains a unimodal information reconstruction. Additionally, MVLM acquires general knowledge by randomly masking images and texts, it fails to effectively model the referential relationship. Hence, we propose Referring MIM and Referring ML.M methods. Specifically, as shown in Fig. 2, our proposed Referring MIM incorporates two additional components: the reconstruction of visual target-relation score and a referring- dynamic masking strategy.
In Referring MIM (Fig. 2), instead of using uni-modal visual tokens [7, we propose to employ visual tokens that dot product with the aggregated text token Estp E I for the reconstruction pur- pose. The reconstruction of Referring MIM involves not only the modality-related content {z] heM, but also the visual target-relation scores (s ) E R.N.x4, We utilize visual target-relation head which consists of a three-layer perceptron (MLP) to predict the scores. The scores represent the J N. and the referred region B = (Ie, Mo, Wrr hr,,, where
distance between each patch token {x;M), (Nies Me, Wr, hy) denote the center coordinate and the width and height of the referred region. It encompasses four masks, i.e.. x-, y+, Ws, h- masks. which represent the normalized horizontal and vertical distances from the referred center, i.e., ((x- -x))x//1 xe)/W, (y ye)/H), and the proportion of width and height on the the referred region, i.e., (P/wr, P/hr), respectively, where (æ, y) denote the center coordinate of each patch. We denote as dot product operation. Finally, the training loss of Referring MIM is defined as:
CReferring MIM
Σ RETIEM.
log p(zi & eSEP))
Σ
Σ
log p(s;"" e esEP) )). (3)
TET NET.EE..No
Referring-aware dynamic image masking strategy. As shown in Fig. 4, among the ex- isting masking strategies, MAE [27] adopts a high-ratio random masking while BEÎT- [87] uses low-ratio block-wiss wise random masking, neither of which effectively directs attention to the referred region. SemMAE [43] proposes semantic-guided masking that requires additional bulky semantic mod- els and limits its generality. To enhance the model's 's understanding of the referred region through surrounding visual context and text semantics, we propose a referring-awant dy- namic masking strategy shown in Algo. 1.
Algorithm 1 Referring--ware Dynamic Masking
Input: N. image patches, N, (hrp x Wry) referred patches. Output: Dynamic masked positions M.
4- Rand Select 9 N2 numbers in [1, No)
New M E B NN,N, {[Mi). No MM 1 if c, else 0}
M <- MM reshape as M g
>> II-centext masking
New M, E RRrpppppe Xhrp with all as 0 & Referred masking
while [M < 7- N, do
< R1nd(1, (1, 7.7 .N, -I-M)
D Block size
Rand(a,
Aspect ratio of block
wa vs/r; hi Vs.T Width, height of block
L Rand(O, Wep w,): t- Rand(O, hrp lws) (i.( = 1 iE [1, + ws). E [t, t ha)
end
M(rap : Zap + Weps Vap yap hrp) M. return M.
The strategy avoids the drawbacks of the aforementioned methods and directs the model's attention to the referred region. Specifically, we denote the shape after patch reshaping of the image as (h, w), where h H/P, W/P, and N, = h x To maximize the masking of the
5

---

Resal::
Unified Madality- shared Encoder
Unified Modality- shared
Cunstraiat
Image
Image
séiie green,
bise grwe, and yallore pathi-
Referring- based REC head
Referring- based RES hrad
and jetfere palkae patka-
det os
Box mark
dots
Expression test
Expression text
(a) Referring Expression Comprehension task
(b) Referring Expression Segmentation task
Figure 3: Illustration of the referring-based grounding and segmentation transfer.
referred region (Ie, Ms, Wyy hy), where Via, ys represent the starting coordinates of the referred region, we introduce margin 7n its surroundings and denote its patch coordinates (Faps Maps Wrys hrp). i.e. Zap = / m. Wry 2 P] + m, Usp and hrp are similar to that of Tap and Wyp- where [-] indicates rounding down to an integer. Thus, the number of referred patches denote as N = hrp X Wyp. Then, as shown in Algo. 1. to ensure that the model allocates appropriate attention to the in-contextual information around the referred region, we utilize a random masking with relatively low ratio / for its surroundings. Simultaneously, we employ a block-we wiic masking approach with a high ratio y the extended area of the region. Since referred regions vary across different image-text pairs, each sample's 's entire masking ratio a is dynamically determined:
= [3 (N₂- N₂) + N.)/N,-
3.3 Referring-awar mask language modeling
Similarly, in Referring MLM, instead of using uni-modal linguistic tokens [87, 47, 2], we propose to employ linguistic tokens that dot product with the aggregated visual token Eats E IR IgID for the reconstruction purpose. The reconstruction of Referring MLM involves not only the modality-related content {4 he, but also the semantic target-relation scores {s Ji=1- ( The score represents the correlation between the referred target and the language token, which obtained by a teacher model (i.e.. BEiT-3 model with performed image-text contrastive intermediate tuning) with calculating the weighted sum of the normalized similarity between the language token {w;} ;=1 and the aggregated visual token
reg of referred region, as well as the aggregated visual token ecis of entire image:
Areg a<<< CLS
{ >) + Aimg al<<
ECLS
) >),
(5)
where <,,-> > denotes cosine similarity operation, a denotes the softmax normalization.A. shown in Fig. 2, we utilize semantic target-relation head which consists of a three-layer MLPs and softmax normalization to predict the scores. Finally, the training loss of Referring MLM is defined as:
Referring MLM
Σ
log p(t:
(w, ecis))
Σ Σ log Phr(s (w:"" eais)), (6)
WETiEM..
WET FE[L.M]]
where Pkt represents probabilistic prediction with Kullback-Leibler divergence [28].
3.4 Referring-based grounding and segmentation transfer
The modeling of visual and language in a unified feature space eliminates the need for the commonly- used Transformer-basee
fusion en-/decoder [14, 54, 55] and various early-stage interaction techniques [15, 79, 92] to further uniform the visual and language features. Additionally, since the referential relationship is modeled by MRefM during pre-training we can accurately regress the results of grounding and referring segmentation using the output tokens, without relying on the widely-used special grounding tokens (e.g., [Region] token [14, 15, 98, 91, 92], query anchors [55, 54]).
Referring expression comprehension. As illustrated in Fig. 3-(a), based on Referring MIM, we initially perform a similarity operation between visual tokens [ E RN..D and aggregated language token ESEP E I to obtain a softmax-normalize similarity mask Maim E R This mask is then replicated and multiplied back to each hidden dimension of the visual tokens. Subsequently, the visual tokens are summed to yield reduced tokens, which are finally subjected to regress the prediction box B = (ice de, wr. hr,) using 3-layer MLPs:
N.
B- = MLP
Σ (Repeat(o(<< CESP" [a >)) O (MP((
To enhance the accuracy of cross-modal similarity, we propose treating the similarity as a coarse- grained downsampling bounding box mask Miow hxv and imposing segmentation loss (i.e.,

---

Table I: Comparison with latest SoTA methods on the five datasets for REC/PG tasks with single-
dataset fine-tuning setting. We highlight best result of base model in red and bold for large model.
Visual
Language
RefCoCo
ReCOOC++
ReCCOCOg
Referit
Flickr
Methods
Venee
Backbone
lesiA
Bes
Gest.
Single-ditaset line-tuming selting ss uni-mnudal pre-traized clusessel detector and language
model:
(traditiona nettimg)
TranVV [1:]
10CV'21
RNIGI-DETR
BERT-R
81.02
82.72
78.35
64.82
70.3
56.94
68.67
67.73
70.73
79.10
Word2Pix [103]
TNNLS:22
RNI0L-DETR
BERT-B
81.20
84.39
78.12
69:74
76.11
61.34
70.81
71.34
QRNet [96]
CVPRC22
Swin-S [60]
BERT-B
84.01
85.85
82.34
72.94
76.17
63.81
71.89
73.03
74.61
81.95
VG-LAW [79]
CVPR'23
[46]
BERT-B
86.06
K2.87
50
66.69
75.31
75.95
Trn5VG-+(11]
TPAMP'23
VIT-DeI [46]
BERT-B
86.28
58.37
80.97
75.
90.45
66.28
76.18
76.30
74.70
81.49
Single-dntsset fine-tuning sctting w. vision-lnngugg-
self-spervisod pre-trained modil::
CLIP-VG [91]
TMM
CLIP-B
CLIP-B
84.29
87.76
78.43
09:55
77.33
57.02
73.18
72.54
70.89
81.99
IMRI [108]
T3
CLIPH
CLIP-B
82.97
87.30
74.02
71.17
79.82
57.01
71.96
72.04
68.25
79.90
Dynamic- MDETR
TEAMI'23
C
CLIP-B
$8.82
80.12
74.83
$1.70
63.44
74.14
74.49
70.37
HIVG-B [X2]
ACMMM
CLIP-R
CLIPP-
57.32
89.85
83.27
73.06
68.1
78.29
79.79
75.22
HIVG-L (92]
ACMMM 24
CLIPL
CLIP-L
88.14
91.09
83:71
80. 10
86.77
70.53
80.78
80.25
76.25
82.10 16
OneRef-B (vurs)
N4 NoarPSS244
B6T3-B
B6TT3-B
88.75
90.95
85.34
50.43
56. 46
74.26
83.68
83.52
77.17
83.61
OneRef-1.
BEIT3-L
RRIT3-1
83.73
88.11
84.75
Table 2: Comparison with latest SoTA methods for REC task with dataset-mixed intermediate pre- training setting. 'RefC' represents the mixup of RefCOCO/+/g training data. indicates RefC has been used during pre-training. "G-DINO-L." denotes '0365,01,cr Caaadd..apMMC.COOOC0
Visulllangugee
Internediute
Dasa
ReCOCO
ReCooCOy
Baxiboos
preczin data
size
val
lest/
testB
v2l
testA
textB
val
18st
Dntawwt-mived intermediate pre-training wetting (v. bax-level datoost-miind open-sst detection pre-trained modell)
MDETR [33]
ICCV221
RNII01BBBEXT-B
Golai Rull.
6.5M
56.75
89.58
81.41
79.52
64.09
70.62
81.64
50.89
YORO'
BOCV'22
BERT-B
ReIC
$2.90
55.64
73.50
64.50
74.30
DETRETR [54]
AAA122
RNIOL/ BERT.B
Refa
59.63
91.04
51.66
73.21
52.76
$3.44
Grounding DINO B
arXan225
Swin B B
0365, GoloG. RefC
7.2M
39.19
91.2
35.99
$1.05
$7.40
74.71
84.15
$4.94
Gronding DINO-L
a3Xw
Swin L BBRFBB
G- GEDNN-11
21.4M
90.56
95:
86.24
62.75
$8.95
75.92
96.1 13
67.00
CyCo (64)
AAA134
WTT BERT-B
VG KG.SBU.OCMM..
>130M
59.47
91.87
85.33
50.40
$7.00
69.30 87
$1.51
81.
H:V0-8 [12]
ACMMMM224 24
CLIP- IP-B
Refert Flickr
0.5M
90.56
02.55
87.33
$3.08
67.83
76.
84.71
84
HIVG-. [92]
2CMMMM24
77:37
96.85
Fime-tuning willing
mulli--ask ris-ouperviedd
pre-trsined medel:
UsIAR [97]
ECCV'22
RNIO1/BBBXXTB
vG.COCO.uk.
>20
58.50
91.06
83.75
50:97
55.36
71.55
94.58
OBN-B B [85]
ICML222
OFA BOBABB
58.48
90.67
83.30
51.39
87.15
74.29
$2.29
$2.31
OFA L [85]
ICML/2
OBA L/ L
90.05
92.93
85.26
$5.90
39.57
79.22
$5.89
$6.55
Vine-tuning wetting
grounding multinudaa large langnage model (GMLLLM):
Shikm=78! [10]
arXi'25
CLIPL (R0n)77[(11)
RoIC.NG
87.01
90.61
80.34
51.60
67.36
72.12
K2.19
Fanet-78 [100]
CLIP-L/ Vicuna 7B]12]
[100]
38M
57.49
91.35
82.45
50.73
67.334
73.34
53.93
54.76
LION 4B [9]
CVPR124
EVA GJ21 WFlanTS 3B
vG..CO...
3.60
59.73
92.29
34.82
53.60
59.72
77.34
$5.09
83:63
LION 120 [9]
CVPR/24
DWA PTlanTS LID
VG.COCO.ss
3.6M
39.80
95.02
85.57
$3.95
59.22
78.06
$5.52
85.74
OmeRet-BB (unsopervise))
DETT By DETS
RefC Refectt
0.5M
39.16
92.05
87.26
83.18
$8.55
77.66
$4.72
88.17
OneReé-B (0.23)
NenrIPS224
BETTS-B INBTT--B
Refeett
87.32
OneRo6-1/ (0.6R) NenelPS
R6T1-11 NRR733.
RefC.
0.5M
93.21
95.43
90.11
58.38
92.11
82.70
87.81
88.83
Focal loss [51] and Dice/F-1 loss [63]) on the sigmoid activated similarity mask Msim E Ro with coefficient >fhbox bbarr and boz boz as the box mask constraints:
Lbox_mask_ce www.aacoosssaiiiiiiie
bo fouul sma Mbbx) + Ad Coxce (M simm: Mboo).
(8)
Consequently, the loss function for the REC task can be reformulated as the weighted sum of vanilla grounding loss (i.e., smooth I.1 loss [25] and Giou loss [73]) and box mask constraints:
CREC XI, (B. + AgiouL (B, B3 Lwoo_oosk_ceecccmcee maak__cnsttrantte
Referring expression segmentation. As illustrated in Fig. 3-(b), the implementation of referring segmentation can be regarded as a simplified version of grounding. Initially, we employ a 3-yayer deconvolution to up-sample the visual token "2 E #// Subsequently, cosine similarity operations are performed on the up-sampled visual tokens and the aggregated language token. The resulting similarity mask is then utilized as the final predicted mask Mseg E TR. xW after applying 1- 1-layer bilinear interpolation. We denote the ground truth segmentation mask as Maeg E R HW then the loss function for RES is defined as follows:
LRSS Ag_segfffooo (M Meggg) + >dd_ Set M Mseg).
(10)
4 Experiments
4.1 Experimental setups
Datasets and evaluation metries. Our method is validated in the REC, RES, and PG tasks with five widely used datasets, namely three REC/RES datasets (RefCOCO/+/g [101, 62], as well as two PG datasets (ReferltGame [34] and Flickr30k Entities [68]). In PG, the query pertains to a specific phrase,
7

---

Table 3: Comparison with latest SoTA methods (mloU metric) on the three datasets for RES task with both single-dataset fine-tuning setting and dataset-mixed intermediate pre-training setting.
Ssqual/Language
Intermediate pretrain data
RefCOCO
RefCOCO+
RefOOCOg
Methods
Venue
Backbone
val
testA
testB
val
lestA
testB
val
test
Single-dataset fine-tuning setting
unl-modaal pre-trained close-set segmentation model:
(traditional setting)
ReCTR [45]
NIPS'21
RNIOI BEKT-8
74.34
76.77
70.87
66.75
70.58
59:40
66.63
67.39
SeqTR [107]
2CCV
DN53 GRU
71.70
73.31
69.82
63.04
66.73
58.97
64.69
65.74
LAYT [94]
CVPR'22
Swin-B/ BERT-B
74.46
70.94
65.81
59.23
63.34
VG-LAW [79]
CVPR'23
VeT-Det/ BERT-B
75.05
77.36
71.69
66.61
70.30
58.14
65.36
65.13
Single-dataset fine-tuning setting
vision-laguaage self-supervised pre-trained model:
CRIS [89]
CVPR'22
CLIP-L/CLI
70.47
73.18
66.
62.27
68.06
53.68
59.87
60.36
JMCELN[52]
EM
CLIP-B/CLIPBB
74.40
77.69
70.43
66.99
72.69
57.34
64.08
64,99
RISCLIP-B [35]
NAACL'24
CLIP-B/ CLIP-B
75.68
72.46
69.16
60.68
67.62
67.97
RISCLIP-L [35]
NAACL/24
CLIP-L/
78.87
81.44
75.41
74.38
78.77
66.84
71.82
71.65
OneRef-B (ours)
NeurIPS'24
BETT3-/ BGTT3BB
77.57
79.05
75.11
71.25
65.45
69.37
69.70
OneRef-I. (ours)
NearlPS 24
BRTT3-1/ BRTT3-
80.09
82.19
77.51
75.17
79.38
70.17
73.18
73.76
Dutuset- intermediute pre-training setting:
PolyFomer.B [S33
CVPR'23
Swin-B/ BERT-R
RefC:
75.96
77.09
73.22
70.65
74.51
64.64
69.36
RISCLIP-D
24
CLIP-B/ CLIP-B
RetC
75.68
78.01
72.46
72.46
74.30
61.37
69.49
69.53
[3S
NAACL'24
CLIP-L/ CLIP4.
RefC
79.53
81.78
75.78
74.88
78.88
68.09
73.45
74.52
OneRef-B' (unsupervised)
BETT3-B/ BEIT3-B
RefC
78.20
79.26
75.92
72.54
75.54
67.39
71.28
71.13
OueRef-B (ours)
N1wrIP'221
BETT3-B/ BEIT3-B
RetC
79.83
81.86
76.99
74.68
77.90
69.58
74.06
74,92
OueRef-L, (ours)
NearIPS'24 24
BB/T3-L/ BEIT3-L
RefC
81.26
83.06
79.45
76.60
80.16
72.95
75.68
76.82
Table 4: Ablation of MRefM on mixup pre- pre- training setting.
Table 5: Ablation of the task heads.
image masking stratogy
ReCCOCO-
ReCCOCOg
Re/OOCOg
MIM
MLM
val
testA
test
val
Usi
Architecture (Fine-tuning serting)
val ledi
x
X
X
78:56
83 83.36
71.72
80.41
80.52
full modkl
83.52
vanilla
vanilla
random
79.68
84.59
72.11
81.35
81.11
full model w/o box mask loss
$2.54
82.02
vanilla
vanilla
referrig-sware
80.06
85.77
73.96
81.96
82.16
w. fusion encoder + region token
78.93
78.51
Ref MIM
vanilla
referring-awe -ware
83.64
88.3
76.58
83.55
85.86
full moddl RES
69.70
vanilla
Ref
referiig-sware
81.52
86 86.87
82.88
84.32
deconv aller similarity operation
67.98
Ref MIM
Ref MLM
random
85.08
89. 19.12
78.56
85.57
86.89
4 d-yayer deceav wio lincar upsample
68.33
68.90 68.96
Ref MIM
Ref MLM
referring-ane aawar
86.38
90.38
79.47
86.82
87.32
2-Jayer decony 2-Jayer upsample
67.51
67.65
while in REC and RES, the query refers to a reference expression. The text of RefCOCO+/g exhibits greater length and complexity in comparison to that of RefCOCO. In REC/PG, we follow previous works [14, 96] that employs Intersection-over-Ul Uviinnn (IoU) as the evaluation metric, i.e., a prediction is deemed accurate only when its IoU exceeds or equals 0.5. We compute the prediction accuracy for each dataset as a performance indicator. While in RES, we follow previous works [79, 35] that employs mean IoU (mloU) and overall IoU (oloU) for each dataset as the indicators. The detailed statistics information regarding these five datasets are provided in the Appendix B.
Experimental details. Since MRefM is proposed on the basis of the traditional MVLM. considering the pre- tre-training cost of MVLM, we adopt BEiT-3 [87] base and large model as our initial weights and then perform intermediate MRefM pre-training on the tesk-relevant dataset. Such intermediate pre-training is common in existing grounding works [55, 54, 33). As described in Sec. 1. to verify the effectiveness of our MRefM approaches, we conduct extensive experiments on three settings: (1) The basic single-dataset fine fine-tuning setting. This setting does not require additional training data and aligns with existing supervised and self-supervised transfer approaches [91, 92, 89, 35]. In this setting, we perform supervised single-dataset intermediate MRefM pre-training before fine- tuning. (2) The setting of fine-tuning with supervised lataset-mixed intermediate pre-training. This setting aligns with existing grounded pre-trained approaches, such as Grounding-DINe DINO [55], DQ-DETR [54] etc. we perform an MRefM intermediate pre- pre-training before fine-tuning. (3) To verify the generality of MReIM, we perform the setting of fine-tuning with unsupervised intermediate MRefM pre-training. There are several ways to obtain the regions in the regional masking modeling works [64, 24, 78], such Felzenswalb- Huttelloooooheeerr (FH) algorithm [23], SAM [38] etc. Thus, we adopt the unsupervised, fast, computcb FH algorithm [23] to generate regions following R-MAE [64]. We then select the referred one using BEiT-3 model with performed image-text contrastive intermediate tuning. More details about the selection of the unsupervised regions, network architecture, training and inference, model hyperparameters etc. are provided in the Appendix C.
4.2 Comparison with state-of-the-art methods
Referring expression comprehension. As shown in Tab. 1 and Tab. 2, we conducted experiments for the REC and PG tasks across three settings. (1) In the single-dataset fine-tuning setting, our base model surpasses the current SoTA method HiVG [92] by 2.07B( 6. (testB,, (tes),
8

---

Original imuge
Random, 75% (b) Blockwise, 40%
(c) Referring-aware dynamic musking
Query: the giraffe walking while another girafte follows
20% 55%
35%
75%
B 50% 90%
Query:
"azzbra eating grass in front of several other eating animals" | m 20%, y - 55% 8 35% 75%
f 50%, y 90%
Figure 4: Illustrations of random masking (MAE) [27], block-wise masking (BEiT) [4], and our referring-aware dynamic masking. a denotes the entire masking ratio, while 3 and y denote the masking ratio beyond and within the referred region.
Table 6: Generality study MRefM on RefCOCOg.
Table 7: Generality of the task heads.
Single-dataset val
Mixup pretrain val
RetCOCOg
Architecture
Backbone
test
Architecture (Fine-tuning setting)
val test
TransVi MRefM-TransVV VG CLIP-VG MReIM-CLIP-GG
DETR BERT-B
68.67
67.73
75.73
75.86
TransV++ (Reproduced by us)
75.04
75.55
DETR / BERT-B
71.51
70.84
78.71
78.69
TransVGtt REC head
76.65
77.09
CLIP-B CLIP-B
73.18
72.54
78.67
78.
LAVT [94]
63.34
63.62
CLIP-B/ CLIP-B
74.22
74.50
80.48
80.83
LAVT our head
65.35
(test,, and 1. 1.50%(test) on the five datasets respectively, while also significantly outperforming the traditional uni-modal detector-basee approach Trans VG++ [15] by 4.37%(testB),, 7.9t%() 7. (tes), (tst), and 2. (test,,,, respectively. (2) In dataset-mixed pre-traiming setting, our base model outperforms HiVG [92] by 1.35%, 2.79%. and 2.63% on RefCOCO/+/g testB/testB/test splits, outperforms Grounding- DINO [55] by 2.59%, 4.76%, and 2.38%. exceeds OFA by 5.28%. 5.18% ,add 5.01% and even surpasses LION [9] - GMLLM model that is 20-60 times larger than ours - by 3. 3.76%, 13% ,and 1. 1.69%. Note that among these works, UniTAB [97]. OFA[85], LION [9] also utilize the MVLM on the pre-training stage. (3) Furthermore,
achieve competitive performance in the unsupervised setting, which shows the generality of MRefM paradigm. Addition- ally, our large-size model exhibits remarkable scalability with further substantial improvements in performance, More detailed results are provided in the Appendix E.
Referring expression segmentation. As presented in Tab. 3 (mloU metric), conducted exper- iments for RES task under three settings. (1) In the single-dataset fine-tuning setting, our base model surpasses the SoTA self-supervised method RISCLIP [35] by 2.65%. 4.77%. and 1.73% on RefCOCO/+/g testB/testB/test splits, respectively, while also significantly outperforming the traditional uni-modal detector- based approach VG-LAW [79] by 3.42%, 7.31%, 31,, and 4.57%, respec- tively. (2) In the dataset-mixed pre-training setting, our base model achieves superior performance compared the SoTA method RISCLIP [35] with improvements of 4.53%, 8.21%, and 5.39%. (3) In the unsupervised pre-training setting, we also achieve competitive performance. Additionally, our large-size model also exhibits remarkable scalability and demonstrates substantial improvement in performance. For oloU metric, the results are presented in Appendix E.2 (Tab. 13).
4.3 Ablation study
The Mask Referring Modeling. In Tab. 4, we conducted ablation studies on MReîM, which included Referring MIM ('Ref MIM'), Referring MLM ('Ref MI.'"., and referring-aware dynamic image masking ("referring-awar aawarr"" The 'vanilla' denotes the vanilla MVLM described Sec. 3. As shown in Tab, 4, referring MIM, referring MLM, and dynamic masking strategy resulted in improvements of 3. 3.70%, 2. 26%, and 1.05% on the RefCOCOg-test dataset, and with an overall improvement of 6.21% demonstrates the effectiveness of our methods. More results are provided in the Appendix E.4.
The referring-aware dynamic masking strategy. Fig. 4 presents a schematic the three masking strategies. In our experiments, as illustrate in Fig. 4-(c), 3 and y demonstrate optimal performance at values of 0.35 and 0.75, respectively. More detailed results are provided in the Appendix E.5.
The enferring-based task heads. We conducted ablation studies on the design of two referring-based task heads. Tab. reveals that our modeling method effectively captures referring information the backbone stage, benefiting from the one-tower structure. This approach is significantly more efficient

---

than the traditional fusion encoder and special token- token-based method. Additionally, our proposed box mask loss also contributes to a performance gain of (tes))
4.4 Generality study
The generality of MRefM. Firstly, we perform an unsupervised MRefM pre-training in Tab. 2 and Tab. 3, both of which achieve competitive performance. Secondly, we replace the backbone and apply MRefM on DETR and CLIP by using Trans VG [14] and CLIP-VG [91] under the two settings. Since the two frameworks do not interact backbone stage, we build MRefM on the fusion encoder. In Tab. 6, MRefM can effectively learn referring representation, resulting in an overall performance gain of about 2.0%. All these findings demonstrate the validity and generality of the MRefM paradigm.
The generality of referring- based task heads. Since both Trans VG++ [15] and LAVT [94] have modality interactions at backbone stage, attempted to apply our task heads to both frameworks.
Trans VG++ is reproduced by us since its code is not available. Tab. 7 shows that our proposed task heads achieve 1.5+% improvement in both REC and RES, offering a new avenue for future research.
Conclusion
In this paper. we propose novel, highly concise, and feature space unified one-tower referring frame- work. Additionally, we pioneer the exploration of mask modeling in referring tasks by introducing MRefM paradigm to capture the referential relationships between vision and text. We demonstrate the effectiveness and generality of MRefM across three settings on REC, PG, and RES tasks, consistently achieving groundbreaking results. Furthermore, leveraging unsupervised methods enables potential large-scale pre-training of MRefM in the future, presenting a new direction for referring tasks.
Acknowledgement
This work was supported in part by the National Natural Science Foundation of China under Grants 62036012, U23A20387, 62322212, 62072455, in part by Pengcheng Laboratory Research Project under Grant PCL2023A08, in part by National Science and Technology Major Project under Grant 2021ZD01 12200, in part by Alibaba Innovative Research Program, and also in part by CAS Project for Young Scientists Basic Research (YSBR-116).
References
[1] Hassan Akbari, Svebor Karaman, Surabhi Bhargava. Brian Chen, Carl Vondrick, and Shih-Fu Chang. Multi-level multimodal common semantic space for image-phrase grounding. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 12476-12486, 2019.
[2] Tarik Arici, Mehmet Saygin Seyfioglu, Tal Neiman, Yi Xu, Son Train, Trishul Chilimbi, Belinda Zeng, and Ismail Tutar. Mlim: Vision-and-language model pre-training with masked language and image modeling. arXiv preprint arXiv:2199. /2/78, 2021.
[3] Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. Multimae: Multi-modal multi-task masked autoencoders. In European Conference Computer Vision, pages 348-367. Springer, 2022.
[4] Hangbo Bao, Li Dong. Songhao Piao, and Furu Wei. Beit: Bert pre-training image transformers. In International Conference on Learning Representations, 2021.
[5] Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified vasion-language pre-training with modaat--ffmodaliy- experts. Advances in Neural Information Processing Systems, 35:32897-322 32912, 2022.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
[7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier. Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pages 213-229, 2020.
[8] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre- pe-training to recognize long-tail tail visual concepts. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 3558 3568,, 2021.
10

---

[9] Gongwei Chen, Leyang Shen, Rui Shao, Xiang Deng, and Liqiang Nie. Lion: Empowering multimodal large language model with dual-level visual knowledge. arXiv preprint arXiv:231 1.1660 2024.
[10] Kegin Chen, Zhao Zhang. Weili Zeng. Richong Zhang. Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's 's referential dialogue magic. arXiv preprint arXiv:2306. 1.6...1517 15/55, 2023.
[11] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu, Uniter: Universal image-texi representation learning, In European conference on computer vision, pages 104-120. Springer, 2020.
[12] Wei-Lin Chiang, Zhuohan Li, 7i Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuma: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
[13] Yong Xien Chng, Henry Zheng, Yizeng Han, Xuchong Qiu, and Gao Huang. Mask grounding for referring image segmentation. arXiv preprint arXiv:2312.12198, 2023.
[14] Jiajun Deng, Zhengyuan Yang, Tianlang Chen, Wengang Zhou, and Houqiang Li. Transvg: End-to-end visual grounding with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1769-1779. 2021.
[15] Jiajun Deng, Zhengyuan Yang. Daqing Liu, Tianlang Chen, Wengang Zhou, Yanyong Zhang. Houqiang Li, and Wanli Ouyang. Transvg++: End-to-end visual grounding with language conditioned vision transformer. IEEE Transactions on Pottern Analysis and Machine Intelligence, 2023.
[16] Jacob Devlin, Ming-Wes Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint (510::880000400. 2018.
[17] Henghui Ding, Chang Liu, Suchen Wang. and Xudong Jiang. Vision- transformer and query generation for referring segmentation, In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 16321-16330, 2021.
[18] Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. VIt: Vision-language transformer and query generation for referring segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(6):7900-77) 7916, 2022.
[19] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11 11122090...99,.......1 2020.
[20] Hugo Jair Escalante, Carlos A Hernández, Jesus A Gonzalez, Aurelio López-López Manuel Montes, Eduardo F Morales. L Enrique Sucar, Luis Villaseñor, and Michael Grubinger. The segmented and annotated iapr benchmark. Computer Vision and Image Understanding (CVIU), 1 114:419-428, 2010.
[21] Yuxin Fang. Wen Wang. Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang. Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19358- 19369, 2023.
[22] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(12)):1-99 2022.
[23] Pedro F Felzenszwalb and Daniel P Huttenlccher. Efficient graph-based image segmentation. International journal of computer vision, 19:167--8 181, 2004.
[24] Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurmans, Sergey Levine, and Pieter Abbeel. Multimodal masked autocncoders learn transferable representations. In First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022, 2022.
[25] Ross Girshick. Fast r-cnn. Proceedings of the IEEE international conference on computer vision, pages 1440-1448, 1448, 2015.
[26] Kaiming Hc, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, pages 2961-2969, 2017.
[27] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.

---

[28] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. stat, 1050: 9, 2015.
[29] Chih-Hui Ho, Srikar Appalaraju, Bhavan Jasani, R Manmatha, and Nuno Vasconcelos. Yoro-lightweight end to end visual grounding. In Computer Vision-ECCC 2022 Workshops: Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VilI, pages 3-23. Springer, 2023.
[30] Richang Hong, Daqing Liu, Xiaoyu Mo, Xiangnan He, and Hanwang Zhang. Learning to compose and reason with language tree structures for visual grounding. JEEE TPAMI, 2019.
[31] Ronghang Hu, Huazhe Xu. Mareus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. Natural language object retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016.
[32] Ziling Huang and Shin ichi Satoh. Referring image segmentation via joint mask contextual embedding learning and progressive alignment network. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7753-7762, 2023.
[33] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mddtr-modulatee detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780-1790 1790, 2021.
[34] Sahar Kazemzadeh, Vicente Ordonez. Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787-798, 2014.
[35] Seayeon Kim, Minguk Kang, Dongwon Kim, Jaesik Park, and Suha Kwak. Extending clip's image-text alignment to referring image segmentation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4611 4628, 2024.
[36] Sungwoong Kim, Daejin Jo, Donghoon Lee, and Jongmin Kim. Magvlt: Masked generative vision-and- language transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23338-23348, 2023.
[37] Wonjae Kim, Bokyung Son, and Iodoo Kim. Vilt: Vision-and- Vision-and-language transformer without convolution or region supervision. In International Conference on Machine Learning, pages 5583-5594, PMLR, 2021.
[38] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Speneer Whitehead, Alexander C Berg. Wan- Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 00116-426. 2023.
[39] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata. Joshua Kravitz, Stephanie Chen, Yannis Kalantidis. Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
[40] Alex Krizhevsky, Iya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NewrlPS, 2012.
[41] Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. EMNLP 2018, page 66, 2018.
[42] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov. Matteo Malloci. Alexander Kolesnikov, et al. The open images dataset v4; Unified image classification, object detection, and visual relationship detection scale. International journal of computer vision, 128(7):195) (1567))995-1981. 2020.
[43] Gang Li, Heliang Zheng, Daqing Liu, Chaoyue Wang, Bing Su, and Changwen Zheng. Semmae: Semantic-guided masking for learning masked autoencoders. Advances in Neural Information Processing Systems, 35: 14290- -4400,, 2022.
[44] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694-9705. 2021.
[45] Muchen Li and Leonid Sigal. Referring transformer: A one-step approach to multi-tast task visual grounding. Advances in Neural Information Processing Systems, 34: 19652-19664,, 2021.
12

---

[46] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In European Conference on Computer Vision, pages 280-296. Springer, 2022.
[47] Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language- image pre-training via masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23390-23400, 2023.
[48] Zhitian Li, Wuhao Yang, Linhui Xiao, Xingyin Xiong, Zheng Wang, and Xudong Zou, Integrated wearable indoor positioning system based on visible light positioning and inertial navigation using unseented kalman filter. In 2019 1 11th International Conference on Wireless Communications and Signal Processing (WCSP), pages 1-6. IEEE, 2019.
[49] Zizhang Li. Mengmeng Wang. Jianbiao Mei, and Yong Liu. Mail: A unified bask-image--angee trimodal network for referring image segmentation. arXiv preprint 2rXiv22111.10444 2021.
[50] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[51] Tsung-Yl Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-290000090 2988, 2017.
[52] Daqing Liu, Hanwang Zhang, Feng Wu, and Zheng-Jun Zha. Learning to assemble neural module tree networks for visual grounding. In ICCV, pages 4673-4682, 2019.
[53] Jiang Liu, Hui Ding, Zhaowei Cai, Yuting Zhang, Ravi Kumar Satzoda, Vijay Mahadevan, and R Man- matha. Polyformer: Referring image segmentation as sequential polygon generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18653 18663,, 2023.
[54] Shilong Liu, Shijia Huang, Feng Li, Hao Zhang, Yaoyuan Liang, Hang Su, Jun Zhu, and Lei Zhang. Dq-detr: Dual query detection transformer for phrase extraction and grounding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1728-1736, 2023.
[55] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Yang, Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set ohject detection. arXiv preprint arXiv:2303.054909 2023.
[56] Xihui Liu, Zihao Wang, Jing Shao, Xiaogang Wang, and Hongsheng Li. Improving referring expression grounding with cross cross-modal attention attention-guided crasing. In CVPR, pages 1950-1959, 2019.
[57] Yaho Liu, Jinghua Wang, Chao Huang, Yaowei Wang, and Yong Xu. Cigar: Cross-modality graph reasoning for domain adaptive object detection. In Proceedings of the IEFRICVF Conference on Computer Vision and Puttern Recognition, pages 23776-23786. 2023.
[58] Yabo Liu, Jinghua Wang, Linhui Xiao, Chengliang Liu, Zhihao Wu, and Yong Xu. Foregroundness-avv aware task disentanglement and self-paced curriculum learning for domain adaptive object detection. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[59] Yunfei Liu, Zhitian Li, Liahui Xiao, Shuaikang Zheng, Pengcheng Cai, Haifeng Zhang, Pengcheng Zheng. and Xudong Zou, Edo-calibr: visual-aided imu calibration based on froqueney-domaie optimization. Measurement Science and Technology. ((44)045108 2023.
[60] 7e Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages
2021.
[61] Ziyang Luo, Yadong Xi, Rongsheng Zhang, GongZheng Li, Zeng Zhao, and Jing Ma. Conditioned masked language and image modeling for image-text dense retrieval. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 130-140, 2022.
[62] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016.
[63] Fausto Milletari, Nassir Navab, and Seyed-Ahman Ahmadi, V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV). pages (65- Icee, 2016.
13

---

[64] Duy Kien Nguyen, Yanghao LI, Vaibhav Aggarwal, Martin R Oswald, Alexander Kirillov, Cees GM Snoek, and Xinlei Chen. R-mae: Regions meet masked autoencoders. In The Twelfth International Conference on Learning Representations. 2023.
[65] Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned photographs. Advances neural information processing systems, 24, 2011.
[66] Fang Peng. Xiaoshan Yang. Linhui Xiao, Yaowei Wang, and Changsheng Xu. Sgva-clip: Semantic- guided visual adapting of vision-language models for few-shot image classification. IEEE Transactions on Multimedia, 2023.
[67] Zhiliang Peng. Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image mod-
cling with vector-quantized visual tokenizers. ArXiv, abs/2208. 06366, 2022. URL https://apl. momanticscholas ,
251554649.
[68] Bryan A Plummer, Liwei Wang. Chris M Cervantes, Juan Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2641-2649, 2015.
[69] Yanyuan Qiao, Chaorui Deng, and Qi Wu. Referring expression comprehension: A survey methods and datascts. IEEF Transactions on Multimedia, 23:4426-4400, 2020.
[70] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763 PMLR, 2021.
[71] J Redmon. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference computer vision and pattern recognition, 2016.
[72] Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv: (o::Xiv:88444 1804.02667, 2018.
[73] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and loss for bounding box regression. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 658-666, 2019.
[74) Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, and Bernt Schiele. Grounding of textual phrases in images by reconstruction. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14, pages 817-834. Springer, 2016.
[75] Christoph Schuhmann, Richard Vencu, Romain Bcaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Thco Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open datasct of clip-filterdd 400 million text pairs. arXiv preprint arXiv:2111.02114 2021.
[76] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quae Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-expers layer. International Conference on Learning Representations, 2016.
[77] Fengyuan Shi, Ruopeng Gao, Weilin Huang, and Limin Wang. Dynamic mdetr: dynamic multi- modal transformer decoder for visual grounding. IEFE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[78] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wajciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15638- 15650, 2022.
[79] Wei Su, Peihan Miao, Huanzhang Dou, Gaoang Wang, Liang Qiao, Zheyang Li, and Xi Li. Language adaptive weight generation for multi-task visual grounding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10866, 2023.
[80] Jiajia Tang, Kang Li, Ming Hou, Xuanyu Jin, Wanzeng Kong, Yu Ding, and Qibin Zhao. Mmt Multi-way modti-modal transformer for multimodal learning. In IJCAL, pages 3458-3465, 2022.
[81] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava. Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 307.020. 2023.
14

---

[82] Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 30, 2017.
[83] Haoqing Wang, Yehui Tang. Yunhe Wang. Jianyuan Guo, Zhi-Hong Deng, and Kai Han. Masked image modeling with local multi-scale reconstruction. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 21 2122-1131, 2023.
[84] Ning Wang, Jiajun Deng, and Mingbo Jia. spcle-consistency learning for captioning and grounding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 5535 5543,,5533555 2024.
[85] Peng Wang. An Yang. Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma. Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence- to-sequence learning framework. In International Conference on Machine Learning, pages 33318-23344 PMLR, 2022.
[86] Peng Wang. Shijie Wang, Junyang Lin, Shuai Bai, Xiaohuan Zhou, Jingren Zhou, Xinggang Wang, and Chang Zhou. One-peace: Exploring one general representation model toward unlimited modalities. arXiv preprint arXiv:2305. 11172, 2023.
[87] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorek, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, ct al. Image as a foreign language: Beit pre- training for vision and vision-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19175-19186, 2023.
[88] Xudong Wang, Shufan Li, Konstantinos Kallidromitis, Yusuke Kato, Kazuki Kozuka, and Trevor Darrell. Hicrarchical open-vocabllary universal image segmentation. Advances in Neural Information Processing Systems, 36, 2024.
[89] Zhaoqing Wang. Yu Lu, Qiang Li. Xungiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1 11686-11695, 2022.
[90] Linhui Xiao, Jinge Wang. Xiaosong Qiu, Zheng Rong, and Xudong Zou. Dynamic-slam: Semantic monocular visual localization and mapping based on deep learning in dynamic environment. Robotics and Autonomoux Systems, 1 117:1-16, 2019.
[91] Linhui Xiao, Xiaoshan Yang, Fang Peng, Ming Yan, Yaowei Wang, and Changsheng Xu. Clip-vg: Self-pacee curriculum adapting of clip for visual grounding. IEFF Transactions on Multimedia, 2023.
[92] Linhui Xiao, Xiaoshan Yang. Fang Peng, Yaowei Wang, and Changsheng Xu. HiVG: Hicrarchical multi- modal ina-grained modulation for visual grounding. In Proceedings of the 32nd ACM International Con- ference on Multimedia (MM 24), 2024. URL //tptree///peeevieeee net/ Wiw.rnn 1kKKZ.
/
[93] Bin Yan. Yi Jiang. Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Universal instance perception as object discovery and retrieval. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15325-15336. 2023.
[94] Zhao Yang, Jiqqi Wang. Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. Lavt: Language- aware vision transformer for referring image segmentation. In CVPR, pages 18155-18165, 2022.
[95] Zhengyuan Yang, Boqing Gong, Liwei Wang, Wenbing Huang, Dong Yu, and Jiebo Luo. A fast and accurate sne-stage approach to visual grounding. In ICCV, pages 4683-4693, 2019.
[96] Zhengyuan Yang, Tianlang Chen, Liwei Wang, and Jiebo Luo. Improving one-stage visual grounding by recursive sub-query construction. In ECCV, pages 387-404, 2020.
[97] Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. Unitab: Unifying text and box outputs for grounded vasion-language modeling, In European Conference on Computer Vision, pages 521-539. Springer, 2022.
[98] Jiabo Ye, Junfeng Tian, Ming Yan, Xiaoshan Yang, Xuwu Wang. Ji Zhang, Liang He, and Xin Lin. Shifting more attention to visual backbone: Query-modulated modulated refinement networks for end-to-end visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15502-15512 15512, 2022.
[99] Jiabo Ye, Junfeng Tian, Ming Yan, Haiyang Xu, Qinghao Ye, Yaya Shi. Xiaoshan Yang. Xuwu Wang. Ji Zhang, Liang He, et al. Uniqmet: Unifying referring expression grounding and segmentation with qrnet. ACM Transactions on Multimedia Computing, Communications and Applications. 2024.
15

---

[100] Нaoxuan You. Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang. Liangliang Cao. Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. In The Twelfth International Conference on Learning Representations, 2024.
[101] Licheng Yu, Patrick Poirson. Shan Yang, Alexander C Berg. and Tamara Berg, Modeling context in referring expressions. Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69-85. Springer, 2016.
[102] Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara Berg. Mattnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1315- 2018.
[103] Heng Zhao, Joey Tianyi Zhou, and Yew-Soon Ong. WordZpix: Word to pixel cross-attention transformer in visual grounding. IEEE Transactions on Neural Networks and Learning Systems, 2022.
[104] Zijia Zhao, Longteng Guo, Xingjian He, Shuai Shao, Zehuan Yuan, and Jing Liu. Mamo: masked multimodal modeling for fine-grained vision-language representation learning. arXiv preprint arXiv:221004418. 2022.
[105] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühh. Objects as points. arXiv preprint 2019.
[106] Yiyi Zhou, Rongrong Ji, Gen Luo, Xiaoshuai Sun, Jinsong Su, Xinghao Ding, Chia- The Lin, and Qi Tian. A real-time global inference network for onc-stage referring expression comprehension. 2021.
[107] Chaoyang Zhu, Yiyi Zhou, Yunhang Shen, Gen Luo, Xingjia Pan, Mingbao Lin, Chao Chen, Liujuan Cao, Xiaoshuai Sun, and Rongrong Ji. Segtr: simple yet universal network for visual grounding. In European Conference on Computer Vision, pages 598 615. Springer, 2022.
[108] Hong Zhu, Qingyang Lu, Lei Xue, Mogen Xue, Guanglin Yuan, and Bineng Zhong, Visual grounding with joint multi-mode representation and interaction, IEEE Transactions on Instrumentation and Measurement, 2023.
[109] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like like visual explanations by watching movics and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19-27, 2015.
16

---

Appendix
We provide an overview of the Appendix below:
Appendix A: Explanation of the task definition
Appendix B: Introduction of the datasets
Appendix B.1 The five referring datasets.
Appendix B.2 Explanation of the dataset abbreviations.
Appendix B.3 Comparison of datasets used in the pre-trained models.
Appendix C: Implementation details
Appendix D: Technical remarks
Appendix D.I Further explanation for the effectiveness mechanism of the visual target- relation score.
Appendix D.2 The selection of the unsupervised regions.
Appendix D.3 Referring-aware text masking.
Appendix D.4 The difference of the task heads between ours with other frameworks.
Appendix E: Extra experimental results
Appendix E, The results on phrase grounding task under mixup pre-training setting.
Appendix E.3 Computational costs analysis compared with SoTA methods.
Appendix E.4 Complete ablation study of MRefM.
Appendix E.5 Ablation study of the mask ratio in referring- aware dynamic masking.
Appendix F: Visualization of the results
Appendix G: Further discussions
Appendix G.I Limitations.
Appendix G.2 Broader impacts.
Explanation of the task definition
As explained in Sec. of the main text, Visual Grounding (VG) aims to grounding region referred by a query text a specific image. The generalized visual grounding includes Referring Expression Comprehension (REC), Phrase Grounding (PG), and Referring Expression Segmentation (RES) tasks. However, in recent years, REC and RES have often been studied separately. Therefore, in numerous works [14, 15, 79, 92, 91, 98], visual grounding specifically refers to REC and PG tasks, which involve grounding a rectangular region. In this paper, we follow the mainstream and have not clearly separated the "grounding" from "generalized visual grounding" and "REC and PG tasks". When expressing the experimental task, "grounding" usually refers to REC or PG tasks, so as to discuss parallelly with the RES task.
B Introduction of the datasets
B.1 The five referring datasets
We present the detailed descriptions of the five referring datasets used our experimental study on the REC, PG, and RES tasks. Tab. 8 presents the detailed statistics.
These three datasets belong the Referring Expression Comprehension (REC) and Referring Expression Segmentation (RES) tasks, and the images of these three datasets derived from MSCOCO [50]. Expressions RefCOCO [101] and RefCOCO+ [101] are collected by the two-player game proposed in ReferitGame [34]. There are two test splits called "testA" and "testB"". Images in "testA" only contain multiple people annotation.
contrast, images in
"testB" contain all other objects. Expressions in RefCOCOg [62] are collected on Amazon Mechanical Turk non-interactive setting. Thus, the expressions in RefCOCOg are longer and more complex. RefCOCOg has "google" and "umd" splits. The "google" split does not have public test set,
17

---

Table 8: The detailed statistics of RefCOCO [101], ReiCOCO+ [[01], RefCOCOg [62], ReferltGame [34] and Flickr30K Entities [68] datasets. We represent test split and testA split in the same column.
total queries
train queries
test(A)
testB queries
Dataset
Images
Instances
queries
queries
RefCOCO [101]
19,994
50,000
142,210
120,624
10,834
5,657
5,095
Re(COCO+[101]
19,992
49,856
141,564
120,191
10,768
5,726
4,889
RefCOCOg [62]
25,799
49,822
95,010
80,512
4,896
9,602
ReferltGame(44
20,000
19,987
120.072
$4,127
5,842
60,103
Flickr30k [68]
31,783
427,000
456,107
427,193
14,433
14,481
Table 9: Comparison of datasets used in the pre-trained models of the comparable methods.
Pretmined model
Uni-mrddal Image// Data sine
Uni-medsl text Data sirne
Imaageient pains Duts size
Total
CLIP [70]
LATON-600M [25] /400M
400M
UnITAB [96]
imagestett paire CCSME), etc : image-tret-box poine COCO, VD. RefC. 0365, SUR, et:: from multiple dewmettrea tack
>2MM
OFN [55]
ImageNet 21K. 40M
liltered BookCorpusl 109]etc./ 140GB CC12ME SBC(s)), COCOEST /G 22M
40M+150GB-2LM
DVA [21]
Mergedi 2BLLAOON 2000=+COYO 700M)/>20
>:2000M
Flan13 [2]]
fillere craed dana 75750G
>-5003B
ONE-PEACE [6]]
image test pains LAION-2B 175], audio text pains: 2.4M 8000 hours
>2000MM
BErT-3 (87]
ImageNet 21K] / 14M
BookCorpus| (0)), 150GB UCT2ME SBU (BU[))) COCO VG /21M 14M+1500BB-210
and exists an overlap between the training and validation image sets. The "umd" split does not have this overlap. Therefore, to prevent data leakage of the test set and following previous studies [79, 102], we exclude the "google" split in the fine-tuning settting and dataset-mixed pre-training setting. Thus, we trained and tested the RefCOCOg dataset only on the "umd" split.
ReferItGame. ReferltGame [34] (short as Referlt) belongs to the Phrase Grounding (PG) task, which contains images from SAIAPRI2 [20] and collects expressions through two-player game. In this game, the first player is shown an image with an object annotation and is asked to write a natural language expression referring to the object. The second player then shown the same image along with the written expression and is asked to click on the corresponding area of the object. If the clicking is correct, both players receive points and swap roles. If not, a new image will presented.
Flickr30k Entities. Flickr30k Entities (short as Flickr30k) [68] belongs to the phrase grounding task, which contains images in Flickr30k dataset. The query sentences short noun phrases in the captions of the image. The queries are simpler and easier to understand compared to ReSCOCO/+/g. Therefore, the ambiguity of the expression is heightened simultaneously, resulting in a relative increase in noise.
B.2 Explanation of the dataset abbreviations
In Tab. 2 of the main text, we provide abbreviations for the datasets used intermediate pre-training. Specifically, 'GoldG' (proposed in MDETR [33]) is mixed region-level fine-grained dataset created by combining three datasets- Flickr30k [68], MS COCO [50], and Visual Genome [39] along with annotated text data for detection, REC and QGA tasks. It has size of approximately 6.2M. "O365' refers the Object365 [105] dataset, 'SBU' stands for SBU caption [65], 'VG' represents the Visual Genome [39] dataset, and 'OI' stands for OpenImage [42] dataset.
B.3 Comparison of datasets used in the pre-trained models
As presented in Tab. 9, we conducted an analysis of the datasets employed by the backbone models compared in Tab. 2 within the dataset-mixed pre-training setting, From the Tab. 9, it is evident that BETT-3 and OFA utilize comparable datasets for pre-training. Conversely, other compared works in Tab. 2, such Shikra [10], Ferret [100], LION [9], and other models, such ONE-PEACE [86] (a tri-modality foundation model), employ significantly larger amounts of data than BEÎT-3. Consequently, our method does not possess any advantage concerning the volume of data used in pre-training.
C Implementation details
Network Architecture. The detailed network structure of our framework shown in Tab. 10. We employ BEiT-B/16 and BEiT-L/16 the backbone for our OneRef base and large version,
18

---

Table 10: Network structure of proposed OneRef framework.
Model
Backbone
Input resolution
One-tower Transformer
All perameters
layers
dimension
heads
(include all MoE heads)
OneRef-B OneRef-L
BEIT-R/16 BEiT-L/16
384
12
768
12
234M (RFC), 267M (RES)
384
24
1024
16
639M (REC). 679M (RES)
Table 11: Hyperparameters of our framework during training. Ir. denotes the learning rate.
Value
Item
basc model
large model
optimizer
Adam 110
Epoch for MRefM pre-training Ir for MRefM pre-training
0.5x10
weight decay
0,5x10-5
patch size
16x 16
Initial value of aspect ratio in MIM
0.3
mask ratio / Referring MIM
0.75
mask ratio 7 Referring MIM
0.35
mask ratio in Referring MLM
0.40
Areo and Azmo Referning MLM
1,
batch size for MRefM pro-training
32
8
Epoch REC/EES transfer
Ir. for REC/RES transfer
x,3
Aiy Agtou RBC
Ay, Aa in REC
20, 2
Aseg_ Asea_a in RES
20, 2
batch size for REC/RES transfer
32 16
8/6
respectively. the structure OneRef-B, the one-tower encoder are 12-layer Transformers with the hidden embedding dimension of 768. In the structure of OneRef-L. the one-tower encoder is 24-layer Transformers with the hidden embedding dimension of 1024. The one-tower encoder encodes both textual and visual modalities. Due to the utilization of 3-layer deconvolution, RES exhibits slightly higher number model parameters compared to the REC task.
Training Details. The batch size for pre-training the base model and large model are (32, 8), while they are (32, 8) and (16. 6) for transferring to the REC and RES tasks, respectively. Our model is optimized and-to-end by using the AdamW W optimizer and a cosine learning scheduler with an initial learning rate of 0.5 10 for 110 epochs during the pre-training stage. During REC/RES transfer stage, the learning rates is 0.3 x 10-1 with 20 epochs. The framework and experiments in our study were conducted using PyTorch. For MRefM pre- training,, the base model took 15 hours on 32 NVIDIA A100 GPUs, while the large model took 50 hours on the same number of GPUs. As for REC/RES transfer fine-tuning training, it took an average of 3 hours for the base model and hours for the large model to process one dataset on 8 A100 GPUs.
Inference Details. Unlike previous methods, such Trans VG++ [15), QRNet [98], etc. which heavily rely on high-resolution images like 640x640, we adopt smaller resolution of 384x384. To ensure compatibility, we employ long edge alignment and short edge pad filling scheme to the image. We include [SEP] and [EOS] token at the beginning and the end of the input text, and align a fixed length of 64 by padding empty tokens.
Model Hyperparameters. We summarize and report the hyperparameter settings of the OneRef framework in Tab. 11.
D Technical remarks
D.1 Further explanation for the effectiveness mechanism of the visual target-relation score
(1) The purpose of designing the Referring MIM algorithm. In the existing MIM paradigm, reconstruction limited to solely relying on the visual features within the image. To enhance content reconstruction by leveraging cross-modal information as much as possible, our Referring MIM approach incorporates visual target-relation scores alongside visual modality content during reconstruction. This modeling approach presents increased difficulty it necessitates reliance
19

---

Befering discouu
Refered
Psich
Pe
(x x)yW
G 9a/H
P/We
P/Ay
y-mmask
w--nmak
I-mask
The Visual Target-relation Score
Referring text: the giraffe walking while another giraffe follows it.
Figure 5: The reconstruction of the visual target-relation score s E N.x, (x, y) represents the coordinates of a general image patch, and the patch size. By slicing the predicted score, four masks can be derived. The score represents the spatial distance and the relative size between the current patch region and the referred region.
textual information for reconstructing the two visual branch. Consequently, our model achieves a more comprehensive understanding of both visual and textual information. In this way, the model not only can perceive the information of the image modality itself but also have a more accurate understanding of the location and correlation of the key object features in different regions.
(2) How and why the visual target-relation score (i.e., the x-, y', W-, and h-masks) works. We provide a clearer illustration in Fig. 5 for further explanation. As mentioned in Sec. 3.2, this score represents the spatial distance between the current patch region and the referred region, it enables implicit deployment of grounding capability within each token of the model. When reconstructing the visual features and target-relation score of each local patch, the model actually needs have an global and comprehensive understanding of the text modality information and the visual information. On this basis, the model needs to rely on the reconstructed visual features of the local patch to implicitly predict the specific location and size of the referred object, and then accurately predict the visual target-relation score. Finally, Referring MIM can enhance the model's s global and multimodal general visual representations,
understanding of textual and visual information, and then learn which can have better generalization ability when deployed to downstream referring tasks.
The proposed Referring MIM is own design, which is mainly used to improve the defects existing in MAE [27]/BEiT [4]. We can find the rationale of our method in some classic computer vision works, such as the YOLO series works [71]. which predicts the location, size, confidence, and category of the object box corresponding to each grid cell based on the global understanding of the image. YOLO etc. [71] also confirmed that the object detection model obtained in this way has stronger generalization ability when transfer to detection tasks that differ greatly from the training data compared with other detectors.
D.2
The selection of the unsupervised regions
The process of selecting unsupervised regions bears resemblance to weakly-superrised visual ground- ing. Drawing inspiration from ALBBFF's method [44] for weakly-supervised grounding, we employ BEiT-3 model with performed image-text contrastive tuning encode both the image and text, thereby obtaining cross-modal text-to-image attention map for selection. Subsequently, leveraging the cross-modal attention and modular parsing of textual sentences provided by MAttNet [102] enables us to derive scores for each proposal. Finally, we select the region with the highest score as objective in Referring MRefM.
D.3 Referring-aware text masking
In referring MLM, we utilize referring- aware text masking strategy. Specifically, we preferentially mask out the referential subject the expression text on the basis a random mask, and the subject is obtained by the NLP parsing tool (e.g., spaCy). Since this small technical point does not observe a significant performance gain as the referring-awant dynamic image masking strategy, we do not provide additional ablation experiments.
20

---

Table 12: Comparison with latest SoTA methods for PG task with dataset-mixed intermediate pre- training setting. 'RefC' represents the mixup of RefCOCO/// training data. i indicates RefC has been used during vre-training.
Visual/I anguage
Intermediate
Data size
Referlt test
Flickr test
Methods
Venue
Backbone
pretrain
Datasct-mixed intermediate pre-training setting
MDETR [33]
ICCV'21
RNIOL/RoBERT-B
GoldG.RefC
6.5M
83,80
YORO! [29]
ECCY'22
VILT [ BERT-B
GoldG,RefC
6.5M
71.90
UniTAB [97]
ECCY'22
RNIO1/RoBERT-B
VG,COCO,ete
>20M
79.38
HiVG-B [92]
2CMMM224
CLIP-B CLIP-B
Reft.Reffett.Fiick
0.8M
77.75
82.08
HiVG-1.1 [92]
ACMMM
CLIPAL//CIPP.
Reft,
0.8M
78.16
82.63
OneRef-B' (0.2B)
NeurlPS 24
BEIT3-B/ BEIT3-B
RefC.Referit
0.5M
79.66
84.01
OneRef-L. (0.6B)
NeurlPS'24 24
BEIT3-L/ BETT3-L
RefC,ReferIt
0.5M
83.22
85.13
Table 13:
Comparison with latest SoTA methods (oloU metric) on the three datasets for RES task with both single-datasct fine-tuning setting and dataset-mixet intermediate pre-training sctting. t indicates RefC has been used during pre-training.
Visnal/ Languge
Intermediate
RefCOCO
RefCDCO VostA
RefCOCO0
Methods
Venue
Backbone
pectrain data
val
bestA
BoadB
lesiB
test
Single-dataset fine tuninng setting
modal pro-trained close-set segmentation model: (traditional setting)
LAVI [94]
CVPR'22
Swin B/ BERT-B
72.73
75.82
68.79
62.14
68.38
55.10
61.24
62.09
Single-dataset fine-tuning setting
vision language self self-suerviseed pre-trainee model:
RISCLIF-B [35]
NAACL221
CLIP-B/ CLIP-B
73:57
76.46
RISCLIP-L,P35
NAACU/24
CLIP-L/ CLIP-L
76.92
80.99
73.04
71.24
76.99
61.56
67.96
68.71
OmeRaf-B (ours)
24
BETT3-8/ BETT3-
77.55
80.96
73.53
70.82
74.53
64.06
70.68
70.61
OneRef-I. (ours)
N4urPP5224
BETT3-1/ BEIT3-L
S0L.88
82.78
78.27
74.25
69.85
Datuset-mied Intermediate pre-training setting:
HIPIR! [88]
23
RNSOCLIP BERT-B
Re66, PACO
78.30
66.20
69 80
UNINEXT [93]
CVPR'23
RN50 / BERT-B
RefC,0365 S3T
77:90
79.68
75.77
66.20
71.22
59.01
70.04
70:52
PolyFormer [69]
CVPR'23
Swin B BERT-B
RefC
74,82
76.64
71.06
67.64
72.89
59.33
67.76
69.05
PolyFormer-U [53]
CVPR'23
Swin-L/ BERT-B
RefC
75.96
78.29
73.25
69.33
74.56
61.87
69:20
70.19
OneRef-B (ours)
24
BEIT3-8/ / BETT3-B
RelCC
81.06
83.05
77. 80
72.24
77.32
67.08
75.14
77.21
OneRef-1.'
NemrPS'24
BEIT3-1/ BETT3-I.
83.62
79.54
70.65
79.18
D.4 The difference of the task heads between ours with other frameworks
Recently, several multi-task visual grounding studies [79, 45] have incorporated both grounding and segmentation task heads into their frameworks. Most relevance to our work is VG-LAW [79], which simplifies the implementation of grounding and segmentation heads by eliminating Transformer-base fusion encoders through visual adaptive weights generation. In contrast, for REC headers, we propose a box mask constraint based on cross-modal cosine similarity that significantly enhances the accuracy of such grounding approach. For the RES head, instead of employing adaptive weights generation, we directly obtain segmentation masks using cosine similarity for the visual tokens upsampled by 3-layer deconvolution.
E
Extra experimental results
E.1
The results on ReferIt and Flickr30k dataset under mixup pre-training setting
The results of our framework on the PG task (i.e., ReferIt [34] and Flickr30k [68] datasets) under the mixup pre-training setting are presented in Tab. 12. It is worth noting that majority of studies conducted under this setting have not provided these results, thus only several works are included in the table. As shown in Tab. 12, our base model outperforms HiVG by 1.91% and 1.93% on the two datasets, and also achieves SoTA performance.
E.2 The results for RES task under oloU metric
The results for RES task under oloU metric are presented in Tab. 13. oloU is calculated as the ratio between the total intersection area and the total union area of test samples, each of which is consists of both text query and an image. This metric particularly favors larger objects. As indicated in Tab. 13, (1) in the single-dataset fine-tuning setting, our base model outperforms RISCLIP [35] by 3. 3.77%. 8.87%. and 5.52% RefCOCO/+/g testB/testB/test split, respectively. (2) Similarly, in the
21

---

Table 14: Comparison of the computational cost in REC task. The results obtained on RefCOCO dataset. The testing environment is NVIDIA A100 GPU. f indicates that the model's code is not publicly available, and the replicated estimation results are shown. The backbone parameters of our UniRef model only include the actual calculated parameters, specifically those of the V-L. expert head in MoF, while excluding the parameters of unused visual and language expert heads and their uni-modal branches. We highlight the best result in bold. (FPS: images / (GPU second))
Backbone param..
Fusion-head param..
Total
FLOPs (G)
Fine-tune FPSt
Test FPST
testA time]
testA Acc.f
Model
param..
TransVG [14]
150M
21M
171M
214
22.8
59.6
95s
82.7
QRNet [98]
252M
21M
273M
540
9,4
50.9
111s
85.9
VransV++++ [15]
161M
10M
171M
396
644s
88.4
MDETR [33]
150M
135M
185M
642
4.7
19.9
283s
89.6
Grounding-D DINO [55]
156M
15M
172M
464
8.3
681s
91.8
UniRef (Ours)
147M
1.7M
149M
162
55.8
83.2
68s
94.3
Table 15: Complete ablation study of MRefM using our OneRef-base model in REC task on both single-dataset fine-tuning setting and mixup intermediate pre-training setting. (Acc@0.5(%))
image masking
RefCOCO testA
RefCOCO+ testA
RefCOCOg
MIM
MLM
strategy
TestB
val
test
Single-datuset fine-tuning setting:
X
X
x
85.23
88.13
83.82
78.56
83.36
71.72
80.41
80,52
vanilla
vanilla
block-wise referring-aware
85.75
88.
83,43
78.47
84.27
71.66
81.06
81.19
Ref MIM
Ref MLM
88.75
90.95
85,34
80.43
86.46
74.26
83.68
83.52
Dataset-mixed intermediate pre-training setting:
(main)
X
X
x
85.23
88.13
83.82
78.55
83.36
71.72
80.41
80.52
vanilla
vanilla
random
86.60
89.86
84.96
79.68
84.59
72.11
81.35
81.11
vanilla
vanilla
referring-aware
86.71
90.58
85.33
80.06
85.77
73.96
81.96
82.16
Ref MIM
vanilla
referring-aware
88.86
92.12
86.89
83.64
88.26
76.58
83.55
85.86
vanilla
Ref MLM
referring-aware
87.26
91.68
86.37
81.52
86.87
75.93
82.88
84.32
Ref MIM
Ref MLM
rundom
90.56
93.55
88.23
85,08
89.12
78.56
85.57
86.89
Ref MIM
Ref MLM
block wise
90.07
93.32
88.21
84.55
88.83
77.98
84.71
86.69
Ref MIM
Ref MLM
referring-aware
91.89
94.31
88.58
86.38
90.38
79.47
86.82
87.32
dataset-mixed intermediate pre-training setting, our base model surpasses UNINEXT [93] by 2.03%,
8.07%, and 6.69% on RefCOCO/+/g testB/testB/test split, respectively. Furthermore, our large model exhibits remarkable scalability with additional performance enhancements.
E.3 Computational costs analysis compared with SoTA methods
In this paper, we highlight two significant advantages of our model architecture over other frameworks: (a) Instead of using Transformer to fuse visual and language features, we only employ a simple lightweight task head; (b) Our one-tower architecture eliminates the need for early interaction techniques in the backbone network, thereby reducing the computational complexity of the model.
We compare the energy efficiency of our model with several well-known SoTA works on the REC task from various perspectives, including the number of parameters, computational complexity (FLOPs), inference speed (FPS), and test time (s). As can be seen from Tab. 14, due to the simplification of our model's structure, the number of parameters and the calculation complexity are significantly lower than other well-known models. Specifically, our feature fusion and grounding head module only require 1.7M parameters, while other methods use 20M, meaning we only have about 8.5% of their parameter count. Additionally, our computation is only 34.9% of Grounding-DDin -IIO and 25.2% of MDETR. Moreover, our inference speed is 10 faster than Grounding DINO and VG++ (the speed also related to the image size used by the model). Despite these advantages, thanks to the modality feature space, outperform all these well-known works.
E.4 Complete ablation study of MRefM on single-dataset fine-tuning and mixup pretrain settings
The complete ablation results of MRefM on both single-dataset fine-tuning and mixup pretrain settings are provided in Tab. 15, which serves as supplement Tab. in the main text. In the table, the masking ratio is set to 0.4 when using block-wise or random masking strategies.
22

---

Table 16: Ablation study of the mask ratio in referring aware dynamic masking strategy Ref-
COCOggval) dataset.
mask ratio
RefCOCOg Acc@0.5(%)
mask ratio
RefCOCOg
B
Acc@0.5(%)
0.20
0.75
83,40
0.35
0.50
85.98
0.30
0.75
85.01
0.35
0.60
86.32
0.35
0.75
86.82
0.35
0.70
86,62
0.40
0.75
86.62
0.35
0.75
86.82
0.50
0.75
86.23
0.35
0.80
86.19
0.60
0.75
84.72
0.35
0.85
85.56
0.70
0.75
83.39
0.35
0.90
84.87
Query text: arm right. "
Query text: front wine glass."
RefCOCO
Query text: "the young child on the left sitting near the old person.
Hound TUR
Query "woman the left of old lady.
CUID BILR
Original image
Ground truth
RES Prediction
REC Prediction
Cross-modal feature
Figure 6: Qualitative results of our OneRef framework on the RefCOCO-val split. Each example shows two different query texts. From left to right: the original input image, the ground truth with box and segmentation mask (in green), the RES prediction of OneRef (in cyan), the REC prediction of OneRef (in cyan), and the cross-modal feature.
E.5 Ablation study of the mask ratio in referring-aware dynamic masking strategy
As depicted in Tab. 16, we conducted ablation experiments on the mask ratio within our proposed referring-awan dynamic image masking strategy. It is observed that while high mask ratio of 0.75 employed for the pixel reconstruction of MAE [27], achieving better results for BEiT's feature reconstruction requires mask ratio ranging from approximately 0.4 to 0.45. In our proposed approach, favorable outcomes can be attained by setting and 7 to 0.35 and 0.75, respectively: where 3 represents the mask ratio beyond the referred region and 7 denotes the mask ratio within Experimental statistics show that our entire mask rate a in each sample is about 0.4 ~ 0.5.
Visualization of the results
As shown in Fig. Fig. 7, and Fig. 8, we present the qualitative grounding and referring segmentation results with several relatively challenging examples. Each example shows two different query texts. The cross-modal features are obtained by the cosine similarity between the [SEP] language token and the vision tokens on REC transfer model of OneRef-B. These results demonstrate the strong semantic comprehension capability of our OneRef model complex understanding and cross-modal grounding.
23

---

Query text "white tourist ship closest viewer.
Bouremum
Query text: "a decorated passenger ship waiting for departure in port.
ReFCOCO+
Query text: "woman with blond hair.
Query text: "a blue shirt giving presentation.
Hile:
Original image
Ground truth
RES Prediction
REC Prediction
Cross-odaa feature
Figure Qualitative results on the RefCOCO+vval dataset. The annotation is the same as Fig.
Query text: "a plate of cookies and banana slice, raisins, and craisins."
gound mum
pred. lour 0.92
Query text: "a bowl of pasta next to bowl of salad.
ground Truth
prec IOLE 0.33
RefCOCOg
Query text: "a group of six oranges wooden table surrounded several other fruits and vegetables." "
UI
red DUr
Query text: "the bag of vegetables behind all of the other fruits and vegetables."
rll lou 0.03
Original image
Ground truth
RES Prediction
REC Prediction
Cruss-modal feature
Figure Qualitative results on the RefCOCOg-val dataset.
The annotation is the same as Fig.
24

---

G
Further discussions
G.1
Limitations
Firstly, despite achieving remarkablc grounding and segmentation results, the pre-training in this paper solely relies on the comparatively limited RefC dataset, as opposed other studies with larger datasets.
Secondly, the MRefM paradigm necessitates additional referential bounding boxes supervised data compared to self-supervised pre-training. Therefore, we explore the potential of unsupervised pre-training for MRefM. However, when utilizing image-text pairs obtained from web crawling, there no guarantee that the referred regions will exhibit strong correlation with the text due to many texts describing the entire image. This aspect introduces certain challenges and biases during large-scale pre-training of MRefM. Consequently, this paper should serve as an inspiration for subsequent researchers to propose more convenient plug-and-pla and-play modeling methods.
G.2 Broader impacts
OneRef demonstrates strong grounding and referring segmentation capabilities, while MRefM represents novel modeling paradigm for referential relationship. This facilitates users easily utilize our model (e.g., OneRef-L) for their own needs by simply providing some appropriate text queries. However, this also raises concerns about how OneRef models with strong understanding capabilities could be used inappropriately in the community, such as for large-scale illegal video surveillance. The open-set grounding capabilities could be manipulated through specialized textual cues to facilitate targeted detection or human tracking instead of generic ones. This manipulation could introduce biases in the detector and result in unfair predictions.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the claims made, including the contributions made in the paper and important assumptions and limitations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of this work in the Appendix G.I.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
26

---

Justification:
We have provided the full set of assumptions and complete proof in Sec. 3 of the main text.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have fully disclosed the information needed to reproduce the main experimental results of the paper in Sec, 4 of the main text and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
27

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: We answer No mainly for the following acceptable reasons: (1) The data we use are all publicly available and have been detailedly introduced in the paper. Researchers can acquire the data according to the provided reference information. (2) Due to time constraints, were unable to compile and submit the anonymous code the time of submission. However, as stated in the abstract, all models and code will be promptly made public after the decision is reached on this paper. (3) The implementation details of our work have been thoroughly explained both the main text and supplementary materials. Even without publicly available code, researchers reproduce it based the information given in this paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/ public/guides Saaiides/oodeSubmiisee oniooionoooolll ooiicy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //ippscccpppii wttpccccppppcccccccccccccccccclt cerooliccguides... gwwggguid///// /CoeeSumii wswonnolll.. onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have specified all the training and test details in the Sec. 4 of the main text and Appendix B.I.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The error bars are not reported because it would be too computationally expensive. Besides, is not crucial for interpreting the experimental results in this task topic.
28

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the eπror bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We provide sufficient information on the computer resources Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https http/enrrip.eee httpp///////.cctccccciieeeeeeeee
Answer: [Yes]
Justification:
We have conformed in every respect with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]

---

We have discussed both potential positive societal impacts and negative societal impacts of the work performed in Appendix G.2.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-us should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited the original paper (e.g., BEYT-3 [87]) that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
30

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi wwwrsswithhoeee thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing research with human subjects.
Guidelines:
31

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "JcNRZCoKLs", "reviewer": "Reviewer_a3fR", "summary": "This manuscript proposes UniRef, a framework aimed at unifying visual and linguistic feature spaces for referring expression comprehension and segmentation. The key presented is the Masked Referring Modeling (MRefM) paradigm, which includes referring-aware MIM and MLM. \nThis approach seeks to streamline the architecture by eliminating the need for separate modality-specific encoders and complex interaction modules, achieving sota performance on several datasets.", "questions": "See weakness.", "limitations": "None.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The introduction of the MRefM paradigm effectively captures the referential relationship between visual and linguistic features, contributing to the robustness and accuracy of the model.\n2. The integration of visual and linguistic feature spaces into a unified framework simplifies the model architecture and potentially improves computational efficiency.\n3. The authors provide extensive experimental results across five datasets, demonstrating the effectiveness of the proposed approach in outperforming existing methods.", "weaknesses": "1. The technical contribution of the proposed method appears to be insufficient, as it primarily adapts traditional Masked Autoencoders into a 'Referring MAE', despite the rich experimentation presented.\n\n2. Section 3.2 is intricate and challenging to follow. I recommend a thorough proofreading and revision of this section to enhance its clarity and readability.\n\n3. Could you explain the rationale behind designing the system to encompass four masks: x-, y-, w-, and h-masks?\n\n4. The manuscript asserts that the approach is lightweight, raising questions about its real-time applicability. It would be helpful if the authors detailed the computational demands and processing speed of the method in practical scenarios.\n\n5. (Minor) While the manuscript has discussed the limitations of the proposed approach, I am interested in understanding how MrefM performs in more general contexts. For instance, evaluating the backbone linear probing (or its integration with LLMs) could demonstratebroader applicability. This should certainly be considered for future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful feedback. Please find below our responses to the questions raised in the review:\n\nFirstly, due to the character limit in the reply box, we have included the figure and experimental tables in the PDF file at the top of this page. Please click on the file to view the corresponding figure and tables.\n\n> **Q1. The technical contribution of the proposed method appears to be insufficient, as it primarily adapts traditional Masked Autoencoders into a 'Referring MAE', despite the rich experimentation presented.**\n\nWe would like to highlight our technical contributions from the following points, while explaining the benefits resulting from our approach.\n\n- Firstly, to address the limitation of the previous MIM paradigm in capturing cross-modal referential relations, we propose the Referring-aware MIM paradigm, which enhances the backbone network's ability to comprehend fine-grained cross-modal information.\n\n- Secondly, we propose the referring-aware dynamic image masking strategy that improves the previous random image masking strategy and effectively directs the model's attention to the referred region.\n\n- Thirdly, we propose a referring-aware MLM paradigm that not only empowers the existing backbone model to reconstruct masked words based on contextual information but also enhances the model's attention and weighting towards crucial referring words.\n- Fourthly, based on the proposed MRefM paradigm, we designed two lightweight task heads and introduced a remarkably simplified one-tower grounding and referring segmentation framework, thereby obviating the necessity for complex cross-modal interaction techniques and cumbersome fusion codec modules. This approach offers a new solution to the fields of grounding.\n\nThe existing methods, such as MAE, reconstruct the image based on its context information and can only learn the uni-modal representation. With our proposed method, we are able to learn more general cross-modal representations and thus enhance the model's global reasoning ability.\n\n> **Q2. Section 3.2 is intricate and challenging to follow. I recommend a thorough proofreading and revision of this section to enhance its clarity and readability.**\n\nAfter receiving the comments on this rebuttal, we conducted a thorough proofreading of Sec. 3.2 and identified some challenging aspects in its writing. Here are several points that may cause confusion. We will update in the revised version.\n\n- 1. Paragraph 1 in Sec. 3.2, the explanation of Referring MIM's motivation is insufficient. \n\n     Explanation: This issue will be explained in Q3 - point 1.\n\n- 2. Paragraph 2 in Sec. 3.2,  the effectiveness mechanism of the four x-, y-, w-, and h-masks needs to be further explained.\n\n     Explanation: This issue will be explained in Q3 - point 2. To facilitate explanation, we further draw Figure R1 based on Figure 2 of the main text, which also will be included in Section 3.2 of the revised version.\n\n- 3. Paragraph 4 in Sec. 3.2, this part lacks a global execution logic of 'Referring-aware dynamic image masking strategy' .\n\n     Explanation:  We have revised the writing logic of paragraph 4.\n\nWe hope the above-mentioned points meet the reviewer's requirements. \n\n>**Q3.Could you explain the rationale behind designing the system to encompass four masks: x-, y-, w-, and h-masks?**\n\n- The response to this issue is placed in the global rebuttal field. Please refer to Common Question 1 above on this page.\n\n>**Q4. It would be helpful if the authors detailed the computational demands and processing speed of the method in practical scenarios.**\n\nWe analyze our gains of computational efficiency in Tab. R1. As can be seen from the table, due to the simplification of our model's structure, the number of parameters and the calculation complexity are significantly lower than other well-known models. Specifically, our modality fusion and grounding head module only require 1.7M parameters, while other methods use 20M, meaning we only have about 8.5% of their parameter count. Additionally, our computation is only 34.9% of Grounding-DINO and 25.2% of MDETR. Moreover, our inference speed is 10 times faster than Grounding-DINO and TransVG++ . Despite the reduction in computational requirements, we already outperform all of these well-known works \n\n>**Q5.(Minor) While the manuscript has discussed the limitations of the proposed approach, I am interested in understanding how MrefM performs in more general contexts.**\n\nTo verify the generality of MRefM, as shown in Tab. R2 and Tab. R3, we follow the experimental test framework of BEiT-3 and conducted VQA fine-tuning experiments (Tab. 2), as well as cross-modal retrieval experiments (Tab. 3) on the MS COCO and Flickr30K datasets.\n\nOur proposed MRefM is a fine-grained multi-modal pre-training paradigm that significantly enhances cross-modal tasks that involving logical reasoning and referring. As shown in the tables, our MRefM pre-training also leads to a considerable improvements in both VQA and retrieval tasks, demonstrating the effectiveness of our MRefM in cross-modal representation learning. Additionally, integrating MRefM into LLMs is a interesting directions, we will attempt corresponding studies in the future works.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XiEIJ5ntDV", "reviewer": "Reviewer_vP75", "summary": "This paper proposes a Mask Referring Modeling (MRefM) paragram and a unified and extremely concise grounding and referring segmentation framework named UniRef that no longer requires the fusion or interaction of the Transformer structure and the special grounding tokens. A masked referring modeling (MRefM) is proposed to model the referential relationship, which encompasses referring-aware mask image modeling and referring-aware mask language modeling.", "questions": "See weakness.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "A mask-referring modeling paragram is proposed to model the referential relation between visual and language effectively. They also propose a one-tower framework for grounding and referring segmentation in a unified modality-shared feature space. Experiments demonstrate the effectiveness of this method and its components.", "weaknesses": "1. Lack of discussion of related work [a]. I suggest discussing the difference between this paper's referring-aware mask language modeling and masked contrastive learning in [a]. \n2. Lack of computation costs analysis with other methods, including parameters/FLOPs/speed. \n\n[a] VLT: Vision-Language Transformer and Query Generation for Referring Segmentation. In TPAMI 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful feedback. Please find below our responses to the questions raised in the review. \n\n> **Q1. Lack of discussion of related work [a]. I suggest discussing the difference between this paper's referring-aware mask language modeling and masked contrastive learning in [a].**\n\nWe will include a comparative discussion between our Referring-aware Mask Language Modeling (Referring MLM) approach and the Masked Contrastive Learning (MCL) approach proposed in VLT [a] in Sec. 2.2 and 3.3 of revised paper. Specifically, we currently briefly present the difference discussion during this rebuttal stage below:\n\n- Firstly, the MCL proposed in VLT [a] constructs contrastive learning for three different types (i.e., same image same object (SISO), same image different object (SIDO), different image (DI)) of referring samples in the training batch. The aim is to make the features with SISO sample type as close as possible and the features with SIDO sample type as far away as possible. At the same time, MCL randomly masks prominent words in the query text of SISO type in a probability-guided manner to construct new positive samples, thereby increasing sample discrimination and model generalization. MCL to some extent proves that text masking is effective in referring tasks.\n\n- Secondly, our Referring MLM involves masking the query text using a referring-aware text masking strategy and reconstructing the linguistic content as well as the semantic target-relation score of the masked words. Our approach differs from MCL. Specifically, while both Referring MLM and MCL employ text masking techniques, our proposed Referring MLM requires not only text masking but also the reconstruction of textual content and the target-relation probability distribution of the masked text. Our proposed Referring MLM paradigm is a relatively more comprehensive approach to perform mask modeling.\n\n> **Q2. Lack of computation costs analysis with other methods, including parameters/FLOPs/speed.**\n\nWe compare the energy efficiency of our model with several well-known state-of-the-art works on the REC task from various perspectives, including the number of parameters, computational complexity (FLOPs), inference speed (FPS), and test time (s). The results are presented in Table R1, which can be found in the rebuttal PDF file located at the top of this page.\n\nIn this paper, we highlight two significant advantages of our model architecture over other frameworks: (a) instead of using a Transformer to fuse visual and language features, we only employ a simple lightweight task head; (b) Our one-tower architecture eliminates the need for early interaction techniques in the backbone network, thereby reducing the computational complexity of the model. \n\n As can be seen from the Table R1, due to the simplification of our model's structure, the number of parameters and the calculation complexity are significantly lower than other well-known models. Specifically, our feature fusion and grounding head module only require 1.7M parameters, while other methods use 20M, meaning we only have about 8.5% of their parameter count. Additionally, our computation is only 34.9% of Grounding-DINO and 25.2% of MDETR. Moreover, our inference speed is 10 times faster than Grounding-DINO and TransVG++ (the speed also related to the image size used by the model). Despite these advantages, thanks to the modality-shared feature space, we outperform all these well-known works. We will include this experiment in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pTXXMjmWlo", "reviewer": "Reviewer_SMKQ", "summary": "The authors propose a unified one-tower referring framework that introduces the MRefM paradigm to capture the referential relationships between vision and text. They demonstrate the effectiveness and generality of MRefM across three settings on REC, PG, and RES tasks, consistently achieving good results.", "questions": "See weaknesses.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method is structurally simpler than previous methods and achieves higher grounding and RES performance.\n\nThe ablation experiments are thorough, validating the effectiveness of the two Ref mask modeling strategies.", "weaknesses": "Adapting BEiT, a robust general-purpose multimodal model, to downstream grounding tasks and achieving performance improvement does not seem to be a particularly interesting finding. Additionally, it is puzzling why introducing two types of relation scores in MIM and MLM would enhance the model's referring performance. Especially regarding the four types of masks in the Visual target-relation score, their effective mechanism is not intuitive and lacks explanation. Why did the authors decide to introduce referring information by predicting these four scores? Are there any reference works?\n\nIn Table 3, comparisons with some related works, such as UNINEXT [1] and HIPIE [2], are missing.\n\n[1] Universal instance perception as object discovery and retrieval. CVPR 2023.\n\n[2] Hierarchical Open-vocabulary Universal Image Segmentation. NeurIPS 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful feedback. Please find below our responses to the questions raised in the review. \n\nFirstly, due to the character limit in the reply box, we have included the figure and experimental tables in the PDF file at the top of this page. Please click on the file to view the corresponding figure and tables.\n\n> **Q1. Adapting BEiT, a robust general-purpose multimodal model, to downstream grounding tasks and achieving performance improvement does not seem to be a particularly interesting finding.**\n\nWe would like to highlight the value of our work from two perspectives.\n\n- Firstly, although BEiT-3 is a general-purpose foundation model trained by the Mask Visual Language Modeling (MVLM) paradigm, it does not perform well in grounding and referring tasks. Our work is the pioneer attempt to implement referring tasks based on the BEiT-3 model. We propose the one-tower UniRef framework, which significantly improves fine-grained cross-modal perception compared to the original BEiT-3 model. It offers a new direction for the cross-modal perception of grounding fields\n- Secondly, our aim is not only to transfer the BEiT-3 model to the grounding task, but also to propose a novel paradigm with certain generality. Specifically, while the existing MVLM is a general pre-training paradigm, it can only learn coarse-grained visual and linguistic knowledge. However, fine-grained cross-modal referring relations are common in our lives, and such a paradigm cannot model these referring relations; no previous work has explored them either. Therefore, our aim is to learn these subtle referring relations with the help of the general mask modeling paradigm.  As shown in the experimental results of Table 2 and Table 3, our approach has also been proven effective for VQA and cross-modal retrieval tasks. Our work represents a novel attempt that we believe can provide valuable insights for future research.\n\n> **Q2. Additionally, it is puzzling why introducing two types of relation scores in MIM and MLM would enhance the model's referring performance.** \n\nWe explain the intrinsic mechanism of the two relation scores from the perspective of MIM and MLM, respectively.\n\n-  In the current MIM paradigm, reconstruction is limited to relying solely on the visual features within the image. To enhance content reconstruction by leveraging cross-modal information as much as possible, our Referring MIM approach incorporates visual target-relation scores alongside visual modality content during reconstruction. This modeling approach presents increased difficulty as it necessitates reliance on textual information for reconstructing two more complex visual branch. Consequently, our model achieves a more comprehensive understanding of both visual and textual information. In this way, the model not only can perceive the information of the image modality itself but also have a more accurate understanding of the location and correlation of the key object features in different regions.\n\n-  Similarly, the Referring MLM also aims to enhance the global comprehension and reasoning capabilities of the model for both visual and textual information. Specifically, existing MLM methods solely rely on contextual information within the text to reconstruct masked words. In addition to leveraging image modality information to restore masked words as much as possible, we guide the model in identifying specific words within the grounding text that require special attention during the referring process. The learning process of semantic target-relation score bears resemblance to knowledge distillation. In this way, the model can acquire a more comprehensive understanding of the referred text.\n\n\n> **Q3. Especially regarding the four types of masks in the Visual target-relation score, their effective mechanism is not intuitive and lacks explanation. Why did the authors decide to introduce referring information by predicting these four scores? Are there any reference works?**\n\n- The response to this issue is placed in the global rebuttal field. Please refer to Common Question 1 above on this page.\n\n\n> **Q4. In Table 3, comparisons with some related works, such as UNINEXT [1] and HIPIE [2], are missing.**\n\nAs shown in Table R4,  we compare our RES task result with UNINEXT [1] and HIPIE [2] under oIOU metric, and we will include the results in Tables 2 and Table 3 of the revised paper.  \n\nBesides, both of the two works belongs to the setting of multi-task multiple dataset-mixed training. Comparing with these two works, our base model exceeded UNINEXT by 2.79%(testA), 5.15%(testA), and 2.87%(test) respectively in the REC task for the RefCOCO/+/g three datasets. In the RES task, our base model surpassed UNINEXT by 1.42%(val), 3.85%(val), and 2.51%(val) respectively, and surpassed HIPIE by 1.02%(val), 3.85%(val), and 2.75%(val) in the same three datasets. It should be noted that while our model only utilizes RefC's data for intermediate pre-training, both UNINEXT and HIPIE employ more additional datasets as well as multi-task pre-training. Despite this distinction, our model already demonstrates superior performance compared to theirs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers, area chairs, senior area chairs, and program chairs,

We sincerely thank for the valuable and the thoughtful comments. It is pleasure that this work has been recognized by the three reviewers, including "the MRefM paradigm effectively captures the referential relationship", "a simple yet effective architecture", "the experiments are thorough", etc.  The commonly concerns within the three reviewers lie in (a) The mechanism for the effectiveness of visual target-relation score is not clearly explained.  (b) Lack of computational efficiency comparison experiments. (c) Include several relevant references. To address these concerns, we have carefully provided detailed explanations point-by-point within each reviewer's response box and have also included the relevant experiments, with the results presented in the rebuttal PDF file. We look forward to a better appreciation of this manuscript that incorporates our great efforts. Furthermore, this manuscript has been carefully revised according to the suggestions of the reviewers. 

The followings are our detailed responses. We greatly thank the constructive suggestions that significantly help improve the quality of our paper.

Best regards,

The authors.


------
------

Due to the character limit in the response field, we address the first commonly asked question below.

> **Common Question 1: The mechanism for the effectiveness of visual target-relation score is not clearly explained..**
>
> * **(Reviewer a3fR)** Could you explain the rationale behind designing the system to encompass four masks: x-, y-, w-, and h-masks?   
> * **(Reviewer SMKQ)** Regarding the four types of masks in the visual target-relation score, their effective mechanism is not intuitive and lacks explanation. Why did the authors decide to introduce referring information by predicting these four scores? Are there any reference works?

In order to provide a clearer explanation, we would like to address the rationale behind the visual target-relation score (i.e., the four x-, y-, w-, and h-masks) designed in Sec. 3.2 of the main text from two perspectives. Specifically:

- **Point 1**:  The purpose of designing the Referring MIM algorithm.

  In the existing MIM paradigm, reconstruction is limited to solely relying on the visual features within the image. To enhance content reconstruction by leveraging cross-modal information as much as possible, our Referring MIM approach incorporates visual target-relation scores alongside visual modality content during reconstruction. This modeling approach presents increased difficulty as it necessitates reliance on textual information for reconstructing the two visual branch. Consequently, our model achieves a more comprehensive understanding of both visual and textual information. In this way, the model not only can perceive the information of the image modality itself but also have a more accurate understanding of the location and correlation of the key object features in different regions.

- **Point 2:** How and why the visual target-relation score (i.e., the x-, y-, w-, and h-masks) works？

  To facilitate the explanation, we provide a clearer illustration of the four masks in Fig. R1 within the rebuttal PDF file. As mentioned in Sec. 3.2 of the paper, this score represents the spatial distance between the current patch region and the referred region, it enables implicit deployment of grounding capability within each token of the model. When reconstructing the visual features and target-relation score of each local patch, the model actually needs to have an global and comprehensive understanding of the text modality information and the visual information of the image. On this basis, the model needs to rely on the reconstructed visual features of the local patch to implicitly predict the specific location and size of the referred object, and then accurately predict the visual target-relation score. Finally, Referring MIM can enhance the model's global and multimodal understanding of textual and visual information, and then learn more general visual representations, which can have better generalization ability when deployed to downstream referring tasks.

  The proposed Referring MIM is our own design, which is mainly used to improve the defects existing in MAE/BEiT, and we have not found a similar method in the existing work. However, we can find the rationale of our method in some classic computer vision papers, such as the YOLO series works [1], which predicts the location, size, confidence, and category of the object box corresponding to each grid cell based on the global understanding of the image. The paper [1] also confirmed that the object detection model obtained in this way has stronger generalization ability when transfer to detection tasks that differ greatly from the training data compared with other detectors.

  [1] Redmon, Joseph, et al. "You only look once: Unified, real-time object detection." CVPR. 2016.

Meta-review: The paper initially received borderline accepts from all reviewers with one reviewer adjusting their score favorably after the author rebuttal. For those reviewers who have not responded, the authors' rebuttal appears to effectively address the main concerns. Therefore, based on the recommendations, the AC also recommends the acceptance of the paper.
Decision: Accept (poster)