Title: OneRef:  Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling
Venue: NeurIPS 2024
Accepted: yes
Keywords: Visual Grounding

Referring Expression Comprehension

Referring Image Segmentation

Multimodality

Masked Visual Language Modeling
Abstract: Constrained by the separate encoding of vision and language, existing grounding and referring segmentation works heavily rely on bulky Transformer-based fusion en-/decoders and a variety of early-stage interaction technologies. Simultaneously, the current mask visual language modeling (MVLM) fails to capture the nuanced referential relationship between image-text in referring tasks. In this paper, we propose **OneRef**, a minimalist referring framework built on the modality-shared one-tower transformer that unifies the visual and linguistic feature spaces. To modeling the referential relationship, we introduce a novel MVLM paradigm called Mask Referring Modeling (**MRefM**), which encompasses both referring-aware mask image modeling and referring-aware mask language modeling. Both modules not only reconstruct modality-related content but also cross-modal referring content. Within MRefM, we propose a referring-aware dynamic image masking strategy that is aware of the referred region rather than relying on fixed ratios or generic random masking schemes. By leveraging the unified visual language feature space and incorporating MRefM's ability to model the referential relations, our approach enables direct regression of the referring results without resorting to various complex techniques. Our method consistently surpasses existing approaches and achieves SoTA performance on both grounding and segmentation tasks, providing valuable insights for future research. Our code and models are available at https://github.com/linhuixiao/OneRef.

OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling

Abstract

Constrained by the separate encoding of vision and language, existing grounding and referring segmentation works heavily rely on bulky Transformer-based fusion en-/decoders and a variety of early-stage interaction technologies. Simultaneously, the current mask visual language modeling (MVLM) fails to capture the nuanced referential relationship between image-text in referring tasks. In this paper, we propose OneRef , a minimalist referring framework built on the modality-shared one-tower transformer that unifies the visual and linguistic feature spaces. To modeling the referential relationship, we introduce a novel MVLM paradigm called Mask Referring Modeling ( MRefM ) , which encompasses both referring-aware mask image modeling and referring-aware mask language modeling. Both modules not only reconstruct modality-related content but also cross-modal referring content. Within MRefM, we propose a referring-aware dynamic image masking strategy that is aware of the referred region rather than relying on fixed ratios or generic random masking schemes. By leveraging the unified visual language feature space and incorporating MRefM's ability to model the referential relations, our approach enables direct regression of the referring results without resorting to various complex techniques. Our method consistently surpasses existing approaches and achieves SoTA performance on both grounding and segmentation tasks, providing valuable insights for future research. Our code and models are available at https://github.com/linhuixiao/OneRef .

1 Introduction

Visual Grounding (VG) aims to ground a region referred by a expression query text in a specific image. The generalized VG / referring tasks include Referring Expression Comprehension (REC) [69, 62, 101, 31, 14, 91, 90, 48], Phrase Grounding (PG) [1, 74], and Referring Expression/Image Segmentation (RES/RIS) [69, 94, 89]. In REC/PG, the grounding region is represented by a rectangular boundary box, while in RES/RIS, it is represented by an irregular fine-grained segmented mask of the referred object. Unlike object detection [57, 58] or instance segmentation [26], which usually relies on a close-set of categories to detect or segment multiple regions that satisfy the object label, visual grounding is not limited to fixed categories. It requires understanding the semantics of the query text and then grounding or segmenting specific areas. Therefore, visual grounding is a task that strongly relies on the multimodal interaction and alignment of visual and linguistic features.

Since the introduction of BERT [16] and ViT [19, 7], the state-of-the-art (SoTA) grounding works have widely adopted a pre-training and fine-tuning paradigm. As illustrated in Fig. 1, existing studies

Unified

shared

Encoder

Image

Text

(a) Previous modality-dependent encoding structure

(b) One-tower structure  (ours)

Vision

Encoder

Language

Encoder

Fusion

Transformer

and

Grounding

Head

Interaction

Image

Text

Results

lightweight

Head

Results

(i) One Transformer Encoder

(ii) One Transformer Encoder

+ One Decoder

(iii) Regression with language-

guided vision token

Types:

employing pre-trained models, either utilizing uni-modal pre-trained models to separately transfer visual and language knowledge [14, 15, 98, 33, 55] or utilizing multimodal pre-trained models [91, 77, 89, 35], primarily fall into three typical architectures: (i) two modality encoders combined with a cross-modal fusion encoder, exemplified by TransVG etc . [11, 14, 98, 89, 91, 35, 92]; (ii) additionally incorporating a decoder, exemplified by MDETR etc . [33, 45, 97, 85, 54, 53, 77, 55]; (iii) direct regression based on language-guided visual features, such as LA VT, TransVG++, etc . [94, 15, 79, 99]. However, incorporating modality-dependent encoders in these studies presents a challenge for seamlessly integrating the two modalities into a unified feature space. Consequently, these works not only require an additional cross-modal Transformer-based [82] en-/decoder ( (i) and (ii) ), but also propose a variety of careful-designed interaction structures for modality-dependent encoders to facilitate early-stage fine-grained cross-modal alignment [15, 98, 92, 79, 54, 35, 55, 59], such as adapter [15, 35], cross-modal bridge [92], weight generation [79], image-text cross-attention [55, 54], etc . Therefore, these methods not only entail a large number of parameters but also involve intricate processes. Considering these critical limitations, we aim to explore simpler modality-shared grounding frameworks that can unify vision and language within a unified feature space, thereby obviating the necessity of the elaborate interaction modules, bulky fusion Transformer en-/decoders, as well as the special grounding tokens.

With the advancement of pre-training [70, 66], several studies have been conducted to explore unified modality-shared multimodal frameworks. YORO [29] implemented a shared encoder based on ViLT [37]. However, its modeling approach tends to overshadow the uni-modal knowledge and requires the encoder to incorporate additional query anchors, limiting its applicability for transfer with common pre-trained models. ONE-PEACE [86] has designed seven expert branches based on Mix-of-Expert (MoE) [5, 76, 22] to construct a three-modality foundation model to realize the integration of image, text, and audio modalities. However, their research employed extensive tri-modal data without exploring the potential utilization of MVLM for modeling the referring tasks. BEiT-3 [87] is built upon multi-way Transformer [5, 80], which adopts three MoE heads ( i . e ., vision, language, visionlanguage) and a modality-shared structure that effectively unifies vision and language within a shared feature space. It demonstrates notable advantages across various classification-like cross-modal fields ( e . g ., Retrieval, VQA etc . ). However, no prior research has explored the utilization of BEiT-3 for achieving transfer in referring tasks. Consequently, our objective is to explore more concise and efficient referring grounding and segmentation transfer within a unified feature space on the one-tower model of BEiT-3. However, BEiT-3 model is pre-trained utilizing a generic Mask Vision Language Modeling (MVLM) approach, and this masking paradigm lacks fine-grained cross-modal referring ability and cannot effectively model the intricate referential relationship between images and text. As a result, there exists a significant gap when applying BEiT-3 to the regression-like referring tasks. Therefore, exploring how to incorporate fine-grained cross-modal referring capability into the mask modeling paradigm becomes an important research issue that has not been addressed yet.

In this paper, we propose a novel paradigm called Mask Referring Modeling ( MRefM ) , as well as a unified and extremely concise grounding and referring segmentation framework named OneRef that no longer requires the fusion or interaction Transformer structure and the special grounding tokens.

Firstly , we propose MRefM paradigm to enhance the referring capability of BEiT-3 in a flexible manner. MRefM consists of two components: Referring-aware Mask Image Modeling ( Referring MIM ) and Referring-aware Mask Language Modeling ( Referring MLM ). The conventional MVLM is typically trained alternately or randomly with uni-modal MIM and MLM. In contrast, Referring MIM and Referring MLM are required to reconstruct two distinct types of content: their own modality-related content and cross-modal referring information. Specifically, (i) Referring MIM employs visual tokens after the dot product operation with the aggregated text token for reconstruction purposes. This not only entails reconstructing masked visual features itself but also necessitates

reconstructing the visual target-relation score, which indicates the distance between the current token and the grounding region. The score encompasses four dimensions: horizontal and vertical distance to the grounding center, as well as width and height of the grounding region. In order to enhance the model's understanding capability for referred regions, we propose a referring-aware dynamic image masking strategy that replaces traditional ratio-fixed random masking so that referred regions are reconstructed with a relatively high mask ratio. (ii) Referring MLM employs text tokens after the dot product operation with the aggregated visual token for reconstruction purposes. This not only involves reconstructing masked text itself but also requires reconstructing semantic target-relation scores that represent the correlation degrees between current text tokens and referred image regions.

Secondly , existing grounding and segmentation models commonly employ a [Region] token and multiple query anchors to regress results. However, embedding the region token in backbone will disrupt the pre-trained model [15], and the query anchor also depends on the decoder [33]. With the unified feature space established by modality-shared encoder, we no longer need additional crossmodal en-/decoders to fuse uni-modal features, enabling us to more effectively leverage the knowledge acquired by pre-trained backbone. Benefiting from MRefM paradigm, the visual token inherently contains referring information. Consequently, we can discard special grounding token/anchors and directly construct lightweight and highly concise grounding and segmentation task heads based on the dot product operation within Referring MIM to unify the referring framework.

Contributions: Our contributions are threefold: (i) We pioneer the application of mask modeling to referring tasks by introducing a novel paradigm called mask referring modeling. This paradigm effectively models the referential relation between visual and language. (ii) Diverging from previous works, we propose a remarkably concise one-tower framework for grounding and referring segmentation in a unified modality-shared feature space. Our model eliminates the commonly used modality interaction modules, modality fusion en-/decoders, and special grounding tokens. (iii) We extensively validate the effectiveness of MRefM in three referring tasks on five datasets. Our method consistently surpasses existing approaches and achieves SoTA performance across several settings, providing a valuable new insights for future grounding and referring segmentation research.

2 Related work

2.1 Referring expression comprehension (REC) and segmentation (RES)

(i) REC. The recent supervised REC task, also known as visual grounding in a narrow sense, can be broadly categorized into five main approaches : (1) Fine-tuning with a uni-modal pre-trained language model and a closed-set detector. This setting is exemplified by TransVG [14], which builds upon the two-stage [102, 56, 52, 30] and one-stage [96, 95, 106] methods from the CNN era. It is considered the most conventional and extensively studied approach. (2) Fine-tuning with a pre-trained uni-modal language model and an open-set detection model pre-trained on box-level datasets mixed with multiple data sources. MDETR [33] represents this type of setting, where Fig. 1(a)-(ii) plays a dominant role in its model structure. (3) Fine-tuning with multimodal self-supervised pre-trained models. CLIP-VG [91] serves as an example for this category, introduced primarily through the proposal of CLIP [70]. (4) Multimodal and multi-task mix-supervised pre-trained models. These methods typically combine multiple tasks while mixing datasets from each downstream task, employing mixed pre-training that incorporates both self-supervision and fine-grained supervision. UniTAB [97], OFA[85], etc . , represent such approaches where visual grounding often acts as one of the pre-training tasks. (5) Grounding multimodal large language models (GMLLMs). These methods influenced by works like GPT [6] or LLAMA [81] etc . These models integrate visual backbones into Large Language Models (LLMs) to generate grounding results rather than relying on regression techniques. Our approach mainly falls under type (3). (ii) RES. The development and approach categories of RES [49, 13, 35, 89, 79, 94, 93, 88] are generally similar to those of REC. However, the key distinction lies in the finer granularity of RES's output, which necessitates separate study from REC. In terms of model architecture, RES works predominantly employ two modality-dependent encoders and a decoder to generate the segmentation mask. Our work stands out as the first endeavor to explore RES within a unified multimodal feature space under a one-tower structure.

2.2 Mask vision language modeling

Motivated by the success of MLM [82] in BERT [16], MAE [27] and BEiT [4] have primary shifted their attention to MIM [21, 83, 3]. Subsequently, exemplified by BEiT-3 [87], numerous MVLM

则x-mask 处会无法显示

过渡层，在打印整页ppt处

勾选高质量

Referring-aware

Mask Image Modeling

Referring-aware

Mask Language Modeling

' the giraffe

walking while

another giraffe

follows it. '

Text

Image

' the

𝒆[୑]

walking

𝒆[୑]

another

giraffe

𝒆[୑]

it. '

One-tower

Encoder

[SEP]

[EOS]

[CLS]

Reconstructed

visual content

x-mask

y-mask

w-mask

h-mask

Referring MIM head

Visual target-relation score

Referring MLM head

'the giraffe walking while another

The  giraffe  walking  while  another  giraffe  follows  it

Semantic target-relation score

Reconstructed language content

Referring-aware

dynamic image masking

Referring-aware

text masking

𝑥 ௜ ℳ

௜ୀଵ

ேೡ

𝓮ୗ୉୔

𝑤௜ ℳ

௜ୀଵ

ெ

𝓮େ୐ୗ

---- KL divergence

𝓏௜

௜ୀଵ

ேೡ

𝑠 ௜

௩௧

௜ୀଵ

ேೡ

𝓉௜

௜ୀଵ

ெ

𝑠 ௜

௦௧

௜ୀଵ

ெ

⨀

⨀

MLP

Norm

MLP

MLP

MLP

Softmax

Shared Multi-Head Self-Attention

Vision-Language Expert

× N

works [61, 47, 36, 104, 2] have emerged, with most of these works implementing randomly alternating uni-modal MIM and MLM. Most relevant to our work are mask region modeling (known as MRM) [64, 83], which can be either unimodal MIM ( e . g ., R-MAE [64]) or employ more fine-grained regional data and contrastive learning to reconstruct the alignment between regions and object labels ( e . g ., ConLIP [61], VLT [17, 18] etc . ). However, our work focuses on modeling the fine-grained referential relationship within image and text, so as to enhance the cross-modal referring capability, which is significantly different from these works.

3 Methodology

In this section, we propose our multimodal Mask Referring Modeling ( MRefM ) paradigm, which includes Referring MIM and Referring MLM, as well as a feature space unified grounding and segmentation framework OneRef . We will introduce these methods in the following sections.

Following BEiT-3 [87], we employ a multimodal modality-shared Transformer [5] as the underlying backbone network. Initially, we perform mask-then-predict MRefM pre-training, and followed by transfer fine-tuning on the referring tasks. As shown in Fig. 2, the MRefM pre-training stage consists of two components: Referring-aware Mask Image Modeling ( Referring MIM ) and Referring-aware Mask Language Modeling ( Referring MLM ). Both modules aim to reconstruct two types of content: modality-related content within each modality and cross-modal fine-grained referring content.

3.1 Preliminaries

BEiT-3 [87] utilizes MIM, MLM, and MVLM for processing image, text, and image-text pairs respectively to facilitate the acquisition of general representations through MoE heads and shared multi-head self-attention. Notably, MVLM involves alternate training of MIM and MLM. Specifically:

(i) Vanilla mask image modeling. We denote x ∈ R H × W × 3 as the input image, and it is tokenized by a convolution projection to N v = HW/P 2 patches { x p i } N v i =1 , where x p ∈ R N v × D , H,W are the image size, and P is the patch size, D is the hidden dimension of the unified feature space. Then, we leverage a specific masking strategy to mask a specific number of image patches. The masked position is termed as M v . Thus, a shared learnable embedding e [ M ] is used to replace the masked image patch embeddings x p i if i ∈ M v . Subsequently, we prepend a learnable [CLS] token to the input, i . e ., [ e CLS , { x p i } N v i =1 ] , and feed them to the one-tower Transformer. Next, we utilize a MIM head which consists of a linear projection and a softmax classifier to predict the visual tokens of the masked positions based on the corrupted image x M . The visual tokens are obtained by the image tokenizer VQ-KDCLIP proposed in BEiT v2 [67], which provides supervisions for the MIM self-supervised learning procedure. The visual tokens of the original image are denote as { z i } N v i =1 , and I denotes the pre-training images. Then, the training loss of MIM is defined as:

(ii) Vanilla mask language modeling. The input text is tokenized and projected to the word embeddings { w i } M i =1 by a SentencePiece tokenizer [41] with vocabulary size of 64010, where w ∈ R M × D , M is the length of tokenized text sequence. Then, following BEiT-3 [87], we randomly mask the text tokens with a fixed masking ratio δ . The masked position is termed as M w . Thus, a shared learnable embedding w [ M ] is used to replace the masked word tokens w i if i ∈ M w . We prepend a learnable special tokens [SEP] and an end-of-sequence token [EOS] to the sequence, i . e ., [ e SEP , { w i } i M =1 , e EOS ] , and feed them to the one-tower Transformer. Similarly, we utilize a MLM head which consists of a linear projection to predict the text tokens of masked positions based on the corrupted text data w M . The original textual tokens are denoted as { t i } i M =1 , and T denotes the pre-training text sequences. Then, the training loss of MLM is defined as:

3.2 Referring-aware mask image modeling

After concatenating the visual and text tokens and feeding them into the modality-shared encoder, the vanilla MVLM is commonly implemented through the alternating use of MIM and MLM [87]. Despite the multimodal features are interact within the modality-shared encoder, it fundamentally remains a unimodal information reconstruction. Additionally, MVLM acquires general knowledge by randomly masking images and texts, it fails to effectively model the referential relationship. Hence, we propose Referring MIM and Referring MLM methods. Specifically, as shown in Fig. 2, our proposed Referring MIM incorporates two additional components: the reconstruction of visual target-relation score and a referring-aware dynamic masking strategy.

In Referring MIM (Fig. 2), instead of using uni-modal visual tokens [87, 47, 2], we propose to employ visual tokens that dot product with the aggregated text token e SEP ∈ R 1 × D for the reconstruction purpose. The reconstruction of Referring MIM involves not only the modality-related content { z i } i ∈M v but also the visual target-relation scores { s vt i } N v i =1 ∈ R N v × 4 . We utilize a visual target-relation head which consists of a three-layer perceptron (MLP) to predict the scores. The scores represent the distance between each patch token { x M i } N v i =1 and the referred region B = ( x c , y c , w r , h r ) , where ( x c , y c , w r , h r ) denote the center coordinate and the width and height of the referred region. It encompasses four masks, i . e ., x-, y-, w-, h- masks , which represent the normalized horizontal and vertical distances from the referred center, i . e ., (( x -x c ) /W, ( y -y c ) /H ) , and the proportion of width and height on the the referred region, i . e ., ( P/w r , P/h r ) , respectively, where ( x, y ) denote the center coordinate of each patch. We denote ⊙ as dot product operation. Finally, the training loss of Referring MIM is defined as:

Referring-aware dynamic image masking strategy. As shown in Fig. 4, among the existing masking strategies, MAE [27] adopts a high-ratio random masking while BEiT3 [87] uses a low-ratio block-wise random masking, neither of which effectively directs attention to the referred region. SemMAE [43] proposes a semantic-guided masking that requires additional bulky semantic models and limits its generality. To enhance the model's understanding of the referred region through surrounding visual context and text semantics, we propose a referring-aware dynamic masking strategy as shown in Algo. 1.

The strategy avoids the drawbacks of the aforementioned methods and directs the

Algorithm 1 Referring-aware Dynamic Masking

Input: N v image patches , N r ( h rp × w rp ) referred patches . Output: Dynamic masked positions M .

model's attention to the referred region. Specifically, we denote the shape after patch reshaping of the image as ( h, w ) , where h = H/P , w = W/P , and N v = h × w . To maximize the masking of the

Unified

shared

Encoder

Image

∑

[SEP]

{

𝑥𝑖

}

𝑖

=1

𝑁

𝒆

ESP

ℳ𝑠𝑖𝑚

∈ ℝ ℎ

×

𝑤

Referring-based

REC head

ℬ

Constraint

Unified

shared

Encoder

Bilinear

[SEP]

{

𝑥𝑖

}

𝑖

=1

𝑁

𝒆

ESP

Deconvolution

Deconvolution

Deconvolution

∈ ℝ 𝐻

/2×

𝐻

/2

ℳ𝑠𝑒𝑔

Result:

' a white cake

with blue, green,

and yellow polka-

dots on it. '

Image

' a white cake

with blue, green,

and yellow polka-

dots on it. '

Result:

Box mask

Referring-based

RES head

Expression text

Expression text

MLP

ℳ𝑏𝑜𝑥

(a) Referring Expression Comprehension task

(b) Referring Expression Segmentation task

referred region ( x s , y s , w r , h r ) , where x s , y s represent the starting coordinates of the referred region, we introduce a margin m to its surroundings and denote its patch coordinates as ( x sp , y sp , w rp , h rp ) , i . e ., x sp = ⌊ x s /P ⌋ -m , w rp = ⌊ w r /P ⌋ + m , y sp and h rp are similar to that of x sp and w rp , where ⌊·⌋ indicates rounding down to an integer. Thus, the number of referred patches is denote as N r = h rp × w rp . Then, as shown in Algo. 1, to ensure that the model allocates appropriate attention to the in-contextual information around the referred region, we utilize a random masking with a relatively low ratio β for its surroundings. Simultaneously, we employ a block-wise masking approach with a high ratio γ in the extended area of the region. Since referred regions vary across different image-text pairs, each sample's entire masking ratio α is dynamically determined:

3.3 Referring-aware mask language modeling

Similarly, in Referring MLM, instead of using uni-modal linguistic tokens [87, 47, 2], we propose to employ linguistic tokens that dot product with the aggregated visual token e CLS ∈ R 1 × D for the reconstruction purpose. The reconstruction of Referring MLM involves not only the modality-related content { t i } i ∈M w but also the semantic target-relation scores { s st i } i M =1 . The score represents the correlation between the referred target and the language token, which is obtained by a teacher model ( i . e ., a BEiT-3 model with performed image-text contrastive intermediate tuning) with calculating the weighted sum of the normalized similarity between the language token { w i } i M =1 and the aggregated visual token e reg CLS of referred region, as well as the aggregated visual token e img CLS of entire image:

where < · , · > denotes cosine similarity operation, σ denotes the softmax normalization.As shown in Fig. 2, we utilize a semantic target-relation head which consists of a three-layer MLPs and a softmax normalization to predict the scores. Finally, the training loss of Referring MLM is defined as:

where p kl represents a probabilistic prediction with Kullback-Leibler divergence [28].

3.4 Referring-based grounding and segmentation transfer

The modeling of visual and language in a unified feature space eliminates the need for the commonlyused Transformer-based fusion en-/decoder [14, 54, 55] and various early-stage interaction techniques [15, 79, 92] to further uniform the visual and language features. Additionally, since the referential relationship is modeled by MRefM during pre-training, we can accurately regress the results of grounding and referring segmentation using the output tokens, without relying on the widely-used special grounding tokens ( e . g ., [Region] token [14, 15, 98, 91, 92], query anchors [55, 54]).

Referring expression comprehension. As illustrated in Fig. 3-(a), based on Referring MIM, we initially perform a similarity operation between visual tokens { x i } N v i =1 ∈ R N v × D and aggregated language token e SEP ∈ R 1 × D to obtain a softmax-normalized similarity mask M sim ∈ R h × w . This mask is then replicated and multiplied back to each hidden dimension of the visual tokens. Subsequently, the visual tokens are summed to yield reduced tokens, which are finally subjected to regress the prediction box ˆ B = (ˆ x c , ˆ y c , ˆ w r , ˆ h r ) using a 3-layer MLPs:

To enhance the accuracy of cross-modal similarity, we propose treating the similarity as a coarsegrained downsampling bounding box mask M box ∈ R h × w and imposing segmentation loss ( i . e .,

Focal loss [51] and Dice/F-1 loss [63]) on the sigmoid activated similarity mask M sim ∈ R h × w with coefficient λ f _ box and λ d _ box as the box mask constraints:

Consequently, the loss function for the REC task can be reformulated as the weighted sum of vanilla grounding loss ( i . e ., smooth L1 loss [25] and Giou loss [73]) and box mask constraints:

Referring expression segmentation. As illustrated in Fig. 3-(b), the implementation of referring segmentation can be regarded as a simplified version of grounding. Initially, we employ a 3-layer deconvolution to up-sample the visual token to x up ∈ R H/ 2 × W/ 2 . Subsequently, cosine similarity operations are performed on the up-sampled visual tokens and the aggregated language token. The resulting similarity mask is then utilized as the final predicted mask ˆ M seg ∈ R H × W after applying 1-layer bilinear interpolation. We denote the ground truth segmentation mask as M seg ∈ R H × W , then the loss function for RES is defined as follows:

4 Experiments

4.1 Experimental setups

Datasets and evaluation metrics. Our method is validated in the REC, RES, and PG tasks with five widely used datasets, namely three REC/RES datasets (RefCOCO/+/g [101, 62]), as well as two PG datasets (ReferItGame [34] and Flickr30k Entities [68]). In PG, the query pertains to a specific phrase,

while in REC and RES, the query refers to a reference expression. The text of RefCOCO+/g exhibits greater length and complexity in comparison to that of RefCOCO. In REC/PG, we follow previous works [14, 96] that employs Intersection-over-Union (IoU) as the evaluation metric, i . e ., a prediction is deemed accurate only when its IoU exceeds or equals 0.5. We compute the prediction accuracy for each dataset as a performance indicator. While in RES, we follow previous works [79, 35] that employs mean IoU (mIoU) and overall IoU (oIoU) for each dataset as the indicators. The detailed statistics information regarding these five datasets are provided in the Appendix B.

Experimental details. Since MRefM is proposed on the basis of the traditional MVLM, considering the pre-training cost of MVLM, we adopt BEiT-3 [87] base and large model as our initial weights and then perform intermediate MRefM pre-training on the task-relevant dataset. Such intermediate pre-training is common in existing grounding works [55, 54, 33]. As described in Sec. 2.1, to verify the effectiveness of our MRefM approaches, we conduct extensive experiments on three settings : (1) The basic single-dataset fine-tuning setting. This setting does not require additional training data and aligns with existing supervised and self-supervised transfer approaches [91, 92, 89, 35]. In this setting, we perform supervised single-dataset intermediate MRefM pre-training before finetuning. (2) The setting of fine-tuning with supervised dataset-mixed intermediate pre-training. This setting aligns with existing grounded pre-trained approaches, such as Grounding-DINO [55], DQ-DETR [54] etc . we perform an MRefM intermediate pre-training before fine-tuning. (3) To verify the generality of MRefM, we perform the setting of fine-tuning with unsupervised intermediate MRefM pre-training . There are several ways to obtain the regions in the regional masking modeling works [64, 24, 78], such as Felzenswalb-Huttenlocher (FH) algorithm [23], SAM [38] etc . Thus, we adopt the unsupervised, fast, image-computable FH algorithm [23] to generate regions following R-MAE [64]. We then select the referred one using a BEiT-3 model with performed image-text contrastive intermediate tuning. More details about the selection of the unsupervised regions, network architecture, training and inference, model hyperparameters etc . are provided in the Appendix C.

4.2 Comparison with state-of-the-art methods

Referring expression comprehension. As shown in Tab. 1 and Tab. 2, we conducted experiments for the REC and PG tasks across three settings . (1) In the single-dataset fine-tuning setting, our base model surpasses the current SoTA method HiVG [92] by 2.07 % (testB), 6.15 % (testB), 4.73 % (test),

Query:

' the giraffe walking while another giraffe follows it. '

Query:

"a zebra eating grass in front of several other eating animals"

Original image

(a) Random

,

𝛼

= 75%

(b) Blockwise

,

𝛼

= 40%

(c) Referring-aware dynamic masking

𝛽

= 20%,

𝛾

= 55%

𝛽

= 35%,

𝛾

= 75%

𝛽

= 50%,

𝛾

= 90%

𝛽

= 20%,

𝛾

= 55%

𝛽

= 35%,

𝛾

= 75%

𝛽

= 50%,

𝛾

= 90%

1.95 % (test), and 1.50 % (test) on the five datasets respectively, while also significantly outperforming the traditional uni-modal detector-based approach TransVG++ [15] by 4.37 % (testB), 7.98 % (testB), 7.22 % (test), 2.47 % (test), and 2.12 % (test), respectively. (2) In dataset-mixed pre-training setting, our base model outperforms HiVG [92] by 1.35 % , 2.79 % , and 2.63 % on RefCOCO/+/g testB/testB/test splits, outperforms Grounding-DINO [55] by 2.59 % , 4.76 % , and 2.38 % , exceeds OFA by 5.28 % , 5.18 % ,and 5.01 % , and even surpasses LION [9] - a GMLLM model that is 20-60 times larger than ours - by 3.76 % , 2.13 % ,and 1.69 % . Note that among these works, UniTAB [97], OFA[85], LION [9] also utilize the MVLM on the pre-training stage. (3) Furthermore, we achieve competitive performance in the unsupervised setting, which shows the generality of MRefM paradigm. Additionally, our large-size model exhibits remarkable scalability with further substantial improvements in performance. More detailed results are provided in the Appendix E.

Referring expression segmentation. As presented in Tab. 3 (mIoU metric), we conducted experiments for RES task under three settings . (1) In the single-dataset fine-tuning setting, our base model surpasses the SoTA self-supervised method RISCLIP [35] by 2.65 % , 4.77 % , and 1.73 % on RefCOCO/+/g testB/testB/test splits, respectively, while also significantly outperforming the traditional uni-modal detector-based approach VG-LAW [79] by 3.42 % , 7.31 % , and 4.57 % , respectively. (2) In the dataset-mixed pre-training setting, our base model achieves superior performance compared to the SoTA method RISCLIP [35] with improvements of 4.53 % , 8.21 % , and 5.39 % . (3) In the unsupervised pre-training setting, we also achieve competitive performance. Additionally, our large-size model also exhibits remarkable scalability and demonstrates a substantial improvement in performance. For oIoU metric, the results are presented in Appendix E.2 (Tab. 13).

4.3 Ablation study

The Mask Referring Modeling. In Tab. 4, we conducted ablation studies on MRefM, which included Referring MIM ('Ref MIM'), Referring MLM ('Ref MLM'), and referring-aware dynamic image masking ('referring-aware'). The 'vanilla' denotes the vanilla MVLM described in Sec. 3.1. As shown in Tab. 4, referring MIM, referring MLM, and dynamic masking strategy resulted in improvements of 3.70 % , 2.16 % , and 1.05 % on the RefCOCOg-test dataset, and with an overall improvement of 6.21 % , demonstrates the effectiveness of our methods. More results are provided in the Appendix E.4.

The referring-aware dynamic masking strategy. Fig. 4 presents a schematic of the three masking strategies. In our experiments, as illustrate in Fig. 4-(c), β and γ demonstrate optimal performance at values of 0.35 and 0.75, respectively. More detailed results are provided in the Appendix E.5.

The referring-based task heads. We conducted ablation studies on the design of two referring-based task heads. Tab. 5 reveals that our modeling method effectively captures referring information at the backbone stage, benefiting from the one-tower structure. This approach is significantly more efficient

than the traditional fusion encoder and special token-based method. Additionally, our proposed box mask loss also contributes to a performance gain of 1.50 % (test).

4.4 Generality study

The generality of MRefM. Firstly, we perform an unsupervised MRefM pre-training in Tab. 2 and Tab. 3, both of which achieve competitive performance. Secondly, we replace the backbone and apply MRefM on DETR and CLIP by using TransVG [14] and CLIP-VG [91] under the two settings. Since the two frameworks do not interact at backbone stage, we build MRefM on the fusion encoder. In Tab. 6, MRefM can effectively learn referring representation, resulting in an overall performance gain of about 2.0 % . All these findings demonstrate the validity and generality of the MRefM paradigm.

The generality of referring-based task heads. Since both TransVG++ [15] and LAVT [94] have modality interactions at backbone stage, we attempted to apply our task heads to both frameworks. TransVG++ is reproduced by us since its code is not available. Tab. 7 shows that our proposed task heads achieve a 1.5+ % improvement in both REC and RES, offering a new avenue for future research.

5 Conclusion

In this paper, we propose a novel, highly concise, and feature space unified one-tower referring framework. Additionally, we pioneer the exploration of mask modeling in referring tasks by introducing MRefM paradigm to capture the referential relationships between vision and text. We demonstrate the effectiveness and generality of MRefM across three settings on REC, PG, and RES tasks, consistently achieving groundbreaking results. Furthermore, leveraging unsupervised methods enables potential large-scale pre-training of MRefM in the future, presenting a new direction for referring tasks.

References

Hassan Akbari, Svebor Karaman, Surabhi Bhargava, Brian Chen, Carl Vondrick, and Shih-Fu Chang. Multi-level multimodal common semantic space for image-phrase grounding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12476-12486, 2019.

Tarik Arici, Mehmet Saygin Seyfioglu, Tal Neiman, Yi Xu, Son Train, Trishul Chilimbi, Belinda Zeng, and Ismail Tutar. Mlim: Vision-and-language model pre-training with masked language and image modeling. arXiv preprint arXiv:2109.12178 , 2021.

Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. Multimae: Multi-modal multi-task masked autoencoders. In European Conference on Computer Vision , pages 348-367. Springer, 2022.

Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations , 2021.

Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified vision-language pre-training with mixture-of-modalityexperts. Advances in Neural Information Processing Systems , 35:32897-32912, 2022.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV , pages 213-229, 2020.

Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3558-3568, 2021.

Gongwei Chen, Leyang Shen, Rui Shao, Xiang Deng, and Liqiang Nie. Lion: Empowering multimodal large language model with dual-level visual knowledge. arXiv preprint arXiv:2311.11860 , 2024.

Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm's referential dialogue magic. arXiv preprint arXiv:2306.15195 , 2023.

Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision , pages 104-120. Springer, 2020.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) , 2(3):6, 2023.

Yong Xien Chng, Henry Zheng, Yizeng Han, Xuchong Qiu, and Gao Huang. Mask grounding for referring image segmentation. arXiv preprint arXiv:2312.12198 , 2023.

Jiajun Deng, Zhengyuan Yang, Tianlang Chen, Wengang Zhou, and Houqiang Li. Transvg: End-to-end visual grounding with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1769-1779, 2021.

Jiajun Deng, Zhengyuan Yang, Daqing Liu, Tianlang Chen, Wengang Zhou, Yanyong Zhang, Houqiang Li, and Wanli Ouyang. Transvg++: End-to-end visual grounding with language conditioned vision transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vision-language transformer and query generation for referring segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 16321-16330, 2021.

Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vlt: Vision-language transformer and query generation for referring segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(6):7900-7916, 2022.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Hugo Jair Escalante, Carlos A Hernández, Jesus A Gonzalez, Aurelio López-López, Manuel Montes, Eduardo F Morales, L Enrique Sucar, Luis Villaseñor, and Michael Grubinger. The segmented and annotated iapr tc-12 benchmark. Computer Vision and Image Understanding (CVIU) , 114:419-428, 2010.

Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1935819369, 2023.

William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research , 23(120):1-39, 2022.

Pedro F Felzenszwalb and Daniel P Huttenlocher. Efficient graph-based image segmentation. International journal of computer vision , 59:167-181, 2004.

Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurmans, Sergey Levine, and Pieter Abbeel. Multimodal masked autoencoders learn transferable representations. In First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022 , 2022.

Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision , pages 1440-1448, 2015.

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV , pages 2961-2969, 2017.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000-16009, 2022.

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. stat , 1050: 9, 2015.

Chih-Hui Ho, Srikar Appalaraju, Bhavan Jasani, R Manmatha, and Nuno Vasconcelos. Yoro-lightweight end to end visual grounding. In Computer Vision-ECCV 2022 Workshops: Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VIII , pages 3-23. Springer, 2023.

Richang Hong, Daqing Liu, Xiaoyu Mo, Xiangnan He, and Hanwang Zhang. Learning to compose and reason with language tree structures for visual grounding. IEEE TPAMI , 2019.

Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. Natural language object retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2016.

Ziling Huang and Shin'ichi Satoh. Referring image segmentation via joint mask contextual embedding learning and progressive alignment network. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 7753-7762, 2023.

Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1780-1790, 2021.

Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , pages 787-798, 2014.

Seoyeon Kim, Minguk Kang, Dongwon Kim, Jaesik Park, and Suha Kwak. Extending clip's image-text alignment to referring image segmentation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages 4611-4628, 2024.

Sungwoong Kim, Daejin Jo, Donghoon Lee, and Jongmin Kim. Magvlt: Masked generative vision-andlanguage transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23338-23348, 2023.

Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. In International Conference on Machine Learning , pages 5583-5594. PMLR, 2021.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4015-4026, 2023.

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision , 123:32-73, 2017.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS , 2012.

Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. EMNLP 2018 , page 66, 2018.

Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International journal of computer vision , 128(7):1956-1981, 2020.

Gang Li, Heliang Zheng, Daqing Liu, Chaoyue Wang, Bing Su, and Changwen Zheng. Semmae: Semantic-guided masking for learning masked autoencoders. Advances in Neural Information Processing Systems , 35:14290-14302, 2022.

Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems , 34:9694-9705, 2021.

Muchen Li and Leonid Sigal. Referring transformer: A one-step approach to multi-task visual grounding. Advances in Neural Information Processing Systems , 34:19652-19664, 2021.

Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In European Conference on Computer Vision , pages 280-296. Springer, 2022.

Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling languageimage pre-training via masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23390-23400, 2023.

Zhitian Li, Wuhao Yang, Linhui Xiao, Xingyin Xiong, Zheng Wang, and Xudong Zou. Integrated wearable indoor positioning system based on visible light positioning and inertial navigation using unscented kalman filter. In 2019 11th International Conference on Wireless Communications and Signal Processing (WCSP) , pages 1-6. IEEE, 2019.

Zizhang Li, Mengmeng Wang, Jianbiao Mei, and Yong Liu. Mail: A unified mask-image-language trimodal network for referring image segmentation. arXiv preprint arXiv:2111.10747 , 2021.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision , pages 740-755. Springer, 2014.

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision , pages 2980-2988, 2017.

Daqing Liu, Hanwang Zhang, Feng Wu, and Zheng-Jun Zha. Learning to assemble neural module tree networks for visual grounding. In ICCV , pages 4673-4682, 2019.

Jiang Liu, Hui Ding, Zhaowei Cai, Yuting Zhang, Ravi Kumar Satzoda, Vijay Mahadevan, and R Manmatha. Polyformer: Referring image segmentation as sequential polygon generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18653-18663, 2023.

Shilong Liu, Shijia Huang, Feng Li, Hao Zhang, Yaoyuan Liang, Hang Su, Jun Zhu, and Lei Zhang. Dq-detr: Dual query detection transformer for phrase extraction and grounding. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 1728-1736, 2023.

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 , 2023.

Xihui Liu, Zihao Wang, Jing Shao, Xiaogang Wang, and Hongsheng Li. Improving referring expression grounding with cross-modal attention-guided erasing. In CVPR , pages 1950-1959, 2019.

Yabo Liu, Jinghua Wang, Chao Huang, Yaowei Wang, and Yong Xu. Cigar: Cross-modality graph reasoning for domain adaptive object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23776-23786, 2023.

Yabo Liu, Jinghua Wang, Linhui Xiao, Chengliang Liu, Zhihao Wu, and Yong Xu. Foregroundness-aware task disentanglement and self-paced curriculum learning for domain adaptive object detection. IEEE Transactions on Neural Networks and Learning Systems , 2023.

Yunfei Liu, Zhitian Li, Linhui Xiao, Shuaikang Zheng, Pengcheng Cai, Haifeng Zhang, Pengcheng Zheng, and Xudong Zou. Fdo-calibr: visual-aided imu calibration based on frequency-domain optimization. Measurement Science and Technology , 34(4):045108, 2023.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Ziyang Luo, Yadong Xi, Rongsheng Zhang, GongZheng Li, Zeng Zhao, and Jing Ma. Conditioned masked language and image modeling for image-text dense retrieval. In Findings of the Association for Computational Linguistics: EMNLP 2022 , pages 130-140, 2022.

Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2016.

Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV) , pages 565-571. Ieee, 2016.

Duy Kien Nguyen, Yanghao Li, Vaibhav Aggarwal, Martin R Oswald, Alexander Kirillov, Cees GM Snoek, and Xinlei Chen. R-mae: Regions meet masked autoencoders. In The Twelfth International Conference on Learning Representations , 2023.

Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned photographs. Advances in neural information processing systems , 24, 2011.

Fang Peng, Xiaoshan Yang, Linhui Xiao, Yaowei Wang, and Changsheng Xu. Sgva-clip: Semanticguided visual adapting of vision-language models for few-shot image classification. IEEE Transactions on Multimedia , 2023.

Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. ArXiv , abs/2208.06366, 2022. URL https://api. semanticscholar.org/CorpusID:251554649 .

Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2641-2649, 2015.

Yanyuan Qiao, Chaorui Deng, and Qi Wu. Referring expression comprehension: A survey of methods and datasets. IEEE Transactions on Multimedia , 23:4426-4440, 2020.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

J Redmon. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition , 2016.

Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767 , 2018.

Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 658-666, 2019.

Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, and Bernt Schiele. Grounding of textual phrases in images by reconstruction. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14 , pages 817-834. Springer, 2016.

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 , 2021.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations , 2016.

Fengyuan Shi, Ruopeng Gao, Weilin Huang, and Limin Wang. Dynamic mdetr: A dynamic multimodal transformer decoder for visual grounding. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15638-15650, 2022.

Wei Su, Peihan Miao, Huanzhang Dou, Gaoang Wang, Liang Qiao, Zheyang Li, and Xi Li. Language adaptive weight generation for multi-task visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10857-10866, 2023.

Jiajia Tang, Kang Li, Ming Hou, Xuanyu Jin, Wanzeng Kong, Yu Ding, and Qibin Zhao. Mmt: Multi-way multi-modal transformer for multimodal learning. In IJCAI , pages 3458-3465, 2022.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS , 30, 2017.

Haoqing Wang, Yehui Tang, Yunhe Wang, Jianyuan Guo, Zhi-Hong Deng, and Kai Han. Masked image modeling with local multi-scale reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2122-2131, 2023.

Ning Wang, Jiajun Deng, and Mingbo Jia. Cycle-consistency learning for captioning and grounding. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 5535-5543, 2024.

Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequenceto-sequence learning framework. In International Conference on Machine Learning , pages 23318-23340. PMLR, 2022.

Peng Wang, Shijie Wang, Junyang Lin, Shuai Bai, Xiaohuan Zhou, Jingren Zhou, Xinggang Wang, and Chang Zhou. One-peace: Exploring one general representation model toward unlimited modalities. arXiv preprint arXiv:2305.11172 , 2023.

Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for vision and vision-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19175-19186, 2023.

Xudong Wang, Shufan Li, Konstantinos Kallidromitis, Yusuke Kato, Kazuki Kozuka, and Trevor Darrell. Hierarchical open-vocabulary universal image segmentation. Advances in Neural Information Processing Systems , 36, 2024.

Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11686-11695, 2022.

Linhui Xiao, Jinge Wang, Xiaosong Qiu, Zheng Rong, and Xudong Zou. Dynamic-slam: Semantic monocular visual localization and mapping based on deep learning in dynamic environment. Robotics and Autonomous Systems , 117:1-16, 2019.

Linhui Xiao, Xiaoshan Yang, Fang Peng, Ming Yan, Yaowei Wang, and Changsheng Xu. Clip-vg: Self-paced curriculum adapting of clip for visual grounding. IEEE Transactions on Multimedia , 2023.

Linhui Xiao, Xiaoshan Yang, Fang Peng, Yaowei Wang, and Changsheng Xu. HiVG: Hierarchical multimodal fine-grained modulation for visual grounding. In Proceedings of the 32nd ACM International Conference on Multimedia (MM '24) , 2024. URL https://openreview.net/forum?id=NMMyGy1kKZ .

Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Universal instance perception as object discovery and retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15325-15336, 2023.

Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. Lavt: Languageaware vision transformer for referring image segmentation. In CVPR , pages 18155-18165, 2022.

Zhengyuan Yang, Boqing Gong, Liwei Wang, Wenbing Huang, Dong Yu, and Jiebo Luo. A fast and accurate one-stage approach to visual grounding. In ICCV , pages 4683-4693, 2019.

Zhengyuan Yang, Tianlang Chen, Liwei Wang, and Jiebo Luo. Improving one-stage visual grounding by recursive sub-query construction. In ECCV , pages 387-404, 2020.

Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. Unitab: Unifying text and box outputs for grounded vision-language modeling. In European Conference on Computer Vision , pages 521-539. Springer, 2022.

Jiabo Ye, Junfeng Tian, Ming Yan, Xiaoshan Yang, Xuwu Wang, Ji Zhang, Liang He, and Xin Lin. Shifting more attention to visual backbone: Query-modulated refinement networks for end-to-end visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15502-15512, 2022.

Jiabo Ye, Junfeng Tian, Ming Yan, Haiyang Xu, Qinghao Ye, Yaya Shi, Xiaoshan Yang, Xuwu Wang, Ji Zhang, Liang He, et al. Uniqrnet: Unifying referring expression grounding and segmentation with qrnet. ACM Transactions on Multimedia Computing, Communications and Applications , 2024.

Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. In The Twelfth International Conference on Learning Representations , 2024.

Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14 , pages 69-85. Springer, 2016.

Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1307-1315, 2018.

Heng Zhao, Joey Tianyi Zhou, and Yew-Soon Ong. Word2pix: Word to pixel cross-attention transformer in visual grounding. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Zijia Zhao, Longteng Guo, Xingjian He, Shuai Shao, Zehuan Yuan, and Jing Liu. Mamo: masked multimodal modeling for fine-grained vision-language representation learning. arXiv preprint arXiv:2210.04183 , 2022.

Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. arXiv preprint arXiv:1904.07850 , 2019.

Yiyi Zhou, Rongrong Ji, Gen Luo, Xiaoshuai Sun, Jinsong Su, Xinghao Ding, Chia-Wen Lin, and Qi Tian. A real-time global inference network for one-stage referring expression comprehension. 2021.

Chaoyang Zhu, Yiyi Zhou, Yunhang Shen, Gen Luo, Xingjia Pan, Mingbao Lin, Chao Chen, Liujuan Cao, Xiaoshuai Sun, and Rongrong Ji. Seqtr: A simple yet universal network for visual grounding. In European Conference on Computer Vision , pages 598-615. Springer, 2022.

Hong Zhu, Qingyang Lu, Lei Xue, Mogen Xue, Guanglin Yuan, and Bineng Zhong. Visual grounding with joint multi-modal representation and interaction. IEEE Transactions on Instrumentation and Measurement , 2023.

Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision , pages 19-27, 2015.

Appendix

We provide an overview of the Appendix below:

Appendix A: Explanation of the task definition

Appendix B: Introduction of the datasets

-Appendix B.1 The five referring datasets.

-Appendix B.2 Explanation of the dataset abbreviations.

-Appendix B.3 Comparison of datasets used in the pre-trained models.

Appendix C: Implementation details

Appendix D: Technical remarks

-Appendix D.1 Further explanation for the effectiveness mechanism of the visual targetrelation score.

-Appendix D.2 The selection of the unsupervised regions.

-Appendix D.3 Referring-aware text masking.

-Appendix D.4 The difference of the task heads between ours with other frameworks.

Appendix E: Extra experimental results

-Appendix E.1 The results on phrase grounding task under mixup pre-training setting.

-Appendix E.3 Computational costs analysis compared with SoTA methods.

-Appendix E.4 Complete ablation study of MRefM.

-Appendix E.5 Ablation study of the mask ratio in referring-aware dynamic masking.

Appendix F: Visualization of the results

Appendix G: Further discussions

-Appendix G.1 Limitations.

-Appendix G.2 Broader impacts.

A Explanation of the task definition

As explained in Sec. 1 of the main text, Visual Grounding (VG) aims to grounding a region referred by a query text in a specific image. The generalized visual grounding includes Referring Expression Comprehension (REC), Phrase Grounding (PG), and Referring Expression Segmentation (RES) tasks. However, in recent years, REC and RES have often been studied separately. Therefore, in numerous works [14, 15, 79, 92, 91, 98], visual grounding specifically refers to REC and PG tasks, which involve grounding a rectangular region. In this paper, we follow the mainstream and have not clearly separated the 'grounding' from 'generalized visual grounding' and 'REC and PG tasks'. When expressing the experimental task, 'grounding' usually refers to REC or PG tasks, so as to discuss it parallelly with the RES task.

B Introduction of the datasets

B.1 The five referring datasets

We present the detailed descriptions of the five referring datasets used in our experimental study on the REC, PG, and RES tasks. Tab. 8 presents the detailed statistics.

RefCOCO/RefCOCO+/RefCOCOg. These three datasets belong to the Referring Expression Comprehension (REC) and Referring Expression Segmentation (RES) tasks, and the images of these three datasets derived from MSCOCO [50]. Expressions in RefCOCO [101] and RefCOCO+ [101] are collected by the two-player game proposed in ReferitGame [34]. There are two test splits called 'testA' and 'testB'. Images in 'testA' only contain multiple people annotation. In contrast, images in 'testB' contain all other objects. Expressions in RefCOCOg [62] are collected on Amazon Mechanical Turk in a non-interactive setting. Thus, the expressions in RefCOCOg are longer and more complex. RefCOCOg has 'google' and 'umd' splits. The 'google' split does not have a public test set,

and exists an overlap between the training and validation image sets. The 'umd' split does not have this overlap. Therefore, to prevent data leakage of the test set and following previous studies [79, 102], we exclude the 'google' split in the fine-tuning settting and dataset-mixed pre-training setting . Thus, we trained and tested the RefCOCOg dataset only on the 'umd' split.

ReferItGame. ReferItGame [34] (short as ReferIt) belongs to the Phrase Grounding (PG) task, which contains images from SAIAPR12 [20] and collects expressions through a two-player game. In this game, the first player is shown an image with an object annotation and is asked to write a natural language expression referring to the object. The second player is then shown the same image along with the written expression and is asked to click on the corresponding area of the object. If the clicking is correct, both players receive points and swap roles. If not, a new image will be presented.

Flickr30k Entities. Flickr30k Entities (short as Flickr30k) [68] belongs to the phrase grounding task, which contains images in Flickr30k dataset. The query sentences are short noun phrases in the captions of the image. The queries are simpler and easier to understand compared to RefCOCO/+/g. Therefore, the ambiguity of the expression is heightened simultaneously, resulting in a relative increase in noise.

B.2 Explanation of the dataset abbreviations

In Tab. 2 of the main text, we provide abbreviations for the datasets used in intermediate pre-training. Specifically, 'GoldG' (proposed in MDETR [33]) is a mixed region-level fine-grained dataset created by combining three datasets - Flickr30k [68], MS COCO [50], and Visual Genome [39] - along with annotated text data for detection, REC and QGA tasks. It has a size of approximately 6 . 2 M. 'O365' refers to the Object365 [105] dataset, 'SBU' stands for SBU caption [65], 'VG' represents the Visual Genome [39] dataset, and 'OI' stands for OpenImage [42] dataset.

B.3 Comparison of datasets used in the pre-trained models

As presented in Tab. 9, we conducted an analysis of the datasets employed by the backbone models compared in Tab. 2 within the dataset-mixed pre-training setting. From the Tab. 9, it is evident that BEiT-3 and OFA utilize comparable datasets for pre-training. Conversely, other compared works in Tab. 2, such as Shikra [10], Ferret [100], LION [9], and other models, such as ONE-PEACE [86] (a tri-modality foundation model), employ significantly larger amounts of data than BEiT-3. Consequently, our method does not possess any advantage concerning the volume of data used in pre-training.

C Implementation details

Network Architecture. The detailed network structure of our framework is shown in Tab. 10. We employ BEiT-B/16 and BEiT-L/16 as the backbone for our OneRef base and large version,

respectively. In the structure of OneRef-B, the one-tower encoder are 12-layer Transformers with the hidden embedding dimension of 768. In the structure of OneRef-L, the one-tower encoder is 24-layer Transformers with the hidden embedding dimension of 1024. The one-tower encoder encodes both textual and visual modalities. Due to the utilization of a 3-layer deconvolution, RES exhibits a slightly higher number of model parameters compared to the REC task.

Training Details. The batch size for pre-training the base model and large model are (32, 8), while they are (32, 8) and (16, 6) for transferring to the REC and RES tasks, respectively. Our model is optimized end-to-end by using the AdamW optimizer and a cosine learning scheduler with an initial learning rate of 0 . 5 × 10 -4 for 110 epochs during the pre-training stage. During REC/RES transfer stage, the learning rates is 0 . 3 × 10 -4 with 20 epochs. The framework and experiments in our study were conducted using PyTorch. For MRefM pre-training, the base model took 15 hours on 32 NVIDIA A100 GPUs, while the large model took 50 hours on the same number of GPUs. As for REC/RES transfer fine-tuning training, it took an average of 3 hours for the base model and 8 hours for the large model to process one dataset on 8 A100 GPUs.

Inference Details. Unlike previous methods, such as TransVG++ [15], QRNet [98], etc . , which heavily rely on high-resolution images like 640 × 640, we adopt smaller resolution of 384 × 384. To ensure compatibility, we employ a long edge alignment and short edge pad filling scheme to the image. We include [SEP] and [EOS] token at the beginning and the end of the input text, and align it to a fixed length of 64 by padding empty tokens.

Model Hyperparameters. We summarize and report the hyperparameter settings of the OneRef framework in Tab. 11.

D Technical remarks

D.1 Further explanation for the effectiveness mechanism of the visual target-relation score

(1) The purpose of designing the Referring MIM algorithm. In the existing MIM paradigm, reconstruction is limited to solely relying on the visual features within the image. To enhance content reconstruction by leveraging cross-modal information as much as possible, our Referring MIM approach incorporates visual target-relation scores alongside visual modality content during reconstruction. This modeling approach presents increased difficulty as it necessitates reliance on

Referred region

Patch region

𝒑𝟏(𝒙𝟏 , 𝒚 𝟏 )

𝒑𝟐(𝒙𝟐 , 𝒚 𝟑 )

𝒑𝟑(𝒙𝟑 , 𝒚 𝟑 )

Referring direction

(𝒙𝒄 , 𝒚

𝒄 )

𝒉𝒓

𝒘𝒓

x-mask

y-mask

w-mask

h-mask

(𝑥 - 𝑥ୡ )/𝑊

(𝑦 - 𝑦 ୡ)/𝐻

𝑃/𝑤௥

𝑃/ℎ௥

𝑊

𝐻

The Visual Target-relation Score

textual information for reconstructing the two visual branch. Consequently, our model achieves a more comprehensive understanding of both visual and textual information. In this way, the model not only can perceive the information of the image modality itself but also have a more accurate understanding of the location and correlation of the key object features in different regions.

(2) How and why the visual target-relation score ( i . e ., the x-, y-, w-, and h-masks ) works. We provide a clearer illustration in Fig. 5 for further explanation. As mentioned in Sec. 3.2, this score represents the spatial distance between the current patch region and the referred region, it enables implicit deployment of grounding capability within each token of the model. When reconstructing the visual features and target-relation score of each local patch, the model actually needs to have an global and comprehensive understanding of the text modality information and the visual information. On this basis, the model needs to rely on the reconstructed visual features of the local patch to implicitly predict the specific location and size of the referred object, and then accurately predict the visual target-relation score. Finally, Referring MIM can enhance the model's global and multimodal understanding of textual and visual information, and then learn more general visual representations, which can have better generalization ability when deployed to downstream referring tasks.

The proposed Referring MIM is our own design, which is mainly used to improve the defects existing in MAE [27]/BEiT [4]. We can find the rationale of our method in some classic computer vision works, such as the YOLO series works [71], which predicts the location, size, confidence, and category of the object box corresponding to each grid cell based on the global understanding of the image. YOLO etc . [71] also confirmed that the object detection model obtained in this way has stronger generalization ability when transfer to detection tasks that differ greatly from the training data compared with other detectors.

D.2 The selection of the unsupervised regions

The process of selecting unsupervised regions bears resemblance to weakly-supervised visual grounding. Drawing inspiration from ALBEF's method [44] for weakly-supervised grounding, we employ a BEiT-3 model with performed image-text contrastive tuning to encode both the image and text, thereby obtaining a cross-modal text-to-image attention map for selection. Subsequently, leveraging the cross-modal attention and modular parsing of textual sentences provided by MAttNet [102] enables us to derive scores for each proposal. Finally, we select the region with the highest score as our objective in Referring MRefM.

D.3 Referring-aware text masking

In referring MLM, we utilize a referring-aware text masking strategy. Specifically, we preferentially mask out the referential subject of the expression text on the basis of a random mask, and the subject is obtained by the NLP parsing tool ( e . g ., spaCy). Since this small technical point does not observe a significant performance gain as the referring-aware dynamic image masking strategy, we do not provide additional ablation experiments.

D.4 The difference of the task heads between ours with other frameworks

Recently, several multi-task visual grounding studies [79, 45] have incorporated both grounding and segmentation task heads into their frameworks. Most relevance to our work is VG-LAW [79], which simplifies the implementation of grounding and segmentation heads by eliminating Transformer-based fusion encoders through visual adaptive weights generation. In contrast, for REC headers, we propose a box mask constraint based on cross-modal cosine similarity that significantly enhances the accuracy of such grounding approach. For the RES head, instead of employing adaptive weights generation, we directly obtain segmentation masks using cosine similarity for the visual tokens upsampled by a 3-layer deconvolution.

E Extra experimental results

E.1 The results on ReferIt and Flickr30k dataset under mixup pre-training setting

The results of our framework on the PG task ( i . e ., ReferIt [34] and Flickr30k [68] datasets) under the mixup pre-training setting are presented in Tab. 12. It is worth noting that a majority of studies conducted under this setting have not provided these results, thus only several works are included in the table. As shown in Tab. 12, our base model outperforms HiVG by 1.91 % and 1.93 % on the two datasets, and also achieves SoTA performance.

E.2 The results for RES task under oIoU metric

The results for RES task under oIoU metric are presented in Tab. 13. oIoU is calculated as the ratio between the total intersection area and the total union area of all test samples, each of which is consists of both a text query and an image. This metric particularly favors larger objects. As indicated in Tab. 13, (1) in the single-dataset fine-tuning setting, our base model outperforms RISCLIP [35] by 3.77 % , 8.87 % , and 5.52 % on RefCOCO/+/g testB/testB/test split, respectively. (2) Similarly, in the

dataset-mixed intermediate pre-training setting, our base model surpasses UNINEXT [93] by 2.03 % , 8.07 % , and 6.69 % on RefCOCO/+/g testB/testB/test split, respectively. Furthermore, our large model exhibits remarkable scalability with additional performance enhancements.

E.3 Computational costs analysis compared with SoTA methods

In this paper, we highlight two significant advantages of our model architecture over other frameworks: (a) Instead of using a Transformer to fuse visual and language features, we only employ a simple lightweight task head; (b) Our one-tower architecture eliminates the need for early interaction techniques in the backbone network, thereby reducing the computational complexity of the model.

We compare the energy efficiency of our model with several well-known SoTA works on the REC task from various perspectives, including the number of parameters, computational complexity (FLOPs), inference speed (FPS), and test time (s). As can be seen from Tab. 14, due to the simplification of our model's structure, the number of parameters and the calculation complexity are significantly lower than other well-known models. Specifically, our feature fusion and grounding head module only require 1.7M parameters, while other methods use 20M, meaning we only have about 8.5 % of their parameter count. Additionally, our computation is only 34.9 % of Grounding-DINO and 25.2 % of MDETR. Moreover, our inference speed is 10 × faster than Grounding-DINO and TransVG++ (the speed also related to the image size used by the model). Despite these advantages, thanks to the modality-shared feature space, we outperform all these well-known works.

E.4 Complete ablation study of MRefM on single-dataset fine-tuning and mixup pretrain settings

The complete ablation results of MRefM on both single-dataset fine-tuning and mixup pretrain settings are provided in Tab. 15, which serves as a supplement to Tab. 4 in the main text. In the table, the masking ratio is set to 0.4 when using block-wise or random masking strategies.

Query text: ' arm on right. '

Query text: ' front wine glass. '

Query text : 'woman to the left of old lady. '

Query text: 'the young child on the left sitting near the old person. '

Original image

Ground truth

REC Prediction

Cross-modal feature

RES Prediction

gpred,iou:0.87

pred, iou:0.96

pred,iou:0.89

gpred,iou:0.81

RefCOCO

E.5 Ablation study of the mask ratio in referring-aware dynamic masking strategy

As depicted in Tab. 16, we conducted ablation experiments on the mask ratio within our proposed referring-aware dynamic image masking strategy. It is observed that while a high mask ratio of 0.75 is employed for the pixel reconstruction of MAE [27], achieving better results for BEiT's feature reconstruction requires a mask ratio ranging from approximately 0.4 to 0.45. In our proposed approach, favorable outcomes can be attained by setting β and γ to 0.35 and 0.75, respectively; where β represents the mask ratio beyond the referred region and γ denotes the mask ratio within it. Experimental statistics show that our entire mask rate α in each sample is about 0 . 4 ∼ 0 . 5 .

F Visualization of the results

As shown in Fig. 6, Fig. 7, and Fig. 8, we present the qualitative grounding and referring segmentation results with several relatively challenging examples. Each example shows two different query texts. The cross-modal features are obtained by the cosine similarity between the [SEP] language token and the vision tokens on REC transfer model of OneRef-B. These results demonstrate the strong semantic comprehension capability of our OneRef model in complex text understanding and cross-modal grounding.

RefCOCO+

Query text : 'white tourist ship closest viewer. '

Original image

Query text: 'a decorated passenger ship is waiting for departure in a port. '

Query text: 'woman with blond hair. '

Query text: 'a blue shirt person giving presentation. '

oundtruth

pred, iou: 0.7

pred,iou:0.65

ground

pred, io

Note

Note:

Note:

pred,iou:0.83

Ground truth

REC Prediction

Cross-modal feature

RES Prediction

Query text: 'a plate of cookies and banana slice , raisins , and craisins. '

Query text: 'a bowl of pasta next to a bowl of salad. '

Query text: 'a group of six oranges on a wooden table surrounded by several other fruits and vegetables. '

Query text: 'the bag of vegetables behind all of the other fruits and vegetables. '

RefCOCOg

Original image

Ground truth

REC Prediction

Cross-modal feature

RES Prediction

pred,iou:0.92

pred,iou:0.93

MA

pred,iou:0.9

pred,iou:0.63

G Further discussions

G.1 Limitations

Firstly, despite achieving remarkable grounding and segmentation results, the pre-training in this paper solely relies on the comparatively limited RefC dataset, as opposed to other studies with larger datasets.

Secondly, the MRefM paradigm necessitates additional referential bounding boxes as supervised data compared to self-supervised pre-training. Therefore, we explore the potential of unsupervised pre-training for MRefM. However, when utilizing image-text pairs obtained from web crawling, there is no guarantee that the referred regions will exhibit strong correlation with the text due to many texts describing the entire image. This aspect introduces certain challenges and biases during large-scale pre-training of MRefM. Consequently, this paper should serve as an inspiration for subsequent researchers to propose more convenient plug-and-play modeling methods.

G.2 Broader impacts

OneRef demonstrates strong grounding and referring segmentation capabilities, while MRefM represents a novel modeling paradigm for referential relationship. This facilitates users to easily utilize our model ( e . g ., OneRef-L) for their own needs by simply providing some appropriate text queries. However, this also raises concerns about how our OneRef models with a strong understanding capabilities could be used inappropriately in the community, such as for large-scale illegal video surveillance. The open-set grounding capabilities could be manipulated through specialized textual cues to facilitate targeted detection or human tracking instead of generic ones. This manipulation could introduce biases in the detector and result in unfair predictions.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state the claims made, including the contributions made in the paper and important assumptions and limitations.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the limitations of this work in the Appendix G.1.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have provided the full set of assumptions and a complete proof in Sec. 3 of the main text.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have fully disclosed all the information needed to reproduce the main experimental results of the paper in Sec. 4 of the main text and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We answer No mainly for the following acceptable reasons: (1) The data we use are all publicly available and have been detailedly introduced in the paper. Researchers can acquire the data according to the provided reference information. (2) Due to time constraints, we were unable to compile and submit the anonymous code at the time of submission. However, as stated in the abstract, all models and code will be promptly made public after the decision is reached on this paper. (3) The implementation details of our work have been thoroughly explained in both the main text and supplementary materials. Even without publicly available code, researchers can reproduce it based on the information given in this paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have specified all the training and test details in the Sec. 4 of the main text and Appendix B.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The error bars are not reported because it would be too computationally expensive. Besides, it is not crucial for interpreting the experimental results in this task topic.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide sufficient information on the computer resources in Appendix C. Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have conformed in every respect with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed both potential positive societal impacts and negative societal impacts of the work performed in Appendix G.2.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the original paper ( e . g ., BEiT-3 [87]) that produced the code package or dataset.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The authors propose a unified one-tower referring framework that introduces the MRefM paradigm to capture the referential relationships between vision and text. They demonstrate the effectiveness and generality of MRefM across three settings on REC, PG, and RES tasks, consistently achieving good results.

STRENGTHS:
The proposed method is structurally simpler than previous methods and achieves higher grounding and RES performance.

The ablation experiments are thorough, validating the effectiveness of the two Ref mask modeling strategies.

WEAKNESSES:
Adapting BEiT, a robust general-purpose multimodal model, to downstream grounding tasks and achieving performance improvement does not seem to be a particularly interesting finding. Additionally, it is puzzling why introducing two types of relation scores in MIM and MLM would enhance the model's referring performance. Especially regarding the four types of masks in the Visual target-relation score, their effective mechanism is not intuitive and lacks explanation. Why did the authors decide to introduce referring information by predicting these four scores? Are there any reference works?

In Table 3, comparisons with some related works, such as UNINEXT [1] and HIPIE [2], are missing.

[1] Universal instance perception as object discovery and retrieval. CVPR 2023.

[2] Hierarchical Open-vocabulary Universal Image Segmentation. NeurIPS 2023.

QUESTIONS:
See weaknesses.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-02

comments: SUMMARY:
This paper proposes a Mask Referring Modeling (MRefM) paragram and a unified and extremely concise grounding and referring segmentation framework named UniRef that no longer requires the fusion or interaction of the Transformer structure and the special grounding tokens. A masked referring modeling (MRefM) is proposed to model the referential relationship, which encompasses referring-aware mask image modeling and referring-aware mask language modeling.

STRENGTHS:
A mask-referring modeling paragram is proposed to model the referential relation between visual and language effectively. They also propose a one-tower framework for grounding and referring segmentation in a unified modality-shared feature space. Experiments demonstrate the effectiveness of this method and its components.

WEAKNESSES:
1. Lack of discussion of related work [a]. I suggest discussing the difference between this paper's referring-aware mask language modeling and masked contrastive learning in [a]. 
2. Lack of computation costs analysis with other methods, including parameters/FLOPs/speed. 

[a] VLT: Vision-Language Transformer and Query Generation for Referring Segmentation. In TPAMI 2023.

QUESTIONS:
See weakness.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript proposes UniRef, a framework aimed at unifying visual and linguistic feature spaces for referring expression comprehension and segmentation. The key presented is the Masked Referring Modeling (MRefM) paradigm, which includes referring-aware MIM and MLM. \nThis approach seeks to streamline the architecture by eliminating the need for separate modality-specific encoders and complex interaction modules, achieving sota performance on several datasets.\n\nSTRENGTHS:\n1. The introduction of the MRefM paradigm effectively captures the referential relationship between visual and linguistic features, contributing to the robustness and accuracy of the model.\n2. The integration of visual and linguistic feature spaces into a unified framework simplifies the model architecture and potentially improves computational efficiency.\n3. The authors provide extensive experimental results across five datasets, demonstrating the effectiveness of the proposed approach in outperforming existing methods.\n\nWEAKNESSES:\n1. The technical contribution of the proposed method appears to be insufficient, as it primarily adapts traditional Masked Autoencoders into a 'Referring MAE', despite the rich experimentation presented.\n\n2. Section 3.2 is intricate and challenging to follow. I recommend a thorough proofreading and revision of this section to enhance its clarity and readability.\n\n3. Could you explain the rationale behind designing the system to encompass four masks: x-, y-, w-, and h-masks?\n\n4. The manuscript asserts that the approach is lightweight, raising questions about its real-time applicability. It would be helpful if the authors detailed the computational demands and processing speed of the method in practical scenarios.\n\n5. (Minor) While the manuscript has discussed the limitations of the proposed approach, I am interested in understanding how MrefM performs in more general contexts. For instance, evaluating the backbone linear probing (or its integration with LLMs) could demonstratebroader applicability. This should certainly be considered for future work.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}