Scene Graph Enhanced Pseudo-Labeling for Referring Expression Comprehension
Authors: Cantao Wu
Yi Cai
Liuwu Li
Jiexin Wang
Venue: EMNLP 2023 Findings (2023)
Keywords: Referring Expression Comprehension
Visual Grounding

Abstract: Referring Expression Comprehension (ReC) is a task that involves localizing objects in images based on natural language expressions. Most ReC methods typically approach the task as a supervised learning problem. However, the need for costly annotations, such as clear image-text pairs or region-text pairs, hinders the scalability of existing approaches. 
In this work, we propose a novel scene graph-based framework that automatically generates high-quality pseudo region-query pairs. Our method harnesses scene graphs to capture the relationships between objects in images and generate expressions enriched with relation information. To ensure accurate mapping between visual regions and text, we introduce an external module that employs a calibration algorithm to filter out ambiguous queries. Additionally, we employ a rewriter module to enhance the diversity of our generated pseudo queries through rewriting. 
Extensive experiments demonstrate that our method outperforms previous pseudo-labeling methods by about 10%, 12%, and 11% on RefCOCO, RefCOCO+, and RefCOCOg, respectively. Furthermore, it surpasses the state-of-the-art unsupervised approach by more than 15% on the RefCOCO dataset.

Scene Graph Enhanced Pseudo-Labeling for Referring Expression Comprehension
Cantao Wu', Yi Cai Liuwu Li', Jiexin Wang 1.2.+
School of Software Engineering, South China University of Technology 2 Key Laboratory of Big Data and Intelligent Robot (South China University of Technology) Ministry of Education mhotaotta04122gmalll com
{ycai, jiexinwang] edu. cn liuwu. ww@outlook...... com
Abstract
Original Query from RefCoCo dataset: the bowl that los blueberries on
Pseudo- Q Query:
Referring Expression Comprehension (ReC) is a task that involves localizing objects in im- ages based on natural language expressions. Most ReC methods typically approach the task as supervised learning problem. However, the need for costly annotations, such as clear image-text pairs region-text pairs, hinders the scalability of existing approaches. In this work, we propose a novel scene graph-based framework that automatically generates high- quality pseudo region-query pairs. Our method harnesses scene graphs capture the relation- ships between objects in images and generate expressions enriched with relation information. To ensure accurate mapping between visual re- gions and text, we introduce an external mod- ule that employs a calibration algorithm to fil- ter out ambiguous queries. Additionally, employ a rewriter module to enhance the diver- sity of our generated pseudo queries through rewriting. Extensive experiments demonstrate that our method outperforms previous pscudo- labeling methods by about 10%, 12%, and 11% on RefCOCO, RefCOCO+. and RefCOCOg, respectively. Furthermore, it surpasses the state- of oh-theeart art unsupervised approach by more than 15% on the RefCOCO dataset.
white bowl White bowl
SGEPG Query.
left top bowl with blueberry above table
on the right side Uhere's white bowl with green brocoali inside
Figure 1: Example of queries generated by Pseudo-Q and SGEPG methods, based in an image from the RefCOCO dataset. Note that the query generated by SGEPG without rewriting is shown in red, while the query generated by SGEPG with rewriting is shown in blue.
different objects
Deng et al., 2021; Yang et al., 2022: Ye et al., 2022) trained on manually annotated region-query data. However, constructing large-scale ReC datasets, especially for textual queries referring the ob- jects, is rather time-consuming. To obtain the clear textual queries, annotators need to provide specific visual information about the referred object, such as its category and attributes. Moreover, due to the presence of multiple similar and confusing objects in images, queries often require additional details to distinguish the referred object, such spatial in- formation (left, right), object state (running, jump- ing), and object relationships (e.g.. person holding ball). Although some weakly-supervies methods (Chen al., 201 Sa; Datta et al., 2019; Gupta et al., 2020) partially alleviate the annotation burden, they still rely on paired image-query data, limiting their applicability in the real-world scenes.
1 Introduction
Referring expression comprehension (ReC) is an important -inual-linguisii task that aims to locate objects in images based on given textual referring expressions. Referring expressions, such as "give me that apple in the basket," " are commonly used in social communication. The ability to accurately comprehend such expressions in real-world scenar- ios is also essential for robots and other intelligent agents to enable natural interactions (Fang et al., 2015; Qi et al., 2020). In recent years, considerable progress has been made in ReC task, particularly for fully-supervised models (Chen et al., 201 8b;
To address the annotation challenges, several works (Yeh et al., 2018; Wang and Specia, 2019; Jiang et al., 2022) have explored unsupervised ap- proaches that do not rely paired annotations.
Nonetheless, these approaches either employ statis- tical hypothesis testing, make simple assumptions, or only investigate the shallow relation between objects, leading to poor performance in complex scenes. For instance, Pseudo-Q (Jiang et al., 2022),
Cooreesponding author

---

which adopts method of generating pseudo-labels train a supervised model. It utilizes an offline ob- ject detector to extract salient objects from images and generates pseudo queries using predefined tem- plates with object labels and attributes. However, Pscudo-Q has two limitations: (1) It fails to capture deep relation information among objects in images, leading to pseudo queries that only partially cover the semantic space of referring expressions and deviate significantly from human expressions. (2)
The experimental results demonstrate the effec- tiveness of our method, which significantly reduces the manual labeling cost by about 40% on the RefCOCO dataset, with only a slight decrease performance compared to the fully supervised set- ting. In addition, our proposed method consistently outperforms the state-of-the-art unsupervised ap- proach by about 10-13 points the RefCOCO (Yu et al., 2016), RefCOCO+ (Yu et al., 2016), and RefCOCOg (Mao al., 2016) datasets. Our main contributions can be summarized as follows:
The absence of a validation mechanism for the gen- erated pscudo queries introduces the possibility of producing incorrect instances. Both limitations can
We propose the first scene graph-based pscudo- labeling method for unsupervised ReC task, which effectively represents the key visual fea- tures of different objects and captures their re- lationships, resulting in region-query pairs that express richer relational information.
lead to degradation in model performance.
In this paper, we present SGEPG (Scene Graph Enhanced Pscudo-Query Generation), an unsuper- vised framework designed to generate high-quality region-query pairs. SGEPG leverages the gener- ated scene graph extract rich semantic informa- tion about objects and capture their complex re- lationships, enabling the selective generation of region-query pairs from the input image. We next introduce a straightforward yet effective strategy based on the scene graph, which helps filter out ambiguous queries, It is crucial to address ambigu- ity as unclear region-text pairs can yield incorrect supervisory signals, which ultimately degrade the performance of the ReC model. Additionally, propose a module rewrite some of the gener- ated queries. This rewriting process enhances the diversity of the generated pseudo queries while also making them more akin to human expressions. As depicted in Figure 1, compared to Pseudo-Q. Q. the SGEPG method generates pseudo queries that are both unambiguous and exhibit higher quality. Moreover, the generated queries by SGEPG also demonstrate more complex inference paths, such as the paths from "bowl" to the "blueberries".
To address the issue of generating ambiguous queries that yield incorrect supervisory signals and degrade performance, we introduce dis- ambiguation module based on the scene graph structure to further align the region-query modal- ities. Additionally, we propose rewriting some queries to increase the diversity and make them more closely resemble human expressions.
Experimental results demonstrate that SGEPG not only effectively alleviates the reliance on manual annotation, but also outperforms coun- terparts and achieves comparable performance to large pre-trained vision- language methods.
2 Related Work
2.1 Referring Expression Comprehension
The task of Referring Expression Comprehension (ReC) plays crucial role in applications such as robot navigation and visual question answering. ReC methods can be roughly classified into three types: fully supervised (Deng et al., 2021; Yang et al., 2022), weakly supervised (Gupta et al., 2020; Liu al., 2019a, 2021; Sun al., 2021), and un- supervised (Jiang et al., 2022: Subramanian et al.. 2022; Wang and Specia, 2019; Yeh et al., 2018). Although fully supervised methods have achieved state-of ofte--of-the-ar results in the ReC task (Kamath et al., 2021; Yang et al., 2022), they heavily rely vast and expensive manually annotated region- query pairs for training. In contrast, weakly super- vised methods aim alleviate this reliance by train- ing the model using only image-text pairs. Most weakly supervised methods (Chen et al., 2018a; Liu et al., 2021) use object detectors to obtain po-
Specifically, the SGEPG framework comprises four essential components: Core Scene Graph Generation, which is responsible for obtaining an informative core graph to capture object rela- tions; Non-Ambiguous Query Generation, which focuses on generating unambiguous queries; Diver- sity Driven Query Rewriter, which is designed to rewrite some of the generated queries, aiming to increase the diversity and alleviate the differences between pseudo-queries and human expressions; Visual-L Visual-Language Module, which aims to capture the contextual relationship between the image re- gions and corresponding pseudo queries.

---

tential regions corresponding the expressions. treating them as alternatives to missing bounding boxes for training the model. However, annotat- ing the expressions in ReC datasets is the most time-consuming and labor-intensive process.
sentation that provides important contextual clues. Therefore, they propose utilizing triples (subject, relation, object) from scene graphs as additional semantic information, along with visual features, as input to generate caption. Yang et al. (2019) introduce Scene Graph Auto-Encoder module and incorporate inductive language bias into an encoder-decoderr image captioning framework to produce more human-like captions.
To address the need for manual annotations and reduce annotation costs, there is growing interest in unsupervised methods (Jiang et al., 2022; Subra- manian et al., 2022; Wang and Specia, 2019; Yeh et al., 2018). For instance. Wang and Specia (2019) employ multiple detectors (e.g., RaNNer-RCNN (Gir- shick, 2015)) to detect objects, scenes, and colors in images. By comparing the similarity between visual regions and the nouns in the query, the cor- responding region related to the query can be iden- tified. ReCLIP, proposed by Subramanian et al. (2022), introduces the use of CLIP (Radford et al., 2021) to assign scores to various image regions based on their similarity to the query. The region with the highest similarity score is then selected the final result. While these methods leverage prior knowledge to analyze the alignment between visual regions and text, the inference speed is rel- atively slow due to the encoding and scoring of image regions and text in each step. One particu- lar unsupervised approach, Pseudo-Q (Jiang et al., 2022). stands out by considering using pseudo- labelling techniques. The generated pseudo-labels can be employed to train supervised ReC models, allowing the models to acquire grounding abilities without the requirement for manually labeled data, However, Pseudo-Q fails to capture deep relation information among objects images and does not address the ambiguity issue, leading to poor quality queries, as shown in Figure 1.
The task of scene graph generation (Dai et al., 2017; Xu et al., 2017; Zhang et al., 2022; Zellers et al., 2018; Tang et al., 2019) aims to obtain a complete graph structure representation of an im- age, where nodes represent object categories and edges indicate relationships between objects (e.g.. verson-riding-bike Early works on scene graph generation focused on local visual information and overlooked contextual information in images. To address this issue, Xu et al. (2017) proposed an end-to-end model that improves visual relation- ship prediction by iteratively propagating contex- tual information across the scene graph's topology. Furthermore, most scene graph generation mod- cls struggle with data distribution issues, such as long- tail distributions. This poses challenges in ac- curately predicting frequent relationships that lack informative details (c.g., on, at), limiting the appli- cability of scene graph models in practical tasks. To address this, Zhang et al. (2022) propose In- ternal and External Data Transfer. This approach automatically generates enhanced dataset, mit- igating data distribution problems and providing more coherent annotations for all predictions.
Methodology
Unlike the existing pscudo-labeling methods (Cheng et al., 2021; Feng et al., 2019; Jiang et al., 2022), SGEPG incorporates scene graphs, enabling the generation of queries with inferred paths and rich relational information.
In this section, we present the SGEPG frame- work, which aims to automatically generate region- query pairs from given images and train models to tackle ReC task using these pseudo labels. As depicted in Figure 2, the SGEPG framework con- sists of four key components: Core Scene Graph Generation, which is responsible for obtaining an informative core graph to capture object rela- tions; Non-Ambiguous Query Generation, which focuses on generating unambiguous queries; Diver- sity Driven Query Rewriter, which is designed to rewrite some of the generated queries, aiming increase the diversity and alleviate the differences between pseudo-queries and human expressions; Misual-Languee Module, which aims to capture the contextual relationship between the image re-
2.2 Scene Graph
Scene graphs contains structured semantic infor- mation about images, including object categories, attributes, and pairwise relationships. They can pro- vide valuable prior information and have proven useful for various tasks like image captioning (Yang et al., 2019; Li and Jiang, 2019; Zhong et al., 2020). For instance, Li and Jiang (2019) argue that previous image caption approaches, which treat each object individually, lack structured repre-

---

gions and corresponding pseudo queries. Note that in this work, we mainly focus on the first three mod- ules, with the objective of generating pseudo-labels of good quality for training visual- visual-language mod- els designed for the ReC task. To accommodate space limitations, certain methodological details may have been omitted. For a more comprehensive understanding, please refer to Appendix A for de- tailed information regarding the methods employed in this study.
sions. Therefore, we next extract these three key aspects from the image:
1. Category. Inspired by two stage ReC methods (Yu et al., 2018), we employ an object detector to obtain object proposals (e.g., "skier" in Figure 2).
2. Attribute. To extract attributes like material (e.g., wooden), color (e.g., red), and status (e.g.. standing), we utilize an attribute classifier (Ander- son et al., 2018). However, existing models have a constraint in extracting only the most confident attribute for each object, even though an object can have multiple attributes.
3.1 Core Scene Graph Generation
The objective of this module is to obtain an in- formative core graph that accurately describes the objects in the image and their relationships. The scene graph, denoted G (V, E), is a directed graph consisting of vertices V representing ob- ject instances characterized by bounding boxes with object category labels, and edges E repre- senting relationships between objects. Each edge e, = (U,, ry) indicates a relation type r; be- tween subject and object t...
3. Spatial Information. Humans often use spa- tial information to refer an object, such as "right apple", "top guy". We obtain spatial information by comparing the center coordinates of object bound- ing boxes. Specifically, we derive horizontal spatial information (e.g., left, right) and vertical spatial information (c.g., top, bottom) by comparing the object center coordinates along these dimensions. After extracting the above three information, we first update the categories of the vertices in the pre- liminary scene graph with the obtained category in- formation. Next, we introduce attribute nodes and spatial nodes into the scene graph, representing the attribute information and spatial relationships with the corresponding object nodes, that are denoted as [Atr], [Spatial], respectively. As a result, we generate informative core scene graph that en- compasses the three crucial pieces of information, as illustrated the lower left corner of Figure 2.
Firstly, we utilize scene graph generation tech- niques to construct a preliminary scene graph for a given image. Specifically, we employ the model proposed by Zhang et al. (2022), as discussed in Section 2.2. However, the generated preliminary scene graph may not be complete due to the limi- tations of current scene graph approaches. These approaches often provide coarse and insufficient ob- ject categories in the generated graph, represented by generic labels such as "flower" instead of spe- cific labels like "rose".! Furthermore, the generated scene graph only contains the relationship informa- tion between objects and fails to capture attribute details of each object, such as the color of a rose. Thus, the preliminary scene graph lacks specific information about object attributes.
3.2 Non-Ambiguous Query Generation
The core scene graph captures the rich semantic in- formation of the entire image, which can be utilized to generate queries by populating its information into designed templates.? However, using the en- tire scene graph may result in queries that are too complex and contain excessive specific informa- tion for referring the objects. In ReC tasks, queries are typically concise and only describe partial in- formation relevant to the objects being referred to, rather than providing an overall description of the scene. To address this, we focus on subgraphs that contain the nodes corresponding to the objects of interest. These subgraphs capture partial informa- tion about the objects, which can be leveraged to obtain concise pseudo queries.
To address the aforementioned limitations, we expand the preliminary graph by introducing ad- ditional nodes and edges to construct a complete scene graph. Following the insights from (Jiang et al., 2022), the referring expressions typically involve three crucial aspects: category, attriutes and spatial information. Additionally, humans nat- urally combine these visual semantic cues when referring to specific target objects, particularly in complex scenes with multiple objects of similar classes, reducing ambiguity in the referring expres-
However, the ambiguity issue may arise if there are similar subgraphs present in the scene graph.
NNote that the model proposed by Zhang et al. (2022) only supports 150 classes.
Please refer to Appendix B for the designed templates.

---

mbiguous Query Generation
Core Scone Graph Generation
prelimitary scene graph
A Amiquuity Resplver Strategg
Visual-L Language Module
Sy Check Opention: 84 S Gg
skirt
bEEK
coaL
Frcatio
Hack
Person
sauting
GEAT
skine
night
Person
siriedes
Fusion
coot
pants
Fillinng Temeplate
diver
tark
Pseuto Queiles
juahi bipartal)
1.iee wertin idwk cod
Ta buni bwill begt
skdrr
тá [Atri juhni Indl
dik buck left
Core Scone G raph
parts
15:
Jukie freet of woods.
dark
Mick
wooder
Blakk
Rewritive Parati (jartine
glunsem
house
glasses
backpack
Lis
skine
luft
skine
Lnght
S b s or
Selact
Lakie vouriay liack coe. and parts
Pseudo queries
skar decaing cou.
lause
coat
black
Eark
3. sker list of woodan Lasseikk
Diversity Driven Query Rewwiter
Cbjectandd
(ARa/15pauu)
Figure 2: The framework of SGEPG, which consists of four modules: Core Scene Graph Generation, Non- Ambiguous Query Generation, Diversity Driven Query Rewriter, and Visual-Language The diagram also illustrates how SGEPG generates queries related to Person
An ambiguous query can refer to multiple objects in the image, which might lead to incorrect super- vision signals detrimental for effective learning of visual- Lisual-language module. Existing pseudo query- based models, such as Pseudo-Q, struggle with this issue and may generate ambiguous queries that refer to different objects, as depicted Figure 1, To address this issue, we propose a simple yet ef- fective Ambiguity Resolver Strategy based on the characteristics of the scene graph. The main idea is that a query referring to an object is non- ambiguous if its corresponding subgraph is not subgraph of any other subgraphs representing dif- ferent objects in the scene graph. Specifically, let
rewriter component. It aims to to enhance the di- versity and bridge the gap with human expressions, while preserving semantic consistency. To achieve this, GPT-3.5,, poweful large pre-trained language model is utilized rewrite the queries. GPT-3.5 also has the capability to incorporate additional prior knowledge into queries. For example, may animal" associat-
rewrite "giraffe" as ing the abstract concept of a "long-necked" with the entity of "giraffe". This integration of prior knowl- edge helps the SGEPG generate more accurate and meaningful queries. Furthermore, we introduce important hyperparameter a, which determines the proportion of generated pseudo queries that will undergo the rewriting process.
G and GB denote the graphs of objects A and B, respectively, and S^ represent the selected sub- graph from G for generating the query about A. To resolve ambiguity, S should satisfy the condi- tion SA & GB. For instance, in the left corner of the Non-Ambiguous Query Generation module on the right side of Figure 2, the subgraph represent- ing skier-pants-dark of "Person A" is removed as is identical subgraph of "Person B".
3,4 Visual-Language Module
In this module, our objective to train visual- language model that learns the mapping between image regions and textual descriptions, using lseudo-labels obtained from the previous modules. To achieve this, we adopt similar architecture to VLTVG (Yang et al., 2022), visual- language model designed for the ReC task through super- vised learning. The VLTVG model consists of four components: a visual encoder, a language encoder, a fusion module, and a multi-stage cross- modsl decoder. To extract visual features from the given
3.3 Diversity Driven Query Rewriter
To address the limited diversity and fluency issues pscudo queries generated by directly filling sub- graphs into predefined templates, we introduce a

---

image, we employ DETR (Carion et al., 2020) as the visual encoder within our model. For the tex- tual modality, we utilize BERT (Devlin al., 2018) as the language encoder to obtain feature represen- tations of the corresponding expressions. These visual and textual representations are then fused using a fusion module. Finally, the fused features are passed through multi-stage cross-modal de- coder, which generates the coordinates for the final predicted bounding box for the referred object in the image. Note that while we have specifically uti- lized this visual language model architecture, it can be replaced by any other supervised ReC models.
(Subramanian et al., 2022), CPT (Yao et al., 2021), and Pseudo-Q(Jiang et al., 2022) as well as weakly- supervised methods like VC (Zhang et al., 2018), ARN (Liu et al., 2019a), KPRN (Liu et al., 2019b), and DTWREG (Sun et al., 2021). We also include the random guess results and the -tatc- results achieved by MDETR (Kamath et al., 2021), It is worth noting that Pseudo-Q. like our SGEPG approach, also relies on generating pseudo queries, as mentioned earlier. However, Pseudo-Q Q employs different visual-language model than VLTVG OYang et al., 2022), which we have chosen as our visual language model. To ensure a fair compari- son, we train VLTVG model using the region-text pairs generated by Pseudo-Q, and refer to it as Pseudo Pseudo-Q-adlppice Q-adaptive in our evaluation.
4 Experiment
4.1 Datasets and setups
We evaluate SGEPG on three datasets: RefCOCo, RefCOCO+ and RefCOCOg. The images in these datasets are from the MSCOCO (Vinyals et al.. 2016) and we follow the same train/val/test splits from (Subramanian et al., 2022). The number of training images for these three datasets is 16994, 16992, and 24698, respectively. Compared to Ref- COCO, the queries in RefCOCO+ do not include spatial words (e.g. "left", "top") to describe objects, and RefCOCOg queries are typically longer and involve more interactions between objects. The re- ferred objects in testA split are all human, the testB split comprises only non-human objects, while val split does not have such restriction.
Accuracy (96)
79
val
testA
- testB
74
30
Percent of Pseudo-samples (%)
(a)
85
Ours
Pseudo-Q
824
Accuracy
83
For each image in the RefCOCo, RefCOCO+, and RefCOCOg datasets, select the top ob- jects based on their confidence scores and generate 3 queries for each selected object. The hyperpa- rameter a for the Rewriter is set to 50%. The linuual- component of our proposed model is trained independently for 30 epochs using the generated pscudo-labels of the images from each dataset, with the remaining hyperparameter settings consistent with those reported in the VLTVG paper. The evaluation of the trained model is conducted on the respective test set of each dataset. Thus, dur- ing the training stage of our approach, we do not utilize any manually annotated datasets, i.c., the human annotated image region-query pairs, which are solely used for evaluation purposes.
50
Percent of Pseudo-samples (%)
(b)
Figure Experiments of reducing the manual labeling cost. Figure (a): results of our method three splits of RefCOCO; Figure (b): comparision results between our method and Pseudo-Q on val split of Re(COCO.
4.3 Main Results
The results on RefCOCO, RefCOCO+, and Ref- COCOg are presented in Table 1. We begin by comparing our proposed SGEPG model with the unsupervised methods: ReCLIP, and Pscudo-Q. We observe that our proposed SGEPG model demon- strates superior performance. In comparison ReCLIP, the state-of-the- unsupervised method which relics on pre-trained visual- lisuul-language models, SGEPG surpasses ReCLIP by a remarkable 15% on
4.2 Baselines
We compare the performance of our SGEPG ap- proach with several existing methods, including unsupervised methods such as ReCLIP, GradCAM

---

RefCOCO
RefCOCO+
RefCOCOg
method
Sup
val
testA
test
testA
testB
val
test
Random
15.73
13.51
19.20
16.29
13.37
19.60
18.12
19.10
Supervised SOTA
Full
87.51
90.40
82.67
81.13
85.52
72.96
83.35
81.64
VC(Zhang et al., 2018)
33.29
30.13
34.60
31.58
ARN(Liu et al., 2019a)
34.26
36.43
33.07
34.53
36.01
33.75
Weak
KPRN(Liu et al.. 2019b)
35.04
34.74
36.98
35.96
35.24
36.96
DTWREG(Sun et al., 2021)
39.21
41.14
37.72
39.18
40.10
38.08
CPT(Yao al., 2021)
CPT-BIK w/ VinVL
No
26.9
27.5
27.4
25.4
25.0
27.0
32.1
32.3
CPT-Segg w/ VinVL
32,2
36.1
30.3
31.9
35,2
28.8
36.7
36.5
CLIP
GradCAM
No
44.65
53.49
36.19
49.41
59.66
38.62
52.29
51.28
ReCLIP
54.04
58.60
49.54
55.07
60.47
47.41
60.85
61.05
Pseudo-labeling
Sseudo-Q(Jiam) et al., 2022)
56.02
58.25
54.13
38.88
45.06
32.13
46.25
47.44
Pscudo-Q-adapted
59.44
61.87
56.47
42.76
48.27
36.90
50.35
49.68
SGEPG (Ours)
69.61
72.34
65.67
55.21
60.67
46.76
61.33
60.61
Table 1: Comparison with ReC methods on RefCOCO/+/g in terms of top-1 accuracy (%). Bold indicates the best performance of the column except for the supervised method. Sup column indicates the method category: Full. Weak, No represent fully supervised, weakly supervised and unsupervised, respectively.
couch with zed cushion
black glasses blue blanket watching outside fence
man holding puper
boat with writings
food ahave fries side with
train number 242
broccol; middle and with brown stuff
Figure 4: Visualization examples of detection results. The first of images displays the visualized detection results of both SGEPG and Pseudo-Q. The second row showcases four instances where SGEPG encountered detection failures, with the corresponding results from Pseudo-Q included for comparison. The green bounding boxes and queries are ground truth and the red bounding boxes and yellow bounding boxes are the detection results of SGEPG and Pseudo-Q, respectively.
Comeer I:A bowl
bluebenry above
upper siting is dressed in Khaki
Mas in red jocket
helmee slanding white snow
sible
In the 6 olclods position, there's man scated
there's
white
bowl broccoli
black being held by the man
skier wearing jocket
dircction. a sitting
with
walking on the white snow
man is observing a piratle
Figure 5: The visualization results of pseudo-labels generated by SGEPG. Different colored queries corresponding to different objects.

---

RefCOCO
RefCOCO+
objects are directly used to fill the templates and generate pseudo queries. Moreover, as the ARS relies on the scene graph, the model without SG
val
testA
testB
val
testA
testB
w/o SG
63.45
63.9
63,47
43.8
45.81
44.14
w/o ARS
65.
63.81
64.65
48.5
50.47
46.02
w/o DDRQ
68. 82
72.46
64,99
53,78
58.26
cannot utilize this strategy to address ambiguity. 46. 14
SGEPG
69.61
72.34
65.67
55.21
60.67
46.76
For the models without ARS or DDQR, we either exclude the strategy or remove the DDQR compo- nent directly from the SGEPG model. The results of this ablation study are presented in Table 2. Firstly, when comparing the performance of the model without SG to the complete SGEPG, we observe that the complete SGEPG achieves ab- solute performance improvements of 6.16% and 11 1..41 % on the validation sets of RefCOCO and RefCOCO+, respectively. This also indicates that the SG can effectively improve the model's 's ability to align visual and textual information, especially on challenging datasets like RefCOCO+. Secondly, the complete SGEPG also outperforms the model without the ARS, with absolute improvements of 8.53% and 10.2% on the RefCOCO/+ testA, respec- tively. Finally, we observe that when the DDQR module is removed, the model's 's performance de- crease across both datasets. The above findings demonstrate that the modules proposed by us can provide more accurate supervision signals to the ReC model and improve its performance.
Table 2: Main ablation test results on the RefCOCO/+.
replace rate (a)
0%
25%
37.5%
50%
75%
val
53.78
54,29
54.94
55.21
55.33
testA
58.26
60.15
58.99
60.67
60.9
testB
46.14
46.41
46.7
46.76
46.8
Table 3: Rewriting ratio test results of DDQR on Ref- COCO+,
the RefCOCO dataset, while achieving comparable performance on RefCOCO+ and RefCOCOg. Re- garding Pseudo-Q, which is also based on pseudo- labeling, our SGEPG model outperforms it by an average margin of 10% to 12.5%. This demon- strates that the queries generated by SGEPG for referring to objects can better ensure accurate map- ping between the two modalities.
4.4 Efficiency Improvement of Manual Annotation
We then conduct experiments to assess the capabil- ity of SGEPG in reducing manual labeling costs. Following Pseudo-Q, replace the manually notated region-text pairs whose queries contain spatial relationships in RefCOCO with the pseudo- labels generated by SGEPG. The results, depicted in Figure 3(a), demonstrate the significant reduc- tion in manual labeling costs achieved by SGEPG. When about 40% of the manually annotated region- text pairs are replaced with our generated pseudo- labels, there is only minimal impact on the model's compared SGEPG
4.6 Effect of Different Replacing Ratios in Rewriter
In this section, we conduct experiments to exam- ine the impact of rewriting ratios a in the DDRQ. We perform rewriting on pseudo queries with a values of 25%, 37.5%, 50%, and 75%. The final re- sults are presented in Table 3. The results indicate that as the rewriting ratio increases, the model 's performance improves. This finding suggests that incorporating more diverse expressions can allevi- ate the discrepancy between pscudo queries gener- ated by artificial templates and human expression. However, there is a noticeable diminishing return effect: the more sentences are rewritten, the train- ing cost increases linearly while the performance gain becomes limited. Therefore, we strike bal- ance between training cost and model performance, and find that rewriting ratio of 50% is appropriate.
performance. In addition, with Pseudo-Q in Figure 3(b), which indicates that our SGEPG model has the ability to generate supe- rior quality pscudo-labels compared to Pseudo-Q.
4.5 Ablation Study
In this section, we perform extensive ablation stud- ies on the RefCOCO and RefCOCO+ datasets to evaluate the effectiveness of the key aspects of SGEPG, including scene graph (SG), ambiguity re- solver strategy (ARS), and Diversity Driven Query Rewriter (DDQR). To evaluate the impact of the SG component, we compare the performance the model without SG. In this scenario, the category, attribute, and spatial information of the detected
4.7 Qualitative Analysis
Qualitative examples. To validate that our ap- proach can provide queries with rich relational information and enhance the model's perception of relationships, we conducted visualizations of some generated examples in Figure 5. Further-

---

more, we performed visual comparisons with the results from Pscudo-Q on selected cases, directly comparing them (first row of Figure 4). We ob- serve that SGEPG effectively captures the complex relational information among objects the scene, while Pscudo-Q tends to randomly select an object from the query's s nouns the detection object. For instance, in the third example in first row, Pscudo- Q generates detection results solely based the word "Person" in the sentence, completely disre- garding the "holding" relationship and the "paper" object. This also suggests that SGEPG capable of better capturing the relationships between objects.
sea. Consequently, the generated pseudo queries may not adequately cover these object classes, po- tentially affecting the comprehensiveness of the generated queries.
Acknowledgement
This work supported by the National Natural Science Foundation of China (62076100), Funda- mental Research Funds for the Central Universi- ties, SCUT (x2riD2230080), the Science and Tech- nology Planming Project of Guangdong Province 102000101100002 CAAl-Huawei MindSpore Open Fund, CCF-Zhipu AT Large Model Fund.
Failure case. Additionally, while second row of Figure 4 presents cases where SGEPG encountered prediction errors that can primarily be attributed the two limitations of the scene graph. Firstly, the scene graph may lack the ability to recognize text present in images. Secondly, the scene graph generation model may struggle in complex scenes, resulting in inaccurate scene graph generation.
References
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. ceedings of the HEEF conference on computer vision and pattern recognition, pages 6077-6086.
Pro-
5 Conclusion
Geoffrey H Ball and David J Hall. 1965. Isodata, novel method of data analysis and pattern classification. Technical report, Stanford research inst Menlo Park CA.
In this paper, we propose SGEPG, a novel pseudo- labeling ReC method that leverages scene graphs to improve the generation of pseudo queries. By utilizing the scene graph, SGEPG captures object relationships and generates pseudo queries that are rich in relation information, thereby improving the training of ReC models. Additionally, we propose a simple yet effective strategy to filter out ambiguous generated queries. Our method outperforms pre- vious pscudo-labeling approaches on three bench- mark datasets and achieves comparable or superior performance compared to other unsupervised ReC methods. Furthermore, SGEPG offers the advan- tage of significantly reducing manual labeling costs while maintaining the model's performance.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In Computer Vision-ECCV 2020: 16th European Conference, Glusgow, UK, August 23- 28, 2020, Proceedings, Part 1 16, pages 213-229. Springer.
Kan Chen, Jiyang Gao, and Ram Nevatia. 2018a. Knowledge aided consistency for weakly supervised phrase grounding, In Proceedings of the IEEE Con- ference Computer Vision and Patlern Recognition, pages 4042-4050.
Xinpeng Chen, Lin Ma. Jingyuan Chen, Zequn Jie, Wei Liu, and Jiebo Luo. 2018b. Real- referring ex- pression comprehension by singlc-stage grounding network. arXiv preprint arx1v:1812.03326.
Limitations
While our method has demonstrated excellent per- formance on three datasets, there are still some lim- itations. Firstly, the quality of the generated results relies on the accuracy of the scene graph genera- tion. In complex scenes, there is no guarantee that all objects and their relationships can be correctly identified, which may impact the effectiveness of our approach. Secondly, the capabilities of the scene graph are limited in terms of object detection. It can only detect a specific set of objects and may fail to identify certain objects such as the sky or
Cheng Cheng, Chunping Li, Youfang Han, and Yan Zhu. 2021. A semi-supervised deep learning image cap- tion model based on pseudo label and n-gram. Inter- national Journal of Approximate Reasoning, 131:93- 107.
Bo Dai, Yuqi Zhang, and Dahua Lin. 2017. Detecting visual relationships with deep relational networks. In Proceedings of the IEEE conference on computer vision and Pattern recognition, pages 3076-3086.
Samyak Datta, Karan Sikka, Anirban Roy, Karuna Ahuja, Devi Parikh, and Ajay Divakaran. 2019. Align2ground: Weakly supervised phrase grounding

---

guided by image-caption alignment. In Proceedings the IEEE/CVF International Conference on Com- puter Vision, pages 2601-2610.
Yongfei Liu, Bo Wan, Lin Ma, and Xuming He. 2021. Relation-aware instance refinement for weakly su- pervised visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- lern Recognition, pages 5612-5621.
Jiajun Deng, Zhengyuan Yang. Tianlang Chen, Wen- gang Zhou, and Houqiang Li. 2021. Transvg: End- to-end visual grounding with transformers. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 1769-1779.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Murphy. 2016. Generation and comprehension of unambiguous ob- ject descriptions. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 11-20.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810004805.
Yuankai Qi. Qi Wu, Peter Anderson, Xin Wang. William Yang Wang, Chunhua Shen, and Anton van den Hengel. 2020. Reverie: Remote embodied visual referring expression in real indoor environ- ments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9982-9991.
Rui Fang. Malcolm Doering, and Joyce Y Chai. 2015. Embodied collaborative referring expression genera- tion in situated human-robot interaction. In Proceed- ings of the Tenth Annual ACM/IEEE International Conference on Human-Roben RRobot Interaction, pages 271- - 278.
Yang Feng, Lin Ma. Wei Liu, and Jiebo Luo. 2019. Unsupervised image captioning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 4125-4134.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin. Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International confer- ence on machine learning, pages 8748-8763. PMLR.
Ross Girshick. 2015. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440-1448.
Sanjay Subramanian, Will Merrill, Trevor Darrell, Matt Gardner, Sameer Singh, and Anna Rohrbach. 2022. Reclip: A strong zero-shot baseline for re- ferring expression comprehension. arXiv preprint arXiv:2204.05991.
Tanmay Gupta, Arash Vahdat, Gal Chechik, Xi- aodong Yang, Jan Kautz, and Derek Hoiem. 2020. Contrastive learning for weakly supervised phrase grounding. In European Conference on Computer Vision, pages 752-768. Springer.
Mingjie Sun, Jimin Xiao, Eng Gee Lim, Si Liu, and John Y Goulermas. 2021. Discriminative triad match- ing and reconstruction for weakly referring expres- sion grounding. IEEE transactions on pattern analy- sis and machine intelligence, 43(1
Haojun Jiang. Yuanze Lin, Dongchen Han, Shiji Song. and Gao Huang. 2022. Pseudo-q: Generating pseudo language queries for visual grounding. In Proceed- ings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 15513-15523.
Kaihua Tang, Hanwang Zhang, Baoyuan Wu, Wenhan Luo, and Wei Liu. 2019. Learning to compose dy- namic tree structures for visual contexts. In Proceed- ings of the IEFF/CVF conference on computer vision and pattern recognition, pages 6619-6621 6628.662..
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Car- ion. 2021. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1790-
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Du- mitru Erhan. 2016. Show and tell: Lessons learned from the 2015 mscoco image captioning challenge. IEEE transactions on paltern analysis and machine intelligence, 39(4))652-663.
Xiangyang Li and Shuqiang Jiang. 2019. Know more say less: Image captioning based on scene graphs. IEEE Transactions on Multimedia, 21(8):21
Xuejing Liu, Liang Li, Shuhui Wang, Zheng-Jun Zha,
Josiah Wang and Lucia Specia. 2019. Phrase localiza- tion without paired training examples. In Proceed- ings of the IEEF/CVF International Conference on Computer Vision, pages 4663-4672
Dechao Meng, and Qingming Huang. 2019a. Adap- tive reconstruction network for weakly supervised re- ferring expression grounding. In Proceedings of the IEFF/CVF International Conference Computer Vision, pages 2611 1-220..
Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei- Fei. 2017. Scene graph generation by iterative mes- sage passing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5410-5419.
Xuejing Liu, Liang Li. Shuhui Wang, Zheng-Jun Zha, Li Su, and Qingming Huang. 2019b. Knowledge- guided pairwise reconstruction network for weakly supervised referring expression grounding. In Pro- ceedings of the 27th ACM International Conference on Multimedia, pages 539-547.
Li Yang, Yan Xu. Chunfeng Yuan, Wei Liu, Bing Li, and Weiming Hu. 2022. Improving visual grounding

---

with risual-linguise verification and iterative rea- soning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9499-9508.
A
More Details of the Scene Graph Generation
A.1 Spatial
Xu Yang, Kaihua Tang, Hanwang Zhang, and Jianfei Cai. 2019. Auto-encoding scene graphs for image captioning. In Proceedings of the IEEF/CVF Confer- ence Computer Vision and Pattern Recognition, pages 10685-10694.
In this section, we will elaborate on some addi- tional details regarding the extraction of spatial information. addition to the four cardinal direc- tions (up, down, left, and right) that we are familiar with, we have incorporated some extra spatial de- scriptions to help the model better learn the concept of "spatial"..
Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. 2021. Cpt: Col- orful prompt tuning for pre-trained vision-language p models. arXiv preprint arXiv:2109. 1797.
Firstly, we introduce the concept of "corner". Sometimes, humans use the term "corner" to re- fer to objects that are located in the corners of an image. Therefore, when extracting the spatial in- formation of objects, we also determine whether the object is positioned at the edges or corners of the image. In this work, we consider an object to have the "corner" attribute if it satisfies the follow- ing conditions: Firstly, the object's bounding box should not be located in the center of the image. Secondly, the object's bounding box should have at least two edges that are positioned at the image's s edges. Objects that meet these criteria are assigned the additional "corner" attribute.
Jiabo Ye, Junfeng Tian, Ming Yan, Xiaoshan Yang. Xuwu Wang, Ji Zhang, Liang He, and Xin Lin, 2022. Shifting more attention to visual backbone: Query- modulated retinement networks for end-to-end visual grounding. In Proceedings of the IEEE/CVF Confer- ence Computer Vision and Pattern Recognition, pages 15502-15512.
Raymond A Yeh, Minh Do. and Alexander G Schwing. 2018. Unsupervised textual grounding: Linking words to image concepts. In Proceedings of the IEEE Conference on Computer Vision and Put- tern Recognition, pages 6125-6134.
Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara I. Berg. 2018 Mattnet: Modular attention network for referring expression comprehension. In Proceedingy of the IEEE Confer- ence Computer Vision and Pattern Recognition, pages 1307-1315.
Secondly, we have observed that people often use the term "alone" refer an object that ap- pears to be "solitary". Therefore, when there are multiple instances of a certain object category in an image (typically three or more), we employ clus- tering algorithms(Ball and Hall, 1965) based on coordinates and object bounding box sizes to at- tempt to divide them into two groups. If one of the groups contains exactly one object, consider that object to be "alone" and assign it the additional "alone" attribute.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander Berg. and Tamara L Berg. 2016. Modeling context in referring expressions. In European Conference on Computer Vision, pages 69-85. Springer.
Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. 2018. Neural motifs: Scene graph parsing with global context. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 5831-5840.
A.2 Scene Graph Cleaning
Ao Zhang, Yuan Yao, Qianyu Chen, Wei Ji, Zhiyuan Liu, Maosong Sun, and Tat-Seng Chua. 2022. Fine- grained scene graph generation with data transfer. arXiv preprint arXiv:2203.1 11654.
While we can utilize scene graph generation models to extract scene graphs from images, the obtained scene graphs may contain errors due to limitations in model performance. Although most incorrectly predicted relations can be filtered out based on confidence scores, some contradictory results may still remain. For example, scene graph may con- tain two conflicting triples such as "boy-sitting at- chair" and "boy-standing on-floor," which contra- diets common sense as person cannot be both "sitting" and "standing." " In such cases, we only retain the triple with the highest confidence level.
Hanwang Zhang, Yulei Niu, and Shih-Fu Chang. 2018. Grounding referring expressions in images by vari- ational context. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 4166-
Yiwu Zhong, Liwei Wang, Jianshu Chen, Dong Yu, and Yin Li. 2020. Comprehensive image captioning via scene graph decomposition. European Conference on Computer Vision, pages 21 211-229. Springer.

---

B Designed Template
Prompt: "Your "Your goal is as human-being to gen- erate diversified, accurate and semantically rich referring expressions according to a description of the referred object. The requirement of generated expression: The category, color, position, texture, relationship with other objects or others segments (entities) may be reorganized with some alternative expressions. I will give you some descriptions, and then you generate referring expressions for per description. You should return a dict in which the key is the description and the values are the 3 ex- pression involed a list, the format looks like ("De- scription": "xxxxx, "Values": ("1": "xxxxx", "2": "xxxxx" "3": "xxxxx"] J. Don't i leave out any sen- tences. Now, begining your generation following the goal, requirement and format:"
Due to the fact that scene graphs are graph struc- tures while natural language a sequential text, is necessary to find an appropriate method to con- vert graph structures into text sequences. In this paper, we employ method that involves populat- ing manually designed template with information extracted from the scene graph. The scene graph encompasses four main types of information: 1. Object categories, 2. Object attributes, 3. Rela- tionships between objects and 4.spatial. Therefore, the elements to be filled in the manually designed template primarily consist of these three types of information. Here are the main templates used in this paper:
[attr] [sub]
Input: center tallest man in glasses
Output: / ""Descriptoo"., "Cennter tallest man in glasses": ("I":"At the center, the tallest man wearing glasses is present.", " """""""aa"ssssss ""."Gaasssswearinng man who is the tallest is in the center.", "T":"The center position is occupied by man, who is the tallest and wears glasses." "]]
|sub] [rel] [obj]
[sub] in/on [spatial] (corner I ]
[attr] [sub] in/on [spatial] [comner a
[attr] [sub] [rel] [obj] in/on [spatial]
Input: corner left top bowl with blueberry above table
[attr] [sub] [rel] [obj] and [rel] [obj]
Output: { ""Description": "corner left top bowl with blueberry above table", "Values": { "1"" "The blueberry-filled bowl is positioned in the top left corner, above the table.", "2": "Above the table, there's 'ss''s a bowl in the top left corner, and it contains blueberries.", ", "3": "In the top left corner, there's a bowl placed above the table, and it has blueberries inside." " )
[attr] [sub] [rel] [attr] [obj] and [rel]
[attr] [obj]
[attr] [sub] [rel] [attr] [obj] and [rel] [attr]
[obj] in [spatial] [corner aalone)
In this context, "attr" refers to the attribute of an object, "sub" represents the subject, "obj" repre- sents the object, "spatial" indicates the spatial, and "rel" represents the relationship between subject and object. Additionally, "corner" and "alone" are optional and indicate whether the object is located in the corner of the image and whether it is "alone," respectively. By filling in the scene graph infor- mation into the templates, we can obtain pseudo queries generated based on the scene graph. Please note that the word order may vary, and "spatial" could be placed at the beginning of the sentence.
It is important to note that not every rewriting attempt is successful. There are instances where GPT-3.55 may fail to rewrite or generate unexpected statements, such outputs that do not meet the expected format provide insufficient information. For such cases where rewriting is not possible, we simply skip them and proceed without making any changes.
D Additional Experiments
D.1 Effect Expressions Number per Object and Object Number per Image
Prompt of Rewriter
We use GPT3.5 (ChatGPT) to rewrite the pscudo queries have constructed. We employ prompt communicate to GPT3.5 that our task objec- tive to rewrite sentences, with each sentence being rewritten three times. The following is the specific prompt we use, along with an example:
In this section, investigate the impact of the number of generated pseudo queries for each ob- ject, as well as the number of pseudo-labeled ob- jects generated per image, as shown in Table 4. Please note that in this experiment, by default, 4 objects are selected per image to generate pseudo-

---

Expression number per object
Object number per image
2
3
4
2
5
SGEPG
66.17
68.21
68.6
68.81
62.3
67.61
68.82
68.6
68.02
Table 4: Results of different expressions numbers per object and different object numbers per image on RefCOCO.
RefCOCO testA
RefCOCO+ testA
val
testB
val
testB
preliminary scene graph
36.23
45.31
25.32
37.21
46.11
26.55
Core scene graph (SGEPG)
69.61
72.34
65.67
55.21
60.67
46.76
Table 5: preliminary scene graph generation results on RefCOCO and ReiCOCO+.
labels, and statements are generated for each object.
Experimental results are shown in Table 5. No- tably, we observed that training the model solely with preliminary scene graphs, which are noisy and unprocessed, led to poor performance. This finding underscores the importance of accurate and refined core scene graphs in our approach. These core scene graphs, complemented with additional information from off-the-shelf object detectors and attribute classifiers, play crucial role in enhane- ing the quality of generated pseudo-labels and ul- timately improving the performance of visual- language model.
Intuitively, generating larger number of pseudo queries for each object allows for broader cov- erage of the semantic space. The experimental results demonstrate that increasing the number of expressions generated per object leads to improved performance, while insufficiently generated expres- sions result in limited coverage the semantic space, leading to a decline in model performance. However, excessively generating expressions for each object can significantly increase training time and increase costs. Hence, to strike balance be- tween performance and training efficiency, have set this value thermore, as the number objects increases, the number of generated pscudo queries also rises, lead- ing to an improvement our model's performance. However, is important to note that our approach utilizes the confidence score from the object de- tector to select salient objects. Objects with low confidence
our other experiments. Fur-
prone to providing incorrect in- formation regarding object categories and visual re- gions. Thus, when the number of objects is higher, the likelihood of introducing such erroneous in- formation and negatively impacting the model's s performance increases.
D.2 Effect of Core Scene Graph
In order to assess the impact of scene graph gen- eration quality and key information on the results of pseudo-label generation, we also attempted to generate pseudo-labela directly using preliminary scene graph rather than core scene graph. We performed the experiments on both the RefCOCo and RefCOCO+ datasets. these experiments, we generated pscudo-labed directly using the prelim- inary scene graph without any further processing.


Reviews:
[{"review_id": "ttsZ9iqQYB", "reviewer": "Reviewer_3Jqz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and we have carefully considered the important issues you raised.\n\n**(a)\tThis paper presents a highly effective engineering method for ReC. However, it should be noted that the proposed framework incorporates some combinatorial and heuristic aspects. In particular, the Non-Ambiguous Query Generation procedure relies on a sophisticated filtering template. It would be helpful if the author could clarify the impact of these heuristic components.**\n\n**A:** We appreciate your insightful comments regarding the combinatorial and heuristic aspects of our proposed framework. Indeed, the proposed framework involves some filtering templates, but it's worth clarifying that the templates used in the Non-Ambiguous Query Generation are straightforward, as detailed in Appendix B. Our approach doesn't intricately design complex templates; rather, it populates available subgraph information into basic fillable templates. For instance, a subgraph obtained after applying Ambiguity Resolver Strategy (ARS), like \"man-wearing-shirt,\" is populated into a template like \"[sub] [rel] [obj]\". However, templates such as \"[attr] [sub]\" or \"[sub] [rel1] [obj1] [rel2] [obj2]\" can not be suitable for filling with the information from this subgraph, as it might not completely align with the content of these templates (e.g., missing attribute or additional relationship information).\n\nThe ARS, described in Section 3.2 of the paper, serves as the tool to filter subgraphs. The guiding principles of ARS, along with illustrative examples, can be found in Lines 374 to 388 of the paper. We have also conducted ablation experiments in Section 4.5 to demonstrate the effectiveness of ARS, and the results are presented in Table 2.\n\n**(b)\tSince the linguistic expression rewriting utilizes the powerful GPT-3.5 language model, it would be interesting to understand the extent of randomness and deviation that may arise from the influence of GPT-3.5. Is there any studies or analyses on this aspect?**\n\n**A:** We appreciate your interest in understanding the impact of linguistic expression rewriting using the GPT-3.5 language model in our work. However, we would like to clarify a few points in this regard:\n1. First of all, it's important to emphasize that our primary focus is on the utilization of scene graphs to generate high-quality pseudo-labels for ReC. As can be observed from Table 2, while  Diversity Driven Query Rewriter (DDQR) is a part of our framework and does contribute to further enhancing the model's query diversity and fluency, its impact on the overall model performance is not as substantial as the Scene Graph (SG) and Ambiguity Resolver Strategy (ARS).\n2. To provide some insights, we conducted an additional experiment to evaluate the semantic consistency of the rephrased sentences generated by DDQR. Among a randomly selected subset of 100 pseudo-label sentences subjected to DDQR rewriting, each sentence was rewritten three times, resulting in 300 rewritten pseudo-labels. Remarkably, 280 of these rewritten statements maintained the same semantic meaning as the original sentences. Additionally, during the rewriting process, out of the 100 original sentences, 87 instances generated multiple rewritten sentences that were consistent with the original sentences.\n3. Additionally, there have been studies analyzing the diversity and consistency of rewrites generated by GPT-3.5, as referenced in [1][2]. While our focus was not solely on DDQR, these analyses further support the effectiveness and reliability of the rewriting process in our approach.\n\n*[1]: Jang M, Lukasiewicz T. Consistency analysis of chatgpt[J]. arXiv preprint arXiv:2303.06273, 2023.*\n\n*[2] Feng Y, Qiang J, Li Y, et al. Sentence simplification via large language models[J]. arXiv preprint arXiv:2302.11957, 2023.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed methods demonstrate impressive performance on benchmarks for unsupervised ReC.\n2. Extensive experiments confirm the effectiveness of the proposed model components.\n3. The paper is easily understandable and straightforward to follow.", "reasons_to_reject": "1. This paper presents a highly effective engineering method for ReC. However, it should be noted that the proposed framework incorporates some combinatorial and heuristic aspects. In particular, the Non-Ambiguous Query Generation procedure relies on a sophisticated filtering template. It would be helpful if the author could clarify the impact of these heuristic components.\n2. Since the linguistic expression rewriting utilizes the powerful GPT-3.5 language model, it would be interesting to understand the extent of randomness and deviation that may arise from the influence of GPT-3.5. Is there any studies or analyses on this aspect?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper tackles label-efficient unsupervised referring expression comprehension (ReC). Specifically, it presents a scene-graph-based approach to generate high-quality pairs of image regions and linguistic expressions, facilitating joint representation learning at the region level. Initially, the image is embedded into a scene graph using heuristic rules. To tackle the ambiguity inherent in rule-based graph representations of diverse regions, a method for generating region queries is proposed to resolve ambiguity. Additionally, a rewriting module based on GPT-3.5 is employed to enhance linguistic expression diversity. The proposed approach demonstrates clear superiority over previous unsupervised methods.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The pipeline figure (Fig. 2) could benefit from further improvement. For instance, the notation 'F' used in the figure is unclear and could be made more explicit. Additionally, the image layout could be further adjusted to enhance clarity and comprehension."}}, {"review_id": "hrEAWQ3Q8G", "reviewer": "Reviewer_G4b6", "summary": "", "questions": "- Question A: A preliminary scene graph is complemented with the additional information from the off-the-shelf object detector and attribute classifier and spatial information of the bounding box of each object. Have you conducted experiments using a preliminary scene graph, not utilizing a core scene graph? This ablation study would be helpful for readers to understand the impact of the quality of generated scene graphs in the pseudo-query generation process.\n- Question B: This paper adopts GPT-3.5 to make pseudo queries to be diverse and fluent. An additional qualitative analysis of the rewriter would be needed to validate the effectiveness of the rewriting strategy.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and we have carefully considered the important issues you raised.\n\n**(a)\tAlthough SGEPG is model-agnostic, this paper conduct experiments using a specific model, VLTVG, to validate the proposed method. The additional experiments using the other VLM would be helpful to emphasize the effectiveness of the scene graph-enhanced pseudo-labeling approach.**\n\n**A:** We appreciate your suggestion for a broader exploration of model compatibility. To underline the efficacy of our scene graph-enhanced pseudo-labeling approach, we have conducted additional experiments using the QRNet [1], in addition to VLTVG. Specifically, we present the results of training QRNet using pseudo-labels generated by our SGEPG method (\"SGEPG-QRNet\") and the Pseudo-Q approach (\"Pseudo-Q-QRNet\"). These experiments were conducted on the RefCOCO and RefCOCO+ datasets.\n\nHowever, it's important to note that due to computational resource limitations and time constraints, we were unable to include the DDQR in our approach when working with QRNet, that further improves performance when using VLTVG. Additionally, we made slight adjustments to the QRNet model's parameter settings during training, including reducing the image size from 640 to 384 pixels and the number of training epochs from 160 to 20. Despite these modifications, our results still offer valuable insights, clearly demonstrating the effectiveness of our SGEPG method. The achieved performance improvements over Pseudo-Q-QRNet underscore the impact of incorporating scene graphs in the pseudo-label generation process.\n\n|                |       | RefCOCO |       |       | RefCOCO+ |       |\n| -------------- | ----- | ------- | ------- | ------- | -------- | ----- |\n|                | val   | testA   | testB   | val   | testA    | testB |\n| Pseudo-Q-QRNet | 56.09 | 59.96 | 54.62   | 37.08 | 44.78    | 32.82 |\n| SGEPG-QRNet    | 62.62 | 62.61 | 60.38   | 44.39 | 47.04    | 40.00 |\n\n*[1] Ye J, Tian J, Yan M, et al. Shifting more attention to visual backbone: Query-modulated refinement networks for end-to-end visual grounding[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 15502-15512.*\n\n**(b)\tA preliminary scene graph is complemented with the additional information from the off-the-shelf object detector and attribute classifier and spatial information of the bounding box of each object. Have you conducted experiments using a preliminary scene graph, not utilizing a core scene graph? This ablation study would be helpful for readers to understand the impact of the quality of generated scene graphs in the pseudo-query generation process.**\n\n**A:** We appreciate your suggestion and have already conducted the experiments as suggested to assess the impact of using a preliminary scene graph on the pseudo-query generation process. We performed the experiments on both the RefCOCO and RefCOCO+ datasets. In these experiments, we generated pseudo-labels directly using the preliminary scene graph without any further processing. \nNotably, we observed that training the model solely with preliminary scene graphs, which are noisy and unprocessed, led to poor performance. This finding underscores the importance of accurate and refined core scene graphs in our approach. These core scene graphs, complemented with additional information from off-the-shelf object detectors and attribute classifiers, play a crucial role in enhancing the quality of generated pseudo-labels and ultimately improving the performance of the visual-language model.\n\n|                            |       | RefCOCO |       |       | RefCOCO+ |       |\n| -------------------------- | ----- | ------- | ----- | ----- | -------- | ----- |\n|                            | val   | testA   | testB | val   | testA    | testB |\n| preliminary scene graph    | 36.23 | 45.31   | 25.32 | 37.21 | 46.11    | 26.55 |\n| Core scene graph （SGEPG） | 69.61 | 72.34   | 65.67 | 55.21 | 60.67    | 46.76 |\n\n**(c)\tThis paper adopts GPT-3.5 to make pseudo queries to be diverse and fluent. An additional qualitative analysis of the rewriter would be needed to validate the effectiveness of the rewriting strategy.**\n\n**A:** We appreciate your interest in understanding the impact of linguistic expression rewriting using the GPT-3.5 language model in our work and the effectiveness of the rewriting strategy. However, we would like to clarify a few points in this regard:\n\n1. First of all, it's important to emphasize that our primary focus is on the utilization of scene graphs to generate high-quality pseudo-labels for ReC. As can be observed from Table 2, while DDQR is a part of our framework and does contribute to further enhancing the model's query diversity and fluency, its impact on the overall model performance is not as substantial as the Scene Graph (SG) and Ambiguity Resolver Strategy (ARS).\n2. To provide some insights, we conducted an additional experiment to evaluate the semantic consistency of the rephrased sentences generated by DDQR. Among a randomly selected subset of 100 pseudo-label sentences subjected to DDQR rewriting, each sentence was rewritten three times, resulting in 300 rewritten pseudo-labels. Remarkably, 280 of these rewritten statements maintained the same semantic meaning as the original sentences. Additionally, during the rewriting process, out of the 100 original sentences, 87 instances generated multiple rewritten sentences that were consistent with the original sentences.\n3.  Additionally, there have been studies analyzing the diversity and consistency of rewrites generated by GPT-3.5, as referenced in [1][2]. While our focus was not solely on DDQR, these analyses further support the effectiveness and reliability of the rewriting process in our approach.\n\n*[1]: Jang M, Lukasiewicz T. Consistency analysis of chatgpt[J]. arXiv preprint arXiv:2303.06273, 2023.*\n\n*[2] Feng Y, Qiang J, Li Y, et al. Sentence simplification via large language models[J]. arXiv preprint arXiv:2302.11957, 2023.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper is well-written and easy to follow. The motivation and design choice of the proposed method is simple and intuitive. The proposed method shows consistent performance improvement on the three benchmark datasets for referring expression comprehension.", "reasons_to_reject": "- Although SGEPG is model-agnostic, this paper conduct experiments using a specific model, VLTVG, to validate the proposed method. The additional experiments using the other VLM would be helpful to emphasize the effectiveness of the scene graph-enhanced pseudo-labeling approach.", "questions_to_authors": "- Question A: A preliminary scene graph is complemented with the additional information from the off-the-shelf object detector and attribute classifier and spatial information of the bounding box of each object. Have you conducted experiments using a preliminary scene graph, not utilizing a core scene graph? This ablation study would be helpful for readers to understand the impact of the quality of generated scene graphs in the pseudo-query generation process.\n- Question B: This paper adopts GPT-3.5 to make pseudo queries to be diverse and fluent. An additional qualitative analysis of the rewriter would be needed to validate the effectiveness of the rewriting strategy.", "paper_topic_and_main_contributions": "- This paper proposes scene graph enhanced pseudo-query generation (SGEPG) for referring expression comprehension. The proposed method shows significant performance improvements in RefCOCO, RefCOCO+, and RefCOCOg benchmark datasets.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- L52: object state(running,\n- L576: examples.To"}, "extra_text": {}}, {"review_id": "0mPr2pz3MJ", "reviewer": "Reviewer_2nSK", "summary": "", "questions": "Question A: Following the concern above, I am wondering how much performance gain on the RefCOCO dataset is gained by using the scene graph vs. extracting spatial information. Could you please provide metrics for the ablation study on the use of spatial information on the RefCOCO dataset?\nQuestion B: The core scene graph generation flow is interesting with a preliminary graph generated with a model supporting a limited number of classes, and then decorated with more fine-grained categories, attributes, and spatial information with a separate extractive model. The paper cited the limitation of the scene graph generation model for the two-stage approach. Do you think that extending the capabilities of the scene graph generation model (to support additional classes and attributes/spatial information extraction) would provide better or similar results?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and we have carefully considered the important issues you raised.\n\n**(a)\tHow much performance improvement the approach can provide in more general cases is less clear. In particular, the RefCOCO dataset contains spatial words in the description that can be better captured by the extracted spatial information and not inherently validating the gain from using scene graphs or capturing relationships. As expected, we observe a larger performance gain using this approach on the RefCOCO dataset compared to the second place approach. For the other two datasets (RefCOCO+ and RefCOCOg), which are more complex and do not contain spatial relations, the performance of the system using the proposed approach is very close to the ReCLIP approach.**\n\n**A:** We appreciate your thorough analysis and concerns. We would like to address these points and provide further clarification:\n\n1. We agree that handling spatial information play a crucial role in referring expression comprehension tasks. However, approaches like Pseudo-Q and ReCLIP have also introduced specific methods or modules to address spatial information. For example, Pseudo-Q incorporate spatial words into pseudo-labels, and ReCLIP employs a dedicated \"Spatial Relation Resolver\" module for handling spatial information. And we also found that RefCOCOg dataset also contains spatial information words and descriptions with redundant information. \n2. In our experiments, we consistently outperform Pseudo-Q, which is also based on pseudo-labels, across various datasets. This demonstrates that our method, with the incorporation of scene graphs and the proposed improvements, is more effective in capturing object relationships, not limited to spatial information, and producing accurate pseudo-labels for training the visual-language model.\n3. Our method's consistent performance across different dataset complexities is noteworthy. ReCLIP's unique pattern of performing well on complex datasets (RefCOCO+/g) and performing poor on simpler ones (RefCOCO) can be attributed to its origin in CLIP, which excels in handling complex textual scenes and struggles with shorter text and visually similar objects. On the other hand, our approach remains stable on complex datasets (RefCOCO+/g) and performs exceptionally well even on datasets with simpler textual descriptions (RefCOCO). This also suggests that our method's effectiveness extends beyond the specific linguistic characteristics of individual datasets.\n4. In our ablation experiments (Section 4.2), we conducted comprehensive analyses of the influence of scene graphs on model performance. The results, presented in Table 2, showcase significant gains from scene graphs for both RefCOCO and RefCOCO+ datasets. This holds true even for RefCOCO+, which lacks spatial words. For instance, on the validation set of RefCOCO+, the inclusion of scene graphs improves the model's performance from 43.8 to 48.5, reflecting a substantial enhancement.\n\n**(b)\tFollowing the concern above, I am wondering how much performance gain on the RefCOCO dataset is gained by using the scene graph vs. extracting spatial information. Could you please provide metrics for the ablation study on the use of spatial information on the RefCOCO dataset?**\n\n**A:** We appreciate your suggestion and have already conducted the experiments as suggested to assess the impact of using spatial information. We removed the utilization of spatial information while retaining the scene graph-based approach. The results clearly demonstrate a significant decrease in performance when spatial information is excluded, which underscores the importance of spatial information in the ReC domain.\n\nMoreover, it's worth noting that the consideration of spatial information is not unique to our approach but rather reflects the practical demands of the ReC task. Baselines like Pseudo-Q and ReCLIP also incorporate mechanisms to deal with spatial information, as mentioned in the first reponse. We hope these insights address your query and provide a clearer perspective on the importance of spatial information in our approach.\n\n|                       |       | RefCOCO |       |\n| --------------------- | ----- | ------- | ----- |\n|                       | val   | testA   | testB |\n| w/o spatial           | 45.43 | 53.76   | 34.60 |\n| with spatial（SGEPG） | 69.61 | 72.34   | 65.67 |\n\n\n**(c)\tThe core scene graph generation flow is interesting with a preliminary graph generated with a model supporting a limited number of classes, and then decorated with more fine-grained categories, attributes, and spatial information with a separate extractive model. The paper cited the limitation of the scene graph generation model for the two-stage approach. Do you think that extending the capabilities of the scene graph generation model (to support additional classes and attributes/spatial information extraction) would provide better or similar results?**\n\n**A:** Thank you for raising this interesting point. Expanding the capabilities of the scene graph generation model to encompass a broader range of object categories and attributes is indeed an appealing idea. This expansion could potentially lead to the capture of more accurate semantic information from images, resulting in higher-quality pseudo-labels and, in turn, improved model performance. However, there might be a potential issue here. Directly increasing the number of recognized object categories in the scene graph generation model could also introduce complexities during training. For instance, inferring relationships between a larger set of objects might become more intricate, potentially leading to convergence issues. Yet, it's worth noting that technology is continuously advancing, and our approach could directly benefit from the ongoing progress in the field of scene graph generation models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The scene graph-based pseudo-labeling approach is novel and is shown to outperform previous pseudo-labeling methods on standard datasets.\n2. The choice of scene graph as well as the design of various techniques to further improve the performance of the approach is well motivated, described in detail, and the benefits are validated in the ablation studies.", "reasons_to_reject": "How much performance improvement the approach can provide in more general cases is less clear. In particular, the RefCOCO dataset contains spatial words in the description that can be better captured by the extracted spatial information and not inherently validating the gain from using scene graphs or capturing relationships. As expected, we observe a larger performance gain using this approach on the RefCOCO dataset compared to the second place approach. For the other two datasets (RefCOCO+ and RefCOCOg), which are more complex and do not contain spatial relations, the performance of the system using the proposed approach is very close to the ReCLIP approach.", "questions_to_authors": "Question A: Following the concern above, I am wondering how much performance gain on the RefCOCO dataset is gained by using the scene graph vs. extracting spatial information. Could you please provide metrics for the ablation study on the use of spatial information on the RefCOCO dataset?\nQuestion B: The core scene graph generation flow is interesting with a preliminary graph generated with a model supporting a limited number of classes, and then decorated with more fine-grained categories, attributes, and spatial information with a separate extractive model. The paper cited the limitation of the scene graph generation model for the two-stage approach. Do you think that extending the capabilities of the scene graph generation model (to support additional classes and attributes/spatial information extraction) would provide better or similar results?", "paper_topic_and_main_contributions": "This paper presents a pseudo-labeling approach to generate training examples for the Referential Expression Comprehension (ReC) task in an unsupervised manner using scene graphs. Specifically, the approach consists of three components to generate and decorate a core scene, filter out ambiguous queries, and rewrite the query for greater diversity. The main contributions of this paper are\n1. the scene graph-based pseudo-labeling framework,\n2. techniques such as the disambiguation module and the rewriting component to improve the quality of pseudo-labeling, and\n3. quantitative results showing that the high quality of the data enables better model performance compared to previous unsupervised approaches.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Missing spaces between word and punctuation in places such as line 52 (after \"object state\") and line 576 (after \"Qualitative examples.\").\n2. Inconsistent fonts on line 374."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces an unsupervised technique for generating training data for Referring Expression Comprehension (ReC). The proposed method initiates by constructing a core scene graph, subsequently filters out ambiguous queries, and lastly rewrites the query using GPT-3.5 to enhance the quality of the pseudo labels.

When tested on standard benchmarks, this data generation method has shown efficacy across various academic ReC datasets. However, questions about its generalizability have emerged. Notably, the most significant improvement is evident on RefCOCO, which contains spatial terms, making scene graphs apt for capturing principal relations between objects. This might not necessarily translate to datasets outside this specific domain. Additionally, there are reservations about the template-based filtering step's adaptability to diverse distributions and the robustness of its underlying heuristics.
Decision: Accept-Findings