Segment-Level and Category-Oriented Network for Knowledge-Based Referring Expression Comprehension
Year: 2023

Abstract:
Knowledge-based referring expression comprehension (KB-REC) aims to identify visual objects referred to by expressions that incorporate knowledge. Existing methods employ sentencelevel retrieval and fusion methods, which may lead to issues of similarity bias and interference from irrelevant information in unstructured knowledge sentences. To address these limitations, we propose a segment-level and category-oriented network (SLCO). Our approach includes a segment-level and promptbased knowledge retrieval method to mitigate the similarity bias problem and a categorybased grounding method to alleviate interference from irrelevant information in knowledge sentences. Experimental results show that our SLCO can eliminate interference and improve the overall performance of the KB-REC task. ‡

heading: 1 Introduction
text: Referring expression comprehension (REC), a.k.a. visual grounding, aims to identify a visual object referred to by a referring expression that disambiguates multiple objects (Cirik et al., 2018; Qiao et al., 2021). As a core task of languagevision fields, REC benefits many downstream multimodal tasks, e.g., robotics (Berg et al., 2020; Wang et al., 2022) and vision-and-language navigation (Qi et al., 2020; Gao et al., 2021).
To explore a broader domain of knowledge, Wang et al. extend the REC task to knowledgebased referring expression comprehension (KBREC), and propose a baseline model and benchmark (Wang et al., 2020). This task requires the use of external knowledge (e.g., commonsense and encyclopedia) to refer to objects. This necessitates the model’s ability to retrieve knowledge related to expressions and associate it with image and expression, enabling localization of the referent.
*Equal contribution. †Corresponding author. ‡The code is available at https://github.com/Buki2/SLCO.
The existing method ECIFA (Wang et al., 2020) retrieves and fuses knowledge in a sentence-level framework. They utilize sentence-level similarity to retrieve the most similar unstructured knowledge sentences from external knowledge bases, e.g., descriptive sentences from Wikipedia. Then they fuse all these retrieved knowledge sentences with expressions to locate the referent. However, ECIFA still has two limitations. Firstly, the sentence-level similarity method exhibits a similarity bias problem (Bogatu et al., 2022). This means that although the retrieved knowledge sentences are lexically similar to the query expression, they may not be the intended knowledge for understanding the expres-
8745
sion. Consequently, the irrelevant knowledge may result in an incorrect localization of the referent due to error propagation. As shown in Fig. 1(a), the intended knowledge for this expression is about pillows, while existing methods retrieve knowledge all about sofas due to sentence similarity with the expression. As a result, the lack of knowledge about pillows leads to localizing the incorrect object, the sofa. Secondly, the retrieved unstructured knowledge sentences may contain a large amount of information that is unrelated to the referent, leading to interference in object localization. As shown in Fig. 1(a), the irrelevant information “on the floor” in the second knowledge sentence may mislead the model to focus on objects located on the ground, rather than the intended focus of “on the sofa” in the expression, resulting in an incorrect localization of the sofa on the floor. Even the irrelevant information “on the bed” in the ground-truth knowledge sentence may potentially mislead the model to localize the incorrect object on the bed.
Based on statistical analysis, we find that most knowledge-based referring expressions can be divided into two segments according to the information contained: (1) Visual segments (e.g., “on the sofa” in Fig. 1(b)), which can be interpreted based on visual content, such as color, shape, and relative position of objects; (2) Knowledge segments (e.g., “used for sleeping” in Fig. 1(b)), which require additional knowledge beyond the visual content to be understood, such as function and non-visual object attributes. Distinguishing these two types of segments and discarding the visual segment during knowledge retrieval can help to solve the similarity bias problem. Moreover, for grounding, it is only necessary to know the category of objects corresponding to the knowledge segment, and detailed descriptive knowledge about the object is not required. Therefore, we employ a category-oriented method to retrieve knowledge categories and fuse them with the visual segment for object localization, which can avoid irrelevant information from knowledge sentences. For example, in Fig. 1(b), the knowledge segment can identify which object categories are used for sleeping and narrows the target to pillows and sofas. Then, these categories associated with the visual segment distinguish multiple instances of pillows and accurately locates the referent one on the sofa.
In this paper, we propose a segment-level and category-oriented network (SLCO), which utilizes
knowledge segments to retrieve knowledge categories and delegate them to visual segments for grounding target objects. It consists of three modules: a segment detection module, a prompt-based retrieval module, and a category-based grounding module. Firstly, the segment detection module identifies visual and knowledge segments. Then, inspired by the excellent knowledge retrieval ability of prompt learning (Shin et al., 2020; Zhong et al., 2021), we present a prompt-based retrieval module that uses knowledge segments as hints to elicit knowledge categories from generic language models. Finally, the category-based grounding module associates the retrieved knowledge categories with visual segments for target object localization.
The contributions can be summarized as follows:
• We propose a segment-level and prompt-based retrieval method that can retrieve object categories corresponding to knowledge segments, thereby addressing the similarity bias problem and reducing incorrect knowledge retrieval.
• We propose a category-based method to associate knowledge categories with visual segments for object localization, thereby alleviating the interference from irrelevant information in knowledge sentences.
• Experimental results on the KB-Ref dataset show that our SLCO can eliminate interference and improve the overall performance.

heading: 2 Related Work

heading: 2.1 Referring Expression Comprehension
text: REC is a fundamental task in the multimodal field. Existing methods are twofold based on the alignment pattern used. Two-stage methods (Wang et al., 2019; Yu et al., 2018) involve an initial stage for detecting boxes, followed by a second stage for ranking these boxes based on an expression. In contrast, one-stage methods (Yang et al., 2020; Huang et al., 2021; Li et al., 2021; Deng et al., 2021; Yang et al., 2022) integrate both visual and textual features to directly regress the bounding box.
To extend this task to a broader knowledge domain, (Wang et al., 2020) introduced a knowledgebased REC task with a benchmark dataset and a two-stage model. This model retrieves knowledge by sentence similarity and fuses it with expression and image for object ranking and selection. However, this model has the problems of similarity bias
and interference from knowledge sentences. To solve these problems, we try to identify parts of expressions that require knowledge and retrieve it in a segment-level and category-oriented manner.

heading: 2.2 Knowledge Retrieval
text: Early attempts in knowledge retrieval primarily relied on similarity measures and matching methods. However, these methods lack the flexibility to handle the variability of natural language. Recently, prompt-based methods (Petroni et al., 2019; Liu et al., 2021) have been shown to possess superior knowledge retrieval capabilities. Many studies (Shin et al., 2020; Qin and Eisner, 2021; Zhong et al., 2021) have focused on prompt engineering to identify effective templates for knowledge retrieval. However, existing methods typically assume that the subject/object and relation are known, and the task is to identify the corresponding object/subject. Nevertheless, in real-world scenes, it is more valuable to analyze which parts of a sentence require knowledge, rather than solely relying on the prespecified subject/object and relation. In this paper, we propose a method for detecting parts of sentences that require knowledge and automatically generating prompts for knowledge retrieval.

heading: 3 Proposed Method
text: An illustration in Fig. 2, SLCO contains three main modules: (1) A segment detection module, which identifies knowledge segments and visual segments
in an expression; (2) A prompt-based retrieval module, which employs knowledge segments as hints to elicit knowledge category from language models; and (3) A category-based grounding module, which associates knowledge category and visual segments for object localization.

heading: 3.1 Segment Detection
text: The main idea of this module is to identify parts of expressions that cannot be inferred solely from images as knowledge segments, and parts with corresponding visual features as visual segments. Given a referring expression x = {x1, ..., xn} comprising n tokens, a knowledge segment can be defined as a subset of the expression xkn ⊆ x, which consists of tokens associated with external knowledge.
Given an image and a referring expression, we first encode visual features v with a convolutional backbone and encode textual features x using a pretrained language model. Due to the difficulties in aligning long expressions with visual features all at once, inspired by sentence decomposition (Yang et al., 2020; Li et al., 2021), we perform a subsentence generation to break the expressions into subsentences by T iterations. The features of subsentence xsub(t) at the t-th iteration are:
xsub(t) = ssub(t) · x, (1)
ssub(t) = Conv(vsub(t−1) · x · ssub(t−1)), (2) where ssub(t) is a score to determine the position of subsentence, with a matrix of ones as an initial
value. In iteration, we use a FiLM method (Perez et al., 2018) for feature projection and obtain visual features of the subsentences vsub(t), as follows:
vsub(t) = ReLU(vsub(t−1) ⊙ Linear(xsub(t)) +Linear(xsub(t))),
(3)
where ⊙ represents element-wise multiplication. To determine which parts of expressions have corresponding visual features, we present a segment activation method that attends visual features to textual features, in contrast to the text-to-image alignment scheme of most REC models. Concretely, we first concatenate the results of T subsentence features to obtain xsub and the visual features of T subsentences to obtain vsub, respectively. Then, to identify activated parts of expressions, we use cross-modal attention to attend the visual features vsub to the textual features xsub. The visiondependent subsentence features x′sub are: x′sub = σ(xsub · v⊤sub) · vsub, (4) where σ represents a softmax function. Finally, we project these features into two scores via MLP: one representing the parts of the sentence activated by visual context and the other representing the unactivated parts. We then obtain visual segment xvi and knowledge segment xkn by multiplying the sentence features with these scores, as follows:
xkn = σ(Linear(x ′ sub))⊙ x+ x, (5)
and xvi = 1− xkn. During the early stages of model training, the visual features and textual features obtained by single-modal encoders are relatively independent, which makes it challenging to align their representations. To tackle this issue, we introduce a pseudosupervision strategy to supervise the detection process. In particular, to generate pseudo-annotations,
we extract common substrings between expressions and their corresponding reference knowledge, as well as extract knowledge guide words (e.g., “used for” and “made up of”) along with the following words. Subsequently, the extracted knowledgerelated words are merged and converted into tokenslevel scores, where 1 indicates knowledge segments and 0 indicates visual segments. Finally, we use a mean squared error (MSE) loss Lseg to reduce the discrepancy between pseudo-annotations and the predicted scores of knowledge segments.

heading: 3.2 Prompt-Based Retrieval
text: After the detection of knowledge segments, this module conducts segment-level knowledge retrieval of object categories to which the intended knowledge in expressions pertains.
Firstly, we use words in the expression that have a higher score than the median of the knowledge segment scores to regenerate knowledge segments for prompts. Considering that longer knowledge segments are primarily descriptive statements, while shorter segments pertain to similar objects or synonyms, we devise two prompt templates for these scenes. They are in the forms of “A is xkn” and “ is a kind of xkn”, where xkn is a knowledge segment in the input slot. When given a prompt, a pretrained language model fPLM predicts the probability of different tokens z ∈ Z that could potentially fill the answer slot. These predictions are then filtered through labels of objects in an image to narrow down potential answers. The top-M highest-scoring tokens ẑ are:
ẑ = argmax z∈Z
P (fPLM (z)|xprompt). (6)
To enhance the category-oriented retrieval ability of this module, we expand the pretrained language
model’s knowledge by incorporating the knowledge bases on which KB-Ref is based. Specifically, inspired by entity-level masking (Sun et al., 2019), we mask knowledge categories in knowledge sentences for fine-tuning the language model.

heading: 3.3 Category-Based Grounding
text: In this module, we associate the retrieved knowledge categories with the detected visual segments for visual grounding. According to (Akula et al., 2020), the position of knowledge categories in a sentence may significantly impact its meaning. Thus, we consider three association forms that can accommodate most situations. In particular, features of each candidate knowledge category xz(m) are integrated into the beginning, middle, and end of a visual segment to obtain ybeg(m), ymid(m), and yend(m), respectively. Then, we perform category association to concatenate and linear project these features into category-associated sentence features ybme(m) for the m-th knowledge category.
During iteration, features of top-ranked categoryassociated sentences with high probability value tend to largely accumulate, facilitating the model to learn important category information. Therefore, we present a knowledge accumulation method to iteratively incorporate textual information of M category-associated sentences into visual features using multi-head attention, as shown in Fig. 3. Specifically, there are two parallel branches. At each step m of the iteration, the first branch fuses the m-th category-associated sentences ybme(m) with the (m− 1)-th visual features u1(m−1) to obtain u1(m), as follows:
u1(m) = σ( u1(m−1) · y⊤bme(m)√
dy ) · ybme(m), (7)
where dy is the dimension of ybme(m). Additionally, the u1(m) is used to compute the weight of each sentence concerning the visual features. We follow the calculation of the verification score in (Yang et al., 2022) to compute this weight. After M iterations, the weights are summed element-wise to obtain category-activated weight svi. The second branch also fuses ybme(m) with the (m− 1)-th visual features u2(m−1) to obtain u2(m) in each step of iteration. After M iterations, these features u2(m) are concatenated and projected via MLP, then added to the original visual features, obtaining u′2. Finally, the category and segment-related visual features vkn are obtained as follows:
vkn = σ( u′2 · u′⊤2√
du ) · v ⊙ svi + v, (8)
where du is the dimension of u′2. After activating objects on visual features with knowledge categories and visual segments, we employ a variant of the Transformer decoder (Vaswani et al., 2017) to further distinguish multiple instances of similar objects. It comprises 6 layers of multi-head attention and point-wise fully connected. In each layer, two self-attentions are replaced with cross-modal attention. In one instance, visual features serve as query, text features serve as key and value, and the other is reversed. The resulting features are then projected into fourdimensional object coordinates via MLP.

heading: 3.4 Training Objective
text: The proposed SLCO is trained by a joint loss containing an MSE loss Lseg and a diversity loss (Yang et al., 2020) Ldiv for segment detection, as well as a smooth L1 loss Ll1 and a GIoU loss (Rezatofighi et al., 2019) Lgiou for grounding, as follows:
L = λsegLseg + λdivLdiv + λl1Ll1 + λgiouLgiou, (9) where λ are trade-off factors.

heading: 4 Experiment

heading: 4.1 Experimental Setup
text: Dataset. KB-Ref (Wang et al., 2020) is the first and currently the only dataset for the KB-REC task. It includes 43,284 knowledge-based referring expressions for objects in 16,917 images from Visual Genome (Krishna et al., 2017). The knowledge involved is derived from Wikipedia, ConceptNet, and WebChild, which are reformed into unstructured sentences. We follow the official data splits. Evaluation Metrics. Following (Wang et al., 2020), accuracy is an average of the number of predictions with IoU greater than 0.50. Given the absence of ground-truth boxes in practical applications, the model inputs available for experiments are images and expressions, without ground-truth boxes. For knowledge retrieval, accuracy is an average of the number of predictions that the groundtruth knowledge category is within the top-M results, i.e., Acc@M . We obtain the ground-truth category corresponding to the ground-truth knowledge from the KB-Ref dataset.
Implementation. During training, we finetune the knowledge retrieval module on a 2080Ti GPU and then end-to-end optimize the remaining modules on two P100 GPUs. The height and width of the input image are resized to 640 and the max length of the expression is set to 40. We use a ResNet-101 (He et al., 2016) initialized with weights from DETR (Carion et al., 2020) as the visual backbone, and BERT (Devlin et al., 2019) as the language model. We then follow the preprocessing of (Yang et al., 2022). For training, we use the AdamW optimizer to train SLCO with a batch size of 28 and a total of 90 epochs. The initial learning rate for feature encoders is set to 10−5, and the other modules to 10−4. For the first 10 epochs, we freeze the weights of feature encoders.
In the prompt-based retrieval module, we use a LAMA framework (Petroni et al., 2019) and an uncased BERT large model as fPLM based on (Shin et al., 2020) which suggests its effectiveness for knowledge retrieval among different PLMs. We employ a Faster R-CNN (Ren et al., 2017) pretrained on Visual Genome (Krishna et al., 2017) to identify object labels in an image. The number of retrieved knowledge M is set to 3.
We follow (Yang et al., 2022) to set smooth L1 and GIoU loss for object localization. The tradeoff between these loss parameters has been tuned to be 5:2. Additionally, we perform a grid search and find the optimal parameters 10 and 0.125 for the MSE loss and diversity loss in the segment detection module. Accordingly, we set λseg, λdiv, λl1, λgiou as 10, 0.125, 5, and 2 in Eq.(9).
Baseline Models. With regards to two-stage methods, LGRANs (Wang et al., 2019) uses a graph-
based attention method to infer inter-object relationships. MAttNet (Yu et al., 2018) employs three modules to handle the grounding of object appearance, location, and relationship. ECIFA (Wang et al., 2020) retrieves knowledge by cosine similarity between expressions and knowledge sentences, and then uses a stack of LSTM to fuse all the knowledge sentences with expressions. In regard to onestage methods, LBYLNet (Huang et al., 2021) uses a landmark convolution method to encode object features. ReSC (Yang et al., 2020) utilizes a recursive framework to fuse visual and textual features. Based on it, BBA (Li et al., 2021) employs a bottom-up and bidirectional framework to align multimodal features. TransVG (Deng et al., 2021) constructs a Transformer-based grounding framework. VLTVG (Yang et al., 2022) further extracts text-conditioned discriminative visual features.
Models for the ordinary REC task lack mechanisms to acquire external knowledge and interact with multimodal information. Therefore, following (Wang et al., 2020), we train all models using their default implementations in the ordinary REC training manner on the KB-Ref dataset.

heading: 4.2 Main Results
text: The results in Table 1 show that ECIFA performs better than other two-stage methods, as it explicitly incorporates external knowledge from multiple knowledge bases. As for one-stage methods, BERTbased models generally outperform LSTM-based ones. The reason lies in that the implicit knowledge from the pretrained language model BERT enhances the comprehension of knowledge-based referring expressions. Moreover, the results show
that our proposed SLCO achieves a performance gain of up to 2.92%, demonstrating the effectiveness of the segment-level and category-oriented strategy. Furthermore, SLCO is the first model that is able to associate both implicit and explicit knowledge from pretrained language models.
Additionally, we conduct an evaluation of the inference time. Our model exhibits an inference time of 0.117 seconds per sample, whereas the baseline model ECIFA necessitates 0.367 seconds.

heading: 4.3 Ablation Study
text: We conduct a series of experiments to verify the effectiveness of three main modules (cf. Table 2). Effectiveness of Segment Detection Module. There is an average decrease of 1.57% when we remove this module and its loss functions. This result validates the effectiveness of our segment-level method, which solves the similarity bias problem in the sentence-level method. Effectiveness of Prompt-Based Retrieval Module. To evaluate the importance of this module, we replace the retrieved knowledge categories with empty strings. The results show that removing this module leads to an average decline of 1.38%, indicating the value of knowledge categories for visual grounding. Moreover, this experimental setup corresponds to using only implicit knowledge, which is similar to the knowledge sources employed by the state-of-the-art one-stage methods in Table 1. Nevertheless, our method has better performance than these methods. Effectiveness of Category-Based Grounding
Module. We ablate this module as well as the prompt-based retrieval module, thus the visual grounding process is performed by visual segments only. Results show that this reduces the model performance by 1.67% on average. The reason lies in that model lacks the ability to associate knowledge categories, making it hard to understand referring expressions and localize the correct referent.

heading: 4.4 Evaluation of Segment Detection Method
text: To explore methods for identifying knowledge and visual segments, we construct a parsing method to compare with the proposed detection method. The parsing method divides the expressions according to the constituency parsing in the syntax. Based on observation, knowledge information mostly appears in predicates or subordinate clauses of expressions. Thus, we take the first half of the parsed sentence as a visual segment and the second half as a knowledge segment.
In Table 3, it can be observed that the parsing method underperforms in both knowledge retrieval and KB-REC. The reason is that the diversity of natural language expressions poses a challenge in identifying knowledge segments based on specific rules, as they may appear in various positions within sentences. Moreover, the results demonstrate the flexibility of our segment detection method in recognizing knowledge and visual information in expressions, which improves the performance of both knowledge retrieval and KB-REC.

heading: 4.5 Evaluation of Knowledge Retrieval Method
text: In block A of Table 4, we compare the proposed method with the cosine similarity method in the baseline model ECIFA (Wang et al., 2020). Results show that replacing the entire expression in the cosine similarity method with the knowledge segments obtained by our segment detection module can significantly improve the performance of knowledge retrieval. Additionally, our promptbased knowledge retrieval method significantly outperforms the sentence-level similarity method, indicating that our segment-level and category-oriented method can effectively alleviate the similarity bias problem. Moreover, we evaluate the inference time of different retrieval methods under the same settings. The results show that our prompt-based method demonstrates high efficiency in retrieving knowledge with 0.020s per sample, which is 70 times faster than the cosine similarity method in ECIFA which takes 1.400s.
Results in block B of Table 4 show that finetuning a language model using category masks improves the model’s capacity to retrieve categories. It contributes to our category-oriented method and alleviates the interference of irrelevant information from unstructured knowledge sentences.

heading: 4.6 Evaluation of Knowledge-Based Grounding Method
text: Block A of Table 5 shows comparison results for associating the textual features of knowledge categories and expressions. It can be seen that associating knowledge categories with visual segments is superior to that with the entire expressions. This is because visual segments concentrate on disambiguation at the instance level and reduce interference from irrelevant parts of expressions.
In block B of Table 5, we evaluate the performance of various methods for associating multiple category-associated sentences and visual features. There are three settings for the association: Multiple sentences and visual features are processed separately by multiple attention mechanisms, and then their results are (1) concatenated or (2) added together; and (3) the attention mechanism is iteratively applied to multiple sentences. The results indicate that the iterative method is the most effective, as it accumulates features from the top-1 category-associated sentence, which is more likely to contain accurate knowledge. In contrast, the concatenation method and the addition method treat all sentences equally, making it difficult to determine which sentences are more important.
The results in Fig. 4 show that the association of the top-3 knowledge categories with expressions achieves the best performance. As the number of knowledge decreases below three, the overall accuracy of knowledge retrieval diminishes, resulting in a degradation of grounding performance. Conversely, when the number of knowledge exceeds three, an excessive number of candidate knowledge categories may impede the model’s ability to accurately associate knowledge for object localization.

heading: 4.7 Qualitative Results
text: As shown in Fig. 5(a) and Fig. 5(b), the baseline model is interfered with by the words “stove” and “desk” in expressions, leading to incorrect results of knowledge retrieval and object localization. In contrast, SLCO effectively avoids this problem by utilizing knowledge segments to retrieve knowledge categories. As shown in Fig. 5(a), SLCO accurately retrieves relevant knowledge categories based on the knowledge segments, and then activates multiple objects related to the retrieved categories in the visual features shown in Fig. 5(c). Then, in
Fig. 5(d), the decoder further refines the objects by visual segments “above the stove” and “in front of the man”, thereby accurately localizing the referent. Qualitative results show that SLCO can solve the issues of similarity bias and interference by irrelevant information in knowledge sentences.

heading: 5 Conclusion
text: In this paper, we propose a segment-level and category-oriented network to endow the model with the ability to identify and utilize the knowledge and visual segments in a targeted manner. Specifically, the proposed method uses knowledge segments to retrieve knowledge, which addresses the similarity bias problem in the sentence-level method. Additionally, our category-oriented retrieval method can elicit knowledge categories from language models, mitigating the interference from irrelevant information in knowledge sentences. Experimental results demonstrate the effectiveness of the proposed method in addressing two limitations of the existing methods, thus improving the accuracy of the KB-REC task. In future work, we will explore more fine-grained information in expressions and combine it with knowledge and visual content.
Limitations
To better understand the limitations of the proposed method, we conducted an error analysis by randomly selecting 100 incorrect predictions and categorizing their error types. The results revealed that 32% of errors were caused by grounding issues,
specifically, an inability to distinguish between multiple objects of the same category, despite having knowledge category of the referent object. The results indicate that there is a need for improvement in the ability to discriminate visual objects, especially for object categories with long-tailed distributions. Additionally, the results show that 20% of errors are due to imprecise object detection, particularly for small objects. This highlights the need for optimization of the visual encoder and loss function. Moreover, 14% of errors are attributed to incorrect knowledge retrieval. To address this, incorporating more fine-grained information in expressions for retrieval should be considered as a future research direction. Furthermore, 34% of incorrect predictions can be attributed to issues with the ground-truth annotations, which may negatively impact the model’s learning process.

heading: Acknowledgments
text: This work is supported by the National Natural Science Foundation of China (62076100, 61976094, and 62276072), the Guangxi Natural Science Foundation (No. 2022GXNSFAA035627), Fundamental Research Funds for the Central Universities, SCUT (x2rjD2220050), the Science and Technology Planning Project of Guangdong Province (2020B0101100002), CAAI-Huawei MindSpore Open Fund, CCF-Zhipu AI Large Model Fund, and the Open Research Fund of Guangxi Key Laboratory of Multimedia Communications and Network Technology.

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: After the conclusion in Section 5 and before the references.
7 A2. Did you discuss any potential risks of your work? The task in this paper does not identify potential risks and harmful effects for the time being.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: The introduction in Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? The method in Section 3 and the experiments in Section 4.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: The method in Section 3 and the experiments in Section 4.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: In the abstract, we claim that our code will be made available to the public.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? The implementation in Section 4.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We do not propose a new dataset.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? We do not have documentation of the artifacts.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. The dataset in Section 4.
C 3 Did you run computational experiments? The implementation in Section 4.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? The implementation in Section 4.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? The implementation in Section 4.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? The experiments in Section 4 and the limitations after the conclusion.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? The implementation in Section 4.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In future work, we will explore more fine-grained information in expressions and combine it with knowledge and visual content. Limitations
To better understand the limitations of the proposed method, we conducted an error analysis by randomly selecting 100 incorrect predictions and categorizing their error types. The results revealed that 32% of errors were caused by grounding issues,
specifically, an inability to distinguish between multiple objects of the same category, despite having knowledge category of the referent object. The results indicate that there is a need for improvement in the ability to discriminate visual objects, especially for object categories with long-tailed distributions. Additionally, the results show that 20% of errors are due to imprecise object detection, particularly for small objects. This highlights the need for optimization of the visual encoder and loss function. Moreover, 14% of errors are attributed to incorrect knowledge retrieval. To address this, incorporating more fine-grained information in expressions for retrieval should be considered as a future research direction. Furthermore, 34% of incorrect predictions can be attributed to issues with the ground-truth annotations, which may negatively impact the model’s learning process