CLEVR-Implicit: A Diagnostic Dataset for Implicit Reasoning in Referring Expression Comprehension
Authors: Jingwei Zhang
Xin Wu
Yi Cai
Venue: EMNLP 2023 Main (2023)
Keywords: referring expression comprehension
implicit reasoning
prompt tuning

Abstract: Recently, pre-trained vision-language (VL) models have achieved remarkable success in various cross-modal tasks, including referring expression comprehension (REC). These models are pre-trained on the large-scale image-text pairs to learn the alignment between words in textual descriptions and objects in the corresponding images and then fine-tuned on downstream tasks. However, the performance of VL models is hindered when dealing with implicit text, which describes objects through comparisons between two or more objects rather than explicitly mentioning them. This is because the models struggle to align the implicit text with the objects in the images. To address the challenge, we introduce CLEVR-Implicit, a dataset consisting of synthetic images and corresponding two types of implicit text for the REC task. Additionally, to enhance the performance of VL models on implicit text, we propose a method called Transforming Implicit text into Explicit text (TIE), which enables VL models to reason with the implicit text. TIE consists of two modules: (1) the prompt design module builds prompts for implicit text by adding masked tokens, and (2) the cloze procedure module fine-tunes the prompts by utilizing masked language modeling (MLM) to predict the explicit words with the implicit prompts. Experimental results on our dataset demonstrate a significant improvement of 37.94\% in the performance of VL models on implicit text after employing our TIE method.

CLEVR-Implicit: A Diagnostic Dataset for Implicit Reasoning in Referring Expression Comprehension
Jingwei Zhang' Xin Wu',; Yi Cai
'School Software Engineering, South China University of Technology 2 Keyy Laboratory of Big Data and Intelligent Robot (South China University of Technology) Ministry of Education ycai@scut, edu. cn
Abstract
Recently, pre-trained vision-language (VI.) models have achieved remarkable success in various cross-moot tasks, including referring expression comprehension (REC). These mod- els are pre-trained on the large-scale image-text pairs to learn the alignment between words in textual descriptions and objects in the corre- sponding images and then fine-tuned on down- stream tasks. However, the performance of VI. models is hindered when dealing with implicit text, which describes objects through compar- isons between
left of only two objects the same color and material
objects rather than explicitly mentioning them. This is because the models struggle to align the implicit text with the objects in the images. To address the chal- lenge, we introduce CLEVR-Iman Imppiic,, dataset consisting of synthetic images and correspond- ing two types of implicit text for the REC task. Additionally, to enhance the performance models implicit text, we propose method called Transform Implicit text into Explicit text (TIE), which enables VI. models to process with the implicit text. TIE consists of two modules: (1) the prompt design module builds prompts for implicit text by adding masked tokens, and (2) the cloze procedure module fine-tunes the prompts by utilizing masked lan- guage modeling (MLM) predict the explicit words with the implicit prompts. Experimental results on our dataset demonstrate significant improvement of 37.95 94% in the performance of VL models on implicit text after employing our TIE method.
The right one the only two objects with different shape and material
Figure I: Examples from CLEVR- There are two types of implicit text: Same and Different.
REC involves identifying a specific object in an im- age based on corresponding textual descriptions. A common approach is to pre-train vision-language model large-scale image-text pairs to leam the alignment between words in textual descriptions and objects in the images. Fine-tuning then per- formed to optimize model performance (Lu et al., 2019; Su et al., 2019; Kamath et al., 2021).
Despite the state-of-the-art results demonstrated by these methods, they still exhibit limitations. Pre- trained models are good at processing explicit text, which describes the appearance attributes of the objects in the image, whether they are reference target objects. For example, consider the phrase "TThe object left on the red cube.". models rely on the explicit word "red cube" locate the referred object the image. However, they encounter chal- lenges when it comes to implicit reasoning, which is processing the implicit text in REC. Implicit
1 Introduction
In recent years, pre-trained vision-language models have demonstrated remarkable achievements in var- ious vasion-language downstream tasks (Lu al., 2019; Su et al., 2019; Kim et al., 2021; Yao et al., 2021: : Kamath et al., 2021; Liu et al., 2022), in- cluding referring expression comprehension (REC).
"Equal contribution.
Conresponding author.

---

reasoning occurs when the textual expression in- directly describes an object through a comparison between two or more objects, one type of implicit text is formatted like "The left one of two same color objects". The phrase "two same color ob- jects" does not describe the appearance attributes of any objects, only the relationships between them, and there is no direct reference for the models to infer the target. Consequently, models struggle to align "two same color objects" with any objects in the image. To evaluate the performance gap between explicit and implicit text, we conducted experiments using existing methods on both types of text separately. Preliminary results indicate that the performance on implicit text is approximately 42% lower compared to explicit text.
Table I: The attributes and example values the objects in CLEVR. We build the CLEVR-Implicit based these objects.
Attribute color material shape
Example values
red, wwl.ue.caa...urll..rrr.....
rebber
cube.cyllnneersphere
size
large,small
spatial
left.right
Implicit. Experimental results demonstrate a sub- stantial difference in the performance of REC mod- cls on our dataset before and after applying TIE. We summarize our contributions as follows:
We analyze the limitations of existing refer- ring expression comprehension models on the implicit text and construct the CLEVR- Implicit dataset.
Besides, the implicit text also plays significant role in the domain of human-comme human-computer interaction. However, existing referring expression datasets, such as RefCOCO (Yu et al., 2016), RefCOCO+ (Yu et al., 2016), RefCOCOg (Mao et al.. 2016), and CLEVR-Ref+ (Liu et al., 2019), primarily con- sist of explicit text. Unfortunately, there is currently dataset available that is specifically designed to provide research data on implicit text.
We propose the TIE method to enable pre- trained REC models process the implicit text by transforming the implicit text to ex- plicit text.
We evaluate the existing models on the CLEVR-Implici and analyze the differences between the models before and after applying TIE.
To address the lack of research data and methods for implicit text, we propose a synthetic dataset called CLEVR-Implicit, specifically for implicit reasoning in REC. Unlike existing datasets, the im- plicit text we constructed is a subset of all types of implicit text, since they describe synthetic im- ages, and do not need consider external knowl- edge. Specifically, We focus on the more difficult types (only containing relationships between ob- jects without any description of their appearance attributes). We evaluate the current pre-trained REC models on our CLEVR-Implicit dataset with these types of implicit text.
2
Related Works
2.1
Referring Expression Comprehension
Referring expression comprehension (REC) aims to localize the region in an image that corresponds to a given sentence description. In recent years, there has been significant increase in research attention towards this task, driven by the increas- ing applications of REC in the domain of human- computer interaction. The related task of refer- ring expression generation (REG) has been studied for many years (Krahmer and Van Deemter, 2012: Mitchell et al., 2013; 3; Reiter and Dale, 1992) in the natural language processing (NLP) domain.
In real-world scenarios, most humans will first extract explicit information from visual context and text information when processing these types of implicit text, and then reason the target object from the extracted explicit information. Inspired by the thinking mode of humans, we simulate the process of humans reasoning about the implicit text and introduce a method called TIE (Transform the Implicit text into Explicit text), which enables pre-trained models to process the implicit text by extracting the explicit information. By employing TIE, REC models facilitate alignment between ex- plicit words and corresponding images on CLEVR-
Several real-world datasets, such as RefCOCO, RefCOCO+, and RefCOCOg, have been proposed based the MSCOCO dataset (Vinyals al., 2016) for this task. CLEVR-Ref+ (Liu et al., 2019) is synthetic dataset that contains geometric ob- jects. These datasets have been widely adopted as benchmarks for evaluating the performance re- ferring expression models. MMI (Mao et al., 2016)

---

directly predicted the bounding box by ranking the region proposals. SLR (Yu et al., 2017) utilized a joint embedding model to predict the referred object by learning the visual and language repre- sentations in an embedding space. MattNet (Yu et al., 2018) adapted the attention mechanism to parse expressions into three modules and predict the visual regions by weighted combination of module scores. VL.-BERT (Su et al., 2019) adopted a Transformer model and concatenates the visual and language embeddings as inputs. MDETR (Ka- math et al., 2021) constructed an end-to-end system based on the DETR (Carion et al., 2020) detector and a language model to learn the alignment be- tween objects and phrases. VLTVG (Yang et al., 2022) improves comprehension ability by utilizing risual-linguisse verification module to generate discriminative feature representations.
Table 2: The statistic data of CLEVR-Implictt.
Attribute num
Same
Different
3000
2
3000
1553
3
3000
3000
4
2905
3000
which involves converting the implicit text within the CLEVR-Implicit dataset into explicit text. This explicit text is then utilized the referring expres- sion comprehension task through the fine-tuning process of pre-trained vision-language models.
3.1 Dataset Construction
3.1.1 CLEVR-Implicit Overview
CLEVR-Implicit Dataset uses subset of the CLEVR (Johnson et al., 2017) scenes (10K im- ages for all, 60% for the train set and 40% for the validation and test set). Table 1 provides an overview of the attributes and values associated with the objects in CLEVR. In order to enhance the difficulty of the dataset, filter and choose the images that contain CLEVR-Rett provides scene graphs about the at- tributes (such as color, material, shape, and size) of objects, as well as the spatial relationships between them within the image, we generate corresponding referring expressions for each image. A total of 19K referring expressions are generated, averaging 1.9 referring expressions for each image. Finally, the bounding box of the referred object is assigned as the label for each referring expression in the CLEVR Impliiit Dataset.
2.2 Prompt Learning
Prompt learning, initially introduced in the field NLP, is used to reformulate downstream tasks to the form of the pre-training task masked lan- guage modeling (MLM) (Devlin et al., 2018). This approach enables models to effectively utilize the knowledge during the pre-training stuge (Liu et al., 2023). Concretely, prompt engineering reformu- lates the downstream task into "fill-n-tthebllan cloze task with the same objective function as the MLM task. The effectiveness of prompt learning heavily relies on the design of appropriate prompt templates. Even slight variations in the prompt template can yield sigmificant differences in perfor- mance. Therefore, the selection of suitable prompt templates becomes crucial for achieving desirable results.
more objects. Given that
3.1.2 Implicit Text Generation
Prompt learning has also been explored in the multi-modal domain through various studies. CLIP (Radford et al., 2021) pre-trained on large-scale image-text pairs and designs manual prompts like "The photo of [MASKJ" to facilitate image classifi- cation by predicting the masked token. DPT (Liu al., 2022) reformulated the visual question an- swering (VQA) task the MLM by converting the input (question) to a declaration with the masked token.
To generate implicit text that describes objects through a comparison between two or more objects, utilize the scene graphs provided by CLEVR- Ref+. A program is employed to extract the at- tributes of all the objects present in the images. This allows for the comparison and analysis of their attributes, enabling the generation of implicit text that captures the relationships between objects without explicitly mentioning any specific object in the image.
For an image, we compare the attributes of each two objects and find each set of objects that sat- isñiss the requirement for generating implicit text. In CLEVR-Implici,, we set up two types of im- plicit text: "same" and "different". The "same"
3 Methodology
This section begins by explaining the construction process of the -EEVR-Implici dataset. Subse- quently, we introduce our proposed TIE method,

---

The ehjectooth
Encwder-Decoder
[MAS]] (AUASK] makrib
Visinn-Languuge Transfrmmrr
The lult oee stjicts with green oderr (MASS]
The lebt of abjict with wome groms nder (NUNE]
The lutt oft abjicts with cri odar
The efw abjicta oyn
Implicit Explicit
Expecsioo Cumprehendin
The framework of our TIE method and subsequent REC process. The implicit text in CLEVR-Implici is converted to an implicit prompt which contains a masked token before the attribute. Then vision- saiiunla model predicts the explicit words by Multi-step and Path-select prediction strategies in the MLM task. After TIE, the output explicit text and the image are fed to referring expression model to predict the result.
Figure 2:
type represents that the text refers to the object by describing the of certain attributes between objects
prioritize maximizing the number of attributes in- volved in attribute comparisons. For example, if the target is in group of objects that share the same color only, we must ensure no other group of objects that can satisfy the same color and matcrial. In cases where such a group of objects exists, we will change the target and modify the implicit text to describe the corresponding number of attributes until no additional attributes can be included. More details of the implicit text generation are shown in Appendix A.
objects (e.g. "The left of the only with the same color and material"), while the "dif- ferent" type is the opposite of "same" (c.g., "The left object that is different shape and size from the other one). Table 2 provides statistics regarding the distribution of "same" and "different" type samples in the CLEVR-Implicit dataset.
It is worth mentioning that the "same" type of referring expressions also exist in the CLEVR- Ref+ dataset. The key difference in "same" type text between CLEVR-Implicit and CLEVR-Ref+ whether the explicit words are hidden. The "same" type of text in CLEVR-RE Ref+ is like "The objects that have the same size as the red sphere", which contains the explicit word "red sphere". The ex- plicit context provides REC models with refer- ence object and helps them locate the target. Oppo- sitely, the "same" type of implicit text in CLEVR- Implicit is formatted like "The two objects with the same size". which hides explicit words. Without explicit context and hard to find the reference ob- ject, REC models are forced to compare every two objects to identify what the "size" is and whether they are the same. That illustrates that implicit text in CLEVR- Implicit is more challenging.
The implicit text in CLEVR-Implicit follows standardized format based on several templates and rules. Especially, it is common for the same sentence to describe different target objects across different images. By employing homogeneous text, we purposefully direct the model's focus toward the image itself, thereby effectively testing the model's reasoning ability.
3.2 TIE Method
3.2.1 TIE Overview
To address the challenge of implicit text compre- hension, we propose a method called Transform Implicit text to Explicit text (TIE). The objective of TIE is to incorporate explicit words into the original implicit text, enabling the model to rea- son more effectively. Our approach consists of two key modules: (1) The prompt design module builds prompts for implicit text by adding masked tokens. By strategically designing the prompts, we prepare the groundwork for the next module. (2)
We pay more attention to the reference sin- gle object the context of REC, During the gen- eration process, we take precautions to avoid text ambiguity, ensuring that the implicit text describes only one object in the image. To achieve this, we

---

The cloze procedure module uses a pre-trained vision-language model to predict the explicit words with the implicit prompts.
within the masked tokens. And we will introduce three prediction strategies: Base, Multi-step. and Path-select.
By leveraging these two modules, TIE enables the integration of explicit words into the implicit text and facilitates indirect reasoning on the im- plicit text effectively.
Base In the prediction stage, the prediction of multiple attribute values in a sample is different from the prediction of a single attribute value. For implicit text containing multiple attributes, a straightforward prediction strategy is to predict all attribute values. However, this strategy tends to de- crease the correlation among the predicted values, resulting in potential discrepancies where multiple predicted values may not refer to the same group of objects. Consequently, incorrect prediction results may arise.
3.2.2 Prompt Design Module
The prompt design module converts the implicit text into format suitable for the Masked Lan- guage Modeling (MLM) task. One approach to adding explicit words is by obtaining the specific attribute values involved in the comparison of at- tributes in the implicit text (e.g.. "What is the color in 'The left one of the only two objects with the same color'?"). Consequently, the prompt design module generates the prompt like "The left one of the only two objects with the same [MASK] color" for an implicit text sample. It's important to note that the conversion process differs between the "same" and "different" types of implicit text. In the "same" " type, only one identical attribute value needs to be predicted, while in the "different" type, two different attribute values must be predicted. Therefore, in the "different" type, two masked to- kens are added before each attribute in the implicit text. This process of converting the implicit text can be seen as translation task, where the source text is the original implicit text, and the target text is the input format required for the MLM task. To accomplish this conversion process, we create an implicit prompt dataset using the implicit text from CLEVR- Implicit as the source text, and the target text is text with the masked token preceding cach attribute in the implicit text. Subsequently, we train an encoder-decoder model (T5, Raffel et al., 2020) on the implicit prompt dataset to add the appropri- ate number of masked tokens corresponding to the attributes for each sample in CLEVR- I.pliiim
Multi-step To address the aforementioned issue, we propose an enhanced strategy named multi-step prediction. Specifically, when predicting a sample with multiple attributes, the model outputs only one attribute value at time. Subsequently, it incorpo- rates the previous prediction as input to predict the next attribute value. Compared with the method of predicting all attribute values at one time, this strategy enhances the correlation among multiple attribute values. For instance, if the original text is "The left one of the only two objects with the same [MASK] color and [MASK] material," the corresponding first-step output is adjusted to "The left one of the only two objects with the same red color and [MASK] material." And the second-step output is "The left one of the only two objects with the same red color and metal material."
Path-select We find that the Multi-step strategy has a potential challenge where the accuracy of predictions relies on the first attribute. If the first attribute prediction is incorrect, it might trigger a chain reaction Icading to subsequent attributes be- ing errors. To mitigate this issue, we search for some objects that are not the targets as negative samples. They are characterized by partially sat- isfying the constraints of implicit text. Part of the attributes that do not satisfy are marked [NONE]. For instance, for a target sample "The same red color and rubber material" in cloze dataset, the cor- responding negative sample might be "The same blue color and [NONE] material".
3.2.3 Cloze Procedure Module
The training objective of the cloze procedure mod- ule aligns with the MLM task, a commonly em- ployed pre-traiming task in various pre-trained mod- els, so another cloze dataset is devised for the MLM training stage. In this dataset, the input consists of implicit text containing masked tokens and an im- age description, while the target comprises explicit text containing the ground-truth attribute values. In this module, we tunettune a pre-trained vision- language model ViLT to predict a specific word
During the prediction stage, when the model ob- tains the optimal Soflmax result for an attribute and predicts [NONE] for the next attribute, rec- ognizes this as an incorrect predicted path. Sub- sequently, the model reverts to the prediction of the preceding attribute, considering the suboptimal

---

ViLT (Kim et al., 2021)
Same
Different
Total
MLM
val
test
val
test
val
test
Base
80.54
81.97
25.73
23.35
61.99
60.48
Multi-step
92.55
91.42
87.05
85.47
91.45
91.50
Path-select
91.76
91.35
85.58
87.50
91.48
90.37
Table 3: The TIE experiment results on CLEVR-Implict. We use three different prediction strategies to compare the performance of ViLT.
Method
Before
TIE
After
TIE
Ground-trute
val
test
val
test
val
test
MDETR (Kamath al., 2021)
53.57
53.89
91.51
92.19
96.01
95.66
VLTVG (Yang et al., 2022)
58.83
58.46
84.48
86.15
95.97
96.85
Table Referring Expression Comprehension results performed by MDETR and VLTVG. Before TIE means that the models are trained on the implicit text before the input transformation of the TIE method, and After TIE means that the models are trained on the explicit text after TIE.
Softmar result, which guides the model toward the correct prediction.
samples, where attribute values differ between ob- jects, the order of the predicted values for single attribute can be reversed.
4 Experiments
We present the results separately for training on the "same" and "different" types of samples, and the "Total" column represents the combined per- formance on both types of implicit text. In the Base strategy, we observe that "different" samples are more challenging to understand compared to "same" samples, resulting in decrease in perfor- mance of up to 54.81%. %.. For "different" samples, the model needs to output two values to predict single attribute, resulting in twice as many explicit words be predicted compared to "same" samples. In contrast, the Multi-step strategy exhibits signifi- cant improvements for both types of text. It shows a 12.01% % increase in performance for "same" sam- ples and a remarkable 61 .322 increase for "differ- ent" samples. A reasonable explanation is that the prediction of each explicit word takes into account all previously predicted results. Multi-step strategy mitigates cases where the initial predictions are cor- rect, but subsequent predictions suffer from errors. leading to more accurate overall predictions.
4.1
Implemention Details
For the REC task on the CLEVR-Implicit dataset, we employ two models: MDETR (Kamath et al., 2021) and VLTVG (Yang et al., 2022). MDETR various datasets
utilizes weights pre-trained such COCO (RefCOCO, RefCOCO+, Ref- COCOg), Visual Genome (Krishna al., 2017), and Flickr30k (Plummer et al., 2015), and VLTVG uses weights pre-trained on RefCOCOg. And we fine-tune them on the CLEVR-Implicit dataset. During the training stage, we use the settings of learning rate le-4, learning rate of backbone le-4, batch size 16 and weight decay lc-5 to train for 60 epochs. In the input transformation experiment, utilize the T5-small model for the prompt design module. As for the cloze procedure module, we select ViLT (Kim al., 2021). ensure a fair comparison, all different prediction methods are configured with the AdamW optimizer, a learning rate of Ic-4, a batch size of 32, and an image size of 384x384 to train for 70 epochs.
From the comparing results of the Path-select and Multi-step strategies, we observe some differ- ences when incorporating negative samples into the training set. The Path-select results on the val- idation set exhibit slight decrease of 0.79% and 1.47% compared Multi-step. We speculate that this because the majority of incorrect positions in wrongly predicted samples tend to oceur in the second attribute or later, and the prediction of the first attribute in these samples is generally accurate.
4.2 Results of TIE
Table 3 presents the results of the TIE experiment. The evaluation method is to assess whether the predicted explicit words match the corresponding ground-truth values. For samples containing multi- ple attributes, necessary to ensure the accurate prediction all explicit words. otherwise, it is considered incorrect. the case of "different" type

---

200
595e
Save Mulii stop
Multo stop
Puth-aated
20
20
Number
Mumber
(a) attributes
(b) different attributes
Figure 3: 3(a) is the performance of three prediction strategies on "same" type samples in test set, and 3(b) is the
performance three prediction strategies on "different" type samples in test
However, on the test set of "different" samples, Path-select surpasses Multi-step by 2.03% in terms of performance. Particularly, the performance of Multi-step and Path-select in the "Total" column is comparable, indicating that the addition of negative samples can enhance the model's robustness. And negative samples enable the model to choose the correct path when the first attribute prediction is incorrect.
Method
Train on CLEVR-Ref+
val
test
MDETR 47.02
45.85
VLTVG
47.99
47.11
Table 5: The performances of REC models which are trained on CLEVR-Ref+ and tested on CLEVR- Implicit.
VLTVG on CLEVR- Reff
val
test
4.3 Results of Referring Expression
Before TIE
66.43
65.22
After TIE
68.05
66.46
Table 4 presents the results of the referring expres- sion experiment. We evaluate the performance of MDETR and VLTVG, both before and after ap- plying the TIE based on the Multi-step prediction strategy. Additionally, we provide the performance of the models trained ground-truth explicit sam- ples reference. The evaluation metric is Inter- section over Union (IoU), with a threshold value of 0.5. We observe that prior to TIE, the perfor- mance both models trained on implicit text is relatively poor, indicating limited reasoning ability when dealing with implicit text. However, after applying TIE, there is a significant improvement in performance for both models (53.57% vs. 91.51% for MDETR and 58.83% 84.48% for VLTVG). However, compared with Ground-truth, there is certain gap performance, because limited by the impact of TIE performance.
Table 6: The performances of VLTVG respectively original CLEVR-ReF- (Before TIE) and CLEVR-Ref+ after inference by TIE trained on CLEVR-Implicit.
Ref+. Table shows that the results of MDETR and VLTVG models are all lower than "Before TIE" Column results in Table4. That indicates that the ex- perience that the models learn from CLEVR- cannot help them process the implicit text, because the samples in CLEVR-Ref+ contain explicit infor- mation to help the model locate the target, while the samples in CLEVR-Impllcit hide the informa- tion. Therefore, we believe that it's challenging process the implicit text in our CLEVR- Implicit dataset.
Affect of TIE on explicit text Since the TIE method converts implicit text into explicit text, in addition to implicit text, also need analyze whether the explicit text after applying TIE will affect the performance of REC models. To ver- ify this problem, we evaluate the performance VLTVG respectively on original CLEVR-ReS and
4.4 Results of Contrast Experiment
Difficulty of CmmvicImplictt In order to contex- tualize the difficulty of CLEVR-Implicit, we eval- wate the performance REC models on CLEVR- Implicit when the models are trained on CLEVR-

---

The left of the only objects with the same shape, color and size in the image
sphere, brown, large
The left one same [MASK] shape, [MASK] color [MASK] size the image
objects with the
stlti-step: cylinder, yellow; large
sylinder, yellow, large
GT: cylinder, yellow, large
(a) example of comparison from Base, sulti-step and Path-select prediction strategies
Before:
After:
The left one of the only two objects with the same shape, color and size the image
The left one of the only two objects with the same cylinder shape, yellow color large image
(b) REC before TIE
(c) REC after TIE
Figure 4: Examples from TIE and referring expression comprehension experiments. (a) an example of the comparison from Base, Multi-step and Path-select prediction strategies. O, P and GT denote original implicit text, generated prompt text and ground-truth explicit words. (b) Visualization of the predicted bounding box from VLTVG before TIE and (c) after TIE. green and red rectangles denote ground-truth and predicted bounding box.
CLEVR-Rett Ref+ after inference by TIE trained on CLEVR- CLEVR-Implicit. Table 6 shows that TIE does not affeet performance (even better on VLTVG) on other types of referring expressions besides the implicit text, and TIE robust in a broader class referring expressions.
Method
Train-i
Train-ge
Train-ge Val-pe
Val-pe
Val-i
MDETR
31.48
46.25
91.98
VLTVG
53.85
48.34
85.90
Table 7:
The contrast experiment for REC. The i suffix denotes the set with implicit samples. The suffix denotes that the explicit words in samples are predicted by TIE. The ge denotes the explicit words in samples is ground-truth.
Contrast between explicit and implicit As shown in Table 7, evaluate the performance of MDETR and VLTVG models separately pre- dicted explicit text and implicit text when the mod- els are trained on implicit text and ground-truth explicit text respectively.
attribute samples. Interestingly. the performance on four-attribute samples is second only that of one- attribute samples. A reasonable conjecture is that of objects is identical when all four properties are the same in CLEVR images, so the model only needs to find two objects that are identical. Figure 3(b) shows that predicting four-attribute "different" samples has the most significant challenge on the Base strategy. However, on the Multi-step strat- egy, the performance on four-attribute "different" samples surpasses that of threc-attribute samples. This indicates that the Multi-stey stet prediction strat- egy effectively mitigates the impact of increased number of predicted words, resulting in improved performance.
When both validation sets provide predicted ex- plicit words (Column vs Column 3), the perfor- mance of the model trained on the implicit training set is significantly decreased (MDETR: 31.48% vs 91.98%, VLTVG: 53.85% VS 85.90%) compared with that on the explicit training set, which is evi- dence that explicit words play crucial role in models' comprehension and reasoning processes. The same situation occurs when exchanging im- plicit and explicit states on train and validation sets (Column 2 Column 3).
4.5 Performance on different number of attributes
Figure 3 illustrates the performance of ViLT "same" and "different" samples with different bers of attributes. In Figure 3(a), we observe that the model achieves the highest performance when predicting one-attribute samples, while the lowest performance observed when predicting three-
4.6 Case Study
Figure 4 indicates an example of the comparison of different prediction strategies in the TIE experi- ment, and we visualize the ground-truth bounding box and the bounding box predicted from VLTVG

---

Table Examples about the application of implicit text in daily life.
Real-world Implicit Text Examples
Two persons are doing the same movement. His height different from each other. His pants and shoes are the same color. The same brand of cosmetics on the table. Take the drugs used for the same symptom.
before and after TIE in the REC experiment. In Figure 4(a), with the attributes of the example "shape" and "color", the Base strategy mistakenly predicts "sphere" and "brown". We observe that from the image there group of objects that is both "sphere" and another group of objects that is both "brown", but there are no such two objects that satisfy "sphere" and "brown" simultaneously. This indicates that the Base strategy performs poorly aullii-ttribbte samples due to the weak correlation between attributes, leading the error that differ- ent attributes refer different groups of objects. contrast, the Multi-step and Path-select strate- gies enhance the correlation between attributes and make predictions step-by-step, leading to more ac- curate predictions.
Original Implicit Text: The left one of two persons with the same posture and gender
After Prompt Design Module: The left one of two persons with the same [MASK] posture and [MASK] gender
After Cloze Procedure Module: The left one two persons with the same walking posture and woman gender
Figure 5: A real-world image from Visual Genome
Genome, consider the implicit text like "The left one two persons with the same posture and gen- der". Through the TIE method, the prompt can be built before the "posture" and "gender" by prompt design module, like "The left one of two persons with the same [MASK] posture and [MASK] gen- der". The prompt guides the cloze procedure mod- ule predict the explicit words "walking" and "woman" like "The left one of two persons with the same walking posture and woman gender". By integrating TIE approach, current models can also process the implicit text on the real-world scenar- ios.
In Figure 4(b), by taking the implicit text as in- put, VLTVG incorrectly predicts the referred object as brown cube. While VLTVG predicts the re- ferred object accurately when taking the explicit text after TIE as input. The comparison result indi- cates that current models are being confused when performing implicit reasoning based on the logical relation conveyed by the implicit text. Only when explicit words are explicitly included, the models can align the explicit words with the corresponding objects the image for explicit reasoning.
Conclusion
In this paper, we analyze the significance of im- plicit reasoning. along with the lack of investigation in this domain. To address this, we introduce the CLEVR-Implich dataset for implicit reasoning in the REC task. This dataset utilizes synthesized im- ages to generate two types implicit text: "same" and "different." " Evaluating the performance of ex- isting pre-trained models on CLEVR-Implicit, we observe model confusion when attempting to align implicit text with images. To overcome this chal- lenge, propose the Transform Implicit text to Explicit text (TIE) method, which converts implicit text into explicit text. Our experiments demon- strate that the reasoning capability of the models improves significantly through the alignment explicit words and images.
5 Implicit Text in Real-world Scenarios
The implicit descriptions in real-world scenarios are more complex and difficult because they not only describe the targets implicitly through the ge- ometrical attributes of color, material, etc., but also contain more attributes as well as external factors. Table 8 shows the implicit text examples in real- world daily life. It is important that even though generated implicit text relies the CLEVR images and the provided annotations, our proposed TIE method can still extend the scope beyond the CLEVR domain to real-world scenarios. For exam- ple, based on Figure 5 which is chosen from Visual

---

Limitations
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin John- son, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vi- sion using crowdsourced dense image annotations. International journal of computer vision, 123(1):32-
The limitation of our work is that (LEVR-Implices is built on synthetic images, so it's straightforward to generate corresponding implicit text based on the annotation of the synthetic images. Besides, since the synthetic images contain only geomet- ric objects, our implicit text can only be generated within the scope of the object's attributes. There- fore, in future work, we will annotate more types of implicit text based on real-world scenarios. This will allow us to explore the mechanisms of model understanding of implicit text.
73.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig, 2023. Pre- train, prompt. and prediet: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35.
Runtao Liu, Chenxi Liu, Yutong Bai, and Alan L Yuille. 2019. Clevr-ref+: Diagnosing visual reason- ing with referring expressions. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 4194.
Acknowledgements
This work was supported by the National Natural Science Foundation of China (62076100), Funda- mental Research Funds for the Central Universi- ties, SCUT (x2:)D2230080), the Science and Tech- nology Planning Project of Guangdong Province (2020B0101 100002), CAAI-Huawei MindSpore Open Fund, CCF-Zhipu Al Large Model Fund.
Yuhang Liu, Wei Wei, Daowan Peng, and Feida Zhu. 2022. Declaration-based prompt tuning for visual question answering. arXiv preprint arXiv:225.002556.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguis- tic representations for vision-and aadn--nn--anggae and-language tasks. Ad- vances in neural information processing systems, 32.
References
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. 2016. Generation and comprehension of unambiguous ob- ject descriptions. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 11-20.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In European conference on computer vision, pages 1239.22.2622 Springer.
Jacob Devlin, Ming-We Wei Chang, Kenton Lec, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint 18100080.004000
Margaret Mitchell, Kees Van Deemter, and Ehud Reiter. 2013. Generating expressions that refer visible objects. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 1174-1184 184.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901-2910.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image- to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Car- ion. 2021. Mddtr-modulated detection for end-to-end modal understanding. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1780-1790.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark. et al. 2021. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, pages PMLR.
Wonjae Kim, Bokyung Son, and Iidoo Kim. 2021. Vilt: Vision-and dadion-and-languee language transformer without convolu- tion or region supervision. In International Con- ference Machine Learning, pages 5583-5594, PMLR.
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res,, (1(
Emiel Krahmer and Kees Van Deemter. 2012. Computa- tional generation of referring expressions: survey. Computational Linguistics, (731)1133-111

---

Ehud Reiter and Robert Dale. 1992. A fast algorithm for the generation of referring expressions. In Pro- ceedings of the 14th Conference Computational Linguistics Volume page 232-238. Association for Computational Linguistics.
(2.4), (3.4), the numbers are object ids). We will generate the corresponding implicit text if a certain pair satisfies the "same" or "different" condition. Suppose the (2,3) combination objects' shape and color are both cube and blue, and guarantee that no other pairs satisfy the condition "same shape and color" to avoid the ambiguity in the REC task, then we will generate the "same" type of implicit text as "The two objects with the same shape and color". Due to the only one target of the REC task, we must choose one of the (2,3) objects be the target. We employ a random selection for the target based on the spatial relationship between and 3. If object 2 is left of object 3 and object 2 is selected, the finally generated implicit text is "The left one of the two objects with the same shape and color". The "different" types of text are generated in the same way. This consistent pattern of implicit text generation in the CLEVR- IImpicitt dataset ensures both the concealment of explicit information and the reference of target objects in the images.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019. VI-bert: Pre-training of generic visual- linguisti representations. arXiv preprint arXiv: 1908.08530
Oriol Vinyals. Alexander Toshev, Samy Bengio, and Du- mitru Erhan. 2016. Show and tell: Lessons learned from the 2015 mscoco image captioning challenge. IEEE transactions on pattern analysis and machine intelligence, 3944)655-6663
Yang, Yan Xu, Chunfeng Yuan, Wei Liu, Bing Li. and Weiming Hu. 2022. Improving visual grounding with visual-linguisie verification and iterative soning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pages 9499-9508.
Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. 2021. Cpt: Col- orful prompt tuning for pre-trained vision-language models. arXiv preprint arxiv 11t. 11977.
Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. 2018. Mattnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 1307-1315.
Licheng Yu, Patrick Poirson, Shan Yang. Alexander C Berg, and Tamara Berg. 2016. Modeling context in referring expressions. In European Conference on Computer Vision, pages 69-85. Springer.
Licheng Yu, Hao Tan, Mohit Bansal, and Tamara L Berg. 2017. joint speaker-listenee siatee--iiiteee--rnfforeee reinforcer model for referring expressions. In Proceedings of the IEEE Conference Computer Vision and Pattern Recog- nition, pages 7282-7290.
A More Details of Implicit Text Generation
To clarify on our implicit referring expression generation process, we'd like to elaborate on the method used to create the ELEVR-Implicin dataset from the existing CLEVR images and their asso- ciated annotations. These annotations contain ob- ject attributes, as well as spatial relationships be- tween objects. For each image, will traverse all the objects in the image by comparing whether the attribute values between two objects are the same/different each time. For example, if an image contains objects, this leads to C(4,2)=6 object pair comparisons (e.g., (1,2), (1,3), (1,4), (2,3),


Reviews:
[{"review_id": "SVleq8fJaV", "reviewer": "Reviewer_avEK", "summary": "", "questions": "1. Examples in Table 1: I would not say this is the same type of implicit texts as in the dataset that is proposed. Are these good examples then really? These require some type of external reasoning beyond image to really understand and identify the object (by understanding what makes it unique and different from others). It is really confusing to see these examples here as they are different from what authors mean by implicit descriptions (or is it?).\n\n2. Section 3.1: so, there are 2 objects per image? Some objects are easier to identify, some harder. Can you provide more details how you generated referring expressions? Did you just combine annotations? Did you filter them in some way?\n\n3. Line 309: MASk is added right before the attribute. This introduces structural biases. Some colors are more frequent and therefore preferred by the model and will be predicted more correctly more often. Models will learn from these biases. Do you think it's a problem?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive suggestions on this paper. The comments are helpful to improve our work. We have carefully studied each comment one by one. We will respond point by point to your review and questions about the paper as follow:\n\n**Response-to-Reject1**: Thank you for the constructive comment. We highlight our contribution is introducing a method to simulate a human-like process for extracting explicit information from implicit textual and visual context. We argee that existing models work well in explicit text, which is validated in existing approaches. Our novelty lies in the methodology we propose to acquire explicit information. The task of transforming the implicit text to explicit involves the fusion of multimodal, extending beyond the domain of text alone. We have not found the mechanism that can enable the models to directly understand the implicit text, but we simulate the process of humans reasoning about the implicit text inspired by the thinking mode of most people on implicit reasoning (extracting the explicit information from implicit text and visual context), and propose the TIE method to enable models to do the implicit reasoning. It begins by transforming implicit text into explicit text and subsequently applies REC reasoning, a process we term \"implicit reasoning\". We believe our method does not eliminate the purpose of our paper, which revolves around implicit reasoning. We'd also like to clarify that the mention of \"understanding the implicit text\" on Line 109-110 pertains to our proposed dataset's future prospects, signifying our ongoing research trajectory. We acknowledge the potential for exploring end-to-end methods for learning directly from implicit descriptions, as you've aptly highlighted. This aligns with our future research directions, and we're actively investigating such approaches. In summary, we thank you for recognizing the significance of our contributions. This review has greatly aided in refining our work, and we're committed to addressing your concerns and insights comprehensively.\n\n**Response-to-Reject2**: (1) Thank you for your suggestion, we will cite these REG related papers. It is crucial to note that while we acknowledge the relatedness of our work to REG tasks, our approach diverges significantly. Our method involves the fusion of multimodal features and the extraction of explicit information through prompt tuning and aims to tackle the challenges posed by multimodal implicit data. Our primary focus centers on the referring expression comprehension task within the context of the multimodal domain.\n\n(2) Regarding your confusion of the examples in Figure 1, we agree that the text could be considered redundant. It's important to acknowledge that redundancy in textual descriptions can indeed arise in multimodal referring expression related datasets. A strong multimodal referring expression model demonstrates the capability to reason through redundant descriptions. A similar example is from CLEVR-Ref+ [1] dataset: “The objects that are either the fourth one of the thing(s) from right or the fourth one of the thing(s) from front that are to the left of the first one of the cylinder(s) from right”, where complex spatial relationships between objects are intentionally included to elevate textual reasoning complexity, despite clearer descriptions being feasible. Another example is from RefCOCOg [2]: “Green apple on the bottom-left corner, under the lemon and on the left of the orange”. Part of description “green apple on the bottom-left corner” is sufficient for object reference, yet redundant or supplementary details are incorporated for textual challenge. In our work, the focus doesn't hinge on the specific textual form, but rather on addressing the issue of implicit text concealing explicit information. Our goal centers around unveiling this hidden information rather than the structure of the text itself.\n\n[1] Liu R, Liu C, Bai Y, et al. Clevr-ref+: Diagnosing visual reasoning with referring expressions[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019: 4185-4194.\n\n[2] Mao J, Huang J, Toshev A, et al. Generation and comprehension of unambiguous object descriptions[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 11-20.\n\n**Response-to-Reject3**: Thank you for bringing up the GQA dataset in the context of reasoning and visual question answering. It’s important to note that while GQA presents valuable challenges for explicit reasoning, our dataset stands apart due to its emphasis on implicit descriptions. The examples from the GQA like:\n\n“Is the bowl to the right of the green apple?”,\n\n“Is the wine glass the same color as the beer bottle?”,\n\n“Does the soap dispenser have the same material as the faucet?”.\n\nThey predominantly employ explicit words like “bowl”, “green apple”, “wine glass”, “beer bottle”, “soap dispenser” and “faucet”. These explicit cues provide a foundation for models to extract information and reason within the visual context. Conversely, our implicit dataset takes a distinct approach. It comprises implicit textual descriptions generated through attribute comparisons among multiple objects in the image. Consequently, our implicit text employs terms like \"object\" \"same color\" and \"shape\" deliberately refraining from revealing explicit object identifiers. This design compels models to decipher the underlying explicit information concealed within the implicit text before undertaking any reasoning process. Therefore, the differences between our dataset and other datasets stem from the unique nature of our implicit descriptions. These differences underscore the complexity of our task, necessitating the development of methods capable of discerning concealed explicit information before embarking on subsequent reasoning steps.\n\n**Response-to-Reject4**: Thanks for your suggestion to explore the literature related to REG tasks. Investigating REG methodologies will contribute to further refinement of our approach. It’s important to note that the task of transforming implicit text to explicit text is not a REG task, while the input of our method contains both visual and textual modalities, and the target object is not indicated in the input, so the process is more complex than REG. We agree with you about the explicit descriptions have a lot of variety and deviation. However, we maintain that explicit reasoning lies not solely in the form, but in the presence of explicit information within the text. The ability of models to perform explicit reasoning hinges on the identification and utilization of these explicit cues. And our goal centers around unveiling this hidden information in implicit text.\n\n**Response-to-Reject5**: Thank you for pointing out the problems in the writing of the paper. We will modify the paper in the revision stage.\n\n**(1) Clarification of TIE modules and predicting strategies. (Line 277-284):**\nWe introduce two modules in TIE. The prompt design module utilizes T5 model to build prompts for implicit text. While the cloze procedure module prompts the ViLT model to fill in the hidden information. We are sorry about make this confusing, and we will make this clearer. Additionally, Base approach is our predicting strategy to predict all attribute values for implicit text with multiple attributes. in contrast to Multi-step approach which involves step-wise predictions.\n\n**(2) Explanation on negative example construction (Line 374):**\nWe recognize the need for a more comprehensive explanation of how negative examples are constructed. To illustrate this, consider an image containing four objects, two are red and rubber, while other two are blue, but rubber and metal. The label (explicit text) in the cloze procedure module might be “The left one of two objects with the same red color and rubber material”. When the model is predicting the value of the first attribute(color), it could yield “red” and “blue” as candidates. However, the two blue objects do not align with the second attribute (material) due to their different materials. To ensure robustness, we build the negative sample “The left one of two objects with the same blue color and [NONE] material”. This enables model to re-predict the output when encountering the [NONE] token. As aptly noted in the opinion, for the “same” type of the implicit text, part of attributes of their corresponding negative samples will have different values. It's worth noting that not all samples in our dataset have corresponding negative examples due to the constraints associated with the negative sample creation process. \n\n**(3) Explanation of [NONE] token in attribute fields (Line 381):**\nLine 381 is also the explanation of the example we mentioned in Line 376-377. As our explanation in above negative samples, we will add [NONE] token to the attributes which dissatisfy the condition of description. Material field is not important, [NONE] token may be inserted in any attributes.\n\n**Response-to-Reject6**: As our explanation in Response-to-Reject5, T5 is used to build prompts for implicit text because it is simple enough but effective.\n\n**Response-to-Reject-Summary**: We believe our work goes beyond giving the setup and describing results which are obvious. As articulated in the response to reason 1, our work is to enable models to reason the implicit text. The reasoning process is structured into two steps: the first step is to retrieval the explicit information through our TIE method with the prompt tuning and some predicting strategies. And the second step is the REC task. By splitting into two steps, our TIE method can be plug-and-play and can be integrated to other referring expression models to do the implicit reasoning, and not only the setup.\n\n**Response-to-Question1**: Thanks for the constructive comment. Our proposed dataset focuses on synthesis images, so the corresponding implicit text does not cover the objects or attributes of the real-world scenes, so we plan to incorporate the discussion of daily life cases as an additional section in our paper. Table 1 shows the cases that the implicit text is possible to appear in our daily life, and our goal is to underscore the importance of researching implicit text occurrences. And the additional section will serve to showcase the applicability of our TIE framework in real-world scenarios.\n\n**Response-to-Question2**: (1) In order to enhance the difficulty of our dataset, we have filtered and chosen the images that contains 3 or more objects. It's important to note that the variance in recognition difficulty across different implicit referring expressions is not arbitrary but is determined by the amount of explicit information that models need to extract from the implicit text.\n\n(2) To shed light on our implicit referring expression generation process, we'd like to elaborate on the method used to create the CLEVR-Implicit dataset from the existing CLEVR images and their associated annotations. These annotations contain object attributes, as well as spatial relationships between objects. For each image, we will traverse all the objects in the image by comparing whether the attribute values between two objects are the same/different each time. For example, if an image contains 4 objects, this leads to C(4,2)=6 object pair comparisons (e.g., (1,2), (1,3), (1,4), (2,3), (2,4), (3,4), number is object id). We generate the implicit text if a certain pair satisfy the condition. Suppose the (2,3) combination objects’ shape and color are all cube and blue, and guarantee that no other pairs satisfy the condition “same shape and color” for avoiding the ambiguity when doing REC, then we will generate the implicit text like “The two objects with the same shape and color”. Due to the only one target of the REC task, we must choose one of the (2,3) objects to be the target. We employ a random selection for the target based on the spatial relationship between 2 and 3. If object 2 is left of object 3 and object 2 is selected, the final implicit text is “The left one of the two objects with the same shape and color.” This consistent pattern of implicit text generation in the CLEVR-Implicit dataset ensures both the concealment of explicit information and reference of target objects in the images.\n\n**Response-to-Question3**: Thank you for addressing the concern. We will incorporate the frequency data of every attribute and their values to the Table 3. Regarding the color values, we have calculated the frequency of red, blue, green, brown, yellow, purple, gray, cyan, all of which fall within the range of 2302 to 2456. We took that into account, and balanced the number of different colors. Therefore, the frequency gap is relatively small. We believe that there is no biases that some colors are more frequent and will be predicted more correctly.\n\nThanks for pointing out the formatting issue with citations and the confusing sentence, we will correct them in the revision stage.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It is an interesting idea: things are often not described explicitly and understanding how we can build better models and dataset for more implicit reasoning (and understanding of implicit descriptions) is important.", "reasons_to_reject": "I think this paper has too many weaknesses to be accepted to EMNLP.\n\n1. The main problem, I think, is that the paper does not do what it promises when it introduces the idea. Authors develop a dataset for understanding implicit descriptions, but their experiments reveal sort of expected results where models are much better in identifying objects which are described explicitly. Plus, authors actually make implicit descriptions explicit and test models on those and show that models are indeed better on explicit descriptions. This eliminates the purpose of the paper introduced in the beginning - the problem of understanding implicit descriptions. In order for it to be a better paper, experiments should focus on actually learning from implicit descriptions and not simplifying the task for the model by making descriptions explicit.\n2.  A lot of relevant literature is missing. Specifically, papers on referring expression generation - NLG community has a lot of ideas which are useful for this paper. Check papers by Emil Krahmer, Ehud Reiter, Kees van Deemter, Robert Dale etc. Reading this literature and taking into account ideas about REG is needed in this paper, as, for example, it is unclear why descriptions in Figure 1 are redundant (why would you say 'the right one' as it is already half-explicit?) and REG literature has a lot of ideas about informativeness of descriptions, the order of attributes, the important of these attributes etc.\n3. Line 081: there are definitely some super related datasets that have implicit descriptions/questions. For example, Drew A. Hudson, Christopher D. Manning: GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering. And I would also encourage authors to explicitly motivate claims like the one in Line 086-088.\n4. The idea of building prompts and prompt learning makes the task very simple as these are more of a template-based methods. Simplicity in this case is not correlating with how real explicit descriptions are - they do not follow a specific template (in fact, there is a lot of variety and deviation). Literature on REG is helpful here for the authors.\n5. Writing in the paper is sometimes just too complicated. Line 277-284 can be really made clearer. Description of the Base model is very confusing. Line 374: do you maybe want to say that attributes in negative examples all have different values? Line 381: only material field? Why only material field is so important?\n6. Some modelling choices are only lightly mentioned, e.g. lines 319. Why T5 and what is the purpose of using T5? It was not clear to me.\n\nI think the paper is describing results which were obvious given the setup (which is ok, but it does not make the paper exciting) and authors deviated from the initial idea in the introduction (given that I understood it correctly).", "questions_to_authors": "1. Examples in Table 1: I would not say this is the same type of implicit texts as in the dataset that is proposed. Are these good examples then really? These require some type of external reasoning beyond image to really understand and identify the object (by understanding what makes it unique and different from others). It is really confusing to see these examples here as they are different from what authors mean by implicit descriptions (or is it?).\n\n2. Section 3.1: so, there are 2 objects per image? Some objects are easier to identify, some harder. Can you provide more details how you generated referring expressions? Did you just combine annotations? Did you filter them in some way?\n\n3. Line 309: MASk is added right before the attribute. This introduces structural biases. Some colors are more frequent and therefore preferred by the model and will be predicted more correctly more often. Models will learn from these biases. Do you think it's a problem?", "paper_topic_and_main_contributions": "The paper introduces a dataset for referring expression comprehension task. The important property of this dataset is that it contains descriptions of objects (in artificial environment) which are not explicit, e.g. the object is described through some property and not mentioned explicitly. Authors describe how they built the dataset and what type of models they test and how to evaluate them on this dataset. Scores and results suggest that models rely on explicit mentions of objects as identifying objects from implicit descriptions is too hard.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "There is a consistent formatting issue with citations, e.g. MDETR(Kamath... There must be a space between reference and what is cited.\n\nLine 156: confusing part of the sentence"}, "extra_text": {}}, {"review_id": "Kc7UlZ1r6Z", "reviewer": "Reviewer_Gmcn", "summary": "", "questions": "- The CLEVR-Ref+ paper includes analysis on expressions that involve the \"same\" operation. However, there is no discussion of it in this paper. How are their findings different from the findings in this paper? Discussion of this would help better identify the contributions of this paper.\n- Why does the prompt design module have to be trainable? At least for the CLEVR domain, it seems like you can use a simple rule to identify words that express an attribute and insert an appropriate number of masked tokens before the word based on whether the expression contains \"same\" or \"different\". Is there additional strategy to it?\n- In the terms used in Table 6, are the results in Table 5 based on the Train pe/Val pe setting? Making this explicit would help understand when and how TIE is used in training and during inference.\n- The discussion of prompt learning in 2.2 is confusing. The wording suggests that the prompts are being learned (improved based on data), but the discussed methods are instances of using prompts to reformulate inputs to make them closer to the model's training distribution, with the changes being performed manually (at least in most of the methods discussed). This section could be improved to be more relevant to the proposed method.\n- Additional examples of the \"different\" samples would also help better understand the construction of that portion of the dataset.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive suggestions on this paper. The comments are helpful to improve our work. We have carefully studied each comment one by one. We will respond point by point to your review and questions about the paper as follow:\n\n**Response-to-Reject1**: Thank you for the constructive comment. Based on our analysis, the situation about implicit text is more likely to appear in the text which contains the same and different types. The implicit text, which hides the explicit information but can refers to an object, often emerges from the comparison between 2 or more objects. Therefore, in CLEVR-Ref+, some classes of referring expressions include: (1) directly describe the target, (2) describe the target by one or several spatial relations, (3) AND, OR and other logics etc. Existing models rely on the explicit words/information to assign the visual features so that they can do the reason regardless of the type of referring expression. But the implicit referring expressions built in CLEVR-Implicit hide the explicit information and force models to reason the implicit text directly, which cause the poor performance of current models. Besides, our proposed TIE method can generalize to other types of referring expressions, like the type of integer comparison. For example, the implicit text of integer comparison might manifest as “The middle one object in a group that have the highest count of a certain color” or more complex like “The amount of a group of objects with a certain color is two more than another group of objects with another color, which object is the left one of this group?” TIE can also build prompts for this type of examples, like “The middle one object in a group that have the largest      number of a certain color”, and then predict the amount (explicit information) of this group of objects. After transformation by TIE, current models can process this type of implicit text as well.\n\n**Response-to-Reject2**: Thank you for the constructive comment. It’s true that our generated implicit text relies on the CLEVR images and the provided annotations. And our proposed TIE method can also expand the scope beyond the CLEVR domain to the real-world photographs. For example, based on an Visual Genome [1] image (You can check the image in this anonymous link: https://anonymous.4open.science/r/example_image-C3BD/25.jpg), consider the implicit text like “The left one of two persons with the same posture and gender”. By our TIE method, prompts can be built before the “posture” and “gender”, like “The left one of two persons with the same      posture and      gender”. The prompt guides the model to predict the explicit words “walking” and “woman” like “The left one of two persons with the same walking posture and woman gender”. By integrating TIE approach, current models can also make the implicit reasoning on the real-world scenarios. Besides, thanks for your suggestion about additional discussion, we will add discussion about how the real-world scenarios fit within our TIE method.\n\n[1] Krishna R, Zhu Y, Groth O, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations[J]. International journal of computer vision, 2017, 123: 32-73.\n\n**Response-to-Reject3**: Thank you for your suggestion about the additional experiments. We have trained both MDETR and VLTVG on CLEVR-Ref+ and tested on CLEVR-Implicit, following the experimental setup in our paper. We get the results in REC task:\n\n|  | val | test |\n| :----:| :----: | :----: |\n| MDETR | 47.02 | 45.85 |\n| VLTVG | 47.99 | 47.11 |\n\nThey are lower than the performance of column “Before TIE” of Table 5. This might be able to illustrate the challenge of reasoning the implicit text in our CLEVR-Implicit dataset. Moreover, we test several examples of our dataset on ViperGPT, one of the generated code is provided below:\n\n```python\ndef execute_query(image):\n    image_patch = ImagePatch(image)\n    objects = image_patch.find(\"object\")\n    if len(objects) < 2:\n        return \"Not enough objects found in the image.\"\n    \n    reference_object = objects[0]\n    matching_objects = []\n    \n    for obj in objects[1:]:\n        if obj.verify_property(\"object\", \"color\") == reference_object.verify_property(\"object\", \"color\") and \\\n           obj.verify_property(\"object\", \"size\") == reference_object.verify_property(\"object\", \"size\") and \\\n           obj.verify_property(\"object\", \"shape\") == reference_object.verify_property(\"object\", \"shape\") and \\\n           obj.verify_property(\"object\", \"material\") == reference_object.verify_property(\"object\", \"material\"):\n            matching_objects.append(obj)\n    \n    if len(matching_objects) == 0:\n        return \"No matching objects found.\"\n    \n    matching_objects.sort(key=lambda x: x.left)\n    return matching_objects[0]\n```\nThe test example is “The left object that is the same color, size, shape and material as the other one”. The generated code has two errors: (1) ViperGPT incorrectly treats the first of all objects as the reference object by default (reference_object = objects[0]), while the correct way involves traversing every two objects combination. (2) There is a problem with the search code. Although ViperGPT can understand the concept of “same” relation and attempt to find the target object by traversing the color, size, shape and material relationship between all objects and reference object, it cannot understand the implicit referring expression and regards the categories of attributes like color and size as the specific attribute values, so LLM does not understand the logic of traversal very well. Besides, existing models are pre-trained on the image-text (with explicit words) pairs so that they have limitations in processing the implicit reasoning. Our method for extracting the explicit information might be a simple but effective approach to overcome the implicit reasoning challenge of current models. And we will add the above discussion into our paper on the next modification.\n\n**Response-to-Question1**: Thank you for the constructive comment. We will add a discussion about the difference of “same” type between CLEVR-Ref+ and CLEVR-Implicit in our paper. The key difference on “same” type text between CLEVR-Implicit and CLEVR-Ref+ is whether the explicit information is hidden. The “same” type of text in CLEVR-Ref+ is like “The objects that have the same size as the red sphere”, which contains explicit information “red sphere”. This explicit context provides REC models with a reference object and helps them locate the target. But our “same” type of implicit text is like “The two objects with the same size”, which hides explicit information. Without explicit context and hard to find the reference object, REC models are forced to compare every two objects to identify what the size is and whether they are same. That is more challenging.\n\n**Response-to-Question2**: It is possible to use a simple script to add masked tokens to text, but considering the possibility of applying it to real-world scenarios in the future (with more attributes and relationships), we use a language model T5, which is also easy to train and can be extended to more complex real-world scenarios.\n\n**Response-to-Question3**: Thank you for the suggestion about this. It’s true that “After TIE” in Table 5 is the same as the “Train-pe/Val-pe” setting and “Ground-truth” is the same as the “Train-ge/Val-ge” setting. We will make this part clearer.\n\n**Response-to-Question4**: Thank you for the suggestion about prompt learning in 2.2. We will improve this section to make it closer to our proposed method.\n\n**Response-to-Question5**: Thank you for the constructive comment. We will discuss more about the “different” samples in our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper undertakes a systematic and thorough analysis of how models perform on a specific aspect of reasoning about language about images. \n- The paper finds that using models (which may be noisy) to expand on implicit information in images can still enable effective reasoning and improve model capabilities. \n- The paper highlights the issues arising from predicting multiple pieces of implicit information and how errors can accumulate in that setting, and also sketches a solution to the problem.", "reasons_to_reject": "- While the paper performs multiple analyses on the proposed CLEVR-Implicit dataset, it doesn't relate the sub-problem of REC in the specific case of \"same\"/\"different\" expressions to the broader problem of REC. Do the issues with grounding these expressions persist when the model is training on a broader class of referring expressions (the answer to this is yes based on results from the CLEVR-Ref+ paper, but a more thorough study would be valuable in this context)? Does the proposed method affect performance on other types of referring expressions? If so, how?\n- While the proposed method is effective on the CLEVR domain, the path to enabling this in REC with photographs is unclear. The proposed method seems to rely on components that assume access to information provided by the underlying program that generates the CLEVR image (such as an enumerable set of attributes, a templated sentence structure that can be easily manipulated to insert additional information, etc.). While expanding the scope to photographs may be beyond the scope of this work, additional discussion on how this might fit within the framework of the TIE method would improve the paper. The limitations section does discuss the role of synthetic data, but it isn't immediately clear how TIE can be made to work on more complex domains.\n- Related to the above two points is the question of comparison to baselines not specifically tailed to the CLEVR-Implicit dataset. The paper is missing results that would help contextualize the difficulty of the task such as a model trained on CLEVR-Ref+ for the same task, and tested on CLEVR-Implicit. Another recent line of work [1] uses large models of code to combine the results of lower-level visual processing modules for complex visual reasoning tasks like REC. These models have been shown to be effective at a task such as REC, require no domain-specific training, and seamlessly integrate into a solution to the more generic version of the REC task. The paper doesn't clearly articulate why the approach of making the implicit information explicit might be a better approach than a modular, compositional form of explicit reasoning. This might even be a suitable baseline, but its use as one is contingent on the authors having the access and resources to closed models like Codex, or being able to get strong numbers with an open-source code model (which would be informative if possible).\n\n[1] Sur'is, D., Menon, S., & Vondrick, C. (2023). ViperGPT: Visual Inference via Python Execution for Reasoning. ArXiv, abs/2303.08128.", "questions_to_authors": "- The CLEVR-Ref+ paper includes analysis on expressions that involve the \"same\" operation. However, there is no discussion of it in this paper. How are their findings different from the findings in this paper? Discussion of this would help better identify the contributions of this paper.\n- Why does the prompt design module have to be trainable? At least for the CLEVR domain, it seems like you can use a simple rule to identify words that express an attribute and insert an appropriate number of masked tokens before the word based on whether the expression contains \"same\" or \"different\". Is there additional strategy to it?\n- In the terms used in Table 6, are the results in Table 5 based on the Train pe/Val pe setting? Making this explicit would help understand when and how TIE is used in training and during inference.\n- The discussion of prompt learning in 2.2 is confusing. The wording suggests that the prompts are being learned (improved based on data), but the discussed methods are instances of using prompts to reformulate inputs to make them closer to the model's training distribution, with the changes being performed manually (at least in most of the methods discussed). This section could be improved to be more relevant to the proposed method.\n- Additional examples of the \"different\" samples would also help better understand the construction of that portion of the dataset.", "paper_topic_and_main_contributions": "The paper studies the problem of grounding referring expressions involving relations between objects (specifically the \"same\" and \"different\" relation between different attributes) where the attributes of the object are referred to implicitly. The paper constructs a synthetic dataset in the widely adopted CLEVR framework, with referring expressions generated using templates. Using the dataset, the paper builds referring expression comprehension (REC) models that identify the object in the image being referred to. The paper finds that existing approaches to REC perform poorly on this task, and propose an algorithm that identifies and makes attributes of the objects in the image explicit, which allows existing approaches to reason more effectively. They find that this idea of making information explicit helps overcome the shortcomings of models in this type of reasoning.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Stylistic: the space before an opening parenthesis is missing in many places (such as line 037, 047, etc.)"}, "extra_text": {}}, {"review_id": "eqhkL7llol", "reviewer": "Reviewer_WAk9", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comment. We will explore the implicit text in the real-world scenarios currently, which contain diverse attributes. And we will discuss about how the real-world scenarios fit within our TIE framework. Our proposed TIE method can also expand the scope beyond the CLEVR domain to the real-world photographs. For example, based on an Visual Genome [1] image (You can check the image in this anonymous link: https://anonymous.4open.science/r/example_image-C3BD/25.jpg), consider the implicit text like “The left one of two persons with the same posture and gender”. By our TIE method, prompts can be built before the “posture” and “gender”, like “The left one of two persons with the same      posture and      gender”. The prompt guides the model to predict the explicit words “walking” and “woman” like “The left one of two persons with the same walking posture and woman gender”. By integrating TIE approach, current models can also make the implicit reasoning on the real-world scenarios.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of the paper is simple but interesting. It shows a limitation of current VLMs on implicit reasoning.\n2. The paper reveals that if the implicit text is converted to explicit text, then VLMs perform much better.\n3. The paper is well organlzed and easy to follow.\n", "reasons_to_reject": "It is a great attempt to evaluate VLM's implicit reasoning on text. However the proposed data is synthetic data and only covered few attributes. It would be great to explore a more realistic setup including diverse attributes.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduced the CLEVR-Implicit dataset for implicit reasoning in the referring expression comprehension task. In their analysis, they showed that models struggle to align the implicit text with the objects in the images. To address the limitation, they proposed Transform Implicit text into Explicit text (TIE).", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hhZCwcTXEE", "reviewer": "Reviewer_rgpy", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive suggestions on this paper. The comments are helpful to improve our work. We have carefully studied each comment one by one. We will respond point by point to your review and questions about the paper as follow:\n\n**Response-to-Reject1**: Thanks for the constructive comment. Based on our experiment results, the REC performance gap of before and after TIE is very huge, but the only difference is whether explicit information is included in the text. We have also made an additional ablation experiment with trained on CLEVR-Ref+ and tested on CLEVR-Implicit, and get the results in REC task: \n\n|  | val | test |\n| :----: | :----: | :----: |\n| MDETR | 47.02 | 45.85 |\n| VLTVG | 47.99 | 47.11 |\n\nTherefore, we believe that existing models are pre-trained on the text with explicit information so that they can only process the explicit reasoning. But implicit text hides the explicit information and force the models to extract the explicit information by traversing all the objects in images to do the reasoning. However, existing models don’t have this capability. Therefore, we divided the implicit reasoning into two steps: TIE and then REC. By splitting into two steps, our TIE method can be plug-and-play and can be integrated to other referring expression models to do the implicit reasoning.\n\n**Response-to-Reject2**: Thank you for the constructive comment. We will make an attempt to fine-tune VL models with TIE in the future work. We are also exploring an effective end-to-end method to do the implicit reasoning more efficient.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper points out an important issue, implicit text matching, for vision-and-langugae learning. It builds a useful dataset on this task and reveals the weaknesses of existing vision-language models on implicit text matching.\n2. This paper proposes a novel and effective methods to transform implicit text into explicit text. In experiments, the proposed method can significantly improve the vision-language models on matching implicit texts.", "reasons_to_reject": "1. It would be better if the authors can design experiments to discuss the reason why existing vision-language cannot match implicit texts. Does it is because that current vision-language model lacks spatial inference ability or other important ability? (solved by authors in the rebuttal)\n2. The authors can also try to fine-tune vision-language models with the TIE, which will better highlight the role of TIE.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper points out the issue that existing vision-language model struggles to deal with implicit text. It constructs a dataset of implicit text matching for related research. To improve the ability of vision-language models to match implicit text, this paper proposes a novel method to utilize masked language models to transform implicit text into explicit text. The proposed method can significantly improve ability of vision-language models on matching implicit text.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewer avEK gave some very detailed critical feedback on the framing of the paper: it only looks at a subset of types of implicit reasoning, and the connection to the implicit reasoning that humans do is a bit tenuous. The authors and this reviewer had a detailed and productive discussion, and I'm confident that the authors will incorporate some of these framing suggestions in the camera ready version of the paper. 

Concerns about the generality of the dataset were also raised by Reviewer Gmcn -- "same"/"different" problems are only one type of REC, and CLEVR does not use real photographs. The author response helped address these, illustrating how to apply the TIE method on Visual Genome, and making a case that findings on the dataset, and the TIE method, are also applicable to other types of REC.

All together, I'm convinced that this work is sound. In terms of excitement and likely impact, it seems that this dataset, and the proposed method, will probably be a useful component in measuring abilities on one type of implicit reasoning.
Decision: Accept-Main