ReCoS: A Novel Benchmark for Cross-Modal Image-Text Retrieval in Complex Real-Life Scenarios

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 10:39:10

Abstract: Image-text retrieval stands as a pivotal task within information retrieval, gaining increasing importance with the rapid advancements in Visual-Language Pretraining models. However, current benchmarks for evaluating these models face limitations, exemplified by instances such as BLIP2 achieving near-perfect performance on existing benchmarks. In response, this paper advocates for a more robust evaluation benchmark for image-text retrieval, one that embraces several essential characteristics. Firstly, a comprehensive benchmark should cover a diverse range of tasks in both perception and cognition-based retrieval. Recognizing this need, we introduce ReCoS, a novel benchmark specifically designed for cross-modal image-text retrieval in complex real-life scenarios. Unlike existing benchmarks, ReCoS encompasses 12 retrieval tasks, with a particular focus on three cognition-based tasks, providing a more holistic assessment of model capabilities. To ensure the novelty of the benchmark, we emphasize the use of original data sources, steering clear of reliance on existing publicly available datasets to minimize the risk of data leakage. Additionally, to strike a balance between the complexity of the real world and benchmark usability, ReCoS includes text descriptions that are neither overly detailed, making retrieval overly simplistic, nor under-detailed to the point where retrieval becomes impossible. Our evaluation results shed light on the challenges faced by existing methods, especially in cognition-based retrieval tasks within ReCoS. This underscores the necessity for innovative approaches in addressing the complexities of image-text retrieval in real-world scenarios.

Main paper: 
 introduction: 
 Image-text retrieval, as a fundamental and crucial problem in information retrieval, has attracted extensive attention in recent years. It aims to bridge the heterogeneous modality gap and achieve semantical matching by bidirectional retrieval, usually consisting of two subtasks: image-to-text (i2t) retrieval and text-to-image (t2i) retrieval. Text-to-image retrieval aims to search the target image from the whole candidate image pool given the text query, while image-to-text retrieval requires the model to search at least one target text description from the candidate text pool given the image query [3] .Early works such as VSE++ [9] and DPC [43] , mapping images and text via convolutional neural networks, only roughly capture modalities' global correspondence, lacking fine-grained vision-language interaction. Consequently, researchers begin to focus on fine-grained retrieval [12, 26, 27, 42] , such as SCAN [14] , which uses Faster R-CNN [30] to encode images more finely and aligns image objects with sentence words. Recently, Visual Language Pretraining models such as BLIP-2 [18] and X2-VLM [40] , acquiring alignment knowledge from image-text pairs through selfsupervised tasks, have been used for image-text retrieval tasks and achieved near-perfect performance on widely used image-text retrieval benchmarks, i.e., MSCOCO-Test-5K and Flickr30K-Test-1K.The benchmarks MSCOCO [22] and Flickr30k [37] have established themselves as widely recognized standards for image-text retrieval assessments. Chen et al. [3] brought attention to challenges associated with "coarse-grained" images and texts, including issues like small retrieval pool sizes and insufficient text descriptions, limiting the evaluation of fine-grained cross-modal semantic matching.Subtracting the two numbers displayed, 42 and 17, equals 25.Madeline Carroll appears to be in her teenage years, closing her lips and throwing sand forward.The tennis player is wearing white shorts and a black shirt with a hint of orange detailing.There are four successive shots blended into one, showing a skateboarder in a green tee progressing through a move.An elderly man dressed in a classic suit and black glasses sits engrossed in his book on a public bench.A white dog wearing a safety vest is situated on a surfboard, with a man standing beside it to the left.With a white wall serving as a backdrop, a boy in a striped shirt is intently holding and eating a donut.Four tall glasses stand in a row, with the first two showcasing amber and yellow hues.A blue and yellow striped shield graphic on the white police car reads \"PHILADELPHIA POLICE\" at the top in bold letters.The execution result of the Python code snippet is the integer 7, which is the sum of the values assigned to variables x and y.The English translation corresponding to the text in the text-only picture is 'I am learning English.'.The skateboarder wearing a white top is in motion, with his body slightly bent forward and his legs raised at differing heights above the concrete surface. In response, refined benchmarks, MSCOCO-FG and Flickr30k-FG, were introduced, aimed at improving the initial coarse-grained images and text descriptions. However, our manual evaluation revealed that MSCOCO-FG and Flickr30k-FG have reduced task difficulty due to the introduction of overly detailed text descriptions. For instance, as illustrated in Figure 1 , the inclusion of the phrase "of a train" significantly simplifies the retrieval of the sample.Recognizing the rapid advancements in Visual-Language Pretraining models, existing image-text retrieval benchmarks encounter challenges in effectively assessing the capabilities of these models. Notably, instances such as BLIP2 achieving near-perfect performance on MSCOCO and Flickr30K underscore the limitations of current benchmarks. To address these concerns and stay abreast of the dynamic landscape of image-text retrieval tasks, the imperative development of a new, comprehensive image-retrieval evaluation benchmark becomes apparent.We posit that a comprehensive evaluation benchmark for imagetext retrieval tasks should exhibit the following four characteristics:(1) Comprehensive Coverage: Encompassing a wide array of subtasks, going beyond the recognition of specific objects. This includes aspects such as existence, count, position, color, and the composition of perceptual information with knowledge in Large Language Models (LLM) to handle more complex retrieval tasks, such as OCR retrieval, code image retrieval, landmark building retrieval, and celebrity retrieval. In pursuit of these objectives, we have curated a comprehensive evaluation benchmark for cross-modal image-text retrieval in complex real-life scenarios, aptly named ReCoS. The main contributions of this paper include:(1) We review current image-text retrieval benchmarks and delineate four essential characteristics that a comprehensive evaluation benchmark for image-text retrieval tasks should possess.(2) As illustrated in Figure 2 , ReCoS comprises 12 sub-tasks in the domain of image-text retrieval. These tasks encompass coarse-grained recognition-based retrieval, covering seven types of retrieval tasks that emphasize aspects such as color, count, location, figure, object, action, and scene. Additionally, ReCoS incorporates fine-grained recognition-based retrieval tasks, including Optical Character Recognition (OCR) and the identification of celebrities. The benchmark extends to cognition-based retrieval, involving common numerical calculations, code reasoning-based retrieval to assess the model's ability to recognize and execute simple code, and text translation-based retrieval to evaluate proficiency in understanding multiple languages. (3) We have created three benchmark versions: 𝑅𝑒𝐶𝑜𝑆 𝑣0 , 𝑅𝑒𝐶𝑜𝑆 𝑣1 , and 𝑅𝑒𝐶𝑜𝑆 𝑣2 . The construction of 𝑅𝑒𝐶𝑜𝑆 𝑣1 and 𝑅𝑒𝐶𝑜𝑆 𝑣2 are built upon their predecessors, introducing heightened task difficulties. (4) We evaluated several representative image-text retrieval models on our new benchmarks and further analyzed their capabilities in complex real-life scenarios.In conclusion, our datasets serve as a significant catalyst for research, prompting the exploration of innovative approaches that seamlessly integrate both perceptual and cognitive aspects. The goal is to achieve more comprehensive and nuanced retrieval results, driving advancements in the field of multimodal retrieval systems. 
related work 2.1 image-text retrieval methods: 
 In recent years, image-text retrieval has been extensively studied, and existing works can be roughly categorized into two categories. Non-pretraining models. Early works [7, 33, 34, 36, 44] , such as VSE++ [9] and DPC [43] , mapping images and text via convolutional neural networks, only roughly capture modalities' global correspondence, lacking fine-grained vision-language interaction. Consequently, researchers begin to focus on fine-grained retrieval [12, 26, 27, 42] , such as SCAN [14] , which uses Faster R-CNN [30] to encode images more finely and aligns image objects with sentence words. Subsequent non-pretraining models basically continue the ideas of VSE++ and SCAN, mainly improving the image-text alignment methods. The main directions of improvement include: 1) Optimizing image encoding methods. For example, the VSRN model [20] encodes image information more finely and uses Graph Convolutional Networks (GCN) [23] and Gated Recurrent Units (GRU) [1] to perform local-global semantic reasoning. 2) Improving the attention mechanism for image-text alignment. BFAN [23] proposes a bidirectional focal attention mechanism to eliminate the impact of irrelevant parts in images and text on the calculation of similarity scores. GSMN [24] uses a more complex way to align image-text, first constructs the relationship graphs of images and text separately, and then matches at the node level and structure level based on the two relationship graphs. SGRAF [6] utilizes GCN to capture the alignment between local and global information, and proposes an attention mechanism based on similarity filtering (SAF). 3) Introducing external knowledge. KASCE [31] and SGM [35] construct external scene graphs to enhance visual relationship learning. 4) Improving retrieval efficiency. Pan et al. [28] believe that the redundant part of cross-modal attention alignment is meaningless, and propose a fine-grained cross-modal alignment network (FCA-Net) to improve the efficiency of image-text retrieval. LexLIP [25] explores a Lexicon-Based cross-modal retrieval method, effectively reducing the computational cost of retrieval. Pretraining models. Visual Language Pretraining is designed to acquire visual language alignment knowledge from a vast collection of image-text pairs through self-supervised tasks. The structure of these models can be broadly categorized into single-stream, dualstream, and hybrid models.Single-stream models, such as UNITER [4] and OSCAR [21] [10] [41] , employ a single encoder to process both images and text, facilitating the learning of alignment between visual and language information. UNITER utilizes a multi-layer Transformer encoder as its core component. OSCAR, in addition to basic image-text pairs, incorporates object tags on images to serve as a connection between the two modalities.Dual-stream models, like CLIP [29] [8] [13] , leverage two independent encoders for processing images and text. The features are then fused and aligned in subsequent stages. CLIP's image encoder may be based on ResNet or ViT structure, while the text encoder follows the Transformer structure. CLIP achieves largescale contrastive learning pretraining on 400M image-text pairs, demonstrating zero-shot effectiveness on various tasks.To leverage the strengths of both structures, hybrid models have emerged [2] . Models such as ALBEF [19] and BLIP [18] excel in modality alignment learning. ALBEF incorporates a multimodal encoder at the back, while BLIP, an enhanced version of ALBEF, retains ITC and ITM pretraining tasks but replaces the MLM task with the LM task to generate image descriptions. BLIP integrates text encoders and decoders for these tasks, sharing parameters across corresponding structural layers. The upgraded version, BLIP-2 [17] , establishes a connection between the visual and text large models through a lightweight Q-Former model. In the domain of image-text retrieval, BLIP-2 has achieved nearly perfect state-ofthe-art results.MSCOCO [22] and Flickr30k [37] are prominent datasets in imagetext retrieval. Flickr30k comprises 31,783 images depicting everyday activities, each with five crowd-sourced descriptions. This dataset has 29,783 training images, 1,000 validation images, and 1,000 test images. On the other hand, MSCOCO is larger, with 123,287 images covering 91 common object categories. Each MSCOCO image has manually added bounding boxes, segmentation, and five descriptions. The dataset includes 113,287 training images, 5,000 validation images, and 5,000 test images.However, Chen et al. [3] highlighted challenges associated with "coarse-grained" images and texts. They identified two primary issues: 1) small image retrieval pool sizes, resulting in significant variation or semantic sparsity, making retrieval targets easily distinguishable without the need for fine-grained semantic understanding; 2) a considerable number of text descriptions lacking detail. These challenges impose limitations on evaluating a model's fine-grained cross-modal semantic matching capabilities. In response, Chen et al. introduced two benchmarks, MSCOCO-FG and Flickr30k-FG, by refining the initially coarse-grained images and text descriptions in the original MSCOCO and Flickr30k datasets.Nevertheless, the existing datasets mentioned above fall short in capturing intricate real-life scenarios, including code recognition and numerical computation. This inadequacy poses a challenge for conducting a thorough evaluation of model performance. In this paper, we introduce a pioneering benchmark tailored for imagetext retrieval within complex real-life contexts. Our benchmark addresses this limitation by encompassing a wider array of subtasks within intricate scenarios, such as OCR retrieval, code image retrieval, landmark building retrieval, and celebrity retrieval. This diverse set of challenges enables a more exhaustive evaluation of model capabilities. The proposed benchmark stands as a rigorous assessment tool for image-text retrieval, particularly well-suited for evaluating the effectiveness of Visual Language Pretraining models.  
benchmark: 
 In this section, we present the process of building our 𝑅𝑒𝐶𝑜𝑆 benchmark, comprising three versions: 𝑅𝑒𝐶𝑜𝑆 𝑣0 , 𝑅𝑒𝐶𝑜𝑆 𝑣1 , and 𝑅𝑒𝐶𝑜𝑆 𝑣2 . We initiated by selecting datasets from diverse domains outside of image-text retrieval, including synthetically generated datasets, to mitigate data leakage risks. Subsequently, we executed four key steps in a scholarly manner, as illustrated in Figure 4 . The following Section 3.1 first involves curating candidate images from these datasets to meet the specific requirements of the 12 subtasks. Section 3.2 delineates the detailed annotation process applied to the selected images. Section 3.3 discusses the introduction of additional similar images and the generation of perplexing textual descriptions to enhance the difficulty of the image-text retrieval task. These descriptions closely mirror those of the original images (see the appendix for details). Section 3.4 elaborates on the further augmentation of task complexity by introducing additional similar images devoid of annotations. Section 3.5 delineates the disparities in image quantity and subtask difficulty among the three versions.We initially collected 500 images to create the candidate image set D, as depicted in Figure 3 , to address the following 12 types of retrieval tasks: (1) Coarse-Grained Recognition-Based Retrieval. This category involves seven types of retrieval tasks focusing on the recognition of color, count, location, figure, object, action, and scene. Each task type includes 50 images, totaling 350 images. Specifically, we select images related to these seven categories and generate corresponding descriptions based on the specific categories, ensuring that the descriptions covered different categories of words. The images for these tasks were gathered from COCO 1 with only images. (2) Fine-Grained Recognition-Based Retrieval. This category includes two types of retrieval tasks focusing on Optical 1 https://cocodataset.org/ Character Recognition (OCR) with images sourced from Totoltext [5] and the recognition of celebrities with images manually captured from videos on public websites. OCR is also a task for testing the basic capabilities of the model. The focus is on text recognition in pictures. Due to the diversity of scenes, OCR is still very difficult [38] . Tasks that include celebrities examine whether the model can specifically identify celebrities in the picture. Each task contains 30 images, totaling 60 images. (3) Numerical Calculation-Based Retrieval. This category comprises 30 manually created images, predominantly featuring common numerical calculation scenarios, including simple addition, subtraction, multiplication, division calculations, and area calculations of two-dimensional images.Huang et al. [11] proposed that if large models are to move towards general artificial intelligence, they must have strong multi-modal cognitive capabilities. The primary objective of this category is to assess the model's ability to recognize handwritten digits and its reasoning proficiency for simple numerical calculation problems. (4) Code Reasoning-Based Retrieval. In this category, there are 30 images of code generated by GPT-4, manually modified and verified. At present, there are some researches on code generation [32] , but there is still a lack of research on the understanding of image code recognition. The primary task is to assess the model's capability to recognize code images and its understanding of simple code. (5) Text Translation-Based Retrieval. There are already many large models that support multiple languages, and we hope to verify their translation capabilities. This category is established to assess the model's understanding of both Chinese and English languages. All 30 images in this category are manually designed. Given the current capabilities of existing models for Chinese and English, our current version focuses on designing simple Chinese-English translation tasks.Generation of Image Descriptions. Refinement of the process outlined in the dashed box in Figure 4 encompasses the following four steps applied to the candidate image set D to form a simple version image-text pair dataset 𝑅𝑒𝐶𝑜𝑆 𝑣0 :• Generate image description.(1) For a given sample image 𝑥 ∈ D, generate five text descriptions {𝑡 1 (𝑥), • • • , 𝑡 5 (𝑥)} using either GPT-4 or manual annotation. • Retrieve Top-k images.( (1) Conduct a grammar check and manual verification to ensure the quality of the text description. (2) Output the 5 refined descriptions for each image.Validation of of text description. In the aforementioned process, evaluating the discriminativity of the text description is a crucial step. This assessment is achieved by computing the discriminativity of the text description 𝑡 as 𝑑𝑖𝑠 (𝑡) = entropy(softmax(sim(𝑡, S(𝑡)))), where:(1) sim(𝑡, S(𝑡)) calculates the cosine similarities among 𝑡 and the images in S(𝑡). (2) softmax(.) generates classification probabilities using the softmax function with temperature parameters set to 1. (3) entropy(.) outputs the entropy of the probability distribution.The value of 𝑑𝑖𝑠 (𝑡), ranging between 0 and log 𝑘, serves as an indicator of BLIP2's confidence in the retrieval result. The minimum value of 0 signifies high certainty in the retrieval, suggesting an overly simplistic retrieval task associated with an over-detailed text description. On the other hand, the maximum value of log 𝑘 indicates uncertainty in the retrieval, suggesting an overly difficult retrieval task linked to an under-detailed text description. The introduction of a parameter, denoted as 𝜂 ∈ (0, 1), plays a crucial role in the selection of text descriptions for regeneation. Specifically, we identify descriptions where 𝑑𝑖𝑠 (𝑡) ∉ [ 1-𝜂 2 log 𝑘, 1+𝜂 2 log 𝑘]. text descriptions falling outside this range are considered unqualified, indicating the need for regeneration. By adjusting the value of 𝜂, we provide a mechanism to either expand or narrow the scope for identifying unqualified text descriptions. This flexibility empowers the annotator to fine-tune the discriminativity threshold according to the desired level of certainty in the retrieval task. A larger 𝜂 not only widens the range for identifying unqualified descriptions but also provides nuanced control over the delicate balance between task simplicity and difficulty in the annotation process. This parameter serves as a versatile tool, enabling customization based on the specific requirements and intricacies of the annotation task.In pursuit of heightened task difficulty, we augmented the challenge by introducing similar images paired with corresponding text descriptions. Specifically, we appended an additional 500 image-text pairs to 𝑅𝑒𝐶𝑜𝑆 𝑣0 , thereby creating a more intricate dataset denoted as 𝑅𝑒𝐶𝑜𝑆 𝑣1 . Our approach involves a semi-automated method, characterized by the following three main steps:• Similar image selection. For each image 𝑥 ∈ D, we identified a similar image 𝑥 ′ from the source 𝑥 was sampled from. BLIP2 was employed to compute the image embedding, facilitating the selection of the most similar image. • Text description generation for similar images. Utilizing GPT-4, we generated proper text descriptions 𝑡 ′ for the selected similar images 𝑥 ′ . This process included the following steps: (1) Generate descriptions by referring to 𝑡 (original description). ( 2) Generate descriptions focusing on color, quantity, and other attributes. (3) Attempt to replace core attributes/keywords in provided descriptions based on image details. (4) Minimize word changes from the referenced description.Ensure each generated description is grammatical, logical, and uniquely detailed. • Conduct manual checks and verifications that the generated similar description matches the similar image.To enhance the complexity of our benchmark, we implemented an image expansion strategy that involved introducing more intricate images into the dataset. This was achieved by incorporating additional images that exhibit similarities to the original ones but lack corresponding text descriptions. For each image 𝑥 ∈ D, we identified twenty similar images from the same source as 𝑥, resulting in a set of 20000 images. These images were then merged with 𝑅𝑒𝐶𝑜𝑆 𝑣1 , and the resulting set underwent refinement to eliminate duplicates, yielding 𝑅𝑒𝐶𝑜𝑆 𝑣2 with a total of 15982 images. It is important to note that out of these, only 1000 images are paired with corresponding text descriptions.  
experiments: 
 In this section, we conduct an evaluation of several classic imagetext retrieval models on our novel benchmarks. Furthermore, we present a comprehensive analysis of the performance of these models, delving into the intricacies of the various sub-tasks and versions incorporated within our benchmarks.Benchmarks: In this experiment, we employ the following five benchmarks:• Flickr30K and MS-COCO benchmarks are derived from everyday life scenarios. The primary distinction lies in the number of test images: the former includes 1,000 images, while the latter encompasses 5,000. • Flickr30K-FG and MS-COCO-FG benchmarks [3] offer more detailed textual descriptions while maintaining the image quantity of their original datasets, 𝐹𝑙𝑖𝑐𝑘𝑟 30𝑘 and 𝑀𝑆 -𝐶𝑂𝐶𝑂. • ReCoS benchmark, introduced in this paper, comprises 1, 000 test images depicting complex real-life scenarios. To assess its performance against previous benchmarks, we conducted comparative experiments using the standardized image-text pairs of 𝑅𝑒𝐶𝑜𝑆 𝑣1 . However, in practice, 𝑅𝑒𝐶𝑜𝑆 𝑣2 (with some images lacking annotations) poses significantly greater difficulty than 𝑅𝑒𝐶𝑜𝑆 𝑣1 , enabling a more comprehensive evaluation of models' capabilities in image-text retrieval, particularly in fine-grained comprehension of text for image retrieval.Baselines and Implementation details: All experiments are conducted on NVIDIA V100 GPUs, and we use as large models as possible. We used the default configuration in LAVIS [16] for CLIP, ALBEF, BLIP, and BLIP2, and the experimental setup in [39] for X-VLM and X2-VLM. Additionally, VSE++ adhered to the configuration from [9] , and SCAN followed that from [14] , VSRN followed that from [20] .Evaluation Metrics: Following established research practices [9, 15, 20] , we evaluate image-text retrieval performance using Recall@k (R@k). This metric quantifies the percentage of queries accurately retrieving the ground truth within the top-k results, where k is selected from values of 1, 5, and 10.Table 2 displays the experimental results of nine models across the five benchmarks. Initially, we note that the recall rates (R@K) of non-pre-trained models, namely VSE++, SCAN, and VSRN, are notably lower than those of pre-trained models, including CLIP, AL-BEF, X-VLM, X2-VLM, BLIP, and BLIP2. However, we also observed performance declines in both pre-trained and non-pre-trained models specifically on our benchmark 𝑅𝑒𝐶𝑜𝑆 𝑣1 when compared to the other four existing benchmarks. This decrease highlights the challenges in our benchmarks regarding accurately retrieving correct images for a given text description in the presence of numerous similar images, and vice versa. X2-VLM and BLIP2 demonstrated superior performance across most tasks, indicating their effectiveness. Nevertheless, our benchmark poses a substantial challenge even for both X2-VLM and BLIP2, motivating the research community to enhance them in future work.To investigate the difficulty of different substaks in our benchmark 𝑅𝑒𝐶𝑜𝐶 𝑣1 , we choose the representative coarse-grained model method VSE++, fine-grained method SCAN and VSRN, and two pre-trained models CLIP and BLIP2 to perform experiments across the 12 sub-tasks. The results are illustrated in Figure 6 and 7 . In image-to-text and text-to-image retrieval tasks, current models, particularly emphasizing pre-trained models like BLIP2 and CLIP, have showcased exceptional performance. They achieved perfection in figure and scene recognition-based retrieval tasks and consistently maintained over 90% accuracy across five coarsegrained and two fine-grained recognition-based retrieval tasks. The notable superiority of BLIP2 and CLIP, compared to non-pre-trained models, is credited to their robust feature extraction capabilities Despite the exceptional performance of models such as BLIP2 and CLIP in perceptual tasks, where they achieve near-perfect results in recognition-based retrieval tasks, particularly for figures and scenes, they face substantial challenges in cognition-based retrieval tasks. A selection of samples from cognition-based retrieval tasks, where both BLIP2 and CLIP struggle, is illustrated in Figure 7 . Even in tasks that demand linguistic understanding, such as text-translationbased retrieval, all models fall short, with some achieving zero accuracy. This highlights a limitation in their capacity to effectively integrate perceptual information with deeper cognitive knowledge.The collective struggle of all models in cognition-based tasks indicates a critical gap in existing methodologies. Our dataset emerges as a valuable resource, prompting a shift in research focus and challenging the community to explore innovative approaches that seamlessly integrate both perceptual and cognitive aspects.In our evaluation, we employed CLIP to assess its retrieval performance across the three versions of our benchmarks, as detailed in Table 3 . It is important to note that we refrained from conducting the image-to-text task on 𝑅𝑒𝐶𝑜𝑆 𝑣2 due to the absence of text descriptions for the additional images introduced based on 𝑅𝑒𝐶𝑜𝑆 𝑣1 .The results displayed in Table 3 unveil a consistent decline in performance from 𝑅𝑒𝐶𝑜𝑆 𝑣0 to 𝑅𝑒𝐶𝑜𝑆 𝑣2 , signifying a notable escalation in task difficulty. Particularly noteworthy is the more pronounced decrease in R@1 compared to R@10, indicating a heightened challenge in achieving accurate retrieval results. This decline in recall rates sheds light on CLIP's limitations in effectively differentiating between similar images or semantic sentences. It prompts a deeper investigation into enhancing CLIP's discriminative capabilities, specifically tailored to the challenges posed by our benchmark. Image → Text Text → Image R@1 R@5 R@10 R@1 R@5 R@10  
conclusions: 
 In this study, we scrutinize common benchmarks for image-text retrieval and find that they fail to fully assess the true capabilities of fine-grained cross-modal semantic alignment due to the coarse granularity of images and texts, excessive descriptive information, and overly simplistic retrieval scenarios. Therefore, We propose 𝑅𝑒𝐶𝑜𝑆, a novel image-text benchmark, designed to address challenges in cross-modal retrieval in real-life scenarios, with three versions: 𝑅𝑒𝐶𝑜𝑆 𝑣0 , 𝑅𝑒𝐶𝑜𝑆 𝑣1 , and 𝑅𝑒𝐶𝑜𝑆 𝑣2 .To compare with existing benchmarks, we select the moderately challenging 𝑅𝑒𝐶𝑜𝑆 𝑣1 as the standard benchmark, where each image is annotated. By evaluating representative image-text retrieval models on 𝑅𝑒𝐶𝑜𝑆, we demonstrate and analyze the models' fine-grained semantic understanding capabilities across various subtasks through extensive experiments.The experimental results indicate that even state-of-the-art retrieval models exhibit certain limitations in real-world scenarios. In the future, we aim to extend more complex retrieval subtasks, such as complex charts, intricate flowcharts, with the hope that the novel benchmark will inspire further research into cross-modal retrieval.  
 
 Appendix: ACM MM, 2024, Melbourne, Australia Anon. 


Decision: Except for one reviewer, there is a general consensus to accept this paper.
The authors' rebuttal successfully addressed most of the concerns raised by the reviewers.
Some typos and minors were noted. Please, in case of acceptance double check.
The major strength mentioned by the reviewers is the usefulness of the proposed novel benchmark dataset for image-text retrieval. The introduction of this dataset promotes further development in the field of multimodal retrieval.
The organization of the article is excellent, and the construction process of the dataset is described in detail.

Reviews: review: paper_summary: paper_summary: This paper focuses on the task of image-text retrieval. It is argued that with the rapid development of visual language pre-training models, the existing image-text datasets are relatively homogeneous in composition and cannot objectively and correctly evaluate the performance of the models from multiple perspectives. Therefore, this paper proposes an evaluation criterion called ReCoS in this area, which collects and generates new data to challenge existing graphic-text retrieval methods for more fine-grained and complex scenarios.
strength_weakness: strength_weakness: a.	Compared with previous methods for improving benchmarks, this paper introduces a variety of additional tasks such as code image retrieval, landmark construction retrieval and celebrity retrieval to make the evaluation results more comprehensive.
b.	This paper proposes an easier-to-use quantitative method for assessing the quality of generated text descriptions.The method proposed in this paper is easy to follow.
c.	This paper presents several different versions of benchmarks that can be effectively evaluated on a scale of difficulty from easy to hard.
limitations: limitations: a.	In the validation of of text description module ,the description of how to use the cross-entropy function to calculate dis(t) is not clear, and there is no way to know whether additional terms are added to expand the range of dis(t) to [0,logk], and the introduction of hyperparameters is not analyzed with appropriate sensitivity,and it is difficult to obtain useful information in the upper part of Figure 5.
b.	In the image confusion module mentioned in this article, for example, “(2) Generate descriptions ... (3) Attempt to ...” It is not possible to obtain information on the specific methodology and criteria used for the assessment, and it would be appreciated if the relevant material could be supplemented.
c.	Among the benchmarks proposed in this paper, the amount of data in “ReCoSv1”, which can be used for the evaluation of bi-directional retrieval is on the low side, which weakens the persuasive power of the results to a certain extent.

Typos and minors
The whole manuscript should be carefully checked for typos, grammar and syntax, as there are some of them. The following are some example:
a.	In lines 581 and 582, that is, “Specifically, we identify descriptions... .logk].text descriptions...” section. The period before “text” here should be replaced with a comma, otherwise the break could be easily misinterpreted.
b.	Suggest revising the logic of the article, the  readablity of this paper is weak.
paper_summary: paper_summary: In this paper, the authors propose a more robust evaluation benchmark specifically designed for cross-modal image-text retrieval in complex real-life scenarios. The delineate four essential characteristics that a comprehensive evaluation benchmark for image-text retrieval tasks should possess. Furthermore, the authors evaluated several representative image-text retrieval models on their proposed benchmark dataset, including both non-pre-trained models and pre-trained models, to further analyze their capabilities in complex real-world scenarios.
strength_weakness: strength_weakness: 1、The article summarizes four essential characteristics that a foundational dataset for image-text retrieval tasks should possess and introduces a new benchmark dataset. The introduction of this dataset promotes further development in the field of multimodal retrieval.
2、The organization of the article is excellent, and the construction process of the dataset is described in detail.
limitations: limitations: 1、The dataset proposed by the authors has three different versions, but during the experiments, only the results on versions ReCoSv1 and ReCoSv2 were observed. Therefore, is the smaller dataset version ReCoSv0 useful?
2、The new dataset proposed by the authors is relatively small, and the specific division of the training set, validation set, and test set is unclear.
3、Regarding the experimental results in Table 2, there is a question about how these results were obtained. On which datasets were the various comparison models trained, especially the experimental results for ReCoSv1?
4、The authors' experimental analysis is not particularly clear. I believe that the potential improvement points of existing image-text retrieval methods should be analyzed in detail based on the experimental results and the proposed new dataset.
5、The authors should provide more detailed samples from the newly created dataset. The samples shown in Figure 2 are too simplistic.
paper_summary: paper_summary: The paper introduces ReCoS, a novel benchmark for cross-modal image-text retrieval designed to address the limitations of existing benchmarks. ReCoS is tailored for complex real-life scenarios and emphasizes cognition-based tasks, aiming to provide a more holistic assessment of model capabilities. The benchmark includes 12 retrieval tasks, focusing on three cognition-based tasks, and utilizes original data sources to ensure novelty and minimize data leakage risks. The paper also presents three versions of the benchmark with varying difficulty levels and evaluates several representative image-text retrieval models on these benchmarks. The results highlight the challenges faced by existing methods, particularly in cognition-based retrieval tasks, and underscore the need for innovative approaches to handle real-world complexities. However, the authors should consider providing more insights into how the benchmark can be generalized and ensuring that it does not inadvertently favor certain model architectures or tasks. Additionally, a discussion on the potential computational requirements and resource availability for using ReCoS would be beneficial for the research community.
strength_weakness: strength_weakness: 1.Comprehensive Benchmark Design: ReCoS covers a wide range of subtasks, going beyond object recognition to include aspects like color, count, position, and complex retrieval tasks such as OCR, code image retrieval, and text translation.
2.Original Data Sources: The use of original data sources helps to ensure the novelty of the benchmark and reduces the risk of overfitting to existing datasets.
3.Balanced Text Descriptions: The benchmark strikes a balance between overly detailed and under-detailed text descriptions, which is crucial for creating a fair and challenging evaluation environment.
4.Real-World Complexity Alignment: ReCoS is designed to reflect the intricacies of real-world scenarios, providing a more accurate assessment of Visual-Language Pretraining models.
5.Incremental Difficulty Levels: The creation of three versions of the benchmark with varying difficulty levels allows for a tiered evaluation of model capabilities.
6.Thorough Evaluation: The paper includes an extensive evaluation of several state-of-the-art models, providing insightful analysis into their performance across different tasks.
limitations: limitations: 1.Complexity of Benchmark: While the complexity of ReCoS is a strength, it may also be a weakness as it could potentially make the benchmark too challenging for current models, possibly slowing down research progress in the short term.
2.Resource Intensity: The use of original data sources and the creation of a comprehensive benchmark may require significant resources, which could limit accessibility for smaller research groups.
3.Potential for Bias: As with any new benchmark, there is a risk of introducing bias towards certain types of models or tasks. Ensuring a wide representation of model architectures and tasks is crucial.
4.Generalization Concerns: The paper does not extensively discuss how well the results from ReCoS generalize to other real-world scenarios outside the benchmark.
5.Overemphasis on Cognition-Based Tasks: While cognition-based tasks are important, there might be an argument for a more balanced approach that equally emphasizes perceptual tasks, given their relevance in many real-world applications.
6.Evaluation Methodology: The paper could benefit from a more detailed discussion on the evaluation methodology, including how the models were selected and how their performance was compared.
paper_summary: paper_summary: This paper proposes a novel benchmark for the task of cross-modal retrieval, it addresses 12 retrieval subtasks and contains three variants.
strength_weakness: strength_weakness: The author conducted thorough evaluations of existing methods on said benchmark and compared the results with those of existing datasets. The paper is well-written, in that it conveys the motivation and methodology of the proposed benchmark clearly.

The benchmark carries several merits compared to the existing ones, such as the inclusion of assessing the models' confidence through entropy.
limitations: limitations: **Novelty and the amount of contribution**

While the benchmark is insightful and aligns well with the motivation, I am afraid that it alone may not constitute a sufficiently novel submission. 

Indeed, this entire paper surrounds the benchmark, with no additional contributions on the task in terms of, e.g., method or other detailed analysis on the performance (4.3 includes some discussions on the sub-tasks and their performance, though I personally find it not enough to be claimed as a major contribution).


**The diversity of the sub-tasks**

As the authors show in Figure 7, with the ReCos$_{v1}$ metric the current models suffer greatly from the three sub-tasks, *code reasoning*, *text translation* and *numerical calculation*, meanwhile performing relatively well (take BLIP2 for instance) on the remaining ones.

To an extent, this result is to be expected because cognition-based sub-tasks are inherently harder, and one might even argue that they are entirely on a different landscape -- which brings me to my concern as to why they shall be mixed with all the other recognition-based sub-tasks. If one wishes for a retrieval model to excel at text translation, for example, one should either aim to build an AGI (alas we are not there yet), or enlist a translation model to help instead of relying on the retrieval mode alone.

All in all, I am not against evaluating a retrieval model on different sub-tasks, I am just not convinced that the author's choice of the subtasks is entirely justifiable -- and to consider that this benchmark, once released, shall encourage others to test and develop on it, one need to consider the implications and impact this benchmark will bring to future research.

**The current state-of-the-art on the proposed benchmark**

Given that part of the author's motivation is to address the fact the state-of-the-art methods on the existing metrics have reached near-perfection, the newly proposed benchmark needs to be more challenging. Indeed, in Table 2 we observe a noticeable difference in the values. However, this is compared with the *v1* metric, I am a bit curious about how the *v0* compares. 

Also, a note that even with the *v1* metric, the performance difference compared to the existing benchmarks is not extremely large, which brings two questions:

 * Will this newly proposed benchmark last long before it reaches the point where it can no longer distinguish state-of-the-art models (just like the current ones)?
* How much of the performance drop is contributed by the much harder cognitive-based sub-tasks discussed above?

**Minor comments**

It would be great if the author could provide some information on the inference cost of the three new metrics, especially the time that takes for one evaluation.