DQG: Database Question Generation for Exact Text-based Image Retrieval

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 09:22:58

Abstract: Screening similar but non-target images in text-based image retrieval is crucial for pinpointing the user's desired images accurately. However, conventional methods mainly focus on enhancing text-image matching performance, often failing to identify images that exactly match the retrieval intention because of the query quality. User-provided queries frequently lack adequate information for screening similar but not target images, especially when the target database (DB) contains numerous similar images. Therefore, a novel approach is needed to extract valuable information from users for effective screening. In this paper, we propose a DB question generation (DQG) model to enhance exact cross-modal image retrieval performance. Our DQG model learns to generate effective questions that precisely screen similar but non-target images using DB contents information. By answering the questions generated from our model, users can reach their desired images by only answering the presented questions even within DBs with similar content. Experimental results on publicly available datasets show that our proposed approach can significantly improve exact cross-modal image retrieval performance. Code is available in the supplemental materials and will be publicly available.

Main paper: 
 introduction: 
 Image retrieval through text query, also known as text-based image retrieval, is one of the most fundamental research topics that should grasp the user's retrieval purpose [30, 58] . The primal aim is to retrieve images similar to the provided query, commonly known as similar image retrieval, and it is a critical task for various Web services such as search engines, recommendation systems, and ecommerce services [7, 13, 60] . Recent advancements in text-based image retrieval have primarily focused on enhancing the performance of similar image retrieval by leveraging shared latent spaces to compute similarities between query texts and candidate images [43] . While these methods excel in similar image retrieval, they often fall short in facilitating the retrieval of specific target images that precisely match users' retrieval intentions, referred to as exact image retrieval. This challenge arises due to the difficulty users face in providing sufficiently detailed query information.Exact image retrieval works well when users provide an appropriate query that contains ample information to uniquely identify the desired image from the target database (DB). However, consistently providing such precise queries poses challenges, given users' limited knowledge about the entire target DB's content. User-provided queries can correspond to multiple candidate images, leading to decrease retrieval performance, even in state-of-the-art text-image foundation models like CLIP [33] and BLIP [24] . This problem escalates significantly when the target DB contains abundant similar content, as it increases the number of candidate images related to queries. For instance, retrieving a specific desired image using the text query "A photo of travel with my friends" from a DB abundant in travel images poses significant challenges. Given the diverse applications of exact image retrieval such as lifelog search [46] and landmark search [53] , dealing with the appropriate query provision difficulty is an essential topic.One promising approach to alleviate query provision challenges is integrating a question-generation (QG) scheme into conventional text-based image retrieval methods. In the adjacent field of document information retrieval, the QG-based retrieval scheme has been widely discussed for enhancing exact information retrieval performance in addition to the conventional retrieval scheme [1, 56] . By introducing a QG module, retrieval systems can better understand users' retrieval intentions through text-based interactions, enabling users to provide additional clues for finding specific target information by answering provided questions. Referring to the performance improvement in the document information retrieval field, the QG module is likely to contribute to exact image retrieval performance improvement by alleviating the query provision difficulty.The QG module for exact image retrieval needs to adaptively generate questions tailored to each target DB. When generated questions align with multiple candidate images, similar but nontarget images cannot be screened even though the retrieval system receives the answers from users. Therefore, the QG module for exact image retrieval must generate questions capable of screening similar candidate images within the target DB adaptively. While manually preparing predefined question labels is a theoretical solution for adaptive QG, it demands considerable effort due to the diverse nature of image retrieval across different target DBs. Consequently, an adaptive QG module that learns suitable questions solely from target DB contents is highly desirable.In this paper, we propose a DB question generation (DQG) approach for enhancing exact text-based image retrieval. Our approach focuses on learning the appropriate question on the target DB only from those candidate images. Here, the appropriate questions must possess two key characteristics: effectively screening similar candidate images and being grammatically understandable to users. Our DQG module generates questions from DB information input, and the generated questions are optimized to meet the two characteristics by minimizing retrieval-based loss and grammatical-based loss. Specifically, our learning scheme transfers the knowledge of the other vision and language models solely using the candidate images on DBs, enabling automatic question generation without retrieval task-specific question labels. The generated questions by our DQG module guide users to their desired images by screening similar but non-target candidate images based on user responses. In other words, users can reach their desired images by only answering the presented questions with our approach. Experimental results show that our approach can improve the exact image retrieval performance and generate grammatically correct questions.The contributions of this paper are summarized as follows.DQG approach for exact text-based image retrieval Our DQG approach effectively screen similar but non-target candidate images, guiding users to their desired images through provided questions.Our DQG module is trained to generate the suitable questions from DB information input without using any question labels for retrieval tasks.Experimental results demonstrate notable enhancement in exact image retrieval performance compared to the current state-of-the-art BLIP method, by +13.1%, +7.5%, and +10.4% in the R@1 metric on the MSCOCO, VG, and biased-MSCOCO datasets, respectively. 
related works 2.1 text-based image retrieval: 
 Our approach focuses on screening similar but non-target candidate images in a text-based image retrieval (hereinafter referred to as TBIR) paradigm. Most recent TBIR methods aim to map a query text and candidate images into the latent space E [5, 11, 37, 42] . That is, they train the two mapping functions Enc L (•): L → E and Enc V (•): V → E, where L and V are lingual and visual spaces, respectively. By embedding a query text and candidate images via the trained mapping functions Enc L (•) and Enc V (•), similarities between the text query and candidate images are calculated on the space E. Conventionally, TBIR is realized based on statistical correlation analysis, such as canonical correlation analysis [18] . However, the advent of deep neural network technologies has revolutionized representation learning, enabling the training of robust mapping functions Enc L (•) and Enc V (•) [12, 20, 22, 24, 33, 59] . Based on the hinge loss, Kiros et al. [22] trained Enc L (•) and Enc V (•) so that the similarities between correct text-image pairs are higher than those between other different pairs. Faghri et al. [12] improved the method used by Kiros et al. [22] by considering the number of images between a text query and the retrieval target image. The existing TBIR performance has been improved by focusing on the loss function and network architectures.While existing TBIR methods excel in similar image retrieval tasks, our work targets the specific retrieval of images exactly matching users' intentions. By integrating DQG-based interaction for re-ranking, our system leverages user feedback to screen similar but non-target images more effectively.Re-ranking has been studied in various retrieval tasks such as person re-identification and object retrieval, and methods for TBIR tasks have also been proposed [3, 4, 30, 36, 54] . Our retrieval approach aligns with re-ranking methodologies as it incorporates additional user information for screening similar candidate images. Therefore, we review the difference between our approach and similar works in this subsection. Based on the necessity of user interaction, re-ranking methods can be roughly classified into two categories: self re-ranking and feedback-based re-ranking.Self re-ranking aims to improve retrieval performance by estimating the key information from the top-ranked images of the initial retrieval results. Several self-re-ranking methods [44, 45, 55] rank text labels in reverse order utilizing each initial ranked image as a query and re-rank the initial retrieval results based on these results. Although these self-re-ranking methods can improve the initial retrieval performance without any feedback information, they cannot obtain additional information from users. Therefore, it is difficult for self-re-ranking methods to deal with query provision difficulty.Feedback-based re-ranking aims to improve retrieval performance based on user feedback. Several learning-based methods [17, 39, 41, [50] [51] [52] allow users to provide feedback on retrieval results via natural language. In the fashion domain, Guo et al. proposed a reinforcement learning-based re-ranking method [17] . By learning the texts that describe differences of images, they can estimate user desired images from feedback comments on the top-ranked image. Although its method enables users to provide natural languagebased feedback, there is no guarantee that the feedback provided by the users effectively clarifies their query text. Besides, users are required to consider additional natural language-based queries for the re-ranking. Also, as the most relevant methods, Yanagi et al. [50, 52] proposed re-ranking methods that receive information only about objects in the target image. Although these methods receive additional information for screening similar candidate images, their retrieval performance heavily depends on the objects in the database, and then the feedback is not always effective.Following the growth of the document information retrieval field [1, 56] , we introduce the DQG for exact TBIR settings. Our approach estimates the most effective questions by learning the target DB without using any labels for the retrieval task. With this procedure, users can reach their desired single image effectively by simply answering the presented questions. 
dqg for exact text-based image retreival: 
 Our approach consists of the following two steps: initial text-based image retrieval and DQG-based screening. An overview of our approach is shown in Fig. 2 . By transferring the knowledge of the other vision and language tasks, our approach learns how to generate adequate questions for the screening in the target DB without using any labels for retrieval. First, we calculate the initial retrieval results by computing the similarities between the query text T and candidate images I 𝑛 (𝑛 = 1, . . . , 𝑁 ; 𝑁 being the number of candidate images). Based on the initial retrieval results, we calculate the features of the target DB using the DQG encoder module G en (•).Then, the calculated DB features are passed into the DQG decoder module G dec (•), and the G dec (•) generates free-form questions for the screening. Next, we receive feedback answers toward the generated question. Here, in the training phase, it takes a lot of costs to manually prepare feedback answers until the training convergence, we obtain the feedback answer from the QA modules by assuming them as users. Finally, the retrieval results are calculated based on the feedback answer.To learn how to generate adequate questions without using question labels for retrieval, we transfer the knowledge of various vision and language modules pre-trained on pretext tasks. As a preparation, we pre-train the image description module I2T(•), the DQG encoder and decoder modules {G en (•), G dec (•)}, and the QA word encoder and decoder modules {A word (•), A dec (•)} with various vision and language tasks. The image description module I2T(•) is pre-trained on the image captioning pretext task [49] using the text-image paired dataset [28] . Also, the DQG and QA modules are pre-trained on the visual question generation and answering pretext tasks [2, 26] using the QA-image paired dataset [16] . Here, the QA-image paired dataset contains 𝐶 dictionary words dic 𝑐 (e.g., "car" and "dog"), and the pre-trained A word (•) can convert these words into 𝐷 word -dimensional features 𝒇 word 𝑐 as follows:EQUATIONEQUATIONThe calculated 𝑭 word is utilized for connecting the question generation and answering modules in the following subsections. We focus on training the DQG decoder module G dec (•) to generate questions that can effectively screen similar but non-target images. Besides, the generated questions should be grammatically correct since we provide the generated questions to users. For guaranteeing the grammatical correctness of the generated questions, we introduce the discriminative module D (•) that can classify the real and generated questions. Specifically, we train the discriminative module D (•) using the real questions 𝑞 real contained in the QA-image paired dataset. The trained D (•) is used for the loss calculation. In the training, the real questions are converted to grammatically incorrect questions 𝑞 gen via word swapping, erasing, or inserting. D (•) is trained for minimizing the binary cross-entropy loss 𝐿 pre as follows:EQUATIONAfter the training, it is expected that D (•) can classify the real and generated questions. In the following sections, we refer the trained discriminative module as D (•). The loss calculation with D (•) is described in Subsec. 3.3.Initial text-based image retrieval. We calculate the initial ranking of the candidate images I 𝑛 from a query text T. Theoretically, Eq.( 4)Eq.( 6)-( 8)Eq.( 9)Eq.( 12)Eq.( 11)Eq.( 10)・・・ ・・・ Answer label ・・・ DB Feature DQG Figure 2:Overview of the proposed approach. At first, we calculate the initial retrieval results by computing the similarities between the query text and candidate images. Based on the calculated initial results, we aggregate the visual features of the candidate images using the QG encoder module G en (•). The aggregated visual features are passed into the QG decoder module G dec (•), and the QG decoder module generates questions for the screening. Next, our approach obtains the feedback answer from the QA modules by assuming them as users. Finally, the retrieval results are calculated based on the feedback answer.since an arbitrary text-based image retrieval method can be used for the first step, we explain our approach with reference to the most basic text-based image retrieval architecture [22] .First, text and image features (𝒇 L ∈ R 𝐷 E and 𝒇 V 𝑛 ∈ R 𝐷 E ) are calculated from T and I 𝑛 , respectively, where 𝐷 E represents the dimension of the embedded features. Using the two trained embedding functions Enc L (•) and Enc V (•), which are provided by the arbitrary conventional text-based image retrieval method, we embed T and I 𝑛 into the shared latent space as follows:𝒇 L = Enc L (T), 𝒇 V 𝑛 = Enc V (I 𝑛 ).Then, the proposed approach calculates the cosine similarities 𝑠 𝑛 between embedded features 𝒇 L and 𝒇 V 𝑛 . We rank the candidate images I 𝑛 as R 𝑘 (𝑘 = 1, . . . , 𝑁 ) in descending order of 𝑠 𝑛 . Namely, R 𝑘 represents the 𝑘th ranked candidate image using T as a query. The initial retrieval results R 𝑘 are provided to the users. Besides, the users can conduct the screening if they are not satisfied with the initial retrieval results R 𝑘 .DQG-based screening. We screen the initial retrieval results R 𝑘 via DQG model. The proposed approach firstly calculates DB features 𝒇 G ∈ R 𝐷 G from R 𝑘 using the DQG encoder module G en (•) as follows:EQUATIONEQUATIONwhere 𝛼 is a hyperparameter that balances the importance of ranking. The calculated DB features 𝒇 G are then passed into the DQG decoder module G dec (•). Afterward, the DQG decoder module G dec (•) calculates the likelihoods of dic 𝑐 for 𝑡-th word (𝑡 = 1, . . . ,𝑇 G ; 𝑇 G being the number of estimated words) 𝒍 𝑡 ∈ R 𝐶 as follows:𝒍 𝑡 = G dec (𝒇 G ).Next, we screen the initial retrieval results by comparing the automatically annotated answer labels in each candidate image and feedback answers from the user for the estimated question. The answer labels a 𝑛 can be obtained via the QA modules {A word (•), A dec (•)}. First, we convert 𝒍 𝑡 to one-hot vector 𝒉 𝑡 ∈ R 𝐶 for propagating the estimated question to the QA modules. In the test phase, for determining the 𝑡-th word, we convert 𝒍 𝑡 to one-hot vector 𝒉 𝑡 via the arg max(•) function. Here, since the arg max(•) function is a nondifferentiable procedure, it cannot be used in the training phase. Therefore, in the training phase, we convert 𝒍 𝑡 to one-hot vector 𝒉 𝑡 via differentiable Gummbel-Softmax-based hard sampling1 [19] . We obtain the answer labels a 𝑛 from 𝒉 𝑡 and the candidate images I 𝑛 as follows:EQUATIONEQUATIONEQUATIONwhere 𝑭 word converts input words into the 𝐷 word -dimensional features (described in Subsec. 3.1). Similarly, we obtain the feedback answer a user targeted toward the inferred question. In the test phase, by extracting the maximum index of the estimated likelihoods 𝒍 𝑡 via the arg max 𝑡 (•) function, the proposed approach obtains the generated question q 𝑡 . The generated question q 𝑡 is then presented to the user, and the user gives the feedback answer a user targeted toward the question. In contrast, in the training phase, it takes a lot of time to manually prepare answers until the training convergence. By considering the QA module as users, we denote the answer label a 𝑛 corresponding to the target image as a user . Finally, we calculate the screening similarities ŝ𝑛 based on the initial similarities 𝑠 𝑛 as follows:EQUATION)where 𝛽 is a hyperparameter that balances the effect of the screening, and sim text (•) is a text similarity function. We rank the candidate images I 𝑛 as R𝑘 in descending order of ŝ𝑛 .To generate questions that can screen similar images in DB, we train the DQG decoder module G dec (•) using pseudo image-query texts pairs prepared for the target DB. Since query text labels are generally not contained in each target DB, we prepare pseudo query text labels based on the candidate images in the target DB. Specifically, we generate texts T psu 𝑛 that represent candidate images I 𝑛 via the pre-trained image description module I2T(•). Namely, T psu 𝑛 and I 𝑛 are pseudo pairs that can be automatically prepared from the target DB.Based on the calculated similarities ŝ𝑛 and the inferred one-hot vector 𝒉 𝑡 , we calculate a loss 𝐿, consisting of two types of loss: retrieval loss 𝐿 rank and gramatical loss 𝐿 D , as follows:EQUATION)where 𝛾 is a hyperparameter that balances the importance of loss. Following the conventional text-based image retrieval methods, for all pseudo query text labels T psu 𝑛 , we calculate the retrieval loss 𝐿 rank as follows:EQUATION)where 𝛿 is a margin hyperparameter and 𝑚 = 1, 2, . . . , 𝑁 . By training the DQG decoder module G dec (•) to minimize the re-ranking loss 𝐿 rank , G dec (•) can generate questions that can screen similar but non-target images.Although the retrieval loss 𝐿 rank helps with the improvement of the retrieval performance, there is no guarantee that the generated questions are grammatically correct for users. Therefore, DQG modules can generate questions that are incorrect for users using only the 𝐿 rank . To avoid such occurrence, we introduce grammatical loss 𝐿 D that can guarantee the grammatical correctness of the generated questions. Specifically, for all generated questions corresponding to T psu 𝑛 , we calculate the grammatical loss 𝐿 D using the trained discriminative module D (•) (described in Subsec. 3.1) as follows:EQUATIONHere, the discriminative module D (•) is trained to output a higher and lower value for the real and non-real questions, respectively. Namely, if the generated questions are similar to the real questions, 𝐿 D becomes lower. By introducing 𝐿 D , it is expected that the generated questions are grammatically correct for users. 
experiments: 
 To evaluate the effectiveness of our DQG-based exact image retrieval, we conducted experiments using two major open datasets and one dataset with similar images. Specifically, we defined and evaluated the following research questions.(1) Whether our approach can enhance the performance of the exact text-based image retrieval methods? (Subsec. 4.2) (2) Whether the questions generated by our approach are grammatically correct or not? (Subsec. 4.3) (3) Whether our approach is effective for DBs with a lot of similar images? (Subsec. 4.4) (4) Whether our approach can effectively screen similar images than the conventional re-ranking approaches? (Subsec. 4.5)Dataset. Following the recent text-based image retrieval methods, we used the following two large-scale datasets with text-image pairs: MSCOCO and Visual Genome. The MSCOCO dataset consists of images and corresponding texts that describe the contents of a paired image. This dataset is adopted by the most recent text-based image retrieval methods. Following the widely used data splits provided by [22] , 123,287 and 5,000 images are used for the pre-training and target DB, respectively. Visual Genome [23] The Visual Genome dataset consists of images and corresponding texts that describe the particular region of the paired image. Following the conventional methods, 75,578 and 32,422 images are respectively used for the target DB [48, 57] . Note that since Visual Genome dataset does not have question and answer labels, we utilized the MSCOCO pretraining dataset for pre-training each module.Implementation details. We compared our approach with some recently proposed text-based image retrieval methods [8, 9, 20, 21, 24, 25, 33, 38] . Note that we implemented all comparative methods based on the open-source codes provided by the authors. Furthermore, to evaluate the effectiveness of our training, we compared the retrieval performance calculated using the DQG module without optimization in Subsecs. 3.2 and 3.3 (hereinafter referred to as Ours w/o opt). Namely, it conducts the re-ranking with the DQG module after the pre-training in Subsec. 3.1. Also, to evaluate the effectiveness of 𝐿 D , our approach without 𝐿 D is used for comparison (hereinafter referred to as Ours w/o 𝐿 D ). In our approach, the DQG, QA, and image description modules are constructed following [26] , [34] , and [10] , respectively. Also, we applied the transformer network for our discriminative model. For the parameters in the equations, we set 𝛾 = 0.3 following the conventional researches and experimentally set 𝛼 = 0.9, 𝛽 = 0.6, and 𝛿 = 0.3. In place of the function sim text (•), we used a cosine similarity function using GloVe [31] word features. Each module is pre-trained following their papers, and our training is conducted until the losses converged. Answer preparation. To evaluate the effectiveness of our approach, answers to the generated questions should be required. However, preparing answers to all generated questions comes at a great cost. Even if we manually prepare the answers to all generated questions, the experimental objectivity and repeatability will not be guaranteed since such manual preparation is always required for considering novel approaches. To ensure the experimental objectivity and repeatability, inspired from the dialog system researches [14, 40] , we prepared answers to the generated questions by assuming the QA model as a user. To avoid overfitting toward the QA module and to ensure experimental objectivity, we use the QA model that has different QA module architectures in the training phase ({A word (•), A dec (•)}). However, using the different model architectures in the training and evaluation, there is no guarantee that the QA model will always answer the presented questions accurately and that its model is closer to the user behavior. Even if we assume the QA model as a user, our approach cannot unfairly identify the target images and does not use any label information.We first evaluate the research question "Whether our approach can further enhance the text-based image retrieval performance ?" by introducing our approach to the conventional text-based image retrieval methods. Mean rank, median rank, and Recall@𝑘(R@𝑘) are used for the evaluation metrics, following the conventional text-based image retrieval methods. R@𝑘 is defined as follows:R@𝑘 = 𝑔 𝑘 𝐽EQUATIONwhere 𝐽 and 𝑔 𝑘 represent the number of queries for evaluation and the number of queries that can rank relevant images in the top-𝑘 retrieval results, respectively. Here, we define a relevant image as an image associated with a query text in datasets. Namely, there is only one paired image in 𝑁 candidate images for each text query. Note that, since there is only one ground truth for each query, the other evaluation metrics (e.g., MAP, NDCG) are not used in the recent text-based image retrieval methods. Experimental results with MSCOCO and Visual Genome datasets are shown in Tables 1 and 2 , respectively. In each table, "Kiros '14 +Ours" reveals results using the method by Kiros '14 and our approach, respectively. We can see that our approach can drastically enhance the retrieval performance of all baseline text-based image retrieval methods as shown in Tables 1 and 2 . Specifically, the enhancement of mean and median ranks reveals that our approach stably and effectively distinguishes similar contents in the target DB. Furthermore, we observe that our approach contributes to the improvement of the retrieval performance compared with "Ours w/o opt." These results mean that the optimization with our loss is effective for screening. Surprisingly, our approach outperforms "Ours w/o 𝐿 D ". Thus, introducing 𝐿 D is also effective for enhancing retrieval performance. From the results obtained after introducing 𝐿 D , we consider that 𝐿 D has the regularization ability to prevent overfitting toward the QA modules. Further analysis of the results can lead to further improvement of the retrieval performance. Additionally, examples of retrieval results are also shown in Fig. 1 . From these results, we confirmed that our approach can improve the retrieval performance of the conventional text-based image retrieval methods in various datasets. Evaluations of our approach are conducted based on two large-scale datasets in Subsec. 4.2. Although the effectiveness of our approach can be verified for these large-scale datasets, whether our approach is effective for DBs with a lot of similar contents (hereinafter referred to as biased-DBs) is not guaranteed. In this subsection, we verify them by conducting experiments on the biased-DBs. To the best of our knowledge, publicly available biased-DBs do not exist. Therefore, we construct an artificially biased-DB by assuming DBs with similar objects are biased-DBs.For constructing the artificially biased-DB, an object label contained in most images of the MSCOCO target DB is calculated, and we reconstructed the MSCOCO target DB so that the images of its DB absolutely contain the object label. The calculated label is "person", and the number of the extracted images is 2, 628. We define these extracted images as a biased-MSCOCO DB. Namely, the images in the biased-MSCOCO DB include similar contents related to "person". We compare the retrieval performance using the biased-MSCOCO DB. Also, the other settings follow Subsec. 4.2.The experimental results are shown in Table 4 . As shown in this table, the same trend as in the two large-scale datasets can be observed in the biased-MSCOCO DB. From these results, we have verified the effectiveness of the proposed approach for a biased-DB.For evaluating the research question "Whether our approach can effectively screen similar but non-target images than the conventional re-ranking approaches ?", we compare our approach with the conventional feedback-based re-ranking approaches. Broadly, comparing the re-ranking approaches that receive different feedback is extremely difficult [35] . Therefore, the experiments were conducted so as to maximize the performance of each comparative method [15, 27, 29, 32, 47, 50, 52] as far as possible. Most conventional approaches re-rank the retrieval results by asking users to select relevant images in the top-ranked images. For realizing the above procedure, in our experiments, we considered images containing the same object labels of the target image as relevant images following the experiments in [50, 52] . In the experiments, at first, the initial retrieval results are computed based on the text-based image retrieval method of PVSE [38] . Next, the initial retrieval results were re-ranked based on each re-ranking approach. Tables 5, 6 and 7 show experimental results on MSCOCO, Visual Genome, and biased-MSCOCO DB. As shown in each table, we can see that our approach improves the retrieval performance of baseline methods compared with the conventional approaches. These results imply that our approach is more effective for enhancing the retrieval performance of the conventional text-based image retrieval methods than the conventional re-ranking approaches. 
conclusions: 
 In this paper, we introduced a novel approach called Database Question Generation (DQG) to enhance the performance of exact text-based image retrieval systems. Our approach learn the appropriate question on the target DB only from those candidate images, facilitating the screening of similar but non-target images. By responding to these generated questions, users can effectively retrieve their desired images, even from queries with limited information. Our experimental results underscore the efficacy of our approach in significantly improving exact text-based image retrieval performance across diverse datasets, confirming its potential in advancing the state-of-the-art in image retrieval methodologies. While our study marks a crucial step forward, our current model and loss architectures represent initial implementations and can benefit from refinement and sophistication. Additionally, in-depth analyses and discussions on the grammatical correctness and semantic coherence of the generated questions would contribute significantly to the robustness of our approach. For future works, we will tackle both architecture improvement and further grammatical correctness verification. 
 
 Appendix: https://pytorch.org/docs/stable/generated/torch.nn.functional.gumbel_softmax. html 


Decision: The authors seem to have done a good job addressing reviewers' concerns during the rebuttal phase, as reflected by a rating increase from the majority of the reviewers:

R1: from borderline reject to weak accept
R2: from borderline reject to weak accept
R3: from borderline accept to weak accept
R4: kept the original weak accept rating

However, for the fairness to other authors, the initial major concerns of the reviewers indicate the lack of clarify on the justification of the contributions and/or lack of technical clarity. 

To summarize, this paper introduces a novel Deep Question Generation (DQG) learning scheme for text-based image retrieval without requiring task-specific labels, demonstrating performance improvements. While the approach shows promise, there are also a few limitations as pointed out by reviewers: 1) insufficient comparative study with SOTA methods in the initial submission, despite the additional experimental results presented during rebuttal, 2) concern about potential overfitting risks (would be nicer to theoretically provide insights into this, or provide cross-dataset validation results.)

Reviews: review: paper_summary: paper_summary: This paper introduces a DB question generation (DQG) model to enhance exact cross-modal image retrieval performance, which learns to generate effective questions that precisely screen similar but non-target images using DB contents information.
strength_weakness: strength_weakness: 1. Motivation and problem formulation is solid, with examples that help readers understand.
2. The theoretical approach is reasonable.
limitations: limitations: 1.The methods cited in the ablation experiments are a bit old. And in the comparison experiments in Table 5, except for RQA, other methods are relatively old，it is not very convincing.
2.The innovation is not obvious, and there is no detail on how to effectively control question quality.
paper_summary: paper_summary: This paper discusses a significant challenge in Text-based Image Retrieval (TBIR) systems: how to enhance the system's performance during precise image retrieval tasks. To address this issue, the paper proposes a novel solution, namely integrating a Question Generation (QG) module. This module generates questions that help the system better understand the user's retrieval intentions, thereby improving the accuracy of the retrieval.
strength_weakness: strength_weakness: 1. **Enhanced User Interaction**: The idea of generating targeted questions to guide users in providing more specific information is intriguing. It has proven effective in the domain of document information retrieval but is relatively rare in image retrieval. This method could also increase user interaction with the system, thus enhancing retrieval precision.

2. **Extensive experimental results and supplementary materials**: Comprehensive experiments and provided materials make this work more convincing.
limitations: limitations: 1. **User Burden**: Although the Question Generation module can help improve retrieval accuracy, it may also increase the user's operational burden. Users need to respond to questions generated by the system, which could lead to a decline in user experience, especially in scenarios where quick retrieval results are required.

2. **User Motivation**: Users have two options: 1) browse N photos and then make a decision (do not use DQG), or 2) answer questions generated by the system (use DQG). The authors should explain the advantages of option 2 over option 1, thereby supporting the motivation for this paper.

3. **Issues with Article Writing**: The methods described in this paper (beyond the initial text-based image retrieval) involve complex structures and processes, such as the DQG-encoder, DQG-decoder, QA-module, and Discriminative module. The inputs and outputs of each module, the insights behind their designs, and specific technical details should be clearly explained. The paper is very brief in introducing these modules, merely describing the pipeline process, making it difficult to understand the design and principles of the methods.

4. **Other Questions**: 1) The initial text-based image retrieval generates N images; how is N determined? 2) The paper mentions the use of a transformer in the discriminative module, but what are the network structures of the other parts? 3) The main role of the DQG decoder is to generate questions; does this module use pre-trained model parameters or is it trained from scratch? 4) Why is grammaticality used to measure the quality of the generated questions in natural language aspects? To make the question easy to understand, grammar is just one aspect; overall fluency is equally important. Have the authors considered using NLP's perplexity metric, which might be more appropriate?

Overall, the ideas presented in this work are intriguing and worth exploring. They might have more interesting applications on mobile devices, such as for precise local image data retrieval, but there are shortcomings in the writing, motivation for use, and other issues mentioned above. This work is interesting and if the concerns above are addressed, I would be glad to raise the score.
paper_summary: paper_summary: This paper introduces a DQG learning scheme without task-specific labels and DQG question generation model, which reach desired images by answering generated questions, further to enhance exact cross-modal image retrieval performance. Experimental results on publicly available datasets show that DOG model improve image retrieval performance. The motivation is relatively novel, and experiments result shows the proposed method is effective.
strength_weakness: strength_weakness: 1. This paper introduces a learning scheme without task-specific labels, which to generate the suitable questions from DB information input.
2. This paper proposes a DQG model to enhance exact cross-modal image retrieval performance, which reach desired images by answering generated questions.
3. The proposed model do experiments on different dataset and compared result with popular methods.
limitations: limitations: 1. How to ensure the quality of the generated question, and is it taken into account when building the dataset?
2. No clear explanation with how to generate suitable questions without any question labels for retrieval tasks.
paper_summary: paper_summary: In this paper, the authors introduced a novel approach called Database Question Generation (DQG) to enhance the performance of exact text-based image retrieval systems. The proposed approach learns the appropriate question on the target DB only from those candidate images facilitating the screening of similar but non-target images. The experimental results demonstrate the efficacy of the approach in significantly improving exact text-based image retrieval performance across diverse datasets.
strength_weakness: strength_weakness: The main strengths of this paper include,
1. the proposed DQG approach for exact text-based image retrieval was developed.
2. the new DQG learning scheme without task-specific labels
3. significant performance improvement
limitations: limitations: Here are some limitations in the text:

1. Lack of Comparative Analysis with Recent Advances: while the paper compares the DQG approach with a few existing methods, it lacks a thorough comparative analysis with the latest state-of-the-art models that utilize newer techniques in recent years for the same task. only DiVE was published in 2023, most of the rest methods were developed in 2019 and 2021.

2. Potential Bias and Overfitting: The method's reliance on pre-trained modules and the absence of a robust validation for the discriminative model might lead to overfitting or biases in question generation. The authors of this paper do not sufficiently discuss measures to mitigate these risks.