<figure><img src="image_1.png" /></figure>

HAL
open science

# PQPP: A Joint Benchmark for Text-to-Image Prompt and Query Performance Prediction

Eduard Poesina, Adriana Valentina Costache, Adrian-Gabriel Chifu, Josiane

Mothe, Radu Tudor Ionescu

► To cite this version:

Eduard Poesina, Adriana Valentina Costache, Adrian-Gabriel Chifu, Josiane Mothe, Radu Tudor Ionescu. PQPP: A Joint Benchmark for Text-to-Image Prompt and Query Performance Prediction. Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR 2025), Jun 2025, Nashville, United States. hal-05149275

HAL Id: hal-05149275

https://hal.science/hal-05149275v1

Submitted on 7 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

Public Domain
# PQPP: A Joint Benchmark for Text-to-Image
## Prompt and Query Performance Prediction

Eduard Poesina¹,*, Adriana Valentina Costache¹,*, Adrian-Gabriel Chifu²,
Josiane Mothe³, Radu Tudor Ionescu¹,*

¹University of Bucharest, ²Aix-Marseille Université, ³Université Toulouse Jean-Jaurès

## Abstract

Text-to-image generation has recently emerged as a viable alternative to text-to-image retrieval, driven by the visually impressive results of generative diffusion models. Although query performance prediction is an active research topic in information retrieval, to the best of our knowledge, there is no prior study that analyzes the difficulty of queries (referred to as prompts) in text-to-image generation, based on human judgments. To this end, we introduce the first dataset of prompts which are manually annotated in terms of image generation performance. Additionally, we extend these evaluations to text-to-image retrieval by collecting manual annotations that represent retrieval performance. We thus establish the first joint benchmark for prompt and query performance prediction (PQPP) across both tasks, comprising over 10K queries. Our benchmark enables (i) the comparative assessment of prompt/query difficulty in both image generation and image retrieval, and (ii) the evaluation of prompt/query performance predictors addressing both generation and retrieval. We evaluate several pre- and post-generation/retrieval performance predictors, thus providing competitive baselines for future research. Our benchmark and code are publicly available at https://github.com/Eduard6421/PQPP.

## 1. Introduction

In recent years, more and more people embraced the use of large language models (LLMs) instead of traditional search engines [1, 7, 62]. The advent of generative diffusion models [9, 18, 33, 40] capable of generating high-quality and realistic images triggered a similar trend in text-to-image retrieval. This paradigm shift in information retrieval (IR) calls for an extensive exploration of research topics related to generative models. One is query performance prediction (QPP), which refers to the process of estimating the performance level of an IR system for a given query [8], this being an active research topic in IR [13, 16, 20, 31, 36]. In

the context of generative AI for text-to-image generation, if a prompt is predicted to be difficult, the system could initiate a conversation to refine the prompt in order to overcome the difficulty and improve the final output. Moreover, the system could indicate to the user its inability to provide a satisfactory image, or it could give positive feedback to the user when the prompt is predicted as easy. In general, when a prompt/query is predicted as difficult for a generation/retrieval system, additional processes can be activated, such as the automatic reformulation, the allocation of extra resources, or the addition of pre- or post-processing. We discuss further applications and concrete use cases of prompt performance prediction in Appendix 8.

With the growing popularity of generative models and retrieval augmented generation, understanding prompt effectiveness in both generation and retrieval becomes increasingly essential. While text-to-image generation and retrieval share common elements, they differ fundamentally in terms of task requirements and success criteria. In text-to-image generation, prompts must guide models to produce images that not only capture the specified elements, but also meet aesthetic and contextual expectations. Conversely, in retrieval, the query role is to retrieve pre-existing images that match the query as closely as possible within a given dataset, where success depends on how well the query aligns with the available data. Studying both tasks together is critical. Examining prompt/query difficulty across generation and retrieval allows us to determine whether the factors that make a prompt challenging in generation (e.g. complexity, specificity, ambiguity) are the same as those in retrieval. To the best of our knowledge, there is no prior study that jointly analyzes the difficulty of prompts/queries in text-to-image generation and retrieval using human relevance judgments. In this study, we introduce the first dataset of prompts/queries which are manually annotated in terms of both image generation performance and image retrieval performance. Additionally, we provide baseline results with various pre- and post-performance predictors, setting a standard for future multimodal QPP research.

Our new Prompt and Query Performance Prediction

<footnote>*Corresp. author: raducu.ionescu@gmail.com. °Equal contrib.</footnote>
<figure><img src="image_2.png" /><figcaption>Figure 1. We select a set of 10K captions from MS COCO [30] using k-means clustering, which are further merged with prompts from DrawBench [41]. Next, we collect human relevance judgments in two scenarios: image generation and image retrieval. For each prompt/query, we generate images with two diffusion models (Stable Diffusion XL [35] and GLIDE [33]) and retrieve images from MS COCO with two vision-language models (CLIP [37] and BLIP-2 [27]). Based on the collected relevance judgments, we score each prompt/query in terms of generation and retrieval performance, respectively. Finally, we train and evaluate multiple prompt/query performance predictors on the proposed benchmark. Best viewed in color.</figcaption></figure>

(PQPP) benchmark comprises 10,200 text samples that serve as prompts for text-to-image generation and queries from text-to-image retrieval. To obtain representative prompts/queries for both generation and retrieval, we combine the 200 prompts from DrawBench [41] with 10,000 diverse captions from the Microsoft Common Objects in Context (MS COCO) dataset [30]. While DrawBench is a recently proposed set of prompts that aims to test the capability of generative models in diverse scenarios, MS COCO is a well-established data collection that allows cross-task studies and is cost-effective. MS COCO provides high-quality image annotations, as well as a wide range of everyday scenes and objects, being widely used across several computer vision tasks, e.g. object detection, object segmentation, and image captioning. Notably, with the advent of text-to-image generation, MS COCO has also been employed in many works for the text-to-image generation task [5, 24, 41, 55]. To select a representative and diverse set of prompts from over 590K captions in MS COCO, we employ a k-means clustering algorithm on the embedding space of a sentence BERT model [47], where $k = 10,000$.

To analyze prompt performance in both generation and retrieval, we examine the same set of 10,200 prompts across two generative models and two retrieval models (see Figure 1). For the generation task, we use SDXL [35] and

GLIDE [33], each generating two images per prompt. Both models are state-of-the-art diffusion models, yet they rely on different designs, which leads to variability in terms of image quality and relevance. Next, we collect over 247K relevance judgments from 147 human annotators for 40,800 images generated by the chosen diffusion models. For the retrieval task, we employ two state-of-the-art vision-language models, CLIP [37] and BLIP-2 [27], to retrieve images for the 10,200 queries. These models use different architectures and training data, impacting their retrieval performance for the same query. We start from preliminary relevance judgments determined via the pre-trained sentence BERT [47] applied on captions. If a caption from MS COCO is similar to a query in the BERT embedding space, the image corresponding to the respective caption is added to the set of images that are potentially relevant to the respective query. Next, we collect 1.39M ground-truth relevance judgments from 91 human evaluators to annotate the 10,200 queries. In summary, we collect over 1.6M annotations to estimate the difficulty score of each query included in PQPP, from both generation and retrieval perspectives.

We conduct preliminary experiments to compare prompt/query difficulty in image generation vs. image retrieval. Our findings show that there is a very low correlation between the two tasks (see Section 4), which justi-
fies the need to study the novel task of prompt performance prediction in image generation. We also carry out experiments with multiple pre- and post-generation/retrieval performance predictors, providing a set of competitive baselines for future research. We find that a strong supervised pre-generation/retrieval predictor is a worthy competitor for post-generation/retrieval predictors in both text-to-image generation and retrieval. To further demonstrate the usefulness of PQPP, we carry out cross-model, cross-dataset and cross-task experiments, evaluating the generalization capabilities of performance predictors in challenging settings.

In summary, our contribution is threefold:

* We propose the first joint benchmark for prompt and query performance prediction.
* We collect over 1.6M relevance judgments from human annotators to score a total of 10,200 queries in terms of generation and retrieval performance.
* We experiment with multiple pre-generation/retrieval and post-generation/retrieval performance predictors to obtain competitive results, which serve as baselines for future work.

## 2. Related Work

Studies on QPP initially focused on textual ad hoc retrieval, where both pre- and post-retrieval predictors were considered for sparse retrieval models [11, 23, 32, 56]. Some recent studies investigated dense (neural network) retrieval models [3, 13, 14, 20, 57], as well as diverse tasks, such as conversational search [21, 31, 42]. We discuss QPP in textual ad hoc retrieval in Appendix 9.

**QPP in text-to-image retrieval/generation.** QPP in a multimodal (e.g. text-to-image) context is a relatively new research area [6, 26, 34, 50, 52]. The exploration of QPP in the context of text-to-image retrieval and generation is gaining significant interest, particularly with the rapid advancements in generative methods [9]. This research domain, distinct for its multimodal nature, aims to enhance the prediction of text query effectiveness for retrieving relevant images. Initial studies, such as those of Xing et al. [53] and Tian et al. [48, 49], have laid the groundwork by exploring query difficulty prediction in image retrieval, utilizing machine learning algorithms and assessing the utility of various features and information sources.

Further contributions by Li et al. [28] delve into the challenges of estimating query difficulty with unigram language models and visual word verification, highlighting the complexities of aligning text-based queries with visual data. Meanwhile, the development of a self-supervised framework for Content-Based Image Retrieval (CBIR) systems, addressing the scarcity of labeled data through synthetic data and rank-based feature training, marks a significant advancement [50]. Recent efforts, such as those of Wu et al. [52] and Pavlichenko and Ustalov [34], focus on integrating

human feedback into text-to-image models to refine prediction accuracy and enhance the visual appeal of generated images. Kumari et al. [26] introduced an approach to concept ablation in text-to-image synthesis, aiming to selectively prevent the generation of specific concepts.

The closest work to our research is that of Bizzozzero et al. [6], which introduced the concept of *prompt performance prediction*. The authors assessed the effectiveness of prompts in generating images. However, in their study, the ground-truth is automatically generated, which may introduce a significant bias in the evaluation. In contrast, we are the first to explore the prompt performance prediction task with respect to human relevance judgments. In addition, we introduce a benchmark that provides performance measurements for the same set of prompts/queries in both generation and retrieval, enabling the comparative study of QPP across both tasks. Another novel contribution of our work is the study of post-generation prompt performance predictors.

**Human feedback in text-to-image generation.** Human feedback can provide fundamental insights for text-to-image generators, which can harness the provided annotations to increase the quality of the generated images. To this end, several works [25, 29, 52, 54] collected human feedback for generated images and developed novel benchmarks to train and evaluate generative models. Kirstain et al. [25] obtained preference annotations by asking humans to choose an image from a pair of generated images. Wu et al. [52] also collected a dataset of human preferences on generated images. The dataset was used to train a classifier to output a human preference score. The classifier was further used to fine-tune Stable Diffusion. Xu et al. [54] asked workers to rank several generated images and rate them based on their quality. The annotations were used to train a reward model, which provided feedback for training diffusion models. Unlike previous studies, Liang et al. [29] collected rich annotations consisting of fine-grained scores, implausible regions and misaligned words. The annotations were further used to improve image generation.

While we recognize the outstanding merits of previous studies that collected human feedback for improving text-to-image generation [25, 29, 52, 54], we would like to highlight that the focus of our work is different. To our knowledge, our work represents the first attempt to study the task of predicting the performance of text-to-image generators on a given prompt.

## 3. Proposed Benchmark

To develop our novel Prompt and Query Performance Prediction (PQPP) benchmark, we first harness the images and captions from the MS COCO [30] training set, which comprises approximately 118K images. Each of these images is accompanied by 5 to 7 descriptive captions, providing a
<figure><img src="image_3.png" /><figcaption>Figure 2. An example showing the annotation interface for a random prompt and the images associated with the respective prompt. For each image, the annotator can select one of the following options: high relevance, low relevance, no relevance and unrealistic. The relevance judgments of an annotator are shown for illustrative purposes. The locations of images are randomly generated, each time they are displayed. Hence, the annotators do not see the images in the same order, which eliminates positional biases. Best viewed in color.</figcaption></figure>

rich context for our study.

To establish a unified foundation for text-to-image generation and retrieval, we select a subset of 10,000 captions to be used across tasks either as *queries* for text-to-image retrieval, or as *prompts* for text-to-image generation. We aim for a wide variety of captions that can be clearly associated with images. The selection process entails identifying the most relevant caption for each image (from the small set of captions available in MS COCO for the respective image), where the (caption, image) similarity is measured via the cosine similarity in the CLIP embedding space [37]. This reduces the number of captions to about 118K. The resulting captions are further processed by a sentence transformer based on BERT [47] to extract sentence embeddings. Aiming to enhance the distinctiveness of our final set of captions, we next apply the k-means clustering algorithm based on k-means++ initialization on the extracted embeddings. For each cluster, we select the nearest caption to the center of each cluster and include it in our final set of captions. We set $k=10,000$ to obtain 10K prompts/queries. Although MS COCO captions have been used before to train text-to-image generators [5, 24, 41, 55], prompts written by users can often be more descriptive [10]. To this end, we add all the prompts from DrawBench [41] to our set, resulting in a combined set of 10,200 prompts/queries. We next describe the procedures for collecting human relevance judgments on both tasks.

## 3.1. Prompt Performance Assessment

**Generative models.** To generate images for our set of prompts, we employ the well-established SDXL [35] and GLIDE [33] generative models. For the first model, we choose the Stable Diffusion XL (base-1.0) variant, which produces images with a resolution of 1024 × 1024 pixels. In contrast, GLIDE initially produces images at a resolution of 64 × 64 pixels. The images are subsequently upsampled to a resolution of 256 × 256 pixels using the upsampling model integrated in GLIDE. For each prompt, we generate two images with SDXL and two images with GLIDE.

**Control and calibration.** Along with the four generated

images, we include the image from MS COCO associated with each prompt into the annotation process. The ground-truth image is included to calibrate user annotations (assuming that the ground-truth image should be labeled as highly relevant) or to exclude annotators that are not seriously engaged in the annotation process. To further ensure the quality of human relevance judgments, we implement a verification process based on a control set of 100 prompts. The control prompts are independently annotated by three authors of this study, following the same protocol as the other annotators. The Fleiss' $\kappa$ coefficient among the control annotators is 0.55. While running the full-scale annotation process, a control prompt is randomly inserted among every five prompts. This allows us to set a minimum threshold for the Cohen's $\kappa$ coefficient to accept the relevance judgments of an annotator. The relevance judgments of an annotator are included into the study if the respective annotator (i) has at least a moderate agreement<fnref n="1" /> ($\kappa$ > 0.4) with each annotator in the control group, and (ii) has annotated more than 95% of the ground-truth images as highly relevant.

**Annotation interface.** We use a custom web interface to collect relevance judgments. The prompts are presented in a random order to each annotator. Each prompt is accompanied by four generated images and one ground-truth image, which are mixed and displayed in a random order, as shown in Figure 2. For each image, annotators are given a choice among four evaluative categories: high relevance, low relevance, no relevance, and unrealistic. Participants receive thorough instructions (summarized below) before starting the annotation process. The ratings are to be assigned as follows: *high relevance* (score 2) – the image depicts over half of the concepts mentioned in the prompt; *low relevance* (score 1) – the image captures at least one concept, but fewer than half; *no relevance* (score 0) – the image is unrelated, yet realistic; *unrealistic* (score -1) – the image exhibits notable generation artifacts.

**Annotation process.** Our aim is to collect at least three annotations per image. In order to collect the required number

<footnote n="1">¹Wiki article on Cohen's κ: Interpreting Magnitude</footnote>
<table><caption>Table 1. Statistics about the annotators enrolled in the annotation process for generated images.</caption><thead><tr><th>Statistic</th><th>Min</th><th>Mean</th><th>Max</th></tr></thead><tbody><tr><td>#annotations per person</td><td>30</td><td>1,681</td><td>15,845</td></tr><tr><td>Fleiss' κ</td><td>0.41</td><td>0.54</td><td>1.00</td></tr></tbody></table>

of annotations (10,200 prompts × 5 images × 3 annotations = 153,000 annotations in total), we recruit 173 annotators. Based on our selection criteria, 26 annotators are excluded from the process. This leaves us with 147 valid annotators. Given the asynchronous nature of the annotation process, several prompts ended up having more than three annotations, leading to a total of 247,050 annotations. When a prompt has more than three relevance judgments, we keep the annotations provided by the top three annotators with the highest Cohen's κ coefficients (with respect to the control prompts). Some statistics about the enrolled annotators are presented in Table 1. Notably, the Fleiss' κ coefficient computed across all annotators is consistent with that of the control annotators. Upon excluding the annotations corresponding to the ground-truth images, we find that most images are voted as highly relevant (see Figure 7 in Appendix 10), confirming that SDXL and GLIDE generally produce relevant results.

**Measuring prompt performance.** To derive the final prompt performance in image generation, we first map the relevance categories to numerical values, as follows: *high relevance* is mapped to 2, *low relevance* to 1, *no relevance* to 0, and *unrealistic* to -1. To exclude outlier annotations, we group the four relevance categories into two high-level categories. The first category, combining the *high relevance* and *low relevance* annotations, represents images that are at least somewhat relevant to the prompt. The second category, combining the *unrealistic* and *no relevance* labels, represents images that are not acceptable for the given prompt, either because they are irrelevant or unrealistic. We employ a majority voting mechanism on the high-level categories to decide if a generated image is either *relevant* or *irrelevant*. The Fleiss' κ coefficient for these categories is 0.75, suggesting that the annotations are more consistent at this coarse level. Since there are only two high level categories and three annotations per image, there is no need to break ties (a majority always exists). The majority voting is performed to rule out outlier annotations. The final relevance of an image is given by averaging the scores (between -1 and 2) associated with the votes forming the majority. The performance of a prompt is given by the average relevance score computed across the generated images. We further refer to the resulting score as *human-based prompt performance* (HBPP).

More details about the prompt performance assessment are discussed in Appendix 10.

## 3.2. Query Performance Assessment

**Retrieval models.** For text-to-image retrieval, we employ two distinct vision-language models: CLIP [37] and BLIP-2 [27]. For CLIP, we select the ViT-Base [19] architecture with patches of 32 × 32 pixels. For BLIP-2, we choose the ViT-Large backbone. These models are pre-trained on natural images, which makes them suitable for image retrieval on MS COCO.

**Annotation process.** For the retrieval setting, we devise a semi-automatic labeling process to generate reference (ground-truth) relevance judgments for the 10,200 queries. We first employ an automatic process to restrict the number of retrieved images to 2,000 per query. We exploit the structure of the MS COCO dataset based on (image, caption) pairs to generate preliminary relevance judgments using sentence BERT [47]. More specifically, we compute the cosine similarity in the embedding space of sentence BERT between each query in PQPP and each caption in MS COCO. Based on a preliminary exploratory data analysis, we set the cosine similarity threshold to 0.7 to determine a comprehensive set of potentially relevant results. The image corresponding to each caption that has the cosine similarity with a query higher than 0.7 is added to the preliminary set of relevant results for the respective query. For some short and generic text queries, the preliminary set may contain thousands of potentially relevant images. For such queries, we refine the results ranked below 1,000 using a bag-of-words representation. More specifically, a low-rank image is kept only if the bag-of-words representation of the query is included in the bag-of-words representation of the aggregated captions of the respective image. All queries are limited to 2,000 images in the preliminary set that undergoes manual labeling. The preliminary steps described above generate a total of 1,393,363 images, which are further subject to rigorous manual review. There are 100 annotators involved in the manual annotation of the potentially relevant images. The annotators are asked to label each image as relevant or irrelevant to the corresponding query. Each image is annotated by two annotators. An image is kept in the relevant set if it is voted as relevant by one annotator. Two of the evaluators, who are also the main authors of this paper, annotated a set of 4,200 queries. These 4,200 queries are used as control queries for the other enrolled annotators. The remaining queries were randomly divided into batches of 50 queries. In each batch, there are 5 control queries, which are used to exclude annotators that provide poor relevance judgments. There are 98 human evaluators who annotated between 1 and 4 batches. We employed the $F_1$ measure on relevant images for control queries to estimate the quality of the relevance judgments, and set a threshold of 0.4 to accept annotations. There are 7 annotators who were excluded from the annotation process based on the considered threshold. For the remaining annotators, we obtain a mean
<table><caption>Table 2. Pearson and Kendall τ correlation coefficients between the performance levels measured in image generation vs. image retrieval. According to a Student's t-test, the results marked with † and ‡ are significantly better than the random chance baseline at p-values 0.01 and 0.001, respectively.</caption><thead><tr><td>Metric</td><td>Pearson</td><td>Kendall</td></tr></thead><tbody><tr><td>HBPP vs. P@10</td><td>0.135<sup>‡</sup></td><td>0.093<sup>‡</sup></td></tr><tr><td>HBPP vs. RR</td><td>0.072<sup>‡</sup></td><td>0.048<sup>†</sup></td></tr><tr><td>P@10 vs. RR</td><td>0.560<sup>‡</sup></td><td>0.512<sup>‡</sup></td></tr></tbody></table>

$F_1$ score of 0.727. The minimum $F_1$ score is 0.447, which is significantly higher than the $F_1$ score of 0.150 of the random chance baseline. The manual annotation process reduced the total number of relevant images to 530,360. In other words, the annotators removed almost two thirds of the originally retrieved images.

**Measuring query performance.** To determine the performance level of system for a given query, we employ two alternative measures of retrieval effectiveness, namely the precision for the top $n$ retrieved results ($P@n$) and the reciprocal rank (RR). The precision@n is the ratio between the number of true positive images and $n$. Since $P@10$ is often used in text QPP [56], we adopt the same measure and set $n = 10$ for our benchmark. The reciprocal rank of a query is given by the ratio between 1 and the rank of the first relevant result. We estimate the $P@10$ and RR measures for both CLIP and BLIP-2.

### 3.3. Evaluation Protocol

We divide the annotated prompts/queries into 6,080 for training, 2,040 for validation, and 2,080 for testing. To evaluate performance predictors, we measure the Pearson and Kendall $τ$ correlation coefficients between the predicted and the ground-truth performance levels of all test queries, following conventional evaluation procedures in text [56, 60] and image [36] QPP. Furthermore, we test the significance of the results with respect to the random chance baseline using Student's t-testing [39].

## 4. Generation vs. Retrieval Performance

For each query, our benchmark provides performance measurements in both generation and retrieval settings. Taking advantage of the structure of the PQPP benchmark, we next analyze the correlation between the studied tasks: prompt performance prediction (in text-to-image generation) and query performance prediction (in text-to-image retrieval). We present the correlation results in Table 2. Although the correlations are statistically significant, the empirical analysis reveals surprisingly low correlations between the ground-truth performance measurements for the generative and retrieval tasks. This observation indicates that the tasks are rather orthogonal, confirming that image generation requires the development of dedicated prompt performance predictors.

<figure><img src="image_4.png" /><figcaption>Figure 3. Representative prompts/queries that exhibit high or low performance in text-to-image generation (on the horizontal axis) and text-to-image retrieval (on the vertical axis).</figcaption></figure>

Aside from the quantitative results presented in Table 2, we manually inspect prompts/queries from the following four categories: (1) high performance in both generation and retrieval; (2) high performance in generation, low performance in retrieval; (3) low performance in generation, high performance in retrieval; and (4) low performance in both generation and retrieval. We present illustrative captions of the four categories in Figure 3. For the first category, the queries usually refer to an animate object performing an action in a location. There are no or few attributes to describe the objects or the location, which increases the likelihood of finding many matches during retrieval, and reduces the constraints during generation. The queries from the second category include inanimate objects with specific descriptions. Generative models seem to handle many specific attributes rather well, but the retrieval models have a hard time finding images that match the specific descriptions. The third category comprises queries about a person performing an action specific to some sport. The MS COCO database contains many images of people playing sports, which makes the retrieval task pretty easy. However, generative models struggle to generate a person in a specific pose, performing a certain action that typically involves interacting with an object, e.g. tennis racket or baseball bat. For the fourth category, representative queries refer to groups of people, e.g. two men, performing specific actions in specific locations, which are rather uncommon. Queries in this category can also refer to uncommon objects with specific attributes, which can even be misspelled, e.g. "gel wrist wrest". Prompts/queries describing rare situations/objects are difficult for both generative and retrieval models. On the one hand, there is some common ground between prompt/query performance in generation and retrieval, as confirmed by the examples from the first and fourth categories. On the other hand, the examples from the second and third categories indicate that there are tasks.
specific characteristics placing prompt and query performance prediction at opposite poles. Therefore, we conclude that the task of prompt performance prediction merits further investigation, motivating the utility of our novel benchmark.

# 5. Predictors

We briefly present the chosen performance predictors below. We provide details about implementation choices and hyperparameter tuning in Appendix 11.

## 5.1. Pre-generation/retrieval Predictors

**Basic text predictors.** Building on linguistic features to predict query difficulty [32], we extract a comprehensive suite of pre-retrieval linguistic indicators: the diversity of concepts (total WordNet synsets per prompt/query), lexical density (number of words per prompt/query), morphological complexity (average word length), and the frequency of specific grammatical structures (proper nouns, acronyms, numerals, conjunctions and prepositions). We test a wide variety of basic predictors, but we only report the results of the top two predictors, namely the number of synsets (#synsets) and the number of words (#words). Results with the other basic text predictors are reported in Appendix 12.

**Fine-tuned BERT.** We explore the potential of fine-tuning the BERT model [17] as a pre-retrieval performance predictor. We select the *base* architecture based on cased inputs, since the queries contain named entities. The fine-tuning process involves attaching a custom regression head to the pre-trained BERT backbone, which learns to predict the performance of prompts/queries in image generation and retrieval, respectively.

## 5.2. Post-generation/retrieval Predictors

**Fine-tuned CLIP.** Our first post-generation/retrieval predictor is based on fine-tuning a CLIP model on (query, image) pairs. We use Long-CLIP [58] with a ViT-B/32 backbone. We attach a regression head for the image generation scenario, and a binary classification head for the image retrieval scenario. The utilization of CLIP-based embeddings is aimed at leveraging the model's capacity to jointly represent text and image modalities in a single latent space. The fine-tuned CLIP learns to predict the relevance judgment of each generated or retrieved image for a given prompt/query. The performance of the prompt/query is then predicted by aggregating the predicted relevance judgments.

**Correlation-based CNN.** Following the work of Sun et al. [46], we employ a convolutional neural network (CNN) trained on the set of generated or retrieved images, respectively. The CNN model takes a correlation matrix between all image pairs as input. The correlation of an (image, image) pair is given by the cosine similarity between the (pre-trained) CLIP embeddings of the respective images.

<figure><img src="image_5.png" /><figcaption>Figure 4. t-SNE visualization of the test prompts embedded in the latent space of the BERT predictor fine-tuned on image generation with GLIDE. The ground-truth HBPP performance is encoded via a color map from green (high) to red (low). The visualization confirms that the fine-tuned BERT predictor learns a meaningful representation of the prompts. Best viewed in color.</figcaption></figure>

# 6. Experiments and Results

**Main results.** In Table 3, we report the results of the best performing predictors for both generative and retrieval tasks (see Table 5 for additional predictors). For image generation, the fine-tuned CLIP shows the highest correlation with actual performance, achieving a Pearson coefficient of 0.649 for GLIDE. Interestingly, the second-best predictor is the fine-tuned BERT, which does not even see the generated/retrieved images. While there is a clear ranking of the predictors for the generative task, the correlations reported on the retrieval task do not point towards a clear winner. The top correlations are divided among the fine-tuned BERT, the fine-tuned CLIP and the correlation CNN, respectively. Moreover, we find that it is generally easier to predict P@10 than RR. Considering the overall results, an interesting finding of our study is that the pre-retrieval fine-tuned BERT is a worthy competitor for the post-retrieval predictors, being consistently better than the correlation CNN and even surpassing the fine-tuned CLIP in a few cases. In addition, our findings suggest that simple pre-retrieval features, e.g. #synsets or #words, are not able to capture the complexity of prompts/queries in text-to-image tasks. Nevertheless, statistical testing shows that all the supervised predictors are always significantly better than random chance, indicating that the proposed benchmark is approachable.

Figure 4 illustrates a t-SNE visualization of the test prompts embedded in the latent space of the BERT predictor fine-tuned on the generation task. We observe that the learned embedding correlates well with the ground-truth HBPP values, explaining the high accuracy of the fine-tuned BERT predictor on the generation task. An analogous visualization for the retrieval task is shown in Figure 8.

**Cross-model results.** While the common approach in QPP literature is to predict the performance of a system for a given query, our benchmark enables the assessment of performance predictors across models. With the rapid advancements in neural architectures, novel generative and retrieval
<table><caption>Table 3. Results of the prompt/query performance predictors for the generative and retrieval settings on the PQPP test set. On the generative task, we report the correlation of the predicted value with the HBPP performance of SDXL and GLIDE, respectively. On the retrieval task, the correlation is computed for the P@10 and RR scores of CLIP and BLIP-2, respectively. For each task and model, the highest correlation is highlighted in bold. According to a Student's t-test, the results marked with † and ‡ are significantly better than the random chance baseline at p-values 0.01 and 0.001, respectively. Additional predictors are reported in Appendix 12.</caption><thead><tr><th rowspan="3">Predictor Type</th><th rowspan="3">Predictor Name</th><th colspan="4">Generative Task</th><th colspan="8">Retrieval Task</th></tr><tr><th colspan="2">GLIDE</th><th colspan="2">SDXL</th><th colspan="4">CLIP</th><th colspan="4">BLIP-2</th></tr><tr><th colspan="2">HBPP</th><th colspan="2">HBPP</th><th colspan="2">P@10</th><th colspan="2">RR</th><th colspan="2">P@10</th><th colspan="2">RR</th></tr><tr><th></th><th></th><th>Pearson</th><th>Kendall</th><th>Pearson</th><th>Kendall</th><th>Pearson</th><th>Kendall</th><th>Pearson</th><th>Kendall</th><th>Pearson</th><th>Kendall</th><th>Pearson</th><th>Kendall</th></tr></thead><tbody><tr><td rowspan="3">Pre-</td><td>#synsets</td><td>-0.112<sup>‡</sup></td><td>-0.076<sup>‡</sup></td><td>-0.087<sup>‡</sup></td><td>-0.080<sup>‡</sup></td><td>-0.110<sup>†</sup></td><td>-0.058<sup>‡</sup></td><td>-0.034</td><td>-0.012</td><td>-0.115<sup>‡</sup></td><td>-0.070<sup>‡</sup></td><td>-0.038</td><td>-0.010</td></tr><tr><td>#words</td><td>-0.090<sup>†</sup></td><td>-0.084<sup>‡</sup></td><td>-0.105<sup>‡</sup></td><td>-0.109<sup>‡</sup></td><td>-0.133<sup>‡</sup></td><td>-0.104<sup>‡</sup></td><td>-0.035</td><td>-0.026</td><td>-0.175<sup>‡</sup></td><td>-0.136<sup>‡</sup></td><td>-0.038</td><td>-0.015</td></tr><tr><td>Fine-tuned BERT</td><td>0.566<sup>‡</sup></td><td>0.406<sup>‡</sup></td><td>0.281<sup>‡</sup></td><td>0.232<sup>‡</sup></td><td>0.451<sup>‡</sup></td><td>0.277<sup>‡</sup></td><td><strong>0.221<sup>‡</sup></strong></td><td><strong>0.176<sup>‡</sup></strong></td><td><strong>0.511<sup>‡</sup></strong></td><td>0.328<sup>‡</sup></td><td>0.168<sup>‡</sup></td><td>0.139<sup>‡</sup></td></tr><tr><td rowspan="2">Post-</td><td>Fine-tuned CLIP</td><td><strong>0.649<sup>‡</sup></strong></td><td><strong>0.474<sup>‡</sup></strong></td><td><strong>0.380<sup>‡</sup></strong></td><td><strong>0.246<sup>‡</sup></strong></td><td><strong>0.473<sup>‡</sup></strong></td><td><strong>0.299<sup>‡</sup></strong></td><td>0.200<sup>‡</sup></td><td>0.149<sup>‡</sup></td><td>0.498<sup>‡</sup></td><td><strong>0.358<sup>‡</sup></strong></td><td>0.166<sup>‡</sup></td><td>0.150<sup>‡</sup></td></tr><tr><td>Correlation CNN</td><td>0.548<sup>‡</sup></td><td>0.393<sup>‡</sup></td><td>0.159<sup>‡</sup></td><td>0.107<sup>‡</sup></td><td>0.270<sup>‡</sup></td><td>0.186<sup>‡</sup></td><td>0.189<sup>‡</sup></td><td>0.162<sup>‡</sup></td><td>0.159<sup>‡</sup></td><td>0.133<sup>‡</sup></td><td><strong>0.206<sup>‡</sup></strong></td><td><strong>0.158<sup>‡</sup></strong></td></tr></tbody></table>

<table><caption>Table 4. Cross-model results of the prompt/query performance predictors for the generative and retrieval settings on the PQPP test set. On the generative task, we report the correlation results for two cross-model settings: SDXL→GLIDE and GLIDE→SDXL. On the retrieval task, we report the correlations for CLIP→BLIP-2 and BLIP-2→CLIP, respectively. For each task, the highest correlation is highlighted in bold. According to a Student's t-test, the results marked with † and ‡ are significantly better than the random chance baseline at p-values 0.01 and 0.001, respectively.</caption><thead><tr><th rowspan="3">Predictor Type</th><th rowspan="3">Predictor Name</th><th colspan="4">Generative Task</th><th colspan="8">Retrieval Task</th></tr><tr><th colspan="2">SDXL→GLIDE</th><th colspan="2">GLIDE→SDXL</th><th colspan="4">BLIP-2→CLIP</th><th colspan="4">CLIP→BLIP-2</th></tr><tr><th colspan="2">HBPP</th><th colspan="2">HBPP</th><th colspan="2">P@10</th><th colspan="2">RR</th><th colspan="2">P@10</th><th colspan="2">RR</th></tr><tr><th></th><th></th><th>Pearson</th><th>Kendall</th><th>Pearson</th><th>Kendall</th><th>Pearson</th><th>Kendall</th><th>Pearson</th><th>Kendall</th><th>Pearson</th><th>Kendall</th><th>Pearson</th><th>Kendall</th></tr></thead><tbody><tr><td rowspan="2">Pre-</td><td>Fine-tuned BERT</td><td>0.165<sup>‡</sup></td><td>0.128<sup>‡</sup></td><td>0.150<sup>‡</sup></td><td>0.087<sup>†</sup></td><td>0.420<sup>‡</sup></td><td>0.264<sup>‡</sup></td><td>0.134<sup>†</sup></td><td>0.112<sup>‡</sup></td><td>0.431<sup>‡</sup></td><td>0.270<sup>‡</sup></td><td><strong>0.165<sup>‡</sup></strong></td><td>0.133<sup>‡</sup></td></tr><tr><td>Fine-tuned CLIP</td><td><strong>0.256<sup>‡</sup></strong></td><td><strong>0.186<sup>‡</sup></strong></td><td><strong>0.179<sup>‡</sup></strong></td><td><strong>0.089<sup>†</sup></strong></td><td><strong>0.449<sup>‡</sup></strong></td><td><strong>0.291<sup>‡</sup></strong></td><td>0.121<sup>†</sup></td><td>0.122<sup>‡</sup></td><td><strong>0.453<sup>‡</sup></strong></td><td><strong>0.337<sup>‡</sup></strong></td><td>0.151<sup>‡</sup></td><td><strong>0.154<sup>‡</sup></strong></td></tr><tr><td rowspan="2">Post-</td><td>Fine-tuned CLIP</td><td>0.131<sup>‡</sup></td><td>0.078<sup>‡</sup></td><td>0.096<sup>†</sup></td><td>0.020</td><td>0.155<sup>‡</sup></td><td>0.130<sup>‡</sup></td><td><strong>0.206<sup>‡</sup></strong></td><td><strong>0.162<sup>‡</sup></strong></td><td>0.228<sup>‡</sup></td><td>0.167<sup>‡</sup></td><td>0.155<sup>‡</sup></td><td>0.130<sup>‡</sup></td></tr><tr><td>Correlation CNN</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr></tbody></table>

models constantly emerge. Therefore, testing the generalization capacity of performance predictors across models is of utter importance. To this end, we report cross-model results on PQPP in Table 4. We include only supervised predictors in this evaluation, since basic (unsupervised) pre-retrieval predictors do not depend on training data. On the one hand, we observe that performance predictors exhibit large score drops when tested across generative models, although most of their correlations remain statistically significant. On the other hand, the predictors seem to be more consistent when tested across retrieval models. In a few cases, the correlation scores are even higher when testing is performed across models. For instance, the Pearson correlation of the correlation CNN for the RR measure is 0.189 when the predictor is trained and tested on CLIP, but the correlation grows to 0.206 when the training is performed on BLIP-2 and the evaluation is performed on CLIP.

**Additional results.** In Appendix 12, we present the complete set of quantitative experiments, including cross-dataset experiments, cross-task experiments, experiments on individual datasets, and experiments with automatic assessment measures. In Appendix 13, we analyze the generation/retrieval results from a qualitative perspective.

## 7. Conclusion and Future Work

In this paper, we have presented the first manually-labeled benchmark for prompt performance prediction in the context of prompt-to-image generation. Our benchmark is also applicable for query-to-image retrieval, enabling the direct comparison of the performance prediction task in generation vs. retrieval scenarios. PQPP is a versatile resource, enabling the evaluation of predictors in various scenarios, such as in-domain, cross-model, cross-dataset and cross-task. Our dataset and code are made publicly available.

One direction for future work is to develop a model that combines different pre-retrieval predictors. Combining predictors in a supervised manner has shown its effectiveness in text [16] and image [36] QPP. Another important direction is to organize a shared task associated with the proposed PQPP benchmark, so that the research community can further explore the novel task of prompt performance prediction in image generation.

**Acknowledgments.** This research is supported by the project "Romanian Hub for Artificial Intelligence - HRIA", Smart Growth, Digitization and Financial Instruments Program, 2021-2027, MySMIS no. 334906.
References

[1] Qingyao Ai, Ting Bai, Zhao Cao, Yi Chang, Jiawei Chen, Zhumin Chen, Zhiyong Cheng, Shoubin Dong, Zhicheng Dou, Fuli Feng, Shen Gao, Jiafeng Guo, Xiangnan He, Yanyan Lan, Chenliang Li, Yiqun Liu, Ziyu Lyu, Weizhi Ma, Jun Ma, Zhaochun Ren, Pengjie Ren, Zhiqiang Wang, Mingwen Wang, Ji-Rong Wen, Le Wu, Xin Xin, Jun Xu, Dawei Yin, Peng Zhang, Fan Zhang, Weinan Zhang, Min Zhang, and Xiaofei Zhu. Information Retrieval meets Large Language Models: A strategic report from Chinese IR community. *AI Open*, 4:80–90, 2023. 1

[2] Negar Arabzadeh, Fattane Zarrinkalam, Jelena Jovanovic, and Ebrahim Bagheri. Neural embedding-based metrics for pre-retrieval query performance prediction. In *Proceedings of ECIR*, pages 78–85, 2020. 5, 6

[3] Negar Arabzadeh, Fattane Zarrinkalam, Jelena Jovanovic, and Ebrahim Bagheri. Neural embedding-based metrics for pre-retrieval query performance prediction. In *Proceedings of ECIR*, pages 78–85, 2020. 3

[4] Negar Arabzadeh, Maryam Khodabakhsh, and Ebrahim Bagheri. BERT-QPP: Contextualized pre-trained transformers for query performance prediction. In *Proceedings of CIKM*, pages 2857–2861, 2021. 3

[5] Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All Are Worth Words: A ViT Backbone for Diffusion Models. In *Proceedings of CVPR*, pages 22669–22679, 2023. 2, 4

[6] Nicolas Bizzozzero, Ihab Bendidi, and Olivier Risser-Maroix. Prompt Performance Prediction for Image Generation. In *Proceedings of ICIP*, pages 2208–2214, 2024. 3

[7] Kevin Matthe Caramancion. Large Language Models vs. Search Engines: Evaluating User Preferences Across Varied Information Retrieval Scenarios. *arXiv preprint arXiv:2401.05761*, 2024. 1

[8] David Carmel and Elad Yom-Tov. Estimating the query difficulty for information retrieval. *Synthesis Lectures on Information Concepts, Retrieval, and Services*, 2(1):1–89, 2010. 1, 2

[9] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion Models in Vision: A Survey. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 45(9):10850–10869, 2023. 1, 3

[10] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Reverse Stable Diffusion: What prompt was used to generate this image? *Computer Vision and Image Understanding*, 249:104210, 2024. 4

[11] Steve Cronen-Townsend, Yun Zhou, and W. Bruce Croft. Predicting query performance. In *Proceedings of SIGIR*, pages 299–306, 2002. 3, 2

[12] Ronan Cummins, Joemon Jose, and Colm O'Riordan. Improved query performance prediction using standard deviation. In *Proceedings of SIGIR*, pages 1089–1090, 2011. 2

[13] Suchana Datta, Debasis Ganguly, Derek Greene, and Mandar Mitra. Deep-QPP: A pairwise interaction-based deep learning model for supervised query performance prediction. In *Proceedings of WSDM*, pages 201–209, 2022. 1, 3

[14] Suchana Datta, Debasis Ganguly, Josiane Mothe, and Md Zia Ullah. Combining Word Embedding Interactions and LETOR Feature Evidences for Supervised QPP. In *Proceedings of QPP++*, pages 7–12, 2023. 3

[15] Claude De Loupy and Patrice Bellot. Evaluation of document retrieval systems and query difficulty. In *Proceedings of LREC*, pages 32–39, 2000. 2

[16] Sébastien Déjean, Radu Tudor Ionescu, Josiane Mothe, and Md Zia Ullah. Forward and backward feature selection for query performance prediction. In *Proceedings of SAC*, pages 690–697, 2020. 1, 8, 3

[17] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In *Proceedings of NAACL*, pages 4171–4186, 2019. 7

[18] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. In *Proceedings of NeurIPS*, pages 8780–8794, 2021. 1

[19] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In *Proceedings of ICLR*, 2021. 5

[20] Guglielmo Faggioli, Thibault Formal, Stefano Marchesin, Stéphane Clinchant, Nicola Ferro, and Benjamin Piwowarski. Query Performance Prediction for Neural IR: Are We There Yet? In *Proceedings of ECIR*, pages 232–248, 2023. 1, 3

[21] Helia Hashemi, Hamed Zamani, and W. Bruce Croft. Performance prediction for non-factoid question answering. In *Proceedings of SIGIR*, pages 55–58, 2019. 3

[22] Claudia Hauff, Djoerd Hiemstra, and Franciska de Jong. A survey of pre-retrieval query performance predictors. In *Proceedings of CIKM*, pages 1419–1420, 2008. 2

[23] Ben He and Iadh Ounis. Inferring query performance using pre-retrieval predictors. In *Proceedings of SPIRE*, pages 43–54, 2004. 3, 2

[24] Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling Up GANs for Text-to-Image Synthesis. In *Proceedings of CVPR*, pages 10124–10134, 2023. 2, 4

[25] Yuval Kirstain, Adam Polyak, Uriel Singer, ShahbulandMTiana, Joe Penna, and Omer Levy. Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation. In *Proceedings of NeurIPS*, pages 36652–36663, 2023. 3

[26] Nupur Kumari, Bingliang Zhang, Sheng-Yu Wang, Eli Shechtman, Richard Zhang, and Jun-Yan Zhu. Ablating concepts in text-to-image diffusion models. In *Proceedings of ICCV*, pages 22691–22702, 2023. 3

[27] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. In *Proceedings of ICML*, pages 19730–19742, 2023. 2, 5

[28] Yangxi Li, Bo Geng, Linjun Yang, Chao Xu, and Wei Bian. Query difficulty estimation for image retrieval. *Neurocomputing*, 95:48–53, 2012. 3
[29] Youwei Liang, Junfeng He, Gang Li, Peizhao Li, Arseniy Klimovskiy, Nicholas Carolan, Jiao Sun, Jordi Pont-Tuset, Sarah Young, Feng Yang, et al. Rich human feedback for text-to-image generation. In Proceedings of CVPR, pages 19401–19411, 2024. 3

[30] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence C. Zitnick. Microsoft COCO: Common Objects in Context. In Proceedings of ECCV, pages 740–755, 2014. 2, 3

[31] Chuan Meng, Negar Arabzadeh, Mohammad Aliannejadi, and Maarten de Rijke. Query performance prediction: From ad-hoc to conversational search. In Proceedings of SIGIR, pages 2583–2593, 2023. 1, 3

[32] Josiane Mothe and Ludovic Tanguy. Linguistic features to predict query difficulty. In Proceedings of SIGIR, pages 7–10, 2005. 3, 7, 2

[33] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. In Proceedings of ICML, pages 16784–16804, 2021. 1, 2, 4

[34] Nikita Pavlichenko and Dmitry Ustalov. Best prompts for text-to-image models and how to find them. In Proceedings of SIGIR, pages 2067–2071, 2023. 3

[35] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. In Proceedings of ICLR, 2024. 2, 4

[36] Eduard Poesina, Radu Tudor Ionescu, and Josiane Mothe. iQPP: A Benchmark for Image Query Performance Prediction. In Proceedings of SIGIR, pages 2953–2963, 2023. 1, 6, 8

[37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of ICML, pages 8748–8763, 2021. 2, 4, 5

[38] Fiana Raiber and Oren Kurland. Query-performance prediction: setting the expectations straight. In Proceedings of SIGIR, pages 13–22, 2014. 2, 3

[39] Haggai Roitman. An extended query performance prediction framework utilizing passage-level information. In Proceedings of SIGIR, pages 35–42, 2018. 6

[40] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. In Proceedings of CVPR, pages 10684–10695, 2022. 1

[41] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In Proceedings of NeurIPS, pages 36479–36494, 2022. 2, 4

[42] Mohammadreza Samadi and Davood Rafiei. Performance Prediction for Multi-hop Questions. arXiv preprint arXiv:2308.06431, 2023. 3

[43] Anna Shtok, Oren Kurland, David Carmel, Fiana Raiber, and Gad Markovits. Predicting query performance by query-drift estimation. ACM Transactions on Information Systems, 30 (2):1–35, 2012. 2

[44] Ashutosh Singh, Debasis Ganguly, Suchana Datta, and Craig McDonald. Unsupervised query performance prediction for neural models with pairwise rank preferences. In Proceedings of SIGIR, pages 2486–2490, 2023. 3

[45] Karen Sparck Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28(1):11–21, 1972. 2

[46] Shaoyan Sun, Wengang Zhou, Qi Tian, Ming Yang, and Houqiang Li. Assessing image retrieval quality at the first glance. IEEE Transactions on Image Processing, 27(12): 6124–6134, 2018. 7

[47] Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks. In Proceedings of NAACL, pages 296–310, 2021. 2, 4, 5

[48] Xinmei Tian, Yijuan Lu, and Linjun Yang. Query difficulty prediction for web image search. IEEE Transactions on Multimedia, 14(4):951–962, 2011. 3

[49] Xinmei Tian, Qianghuai Jia, and Tao Mei. Query difficulty estimation for image search with query reconstruction error. IEEE Transactions on Multimedia, 17(1):79–91, 2014. 3

[50] Lucas Pascotti Valem, Vanessa Helena Pereira-Ferrero, and Daniel Carlos Guimarães Pedronette. Self-supervised regression for query performance prediction on image retrieval. In Proceedings of AIKE, pages 95–98, 2023. 3

[51] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis. arXiv preprint arXiv:2306.09341, 2023. 6, 7

[52] Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human Preference Score: Better aligning text-to-image models with human preference. In Proceedings of ICCV, pages 2096–2105, 2023. 3

[53] Xing Xing, Yi Zhang, and Mei Han. Query difficulty prediction for contextual image retrieval. In Proceedings of ECIR, pages 581–585, 2010. 3

[54] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Image Reward: Learning and evaluating human preferences for text-to-image generation. In Proceedings of NeurIPS, pages 15903–15935, 2024. 3

[55] Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Paths. In Proceedings of NeurIPS, pages 41693–41706, 2024. 2, 4

[56] Elad Yom-Tov, Shai Fine, David Carmel, and Adam Darlow. Learning to estimate query difficulty: including applications to missing content detection and distributed information retrieval. In Proceedings of SIGIR, pages 512–519, 2005. 3, 6
[57] Hamed Zamani, W. Bruce Croft, and J. Shane Culpepper. Neural query performance prediction using weak supervision from multiple signals. In *Proceedings of SIGIR*, pages 105–114, 2018. 3

[58] Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-CLIP: Unlocking the Long-Text Capabil-ity of CLIP. In *Proceedings of ECCV*, pages 310–325, 2024. 7

[59] Zhongmin Zhang, Jiawei Chen, and Shengli Wu. Query per-formance prediction and classification for information search systems. In *Proceedings of APWeb-WAIM*, pages 277–285, 2018. 2

[60] Ying Zhao, Falk Scholer, and Yohannes Tsegay. Effective pre-retrieval query performance prediction using similarity and variability evidence. In *Proceedings of ECIR*, pages 52–64, 2008. 6, 2

[61] Yun Zhou and W. Bruce Croft. Query performance predic-tion in web search environments. In *Proceedings of SIGIR*, pages 543–550, 2007. 3

[62] Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji-Rong Wen. Large language models for information retrieval: A survey. *arXiv preprint arXiv:2308.07107*, 2023. 1