Discrete Prompt Optimization via Constrained Generation for Zero-shot Re-ranker
Year: 2023

Abstract:
Re-rankers, which order retrieved documents with respect to the relevance score on the given query, have gained attention for the information retrieval (IR) task. Rather than fine-tuning the pre-trained language model (PLM), the large-scale language model (LLM) is utilized as a zero-shot re-ranker with excellent results. While LLM is highly dependent on the prompts, the impact and the optimization of the prompts for the zero-shot re-ranker are not explored yet. Along with highlighting the impact of optimization on the zero-shot re-ranker, we propose a novel discrete prompt optimization method, Constrained Prompt generation (Co-Prompt), with the metric estimating the optimum for reranking. Co-Prompt guides the generated texts from PLM toward optimal prompts based on the metric without parameter update. The experimental results demonstrate that Co-Prompt leads to outstanding re-ranking performance against the baselines. Also, Co-Prompt generates more interpretable prompts for humans against other prompt optimization methods.

heading: 1 Introduction
text: Information retrieval (IR) is the task of searching for documents relevant to a given query from a large corpus. As re-ranking the fetched documents from the retriever effectively enhances the performance and the latency, recent studies have suggested several kinds of re-rankers by fine-tuning pre-trained language models (PLM) (Nogueira and Cho, 2019; Nogueira et al., 2020). Furthermore, Sachan et al. (2022) show that large-scale language models (LLMs) such as GPT-3 (Brown et al., 2020) can be exploited as a zero-shot reranker with the prompt describing the task. They also highlight the importance of an appropriate prompt to elicit the full performance of LLMs, rather than updating the parameters. They choose
∗ Corresponding author
an optimal prompt among the handcrafted candidates by cross-validation. However, such a manual search for the discrete prompts is highly expensive and sub-optimal in transferability.
To resolve the issue, several methods are proposed for automatically optimizing the discrete prompt. They focus on text classification or maskfilling task while underestimating the open-ended generation (Shin et al., 2020; Gao et al., 2021; Prasad et al., 2022). Recently, Deng et al. (2022) address the discrete prompt optimization applicable to generation tasks with reinforcement learning by designing the reward function, which measures the generated text belonging to a discrete label. Since there are tasks that are still not aligned, requiring a continuous score of output, we aim at a prompt optimization for one of such tasks: re-ranking.
In this paper, we propose Constrained Prompt generation, Co-Prompt, as left-to-right discrete prompt optimization without additional model training. By defining the metric of prompt optimum for re-ranking, we interpret the searching process of the optimal prompt as constrained generation with two modules: a zero-shot re-ranker as a discriminator and any decoder-only PLM as a generator. The discriminator calculates the likelihood (i.e., metric) that the prompt sequence is optimal for guiding an LLM to distinguish relevant
960
documents among the large set for a given query. The generator samples the prompt tokens having a high prior from the previous prompt sequences for effectively restricting the prompt candidates for discriminator to evaluate. An overview of Co-Prompt is shown in Figure 1.
We validate our method, Co-Prompt, against other optimization baselines on two LLMs, T0 (Sanh et al., 2022) and OPT (Zhang et al., 2022), with two benchmark datasets, MSMARCO (Nguyen et al., 2016) and Natural Question (Kwiatkowski et al., 2019). Experimental results show that Co-Prompt consistently generates well-performing prompts regardless of LLMs and datasets over the baselines. The qualitative analyses also support the interpretability of the prompts generated by Co-Prompt, similar to human language patterns.
Our contributions in this work are threefold:
• We highlight the impact of optimal prompt on a zero-shot re-ranker by exploiting the optimization methods.
• We propose Co-Prompt, a novel discrete prompt optimization via constrained generation for a zero-shot re-ranker.
• We experimentally show that Co-Prompt consistently guides the re-ranker well against the baselines and its output is similar to human language patterns.

heading: 2 Related Work
text: Document Ranking with Generative Model Using the generative model is one of the dominant methods for ranking the retrieved documents by defining the relevance score as the query likelihood score (Nogueira dos Santos et al., 2020; Ju et al., 2021). More recently, Sachan et al. (2022, 2023) showed that the LLM serves as either a zero-shot re-ranker or a training module of an unsupervised dense retriever. However, unlike ours, they require carefully designed manual prompts, which may have a limitation in transferability.
Prompt Optimization As prompting is considered a key variable when exploiting LLMs for various NLP tasks, finding the optimal prompt has become important to get the best performance out of the LLMs (Kojima et al., 2022; Xie et al., 2022). Recently, the prompt optimization work has focused on discrete prompt search (Shin et al., 2020; Gao et al., 2021; Deng et al., 2022) or soft prompt
learning over a continuous space (Liu et al., 2021; Qin and Eisner, 2021; Lester et al., 2021). While the existing optimization methods mainly consider text classification or mask-filling task, their applicability to re-ranking is yet underexplored. In this paper, we target at optimizing discrete prompts for zero-shot re-ranker to get higher relevance scores for more relevant pairs via constrained generation.
Constrained Generation Constrained generation aims at deriving the text sequences that follow a certain constraint (Keskar et al., 2019). Utilizing a discriminator for guiding the generation toward the constraint via the Bayes’ rule is one of the widely used constraint generation methods (Dathathri et al., 2020; Krause et al., 2021; Chaffin et al., 2022). Inspired by the effectiveness of the discriminator-based method, we adopt the zero-shot re-ranker as a discriminator when generating optimal discrete prompt sequences.

heading: 3 Method

heading: 3.1 Preliminaries
text: An LLM re-ranks the retrieved document d concerning the relevance score with a given query q as the query generation score:
logP (d|q) ∝ logP (q|d, ρ)
= 1 |q| ∑
t
logP (qt|q<t, d, ρ), (1)
where |q| denotes the token length of the query q and ρ is a natural language prompt guiding an LLM to generate the query q. Since the prompt ρ is the only controllable variable in Equation 1, searching for an optimal prompt is a simple yet effective way to enhance the performance of LLMs. Thus, in this work, we focus on a prompt optimization strategy.

heading: 3.2 Constrained Prompt Generation
text: We define the optimal prompt ρ∗ for the re-ranker which maximizes the query generation scores:
ρ∗ = argmax ρ E(di,qi)∈D[P (qi|di, ρ)], (2)
where D is the dataset for the retriever, consisting of pairs of a query and its relevant document.
We solve the task of searching the optimal prompt ρ∗ for the document-query pair dataset D with discriminator-based constrained generation. The generation is guided by the Bayes’ rule:
P (ρt|D, ρ1:t−1) ∝ PMD (Ds|ρ1:t)PMG(ρt|ρ1:t−1), (3)
Algorithm 1: Co-Prompt: a beam search-based prompt generation algorithm with a discriminator and a generator. Ds: document-query pairs, B: beam width, L: maximum prompt length, N : the number of final prompts, V: vocabulary set
Require: Ds, B, L,V begin
P1 ← {Start-Token} for t = 1, . . . , L do
Pt+1 ← ∅ foreach ρ1:t ∈ Pt do
St+1 ← topK K=B,ρt+1∈V PMG (ρt+1|ρ1:t)
Pt+1 ← Pt+1∪{ρ1:t+1|ρ1:t⊕ρt+1 ∈ St+1} end Pt+1 ← topK
K=B,ρ1:t+1∈Pt+1 PMD
(Ds|ρ1:t+1)
end P ← ∪t∈[1,L]Pt R← topK
K=N,ρ∈P PMD
(Ds|ρ)
return R end
where MD is a zero-shot re-ranker serving as a discriminator, MG is a decoder-only PLM as a generator, and Ds is a dataset sampled from D.
Discriminator The discriminator MD measures how effectively the prompt sequence ρ1:t guides the zero-shot re-ranker to generate the query from the given document by computing the likelihood PMD(Ds|ρ), defined as the expectation of relevance score between document-query pairs (qi, di) of the sampled dataset Ds with the prompt ρ:
PMD (Ds|ρ) = E(di,qi)∈Ds [PMD (qi|di, ρ)]. (4)
We use this likelihood as the metric for prompt optimum. The other option of PMD is shown in Appendix B.1.
Generator The generator MG samples the pool of prompts to be evaluated by a discriminator since computing Equation 3 of all possible tokens in the vocabulary requires a prohibitively high computational cost. The decoder-only PLM is exploited to sample prompt tokens ρt having a high prior PMG(ρt|ρ1:t−1) in a zero-shot manner.
We combine these modules to optimize the prompt by iteratively performing two steps: candidate generation and evaluation. We choose to use a beam search as a decoding strategy for left-toright prompt generation. The detailed steps of the decoding strategy are shown in Algorithm 1.

heading: 4 Experimental Setups
text: We describe the experimental setups for validating the performance of the prompts. Our code is publicly available at github.com/zomss/Co-Prompt.
Datasets We employ two information retrieval datasets: 1) MS-MARCO (Nguyen et al., 2016), collected from the Bing search logs, and 2) Natural Question (NQ, Kwiatkowski et al. (2019)),
fetched from Google search engines. We only use the document data of the dataset for evaluation. More information is shown in Appendix A.1.
Evaluation Metrics We evaluate the results by two metrics, ACC and nDCG. 1) ACC is the percentage of the relevant documents in the total retrieved ones. 2) nDCG, normalized discounted cumulative gain, reflects that the more relevant documents should record higher ranks.
Retriever & Re-ranker We select two widely used sparse and dense retrievers as our retrievers, which are 1) BM25 (Robertson and Zaragoza, 2009) and 2) DPR (Karpukhin et al., 2020), respectively. For the zero-shot re-ranker, we use 1) T0 (Sanh et al., 2022) and 2) OPT (Zhang et al., 2022). We describe more detailed information in Appendix A.3 and A.4.
Prompt Baselines We compare Co-Prompt against four baselines: 1) Null Prompt is an empty prompt without any token. 2) P-Tuning is a soft prompt optimization method that yields prompt embeddings from the prompt encoder (Liu et al., 2021). 3) RL-Prompt is a discrete prompt optimization method by training policy network (Deng et al., 2022). Note that we modify RL-Prompt and P-Tuning applicable to the re-ranking task. 4) Manual Prompt, suggested by Sachan et al. (2022), is given as "Please write a question based on this passage", following the assumption that it is one of the best prompts that humans can find. Last, 5) CoPrompt, our proposed method, is a discrete prompt optimization method in left-to-right zero-shot generation. The implementation details of baselines are shown in Appendix A.5.
Implementation Details The discriminator MD is the same model as the zero-shot re-ranker. Since the generator MG should be a decoder-only model, in the case of T0, GPT2-Large (Radford et al., 2019) is utilized as the generator. OPT, a decoderonly model, is used as both the discriminator and the generator. We use the start token as "Please" for a direct comparison with the manual prompt and fix the beam width B as 10 and the maximum prompt length L as 10 in our experiment.
Environment We conduct all experiments including prompt searching and document re-ranking on V100 32GB GPUs. We use BEIR (Thakur et al., 2021) framework1 for re-ranked result evaluation and passage retrieval datasets. Also, the retrievers, BM25 and DPR, are from the same framework. We employ T0 and OPT with 3B and 2.7B parameters each for the discriminator and the re-ranker publicly open on the Huggingface model hub2 (Wolf et al., 2020).

heading: 5 Result
text: In this section, we show the overall results of our method, Co-Prompt, with a detailed analysis.
Overall Results As shown in Table 1, Co-prompt consistently shows a robust performance gain in all scenarios, regardless of LLM, the dataset, and the retriever. Specifically, Co-Prompt, applied to OPT, achieves better results than the other methods. This indicates that the prompts generated by our
1http://beir.ai/ 2https://huggingface.co/models
method are more appropriate to play the role of an instruction to guide LLMs against other prompt optimization methods. More detailed results of reranked performance with various metrics are shown in Appendix B.3.
Impact of Start Tokens We exploit other options of start token such as "Score" and "This" as shown in Table 2. Regardless of the start tokens, Co-Prompt consistently generates prompts eliciting the performance of LLM efficiently. However, we observe that finding the optimal start token for the dataset is important to achieve better results.
Impact of Generator As shown in Table 3, even if different generators are used, the generated prompts by different generators guide the zero-shot re-ranker efficiently. Still, the differences in performance are caused by a vocabulary mismatch between the two modules. We see that, although our method does not vary significantly in performance to the generator, a more suitable generator may be necessary for better results.
Relevance Score We analyze the distributions of relevance scores between positive or negative document-query pairs. As the negative documents for a given query are retrieved from BM25, the negative ones are related to the query but unable to directly find the answer. As shown in Figure 2, we point out that the distribution difference exists between pairs despite some overlap. Also, an LLM
can distinguish which pair is positive, even without a prompt. However, we observe that the effect of discrete prompt optimization on the zero-shot reranker is in the direction of increasing the mean and variance of the relevance score.
Case Study of Prompts Table 2 shows the discrete prompts generated by our method and discrete prompt baselines when exploiting OPT as a reranker. While the prompts from the RL-prompt are ungrammatical gibberish close to a random word sequence, our method, Co-Prompt, generates interpretable prompts for humans, following human language patterns, and surpasses the performance of the other discrete prompts. Also, the word ‘question’, one of the keywords describing the task, is included in the prompts from Co-Prompt regardless of the datasets. This implies that the prompts from our method can provide a natural user interface to improve human understanding of how LLMs work. See Appendix B.3 for more examples of Co-Prompt.

heading: 6 Conclusion
text: In this paper, we propose Co-Prompt, left-to-right prompt optimization for zero-shot re-ranker via constrained generation. Co-Prompt effectively restricts prompt candidates and evaluates the optimum of these prompts without any parameter updates. We experimentally show that our method achieves consistently outperforming performance across all experiments. Also, the impact of prompt optimization including baselines on the zero-shot re-ranker highlights its importance. We also present an interesting outcome in that the optimal prompt is interpretable for human. For future work, we plan to expand our method to other open-ended generation tasks using LLMs.

heading: Limitations
text: As shown in Table 1, our method is experimentally demonstrated to be effective for two LLMs. However, OPT, a decoder-only model, is more suitable for the prompts generated by Co-Prompt. This seems to be because T0, the encoder-decoder model, requires a separate generator such as GPT2. The performance of prompts may vary to the generator involved in the vocabulary and training process. Also, there is a trade-off between search time and performance. While increasing the beam size and the number of document-query pairs enhances the probability of finding a more optimal
prompt, it makes the search time proportionally longer.

heading: Ethics Statement
text: Our work contributes to enhancing the retrieval performance of a zero-shot re-ranker by optimizing the discrete prompt via constrained generation. We are keenly aware of the possibility of offensive or upsetting prompts caused by bias of the generator itself even though there were no such prompts in our experiments. Because there is no additional training for prompt optimization, our method has difficulty removing the bias of the language model itself. As studies on reducing the bias of language models or filtering out inappropriate expressions in texts are being actively conducted, these problems are expected to be sufficiently resolved in the future.

heading: Acknoledgements
text: This work was supported by Institute for Information and communications Technology Promotion (IITP) grant funded by the Korea government (No. 2018-0-00582, Prediction and augmentation of the credibility distribution via linguistic analysis and automated evidence document collection).

heading: A.1 Datasets
text: We employ two information retrieval datasets for evaluating the performance of the zero-shot reranker with the prompts. 1) MS-MARCO (Nguyen et al., 2016) contains about 8M passages and 6,980 queries in development split collected from the Bing search logs. Because of the diversity of topics and contents with the large training set, recent work exploits MS-MARCO for retriever training (Nogueira and Cho, 2019; Qu et al., 2021). 2) Natural Question (NQ, Kwiatkowski et al. (2019)) contains about 2M passages of Wikipedia articles and 3,452 queries in test split collected from Google search engines. Also, NQ, one of the popular open-domain question datasets, is exploited as training data of dense retrievers (Karpukhin et al., 2020). Both datasets are the benchmarks for evaluating information retriever systems (Thakur et al., 2021). Only 1,500 document-query pairs from MSMARCO test split and NQ development split each are utilized for the prompt optimization.

heading: A.2 Metrics
text: As mentioned in Section 4, we employ two metrics, 1) ACC and 2) nDCG. In addition, we use one more metric. 3) MAP is the mean average precision of the relevant documents’ ranks for a given query.

heading: A.3 Retrievers
text: We use two types of retrievers, sparse and dense retrievers, for retrieving documents re-ranked by LLMs. 1) BM25 (Robertson and Zaragoza, 2009) is a representative sparse retriever computing the relevance score between a document and a query based on term frequency and inverse document frequency. BM25 has been widely employed because of its fast speed and effective performance. 2) DPR (Karpukhin et al., 2020) interprets training dense retrieval as metric learning problems. The biencoder initialized with BERT (Devlin et al., 2019) is trained with contrastive learning exploiting positive and negative passages for a given query. It shows outperforming results over traditional sparse retrievers.

heading: A.4 Zero-shot Re-rankers
text: We employ two LLMs, T0 and OPT, as re-rankers with the prompt. 1) T0, one of the T5 series (Raffel et al., 2020), consists of transformer encoder-
decoder layers. The models are fine-tuned versions of T5 for multi-task learning with prompted datasets. 2) OPT, a publicly open model, consists of decoder-only transformer layers. Its performance is comparable to those of GPT-3 models. We exploit OPT instead of GPT-3 due to academic budget.
The template is needed when trasmitting a document, a prompt and a query to zero-shot reranker together. Following the template setting of UPR, the template used in the experiments is "Passage: {document} {delimiter} {prompt} {delimiter} {query}". The delimiters used in the experiments are " " for T0 and "\n" for OPT.

heading: A.5 Baselines
text: Manual Prompt Sachan et al. (2022) not only proposed unsupervised passage re-ranker exploiting LLMs but also carefully selected the optimal prompt among handcrafted candidates validated by the re-ranked result at BM25 passages of NQ development set. The manually optimized prompt "Please write a question based on this passage" effectively guides zero-shot re-rankers to generate the query corresponding to the document.
P-tuning Liu et al. (2021) proposed P-tuning3, generating soft prompts (i.e., continuous prompt embeddings), not discrete ones. They employed the prompt encoder consisting of long-short term memory layers trained to return the optimal soft prompts for the task. While the method mainly focuses on the text classification task, we define the loss objective as query generation log-likelihood for application to re-ranking. The prompt encoder is trained with document-query pairs for 10 epochs to generate 10-length soft prompts.
RL-Prompt Deng et al. (2022) proposed discrete prompt generation, applicable to open-ended generation tasks, with reinforcement learning. They validated the method applicable to text style transfer, one of open-ended text generation techniques. In order to align to the re-ranking task, we define the reward for the policy network as query generation log-likelihood from the document and the prompt. Following the setting mentioned in RL-Prompt4, the 5-token length prompt is created through 12,000 training steps with a policy network model.
3https://github.com/THUDM/P-tuning 4https://github.com/mingkaid/rl-prompt

heading: B Analysis

heading: B.1 Likelihood PMD(Ds|ρ1:t)
text: In this section, we call the likelihood proposed in Equation 4 as the base metric. We consider the other option of likelihood PMD(Ds|ρ1:t) in a contrastive manner and also show the compared result with base metric in Table 5.
Contrastive Measurement The query generation score should be high for positive documentquery pairs D+s and low for negative pairs D − s . In a contrastive manner, the likelihood exploits the contrast between Pbase(D+s |ρ) and Pbase(D−s |ρ) as follows:
Pcont(Ds|ρ) = Pbase(D + s |ρ)
Pbase(D + s |ρ) + Pbase(D−s |ρ)
(5)
As shown in Table 5, base metric gains a certain level of performance regardless of the dataset and LLM, whereas contrastive metric shows inferior performance over MS-MARCO.
B.2 Impact of Generator We show more detailed results of the prompts from the different generators in table 4. While the generated prompts follow human language patterns, there are some differences in used words.

heading: B.3 Detailed Results
text: We evaluate the performance of zero-shot re-ranker with various metrics at Top-20 and Top-100 documents, as shown in Table 6. Co-Prompt is ranked
1st or 2nd on every metric across all experiments. On the other hand, the manual prompt, optimized for NQ, records inferior performance over MSMARCO. Also, other optimization methods, RLPrompt and P-Tuning, fail to achieve the best record in all experiments. This shows that the optimal prompt for zero-shot re-ranker is made from our method, Co-Prompt.
In addition, when confirming qualitatively generated prompts, the outputs from Co-Prompt are similar to human language patterns compared to RL-Prompt. The keyword "question" is included in most of the prompts generated by Co-Prompt. Considering that other optimization methods produce dense prompt embedding or ungrammatical gibberish, Co-Prompt suggests a new direction in which a prompt can function as a natural user interface to understand a black-box model.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: I discuss the limtations at section "Limitation".

heading: 3 A2. Did you discuss any potential risks of your work?
text: I discuss the potential risk at section "Ethics Statement".

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: I write at section "Abstract" and 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4 & Appendix A

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4 & Appendix A

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 4 & Appendix A

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4 & Appendix A

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 4 & Appendix A

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4 & Appendix A
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4 & Appendix A
C 3 Did you run computational experiments? section 5 Result

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix A
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4 & Appendix A

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? section 5 Result

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4 & Appendix A
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants?

heading: Left blank.
text: D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: As studies on reducing the bias of language models or filtering out inappropriate expressions in texts are being actively conducted, these problems are expected to be sufficiently resolved in the future