Query2doc: Query Expansion with Large Language Models
Authors: Liang Wang
Nan Yang
Furu Wei
Venue: EMNLP 2023 Main (2023)
Keywords: query expansion
large language models
information retrieval
TL;DR: a simple and effective approach for query expansion by prompting large language models

Abstract: This paper introduces a simple yet effective query expansion approach, denoted as query2doc, to improve both sparse and dense retrieval systems. The proposed method first generates pseudo-documents by few-shot prompting large language models (LLMs), and then expands the query with generated pseudo documents. LLMs are trained on web-scale text corpora and are adept at knowledge memorization. The pseudo-documents from LLMs often contain highly relevant information that can aid in query disambiguation and guide the retrievers. Experimental results demonstrate that query2doc boosts the performance of BM25 by 3% to 15% on ad-hoc IR datasets, such as MS-MARCO and TREC DL, without any model fine-tuning. Furthermore, our method also benefits state-of-the-art dense retrievers in terms of both in-domain and out-of-domain results.

Query2doc: Query Expansion with Large Language Models
Liang Wang and Nan Yang and Furu Wei
Microsoft Research
Iwangliang.nayya.uveei microsoft.com
Abstract
2009) have only shown limited success on popular datasets (Campos et al., 2016), and most state-of- the-art dense retrievers do not adopt this technique. In the meantime, document expansion methods like doc2query (Nogueira et al., 2019) have proven to be effective for sparse retrieval.
This paper introduces simple yet effec- tive query expansion approach, denoted as query2doc, to improve both sparse and dense retrieval systems. The proposed method first generates pseudo-documents by few-shot prompting large language models (LLMss, and then expands the query with generated pseudo- documents. LLMs are trained on web-scale text corpora and are adept at knowledge mem- orization. The pseudo-documents from LLMs often contain highly relevant information that can aid query disambiguation and guide the retrievers. Experimental results demon- strate that query2doc boosts the performance of BM25 by 3% 15% on hoc IR datasets, such MS-MARCO and TREC DL, with- out any model fine-tuning. Furthermore, our method also benefits state-of-the-art dense re- trievers in terms of both in in-domain and out-of- domain results.
this paper, we demonstrate the effectiveness of LLMs (Brown et al., 2020) as query expan- sion models by generating pseudo-documents con- ditioned on few-shot prompts. Given that search queries are often short, ambiguous, or lack neces- sary background information, LLMs can provide relevant information guide retrieval systems. they memorize an enormous amount knowledge and language patterns by pre-training on trillions of tokens.
Our proposed method, called query2doc, gen- erates pseudo-documents by few-shot prompting LLMs and concatenates them with the original query to form a new query. This method is simple to implement and does not require any changes in training pipelines or model architectures, making it orthogonal to the progress in the field of LLMs and information retrieval. Future methods can casily build upon our query expansion framework.
1 Introduction
Information retrieval (IR) aims locate relevant documents from a large corpus given user is- sued query. is a core component modern search engines and researchers have invested for decades in this field. There are two mainstream paradigms for IR: lexical-based sparse retrieval, such as BM25, and embedding-baseed dense re- trieval (Xiong al., 2021; Qu et al., 2021). Al- though dense retrievers perform better when large amounts of labeled data are available (Karpukhin et al., 2020), BM25 remains competitive on out-of- domain datasets (Thakur et al., 2021).
For in-domain evaluation, we adopt the MS- MARCO passage ranking (Campos al., 2016). TREC DL 2019 and 2020 datasets. Pseudo- documents are generated by prompting an im- proved version of GPT-3 rext-davinci-003 from OpenAl (Brown et al., 2020). Results show that query2doc substantially improves the off-the- shelf BM25 algorithm without fine-tuning any model, particularly for hard queries from the TREC DL track. Strong dense retrievers, in- cluding DPR (Karpukhin et al., 2020), SimLM (Wang al., 2023), and E5 (Wang et al., 2022) also benefit from query2doc, although the gains tend to be diminishing when distilling from a strong cross-encoder based re-ranker. Experi- ments zero-shot OOD settings demonstrate that
Query expansion (Rocchio, 1971; Lavrenko and Croft, 2001) is long-standing technique that rewrites the query based on pseude-relevance feedback external knowledge sources such as WordNet. For sparse retrieval, can help bridge the lexical gap between the query and the docu- ments. However, query expansion methods like RM3 (Lavrenko and Croft, 2001; Ly and Zhai,

---

our method outperforms strong baselines on most datasets. Further analysis also reveals the im- portance of model scales: query2doc works best when combined with the most capable LLMs while small language models only provide marginal im- provements over baselines. To aid reproduction, we release all the generations from lexi-davinci- 003 at https://huggingee co/ datasets/ intfloaa/qqerryaoocmsmmnrr
Here, "concat" denotes the string concatenation function. q is used as the new query for BM25 retrieval. We find that = 5 is a generally good value and do not tune it on a dataset basis.
Dense Retrieval The new query q² is sim- ple concatenation of the original query q and the pseudo-document d' separated by [SEP]:
qt concat(g, [SEP], d')
(2)
2 Method
For training dense retrievers, several factors can influence the final performance, such as hard nega- tive mining (Xiong et al., 2021), intermediate pre- training (Gao and Callan, 2021), and knowledge distillation from cross-encoder based re-ranker (Qu ct al., 2021). In this paper, investigate two settings to gain more comprehensive understand- ing of our method. The first setting is training DPR (Karpukhin et al., 2020) models initialized from BERT base with BM25 hard negatives only. The op- timization objective is a standard contrastive loss:
LLM Prompts
Write passage that answers the given query:
Query: what state this zip code
Welcome to TEMPE, AZ 85282. 85282 rural zip code in Tempe, Arizona, The population primarily white...
Passage:
Query:
when was pokemon green released
Passage:
Output
chyy
Pokemon Green was released in Japan on February 27th, 1996. was Pokemon series of games and served the basis for Pokemon Red and Blue, which were released in the US in 1998. The original Pokemon Green remains a beloved classic among fans the series.
Leont loo ch ha + EdEN hy
(3)
where h, and ha represent the embeddings for the query and document, respectively. N denotes the set of hard negatives.
The second setting is to build upon state-of-the- art dense retrievers and use KL divergence distill from a cross-encoder teacher model.
Figure I: Illustration of query2doc few-shot prompting. We omit some in-context examples for space reasons.
Given a query 4. we employ few-shot prompting to generate a pseudo-documen d' as depicted in Figure 1. The prompt comprises a brief instruction "Write passage that answers the given query:" and & labeled pairs randomly sampled from training set. We use k 4 throughout this paper. Subse- quently, we rewrite q new query q² by con- catenating with the eseudo-documeet d'. There are slight differences in the concatenation operation for sparse and dense retrievers, which we claborate on in the following section.
min DKL(Pce Pstm) + L
(4)
Pee and Psu are the probabilities from the cross- encoder and our student model, respectively. a is coefficient to balance the distillation loss and contrastive loss.
Comparison with Pseudo-relevance Feedback Our proposed method is related the clas- sic method of pseudo-relevance feedback (PRF) (Lavrenko and Croft, 2001; Ly and Zhai, 2009). conventional PRF, the feedback signals for query expansion come from the top-k documents ob- tained in the initial retrieval step, while our method prompts LLMs to generate pseudo-documentss Our method does not rely on the quality of the initial retrieval results, which are often noisy or irrelevant. Rather, it exploits cutting-edge I.LMs to generate documents that are more likely to contain relevant terms.
Sparse Retrieval Since the query is typically much shorter than pseuud-drccumnnt, to balance the relative weights of the query and the pscudo- document, boost the query term weights by repeating the query n times before concatenating with the pseudo-document d':
qdt = concat( (g) x n, d')
(1)

---

MS MARCO dev
TREC DL 19
TREC DL 20
Method
Fine-tuning
MRR @ I0
R@50
R@ lk
nDCG@ 10
nDCG@ 10
Sparse retrieval
BM25
18.4
58.5
85.7
51.2"
47.7"
+ query2doc
21.4*30
65.3+6.8
91.8 1
66,2*15.0
62,9-15.
BM25 RM3
15.8
56.7
86.4
52.2
47.4
doc' JocTsquery (Nogueira and Lin)
27.7
75.6
94.7
64.2
Dense retrieval w/o distillation
ANCE (Xiong et al., 2021)
33.0
95.9
64.5
64.6
HyDE (Gao et al., 2022)
61.3
57.9
DPRerr-lase (our impl.)
33.7
80.5
95.9
64.7
64.1
+ query2doc
35.1+14
82.6
97.2*1.3
68.7+10
67.1*38
Dense retrieval w/ distillation
RockerQAv2 (Ren et al., 2021)
38.8
86.2
98.1
AR2 (Zhang al., 2022)
39.5
87.8
98.6
SimLM (Wang et al., 2023)
87.8
98.7
71.4
69.7
query2doc
41.5
88,0
98.8+0.1
72,9*15
71.6*1.
base + KD (Wang al., 2022)
40.7
87.6
98.6
74.3
70.7
query2doc
41,5
88,1
98,7+t
74.9*06
72.5
Table I: Main results on the MS-MARCO passage ranking and TREC datasets. The "Fine-tuning" column indi- cates whether the method requires fine-tuning model on labeled data or not. x: our reproduction.
3 Experiments
are long-tailed entity-centric queries, which benefit more from the exact lexical match. The traditional query expansion method RM3 only marginally improves the R k metric. Although the docu- ment expansion method docTry achieves bet- ter numbers on MS- MS-MARCO dev set, it requires training a T5-based query generator with all the available labeled data, while "BM25 + query2doc" does not require any model fine-tuning.
3.1 Setup
Evaluation Datasets For in-domain evaluation, we utilize the MS-MARCO passage ranking (Cam- pos et al., 2016), TREC DL 2019 (Craswell et al., 2020a) and 2020 (Craswell al., 2020b) datasets. For zero-shot out-of-domain evaluation, we select five low-resource datasets from the BEIR bench- mark (Thakur et al., 2021). The evaluation met- rics include MRR@ 10, R R@k & (k E {50, Ik)), and nDCG@ 10.
For dense retrieval, the model variants that com- bine with query2doc also outperform the corre- sponding baselines on all metrics. However, the gain brought by query2doc tends diminish when using intermediate pre-training or knowledge distil- lation from cross-encoder re-rankers, as shown by the "SimLM + "dor" and "E5 + query2doc" results.
Hyperparameters For sparse retrieval including BM25 and RM3, we adopt the default implementa- tion from Pyserini (Lin et al., 2021). When training dense retrievers, we use mostly the same hyper- parameters as SimLM (Wang et al., 2023), with the exception of increasing the maximum query length to 144 to include geeudo-documenty When include 4 in-context exam-
For zero-shot out-of-domain retrieval, the results mixed as shown in Table 2. Entity-centric datasets like DBpedia see the largest improvements. On the NFCorpus and Scifact datasets, we observe a minor decrease ranking quality. This is likely due to the distribution mismatch between training and evaluation.
prompting LLMs, ples and use the default temperature of to sample most 128 tokens. For further details, please refer Appendix A.
3.2 Main Results
Analysis
In Table 1, we list the results the MS-MARCO
passage ranking and TREC DL datasets. For sparse retrieval, "BM25 query2doc" beats the BM25 baseline with over 15% improvements on TREC DL 2019 and 2020 datasets. Our manual inspection reveals that most queries from the TREC DL track
Scaling up LLMs is Critical For our proposed method, question that naturally arises is: how does the model scale affect the quality of query expansion? Table shows that the performance steadily improves as we go from the 1.3B model

---

DBpedia
NFCorpus
Scifact
Trec-Covid
Touche2020
BM25
31.3
32.5
66.5
65.6
36.7
query2doc
37.0+5.
34,9*24
68.6
72.2
39,8+3.1
SimLM (Wang et al., 2023)
34.9
32.7
62.4
55.0
18.9
query2doc
38,3+3.4
32,1-06
59,5-29
59,9+4.9
25.6+6.7
base + (Wang et al., 2022)
40.7
35.0
74.1
30.9
query2doc
42,4*1.7
35,2
67.5-29
75.1
31.7*08
Table 2: Zero-shot out-of-domain results on 5 low-resource datasets from the BEIR benchmark (Thakur al., 2021). The reported numbers are nDCG@ 10. For fair comparison, the in-context examples for prompting LLMs come from the MS-MARCO training set.
params
TREC 19
TREC 20
1%, regardless of the amount of data used for fine- tuning. This observation highlights that our contri- bution is orthogonal to the continual scaling up of
BM25
51.2
47.7
w/ babbage
1.3B
52.0
50.2
w/ curie
6.7B
55.1
50.1
supervision signals.
w/ davinci-001
175B
63.5
58.2
w/ davinci-003
175B
66.2
62.9
TREC 19
TREC 20
w/ gpt-4
69.2
64.5
BM25 + query2dac
66.2
62.9
w/ query only
51.2
47.7
Table 3: Query expansion with different model sizes. Even though GPT-4 performs best, we are unable to apply in the main experiments due quota limits.
w/ pscudo-doc only
48.7
44.5
Table 4:
Using the concatenation of the original query and the generated pseudo-documents perform substan- tially better.
to 175B models. Empirically, the texts generated by smaller language models tend be shorter and contain more factual errors. Also, the "davinci-003" model outperforms its earlier version "'dvvinei0001 by using better training data and improved instruc- tion tuning. The recently released GPT-4 (OpenAI, 2023) achieves the best results.
How to Use Pseudo-documents In this paper, we concatenate the original query and pseudo- documents as the new query. Alternatively, one can solely use the pseudo-documents, as done in the approach of HyDE (Gao et al., 2022). The results presented in Table 4 demonstrate that the original query and pseudo-document and their combination leads substantially better performance in sparse retricval.
complementary,
36
32.1
35.7
MRR on dev S8t
32
328
Case Analysis Table 5, we show two queries along with their respective pseudo-documents and groundtruth. The pscudo-documents, which are generated by LLMs, offer detailed and mostly ac- curate information, thereby reducing the lexical mismatch between the query and documents. In some cases, the pseudo-documents are sufficient meet the user's information needs, rendering the retrieval step unnecessary. However, is worth noting that the LLM generations may contain fac- tual errors. For instance, in the second query, the theme song "It's a Jungle Out There" was used as of season two in 2003, not 2002 Although such may appear subtle and difficult verify, they pose a significant challenge building trustworthy systems using LLMs.
31.4
28
24
DPR w/o query2doc
DPR w/ query2dac
20
% labeled data for fine-tuning
Figure 2: MRR MS-MARCO dev set w.r.t the per- centage of labeled data used for fine-tuning.
Performance Gains are Consistent across Data Scales Figure/ 2 presents comparison between variants of DPR models. which differ in the amount of labeled data used. The results show that the "DPR + query2doc" variant consistently outperforms the DPR baseline by approximately
Refer to //en, wttpp//ii...g//iiiittt...a Lungle_Qut_There__CC

---

query
who killed nicholas ii of russia
L.LM generation
Nicholas II of Russia, the last Tsar of Russia, executed by Bolshevik revolutionaries on July 17th, 1918, along with his wife Alexandra and their five children. The exccution was ordered by Vladimir Lenin, the leader of the Bolshevik Party. The actual execution was carried out by firing squad of Bolshevik troops, and the bodies were later buried mass grave... (1868-1918). Nicholas II was the last emperor, or tsar, of Russia, serving from 1894 to 1917. Nicholas, his wife, and their five children were killed by the Bolsheviks, members of revolutionary group (led by Lenin) who seized control of the government in Russia during the October Revolution (scc Russian Revolution) of 1917.
Groundtruhh
query
who sings monk theme song
LLM generation
The theme song for the television show Monk is entitled "It's a Jungle Out There" and is sung by American singer-songwriter Randy Newman. The song was written specifically for the show, and it has been used as the theme song since the series premiered in 2002. has been praised by crities and fans alike and is often regarded one of the best theme songs in television history.
Groundtruth
cxists and is an alternate of. The Monk theme song is It's Jungle Out There by Randy Newman. The Monk theme song It's Jungle Out There by Randy Newman.
Table 5: Examples from the TREC DL 2020 dataset. Bold texts are the overlapping words between groundtruth and pseudo-documents generated from LLMs. The italicized red sentence demonstrates a factual error in language model generations.
5 Related Work
eters, exhibiting unparalleled generalization ability across various tasks. LLMs can follow instruc- tions in a zero-shot manner or conduct in-context learning through few-shot prompting. Labeling a few high-quality examples only requires minimal human effort. In this paper, we employ few-shot prompting to generate pseudo docuum-dsssmnnts from a given query. A closely related recent work HyDE (Gao et al., 2022) instead focuses the zero- shot setting and uses embeddings of the pseudo- documents for similarity search. HyDE implicitly assumes that the groundtruth document and pseudo- documents express the same semantics in different words, which may not hold for some queries. In the field of question answering, RECITE (Sun et al., 2022) and GENREAD (Yu et al., 2022) demon- strate that LLMs are powerful context generators and can encode abundant factual knowledge. How- ever, as our analysis shows, LLMs can sometimes generate false claims, hindering their practical ap- plication in critical areas.
Query Expansion and Document Expansion are two classical techniques to improve retrieval quality, particularly for sparse retrieval systems. Both techniques aim to minimize the lexical gap be- tween the query and the documents. Query expan- sion typically involves rewriting the query based on relevance feedback (Lavrenko and Croft, 2001; Rocchio, 1971) or lexical resources such as Word- Net (Miller, 1992). In cases where labels are not available, the top-k retrieved documents can serve as pseudo-relevance feedback signals (Lv and Zhai, 2009). Liu et al. fine-tunes an encoder-decoder model to generate contextual clues.
In contrast, document expansion enriches the document representation by appending additional relevant terms. Doc2query (Nogucira et al., 2019) trains seq2seq model predict pseudo-queries based on documents and then adds generated pseudo-queries to the document index. Learned sparse retricval models such as SPLADE (Formal et al., 2021) and uniCOIL (Lin and Ma, 2021) also learn document term weighting in an end-to-end fashion. However, most state-of-the-art dense re- trievers (Ren et al., 2021; Wang et al., 2023) do not adopt any expansion techniques. Our paper demon- strates that strong dense retrievers also benefit from query expansion using LLMs.
6 Conclusion
This paper presents a simple method query2doc to leverage LLMs for query expansion. It first prompts LLMs with few-shot examples to gener- ate pseudo-documents and then integrates with ex- isting sparse dense retrievers by augmenting queries with generated pseudo-documents. The un- derlying motivation is to distill the LLMs through prompting. Despite its simplicity, empirical evalua- tions demonstrate consistent improvements across various retrieval models and datasets.
Large Language Models (LLMs) such as GPT-3 (Brown et al., 2020), PaLM (Chowdhery et al., 2022), and LLaMA (Touvron et al., 2023) are trained on trillions of tokens with billions of param-

---

Limitations
Majumder, Li Deng, and Bhaskar Mitra. 2016. Ms marco: A human generated machine reading com- prehension dataset. ArXiv preprint, abs/161 1.09268.
LLM call
Index search
BM25
16ms
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gauray Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez. Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben- ton C. Hutchinson, Reiner Pope, James Bradbury. Ja- cob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghe- mawat, Sunipa Dev, Hemryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fe- dus. Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankara- narayana Pillai, Marie Pellat, Aitor Lewkowyez. Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Saeta, Mark Díaz. Orhan Firat, Michele Catasta. Jason Wei, Kathleen S. Meier-Hellsterm, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language modeling with pathways. ArXiv preprint, ahs/2204.02311.
+ +qery2doc
2000ms
177ms
Table 6: Latency analysis for retrieval systems with our proposed query2doc. We retrieve the top 100 results for MS-MARCO dev queries with single thread and then average over all the queries. The latency for LLM API calls depends server load and is difficult to precisely measure.
An apparent limitation is the efficiency of re- trieval. Our method requires running inference with LLMs which can be considerably slower due to the token-by-token autoregressive decoding, Moreover, with query2doc, searching the inverted index also becomes slower as the number of query terms in- creases after expansion. This is supported by the benchmarking results in Table 6. Real-world de- ployment of our method should take these factors into consideration.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. 2020a. Overview of the tree 2019 deep learning track. ArXiv preprint, abs/2003.07820.
References
Alexander Bondarenko, Maik Frübe, Johannes Kiesel, Shahbaz Syed, Timon Gurcke, Meriem Beloucif, Alexander Panchenko, Chris Biemann, Benno Stein, Henning Wachsmuth, et al. 2022. Overview of touché 2022: argument retrieval. In Interna- tional Conference of the Canss-Language Evalua- fion Forum for European Languages, pages 311- 336. Springer.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Fernando Campos, and Ellen M. Voorhees. 2020b. Overview of the tree 2020 deep learning track. ArXiv preprint, abs/2003.07820.
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021. Splade: Sparse lexical and expan- sion model for first stage ranking. Proceedings of the 44th International ACM SIGIR Conference Research and Development in Information Retrieval.
Vera Boteva, Demian Gholipour, Artem Sokolov, and Stefan Riezler. 2016. A full-text learning to rank dataset for medical information retrieval. In Euro- pean Conference on Information Retrieval, pages 716-722. Springer.
Luyu Gao and Jamie Callan. 2021. Condenser: a pre- training architecture for dense retrieval. In Proceed- ings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 981-993, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu. Clemens Winter, Christopher Hesse, Mark Chen, Fric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark. Christopher Berner, Sam Mc- Candlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are shot learn- ers. Advances Neural Information Processing Systems 33: Annual Conference on Neural Informa- tion Processing Systems 2020, NeurlPS 2020, De- cember 6-12, 2020, virtual.
Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan, 2022. Precise zero-shot dense re- trieval without relevance labels. ArXiv preprint, abs/2212.10496.
Faegheh Hasibi, Fedor Nikolaev, Chenyan Xiong. Krisztian Balog, Svein Erik Bratsberg, Alexander Kotov, and Jamie Callan. 2017. Dbpedia-entity v2: A test collection for entity search. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017, pages 1265-1268. ACM.
Daniel Fernando Campos. Tri Nguyen, Mir Rosenberg. Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan

---

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pages 6769- 6781, Online. Association for Computational Lin- guistics.
Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021. RocketQAv2: A joint training method for dense passage retrieval and passage re-ranking. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 2825-2835, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Victor Lavrenko and W. Bruce Croft. 2001. Relevance- based language models. ACM SIGIR Forum, 51:260 - 267.
J. Rocchio, 1971. Relevance feedback in information retrieval.
Zhiqing Sun, Xuezhi Wang. Yi Tay, Yiming Yang, and Denny Zhou. 2022. Recitation-auurmented language models. ArXiv preprint, abs/2210.01296.
Jimmy J. Lin and Xueguang Ma. 2021. A few brief notes deepimpact, coil, and a conceptual frame- work for information retrieval techniques. ArXiv preprint, abs/2106. 148077
Nandan Thakur, Nils Reimers, Andreas Rücklé Ab- hishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogeneous benchmark for zero-shot evalua- tion of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Sys- lems Datasets and Benchmurks Track (Round 2).
Jimmy J. Lin, Xueguang Ma, Sheng-Chier Lin, Jheng- Hong Yang. Ronak Pradeep, Rodrigo Nogueira, and David R. Cheriton. 2021. Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations. Proceedings of the 44th International ACM SIGIR Conference on Re- search and Development in Information Retrieval.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ane Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur' clien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023, Llama: Open and efficient foundation language mod- els. ArXiv preprimt, abs/2302. 13971.
Linqing Liu, Minghan Li, Jimmy Lin, Sebastian Riedel, and Pontus Stenetorp. 2022. Query expansion us- ing contextual clue sampling with language models. ArXiv preprint, abs/2210.0709..
Yuanhua Lv and ChengXiang Zhai. 2009. A compara- tive study of methods for estimating query language models with pseudo feedback. Proceedings of the 18th ACM conference on Information and knowl- edge management.
Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2021. Trec-covid: constructing a pandemic information re- trieval test collection. In ACM SIGIR Forum, vol- ume 54, pages 1-12. ACM New York, NY, USA.
George A. Miller 1992 WordNet: A lexical database for English. In Speech and Natural Language: Pro- ceedings of a Workshop Held at Harriman, New York, February 23-26. 1992.
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verify- ing scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 7534-7550, On- line. Association for Computational Linguisties.
Rodrigo Nogueira and Jimmy Lin. From doc2query to doctittqquery.
Rodrigo Nogueira, Wei Yang, Jimmy J. Lin, and Kyunghyun Cho. 2019. Document expansion by query prediction. ArXiv preprint, abs//194.008377 08375.
I Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang. Daxin Jiang, Rangan Majumder. and Furu Wei. 2022. Text embeddings by weakly- supervised contrastive pre-training. ArXiv preprint, abs/2212.03533.
OpenAT. 2023. abs/2303.08774.
Gpt-4 technical report. ArXiv,
Ofr Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. arxiv preprint
I Liang Wang. Nan Yang. Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2023. SimLM: Pre-training with repre- sentation bottleneck for dense passage retrieval. In Proceedings of the 61 Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 2244 2258, Toronto, Canada. Association for Computational Linguistics.
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. RocketQA: An opti- mized training approach to dense passage retrieval for open-domain question answering. In Proceed- ings of the 2021 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5835-5847 Online, Association for Computational Linguistics.
Lee Xiong. Chenyan Xiong. Ye Li, Kwok-Fung Tang. Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neigh- bor negative contrastive learning for dense text re- trieval. In 9th International Conference on Learning

---

Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
which costs nearly 5k dollars. Most API calls are used generate pseudo-documens for the training queries.
W. Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingx- uan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2022. Generate rather than retrieve: Large language models are strong context generators. ArXiv preprint, abs/2209.1006s
For GPT-4 prompting, we find that it has ten- dency to ask for clarification instead of directly generating the pseudo-documents To mitigate this issue, we set the system message to "You are asked to write a passage that answers the given query. Do not ask the user for further clarification.. Regarding out-of-domain evaluations on DBpe- dia (Hasibi et al., 2017), NFCorpus (Boteva et al., 2016), Scifact (Wadden et al., 2020). Trec-Covid (Voorhees et al., 2021), and Touche2020 (Bon- darenko et al., 2022), SimLM's results are based on the released checkpoint by Wang al..
Hang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lv, Nan Duan, and Weizhu Chen. 2022. Adversar- The
ial eerrkeeerrankkr for dense text retrieval. Tenth International Conference on Learning Repre- sentations, ICLK 2022, Virtual Event, April 25-29. 2022. OpenReview.net.
A Implementation Details
DPR
w/ distillation
For ablation experiments in Figure 2, we fine- tune for 40 epochs 18k steps, whichever is reached first.
learning rate
x 10-5
10-5
PLM
BERT
SimLM/ E5 base-unspp
# of GPUs
4
warmup steps
1000
1000
batch size
64
64
B Exploring Other Prompting Strategies
epoch
3
n.a.
0.2
negatives depth
1000
200
DL 2019
DL 2020
query length
144
144
Query2doc
69.2
64.5
passage length
144
144
+ iterative prompt
68.6
64.8
of negatives
15
Table 8: GPT-4 with iterative prompting to self-verify the generated pseudo-documents.
Table 7: Hyper-parameters for training dense retrievers MS-MARCO passage ranking dataset.
Instead of generating pseudo-documents in one round, recent work (Press et al., 2022) proposes to iteratively prompt the LLM to improve the gen- eration quality. We explore this intuition by ask- ing GPT-4 rewrite its own generated pseudo- documents with the following prompt template:
For dense retrieval experiments in Table 1, we list the hyperparameters in Table 7. When training dense retrievers with distillation from cross-encoder, we use the same teacher score released by Wang et al.. The SimLM and E5 checkpoints for initialization are pub- licly available at co/ co//ittfloattsiinmmm-e----ssmss https:://uggiingaace wtt.eeeloonatt//////////. e5- e5-base-unsupervised.
You are asked to rewrite the passage that poten- tially answers the given query. You should only correct the factual errors in the passage, do not ask for clarification or make unnecessary changes.
and
To compute the text embeddings, we utilize the [CLS] vector for SimLM and mean pooling for E5. This makes sure that the pooling mechanisms remain consistent between intermediate pre- rve-training and fine-tuning. The training and evaluation of a dense retriever take less than 10 hours to finish.
Query: [[query]]
# Begin of passage [1 [[passage]] End of passage
When prompting LLMs, we include 4 in-context examples from the MS-MARCO training set. To increase prompt diversity, we randomly select examples for each APT call. A complete prompt is shown in Table 11. On the budget side, we make about 550k API calls to OpenAl's service,
Empirically, we find that GPT-4 makes very few changes to the generated pseudo-documents, which suggests that the pseudo-documents are already of high quality or GPT-4 is not capable of correcting its own errors. The results are shown in Table 8.

---

query
allows employers who sings hey good looking
LLM generation
Hey Good Lookin' was originally written and recorded by the country music singer Hank Williams in 1951. The song has since been covered by several other artists and bands, including Jimmy Buffett, Conway Twitty, and Willie Nelson.
Groundtruth
Hey, Good Lookin'. (1951) Hey, Good Lookin" is 1951 song written and recorded by Hank Williams, and his version
inducted into the Grammy Hall of Fame 2001. Since its original 1951 recording it has been covered by variety of artists.
Query
trumbull marriott fax number
LLM generation
The fax number
Trumbull Marriott Shelton 203-378-4444,
Groundtruth
Business name: Trumbull Marriott Merritt Parkway; Address: 180 Hawley Lane Trumbull, Connecticut 06611; Phone number: 203-378-4955 Fax number: 203-378-1444 1400; Business hours: 24; Credit cards accepted: Yes; Number of employees: 10-19; Map:
query
what is hra and hsa
LLM generation
HRA stands for Health Reimbursement Arrangement. It is a type of benefit plan where an employer reimburses employees for qualifying medical expenses. to set aside money taxefree to reimburse employees for medical expenses that are not covered under employer's ' health insurance plan. HSA stands for Health Savings Account. It a savings account that is used in conjunction with qualified High Deductible Health Plan (HDHP). An HSA enables an individual to pay for current health expenses and save future health expenses on tax free basis. Contributions to the account are tax
Groundtruth
The key difference between HRA HSA HRA is an employer funded health benefit plan that reimburses for medical expenses including personal health insurance policy pre- miums of employees whereas HSA available to taxpayers in the United States who are enrolled in High-Deductible Health Plan (HDHP).
tax advantaged health benefit plan exclusively
Table 9: More examples of LLM generations. The format is the same in Table 5.
DL 2019
DL 2020
Average 64.8
60.9
Std dev.
+1.14
#166
Table 10: Sparse retrieval results of query2doc across 3 random runs. The randomness comes from the se- lection of few-shot examples and the auto-regressive sampling LLMSs
C Results Across Multiple Runs
In our method, there two sources of random- ness: the selection of few-shot examples and the auto-regressive top-p sampling of L.Mss. To quan- tify the variance of our method, we report the av- erage and standard deviation of sparse retrieval results across 3 random runs in Table 10. One pos- sible improvement is to select few-shot examples based on semantic similarity to the query. We leave this for future work.

---

Write passage that the given query:
Query: what state is this zip code 85282
Passage: Welcome to TEMPE, AZ 85282. 85282 is rural zip code in Tempe, Arizona. The population primarily white, and mosty single. At $200,200 the average home value here bit higher than metro area, so this probably isn't the place to look for housing 33 degrees latitude (Fun Fact: this the
average for the bargains.5282 5282 Zip code is located in the Mountain time same latitude as Damascus, Syria!) and -1 12 degrees longitude.
Query: why gibbs model of reflection good Passage: In this reflection, I am going
Gibbs (1988) Reflective Cycle. This model is recognised framework for my reflection. Gibbs (1988) consists of six stages to complete one cycle which is able improve my nursing practice continuously and learning from the experience for better practice the future.n a conclusion of my reflective assignment, I mention the model that chose, Gibbs (1988) Reflective Cycle my framework of my reflective. I state the reasons why I am choosing the model well discussion on the important of doing reflection in nursing practice.
prompts
Query: what does a thousand pardons means
Passage: Oh, that's all right, that's all right, give rest; mind about the direction, hang the direction- I beg pardon, I beg a thousand pardons, am not well to-day; pay no attention when soliloquize, is old habit, an old, bad habit, and hard get rid of when one's digestion is all disordered with eating food that was raised forever and ever before he was born; good land! a man can't an't keep his functions regular spring chickens thirteen hundred years old.
Query: what is
warning
Passage: Macro virus warning appears when no macros exist in the file in Word. When you open Microsoft Word 2002 document template, you may receive the following macro virus warning, even though the document or template does not contain macros: CA<<aat>><cille name>contains macros. Macros may contain viruses.
Query: when was pokemon green released
Passage:
Pokemon Green was relcased Japan on February 27th, 1996. It the first in the Pokemon series of games and served as the hasis for Pokemon Red and Blue, which were released in the US in 1998. The original Pokemon Green remains heloved classic among fans of the series.
output
Table 11: The full prompt used for the example in Figure 1.


Reviews:
[{"review_id": "RIrjQ762Lb", "reviewer": "Reviewer_dbnL", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments! We are encouraged to learn that you are impressed by the comprehensive experiments presented in our paper.\n\n**Q1: On limited novelty in modeling**\n\nWhile we did not introduce any new modeling architecture or training methods, our paper's main contribution lies in demonstrating through empirical evidence that LLMs can substantially enhance retrieval systems via query expansion. We believe this is an important finding that can have a big impact on the field, and we urge for the adoption of simple yet effective methods that can enhance retrieval systems.\n\n**Q2: On the discussion of query2doc as a Pseudo-Relevance Feedback method**\n\nWe agree with the reviewer that query2doc does not strictly align with traditional PRF methods, as query2doc does not have an initial retrieval step to find top-ranked documents. Instead, the pseudo-documents in query2doc come from prompting off-the-shelf LLMs. Despite this difference, they share quite some similarities, therefore we include this comparison to help readers better understand this field.\n\nWe are also thankful for the reviewer's helpful suggestions on presentation improvement.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Simple and effective query expansion model using LLM with a good set of experimentation. ", "reasons_to_reject": "Limited novelty in modeling. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper discusses a simple query expansion approach using LLM. It is an incremental work compared to the HyDE model. The only difference between the two is that query2doc combines the original query with the generated pseudo-document, while HyDE only uses the generated document as the query to retrieve. \n\nHowever, in my opinion, the contribution of this work mainly comes from experimentation. While HyDE only experimented with Contriever, this paper experiments with multiple retrieval models, including sparse and dense ones, which demonstrates the robustness and generalizability of the work. I am willing to vote for this paper based on its completeness of the experimentation, which is often less than ideal in retrieval papers submitted to *CL venues. \n\nAs a minor comment, the authors mentioned that the query2doc approach could be considered a pseudo-relevance feedback approach, which is, in fact, not the case. The proposed method is a query expansion technique that does not use the top-ranked documents as feedback signals. PRF is a set of query modification approaches that blindly assumes the top-ranked documents are relevant and modifies(often expands) the query based on them. Despite the similarity, query2doc should not be considered a PRF approach. However, the comparison is still a good one since both of them modify the query. \n\nIn summary, the authors provided a good set of experiments to demonstrate the generalizability and robustness of the proposed method. Despite the limited technological novelty in the modeling part, the experimentation is good. Therefore, I vote for acceptance for the paper.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- doc2query itself is a document expansion technique instead of a retrieval model. Since it uses BM25 underneath, it should be listed under BM25 in Table 1. "}, "extra_text": {}}, {"review_id": "UtX1kGgY1R", "reviewer": "Reviewer_hWUX", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable review! We are thrilled to hear that you find our work interesting.\n\n**Q1: I didn't see any details about the computational complexity / budget used.**\n\nThanks for pointing this out.\n\nFor computational complexity, we provide the information on the hyperparameters and the number of GPUs used in Table 6. Training and evaluation of a dense retriever take less than 10 hours to finish.\n\nOn the budget side, we make about 550k API calls to OpenAI's service, which costs nearly 5k dollars. Most API calls are used to generate pseudo-documents for the training queries.\n\nWe will add these details in the future version of the paper.\n\n**Q2: I would prefer to see additional standard metrics in the main evaluation (Table 1).**\n\nFor space reasons, we list the most widely adopted metrics for the corresponding datasets in Table 1. We find that other metrics such as MAP or Recall with different thresholds correlate with these main metrics quite well. We will consider adding additional metrics in the Appendix part.\n\n**Q3: Statistical significance tests are not reported.**\n\nWe will update the paper numbers by averaging over different random runs and report significance tests whenever applicable to ensure the robustness and reliability of our results.\n\nThank you once again for your feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Using LLMs as proposed in this paper is a straightforward step to improve the quality of query expansion.\n- The paper includes an interesting analysis section containing experiments for assessing the effect of model size and labeled data size on query expansion performance.\n- The proposed method is evaluated on three standard datasets and is compared to several methods of different types - and shows strong performance.", "reasons_to_reject": "- I didn't see any details about the computational complexity / budget used.\n- I would prefer to see additional standard metrics in the main evaluation (Table 1).\n- Statistical significance tests are not reported.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the authors use LLMs for the task of query expansion.\nThe authors propose to generate pseudo-relevant documents using an LLM and concatenate them to the original query. They experiment with both sparse and dense retrievers, and report strong performance for both of them.\nThe proposed method is evaluated on three standard datasets and is compared to several methods of different types.\nFinally, the authors analyze the effect of model and labeled data size on query expansion performance, and found positive correlations for both.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9pjaTo9VDB", "reviewer": "Reviewer_VHFs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback!\n\n**Q1: The proposed method is highly empirical and lacks theoretical justification, particularly in the case of repeating q for n times for sparse retrieval.**\n\nWe appreciate your feedback and acknowledge that this paper is mostly empirical, aligning with the theme of the EMNLP conference which focuses on empirical methods in NLP.\n\nIn regard to repeating q for n times for sparse retrieval, we would like to clarify that this is a form of term weighting. The BM25 matching score is a sum over all query terms. However, the generated pseudo-documents are often much longer than the original query, causing the BM25 score of the original query to be overwhelmed by the score of the pseudo-document. To balance their relative weights, we repeat the original query for n times. We will elaborate on this decision in the revised version.\n\n**Q2: The evaluation does not compare the performance of dense retrievers using query2doc with other means of query/doc expansion, such as DPR/ES+Doc2Query.**\n\nIn terms of the evaluation, we aim to demonstrate that existing dense retrievers can be improved by using our query expansion techniques while keeping all other factors unchanged. This is an orthogonal contribution from document expansion, which can also be applied to further boost performance.\n\n**Q3: It might be useful to add a validation step that asks the LLM whether the generated passage answers the query. Additionally, a failure analysis should be included to allow deeper understanding of the limitations of the proposed approach.**\n\nWe agree that asking the LLM to verify its own generations is a promising idea to explore, and we will add this part of analysis in the future revision.\n\nRegarding the failure analysis, we have included several cases in Table 7 and 8 in the Appendix, demonstrating that LLMs may hallucinate facts in the generated pseudo-documents, which can have a negative impact on retrieval systems. However, based on our manual inspection, such cases are relatively rare for LLMs like davinci-003 / gpt-4. Thank you for bringing this up.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposes an approach to query expansion, query2doc, that can significantly improve the performance of both sparse and dense retrieval systems on ad-hoc IR datasets.\n\n2. The paper provides experimental results that demonstrate the effectiveness of query2doc and its benefits to state-of-the-art dense retrievers.\n\n3. The paper makes a computationally-aided linguistic analysis contribution by proposing a new approach to query expansion that leverages large language models.", "reasons_to_reject": "1. The proposed method is highly empirical and lacks theoretical justification, particularly in the case of repeating q for n times for sparse retrieval. It is empirical in the sense that the methodology does not appear to be able to generalize to other search tasks/datasets.\n\n2. The evaluation does not compare the performance of dense retrievers using query2doc with other means of query/doc expansion, such as DPR/ES+Doc2Query.\n\n3. The proposed approach prompts the LLM to make a guess about what a relevant document looks like, which could be incorrect and lead to degraded retrieval effectiveness. It might be useful to add a validation step that asks the LLM whether the generated passage answers the query. Additionally, a failure analysis should be included to allow deeper understanding of the limitations of the proposed approach. (Update: the case study is actually at the very end of the appendix.)", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper is about a query expansion approach called query2doc, which aims to improve both sparse and dense retrieval systems in information retrieval. The paper addresses the problem of query disambiguation and how to guide the retrievers to improve performance on ad-hoc IR datasets. The main contribution of this paper is the introduction of query2doc, which generates pseudo-documents through large language models to aid in query disambiguation and guide the retrievers. The paper shows that query2doc can significantly boost the performance of retrieval systems, particularly on ad-hoc IR datasets. This paper makes a computationally-aided linguistic analysis contribution by proposing a new approach to query expansion that leverages large language models. The paper also provides publicly available software for query2doc.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes Query2doc to generate a potential relevant passage with LLMs and uses it to expand the query. The experiments show the effectiveness of the approach.
The experiments confirm the usefulness of LLMs for improving ad hoc IR in this way. It can inspire other researchers to work in a similar direction in the future.
Decision: Accept-Main