LLM-Augmented Retrieval: Enhancing Retrieval Models Through Language Models and Doc-Level Embedding

Authors: Mingrui Wu, Sheng Cao
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: information retrieval, text retrieval, artificial intelligence, large language models, data augmentation

Abstract:
Recent advancements in embedding-based retrieval, also known as dense retrieval, have shown state of the art results and demonstrated superior performance over traditional sparse or bag-of-words-based methodologies. This paper presents a model-agnostic document-level embedding framework enhanced by large language model (LLM) augmentation. The implementation of this LLM-augmented retrieval framework has significantly enhanced the efficacy of prevalent retriever models, including Bi-encoders (Contriever, DRAGON) and late-interaction models (ColBERTv2). Consequently, this approach has achieved state-of-the-art results on benchmark datasets such as LoTTE and BEIR, underscoring its potential to refine information retrieval processes.

Decision: reject
Reviewer scores: 5, 8, 5, 5

Reviews and discussion:
<thread>
creation date:
1730179430711

thread_type:
official_review

<thread_object>
creator: reviewer_zrax

<content>
summary:
This paper proposes an LLM-augmented retrieval framework that uses a large language model (LLM), specifically LLaMA-70B, to generate pseudo-queries for document retrieval. For each document, the framework includes synthetic queries generated by the LLM, existing or LLM-generated titles, and document chunks. These text fields are embedded and combined with the query embedding using inner product, followed by weighted sum and max pooling. Evaluations are conducted with Contriever, Dragon, and ColBERT on the LoTTE dataset and a subset of the BEIR benchmark, with considerable improvements.

strengths:
1. The motivation is solid, with a straightforward idea: using pseudo-queries to augment document representation, an important direction given the short context windows of current embedding models that limits their effectiveness.

2. The observed improvement appears substantial on the presented datasets.

3. The authors provide some valuable insights that could be helpful, such as the impact of titles and approaches to handle cases where titles are missing, which aligns well with real-world scenarios.

weaknesses:
1. The cost of this solution appears too high, given that it relies on LLaMA-70B, and all documents require query generation. Additionally, I have concerns regarding potential data leakage in the LLM, as it may have previously encountered these query-document pairs, potentially generating similar queries based on the document. This issue could be addressed if the authors provided examples and analysis.

2. The inference process requires weighting strategies to combine the relevance between the query and different fields of the document. I have concerns about the scalability of this solution, as the presented dataset contains a relatively small number of documents (see Q1).

3. Some presentation issues: (i) The inconsistent capitalization of "Bi-encoder" is concerning, with both "Bi-encoder" and "bi-encoder" used interchangeably throughout the paper, which appears unprofessional. Additionally, on line 19, to my knowledge, ColBERTv2 falls under the category of bi-encoders but employs a more complex relevance scoring process. (ii) Certain findings in sections 3.1.2, 3.1.3, and others could be more effectively presented in an analysis section following the experiments section, providing a clearer flow and focused discussion on results.

questions:
1. I am curious whether this method could be effective on larger document benchmarks, such as MSMARCO and FEVER within the BEIR benchmark. Could the authors provide results on these datasets?

2. Could the authors provide an analysis of the costs associated with using an LLM to enhance these tasks, such as the number of tokens or text checks required for generation?

3. It would be interesting to see the performance when using the original query to search against the pseudo queries, to quantify any potential leakage from the LLM (i.e., the ability to recover the query based on the document).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730609912411

thread_type:
official_review

<thread_object>
creator: reviewer_fyxe

<content>
summary:
This paper proposes a novel retrieval framework called to improve the performance of current retrieval models. The framework achieves this by enriching document embeddings with information from large language models (LLMs), such as synthetic queries and titles. The proposed document-level embedding approach integrates this augmented information with existing document chunks, and it's shown to be effective for both bi-encoder and late-interaction retrieval models. The authors demonstrate the framework's effectiveness by achieving state-of-the-art results on benchmark datasets LoTTE and BEIR.

strengths:
The paper is clear and the results are good.

weaknesses:
It's a rather straightforward model. The novelty is simply calling some LLM to generate titles and queries for each doc, which the retriever will later use to retrieve the document. I think ICLR requires more model novelty.

questions:
All clear to me.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692094429

thread_type:
official_review

<thread_object>
creator: reviewer_z7jy

<content>
summary:
The paper presents an approach to document expansion using LLMs to enhance the zero-shot retrieval effectiveness of existing state-of-the-art bi-encoder retrievers (Contriever, DRAGON and ColBERTv2).

strengths:
1. The proposed method is straightforward and easy to implement and shows significant improvements for existing state-of-the-art retrievers.
2. The proposed approach is useful, which combines the document embedding from different sources and forms a single document embedding. Thus, the index size is still the same for single-vector dense retrieval.

weaknesses:
1. Although the proposed approach is simple and effective, I think the major concern for me is the limited discussion in ablations and comparisons, which are important for the readers who want to adopt this approach. I'm willing to raise score if the authors can address the concern below: (1) As a reader, one may want to know the quality and cost tradeoff of the generated titles and queries. For instance, the paper prompts Llama-70B to make it work; how about using smaller size of Llama; 7B, 13B etc.? and the additional cost of document indexing with LLM document expansion should be reported. (2) There are no ablations on the three combinations: chunk + query, chunk + title and query + title, which helps readers to choose the combinations with a balanced tradeoff between effectiveness and cost  (3) A comparison with LLM-based retrievers, such as RepLlama[1] or MistralE5[2], in terms of retrieval effectiveness, query latency and indexing time, can provide more useful information for readers to choose which approach to use. (4) Can we apply this approach to multilingual retrieval tasks?
2. Although the proposed method show improvements over existing SoTA retrievers, the proposed approach is a bit incremental. I think the approach to augment the document is very similar to the previous work[3][4]. The authors should make a comparison with the previous work in Related Work to make the contribution more clear.
3. Some experimental details are missing (see Questions).

[1] Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. 2023. Fine-tuning llama for multi-stage text retrieval.
[2] Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023. Improving text embeddings with large language models.
[3] Rodrigo Nogueira and Jimmy Lin. 2019. From doc2query to docTTTTTquery.
[4] Yongqi Li, Nan Yang, Liang Wang, Furu Wei, and Wenjie Li. 2023. Multiview identifiers enhanced generative retrieval.

questions:
1. Clarified on implementation details: according to Figure1, for each passage coming from the same document, it seems you use the same synthetic queries and titles? However, in the experiments, you mention you use the original chunk from the datasets; then, how do you know which chunks are coming from the same document and use the document to generate synthetic queries and titles for those chunks.
2. Which dev set are used to tune the hyperparameters, $w_{query}, w_{title}, w_{chunk}$?
3. From the prompts, it seems that the models are instructed to output multiple relevant queries; however, only one generated query is used for document expansion. How do you choose the query among all the generated ones? Or if you use multiple generated queries, how do you combine them?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730810942617

thread_type:
official_review

<thread_object>
creator: reviewer_nf7f

<content>
summary:
This paper introduces a novel, model-agnostic approach for document embedding, leveraging an augmented document field comprising synthetically generated queries, titles, and document chunks. With a new similarity function, the method computes a similarity score between the query and document field as a proxy for query-document relevance. The method outperforms using naive document embeddings from three existing retrievers (Contriever, DRAGON, ColBERTv2) on two benchmarks (BeIR, LoTTE).

strengths:
- I like the idea of enhancing document embeddings prior to the inference step. Specifically, this can be pre-computed, offering practical advantages over query expansion methods which have been studied extensively.
- Proposed method can be applied to both bi-encoder and token-level late-interaction models, and it achieves consistently strong results across most datasets.

weaknesses:
- There lacks detailed analysis of how (synthetically generated) document fields help the retrieval.  For example, how many queries were generated per document on average? The limitations section mentions that the augmented text may be as long as the original text, which suggests a substantial number of queries—clarification here would be valuable. Also, could you include some qualitative examples? 
- In the ablation study, weights for each field are exclusively set to 1.0. However, the “left term” at equation (1) remains active, meaning the query-chunks still affect the results. This raises concerns about whether the ablation study truly isolates the impact of each field, especially for the query and title.
- The results are evaluated only on the subset of BeIR benchmark without any explanations.

questions:
- How was the chunk size determined?
- I suggest a retouch on line 348-350 “augmentation of document embeddings with LLMs can substantially elevate the recall performance of retrieval models without necessitating additional fine-tuning”. I don’t think one model is simply fine-tuned relative to another, but it’s their complexity that differs.
- Could you cite some prior work on chunk-level embeddings (line 246-248)? Using a single document-level embedding, such as truncating, seems to be a common practice. In such cases, the proposed method will increase the document index size by |doc_len| / |chunk_size|.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732304799765

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal

comment:
# Rebuttal

We sincerely appreciate the insightful feedback provided by all reviewers. We are pleased to note that the reviewers recognize our paper as achieving robust, significant, and substantial results [**nf7F, z7Jy, fyxe, ZRAX**] through a straightforward method [**z7Jy, ZRAX**] that is easy to implement.


## Highlights and Common Questions
We summarize several highlights and common feedback as below.


### Our novelty is primarily derived from the following aspects
1. We present a **training-free**, straightforward yet effective method that demonstrates improvements over existing state-of-the-art (SoTA) retrievers [**nf7F, z7Jy, fyxe, ZRAX**].
2. Our approach maintains inference speed without any sacrifice, and the total cost is minimized as the retriever index is pre-computed only once prior to the inference step [**nf7F**]. Additionally, the index size remains consistent with that of single-vector dense retrieval while including richer information from multiple fields [**z7Jy**].
3. We provide valuable insights, such as the impact of titles on information retrieval and the corresponding methods to employ when titles are absent [**ZRAX**].
4. Our solution is **model-agnostic** (also multilingual-agnostic) and can be applied to both Bi-encoder and token-level late-interaction models [**nf7F**].


### We have enhanced the presentation of the paper with the following improvements
1. We have added a quantitative summary of the augmentation details, including the number of queries/tokens generated [**nf7F, ZRAX**] and the query match ratio [**ZRAX**], in Table 4, Section 4.5 with **blue** color, where we also discuss the cost and latency analysis of our proposed method. In short, the cost impact is limited.
2. We have included qualitative examples [**nf7F, z7Jy, ZRAX**] of synthetic queries: Table 11 in the Appendix, highlighted in **blue** color, uses four documents as examples to illustrate the sample synthetic relevant queries generated by LLMs.
3. We have improved the description of hyperparameter selection and experiments details [**nf7F, z7Jy**], highlighted in **orange** color in Section 4.3.
4. We also improved our notations in equations in **teal** color to address reviewer’s questions [**z7Jy**] in Section 3.2.1 regarding how synthetic queries work in doc-level embedding.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes an LLM-based method for document expansion in document retrieval. It enhances each document with synthetic queries, titles, and chunks generated by the LLM. The proposed method is training-free and model-agnostic because of the use of LLM. The method demonstrates substantial performance improvements across various retrievers (Contriever, DRAGON, ColBERTv2) and benchmarks (LoTTE, BEIR). Reviewers found the approach straightforward (z7Jy, ZRAX) and practical to implement (nf7F, z7Jy), with substantial empirical gains (fyxe, ZRAX).

The reviewers also raised several concerns:
1. The high inference cost of LLM-generated augmentations and the unfair comparison with previous methods in terms of inference efficiency (z7Jy, ZRAX).
2. The limited novelty, as the method is incremental with respect to existing document augmentation techniques (z7Jy, fyxe).
3. Concerns about the evaluation metrics (nf7F, z7Jy) and datasets (nf7F, ZRAX).

During the rebuttal, the authors provided additional results to address concern 3; however, concerns 1 and 2 were not sufficiently addressed. A rejection is recommended. We encourage the authors to improve the current version based on reviewers’ feedback for future publication.