IterKey: Iterative Keyword Generation with LLMs for Enhanced Retrieval Augmented Generation
Authors: Kazuki Hayashi
Hidetaka Kamigaito
Shinya Kouda
Taro Watanabe
Venue: COLM 2025 (2025)
Keywords: retrieval-augmented generation
RAG
sparse retrieval
LLM
Iterative
TL;DR: We introduce IterKey, an LLM-based iterative keyword generation method that optimize the Retrieval-Augmented Generation process, improving accuracy by refining keywords and self-evaluating responses.

Abstract: Retrieval Augmented Generation (RAG) has emerged as a way to complement the in-context knowledge of Large Language Models (LLMs) by integrating external documents.

However, real-world applications demand not only accuracy but also interpretability. 
Dense retrieval methods provide high accuracy but lack interpretability, while sparse retrieval is transparent but often misses query intent due to keyword matching.
Thus, balancing accuracy and interpretability remains a challenge.

To address these issues, we introduce IterKey, an LLM-driven iterative keyword generation framework that enhances RAG via sparse retrieval. 
IterKey consists of three LLM-driven stages: generating keywords for retrieval, generating answers based on retrieved documents, and validating the answers. If validation fails, the process iteratively repeats with refined keywords.

Across four QA tasks, experimental results show that IterKey achieves 5% to 20% accuracy improvements over BM25-based RAG and simple baselines. Its performance is comparable to dense retrieval based RAG and prior iterative query refinement methods using dense models.
In summary, IterKey is a novel BM25-based iterative RAG framework that leverages LLMs to balance accuracy and interpretability.

Published as a conference paper at COLM 2025
ITERKEY: Iterative Keyword Generation with LLMs for Enhanced Retrieval Augmented Generation
Kazuki Hayashi', Hidetaka Kamigaito', Shinya Koudał, Taro Watanabe "Nara Institute of Science and Technology FTDSE Inc. (hayashi wwa.kik.kkkkkk h14, tamigaito.h, aaro)eis.naist..
Abstract
Retrieval Augmented Generation (RAG) has emerged way to com- plement the in-context knowledge of Large Language Models (LLMs) by integrating external documents. However, real-world applications demand not only accuracy but also interpretability. Dense retrieval methods provide high accuracy but lack interpretability, while sparse retrieval is transparent but often misses query intent due to keyword matching, Thus, balancing accuracy and interprelability remains challenge. To address these issues, we introduce ITERKEY, an LLM-driven iterative keyword generation frame- work that enhances RAG via sparse retrieval. ITERKEY consists of three LLM-driven stages: generating keywords for retrieval, generating answers based on retrieved documents, and validating the answers. If validation fails, the process iteratively repeats with refined keywords. Across four QA tasks, experimental results show that ITERKEY achieves 5% 20% accuracy im improvements over BM25-based RAG and simple baselines. Its performance is comparable to dense retrieval based RAG and prior itera- tive query refinement methods using dense models. In summary, ITERKEY is a BM25-based iterative RAG framework that leverages LLMs to balance accuracy and interpretability.
1 Introduction
Large Language Models (LLMs) (OpenAI, 2023; Chiang et al., 2023; Dubey et al., 2024; Abdin et al., 2024) achieve strong performance in many natural language processing tasks but still face issues like hallucinations, outdated knowledge, and complex multi-hop rea- soning (Kandpal et al., 2023; Zhang et al., 2023b; Gao et al., 2024). These challenges are tasks (Lee et al., 2019; Zellers et al., 2018), where
pronounced in knowledge-intenste necessary information may not be fully stored or easily retrieved within model parameters.
Retrieval Augmented Generation (RAG) improves the accuracy and relevance of generated responses by integrating external knowledge, making it particularly effective for open domain question answering (Lewis et al., 2020; Izacard & Grave, 2021; Shuster al., 2021; Ram et al., 2023; Izacard et al., 2024). Although improvements to individual RAG com- ponents such as query expansion, document reranking, and retrieval tuning have been extensively studied (Gao et al., 2024; Fan et al., 2024), real-world applications demand both interpretability and accuracy from the retrieval component. Dense retrieval methods are highly accurate but suffer from limited interpretabilitt, while sparse retrieval methods are more transparent but often struggle to capture the underlying intent user queries (Kang et al., 2025; Cheng et al., 2024; Ayoub et al., 2024; Llordes et al., 2023).
To address this, we propose ITERKEY: Iterative Keyword Generation with LLMs, a BM25- based method that improves RAG through iterative keyword refinement and self evaluation. ITERKEY operates in three stages: keyword generation, answer generation, and answer validation. In each cycle, LLM extracts keywords, generates an answer, and checks correctness. the validation fails, it updates the keywords and repeats the process until satisfactory answer is obtained. By leveraging sparse BM25 retrieval, the contribution of

---

Published as a conference paper at COLM 2025
Keyword Generation
Iterkey
Talk to AI
User Query
What the name of the spocacroft that first landed humans Moon?
Keyword Regeneration
Keywords
londing,
New Keywords
Spacecraft, Name, Moon, Space
Moon landing, Apollo Spacecraft,
Lef
stenttive thinking.. thenctive thinking...
Retrieval
module nome, Apollo program
User Query
Keywords
Previous Keywords
4
In landing, Mimains
External
Spac
craft. No
Moon, Space
Engle!
"Eagle," the luncr module of Apollo 11 that first landed humons the Moon
False
User Query
+
Answer
Docs
Space Shuttle Challenger
2
Ohl Thank you!
Answer Space Shuffle Challenger
3
True
Answer Generation
Answer Verification
Figure 1: IterKey iteratively generates keywords, produces answers, and validates them using an LLM until the correct response is reached, thereby improving RAG accuracy.
each keyword to the score is explicit, enabling the tracking of generated keywords, retrieved documents, and validation results, which in turn improves accuracy and traceability.
Across four open domain QA datasets, ITERKEY improved retrieval performance by up to 20% over BM25 baselines and achieved 5% 20% accuracy gains over vanilla methods and BM25-based RAG, with performance comparable to Dense model based RAG and iterative query refinement methods. These results indicate that LLMs can infer and expand keywords effectively, highlighting the practical potential of LLM-driven RAG refinement that balances performance and interpretability in supporting human search efforts.
2 ITERKEY: Iterative Keyword Generation with LLMs
Sparse retrieval algorithms, such as BM25 (Robertson & Zaragoza, 2009), efficiently handle large datasets by ranking documents based on keyword frequency and uniqueness. They are interpretable, fast, and training-free
making them suitable for scalable, low-resource settings. Their transparency lets users see which keywords influenced retrieval. However, they often miss nuanced or implicit query intent, limiting effectiveness compared to dense methods (Llordes al., 2023; Izacard et al., 2022; Chen et al., 2023; Wang et al., 2022b). To address this limitation, we propose IterKey, a method that extends query expansion research (Wang et al., 2023a;b; Gao et al., 2023) by leveraging LLMs to iteratively refine the retrieval process in RAG. Specifically, IterKey uses the self-validation capabilities of LLMs (Wang et al., 2024) improve sparse retrieval through iterative keyword generation, document retrieval, answer generation, and validation. By using BM25 for retrieval, all keywords that contribute to the search remain explicit, allowing the generated keywords, retrieved documents, and validation results to be inspected directly. This design aims to pre- serve the interpretability of sparse retrieval while improving accuracy. Table summarizes the prompts used in each step of ITERKEY. We further explain the motivation behind each prompt design, with concrete examples provided in Figure to illustrate the full process.
Step 1: Keyword Generation Given a user query 9, the LLM generates a set of critical
keywords XCO essential for retrieving relevant documents, By leveraging BM25 retrieval and the query's 's intent, an LLM generates critical keywords from the initial query for retrieval. This step follows recent work on LLM-based query expansion (Xia et al., 2025; Lei et al., 2024) and enables the LLM to capture semantic and contextual keywords that sparse retrieval
2

---

Published as a conference paper at COLM 2025
Step
Prompt
Step I: Initial Key-
You are an assistant that generates keywords for information retrieval. important keywords related to the Query: (q) Focus on keywords that are relevant and likely to appear in documents for BM25 search in the RAG framework.
System:
word Generation
User: Generate
Output the keywords as: ['keyword1", "keyword2", "keyword3", ...] Separate each keyword with comma and do not include any additional text.
Step 2: Answer System: You are an assistant that generates answers based on retrieved docu- Generation using ments.
RAG
User: Here is question that you need to answer: Query: (q)
Below are some documents that may contain information relevant to the question.
Consider the information in these documents while combining it with your own knowledge to answer the question accurately.
Documents:
Provide clear and concise answer. Do not include any additional text.
Step 3: Answer Val- System: You are an assistant that validates whether the provided answer is correct.
idation
User:
Is the following answer correct?
Query: (
Answer:
Previous Retrieval Documents: Docs
Respond 'True' or 'False'. Do not provide any additional explanation or text.
Step Keyword Regeneration
System: You refine keywords to improve document retrieval for BM25 search in the RAG framework.
User: Refine the keyword selection process to improve the accuracy of retrieving documents with the correct answer.
Query: {q]
Previous Keywords: (K)
Provide the refined list of keywords this format: l'keywordl", "keyword2", ...)
Separate each keyword with comma and do not include any additional text.
Table 1: IterKey prompts for iterative keyword refinement and answer validation.
often misses, resulting in more effective document retrieval. In our example, for the query "What is the name of the spacecraft that first landed humans on the Moon?", LLM generates important keywords like Moon landing, Spacecraft, First humans.
Step 2: Answer Generation The original query q is expanded with the generated keywords K' to form an enhanced query q + KC' This expanded query is used to retrieve a set of top-k documents Dí = d2,,d2..... ..) from an external corpus using a BM25-based retriever. Using the retrieved documents, the LLM generates an answer a' to the original query 9. For example, the LLM generates the answer Space Shuttle Challenger using the expanded query.
Step 3: Answer Validation Recent studies (Asai et al., 2024; Yao et al., 2023) suggest that allowing an LLM critique its own answer after generation can improve answer validity. Building on this insight, we let the LLM act as a judge to decide whether the generated answer supported by the retrieved documents, using only the logical consistency between answer and evidence as the criterion (Zheng et al., 2023). The LLM is required to choose between 'True' or 'False' using force decoding, where the output is constrained to these two options. The generation probabilities for both are evaluated, and the option with the higher probability is selected as the answer. "True' is selected, the answer is considered correct, the process concludes, and is returned as the final answer If False' is selected, the process restarts to refine the retrieval and generation steps. After reviewing the documents, the LLM finds Space Shuttle Challenger incorrect and returns 'alse' in our example.
Step 4: Keyword Regeneration This step follows recent work iterative relevance feedback with LLMs (Mackie et al., 2023; Liu et al., 2024). If the LLM responds with 'alsse' in the validation step, we regenerate new set of keywords K to improve retrieval. The regeneration process uses the original query and the previous keywords as cues, allowing the LLM to refine its understanding of the query intent and progressively retrieve more relevant documents. The new keywords Kitl are then used to form a new expanded query q + K 11, and Steps 2 through 4 are repeated. This iterative process continues until the
3

---

Published as a conference paper at COLM 2025
validation step returns 'True' or a predefined maximum number of iterations i For example, after receiving 'Fllse, the LLM generates new keywords like Apollo 11, Lunar module name, and the correct answer Engle is finally retrieved in Figure 1.
is reached.
3 Experiments
3.1 Settings
Datasets and Evaluation Methods We evaluated ITERKEY on four open domain QA datasets: Natural Questions (NQ) (Kwiatkowski et al., 2019), EntityQA (Li et al., 2019), We- bQA (Chang et al., 2022), and HotpotQA (Yang et al., 2018). Following prior work (Trivedi et al., 2023; Jiang et al., 2023b; Feng et al., 2024), we randomly sampled 500 entries per dataset due to computational cost and conducted the evaluation under zero-shot settings. We evaluate generated answers using the Exact Match (EM) metric (Rajpurkar et al., 2016), which considers an answer correct if it matches any reference after normalization (lower- casing, removing articles and punctuation, and whitespace consolidation). To assess the impact of query expansion, we compute recall as the percentage of retrieved documents that contain at least one reference answer. We used the December 2018 Wikipedia dump as the retrieval corpus for all datasets (Izacard et al., 2024) following (Feng et al., 2024).
Retrieval Models We utilized BM25 (Robertson & Zaragoza, 2009), implemented using BM255 (Lù, 2024)', as the base retriever. Additionally, we adopted three Dense Models: Contriever (Izacard et al., 2022), BGE (Chen et al., 2023), and E5 (Wang et al., 2022b), under identical conditions to compare their retrieval performance against sparse models.
LLMs To examine the practical utility of ITERKEY across different model settings, we evaluate using four LLMs with varying capabilities: Llama-3.1 8B and 70B (Dubey al., 2024), Gemma-2 (Team et al., 2024), and Phi-3.5-mini (Abdin et al., 2024). This analysis examines how generation and validation abilities impact each component, offering insights for practical model selection. All models can produce the structured outputs required by each ITERKEY step. See Appendices and for details.
Comparative Baseline As a representative iterative baseline, we adopt ITRG (Iterative Retrieval-Generation Synergy) (Feng et al., 2024), the strongest publicly available method in this line of work. ITRG repeatedly concatenates L.M--gnerated answer passages with the original query refine retrieval quality, In contrast, IterKey generates explicit keyword lists and introduces an answer validation step that halts iteration once the answer is deemed correct. ITRG integrates established single-pass query expansion techniques (Wang al., 2023b;a; Gao et al., 2023), positioning it as an appropriate, advanced comparator.
3.2 Results
IterKey Baseline & RAG (BM25) Table 2 shows that ITERKEY consistently improves accuracy over the Baseline across all models, achieving gains of 10% to 25%. Particularly, Llama-3.1 models of 8B and 70B parameters show notable improvements with TTERKEY, with the 8B model reaching accuracy levels similar to the 70B model. This demonstrates ITERKEY's effectiveness in enhancing the performance of smaller models, allowing them to achieve accuracy levels closer to larger ones. Further comparison with BM25-based RAG reveals a 5% to 10% accuracy increase on Llama-3.1 when using ITERKEY. However, ITERKEY performs comparably to RAG (BM25) on Phi-3.5-mini, while falling behind RAG (BM25) on Gemma-2, with no notable improvements on the other tasks. These findings highlight the effectiveness of ITERKEY, although the benefits vary across models.
IterKey vs. RAG (E5) & ITRG As shown in Table 2, ITERKEY with BM25-based retrieval outperforms E5 based RAG 3lama-3.1 models. Notably, the Llama-3.1 70B model achieves
thttpss://gitieee com/xhlucaa//m255
4

---

Published as a conference paper at COLM 2025
Method
Model
Size (B)
Entity QA
HotpotQA
Natural QA
WebQA
Llama-3.1
8B
33.6
31.2
40.6
53.4
Vanilla
Llama-3.1
70B
45.2
41.4
46.0
54.0
Gemma-2
9B
10.6
11.6
9.2
20.8
Phi-3.5-mini
3.8B
24.6
25.4
25.8
44.0
Llama-3.1
8B
54.0
47.0
44.8
51.4
RAG (BM25)
Ilama-3.1
70B
54.6
46.2
43.4
47.4
Gemma-2
9B
47.9
39.6
33.2
41.6
Phi-3.5-mini
3.8B
42.2
32.6
Llama-3.1
8B
52.9
47.7
49.6
48.2
RAG (ES)
Llama-3.1
70B
57.0
51.0
49.4
48.8
Gemma-2
9B
52.2
40.8
41.5
41.8
Phi-3.5-mini
3.8B
50.2
44.6
37.0
41.4
1.ama-3.1
8B
60.6
53.4
53.6
56.2
ITRG Refresh (E5)
Llama-3.1
70B
60.7
52.9
53.3
51.6
Gemma-2
9B
54.2
47.6
47.4
48.5
Phi-3.5-mini
3.8B
54.3
47.1
36.2
44.6
Llama-3.1
8B
62.0
49.7
30.2
53,3
IterKey (BM25)
Llama-3.1
54.9
53.8
Gemma-2
9B
36.1
26.1
35.9
36.1
Phi-3.5-mini
3.8B
42.7
36.8
Table 2: 'Vanilla' uses no retrieval. 'RAG (BM25) and 'RAG (E5) perform a single retrieval step on the original query using BM25 and E5, respectively. Other dense retriever results are in Table 18. TTRG Refresh (E5y is prior iterative refinement method, reproduced here with E5 (its best dense retriever) and five query expansions. Two approaches, Refine and Refresh, exist; see (Feng et al., 2024) for details. Refine results are in Table 19, while Refresh is in the main text. Our 'IterKey (BM25) iteratively generates and refines keywords for up to five retrieval iterations to optimize answer quality. In the table, underlined values mark the best performance for each model, and bold values the highest accuracy overall.
RM25
E5
1ama--31111 3.188
Llana 3,1-708
Gentan 2-98
Phu-3 m-mini--333BB
Percentage (%)
60.0
65.2
70.0
63.8
59.6
56.0
56b
56.
60.0
31
4
40.0
31
Top- Recall Herkey (BM25)
Percetinge (%)
80.0
63
68.
59
53.3
56.9
54
47
43.0
40.0
5
Top-K Recall Berkey (E5)
Percanage (%)
50.0
5100
69
2o
2
66
5
76
62
700
60.0
51.0
40.0
316
20.0
4
5
2
5
2
4
Top-K Recall
ITRG (E5)
Figure 2: The figure shows the Top-K retrieval recall rates for each model the Entity QA task, representing the proportion of retrieved documents containing the correct answer. The baseline methods, 'BM25' and 'E5,' use a single retrieval step based on the original query. Our proposed method, 'IterKey (BM25), is compared with 'IterKey (E5y for dense retrieval and the prior work TTRG (E5).' Recall rates are averaged over five iterations.
the highest accuracy across all tasks, outperforming ITRG, an iterative refinement method that employs dense retriever. Moreover, the Llama-3.1 8B model attains accuracy compa- rable to ITRG, demonstrating that ITERKEY can effectively compete with dense retrieval based iterative strategies even at smaller scales. However, ITERK effectiveness does not

---

Published as a conference paper at COLM 2025
extend uniformly across all models. While ITRG consistently improves performance for Gemma-2 and Phi-3.5-mini, ITERKEY does not achieve comparable gains in these models. Additionally, compared to E5 based RAG, its performance remains lower across most tasks.
Analysis
4.1 Recall and QA Performance in IterKey
Figure 2 shows recall rates (%) on the Entity QA task for three methods and four models, av- eraged over five iterations. For comparison, BM25 and E5 (leftmost) show a single retrieval step using the original query. Recall is the percentage of retrieved documents containing the correct answer. IterKey (BM25) improves recall by approximately 20% over standard BM25. For the Llama-3.1 models, the top-5 recall is about 10% higher than that of E5, indicating that LLMs effectively generate and expand keywords to enhance retrieval. Although Gemma and Phi outperform BM25, their recall remains below E5, implying keyword generation quality depends heavily on the model. Despite achieving approximately 60% top-3 recall, Gemma and Phi exhibit QA accuracy that is 20-30 percentage points lower (see Table 2), suggesting weaknesses in answer generation or validation significantly reduce performance.
Comparing IterKey (BM25) and IterKey (E5) reveals similar Top-1 performance; however, IterKey (BM25) achieves 10% higher Top-5 precision and higher overall QA accuracy, see Table 6. This likely reflects that plain keyword lists align well with BM25, while E5, trained for natural-language queries, handles keyword strings less effectively. These results show that our iterative keyword refinement is especially effective for sparse retrieval models.
To further analyze how accuracy evolves over iterations, Table reports per-iteration results. Most performance gains occur in the first one or two iterations, with accuracy improving by up to 7%. After the third iteration, improvements taper off, indicating diminishing returns. This demonstrates that IterKey balances efficiency and effectiveness by capturing most of the benefits early while retaining the flexibility to continue when necessary.
Step
Step
Step
Step 5
Model
Stop 2
Step 3
Step
Step
Mean
Llama-3.1-8B
10.5
9.57
8.59
8.20
36.9
9.21
Llama-3.1-8B
2.12
1.65
1.53
1.32
6.62
1.66
Llama-3.1-70B
9.34
3.65
3.78
3.26
20.0
5.01
Llama-3.1-708
1.93
1.44
1.15
1.25
5.77
1.44
Gemma-2
14.1
8.56
5.70
4.59
33.0
8.25
Gemma-2
2.22
1.61
1.27
1.12
6,22
1.55
Phi-3.5-mini
18.1
7.12
6.00
5.07
36.3
9.07
Phi-3.5-mini
2.03
0.69
1.04
0.95
4.71
1.18
Table 3: Average number of new keywords generated per query each iteration.
Table 4: Average number of new documents retrieved per query at each iteration.
4.2 Keyword and Document Expansion by Iteration
To clarify the role of each iteration in our method, we show the average number of new keywords and documents added at each step. Table presents the average number of newly generated keywords per query at each iteration from Step 2 to Step 5. Table 4 shows the corresponding statistics for newly retrieved documents. These results show that each iteration continues to introduce new information. On average, each step generates between 3 and 18 additional keywords per query, resulting in total of 20 to 37 new keywords after four steps. Similarly, each iteration retrieves approximately 0.7 to 2.2 new documents per query, with total of 4.7 to 6.6 documents accumulated through the full process. These findings indicate that later iterations are effective in generating novel keywords and retrieving previously unseen documents. The iterative process enables the method to maintain both efficiency and coverage, it allows early termination for easier queries while providing additional refinement for more challenging cases.
4.3 Keyword Generation Step
Based on these results, we conducted additional experiments to further analyze the impact of keyword quality ITERKEY's performance. Specifically, evaluated the effects of

---

Published as a conference paper at COLM 2025
Model
Iter 1
Iter
Iter 3
Iter 4
Iter
Llama-3.1 8B
62.2
65,4
+3.2
69.6
+7.4
70.5
+8.3
72.6
+10.4
Llama-3.1 70B
65.7
70.2
+4.5
73.5
+7.8
74.4
+8.7
75.7
+10.0
Gemma-2 9B
53.2
57.6
+4.4
60.0
+6.8
61.2
+8.0
63.2
+10.0
Phi-3.5 mini
57.5
64.5
+7.0
67.9
+10.4
68.8
+11.3
69.6 +12.1
Table 5: Accuracy (%) on the Entity QA task for each iteration. Values in blue boxes show absolute gains from Iter 1.
Method
Model
Size (B)
Entity QA
HotpotQA Natural QA
WebQA
Llama-3.1
8B
62.0
49.7
50.2
53.3
IterKey (BM25)
Llama-3.1
70B
63.5
54.9
53.8
57.7
Gemma-2
9B
36.1
26.1
35.9
36.
Phi-3.5-mini
3,8B
47.9
42.7
36.8
41.2
Llama-3.1
8B
-1.8
60.2
+4.1
53.8
+1.2
51.4
-2.7
50.6
Iterkey (E5)
Llama-3.1
70B
-1.3
62.2
-2.1
52.8
-0.6
53.2
-577
52.0
Gemma-2
9B
-1.6
34.5
-0.9
25.2
-0.7
35.2
-3.7
32.4
Phi-3.5-mini
3.8B
-0.5
47.4
+1.8
44.5
-2.4
34.4
-1.0
40.2
Table 6: Performance of ITERKEY with 'BM25' and the Dense Model ('E5'. This table shows
the QA performance when replacing 'BM25' with 'E5' under consistent settings.
using high quality keywords generated by Llama 3.1 70B, which achieved the highest recall, and low quality keywords from Gemma 2, which had the lowest recall. As shown in Table using high-quality keywords consistently improved accuracy across all models, notably resulting in about 5% gain for Gemma 2. Conversely, employing low-quality keywords degraded performance for all models and tasks, with Llama 3.1 (8B) experiencing particu- larly significant accuracy drop of 4.5% on EntityQA. These results clearly demonstrate that keyword quality plays a crucial role in ITERKEY's performance.
Notably, we observed that even when provided with high-quality keywords, Gemma-2's accuracy still lagged considerably behind Llama-3.1. Given that ITERKEY iteratively refines retrieval and validation, this suggests that Gemma-2's primary performance bottleneck lies in its validation capability rather than in keyword generation.
4.4 Answer Validation Step
We conduct additional experiments to examine validation model quality affects IterKey performance. We compare the high quality validation model Llama 3.1 70B, which achieves the highest baseline accuracy, with the low quality validation model Gemma, which shows the lowest baseline accuracy. As shown in Table 5, using high quality validation model consistently improves accuracy across all base models. Gemma improves by more than five points on average over four tasks. In contrast, Gemma lowers accuracy for all models, with LLaMA-based models dropping by over ten points, showing validation quality is critical for IterKey and that Gemma's limited validation capability severely constrains its performance.
We also examined how IterKey's Answer Validation step performs over repeated iterations. Specifically, we iterated up to five times, including after 'True' judgment, and compared accuracy across three settings to assess each model's validation performance. As shown in Table 9, comparing Verified with Base' reveals substantial discrepancies in validation accuracy across all models, with Gemma-2 showing the largest gaps (16.9% in HotpotQA and 15.2% in Entity QA). These results indicate that validation errors are more prominent in weaker models, highlighting the challenge of ensuring reliable answer verification.
The miss True rate ('Base' vs. W'rifiedTrree" measures the tendency to accept wrong answers as correct, while the miss False rate ('VerifiedTrue' vs. 'VerifiedAAll) quantifies failure to recognize correct answers under stricter validation. For Llama-3.1-8B, Llama-3.1-70B, and Phi-3.5-mini, 'miss True' is consistently higher than 'miss False,' indicating tendency to
7

---

Published as a conference paper at COLM 2025
Method
Model
Size (B)
Entity QA
HoipoiQA Natural QA
WebQA
1lama-3.1
8B
62.0
49.7
50.2
53.3
Iterkey
Llama-3.1
70B
63.5
54.9
53.8
57.7
Gemma-2
9B
36.1
26.1
35.9
36.1
Phi-3.5-mini
3.8B
47.9
42.7
36.8
41.2
Llama-3.1
8B
63.4
52.8
+1.4
51.6
+0.7
54.0
Iterkey
1lama-3.1
70B
63.5
54.9
53.8
57.7
w/ HQ Keywords
Gemma-2
9B
+3.9
40.0
44.4
30.5
+1.6
37.5
+3.9
40.0
Phi-3.5-mini
3.8B
+1.9
49.8
+1.5
44.2
+1.2
38.0
+2.2
43.4
Llama-3.1
8B
-6.0
56.0
-0.5
49.2
50.2
-1.4
51.9
IterKey
Llama-3.1
70B
57.1
-2.6
52.3
-3.3
50.5
52.6
w/ LQ Keywords
Gemma-2
9B
36.1
26.1
35.9
36.1
Phi-3.5-mini
3.8B
-2.8
45.1
-2.2
40.5
-2.4
34.4
+0.2
41.4
Table 7:
Ablation study keyword quality in ITERKEY. 'High-quality (HQ) keywords' are generated by Llama-3.1-70B, the model with the highest retrieval recall, and reused by smaller models with up to 10x fewer parameters, consistently boosting performance. 'Low- quality (LQ) keywords,' generated by Gemma-2-99 degrade performance across models.
Method
Model
Size (B)
Entity QA
HotpotQA Natural QA
WebQA
Llama-3.1
8B
62.0
49.7
50.2
53.3
Iterkey
Llama-3.1
63.5
54.9
53.8
57.7
Gemma-2
9B
36.1
26.1
35.9
36.1
Phi-3.5-mini
3.8B
47.9
42.7
36.8
41.2
Llama-3.1
8B
+0.5
62.5
+2.0
51.7
+2.0
52.2
+1.2
54.5
IterKey w/
Llama-3.1
70B
63.5
54.9
53.8
57.7
HQ Validation Model
Gemma-2
9B
+7.4
43.5
+8.0
34.1
+4.6
40.5
+3.3
39.4
Phi-3.5-mini
3.8B
+2.9
50.8
+1.9
44.6
+1.0
37.8
+2.2
43.4
Llama-3.1
8B
-10.4
51.6
-8.0
41.7
44.5
-7.1
46.2
IterKey w/ LQ Validation Model
Llama-3.1
70B
-10.7
52.8
-11.9
43.0
-11.4
42.4
-5.9
51.8
Gemma-2
9B
36.1
26.1
35.9
36.1
Phi-3.5-mini
3.8B
-2.6
45.3
-2.6
40.1
32.8
-3.0
38.2
Table 8: Performance comparison of the IterKey validation step. The 'High Quality' uses Llama-3.1-70B for the validation step, whereas the 'Low Quality' employs Gemma, the model with the lowest baseline accuracy. Only the validation step is modified, and the performance changes across four tasks (improvements in blue, degradations in red) are shown.
accept incorrect answers and limiting accuracy gains. In contrast, Gemma-2 shows higher 'miss False' rate, reflecting difficulty identifying correct answers under stricter validation.
Overall, these results show that validation errors, particularly 'miss True,' significantly constrain accuracy improvements. Thus, a model's validation capability is essential when applying ITERKEY. Moreover, adherence to output format does not necessarily imply strong instruction comprehension. Even the models that follow formatting constraints may fail to apply validation rules accurately, as seen in Gemma-2's validation errors. This aligns with prior studies (Kung & Peng, 2023; Zhou al., 2023b; Liang et al., 2024) emphasizing the distinction between format adherence and instruction comprehension in LLMs.
4.5 Keyword Regeneration Prompt
We compared IterKey (BM25), which reuses previous keywords proxy for documents in subsequent iterations, with a document-by-doccmeen
approach that regenerates and concatenates keywords for each retrieved document for the next retrieval. Since we use three documents at retrieval time, in a single iteration, the model ends up generating keywords two additional times. The prompt
used is shown in Table 10.
As shown in Table 11, the approach that generates keywords based on each retrieved document achieves higher recall for Gemma-2. However, for Llama-3.1 (8B, 70B) and Phi- 3.5-mini, our original IterKey (BM25) approach still produces higher accuracy. Moreover, from cost performance perspective, using previously generated keywords proxy for
8

---

Published as a conference paper at COLM 2025
Setting
Model
Size (B)
Entity QA
HotpotQA Natural QA
WebQA
Llama-3.1
88
62.0
49.7
50.2
53.3
Base
1lama-3.1
70B
54.9
53.8
57.7
Gemma-2
9B
36.1
26.1
35.9
36.1
Phi-335-mini
3.8B
47.9
42.7
36.8
41.2
Llama-3.1
8B
63.9
55.0
+4.3
54.5
VerifiedTrue
Llama-3.1
70B
+2.3
65.8
+2.1
57.0
+4.0
57.8
+4.7
62.4
Gemma-2
9B
+4.9
41.0
34.2
+3.5
39.4
44.4
40.5
Phi-3.5-mini
3.8B
+4.6
52.5
+4.4
47.1
+3.7
40.5
+5.6
46.8
Llama-3.1
8B
14.6
66.6
8.9
58.6
+10.01
60.2
+7.5
60.8
VerifiedAll
Llama-3.1
70B
+4.3
67.8
+6.6
61.5
+10.2
64.0
+6.5
64.2
Gemma-2
9B
+
14.3
50.4
+ 16.4
42.5
+7.9
43.8
11.5
47.6
Phi-3.5-mini
3.8B
+7.7
55.6
+7.5
50.2
+8.6
45.4
+11.2
52.4
Table 9: Accuracy results for different models and settings across four QA tasks. Settings: 'Base' uses the first "True" in the Answer Verification step the answer, then stops. "Ver- ified'True' checks if any iteration judged "True" contains the correct answer. 'VerifiedAll' checks all iterations, both "True" and "False," for the correct answer.
Step
Prompt
Step
the provided keywords to enhance document retrieval accuracy Regen- BM25 search in the RAG framework.
Key-
System:
word
eration
with
User: Please refine the keyword selection process to improve the accuracy retrieving documents containing the correct
Does
Query: q
Previous Keywords: (K)
Previous Reirieval Documents: Docs }
Provide the refined list of keywords this format: ["keywordl", "keyword2", ...
Ensure each keyword is separated by comma, and do not include any additional text.
Table 10: Compared the Step prompt in Table 1, this prompt incrementally adds 'Previous Retrieval Documents' at a time and regenerates new keywords at each step.
documents remains more efficient. Hence, while the document--y- approach is effective for Gemma-2, our method of using previously generated keywords as a proxy offers more favorable balance of accuracy and cost in most cases.
4.6 Computational Cost
We evaluated how well ITERKEY balances accuracy and computational cost compared to other methods. Table 12 compares runtime for the Entity QA dataset using Llama-3.1-SB For fair comparison, both ITERKEY and ITRG used E5 for retrieval. Inference on an NVIDIA RTX 6000 Ada GPU, while BM25 retrieval used an Intel Xeon Platinum 8160 CPU. As result, IterKey achieved comparable accuracy to ITRG while using only approximately 80% of the latency per question, with an average latency of 3.43 seconds.
performed
Table 13 shows the average iterations required by ITERKEY across all QA tasks and mod- els. The count stays below 1.5, with each iteration involving three LLM calls: keyword generation, answer generation, and validation. Multiplying the 3.43 base latency by the average 1.5 iterations yields x 5.15 per question. In summary, while iterative reasoning introduces a certain overhead (about 4 seconds slower than non-iterative baselines), the per- question latency remains well within acceptable bounds for real-time QA with GPU-based inference. Given the substantial improvements in accuracy and robustness, we consider this reasonable trade-off in practical settings where reliability and interpretability are essential.
Related Work
Retrieval Augmented Generation (RAG) RAG enhances response accuracy and relevance by incorporating external knowledge, making effective for open-domain QA (Lewis et al., 2020; Izacard & Grave, 2021; Shuster et al., 2021; Izacard et al., 2024). It also addresses the limitations of in-context learning, which struggles with complex questions and outdated information (Liu et al., 2022; Petroni et al., 2019; Borgeaud et al., 2022; Roberts et al., 2020).

---

Published as a conference paper at COLM 2025
Model
Size (I)
Top1 (%)
Top2 (%)
Top3 (%)
Top4 (%)
Top8
1a1
51.4
60.8
66.8
69.8
72.6
Heereey (BM25)
Llama-3.1
70B
56.6
65.2
70.0
73.8
75.7
Gemma-2
98
35.6
47.2
55.4
59.2
63.2
Pni-3.5-mins
3.8B
46.4
58.2
63.8
67.8
69.6
Heereey (BM25)
1ama-331
1aama-311
70B
+3.6
59.0
55.0
65.4
60.0
68.2
63.4
65.2
704
-29
466
72.8
68.0
document-by-document
Gemma-2
58.2
62.2
65.4
Phi-3.5 m-nin
55.8
60.6
64
Table 11: Retrieval performance comparison between the original IterKey (BM25) and the
document-by-documee keyword regeneration approach.
Step
RAG(BM25) RAG(ES) ITRG IterKey
Query Expansion
1.33
Model
Size (B)
Entity
Hotpot
Natural
Web
Retrieval
0.44
0.11
0,88
0.33
Llama-3.1
8B
1.27
1.38
1.35
1.36
Answer Validalion
Answer Generation
0.64
0.72
Llama-3.1
70B
1.15
1.20
1.23
1.10
Gemma-2
1.33
1.36
1.34
1.37
Total
1.08
0.73
4.26
3.43
Phi-3,5-mini
3.8B
1.38
1.32
1.28
1.29
Table 12: Average latency (s) for each step in Entity QA with LLaMA-3.1-8B.
Table 13: Average iterations for ITERKEY.
Recent work has focused on improving components such as query expansion, retrieval tuning, summarization, and reranking (Gao et al., 2024; Fan et al., 2024). RAG's effectiveness depends on both its components and their interaction (Jiang et al., 2022; Zhang et al., 2023a), yet these processes often function independently, making coherence difficult (Shi et al., 2024; Guo et al., 2023). Even when relevant documents are retrieved, they may not be fully used in generating responses (Ram et al., 2023; Dai et al., 2023; Xu et al., 2024; Asai et al., 2024), and retrieval failures can lead to irrelevant outputs (Jiang et al., 2023b; Guo et al., 2023).
Query and Document Expansion Recent methods improve retrieval by generating pseudo documents from queries, benefiting both sparse and dense systems (Wang et al., 2023a). Generating hypothetical documents has also enhanced zero-shot dense retrieval without relevance labels (Gao et al., 2023). These approaches leverage LLMs to address short or ambiguous queries by adding relevant context (Zhang et al., 2023b; Asai et al., 2023).
Iterative and Chain-of-Thought Retrieval Iterative retrieval methods leverage interme- diate reasoning to refine queries for complex, multi-step information needs, integrating generative and Chain-of-Thoughh (CoT) reasoning to dynamically update queries and re- sponses (Shao et al., 2023; Kim et al., 2023; Feng et al., 2024; Sun et al., 2023; Jiang et al., 2023b; Arora et al., 2023; Wang et al., 2022a). CoT-guided retrieval further enhances docu- ment relevance, factual accuracy, and reduces hallucinations (Trivedi et al., 2023; Creswell et al., 2023; Zhou et al., 2023a; Yao et al., 2023; Wei et al., 2022; Kojima et al., 2024; Zheng et al., 2024). Both supervised and unsupervised iterative approaches excel in multi-hop QA and knowledge-intenste
tasks, making them effective even in few-shot and zero-shot scenarios (Das et al., 2019; Xiong et al., 2021; Khattab et al., 2023).
Conclusion
We introduce ITERKEY, novel BM25-based framework that refines Retrieval Augmented Generation (RAG) by dividing the process into three stages: keyword generation, answer generation, and answer validation, all performed iteratively by an LLM. Experimental results across four open domain QA datasets show that ITERKEY improves retrieval perfor- mance by 20% over standard BM25 baselines and achieves 5% to 20% accuracy gains over vanilla and BM25-based RAG. It also delivers performance comparable to Dense model- based RAG and prior iterative refinement methods. These results demonstrate that LLMs can infer and refine keywords effectively to improve retrieval and answer quality, while preserving the transparency of sparse retrieval. Our findings highlight the potential of LLMs to enhance both accuracy and interpretability in real-world RAG applications.
10

---

Published as a conference paper at COLM 2025
Ethical Considerations
Language models are known to sometimes generate incorrect or potentially biased informa- tion. This issue is particularly significant when sensitive questions are posed to the model. While our retrieval-augmented approach is expected to mitigate this problem to some extent by grounding responses in external sources, does not fully eliminate the risk of biased or offensive outputs. Therefore, careful consideration is required when deploying such systems in user facing applications to avoid unintended harm. All datasets and models used in this work are publicly available under permissible licenses. The Natural Questions dataset is provided under the CC BY-SA 3.0 license, and the WebQuestions dataset is also available under the CC BY-SA 3.0 license. The data related to EntityQA are distributed according to their respective licensing terms. Additionally, the use of these datasets is permitted for research purposes. Note that in this work, used AI assistant tool, ChatGPT, for coding support.
References
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mah- moudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang. Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guan- hua Wang, Lijuan Wang, Shuohang Wang, Xin Wang, Yu Wang, Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Wei- jian Xu, Jilong Xue, Sonali Yadav, Fan Yang, Jianwei Yang, Yifan Yang, Ziyi Yang, Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL hwtph:tt/////iitii http:///rxiii.oorr////////////// 14219.
Daman Arora, Anush Kini, Sayak Ray Chowdhury, Nagarajan Natarajan, Gaurav Sinha, and Amit Sharma. Gar-meels-rag paradigm for zero-shot information retrieval. CoRR, abs/ 2310.22158, 2023. URL http: /ttp:///bblc /ttlp/llpp..// fncllp.unii-- tniitrrir... de/ de/db/ http:///uuuaasssor//orrrr/////// 2onllaab-2311-220
Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. Task-aware retrieval with instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 3650-3675, Toronto, Canada, July 2023. Asso- ciation for Computational Linguistics. doi: 10. 10.18653/ (6653/vl/// www../...iiinns-aae.. https://aclanthollee org/2023. wtt.2223...iii findings-acl. 225.
URL
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth Interna- tional Conference on Learning Representations, 2024. URL httpss//opeereeeee id=hSyW5go0v8.
net/forum?
Michael Antonios Kruse Ayoub, Zhan Su, and Qiuchi Li. A case study of enhancing sparse retrieval using llms. In Companion Proceedings of the ACM Web Conference 2024, WWW '24
11

---

www.orrmmiigggeeWWWM.... Published as a conference paper at COLM 2025
Pp. 1609-1615, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400701726. doi:
URL https:://oi..ooo /t/.//0.....r111111144333333333 1145/359333.
3651945.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang. Sheng- guang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report, 2023. URL wttph::t///ee //axxiv. wogg ha/aabbs//30 16609.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saf- fron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 2206-2240. PMLR, 17-23 Jul 2022. URL p.lr.pres//V wttps//v/2/bbrrgauude html.
Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. Webqa: Multihop and multimodal qa. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16495-16504, June 2022.
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, multi nulti-functionality. multi-granularity text embeddings through self-knowledge distillation, 2023.
Yiruo Cheng, Kelong Mao, and Zhicheng Dou. Interpreting conversational dense retrieval by rewriting-enhanced inversion of session embedding. In Lun-Wei Ku, Andre Martins, and Vivek Šrikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2879-2893, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v hvl3 (0663//v1/024.ac-/// 2022.acl--ong. 159. URL https:://aclatthollog acc- acl-long. 159/.
Wei-Lin Chiang. Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph F. Gonzalez, Ion Stoica, and Eric Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023,
URL http:pss//inssyssorr////////2/// -0-3ivvicaa.
Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inferencee Exploiting large language models for interpretable logical reasoning. In The Fleventh International Conference on Learning Represenfations, 2023. URL https: https://openrreview. net/forumiid= 3PF3Wg60-A4.
Zhuyun Dai, Vincent Y Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith Hall, and Ming-Wei Chang. Promptagator: Few-shot dense retrieval from examples. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreviees
Rajarshi Das, Ameya Godbole, Dilip Kavarthapu, Zhiyu Gong, Abhishek Singhal, Mo Yu, Xiaoxiao Guo, Tian Gao, Hamed Zamani, Manzil Zaheer, and Andrew McCallum. Multi- step entity-centric information retrieval for multi-hop question answering. In Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen (eds.), Proceedings of the 2nd Workshop Machine Reading for Question Answering, pp. 113-118, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: /1/3/191119 URL https: //acl////l/laa aaacannnaologyyy www.:ggg..rgg///// 0r8/
12

---

Published as a conference paper at COLM 2025
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Carcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, lliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, I'engchuan Zhang, l'engwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stujnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenmaa, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching- Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Fsteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun
13

---

Published as a conference paper at COLM 2025
Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Ibrahim Damlai, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Carg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vitor Albiero, Vlad lonescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The Ilama 3 herd of models, 2024. URL htttp:://arrett https:///arri..oo/aa//4/000 21783.
Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A survey on rag meeting llms: Towards retrieval- atgmenee- large language models, 2024. URL https::t///// //arxi///vvvv///// htt//bb//20555 06211.
Zhangyin Feng, Xiaocheng Feng, Dezhi Zhao, Maojin Yang, and Bing Qin. Retrieval- generation synergy augmented large language models. In ICASSP 2024 2024 LEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 11661- 11665, 2024. doi: 10.1109/ICASE 000SP448885.222410425
Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. Precise zero-shot dense retrieval without relevance labels. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Anmull Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1762-1777, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653 2t1//22.acllooogg... URL https://aclantteen acl-long.99 99.
org/2023.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey, 2024. URL htttpp://aaxiwworr//aaasa//////
10997.
Zhicheng Guo, Sijie Cheng, Yile Wang, Peng Li, and Yang Liu. Prompt-guided retrieval augmentation for non-knowledge-ntensivee tasks. In Anna Rogers, Jordan Boyd-Graber,
14

---

Published as a conference paper at COLM 2025
and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, PP. pp. 10896-10912, Toronto, Canada, July 2023. Association for Computational Lin- guistics. doi: 10.18653/ /12033/1v/// htl/222..inninngssss.ss.sss findings-acl. 693.
URL https:///allanthooogg 2023..
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative mod- els for open domain question answering. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume, pp. 880, Online, April 2021. As- sociation for Computational Linguistics. doi: 10. 10.18653/ /v/5//11/// /22222121eaa maci-mai....7 URL https://aclanthologgy orgh org/2021 eacl-main. 74.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Ar- mand Joulin, and Edouard Grave. Unsupervised dense information retrieval with con- 09118.
trastive learning, 2022. URL
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: few-shot learning with retrieval augmented language models. J. Mach. Learn. Res., 24(1), March 2024. ISSN 1532-4435.
Albert Q- Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anm Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023a. URL http:://arrt..org//b//3311 wrgp/bb/////...oo. 06825..
Zhengbao Jiang, Luyu Gao, Zhiruo Wang, Jun Araki, Haibo Ding, Jamie Callan, and Graham Neubig. Retrieval attention: End-to-end learning of retrieval and reading within a single transformer. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 2349, Abu Dhabi, United Arab Emirates, December 2022. Associa- tion for Computational Linguistics. doi: https://aclantholoogy org/20222 emnlp-main.14
URL
Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yo Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 9999, Singapore, December 2023b. Association for Computational Linguistics. doi: 10.18653 /v1 htt1//22.emmmpppp m00..emmnp.mmii.. 495. URL https: hwwp//oooonntttoo..ooooo or org/2023. emnlp-main 495.
of
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 15696-15707 PMLR, 23-29 Jul 2023. URL https://prroceedingg mlr.
html.
Hao Kang, Tevin Wang, and Chenyan Xiong. Interpret and control dense retrieval with http:///wwxvvvooora//////s
sparse latent features, 2025. URL
Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. Demonstrate-eeardhpprdde Composing retrieval and language models for knowledge-intensive nip, 2023. URL http::://aaxxi.orr/abbeee
14024.
Gangwoo Kim, Sungdong Kim, Byeongguk Jeon, Joonsuk Park, and Jaewoo Kang. Tree of clarifications: Answering ambiguous questions with retrieval-augmeened large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 996-1009, Singapore December 2023. Association for Computational Linguistics. doi: 10.18653/v1/200 emnlp-main.63. URL htttps:/aclaanthllloe or org/2023. emnlp-main.66
15

---

Published as a conference paper at COLM 2025
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Proceedings of the 36th International Conference Neural Information Processing Systems, NIPS '22, Red Hook, USA, 2024. Curran Associates Inc. ISBN 9781713877
Po-Nien Kung and Nanyun Peng. Do models really learn to follow instructions? empirical study of instruction tuning. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Compulational Linguistics (Volume 2: Short Papers), pp- 1317-1328, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: (0.18653/v1/2000 acl.soort.. aclan wtg. acl-short.113//
URL https://
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszko- reit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019. doi: 10.1162 taacllll_a_a 00266. URL https: https://aclanntoo...e
org/Q19-1028
Kenton Lee, Ming-Wes Chang, and Kristina Toutanova. Latent retrieval for weakly super- vised open domain question answering. In Anna Korhonen, David Traum, and Lluis Märquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Lingwistics, pp. 4086-6096, Florence, Italy, July 2019. Association for Computational Lin- guistics. doi: 10.18653/v1/PP /P10/ 1612. URL https://aclanttolllgg orrp199--1///11111111 1612.
Yibin Lei, Yu Cao, Tianyi Zhou, Tao Shen, and Andrew Yates. Corpus-steered query expansion with large language models. In Yvette Graham and Matthew Purver (eds.), Pro- ceedings of the 18th Conference of the European Chapter of the Association for Computational Lin- guistics (Volume Short Papers), pp. 393-401, St. Julian's, Malta, March 2024. Association for Computational Linguistics. URL https: / www.aacooanttoyyy..ocm.ooooooooe www.lloyy..ggg////// org/2024 eacl- short.34/.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensitnn nlp tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS 20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829548
Xiaoya Li, Fan Yin, Zijun Sun, Xiayu Li, Arianna Yuan, Duo Chai, Mingxin Zhou, and Jiwei Li. Entity-relation extraction as multi-turn question answering. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Compulational Linguistics, pp. 1340-1350, Florence, Italy, July 2019. Association for URL htttps://aclatthon
Computational Linguistics. doi: org/P19-1129.
Shihao Liang, Runchu Tian, Kunlun Zhu, Yujia Qin, Huadong Wang, Xin Cong, Zhiyuan Liu, Xiaojiang Liu, and Maosong Sun. Exploring format consistency for instruction tuning, 2024. URL https: //txx//axvvvv///////a//eee/aawvl http:///aaxxv.o////////////// orw. 15504.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? In Eneko Agirre, Marianna Apidianaki, and Ivan Vulié (eds.), Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pp. 100- 114, Dublin, Ireland and Online, May 2022. Association for Computational Linguistics. doi: 10. /v1 /ttp:/22222deelo11111 URL https://aclantthlolg www.://...gg......eee/ee de 1000
Yanming Liu, Xinyue Peng, Xuhong Zhang, Weihao Liu, Jianwei Yin, Jiannan Cao, and Tianyu Du. RA-ISF: Learning to answer and understand from retrieval augmentation via iterative self-feedback. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 4730-4749, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/ /vl 2024. 2024.findings-al... 281. URL https: http:////aaaaalog/...../ org/2024. findings-acl 281/
16

---

Published as a conference paper at COLM 2025
Michael Llordes, Debasis Ganguly, Sumit Bhatia, and Chirag Agarwal. Explain like am bm25: Interpreting a dense model's ranked-list with a sparse approximation, 2023. URL 12631.
Xing Han Lù. Bm25s: Orders of magnitude faster lexical search via eager sparse scoring, 03618.
2024. URL
lain Mackie, Shubham Chatterjee, and Jeffrey Dalton. Generative relevance feedback with large language models. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieoal, SIGIR '23, pp. 2026-2031. ACM, July 2023. URL http:: doi orw...rg///00 1145/3539618 3591 1922.
doi: 10.1 01445/553618.35991
OpenAL. Gpt-4 technical report. ArXiv, abs /bbb//333.88774, 2023. URL abs/2303. 08774.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Language Processing (EMNLP-IJCNLP), pp. 2463-2473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653 10.18653vv1/D19-1111 v1/D19-1250. URL fttpp:://allanthooogyy..
Natural
1250.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653 vv1 DDD6-2644
URL httpttttts///clatttttt wwwpthhhoooggooo//1eeeeo wrg.//o-g///// 1264.
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton- Brown, and Yoav Shoham. In-context retrieevllaugmeened language models. Transactions of the Association for Computational Linguistics, 11:1316-1331, 2023. doi: 10.1162/tacl_a
00605. URL https::/aclaattool//
org/ 22t.:/223... tacl-1.75.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1416,,4,,, Online, November 2020. Association for Computational Linguistics. doi: 10.18653 / 2220...0mlpmmaii..... cellanthlooy.orgg// org/2020 emnlp-man 437.
URL https://
Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval, 3:333-388 389,, 01 2009. doi: 10.1561/1500000019
Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. Enhancing retrieval-augmenter large language models with iterative -enerraion synergy. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 9248 9274,, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1 /ttp//ll2222./// w0t.:/2..inningn-eee... andingg-emmlp.622 2023 f findings-emnlp. 620.
URL https::/acllatthottt
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. REPLUG: Retrievalaaugmented black-box language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Papers), pp. 8371-8384, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 1011866/vvll v220/44./222.////// 120/2222aaaaacooog 463. URL https://acllatthllloe org/2244 wttpg///2...aaala111a1 long. 463.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmen- tation reduces hallucination in conversation. In Marie-Francine Moens, Xuanjing Huang,
17

---

Published as a conference paper at COLM 2025
Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Association for Computational Lin- guistics: EMNLP 2021, pp. 3784-3803, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/ v1 fin.....niings--eii. URL
wwcplloyyyoogg//g/ org/2021. fiindffgsneeniinnnn 320.
Weiwei Sun, Hengyi Cai, Hongshen Chen, Pengjie Ren, Zhumin Chen, Maarten de Rijke, and Zhaochun Ren. Answering ambiguous questions via iterative prompting. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7669- 7683, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 18653/ w1/2023.acl-long. 424. URL https: hatp:://aclanthonn hwwp///olaaattoooo...o.oooo.../a org acl aol-long--.. .244.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George- Christian Muraru, Grigory Rozhdestvenskly, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Ferret, Justin C Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open models based on gemini research and technology, 2024. URL https: / /ttx//ixivvvv///aa//ee////a http::///axxvv.ooabbbbbss/ 24ee//2433.. 2403.0895.
Jeff Stanway, Jenny Brennan, Jeremy
Chen,Johan Jeremy
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brían Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https://arxxxt http::://rxxw.org//bb////2 orgg 09288.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with thain-of-thought reasoning for knowledge-intensive multi-step questions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meefing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10014-10037, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: v1t0///./1112233all1mmc 557. URL https:/ /aopps://aclae http:////aaaaaatoooooooooo org www.o//g..//////e/ acl-long. 557.
Boshi Wang, Xiang Deng, and Huan Sun. Iteratively prompt pre-trained language models chain of thought. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 2714-2730,
modglls for
18

---

Published as a conference paper at COLM 2025
Abu Dhabi, United Arab Emirates, December 2022a. Association for Computational URL https::/aclannthllogy org/
Linguistics. doi: 2022. eenleemman- 174.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by reakly-supervisse contrastive pre-training. arXiv preprint arXio:2212.03553,, 2022b.
Liang Wang, Nan Yang, and Furu Wei. Query2doc: Query expansion with large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9414-9423, Singapore December 2023a. Association for Computational Linguistics. doi: emnlp-main.585. URL https: h/clat/hooy/l wttpe//yrrrr//3/3/// http/222...mmmmmaaaaa..mmm emnlp-main. 585.
Liang Wang, Nan Yang, and Furu Wei. Query2doc: Query expansion with large language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023b. URL httpts:///pppeeeeeeee/
pet//orrm?idd-HHENNee BII
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math sath-sheppher: Verify and reinforce LLMs step-by-step without human annotations. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1426-94399 Bangkok, Thailand, August 2024. Association for Computational org//024/ 2024.
Linguistics. doi: 10.18653 (118653/vv/2022/ wl/o024..cl-oong.5//. URL acl-long.511 510.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL (wrfformmmii__vjOMMeSSS
Yu Xia, Junda Wu, Sungchul Kim, Tong Yu, Ryan A. Rossi, Haoliang Wang, and Ju- lian McAuley. Knowledge-aware query expansion with large language models for textual and relational retrieval. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Pp. 4275-4286, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-99176-189-6 doi: 10.18653/vi /v1///11////// 2aw..naacl-lonn... https://aclanttollgg/ wwwpethhooog.oooo////1ee wwg. naacl- 216/
URL
Wenhan Xiong, Xiang Li, Srini Iyer, Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mehdad, Scott Yih, Sebastian Riedel, Douwe Kiela, and Barlas Oguz. Answering complex open-domain questions with multi-hop dense retrieval. In International Conference on Learning Representations, 2021. Utt.tt/ppe///ooereeee//////////o
Fangyuan Xu, Weijia Shi, and Eunsol Choi. RECOMP: Improving retrieval-augmmeted LMs with context compression and selective augmentation. In The Twelfth International Conference on Learning Representations, 2024. URL https://openrevieee wwwprreeewweeeeeforunnneee d= m1JLVigNHp.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369 2380, Brussels, Belgium, October-November 2018. As- sociation for Computational Linguistics. doi: 10.18653/v1/D18- 1259.. 1259.
URL https:
//aclanthology.orgln
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICI.R,, 2023.
19

---

Published as a conference paper at COLM 2025
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. SWAG: A large-scale adver- sarial dataset for grounded commonsense inference. Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 93-104, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. 12.16633/11/D18-10099 URL
Peitian Zhang, Shitao Xiao, Zheng Liu, Zhicheng Dou, and Jian- Yun Nie. Retrieve anything to augment large language models, 2023a.
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. Siren's s song in the ai ocean: A survey on hallucination in large language models, 2023b. URL https: http:///wwwvv..o/aa///a///// orgg/bs//3099 01219.
Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed H. Chi, Quoc V Le, and Denny Zhou. Take a step back: Evoking reasoning via abstraction in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: /tppnreev//eeeeewww net/ mtttf/oormmmdddb3jjsseee
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
URL https://openreeiiew www.reewwwwweeefooomiiiiiioo net/forum2IdduuccPPP))
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh Interna- fional Conference on Learning Representations, 2023a. URL /tppe://ope wttpeeiwwww...efrrreeeeee net/forum? id=WZH7099tgFM.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-followiing evaluation for large language models, 2023b.
URL htttps://aaae www..v..rrg/ab///111111.. 07911.
A Limitations
The performance of ITERKEY is highly dependent on the capabilities of the underlying large language model (LLM). For example, Gemma-2, recent high-performance model, showed notable drop in accuracy despite adhering to the output format and instructions. This sug- gests that even advanced models may struggle to effectively perform keyword generation or answer validation, limiting the range of LLMs suitable for ITERKEY. This aligns with prior studies (Kung & Peng, 2023; Zhou al., 2023b; Liang al., 2024) emphasizing the distinction between format adherence and instruction comprehension in LLMs.
ITERKEY introduces additional computational cost compared non-iterative RAG methods. Each iteration involves keyword generation, retrieval, answer generation, and validation, resulting in higher runtime and resource usage, especially with large models or large-scale applications. Additionally, since ITERKEY operates under fixed iteration limit, it may fail to resolve highly ambiguous or complex queries within the allotted steps.
Finally, the answer validation step relies on textual pattern matching, which can misjudge semantically correct answers expressed in different formats such as abbreviations or para- phrases. As result, true positives may be overlooked, leading to underestimation of model performance. This limitation particularly affects the accuracy of the ablation analysis, where validation quality is central factor.
20

---

Published as a conference paper at COLM 2025
In this study, to ensure fair comparison of performance across multiple models, all exper- iments were conducted on a single NVIDIA RIX 6000 Ada GPU, with half quantization utilized for model generation. However, due to resource constraints, the Llama-3.1 70B model was loaded and inferred in 4-bit mode. We use LLMs for three steps: keyword gen- eration, answer generation, and answer val- idation. For each step, an appropriate Max Token Length was set, shown in Table 14. The same settings were applied each model for performance comparison purposes.
STEP
Max Token Length
Keyword Generation
50
Answer Generation
50
Answer Validation
30
Table 14: Max token length for each step.
Model
Params
HuggingFace Name
Table 15 provides details of the language mod- els used in experiments, including their parameter sizes and Hugging Face model identifiers. These models were selected based on their performance and availability for fair evaluation across different LLM architectures.
Llama-3.1
8B
meta-llama/ meta-llmma/Llama33 3 I-St-
1lama-3.1
706
meta-llama 1ama-3.1 1-700 Instruct
Gemma-2
9B
google 1ooel-/Gmmm--299--t
Phi-3.5-mini
3.8B
micoosof//hh335--iiiinmmttt
Table 15: Details of the LMs for the exper- iments.
B Details of Experimental Setting
C Selection of Models for Iterkey
For the selection of models used Iterkey, we conducted preliminary experiments with multiple candidate models. Based the results, selected models according to two main criteria (Section 2) required to implement our method without human intervention.
1. Adherence to Specified Output Format:
The model must follow the designated generation format. In keyword generation and validation, should produce outputs that conform to the specified format with minimal post-processing, generating lists without unnecessary extra text.
2. Accurate Understanding of Instructions: The model must accurately comprehend the intent of the instructions. It should generate outputs based on a correct under- standing of the instructions. During validation and keyword regeneration, it should produce consistent outputs aligned with the task requirements.
Models such as Llama 2 (Touvron et al., 2023), Qwen 1.5 (Bai et al., 2023), Vicuna (Chiang et al., 2023), and Mistral (Jiang et al., 2023a) were not adopted for ITERKEY as they either produced erroneous outputs failed to adhere to the specified format, making them unsuitable for use in this context. Additionally, all selected models received instruction tuning, as models without instruction tuning were unable to generate controlled outputs effectively.
D Effect of Chain of Thought Reasoning
To examine whether explicit Chain of Thought (CoT) reasoning can further boost ITERKEY, we replaced the original single sentence prompts in Answer Validation (Step 3) and Keyword Regeneration (Step 4) with step by step versions. Specifically, in Step 3 the model first writes out its reasoning, citing the retrieved documents, and only then outputs 'True' or False'; in Step it diagnoses why the previous keywords were insufficient before proposing refined comma separated list. These changes preserve the BM25 backbone and add no extra retrieval overhead. Table 16 lists the modified prompts.
Table 17 compares the original ITERKEY (BM25) with its CoT variant on four QA benchmarks. Across models and tasks, the CoT variant fails to produce consistent trend: NaturalQA
21

---

Published as a conference paper at COLM 2025
Step
Prompt (CoT version)
Step 3: Answer
System: You are an assistant that verifies whether an answer is correct
Validation (CoT)
based on on retrieved documents.
User: Question: {q)
Answer: [a]
Documents: Docs
Let's think step by step.
Chain of thought: (Explain your reasoning step step, Refer to the documents when relevant.)
Conclusion: True False
Do not output anything besides the above format.
Step 4: Keyword Regeneration (CoT)
System: You refine keywords to improve document retrieval for BM25 search in the RAG framework.
User: The previous keywords failed to retrieve useful documents for the
query: [q]
Here the previous keywords: prev keywords
Let's think step by step. Check if they are general, specific, or missing important concepts.
Then, refine them to produce a more effective list of keywords for retrieving documents that contain the correct answer.
Output the keywords in the exact format: ["kw1", "kw2", ...] Refined Keywords:
Table 16: CoT enhanced prompts used in Step 3 (Answer Validation) and Step 4 (Keyword Regeneration).
Setting
Model
Size (B)
Entity QA
HotpotQA
Natural QA
WebQA
Llama-3.1
8B
62.0
49.7
50.2
53.3
Llama-3.1
70B
63.5
54.9
53.8
57.7
IterKey (BM25)
Gemma-2
9B
36.1
26.1
35,9
36.1
Phi-3.5-mini
3.8B
47.9
42.7
36.8
41.2
Llama-3.1
8B
-1.6
60.4
+1.1
50.8
+3.0
53.2
-1.1
52.2
IterKey (BM25)
1lama-3.1
70B
-0.3
63.2
+0.5
55.4
+1.6
55.4
-2.1
55.6
CoT
Gemma-2
9B
-3.7
32.4
-1.5
24.6
-5.9
30.0
-5.1
31.0
Phi-3.5-mini
3.8B
+0.3
48.2
-0.3
42.4
-0.2
36.6
-0.6
40.6
Table 17: Score changes when applying CoT to IterKey (blue indicates improvement, red indicates decline).
improves modestly by up to +1.6, whereas EntityQA, HotpotQA, and WebQA show declines of up to -2.8. These results indicate that the clearly defined instructions in each step of ITERKBY are sufficient leverage the LLM's capabilities, making additional CoT prompting unnecessary in this BM25 centric setting.
E
Keyword Numbers Distribution in ITERKEY
Figure shows the average and variance of the number of keywords generated by each model across all tasks and iterations. Most models generate between and 12 keywords, with a maximum of under 30. This distribution helps analyze the relationship between keyword generation and performance in ITERKEY. Despite the size difference, both Llama- 3.1-8B and Llama-3.1-70f exhibit similar trends, peaking at 8-10 keywords with nearly identical distribution shapes. On the other hand, Gemma-2 model generates the highest average number of keywords (12.39) compared to the other models, with more concen- trated distribution. However, as discussed in Section 3.2, 4.1 its recall rate and EM score are lower, particularly in keyword generation and iterative refinement, where it underperforms. This suggests that generating more keywords doesn't necessarily improve accuracy, likely due to issues in the validation step identified earlier. Phi-3.5-mini model shows similar
22

---

Published as a conference paper at COLM 2025
keyword generation distribution to Llama-3.1-8B, but differences in recall rates and EM scores indicate variations in keyword quality. This underscores that performance depends just on the number of keywords but their quality as well.
Figures 4, 5, 6, 7 and Table 20 show how keyword generation evolves over five iterations for each model. Llama-3.1 consistently improves by reducing average keyword count and variance, contributing to strong performance. Llama-3.1-70B maintains stable keyword generation with slightly higher averages and increasing variance, reflecting flexibility without a notable drop in quality. In contrast, Gemma-2 generates the most keywords (mean: 12.8) but shows limited refinement and low variance, possibly limiting dynamic adjustment and lowering recall and EM scores. Phi-3.5-mini starts with a high keyword count but drops sharply, with rising variance, yet shows little performance improvement, likely due to inefficient refinement. These results suggest that keyword refinement is key, with Llama models showing strong link performance, while Gemma-2 and Phi-3.5-mini struggle with dynamic adjustment, potentially affecting their results.
F Impact of Chunk Size on ITERKEY
This study examines the impact of different chunk sizes and Top-k retrieved documents on model performance in QA tasks. Several key trends emerged from the results, as shown in Table 21. In these additional experiments, we used the December 2018 Wikipedia dump as the retrieval corpus for all datasets (Izacard et al., 2024). The text was divided into segments of tokens, with an overlap of 50 tokens at the beginning and end of each chunk to preserve context.
In the 'Chunk256' 'Top3' setting, the Llama-3.1-8B model outperformed other models across all QA tasks, achieving the best performance. This indicates that retrieving Top3' documents improves the model's accuracy over Topl'. On the other hand, when only the 'Top1' document is provided, the larger alama-3.1170B model generally outperformed the Llama-3.1-8B model, especially in tasks like EntityQA and WebQA. This suggests that the 70B model has the capacity to leverage smaller set of relevant information more effectively, maintaining high accuracy with fewer retrieved documents.
Furthermore, when comparing chunk sizes, the results showed that increasing the chunk coparinng size from 'Chunk256' to 'Chunk512' led to a decline in performance, particularly in the 'Top3' configuration. For example, in the 'Chunk512' 'Top3' setting, both the Llama-3.1-8B and Llama-3.1-700 models exhibited significant drops in performance across all tasks, with notable declines in EntityQA and WebQA. This suggests that larger chunk sizes exceed the model's optimal token processing capacity, causing difficulties in handling the larger input context, and consequently leading to performance degradation.
Regarding model size and information efficiency, the performance difference between Llama- 3.1-8B and Llama-3.1-70B reveals an interesting tradeoff. While the 70B model benefits from a larger parameter space, allowing it to excel with fewer documents ("Topl',, the 8B model makes better use of multiple documents ('Top3'. This indicates that smaller models can compensate for their parameter limitations by utilizing more retrieved information, whereas larger models can achieve comparable or better results with minimal inputs.
23

---

Published as a conference paper at COLM 2025
Method
Model
Size (B)
Entity QA
HotpotQA
Natural QA
WebQA
Llama-3.1
8B
33.6
31.2
40.6
53.4
Llama-3.1
70B
45.2
41.4
46.0
54.0
Vanilla
Gemma-2
9B
10.6
11.6
9.2
20.8
Phi-3.5-mini
3.8B
24.6
25.4
25.8
44.0
Llama-3.1
8B
54.0
47.0
44.8
51.4
Llama-3.1
70B
54.6
46.2
43.4
47.4
RAG (BM25)
Gemma-2
9B
47.9
39.6
33.2
41.6
Phi-3.5-mini
3.8B
48.2
42.2
32.6
40.2
Llama-3.1
8B
50.9
46.7
47.2
48.2
Llama-3.1
70B
56.5
52.0
46.4
45.0
RAG (E5)
Gemma-2
9B
52.2
40.8
41.5
41.8
Phi-335-mini
3.8B
50.2
44.6
37.0
41.4
Llama-3.1
8B
48.3
46.7
45.7
47.2
Llama-3.1
70B
53.3
50.0
48,4
48.8
RAG (BGE)
Gemma-2
9B
49.2
36.5
41.6
42.8
Phi-3.5-mini
3.8B
46.2
43,9
36.0
44.8
Llama-3.1
8B
47.4
47.7
43.6
48.2
Llama-3.1
70B
51.0
51.0
47.4
48.8
RAG (Contriever)
Gemma-2
9B
48.5
40.8
38,8
41.8
Phi-3.5-mini
3.8B
44.2
44.6
39.0
41.4
Llama-3.1
8B
62.0
49.7
50.2
53.3
Llama-3.1
70B
63.5
54.9
53.8
57.7
IterKey (BM25)
Gemma-2
9B
36.1
26.1
35.9
36.1
Phi-3.5-mini
3.8B
47.9
42.7
36.8
41.2
Table 18: 'Vanilla' uses no retrieval. 'RAG (BM25), 'RAG (E5y, 'RAG (BGEY, and 'RAG
(Contriever) apply single retrieval step based on the original query, using BM25, E5, BGE, and Contriever respectively. Our proposed 'terKey (BM25) iteratively generates and refines keywords, performing up to five retrieval iterations to optimize answer quality.
24

---

Published as a conference paper at COLM 2025
Method
Model
Size (B)
Entity QA
HotpotQA
Natural QA WebQA
Llama-3.1
8B
33.6
31.2
40.6
53.4
Llama-3.1
70B
45.2
41.4
46.0
54.0
Vanilla
Gemma-2
9B
10.6
11.6
9.2
20.8
Phi-3.5-mini
3.8B
24.6
25.4
25.8
44.0
Llama-3,1
8B
54.0
47.0
44.8
51.4
Llama-3.1
70B
54.6
46.2
43,4
47,4
RAG (BM25)
Gemma-2
9B
47.9
39.6
33.2
41.6
Phi-3.5-mini
3.8B
48.2
42.2
32.6
40.2
Llama-3.1
8B
60.6
53.4
53.6
56.2
Llama-3.1
70B
57.1
52.9
53.3
51.6
ITRG Refine (E5)
Gemma-2
9B
54.2
47.6
47.4
48.5
Phi-3.5-mini
3.8B
54.3
47.1
36.2
45.6
Llama-3.1
8B
58.5
49.2
52.2
56.3
Llama-3.1
70B
60.1
53.2
55.7
52.2
ITRG Refresh (E5)
Gemma-2
9B
50.2
40.8
45.9
49.3
Phi-3.5-mini
3.8B
49.3
43.9
36.0
41.4
Llama-3.1
8B
62.0
49.7
50.2
53.3
Llama-3.1
70B
63.5
54.9
53.8
57.7
IterKey (BM25)
Gemma-2
9B
36.1
26.1
35,9
36.1
Phi-3.5-mini
3.8B
47.9
42.7
36.8
41.2
Table 19:
'Vanilla' does not use retrieval. 'RAG (BM25) apply single retrieval step based on the original query, using BM25. 'TRRG Refine (E5)' is a method that utilizes E5 for iterative refinement. 'TRR Refresh (E5) also uses E5, performing refinement based on query refreshment. The proposed method, 'IterKey (BM25), iteratively generates and refines keywords, performing up to five retrieval iterations to optimize answer quality.
Iteration 1
Iteration 2
Iteration 3
Iteration 4
Iteration 5
Model
Mean
Var
Mean
Var
Mean
Var
Mean
Var
Mean
Var
Llama-3.1 8B
11.2
4.1
9.3
3.9
8.5
4.0
8.1
3.8
8.1
3.9
Llama-3.1 70B
11.2
4.1
10.3
4.8
10.3
4.8
10.6
4.8
10.7
4.9
Gemma-2 9B
12.8
2.5
12.4
3.3
12.3
3.6
12.3
3.6
12.2
3.7
Phi-3.5-mini 3.8B
13.8
3.4
9.4
3.6
8.6
4.0
8.4
4.0
8.6
4.0
Table 20: Results of different models across iterations, displaying both the mean and variance.
25

---

Published as a conference paper at COLM 2025
Llama-3.1-8B --88BB
Llama-3.1-77 1-70B
1400
1400
1200
1200
Mean: 9.00
Mean: 10.62
Frequency
1000
Median: 9.0
Frequency
1000
Median: 10.0
800
Std Dev: 4.11
800
Std Dev: 4.66
600
600
400
400
200
200
10
20 Number of Keyword
10
30
Number of Keyword
Gemma-2-9B
Phi-3.5-mini--.888
1400
1200
1200
1000
Mean: 12.39
1000
Mean: 9.75
Frequency
Median: 13.0
Frequency
Median: 10.0
800
Std Dev: 3.37
Sid Dev: 4.31
600
600
400
400
200
0
10
20
25
30
10
20
30
Number of Keyword
Number of Keyword
Figure 3: The distribution of keywords generated by each LLMs.
Llama-3.1-8B
Prequency
Iteration:
Iteration:2
Iteration: 3
Iteration:
Iteration: 5
Number of Keyword
Figure 4: The distribution of the number of keywords per iteration in LLama 3.1-8B.
Llama-3. 1-70B
16
Iteration:
Iteration:2 2
Iteration: 3
Iteration:
Iterution: 5
Number of Keyword
Figure 5: The distribution of the number of keywords per iteration in LLama 3.1-70B.
26

---

Published as a conference paper at COLM 2025
Gemma-2-9B
Iteration:
Iteratio:22
Iteration: 3
Iteration:
Iteration:3 5
Number of Keyword
Figure 6: The distribution of the number of keywords per iteration in Gemma-2.
Phi-3.5-mini-3.8B
15
15
Iteration:
Iteration:
Iteration:
Iteration:
Iteration:
Number of Keyword
Figure 7: The distribution of the number of keywords per iteration in Phi-3.5-mini.
Method
Model
Size (B)
Entity QA
HotpotQA
Natural QA
WebQA
Llama-3.1
8B
46.2
41.2
47.6
56.4
Llama-3.1
70B
53.1
44.4
49.0
55.0
Chunk256 Top1
Gemma-2
9B
24.0
7.2
10.0
18.8
Phi-33.-mini
3.8B
34.2
28.0
28.8
43.2
Llama-3.1
53.6
47.8
54.8
59.8
Llama-3.1
70B
52.0
47.0
52.6
55.6
Chunk256 Top3
Gemma-2
9B
7.2
7.6
8.1
13.8
Phi-3.5-mini
3.8B
41.8
33.6
35.2
49.4
Llama-3.1
8B
49.6
42.2
47.2
55.2
Llama-3,1
70B
48.7
44.4
50.4
54.3
Chunk512 Topl
Gemma-2
9B
10.6
7.2
10.0
12.0
Phi-3.5-mini
3.8B
35.2
30.0
31.9
46.1
Llama-3.1
8B
22.6
15.6
23.5
29.2
Llama-3.1
70B
29.7
23.4
33.4
37.3
Chunk512 Top3
Gemma-2
9B
0.6
1.6
1.0
2.1
Phi-335-mini
3.8B
16.2
11.0
17.6
24.6
Table 21: Comparison of model performance on QA tasks with different chunk sizes and Top retrieved documents provided to the model during RAG. For example, with 'Chunk256' and 'Top''
27

---

Published as a conference paper at COLM 2025
Question:
Where does the PATH train stop Newark?
Answer: Newark Penn Station
Iteration
Keywords
Model Output Answer: Newark Liberty International Airport Sta- tion
Validation Output Correct?
PATH train, Newark, stops, transportation, stations, Penn Station, downtown Newark, train stations, commute, New Jersey, NJ Transit, rail, Newark PATH stops, PATH Newark stops, Newark train stops
False
False
Newark, New Jersey, NJ Transit, Penn Station, Newark Penn Station, downtown Newark, trans- portation, train stations, rail, commute
The PATH train stops at Newark Penn Station.
True
True
Table 22: Case 1: Correct answer is reached in the second iteration.
Question:
Who wrote and performed "T Can Only Imagine"?
Answer: Bart Millard
Iteration
Keywords
Model Output Bart Millard
Validation Output Correct?
Can Only Imagine, Mer- cyMe, Bart Millard, Chris- tian Song. Christian Music
False
False
Artist, Lead Vocalist, Song-
writer, Christian Band, Con-
temporary Christian Music
2
I Can Only Imagine, Bart Millard, MercyMe, Chris- tian Song. Christian Music Artist, Lead Vocalist, Song- writer, Christian Band, Con- temporary Christian Music Can Only Imagine, Mer- cyMe, Bart Millard, Chris- tian Song, Worship Mu- sic, Christian Music Artist, Lead Vocalist, Songwriter, Christian Band
Bart Millard
False
False
Bart Millard wrote and per- formed "T Can Only Imag- ine".
True
True
Table 23: Case 2: Correct Answer with False Validation.
28

---

Published as a conference paper at COLM 2025
What prolific English engineer, who died on 15 December 1857, contributed to the advent of the hot air engine by way of their open cycle devices? Answer: Sir George Cayley
Question:
Iteration
Keywords
Model Output
Validation Output
Correct?
English engineer, hot gine, open cycle, December 1857
Sir George Cayley
False
False
English engineer, hot air gine, open cycle, Decem- ber 1857, inventor, engine inventor English engineer, hot air gine, open cycle, inventor, 15 December 1857, British engineer, thermodynamic cycle
Sir George Cayley
False
False
3
Sir George Cayley
False
False
engineer, hot air gine, open cycle, thermody- namic cycle, 15 December 1857, British engineer, in- ventor, heat engine
Sir George Cayley
False
False
English engineer, hot air en- gine, open cycle, thermody- namic cycle, 1857, British engineer, inventor, heat gine, Stirling engine, Robert Stirling
Sir George Cayley
False
False
Table 24: Case 3: Model fails to verify the correct answer despite arriving at it.
29


Reviews:
[{"review_id": "h0L2A0a6Sz", "reviewer": "Reviewer_cbBx", "summary": "The paper introduces IterKey, an LLM-driven iterative keyword generation framework designed to enhance Retrieval-Augmented Generation (RAG) by combining sparse retrieval (BM25) with iterative refinement. IterKey operates through three stages: keyword generation, answer generation, and answer validation, iterating until a validated answer is obtained. Experiments on four QA datasets (NQ, EntityQA, WebQA, HotpotQA) demonstrate that IterKey improves BM25-based RAG by up to 20% in accuracy, achieving performance comparable to dense retrieval methods (e.g., E5) and prior iterative approaches (e.g., ITRG). The method leverages LLMs to balance accuracy and interpretability while preserving the transparency of sparse retrieval.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Strong Empirical Results: Demonstrates significant improvements over BM25 baselines (5–20% accuracy gains) and competitive performance against dense retrieval methods across multiple models (Llama-3.1, Gemma-2, Phi-3.5).", "reasons_to_reject": "- Model Dependency: Performance heavily relies on the LLM’s capabilities (e.g., Gemma-2 underperforms despite format compliance), limiting generalizability to weaker or smaller models.\n\n- Computational Overhead: Iterative process increases runtime (1.5–5x slower than non-iterative RAG), raising concerns for scalability and cost in production settings.\n\n- Limited Theoretical Insight: Empirical observations (e.g., optimal keyword counts, layer sensitivity) lack theoretical grounding, leaving open questions about why certain design choices work.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3j3KNRGtkH", "reviewer": "Reviewer_KgZB", "summary": "This paper introduces IterKey, an LLM-driven iterative keyword generation framework that enhances RAG via sparse retrieval. The primary goal of this paper is to balance the accuracy and interpretability by leveraging iterative keyword generation and self-evaluation during RAG process. While the paper is well-designed, there are certain significant concerns that need to be addressed.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-\tThe motivation is clear.\n-\tGood presentation.", "reasons_to_reject": "-\tThere are several significant concerns in the paper that go beyond revision and would require substantial clarification and restructuring to address effectively.\n-\tThe objective of this work is to balance the accuracy and interpretability during RAG. In the experiments, the authors verify the proposed method can improve the RAG accuracy, but the interpretability aspect is not verified.\n-\tThe experiment is not sufficient. Why not compare the proposed method with other RAG frameworks? For example, AdaptiveRAG [1], HippoRAG [2], Beam Retrieval [3], etc.\n\n[1] Jeong, S., Baek, J., Cho, S., Hwang, S. J., & Park, J. C. (2024). Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity. In 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 7036-7050). Association for Computational Linguistics.\n[2] Gutiérrez, B. J., Shu, Y., Gu, Y., Yasunaga, M., & Su, Y. (2024, January). Hipporag: Neurobiologically inspired long-term memory for large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.\n[3] Zhang, J., Zhang, H., Zhang, D., Yong, L., & Huang, S. (2024, June). End-to-End Beam Retrieval for Multi-Hop Question Answering. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (pp. 1718-1731).", "questions_to_authors": "-\tFirst of all, what is the difference between your proposed method and hybrid-based RAG (i.e., RAG that combines search results from sparse and dense vectors)? A clear explanation is needed to understand the novelty and improvement.\n-\tIt seems that the retrieval efficiency highly depends on the number of iterations, but I could not find a discussion about this issue.\n-\tIn step 1, the terms “explicit” and “implicit” are mentioned, but their meanings are not clearly defined. The explanation provided in the paragraph is vague, and the subsequent example does not sufficiently clarify the distinction. A more detailed explanation is needed.\n-\tIn step 2, it is not clear how the enhanced query is formed, is it simply a concatenation of the original query and the generated keywords or what? Figure 1 should be revised to illustrate this step more clearly.\n-\tIn step 3, “LLM verifies whether the generated answer is correct based on the retrieved documents, ensuring the RAG process has been executed correctly.”, how do authors ensure the reliability of the LLM’s verification process. Btw, the caption of Table 13 in Appendix C.1 mentions the keyword generation and answer validation, but the table content does not reflect the keyword generation, which is confusing. It is also unclear whether the LLM is judging the RAG output based on retrieved documents or a golden answer. 1) If LLM judges RAG output based on the retrieved documents, it is redundant as the RAG itself involves LLM generating a response based on retrieved contexts, and you use another LLM to evaluate the output, the validation process cannot be guaranteed (especially considering the largest LLM used for evaluation is 70B in 4-bit mode). 2) If LLM judges RAG output based on golden answer, the prompt in Table 13 does not present a placeholder for the golden answer, which needs clarification. In short, how exactly does your LLM judge the output is correct or not?\n-\tTo better understand the proposed framework, it would be helpful to share the inputs and outputs of each step in this query example “What is the name of the spacecraft …”\n-\tTable 16 is not cited and its mentions in the table caption do not appear in the table content (e.g. underlined values, bold values.)\n-\tTable 17 is also not cited in the paper.\n-\tAppendix D explains that the number of keywords is not critical to performance, but why does the subsequent section devote so much space to presenting the keyword distribution for each model? This seems unnecessary unless there is a correlation between keyword distribution and retrieval performance/efficiency, which is not clearly discussed. A more focused analysis in this regard would be more impactful.", "extra_scores": {}}, {"review_id": "ctY0sptqIj", "reviewer": "Reviewer_NWKh", "summary": "The paper introduces a framework aimed at improving the accuracy and interpretability of the retrieval process in RAG use cases which utilize sparse retrieval. The authors introduce IterKey, which addresses the issue by leveraging a LLM to iteratively generate keywords to be used in sparse retrieval based on an iterative process. The main contributions of the paper include the introduction of a three-stage iterative process for keyword generation, answer generation, and answer validation. In the first stage, an LLM generates keywords from the user query, which are then used to retrieve relevant documents. In the second stage, the LLM generates an answer based on the retrieved documents. Finally, in the third stage, the LLM validates the generated answer. If the validation fails, the process repeats with refined keywords, answer generation and validation until the LLM accepts the answer. The main novelty is the iterative approach and validation of the answer which allows for continuous improvement in the retrieval and generation.\n\nThe paper has an extensive evaluation and analysis section examining the components of the experiments from each aspect of the setup (retrieval, LLMs, cost, answer validation). The results, across four open-domain QA datasets, demonstrate that IterKey achieves significant accuracy improvements over BM25-based RAG and simple baselines, and performs comparably to dense retrieval-based RAG methods. The authors also provide detailed ablation studies analyzing the effects of keyword quality, validation accuracy, cost, and model size. Overall, the method offers a promising balance between retrieval performance and interpretability, with the caveat that its effectiveness depends significantly on the underlying LLM’s capabilities.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposes a well-motivated and clearly described three-stage iterative framework.\n- The paper leverages the use of BM as a sparse retrieval that has high interpretability, this property is key for the claim that IterKey’s practical applicability in scenarios requiring transparency.\n- Robust experimental setup - The evaluation spans four open-domain QA datasets, considers four different LLMs of varying scales (3.8B to 70B), and includes comparisons against dense RAG and prior iterative refinement methods like ITRG.\n- The authors conduct ablation studies on keyword quality and validation model quality, revealing how each stage contributes to final performance.\n- Clear writing and presentation.", "reasons_to_reject": "- Missing comparison to Query Expansion. The paper lacks a comparison to recent query expansion or rewriting techniques that also enhance retrieval in sparse or hybrid setups (https://arxiv.org/abs/2401.06311, https://arxiv.org/abs/2305.03653).\n- The authors do not evaluation or provide insight into how much new information is retrieved at each iteration by tuning the key words. Quantifying the novelty of documents across iterations would help validate the need for iterative keyword refinement over simpler alternatives like re-ranking.\n- Limited iterative behavior in practice - Table 9 shows that, on average, models complete just 1–2 iterations. This raises the question of whether the full iterative loop contributes significantly in practice, or whether most gains come from just the initial keyword refinement.\n- High latency and consequently cost.\n- Many performance discussions emphasize improvements over a vanilla (no-retrieval) baseline. While useful as a sanity check, this comparison is less relevant in practice, where RAG setups are the standard.", "questions_to_authors": "1.\tWhat is the explanation for IterKey(BM25) outperforming IterKey(E5) in Table 3, despite E5 being a stronger retriever in single-shot settings?\n2.\tIn Table 10, the runtime of components is extremely high? Is it total per dataset? I would have been more informative to understand latency per 1 example", "extra_scores": {}}, {"review_id": "Dl094rPSfd", "reviewer": "Reviewer_g9QR", "summary": "The paper introduces IterKey, a novel method for refining keyword generation in RAG systems, specifically optimizing sparse retrieval methods. IterKey addresses the challenge of capturing nuanced query intent in sparse retrieval by iteratively adjusting keywords, thereby improving the performance and relevance of the document retrieval. Extensive experiments demonstrate the effectiveness of IterKey, highlighting its potential to improve RAG-based models.\n\nOn the whole, the method presented in this paper is simple but effective, the experiments are thorough, and the writing is clear and easy to follow.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The method is simple but effective and is easy to follow.\n2. The experiments are extensive, and cover various aspects, demonstrating the method's robustness and generalizability.\n3. The writing is clear and well-structured, making the method easy to understand for audience.", "reasons_to_reject": "1. To my understanding, the proposed method appears to be a form of query expansion, which is fundamentally similar to existing methods such as Q2E [1] and Query2doc [2]. This similarity reduces the perceived novelty of the approach. \n2. The baselines selected for comparison are not comprehensive enough. To better highlight the superiority of the proposed method, it would be beneficial to include additional query expansion techniques as baselines. \n3. I think the proposed method might have design-level issues related to robustness, as detailed in the \"Questions to Authors.\" \n\n﻿[1] Query Expansion by Prompting Large Language Models.\n\n[2] Query2doc: Query Expansion with Large Language Models.", "questions_to_authors": "1. Although the authors demonstrate the effectiveness of the Answer Validation step in their experiments, I still have concerns regarding its robustness. Specifically, how do LLMs determine whether the answer to a question is correct? This is especially questionable when LLMs do not possess the necessary domain-specific knowledge or up-to-date information relevant to the query.\n\n2. In the Keyword Re-Generation step, I am curious about the basis on which LLMs refine keywords. Additionally, does the refinement of keywords actually help in answering the query effectively, especially when the LLM has limited knowledge of the query context or lacks sufficient domain-specific information?\n\n3. I am also curious whether incorporating in-context learning, CoT reasoning, or more complex agent-based workflows could enhance the performance of the proposed method. Could these techniques potentially improve the overall effectiveness of the approach? This is an additional question, not my judgment on this paper.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a RAG approach that uses iterative keyword generation for improved interpretability of the RAG pipeline. The reviewers appreciated the idea, but had multiple concerns in the initial review cycle. These were largely addressed in the author rebuttal and reviewers are now in support of accepting.
Decision: Accept