PRCA: Fitting Black-Box Large Language Models for Retrieval Question Answering via Pluggable Reward-Driven Contextual Adapter
Authors: Haoyan Yang
Zhitao Li
Yong Zhang
Jianzong Wang
Ning Cheng
Ming Li
Jing Xiao
Venue: EMNLP 2023 Main (2023)
Keywords: Retrieval Question Answering
Black-Box LLMs
Retrieval Augmentation
Pluggable Reward-Driven Contextual Adapter

Abstract: The Retrieval Question Answering (ReQA) task employs the retrieval-augmented framework, composed of a retriever and generator. The generators formulate the answer based on the documents retrieved by the retriever. Incorporating Large Language Models (LLMs) as generators is beneficial due to their advanced QA capabilities, but they are typically too large to be fine-tuned with budget constraints while some of them are only accessible via APIs. To tackle this issue and further improve ReQA performance, we propose a trainable Pluggable Reward-Driven Contextual Adapter (PRCA), keeping the generator as a black box. Positioned between the retriever and generator in a Pluggable manner, PRCA refines the retrieved information by operating in a token-autoregressive strategy via maximizing rewards of the reinforcement learning phase. Our experiments validate PRCA's effectiveness in enhancing ReQA performance on three datasets by up to 20% improvement to fit black-box LLMs into existing frameworks, demonstrating its considerable potential in the LLMs era.

PRCA: Fitting Black-Box Large Language Models for Retrieval Question Answering via Pluggable Reward-Driven Contextual Adapter
Haoyan Yang Zhitao Li', Yong Zhang', Jianzong Wang', Ning Cheng', Ming Li Jing Xiao
Ping An Technology (Shenzhen) Co., Ltd., China
New York University *University of Maryland jzwang@1 88. com
Abstract
Current Paradigm
The Retrieval Question Answering (ReQA) task employs the retrieval larrrevala frame- work, composed of a retriever and generator. The generator formulates the answer based on the documents retrieved by the retriever. Incor- porating Large Language Models (L.LMs) as generators is beneficial due their advanced QA capabilities, but they are typically too large be fine-tuned with budget constraints while some of them are only accessible via APIs. To tackle this issue and further improve ReQA per- formance, we propose a trainable Pluggable Reward-Driven Contextual Adapter (PRCA), keeping the generator as a black box. Po- sitioned between the retriever and generator in Pluggable manner, PRCA refines the re- trieved information by operating in a token- autoregressive strategy via maximizing rewards of the reinforcement learning phase. Our ex- periments validate PRCA's effectiveness in en- hancing ReQA performance on three datasets by up to 20% improvement to fit black-box LLMs into existing frameworks, demonstrating its considerable potential in the LLMs era.
Retriever
Corpus
White-Bor
Model
Querry
Documents
?
PRCA-based Paradigm
Generstor Block-Box
Retriever
Corpus
Query
Decuttents
PRCA
Figure I: A comparison between two paradigms for information retrieval and generation. The upper section showcases the traditional method where query pro- cessed by retriever that Top-K documents and then fed to white-box genera- tor. The lower section introduces our proposed PRCA method, which processes extracted Top-K documents from the retriever before feeding them to black-box generator achieve better performance for in-domain tasks.
corpus to fetch the
Recent advances in Large Language Models (LLMs) such as the generative pre-trained trans- former (GPT) series (Brown et al., 2020; Ouyang et al., 2022; OpenAI, 2023) have demonstrated remarkable potential, notably in their zero-shot and few-shot abilities within the realm of QA tasks. Owing to these capabilities, LLMs are ex- cellent choices as generators within the retrieval- augmented framework. However, due to the vast parameters of LLMs, fine-tuning them becomes exceedingly difficult within a limited computation budget. Furthermore, certain LLMs such as GPT-4 (OpenAT, 2023) are dlosed-souree, making it im- possible to fine-tune them. To achieve optimal results on specific datasets, fine-tuning retrieval- augmented models becomes necessary (Guu al., 2020; Lewis et al., 2020b; An al., 2021). Previ- ous attempts to integrate LLMs into the retrieval- augmented framework have met with partial suc-
1 Introduction
Retrieval Question Answering (ReQA) tasks in- volve generating appropriate answers to given ques- tions, utilizing relevant contextual documents. To achieve this, retrieval augmentation is employed (Chen al., 2017; Pan et al., 2019; Izacard and Grave, 2021), and comprised of two key compo- nents: a retriever and generator. The retriever's role is to retrieve relevant documents from a large corpus in response to the question, while the gener- ator uses this contextual information to formulate accurate answers. Such systems alleviate the prob- lem of hallucinations (Shuster et al., 2021), thereby enhancing the overall accuracy of the output.
The work was done when the first author was doing internship at Ping An Technology (Shenzhen) Co., Ltd., China. *Corresponding author: Jianzong Wang.

---

cess but also come with limitations. (Shi et al.. 2023) utilized the logits from the final layer of the LLMs when calculating the loss function, which may not be available to certain powerful LLMs that served via APIs. (Ma et al., 2023) involved fre- quently invoking pricy LLMs and overlooked the impact of the input token length on the accuracy and effectiveness of the system.
for black-box models in inference. Furthermore. the non-open-source nature and large parameter size of these models further contribute to their in- clination towards being perceived as black boxes.
On one hand, LLMs like GPT-4 (OpenAI, 2023) and PaLM (Scao et al., 2023) have showcased im- pressive performance in QA tasks. However, their closed source nature restricts access to these mod- els, making API-based utilization the only feasi- ble option, thereby categorizing them as black-box models.
To overcome these hurdles, we propose a train- able Pluggable Reward-driven Context Adapter (PRCA) that enables one to fine-tune the adapter instead of LLMs under the reurieeal-augmentee framework on specific datasets and achieve higher performance. Furthermore, PRCA distills the retrieved documents information guided by rewards from the generator through reinforcement learning. The distillation of retrieval information through PRCA reduces the length of text input to the generator and constructs context of superior quality, which mitigates the hallucination issues during the answer generation. As shown in Figure 1, PRCA is placed between the retriever and the generator, forming a PRCA-based Paradigm where both the generator and the retriever remain frozen. In general, the introduction of the PRCA-based paradigm brings the following advantages:
On the other hand, training LLMs, exemplified by models like Bloom (Scao et al., 2022) and GLM- 130B (Zeng et al., 2023), impose substantial com- putational demands. Specifically, training Bloom took 3.5 months using 384 NVIDIA A A100 80GB GPUs. Similarly, 1LLM-130B requires two-month training period on cluster of 96 DGX-A100 GPU servers. These resource requirements make it ex- tremely challenging for the majority of researchers to deploy these models. Moreover, LLMs exhibit rapid development speeds. For instance, from LLaMA (Touvron et al., 2023) to Alpaca (Taori et al., 2023) and now Vicuna (Peng et al., 2023), the iterations are completed within a month. It is evident that the speed of training models lags behind the pace of model iterations. Consequen- tially, tuning small-size adapters for any sequence- to-sequence LLMs on downstream tasks could be a simpler and more efficient approach.
Black-box LLMs Integration With the use of PRCA, LLMs can be treated as a black box integrated into the retrieval-augmented framework, eliminating the need for resource- inteesive fine- tuning and restrictions on closed-nature models.
2.2 Retrieval-Augmenter Framework
Various retrieval augmented ideas have been pro- gressively developed and applied to improve the performance in the ReQA task. In the initial stage of research, independent statistical similarity-base retrievers like TF-IDF (Sparck Jones, 1972) and BM25 (Robertson and Zaragoza, 2009) were used as fundamental retrieval engines. They helped in extracting the most rel- evant documents from the corpus for QA tasks (Chen al., 2017: Izacard and Grave, 2021).
Robustness PRCA serves as pluggable adapter that is compatible with various retrievers and generators because PRCA-based paradigm keeps both the generator and retriever frozen.
Efficiency The PRCA-based paradigm en- sures the efficiency of the framework by reducing the text length inputted into the generator and can adapt different retrieval corpus.
2 Related Work
The concept of vectorization was subsequently introduced, where both questions and documents were represented as vectors, and vector similar- ity became a critical parameter for retrieval. This paradigm shift was led by methods such as dense retrieval, as embodied by DPR (Karpukhin et al., 2020). Models based on contrastive learning like SimCSE (Gao et al., 2021) and Contriver (Izac- ard et al., 2022a), along with sentence-level se-
2.1
The Potential of LLMs as Black-Box Models
LLMs have demonstrated remarkable capabilities in downstream QA tasks, even in scenarios with limited or no training data (Wei et al., 2022). This emergence capability enables them to efficiently tackle such tasks, making them potential candidates

---

mantic models such as Sentence-BERT (Reimers and Gurevych, 2019), represented this era. These methods can seen as pre-trained retrievers that boosted the effectiveness the ReQA task.
Contextual Extraction Stage
Input
PRCA
Further development led to the fusion of retrieval and generation components within the ReQA frameworks. This was implemented in systems like REALM (Guu et al., 2020) and RAG (Lewis al., 2020b), where retrievers were co-trained with generators, further refining the performance in the ReQA task.
Reward-Driven Stage
Generator
Reward Model
Top-k
Ootral
PRCA
Recently, advanced approaches like Atlas (Izac- ard et al., 2022b) and RETRO (Borgeaud et al., 2022) have been introduced which could achieve performance comparable large-scale models like Palm (Chowdhery al., 2022) and GPT3 (Brown al., 2020) with significantly fewer parameters.
Reward
An illustration of the two-stage sequential training process for the PRCA. In the first "Contex- tual Extraction Stage". PRCA module is pre-trained domain abstractive summarization tasks. The second "Reward-Drivee Stage", demonstrates the interaction tween retrieved Top-K documents and the PRCA. Here, the PRCA refines the query using both the documents and the original query, producing am optimal context. This context is processed by generator to obtain reward, signifying the quality and relevance the con- text, with the feedback loop aiding in further refining the model's 's output and performance.
Figure 2:
3 Methodology
3.1 Two-Stage Training for PRCA
PRCA is designed to take sequences composed of the given query and the Top-K relevant documents retrieved by the retriever. The purpose of PRCA is to distill this collection of results, presenting a concise and effective context the generator, while keeping both the retriever and the genera- tor frozen. This PRCA-based paradigm introduces two challenges: the effectiveness of the retrieval cannot be directly evaluated due to its heavy depen- dence on the responses generated by the genera- tor, and learning the mapping relationship between the generator's outputs and the input sequence via backpropagation obstructed due the black-box generator. To tackle these issues, we propose two- stage training strategy for PRCA, as illustrated in Figure the contextual stage, supervised learn- ing is employed to train PRCA, encouraging it to output context-rich extractions from the input text. During the reward-driven stage. the generator is treated reward model. The difference between the generated answer and the ground truth serves as a reward signal to further train PRCA. This process effectively optimizes the information distillation to be more beneficial for the generator to answer accurately.
input text. The objective the training process is to minimize the discrepancy between Cextracted and the ground truth context Ciruth and the loss function is demonstrated as follows:
min L(0) 0
1 N
Σ
cii log(/FRCA (8
inputi 0))
(1)
where 0 represents the parameters of PRCA
In the context extraction stage, PRCA is ini- tialized from BART-Large model pre-trained on CNN Daily Mail dataset (Lewis et al., 2020a).
3.3 Reward-Driven Stage
In the reward-driven stage, the objective is align the extracted context Cextracted from the previous stage with the downstream generator, ensuring that the text distilled by PRCA serves effectively to guide the generator's s answering. Given the black- box nature of the generator, direct update of PRCA is not feasible. Therefore, resort to rein- forcement learning to optimize PRCA's parameters. Specifically, the generator offers rewards guide the update of PRCA's parameter, targeting im- prove answer quality. The reward is based on the ROUGE-L score between the generated answer
3.2 Contextual Extraction Stage
the contextual extraction stage, we train PRCA extract textual information. Given an input text Simpor, PRCA generates an output sequence Cextracedd representing the context derived from the

---

and the ground truth O", Meanwhile, it's vital that PRCA retains its skill of information extraction from long texts, as learned in the contextual extrac- tion stage. Our objective is twofold: maximizing generator's s reward and maintaining similarity be- tween updated and original parameters of PRCA after contextual extraction training. Catering to the wwwrrd-drived training where policy actions ma- nipulate sequence tokens, policy optimization, par- ticularly via Proximal Policy Optimization (PPO) (Schulman et al., 2017; Stiennon et al., 2020), is the preferred method. However, when employing a black-box generator as a reward model, we identify certain limitations of using PPO.
T
= EGNNoo
(3)
1=0
where 6/ Rit + (V(44+441 ((s+)); and À as discounting and GAB parameters respectively.
To tackle this issue, we introduce a strategy to estimate Rg- In the PRCA, when the token (EOS) is generated, we can obtain the reward REos by comparing the generated answer against the ground truth. We consider it an accumulation of the reward R2 achieved at each time step t for the generated to- ken. As for Rr, it serves as a target in J(0) to train the critic network Vss) for fitting, symbolizing the average reward of the current action, thereby sessing the advantage of the current policy. For each token, the greater the probability of genera- tion, the more important this token is perceived by the current policy, so we consider its contribution to the total reward to be greater. Therefore, we regard the probability of generating each token as the weight of Roos, and the representation of R, is given by the following:
In (2), we present the PPO's objective function (O).. This function strives to optimize the advan- tage, a value derived from the Generalized Advan- tage Estimation (GAE) (Schulman et al., 2016). The GAE leverages both y and À as discounting factors, adjusting the estimated advantage based on the temporal difference 6/44, depicted in (3). Here, E, (min((r AÇAE, clip(rr(0), - + 1.AAGAR captures the expected advantage. The clip function serves to prevent excessive policy updates by constraining the policy update step, en- suring stability in the learning process. The term ((V(s:) R₂)² is a squared-error term between V(s)) and Rg This term seeks to minimize the difference between the predicted and actual value, ensuring accurate value predictions. However, the critic network V usually initialized to have the same parameter as the reward model (Yao et al., 2023; Fazzie et al., 2023), which is inapplicable when the reward models are black-boxed Addition- ally, the APIs from vendors usually have limited amount of return parameters which may cause the computation of Rg impossible.
R₂ Rgos me(ac(at)
(4)
Reos ROUGE-L(O,O")
33 DKL (TOM))
(5)
ROUGE-L = LCS(X,Y) max (
(6)
where is the number of tokens in one gener- ated context, LCS(X, Y) denotes the length of the longest common subsequence between sequence X and sequence Y. and [X] and |Y| denote the lengths of sequences X and Y, respectively.
J(0) (r( ACAE
0
clip(r:(0),1 1,1--,,1 +e) (.)))GGAM AGAR)
-(V 22
(2)
This method mitigates the challenges associated with calculating RG when interpreting the black- box generator as reward model. A substantial advantage it confers is the requirement of invok- ing the reward model only once for each context generation. Compared to the original PPO that employs the reward model for every token com- putation, our approach reduces the reward model usage to t. which is cost- especially when using LLMs as generators.
where (0) mp(o.1ss) To [[]] is the ratio of the up- dated policy Ty to the original policy T0we Gt represents the action (the next token); s, is the state (the sequence of previous tokens); e is the clip- ping parameter; V is a critic network: V(+)) is the predicted value of state 8;; f3 coefficient that weights the squared-error term; R₁ is the expected return at time t.

---

Table 1: Overview of the data quantities used for train- ing and testing across three benchmark datasets.
Table 2: Hyperparameters settings used in the experi- ments.
Dataset
Train / Test
# of Q
# of C
# of A
Hyperparameters
Value
Train
87.6k 6k
18.9k
87.6k
Leamiing rate
5 10
SQuAD
Test
10.6k
2.
10.6k
Ratch size
1/2/4
Num beams
Train
90.4k
483.5k
90.4k
HotporQA
Temperature
Test
7.4k
66.5k
7.4k
Early Stopping
True
Train Test
45.5k
45.5k
45.5k
Topı
0.0
TopiQCQA
2.5k
2.5k
2.5k
Top,
1.0
4 Experimental Setup
established total of seventy-five baseline figurations on three datasets. For each configura- tion, we evaluated the performance with and with- out the application of PRCA and the difference serves as an indicator of the effectiveness of our proposed approach.
4.1 Datasets
We performed our experiments three QA datasets: SQuAD (Rajpurkar et al., 2016), Hot- poiQA (Yang et al., 2018) and TopiOCQA (Ad- lakha et al., 2022). The complexity of three datasets increases sequentially: SQuAD is a dataset that matches questions, documents, and answers a one-to-one manner. HotpotQA is a multi-hop QA dataset, requiring the synthesis of correct answers from multiple documents. TopiOCQA is conver- sational QA dataset with topic switching.
4.3 GPT-4 Assessment
Notably, we used GPT-4 for evaluation rather than traditional metrics like and BLEU, as these met- rics often misjudged semantically similar sentences. LLMs often output longer textual explanations for answers, even when the correct answer might be word or two. Despite attempts to constrain an- swer lengths, the results weren't ideal. We then evaluated predictions using both manual methods and GPT-4 against golden answers. GPT-4's evalu- ations showed correctness rates of 96%, 93%, and 92% across three datasets, demonstrating its relia- bility and alignment with human judgment. Specifically, the template for GPT-4 assessment is shown follows. Finally, the accuracy rate of answering "Yes" is counted as the evaluation metric.
To align these datasets with our ReQA task, reconstructed all three datasets into the form of (Q, C, A), where Q and denote the question and answer pair, and C represents corpus composed of all the documents in the dataset respectively. In Table 1, we present the number of questions and answers employed in the PRCA training and testing phases for every dataset. Additionally, provide the quantity of documents contained within each respective corpus.
4.2 Baseline Retrievers and Generators
We conducted experiments with five different retrievers, specifically BM25 (Robertson and Zaragoza, 2009), SentenceBert (Reimers and Gurevych, 2019). DPR (Karpukhin al., 2020), SimCSE (Gao et al., 2021), and Contriver (Izac- ard al., 2022a). We also utilized five generators which are T5-large (Raffel et al., 2020), Phoenix- 7B (Chen et al., 2023), Vicuna-7B (Peng et al., 2023), ChatGLM (Du et al., 2022) and GPT-3.5 to assess the effectiveness of PRCA. Note that both the retrievers and generators remain frozen through the experiment.
Template for GPT-4 Assessment
You are now an intelligent assess- ment assistant. Based on the question and the golden answer, judge whether the pre- dicted answer correctly answers the question and give only Yes or No.
Prompt:
Question:
Golden Answer:
Predicted Answer:
Expected Output:
Yes No
By pairing every retriever with each generator,
4.4 Hyperparameter Configurations
'Our experiments were conducted with the default version of GGP--3.5-turbo and GPT-4 between May and June 2023 via https///opnai..com.
To achieve optimal results our PRCA training, careful selection of hyperparameters is pivotal. The

---

Table 3: Comparative results of performance for different retriever and generator combinations in the presence and absence of PRCA integration. The results based the evaluation using three benchmark datasets: SQuAD, HotpotQA, and TopiOCQA, and focus on the selection of the Top-5 most relevant documents.
Retriever
Generator
SQuAD
HotpotQQ
TopiOCQA
T5
0.74-0.03
0.35++0.01
0.27+0.08
Phoenix
0.61+0.02
0.31+0.09
0.25+00.3
BM25
Vicuna
0.59+0.09
0.19+0.13
0.23+0.10
CharGLM
0.67+0.03
0.36+0.04
0.35+00,3
GPT-3.5
0.75+0.02
0.48+00.6
0.44+0.04
T5
0.18-0.06
0.20+0.05
0.28+0.05
Phoemix
0.42+0.00
0.13+0.10
0.26+0.08
SentenceBert
Vicuna
0.36+0.09
0.22+0.03
0,23+0.05
ChatGLMM
0.57+0.04
0.16+0.08
0.28+0.04
GPT-3.5
0,6+0.02
0.34+0.03
0.47+0.03
TS
0.57+0
0.23++002
0.20+0.09
Phoemix
0,56+0.01
0,15+0.09
0.15+0.16
DPR
Vicuna
0.42+0.006
0.16+0.11
0.15+0.14
CharGLM
0.53+0.0
0.16+0.04
0.31+0.07
GPT-3.5
0.69+0.04
0.4140.02
0.31+0.066
TS
0.75+0.01
0,28+0.02
0.18+0.09
Phoenix
0.67+0.02
0.17+0.10
0.17+0.13
SimSCE
Vicuna
0.47+0.06
0.19+0.06
0.10+0.20
ChatGLM
0.75+00.0
0.17+0.05
0.21+0.06
GPT-3.5
0.77+0.00
0.37+0.05
0.31+0.06
TS
0.80-0.08
0.35++003
0.18+0.11
Phoenix
0.69+0.02
0.10+0.11
0. 0.16+0.18
Contriver
Vicuna
0.58+00.8
0.17+0.12
0. 0.1440.19
ChatGLM
0.71+0005
0.13+0.09
0.23+0.05
GPT-3.5
0.80+0.02
0.37+0.05
0.30+0.08
'+' indicutes improvement in performance metrics upon the incorporation of PRCA. The color coding provides a visual representation of the effect: Green signities positive enhancement in performance, while Red indicates decrement.
configuration settings employed in our experiment are stated in Table 2.
generator performance by PRCA across the three datasets is incremental, while the original perfor- mance of the generators across the three datasets is decremental without PRCA, correlating directly with the complexity of the datasets. This is be- cause when faced with more complex issucs, such as multi-hop questions in HotpotQA and topic tran- sitions in multi-turn QA in TopiOCQA, PRCA re- serves and integrates critical information which is beneficial for generators from the retrieved doc- uments. This attribute of PRCA alleviate issues where generators struggle with lengthy texts, fail- ing to answer questions correctly or producing hal- lucinations, thus enhancing performance.
5 Results and Analysis
5.1 Overall Performance
As delineated in Table 3, among the seventy-five configurations, our experimental results suggest that the inclusion of PRCA improves performance in seventy-one configurations. On average, we ob- serve enhancement of 3%, 6%, and 9% on the SQuAD, HotpotQA, and TopiOCQA datasets, re- spectively. This demonstrates that PRCA possesses robustness and can enhance the performance of dif- ferent combinations of retrievers and generators on the ReQA task. As illustrated in Figure 3, the improvements rendered by PRCA to the generators are significant across all three datasets. Particularly on the TopiOCQA dataset, the average improve- ment for generator Vieuna across five different re- trievers reaches 14%. Notably, when SimSCE is the retriever, the enhancement offered by PRCA is 20%.
However, the inclusion of PRCA has a negative effect on the performance of the generator T5 on the SQuAD dataset. This is because the SQuAD dataset is relatively simple, where the answer often directly corresponds to a phrase in the text. As encoder-decoder architecture model, T5 tends extract answers directly rather than infer in-depth based on the context. Therefore, without infor- mation distillation by PRCA from the retrieved
In Figure 3, we notice that the improvement to

---

T5
Phoenix
Baselines
Vicuna ChatGLM
GPT3.5
#120%
TopicOCAA
GPT3.5
Positive Effect PRCA
Effect
0.6
Negafive Effect of PRCA
HotpotQA
GPT4 Aasessment
SQuAD
0.6 GPT4 Accuracy
0.3
+27.1%
-64.%%
Figure 3: Comparison of performance different gen- erators (T5, Phoenix, Vicuna, ChatGLM, and GPT-3.5) on three benchmark datasets: SQuAD, HotpotQA, and TopicOCQA. The horizontal axis represents the GPT-4 assessment accuracy. Bars depict the performance levels each generator, with green and red arrows indicating the enhanced diminished effects due PRCA inte- gration, respectively.
0.1
SQUAD
HotpotQA
TopicOCQA
Figure 4: Performance comparison between PRCA- enhanced baseline models and GPT-3.5 across SQuAD, HotpotQA, and TopicOCQA. Light and dark blue bars represent baseline and GPT-3.5 performance, while striped green indicates PRCA's improvement.
documents, T5 performs well because its features fit well in handling this dataset, capable of directly extracting answers from the context. But under the effect of PRCA, the structure of the text might be altered, and T5's direct answer extraction may lead to some errors, thereby reducing performance.
Table 4: PRCA inference speed test results.
Dataset Precision
GPU
Batch Size
Inference Speed
(token/s)
While in a few configurations, the characteristics of PRCA may have negative effects, for the vast majority of configurations, our experiments vali- date that under PRCA-based paradigm, PRCA can effectively enhance the performance in the ReQA task, demonstrating robustness.
PRCA PRCA PRCA
float32
A100
1
126
float32
A100
2
231
float32
A100
4
492
parameter increment, the performance of these generators improved by 12.0%, 27.1%, and 64.5% respectively. Hence, PRCA has great potential be an efficient way boost the performance of ReQA task while keeping computational resources consumption acceptable. During the inference process, fully-trained PRCA will perform only standard forward propagation and hence introduce limited impact on inference latency. Our inference latency test SQUAD reported in Table 4. This low latency ensures that the system maintains smooth process without significant delays after integrating PRCA, underscoring the high efficiency of PRCA in boosting system performance.
5.2 Efficiency of PRCA
PRCA represents an effective approach for enhancing the performance of the ReQA task without significantly increasing computational demand. Its efficiency is manifested in optimizing parameters to achieve superior results and in simplifying input text, thereby aiding generators in managing complex text.
Parameter Efficiency Figure portrays a comparative analysis between the generators, which gain the maximum improvements with PRCA, and the GPT-3.5 model which operates without PRCA, across datasets. PRCA boasts roughly 0.4 billion parameters, the most signif- icantly improved generators encompass about billion parameters on average, while GPT-3.5 has approximately 1.75 trillion parameters. As demonstrated in Figure 4, with a marginal
Input Simplification As illustrated in Figure 5, analyzed the relationship between reward and token count during reward-driven stage for QA pair in the HotpotQA dataset, with and without PRCA. There's a discernible difference in the re- ward trajectories with and without PRCA. Both reward curves ascend with the increase in token

---

Reward Without PRCA
SQuAD
SQuAD-PRCA
Reward With PRCA
HotpotQA
HotpotQA-PRCA
1.0
TopiOCQA
TopiOCQA-PRCA
0.8
Reward
0.4
GPT4 Assessmentt Accuracy
0.8
0.2
200
400
600
800
0.6
Token Count
Figure 5: A depiction of reward trajectories over in- creasing token counts during the reward-driven stage for QA pair within the HotpotQA dataset. Distinct lines represent rewards achieved with and without the implementation of PRCA, underscoring PRCA's ability extract concise and high-quality text.
0.4
10
13
15
18
20
Top-K Retrieved Documents
count, but the gradient of ascent with PRCA is no- ticeably steeper. This implies that when PRCA is in action, the generator reaches its optimal perfor- mance with a significantly reduced token count.
Figure 6: Comparison of performance with and with- out PRCA with the different number of retrieved docu- ments.
Under the influence PRCA, the generator can derive the correct answer with approximately four times fewer tokens. This indicates that PRCA can distill the retrieved text while ensuring the qual- ity of the generated answer. This simplification process filters redundant information, thereby promoting the generator extract answers more accurately using a more streamlined context. More- over, the reduction in token count enables the gen- erator to process text faster and produce outputs more promptly. Overall, PRCA's efficiency in infor- mation distillation greatly bolsters the generator's ' capacity to manage and interpret complex text.
documents are added. This decline is attributed to the model's diminishing capability to generate accurate answers complex questions due the rise in distracting information and the onset of hal- lucination problems. However, by implementing PRCA, these adverse effects are systematically leviated, which not only reduces the onset of hallu- cinations but also enhances the generator's ability to handle complex queries amidst distractions.
In general, at different Top-K values, PRCA demonstrates positive effects across all three datasets, thereby illustrating the universal appli- cability of PRCA regardless of the quantity re- trieved documents.
5.3 Impact of Top-K Selection
We conducted parameter sensitivity experiments to observe the performance of PRCA when the number of retrieved relevant documents changes. The results presented in Figure 6 show that the SQuAD dataset, both the performance with and without PRCA improve as the number of retrieved documents increases, while the addition of PRCA consistently provides a positive effect across dif- ferent Top-K values. Since the dataset is relatively simple, with the increased likelihood of the correct answer being included in the retrieved documents, both trends exhibit upward trajectory.
5,4 Case Study
When answering the form of Mersenne primes problem, the retrieved text contains two distinct sources of information. One directly specifies the form as 2p-1, accurately reflecting the nature Mersenne primes. The other source misguidedly introduces "factorial primes" as an answer. With- out PRCA's intervention, this diversion leads the generator astray, resulting in erroneous answer of "factorial primes". However, when PRCA is engaged, it sifts through the information, priori- tizing the accurate context. This refined context extraction steers the generator towards the correct answer.
In contrast, without the implementation of PRCA, there is a noticeable drop in performance on the HotpotQA and TopiOCQA datasets when more

---

Case Study without and with PRCA
Effect of PRCA With the
Reward-Drivv Training Stage
Question:
Effect of PRCA Without the
Reward-Driven Training Stage
Of what form do Mersenne primes take?
Golden Answer: 2p-1
TopicOCQA
Part of Retrieved Documents: [Golden
Answer Source] Mersenne primes are prime numbers that are of the form 2p=1, where an arbitrary prime. [Pre- dicted Answer Source] The Sieve of Er- atosthenes, attributed to Eratosthenes, is simple method to compute primes, although the large primes found today with comput- are not generated this way, are prime. Prime numbers of this form known factorial primes.
HolpotQA
-0.08
0.13
SQuAD
Aceuraac GPT4 Assessment Accuracy
Figure An illustration showcasing the impact of the reward-driven stage on PRCA's performance.
Conclusion
Predicted Answer without PRCA: Facto- rial primes
In conclusion, this research successfully introduces a PRCA-based paradigm for ReQA tasks, tack- ling the inherent challenges of fine-tuning LLMs in the retrieval-cnhancemenn framework, especially given their vast parameter size and closed-source natures. PRCA innovatively distills retrieved docu- ments via generator rewards, leading marked improvement in the ReQA task's performance. Ex- perimental outcomes consistently demonstrate the robustness and effectiveness PRCA when paired with various retrievers and generators, indicating its potential to be widely deployed as an adapter on the ReQA task.
Context through PRCA: Mersenne primes are prime numbers that are of the form 2p-1, where p is an arbitrary prime. The Lu- cas Lehmer test is particularly fast for num- bers of this form, so many of the largest primes found today are Mersenne primes. Predicted Answer with PRCA: 2p-1 Note: "_" denotes key information relevant to the question, '*7 represents predicted answers.
Limitations
5.5
Ablation Study of PRCA
While PRCA has shown effectiveness in improving ReQA task performance, it has limitations, includ- ing dependency on generators, convergence issues. and limited integration with retrievers. The reward during reinforcement learning training is derived from the generator, requiring PRCA retraining with different generators, which can be time-consuming. PRCA may also experience difficulties converging in single training session, which impacts the sta- bility and consistency of its performance. Lastly, PRCA's operation
We assessed the impact of PRCA on three datasets using the configurations from section 5.2, which showed maximum improvements. The evaluation is conducted with and without the reward-driven stage to observe the impact of PRCA on the per- formance. As illustrated in Figure 7, without the reward-driven training stage, the effect of PRCA on the entire configuration becomes adverse because PRCA merely simplifies the text without discerning which information is beneficial for the generator to answer questions, resulting in the omission of useful text. In contrast, once the training process incorporates the reward-driven stage, the quality of the context becomes directly aligned with reward values, assisting PRCA in more effectively distill- ing pertinent information. Therefore, the reward- driven stage is vital, allowing PRCA to retain key details while simplifying text, enhancing its overall effect.
pluggable adapter limits its ability to train jointly with retrievers, which means if the retrieval quality not up to par, PRCA's effectiveness could be compromised.
Acknowledgement
Supported by the Key Research and Develop- ment Program of Guangdong Province (grant No. 202 100101400003) and Corresponding author is Jianzong Wang (jzwang@) 18.ccm).

---

References
Zhengxiao Du Yujic Qian, Xiao Liu, Ming Ding. Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. GLM: General language model pretraining with autoregres- sive blank infilling. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 320-335, Dublin, Ireland. Association for Computational Lin- guistics.
Vaibhav Adlakha, Shehzaad Dhuliawala, Kaheer Sule- man, Harm de Vries, and Siva Reddy. 2022. Top- iOCQA: Open-domain conversational question an- swering with topic switching. Transactions of the Association for Computational Linguisties, 10:468- 483.
Chenxin An, Ming Zhong, Zhichao Geng, Jianqiang Yang, and Xipeng Qiu. 2021. Retrievalsum: A re- trieval enhanced framework for abstractive summa- rization. CoRR, 0bs/2109.07943.
Fazzie, FrankLeeeee, BlueRum, ver217, ofey404, Wen- hao Chen, Zangwei Zheng, and Xue Fuzhao. 2023. Colossalchat. https://gi thub. com/hpcaitech/ oolosssall/ttrree/aiinn//ppplll httpi//apppcaaaiin//ctcattttt
Sebastian Borgeaud, Arthur Mensch, Jordan Hoff- mann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, Jean-Bapy -aptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang. Loren Mag- giore. Chris Jones, Albin Cassirer, Andy Brock. Michela Paganimi, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rac, Erich Elsen, and Laurent Sifre. 2022. Improving language models by retrieving from trillions of tokens. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 2206-2240. PMLR.
Tianyu Gao, Xingeheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Process- ing. pages 6894-6910, Online and Punta Cana, Do- minican Republic. Association for Computational Linguisties.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model gra-training. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3929-3938, PMLR.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebas- tian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022a. Unsupervised dense informa- tion retrieval with contrastive learning. Transactions on Machine Learning Research, 08:1-21.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open do- main question answering. In Proceedings of the 16th Conference of the Eurppan Chapter of the Associ- ationfor Computational Linguistics: Main Volume. pages 780, Online. Association for Computa- tional Linguistics.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lu- cas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022b. Few-shot learning with re- trieval augmented language models. arXiv preprint arXiv:2208.03299
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open- domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics ( (Volume 1: Long Papers), pages 1870-1879, Vancouver, Canada. Association for Computational Linguisties.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6769-6781, Online. As- sociation for Computational Linguistics.
Zhihong Chen, Feng Jiang, Junying Chen, Tiannan Wang, Fei Yu, Guiming Chen, Hongbo Zhang. Juhao Liang, Chen Zhang, Zhiyi Zhang, al. 2023. Phoenix: Democratizing chatgpt across languages. arXiv preprint 21Xx:2233441045.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a. BART: Denoising sequence-to-sequenee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Compulational Linguistics,
Aakanksha Chowdhery. Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts. Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.

---

pages 7871-7880, Online, Association for Computa- tional Linguistics.
Stephen Robertson and Hugo Zaragoza. 2009. The prob- abilistic relevance framework: Bm25 and beyond. Foundations and Trends0 in Information Retrieval, 3(4):333-389
Patrick Lewis. Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttle, Mike Lewis, Wen-tau Yih. Tim Rock- täschel, Sehastian Riedel, and Douwe Kiela. 2020b. Netrieval-augmeenn generation for knowledge- intensive nlp tasks. In Advances in Neural Infor- mation Processing Systems, volume 33, pages 9459- 9474. Curran Associates, Inc.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ilié. Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni. François Yvn., Matthias Gallé, al. 2022. Bloom: 176b- parameter open-aceess multilingual language model. arXiv preprint arXiv:2211.00 22222221111000
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2023. Palm 2 technical report. ArXiv, abs/2305. 100/2305.1003..
Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. Query rewriting for retrieval- augmented large language models. arXiv preprint arXiv:2305. 14283.
OpenAI. 2023, Gpt-4 technical report. arXiv preprint arXiv:2302.08774.
John Schulman, Philipp Moritz, Sergey Levine. Michael I. Jordan, and Pieter Abbeel. 2016. High- dimensional continuous control using generalized advantage estimation. In The Fourth International Conference on Learning Representations.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27740.27744. Curran Associates, Inc.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. CoRR, abs/1707.06347.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Min- joon Sco, Rich James, Mike Lewis, Luke Zettle- moyer, and Wen-tau Yih. 2023. Replug: Retrieval- augmented black-box language models. arXiv preprint arXiv:2301. 12652.
Xiaoman Pan, Kai Sun, Dian Yu, Jianshu Chen, Heng Ji, Claine Cardie, and Dong Yu. 2019. Improving question answering with external knowledge. In Pro- ceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 27-37, Hong Kong, China. Association for Computational Linguistics.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3784-3803 Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Baolin Peng, Chunyuan Li, Pengcheng He. Michel Gal- ley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277
Karen Sparek Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28(1):11-21.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Nisan Stiennon, Long Ouyang. Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learn- ing to summarize with human feedback. In Ad- vances in Neural Information Processing Systems, volume 33, pages 3008-3021. Curran Associates, Inc.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Prucessing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Rohan Taori, Ishaan Guirajani. Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Talsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-followiin llama model. https: // gi wtt./////tttttllabcc httpsuu-aaaassttaod__aaappa
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing, pages 3982-3992, Hong Kong, China. Association for Computational Linguis- tics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet. Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint 1302:2302.11971

---

Jason Wei. Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emer- gent abilities of large language models. Transactions on Machine Learning Research, 08:1-30. Survey Certification.
Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. 2018. HotporQA: dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference Empiri- cal Methods Natural Language Processing, pages 2369-2388 Brussels, Belgium. Association for Com- putational Linguistics.
Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari, Xiaoxia Wu, Am- mar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, Zhongzhu Zhou, Michael Wyatt, Molly Smith, Lev Kurilenko, Heyang Qin, Masahiro Tanaka, Shuai Che, Shuaiwen Leon Song, and Yuxiong He. 2023. DeepSpeed-Chat Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales. arXiv preprint arXiv:2308.01320.
Aoban Zeng, Xiao Liu, Zhengxiao Du Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang. Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. 2023. GLM-130b: An open bilingual pre-trained model. In The Eleventh International Conference on Learning Representations.


Reviews:
[{"review_id": "C4w9QWQsVw", "reviewer": "Reviewer_bnUG", "summary": "", "questions": "+ You state that the framework ensure efficiency , but can you provide some latency test of the pipeline\n+ can you better explain why you did not use other datasets with respect to SQUAD?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our work and provide feedback. We appreciate the detailed comments and the opportunity to address your concerns. Please find our responses to each of the issues raised below:\n\nQ1: You state that the framework ensures efficiency, but can you provide some latency test of the pipeline?\n\nResponse: Regarding the efficiency of the PRCA component, we have conducted tests on the SQuAD dataset. The specific test data is as follows:\n\n| Model Data | Precision | GPU   | Batch Size | Inference Speed (token/s) |\n|---------------|------------|----------|-----------|-------------------------|\n| PRCA       | float32   | A100  | 1          | 126                       |\n| PRCA       | float32   | A100  | 2          | 231                       |\n| PRCA       | float32   | A100  | 4          | 492                       |\n\nFor example, we conducted a latency test with a batch size of 1 on 100 texts. The average length was 486 tokens. On an A100 GPU, our PRCA processing speed was found to be 3.86s. This indicates that, upon integration, the PRCA component is efficient and does not introduce significant time overhead.\n\nQ2: Can you better explain why you did not use other datasets with respect to SQUAD?\n\nResponse: We did consider other datasets, including NQ and triviaQA. However, there were some challenges associated with its utilization. The context lengths in NQ and triviaQA are considerably bigger. For example, in a preliminary run of 300 contexts from triviaQA, we found the maximum text length to be 258,027 and an average length of 10,228. Given that our PRCA is trained on the BART model, there are inherent length constraints of 1024. In comparison, our three datasets offer more manageable context lengths, aligning better with our model's capacity. \n\nQ3: Manual evaluation of the results is missing.\n\nResponse: In fact, we have conducted manual evaluations to further validate our results. On three different datasets, we randomly selected 100 prediction-answer pairs generated by the combination of GPT-3.5 and Contriever. We then compared these predictions against the golden answers through both manual evaluation and GPT-4-based evaluation. The results indicate that GPT-4's evaluation accuracy is quite comparable to human assessment, with correctness rates of 96%, 93%, and 92% respectively. These findings suggest that GPT-4's evaluation approach can serve as a robust alternative to manual assessments, reducing the need for intensive manual work. On the other hand, GPT-4 as reported in different works, could serve as a strong evaluation benchmark.[ Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality,  AlpacaEval: An Automatic Evaluator of Instruction-following Models, Judging LLM-as-a-judge with MT-Bench and Chatbot Arena.]\n\nIn conclusion, we believe that the clarifications and additional data provided here address the concerns raised by the reviewer. We remain committed to ensuring the robustness and validity of our approach and are open to any further feedback to improve our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ exhaustive experiments with several baselines on different dataset\n+ interesting approach to improve the LLM outcomes without finetuning the LLM itself\n+ tested on different domains", "reasons_to_reject": "+ missing further experiments on other more complex QA datasets rather than SQAD (e.g: NQ)\n+ manual evaluation of the results missing\n+ the trained model will learn how to optimize with respect to the final LLM\n+ missing latency experiments of the pipeline with and without the context generative model", "questions_to_authors": "+ You state that the framework ensure efficiency , but can you provide some latency test of the pipeline\n+ can you better explain why you did not use other datasets with respect to SQUAD?", "paper_topic_and_main_contributions": "The paper is about a new proposed framework  named PRCA. In this framework between the retriever and the LLM exists a generative model that improve the context for the final answer generation via LLM. The LLM and the retriever are frozen and only the \"middle\" model is trained", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dWRbmNcAAf", "reviewer": "Reviewer_N8VF", "summary": "", "questions": "See reasons to reject.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and questions. Please find our clarifications and responses below:\n\nQ1: Why do we use GPT-4 to evaluate answer accuracy? How about the common metrics like EM and F1 in your table 2?\n\nResponse: EM and F1 are widely used metrics, and we initially attempted to employ them for evaluating the difference between predicted and standard answers. However, their performance was not satisfactory for our use case. A significant limitation we observed was that semantically similar sentences could have a low EM or F1 score. LLMs often output longer textual explanations for answers, even when the correct answer might be a word or two. Although we attempted to introduce prompts to constrain answer length and make EM and F1 metrics more effective, the outcomes were not as expected.\n\nConsider the example:\n\nQuestion: When did Tesla patent the motor?\n\nPrediction Answer: Tesla was granted the patent for the alternating current induction motor in May 1888.\n\nGolden Answer: 1888\n\nThe prediction answer correctly addresses the question. Still, if we were to evaluate it using EM or F1, the score would be undeservedly low, misrepresenting the quality of the model. Hence, we adopted GPT-4-based evaluation, which excels at assessing semantic similarity between sentences.\n\nQ2: In Figure 6 and 7, which generator do you use? I am not sure about the effect of PCRA if we use a strong generation like GPT-3.\n\nResponse: We employed Vicuna as the generator for Figures 6 and 7. Vicuna, as reported officially, possesses about 90% of chatGPT's capabilities, rendering it a potent generator. \n\nFurthermore, in the research paper “TopiOCQA: Open-domain Conversational Question Answering with Topic Switching” introducing the TopiQCQA dataset, experiments demonstrate that GPT-3, a strong generator, achieves only an F1 score of 31.8% on the test set. This suggests that even formidable generators might not attain optimal results on intricate datasets. In our paper's Table 2, when using GPT-3.5 as the generator, PRCA consistently showed positive effects across various datasets. We believe that PRCA is highly beneficial, even when paired with a powerful generator. \n\nQ3: How about the comparison between your approach and the REPLUG (Replug: Retrieval-augmented black-box language models)? Do you have results on their experimented datasets for comparison?\n\nResponse: The dataset used by REPLUG is triviaQA. In a preliminary run of 300 contexts from triviaQA, we found the maximum text length to be 258,027 and an average length of 10,228. Given that our PRCA is trained on the BART model, there are inherent length constraints of 1024. In comparison, our three datasets offer more manageable context lengths, aligning better with our model's capacity. Meanwhile, REPLUG requires direct access to models raw outputs which might not be available from different model vendors. \n\nOn the other hand, due to the closed source nature of the training data of GPT series, stronger generation baseline has a bigger potential data leakage problem and leads to less convincing experiments.\n\nQ4: How about using ChatPGT with few-shot as the context summarizer?\n\nResponse: ChatGPT cannot be trained directly, so few-shot training is not feasible for it. However, if we approach this using an in-context learning method, the comparative results have already been provided in our Table 2.\n\nThank you for your insightful questions. We believe our clarifications provide a comprehensive understanding of our approach and its advantages. We're eager to receive further feedback to refine our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Their proposed PRCA approach achieve consistent improvement with various retriever and generation combinations. Their ablation study demonstrate the effectiveness of PCRA and reward-driven stage in PCRA.", "reasons_to_reject": "There several questions about the papers.\n1. Why do we use GPT-4 to evaluate answer accuracy? How about the common used metrics like EM and F1 in your table 2?\n2. In Figure 6 and 7, which generator do you use? I am not sure the effect of PCRA if we use a strong generation like GPT-3.\n3. How about the comparison between your approach the REPLUG <Replug: Retrieval-augmented 720 black-box language models>. Do you have results on their experimented datasets to have a comparison?\n4. How about using ChatPGT with few-shot to as the context summarizer?", "questions_to_authors": "See reasons to reject.", "paper_topic_and_main_contributions": "This paper focused on retrieval question answering tasks by treating the Large Langauge Models (LLMs) as black box. Specifically, they propose a trainable Pluggable Reward-Driver Contextual Adapter (PRCA) between the retriever and generator. To train PRCA, there are two stages. The first stage is to extract a concise and effective context from retrieved passages with supervised loss between extracted context and ground truth context by using a BART-large model. The second stage is to use the extracted text to guide the generator's answering. They utilize ROUGE-L score as the reward to optimize the PRCA parameter while also maintain the similarity between updated and original parameters.  They conduct experiments on 3 three QA datasets with various retriever. The results show that PRCA can enhance the performance of different combinations of retrievers and generators.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "Bfl48gUefW", "reviewer": "Reviewer_JYC9", "summary": "", "questions": "Q1: I didn't find any descriptions of the critic network, how is it initialized and trained?\n\nQ2: Many experimental details are missing. How does the two-stage training work? Sequentially or iteratively? What are the hyperparameters in training such as learning rate?\n\nQ3: The used backbones are all vanilla language models (i.e. not fine-tuned for ReQA tasks), can the proposed PRCA also work on fine-tuned baselines? Perhaps if we already have a fine-tuned generator, it can produce better results and thus less chance to be improved by external adapters like PRCA.\n\nQ4: Line 442 mentioned that the PRCA has 4B parameters, however, Line 222 said that PRCA is initialized by pre-trained model BART-Large-CNN, which I believe is not that large. Did I miss some part of PRCA? Is it the critic network?\n\nQ5: Following Q3, if the PRCA has 4B parameters, directly fine-tuning those while-box models which actually have a similar size to the PRCA will have a similar cost. I am curious which approach is better.\n\nQ6: What version of GPT3.5 is used?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: I didn't find any descriptions of the critic network, how is it initialized and trained?\n\nResponse: Indeed, we have not employed a critic network in our method. Typically, the initialization of a critic network involves using the initial weights of a reward model. However, since we leverage a black-box generator as our reward model in this work, we don't have access to the critic network's initial weights. \n\nAs stated from line 280 in the paper, we propose a simpler method to compute the reward for each token without requiring a critic network. Our approach innovatively estimates the reward $R_t$ for each token within the PRCA system. By harnessing the generation probability of each token, we calculate its contribution to the total reward. We obtain the primary reward, $R_{EOS}$ when an <EOS> token is produced, comparing the generated answer to the ground truth. Our method's strength lies in its efficiency: it requires the reward model's invocation just once per context generation. This is markedly more efficient than the original PPO, especially beneficial when using Large Language Models as generators.\n\nQ2: Many experimental details are missing. How does the two-stage training work? Sequentially or iteratively? What are the hyperparameters in training such as learning rate?\n\nResponse: Our training process for the PRCA component is structured into two sequential stages: \nContextual Extraction Stage: We begin by initializing the PRCA model with the weights from the BART-Large model. Following this, the model undergoes training using the question-answer pairs from our dataset.\nReward-Driven Stage: Once the first stage is complete, we proceed to this phase where we refine the model weights through reinforcement learning, aiming to distill crucial information. It's worth noting that these stages are sequential; we complete the first before initiating the second.\nFor further clarity, below is a table detailing the hyperparameters utilized during our training:\n\n| Hyperparameter  | Value               |\n|-----------------|---------------------|\n| Learning rate   | 0.00005             |\n| Batch size      | 1/2/4               |\n| num_beams       | 3                   |\n| temperature     | 1                   |\n| early_stopping  | True                |\n| top_k           | 0.0                 |\n| top_p           | 1.0                 |\n\nQ3: The used backbones are all vanilla language models (i.e. not fine-tuned for ReQA tasks), can the proposed PRCA also work on fine-tuned baselines?\n\nResponse: The generator, being a black-box, can originate from diverse vendors. Consequently, there might be no feasibility or cost-effectiveness in fine-tuning them specifically for tasks. Our approach ensures adaptability even when specific task fine-tuning isn't an option.\n\nQ4: Line 442 mentioned that the PRCA has 4B parameters, however, Line 222 said that PRCA is initialized by pre-trained model BART-Large-CNN. Did I miss some part of PRCA?\n\nResponse: We apologize for the oversight. The correct size of PRCA is 0.4B parameters. Line 442 contains a typing error, and we appreciate your attention to detail.\n\nQ5: Following Q3, if the PRCA has 4B parameters, directly fine-tuning those white-box models which actually have a similar size to the PRCA will have a similar cost. I am curious which approach is better.\n\nResponse: Firstly, we apologize for the typing error, and I reiterate that PRCA's actual size is 0.4B. Given this, if one were to fine-tune the white-box models directly, their generally smaller parameter size would likely not reach the performance levels of the black-box models, reinforcing the relevance and utility of our approach.\n\nQ6: What version of GPT3.5 is used?\n\nResponse: We utilized the gpt-3.5-turbo-0301 model, which was released by OpenAI in May 2023.\n\nQ7: As discussed in the Limitation section, the RL algorithms are usually unstable. More investigation into the stability of training can be encouraged to make the approach more applicable. \n\nResponse: Trick 1 - Advantage Normalization: By normalizing the advantages, we ensure that the update step sizes are consistent, which helps in stabilizing the learning process and mitigating extreme policy updates.\n\nTrick 2 - State Normalization: This involves standardizing the state inputs to have zero mean and unit variance. By doing so, we help the policy function to generalize better and ensure smoother training trajectories.\n\nTrick 3 - Learning Rate Decay: Over the course of training, reducing the learning rate can be beneficial. As we approach the optimal policy, a smaller learning rate prevents large oscillations and helps the agent to converge to a stable solution more reliably.\n\nThank you for your insightful questions. We believe our clarifications provide a comprehensive understanding of our approach and its advantages. We're eager to receive further feedback to refine our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Novel method to leverage LLMs in a retrieval framework without accessing their parameters.\n\n2. Sensible pipeline design, i.e., context extraction and reward training.\n\n3. Providing insights to the usage of LLMs in the general QA tasks in NLP community.", "reasons_to_reject": "1. Some details about the approach and the experiments are missing, making it hard to follow and reproduce. (See questions)\n\n2. The authors only provide the results of some vanilla language models (e.g. Vicuna, ChatGLM, GPT-3.5) and them with PRCA.  Although I understand that the proposed method is not in competition with other specific methods, adding more baselines as references could be more informative.\n\n3. Many improvements seem marginal in Table 2, in this case, it is suggested to conduct a significance study to confirm that the improvement brought by PRCA is indeed significant.\n\n4. More evaluation metrics are encouraged, only using GPT4 might be biased and inaccurate.\n\n5. As discussed in the Limitation section, the RL algorithms are usually unstable. More investigation into the stability of training can be encouraged to make the approach more applicable.", "questions_to_authors": "Q1: I didn't find any descriptions of the critic network, how is it initialized and trained?\n\nQ2: Many experimental details are missing. How does the two-stage training work? Sequentially or iteratively? What are the hyperparameters in training such as learning rate?\n\nQ3: The used backbones are all vanilla language models (i.e. not fine-tuned for ReQA tasks), can the proposed PRCA also work on fine-tuned baselines? Perhaps if we already have a fine-tuned generator, it can produce better results and thus less chance to be improved by external adapters like PRCA.\n\nQ4: Line 442 mentioned that the PRCA has 4B parameters, however, Line 222 said that PRCA is initialized by pre-trained model BART-Large-CNN, which I believe is not that large. Did I miss some part of PRCA? Is it the critic network?\n\nQ5: Following Q3, if the PRCA has 4B parameters, directly fine-tuning those while-box models which actually have a similar size to the PRCA will have a similar cost. I am curious which approach is better.\n\nQ6: What version of GPT3.5 is used?", "paper_topic_and_main_contributions": "This paper proposes a Pluggable Reward-driven Contextual Adapter (PRCA) to help retrieval question answering. The main contributions are:\n\n1. PRCA allows using black-box LLMs as black boxes without fine-tuning.\n\n2. The reinforcement learning part provides some insight to the usage of LLMs.\n\n3. Experiments showing PRCA comprehensively improves performance on 3 QA datasets.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Fix typos: \"conducte\" in line 501."}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Summary:** The paper proposes to incorporate an additional module: Pluggable Reward-Driven Contextual Adapter (PRCA) in the retrieval-augmented generation setting (RAG/ReQA). PRCA is plugged between the retriever and the generator: each of the latter two modules are frozen, and only PRCA is trained (PRCA is itself based on a generative LM backbone) using a reinforcement learning approach to refine the retrieved content so as to optimize and provide a better context as prompt for the generator. Specifically, PRCA first extracts a relevant context from the retrieved passages, and then uses a ROUGE-L score to reward/penalize parameter updates while also maintaining similarity to the original parameters. The approach is empirically evaluated on three QA datasets, and beats baselines by significant margins.

All reviewers acknowledged the novelty of the solution proposed in the problem: also highlighting it's practicality to suit real world industry use-cases where the LLM cannot be trained/only API access calls are available. The claims in the paper are supported well empirically with evaluation over 3 datasets. Based on clarifications provided in the author rebuttal, most major concerns raised by the reviewers have been mitigated.

**Recommendations for improvement:** (i) One of the concerns on the paper is around the usage of only automatic evaluation (GPT-4) and no human evaluation to support the claims. The author response has partially addressed this by providing a small pilot study on 100 examples to compare human and automatic evaluation (and achieved parity in the late 90%s). I would encourage the authors to expand on this study, and add it to the main paper to strengthen claims of the paper.

(ii) The authors should add all details from the author response clarifying several of reviewer JYC9's concerns around missing details in the paper, and adding significance tests to the empirical results.
Decision: Accept-Main