ChatQA: Surpassing GPT-4 on Conversational QA and RAG
Authors: Zihan Liu
Wei Ping
Rajarshi Roy
Peng Xu
Chankyu Lee
Mohammad Shoeybi
Bryan Catanzaro
Venue: NeurIPS 2024 poster (2024)
Keywords: large language models
retrieval-augmented generation
RAG
Primary area: natural_language_processing
TL;DR: We introduce ChatQA, a suite of models that outperform GPT-4 on RAG and conversational QA.

Abstract: In this work, we introduce ChatQA, a suite of models that outperform GPT-4 on retrieval-augmented generation (RAG) and conversational question answering (QA).  To enhance generation, we propose a two-stage instruction tuning method that significantly boosts the performance of RAG.  For effective retrieval, we introduce a dense retriever optimized for conversational QA, which yields results comparable to the alternative state-of-the-art query rewriting models, while substantially reducing deployment costs.  We also present the ChatRAG Bench, which encompasses ten datasets covering comprehensive evaluations on RAG, table-related QA, arithmetic calculations, and scenarios involving unanswerable questions.  Our ChatQA-1.0-70B (score: 54.14), built on Llama2, a weaker foundation model than GPT-4, can slightly outperform GPT-4-0613 (score: 53.90) and GPT-4-Turbo-2024-04-09 (score: 54.03) on the ChatRAG Bench, without relying on any synthetic data from OpenAI GPT models.   Notably, Llama3-ChatQA-1.5-70B model surpasses the accuracy of GPT-4-Turbo-2024-04-09 by a margin. These results demonstrate the exceptional quality of the proposed ChatQA recipe. To advance research in this field, we open-sourced the model weights, instruction tuning data, ChatRAG Bench, and retriever for the community.

ChatQA: Surpassing GPT-4 on Conversational QA and RAG
Zihan Liu +1
Wei Ping
Rajarshi Roy
Peng Xu
Chankyu Lee"
Mohammad Shoeybi
Bryan Catanzaro 1
NVIDIA
Abstract
In this work, we introduce ChatQA, suite of models that outperform GPT-4 on retrieval-augmented generation (RAG) and conversational question answer- ing (QA). To enhance generation, we propose two-stage instruction tuning method that significantly boosts the performance of RAG. For effective retrieval, we in- troduce dense retriever optimized for conversational QA, which yields results comparable to the alternative state-of-the-are query rewriting models, while sub- stantially reducing deployment costs. We also present the CHATRAG BENCII, which encompasses ten datasets covering comprehensive evaluations on RAG, table-related QA, arithmetic calculations, and scenarios involving unanswerable questions. Our ChatQA--10-700 (score: 54.14), built on Llama2, weaker foun- dation model than GPT-4, can slightly outperform GPT-4-0613 (score: 53.90) and 1o44-Tuubo-2224-04-402
(score: 54.03) the CHATRAG BENCH, with- out relying on any synthetic data from OpenAl GPT models. Notably, Llama3- ChatQA-1.5-70B model surpasses the accuracy of GPT-44-Trbo-2024-0-444 margin. These results demonstrate the exceptional quality of the proposed ChatQA recipe. To advance research in this field, we open-sourced the model weights, instruction tuning data, CHATRAG BENCH, and retriever for the community: https://chatqa-projjeee github.io/.
by
Introduction
Most recently, ChatGPT (OpenAI, 2022) and its follow ups (OpenAI, 2023; Anthropic, 2023b; Google, 2023) have led to the paradigm shift of building question answering (QA) and retrieval- augmented generation (RAG) system in production and research community. In particular, the following aspects of the models are preferred: The users can interact with the QA models in conversational way, thus one can easily raise follow-up questions. ii) The models are capable of integrating retrieved chunks of evidence in both open-domain or long document settings, where the provided context is much longer than the context window of LLM (e.g., Anthropic, 2023a; Xu al., 2023b). îi) The generalist models can answer any questions with respect to table, arithmetic calculation in zero-shot manner without dataset-specific fine-tuning. while matching the accuracies of fine-tuned models. To this end, we focus on building state-of-the-ar model with all these key capabilities that are essentially important for many real-world applications.
However, building such model that can match the accuracy of the sute-of-the-as proprietary models, e.g., GPT-4 (OpenAI, 2023), is still grand challenge for the research community. In this work, we introduce ChatQA, a family of open-sourced models that can outperform GPT-4 while utilizing relatively weak open- weights foundation model. We also open-source our training data, detailing
*Correspondence to: Zihan Liu <zihanl @nvidia.coii......... Wei Ping <wping wwping@nvidia.com.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

techniques for synthetic data generation, along with alternative human annotation processes aimed at eliminating reliance on OpenAl GPT models for open research purpose.
Specifically, we make the following contributions:
1.
We propose two-stage instruction tuning method and design a dataset curation recipe that can largely enhance LLM's capability of integrating user provided or retrieved context for conversational QA and RAG tasks. We demonstrate that the proposed instruction tuning method significantly outperforms strong alignment baselines or RLHF-base based recipes (e.g.. Llama2-Chat
on RAG and various conversational QA tasks.
2. For retrieval, we show that fine-tuning the single-turn QA retriever on human aanaann-n data or synthetic multi-turn QA dataset works as well as utilizing the state-of-the-a LLM-based query rewriting model, i.e., GPT-3.5-Turbo (OpenAI, 2022). Our result also highlights the promising direction of utilizing synthetic data generation for training customized retriever.
We introduce CHATRAG BENCH, comprehensive benchmark with ten conversational QA datasets, including five datasets with long documents that need retrieval and three datasets with tabular data and arithmetic calculation. We apply ChatQA training recipe on different text foundation models, and show the superb generalization capability of the proposed methods. In terms of average score on CHATRAG BENCH, our ChatQA-1 --00070BB (54.14) based on Llama2 can outperform GPT-4-0613 (53.90) and TTTT----rrbo-222--00 any synthetic data from ChatGPT models. Notably, much smaller Saatmaa--hatQA--.... can perform comparable with GPT-4 models, while -amaa-ChaaQQ-1155-00 outperforms GPT-44-uuro-2224--0444 by a margin.
(54.03) without utilizing
We study the "unanswerable" scenario, where the LLM needs generate "cannot answer" to avoid hallucination. We show that incorporating small amount of "unanswerable" samples significantly enhances model's 's capability to handle it. Our ChatQA--L0-700 outperforms GPT-3.5-Turrl in this regard, while has slight gap compared to GPT-4-0613 (around 3.5%).
We discuss related work in $ 2. We introduce the two-stage instruction tuning method and data curation for ChatQA in $ 3, and study retrieval in conversational QA in 4. We present the experimental setup in $ 5, results in $ 6, and conclude the paper in $ 7.
2 Related Work
2.1 Conversational QA and RAG
Question answering in conversational way naturally improves experiences by addressing follow-up questions. The model can also raise clarification questions for users if necessary, which can reduce hallucination. Thus, it becomes the default format of deploying QA models in production (e.g. OpenAI, 2022; Google, 2023; Anthropic, 2023b). In contrast to the latest LLM-based generalist solution (e.g., OpenAI, 2022), most of the previous studies focus on fine-tuned expert models on specific domains or datasets (Feng et al., 2020; Izacard & Grave, 2021; Chen et al., 2022a; Gao et al., 2022: Nakamura et al., 2022: Adlakha al., 2022: Wu et al., 2023).
In recent years, many conversational QA datasets have been introduced. The models are asked to answer questions based on provided context documents, which involves satrivval-uugmente generation (RAG) the provided documents are longer than the context window of L.M. The provided context documents
be: i) text-only documents from various domains (Feng et al., 2020; Anantha et al., 2021; Saeidi et al., 2018; Adlakha et al., 2022; Aliannejadi et al., 2021; Reddy al., 2019; Qu al., 2020; Wu al., 2023; Deng al., 2022; Guo et al., 2021; Choi al., 2018; Campos et al., 2020), or ii) documents comprising plain text along with tables (Pasupat & Liang, 2015; Nakamura et al., 2022; Chen al., 2022a).
2.2 Retrieval for Multi-Turn QA
RAG is critically important for conversational QA in open-domain setting, e.g., utilizing update-to- date information from search engine, or when the proprietary documents are longer than the context window of L.M. The dense retrievers are usually trained to retrieve the top-k relevant chunks given single question (e-g.. Lin et al., 2023a; Wang et al., 2022a; Izacard al., 2022). In conversational QA, the follow-up questions (c.g., with pronouns referring to entities mentioned in the previous conversation) may have insufficient information for retrieval, while feeding them along with all of the dialogue history can be redundant, thus leading to sub-optimal results.

---

St Supervised Fine-tuning
Stage-2: Eonhnst- Instruction Tuning
Foundation L.M
Sela ELIE dbeskabier
SPT Model
NETEWOR Gred NOWOL TATQA UImmenserO
Chat:QA
Figure I: Two-stage instruction tuning framework for ChatQA.
Conversational Query Rewriting Most of the previous solutions are query rewriting methods. The latest turn of question is rewritten to be a standalone query without additional information from previous dialogue history (Vakulenko et al., 2021a; Ye et al., 2023; Mo et al., 2023), so it can be directly used by retrieval model to retrieve relevant context (Vakulenko et al., 2021b; Mele et al., 2021; Raposo et al., 2022; Mo et al., 2023). Many datasets have been collected to facilitate this line of research (Elgohary al., 2019; Chu al., 2020; Qu et al., 2020; Anantha et al., 2021; Brabant et al., 2022), alongside multiple proposed query rewriting methods (Ishii et al., 2022: Yu et al., 2020; Wu et al., 2022; Del Tredici et al., 2021; Chen et al., 2022b; Galimzhanova et al., 2023). For example, Wu et al. (2022) and Chen et al. (2022b) proposed to use reinforcement learning methods for the query rewriting. Yu et al. (2020) investigated few-shot generative models like GPT-2 for query rewriting. Galimzhanova al. (2023) studied instruction tuned GPT-3.5-T Turbo and showed that it achieved state-of-the results for conversational query rewriting.
Some previous work fine-tune a single-turn query retriever on in-domain conversational query and context pairs (Feng et al., 2020; Gao et al., 2022; Adlakha et al., 2022; Wu et al., 2023), so it can directly take a concatenation of dialog history and current query input. In this work, we focus on the zero-shot evaluation. We fine-tune a single-turn query retriever on high-quality multi-turn dataset. Then, we evaluate zero-shot capability of the fine-tuned retriever on five benehmark datasets. Surprisingly, we find this simple approach can obtain comparable zero-shot results as the state-of-the-ar query rewriting model, i.c., GPT-3,5-Turbo.
Fine-tuning Retriever for multi-turn QA
2.3 Instruction Tuning
The goal of instruction tuning is equip LLMs with the capability follow natural language
instructions (Wei et al., 2022a; Sanh et al., 2022; Mishra et al., 2022; Iyer et al., 2022; Du et al., 2022;
Ouyang et al., 2022: Wang et al., 2023; Zhang al., 2023b; Gao et al., 2023; Chung et al., 2022;
Muennighoff et al., 2022; Xu et al., 2023a; Wang et al., 2022c; Zhou et al., 2023; Albalak et al., 2024). There has been surge in the development of high-quality instruction tuning datasets, including FLAN (Chung et al., 2022), Self- Instruct (Wang et al., 2022b), unnatural Instructions (Honovich al., 2022), Dolly (Conover al., 2023b), and OpenAssistant (Köpf et al., 2023).
Although numerous research on instruction tuning has been conducted, a few works focused on improving RAG or context awareness generation for QA. Lin et al. (2023b) introduced retrieval- augmented instruction tuning method, which appends top-k retrieved chunks for LLM fine-tuning. Wang et al. (2024) applied instruction tuning after propose a two-stage instruction tuning method to improve generation with retrieval or provided context. We find that appending top-k retrieved chunks for LLM fine-tuning does help for wide range of conversation QA tasks (see Appendix $A.1 for details). Similar to the latest work (Zhang al., 2023a), we show that adding a small amount of "unanswerable" samples in instruction tuning prompts the model to respond "cannot answer" when needed, significantly reducing hallucination.
pretraining. In contrast, we
3 ChatQA
In this section, we propose a two-stage instruction tuning method for ChatQA. See Figure 1 for an illustration.. Our method starts with pretrained LLM base model. At stage- we apply supervised fine-tuning (SFT) as in Ouyang et al. (2022) on a blend of instruction-followinn and dialog datasets. After that, our model exhibits good capability to follow instructions. However, its capability for contextualized or RAG-based QA remains limited. Hence, we introduce subsequent stage, called context-enhanced instruction tuning, which is designed specifically for enhancing our model's capability for context-an aware or -uggeeeee generation in conversational QA.
3

---

Multi-Turn Queries
Contrastive Finctuning
Usera beanls anbville pereas
Query Encoder
Axsitaant No hrand they seborris 1t?
41
Extt
E4:
Egn
Corresponingg Contectr
pos
neg
It does not arply coditional edvortinng rechiiictoo Ulowecser they llei odraitakig.....
Context Eacoder
E
neg
110g
1005
Figure 2: Illustration of fine-tuning retriever for multi-turn QA.
3.1 Stage-1: Supervised Fine-tuning
To construct large and comprehensive supervised fine-tuning (SFT) dataset, follow Xu al. (2023b), Wang et al. (2024) and gather combined set of 128K SFT samples from high-quality instruction tuning datasets. It consists of 1) a social dialogue dataset Soda (Kim et al., 2022), 2) a long-form QA dataset ELI5 containing elaborate answers (Fan et al., 2019), 3) FLAN and thain-of-thought datasets (Wei al., 2022b; Chung et al., 2022; Longpre et al., 2023), 4) LLM synthetic instruction tuning datasets, including Self-Instruct (Wang et al., 2022b) and Unnatural Instructions (Honovich al., 2022), and 5) private crowd-sourced conversational dataset, as well as two public human- -writee conversation datasets: OpenAssistant (Köpf et al., 2023), and Dolly (Conover et al., 2023a).
We unify the structure of all the SFT data in a conversational format. We first add "System" role the beginning
up general instruction guiding LLM to provide polite and helpful answers. We also add "User" and "Assistant" roles to incorporate instruction and response pairs from the SFT dataset. We apply fine-tuning using this unified format on an LLM foundation model.
3.2 Stage-2:
Instruction Tuning
To further enhance the model's conversational QA capability over given context, we conduct second stage instruction tuning, which integrates contextualized QA datasets into the instruction tuning blend. One of the key elements for stage-2 is to obtain high-quality document- grounddd conversational QA dataset. We leverage two strategies of collecting such dataset. One to leverage GPT-3.5- generate a synthetic dataset (referred to as SyntheticConvQAA) Another is to collect human-annotated dataset (referred to as reliance OpenAl models. For both datasets, also collect unanswerable data samples the answers cannot be found within the given context. More details can be found in the Appendix B.
which enables us to avoid the
To boost the QA capability in handling tabular documents and arithmetic calculation, we add the TAT- QA dataset (Zhu et al., 2021) which contains both elements. In addition, we integrate contextualized single-turn QA datasets to further strengthen the QA capability of our model. We also retain the stage-1 SFT dataset in the training blend to maintain the model's cneruuctionfolllwwiii capability.
Finally, the training blend for stage-2 2 consists of: 1) A conversational QA dataset: HumanAnno tatedConvQA or SyntheticConvQA., 2) single-turn QA datasets: DROP (Dua al., 2019), Narra- tiveQA (Kočiský et al., 2018), 8), Quoref (Dasigi et al., 2019), ROPES (Lin et al., 2019), SQuAD1.1 (Ra- jpurkar et al., 2016). SQuAD2.0 (Rajpurkar et al., 2018). NewsQA (Trischler et al., 2017), TAT- QA (Zhu et al., 2021), and 3) all of SFT datasets from stage- As for the training of 1.5, we additionally incorporate HybriDial (Nakamura et al., 2022) and our collected around 2K QA pairs within the financial domain to further improve our model's capability in tabular understanding and arithmetic calculations. We follow the similar template as in stage- to unify all single-turn QA and conversational QA datasets. The differences are in parts: 1) Following the system role, we append relevant context for the single-tum question or the multi-turn conversation, and 2) Just before the single-turn question or multi-turn conversation, we integrate further instruction based the answer types of different QA datasets (e.g., short answer, long answer, arithmetic calculation). We the format for SFT dataset from stage-1.
2 Unless specified otherwise, the experiments use Amntaneonnnttt as the default setting. 'Details of the templates for both stage-1 1 and stage-2 instruction tuning as well as the synthetic data generation be found in the Appendix C.

---

Models
Average
Doc2Dial
QuAC
QReCC
TepiDCQA
INSCIT
top-1
top-5
top-1
top-5
top-5
lop-1
top-5
top-5*
top-20"
top-5
top-20
Adlakha al. (2022)
70.40
Wu et al. (2023)
71.10-
ES-amupervised (Wang et al., 2022a)
31,56
59.22
23.02
55. 35.33
43.49
77.68
44,71
$4.99
26.
37.67
20.32
40.44
ES es-aumppevised Rewrite!
33,23
61.02
25.56
58.00
46.00
80.01
45.50
85.89
27.58
39.15
21.53
42.04
ES-umupervised + Pine-tune"
47.79
75.00
45.28
80.96
46.52
80.74
53.37
89.91
41.01
51.07
52.79
72.31
Dragon' (Lin al., 2023a)
43.33
75.61
56.80
46.17
57.68
27.49
46.22
Dragon Rewite!
54.46
80.13
47.60
80.60
47.10
77.15
51.73
85.78
73.07
88.19
52.79
68.92
Deigon Fine-tune'
-2.72
80.67
48.94
83
52.64
50.73
87.17
67.
28
43.49
64.94
SyuthelicConvQA
52.98
81.15
48.64
83.47
54.75
83.23
49.63
86,70
64.48
83.24
47.41
67.13
Table I: Retrieval results across five multi-turn QA datasets with the average top-1 and top-5 recall scores. Compared to rewriting, fine-tuning performs much better on E5-unsupervised and comparable Dragon. *Since the average context length in TopiOCQA and INSCIT is smaller than in other datasets, we report top-5 and top-20 to roughly match the context lengths of top-1 and top-5, respectively, in those datasets. 'The inputs for these two models a concatenation of the dialogue history and the current query. 'The input for this model is the rewritten query. O denotes that the HumanAnnotatedde AnnotatedConvO. dataset is replaced with the SynthericConvQA for fine-tuning. The numbers are not apple-to-apple comparison (e.g., they use the training set for fine-tuning).
4 Retrieval for Multi-Turn QA
In conversational QA tasks, when a document becomes too lengthy to feed directly into LLMs, a retriever that can handle conversational queries becomes essential. This conversational retriever encodes the concatenation of the dialogue history and the current query, and then retrieve relevant context from documents. After that, only the relevant context will be used as inputs for LLMs. The state-of-the-art retrievers, e.g., Dragon (Lin et al., 2023a), are optimized for single-turn queries, resulting in limited generalization capability for multi-turn conversational queries. In Figure 2, we depict our retriever fine-tuning method to alleviate this issue. We propose to use conversational query and context pairs for further fine-tuning a single-turn query retriever to cope with multi-turn queries.
An alternative solution is conversational query rewriting method which uses query rewriter to rewrite the current question based on the conversational history. The rewritten query is then directly used as the input to single-ture tangle-turnnn query retriever for retrieving relevant context. In addition to the embedding and search cost, the query rewriting model introduces a large amount of extra computational expense to generate the rewritten query.
4.1 Fine-tuning Retriever for Multi-turn QA
To build a high-quality fine-tuning dataset, we leverage the conversational QA dataset from either or SyntheticConvQA to construct conversational query and context pairs. For the HumannAnotateer AnnotatedConvQ... we directly take the annotations of the conversational query and context pairs, and use them to further fine-tur tune single-tum query retriever. For the SyntheticConvQA. we first cut each document in the conversational QA dataset into different chunks. Then, we calculate the 4-gram recall score between agent's 's answer and each chunk. After that, we consider the chunk that has the highest recall score as the gold chunk for the current user's question. Finally, the constructed conversational query and context pairs are used fine-tune single-turn query retriever.
4.2 Conversational Query Rewriting
To build powerful conversational query rewriting model, we take GPT-3.5- -urbbo as the rewriter given that Galimzhanova et al. (2023) demonstrated the state-of-the-art query rewriting results using GPT-3.5-Turbe Similar to Galimzhanova et al. (2023), we not only provide GPT-3.5-Turbo with the rewriting task instruction, but also give it few-shot rewriting examples to enhance the quality of rewriting results. More details can be found in Appendix E.
4.3 Comparisons
In Table 1, we compare the query rewriting and fine-tuning methods across five datasets in the zero-shot setting. More details about these datasets can be found in $5.2. We conduct experiments on state-of-the-aar retriever, Dragon (Lin et al., 2023a), and strong unsupervised retriever, E5- unsupervised (Wang al., 2022a), which is not finetuned on MS MARCO (Nguyen et al., 2016). In terms of the experiments on Dragon, we find that fine-tuning performs marginally worse than query rewriting in average top-1 recall by 1.74%, while it achieves better results on average top-5 recall by 0.54%. It demonstrates the effectiveness of the fine-tuning approach for the conversational retrieval.
5

---

In addition, we observe that the results are comparable between using Aontaamdonnttteooooooo and SyntheticConvQA QA for fine-tuning. This highlights that our suman-annotated dataset is in high-quality, and we do not rely on ChatGPT models for building the stateoof-the-ae multi-turn query retriever.
Surprisingly, fine-tuning performs significantly better than rewriting on E5-unsupervised. We con- jecture that since ES-unsupervised does not use guman-annotatee query and context pairs in the pre-training stage, it leads weak generalization for the high-quality rewritten query. In contrast, using a high-quality dataset to fine-tune ES-unsupervised brings a giant boost, with more than a 15% improvement on both average top-1 and top-5 recall scores.
Therefore, fine-tuning a good single- turn retriever on high-quality conversational query context pairs performs on par with leveraging the ttate-o-flee toate-oofthe-arr the-art rewriter. However, rewriting method requires extra computational time for autoregressive generation process and probably also API cost for using powerful models like GPT-3.5-Turbo. In contrast, our proposed multi-turn fine-tuning bypasses these issues. For the QA evaluations across these five datasets, we consistently use the retrieved top-5 results from the fine-tuning approach for all the QA models. We put more results on comparisons between rewriting and fine-tuning methods in the Appendix E.2.
5 Experimental Setup
5.1 Baselines
We develop ChatQA models based on our in-house GPT-|8B, 22B} base models (pretrained with
3.5 trillion tokens), Llama2-(78 (7B, 13B, 70B] base models (Touvron et al., 2023), and Llama3-(8B, 70B} base models (Meta, 2024). We compare ChatQA models against Mlama--Chat-TT7 13B, 70B) and Llamaš-Instruct-fe
70B), which are shown to possess strong instruction following and conversational QA capabilities (Touvron et al., 2023). In addition, we compare against a powerful RAG model, Command R+, which has 104 billion parameters, as well as three very strong OpenAl models: SPT-3.5-Turbb-066
For fair comparison, when retrieval is needed, we use the same top-k retrieved chunks from our best retriever as the context for all baselines and our ChatQA models. Note that we have carefully tuned the instructions for all the baselines to ensure they achieve as good as possible results.* 4
GPT-4-0613, and GPT-4-7Turo-2222--0000
5.2 CHATRAG BENCH: Evaluation Benchmarks
To evaluate the model's capability on conversational QA and RAG, we construct CHATRAG BENCH, collection of 10 datasets covering wide range of documents and question types, which require models to generate responses from (retrieved) context, comprehend and reason over tables, conduct arithmetic calculations, and indicate when questions cannot be found within the context.
Long Document Datasets We collect five conversational QA datasets with long documents: Doc2Dial (D2D) (Feng et al., 2020); QuAC (Choi et al., 2018), QReCC (Anantha et al., 2021), TopiOCQA (TCQA) (Adlakha et al., 2022), and INSCIT (Wu et al., 2023). One can find details of these datasets in Appendix F. Since the documents of these datasets cannot be directly fitted into LLMs with a sequence length of 4K or 8K tokens. Hence, we run our best multi-turn query retriever to get top-k relevant chunks as the inputs. For Doc2Dial, QuAC, and QReCC, we segment documents into around 300-word chunks, and we retrieve top-5 relevant chunks as context for each user question. For TopiOCQA and INSCIT, we follow their original segmentation, resulting in smaller chunks. Hence, we retrieved top-20 chunks to obtain similar context length to the first three datasets (experiments can be found $4.3).
Short Document Datasets To increase the diversity of document lengths, we collect five conversa- tional QA datasets with short documents (less than 1.5K words): CoQA (Reddy et al., 2019), a dataset covering a wide range of domains; DoQA (Campos et al., 2020), a dataset covering cooking, travel and movie domains with unanswerable cases; Co (Chen et al., 2022a), a dataset based on Financial domain with tabular document and requiring arithmetic calculation and complex numerical reasoning; SQA (Pasupat & Liang, 2015); and HybriDial (HDial) (Nakamura et al., 2022). Both SQA and HibriDial have tabular documents from Wikipedia and require reasoning for complex questions."
'The prompts for these baselines can be found in Appendix F.
MMore dataset details can be found in Appendix F.

---

Avg. wio
Avg.
Models
IDDial
All
D2D
QuAC
QReCC
CoQA
DoQA
CRQA
SQA
TOQA
HDial
INSCIT
Llumm2
Hlama2-Chut-75
37.96
38.86
33.27
25.83
46.02
72.28
33.15
36.58
26.14
36.68
47.02
31.67
Llama2-Clat-708
44.64
45.21
36.87
32.47
49.40
80.41
38.97
46.85
37.62
44.31
50.35
34,88
Llama3-Insten Instrucctt
46,76
46.70
33.59
60
45.70
75.26
37.33
58.05
59.72
44.96
46.20
32.59
Llamm3-lnsttuct-700 Instruct-7BB
$2.95
52.52
37.88
36.96
51.34
76.98
41.24
76.60
69.61
49.72
48.59
36.23
Cha1QA-1.00-7
46,96
47.71
41.57
51.61
61.87
30.96
CharQA-1.0-20 1.0-00
53.89
54,14
38.90
41,82
48,05
78.57
$1.94
73.69
69.14
50.98
56.44 44
31.90
53,99
55.17
39.33
39.73
49,03
76.46
49.60
78.46
73.28
49. 96
65.76
30.10
Maama-ChattAA-L555
57.14
58.25
41.26
38.82
51,40
78.44
50.76
81.88
83.82
55.63
68.27
32.31
Command R+
51,40
50.93
33.51
34, 16
49.77
69.71
40:67
71.21
74.07
53.77
46.70
35.76
GPT-3.5. Turbboo0666
50.69
50.37
34.83
37.17
79.33
41.11
73.15
60.63
35.27
GPT-4 061-4-0633 2U7-44TTubb-22244444-9
$4.15
53.90
34.16
40.29
52.01
77.42
43.39
81.28
79.21
45.09
49.81
36.34
54.72
54.03
35.35
40.10
77.73
41.60
84.16
79.98
47.86
Table 2: Zero-shot results on CHATRAG BENCH. ChatQA-1.0 are built on Llama2 base models,
while Llama3-ChatQA-1.. are built Llama-3 base models. Note that Llama3-CharQA-1.5 used training samples from the HDial dataset. Hence, we also compare average scores excluding HDial for the fair comparison.
Models
Avg.
D2D
QuAC
QR&CC
CoQA
DoQA
CFQA
SQA
TCQA
HDial
INSCIT
ChatQA-1 1n-7QA-I.--70B
54.14
38.90
41.82
48.05
78.57
51.94
73.69
69.14
50.98
56.44
31.90
SyytheticCom-QA
54.08
39.19
38.33
48.73
79.83
48.65
76.44
68.63
51.30
55.68
33.98
stage-1"
52.18
38.43
37.52
46.08
73.51
49.42
72.15
72.08
51.28
50.74
30.56
w/o stage-2"
43.22
34.42
25.65
41.88
73.04
28.21
46.64
58.90
37.20
55.52
30.71
w/o single-turn*
52.25
38.30
37,89
47.08
76.74
46.43
72.42
67.1
49.85
53.16
33.18
w/o ContQADuta
48.97
35.47
28.19
41.68
75,80
31.68
73.83
68.05
45,90
58.40
30.70
Table 3: Ablation studies on ChatQA training. + denotes that the HlnnttAonotttedCConn
is
replaced with the SyntheticConvQA. denotes that the stage-1 (SFT) stage-2 tuning is removed from the ChatQA training. denotes that the single-turn QA datasets are removed from the stage-2 training blends. denotes that the fummanAAmottteeCCnnoQ.. is removed from the stage-2 training blends. In both "w/o stage-1" and "w/o single-turn" settings, the SyntheticConvQA data is used.
5.3 Evaluation Metrics
Given that Fl score is the most commonly used automatic metric to assess QA models, we use for all datasets except for ConvFinQA. In ConvFinQA, we follow Chen et al. (2022a) to exact match metric since the answers in ConvFinQA are about extracting numbers from documents as well as arithmetic calculations. Hence, the answer only makes sense when it is exactly the same as the answer. When models generate the arithmetic formula, we will calculate its final result based on a calculator and compare it with the gold answer. In addition, we also conduct human evaluations to assess the correctness of generated answers between our model and GPT-4-0613.
6 Results
6.1 Main Results
Overview In Table we compare different model variants and OpenAl models across 10 conver- sational QA datasets. ChatQA-1100-B/700 greatly surpass Llama2-Chir counterparts, and Llama3- ChatQA-1.5-8B/70B greatly outperform Llama3-Instruct counterparts. For example, ChatQA-1.00 1.0- 70B outperforms Llama2-Chat-70p by an average score of 8.93, and LLamaa-ChatQA--.5--1 surpasses Llama3-Instruct-700 by an average score of 4.19 (Avg. w/o HDial). In addition, ChaiQA- 1.0-70B slightly outperforms GPT-4-0613 and 4PT-4-Tubb, and greatly surpasses Command R+, powerful RAG model that has 104 billion parameters, as well as GPT-3.5-Turbo (Avg. All). Compared to of-te-of-thh-att OpenAl models (i.c., GPT-4-0613 13 and GPT-4-Turbo), Salmma-ChhtQA--La achieves comparable results, and Llama3-ChatQA-1.5577
greatly outperforms both of them.
Furthermore, we compare our models and OpenAl models across different dataset types (e.g.. documents with table or solely text) on CHATRAG BENCH. We observe that Alama33ChatQAA 70B surpasses GPT-4-0613 3 and GPT-4-Turbo in all categories. Detail results are in the Appendix A.2.
Importance of Stage-1 SFT and Stage-2 Tuning In Table 3, we find that removing the stage-1 SFT from the ChaiQA training makes the average score drops 1.9 (from 54.08 to 52.18). Aside from the SQA, removing stage-1 makes the model consistently perform worse on other datasets. Results indicate that stage-1 still plays an important role. It enhances the instruction -ostluction-followinn capability of ChatQA which is beneficial for stage-2 tuning. In addition, we find that removing ChatQA stage-2 We place full results (e.g., models based 1lama--13B and our in house 8B/22B models) in Appendix K.
7

---

Win
Loss
Figure 3: Human evaluation (A/B testing) com- paring ChatQA-1.0-70B GPT-4-0613 on CHATRAG BENCH.
ChuIQA-110-70B 13.8%
vs GPT-4-0613
69.1%
17.1%
tuning makes the average scores greatly drop by 10.92 (from 54.14 to 43.22). This is because the stage-2 tuning (i.e., context-enhanced instruction fine-tuning) enables the model to learn how to effectively leverage information from retrieved or relevant context.
Effectiveness of Single-Turn Data To investigate how single-turn QA datasets affect model's multi-turn QA capability, we conduct an ablation study by removing them from the ChatQA-1.0r 1.0-70 training blends in stage-2. As shown in Table 3, incorporating single-turn QA datasets in the stage-2 training blends generally make the scores increase across all benchmark datasets, leading to an average improvement of 1.83 score. Interestingly, we observe improvement in ConvFinQA, SQA, and HybriDial (table-based datasets), despite the added single-turn QA datasets not having tabular data in the documents. These results align with our intuitions. Adding single-turn QA datasets improves the model's 's capability to effectively leverage relevant context for the answers, resulting in better scores on CHATRAG BENCH.
Effectiveness of Conversational QA Data We further explore the how conversational QA data affect the model's multi-turn QA capability by removing HlumanAnnttttedComy QA data from the ChatQA stage-2 training blends. As illustrated in Table 3, "w/o ConvQA makes the results significantly worse than ChatQA-1.0- 1.0-0BB (average scores degrading from 54.08 to 48.97). We observe large degradation in datasets with text-based documents, such as QuAC, QReCC, and DoQA. However, the degradation in datasets with table-based documents (e.g., ConvFinQA, SQA) are small, thanks to having TAT-QA (Zhu al., 2021) in the training blends.
Human Annotated Data vs. GPT-3.5-Turbo Synthetic Data In Table 3, we also compare ChatQA models using the 7k GPT-3.5-Turbo synthetic dataset (SyntheticConvQA and our collected 7k
Aaiimaannnnotttooooooo QA). First, we find that both achieve comparable results in terms of average scores, which suggests that we do not need to rely on synthetic data from OpenAl models to build the state-of-the- of-the--rtt conversational QA models. Second, we find that using human-annotate data achieved significant improvements on QuAC and DoQA datasets. This can be attributed the fact that the human-annotated data have higher quality unanswerable cases which exists in QuAC and DoQA datasets. Eventually, it leads to the overall improvements on these two datasets. Detail results and analyses on unanswerable cases can be found in $6.2.
dataset
Human Evaluation Despite Fl scores being the most commonly used metrics for evaluating the quality of QA models, there are often multiple ways to answer questions, which makes the automatic metrics less than perfect. Therefore, ChatQA-1.0-700 with GPT-4-0613. In this human evaluation, we ask annotators to verify the facts in ChatQA-1.0-700 and GPT-4's outputs and determine which model provides more accurate response to the question. The human evaluation results averaged over the 10 datasets in CHATRAG BENCH shown in Figure We find that our CharQA-1.0-77
use human evaluations to further compare
and GPT-4 are tie most of the time (69.1% and GPT-4 achieves slightly higher win rate (3.3%) than ours. This further confirms our model has powerful capability produce correct answers."
Additional Ablation Studies & Case Studies We conduct ablation studies on ChatQA training in terms of using top-k chunks as context for stage-2 tuning (See Appendix A. Furthermore, we also study how different factors affect the model's generation quality on the inference stage, in terms of: 1) the number of retrieved context/chunks: 2) context ordering; and 3) different retrievers (See Appendix A.3). In addition to quantitative results, perform detailed qualitative case studies, and compare the output ChaiQA-L.0-705 GPT-3.5-Turb--0613. and GPT-4-0613 in Appendix I.
6.2 Evaluation of Unanswerable Case
Evaluation Setup In this section, we study another aspect of the model's capability, which is to discern if question can be answered within the provided context. Generating an answer in unanswerable will lead to hallucination. To allow this evaluation, require the model to indicate it when no answer can be found in the given context.
Detailed human evaluation setup and full results can be found in the Appendix G.

---

Models
Avg
Avg.
QsAC
QuAC
Avg
DoQA
DoQA (yes*)
ChatRAG
Both
QuAC
(no*)
(yes*)
DoQA
(no*)
1L.-QA-L.0-70B
77.25
80.76
77.66
83.85
73.74
68.81
78.67
54.14
remarkably outperforms existing Command R+
68.11
69.61
41.79
97.42
66.62
46.37
86.87
50,93
13
73.27
78.34
61.91
94.76
68.21
51.99
84.43
50.37
GPT-4-0613
80.73
87.42
83.45
91.38
74.05
74,28
73.82
53,90
G17744Tuubo-2224000-99
80.47
88.73
80.42
97.03
72.21
72.28
72.13
54.03
Table 4: Accuracies on answerable and unanswerable samples across QuAC and DoQA datasets. Avg-Both is the averaged score between QuAC and DoQA. ChatRAG is the average score on the CHATRAG BENCH. * "no" and "yes" denote unanswerable and answerable samples, respectively.
Models
Average
NQ
TriviaQA
HospoiQA
Atlas (1IB) (Izacard et al., 2023) 2023)
39.4
26.7
56.9
34.7
Raven (11B) (Huang et al., 2023)
29.6
65.7
RECOMP (20B) (Xu al., 2024)
42.1
37.0
59.0
30.4
InstructRetro (43B) (Wang et al., 2024)
38.9
65.6
RePlug (65B) (Shi et al., 2023)
44,5
28.8
72.6
32,0
RA-DIT (65B) (Lin et al., 2024)
50.1
35.2
75.4
39.7
Llamuö-Instruct-8B (Meta, 2024)
42.5
30.9
70.7
26.0
Llama3-Instruct-700 Instruct- 70B (Meta, 2024)
53.6
42.7
82.4
35.6
Llama3- Saana--ChhtAA--5-88
52.3
42.4
81.0
33.5
Llama--hatQAA1155-00
58.7
47.0
85.6
42.2
Table 5: Zero-shot exact match scores Natural Questions (NQ), TriviaQA, and HotpotQA, which were evaluated using the data split from the KILT Benchmark (Petroni et al., 2021).
We use QuAC and DoQA datasets which have such unanswerable cases to evaluate such capability. Specifically, for unanswerable case, we consider the model indicating that the question cannot be answered as correct', and as for answerable cases, we consider the model not indicating the question unanswerable as correct (i.e., the model giving an answer). Note that for answerable cases, we only select the samples where correct context is retrieved. In the end, we calculate the average accuracy score of unanswerable and answerable cases as the final metric. We consider this average accuracy as the evaluation metric since it is in the same spirit of F1 metric which measures the harmonic mean of precision and recall scores.
Results In Table 4, we compare our models with OpenAl models across QuAC and DoQA datasets. We observe that OpenAl models show powerful capability in this task, especially for GPT-4-0613. Compared them, our best model (ChatQA-1a 1. 1.0-70B) achieved significantly better average accuracy than GPT-3.5- T.T-.,-TTrr,,, while we still has a slight gap compared to GPT-4 and GPT-4-Turbo (around 3.5% and 3.2%, respectively). Furthermore, we conduct ablation studies in terms of the number of unanswerable samples for training. We find that using small amount of unanswerable samples (e.g., 1.5k) able to achieve remarkable results on the unanswerable evaluation and incorporating more unanswerable samples does not necessarily lead to higher accuracy scores in the unanswerable evaluation (Detail results are in the Appendix H.2).
6.3 Eyaluation on Single-Turn QA and RAG Benchmark
In addition to CHATRAG BENCH, we further evaluate Llama3- ChaiQA-115 models on knowledge- intensive iugle-turn QA datasets: Natural Questions (NQ) (Kwiatkowski et al., 2019), Trivi- aQA (Joshi et al., 2017), and HotpotQA (Yang al., 2018), and compare them against frontier RAG models. We use the Dragon retriever (Lin et al., 2023a) to extract the top-k most relevant contexts, which are then used as the inputs for Llama3-Instruct and Llama3-ChatQA-I.. We report the best results obtained from using the t5p-15, top-20, and top-25 contexts. In Table 5, we show that, despite its significantly smaller model size, Llama3-ChatQA-11 --a.-------a----1- performs better than the state-of-the-art RA-DIT (65B) model. frontier RAG models.
7 Conclusion
In this paper, we build a family of models that surpass GPT-4 on conversational QA and RAG. We introduce CHATRAG BENCH, collection of 10 conversational QA datasets covering comprchensive
'e collect a set of heuristic matching patterns from all generated samples to determine if the model suggests that the question is unanswerable. More details can be found in the Appendix H.

---

evaluations on RAG, table-based reasoning, arithmetic calculations, and unanswerable scenarios. Our CharQA-L.0-70B model built on Llama2 can slightly outperform GPT-4-0613 and GPT-4- Turbo without using any synthetic data from OpenAl GPT models. Remarkably, Llama3-ChatQn 1.5-70B even surpasses GPT-4-Turbo in all categories of CHATRAG BENCH. In addition, we demonstrate that fine-tuning single-turn query retriever using our curated conversational QA data performs comparably to the state-of-the-ar LLM-based query rewriting model, without incurring extra computational time and potential API costs associated with rewriting. Furthermore, we show that incorporating a small amount of "unanswerable" samples can significantly enhance our model's capability to handle scenarios where answers unavailable.
Impact Statement
In this section, we discuss potential positive and negative social impacts that could arise from our ChatQA models.
Positive Impacts First, ChatQA enables users to interact with their data, including documents and tables, and provides accurate responses their questions. This enhances productivity and efficient learning. Second, ChatQA can improve customer support services by quickly retrieving information about products, services, and common issues, allowing customer queries to be addressed more efficiently. Third, ChatQA can assist professionals, such as those in the medical field, by gathering relevant information and improving
Potential Negative Impacts First, although our CharQA models demonstrate impressive accuracy in generating answers, they still have the potential to provide factually inaccurate responses. Second, LLMs can inadvertently learn and perpetuate biases present in their pre-training data, which may lead to biased responses in our QA models as they are further fine-tuned based on these LLMs. Third, there is risk of malicious use of our ChatQA models, such gathering personal information by inputting documents containing relevant personal content.
Limitation
ChatQA models optimized for RAG and conversational QA across various types of questions and documents, making them very useful in many real-world applications. However, they may not perform well on code-related tasks or math reasoning tasks compared to GPT-4 and other versatile frontier models, because ChatQA's instruction-tuninn blend does not include any code generation data. In the future, we plan include more code-related and math reasoning data to support potential use cases.
References
Adlakha, V., Dhuliawala, S., Suleman, K., de Vries, H., and Reddy, S. Topiocqa: Open-domain conversational question answering with topic switching. TACL, 2022.
Albalak, A., Elazar, Y., Xie, S. M., Longpre, S., Lambert, N., Wang, X., Muennighoff, N., Hou, B., Pan, L., Jeong, H. et al. A survey data selection for language models. arXiv preprint arXiv:2402. 16827, 2024.
Aliannejadi, M., Kiseleva, J., Chuklin, A., Dalton, J., and Burtsev, M. Building and evaluating open-domain dialogue corpora with elarifying questions. In EMNLP. 2021.
Anantha, R., Vakulenko, S., Tu, Z., Longpre, S., Pulman, S., and Chappidi, S. Open-domain question answering goes conversational via question rewriting. In NAACL, 2021.
Anthropic. Introducing 100k context windows, 2023a.
Anthropic. Introducing Claude, 2023b.
Brabant, Q. Lecorvé, G., and Barahona, I. M. R. Coqar: Question rewriting on coqa. LREC, 2022.
10

---

Campos, J. A., Otegi, A., Soroa, A., Deriu, J. M.. Cieliebak, M., and Agirre, E. accassccessing domain-speciffs faqs via conversational qa. ACL, 2020.
Chen, Z., Li, S., Smiley, C., Ma, Z., Shah, S., and Wang. W. Y. Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering. In EMNLP, 2022a.
Chen, Z., Zhao, J., Fang, A., Fetahu, B., Rokhlenko, O., and Malmasi, S. Reinforced question rewriting for conversational question answering. In EMNLP, 2022b.
Choi, E., He, H., lyyer, M., Yatskar, M., Yih, W.-t,, Choi, Y., Liang, P., and Zettlemoyer, L. Quac: Question answering in context. In EMNLP, 2018.
Chu, Z., Chen, M., Chen, J., Wang, M., Gimpel, K., Faruqui, M., and Si, X. How ask better questions? a large-scale multi-domain dataset for rewriting ill-formed questions. In AAAI, 2020.
Chung. H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M.,
Brahma, S., Webson, A., Gu, S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Castro-Ros, A., Pellat, M., Robinson, K., Valter, D., Narang, S., Mishra, G., Yu, A., Zhao, V., Huang, Y., Dai, A., Yu, H., Petrov, S., Chi, E. H., Dean, J., Devlin, J., Roberts, A., Zhou, D., Le, Q. V., and Wei, J. language models. arXiv preprini arXiv: 2210. 1416, 2022.
Scaling instruction-fineturee
Conover, M., Hayes, M., Mathur, A., Meng, X., Xie, J., Wan, J., Shah, S., Ghodsi, A., Wendell, P., Zaharia, M., et al. Free dolly: Introducing the world's first truly open instruction-tume tumee 1lm, 2023a.
Conover, M., Hayes, M., Mathur, A., Xie, J., Wan, J., Shah, S., Ghodsi, A., Wendell, P., Zaharia, M., and Xin, R. Free Dolly: Introducing the world's 's first truly open instruction-tued Iim, 2023b.
Dai. Z., Chaganty, A. T., Zhao, V., Amini, Green, M., Rashid, Q., and Guu, K. Dialog inpainting: Turning documents to dialogs. In ICML, 2022.
Dasigi, P., Liu, N. F. Marasović, A., Smith, N. A., and Gardner, M. Quoref: A reading comprehension dataset with questions requiring coreferential reasoning. In EMNLP, 2019.
Del Tredici, M., Barlacchi, G., Shen, X., Cheng, W., and de Gispert, A. Question rewriting for open-domain conversational qa: Best practices and limitations. In CIKM, 2021.
Deng, Y., Lei, W., Zhang, W., Lam, W., and Chua, T.-S. Pacific: Towards proactive conversational question answering over tabular and textual data in finance. In EMNLP, 2022.
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, 7., and Tang, J. Glm: General language model pretraining with autoregressive blank infilling. In ACI., 2022.
Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., and Gardner, M. Drop: reading compre- hension benchmark requiring discrete reasoning over paragraphs. In NAACL, 2019,
Elgohary, A., Peskov, D., and Boyd-Graber, J. Can you unpack that? learning to rewrite questions-in- context. In EMNLP, 2019.
Fan, A., Jernite, Y., Perez, E., Grangier, D., Weston, J., and Auli, M. Eli5: Long form question answering. In ACL, 2019.
Feng, S., Wan, H., Gunasckara, C., Patel, S., Joshi, S., and Lastras, L. doc2dial: A goal-oriented document-groumm grounded dialogue dataset. EMNLP, 2020.
Galimzhanova, E., Muntean, C. I., Nardini, F. M., Perego, R., and Rocchietti, G. Rewriting conversa- tional utterances with instructed large language models. In IEEE/WIC International Conference on Web Intelligence and Intelligent Agent Technology, 2023.
Gao, C., Zhang, W., and Lam, W. Unigdd: A unified generative framework for goal-oriented document- grounded dialogue. In ACL, 2022.
Gao, Y., Xiong. Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., and Wang, H. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312. 10997, 2023.
Google. Introducing bard, 2023.

---

Guo, M., Zhang, M., Reddy, S., and Alikhani, M. Abg-coqa: Clarifying ambiguity in conversational question answering. In AKBC, 2021.
Honovich, O., Scialom, T., Levy, O., and Schick, T. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:2212.09660 2222222....999, 2022.
Huang, J., Ping, W., Xu, P., Shoeybi, M., Chang, K. C.-C., and Catanzaro, B. Raven: In-context learn- ing with retrieval augmented ancderrdeccode language models. arXiv preprint (rXiv:2308.07922 2023.
Ishii, E., Xu, Y., Cahyawijaya, S., and Wilie, Can question rewriting help conversational question answering? In Proceedings of the Third Workshop Insights from Negalive Results in NLP, 2022.
Iyer, S., Lin, X. V., Pasunuru. R., Mihaylov, T., Simig. D.. Yu. P., Shuster, K., Wang, T., Liu, Q.. Koura, P. S., et al. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint 12Xiv:2212. 12017 2022.
Izacard, G. and Grave, É. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, 2021.
Izacard, G., Caron, M., Hosseini, L., Riedel, S., Bojanowski, Joulin, A., and Grave, E. Unsuper- vised dense information retrieval with contrastive learning. Transactions on Machine Learning Research, 2022.
Izacard, G., Lewis, P., Lomeli, M., Hosseini, L.,, Petroni, F., Schick, T., Dwivedi-Yu, J., Joulin, A., Riedel, S., and Grave, E. Atlas: Few-shot leamnnn with retrieval augmented language models. Journal of Machine Learning Research, 2023.
Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. ACL, 2017.
Kim, H., Hessel, J., Jiang, L., Lu, X., Yu, Y., Zhou, P., Bras, R. L., Alikhani, M.. Kim, G., Sap, M., et al. Soda: Million- scalo dialogue distillation with social commonsense contextualization, arXiv preprint 1..i::22/2.106666 2022.
Kočiský, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, M., Melis, G., and Grefenstette, E. The narrativeqa reading comprehension challenge. TACL, 2018.
Köpf. A., Kilcher, Y., von Rütte, D., Anagnostidis, S., Tam, Z.-R., Stevens, K., Barhoum, A., Duc, N. M., Stanley, O., Nagyfi, R., et al. Openassistant conversations democratizinn large language model alignment. arXiv preprint arXiv:2330..03327, 2023.
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., al. Natural questions: benchmark for question answering research. TACL, 2019.
Köpf, A., Kilcher, Y., von Rütte, D., Anagnostidis, S., Tam, Z.-.., Stevens, K., Barhoum, A., Duc, N. M., Stanley. O., Nagyfi, R., ES, S., Suri. S., Glushkov, D., Dantuluri, A., Maguire, A.. Schuhmann, C., Nguyen, H., and Mattick, A. Openassistant conversations democratizing large language model alignment. arXiv preprint arXiv: 2304.07327, 2023.
Lin, K., Tafjord, O., Clark, P., and Gardner, M. Reasoning over paragraph effects in situations. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, 2019.
Lin, S.-C., Asai, A., Li, M., Oguz, B., Lin, J., Mehdad, Y., Yih, W.-t,, and Chen, How to train your dragon: Diverse augmentation towards generalizable dense retrieval. arXiv preprint 002.v:2302.07455. 2023a.
Lin, X. V., Chen, X., Chen, M., Shi, W., Lomeli, M., James, R., Rodriguez, P., Kahn, J., Szil- vasy, G., Lewis, M., et al. Ra-dit: Retrieval-augmente dual instruction tuning. arXiv preprint arXiv:2310.01352, 2023b.
12

---

Lin, X. V. Chen, X., Chen, M., Shi, W., Lomeli, M., James, R., Rodriguez, P., Kahn, J., Szilvasy, G., Lewis, M., Zettlemoyer, L., and Yih, W. RA-DIT: Retrieval auggeette dual instruction tuning. In ICLR, 2024. URL. htthtts:://ppereeiieeeee net/forumm2dd220Tuuuin
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03 03722 2023.
Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W. Tay, Y., Zhou, D., Le. Q. V., Zoph, B., Wei, J., et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301. 10..0...1...,..1 2023.
Mele, I,, Muntean, C. I. Nardini, F. M., Perego, R., Tonellotto, N., and Frieder. O. Adaptive utterance rewriting for conversational search. Information Processing & Management, 2021.
Meta. Introducing meta llama 3: The most capable openly available Ilm to date, 2024.
Mishra, S., Khashabi, D., Baral, C., and Hajishirzi, H. Cross-task generalization via natural language crowdsourcing instructions. In ACL, 2022.
Mo, F., Mao, K., Zhu, Y., Wu, Y., Huang. K., and Nie, J.-Y. Convggr: Generative query reformulation for conversational search. arXiv preprint arXiv:2305. 2023.
Muennighoff, N., Wang. T., Sutawika, L., Roberts, A., Biderman, S., Scao, T. L., Bari, M. S., Shen, S., Yong, Z.-X., Schoelkopf, H., al. Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01788 2022.
Nakamura, K., Levy. S., Tuan, Y.-L., Chen, W. and Wang, W. Y. Hybridialogue: An information- seeking dialogue dataset grounded tabular and textual data, In Findings of ACL, 2022.
Nguyen, T., Rosenberg, M., Song, X., Gao, J., Tiwary, S.,, Majumder, R., and Deng, L.. Ms marco: A human generated machine reading comprehension dataset. choice, 2016
OpenAl. Introducing ChatGPT, 2022.
OpenAl. GPT-4, 2023.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. NeurlPS, 2022.
Pasupat, P. and Liang, P. Compositional semantic parsing on semi-structured tables. In ACL, 2015.
Petroni, F., Piktus, A., Fan, A., Lewis, P., Yazdani, M., De Cao, N., Thome, J., Jernite, Y., Karpukhin, V., Maillard, J., et al. Kilt: benchmark for knowledge intensive language tasks. In NAACL, 2021.
Qu, C., Yang, L., Chen, C., Qiu, M., Croft, W. B., and Iyyer, M. Open-retrieval conversational question answering. In SIGIR, 2020.
Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. Squad: 100,000+ questions for machine comprehension of text. In EMNLP, 2016.
Rajpurkar, P., Jia, R., and Liang, P. Know what you don't 't know: Unanswerable questions for squad. In ACL, 2018.
Raposo, G., Ribeiro, R., Martins, B., and Coheur, L. Question rewriting? assessing importance for conversational question answering. In ECIR, 2022.
Reddy, S., Chen, D., and Manning, C. D. Coqa: A conversational question answering challenge. TACL, 2019.
Saeidi, M., Bartolo, M., Lewis, P., Singh, S., Rocktäschel, T., Sheldon, M., Bouchard, G., and Riedel, S. Interpretation of natural language rules in conversational machine reading. In EMNLP. 2018.
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. I.., Raja, A., et al. Multitask prompted training enables zero-shot task generalization, In ICLR, 2022.
13

---

Shi, W., Min, S., Yasumaga, M., Seo, M., James, R., Lewis, M., Zettlemoyer, L., and Yih, W.-t. Replug: betrieval-augmentee black-box language models. arXiv preprint arXiv:2301.1265. 2023.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv: 09288, 2023.
Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., and Suleman, Newsqa: machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, 2017.
Vakulenko, S., Longpre, S., Tu, Z.., and Anantha, R. Question rewriting for conversational question answering. In WSDM, 2021a.
Vakulenko, S., Voskarides, N., Tu, Z., and Longpre, S. A comparison of question rewriting methods for conversational passage retrieval. In ECIR, 2021b.
Wang. B., Ping, W., McAfee, L.,, Xu, P., Li, B., Shoeybi, M., and Catanzaro, B. Instructretro: Instruction tuning post retrieval-augmenter pretraining. In ICML, 2024.
Wang, I.,, Yang, N., Huang, X., Jiao, B., Yang, L., Jiang, D., Majumder, R., and Wei, F. Text embeddings by weakly-supervise contrastive pre-training. arXiv preprint arXiv:2212.03533, 2022a.
Wang. Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.1 10560, 2022b.
Wang. Y., Mishra, S., Alipoormolabashi, P., Kordi, Y.. Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran, A. S., Arunkumar, A., Stap, D., et al. Super-naturalintrrutiooss Generaliza- tion via declarative instructions on 1600+ nlp tasks. In EMNLP, 2022c.
Wang, Y., Ivison, H., Dasigi, P., Hessel, J., Khot, T., Chandu, K. R., Wadden, D., MacMillan, K., Smith, N. A., Beltagy, I., et al. How far can camels go? exploring the state of instruction tuning on open resources. arXiv preprint arXiv:2306.0473.
2023.
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. In ICLR, 2022a.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E.,, Le, Q. V. Zhou, D., al. Chain-of- thought prompting elicits reasoning in large language models. NeurIPS, 2022b.
Wu, Z., Luan, Y., Rashkin, H., Reitter, D., Hajishirzi, H., Ostendorf, M., and Tomar, G. S. Conqrr: Conversational query rewriting for retrieval with reinforcement learning. EMNLP, 2022.
Wu, Z., Parish, R., Cheng, H., Min, S., Ammanabrolu, P., Ostendorf, M., and Hajishirzi, H. Inscit: Information-seeking conversations with mixed-initiative interactions. TACL, 2023.
Xu, C., Sun, Q. Zheng, K., Geng, X., Zhao, P., Feng. J., Tao, C., and Jiang, D. Wizardlm: Empow- ering large language models to follow complex instructions. arXiv preprint arXiv:2304.1222 12244, 2023a.
Xu, F., Shi, W., and Choi, E. RECOMP: Improving retrieval-auummented LMs with context compres- sion and selective augmentation. In /CLR, 2024. URL hRLLhttpp://oppeeevveee wtw.://wwwweeeooumiiiieee m1JLVigNHHp
Xu, P., Ping, W., Wu, X., McAfee, L., Zhu, C., Liu, Z., Subramanian, S., Bakhturina, E., Shoeybi, M., and Catanzaro, B. Retrieval meets long context large language models. arXiv preprint arXiv:2310.03023 2023b.
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., and Manning, C. D. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In EMNLP, 2018
Ye, F., Fang, M., Li, S., and Yilmaz, Enhancing conversational search: Large language model-aided informative query rewriting. In EMNLP, 2023.
14

---

Yu, S., Liu, J., Yang, J., Xiong, C., Bennett, P., Gao, J., and Liu, Z. Few-shot generative conversational query rewriting. In SIGIR, 2020.
Zhang. H., Diao, S., Lin, Y., Fung, Y. R., Lian, Q. Wang. X., Chen, Y., Ji, H., and Zhang, R-tuning: Teaching large language models to refuse unknown questions. arXiv preprint 2.0007, 2023a.
Zhang, S., Dong, L., Li, X., Zhang, S., Sun, X., Wang. S., Li, Hu, R., Zhang, T., Wu, F., et al. Instruction tuning for large language models: A survey. arXiv preprint 2rX.v:2388..07933 2023b.
Zhou, C.,, Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., al. Lima: Less is more for alignment. arXiv preprint arXiv:2305. /1206, 2023.
Zhu, F., Lei. W., Huang, Y., Wang. C., Zhang, S., Lv. J., Feng, F., and Chua, T.-S. Tat-qa: A question answering benchmark on hybrid of tabular and textual content in finance. ACI. 2021.
15

---

A Additional Results & Analyses
A.1
Using Top-k Chunks as Context for Stage-2 Instruction Tuning
Models
ChatRAG
Avg-ret
Avg-nonret
54.14
42.33
65.96
w/ "top-5" chunks
54.04
42,91
65.16
Table 6: Ablation study using "top-5" retrieved chunk the context for the stage-2 2 instruction tuning. We report average scores on CHATRAG BENCH, five retrieval datasets (Avg-ret) and five non-retrieval datasets (Avg-nonret).
For all the datasets used in stage-2 tuning. the context is provided continuous paragraphs docu- ments containing the answer. In contrast, the model needs to handle the top-k retrieved chunks during inference for long documents. To address this train/test mismatch, we investigate whether replacing some continuous paragraphs with retrieved top-k chunks will enhance the model's 's robustness.
We use NarrativeQA for this study, since each question has a corresponding long document. Initially, we use a summary of the long document as the context, which contains the answer. To incorporate discontinuous contexts, we first cut the long document into 300-word chunks. Then, we use the Dragon retriever to retrieve the top-4 chunks related to the question as additional context. Finally, consider the retrieved four chunks along with the summary of the long document the "top-5" chunks. We use this reconstructed NarrativeQA to replace the original one for the stage-2 instruction tuning.
In Table 6, we observe that using "top-5" chunks as the context for training leads to improvements on the datasets that require retrieval. But it degrades the performance on non-retrieval datasets. Overall, these two models perform comparable. is because incorporating "top-5" retrieved chunks in stage-2 tuning aligns with the inference stage where retrieval is needed, which improves the Avg-ret score. However, mixing continuous and discontinuous documents could make the stage-2 tuning less stable, leading to sub-optimal results non-retrieval datasets. We believe more future work can be conducted regarding the balance of incorporating continuous context and top-k retrieved chunks in 2tage-2 tuning.
A.2 Fine-grained Analyses
Avg-table w/o HDial
Models
Avg-lext
Avg-table
Avg-ret
Avg-nonrel
ChaiQA-L.0-700
48.88
66.42
71.42
42.33
65,96
47.74
72.50
75.87
41.63
68.71
Llama3 Samma-ChaiQAA-L5--PP
49.79
77.99
82.85
43.89
72.62
GPT-3.5-Turbo-006
46.07
60.40
66.89
40.41
60.33
GPT-4-0613
46.96
70.10
80.25
41.58
66.22
GPT-4-Turbo- 04TT444Trboo2224-04444 2024-04-09
46.90
70.67
82.07
41.80
66.27
Table Fine-grained studies on average scores of different dataset types. Avg-text covers datasets
where the documents only have text, including Doc2Dial, QuAC, QReCC, CoQA, DoQA, TopiOCQA, and INSCIT. Avg-table covers datasets with table the documents, including ConvFinQA, SQA, and HybriDial. Avg-ret covers datasets with long documents requiring retrieval, including Doc2Dial, QuAC, QReCC, TopiOCQA, and INSCIT. Avg-nonret covers datasets with short documents which do not require retrieval, including CoQA, DoQA, ConvFinQA, SQA, and HybriDial.
In Table 7, further compared models and OpenAl models across different dataset types in conversational QA benchmarks. ChatQA-1.0- 1.0-70B achieves hetter results in text-only documents (Avg- text), while GPT-4-0613 shows better QA capability in tabular data given (Avg-table). Regarding
Note that, we do not directly use top-5 retrieved chunks for training. because they may not contain the answer. In such cases, fine-tuning the model generate answer could encourage hallucination.
16

---

the datasets that require or do not require retrieval (Avg-ret and von-nonee,), ChatQA-1.0-7000 and GPT-4-0613 are comparable.
Furthermore,
surpasses GPT-4-0613 and GPT-4-Turbo all categories.
A.3 Ablation Studies on Inference Stage
Models
Avg.
Doc2Dial
QuAC
QReCC
TopiOCQQ
INSCIT
ChuiQA-1.0-70B
42.31
39.19
38.33
48.73
51.30
33.98
# of ctx: top-3
41.91
37.20
38.35
48.94
52.78
32.27
# of ctx: top- 10
40.71
37.06
36.95
47.61
49,40
32.53
reverse ordering
42.48
39.08
38.85
49.63
51.16
33.69
swing ordering
42.30
39.35
38.09
49.09
50.98
33.99
random ordering
42.01
39.32
38,28
48.79
50.13
33.51
Dragon Retrieval 40.50
37.92
38.44
47.88
50.39
27.87
Table 8: Ablation studies on input context across datasets that require retrieval. All models use SyntheticConvQA. We study the number of contexts used in inputs (# of ctx), context ordering (reverse, swing, random), and the of retrieved context from the original Dragon. In comparison, ChatQA-1.06 1.0-70B (default setting) uses "Dragon + Fine-tune" to retrieve the top-5 contexts, and arranges them sequentially from the first the fifth context in top-5.
In Table 8, we show ablation studies on how the number of retrieved context/chunks, context ordering, and different retrievers affect the conversational QA and RAG results.
First, we find that using more contexts as inputs do not always improve the results. Utilizing top-5 contexts as input yields better results compared using either top-3 or top-10 contexts. Intuitively, more contexts have higher probability to contain correct answers (better recall scores). As result, using top-5 contexts achieves better results than using top-3. However, as the number of contexts further increases, the model may suffer from "lost in the middle" phenomenon (Liu et al., 2023) and the difficulty of extracting answers from the provided context could also increase, which leads to inferior results by using top-10 10 contexts.
Second, we study how using different orderings of top-5 contexts affects the results. We compare sequential ordering (from 1st context to 5th context) to reverse ordering (from 5th to 1st context), swing ordering (given the "lost in the middle" phenomenon, we arrange the most relevant context to appear at the beginning and the end of the input context. Hence, the ordering becomes [Ist, 3rd, 5th, 4th, 2nd]), and random ordering (random shuffle the top-5 contexts). We find using sequential ordering is comparable to using reverse and swing orderings, and random shuffling is slightly worse. Results indicate that our model excels extracting the correct answer from lengthy contexts, regardless of the answer's location. This is because, during the ChatQA fine-tuning, the answer's location is randomly distributed within the context.
Third, we observe that when we replace "Dragon + Fine-tune" with the original non-finetuned Dragon retriever, the average score drops by 1.81 (from 42.31 40.50). In addition, the score drops significantly in INSCIT dataset (from 33.98 to 27.87) due to the large performance gap between the two retrievers (as shown in Table 1). Basically, when the quality of retrieval improves, it directly enhances the performance of question answering.
17

---

B Conversational QA Data Collection for Stage-2 Instruction Tuning
B.1 Human Annotated Data
In addition to the public available datasets, one of the key elements for stage-2 is to obtain high- quality greument-grounddee conversational QA dataset. We create human--nnotatle conversational QA (called we first collected 7k documents covering diverse topics from the Internet. Then, we instruct the annotator to act both as curious user asking questions (and follow-up questions) about the document and as an agent to give the answers. We create multi-tune conversation for each document, resulting in total of 7k conversational QA dialogues with an average of 5 user-agent turns per dialogue. Details of data collection guideline can be found in the Appendix J.2.
dataset only consisting of 7k dialogues. To build this dataset,
To reduce hallucinated answers in unanswerable cases, we aim to empower our model to explicitly indicate when the answer cannot be found within the given context. To obtain these unanswerable data samples, we requested annotators to identify all parts of the context locations that are relevant to the user's question. Hence, it enabled to construct unanswerable scenarios by deleting the text from the corresponding locations in the context. After deleting the relevant text to the question, we use a sentence, "Sorry. I cannot find the answer based on the context", as the response for the unanswerable question. Finally, we construct another 1.5k user-agent turns with unanswerable annotations, which provides a good trade-off for answerable and unanswerable cases. Detailed results can be found in the Appendix H.2.
B.2 Synthetic Data Generation
To validate the quality of the HumanAAnnosateoCCooAA we leverage GPT-3.5-Turbo to generate synthetic conversational QA dataset given its powerful instruction-follovii and text generation capability. Note that large-scale synthetic data for conversational QA has also been explored in Dai et al. (2022). In this work, we focus on midsize high- sigh-qualtty synthetic data for LLM fine-tuning, One can find the details of synthetic data generation at Appedix
Similar to the able annotations in this synthetic dataset. Since there agent's ' answer, we construct synthetic unanswerable samples from SyntheticConvQA. Specifically, we first cut the document (for each dialogue) into different chunks. Then, we consider unanswerable sample only when there are chunks that have "high overlaps" with the agent's 's answer be removed and the rest of the chunks show "low overlaps" with the agent's 's answer. We use the 4- gram recall score between each chunk and the agent's s answer (which measures the ratios of the answer's 4 4-gram phrases are within each chunk) as the metric to measure their overlaps, and consider it higher than 0.5 as "high overlaps" and lower than 0.1 as "low overlaps".
we construct another 1.5k user-agent turns with unanswer- annotations of the context location for the
valid
18

---

C ChatQA Instruction Tuning Templates
C.1
Stage-1: Supervised Fine-tuning
The format template of LLM inputs in stage-1 as follows:
System:
This is a chat between a user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions based on the context. The assistant should also indicate when the answer cannot be found the context.
User: (Question 1}
Assistant: (Answer 1}
User: (Latest Question}
Assistant:
We use the (Latest Answer) from Assistant as the supervision for model outputs.
C.2 Stage-2: Context-Enhanced Instruction Tuning
Based the stage-1 format template, the LLM inputs in stage-2 adds (Context for Latest Question} and (Instruction) from User, as depicted below:
System: This is chat between user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions based on the context. The assistant should also indicate when the answer cannot found in the context.
{Context for Latest Question}
User: (Instruction] + (Question 1}
Assistant: {Answer 1}
User:
(Latest Question}
Assistant:
Same Stage-1, the (Latest Answer) from Assistant the supervision for model outputs. As for the (Instruction). we use different instructions for different datasets based on the answer types. Details can be found below:
"Please give full and complete answer for the question. This is for datasets with long answers. We use it for the HumanAnmotatcdConnQAA or SymtheticConvQA.
"Answer the following question with short span. The answer needs to be just in a few words. " This is for datasets with short answers. We use it for SQUADI.1, SQuAD2.0, NarrativeQA, DROP, ROPES, NewsQA, and Quoref.
"Answer the following question with number from context or the math arithmeti using -, or /." This is for datasets using arithmetic calculation or extracting numbers from the context. use for the TAT-QA dataset where questions need the arithmetic calculation extracting numbers from the context.
19

---

"Answer the following question with short span, or a full and complete answer. This for datasets with both short and long answers. The model needs tell whether to generate short long answers based on the questions. We use it for the TAT-QA dataset where questions do not need the arithmetic calculation, since we observe the lengths of the corresponding answers can be both short and long.
For the stage-1 SFT data used in stage-2 instruction tuning, keep the same format template in stage-1. In other words, the {Context for Latest Question} and (Instruction} are empty.
D Synthetic Data Generation
The instruction for Gur-o.5-Turbo comprises three parts: 1) system role guiding the model to provide helpful answers, 2) examples of conversational QA indicating the required data types, and 3) document that directs the model to generate conversational QA based on its content. We collect 7k documents (average ~k words per document) from common crawl, which cover wide range of domains. Each document is used for generation of single conversational QA sample, which leads to total of 7k multi-turn QA dialogues with average of 4.4 user-agent turns per dialogue (called SyntheticConvQAN
We use two-step prompting on GPT-3.5-Turbo for obtaining high-quality multi-turn QA conversation. In step-1, we prompt GPT-3.5-Turbo to generate multi-turn QA conversation based given document. However, find that the generated conversation lacks pronouns or common nouns to refer back to words or phrases in dialogue history. Hence, in step-2, we prompt GPT-3.5- Turbo to revise the generated conversation by adding pronouns common nouns. For both stages, we give GPT-3,5-Turbo few-shot examples to ensure a high-quality generation.
The in-context few-shot prompt in step-1 is as follows (aims to generate multi-turn QA conversation):
You are a helpful, pattern-followine assistant.
role-user:
PersonA has access to information the following passage: ###There are certain important aspects of college preparation. The student preparing for college should define their goals and interests. The student should also plan financially and academically. We have checklist that helps students get ready financially and academically. ###
Generate conversation of around four turns between PersonA and PersonU where PersonU needs information from PersonA that found in the passage.
the passage. Use only information
PersonU starts the conversation and only asks questions. Most of PersonU'ss questions are follow-up contimutioos of previous questions.
Conversation:
role-assistant:
PersonU: need to know how prepare for college.
PersonA: You should first define your goals and interest and be sure to know the costs and how plan financially and academically for college.
PersonU: Is there something I can use as a guide to get ready for it?
PersonA: you can our checklist to get ready financially and academically.
PersonU: Any other tip?
PersonA: sure to check out the reasons to attend a College or Career schools.
PersonU: Ok, that sounds good. Can you give me more tips?
PersonA: Another thing to take into consideration
explore your career options.
role-user:
PersonA has access to information in the following passage:
###There are several disability compensation benefits. One of the benefits is the annual clothing allowance. This allowance covers the cost of purchasing new clothes up to limit of 200 dollars annually. ### Generate conversation of around two turns between PersonA and PersonU where
20

---

PersonU needs information from PersonA that is in the passage. Use only information found the passage.
PersonU starts the conversation and only asks questions. Most of PersonU's questions are follow-up or continuations previous questions. Conversation:
role-assistant:
PersonU: heard about a disability benefit for clothing. What is it about?
PersonA: This is disability compensation benefit known annual clothing
allowance.
PersonU: am quite interested about such benefits.
PersonA:
Would you like to find out if you are eligible?
role-user:
PersonA has access information in the following passage:
###Vascepa (icosapent ethyl) is prescription fish oil medication approved for treating adults with extremely high triglycerides, defined as a triglyceride level of 500 mg/dL or higher. While using this medicine, you may need frequent blood tests at your doctor's office. Vascepa is only part of a treatment program that may also include diet, exercise, and weight control. Follow your doctor's instructions very closely. You should not use Vascepa if you are allergic to icosapent.
make sure Vascepa is safe for you, tell your doctor if you have: 1 liver disease; 2 diabetes; a thyroid disorder; disorder; 6 if you are allergic to fish or shellfish; or. 7 if you drink large amounts alcohol, Vascepa (icosapent) works in the liver and bloodstream to reduce very low-density triglycerides. ###
problems with your pancreas;
bleeding or blood-clotting
Generate conversation of around three turns between PersonA and PersonU where PersonU needs information from PersonA that is in the passage. Use only information found in the passage.
PersonU starts the conversation and only asks questions. Most of PersonU's questions are follow-up or contimuations of previous questions.
Conversation:
role-assistant:
PersonU: heard about a medication called Vascepa. What is it used for?
PersonA: Vascepa is a prescription fish oil medication used for treating adults with extremely high triglyceride levels.
PersonU: Are there any side effects or precautions should be aware of?
PersonA: While using Vascepa, you may need frequent blood tests, and it's important
follow your doctor's instructions closely. It's important to talk to your doctor about any allergies or existing conditions
have.
PersonU: effective lowering cholesterol levels?
PersonA: Vascepa is used together with a low-fat diet and other treatments to lower total cholesterol people with severe high cholesterol.
role-user:
PersonA has access to information in the following passage:
(Document)
Generate conversation of around three turns between PersonA and PersonU where PersonU needs information from PersonA that is in the passage. Use only information found in the passage.
PersonU starts the conversation and only asks questions. Most of PersonU'ss questions are follor-up or continuations previous questions. Conversation:
role-assistant:
21

---

The in on-context few-shot prompt in step-2 is as follows (aims to revise the multi-turn QA conversation by adding pronouns and common nouns):
You helpful, pattern-folloving assistant.
role-user:
Given the following conversation between PersonU and PersonA:
###PersonU: I've heard about adobo sauce Mexican cuisine. Can you tell me more about it?
PersonA: Adobo sauce a dark red sauce made ground chiles, herbs, and vinegar. It's used as marinade and cooking sauce in Mexican and Southwestern cuisines.
PersonU:
What are the main ingredients in adobo sauce?
PersonA: The main ingredients are typically chile peppers, garlic, and vinegar. However, there can be variations in the recipe.
PersonU: How can I make adobo sauce at home?
PersonA: One way to make adobo sauce at home is by using smoked jalapeno peppers called chipotles. You can stew the chipotles in sauce with tomatoes, garlic, vinegar, salt, and spices. It's also possible to find canned adobo sauce in the Hispanic foods section of many grocery stores.
PersonU: Are there any other uses for adobo sauce?
PersonA: Adobo sauce can be used as a condiment or as marinade for various dishes. It enhances the flavor of meats and can be used cooking and serving sauce. Additionally, adobo is also used in a steu-like presentation of meat prepared in this sauce.
PersonU: Is adobo sauce easy to preserve?
PersonA: Yes, you can preserve adobo sauce by freezing or canning it when you're done making it. This allows you to store it for up to year and enjoy its flavors whenever you ##l
Modify PersonU'ss questions to use pronouns or common nouns to refer back to words or phrases in PersonU's previous questions or PersonA's previous responses.
role-assistant:
PersonU: I've heard about adobo sauce in Mexican cuisine. Can you provide more information about it?
PersonA: Adobo sauce is dark red sauce made from ground chiles, herbs, and vinegar. It's used as a marinade and cooking sauce in Mexican and Southwestern cuisines.
PersonU: What are the main ingredients in the sauce?
PersonA: The main ingredients are typically chile peppers, garlic, and vinegar.
However, there can be variations in the recipe.
PersonU: How can I make the sauce at home?
PersonA: One way to make it at home is by using smoked jalapeno peppers called chipotles. You can stew them in a sauce with tomatoes, garlic, vinegar, salt, and spices. It's also possible to find canned adobo sauce in the Hispanic foods section of many grocery stores.
PersonU: Are there any other
for it?
PersonA: It can be used as a condiment or as a marinade for various dishes. It enhances the flavor of meats and be used cooking and serving Additionally, it is also used in a steu-like presentation of meat prepared in this sauce.
PersonU: Is it easy to preserve?
PersonA: Yes, you can preserve by freezing or canning when you're done making it. This allows you to store it for up to a year and enjoy its flavors whenever you want.
role-user:
Given the following conversation between PersonU and PersonA:
###PrssonU: I've heard about test called urinalysis. What used for?
PersonA: Urinalysis is test that evaluates sample of your urine. It is used to
22

---

detect and assess a wide range of disorders, such urinary tract infection, kidney disease, and diabetes. It examines the appearance, concentration, and content
of urine.
PersonU: Can it help diagnose bladder cancer?
PersonA: Yes, urinalysis can be used as a screening and diagnostic tool for bladder cancer. One way to test for bladder cancer is to check for blood in the urine, which is called hematuria. This can be done during urinalysis by examining the urine cells under special microscope.
PersonU: What other conditions diseases
be detected through urinalysis?
PersonA: Urinalysis is commonly used to diagnose urinary tract or kidney infections, evaluate causes of kidney failure, screen for the progression of chronic conditions like diabetes mellitus and high blood pressure, and it can be used in combination with other tests to diagnose certain diseases. It is versatile test that helps detect abnormalities in the urinary tract.
PersonU: Where
get urinalysis done?
PersonA: Urinalysis can be performed in various healthcare settings, including
doctors' offices, urgent care facilities, laboratories, and hospitals, You provide a urine sample in a specimen cup, and the sample can be analyzed on-site or sent a laboratory for testing.
PersonU: How much urine is typically needed for urinalysis?
PersonA: Usually, only small amounts urine, ranging from
ml, are required for urinalysis testing. It's non-invasive and relatively simple test that provides valuable information about your health. ###
Modify PersonU's questions to use pronouns or common nouns to refer back to words or phrases PersonU'ss previous questions or PersonA's previous responses.
role-assistant::
PersonU: I've heard about test called urinalysis. What is it used for?
PersonA: Urinalysis is a test that evaluates a sample of your urine. It's used to detect and assess wide range of disorders, such urinary tract infection, kidney disease, and diabetes. It examines the appearance, concentration, and content of urine.
PersonU: Can it help diagnose bladder cancer?
PersonA: Yes, urinalysis
be used cancer. One way to test for bladder cancer is to check for blood in the urine, which is called hematuria. This can be done during urinalysis by examining the urine cells under special microscope.
screening and diagnostic tool for bladder
PersonU: What other conditions diseases be detected through this test?
PersonA: Urinalysis is commonly used to diagnose urinary tract or kidney infections, evaluate causes of kidney failure, screen for the progression of chronic conditions like diabetes mellitus and high blood pressure, and it can be used in combination with other tests to diagnose certain diseases. detect abnormalities in the urinary tract.
a versatile test that helps
PersonU: Where can go to get this test done?
PersonA: Urinalysis can be performed in various healthcare settings, including hospitals. You can provide urine sample in specimen cup, and the sample can be analyzed on-site or sent to a laboratory for testing.
doctors' offices, urgent care facilities, laboratories,
PersonU: How much urine is typically needed for the test?
PersonA: Usually, only small amounts of urine, ranging from 30 to 60 ml, are required for urinalysis testing. It's non-invasive and relatively simple test that provides valuable information about your health.
role-user:
Given the following conversation between PersonU and PersonA:
fmulti-turn QA conversation)
Modify PersonU's questions to use pronouns or common nouns to refer back words
23

---

or phrases in PersonU's previous questions or PersonA's previous responses.
role-assistant:
E More Details and Results for Retrieval in Conversational QA
E.1 Query Rewriting Prompts for GPT-3.5-Turbo
The in-context few-shot query rewriting prompt we use for GPT-3.5-Turbo is as follows:
You are a helpful, pattern followwig assistant.
role-user:
Given the following conversation between PersonU and PersonA:
PersonU: Hello, I would like to know what to do if do not agree with any decision.
PersonA: disagree with our decision about your monthly income adjustment amounts?
PersonU: Where can I find my SHIP contact information?
PersonA: You can find your local SHIP contact information in the back of your
Medicare You 2020 Handbook online.
PersonU: and how do they calculate the adjustments?
Instead of having this entire conversation, how can PersonU get what he she is looking for using a single question? Respond with that question.
role-assistant:
How is the calculation for adjustments made by SHIP determined?
role-user:
Given the following conversation between PersonU and PersonA:
PersonU: need to know
prepare for college.
PersonA: You should first define your goals and interest and be sure to know the costs and how to plan financially and academically for college.
PersonU: Is there something I can use as guide to get ready for it? Instead having this entire conversation, how can PersonU get what or she looking for using single question? Respond with that question.
role-assistant:
What resources or guides can use to help me prepare for college?
role-user:
Given the following conversation between PersonU and PersonA:
(Dialogue History + Latest Question]
Instead of having this entire conversation, how PersonU get what he she is looking for using a single question? Respond with that question.
role-assistant:
E.2 More Results for Retrieval in Conversational QA
In Table 9, we show comprehensive comparisons between fine-tuning and rewriting methods.
Interestingly, we find that compared to only using the rewritten query as input (Dragon + Rewrite (w/ single query only)), giving additional dialog history (Dragon + Rewrite (w/ dialog history)) makes the average scores significantly drop. This is because Dragon is originally pretrained on single-turn queries, will naturally have better generalization ability when single-turn rewritten

---

Models
Average
DociDial
QuACI
QRACC
TopiOCiQ
INSCITT
top-1
top:5
top-1
top:5
top
top:5
top-1
1op:5
top:5"
top-20
top:5*
top 20
Dragon (w/ dialog history)
46.29
73.09
43.33
75.61
56.8
82.86
16.17
81,96
57.68
78.80
27.89
46.22
Dragon + Rewtile (wr dialog histuryy)
47.57
74,12
44.51
76.98
57.23
83.01
16.45
82.60
60.94
81.74
26.69
46 16.22
Dragon Rewrile (wl single query only)
54,46
80.13
47.60
80.60
47.10
77.15
51.73
85.78
73.07
68.19
52.79
68.92
Dragon Fine-tune dialog history?
80.67
50.73
Dragon Fine-tine Rewelt (wl dialog hisorry)
53.17
50.84
49.30
84.64
$5.04
83.23
51.23
87.99
60.50
81.03
49.80
67.33
Table 9: Comprehensive multi-turn retrieval results across five datasets.
query instead multi-turn conversation is provided. And the rewritten query already contains sufficient information from the dialog history.
In addition, we observe that "Dragon + Fine-tune" performs par with "Dragon Fine-tune + Rewrite". In other words, for the multi-turn fine-tuning method, replacing original query with rewritten query as inputs yields comparable results. This is because rewritten query will not provide much additional information for the model since the dialog history has already been provided, and it might even cause negative effects (e.g., results on the TopiOCQA dataset) since it makes the whole conversation input not natural. This once again demonstrates how effective the fine-tuning method is in equipping the model with the capability to comprehend the multi-tume context.
F Details of CHATRAG BENCH
F.1 Dataset Statistics and Details
Doc2Dial (Fen) et al., 2020), a dataset covering DMV, SSA, VA, and Student Aid domains with an average document length of 101K words. We use the test set of Doc2Dial for the evaluation. consists 719 dialogs with 3939 user-agent turns. License this dataset "Apache License 2.0".
QuAC (Choi al., 2018), open- dataset with average document length of 15K words and unanswerable cases. We use the validation set of QuAC for the evaluation since its test set cannot be directly obtained. Its validation set consists of 1000 dialogs with 7354 user-agent turns. Among these 7354 user-agent turns, there are 1486 (around 20.2%) unanswerable questions. For the combined evaluation of answerable and unanswerable questions, response for the unanswerable question as "Sorry. the context." ", same the one set in stage-2 tuning. For the fair comparison, the unanswerable responses in the baseline models (i.e., Llama2-SFT/Char GPT-3.5-Turbo, GPT-4) with the same sentence (Details can be found in Appendix H). License of this dataset is "CC BY-SA 4.0"
set the ground truth cannot find the answer based on replace
QReCC (Anantha et al., 2021), open-domain dataset with an average document length of 5K words and maximum document length of 20K words. We use the test set of QReCC for the evaluation. Its test set includes some dialogue samples source from the QuAC dataset. To avoid overlap with other benchmark datasets, we remove those QuAC-sourced samples, resulting in 2805 user-agent turns. License of this dataset is "Apache License 2.0".
TopiOCQA (ACQA)Adlakhaa et al., 2022) uses the whole Wikipedia as the documents. We use the validation set of TopiOCQA since its test is not available yet. Its validation set consists of 205 dialogs with 2514 user-agent turns. Each question has a total of four human- answers. It has unanswerable cases. However, we do not include this dataset unanswerable case evaluation because we only find 46 questions with no answer being found, which is not sufficient enough for the unanswerable case evaluation. Given that this dataset use the whole Wikipedia as the retrieval corpus, we leverage its topic information for multi-turn retrieval since we focused more on conversational QA over documents which typically narrow down specific topic or domain. License of this dataset Nww.-oooonnnnaaaaaacccccaaaaaaaa
4.0 International".
INSCIT (Wu et al., 2023) use the whole Wikipedia as the documents. We use the validation set of TopiOCQA since its test set is not available yet. Its validation set consists of 86 dialogues with 502 user-agent turns. Each question has an average of 1.9 human-written answers. has unanswerable response for few questions. However, we do not include this dataset in unanswerable case evaluation
25

---

because every question always has at least one answerable response, which means every question is answerable. INSCIT also use the whole Wikipedia as the retrieval corpus. Hence, we also leverage its topic information for multi-turn retrieval for the same reason mentioned in TopiOCQA. License of this dataset is "argiv.org perpetual, non-exclusive license".
CoQA We use the validation set of CoQA since its test set cannot be directly obtained. Its validation set consists 500 dialogues with 7983 user-agent turns. Each question has a total four human- written answers. It has unanswerable cases. However, we do not include this dataset in unanswerable case evaluation for the same reason as in the TopiOCQA dataset. We only find 13 questions with evaluation. CoQA
answer being found, which is not sufficient enough for the unanswerable contains passages from seven domains. Five of these public ones are under the following licenses: Literature and Wikipedia passages collected from MCTest which comes with "MSR-LA license"; Middle/High school exam passages collected from RACE which comes with its own license; and News passages collected from the DeepMind CNN dataset which comes with "Apache license".
shared under "CC BY-SA 4.0 license", Children's 's stories are
DoQA We use the test set of DoQA for the evaluation. Its test consists of 1200 dialogues with 5394 user-agent turns across Cooking, Travel and Movie domains. Among 5394 user-agent turns, there are 1479 (around 27.4% unanswerable questions. We use the strategy the one mentioned in QuAC dataset to incorporate unanswerable samples into the evaluation. License of this dataset is "CC BY-SA 4.0"
ConvFinQA We use the validation set of ConvFinQA since its test set cannot be directly obtained. Its validation consists of 421 dialogues with 1490 user-agent turns. License of this dataset "MIT License".
SQA We use the test of SQA for the evaluation. Its set consists of 1025 dialogues with 3100 user-agent turns. License of this dataset is "Microsoft Research Data License Agreement".
HybriDial We use the test set of HybriDial for the evaluation. Its test set consists of 243 dialogues with 1111 user-agent turns. License of this dataset is "arXiv.org Non-exclusive license".
F22 Details of Incorporating Table as Context
We use the markdown format process the table in the document (i.e., using "I" to separate each cell in the table). However, as the number of rows in a table increases, it is harder for the model to link the values of cells in the last few rows their corresponding headers in the table. To address this issue, we append the headers to the corresponding cell values in each row, so as to make it casier for the model capture the connections between the cells and their respective headers. For example, original markdown table
Boat
Crew | Nation | Date | Meet | Location
Mix
Mahe
2009
Poznan
M2-
Hamish
US |
2012 | Olympics Lucerne
M2+
Igor
1994
Indianapolis
will be converted
Mix
(Boat)
Crew: Mahe | Nation: FR | Date: 2009 | Meet:
Location: Poznan
M2-
(Boat)
Crew: Hamish Nation: US Date: 2012
Meet:
0lympics
Location: Paris
M2+
(Boat)
Crew: Igor Nation: DE
Date: 1994
Meet:
Location: Indianapolis |
We apply this table processing strategy to all tabular datasets.
F.3
Prompts for the Benchmarks
We adopt the prompt format mentioned in $C.2 for benchmarking ChatQA, Llama2-Chat, Llama3- Instruct, GPT-3.5-Turbo, GPT-4, and GPT-4-Turbo. For Lllam2-Chha and Llamma-Instruuc, we modify the prompt based their own templates, and convert the prompt into the chat completion
26

---

template for OpenAl models. We keep the [Context for Latest Question) the same for all models. We adjust the (Instruction] for the baselines (i.e., Llama2-Chat, GPT-3.5-Turbo and GPT- 4) to ensure they do not get sub-optimal results. We use different instructions for test benchmarks with different answer types (e.g., long answer, short answer, arithmetic calculation). The (Instruction) for all models on the benchmarks are as follows:
F3.1 ChatQA
We keep the (Instruction} consistent between the data blends stage-2 and test benchmarks based on different answer types.
We use "Please give full and complete answer the question. for DoQA, IN- SCIT, HybriDial, Doc2Dial, QuAC, and QReCC, since these datasets generally have long answers for the questions.
We use "Answer the following question with a short span, or a full and complete answer." for SQA and TopiOCQA, since these datasets have both short and long answers based on the questions.
We use "Answer the following question with short span. The answer needs to be just in a few words." " for CoQA, since generally has short answers for the questions.
We use "Answer the following question with number from context or the math arithmetic using +, *, or /." . for ConvFinQA, since this dataset requires the model to either extract numbers from the context or do arithmetic calculation. We will calculate the number based on the arithmetic formula the model generates and compare it with the gold answer.
F.3.2 Llama2-Chat & Llama3-Instruct
Note that we tried original Llama2-Char prompt template which gives slightly worse results compared using the one SC.2. We have tried several (Instruction) for Samaa-Chat.. We find the ones below works the best.
We "Please give an answer in just one sentence." " for DoQA, INSCIT, HybriDial, Doc2Dial, QuAC, and QReCC, since these datasets generally have long answers but within one sentence. We notice that is important to give Llama2-Chi specific instruction like "one sentence" instead of "full and complete answer" to prevent the model from generating very long answers.
We use "Answer the following question with short span, TopiOCQA, since this dataset has both short and long answers based on the questions, and the long answers generally within sentence.
sentence. " for
We use "Answer following questions with one or
entities." " for SQA, since the answer for this dataset always consists of one or a list of entities from the con- text.
We use "Answer the following question with a short span. The answer needs to be just a few words." for CoQA, since generally has short answers for the questions.
We use "Answer the following question with just a number from context or just the math arithmetic using +, -, *, /." for ConvFinQA, since this dataset requires the model to either extract numbers from the context or do arithmetic calculation. We extract the arithmetic formula generated by the model and use caleulator to get its final result.
F.3.3 GPT-3.5-Turbo & GPT-4 GPT-4-Turbo
We have tried several (Instruction) for GPT-3.5-Turbo, GPT-4, and GPT-4-Turbo, we find the ones below works the best.
We use "Please give an answer in just one sentence." for DoQA, INSCIT, HybriDial, Doc2Dial, QuAC, and QReCC, since these datasets generally have long answers but within one sentence. Similar to Llama2-Chat, we also find OpenAl models tend to generate quite long answers
hhttps://gpuu alm-utils..m org.org////oo hete/llamm222 2-prompt- prompt-templater
27

---

given the instruction of "full and complete answer". Hence, we make the instruction more specific (i.e., "one sentence") prevent the model from generating very long answers.
We use "Answer the following questions in JUST a few words or one sentence. " for TopiOCQA, since this dataset has both short and long answers based the questions, and the long answers are generally within one sentence.
We use "Answer the following questions with one list of entities. Do not give detailed explanation. Answer needs to be as short as possible." for SQA. We find that OpenAl models often give detailed explanation for SQA dataset unless specifically instructed not to.
We use "Answer the following question with short span. The answer needs to be just a few words." " for CoQA, since generally has short answers for the questions.
We use "Answer the following questions with just a number from context or just the math arithmetic using +, +, or /." for ConvFinQA, since this dataset requires the model to either extract numbers from the context or do arithmetic calculation. We extract the arithmetic formula generated by the model and use calculator get its final result.
G
Human Evaluation
G.1
Evaluation Setup
You will be provided some tea paragraphs (contexrt) a dialoguue history (between Usee and Accistanti final question from User that could answiied based on the paragraphs, two the finsl cuestion. question. Note
You
indicate
found. Nore the dialogue history could be empry.
What you need to do:
After reading the text paragraphs (context), the clelog history, the question, and the reference answer, you have to select the respanse that more factuslly correct
both equely factually coneect select both netheer is factuelly cornect, select neither
Attantion
long as both Please peralize verbosity the response has some content that are NOT cirectly relevant to the question
Consicer both resporces Equally
answer the questioo,, NOT have unventrabic teotucily ncomeot content.
- Please per saliow it some content in the response can NOT be varifiable..
Select option
Tesporset
S(pomest)
Response2
Dialogue History: Sid alogue history:
Both
Finel Queation: S(questoo)
Reterence Answor: S(reterence)
Response1: S(response1)
nesponsa2: 2(moponsa2)
Figure 4: Human evaluation instructions and the interface for annotators.
Subert
We conduct human evaluation across the ten test benchmark datasets. We randomly select 60 samples for each dataset, and each sample is labelled by three annotators, which results in a total of 1800 annotations.
We ask the annotators to verify the facts models' outputs and determine which model provides more accurate response to the question. We use Amazon Mechanical Turk platform to conduct the human evaluation. The details of our human evaluation instructions and the interface shown to annotators illustrated in Figure 4.
28

---

Ours Win
Tie
GPT-4 Win
Average
13.81%
69.09%
17. 10%
Doc2Dial
14.29%
68.00%
17.71%
QuAC
11.7%
73.33%
15.00%
QReCC
11.11%
77.22%
11.67%
CoQA
7.78%
80.00%
12.22%
DoQA
22.78%
57.78%
19.44%
ConvFinQA
16.67%
67.78%
15.55%
SQA
11.11%
61.67%
27.22%
TopiOCQA
19.31
60.699%
20.00%
HybriDial
7.78%
78.33%
13.89%
INSCIT
15.56%
66.11%%
18.33%
Table 10: Human evaluation (A/B testing) comparing our ChatQA-1.00-0B to GPT-4-0613 on CHATRAG BENCH. In terms of average scores, our model and GPT-4 are tie most of the time (~69..1%) and GPT-4-0613 achieves slightly higher win rate (~3.3%% than ours.
G.2 Full Human Evaluation Results
The full human evaluation results over the 10 datasets on CHATRAG BENCH are shown in Table 10. We first find that our ChatQA-1.007001 and GPT-4 are tie most of the time (~69.199 1%), and GPT-4 achieves slightly higher win rate (~3.3%% than ours. This further confirms our model has powerful capability to produce correct answers. Second, we find that our model achieves slightly better win rate than GPT-4 ConvFinQA, which indicates the strong arithmetic calculation ability of our model. Third, find that GPT-4 achieves significantly better win rate in SQA task, which suggests that there is still a gap between our model and GPT-4 on tabular reasoning task.
H Unanswerable Case Evaluation
H.1 Evaluation Setup
For unanswerable case, we consider the model indicating that the question cannot be answered as correct. For answerable cases, we consider the model not indicating the question is unanswerable as correct (i.e., the model giving an answer). We find that ChatQA always generates "Sorry cannot find the answer based on the context" when it finds the question is unanswerable, since we use this sentence for Llama2-Chat, Llama3-Instruct, GPT-3.5-Turbo GPT-4, and GPT-4-Turbo, their outputs for unanswerable questions usually follow some specific patterns. Below, we list the heuristic matching patterns derived from all generated samples in DoQA and QuAC datasets to determine if the model suggests that the question is unanswerable. When we find the generated output contains the pattern listed below, we interpret it as an indication that the question unanswerable, and will replace its answer with "Sorry. I cannot find the answer based on the context." for the F1 evaluation.
response for unanswerable questions in the stage-2 tuning. As
cannot find, can't find, not able to, unable to, does not provide, cannot provide, cannot answer, couldnot answer, can't answer, couldn't answer, cannot be found, cannot be determined, do not have, couldn't find, no information, does not mention, doesn't mention, not explicitly mentioned, not explicitly explain, can not find, could not find, does not specify, doean's 't provide, doesn't specify, there is no, not mentioned, don't have, don't 't know, does not include, doesn 't include, does not contain, doesn't contain, not provided, does not indicate, doesn indicate, does not disclose, doesn 't disclose
H.2 Ablation Studies on the Number of Unanswerable Samples for Training
In Table 11, we find that compared to using SyntheticConvQA. using significantly increase the average accuracies on both QuAC and DoQA. is because the unanswerable annotations are in higher quality for human-annotated data, which significantly improve the accuracy
29

---

Models
Avg
Avg.
QuAC
QuAC
Avg DoQA
DoQA
DoQA
ChatRAG
QuAC
(no*)
(yes*)
(no")
(yes*)
Ik unanswerable
76.88
80.89
75.10
86.67
72.88
64.49
81.26
54.16
1.5k unanswerable'
77.25
80.76
77.66
83.85
73.74
68.81
78.67
54.14
ChaiQA-I.0-70B
2k unanswcrablet
77.10
80.82
77.59
84.05
73.38
67.95
78.80
53.86
2.5k unanswerahlef
75.87
78.81
73.76
83.85
72.93
66.54
79.31
53.78
SymtheticCom-QA"
69.84
72.92
55.38
90.42
66.77
45.09
88.45
54.08
Table 11: Accuracies answerable and unanswerable samples across QuAC and DoQA datasets. Avg-Both the averaged score between QuAC and DoQA, and ChatRAG is the average score on the is replaced with the Synthetic- ConvQA, * "no" denotes the unanswerable samples, while "yes" denotes the answerable samples. WWe conduct ablation studies in terms of the number of unanswerable samples in HumanAmnotat edConvQA. We use 1.5k unanswerable samples for our final ChatQA models, as it produces both high-quality generation and less hallucination.
CHATRAG AG BENCH. denotes that the FlumanAmnottteeConne
for unanswerable cases. Furthermore, we conduct ablation studies in terms of the number of unan- swerable samples in
We find that using small amount of unanswerable samples (e.g.. 1.5k) able to achieve remarkable results on the unanswerable evaluation, as well as the overall CHATRAG AG BENCH score. Interestingly, incorporating more unanswerable samples does not necessarily lead to higher accuracy scores in the unanswerable evaluation.
30

---

I
Case Study
Contest
Converatico
Aubitar Regana
unen procco, secp reggreddnng adkee Apponiing
yourss for tound bourins bisclanged y'aal appoaiig VA. decidiss
regiisi perpery
Ausatani Doyye vesl counly whit expect?
User Whet liors res thand heaing?
Clano 10
dest Petraary is, 2015 you fechow the ack prodoss
desi
yeu recn or alle incheuur 10, st0t nateppiing ei Ripel Appects
heating
User ts secovey no suceeskk repising thant toures
CDPT del's Yee'l fellin for pricess the Reael learing cid receraal abtr putecuyy parkippang Rapal Appcill Midenttation
you
you edig There tio aquull a dociice you mosiee hotler Fibusary
to of femme: Appoil Horning pouc... The board mosice rxpaett...
Oser whic will Lnkon the now procce beand hortnn?
to poodle dioee julgs alves your appoal
ogtng the wisl process winng Foom 1012?
ontar
Gread Traee You willl dellow the aon precess E-ppoatig VA.
you roceval
Arya sumbo dai contretatre unscatle??
Uwer tow old Arya varvinc thi ontrrmmace anvi
appeaed
scea (ive spacsi A vtric wilker bire go beity ledking lig sidi pita sepptthhla dugarr. Night
Aucizaan seya was gaick moegs to man, dogg the aeg
zoot
Gro-4tes The des prondc infermatiio on dhliiku
bapon auss was eo oocco SHIDO
saprice theos subticc walker
and
whal
Usee How clockiid nor they kyy on the
by
Naxef
Taye ef nntert rocrre
Prevervol arma
User wtich of fiecas capurtiia Nouphen noserve?
Carputian Bicophere
2475.8
Charge-1 th Misdow
Unikk Wide Mesios
Cargetiaan Bingghee
11801h
Macuramne, Raddy/ TTyhurhur
GF2--013 Choredaora, Chotes W Mindow, Skydaves.
Manunera
Chrputias Bicekkare
User wilch those
1100/
NatonalPPt Park
402.5
Groad Trade Whilks Wide Mandow
Dele
Locaien
Type
Shate
User whe war al de whick oeglesd knttryed or
34 Apst 1944
Masih Gameat
Na-110
Denroyed
1944 indidag 24 Apdll 1960
23 Sapeeeeer Narhhansa Gormay
18 Aagai 1944
Purk, Friere
05
Ny 109
Deanned
Bvanged
Auvatat: April 27 Axpert Sepiemberr
1944 Golaheer 1864, Novitter 1944 Jamaayy 1945
CIT1- 0013 Agel 1994 Swithh 1944
Sepaaber 1944, Ocnber 1944,
Dcnbeer
Ne-109
9. Denker 1944
Beris Okrmayy
NX-10
cain fuat one plae?
Grvend Trade 24 Apdil 1966, 17 Separber 1914, Deviber 1941, Novambaa
54 Tanuary 19-5
Balls, Germay
M0-10
Dearojed
Table 12: Case studies on ChatQA-I.0-70B and GPT-4-0613, We use blue color to highlight the relevant context for the last user question. And use green and red color to highlight the correct and incorrect parts in the assistant responses.
In Table 12, we show four examples of ChaiQA-LL0-70B and GPT-4-0613 outputs. More examples can be found in Appendix I. The first example (about Board hearing) is simple information seeking problem, and both ChatQA-1.0- 1.0--0BB and GPT-4 answer correctly. In the second example (about Arya), the model needs to find implicit information (highlighted in blue) for giving the answer. GPT-4 tends to be safe on providing an answer, and it replies that the context does not provide exact information about the age which is also correct.
Both the third and fourth examples require the model to have good tabular understanding and reasoning ability. In the third example (about Massif), ChatQA--1007000 -.000BB gives correct answer by comparing the size of preserved area against 3100 ha, while GPT-4 fails do so. In the fourth example (about John B. England), ChatQA--L-0-0BB correctly lists three dates but misses one date, while GPT-4 correctly answers the question.
More examples on MaatQA-----1333 ChatQA-1.0- 10070B, CPT-3.5-Turbo-061 and GPT-4-0613 can be found in Table 13. We use blue color to highlight the relevant context for the last user question. And we use green and red color to highlight the correct and incorrect parts in the assistant responses.
31

---

Responses
Pos
Rider
Time/Redired
User: what racers are shown?
Loris Capimesi
38:04.755
Assistant: Loris Capirossi, Valentino Rossi, MeWilliams, Tohra Ukawa, Henk Van Lagemast, Maurice Bolwork, Roberto Rolfo, Marcellino Lucchi.
2
Valentino Rossi
+0.180
3
Jeremy MeWilliams
+0.534
5
Shiaya Nakano
+0.742
CharQA-1.00 1.00QA-----18:::22 Loris Capinossi
Ralf Waldmann
User: those, what times were rocoreed?
Franco Battnini
+20 889
CharQA-100 BooQQA-L--70B:--11 -1 Lap
Jason Vincent
Stefano Perugini
+21 310
+20.891
Assistunt: 38:04,730, +0. 180, +0.534, +0.537, +1:10.635,
Retirement.
GPT-3.5- Turbo-06--:-:: 38:04.730
10
Anthoey West
466816
User: of those times, which were not actual times!
GPT-4-0613: 3: 38:04.730
Arno Visscher
1:40.635
Ground Lap
24
Henk Van Lagemaat
Lap Resirement Retirement
Assistant: Lap, Retirement
Ret
Maurice Bolwerk
Andre
User: which of was actual finish??
Ret
Roberto Rolfo
Retirement
Marcellino Lucchi
Retirenent
Assistant:
was average backlog 2015?
2014 compared 2013 mot net sales
decreased S 305 million, or 3% in 2014 as compared 2013 space systems operiting resalts inc Iuded the following millionsy:
Assistant: 17400
User: what was in 2014?
ChatQA-L.0-13 20300 214000
sales
backlog year-end
Assistant: 20300
2015
S $9105
17400
CharQA-1.00 17400 20300 21400
2014
9202
20300
User: what is sum those years?
2013
$ 9288
$21400
Turbo-0613: GPT-35--Turbo-6613
GPT-3.5
9288 9202 9108
2015
compared to 2014 space systems 2019
nct sales 2015 decreased S 97 million, compared
User: what avernge backlog 2013?
GPT-4-0613: 17400 + 20300 21400
decreases were
Ground Truth: 17400 20300 21400
partially offset by higher net sales of approximately 235 million for businesses ucquired 2014; and approximately $ 75 million for the orion progrum due
Assistant: 21-000
User: what is the total sum for all years?
Assistant:
CharQA-1.0 1.0-1 There Gymnastics participated in the championship.
The XVI World Rhythmic Cymmastics Champioeships were held Brussels,
CharQA-L.D participated in the championskip.
gymmasts"
User: What the XVI World Rhythmic Gymmastics Championships?
Relgium, on November 20-22, 1992.
GPT-3.5 Turbo 0613; There were total of gymeasts" who participated in this champiooship.
Rank
Gymnast
Country
Point
Assistant:
The XVI World Rhythmic
Ovana Kostina
Russia
9.775
Gymmastics Clampionship were
2
Maria Petrova
Bulgaria
9.700
Brussels, Belgium, on November
3
Diana Popowa
Bulgaria
9.625
does not provide the total number of gymmasts who participated in the XVI World
GPT-4-0613:
Acedo
Spain
5
Irina Delcanu
Romania
9.550
User: How Gymnastics participuted this championship?
Ekaterina
Ukmine
9.525
Rhythmic Gymnastics Championships.
Viktoriya Yani
Ukraine
9.475
Maria Sansaridou
Greece
9-425
gymmasts pariiippred this championship.
"Both models conrect the User's typo of "Gymiastiis"" to "gymossis"
32

---

Context
Conversution
Ascistant Responses
3000 metres steeplechase steeplechase field it abstucl
distance
User Hello, sesplechase?
about 3000
stepleshase
derives name
disance 3000 metres,
horse racing steeplechase.
Asistant: The metres sseeplechase mow comsus distanee for siceplaciuse and fiekd that goes over distance of the 3000 metros. which derives its name the horse racing steeplochase.
CharQN-1 1,0-330: ran Berlin, Adelaide, Stociholm.
Year
Athlete
Location
ChuQA-I 10-08: Kary Bricn can Adelaide and Berlin.
1968
Jouko Kuha (UUN)
Stockholm
1969
Vlalimins Dadinas (URS)
Kizs
1971
Kerry
(AIS)
Adelliide
User. Can you tell me the mes that are past of'??
GPT 3.-.-oo-T Turbo 0613: Kerry O'Bries ran in Berlin, Adklaidd, and Clumpsign.
1972
Anders Gardenud (SWL)
1973
Ban lipdho (KEN)
Helvinki
Assistate Some of the men are Hlenry Rozo and Michael Kanst
GPFT S-PT-4-660 Keny O' Beizen run Beilin and Adelaide.
1974
Andes derdd (SWE)
Helsinki
1976
1975
Anders Gärderud (SW E)
Ander Gardensl ISW
Stockhelm
Monurell
User: Which locations did Kerry O' Beiie for?
Giound Truth: He for both Berlin and Adclasie.
1977
Maclael Karsi (PRC)
Stockheokn
Assistant
ChaiAA- striclly These provisions also deperdient
grounds refural that
EU family member traveling to Schengen coustries, source:
tuael visa-free oaly you have "residence" card of fumily member a unioa cilsen" leuna under orticle 10 directive for if you are citizen a country whose nationals don't i requize visas in the Schenges urea).a A residence card issoed by Romania to the spotse of Romseian citizen is not neassoorly an seticle 10 card. If you do not have on article 10 card, EU countries other than Romania must toate and on the basis of an expedited procedare. The grounds for refussil of that vies and strictly limited, Those
User. tavel from Romania to Schengen auos visa-free?
your troveling with or joining your spouse, case you are ant, you 'll ased segular Schengos visa.
movenant
Ascistant: You can travel visa-free only you have a "residence card of family member a unioa cinzzen issedd under article of the fize movessent direciive
CharQA-I 0-700: Sorry comnor find the snswer hased on the context.
GPT-33 S-Ticha00613 To abtais un article 10 card, you would need to meet the requirements set out in the free
visa free of charge
User have temporayy residence card also, can we travel togetber visa free?
movemment directive and apply for it through the appropraaee channels in Romania.
regular from Romania not yet fail participant Schengen agroeneent, countrics not just Schengen countries, becaue they are haxed on RU right froedom movement
provtaions spouse.
viaa Relased poisted appears your
dependent your traveling with
you'll ll need
have article card, EU countries other than Romania most loore vics free charge the procedure.
Asestent If
expedited
Grund Truth: Sony. Eaannt the contect.
how to ohiain article 10 cand."
GPT-4-0613: The test does not provide informotion
laaee
How get anticle cordf?
apply to
"Is evalastion, petter replace with "Sury. caanno answer based on the context" to monch with the ground truth.
manswerable
Ascistant
tite: Bengal for. "Vulpes bengaleneis" is relatvely small fox with an clongated muzzle, long, pointed cam, and hushy ball abont to 50S the langhs of the head and body, lis donsal pelage vayy varisblee hut moutly grayich and poler ventrally: its legs tend to be brownish or ratous. is more daintily built than "Volpes valpex" The tail busby with prominent black which dielinguidhes from "V. culpes". Back of dark brown with black naked and the lips bluck, with small black hair patches upper part auzzle (shufo
User what is red fox?
CharQA L.0 130: The ears have the same colour the aape or maybe das but nut having a dark pulcht
Ascistant: is the largost of the true fintes and one of the most widely distributed members of the onder Camivora.
in "V. volpes' Extencive variation in cost colour exists scrrss populations and vecomallyy within populations but generally varies from grey to pale brown.
Its rhinsmum
User where did originate?
CharQA-1 00770R- "Valpos hengslemmis" is relatively small fox elongated muzzie. pointed busdy head body.
Assistant The specias Eursiin origin.
the loagth
User how did evolve?
iite:: fox, "Vülges" than Afghun, consac specialised
fox considered more
Asiitust considered more specialised Toras "Vulpes" fenes in the direction sixe and adupeattinn cambirry..
3.5 0613: soery. but couldn't any infimmation about thind type for the given context.
Bengal loses is disecton size odeptttion to carniwer: the shall displays fir fower nectenonn traits than other species...
Adgitan coesae and Bengal
GFT 4 0613: The contest does not provide information on third type fox.
tiee Bengall Fox, The cars have the same colour the mape er mayhe darker. but not having a dank ponch as in "V. vulpes" Extensive variation in coer colour exists across populations and sesconally within populations. but generally varies from pule browi....
User what is the appearance of this third type of fix??
Ground Truth: It relativlly small fox with on
Assistant
elongated muzzle, long. pointed cuss, and busby tall about 50 to 60% of the length of the head and body.
33

---

Context
Conversation
Assistant Responses
Zimmer Holdings, Inc. Notes to consolidated financial statements continued the unaudited pro forma results 2003 include events or changes in circumstances indicare that the carrying $ 90.4 million expense related centerpulse hip and knee value of asset may not be recoverable...
User: was tocal inventories in 2003?
Assistant: 527.7
User: and what 2002?
inventories payments that contingent the growth of implex inventories at december 31, 2003 and 2002, consist product sales through 2006.
Assistant: 257.6
CharQA-1.0-133: -84.3-206.7
User: how much, did the 2003 amount represent in relation to this 2002 onc?
CharQA-1.0-P0 1.000:: 384.3 -06.77206.7
the net value trunsferred at the lollowing tin millions): closing approximately million, which includes.
G.T-3.5 06 13: 384.3 206.7
Assistant: 257.6
0-013: 384.3 206.7
finished goods
progress
inventories
User: portion equivalent to the 2002 amount?
excluding
2003 $ 384.3
90.8
$ $27..7
Ground Truth: - 206.7
2002
202..7
50.9
257.6
made by zimaee to implex purstaant to their existing alliance materials and work in progress 90.8 50.9 arrangement, escrow and other items. the acquisition will be inventory step-up up 52.6 2013 accounted for under the purchase method of accounting.
Assistant: 527.7/257.66
hetween those same two years, what was the change the total of finished goods, millions?
User:
Assistant:
common equity, related stockholder matters and issuer purchases oquity securities following graph compares annual total return of our common stock, the standard pour 2019s 500 composite slock index cs0 500 index 201d) and our poer group (20lcloews peer group 201d) for live ended december 31, 2015.
market registrant
User: from 2010 2011, what the change the value of the loows stock?
Assistant:
User: what change portion the 2010 value of that
the graph assumes that the value of the investment in common stock, the s&p 500 index and the loews peer group was 100 on december 31, 2010 and that dividends
CharQA-1.0-13B: (106.04 100) 100
CharQA-1.0-70B: (106.04 100) 100
seinvested.
Assistant: (97.37 100) 100
common stock
s&p 500
peer group
GPT-3.5 0uuubb-0611:. (106.04 100) 100
User: from 2010 2012, what the change the value of that stock?
2011
97.37
102.11
101.59
GPT-4-0611 (106.04 100) 100
2013
126.23
156.82
145.12
Ground Truth: (106.04 100)
2014
110.59
178.29
A ssistant: 106.04 100
2015
101.72
180.75
144.70
User: how much did this change represent in relation that 2010 value?
The loews peer group companies that are industry competitors of our principal operating subsidiaries...
of the lollowing
dividend information we have paid quarterly eash dividends loews year since 1967.
stock in each
Assistant:
Table 13: Case study for ChatQA-1.0-131 ChaIQA-L-0-70B GPT-3.5-turbo-0613. and GPT-4-0613, We blue color highlight the relevant context for the last user question. And we use green and red color to highlight the correct and incorrect parts in the assistant responses.
34

---

Human Annotated Conversational QA Data
J.1 Statisties of User-Agent Turns
50.00%
40.00%
30.00%
Ratio
19.86%
20.00%
12.36%
10.00%
4.16%
5:23%
4 6.06%
1.05%
1.45
0.51%
0.22%
0.00%
10
12
13
>14
Number of user-agent turns dialogue
Figure 5: Statistics regarding the number of user-agent turns for each dialogue
Figure illustrates the statistics regarding the number of user-agent turns per dialogue over the 7k human annotated conversational QA data.
J.2 Guidelines for Conversational QA Data Collection
In this section, we demonstrate the guidelines provide for the conversational QA data collection. We request the annotators to act as both user and agent to construct the conversational QA sample based the given document, which makes the annotation more efficient. The guidelines consists of three parts: 1) What does conversational QA samples look like; 2) What kinds conversational QA samples need; and 3) What need to annotate.
J.2.1 What does conversational QA samples look like
One conversational QA sample based document that we provide. The user and agent behaviors are as follows:
User behavior: 1) Ask questions to agent based on the given document; 2) Answer questions from agent when agent wants to clarify something.
Agent behavior: 1) Answer questions from user based on the document; 2) Ask questions to the user when the user's question is not clear, too general/broad.
J.2.2
What kinds of multi-turn QA samples we need
We list the requirements for user's questions and agent's 's responses below.
User's Questions
User's questions can refer to his/her previous (or multiple turns before) question.
User's questions can also refer to the previous (or multiple turns before) answer from the agent.
Try pronouns common nouns replace the entities mentioned before.
Try make the user's questions diverse. For the same type of questions, try use different ways express it in different dialogue annotations.
Other user question types need
35

---

Given the answer from agent, ask for extra information (e.g.. What else ...; any other .... there more ...).
Switch the topic and starting a new thread conversation.
Ask two questions at the same time.
Agent's Response
Try to make the agent's answer within 1-2 sentences. If the answer has to be long, try to make it concise.
Try not to directly copy the whole relevant context from the document the answer. Instead, try to construct the answer by paraphrasing the chosen context.
Try to come up with a small proportion of cases where agent asks questions to user to clarify something. Specifically, when the user's question is too broad or not clear, the agent needs to narrow down the scope by asking clarification questions to see which specific aspects that the user cares more about.
J.2.3
What we need annotate
list what we need to annotate each conversation below.
For each document, need annotate the user's questions and corresponding agent's responses. The average number of user-agent turns per conversation needs to be around five.
For each user's question, we need annotate all the relevant context within the document.
36

---

K Results of All Models on CHATRAG BENCH
Models
Avg. w/o HDial
Avg.
D2D
QuAC
QRcCC
CoQA
DoQA
CFQA
SQA
TCQA
HDial
INSCIT
Llamu2 SB
33.24
34.81
30.26
19.21
37.55
62.75
21.76
34,43
12.18
32.88
48.96
28.16
Mama2-Chat- Chattt
33.27
25.
46.02
72.28
33.15
26.14
47.02
31,67
ChulQA-1.0-7B
46.96
47.71
37.88
29.69
46.97
76.61
41.57
51.61
61.87
45.45
54,51
30.96
GPT-SFT-88
32.48
31 00
20.07
37.69
59.24
21.72
40.06
17
52.29
86
ChaiQA-L.0-BB
48.86
49.36
36.76
33.95
45.54
77.90
44.65
61.68
60.74
47.03
53.81
31.50
Llama2 SPT-130*
36.08
37.69
30.68
21 59
38.25
69:52
21.70
41.14
37.85
35.26
52.22
28.73
Jamm2-Chatt133R
39.29
40.34
34.74
27.89
49.19
72.50
32.60
41.54
25.39
39.25
49.82
32.52
ChaIQA-L.0-13B
50.27
50.86
38.05
34.28
48.06
77.23
43.31
65 65.44
66.41
48.88
56.19
30.79
GPT-SFT-228*
37.11
38.83
32.98
22.83
39.92
68.19
22.83
30.54
47.01
38 38.07
54.22
31 66
ChaiQA-L.0-222
53.01
53.53
39.71
36.89
50.47
80.32
44.88
72.08
69.08
48.83
58.20
34.11
Hama2-SFFT-00
25
73.04
46
58,90
55.52
30.71
Jlama2-Chat-70F 70B
44.64
45.21
36.87
32.47
49.40
80.41
38.97
46.85
37.62
44.
50.35
34.88
ChaiQA-1.0
53.89
38.90
48.05
78.57
73.69
69.14
56.44
31.90
Llama3-Instruc-88
46.76
46,70
33.59
33 60
45,70
75.26
37.33
58 58.05
59.72
44,
46.20
32.59
Llema3-Chut(A-15--888
53.99
55.17
39.33
39.73
49.03
76.46
49.60
78.46
73.28
49.96
65.76
30.10
1ama3- Jamm--Instmett708
$2.95
$2.52
37.88
36
51.34
76
41.24
76.60
69.61
49.72
48.59
36.23
Llama Maama--ChnQ4-115-70000
57.14
58.25
41.26
38. 82.82
51.40
78.44
50.76
81. 88
83.82
55.63
68.27
32.31
Command
33.51
34.16
49.77
71.21
46.70
35.76
GPT-3,5- Turbo-0--
30.69
50.37
34.83
37.17
50.46
79.33
41.11
73.15
60.63
44.30
47.42
35.27
GPT-4-0613
34.16
52.01
77.42
79.21
GPT-4-Tubb-2022-04--00
54.72
$4.03
35.35
40.10
51.46
77.73
41.60
84.16
79.98
48.32
47,86
33.75
Table 14: Full zero-shot results on CHATRAG BENCH. SFT denotes that only stage-1 supervised fine-tuning is applied. We add Llama2- 13B and our in-house 8B/22B models.
Models
Avg. HDial
Avg. All
QuAC
QReCC
CoQA
DeQA
CFQA
SQA
TOQA
HDial
INSCIT
Lluma3- Instruct-SB
46.76
46.70
33.59
33,60
45.70
75.26
37.33
58.05
59.72
44.96
46.20
32.59
Hlama--Instruct-- 8B + Stuge-2
54.41
55,59
39.18
36.54
19.13
82.43
48.91
77.99
72.59
50.88
66.21
32.04
53.99
55.17
39.33
39.73
49.03
76.46
49.60
78,46
73.28
49.96
65,76
30.10
Lluna3- Instruct-70B
52.95
52.52
37.5
36.96
51.34
76.98
41.24
76.60
69:61
49.72
48.59
36.23
I LJuma3- Insmai-Instrict-- Stage-2
$6.80
$7.88
40.22
40.58
50.66
76.44
51.48
82.75
80.34
56.21
67.57
32.58
Samma3-CaaQQA-155221
57.14
58.25
41.26
38.82
51,40
55.63
32.31
Table 15: Studies on directly applying stage-2 instruction tuning to the powerful Llama3-Instruct models.
In Table 14, we report zero-shot results of all ChatQA models on CHATRAG BENCH. ChatQA models generally give great improvements compared the corresponding SFT/Chat/Instruct counterparts.
In Table 15, we study the effectiveness of directly applying stage-2 instruction tuning to the powerful Llnmaa-Instruct models. Llama3-Instruc models can be considered as those that have already finished the Stage-1 SFT training. They use a much larger dataset than the stage-1 training for Llama3- ChatQA-1.5. However, find that Instrat-Isttuut-81 Llama3-CharQA-LL.. -.5-..------------ This further suggests that the stage-2 instruction tuning plays a key role in improving model's capability in conversational QA and RAG.
+ Stage-2" performs comparable to
L Training Details
For all ChatQA models, in stage-1 SFT, we use learning rate of 5c-6, and train 1000 iterations with global batch size of 128; and in stage-2 instruction tuning. we use learning rate of 3e-7, and train 3300 iterations with a global batch size of 64. We use 256 NVIDIA A100 GPUs for training ChatQA-1. -.00-00 and Llama3-Cha(AA-1.555 training and around six hours for the stage-2 training. We use 64 NVIDIA A100 GPUs for training ChatQA-1.0-78 and llama3-ChaQAA- 1.5-8B models, and it takes around one and half hours for stage-1 I training and around three hours for stage-2 training.
models, and takes around three hours for stage-
37

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have comprehensive experimental results in & 6.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our models at the beginning of the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
38

---

This work does not propose any theory assumptions and does not include theoretical results.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all the details of the data collection for training and evaluations, which can be found in section 3, 4, 5 and appendix C, D, E and H. We also open-source model weights, training data, evaluation data, and scripts for reproducing our results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
39

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: open-source model weights, training data, evaluation data, and scripts for reproducing our results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / httpssscccppbccccceeeeeeeecc guw/guiiesss CoodeSubmis ssi assoonPoicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Dataset details can be found in section and Appendix F. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: All results are zero-shot and deterministic (generations are using greedy search, retrieval scores are also deterministic). Hence, the results will not have any variances.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
40

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the compute resources information in the training details section of the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics stttps://neurpp::
Caabllcc/Eticshuute ines?
Answer: [Yes]
Justification: Our research is conducted in the paper conform, in every respect, with the
NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both potential societal impacts and negative impacts at the begin- ning of the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
41

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We include licenses of the datasets in the Appendix F.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwode..coo..... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
42

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: provide details explanations how we collect human-annotated data and synthetic data in the Appendix D and Appendix J.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the datasct/codd/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
43

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
44


Reviews:
[{"review_id": "MyDpw4N1Ro", "reviewer": "Reviewer_NmdV", "summary": "The paper introduces ChatQA, a suite of models designed to excel in retrieval-augmented generation (RAG) and conversational question answering (QA). The authors propose a two-stage instruction tuning methodology to bolster generative capabilities and a dense retriever optimized for conversational QA to improve retrieval effectiveness. Notably, the ChatQA models, even when built on foundation models perceived as weaker than GPT-4, demonstrate superior performance on RAG and conversational QA tasks, surpassing GPT-4 in certain benchmarks. The paper further contributes the ChatRAG Bench, a comprehensive benchmark for evaluating RAG and conversational QA models, and releases model weights, training data, the ChatRAG Bench itself, and the retriever to the community.\n\nSoundness:\n\nThe technical claims, experimental methodology, and research design in this paper are well-supported and sound. The central claims are convincingly backed by extensive experimental results and comparisons to established baselines, including GPT-4. The ablation studies further validate the efficacy of the proposed two-stage fine-tuning approach and the significance of the curated datasets. The paper demonstrates a meticulous and rigorous approach, ensuring the reliability and reproducibility of its findings.\n\nPresentation:\n\nThe paper exhibits a clear and well-organized presentation style. The writing is lucid, and the technical concepts are effectively conveyed. The authors adequately contextualize their work within the landscape of prior research, highlighting the novel contributions of their approach. Overall, the paper is well-written and easy to follow, making it accessible to a broad audience.\n\nContribution:\n\nThe paper makes a significant contribution to the field of conversational QA and RAG. The proposed ChatQA models push the boundaries of performance, even outperforming GPT-4 in certain benchmarks. The open-sourcing of model weights, data, and the ChatRAG Bench fosters transparency and collaboration, promoting further advancements in the research community. The paper's findings challenge prevailing assumptions about the necessity of relying on synthetic data from OpenAI GPT models, opening avenues for innovative training strategies.", "questions": "How does the proposed two-stage instruction tuning method compare to other state-of-the-art instruction tuning approaches in terms of efficiency?", "limitations": "The paper does not identify any potential negative societal impact, which may be worth exploring in future work, particularly given the potential for misuse of powerful conversational QA models.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The paper introduces a novel two-stage instruction tuning methodology that significantly enhances the context-aware and RAG-based QA capabilities of LLMs. It also proposes an effective dense retriever optimized for conversational QA.\n2. The research is meticulously conducted, with rigorous experiments and comprehensive evaluations. The ablation studies provide valuable insights into the contributions of various components of the proposed approach.\n3. The paper is well-written and clearly structured, making it easy to follow the authors' line of reasoning. The technical concepts are presented in an accessible manner.\n4. The paper's findings are impactful, demonstrating that state-of-the-art performance in conversational QA and RAG can be achieved without reliance on synthetic data from OpenAI models. The open-sourcing of resources further amplifies the significance of this work.", "weaknesses": "The paper primarily focuses on evaluating the \"unanswerable\" scenario using a small set of samples. A more extensive evaluation involving diverse \"unanswerable\" scenarios would enhance the robustness of the findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your detailed comments and feedback. We will address your questions below.\n\n> “The paper primarily focuses on evaluating the \"unanswerable\" scenario using a small set of samples. A more extensive evaluation involving diverse \"unanswerable\" scenarios would enhance the robustness of the findings.”\n\n- Thank you for your suggestion. We will conduct further studies on “unanswerable” scenarios in future work. We also believe that the open LLM research community needs to invest more in this area. The leading proprietary LLMs also struggle with balancing hallucination and incorrect refusal answers. For example,\n  - “Previous Claude models often made unnecessary refusals that suggested a lack of contextual understanding.” from https://www.anthropic.com/news/claude-3-family\n\n\n> “How does the proposed two-stage instruction tuning method compare to other state-of-the-art instruction tuning approaches in terms of efficiency?”\n\n- We believe our ChatQA training requires much smaller compute resources compared to state-of-the-art instruction tuning approaches (e.g., Llama3-instruct) that use an enormous SFT dataset and even need iteratively online SFT training. \n- For all ChatQA models, we perform one epoch training on a relatively small SFT dataset (128K samples) in our stage-1 training. We use a global batch size of 128 and fine-tune the base model with 1000 iterations. In our stage-2 training, we blend conversational QA, single-turn QA, and the SFT dataset in stage-1 with certain ratios and further fine-tune the model with 3000 iterations using a global batch size of 64. Therefore, in total, our model consumes 128,000 + 64 * 3000 = 320,000 samples, which are much fewer than Llama-3 model alignment training, which takes more than 10M samples.\n\n\n> “The paper does not identify any potential negative societal impact, which may be worth exploring in future work, particularly given the potential for misuse of powerful conversational QA models.”\n\n- Thanks for pointing this out. We have discussed the potential negative impacts in page 22 (right before the Appendix), and the potential misuse of ChatQA models has been included. But, we will put them right after conclusion to make them easier to find.\n\n\nWe really appreciate your review and hope our response addresses all your questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p18O1ochiI", "reviewer": "Reviewer_DkJ4", "summary": "This paper introduces ChatQA, a suite of models that outperform GPT-4 on retrieval-augmented generation (RAG) and conversational question-answering (QA) tasks. The key contributions of the paper include:\n- A two-stage instruction tuning method that improves RAG performance.\n- A dense retriever optimized for conversational QA that performs comparably to state-of-the-art query rewriting models while reducing deployment costs.\n- CHATRAG BENCH, a comprehensive benchmark comprising 10 conversational QA datasets for evaluating RAG, table-related QA, arithmetic calculations, and unanswerable questions.\n- ChatQA-1.0-70B, built on Llama2, which slightly outperforms GPT-4-0613 and GPT-4-Turbo on CHATRAG BENCH without relying on synthetic data from OpenAI GPT models.\n- Llama3-ChatQA-1.5-70B, which surpasses GPT-4-Turbo-2024-04-09 by a good margin.\n- Good performance on single-turn QA and RAG benchmarks, with Llama3-ChatQA-1.5-70B outperforming existing frontier RAG models like Command R+.", "questions": "1. How does the performance of ChatQA models scale with model size? Is there a point of diminishing returns? This analysis could offer insights into the trade-offs between model size and performance.\n2. Are there specific limitations of the current approach that you think are most important to address apart from expanding to code-related tasks or math reasoning tasks?\n3. What do you see as the most promising avenues for future work based on your findings? \n4. Do you have any explanation or intuition as to why incorporating more unanswerable samples beyond 1.5K leads to lower accuracy scores in most of the tasks in Table 11?\n5. You mention that fine-tuning a single-turn query retriever performs comparably to state-of-the-art query rewriting. Can you elaborate on the trade-offs between these two approaches beyond computational cost?", "limitations": "They discuss the limitations at the beginning of the Appendix (Line#908-914).", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper demonstrates several notable strengths across several dimensions:\n- The two-stage instruction tuning method is a novel approach to enhancing RAG performance, combining supervised fine-tuning (SFT) with context-enhanced instruction tuning.\n- The development of a dense retriever optimized for conversational QA without relying on query rewriting is an innovative solution to a common challenge in multi-turn QA systems.\n- The creation of CHATRAG BENCH as a comprehensive evaluation suite for conversational QA and RAG tasks is a good contribution to the field.\n- The empirical results are robust, with the ChatQA models outperforming strong baselines, including GPT-4, across multiple datasets and tasks.\n- The ablation studies provide valuable insights into the contributions of different components of the proposed method.\n- The study on handling \"unanswerable\" scenarios addresses an important challenge in QA systems, contributing to the development of more robust models.", "weaknesses": "- The study primarily focuses on Llama2 and Llama3 of sizes 7B/8B, 70B  as base models. Including a wider range of base models could provide insights into the generalizability of the proposed methods. Also, a detailed analysis of how performance scales with model size could offer insights into the trade-offs between model size and performance.\n- The paper doesn't address potential ethical implications or biases that might be present in the developed models or the CHATRAG BENCH. A brief discussion on this would enhance the paper's comprehensiveness.\n- While the paper provides extensive quantitative results, it doesn't include a detailed qualitative analysis of the types of errors made by the models. This could offer insights into areas for future improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your detailed comments and feedback. We will address your questions below.\n\n---\n> “The study primarily focuses on Llama2 and Llama3 of sizes 7B/8B, 70B as base models. Including a wider range of base models could provide insights into the generalizability of the proposed methods. Also, a detailed analysis of how performance scales with model size could offer insights into the trade-offs between model size and performance.” and “Question 1. How does the performance of ChatQA models scale with model size? Is there a point of diminishing returns? This analysis could offer insights into the trade-offs between model size and performance.”\n\n- Many thanks for your comment. We actually have results on a wider range of base models, including in-house pretrained GPT-{8B, 22B} (pretrained on 3.5T tokens), Llama2-{7B, 13B, 70B}, and Llama3-{8B, 70B}. The full results of all these models are in Appendix K (Table 14) due to the page limit. \n- Thanks for providing this analysis suggestion. In Table-14 (Appendix K), we have studied the model size of 7B, 13B, 22B, 70B for ChatQA-1.0. Note that the 22B model comes from our in-house GPT-22B, and the rest models are from Llama2. We put the ChatRAG average scores of these ChatQA models as follows:\n\n| Models | ChatRAG |\n| ---------- | ------- |\n| ChatQA-1.0-7B  | 46.96 |\n| ChatQA-1.0-13B | 50.27 |\n| ChatQA-1.0-22B | 53.01 |\n| ChatQA-1.0-70B | 53.89 |\n\n- We find that the performance has a large boost from ChatQA-1.0-7B to 13B, and from ChatQA-1.0-13B to 22B. However, the improvement becomes marginal from ChatQA-1.0-22B to 70B. Therefore, a rough estimate suggests that a 22B model might strike a good balance between the model size and performance. But, we believe that more thorough studies are needed to analyze this trade-off in detail.\n\n---\n> “The paper doesn't address potential ethical implications or biases that might be present in the developed models or the CHATRAG BENCH. A brief discussion on this would enhance the paper's comprehensiveness.”\n- Thanks for this suggestion. We will include this discussion in the paper.\n\n---\n> “While the paper provides extensive quantitative results, it doesn't include a detailed qualitative analysis of the types of errors made by the models. This could offer insights into areas for future improvement.”\n- We put case studies in Appendix I due to a page limit. We study some errors made by ChatQA-1.0-13B, ChatQA-1.0-70B, GPT-3.5-Turbo, and GPT-4. We find that ChatQA models are robust in text-based information seeking scenarios, while it will sometimes make mistakes for tabular reasoning questions. We also find that the larger ChatQA model (e.g., ChatQA-1.0-70B) is able to correct the typo in the question or context, while the smaller one (e.g., ChatQA-1.0-13B) might fail to do so.\n\n---\n> “Are there specific limitations of the current approach that you think are most important to address apart from expanding to code-related tasks or math reasoning tasks?”\n- Thank you for raising this question. We believe there are several limitations to the current approach and many opportunities to extend its capabilities:\n  - ChatQA is tested with RAG tasks involving single-step retrieval. It would be interesting to explore and extend its RAG capabilities to handle multiple retrieval steps, requiring joint reasoning to provide accurate answers.\n  - The current version of ChatQA-1.5 only supports an 8K context window. Extending this context window would be beneficial for long-context summarization tasks.\n\n---\n> “What do you see as the most promising avenues for future work based on your findings?”\n- We observe very strong results from relatively small ChatQA models. For example, Llama3-ChatQA-1.5-8B achieves results comparable to GPT-4-Turbo. A promising direction would be to explore the effectiveness of even smaller models (e.g., 1B) through model distillation. Smaller models are ideal for deployment on mobile devices, and conversational QA and RAG are important use cases for users.\n\n---\n> “Do you have any explanation or intuition as to why incorporating more unanswerable samples beyond 1.5K leads to lower accuracy scores in most of the tasks in Table 11?”\n- We conjecture that it might be attributed to the data quality of unanswerable samples.  For HumanAnnotatedData, we asked annotators to identify all parts of the context locations that are relevant to the user’s question. And we construct unanswerable samples by deleting the text from the corresponding locations in the context. There is a possibility that the relevant context for a few questions is not entirely removed, leading to incorrect unanswerable training samples. Hence, a better data filtering strategy can be applied to improve the data quality and potentially improve the overall accuracy.\n\n---\n> “You mention that fine-tuning a single-turn query retriever performs comparably to state-of-the-art query rewriting. Can you elaborate on the trade-offs between these two approaches beyond computational cost?”\n- Query rewriting requires making API calls to powerful LLMs to rewrite the query, which incurs additional costs each time a conversational query comes. Alternatively, we can build a strong instruction following model or query rewriting model ourselves, but in additional computational resources, large amounts of datasets need to be collected for training.\n- Fine-tuning a single-turn query retriever, however, requires careful data collection on conversational query and context pairs. We showcase that collecting around 7k conversational dataset (around 5 conversational query and context pairs for each conversation) is good enough to build a powerful multi-turn query retriever that works comparable to the GPT-3.5-Turbo query rewriting approach.\n\n---\n> “They discuss the limitations at the beginning of the Appendix (Line#908-914).”\n- Thanks for pointing this out. We will put them right after conclusion to make them easier to find.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "11gaytuzDe", "reviewer": "Reviewer_ynR1", "summary": "This paper explore RAG in conversational QA senarios. It proposes a two-stage instruction tuning method for conversational QA in a RAG manner, accompanied by a comprehensive benchmark. The training of LM involves two stages: (i) SFT on dialogue and QA datasets and (ii) context-enhanced instruction tuning which includes SFT on both synthetic and existing datasets. Similar strategies are applied to tune the retriever for multi-turn QA scenarios. The experiments demonstrate that the proposed ChatQA model surpasses various existing large language models in performance.", "questions": "1. Is there any difference between the concepts of \"multi-turn QA\" and \"conversational QA\" in this work? These terms appear to be used interchangeably.\n\n2. Could you provide justification for the proposed retriever tuning strategies?\n\n3. Does the conversational QA scenario differ from traditional QA primarily because it includes additional context? If so, it would be intriguing to assess the impact on QA accuracy when this context is removed. Furthermore, comparing the importance of this context against the content retrieved in the RAG may provide insightful observations.", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This paper effectively underscores the significance of RAG in chat-based QA scenarios, a critical area that has been overlooked in previous research. It offers a detailed and concise overview of the current research landscape, scenario positioning, and existing challenges, providing valuable insights into these areas. Additionally, the proposed benchmark presented in the paper could be valuable for advancing research in this domain.", "weaknesses": "- There is a lack of technical novelty. The paper focuses on fine-tuning a language model using an existing dialogue dataset and self-created data for this scenario. While the approach is clearly outlined, it would be helpful if the authors could further elaborate on the innovative aspects of their methodology to highlight its novelty.\n\n- The paper lacks clear definitions and formalizations of the tasks that need to be addressed, which obscures the distinctions between its concepts and existing works. It took me some time to realize that the \"instruction tuning method\" pertains solely to the construction of training data and is conducted without retrieval. This leads to questions about the method's relevance to the subsequent RAG framework. The connection between the proposed method, which is tailored for LM in chat-based QA, and the RAG remains unclear and needs better delineation. \n\n- Lack of essential ablation studies. Numerous choices regarding training data selection and the construction of synthetic data are only briefly explained. These choices, while seemingly reasonable, necessitate thorough ablation and comparison with standard processes to validate their effectiveness. Additionally, although several strategies for tuning the retriever for this scenario are proposed, there is a lack of analysis and ablation to justify their usefulness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments and feedback. We will address your concerns and questions below.\n\n> “There is a lack of technical novelty…it would be helpful if the authors could further elaborate on the innovative aspects of their methodology to highlight its novelty” \n\n- The innovative aspects of the methodology include: i) We propose a two-stage instruction tuning method, with one stage focusing on general instruction-following capabilities and another focusing on document-grounded and contextualized QA. The effectiveness of this method is validated in ablation study (see Table 3). ii) We introduce a method to obtain a retriever for conversational QA that can utilize the entire conversation history for accurate retrieval from the documents. It performs as well as the SOTA ChatGPT-based query rewriting model but significantly reduces deployment costs.\n\n- In addition to the innovative aspects of methodology,  we think that technical novelty can also come from experimental results, and this work certainly presents a good amount of novel results / findings. For example, i) ChatQA-1.5-70B model is built on a weaker foundation model (Llama3-70B) compared to GPT-4-Turbo. However, it can significantly surpass GPT-4-Turbo (Avg. 58.25 vs. 54.03) with carefully designed instruction tuning recipe, while not utilizing any synthetic data from ChatGPT models.  ii) We demonstrate that the proposed multi-turn query retriever can be as good as the ChatGPT based query rewriting model in conversational RAG setting.  iii) We are one of the first open-source efforts to show compelling results for unanswerable cases (outperforming GPT-3.5, but slightly worse than GPT-4) in a zero-shot QA setting. We believe that novel results and findings are as important as novel ideas for LLM research.\n\n- Last but not least, we open-source the ChatQA models, retriever model, training data, and ChatRAG Bench. We believe this contribution holds significant value for the research community.\n\n----\n> “The paper lacks clear definitions and formalizations of the tasks that need to be addressed, which obscures the distinctions between its concepts and existing works. It took me some time to realize that the \"instruction tuning method\" pertains solely to the construction of training data and is conducted without retrieval. This leads to questions about the method's relevance to the subsequent RAG framework. The connection between the proposed method, which is tailored for LM in chat-based QA, and the RAG remains unclear and needs better delineation.”\n\n- Thanks for raising the question. The goal of the ChatQA work is to address context-rich conversational QA, where the context can involve long documents requiring RAG. In the final evaluation, we use five long-document datasets that need retrieval and five short-document datasets where the documents can fit into LLM prompts. Detailed information can be found in Section 5.2. We will clarify the tasks further in our final manuscript.\n\n- For the construction of instruction-tuning data, we indeed tried to include training data with top-5 retrieved chunks from long documents. We need to put the results in Appendix A.1 due to the space limit. We find that this can slightly improve the RAG results on long document datasets  but it degrades the performance on short documents datasets. Note that, the context-enhanced instruction tuning (stage-2) empowers the ChatQA model to effectively integrate useful information from the “context” for response generation. This context can either be user provided short document, or retrieved top-k contexts from provided long documents. \n\n----\n> “Lack of essential ablation studies. Numerous choices regarding training data selection and the construction of synthetic data are only briefly explained. These choices, while seemingly reasonable, necessitate through ablation and comparison with standard processes to validate their effectiveness.”\n\n- Thanks for your reminder. In Table 3, we have conducted ablation studies on the key components of our proposed training strategies. For example, the effectiveness of adding single-turn QA datasets, the multi-turn QA dataset, stage-1 and stage-2 training, and the comparison between the synthetic and human-annotated multi-turn QA dataset.  \n- In Appendix H.2, we perform ablation studies on selecting the number of unanswerable samples in the instruction tuning dataset (Table 11).  \n- In Appendix A, we provide more ablation studies on utilizing top-k retrieved chunks for instruction-tuning (Table 6).\nWe will highlight the list of ablation studies in the main text for the final version of the paper. \n\n----\n> “Lack of analysis and ablation to justify the usefulness of the proposed retriever tuning. Could you provide justification for the proposed retriever tuning strategies?”\n\n- Existing retrievers, such as Dragon and E5, are designed for single-turn queries and struggle to generalize well to questions within conversations that reference previous dialogue, such as follow-up questions. To address this limitation, our retriever tuning strategy involves further fine-tuning the single-turn retriever using pairs of conversational queries and corresponding contexts (passages) from documents, utilizing the same contrastive loss. This approach enhances the retriever's ability to handle conversational queries more effectively.\n- In addition, we provide ablation studies and analyses on different strategies for tuning the retriever and query rewriting method in Appendix E.2 (Table 9), which further illustrates the effectiveness of our proposed retriever tuning strategy.\n\n----\n> \"Any difference between the concepts of \"multi-turn QA\" and \"conversational QA\" in this work?\"\n- Multi-turn QA and conversational QA are interchangeable terms.\n\n----\n----\nWe will respond to your intriguing final question in the follow-up comment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p0oPfPfg03", "reviewer": "Reviewer_A2Vo", "summary": "This paper proposes a family of fine-tuned models (ChatQA) that surpass GPT-4 on conversational QA and RAG. It introduces a two-stage instruction fine-tuning method to enhance the model’s capability of using the retrieved context for generation. In addition, it shows that fine-tuning a single-turn retriever (Dragon) on human-annotated data can closely match LLM-based query rewrites, thereby eliminating computational and API costs. Lastly, this paper collects a ChatRAG benchmark for evaluating RAG, table-related QA and arithmetic calculations. Experiments on the ChatRAG Bench show that the proposed ChatQA models outperform or closely match the strong generalist model GPT-4.", "questions": "1. L163, in the stage-2 context-enhanced instruction tuning, it seems that you again use all of the SFT datasets from stage-1. Will it lead to overfitting? If you want to maintain the instruction-following ability, why not merge all data from stage-1 and stage-2 and then do multi-task tuning?", "limitations": "N/A", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Overall, the writing of the paper is good. The experiments and ablation studies are sound and comprehensive, demonstrating the effectiveness of the two-stage instruction fine-tuning method and the data curation recipe.\n2. The open-sourced training, the data curation recipes, and the use of the public foundation models (Llama2 and Llama3) could be valuable and beneficial for the community in chasing proprietary models.", "weaknesses": "1. Training an open foundation model with curated instruction data is not new. The paper could be improved if it demonstrates why the selected mixture of training data is effective, and training on them could match GPT-4.\n2. While the ablation studies show the effectiveness of the collected data in training ChatQA models, I believe more fine-grained data selection and analysis could potentially further improve the performance [1][2].\n\n[1] How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources, 2023\\\n[2] A Survey on Data Selection for Language Models, 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and feedback. We will address your questions below.\n\n> “1. Training an open foundation model with curated instruction data is not new. The paper could be improved if it demonstrates why the selected mixture of training data is effective, and training on them could match GPT-4.”\n- Thank you for the suggestion. ChatQA models show compelling results because our collected instruction tuning datasets are designed for conversational QA and RAG.\n  - We collected a conversational QA dataset where the document context is provided for each sample, enabling our models to learn how to locate relevant information and generate accurate answers. Additionally, the data samples are in a conversational format, allowing our models to answer follow-up questions. Furthermore, we include several single-turn QA tasks with document contexts to enhance information-seeking capabilities. To enable our model's tabular understanding and reasoning abilities, we incorporate TAT-QA, which provides a rich source of tables and QA samples involving mathematical calculations and reasoning. \n  - In Table 3, we present comprehensive ablation studies demonstrating the effectiveness of these single-turn QA datasets and the conversational QA dataset, and the comparison between the synthetic and human-annotated conversational QA dataset.  \n- We will elaborate further and provide more quantitative results on why the selected datasets are effective in the final version of this paper.\n\n\n> “2. While the ablation studies show the effectiveness of the collected data in training ChatQA models, I believe more fine-grained data selection and analysis could potentially further improve the performance [1][2].\n\n  > [1] How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources, 2023\n\n  > [2] A Survey on Data Selection for Language Models, 2024”\n\n- Thanks for your great suggestion. We will cite and discuss the papers you mentioned in related work. We will look into more detailed data selection and analysis in our future work.\n\n> “Questions:  1. L163, in the stage-2 context-enhanced instruction tuning, it seems that you again use all of the SFT datasets from stage-1. Will it lead to overfitting? If you want to maintain the instruction-following ability, why not merge all data from stage-1 and stage-2 and then do multi-task tuning?”\n- This is a good question. This two-stage approach is akin to curriculum learning, which first equips the model with basic instruction-following capabilities and then enhances its conversational QA and RAG capabilities. In stage-2, we blend all of the SFT datasets from stage-1 again to ensure the model does not forget its instruction-following capabilities. \n- In Table 3, we conducted this ablation study by training on all datasets merged from stage-1 and stage-2. The result is denoted as \"w/o stage-1\" because it is essentially the same as performing only stage-2 training, which uses all the datasets. We found that this resulted in an overall drop in performance, demonstrating the effectiveness of the proposed two-stage curriculum training.\n\n\nThank you once again for your review. We hope our response satisfactorily addresses all your questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces ChatQA, a set of models designed to excel in retrieval-augmented generation (RAG) and conversational question answering (QA), demonstrating a good performance that, in some cases, surpasses GPT-4. The authors present a two-stage instruction tuning method and a dense retriever specifically optimized for conversational QA, alongside a benchmark called ChatRAG Bench. The study is complemented by the release of model weights, training data, and the benchmark to the community, which could potentially foster further research in this domain.

The technical rigor of the paper is generally solid, with well-supported claims and extensive experimentation that effectively validates the proposed methods. The ablation studies reinforce the methodology's efficacy, particularly the two-stage fine-tuning approach. However, some reviewers have raised concerns about the lack of technical novelty, pointing out that the approach relies heavily on existing techniques and does not significantly advance beyond current methods. 

The presentation of the paper is clear and well-structured, making the complex concepts accessible to a broad audience. Despite this, there are concerns regarding the lack of in-depth analysis of the types of errors made by the models and the absence of discussions around ethical implications or biases. These omissions limit the paper's comprehensiveness and suggest areas for improvement, particularly in providing more detailed qualitative insights and addressing potential societal impacts.
Decision: Accept (poster)