Orca: A Few-shot Benchmark for Chinese Conversational Machine Reading Comprehension
Authors: Nuo Chen
Hongguang Li
Junqing He
Yinan Bao
Xinshi Lin
Qi Yang
Jianfeng Liu
Ruyi Gan
Jiaxing Zhang
Baoyuan Wang
Jia Li
Venue: EMNLP 2023 Findings (2023)
Keywords: dataset
conversational machine reading comprehension

Abstract: The conversational machine reading comprehension (CMRC) task aims to answer questions in conversations, which has been a hot research topic in recent years because of its wide applications. However, existing CMRC benchmarks in which each conversation is assigned a static passage are inconsistent with real scenarios. Thus, model's comprehension ability towards real scenarios are hard to evaluate reasonably. To this end, we propose the first Chinese CMRC benchmark \textbf{Orca} and further provide zero-shot/few-shot settings to evaluate model's generalization ability towards diverse domains. We collect 831 hot-topic driven conversations with 4,742 turns in total. Each turn of a conversation is assigned with a response-related passage, aiming to evaluate model's comprehension ability more reasonably. The topics of conversations are collected from social media platform and cover 33 domains, trying to be consistent with real scenarios. Importantly, answers in Orca are all well-annotated natural responses rather than the specific spans or short phrase in previous datasets.
Besides, we implement three strong baselines to tackle the challenge in Orca. The results indicate the great challenge of our CMRC benchmark.

Orca: A Few-shot Benchmark for Chinese Conversational Machine Reading Comprehension
Nuo Chen!, Hongguang Li², Junqing He², Yinan Bao", Xinshi Lin², Qi Yang', Jianfeng Liu?, Ruyi Gan Jiaxing Zhang', Baoyuan Wang*t, .Jia Lift Hong Kong University of Science and Technology (Guangzhou), Hong Kong University of Science and Technology "Xiaobing.AI
CCNLL, IDEA, Shenzhen, China
Institute of Information Engineering, Chinese Academy of Sciences
Abstract
the given text passage/paragraph and respond ap- propriately to set of questions in a conversation. However, despite notable successes in a variety of types and on a sizable scale, there are still a num- ber of important challenges that have rarely been mentioned in prior CMRC efforts:
The Conversational Machine Reading Compre- hension (CMRC) task aims to answer questions in conversations, which has been hot research topic because of its wide applications. How- ever, existing CMRC benchmarks in which each conversation is coupled with static pas- sage are inconsistent with real scenarios. In this regard, is hard to evaluate model's com- prehension ability towards real scenarios. In this work, we propose the first Chinese CMRC benchmark Orea and further provide zero- shot/few-shot settings evaluate model's gen- eralization ability towards diverse domains. We collect 831 hot-topic driven conversations with 4,742 turns in total. Each turn of conversa- tion ussigned with pepponse-related passage, aiming evaluate model's comprehension abil- ity more reasonably. The topics of conversa- tions are collected from social media platform and cover 33 domains, trying to be consistent with real scenarios. Importantly, answers in Orca are all well-annotated natural responses rather than specific spans or short phrases in previous datascts. We implement two strong frameworks tackle the challenge in Orca. The results indicate there is substantial room for improvement for strong baselines such as ChatGPT on our CMRC benchmark. Our codes and datasets are available at: https: //githubs com/ wwccherppku//rcaaaaa
(1) Conversations in existing CMRC datasets are only grounded on a singlc/static document pas- sage. Each time a question asked, is necessary to consider the passage as the sole source of in- formation. In practice, the natural multi-turn QA is driven by a topic with dynamic knowledge and diverse domains rather than restricted to static evidence passage. Thus, there is a great discrep- ancy between the pattern of training data and real application scenarios.
(2) Many annotated answers in these datasets are restricted span texts in the provided passages. For instance, significant proportion of CoQA an- swers are short phrases from the document (Cam- pos et al., 2020). While span texts straightly taken from the original document and basic phrases like "yes/no" might be seen as ground-truth answers, they do not sound like natural responses in real-life. Besides, current Large Language Models (LLMs) primarily generative in nature. Relying solely on these fixed-length span answers poses limita- tions in effectively evaluating the quality of LLMs through standard evaluation metrics.
1 Introduction
(3) Previous CMRC benchmarks are only built on English corpus, limiting the progress of CMRC in other languages like Chinese. Moreover, ques- tions and knowledge embedded within previous datasets are mostly antiquated, potentially already encompassed within the pre-training corpus of LLMs. As a result, they fail to adequately eval- uate the adaptability and understanding abilities of LLMs for new questions and emerging knowledge.
Conversational Machine Reading Comprehension (CMRC) has aroused an increasing research inter- est in recent years. Accordingly, lots of large-scale CMRC benchmarks (You et al., 2021; Chen et al., 2023b; Campos al., 2020: Zaib et al., 2022) have been proposed, such as CoQA (Reddy et al., 2019), QuAc (Choi et al., 2018). Technically, CMRC aims endow a machine with the ability understand
To this end, we collect Orca, the first few-shot Chinese CMRC benchmark. Concretely, we hire professional annotators to annotate the following
"chennuo26 6 6.mma I com, Work done when Nuo in- temed at AI. refers to conresponding authors.
Work

---

Toplc: 12
China Space Station waiting business Dowatt K4t Technology
f9 2 What's the China Station (Facould)
it Siaeiinn (5 385 cham space Stanee Herdss Art Bam - RYRAARS
[R My Wale Chen space Staion chatary Space baice, logion nane clasa Space
R: 2 Pt. It nome Taangong Space Station
staken sase stabe yazze uaker conaraction Peppic Aopublic Clasa (1S0)) whics eepesicd
53 Have achiived long-tomm
Paltarns:
3221 (e #o g W0 Retteee S
renideecee? (Contirmation)
ALA As 60 Genber in 3831, arrveaze excoutiity emaued the Tambe alcrefChis Spece Statiee, seae China Space Statio lasstatte sfpegte:
R: (PERRRRRR 4AA 137
Chita Space Station has opened
provecoc
on om longsiemm presenoe people.
Fet turmun
www. ? llow
stations the world' (Pactuid)
Edi (BR SRaft rery sym -ith 1hth ty Au Le
Mitshyaaa 30 "yerp MRA 20 Cammats lavel 20111 are
R: 19 : 1190,, These are two space vations world.
spcsoo txcladng Unated Hamaa othr
sdcpeicsity beik by Clara.
Q: 25t 10 ? Why barld this? (Cassal)
Pafmury
25 #HIKKK 7- ta 12 SE. is convenien: for people to work in space for long time, and there aood to retura the carth rosouace roplristtio shoet tanz.
Salds 1 2029 1938 dertt aaat (677
typn spor each fos replra in sboet
line, and
Figure 1: An abridged conversation in Orca. Q. R. and P means query, response, and passage respectively.
components in Orca: 1) Topic, which consists of several sentences to drive the whole conversation; 2) Domain, only one or two words indicate the specific field to which the content of the conversa- tion belongs; 3) Conversations, where each turn is assigned golden knowledgeable passage. Im- portantly, the question and response in each turn are human-crafted, Icading more coherent con- versations. And response- related passages are also manually selected from the search engine. Figure 1 shows an abridged conversation in Orca. In total, we annotate 831 conversations and 4742 turns in Orca. We deliberately limit the size of our dataset to build strong CMRC models under few-shot/zero- shot settings, and without the need to collect data for each target domain.
ests, and has never been included in earlier bench- marks. Hence Orca can serve as a comprehensive tool for assessing the adaptability of current LLMs to novel knowledge and emerging questions. More- over, good results on Orca are of practical interest. 4) We carefully annotate conversations across 33 domains, such as Society, People, Celebrity, Book, Finance. In contrast, the most commonly-used datasets, CoQA only has 7 domains and DoQA contains 3 domains. The variety of data domains makes Orea a better testhed evaluate the gen- eralization of CMRC models. Table shows the comparison between Orca and typical QA datasets. In this paper, we implement two strong frame- works to tackle the challenge in Orca: 1) In-Context Learning with LLMs like GPT-3 (Brown et al., 2020) and ChatGPT; 2) fine-tuning with language models that are less than I billion. In our experi- ments, the latter includes both end-to-end training and three-stage training. We conduct our experi- ments under both zero-shot and few-shot settings. The results indicate that Orca provides challeng- ing evaluation testbed for the CMRC systems.
Our dataset has the following salient features: 1) Each turn is assigned with a passage as the knowl- edge source, i.e., the given passages are dynam- ically changed rather than only one topic-central passage. Orca gives more in-depth background knowledge about the specific topie in this manner, making the following-up dialogue more natural and diversified. Meanwhile, Orca poses greater challenge to existing methods since it necessitates modeling both dialogue history and dynamic evi- dence passages. 2) Answers at each turn are natural and informative responses from human annotation rather than certain spans in the provided passage. With this, we can both evaluate models' compre- hension ability and generation ability. 3) Dialogue topics in Orca are collected from November 2021 to November 2022 on Weibo', one of the most pop- ular social media platforms in China. This means that the collected data reflects real human inter-
2 Orca
2.1 Problem Definition
Generally, the conversational machine reading com- prehension task can be illustrated as: Given a con- versation at turn t, <P, Hi Qt, As>, where P is the evidence passage, He refers to the dialogue history at turn O. Ar denote the QA pairs at the current turn, respectively. The task requires the model to predict the answer A, based understanding the P, H4 and P. A, often refers the spans in P. In contrast, given a conversation at turn t, Orca formalizes the CMRC task as <T, Pr, H4 2, R,>,
Phttps: //mm weibo.cn. cn/

---

Datase
Conversation
Few-sho
Respatae-riittdd Passage
Answer Type
Language
Domaia
SQUIAD (2016)
Spin
Faglish
(3066)
Fere-fomm leat
Faglish
DuReade: (2017)
Fice text
Chinese
QuAC
Spin
English
ColZA (2019)
Free forn text
Lingish
7
Reptiaaed span
bnglish
Orea (ours)
Response
Table I: Comparison of benchmark with other typical QA datasets. Because Orea multi-domain CQA dataset rather than open domain, we don't t compare Orca with some open domain benchmarks. We don't report the domain diversity if the original papers don't tt mention it. Free-form denotes that the answers include span-text, abstractive text and "yes/no".
where T is the topic which drives the whole conver- sation, Pe represents the evidence passage of turn t, RR is the corresponding response. Of note, 1) Since Orea delivers an evidence passage for each round in the conversation, compared to other data sets, it demands the model to have a stronger ca- pacity to comprehend these dynamic knowledge; 2) Each R, is the human-annotated natural response rather than Ar, and thus, Orea requires sequence- to-sequence modeling; 3) H₂ in Orca is allowed to include passages of previous turns.
der to avoid prejudice brought about by individuals, recruit another annotator further check the selected topics. We also require the annotators disregard the topic of the same domain three times in a row, so as to increase the domain diversity. Se- lected topics in Orca are collected from November 2021 to November 2022 on Weibo, guaranteeing the timeliness and diversity of our data.
Conversation Collection After obtaining topics, we split them into five parts equally. Then, we di- vide the ten annotators into pairs (one questioner and one responder) and assign them the topics. We ask them to have conversation around each signed topic. The questioners are demanded ask following up questions as many as possible and responders are asked to give natural responses seriously. The conversation begins with the ques- tioner formulating a free-text question from the information on the chosen topic. To ensure that the responses are accurate, comprehensive as well as informative, we let responder use web search en- gines find knowledgeable passage that includes the information of answering this question.
2.2 Dataset Collection
As previously stated, high-quality dataset is re- quired for the development successful CMRC models. In our dataset, we include the following el- ements: 1) Topic, a collection of several sentences that serves as the conversation's 's central focus; 2) two words designate the specific field to which the content of the conver- sation belongs; 3) Conversations, where each turn is allocated a knowledgeable passage. In particu- lar, Orca collected in three stages: 1) Topic and domain collection; 2) Conversation collection; 3) Quality control. We introduce each of them in the following parts:
Domain where simply
Then, the responder needs to answer the ques- tion based on the manually selected passage and their knowledge. Responses required to be the free-form conversational text, such as sentence with demonstrative pronouns, making the expres- sion more fluent. To facilitate more natural and consistent conversations, the responder should pro- vide additional information about the answer or the knowledge about the topic to encourage the ques- tioner to continue with questions related to the lat- est question. Besides, encourage the questioner to find interesting questions from the recorded di- alogues about the topic on Weibo. Dialogues are ended when more than two unanswerable questions were asked or one of the partners decides to end the conversation.
Topic and Domain Collection With the goal of collecting data that can reflect real human needs and daily life, we treat the social media platform Weibo as the data source. More than 200 million Chinese users share things around them and talk about hot topics on Weibo every day. We employ ten annotators to find out and write down topics that they are interested from the hot-topic list 2 If the selected topic on the list is not a grammati- cally correct sentence, the marker needs to rewrite a few sentences to re-explain the topic. Meanwhile, the domain of this topic is annotated in Orea. In or-
hhttps: //s. ///.wwe wel ba. cww/too///ooooooo opp//ttp/too summa ry rycate realt timehot

---

Statistics
Support
Test
testbed that can evaluate models' ability towards real scenarios in a reasonable way.
Conversations
200
631
831
Turns
1115
3627
4742
3.1
Conversation Analysis
Turns per Conversations
Max.
18
14
18
Ave.
5.75
5.58
5.71
As shown Figure 2 (a), there are 33 different do- mains Orea, covering a large range of topics daily life, such as Sociery, People, Celebrity, Book, Finance. The variety of domains makes Orca more challenging. which can effectively evaluate model's generalization ability towards different domains. To evaluate model's generalization ability to- wards various domains, we extract 200 conversa- tions from Orea as the support set and treat the rest data as test set. Table 2 shows the statistics of Orca. Each conversation contains 5.71 turns on av- erage, which is closer to the number of QA pairs in a human conversation (Campos et al., 2020). There are about 91.9 tokens per passage and 16.35 tokens per response average, indicating the abundant information of manually retrieved passages.
Max.
790
822
822
Passage tokens
Ave.
92.35
90.45
91.90
Max.
19
16
19
Question tokens
Ave.
5.99
5.92
5.97
Max.
380
197
380
Response tokens
Ave.
16.31
16.50
16.35
Table 2: Statistics of collected data of Orca.
Quality Control To avoid bias and further con- trol the data quality, we have made some strategies to ensure the quality of conversations:
We employ another five annotators to check whether an inconsistent problem exists in the conversation. Then, annotators are demanded to revise the questions responses directly delete the turns that cause inconsistency.
In terms of the link between a question and its conversation history, we categorize questions as dependent or independent the conversation his- tory. According to our findings, only 19.9% of questions do not need coreference with the con- versational history and can be answered on their own. Explicit coreference indicators such as he, she, and it appears in over half of the queries (57.8)). These are either entities, people, build- ings or events mentioned in the conversation. The remaining 22.3% lack explicit coreference indica- tors but are indirectly related to an item or event, like simple following-up questions "Why this?", The analysis shows Orca poses challenging linguis- tic phenomena to build effective CMRC models.
Simultancously, gruff questions and responses are revised by annotators, aiming to guarantee natural expressions.
As our annotation work spans about one year, in order to keep the enthusiasm and work ef- ficiency of annotators, quality check by an- notators lasts for about one month. The best- performing annotators will be monetary re- warded, and the worst performers will be re- trained or even separated.
At last, we will manually re-check the quality to further ensure the high quality of Orca.
3.2 Query Analysis
All annotators are Chinese native speakers re- cruited from universities. The annotation costs about 15K US dollars in total, with 2.5 dollars per dialogue turn.
Query types in Orca can be classified into 5 streams: Factoid (39.1 1%), Causal (30.1%), List (13.0%), Confirmation (15.4%) and Hypothetical (2.4%).
Different from existing datasets (Choi et al., 2018; Reddy et al., 2019; Christmann et al., 2019; Saha et al., 2018; Chen et al., 2021b) in which most queries are Factoid, query types in Orea are more diverse. For example, factoid questions accounted for about 60% of CoQA. In contrast, 60.9% of the questions non-factoid questions Orca, where Causal, List, and Confirmation all account for a considerable part.
3 Dataset Analysis
As a result, Orca contains 831 conversations and 4,742 turns. For each conversation, there are 5.71 turns on average, and each turn is assigned with a knowledgeable passage. We randomly split the collected data into support (train) and test set. Con- cretely, the support set and test set contain 200 and 631 conversations, separately. Each turn of con- versation needs to be evaluated in experiments. In short, we aim to provide a challenging evaluation
Figure 2 (b) shows the distribution of the bigram prefixes of queries. We observe that most of the first word of queries are similar to other datasets,

---

A4 People (8.78%)
B Calebrity (7.58%)
1388 Rook (7.46%)
e Finance (6.02%)
Ball
many
did
Hide Society (29,84%)
old
How
Who
Wby
E TV Dama (2.89%)
Sports .4%5
Polities (4.57%)
Health (3.25%)
Show (2.77%)
Whutt
Wen
did
P
R Technology (2.29%)
Blg Intemational
E Miliory (1.81%)
arc
(2.65%)
(2.05%)
(1.33%)
(10,57%)
d08 Naima, Food set Funns 116 Fasiion. Rklh Travel.
Dipt- Campas ERB-Naaur 2013 tame: Wil--im. 2h.p D..--liiarry
-leliaa, 38% Weates Cagenn w En--nnmmimmma
(a)
(b)
Figure 2: (a) Distribution of domains in Orea. (b) Distribution of the bigram prefixes of querys in Orca. For ease of reading, English translation is only provided here.
like QuAC, CoQA and CSQA. However, queries beginning with "Why" and "Is" account for larger proportion compared with other datasets. We find that "Why" pattern is mainly related to queries with Causal type (eg. Why is he popular in China?) and "Is" pattern is mainly related queries with Confirmation type (eg. Is it a developed country?). The diversity and relatively uniform distribution of queries Orca provide conversation scenarios that are as realistic as possible.
(text-ddavinci-O00 ChatGLLM-1100 and the backbone.
ChatGPT (gpt3. (gttr.)- Concretely, given the task descriptions as prompts the train set, we validate the performances of these LLMs on the test set from Orca.
4.2 End-to-End
As most of the responses in Orca are free-form texts, rather than text spans extracted from the given passages, we build our end-to-end pipeline with several widely-used generative models: T5 and Bart. T5 (Raffel et al., 2020) is pre-trained encoder-decoder Transformer model which has achieved strong results on diverse set of bench- marks, such as question answering. We use a pre-trained T5 based on Chinese (Wang et al., 2022) with 784 million parameters for experiments. BART (Lewis et al., 2020) is pre-trained encoder- decoder Transformer model for text generation tasks which has 375 million parameters.
3.3 Response Analysis
According to the statistics all responses Orca, we observe that 69.30% of responses cannot be found as exact text spans from the passage. Only 30.70% % of responses have overlap spans in the given passage (like entities). It indicates that most of the responses in Orea are free-form expressions that are more colloquial, calling for models with strong generative ability.
In our implementation, optimize the model with the widely used negative log-likelihood loss during training. At turn a conversation, the optimized objective can formulated as:
4 Models
In this section, we introduce three strong base- lines to address the challenge Orca: 1) In- context learning with large language models; 2) fine-tuning including end-to-end training and three- stage training which consists of three modules: Query Rewriter, Passage Reader, and Response Rewriter (short for QPR).
C = -log(po(RA T,P., H4, 2+))
(1)
log(po(ralT,, P. Q.)).
4.1 In-context Learning
4.3 QPR Framework
Previous works have proved that directly utiliz- ing LLMs with in-context learning could lead promising performances in zero-shot. In- context learning means the learning procedure that the models learn to make predictions in target task via conditioning on a few input la- bel pairs In our experiments, we employ GPT-3
As explained in Section 1, human cognition pro- towards CMRC can be split into three steps. To this end, given the related passage of each turn, further design QPR that consists of Query Rewriter Passage Reader, and Response Rewriter. In this architecture, given question at turn 1, Query Rewriter module first rewrites the question Q, to a

---

Shots (Sessions)
Method
BLEU-1
BLEU-2
Distinct-1
Distinct-2
ROUGE-L
EM
In-Centext Learning
ChatGL.M- 1300
12.96
9.42
2.21
31.37
23.71
1.05
GPT-3 w/o P
25.41
18.81
5.27
44.84
34.02
2.04
O-shot
GPT-3
32,29
29.05
3.86
44.77
36.77
3.25
ChatGPT w/o P
23,84
17.37
2.68
32.87
31.22
1.38
ChatGPT
35.93
33.21
2.46
38.13
47.72
3,03
GPT-3
33.88
32.59
3.59
40.67
39.22
I-shot
ChatGPT
36.70
32.81
3.13
43.23
47.87
3.35
GPT-3
34.56
33.40
3.05
39.63
41.01
3.41
5-shot
ChatGPT
41.11
37.81
4.31
46.44
57.96
5.03
Pine-Tunimg
QPR
61.10
58.21
4.47
48.51
62.07
14.83
BART
24.72
21.37
3.46
39.72
26.79
1.54
5-shot
T5
42.41
39.70
3.65
41.95
47.39
6.89
BART
38.73
35,30
4.24
44.51
40.87
5.82
10-shot
T5
46.59
43.98
3.58
43.09
57.80
12.54
200-shot
BART
61.59
59.05
5.34
51.56
69.90
25.23
(full data)
T5
68.25
66.11
4.50
46.82
73.06
31.40
Table 3:
Comparison of baselines on Orca, which have significant improvement with p-value < 0.05. means we use conversations without passages to evaluate GPT-3.
"GPT-3 w/o P"
more natural text é. Then Passage Reader mod- ule is responsible for generating answers Ae based (T, Pt, He, 2t). At last, Response Rewriter module takes A, inputs and generates the final predicted response. Considering the fact that Orca lacks labels of re-written questions, we post-train these modules on some benchmarks, and then test the resulting models in zero-shot setting on Orca. The following presents our implementation details.
Experiments
In this section, we first introduce experimental set- tings. Then automatic and human evaluation met- rics are introduced to evaluate model performances. At last, we present extensive experimental results.
5.1 Experimental Setting
We implement experiments under zero-shot/few- shot settings evaluate model's performance wards unseen domains with few samples. Follow- ing existing works (Mi al., 2022), Orea provides multiple few-shot splits to measure variance CMRC models across various splits. Concretely, we split 5 sessions and 10 sessions from the sup- port set and define them as 5-shot and 10-shot, respectively. 0-shot means nonuse of support set. In word, experiments implemented under 0- session, 5-session, 10-session and 200-session set- tings (named as O-shot, 5-shot, 10-shot and 200- shot respectively). For QPR framework, due the deficiency of ground-truth of Query Rewriter and Passage Reader, cannot be fine-tuned based on the support set. Therefore, we only evaluate this system under 0-shot setting. Furthermore, we test LLMs in O-shot and few-shot settings with in- context learning. Due the significant expense of fine-tuning and the max input sequence limit, we use text-davinei-003--66 as backbone ChatGPT. T5 and BART are evaluated under all the settings. We present training details in Appendix
Query Rewriter. Demonstrative pronouns and abridged expressions often appear natural con- versations. To comprehend queries better, we use the dataset from Su et al. (2019) to fine-tune Query Rewriter module based on BART, realizing anaphora resolution and omission completion.
Passage Reader. Given an answer-related passage and the query after Query Rewriter, we need to generate an answer from the passage. We use the dataset from He et al. (2018) to fine-tune gener- ative Passage Reader based on BART, obtaining preliminary answer.
Response Rewriter. crated answer and make the expression ural, we feed the answer and original query with- Query Rewriter into Response Rewriter to get the final response. We use the dataset from Chen et al. (2023a) to fine-tum tune the generative Response Rewriter based on BART.
To further improve the gen-
nat-

---

ChuRGPT ChatGP
13.54
ROUGGT-
412
ROUGG 1I-I11--1
68
$4.73
30
58
28
11/0
Factold
Causal
Confirm
List
Hypo.
Factold
Causal
Confirm
List
Hyyo.
(a) Comparison of ChutGPT and ChatGPT w/o
(b) Comparison of TSS and BART.
Figure 3: Automatic evaluation results of different methods under various query types.
Models
Relev. Comple. Nature.
An earthquake occurred night
Topic
Aba, Sichuan, and lasted night
0-shot
Domain
Society
ChatGPT
2.48
2.50
3.08
Conversation History
QPR System
3.00
3.04
3.30
Question
When did this earthquake occur?
200-shot
Response
happened June 10 00:03
BART 2.92
3.80
3.45
T5
3.12
3.82
3.36
Current Trun
Question
What is the maximum magnitude
Table 4: Human evaluation results (%), which have significant improvement with p-value < 0.05.
Malcom City, Aba Prefecture, Sichuan, June 10, 0:03 a.m., 5.8 magnitude earthquake, followed by several earth- quakes ..... including 9 Malcom City and in Hongyuan County, with maxi- magnitude of 6.0.
Passage
5.2 Evaluation Metrics
Automatic Metrics. We employ the following widely-used metrics to validate the lexical, seman- tic, and diversity aspects the generated responses, including exact matching score (EM), ROUGE- L (Lin, 2004), BLEU-1, BLEU-2 (Papineni et al., 2002), Distinct-1, and Distinct-2 (Li et al., 2016). Human Evaluation. In order to further get full view of how well the model works, we recruit annotators rate the generated response of the best two models under 0-shot setting and 200-shot setting. Annotators are asked to give three scores according to Relevance, Completeness, and Natu- rality with levels of [1, 3, 5), which represent un- readable, mediocre and flawless, respectively. We present the details of scoring rules in the Appendix. For all evaluation metrics, the larger number de- notes better performances. We present annotation details in Table 6.
The maximum magnitude of this earthquake is 6.0.
Response
is at 5.8 level.
BART
is.
Table 5: An error case of BART and T5 under 200-shot settings. We highlight the ground-truth response the evidence text in the table.
GPT-3 and CharGLM-13008 (2) Although we don't fine-tune QPR systems in Orca, it still outperforms ChatGPT under (-shot settings. We argue that two reasons may contribute its performance: The architecture of QPR more consistent with the human cognition process towards CMRC; 2) post- training QPR on several similar Chinese bench- marks brings more benefit.
(3) With the increase of training samples for fine-tuning: model perfor- mances on the test set become better. Comparing ROUGE-L scores under 5-shot and 10-shot set- tings, the improvement of BART and T5 is 12.25 % on average. Learning from only 10 support sam- ples, their performances improve markedly and can catch up with that of the QPR system. Given 200 support samples, end-to-end baselines achieve sig-
5.3 Results
Automatic Evaluations Table reports our base- lines' results of in-context learning and fine-tuning. From the table, we can observe that: (1) Under in-context learning settings, ChatGPT performs best in terms of all metrics when compared with

---

nificant promotions and surpass the QPR system. From the above observations, we can conclude that: (1) The disparity between the performance of LLMs under the in-context learning setup and the results obtained through fine-tuning smaller lan- guage models is striking. Particularly noteworthy is the fact that, despite the substantial difference in parameter count, with T5 being more than two hun- dred times smaller than GPT-3, fine-tuning T5 with a mere five dialogue sessions yields superior out- comes compared to GPT-3. These findings serve as compelling testament to both the difficulty of Orca and the untapped potential of LLMs in tack- ling the CMRC task. Moreover, through our analy- sis of the results, we have observed that ChatGLM and ChatGPT tend to exhibit tendency to refuse responses or provide excessively verbose and con- voluted answers, which primarily contributes to their lower performances in automated metries. (2) Compared with GPT-3, "GPT-3 w/o P", ChatGPT and "ChatGPT w/o P" sheds light that Orea con- tains plenty of conversations with fresh knowledge and it's hard for GPT-3 and ChatGPT to generate accurate responses only based its parameters pre-trained by old knowledge. It also proves the value of passages provided by Orca.
from deficiencies in modeling.
5.4 Analysis
We further explore the model performances on dif- ferent query types. Similarly, we conduct experi- ments on zero-shot and 200-shot settings.
Influence of Query Type Figure 3 (a) reports the results of different types of queries for fine-grained analysis. First and foremost, it is evident that Chat- GPT demonstrates remarkable proficiency in ad- dressing list-type questions and performs worst in confirm-type questions. Moreover, the utilization of knowledge notably enhances ChatGPT's capabil- ities, particularly in the domains of causal and list- based question answering. Focusing on 200-shot setting, Figure 3 (b) shows the results of T5 and BART under different query types. T5 almost out- performs BART under all query types, depending on its advantage in the number of model parameters. We obverse that BART shows poor performance on Causal queries, List queries, and Hypothetical queries in particular. According to Figure 2, these three query types account for 45.5% in Orea, prov- ing that the variety of query types poses a thormy challenge for CMRC research. Moreover, a hori- zontal comparison of figure (a) and (b) reveals a distinct advantage of T5 and BART over ChatGPT in answering confirmation, causal, and factoid-type questions after fine-tuning.
Human Evaluations To comprehensively assess the performance of these models, we randomly sampled 100 dialogue sessions from the test set for manual evaluation. In the table below, we present the results for ChatGPT, QPR, well as BART and T5 fine-tuned on the full data. T5 and BART demonstrate comparable performance across the three perspectives evaluated but exhibit more pro- nounced effects compared to the other two mod- els, proving their good ability to understand the passage, history and question, so as to generate complete and natural responses. Based on the analysis of generated responses, we find that Chat- GPT tends to generate long responses with extra information, ignoring the response completeness sometimes. The human evaluation results basically mathh their performances in terms of automatic evaluations.
Error Analysis of BART and TS We conduct an error case study to show the challenge of Orca. As seen in Table 5, we first find that there is coref- erence phenomenon in the latest question. That is, utilizing it to indicate the earthquake in the above question. Besides, the collected passage doesn't mention the highest earthquake magnitude clearly; instead, it states the average level, which confuses the model. As a result, two models are generated with 5.8 rather than the ground-truth an- swer 6.0. This example demonstrates that not only do the questions in Orca encompass wide range of linguistic phenomena, but it is also challenging to comprehend the gathered passages. More case studies are in Appendix.
In summary, the results under (-shot and few- shot settings prove the effectiveness of the pro- posed three pipelines, but they still remain the chal- lenge in Orca. For instance, none of the models performs above 80% on automatic metrics. From the human evaluation perspective, they achieve the worst results on Reley. showing they still suffer
Error Analysis of ChatGPT and GPT-3 From the table 3, it is evident that even in 1-shot and 5-shot scenarios, the performance of ChatGPT and GPT-3 is not as strong as expected, and falls sig- nificantly behind fine-tuned T5 models. Regarding the specific reasons for the performance of Chat-

---

GPT and GPT-3, our case analysis points to the following factors:
reasonably. Moreover, zero-shot and few-shot set- tings are provided to evaluate model's learning abil- ity towards unseen domains with few samples. Ex- perimental results of our proposed three baselines indicate that Orca provides a challenging testbed for CMRC research.
1. Output in English: In specific cases, despite repeated resampling, ChatGPT tends to gen- crate responses in English. We speculate that this behavior might be influenced by the pres- ence of certain English words unique those contexts in the preceding text.
Limitations
The main target of this paper is towards developing and testing Chinese strong CMRC models. To this end, we collect and propose Orca. More gener- ally, we expect the core contribution of this paper can be seen as a universal benchmark for testing LLMs' ability to new questions and knowledge under CMRC task. Admittedly, the data in the proposed dataset from the Harry Potter Series is relatively small and restricted to Chinese commu- nities. These concerns warrant further research and consideration when utilizing this work to build intelligent CMRC systems in the virtual world.
2. Declining to answer:
In scenarios requiring knowledge or information, even when ample relevant information has been provided to answer the questions. Based on the knowl- edge up until 2021, ChatGPT sometimes re- fuses to respond, citing incomplete informa- tion as its rationale. Here we provide an ex- ample: This dialogue is about a Chinese in- ternet celebrity from 2022, Brother Man from Peking University. The current round of ques- tions is about how Brother Man makes liv- ing, and the provided passage already details the way he earns money. However, ChatGPT still refuses to answer the question, citing a lack of relevant information. We present one detailed example in Appendix F, Table 8.
References
Tom B. Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
3. Closed-domain hallucination: We have ob- served instances where ChatGPT exhibits a closed-domain illusion, mistakenly assuming limitations to its knowledge domain. This can result in responses that appear overly cautious or confined to past information. Concretely, hallucination can be defined as: 1) Intrin- sic hallucination: the generated response is contradictory to the dialogue history the external knowledge sentences. 2) Extrinsic hallucination: the generated response is hard to verify with the dialogue history or the ex- ternal knowledge sentences. We present two detailed examples in Appendix F, Table 8.
Jon Ander Campos, Arantxa Otegi, Aitor Soroa, Jan Mi- lan Deriu, Mark Cieliebak, and Eneko Agirre. 2020. Doga-accessing domain-specift faqs via conversa- tional qa. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7302 -3144
6 Conclusion
We propose the first few-shot benchmark Orca for Chinese CMRC. Unlike existing CMRC datasets that each dialog contains a static passage, we as- sign each turn golden response-related passage in Orca, which is more consistent with real scenarios. All the conversations Orca are hot-topic driven and collected from various domains, responses are natural and informative via human annotation. In this way, model's 's comprehension ability and gener- ation ability towards real scenarios can be evaluated
Nuo Chen, Hongguang Li, Yinan Bao. Baoyuan Wang. and Jia Li. 2023a. Natural response generation for chinese reading comprehension.
Nuo Chen, Fenglin Liu, Chenyu You, Peilin Zhou, and Yuexian Zou. 2021a. Adaptive bi-directional atten- tion: Exploring multi culli-granularityy representations for machine reading comprehension. In ICASSP 2021- 2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 7833- 7837. IEEE.

---

Nuo Chen, Linjun Shou, Ming Gong, and Jian Pei. 2022a. From good to best: Two-stage training for cross-lingual machine reading comprehension. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10501-10508.
for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7871-7880. Association for Computational Linguistics.
Nuo Chen, Linjun Shou, Ming Gong, Jian Pei, and Daxin Jiang. 2022b. Bridging the gap between lan- guage models and cross- cross-lingual sequence labeling. In NAACL-HLI, pages 1909-1923. Association for Computational Linguistics.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A
ob- jective function for neural conversation models. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pages 110-119, The Association for Computational Lin- guistics.
Nuo Chen, Yan Wang. Haiyun Jiang, Deng Cai, Yuhan Li, Ziyang Chen, Longyue Wang, and Jia Li. 2023b. Large language models meet harry potter: A bilingual dataset for aligning dialogue agents with characters.
Chin- Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- lion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguisties.
Nuo Chen, Chenyu You, and Yuexian Zou. 2021b. Self-supervised dialogue learning for spoken con- versational question answering. arXiv preprint arXiv:2106.02158
Fei Mi, Yasheng Wang. and Yitong Li. 2022. CINS: comprehensive instruction for few-shot learning in task dialog systems. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Shtirt-Foourll Conference on Innovative Applications of Artificial Intelligence, TAAI 2022, The Twelveth Symposium Educational Advances in Artificial In- telligence, EAAI 2022 Virtual Event, February 22- March 1, 2022, pages 11076-11084 AAAI Press.
Eunsol Choi, He He, Mohit lyyer, Mark Yatskar, Wen- tau Yih, Yejin Choi, Percy Liang, and Luke Zettle- moyer. 2018. Quac: Question answering in context. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, Brus- sels, Belgium, October 31- - November 4, 2018, pages Association for Computational Linguis-
tics.
Philipp Christmann, Rishiraj Saha Roy, Abdalghani Abujabal, Jyotsna Singh, and Gerhard Weikum. 2019. Look before you hop: Conversational question an- swering over knowledge graphs using judicious con- text expansion. In Proceedings of the 28th ACM International Conference Information and Knowl- edge Management, CIKM 2019, Beijing, China, November 3-7, 2019. pages 729-738. ACM.
Tri Nguyen, Mir Rosenberg, Xia Song. Jianfeng Gao. Saurabh Tiwary, Rangan Majumder, and Li Deng- 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, July 6-12, 2002, Philadelphia. PA, USA, pages 311-318. ACL.
Wei Hc, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, Xuan Liu, Tian Wu, and Haifeng Wang. 2018 Dureader: chinese machine reading comprehension dataset from real-world applications. In Proceedings of the Workshop on Machine Read- ing for Question Answering @ ACL 2018, Melbourne, Australia, July 19, 2018. pages 37-46. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res,, 14:1::11114::7
Wei He, Kai Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, Xuan Liu, et al. 2017. Dureader: a chinese machine reading comprehension dataset from real-world ap- plications. Xrxiv preprint arXiv: 1711:071
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392. Associ- ation for Computational Linguistics.
Etsuko Ishii, Yan Xu, Samuel Cahyawijaya, and Bryan Wilie. 2022. Can question rewriting help con- versational question answering? arXiv preprint arXiv:2204.06239.
Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. Coqa: A conversational question answering challenge. Trans, Assoc. Comput. Linguistics, 7:249- 266.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequence-to-sequent pre-training
Amrita Saha, Vardaan Pahuja, Mitesh M. Khapra, Karthik Sankaranarayanan, and Sarath Chandar.

---

2018. Complex sequential question answering: To- wards learning to converse over linked question an- swer pairs with knowledge graph. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applica- fions of Artificial Intelligence 1)), and the 8th AAAI Symposium Educational Advances in Artifi- cial Intelligencee (EAAI-18). New Orleans, Louisiana. USA, February 2-7, 2018, pages 705-713, AAAI Press.
and 128, respectively. For T3, we fine-tune it for 20 epochs with the initial learning rate of lc-4 and batch size of 8. The max length of input and output sequences are set to 512 and 300, respectively. In this work, we utilize large version of BART and base version of T5' and eext-davnnci-022 version of GPT-3 in our experiments.
B Related Work
Tengtao Song, Nuo Chen, Ji Jiang, Zhihong Zhu, and Yuexian Zou. 2023. Improving retrieval- based dialogue system via syntax-informed attention. In ICASSP 2023-2023 IEÉE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP). pages 1-5. IEEE
More and more MRC datasets have been pro- posed to promote the development of QA sys- tems (Chen et al., 2021a; Song et al., 2023; Ishii et al., 2022; Chen et al., 2022a,b; Ye et al., 2023), Tabel 1 presents the comparison of text-based MRC datasets in recent years. The release of QuAC (Choi et al., 2018) and CoQA (Reddy et al., 2019) have aroused great interest in CMRC. In these datasets, each conversation is assigned a document passage. Questioner asks questions based on the passage and the answerer finds evidence answers from it, However, this pattern is quite different from natural human-human conversations. In reality, a natural conversation is driven by a certain topic and an evidence passage that is available for each turn doesn't 't exist (You et al., 2022). Limiting the con- versation content into a given document passage is inconsistent with real scenarios. Thus, model's 's evaluation on these datasets could not reflect its ability towards realistic scenarios reasonably. Be- sides, almost CMRC datasets are collected from English corpus, limiting the language diversity in CMRC tasks.
Hui Su, Xiaoyu Shen, Rongzhi Zhang, Fei Sun, Peng- wei Hu, Cheng Niu, and Jie Zhou. 2019, Improving multi-turn dialogue modelling with utterance rewriter. Proceedings of the 57th Conference of the Associ- ation for Computational Linguistics, ACL 2019, Flo- rence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pages 22- 31. Association for Computational Linguistics.
Junjie Wang. Yuxiang Zhang, Lin Zhang, Ping Yang. Xinyu Gao, Ziwei Wu, Xiaoqun Dong. Junqing He, Jianheng Zhuo, Qi Yang, Yongfeng Huang, Xiayu Li, Yanghan Wu, Junyu Lu, Xinyu Zhu, Weifeng Chen, Ting Han, Kunhao Pan, Rui Wang, Hao Wang. Xiao- jun Wu, Zhongshen Zeng, Chongpei Chen, Ruyi Gan, and Jiaxing Zhang, 2022. Fengshenbang 1.0: Be- ing the foundation of chinese cognitive intelligence. CoRR, abs/2209.02970)
Qichen Ye, Bowen Cao, Nuo Chen, Weiyuan Xu, and Yuexian Zou. 2023. Fits: Fine-grained two-stage training for knowledge-aware question answering. arXiv preprint 2.02.
Chenyu You, Nuo Chen, Fenglin Liu, Shen Ge, Xian Wu, and Yuexian Zou. 2022. End-to-end spoken conversational question answering: Task, dataset and model. In NAACI-HHILT (Findings), pages 1219-1232. Association for Computational Linguistics.
Unlike existing datasets, we propose Chinese CMRC benchmark with various hot topics and each turn of a dialog is assigned a golden response- related passage. Thus, models' comprehension ability and generation ability towards real scenarios can be evaluated more reasonably. Furthermore, we collect Orca in few-shot settings, challenging mod- els learn unseen domains with few samples. Al- though one can question previous full-data CMRC datasets also could support few-shot training for CMRC models, we argue that it could lead to un- clear comparisons due to the inconsistent settings of different works on these datasets (Chen et al., 2022b). In contrast, we present a single standard benchmark for thorough comparisons.
Chenyu You, Nuo Chen, and Yuexian Zou. 2021. Self- supervised contrastive cross-modality representation learning for spoken question answering. In Find- ings of the Association for Computational Linguis- tics: EMNLP 2021, pages 28-39.
Munazza Zaib, Wei Emma Zhang, Quan Z. Sheng, Ad- nan Mahmood, and Yang Zhang. 2022. Conversa- tional question answering: survey. Knowl. Inf. Syst., 64( 1551333355133
A Training details
We fine-tune BART for 50 epochs with a learning rate of 5c.5. The batch size is set to 10. The max length of input and output sequences are set to 512
hhtpps://gt httttss://gtthu..cmm. t5-pegasus
ogy/

---

Aspect
Soure
Rule
son could be that the related passage is too long generate an accurate response from it.
uncelated The prodiction duren' cover the semunrics of golden response sucurately. It only covens of information mentioned golden response lnvolce sons sodundanr, infermation. The prediction covens the semuntics of golden
The
Rel.
F Case studies of ChatGPT
We present one example of declining to respond and two hallucination examples of ChatGPT in Table 8. Here, give more detailed explanations of hallucination examples.
The expression of prodiction uncomileeted.
The exprasiis prodiction compleied but it's bit awicwadd read with the query. completed
Com.
The expression and smooth.
In the first example, the conversation history re- volves around inquiring about some achievements of Ai Fukuhara. However, there is an explicit topic shift in the current part of the conversation. It starts to ask about who Zhang Jike is and whether he has won Olympic championships. The passage provided contains ample information to accu- rately answer this question. However, the response generated by ChatGPT is entirely inaccurate:
The exprasioo of profiction unnviiral.
The expressios of the production semewhut natural but it's slightly insufiiiin the sbsence demumerative pronoon.
The espression of the prodectinn antural and demonstraiiv pronoun.
Table 6: The details of scoring rules of human evalua- tion.
C Details of Human Evaluation
For human evaluation, we ask annotators to give three scores according to Relevance, Completeness, and Naturality with level of [0, 1, 2), respectively. Table 6 shows the details of scoring rules of the three aspects.
Firstly, significant portion of its response remains entangled in the previous discussion about Ai Fukuhara, leading an answer that is off-topic and redundant.
Secondly, in its response about Zhang Jike, there a glaring inconsistency. initially states that Zhang Jike has won the 2012 Olympic championship but later concludes by stating that he has not won Olympic cham- pionship. This incorrect response showcases both failure to address the current question appropriately and internal inconsistencies in the information provided.
Case Study of GhatGPT with and without Passage under O-shot Setting
As shown in Table and without passage under O-shot setting. "Ghat- GPT w/o p" performs worse on most of the auto- matic metrics. For intuitive analysis, we present cases of GhatGPT w/ and w/o passage in Table 7. It indicates that GhatGPT cannot generate accurate response without passage in some cases, specially the queries that need fresh knowledge. The reason is that GhatGPT pre-trained old knowledge is probably unaware of new knowledge. Moreover, the comparison of GhatGPT with and without pas- sage proves the value golden passages provided by Orea.
evaluate GhatGPT with
In the second example, the current turn's ques- tion in the conversation pertains to how cancer is detected. The provided passage also offers detailed steps for cancer detection. However, ChatGPT's response veers off track by revisiting the previ- ous conversation history, discussing the defini- tion of cancer, its causes, symptoms, and related matters. This response is off-topic and fails to directly address the inquiry about how cancer is detected, outlined in the current question.
E
Case Study of T5 under 200-shot Setting
As shown in Table 9, we present two cases of T5 under 200-shot setting. Table 9()) shows case that T5 generated response is the same as golden response. It indicates that T5 has strong learn- ing ability and has excellent performance under 200-shot setting. Table (() shows case that T5 generated response slightly worse than golden re- sponse. Compared with the golden response, T5's response has obvious information loss. The rea-

---

Topic:
China Space Station waiting for your business trip.
Domain:
(Technology)
Query:
2 2.? (Pactoid)
What's the name of China Space Station?
3 3 2
( # Space Station)
Passage:
M1 Si....
[
China Space Station (Tiangong Space Station, English name China Space Station) is space station system under construction in People's Republic of China...
Response:
Its name is Tiangong Space Station.
GPT-3 w/o P:
T m4 (
called Tiangong
GPT-3 w/ P:
X ( MiChina Space Station
Tiangong Space Station. Its English name is China Space Station.
Query:
E*ASE & 1 A? (Factoid)
How many people can stay there for long time?
2/ ( 4400 450 H. 42 43 1 KA NER
Passage:
T 180m4,
+ E....
The orbital height of the space station is 400-450 kilometers, the inclination is 42-43 degrees. the design life is 10 years, there are long-term residents, the total weight can reach 180 tons for large-scale space applications...
Response:
E TT 17 i A
Three people stay there for long time.
GhatGPT w/o P:
t oI Et 666
Six people can stay there for long time.
GhatGPT P:
E R. Et 16 (
Three people stay there for long time.
Query:
tt 1
(Factoid)
How many space stations in the world?
Passage:
# M 17
Currently (as of June 2021) there only two space stations operating in the world. One is the
International Space Station jointly built by six western space agencies including the United States and Russia, and the other is the Tiangong series of space stations independently built by China.
Response:
[4:37_ F:2 1 ( -
There are two space stations the world.
GhatGPT w/o P:
A tt 3R 1t 1
Currently, there are five space stations in the world.
E 2022 ) th -
GhatGPT
Currently (as of June 2021) there are only two space stations operating in the world. One the International Space Station jointly built by six westemn space agencies including the United States and Russia, and the other is the Tiangong series of space stations independently built by China.
Table 7: Cases GhatGPT with and without passage under 0-shot setting.

---

Examplc Dectining auswer
Who Brother Mus? Brother Man Peking Univenity bora the countyside and the northeest China. sistzr stulled Tsinglua Usivenity, uncded Poking University engaged culine edacation the field of atternet. Did be complete his undergroduste studes at Peking Universit?? Yes, Brother Man gaaduated from Peking Universty.
Cumeesation History
Question
Haw dues muney?
Bruber Min rams monry Brouugh live strzaming unline olucation Beveher Man from Poking University gained pupubbiit sharing his joumey of getting into getting Thinghus Univenitty Bressher Man's story undemttnd life challenging that one might imagine, but you
Boca countryside and in the northeuut with his
Presagg
much stronger than you think.
Generated Response
Bised oz knowledge up unill 2021, can't answer question because don't know who "Bencher Man" mentioned
Esample-1 of Closo--demain hullucination
What nationality Al Falalars? Al Fululam of Japuncse antionlity. Why she so populte China' It's quite simple Ai Fakuhara has spent significant amount of time China, Mozzover, her flucnt Mandario with sortheasiera accent climinutes serve of distance Reyondl her shility speak Chinew: her smishle pessunllit and high emotional intellignnee cracial factors. sie's Known be
Conversation History
crybuby.
Question
Do you know Zhang like? Has be won dhe Olympic Men's Sagles Champsonship?
Zhang like a Chinese male sable senais player 2012, he won the World Table 'lennis 'eam Champiotships and secured the Man's Singles goid medi at the Londoe Olympics in August, coupliting a grand slam of individual champondips that inclades the World Championstips. World Cap, and Olympics.
Passage
Ai Lukahara is of Japanese nationality and her popularity in China can be attnbused several reasons: She has spent a sgnificnt smount of time in Chiiu, eashlishing a derper connectinn with the Chincee people. She is furmt in Monberin wit sortheastem accemt which eliminates any sense disance between her and the Chinese population. Her good nature and high emotional intelligenoe contribute her likability Her umdency shed lears easily solh her perorised charm 2012, the
Generated Response
Champorshipe secured the Women's Singles zold medal 2012 Londos Olympics, achicsing graud slan individial championiips inclides the World Championlips, World Cup, and Olympics. Zhang has not won Olymspic Men's Singles gold medal.
World Tible leunus
Example-2 of Closed- domain hallacination
What is the essence of cancer? Canceris is pricess where normal cells in the buman body undergo changes, transforming from nummal calls, tosppiccal wolls What the comes of (his discsse? They can he categerieal into exugeron factirn and endugenous factors. The symptoms this dieasse include: Systemic cympions: Local sympioms the Symptoms crused wing chemical drugs daily coutine. Adogt healthy distary habits, Avoid smoling. Other aspects. Putients should sctively esgage poevention. early detestion, and early inosimert
Conrersation History
of the cancer What chemnttessppp Chemoticrapy therapeute purposes. How can disease prevented? Maintais a regular
theappy It invalves
Question
How this discase dingnosed?
Different of cancers require difienent types
Commonly used test include blood tests, imaging and clinkca
lovasive
details: lmaging
it absormulity found hugs chest scan chest ussally performed assess the siuution,
X
recommended
conduct a CT scan upper abdemen and the cutire abdomen. but also identiftes any potencal invasion of the adrenal glands and liver. Additionall. a whole body bote scan or cammal MKJ night: be necessary for comgrehessive assesmect. Blood tesis Bloed tost provid: information about the basic functional statss of the hoely, including complase blaed conn, liver function, kidhay function, cuopulation function. etc. Timor markers also detected thnugh bloed test, lavisive teas: For esampls, is the case of lang conces, flheroptic bronchonsopy be
act only helps determise
of
tantor
Phesage
ling hippsy might he performed. If theme's ampiccon digeddive inss tumens endeccopy of stomach or colon might
impurtani Lintaged tymph nodes neck would deperal suepesed type lucation the
lymph node
excisio for examination. It's
specific andiddal panent's condition.
Cancer maligueat originating from transfomeed celk grow mncomtollably
from
incluling inflummuss like ralistion, chumirals,
ganlics Symptome vary
type
Generated Response
include sysemiic effects, changes, and metacasis related issues. Some cancers cured. deponding type and timing After sororecful iestment parimns can long. influrncdd muliple fasties, Treatment incolves sargery. chemothengy, ndiation, imsusctherapy, and more, tallored to cancer specitics and patient bealth. Pollow up case is crucill for mainaining bealth post irvemmnt.
Table 8: Error cases of ChatGPT.

---

Topic:
Why don't 't young people use mobile phone cases? 21
Domain:
(Society)
Query:
What the functions of mobile phone case?
Passage:
F BL (M///
not only used as an ornament, but also can protect the mobile phone...
Response: F F.U.LE 0U003333 (TX (t / (1, HH. 3i84 Bi Bi
It is not only used as an ornament, but also can protect the mobile phone, and it can prevent falling, scratching, water and shock.
Query:
TL & ? (Hypothetical)
T What are disadvantages of using a mobile phone case?
Passage:
F8.... uu F: F BUA EL
It's best not to use mobile phone case for the following reasons: 1. The mohile phone case will affect the heat dissipation of mobile phone, which reduce the life of the mobile phone...
1. FU Mani FU FL de (EE 2. B-A F 1L
Response:
3. (1 A $
28 0R ( T (
The phone case will affect the heat dissipation the phone, resulting a reduction
life the phone.
2. The phone case affects the beauty of the phone and makes the phone lose its ergonomic appearance. 3. Mobile
phone cases are mostly made of chemical synthetic materials, and frequent use is not conducive to health.
3. FI
EB PF PH 1. The phone case will affect the heat dissipation of the phone, resulting a reduction in the life of the phone.
1::
T5:
2. The phone affects the beauty of the phone and makes the phone lose its ergonomic appearance. 3. Mobile
phone cases are mostly made of chemical synthetic materials, and fireuuett use is not conducive to health.
(a) An example that T5 performs well compared with golden response.
Topic:
H A
Two small police dogs are welcomed by Nanjing Metro Police.
Domain:
2
(Society)
Query:
H 2
2 H th (+?
What are the requirements become police dog?
Passage:
F. f (1) 4 E TA #E WE
Police dogs need strong visual and auditory abilities; strong sense smell...
Response:
Wf Police dogs need strong visual and auditory abilities...
km # F. # A
Query:
E
(Causal)
Where does go after retirement?
Passage:
/ E HiSt JJ 4 "...
Most of them will remain in the barracks after retirement...
# #H A H B
Response:
# T BR J W
Most of the them will remain in the burracks after retirement and be placed in the special sanatorium. They will be carefully taken care of by the trainers and enjoy their old age. Some retired police dogs will be adopted.
T5:
E11 /. # AJ #i 1h 41J tt te BLBB E il]l 5. CEE # Most of the them will remain in the barracks after retirement and be placed in the special sanatorium. They will he carcfully taken by trainers and enjoy their old age.
(b) An example that T5 shows slightly worse performance compared with golden response.
Table 9: Two cases of T5 under 200-shot setting.


Reviews:
[{"review_id": "XJVok8RdTQ", "reviewer": "Reviewer_e1ov", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First and foremost, we appreciate the time and effort you dedicated to reviewing our paper and providing valuable feedback. In response to the concerns raised by the reviewer, we offer the following rebuttals.\n\n**Q1**: Add Few-shot experiments of ChatGPT.\n\n>**A1**: Thanks for your thoughtful advice. When constructing the dataset, we create support sets of 5, 10, and 200 shots. Apart from the zero-shot experiments we report, we indeed attempt few-shot trials, such as the 5-shot, which means we add training samples from our 5-shot support set as exemplars in our prompts.  However, given that some of our test samples involved lengthy dialogue histories and passages, sometimes spanning several dozen turns, ***they exceed the maximum token limit of ChatGPT, rendering it incapable of generating a relevant answer***. Given the unfairness of this comparison to ChatGPT under such circumstances, we thus decide not to report these results in the paper.\n\n**Q2**: Add  Experiments with different strategies carefully and report the exact prompts/instructions used in the experiments in the appendix for completeness and reproducibility.\n\n>**A2**:  Thanks for your thoughtful advice. In fact, during our experimentation process, we have already investigated the impact of different prompts on the model's performance. However, due to space constraints, we only report the optimal results in the paper. As shown in the two tables below, we present the various prompts we utilize, along with their corresponding results based on ChatGPT. Just as you suggest, we will include these discussions in our Appendix.\n\n Here, we only provide English translation of our prompts.\n\n| Prompt |Content |\n| :----------- | :-----------|\n|Prompt-1|  The following is a conversation between two users, User A and User B. Your task is to reply to the latest question from User A based on their question and the conversation context. Additional relevant information has been provided to assist you in answering User A's question. Please use the provided knowledge and the conversation history to formulate your response. Please remember you just need to directly give your response and avoid producing irrelevant statements.|\n|Prompt-2| You are a professional  dialogue agent. Below, there will be a conversation about a certain topic involving the user and the assistant. Your task is to respond to the user's latest question based on  the conversation content. To assist you in answering questions, we also provide some relevant knowledge. You need to incorporate the provided knowledge along with the conversation history to formulate your response. Please remember you just need to directly give your response and avoid producing irrelevant statements.|\n|Prompt-3| The conversation involves two users, User A and User B. Your role is to reply a response to the most recent question from User A.Your task is to respond to the latest round of questions from User A from the perspective of User B, based on User A's questions and the conversation context. To assist you in answering User A's question, we have provided some relevant information. You should incorporate the provided knowledge along with the conversation history to give your response. Please remember you just need to directly give your response and avoid producing irrelevant statements. |\n\n| ChatGPT |BLEU-1 |BLEU-2 |Distinct-1 |Distinct-2 |ROUGE-L |EM|\n| :----------- | :-----------: | :----------- | :-----------: | :----------- | :-----------: |:-----------: |\n| Prompt-1 |35.30 |32.90 |2.59 |39.00 |46.14|1.91|\n| Prompt-2 |34.45 |32.60 |3.45 |39.63 |45.67 |3.01|\n| Prompt-3 |35.93 |33.21 |2.46 |38.13 |47.22 |3.03|\n\n**Q3**: Question about quality of the gold label of Table 5 example\n\n>**A3**: Sorry for the misunderstanding.  After conducting a thorough review of our dataset, we've identified an error in Table 5. The question should be \"What was the maximum earthquake magnitude?\" instead of \"What was the earthquake magnitude?\" This is a typo that we will rectify in the upcoming version of our paper.\n\n**Q4**: the role of retrieved documents in the evaluation seems rather unclear. Currently it seems the evaluation assumes that the external documents are provided by an oracle, but in reality I imagine those documents need to be automatically retrieved...? (thus it might make sense to include such evaluation settings as well...?)\n\n>**A4**: Thank you for your thoughtful inquiry. However, there appears to be a minor misunderstanding in your interpretation. As we've illustrated in both the introduction and problem formulation sections of our paper, we focus on the ***CMRC (Conversational Machine Reading Comprehension)*** task, as opposed to the conventional CQA (Conversational Question Answering) task. It's noteworthy that the Orca dataset also aligns with the CMRC category. \n\nThe primary differentiation between these tasks lies in the fact that CQA tasks don't furnish predefined passages or documents as repositories of knowledge. Consequently, models engaged in CQA tasks are mandated to perform document retrieval during conversations in order to procure pertinent information. On the other hand, ****CMRC tasks involve provided passages and conversations, requiring models to understand passage content in conversation****. This difference means Orca doesn't require a document retrieval step, eliminating the need to assess the precision and quality of retrieval documents.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- a new corpus of Chinese language conversational reading comprehension\n\n- compared to the prior resources that assumed the exact phrase from the provided paragraph as answers, this paper aims to support more naturalistic and free-form answers\n\n- compared to the prior resources that assumed one external document to ground the conversation on, this paper aims to support multiple documents to ground the conversation", "reasons_to_reject": "- the experiments for baselines, especially ChatGPT could've been executed better. As is, only zero-shot performance is reported for ChatGPT, while the authors could've included few-shot cases as well. \n\n- Moreover, due to the particularity about how the external document is provided primarily by the responder interleaved in between the conversation turns, how exactly the instruction was phrased for ChatGPT will be very important. I wish the authors experimented with different strategies carefully and reported the exact prompts/instructions used in the experiments in the appendix for completeness and reproducibility.  \n\n- I have a mixed feeling about the quality of the gold label of Table 5 example (and the likely consequences of such noise in the annotation on the baseline performances). it seems debatable to me that the gold answer provided is necessarily the only correct answer (or the most naturalistic answer), especially considering that the preceding interaction was about the earthquake that happened on Jun 10 at 00:03am, combined with the fact that the retrieved document itself characterized that the earthquake as \"a 5.8 magnitude earthquake\" explicitly. If the annotations had such noise, it is likely that only supervised methods that are capable of learning the particularities of the annotations can perform well, which will inherently influence the perceived gap between supervised approaches and ChatGPT performance. While I would be excited to see a failure mode of ChatGPT, this particular annotation just doesn't seem quite right to me\n\n- the role of retrieved documents in the evaluation seems rather unclear. Currently it seems the evaluation assumes that the external documents are provided by an oracle, but in reality I imagine those documents need to be automatically retrieved...? (thus it might make sense to include such evaluation settings as well...?)", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a new Chinese language dataset for machine reasoning comprehension that focuses on conversational QA based on multiple documents. The dataset is constructed with (crowdsourced) human interactions focusing on trendy topics, amounting to 831 conversations and 4742 interactions in total. The paper also presents baseline performances of ChatGPT and others, demonstrating much room for improvements. \n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HehKlCfuqx", "reviewer": "Reviewer_vj1q", "summary": "", "questions": "1. In Table 3, how do you address the issue that the ChatGPT's response is open-ended and sometimes, it may generate some unrelated content to the answer, e.g, \"I am happy to help you.\", \"As an AI language model ..\". How these contents can affect your evaluation.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Firstly, sorry for the typos and we sincerely thank you for dedicating your time and effort to review our paper and provide valuable feedback. In response to the concerns raised, we would like to offer the following clarifications:\n\n**Q1**: Question about evaluation metrics such as BLEU and Distinct.\n\n**A1**: In response to your concern regarding the use of metrics such as BLEU and Distinct in our paper, I appreciate the opportunity to provide a thorough explanation of our choices as authors. \n\nWe acknowledge the concern about using metrics like BLEU and Distinct in evaluating the generated texts of LLMs. While it's true these metrics have limitations in capturing the full quality of LLMs, **there is currently no universally accepted automatic metric that offers a complete evaluation of LLM-generated texts**. While BLEU and Distinct might not fully capture the quality of texts generated by LLMs, they still provide valuable insights into certain aspects of the model outputs such as diversity. In addition to the BELU and Distinct metrics you mentioned, we have also employed the ROUGEL and EM metrics, which focus on fuzzy matching and exact matching, respectively. We aim to evaluate the model's performance from multiple metric perspectives. \n\nMoreover, to further corroborate our automatic evaluation results, we conduct evaluations based on ***three human-constructed metrics***, as shown in Table 4. The results demonstrate a positive consistency between automatic and human evaluations. We look forward to future research that advances the field's understanding of evaluation and provides more accurate and comprehensive metrics for assessing the true quality of these models' outputs.\n\n**Q2**: Question about the challenge of Orca: With the birth of ChatGPT / GPT-4 / Claude, their conversation ability is quite good. For conversational QA, it should be quite easy for them to address, given the external passage for each question.\n\n>**A2**: Thanks for your concerns about the difficulty of Orca, we have several points to clarify:\n\n- Our dataset, based on the CMRC framework, provides dynamic context passages for each question within the conversation. Specifically, at each turn of a dialogue, the model's input of in-context learning could include the task description, the conversation's topic, domain, dialogue history, knowledge from previous rounds, the current question, and corresponding contextual knowledge. As conversations deepen, the number of turns and the length of input gradually increase. Our statistical analysis (shown in the table below) reveals that nearly 1/4 of the dataset has input lengths exceeding 2K tokens, with even 10% of instances surpassing 4K tokens. This presents three significant challenges for LLMs:\n\n    - First, testing their memory and comprehension abilities for longer texts; \n    - Second, demanding effective strategies for modeling lengthy texts when input lengths exceed model limits. Simple in-context learning may not suffice to handle our dataset, as argued in [C].\n   - In Orca, the model's comprehension task is not limited to a simple question-external passage pair. It involves grasping extensive dialogue context, dynamic knowledge updates, and potential topic shifts. \n\nThese challenges continue to remain pertinent and unsolved for current LLMs.\n\n| Input Length |0-500 | 500-2k | 2k-4k |  >4k|\n| :----------- | :-----------: | :-----------: | :-----------: | :-----------: |\n|Percentage (%)| 34.3% | 41.9% | 14.2% | 9.6% |\n\n\n- Recent studies [A] [B] have pointed out the existence of \"closed-domain hallucination\" in current LLMs, where erroneous answers are generated despite the presence of external passages:\n    -  **Intrinsic hallucination**: the generated response is contradictory to the dialogue history or the external knowledge sentences.\n    -  **Extrinsic hallucination**: the generated response is hard to verify with the dialogue history or the external knowledge sentences. Responses with extrinsic hallucination are impossible to verify with the given inputs. “Pickwick hotel” might be “in san diego”, and Djokovic may have been “in the top ten singles players of the world”. However, we do not have enough information to check the truth of these statements. Our dataset, being novel and diverse, increases the probability of such hallucinations, further complicating the evaluation.\n\n| |Example (We omit provided passages of previous turns) |\n|-----------|:--------------------------------------------------|\n| Dialogue History |  What nationality is Ai Fukuhara? Ai Fukuhara is of Japanese nationality. Why is she so popular in China? It's quite simple – Ai Fukuhara has spent a significant amount of time in China. Moreover, her fluent Mandarin with a northeastern accent eliminates any sense of distance. Beyond her ability to speak Chinese, her amiable personality and high emotional intelligence are more crucial factors. Plus, she's known to be a bit of a crybaby. |\n| Current Question |  Do you know Zhang Jike? Has he won the Olympic Men's Singles Championship? |\n| Current Passage |  Zhang Jike is a Chinese male table tennis player. In 2012, he won the World Table Tennis Team Championships and secured the Men's Singles gold medal at the London Olympics in August, completing a grand slam of individual championships that includes the World Championships, World Cup, and Olympics. |\n| Generated Response |  Ai Fukuhara is of Japanese nationality and her popularity in China can be attributed to several reasons: She has spent a significant amount of time in China, establishing a deeper connection with the Chinese people.  In 2012, she won the World Table Tennis Team Championships and secured the Women's Singles gold medal at the 2012 London Olympics, achieving a grand slam of individual championships that includes the World Championships, World Cup, and Olympics. **Zhang Jike has not won an Olympic Men's Singles gold medal.** |\n\n Here, we present a hallucination case of ChatGPT in Orca. In this example, the conversation history revolves around inquiring about some achievements of Ai Fukuhara. However, there is an explicit topic shift in the current part of the conversation. It starts to ask about who Zhang Jike is and whether he has won Olympic championships. The passage we provided contains ample information to accurately answer this question. ***However, the response generated by ChatGPT is entirely inaccurate***: \n\n- Firstly, a significant portion of its response remains entangled in the previous discussion about Ai Fukuhara, leading to an answer that is off-topic and redundant.\n- Secondly, in its response about Zhang Jike, there is a glaring inconsistency. It initially states that Zhang Jike has won the 2012 Olympic championship but later concludes by stating that he has not won an Olympic championship. This incorrect response showcases both a failure to address the current question appropriately and internal inconsistencies in the information provided.\n\n***Taken together, the challenges posed by Orca go beyond the simplicity of question-external passage pairs, extending into the intricacies of dialogue context, dynamic knowledge incorporation, limited max token lengths, and handling topic shifts.*** These challenges remain significant hurdles for current state-of-the-art LLMs and underscore the necessity of addressing them for more effective CMRC.\n\n[A]: Survey of Hallucination in Natural Language Generation\n\n[B]: Performance and exploration of ChatGPT in medical examination, records and education in Chinese: Pave the way for medical AI\n\n[C]: MemoryBank: Enhancing Large Language Models with Long-Term Memory\n\n**Q3**: Question about how to evaluate ChatGPT's performances when its response is open-ended sometimes.\n\n>**A3**: Thank you for raising concerns regarding the open-ended responses from ChatGPT. In our experiments, our prompts specifically request the model to avoid producing irrelevant statements, such as \"as a language model\" or \"as an AI.\" Additionally, we apply post-processing to the outputs, utilizing techniques such as regular expression matching, to remove statements that are unrelated to the answers. The final outputs from our experiment show no presence of such unrelated expressions.\n\n**Q4**: Typo: \"bel pairs In our experiments\" -> \"bel pairs in our experiments\"\n\n>**A4**: Sorry for the typos, we will revise this mistake in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A new benchmark: The paper provides a new benchmark for studying conversational question answering. \n\nThe new benchmark considers a new issue that is ignored by previous datasets, that is each question may involve different passages, instead of just one. \n\nThe new benchmark should be useful to the community. ", "reasons_to_reject": "There are some issues for the proposed new benchmark:\n\n1. Evaluation Metric: the new benchmark emphasize the gold answer for each question a natural sentence instead of phrases that can be extracted from the passage. However, the evaluation metric used for evaluation is BELU and Distinct. \n\nSuch evaluation metrics can not well measure the real performance of models, since BELU only measures the token overlaps between the candidate and reference. For the baseline of ChatGPT, under the zero-shot setting, its response is quite open-ended, so it is very hard to measure ChatGPT's response. \n\nSo, I would think the evaluation metric for this benchmark is not good. \n\n2. I have the doubt for the challenge of the new benchmark. \n\nWith the birth of ChatGPT / GPT-4 / Claude, their conversation ability is quite good. For conversational QA, it should be quite easy for them to address, given the external passage for each question. ", "questions_to_authors": "1. In Table 3, how do you address the issue that the ChatGPT's response is open-ended and sometimes, it may generate some unrelated content to the answer, e.g, \"I am happy to help you.\", \"As an AI language model ..\". How these contents can affect your evaluation.", "paper_topic_and_main_contributions": "The paper presents a new dataset for conversational question answering. Different from previous work one dialogue only involves one passage, the proposed new benchmark considers multiple passages that one dialogue can involve. In each round, there is a passage as the reference. \n\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 342: \"bel pairs In our experiments\" -> \"bel pairs in our experiments\""}, "extra_text": {}}, {"review_id": "dQjxN7Xwt6", "reviewer": "Reviewer_fQ1z", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your meticulous review and invaluable suggestions on our work. We would like to address and clarify the key issues and suggestions you raised:\n\n**Q1**: Question about using BLEU scores to evaluate  LLMs such as BLEU and Distinct.\n\n>**A1**:  In response to your concern regarding the use of metrics such as BLEU and Distinct in our paper, I appreciate the opportunity to provide a thorough explanation of our choices as authors. \n\nWe acknowledge the concern about using metrics like BLEU and Distinct in evaluating the generated texts of LLMs. ***While it's true these metrics have limitations in capturing the full quality of LLMs, there is currently no universally accepted automatic metric that offers a complete evaluation of large model-generated text.*** While BLEU and Distinct might not fully capture the quality of text generated by large models, they still provide valuable insights into certain aspects of the model outputs. In addition to the BELU and Distinct metrics you mentioned, we have also employed the ROUGEL and EM metrics, which focus on fuzzy matching and exact matching, respectively. We aim to evaluate the model's performance from multiple metric perspectives. \n\nMoreover, to further corroborate our automatic evaluation results, we conduct evaluations based on three human evaluated metrics, as shown in Table 4. The results demonstrate a positive consistency between automatic and human evaluations. We look forward to future research that advances the field's understanding of evaluation and provides more accurate and comprehensive metrics for assessing the true quality of these models' outputs.\n\n**Q2**: Analyze the poor performances of ChatGPT\n\n>**A2**:  Thanks for your thoughtful concern about the poor performance of ChatGPT.\n\nIn our analysis, the suboptimal performance of ChatGPT can be mainly attributed to several reasons as outlined below:\n\n- **Output in English**: In specific cases, despite repeated resampling, ChatGPT tends to generate responses in English. We speculate that this behavior might be influenced by the presence of certain English words unique to those contexts in the preceding text.\n- **Declining to answer**: In scenarios requiring new knowledge or information, even when ample relevant information has been provided to answer the questions. Based on the knowledge up until 2021, ChatGPT sometimes refuses to respond, citing incomplete information as its rationale. Here we provide an example: This dialogue is about a Chinese internet celebrity from 2022, Brother Man from Peking University. The current round of questions is about how Brother Man makes a living, and the provided passage already details the way he earns money. However, ChatGPT still refuses to answer the question, citing a lack of relevant information.\n\n|   |                  Example of Declining to answer         |\n| ------------------------------ | ------------------------- |\n| Dialogue History               | Who is Brother Man? Brother Man from Peking University was born in the countryside and grew up in the northeast of China. His sister studied at Tsinghua University, while he attended Peking University and is engaged in online education in the field of the internet. Did he complete his undergraduate studies at Peking University? Yes, Brother Man graduated from Peking University. |\n| Current Question               | How does he earn money?    |\n| Current Passage                | Brother Man earns money through live streaming online education. Brother Man from Peking University gained popularity mainly by sharing his journey of getting into Peking University. Born in the countryside and raised in the northeast, with his sister also getting into Tsinghua University, Brother Man's story made me understand a truth: life is far more challenging than one might imagine, but you are also much stronger than you think. |\n| Generated Response                       | **Based on the knowledge up until 2021, I can't answer this question because I don't know who \"Brother Man\" you mentioned is.** |\n\n- **Closed-domain hallucination**: We have observed instances where ChatGPT exhibits a closed-domain illusion [A][B], mistakenly assuming limitations to its knowledge domain. This can result in responses that appear overly cautious or confined to past information. Concretely, hallucination can be defined as: \n    - **Intrinsic hallucination**: the generated response is contradictory to the dialogue history or the external knowledge sentences.\n    - **Extrinsic hallucination**: the generated response is hard to verify with the dialogue history or the external knowledge sentences. \n\nResponses with extrinsic hallucination are impossible to verify with the given inputs.  We provide two illustrative examples below to further elaborate on this point.\n\n|  |Example-1 of Closed-domain hallucination|\n| ----------- | -----------|\n|Dialogue History|What nationality is Ai Fukuhara? Ai Fukuhara is of Japanese nationality. Why is she so popular in China? It's quite simple – Ai Fukuhara has spent a significant amount of time in China. Moreover, her fluent Mandarin with a northeastern accent eliminates any sense of distance. Beyond her ability to speak Chinese, her amiable personality and high emotional intelligence are more crucial factors. Plus, she's known to be a bit of a crybaby.  |\n|Current Question|Do you know Zhang Jike? Has he won the Olympic Men's Singles Championship? |\n|Current Passage|Zhang Jike is a Chinese male table tennis player. In 2012, he won the World Table Tennis Team Championships and secured the Men's Singles gold medal at the London Olympics in August, completing a grand slam of individual championships that includes the World Championships, World Cup, and Olympics. |\n|Generated Response|Ai Fukuhara is of Japanese nationality and her popularity in China can be attributed to several reasons: She has spent a significant amount of time in China, establishing a deeper connection with the Chinese people. She is fluent in Mandarin with a northeastern accent, which eliminates any sense of distance between her and the Chinese population. Her good nature and high emotional intelligence contribute to her likability. Her tendency to shed tears easily adds to her perceived charm and cuteness. In 2012, she won the World Table Tennis Team Championships and secured the Women's Singles gold medal at the 2012 London Olympics, achieving a grand slam of individual championships that includes the World Championships, World Cup, and Olympics. **Zhang Jike has not won an Olympic Men's Singles gold medal.**|\n\n**In the first example**, the conversation history revolves around inquiring about some achievements of Ai Fukuhara. However, there is an explicit topic shift in the current part of the conversation. It starts to ask about who Zhang Jike is and whether he has won Olympic championships. The passage we provided contains ample information to accurately answer this question.\n\nHowever, the response generated by ChatGPT is entirely inaccurate:\n- Firstly, a significant portion of its response remains entangled in the previous discussion about Ai Fukuhara, leading to an answer that is off-topic and redundant.\n- Secondly, in its response about Zhang Jike, there is a glaring inconsistency. It initially states that Zhang Jike has won the 2012 Olympic championship but later concludes by stating that he has not won an Olympic championship.\nThis incorrect response showcases both a failure to address the current question appropriately and internal inconsistencies in the information provided.\n\n\n| |      Example-2 of Closed-domain hallucination                                                                           |\n|------------|-------------------------------------------------------------------------------------|\n| Dialogue History | What is the essence of cancer? Cancer is a process where normal cells in the human body undergo changes, transforming from normal cells to atypical cells. What are the causes of this disease? They can be categorized into exogenous factors and endogenous factors. The symptoms of this disease include: Systemic symptoms; Local symptoms of the cancer; Symptoms caused by distant metastasis of the cancer... What is chemotherapy? Chemotherapy is short for chemical therapy. It involves using chemical drugs to kill cancer cells for therapeutic purposes. How can this disease be prevented? Maintain a regular daily routine; Adopt healthy dietary habits; Avoid smoking. Other aspects: Patients should actively engage in prevention, early detection, and early treatment. |\n| Current Question | How is this disease diagnosed? |\n| Current Passage | Different types of cancers require different types of tests. Commonly used tests include blood tests, imaging tests, and invasive tests. Here are the details: Imaging tests: In clinical practice, if an abnormality is found in the lungs during a chest X-ray, a high-resolution CT scan of the chest is usually performed to assess the local situation. It's also recommended to conduct a CT scan of the upper abdomen and the entire abdomen. This not only helps determine the size of the lung tumor but also identifies any potential invasion of the adrenal glands and liver. Additionally, a whole-body bone scan or cranial MRI might be necessary for a comprehensive assessment. Blood tests: Blood tests provide information about the basic functional status of the body, including complete blood count, liver function, kidney function, coagulation function, etc. Tumor markers can also be detected through blood tests. Invasive tests: For example, in the case of lung cancer, a fiberoptic bronchoscopy or a lung biopsy might be performed. If there's suspicion of digestive tract tumors, an endoscopy of the stomach or colon might be necessary. Enlarged lymph nodes in the neck might require a lymph node biopsy or even excision for examination. It's important to note that the specific tests conducted would depend on the suspected type and location of the cancer, as well as the individual patient's condition. |\n| Generated Response | Cancer is a malignant tumor originating from transformed cells that grow uncontrollably. It can result from various factors, including external influences like radiation, chemicals, and internal factors such as genetics. Symptoms vary by cancer type but include systemic effects, local changes, and metastasis-related issues. Some cancers can be cured, depending on type and timing. After successful treatment, patients can live long, influenced by multiple factors. Treatment involves surgery, chemotherapy, radiation, immunotherapy, and more, tailored to cancer specifics and patient health. Follow-up care is crucial for maintaining health post-treatment. |\n\n**In the second example**, the current turn's question in the conversation pertains to how cancer is detected. The provided passage also offers detailed steps for cancer detection. However, **ChatGPT's response veers off track by revisiting the previous conversation history, discussing the definition of cancer, its causes, symptoms, and related matters**. This response is off-topic and fails to directly address the inquiry about how cancer is detected, as outlined in the current question.\n\nThis instance demonstrates a lack of relevant and focused response, where ChatGPT misses the mark by not providing a direct answer to the question regarding cancer detection procedures.\n\nWe hope our response can address your concern, and we will add these analyses and cases to our Appendix.\n\n[A]: Survey of Hallucination in Natural Language Generation\n\n[B]: Performance and exploration of ChatGPT in medical examination, records and education in Chinese: Pave the way for medical AI", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a valuable contribution to the field of natural language processing by proposing a dataset for the Chinese Conversational Machine Reading Comprehension task. In addition to proposing the dataset, the authors have provided a comprehensive evaluation of several state-of-the-art models based on LLM and PLM methods. ", "reasons_to_reject": "It is surprising that ChatGPT, when given an answer passage, could not generate a correct answer, as shown by the low BLEU score in Table 3. However, it is important to consider whether the BLEU score accurately reflects answer correctness in this context. While the BLEU score is suitable for evaluating text span, it may not be the most appropriate metric for evaluating natural text generated by LLMs in conversational settings. It is possible that ChatGPT may provide a correct answer but not in the exact format of the golden answer. Therefore, it would be valuable to consider additional metrics to more accurately reflect the answer's correctness. In addition, it would be valuable to analyze the poor performance of ChatGPT in generating correct answers given an answer passage. Specifically, it would be interesting to investigate whether the poor performance is due to a poor understanding of Chinese or a poor retrieval ability to search for correct answer evidence. I am willing to raise the scores if my consideration could be solved. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper has proposed the first Chinese Conversational Machine Reading Comprehension benchmark Orca. The authors have demonstrated the quality and usefulness of the Orca dataset by evaluating several state-of-the-art models on the benchmark. The paper is well-written and presents a clear and comprehensive description of the Orca dataset.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper contributes a conversational machine reading comprehension dataset in Chinese. The performance of models, both commercial (in zero-shot and few shot ICL setting) and open (few and many shot finetuned) was evaluated on the dataset, showing room for improvement. The reviewers all believe (and I agree) that the resource is valuable, and that the task setting is realistic and challenging: conversations grounded in multiple documents, and answers not necessarily being spans in the input contexts. The authors provided additional results and analysis to some of the concerns raised by the reviewers, particularly prompt sensitivity and error analysis of ChatGPT on this dataset.
Decision: Accept-Findings