Natural Response Generation for Chinese Reading Comprehension
Authors: Nuo Chen
Hongguang Li
Yinan Bao
Baoyuan Wang
Jia Li
Venue: EMNLP 2023 Findings (2023)
Keywords: machine reading comprehension
dataset

Abstract: Machine reading comprehension (MRC) is an important area of conversation agents and draws a lot of attention. However, there is a notable limitation to current MRC benchmarks: The labeled answers are mostly either spans extracted from the target corpus or the choices of the given candidates, ignoring the natural aspect of high-quality responses. As a result,
MRC models trained on these datasets can not generate human-like responses in real QA scenarios.
To this end, we construct a new dataset called \textbf{Penguin} to promote the research of MRC, providing a training and test bed for natural response generation to real scenarios.
Concretely, Penguin consists of  200k training data  with high-quality fluent, and well-informed responses. Penguin is the first benchmark towards natural response generation in Chinese MRC on a relatively large scale. To address the challenges in Penguin, we develop two strong baselines: end-to-end and two-stage frameworks. Following that, we further design \textit{Prompt-BART}: fine-tuning the pre-trained generative language models with a mixture of prefix prompts in Penguin. Extensive experiments validated the effectiveness of this design.

Natural Response Generation for Chinese Reading Comprehension
Nuo Chen'; Hongguang Lit, Yinan Bao
Baoyuan Wang and Jia Lifs
'Hnng Kong University of Science and Technology (Guangzhou), Hong Kong University of Science and Technology Xiaobing.Al /chennuo26@gmal comm jialee@ust.hl
Abstract
popular streams' from the answer type perspective: span-extraction (Rajpurkar et al., 2016; Trischler et al., 2016; Cui et al., 2019; Chen et al., 2022b; You et al., 2021a) and multiple choices (Lai et al., 2017; Zellers et al., 2018; Wang et al., 2020). The former requires the model to locate the text span in the given passage as the answer, e.g., SQuAD (Ra- jpurkar al., 2016) and NewQA (Trischler et al., 2016). SQuAD, for example, introduces extracted span answers. The latter needs to choose the an- swer from a list of candidates like RACE (Lai et al.. 2017). Considering most of the existing works are built in English, several efforts have been pro- posed to further advance MRC in Chinese such as DuReader (He et al., 2017). CMRC 2018 (Cui et al., 2019), ReCo (Wang et al., 2020).
Machine reading comprehension (MRC) is an important area of conversation agents and draws lot of attention. However, there is a notable limitation to current MRC bench- marks: The labeled answers are mostly ci- ther spans extracted from the target corpus or the choices of the given candidates, ignoring the natural aspect of high-quality responses. As result, MRC models trained on these datasets can not generate human-like responses in real QA scenarios. To this end, we con- struct a new dataset called Penguin to pro- mote the research of MRC, providing training and test bed for natural response generation to real scenarios. Concretely, Penguin con- sists of 200k training data with high-quality fluent, and well-informed responses. Penguin is the first benchmark towards natural response generation in Chinese MRC on a relatively large scale. To address the challenges in Pen- guin, we develop two strong baselines: end- to-end and two-stage frameworks. Following that, we further design Prompt-BARR BART: fine- tuning the pre-trained generative language mod- els with mixture of prefix prompts in Pen- guin. Extensive experiments validated the ef- fectiveness of this design. Our benchmark and codes are available at https: /github. hoan///wocchengt//eeggg
Nonetheless, despite significant achievements the various types and on a relatively large scale, a significant barrier that has rarely been discussed in previous MRC efforts remains: Annotated an- swers among these datasets are almost limited to spans in the source document the given candi- dates. While spans directly copied from the source document and simple words like "yes/no" can be seen as ground-truth answers, they don't sound like natural responses in real-life scenarios. In reality, the responses of humans are always generated with ficxibility and information rather than being forced to select certain type spans ( (.g., entities and per- son) one of the candidates.
1 Introduction
Machine Reading comprehension (MRC) aims to empower the machine to correctly answer queries based on the given passages or paragraphs, which is an active field in NLP. Much of this popularity be attributed the release of many annotated and publicly available datasets (Rajpurkar al., 2016; Trischler al., 2016; Chen et al., 2022a; You et al., 2022; Chen et al., 2023a). Formally, these MRC efforts can be classified into two most
This work devotes itself to natural response gen- eration for Chinese MRC, catering to real-world QA scenarios. Intuitively, we can try two exist- ing technical routes to generate natural responses towards MRC at the moment: 1) Optimizing gen- crative language models (GLMs) such as BART and T5 with predetermined and inflexible answers on the above benchmarks. 2) Use extremely large-
'Some answer free-form datasets with extracted span an- swers and yes/no/unknown candidates classification also can he seen as the span-cxtraction and multiple choice settings.
Work done when interned at Xiaobing.AI. Indicates Corresponding authors.

---

...Bream, which is traditionally steamed or braised, is a popular fish. Soup should course cooked, soles not suitable dish because its fleshy nature...
responses. To this end, move forward push the boundary MRC: Firstly, we collect the first Chinese generative reading comprehension (GRC) dataset named Penguin to facilitate research of natural response generation for Chinese MRC. For- mally, Penguin poses new challenge to current MRC models: Not only should the responses be accurate, but they also need to be natural (as fluent and informative as possible). Concretely, Penguin contains 200k training and 15K test corpus with high-qualige fluent and informative responses. Ta- ble 2 shows the overview of current MRC datasets and Penguin.
Passage
L - H
BN EL ff T
AT M4
H MA
$6. 1 192 Ao...
Can bream be stewed in soup? 1 f RJ 20
Question
Of Course No, you can't / Uncertain B 1Jj gu A
Candidates
Of Course TTE
BART
Yes! you want to stew soup. think- ing. then we still choose to have.
Secondly, we design two GRC pipelines based this non-trivial dataset: end-to-end; ii) two- stage training. For the former, we directly take the question and corresponding passage as input, gener- ating a sequence of words as responses. The latter consists of two modules: Answerer and Responser. Answerer is responsible for generating initial an- swers based on the question and passage. Then Responser rewrites and expands the generated an- swers into human-like responses. Experimentally, two pipelines achieve promising results in Penguin. Finally, we propose Prompt-BART an approach for fine-tuning pre-trained generative language models using wide variety of prefix prompts in Penguin. The resulting model, as stated in the prompt that accompanied it, boosted the overall performance.
GPT-3
DJ L. (8) AB uu oit Y5 If) if. #B..
2
Bream can used in soups but is gen- erally not recommended.
Expected Response
ol 3 #
12
Table I: An example from ReCo (Wang et al., 2020). The text in bold refers to the answer the candidates or the expected response. English translation also given for comparison.
scale pre-trained GLMs like GPT-3 (more than 100 billion parameters) (Brown et al., 2020) with few- shot in-context learning.
We showcase a multi-choice MRC example from ReCo (Wang et al., 2020) in Table where the query is a typical yes-or-no question and the answer is required to be one of the candidates. We fine- tune BART-large (Lewis et al., 2020) on ReCo and find that the resulting model can accurately predict the answer: "of course" in this case. However, the prediction is accurate but stiff and lacking in detail. On the other hand, we use GPT-3 under few-shot from ReCo and validate its generated response via in Table I. Obviously, the produced result "Yes! If you want to ... we still choose have" is fluent yet incoherent and semantically incorrect. Ideally, the qualified response of humans should be informative and fluent, like "Bream can be used ..... not recommended".. In other words, although these models are proficient, they can't it produce fluent and accurate responses for reading comprehension.
2 Related Work
Machine Reading Comprehension During the past few years, Machine Reading Comprehension (MRC) (Rajpurkar et al., 2016; Trischler et al., 2016; You et al., 2021b; Chen et al., 2021; Song et al., 2023; Chen et al., 2023b) receives lots of attention from research communities and various related benchmarks have been proposed: SQuAD (Rajpurkar et al., 2016), MS-MarCO (Nguyen al., 2016), NewQA (Trischler et al., 2016) and RACE (Lai al., 2017), etc. Following this line of re- search, some works aim to accelerate the develop- ment Chinese MRC on language diversity, such as DuReader (He et al., 2017), CMRC 2018 (Cui et al., 2019), ReCo (Wang et al., 2020).
the same
Based on the above analysis, we can draw the conclusion that it is impossible to build an effec- tive MRC model for natural2 response generation in the absence of a benchmark with well-labeled
These well-designed and large-scale benchmarks facilitate the research of various types of MRC tasks, but they ignore the fact that answers are pri- marily confined to text span extraction from the source document or choosing candidates. In com-
In this work, we argue that natural response are fluent and
informative.

---

parison with natural human responses, these an- swers are less informative and less fluid. In other words, models trained on the above datasets do not have strong ability to generate fluent, natural as well as informative responses. Although recent works (Bi et al., 2019; Li et al., 2021) develop sequence-to-sequent frameworks to address the challenge in MS-MarCO (Nguyen et al., 2016), they still regard some abstractive answers the optimized objective, which consist of some uninfor- mative extracted spans from the passage, leading to responses that are far away from fluent and fluid text. To this end, we propose a new dataset Penguin to encourage further progress in natural response generation of Chinese MRC.
Dataset
Lang.
Answer Type
Size
SQuAD (2016)
EN
Span
100K
RACE (2017)
EN
Mluti-Chhicess
87K
MarCO (2016)
EN
Free-Form
100K
NewsQA (2016)
EN
Span
100K
DuReader (2017)
CN
Free-Form
200k
CMRC (2019)
CN
Span
20K
ReCo (2020)
Multi Multi-Choices
300K
Penguin
CN
Natural Response
200K
Table Datasets Comparison. EN and CN refer English and Chinese, separately.
3.2 Dataset Collection
As aforementioned, a high-quality dataset is pre- requisite for building effective GRC models. Un- fortunately, current datasets don't contain well- informed and natural responses. To facilitate the study of MRC, we collect new dataset: Penguin, in the hopes of creating sophisticated GRC models that can generate natural responses for practical QA situations. Considering constructing such a dataset on large scale is non-trivial, we initialize our dataset from the current Chinese MRC dataset corpus to get raw passugg--uestion-aaswww (<P, Q, A>) triplets, including CMRC 2018, DuReader, and ReCo.
Natural Generative Question Answering Simi- lar to our work, natural generative question answer- ing (NGQA) (Yin et al., 2016; Huang and Zhong, 2018; Bi et al., 2019; Jiang et al., 2022) also aims generating natural responses to questions. Al- though both tasks share similar target, the re- search question and task settings differ: Firstly, NGQA produces responses to simple factoid ques- tions based on the naive entity triplets that are given as external knowledge. In contrast, we focus on yielding fluent responses based on a deep under- standing of the question and passage. Such deep understanding of interaction between question and passage is more challenging than the simple entity triplets. Moreover, GRC closely resembles humans asking questions about a certain topic and generat- ing corresponding replies. Last, previous NGQA efforts were limited to the English corpus, whereas we are expanding it to Chinese.
It is extremely difficult and expensive ask the annotator to start from scratch and write a response R that makes sense and sounds natural for each <P. Q, A triplet. Therefore, we generate informative and fluent responses via the following steps: (1) We first utilize specific response generative model (short for Responser) to generate initial responses for the above triplets; (2) Then, use state-of-the the- semantic matching models alongside certain manually-created criteria exclude samples of in- coherence and semantic shift; (3) Last, we employ three profcssional annotators to rewrite and recheck undesired cases, therefore regulating data quality. As result, we collect a sequence of 4-tuples: <P, Q. A. R> in Penguin, where R. is the labeled re- sponse. In the following, we will introduce the de- tailed procedures of these steps, sequentially. The conceptual process of data collection is presented in Figure 1.
3 DataSet
3.1 Task Definition
Generally, machine reading comprehension task can be illustrated as <P,< Q, A>, where P is the paragraph or the whole passage, Q represents ques- tions, AA refers to answers. Previous works tend to model MRC as a classification task with the ob- jective of selecting A from several candidates or finding the text span of A in P. In contrast, formalize MRC as a sgquence-to-sqquene task, where the objective is to generate fluent and factual responses: R. In this work, we include <P, Q A, R> in Penguin.
Responser Prior to going further, the main concern is how get effective sequence-to- sequence responser model. Furthermore, more pressing than any of these other issues figur- ing out where collect the high-quality Chinese corpus needed to train the responser. Inspired by NGQA works, We first get the English <question,

---

EN
"question, answer, response?
Statistics
Train
Dev
Test
<P, P.Q,A, R>
200k
10k
15k
Machine Translation
Max.
492
486
490
Passage tokens
Min.
24
19
31
CN
quessiin, answer, response
Ave.
84.3
82.45
84.85
Max.
61
37
17
Question tokens
Min.
4
5
5
Responser
P.2.A
Ave.
11.5
11.6
11.5
Max.
20
20
Initial Responses
Answer tokens
Min.
1
1
Ave.
2.8
2.6
Max.
42
26
Data Filtering
Response tokens
Min.
6
6
Ave.
10.1
9.6
Table 3:
Overall data Statistics in Penguin.
Semantie Matching
Filtering Rules
larity score between the response and the answer is below the pre-defined threshold. To be more specific, the generated response just be seen replacing few words the answer in some cases, resulting the extremely high similarity between the answer and the response. e.g., given the ques- tion "How tall is Yao Ming", and the answer "He is 2.29m. ", the generated response could be "Yao Ming is 2.29m." For these cases, the threshold used to filter incoherent samples will be manually larger (more than 0.95). On the other hand, in certain in- stances, where the generated response provides the final piece missing information based on the an- swer, the threshold will be manually lowered (lower than 0.3)* Besides, we employ additional scoring and filtering rules to identify the most grammati- cally correct and contextually relevant response (if any) to each <P, Q, A> triplet. For instance, we re- move the samples where the length of the response is less than the length of the answer plus 5.
Human Annotation
<
Figure 1: Data collection pipeline
answer, response> triplets in (Baheti al., 2020). Then we utilize machine translation systems to translate these data from English to Chinese. Con- sidering translated corpus will inevitably have erro- neous examples, we utilize perplexity (Brown et al.. 1992) to identify and weed out some bad cases. Fol- lowing the line of Yin et al. (2016), Bart- is employed as the backbone of responser to fine-tune on these translated corpus, which takes the ques- tion as well answer as input and then outputs response during training. Subsequently, the result- ing responser models are used to inference on each <P. Q A> triplet from CMRC 2018, DuReader and ReCo, generating initial responses. Coneretely, responser takes the passage and question as input, then outputs the response during inference.
Human Annotation In the above, we automat- ically generate the response via the effective sponser, and then try filter out meaningless examples. To further control the training data qual- ity, we recruit another three professional annotators to re-check each example and revise undesirable cases. detail, each piece of data is verified by two different annotators. These annotators review the generated responses after reading the relevant question, passage, and answer. If the generated response is unnatural, they try rephrasing the state- ment obtain fair response. Thereafter, another annotator rates their annotations' quality, avoiding bias and getting high-agreement data. As result,
Data Filtering Intuitively, the collected initial responses might be stilted, evasive, and semanti- cally challenging to connect with its correspond- ing questions and answers. Therefore, we employ state-of-He the-art art Chinese semantic matching models LCQMC leadboard compute the sentence similarity of the generated response and answer. Roughly, we will throw out an instance the simi-
hhtp:///wwweeeiiiooocccccmmccess dussificationnonnllqme
"We manually select these thresholds based on experi- ments.

---

Type
Example
Percentage
YES/NO Fact Oprion
The design life of Toyota engine?
Extension
0.24 million km Toyota engine has de- sign life 0.24 million km.
62.4%
When
Where
Q: Can carry tea in trains?
Fact
Which
Rewritten
A: Yes
26.3%
bring tea in trains.
R: Of course,
Others
Opinion
YES/NO
Do
Q: Can you fix the Apple 58 if it'ss flooded?
Whar
A: It can be fixed.
Others
11.3%
R: In general, if the iPhone 5s gets water, it can defi- nitely be repaired.
VWN
YOUR
Table 4: Comparison between answer and response in Penguin. For ease of reading, English translation data is only provided here.
Figure 2: Question types in Penguin. Empty colored blocks indicate suffixes that are too rare to show.
Fact, and Opinion. Yes/No questions often asked for confirmation, and Fact questions are mostly about specific entities, timing, or descrip- tion of some facts. Opinion questions always con- tain why/what/how questions, inquiring about the reason or advice of things. We draw the distribu- tion of these three question types from our collected Penguin in Figure 2, showing Penguin contains var- ious questions, catering more realistic scenarios.
we collect high quality 4-tuples: <P, 2.
Test set Annotation We additionally hire eight expert annotators to choose and create questions and their answers from social media and them- selves, bringing the acquired data closer to real- world events and increasing variety. Concretely, cight annotators are split into four smaller groups, and each of them consists of one questioner and one answerer. Questioners first select questions ques- tion the news they interested in from current social media platforms like weibo and zhihu. Then, answerers get related passages that contain cor- responding knowledge about each question from Bing search. Following that, they must select the excerpt from the retrieved ones that is the most relevant to the annotated question at hand. After reading the question and its accompanying passage, they then annotate responses at last. Similarly, recruit annotators recheck the data twice. further guaranteeing data quality. This way, we collect 15k data in our test set.
Response Analysis After deep analysis of sponses and answers in Penguin, classify the connection between them into three broad cate- gories: Extension, Rewritten, Others. Extension refers the response containing exactly con- tent of the answer and going into more detail about it. Of note, although these answers are semanti- cally right, we argue that the responses that extend to them could be more informative and cater to real human responses. Rewritten denotes the sponse totally rewritten the answer make it more informative and natural while retaining the origi- nal meaning. That is say, the response almost has no overlap with the answers in this situation. Others indicates the response may contain part of the answer, as well as expands with lexical, coreferential, casual inferential knowledge. We showcase of each category in Table 4, respectively.
The general statistics in Penguin are presented in Table 3. In total, we collect train, development, and test set, which consists of 200k, 10k, and 15k <P, Q. A. R> data, separately. The collected responses are noticeably more informative than the answers given, as seen by their length.
3.3 Data Statics
Model
The pilot investigations (He et al., 2017) assisted us reaching an agreement on the following question type taxonomy: Yes/No,
Question Analysis
In this paper, we propose two frameworks to ad- dress the challenge Penguin: 1) end-to-end; 2) two stage.

---

P: Brecam, which is traditionally steumed or popalar fish...
End-to-end
Sequence-to-Sequeecc
R: Yes, Breams can be used in soups.
@ Can bream be stewed soup?
F:: Bream, which is traditionally steamed or briised, is popular fish...
(b) Two-stage
Me
Responser
stewed soups,
Q: Can brcam be stewed in soup?
Figure 3: Overview of proposed frameworks: End-to-end; (b) Two-stage. Answerer denotes answer generator module and Responser refers to response generator module. English translation is only given for reading.
which consists of answer generator module (short for Answerer) and a response generator module (short for Responser), seen in Figure (b). two stage framework, Answerer can be thought of as a reading comprehension module that is charge of thoroughly comprehending the passage and the question before generating answer; then Responser rewrites and extends the generated an- swer to more human-like response.
Example <0>
<p>
<>>>
End-to-End
Template
Template 2
Please give a response
based the question and pangraph.
Question:
Template 3
Passage: <PP >
Two Stage
Template - Answerer
Template
Please the question based
To be more specific, we first take X. into An- swerer, which yields initial answer
the question and paragraph.
Template 3
Question is <
Passage <
M
Template 1 Responser
Template 2
La = Σ log(Pa Azj)))
(3)
This task is response rewriting, giving reasonable response based on the question and answer below.
3emplate 3
Question
Then we construct new input X, with generated answer and question:
Answer is <
X, {CCLS] Queistiooeiiom///mm [SEP Answer [SEP]]
Figure 4: Prompt templates.
Subsequently, Responser responsible for read- ing the X, to predict the target response R:
End-to-end Intuitively, we can directly concat question and passage to construct the input X,:
X, = [CLS] Quest [SEP] Passage (SEP]) (1)
[
C, - Σ log(po X, Rey)).
(5)
where [CLS] and [SEP] refer to the beginning and separate symbols. Then pre-trained genera- tive model is utilized take X input, and then generates a response after deeply understanding of the interaction between question and passage, as shown Figure 3 (a). Concretely, we optimize the model with the widely used negative log-likelihood loss during training:
Notice that, Answerer and Responser are both the generative models but not sharing parameters in our experiments.
Hoompt-Tuning Inspired by recent works (Wei et al., 2022; Brown et al., 2020), we fine-tune in Penguin with variety of prefix prompt with dif- ferent template types. Concretely, manually de- sign five templates for each generative module from end-to-end and two-stage frameworks, as several examples shown in Figure We use these prompts to formulate different inputs in training, obtaining the final checkpoint. Experimentally, we choose
L. (og Xx))
[R]
Log(po(ri)X)))) Rei)).
Two stage Besides, we also design a two-stage framework to deal with the challenge in Penguin

---

Framework
BackBone
Dist-1
Dist-2
BLEU-1
BLEU-2
R.-L
EM
Relv.P
Flu.
Info.
TS-base
2.7
33.5
80.8
75.4
79.1
31.4
3.41
3.79
3.84
End-to-End
BART-Large
2.7
34.4
82.0
77.1
80.8
35.1
3,50
3.88
3.84
Prompt-BAN BART
2.8
34.7
77.6
81.0
35.5
3.56
3.87
3.90
T5-base
2.7
33.8
80.4
75.0
79.0
31.0
3.40
3.84
3.87
Two-Stage
BART-Large
2.7
33.7
82.5
77.3
81.2
36.7
3.67
3.87
3.90
Prompt-BARI
2.9
33.8
83.2
78.2
81.8
37.2
3.76
3.87
3.93
Table 5: End-to-end and two-stage frameworks' performances with different backbone models. EN and CN refer to English and Chinese, separately. We highlight the best performances in the table. For smaller versions' results of each backbone model can be seen in Appendix D. R.-L and EM refer ROUGH-L and Exact Match, separately. We run each model three times and report their average results.
BART-large as baseline, the resulting model is named as Prompt-BAr BART after prompt-tuning All prompts can be seen in Appendix G, Table 13.
BART
83.6
$4.1
66.3
182
5 Experiments
ROUGIEL
78
5.1 Evaluation Metric
Different from previous span-extraction and multiple-choice MRC tasks, we use automatic and human metrics evaluate the model performances.
Fact
Opinien
Yes/No
Figure 5: Model performances different question
Automatic Metrics We utilize following commonly-used metrics to evaluate the quality of generated responses: Dist-1, Dist-2 (Li et al., 2016). BLEU-1, BLEU-2 (Papineni et al., 2002), ROUGH-L and Exact Match. We give detailed explanations of these metrics in Appendix A. For the above metrics, the larger the value, the better language modeling performance.
types. Here, we choose BART-large as the backbone.
2020) and BART (Lewis et al., 2020) as our back- bones. In the Appendix B, we briefly introduce each model. Detailed experimental setup of train- ing these models can be seen in Appendix C.
5.3 Results
Automatic Evaluations Table 5 shows main results of models. From the table, we can con- clude the following: 1) The designed two frame- works with different backbone models all achieve competitive results in each automatic metric, prov- ing their efffcctiveness. 2) Importantly, the results of each base model under the two different frame- works are comparable. More interestingly, an end- to-end framework a more advanced and promis- ing system when you take into account that the parameters and inference cost of a two-stage frame- work are almost two times higher. 3) Prompt- tuning can further boost the model's performances in all evaluation metrics, and the improvement is more obvious in two-stage training (e.g., 81.8 81.2 in ROUGH-L).
Human Metries To further get a fuller picture of how well the generation-based model works, we asked annotators to look at the quality and consistency of the generated responses on our test set. These annotators are from third-party and skilled with language tasks but have knowledge of the models. Concretely, we randomly sample 500 examples in our test set for model's evalu- ation. Human annotators are tasked with rating dialogue quality using three principle measures: Fluency (Flu.), Informativeness (Info.) and Rel- evance with Passage (Relv.P). Each criterion is given rating from to 5, with 3, 5 representing poor, mediocre and flawless performances.
5.2 BackBone
To investigate the performances of current state-of- the-art generative models, we choose two popular pre-trained generative models: T5 (Raffel et al.,
Human Evaluations We also draw the follow- ing conclusions from the Table 5: 1) All models achieve good results in fluency, informativeness

---

End-fo-End
Answerer
100%
14.5
10.5
...If you already have loan, the bank will definitely consider the debt ratio, depending on the assets and repayment
Passage
80%
ability...
60%
Question
Will Credit Loan Affect Mortgage?
40%
85.5
92.5
Response
It could affect your Mortgage, but it depends your income level and as- sets.
20%
Generated Response-Oris stage
0%
T5
Credit loan must affect Mortgage.
Fluency
Informative
Relevance
BART
will affect your Mortgage.
Figure 6: Win/Lose human evaluation results (%) between end-to-end and Answer in two-stage pipeline with p-value < 0.05.
Prompt-BA. BART
Credit loan could affect mortgage, but depends assets.
Generated Response-Twe stage
T5
Credit loan must affect Mortgage.
and relevance, showing their generated responses are close to human-like. 2) Prompt-BART in two- stage training performs better in relevance than other models, indicating a better understanding of the passage. In general, human evaluations of these models basically mirror automated evaluations.
BART
Credit loan will affect mortgage
Prompt-BART
Credit loan may affect your Mortgage.
Table case study in our collected Penguin, and generated responses from the large version of baselines are provided. The text in bold refers to the expected response from human annotation.
5.4 Discussion and Analysis
each example in randomly selected 200 test sam- ples from three views, as shown in Figure 6. We can observe that more than 85% of the generated responses are superior to their counterpart answers across all evaluation perspectives, indicating they closer to human-like responses than the gener- ated answers. These Win /Los results also prove the value of Penguin, that is, providing a training and testing bed for natural response generation real- world scenarios.
Aside from the high performance of our proposed models, we are concerned about the following ques- tions: Q1:
What does the model perform when it answers different question types? Q2: Is the gen- erated response more natural to the generated an- swer? Q3: Can extremely large model like GPT-3 almost address the challenge in Penguin? (Ap- pendix 1, Table 8) Q4: What the majority of the improvement brought by the designed prompts? (Appendix E.2, Table 9).
Case Study In addition, we showcase example and its generated responses from Penguin in Table 6. We can see that almost all of the responses from the different models do a pretty good job of roughly capturing the intended meaning. Morcover, Prompt BART could generate more complete and accurate responses, showing its efficiency.
We conduct fine-grained analysis to validate model performances on different ques- tion types. As seen in Figure 5, we can observe: 1) each model performs best on Yes/No questions and worst Fact questions, indicating generat- ing natural responses about specific facts needs deeper understanding of questions and passages; 2) prompt-tuning brings more benefits when an- swering Fact questions (76.5 vs. 74.3). We further present each case on three question types in Table 11 and Table 12.
Answer to Qı:
6 Conclusion
In this paper, we summarize the limitation of cur- rent MRC efforts, and propose a new dataset named Penguin for natural response generation in Chinese MRC. Penguin requires the predicted responses not only be accurate but also natural-sounding, catering real scenarios. Thereafter, we propose end-to-end and two-stage GRC pipelines ad- dress the challenge in Penguin, achieving promis- ing results. Moreover, we design prompt-turing, which involves tine-tuning the pre-trained genera-
Answer to Q2: In order to test whether the gen- crated responses
natural than the fixed answers. Three professional workers asked evaluate the generated responses from end-to- end framework and generated answers from An- swerer in two-stage pipeline. Concretely, they need to choose which is the best one (Win/Lose) for

---

tive language models with a mix of different prefix prompts in Penguin, making the system perform better and more efficiently. As an extension of future work, we will explore other techs to build more human-like GRC models.
Nuo Chen, Hongguang Li, Yinan Bao, Junqing He, Xinshi Lin, Qi Yang. Jianfeng Liu, Ruyi Gan, Ji- axing Zhang, Baoyuan Wang, et al, 2023a. Orca: A few-shot benchmark for chinese conversational machine reading comprehension. arXiv preprint arXiv:2302.138 13619.
Limitations
Nuo Chen, Fenglin Liu, Chenyu You, Peilin Zhou, and Yuexian Zou. 2021. Adaptive bi-directional attention: Exploring multi coalii-ganularityy representations for ma- chine reading comprehension. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7833-7837. IEFE.
The main target of this paper is to generate natu- ral responses for Chinese Reading Comprchension. We present new benchmark named Penguin in the hope of creating an effective GRC model. More generally, we expect the proposed Penguin can fa- cilitate the research of MRC, catering to more re- alistic QA scenarios. Our collected corpus will be soon made public for the whole research commu- nities. Admittedly, one limitation of this work is
Nuo Chen, Linjun Shou, Ming Gong, and Jian Pei. 2022a. From good to best: Two-stage training for cross-lingual machine reading comprehension. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10501-10508.
that the proposed dataset is restricted to Chinese. Another limitation is that our labeled responses not 100% from human annotation due to the high annotation cost. These concerns warrant further re- search and consideration when utilizing this work to build effective GRC models.
Nuo Chen, Linjun Shou, Ming Gong, Jian Pei, and Daxin Jiang. 2022b. Bridging the gap between lan- guage models and cross-lingual sequence labeling. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 1 1909-1923, Seattle, United States. Association for Computational Linguistics.
References
Nuo Chen, Linjun Shou, Tengtao Song, Ming Gong, Jian Pei, Jianhui Chang, Daxin Jiang, and Jia Li. 2023b. Structural contrastive pretraining for cross- lingual comprehension. In Findings of the Associa- tion for Computational Linguistics: ACL 2023, pages 1042-2057 Toronto, Canada. Association for Com- putational Linguistics.
Prithviraj Ammanabrolu, Renee Jia, and Mark O. Riedl. 2022. Situated dialogue learning through procedural environment generation. In ACL (1), pages 8099- 81 16. Association for Computational Linguistics.
Ashutosh Baheti, Alan Ritter, and Kevin Small. 2020. Fluent response generation for conversational ques- tion answering. In ACL, pages 191 2072 Association for Computational Linguistics.
Nuo Chen, Yan Wang, Haiyun Jiang, Deng Cai, Ziyang Chen, and Jia Li. 2022c. What would harry say? building dialogue agents for characters in a story. arXiv preprint 2xxx:2211.0686
Bin Bi. Chen Wu, Ming Yan, Wei Wang. Jiangnan Xia, and Chenliang Li. 2019. Incorporating exter- nal knowledge into machine reading for generative question answering. In EMNLP/IJCNLP (1), pages 2521-2530. Association for Computational Linguis- tics.
Yiming Cui, Ting Liu, Wanxiang Che, Li Xiao, Zhipeng Chen, Wentao Ma, Shijin Wang, and Guoping Hu. 2019. A span-extraction dataset for chinese machine reading comprehension. In EMNLP/JCNNPP (I), pages 5882-5888. Association for Computational Linguistics.
Peter F. Brown, Stephen Della Pietra, Vincent J. Della Pictra, Jennifer C. Lai, and Robert L. Mercer. 1992. An estimate of upper bound for the entropy of english. Comput. Linguistics, 18(1):31 40.
Wei He, Kai Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang. Hua Wu, Qiaoqiao She, Xuan Liu, et al. 2017. Dureader: a chinese machine reading comprehension dataset from real-world ap- plications. arXiv preprint arXiv: 1.50070.3.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin. Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In NewrIPS.
Yanzhou Huang and Tao Zhong. 2018. Multitask leam- ing for neural generative question answering. Mach. Vis. Appl., 2966)10009-100 1017.
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2022. Understanding and improving zero- shot multi-hop reasoning in generative question an- swering. In COLING, pages 1765-1775 Interna- tional Committee on Computational Linguistics.

---

Guokun Lai, Qizhe Xie. Hanxiao Liu, Yiming Yang. and Eduard Hovy. 2017. Race: Large-scale read- ing comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 796- 805. Association for Computational Linguisties.
Pranav Rajpurkar, Jian Zhang. Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392 Associ- ation for Computational Linguistics.
Tengtao Song, Nuo Chen, Ji Jiang, Zhihong Zhu, and Yuexian Zou. 2023. Improving retrieval- based dialogue system via syntax-infe -nnta--iiffome attention. In ICASSP 2023-2023 IEÉE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdeirahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, ACI. 2020, Online, 5-10, 2020, pages 7871-7880. Association for Computational Linguistics.
Adam Trischler, Tong Wang. Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2016. Newsqa: A machine comprehension dataset. arXiv preprint arXiv:1611 1611:0600
Chenliang Li, Bin Bi, Ming Yan, Wei Wang, and Song- fang Huang. 2021. Addressing semantic drift in gen- crative question answering with auxiliary extraction. In Proceedings of the 59th Annual Meeting of the As- sociation for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 2: Short Pupers), pages 942-947, Online. Association for Computational Linguistics.
Bingning Wang, Ting Yao. Qi Zhang, Jingfang Xu, and Xiaochuan Wang. 2020. Reco: A large scale chinese reading comprehension dataset on opinion. In The Thirty-Fourth AAAI Conference on Artificial Intelli- gence, AAAI 2020, The Thirty-Second Innovative Ap- plications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposiun on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 9146- 9153. AAAI Press.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao,
and Bill Dolan. 2016. A enversiy-prooootin ob- jective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages I 110-119, San Diego, California. Association for Computational Linguistics.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quoc V. Le. 2022. Finetuned language models are zero-shot learners. In ICLR. OpenReview.net.
Jun Yin, Xin Jiang, Zhengdong Lu, Lifeng Shang, Hang Li, and Xiaoming Li. 2016. Neural generative ques- tion answering. In Proceedings of the Workshop on Hluman-Compien Question Answering, pages 36-42.
Chin-Yos Yew Lin. 2004. ROUGE: package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Our, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Chenyu You, Nuo Chen, Fenglin Liu, Shen Ge, Xian Wu, and Yuexian Zou. 2022. Ean-to-end spoken conversational question answering: Task, dataset and model. In NAACL-HLT (Findings), pages 1219 1232. Association for Computational Linguistics.
Benjamin Muller, Luca Soldaini. Rik Koncel- Kedziorski, Eric Lind, and Alessandro Moschitti. 2021. Cross-lingual genqa: A language-agnostic generative question answering approach for open- domain question answering. CoRR, 10.02500
Chenyu You, Nuo Chen, and Yuexian Zou. 2021a. Knowledge distillation for improved accuracy in spoken question answering. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7793-7797. TEEF.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: human generated machine reading comprehension dataset. arXiv preprint 161.0966
Chenyu You, Nuo Chen, and Yuexian Zou. 202 Self- supervised contrastive cross-modality representation learning for spoken question answering. In Find- ings of the Association for Computational Linguistics: EMNLP 2021, pages 28-39, Punta Cana, Dominican Republic, Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th anmual meeting of the Association for Computa- tional Linguisties, pages 311-318.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018 SWAG: A large-scale adversarial dataset for grounded commonsense inference. In EMNLP. pages 93-104. Association for Computational Lin- guistics.
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach Learn. Res., 21:140: 1-4667

---

Parameter
Ts
BART
Prompt
Module
BLEU-1
BLEU-2
ROUGII-L
Batch size
8
10
10
Answer
79.3
77.6
83.6
Learning Rate
2e-4
2e-5
2e-5
+Prompt
82.7
80.9
86.3
Epoch
20
Responser 82.0
77.0
80.9
Grudient uccunulation steps
10
+Prompt
82,9
78.1
81.7
Encoder Max Length
512
512
Decoder Max Length
40
128
128
Table 9: Model performances with designed prompts. Here we choose BART-Large as our back- bone model. Notice that, we use ground-truth answers rather than responses to evaluate the performances of Answer.
Table 7: Saper-parameters setup during tinettning
BART and T5.
Module
BLEU-1
BLEU-2
ROUGH--L
GPT-3 (2-shot)
40.8
35.5
46.2
GPT-3 (5-shot)
48.7
42.9
58.0
BART-Large 84.8
80.0
84.3
C Experimental Setup
Table 8: GPT-3 performances with different few-shot examples.
Table 7 shows experimental setup in our experi- ments.
A Automatic Metrics
D More Results
Dist.: One of the most commonly-used metric that evaluate the diversity and informativeness of generated text. In this paper, we utilize Dist-1 and Dist-2 (Li et al., 2016) to validate the model performance.
We present T5-small, BART-small results in Table 10.
More results
E.1 Answer to Q3
BLEU: Another famous metric used evalu- ate the difference between the generated text and its reference. Similarly, we use BLEU-1 and BLEU-2 (Papineni et al., 2002) in this work.
In order to validate whether GPT-3 achieves promis- ing results in Penguin. We train GPT-3 with few- shot examples, and construct mini test set via randomly sampling 200 examples in Penguin's 's test set. Table 8 illustrates their corresponding results, observe there is significant performance gap between GPT-3 and BART-Large. The results show current large language models with more 100 billion-level parameters can not address the chal- lenge in our dataset, proving the difficulty of our Penguin.
An effective metric that measures the longest common subsequence (LCS) between model output and reference (Lin, 2004).
ROUGH-L:
Exact Match: This metric measures the percent- age of model prediction that match the reference exactly. The evaluated score is 1 if the prediction is exactly the same the reference, otherwise O.
E.2 Answer to Q4
B Model Architecture
In this subsection, we are devoted to investigat- ing where the majority of the improvement that designed prompts bring to the two-stage frame- work stems from. We show more detailed re- sults of Answerer and Responser in Table 9, We can easily draw the conclusions that 1) well- performed Answerer is the main prerequisite for well well-performed Responser. 2) The benefits of the designed prompts are most seen in the Answerer module, which enhance Answerer to generate more accurate answers. For instance, Answerer + prompt obtains more than 3 points in BLEU scores.
T5 also strong pre-trained encoder-decoder Transformer-basee model which have achieved strong results on various generative benchmarks (Chen et al., 2022c; Muller et al., 2021; Am- manabrolu et al., 2022). Likewise, we use Chinese T55 with small and base versions in our experi- ments.
BART is a widely used denoising auto coder for pre-training els, which also a standard Transformer-base PLMs. this work, we utilize Chinese B with base and large version.
seuueecc--t-seqqeeee mod-
F Generated Cases of Question Types
hhnnss///gtthbbommmm//ii///hh/oh "hanss//github.comm//aammpppp
We show typical cases in Table 11 and Table 12.

---

Framework
BackBone
Dist-1
Dist-2
BLEU-1
BLEU-2
ROUGH-I
EM
Average
TS-small
2.7
34.5
80.1
74.8
78.9
30.5
50.3
End-to-End
BART-base
2.7
34.4
82.1
77.2
81.0
35.6
52.1
TS-small
2.7
34.0
80.2
75.0
79.0
30.8
50.3
Two-Stage
BART-base
2.8
34.4
82.1
77.3
81.1
36.2
52.3
Table 10: End-to-end and two-stage frameworks' performances with different backbone models. EN and CN refer to English and Chinese, separately. We highlight the best performances in the table.
Question Type
YES/NO (Æ (i)))
three time the line, the child's gastrointestinal function is relatively weak, cating more will gastrointestinal discomfort...
Three-year-oldd baby
Passage
/ TH 162
etfi T. 7 1E1C3 1E 2
good for three-ycar-ee baby litchi?
Question
Answer
Yes. Af
Three-year-old babies cat three litchis, but not much. / My se iT BUR 1 (B TE
Expected Response
Generated Response-One-Stage
TS
They litchi. (/ TV 13 They can eat litchi. fell [J 2i They can eat litchi, fil 1/ 2U
BART
Prompt-BART
Generated Response- Twopoons-Tww-uuggee
TS BART
Three-ycar-old baby can
litchi.
They can eat litchi. f2 Et # Three-ycar-ole baby can cat litchi.
Prompt-BART
(a) A Generated Response example of Yes/No questions.
Question Type
Opinion (3
...Ittss matter of how fast you move, matter understanding. have heen leamiing the piano 10 years and have to practice these ezernies and so on over over again this year. It recommended learn Bayer first, which greatly improves the basic skills....
Passage
te J [i] et. ( (N
i 10 4
I've already learned Faber, so I still need to practice Baier?
Question
(AAK, HALE Tet >J Jggg??
Answer
Of course.
You'de 'd better practice Baier. It improve your basic skills. Hi T 3. JE (N H ACJ) fi (F
Expected Response
Generated Response-One -neage
TS
You need to practice Baier. A-- 9E
BART
You still need to practice Baier: PH You still need to practice Baier. 11
Prompt-BARY
Generated Response- Two Stage
TS
You still need practice Baier:
BART
You still need to practice Baier. [
Prompt-BART
He still needs to practice Baier. 51
(b) Generated Response example of Opinion questions.
Table 11: Typical generated examples from each model Penguin. English translation is also given for comparison.

---

Question Type
Fact (4 (453))
...Zhang Lei is Chinese foothall player and goalkeeper. Zhang Lei was selected Guang- dong Hongyuan Youth Team the age of 13, then played as the main goalkeeper for Dongguan Southem City before moving to Shenzhen 2006. He was selected the 2008 National Olympic Team and 2005 World Youth Charppionstips squad. In 2009, Beijing Guoan exchanged Cheng Yuelei, a backup goalkeeper, plus 400,000 yuan to Shenzhen exchange for Zhang Lei, which was also the last domestic import transaction Guoan in first transfer market of the 2009 season. Zhang Lei did not get a chance play because Beijing Guoan had Yang 7hi as their keeper. He was listed by Beijing Guoan and eventually moved to Changsha Jinde in 2010 after losing their main goalkeeper Song Zhenyu, 2011, Zhang moved to Chongqing Lifan and became the team's main goalkeoper. Served for five years, the first three years to play the main force, the last two years gradually as a substitute. In 2016, he was transferred to Hangzhou Greentown. Zhang Lei joined the Lijiang Flying Tigers on loan from Hangzhou Greentown in 2017....
Beijing
Passage
IA
T LIRE ith 5i 1
1 AA 11 bi K 13 HO A DE 2E /) MA E 20064
# HE AE TH E
91 FOR UI
18 A T082
E
BA
# S
2000
2009
& II A THE
ME F40
BA 3E
SKE
M. 12 th E 2009 th E
RE 9
- 2 1 E.
32 5.
PR
E 1
Bi EJ #K dn Ne E (9 H 8
#
2E200
4F 2: "
ズ DT 18 9F 90 KE
$ 2011
69 4 W
EPj 2 #/ 2016
2017 AK
HHL
In what year did Zhang Lei transfer to Shenzhen? # 444 2 |?
Question
Answer
2006. 2006
He transfers to Shenzhen in 2006. ttt 2006 + -
Expected Response
Generated Response- Stage
TS
He transferred to Shenzhen two years. it's T2
BART
He transferred to Shenzhen in two years. AE 2 He transfers to Shenzhen in 2006. f J 2006 31
Prompt-BART
Generated Response- Twonss---
TS BART Prompt-BART
He transfers to Shenzhen in 2006. 20
He transfers to Shenzhen in 2006. 2006
Zhang Lei transfers to Shenzhen in 2006. 38 2 4000066666650 4F
Table 12: A Generated Response example of Fact questions Penguin. English translation also given for comparison.
G All Prompts
We design five prompts for each module in end- to-end and two-stage frameworks, which are pre- sented in Table 13.

---

Framework
Prompts
Please generate reasonable response based the following question and passage. Question: Q
Passage:
The question is 2. give the reasonable response based on the following paragraph:
reading comprehension problem, please generate reasonable response based following question and passage. Question Q. Passage is P.
One-Stage
4. Please unswer the question according descriptions the question and paragraph. Question is Q.
Passage
The question is Q. Conduct question understanding response generation, knowledge question and response task. Relevant paragraphs are follows: P.
Please generate the reasonable
based the following question and passage. Question: Q
Passage:
The question is 2. give the reasonable answer based the following paragraph:
This reading comprehension problem, please generate reasonable answer based on following question and passage. Question
Passage
Answerer
4. Please answer the question according descriptions question and paragraph. Question is Q.
Passage
5. The question is Q, Conduct question understanding answer generation, knowledge question and answer task. Relevant paragraphs are as follows: P.
1. Please generate the reasonable response based the following question and Question: Q Answer:
The question is 2. give the reasonable response based on the following answer:
This a answer rewriting problem, please generate reasonable response based the following question and answer. Question Answer A.
Responser
4. Please give response according to descriptions of question answer. Question Answer A.
5. The question is Conduct question understanding and response generation, answer rewriting task. Relevant answer is follows: A.
Table 13: All used prompts during training. English translation is only given for reading.


Reviews:
[{"review_id": "rEsDyG3qoL", "reviewer": "Reviewer_DxeM", "summary": "", "questions": "A - (minor point) Although GPT-3 responses do not perform well according to the automatic metrics (Table 8) but I'd be interested to know how they perform in human evaluation and side-to-side comparison with BART models.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First and foremost, we appreciate the time and effort you dedicated to reviewing our paper and providing valuable feedback. In response to the concerns raised by the reviewer, we offer the following rebuttals.\n\n**Q1**: The details of how the initial responser generates the first response before filtering and human annotation are not clear from the main text.\n\n>**A1**: Thanks for your concern about the initial responser. As presented in the main paper Line 229-251, the steps for generating initial responses can be divided into the following stages:\n\n-  We first get the English <question, answer, response> triplets from [A]. Then we utilize machine translation systems to translate these data from English to Chinese.\n\n-  Considering that translated corpus will inevitably have erroneous examples, we then utilize perplexity to identify and weed out some bad cases.\n\n- Bart-large is employed as the backbone of ***responser*** to fine-tune on these translated corpus, which takes the *question* as well as answer as input and then outputs *response* during training.\n\n- Subsequently, the resulting responser is used to infer each data  from collected Penguin, generating *initial responses*. Concretely, responser takes the passage and question as input, then outputs the response during inference.\n\nWe will clarify these texts for easier accessibility in the main paper.\n\n**Q2**: Although GPT-3 responses do not perform well according to the automatic metrics (Table 8) but I'd be interested to know how they perform in human evaluation and side-to-side comparison with BART models.\n\n>**A2**: Thanks for your interest in GPT-3 responses in Penguin. In the following, we present the human evaluation of GPT-3 and our two-stage Prompt-BART. The test samples are 500 which is consistent with human evaluation settings in our main paper (423-430).\n\n| Method |Relevance with Passage|Fluency |Informativeness|\n| :----------- | :-----------: |:----------- | :-----------: |\n|GPT-3|2.90|3.45|3.55|\n|Prompt-BART|3.76|3.87|3.93|\n\nAs shown in the table, Prompt-BART performs significantly better than GPT-3. In our case analysis, we observe that when GPT-3 answers factoid questions, such as those about time and location, it tends to include irrelevant or redundant information, resulting in answers that are neither concise nor always accurate. In contrast, the outputs from the Prompt-BART model are found to be more accurate and succinct during human annotations. In the upcoming version, we will delve deeper into this and present a comprehensive analysis.\n\n[A] Fluent response generation for conversational question answering.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- New large-scale reading comprehension datasets in Chinese language containing fluent responses along with answers.\n- Sound experiments with both automatic and human evaluation and some additional qualitative analysis.", "reasons_to_reject": "- (minor point) The details of how the inital responser generates first response before filtering and human annotation are not clear from the main text.", "questions_to_authors": "A - (minor point) Although GPT-3 responses do not perform well according to the automatic metrics (Table 8) but I'd be interested to know how they perform in human evaluation and side-to-side comparison with BART models.", "paper_topic_and_main_contributions": "The authors introduce a new large-scale Chinese language Reading Comprehension dataset named Penguin. Different from previous works containing <Passage, Question, Answer> tuples, their dataset also provides a fluent answer response that describes the answer in a more human-readable way. To curate their dataset, they collect initial <P,Q,A> tuples from existing datasets and synthetically generate initial responses. These initial responses are then filtered using semantic similarity and other heuristics before showing to the human annotators who rewrite the initial response in case it is not fluent.\nThe final dataset contains a variety of question types including, facts, opinions, and yes/no questions. The authors train multiple models to test the performance of Reading Comprehension including, simple fine-tuning, prompt-tuning, and two-stage answerer and responser.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3dM9X71bkt", "reviewer": "Reviewer_qhwR", "summary": "", "questions": "1. This natural response is no better than the original option/answer, because most MRC questions are from designed-deliberately exams. \n2. This dataset is collected from existing datasets. Thus, it is not novel. \n3. The paper didn’t introduce details about the annotators, who the authors claimed to be professional.  \n4. There is no definition of the question difficultness, and readers cannot tell the real value of these \"natural\" responses.\n5. The metrics, although the author claimed this paper is focused on \"natural\", do not include metrics for naturalness, e.g., PPL, coherence (by ChatGPT), etc.\n6. The authors claimed this dataset to be necessary, does that mean LLMs like chatGPT can never achieve the same level of naturalness？\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Firstly, sorry for the typos and we sincerely thank you for dedicating your time and effort to review our paper and provide valuable feedback. In response to the concerns raised, we would like to offer the following clarifications:\n\n**Q1**: This natural response is no better than the original option/answer, because most MRC questions are from designed-deliberately exams.\n.\n>**A1**: Thanks for your thoughtful concern, we recognize that MRC questions mainly stem from deliberately designed exams. However, our objective is to empower models to generate answers that are more natural and relevant for real-life scenarios beyond just the standardized answers.\n\nMoreover, in order to test whether the generated responses are better than the original answers.  In Figure 6 of the main paper, we delve deeper into the difference in naturalness, information richness, and relevance between the natural responses and standard answers via a human evaluation. ***The figure shows that more than 85% of the generated responses are superior to their counterpart answers across all evaluation perspectives, indicating they are closer to human-like responses than the generated answers.*** The results substantiate that our delineation of natural responses is valuable and necessary.\n\n**Q2**: This dataset is collected from existing datasets. Thus, it is not novel.\n\n>**A2**: Although our training dataset draws partly from existing datasets, the enhancements, annotations, and natural response generation we introduced are distinct. This differentiates our dataset in some respects from other datasets, presenting a fresh perspective for research. Furthermore, as shown in Table 4, our natural responses encompass 62.4% elaboration and 26.3% complete rewrites, meaning nearly 90% is an entirely new dataset in terms of annotated natural responses.\n\n***Moreover, our test set is collected from scratch.*** As introduced in Line 295-314, we additionally hire eight expert annotators to choose and create questions and their answers from social media and themselves, bringing the acquired data closer to real-world events and increasing variety.  Concretely, eight annotators are split into four smaller groups, and each of them consists of one questioner and one answerer:\n\n- Questioners first select questions or question the news they are interested in from current Chinese social media platforms like Weibo and Zhihu.\n- Then, answerers manually get related passages that contain corresponding knowledge about each question from Bing search. \n- Following that, they must select the excerpt from the retrieved ones that is the most relevant to the annotated question at hand. After reading the question and its accompanying passage, they then annotate responses at last. We recruit two annotators to recheck the data twice, further guaranteeing data quality.\n\nThis way, we collect 15k data in our test set. \n\n**Q3**: The paper didn’t introduce details about the annotators, who the authors claimed to be professional.\n.\n>**A3**:  Sorry for missing the introduction of our annotators. During the process of collecting our dataset, we specifically employ a professional, publicly listed annotation company in China named DataTang to handle the labeling tasks. Throughout the annotation process, we provide rigorous training to these annotators, conduct trial annotations, and ensure regular quality feedback and checks. The completion of all annotation tasks involves approximately 10 annotators and takes 3 months, costing 100k RMB. We will clarify this in the upcoming version.\n\n**Q4**: There is no definition of the question's difficulty, and readers cannot tell the real value of these \"natural\" responses.\n\n>**A4**: Figure 2 shows the question types in Penguin: **Yes/No**, **Fact**, and **Opinion**. **Yes/No** questions are often asked for confirmation, and **Fact** questions are mostly about specific entities, timing, or description of some facts.  **Opinion** questions always contain why/what/how questions, inquiring about the reason or advice of things. These various questions show Penguin contains various questions, catering to more realistic scenarios.\n\n**Q5**: The metrics, although the author claimed this paper is focused on \"natural\", do not include metrics for naturalness, e.g., PPL, coherence (by ChatGPT), etc.\n.\n>**A5**: Thanks for your thoughtful concern regarding the naturalness evaluation. As we argued in the Introduction, natural responses are defined as fluent and informative. \nSo, in this paper, we mainly measure the accuracy and diversity of generated responses through automated metrics, using human metrics:  Fluency (**Flu.**), Informativeness (**Info.**) and Relevance with Passage (**Relv.P**) to assess the naturalness and relevance of the answers. \n\nTherefore, we do not use PPL (Perplexity) because it cannot accurately measure the naturalness of answers; it only measures the model's language comprehension ability. \n\nAdditionally, as mentioned in [A], current LLMs like ChatGPT and GPT-4 are not qualified and fair evaluators for text generation. Changing the input sequence of the texts to be evaluated can significantly affect the evaluation results from these LLMs. Hence, for a fair assessment, we use human evaluation to measure the naturalness of generated responses in this paper.\n\n[A]: Large Language Models are not Fair Evaluators\n\n**Q6**: The authors claimed this dataset to be necessary, does that mean LLMs like chatGPT can never achieve the same level of naturalness？\n>**A6**：We're not suggesting that models like chatGPT can't attain a comparable level of naturalness. It is imperative to underscore that within the current landscape of LLMs:\n\n- Our dataset can serve as a high-quality Chinese corpus for instruction fine-tuning (SFT), aiding the open-source community in training a QA model that produces more natural and accurate responses. \n\n- Our test set can also function as a comprehensive test bed to measure the naturalness of LLM responses and their capability in  Machine Reading Comprehension (MRC) tasks.\n\n In essence, Penguin can play a pivotal role in advancing the development of Chinese LLMs, enhancing their capacity to provide more natural answers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A new dataset was built. And the prompt-BART performed better than T5-base and BART-large, given its experiment settings.\n", "reasons_to_reject": "1. This natural response is no better than the original option/answer, because most MRC questions are from designed-deliberately exams. \n2. This dataset is collected from existing datasets. Thus, it is not novel. \n3. The paper didn’t introduce details about the annotators, who the authors claimed to be professional.  \n4. There is no definition of the question difficultness, and readers cannot tell the real value of these \"natural\" responses.\n5. The metrics, although the author claimed this paper is focused on \"natural\", do not include metrics for naturalness, e.g., PPL, coherence (by ChatGPT), etc.\n6. The authors claimed this dataset to be necessary, does that mean LLMs like chatGPT can never achieve the same level of naturalness？\n", "questions_to_authors": "1. This natural response is no better than the original option/answer, because most MRC questions are from designed-deliberately exams. \n2. This dataset is collected from existing datasets. Thus, it is not novel. \n3. The paper didn’t introduce details about the annotators, who the authors claimed to be professional.  \n4. There is no definition of the question difficultness, and readers cannot tell the real value of these \"natural\" responses.\n5. The metrics, although the author claimed this paper is focused on \"natural\", do not include metrics for naturalness, e.g., PPL, coherence (by ChatGPT), etc.\n6. The authors claimed this dataset to be necessary, does that mean LLMs like chatGPT can never achieve the same level of naturalness？\n", "paper_topic_and_main_contributions": "The authors construct a dataset named Penguin to promote the research of MRC, providing a training and test bed for natural response generation to real scenarios. They also train a model named Prompt-BART with a mixture of prefix prompts in Penguin. Extensive experiments validated the effectiveness of this model.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "We showcase of each category in Table 4, respectively. -> We show cases in these categories in Table 4, respectively.\nWhat does the model perform when it answers different question types?  -> How does the model perform / What is the performance of the model...\nAppendix E: More results -> More Results.\n"}}, {"review_id": "3gaNOEA3nz", "reviewer": "Reviewer_7YEK", "summary": "", "questions": "1. Why not compared with T5-large?\n2. There are five templates in the paper, which of them was used in the experiments?\n3. All RQs and their answers should be put in the main text. Because the paper submission needs to remain fully self-contained, as these supplementary materials are completely optional.\n4. The representations should be polished, making the paper more precise and clearer.\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your meticulous review and invaluable suggestions on our work. We would like to address and clarify the key issues and suggestions you raised:\n\n\n**Q1**: The natural response is diverse. Thus, it won't be consistently better than the original option/answer.\n\n>**A1**: In order to test whether the generated responses are more natural than the original answers.  In Figure 6 of the main paper, we delved deeper into the difference in naturalness, information richness, and relevance between the natural responses and standard answers via a human evaluation. \n\n**The figure shows that more than 85% of the generated responses are superior to their counterpart answers across all evaluation perspectives, indicating they are closer to human-like responses than the generated answers.** The results substantiate that our delineation of natural responses is valuable and necessary. While a natural response may not necessarily be superior to the original answer in each case, striving for responses that are better than the original answer in the vast majority of cases has indeed become a significant motivation behind our paper.\n\n**Q2**: The results should be compared with ChatGPT w.r.t. naturalness.\n\n>**A2**: We understand your concerns about the naturalness comparison with ChatGPT. We have indeed considered this and are currently conducting relevant experiments.  In the following, we present the human evaluation of ChatGPT and our two-stage Prompt-BART. The test samples are 500 which is consistent with human evaluation settings in our main paper (423-430). As seen in the table, ChatGPT performs well across three metrics, but still falls slightly behind prompt-BART.\n\n| Method |Relevance with Passage|Fluency |Informativeness|\n| :----------- | :-----------: |:----------- | :-----------: |\n|GhatGPT|3.40|3.56|3.78|\n|Prompt-BART|3.76|3.87|3.93|\n\nHere we show a bad case of ChatGPT in Penguin. In this example, the question revolves around who Zhang Jike is and whether he has won Olympic championships. The passage we provided contains ample information to accurately answer this question. However, the response generated by ChatGPT is entirely inaccurate: In its response about Zhang Jike, there is a glaring inconsistency. **It initially states that Zhang Jike has won the 2012 Olympic championship but later concludes by stating that he has not won an Olympic championship.** This incorrect response showcases both a failure to address the current question appropriately and internal inconsistencies in the information provided.\n\n| Example ||\n| ----------- | -----------|\n|Question|Do you know Zhang Jike? Has he won the Olympic Men's Singles Championship? |\n|Passage|Zhang Jike is a Chinese male table tennis player. In 2012, he won the World Table Tennis Team Championships and secured the Men's Singles gold medal at the London Olympics in August, completing a grand slam of individual championships that includes the World Championships, World Cup, and Olympics. |\n|Generated Response|In 2012, he won the World Table Tennis Team Championships and secured the Women's Singles gold medal at the 2012 London Olympics, achieving a grand slam of individual championships that includes the World Championships, World Cup, and Olympics. **Zhang Jike has not won an Olympic Men's Singles gold medal**.|\n\n\n**Q3**: Why not compare with T5-large?\n\n>**A3**：We are not using T5-large in the paper for two main reasons: \n\n- T5-small and BART-Large have similar parameter sizes. Therefore, to maintain a fair comparison within the same parameter level in our experiments, we are not using T5-large; \n- Within the open-source community, we do not find a public open-source Chinese T5-large model.\n\n**Q4**: There are five templates in the paper, which of them was used in the experiments?\n\n>**A4**: Sorry for the misunderstanding. As shown in paper Line 392-403, we fine-tune in Penguin with all these five different prompts, increasing the size of the dataset.\n\n**Q5**: All RQs and their answers should be put in the main text. Because the paper submission needs to remain fully self-contained, as these supplementary materials are completely optional.\n\n>**A5**: Thanks for your advice, we omit some content of examples in our main paper due to the space limit. We will revise it in the upcoming version.\n\n**Q6**: The representations should be polished, making the paper more precise and clearer.\n\n>**A6**: Thank you for your suggestions. However, we would appreciate it if you could point out specific areas in our paper where the expression is not precise or clear enough. This will allow us to make the necessary improvements.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The new dataset is useful.\n2. The experimental results and analysis are convincing to some extent. \n\n", "reasons_to_reject": "1. The natural response is diverse. Thus, it won't be consistently better than the original option/answer.\n2. The results should be compared with ChatGPT w.r.t. naturalness.\n", "questions_to_authors": "1. Why not compared with T5-large?\n2. There are five templates in the paper, which of them was used in the experiments?\n3. All RQs and their answers should be put in the main text. Because the paper submission needs to remain fully self-contained, as these supplementary materials are completely optional.\n4. The representations should be polished, making the paper more precise and clearer.\n", "paper_topic_and_main_contributions": "The authors propose a new dataset named Penguin for natural response generation in Chinese MRC. Penguin requires the predicted responses not only be accurate but also natural-sounding, catering to real scenarios. They also propose Prompt-BART, an approach for fine-tuning pre-trained generative language models using a wide variety of prefix prompts in Penguin. The resulting model, as stated in the prompt that accompanied it, boosted the overall performance.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "All RQs and their answers should be put in the main text. Because the paper submission needs to remain fully self-contained, as these supplementary materials are completely optional.\nThe representations should be polished, making the paper more precise and clearer."}}]

Author rebuttal:

Meta-review: The paper introduced a new reading comprehension dataset in Chinese named Penguin. The proposed dataset is large, 200k examples. Each one is <Paragraph, Question, Answer, Response>.  The paragraph/question/answer are all from existing datasets, where the Answer is an extractive span from the paragraph. The Response is a more detailed, non-extractive answer to the question, and it is what the dataset user needs to output. 
The Paragraph, Question, Answer are translated from English datasets, followed by an automatic method to generate potential Responses, then the output is filtered automatically then by human annotators. The test set is human-authored. 
The manual effort to annotate the dataset is relatively big given the dataset size. 
The paper presented reasonable baselines (not super strong baselines). They also did human eval to compare Responses vs. Answers, and found Responses to be a lot more helpful to the user. 

Some of the reviewers expressed concerns about the naturalness of the dataset, which is a legitimate concern. 
Regarding evaluation, I would also be interested in finding out human performance and how far it is from the best model performance (e.g., an additional row in table 5), followed by an error analysis for the examples that humans get wrong.
Decision: Accept-Findings