PARROT: Zero-Shot Narrative Reading Comprehension via Parallel Reading
Authors: Chao Zhao
Anvesh Rao Vijjini
Snigdha Chaturvedi
Venue: EMNLP 2023 Findings (2023)
Keywords: narrative reading comprehension
zero-shot learning
TL;DR: A zero-shot approach for narrative reading comprehension

Abstract: Narrative comprehension is a challenging task that requires a deep understanding of the foundational elements of narratives. Acquiring this skill requires extensive annotated data. To mitigate the burden of data annotation, we present Parrot, a zero-shot approach for narrative reading comprehension through parallel reading, which involves two parallel narratives that tell the same story. By leveraging one narrative as a source of supervision signal to guide the understanding of the other, Parrot abstracts the textual content and develops genuine narrative understanding. Evaluation conducted on two narrative comprehension benchmarks demonstrates that Parrot surpasses previous zero-shot approaches and achieves comparable performance to fully supervised models. The code will be available at https://github.com/zhaochaocs/Parrot.

PARROT: Zero-Shot Narrative Reading Comprehension via Parallel Reading
Chao Zhao
Anvesh Rao Vijjini Snigdha Chaturvedi
Department of Computer Science, UNC Chapel Hill
(zhaochao, anvesh, snigdha]@cs. wnc. edu
Abstract
Narrative N: US Nevy
TOPGUN MAVERICK
Narrative N Colonel
Colorel Pate "Maveridk"
Pete "Mawerikk" Mitchel is test pilot in the Mojave Desert. day, Rear Admiral "Hammer" attempted to suspend the hyporsonic flight tost machine "Darkstar" project and intended to reallocate the funds to program command. Before Cain arrived, Miverick decided to push the prototype to its contractually spectied speed of Mach
Narrative comprehension is challenging task that requires deep understanding of the foun- dational elements narratives. Acquiring this skill requires extensive annotated data. To miti- gate the burden of data annotation, we present PARROT, a zero-shot approach narrative reading comprehension through parallel read- ing, which involves two parallel narratives that tell the same story. By leveraging one narrative source of supervision signal to guide the understanding of the other, PARROT abstracts the textual content and develops genuine nar- rative understanding. Evaluation conducted on two narrative comprehension benchmarks demonstrates that PARROT surpasses previous zero-shot approaches and achieves compara- ble performance to fully supervised models. The code will be available at https://gt thub. com/2hhacchocssParrro
Mitchell is test pilot of the Iyperssnk
scramiet program: Rear Admiral Chester "Hammer" Cain plans to cancel "Darkstar" favor of funding a more gromising drone program llo meet the final contract specification the program Maverick unilaterally changes the speed for that day's test from Mach 9 to Mach 10.
Q1: Who is the test pilot? [exact match) Pate "Miveric"" Mitchell
Which program
cancelation? [paraphrase]
03: What's the relation between Maverick and Darkstar? [mutti hop reasaning) test Darkstar
Qa: What motivates Hommer to fund drone program? (character's mativation)
He thinks that program is more promising
as: Why does maverick pilot Darkstar Mach 10? [event causality)
To moet the final contract specification and save the program
Figure I: Illustration of parallel reading. W and V different rendorings of the same story. The key idea is to ask questions from N and encourage the model to answer them from This helps the model in learning deep comprehension skills (as indicated in []).
1 Introduction
Narratives have long been recognized as valu- able resource for linguistic, scientific, cultural, and social learning (Rosen, 1985; Knoespel, 1991; Lyle, 2000; Nash, 2005; Bettelheim, 2010). Nar- rative comprehension. therefore, is considered a fundamental aspect of human intelligence (Bruner, 1997) and an important tool for cognitive develop- ment and meaning-makin (Polkinghorne, 1988). With this motivation, previous research has tack- led the task of narrative reading comprehension, which involves automatically comprehending a given narrative and answering questions related to it (Hirschman et al., 1999; Richardson et al., 2013). However, in comparison to general text com- prehension, which typically focuses on the under- standing of named entities and factual informa- tion (Rajpurkar al., 2016), narrative comprehen- sion presents unique challenges. Specifically, requires understanding the foundational elements narratives. These elements include events along with their temporal and causal connections; settings
such as the time, place, and environment; as well as characters, including their motivations, desires, emotions, and relationships with other characters. Together they exhibit intricate plot structures and involve complex character interactions, making it challenging for machines to comprehend. Despite the availability of extensively annotated data for general text reading comprehension. there is cur- rently lack of sufficient annotated data in the narrative domain, and it is not optimal to directly use models trained general text data for narra- tive reading comprehension. Hence, there is need to develop data-efficient learning approaches for narrative reading comprehension.
To address the aforementioned challenges, our idea to leverage parallel reading: reading two parallel narratives that convey the same story but differ in various aspects of story-telling style. This

---

TOP GUN MAVERICK
Pre-training: Parallel Reading
NER, SRI, Constituency Parsing
Narrative Masking
Mask Prediction
Narrative V
Masking & Prediction
Narrative N
Us Navy Colonel (Pete "Maverick" Mitchell res-ren)
US Novy Colonel [mask]] [who] test pilot of hypersonic "Darkstar" scramjet program, Rear Admiral Chester "Hammer" Cain plans to cancel "Darkstar" favor [mask2 what]. [mask3] whyy), Miverick unilaterallyy changes the target speed for that day's test from Mach Mach 10.
the hypersonic "Darkstar" scramjet program. Rear Admiral Chester "Hammer" Cain plans to cancel "Darkstar"
Colonel Pete "Maverick" Mitchell as test in Mojave
day, Rear Admiral Chester "Hammer" Cain attempted to suspend the hypersonic flight machine "Darkkstar" project and intended reallocate the funds to an unmanned drone under his command. Before Cain arrived, Maverick decided to push the prototype to its contractually specified speed Mach
(funding promising drane program coNsr-vel, ITo meet the final contract specification and save the program stumt)), Mavenic unilaterally changes the target speed for that day's test from Mach 9 Mach 10.
[mask] [who]: Pete "Miverick" Mitchell
[mask2 what] funding a more promising drone program [maska] why/ To moet the final contract spedffication and save the program
(Some plots have been omitted.... ..]
Alteward, Maverick visited Penny's shop, but she was away on yacht cruis. They spent time together maintairing his house. One day, Penny visited with her daughter. Then Maverick took Panny aboard P-51, taking off towards the sunset. Mesnwhile, Rooster looked at a photo of himselt, Goose and Maverick, reminiscing with smile.
Inference: Narrative
P-51
Question Type Identification
Question Transformation
Mask Prediction
Quession: What did Roester do when looked the photo of himsetf, Goose, Maverick? Massed Statement: Rooster (mask1 [whha]] when he looked at the photo of himself, Goose, and Maverick. Answer / [macks [what]: raminisced about his father with a smile
Figure 2: Illustration of the proposed approach, PARROT. During pre-training, we collect two parallel narratives, N and We mask narrative-specife spans in N and pre-train the model predict these spans by reading N During inference, we transform the question into a masked statement, following the pre-training format. Then we apply the pre-trained model to predict the answer based on the narrative and the masked statement. Note that for illustrative purposes, N is shared between pre-training and inference, but in real scenarios, there is overlap.
idea aligns with the classical model of narrative theory (Genette, 1983), which emphasizes the per- spectival nature of narratives - narratives encom- pass not only the sequence of events (the story), but also the ordering, granularity. point-of-view: and localization (the discourse and narrating). Ideally, comprehending either narrative would result in the same understanding of the story. Therefore, we can teach the model to develop reading comprehension skills by asking questions based on one narrative and encouraging the model to answer them by read- ing the parallel narrative. Figure 1 illustrates this concept. We will explain later how we operational- ize this idea of asking and answering questions through masked language modeling.
such implicit information empowers to surpass superficial understanding and grasp the implicit information and underlying meaning within the narrative (as seen in Q4 and Q5 in Figure 1). These advantages have been demonstrated in pedagogy to improve students' reading comprehension abilities (Schumaker al., 1984; Grellet, 1981; Yano al., 1994; Sipe, 2001).
With this idea in mind, we propose PARROT 1 novel pre-training approach for zero-shot narra- tive comprehension. Figure shows overall illustration. It selectively masks important narra- tive elements within one narrative, and then pre- trains the model to predict these masked elements by reading the parallel narrative. To encourage PARROT to learn about a wide array of narrative el- ements, we mask a diverse set of elements covering characters, events, time, place, environments, and more. Lastly, enable PARROT to perform narra- tive reading comprehension in zero-shot manner, we narrow the disparity between the pre-training task of span prediction and the downstream task of reading comprehension by aligning their data formats.
Learning from parallel reading offers two ad- vantages. Firstly, by exposing the model to narra- tive variations of the same story, we discourage its reliance on text-matching and enhance its ability to comprehend paraphrases, integrate information from long contexts, and perform multi-hop reason- ing (as seen in Q2 and Q3 in Figure 1). Secondly, one narrative may contain information that is not explicitly stated in the other narrative, but can be implicitly inferred through a deeper understand- ing of the context. Training a model to deduce
Stands for parallel reading for zero-shot narrative com- prehension.

---

We conducted experiments on two narrative reading comprehension benchmarks, narrativeQA (Kočiský et al., 2018) and FairytaleQA (Xu et al., 2022), to evaluate PARROT. The results demon- strate that without any human annotation, PAR- ROT achieves performance that is comparable to that of fully supervised model. Furthermore, PAR- ROT exhibits superior performance compared to su- pervised models when applied out-of-domain datasets, demonstrating its effectiveness in transfer learning scenarios.
masked language modeling (MLM) based pre- training approach, which incorporates selective span masking strategy to mask essential narrative elements (Sec. 3.1) and parallel reading strategy to learn to predict the masked spans (Sec. 3.2). Next, to utilize the pre-trained model in a zero-shot fashion, we transform the downstream narrative reading comprehension task to match the format of the pre-training task (Sec. 3.3). Figure 2 shows overall illustration.
Our contributions are three-fold:
3.1 Selective Span Masking
We present PARROT, novel pre- pre-training ap- proach for effective zero-shot narrative com- prehension.
In model pre-training, a commonly used technique masked language modeling (MLM), where spans are randomly masked for the model to predict (De- vlin et al., 2019; Raffel et al., 2020). In previous works on pre-training for reading comprehension, named entities and recurring spans were masked they are more closely associated with factual infor- mation (Ram et al., 2021; Bian al., 2021). How- ever, for narrative comprehension, the model needs to understand not just named entities but also vari- ous other narrative elements such as events, causal- ity, temporal relationships, environmental settings, characters, their desires, personality traits, and re- lationships with others, to name few. Previous masking strategies do not cover all these essential elements adequately.
We introduce novel parallel reading strategy that involves utilizing different versions of nar- ratives during pre-training to foster genuine narrative understanding.
Our approach achieves competitive or better performance when compared to supervised models, showcasing its effectiveness in narra- tive comprehension tasks.
2 Related Works
Data-efficient reading comprchension has been addressed by automatically generating (passage, question, answer) triples for pre-training model (Lewis al., 2019, 2021). However, requires annotated data train a question- -ettriotinnnnttiionn mod- ule. One possible solution is to replace it with a heuristic-based question-generation module (Heil- man and Smith, 2010; Li et al., 2020; Lyu et al., 2021). Another option is to replace natural ques- tions with cloze questions by masking the an- swer from the passage (Hermann et al., 2015; Hill al., 2015). Researchers have proposed mask- ing and predicting recurring spans within the pas- sage as answers (Bian et al., 2021; Ram et al., 2021). In contrast to these approaches that focus on general extractive reading comprehension, PAR- ROT specifically addresses the unique challenge of open-ended narrative comprehension and leverages parallel reading for abstractive comprehension and free-form question answering.
Therefore, enhance the model's 's ability to com- prehend narratives, we incorporate a diverse set of masked spans to encourage the learning of a wide range of comprehension skills specific to narratives. We carefully select three types of spans to mask.
Named entities: Named entities play crucial role in narratives as they help identify char- acters and settings (such as time and place) within the narrative. We choose nine types of named entities: Person, Location, Gcopoliti- cal Entity, Facility, Organization, Time, Date, Event, and Products.
Semantic roles: Named entities alone encompass all narrative elements, such as settings like last week and small town, event causality, characters' purpose, and more. Since these narrative elements usually unfold along with events, we focus on the associated arguments of verbs and include five semantic roles: 3 Direction CarriionnAGGMAM (ARGM-DIR), Location
3 PARROT
narrative reading comprehension, the input is narrative A and question q, while the output is a concise answer a. We develop PARROT, zero- shot solution for this problem. PARROT utilizes
We employ Spacy for NER. https://spacy: *We employ AllenNI.P for SRI., https://allem/ .rrrg allennip

---

(ARGM-LOC), Time (ARGM-IMP), Pur- pose (ARGM-PRP), Cause (ARGM-CAU), and Manner (ARGM-MNR).
this noise, we apply two filtering steps: one at the sentence level and another at the span level. At the sentence level, for each sentence & N, we require the Rouge- Precision score (Lin, 2004) between and W to surpass a predefined threshold. If it does not, we do not mask spans from s. This criterion ensures that the remaining sentences in N align closely with the corresponding content in N At the span level, we selectively mask spans in N that directly or indirectly appear in A For spans that correspond their presence in N using exact match. For spans that correspond to semantic roles or constituency phrases, which are more likely to be paraphrased. we adopt more lenient criterion. For them, we calculate the Rouge-1 Precision score between the span and A setting threshold to determine the acceptability of the span candidates for masking. Lastly, we pre-train the model to predict the masked content within N, given the concatenation of the masked narrative, ((m(A)) together with the longer narrative, '' The loss function is
Verb and adjective phrases: A narrative can be seen as sequence of events organized by a narrator (Schank and Abelson, 2013). To directly comprehend events, we identify verb phrases using constituency parsing and mask them. Additionally, we mask adjective phrases to enhance the understanding of nar- rative settings and character characterization. Given a narrative N, we first identify and mask some spans, m(N), and then pre-train a sequence- to-sequence model to predict these spans using the remaining text, AV mm(), and the original narrative, N. We refer to this model as singe The loss function is
named entity, we verify
Single logr (m(N) | Mm
3.2 Parallel Reading
Predicting masked spans using the original narra- tive N may result the model trivially relying on the superficial lexical overlap. To mitigate this issue, we propose "parallel reading". Instead of solely masking and predicting spans within a sin- gle narrative N, we leverage an additional parallel narrative, denoted Ar+, which tells the same story as V but differs in granularity, point-of-view, etc. By simultaneously reading A and N and predicting the masked key information, we encour- age the model to abstract the textual content and foster a genuine and deeper comprehension nar- ratives, avoiding overreliance on superficial textual matching clues. For example, in Figure 2, predict- ing [mask2] and [mask3] in N based on N re- quires more advanced comprehension skills, such understanding character motivation and event causality.
Cparallet - log (m(N) (m (2)
3.3 Adapting Reading Comprehension
In general, after the MLM pre-training, the pre- trained model requires fine-tuning with additional data to adapt the specific downstream task. This fine-tuning is necessary because the pre-training task and the downstream task can be in different formats. However, in this paper, we do not assume access to the availability of any fine-tuning data and directly utilize the pre-trained model in a zero- shot manner. The key insight is that the reading comprehension task can be transformed into the MLM task. For example, in Figure 2, the question "What did Rooster do when he looked al the photo?" can be transformed into "Rooster [mask] when he looked the photo", and the answer can be obtained by filling in the masked part. To achieve this transformation, we use QA2D (Demszky et al., 201 8), which leverages a neural sequence model to generate masked statements from questions.
Here we provide more details regarding paral- lel reading. Without loss of generality, we assume that ' is longer than A. We selectively mask spans in the shorter narrative, .,, and utilize the longer narrative, - as source of evidence to predict the masked spans, since the longer narra- tive is likely to contain the necessary information present in the shorter narrative.
One drawback of this transformation strategy, well as the pre-training strategy, is that the masked statement does not contain the question-type in- formation typically conveyed by the wh-word in questions. For instance, without the original what question with the answer of "reminisced about his father with a smile", the masked statement in our example from Figure 2 can also be interpreted as
However, A might also contain some spans that are not answerable from * Masking such spans can result in noise in the training data. To mitigate
WW employ AllenNIP for constituency parsing.

---

Wh- Type
Span Type
N' into smaller segments and predict the spans within each segment separately. However, we also need to strike a balance as excessively short seg- ments would increase the overall number of train- ing instances. Therefore, we opt to divide A into segments based on every three sentences.
Who
DERPPERSON
When
TIRREDET//ATE ARGM-TMP
Where
NER-LOC/GEOOFAC, ARGM-LOC/DIR
Why
ARGM-CAU/RRP
How
ARGM-MNR, ADJP
What
Others
Datasets for Evaluation: To evaluate the perfor- mance of PARROT, we conduct experiments on two narrative reading comprehension benchmarks: Nar- rativeQA (Kočiský et al., 2018) and FairytaleQA (Xu et al., 2022). Since PARROT is zero-shot, we solely use the test sets of these datasets for evalu- ation. The narratives in FairytaleQA are derived from children's 's stories, while the narratives in Nar- rativeQA consist of plot summaries from books and movie scripts. For NarrativeQA, to avoid any potential overlap with the pre-taining data, we only consider instances derived from books for evalua- tion purposes. The average length of the narratives in these datasets is 150 and 659 tokens, respec- tively, and their test sets contain 1,007 and 10,557 question-answer pairs, respectively.
Table I: The mapping between the type of question and the corresponding type of masked span. This mapping enables the model to identify the appropriate type of question during pre-training.
a how question, leading to the possibility of fill- ing the mask with a different answer such as "felt delighted"'
To mitigate this ambiguity, we introduce a spe- cial type token preceding the mask to provide more accurate information about the question type. This token, as we illustrated in Figure 2, is typically wh-element, such as who and what, which is extracted from the original question. To extract these words, we employ a constituency parser to parse the question and then identify the ele- ments labeled with syntactic tags such as "WHNP", "WHADVP", "WHADJP", or "WHPP". During pre- training, since we lack the actual questions, we in- fer the question type based the type of masked spans. The mapping between the span type and the question type is provided in Table 1.
4.2 Setup
Implementation Details: The underlying model in PARROT is T5-base (Raffel et al., 2020). We chose T5 because has been pre-trained on a simi- lar MLL task. Furthermore, compared with other MLs-basee pre-trained models such as BART, T5 only predicts the masked tokens, making it more computationally efficient. During pre-training, we employ the AdamW optimizer (Loshchilov and Hutter, 2017) with learning rate of 3 10-5 and batch size of 512. We choose large batch size because the pre-training data can be noisy. We incorporate warmup for the first 50 steps and im- plement carly stopping based on the model's perfor- mance on the validation set. Training the models is conducted on four Tesla 3090 GPUs with 24 GB memory, taking approximately 4 hours to complete the pre-training process.
By transforming the question to a masked state- ment during inference and incorporating the ques- tion type during pre- training, we establish a consis- tent data format for both pre-training and inference. This thereby empowers the model to perform zero- shot inference without explicit fine-tuning.
4 Experiments
In this section, we evaluate the performance of PARROT.
4.1 Datasest
Datasets for Pre-training: For parallel reading in the pre-training phase, we utilize NarraSum (Zhao et al., 2022), a dataset of 122K parallel narrative pairs obtained from plot descriptions of movies and TV episodes. After processing, we obtain total of 57.4K paired narratives and 154.5K question- answer pairs. The average lengths of A and + are 125 and 926 tokens, respectively. Each narra- tive pair includes 2.7 masked spans on average.
Baselines: Our first baseline is an information retrieval (IR) baseline adopted by Kočiský et al. (2018), which selects the most similar sentence in the narrative to the given question and considers it as the answer. For computing this similarity, we use TF-IDF based cosine similarity. To establish stronger baselines, we compare PARROT with the model described in Lewis et al. (2021), which auto- matically generates question and answer pairs from the narrative. This involves utilizing an answer
To reduce input length and enhance computa- tional efficiency, we partition the shorter narrative

---

FairytaleQA Rouge-2
Narrative Rouge-2
Rouge-1
Rouge-L
Rouge-1
Rouge-L
T5 Finetuned on FairytaleQA
54. 64 49.64
40.43
54,03
45.29
25.73
44.59
T5 Finetuned NarrativeQA
38.45
49.10
65.05
36.04
64.49
IR (TF-IDF) (Kočiský et al., 2018)
21.64
14.30
20.82
16.61
7.79
15.67
AE-QG (Lewis al., 2021)
43.29
29.81
42.89
53.61
28.11
53,27
Dicuna-138 (Chiang al., 2023)
37.52
20.44
35.98
32.59
17.37
31.46
ChatGPT
44.32
27.10
43.49
41.27
24.63
40.07
PARROTSinEE PARROT
40.32
30.35
40.01
50.01
26.78
49.60
48.56
36.83
48.10
55.71
30.81
55.32
Table 2: Results evaluated on FairytaleQA and NarrativeQA by Rouge scores. PARROT outperforms all baselines and achieves comparable or superior performance compared to supervised models in the out-of-domain setting.
extraction (AE) model and a question generation (QG) model trained on three MRC datasets: NQ (Kwiatkowski et ul., 2019), TriviaQA (Joshi et al., 2017), and SQuAD (Rajpurkar et al., 2016). With these models, generate question and answer pairs from the narratives in NarraSum, and then train reading comprehension model based on T5- base. We refer this baseline as AE-QG.
comparable to those of fully supervised upper- bounds in terms of Rouge-L (48. 10 vs. 54.03 and 55.32 vs. 64.49). These results demonstrate the ef- fectiveness of PARROT in narrative reading compre- hension. When comparing the strategies of single and parallel reading, PARROT achieves significantly higher performance compared to its single-reading counterpart, PARROT on both datasets. This result emphasizes the crucial role parallel read- ing in enhancing model performance.
Additionally, we compare PARROT with Chat- GPT state-of-thhe-at large language model, and Vicuma- 13B (Chiang al., 2023), one of its best open-source alternatives. To use these models, use the instruction "Please generate brief answer rather than complete sentence to the following question based on the provided passage as evidence.", alongside the passage and question that are appended.
We also compare PARROT supervised models under the oft-of-domain setting, i.e., training the supervised model on one dataset and evaluating it on another. These results are displayed in gray font in the table. PARROT demonstrates competitive per- formance on FairytaleQA (Rouge-L of 48.10 vs. 49. 10) and superior performance on Narrative (Rouge-L of 55.32 vs. 44.59). This further demon- strates that PARROT acquire general narrative comprehension skills and effectively apply them diverse narratives.
Lastly, compare with fine-tuned models. We fine-tune the T5-base model the training sets of AarrativeQA and FairytaleQA, resulting in two fine-tuned models. We treat these results as upper bounds due to their supervised nature.
Among the large language model baselines, ChatGPT exhibits stronger performance than Vicuna-13BB AE-QG models also achieve strong performance. However, these models require ad- ditional training data for training the answer ex- traction and question generation components. Fur- thermore, the generated question-anse -nnweer pairs may contain errors, which could potentially impact the model's ' overall performance during pre-training.
Evaluation Measure: Following the official evalu- ation of the two benchmarks, we Rouge scores (Lin, 2004) between the predicted and the gold answers to evaluate the models.
4.3 Results
Table 2 presents the performance of PAR- ROT and baselines on FairytaleQA and Narra- tiveQA datasets. PARROT exhibits superior perfor- mance, significantly surpassing all zero-shot base- lines (approximate randomization (Noreen, 1989; Chinchor, 1992), p < 0.01) Additionally, achieves performance that is 89.0% and 85.8%
4,4 Human Evaluation
To obtain reliable assessment of the model performance, further conduct human evalu- ation via Amazon Mechanical Turk (AMT). We randomly select 100 test instances from the test sets of both datasets. For each instance, we show three independent annotators the question, correct
'ht /chat wpw.tai..pnnioommm
"Wo tried different instructions and select the best- performing one.

---

FairytaleQA
NarrativeQA
PIT 200
IR (TF-IDF)
2.12
2.37
What
AE-QG
2.56
2.91
Vicuna-13B
2.36
2.61
NF
ChatGPT
2.49
2.86
PaRROTangle
2.30
2.78
VITT
PARROT
2.71
3.10
When
DIG
KNGMA ANGNA-HIP
Table 3: Results of human evaluation on FairytaleQA and NarrativeQA.
Figure Distribution of the types of wh-elements and the sources of masked spans in pre-training data.
answers, and the answers generated by various sys- tems. We then ask annotators to rate the quality of the predicted answers on Likert scale ranging from 1 to 5. To maintain the evaluation quality, we require annotators to AMT Masters based the
FairytaleQA
NarrativeQA
Random
8,38 / 8.31
13.54 4.02 13.52 10.67 4.38 10.54
None
6.199/ 11.84
NE
23.89//3.88) 34.98
48.72/ 24,93 48.53
United States, with more than 1,000 HITs approved and an approval rate exceeding 98%. We manually review the annotation results, and if we identify annotators consistently providing low- cow-quality an- notations, we block them and re-assign their tasks, Annotators compensated at rate of $14 per hour, exceeding the local minimum wage.
SR
45.55/34.20 45.21
54.83 29.95 54.46
CP
48.56 36.83 48.10
55.71 30.81 /5..32
Table 4: The contribution of each source of masked spans the final performance (R-1/R-2/R-LL. We start with TS-base with and without further pre-training (Ran- dom and None). We then incrementally introduce named entities (NE), semantic roles (SR), and con- stituency phrases (CP) into the pre-trained data.
Table shows the results of human evaluation. The inter-annotator agreement score is 0.7003 in Gwer's gamma. Results from both datasets, along with the automatic measures, consistently demon- strate that Parrot outperforms the baseline models.
almost all of them fall under the category of verb phrases.
5 Analysis
To investigate the impact of different mask types on the overall performance, we conduct ablation study. During the construction of the pre-training data, we gradually expand the type of masked el- ements from named entities to semantic roles and constituency phrases. We also compare with a ran- dom masking strategy that aligns with the original pre-training objective of T5. The performance of the models trained on these versions of the pre- trained data is presented in Table 4. It reveals that focusing named entities can improve the model performance, which aligns with previous research findings. However, relying solely on named enti- ties insufficient to encompass all narrative ele- ments. By incorporating semantic roles, the model achieves a substantial improvement in performance. By including constituency phrases, we observe a further enhancement. On the contrary, when we continue pre-training with random span masking strategy, do not observe improvement in model performance. These results support our hypothe- sis that incorporating diverse range of masked spans can significantly enhance models' ability of narrative comprehension.
We conduct analysis to better understand the be- havior of PARROT.
5.1 Type of Masked Spans
One of our work's major contributions incor- porating carefully selected and diverse set of masked spans geared toward narrative comprehen- sion. To highlight the diversity, we analyze the dis- tributions of different question types and the types of masked spans in the pre-training data. The re- sults are presented in Figure 3. In terms of question types, the pre-trained data contains six major types: what (41.7%), who (25.6%), when (13.4%), where (10.4%), why (5.6%), and how (3.3%). ). In terms of masked spans, it shows that named entities (NE), semantic roles (SR), and constituency phrases (CP) are evenly distributed within the pre-training data. Specifically, named entities are predominantly rep- resented by PERSON (72.0%) and ORG (16.1%)
categories. Within semantic roles, Time (41.5%) Location (25.5%), and Purpose ( (14.9%) are the top three categories. Within the constituency phrases,

---

Parallel Reading
Parallel Reading
0.75
Rouge:L
Supervised
Rouge- Recal
0.8
Single Reading
Rouge-1, Recail
0.8
Single Reading
0.50
0.25
0.6
0.6
who
where
what
why
how
0.4
0.4
Question Types
0.2
0
10
0
1D
Parrot
Level Abstractiveness
Level of Abstractiveness
Rouge-L
Supervised
0.50
Figure 5: Model performance on FairytaleQA (left) and narrative (right) w.r.t the abstractiveness level be- tween the question and the narrative. We report Rouge-L. Recall to evaluate whether the correct answer is included in the predicted answer.
0.25
character setting
action
causal
prediction
outcome
feeling
Narrative Elements
Figure Fine- Line-grained model performance on Fairy- tableQA w.r.t. the types of questions (top) and narrative elements (bottom).
alyze how parallel reading impacts the model's 's performance when the question is less lexically overlapped with the narrative.
5.2 Decomposition of Model Performance
To accomplish this, we divide the test set into subsets based on the level of abstractiveness be- tween the question and the narrative. More specifi- cally, we first identify the most similar sentence in the narrative with the question the evidence sen- tence, and then use the sum of Rouge-1 precision and Rouge-2 precision between the question and the evidence sentence to approximate the level abstractiveness.. Higher Rouge Precision indicates lower abstractiveness, Figure 5 shows the model's performance based on the degree of abstractiveness between the question and the narrative.
We proceed to conduct thorough analysis of the model's performance finer granularity. To ac- complish this, we partition the FairytaleQA dataset into smaller subsets based on question types and narrative elements, as annotated within the dataset.
Then we evaluate the model's performance on the individual subsets and compare it with the perfor- mance of the supervised model. The results are illustrated in Figure 4.
Comparing the results with the supervised model, PARROT demonstrates competitive performance in questions that involve identifying characters (who) and their activities (what), establishing causal rela- tionships between events (why), and understand- ing the setting of the narrative (where). How- ever, when dealing with more intricate narrative aspects such pinpointing outcomes, predicting unknown events, and deciphering characters' emo- tional states (how), PARROT exhibits larger perfor- mance gap. This particular strength and weakness align with the distribution of the types of masked spans present in the pre-training data. We leave enhancing the comprehension of these narrative components for future work.
The results demonstrate that, in general, as the question becomes increasingly abstractive (the right side of the r-axis), the performance gap be- tween PARROT and PARROT becomes more significant. This finding indicates that compared with PARROT PARROTiS better at understand- ing abstractive questions and finding answers based on genuine comprehension, rather than mere text matching. It supports our motivation that parallel reading enhances the model's ability to compre- hend the underlying meaning of the narrative.
Interestingly, in highly extractive scenarios (the left side of the raaxis), PARROT also outperforms its single-reading counterpart on FairytaleQA. This is because PARROT single tends to directly copy text from the narrative, which sometimes results in rors related to answer resolution when the copied part includes a pronoun instead of the proper entity mention. In contrast, PARROT is capable to select the appropriate entity mention as the answer, rather than mechanically copying the pronoun.
5.3 Impact of Parallel Reading
In addition to incorporating diverse array of nar- rative clements, significant contribution of PAR- is leveraging parallel reading abstract the textual content and comprehend the underlying meaning of the narrative. As discussed in Section 4.3, PARROT achieves better overall performance compared to PARROT In this section, we an-

---

Fine-tuned
Nurrative I: Twu pussed, and the youth longer thought of any danger. Henos, when the flowers begin to bloom the spring, Rose-Red went driving in a small wagon outside the city, near the river And there one of the prince's is servants her, informed master. since he could not conceal the muiler, told him the whole story exacily as it had happened. Said prinee: "The whole blame reproach you. Yet since she is new your wife I will let the whole matter rest. But Melo will have sufler for it!"
60
latter sent
youth,
Rouge e-L
55
Rose-Red. do
Fine-tuned
NarrativeQA FairytaleQA
Question: Why did Tsui tell prince the whole story exactly it had happened?
45
Model Oulputs:
220M
770M
3G
11G
20G
Model Size
Vieuna-133k Rose-Red told the prince the whole story because the prince asked her to.
ChatGPT: To inform the peince of what had happened.
Figure 6: Model performance w.r.t. the size of underly- ing models.
AE-QG: He could not conceal the maiter.
PARROT: Reemss he could not conceal the matter.
Reference: He could not conceal the matter.
"That is a pity," returned the young man politely, but in his beart he rejoiced, for he wished to be alone. Leaving the house, he entered grove trees which stood behind, and took the box from his pocket raised the lid, and out flew the three little birds. "Good master, what shall we do for thee?" asked they, and be unswered. "Bring me seen, and the grandest dress, and glass shocs." "Thy are here, master," said the birds, and they splendid, Mounting the hons he rode into the ground where horses were ussembling for the beasts were there which won many races, left them all behind, and he was first at the winning post. The king's is daughter waited for him gor off his horse, and pat on his old clathes, and hade the hox place some gold in his pockets. After that went back the weaver's 's house, and told him that the gold had been given him by the man who had won the racc, and that the weaver might have for his kindiness him. Question: Why did lan need the finest borse?
Nurrative II:
5.4 Scaling to Larger Models
We further investigate the potential benefits of PAR- ROT'S 's pre-training strategy for larger models. To this end, we experiment with different sizes of the underlying T5 model, namely base (220M), large (770M), XL (3B), XXL (1IB), and UL2 20B (Tay al., 2023), a T5-like large model. We fine-tune the entire parameters for T5 base and T5 large. For larger models, we utilize LORA (Hu et al., 2022) to facilitate more efficient fine-tuning due to the large computation. The results of these models are presented in Figure 6.
that over was
and never had the young
anything
and look his place among them. Many good son
the
prize, he
the wood, and
Model Outputs:
Vicuna- 138:: lan needed the finest horse for the race.
ChatGPT: To participate the great race.
AE-QG: pily.
The results show that as we increase the size of the underlying model, the performance improves gradually, approaching and eventually surpassing the performance of the fine-tuned baselines. This experiment indicates that PARROT can effectively enhance a model's 's ability of narrative comprehen- sion irrespective of its size.
PARROT: To win the race.
Reference: He wanted to win the horse-mee.
Table 5:
Sampled answers generated by different sys- tems. For each example, we show the narrative, model predicted answers, and the referenced answer.
is the primary reason he sought the "finest" horse. Thanks to the long-term reasoning skills acquired during parallel pre-training, PARROT accurately an- swered this question.
5.5 Qualitative Analysis
Table 5 shows two examples of generated answers from different systems. To address the first ques- tion correctly, the model must distinguish that "Tsui" refers to the young man rather than "Rose- Red," who is his wife. Vieuna falls short in making this distinction. ChatGPT offers a consecutive an- swer that lacks useful information. In contrast, PARROT provides the correct answer. The second narrative is notably longer than the first, with the answer to the question not explicitly provided in the text. To answer this question, the model needs to establish a connection between the earlier plot when Ian requested the finest horse and the later plot where he won the race. AB-QG fails to estab- lish this connection, resulting in a lack of useful information. Vieuna and ChatGPT partially answer the question by mentioning the race, but do not em- phasize Tan's 's motivation to "win the race," which
Conclusion
We introduce PARROT, novel zero-shot approach for narrative reading comprehension based on pre- training. By selectively masking significant el- ements within the narrative and pre-training the model to predict these spans through parallel read- ing, PARROT learns to abstract essential textual con- tent and gains a genuine understanding of the narra- tive. Experimental results on two diverse narrative datasets demonstrate the superiority of PARROT, showcasing its effectiveness in enhancing narrative reading comprehension. Our analysis further em- phasizes the significance of employing diverse range of masked spans and leveraging the parallel reading strategy during model pre-training.

---

Acknowledgements
Jerome Bruner. 1997. The culture of education. In The Culture of Education. Harvard university press.
This work was supported in part by NSF grant DRL-21 12635. We thank anonymous reviewers for their thoughtful and constructive reviews.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric P. Xing, 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Limitations
One limitation of our approach, as discussed in the analysis section, is that the selective masking strategy employed PARROT may not adequately encompass some intricate narrative elements such as event prediction and user emotion. Exploring these aspects is left as future work. Another limi- tation is that we rely on specific NLP modules for identifying the spans to be masked and transform- ing natural questions into masked statements. It is woithwhile to investigate the potential of employ- ing large language models to replace these special- ized modules.
Nancy Chinchor. 1992. The statistical significance of the muc-4 results. In Proceedings of the 4th confer- ence on Message understanding, pages 30 50. Asso- ciation for Computational Linguistics.
Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming question answering datasets into nat- ural language inference datasets. arXiv preprint 1rX.vv180.00292.
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Additionally, our focus in this paper is specifi- cally on narrative question answering. While we have demonstrated the effectiveness of PARROT in this particular task, we have not extensively ex- plored its performance in other narrative compre- hension tasks such as summarization, or scenarios where questions are not well-formed. Finally, we only conduct experiments on English datasets. would be beneficial to further explore our approach to other languages.
Gérard Genette. 1983. Narrative discourse: An essay in method, volume 3. Cornell University Press.
Francoise Grellet. 1981. Developing reading skills: A practical guide to reading comprehension exercises. Cambridge university press.
Michael Heilman and Noah A. Smith. 2010. Good question! statistical ranking for question generation. In Human Language Technologies: The 2010 An- nual Conference of the North American Chapter of the Association for Computational Linguistics, pages 609-617, Los Angeles, California. Association for Computational Linguistics.
Ethical Considerations
To address energy consumption concerns, we con- duct the majority of experiments of PARROT using the relatively smaller T5-base model. We develop and test PARROT on publicly released datasets. Since we only pre-trained and tested on few datasets, the developed model may exhibit incor- rect answers for specific questions or demonstrate bias towards certain types of narratives. We would encourage downstream users of PARROT to proac- tively anticipate and mitigate these potential risks.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. Advances in neural information processing systems, 28.
Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The goldilocks principle: Reading children's books with explicit memory representa- tions. CoRR,
Lynette Hirschman, Mare Light, Eric Breck, and John D Burger. 1999. Deep read: A reading comprehension system. In Proceedings of the 37th annual meeting of the Association for Computational Linguistics, pages 322-332.
References
Bruno Bettelheim. 2010. The uses of enchantment: The meaning and importance of fairy tales. Vintage.
Ning Bian, Xianpei Han, Bo Chen, Hongyu Lin, Ben He, and Le Sun. 2021. Bridging the gap between language model and reading comprehension: Unsu- pervised mre via sulf-supervvsion. arXiv preprint arXiv:2107.08582.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Reprexentations.

---

Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of for the Association for Computational Linguistics (Vol- une 1: Long Pupers), pages 1601-1611.
Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cristopher Nash. 2005. Narrative in culture: The uses of storytelling in the sciences, philosophy and litera- lure. Routledge.
Kenneth J Knoespel. 1991. The emplotment of chaos: instability and narrative order. Chaos and order: Complex dynamics in literature and science, pages 100-122.
Eric W Noreen. 1989. Computer- intensivee methods for testing hypotheses. Wiley New York.
Donald E Polkinghorne. 1988. Narrative knowing and the human sciences. Suny Press.
Tomáš Kočiský. Jomathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Ed- ward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Transactions of the Asso- ciation for Computational Linguistics, 6:317-328.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. Journal of Machine Learning Research, 21:1- 67.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee. et al. 2019. Natural questions: benchmark for question answering research. Trunsactions of the Association for Computational Linguistics, 7:453- 466.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natu- ral Language Processing, pages 2383-2392.
Patrick Lewis, Ludovic Denoyer, and Sebastian Riedel. 2019. Unsupervised question answering by cloze translation. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics. pages
Ori Ram, Yuval Kirstain, Jonathan Berant, Amir Glober- son, and Omer Levy. 2021. Few-shot question an- swering by pretraining span selection. In Proceed- ings the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1ith International Joint Conference on Natural Language Processing (Volume I: Long Papers). pages 3066-3079.
Patrick Lewis. Yuxiang Wu, Linging Liu, Pasquale Min- ervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. 2021. Paq: 65 mil- lion probably-asked questions and what you can do with them. Transactions of the Association for Com- putational Linguistics, 9:109811115.
Matthew Richardson, Christopher JC Burges, and Erin Renshaw, 2013. Metest: challenge dataset for the open- machine comprehension of text. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 193- 203.
Zhongli Li, Wenhui Wang. Li Dong, Furu Wei, and Ke Xu. 2020. Harvesting and refining question- answer pairs for unsupervised QA. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6719-6728, On-
Harold Rosen, 1985, Stories and meanings. National Association for the Teaching of English.
line. Association for Computational Linguistics.
Chin-Yew Yew Lin. 2004. ROUGE: package for auto- matic evaluation of summaries. In Text Summarizu- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Roger C Schank and Robert P Abelson. 2013. Scripts, plans, goals, and understanding: An inquiry into human knowledge structures. Psychology press.
Jean B. Schumaker, Pegi H. Denton, and Donald D. Deshler. 1984. The Paraphrasing Strategy. Univer- sity of Kansas.
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint 1711v:1711.05101.
Sue Lyle. 2000. Narrative understanding: Developing a theoretical context for understanding how children make meaning in classroom settings. Journal of Cur- riculum Studies, 32(1):45-65.
Lawrence R. Sipe. 2001. A palimpsest of stories: Young children's 's construction intertextual links among fairytale variants. Reading Research and Instruction, 40(4):333-352.
Chenyang Lyu, Lifeng Shang. Yvette Graham, Jennifer Foster, Xin Jiang, and Qun Liu. 2021. Improving unsupervised question answering via summarization- informed question generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4 4134-4148, Online and
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang. Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. 2023. UL2: Unifying language learning paradigms. In The Eleventh Inter- national Conference on Learning Representations.

---

Ying Xu, Dakuo Wang. Mo Yu, Daniel Ritchie, Bing- sheng Yao. Tongshuang Wu, Zheng Zhang. Toby Li, Nora Bradford, Branda Sun, et al. 2022. Fantastic questions and where find them: Fairytaleqa-an authentic dataset for narrative comprehension. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers).
Yasukata Yano, Michael H. Long, and Steven Ross. 1994. The effects of simplified and elaborated texts on foreign language reading comprehension. Lan- guage Learning, 44(2):189-21
Chao Zhao, Faeze Brahman, Kaiqiang Song, Wenlin Yao, Dian Yu, and Snigdha Chaturvedi. 2022. Nar- raSum: large- dataset abstractive narra- tive summarization. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 182-197, Abu Dhabi, United Arab Emirates, Associ- ation for Computational I Linguistics.


Reviews:
[{"review_id": "niyX9hLZ2N", "reviewer": "Reviewer_jhYe", "summary": "", "questions": "From line 47 - 49, you mentioned that there is still no sufficient data in the narrative domain, and it is not optimal to directly use models trained on general text data for narrative reading comprehension. However, from my understanding, PARROT also  uses a parallel narrative pairs for pre-training, is the data bottleneck the lack of annotated questions and answer pair, or the text?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We have addressed the comments outlined below and will incorporate these revisions in the next version.\n\n---\n\n**Q1:** For experiment results, almost all analysis are based on ROUGE scores. However, ROUGE score is known to have problems and can be easily gamed (Krishna et al. 2021; https://arxiv.org/abs/2103.06332). I am yet not convinced that PARROT is indeed better than the baselines. More automatic evaluation metric as well as human evaluation should be conducted.\n\n**A1:** For additional automatic evaluation metrics, we’ve incorporated both the Exact Match (EM) and F1 score. We also conducted a human evaluation using Amazon Mechanical Turk. We randomly selected 100 test instances from both test sets. For each instance, we show three independent turkers the question, correct answers, and the predicted answers. Turkers were asked to assess the quality of the predicted answers on a Likert scale ranging from 1 to 5. To maintain the evaluation quality, we require turkers to be AMT Masters based in the US, with more than 1,000 HITs approved and an approval rate exceeding 98%. We manually checked the annotation results, blocked turkers who consistently provided low-quality annotations, and re-distributed their tasks. Human judges received a wage rate of $14 per hour.  The IAA score is 0.7003 in Gwet’s gamma.\n\n|       |    EM     |    F1     |  Human  |\n|-------|:---------:|:---------:|:-------:|\n|   IR  | 0.20/0.02 | 19.69/15.12 | 2.12/2.37 |\n|  AE-QG | 22.74/34.90 | 41.62/52.93 | 2.56/2.91 |\n| Vicuna | 2.18/2.96 | 31.63/30.87 | 2.36/2.61 |\n| ChatGPT | 10.72/6.85 | 46.18/39.83 | 2.49/2.86 |\n| Parrot-single | 15.79/29.43 | 37.73/48.94 | 2.30/2.78 |\n|  Parrot  | 23.83/35.89 | 46.59/54.11 | 2.71/3.10 |\n\nResults on both datasets (separated by “/”) show that both extra automated metrics and human evaluation support that Parrot is significantly better than baselines (approximate randomization p<0.05). \n\nThe ROUGE score is problematic when assessing lengthy outputs, as exemplified in the “long-form” QA paper pointed out by the review. LFQA concludes that ROUGE lacks reliability as they have observed that simply copying questions yields higher ROUGE scores than predicting answers, and predicting answers yields higher ROUGE scores than using reference answers. \n\nHowever, our study primarily focuses on the “short-form” QA problem, where ROUGE is a reliable measure. For instance, when the reference contains only a few tokens, a higher ROUGE score effectively demonstrates the system's superiority. Also, none of the misleading behaviors of ROUGE mentioned in the LFQA paper were observed in the two datasets we employed. For these reasons, ROUGE serves as the official evaluation metric in both datasets, and is the sole official measure employed in the FairytaleQA (2022) paper. \n\n---\n\n**Q2:** What does the question and answer look like? Figure 4 briefly mentioned the types but more examples should be added.\n\n**A2:** We have some examples in Figures 1 and 2 in the paper. We will add more examples in the appendix.\n\n---\n\n**Q3:** From line 47 - 49, you mentioned that there is still no sufficient data in the narrative domain, and it is not optimal to directly use models trained on general text data for narrative reading comprehension. However, from my understanding, PARROT also uses a parallel narrative pairs for pre-training, is the data bottleneck the lack of annotated questions and answer pair, or the text?\n\n**A3:** The primary bottleneck lies in the availability of annotated question-and-answer pairs. The raw narrative text should not be a limiting factor. We will clarify it in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- To adapt to the narrative comprehension setup, PARROT increases types of spans to mask that includes named entities, semantic roles, as well as verbs and adjective phrases.\n- PARROT utilize parallel reading corpuses to do narrative reading in a zero-shot manner, which is quite interesting.", "reasons_to_reject": "- For experiment results, almost all analysis are based on ROUGE scores. However, ROUGE score is known to have problems and can be easily gamed (Krishna et al. 2021; https://arxiv.org/abs/2103.06332). I am yet not convinced that PARROT is indeed better than the baselines. More automatic evaluation metric as well as human evaluation should be conducted.\n- What does the question and answer look like? Figure 4 briefly mentioned the types but more examples should be added.", "questions_to_authors": "From line 47 - 49, you mentioned that there is still no sufficient data in the narrative domain, and it is not optimal to directly use models trained on general text data for narrative reading comprehension. However, from my understanding, PARROT also  uses a parallel narrative pairs for pre-training, is the data bottleneck the lack of annotated questions and answer pair, or the text?", "paper_topic_and_main_contributions": "This paper presents PARROT, a zero-shot approach for narrative reading comprehension through parallel reading. PARROT treats one narrative as supervision signal to guide the understanding of the other. PARROT also surpasses previously zero-shot approaches on two narrative comprehension benchmarks, and is comparable to fully supervised models based on ROUGE scores.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XbDKxTMvFz", "reviewer": "Reviewer_bdc5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We have addressed the comments outlined below and will incorporate these revisions in the next version.\n\n**Q1.** The paper has a relatively limited comparison of the generated results. It can further enhance the comparison between the proposed method and baseline results using other methods such as Human Evaluation and Case Studies.\n\n**A1.** For additional automatic evaluation metrics, we’ve incorporated both the Exact Match (EM) and F1 score. We also conducted a human evaluation using Amazon Mechanical Turk. We randomly selected 100 test instances from both test sets. For each instance, we show three independent turkers the question, correct answers, and the predicted answers. Turkers were asked to assess the quality of the predicted answers on a Likert scale ranging from 1 to 5. To maintain the evaluation quality, we require turkers to be AMT Masters based in the US, with more than 1,000 HITs approved and an approval rate exceeding 98%. We manually checked the annotation results, blocked turkers who consistently provided low-quality annotations, and re-distributed their tasks. Human judges received a wage rate of $14 per hour.  The IAA score is 0.7003 in Gwet’s gamma.\n\n|       |    EM     |    F1     |  Human  |\n|-------|:---------:|:---------:|:-------:|\n|   IR  | 0.20/0.02 | 19.69/15.12 | 2.12/2.37 |\n|  AE-QG | 22.74/34.90 | 41.62/52.93 | 2.56/2.91 |\n| Vicuna | 2.18/2.96 | 31.63/30.87 | 2.36/2.61 |\n| ChatGPT | 10.72/6.85 | 46.18/39.83 | 2.49/2.86 |\n| Parrot-single | 15.79/29.43 | 37.73/48.94 | 2.30/2.78 |\n|  Parrot  | 23.83/35.89 | 46.59/54.11 | 2.71/3.10 |\n\nResults on both datasets (separated by “/”) show that both extra automatic metrics and human evaluation support that Parrot is significantly better than baselines (approximate randomization p<0.05). We will add this and case studies to the paper. \n\n---\n\n**Q2.** The authors should include a comparative model that continues pre-training using T5's original pre-training task on the author's pre-training dataset. Furthermore, a comprehensive analysis and comparison should be conducted between PARROT's parallel reading approach and the performance of this model.\n\n**A2.** T5 is already pre-trained on our dataset. The majority of our pre-training dataset is sourced from Wikipedia and online resources that are already included in C4, the pre-training data for T5. In Table 3, we provide a comparison between this T5 (indicated as “none”) and our \"Selective Span Masking\" strategy. The results demonstrate that our strategy significantly enhances model performance.\n\nWe continue to pre-train T5 using the NarraSum dataset using T5's original pre-training task. We utilize two pre-training objectives. The first (M1, single-reading) is the same as T5's original pre-training objective, where we mask spans from a narrative and then train the model to predict these masked spans. For the second objective (M2, parallel reading), we mask spans from N and train the model to predict these spans based on both the masked N and the unmasked N+. Results on two datasets (separated by “/”) are presented below. It demonstrates that continuing pre-training without the selective span masking strategy cannot improve model performance.\n\n| Model | Rouge-1   | Rouge-2   | Rouge-L   |\n|-------|-----------|-----------|-----------|\n| M1    | 7.94/11.05 | 3.24/3.55 | 7.88/11.03 |\n| M2    | 8.38/13.54 | 3.29/4.02 | 8.31/13.52 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.This paper introduces a novel pre-training task called PARROT, which utilizes parallel reading to enhance the model's capability for zero-shot narrative reading comprehension.\n\n2.This paper's empirical experiments validate the significance of \"Selective Span Masking\".", "reasons_to_reject": "1.The paper has a relatively limited comparison of the generated results. It can further enhance the comparison between the proposed method and baseline results using other methods such as Human Evaluation and Case Studies.\n\n2.The authors should include a comparative model that continues pre-training using T5's original pre-training task on the author's pre-training dataset. Furthermore, a comprehensive analysis and comparison should be conducted between PARROT's parallel reading approach and the performance of this model.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The main contribution of the article is the proposal of PARROT, a pre-training method for narrative reading comprehension that leverages parallel narratives. This approach utilizes \"Selective Span Masking\" to cultivate the model's reading comprehension abilities in the narrative domain. This method aims to address the lack of data in the narrative domain, improve data efficiency, and enhance the model's capability for zero-shot narrative reading comprehension.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YK1YEVeAYk", "reviewer": "Reviewer_z7i5", "summary": "", "questions": "1. Does the baseline also utilize the question transformation during the inference?\n2. For PARROT-single pre-training, does it still utilize the all the narratives (i.e., 57.4*2K data)?\n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We have addressed the comments outlined below and will incorporate these revisions in the next version.\n\n---\n\n**Q1:** Such strategy requires extra parallel data, which might not exist in many datasets/tasks especially during the pre-training stage. The authors did not consider such cases to propose some cheap ways to acquire such parallel data. \n\n**A1:** Prior work has presented inexpensive ways to collect the kind of parallel data PARROT needs. For instance, in our research, we use the NarraSum dataset, which is a collection of extensive parallel data acquired in an automatic and inexpensive manner. Other studies have shown that collecting different types of parallel data on a larger scale is feasible, such as dialogue-narration pairs [1], query-document pairs [2], sentence simplification pairs [3], bilingual pairs [4], image-caption pairs [5], and more. \n\n[1] Storytelling with Dialogue: A Critical Role Dungeons and Dragons Dataset, ACL 2020\n\n[2] C-MORE: Pretraining to Answer Open-Domain Questions by Consulting Millions of References, ACL 2022\n\n[3] MUSS: Multilingual Unsupervised Sentence Simplification by Mining Paraphrases, LERC 2022\n\n[4] Unsupervised Bitext Mining and Translation via Self-Trained Contextual Embeddings, TACL 2020\n\n[5] Learning Transferable Visual Models From Natural Language Supervision, ICML 2021\n\n---\n\n**Q2:** Also, utilizing the parallel data for training increase the size of context window, which require larger context window for models and might be expensive.\n\n**A2:** Increasing context is inevitable in reading comprehension. The model must have some extra content to encode the question besides the passage. In Parrot, the only difference is that the model encodes the masked parallel narrative instead of the question. Also, parallel training will not increase the size of the context window too much. As mentioned in lines 339 - 346, we only augment the context by three sentences ( a few dozen tokens), which should not pose a significant problem.\n\n---\n \n**Q3:** The question answering requires the template mapping to transform the question into a masked statement, which might cause the poor generalization to questions that are not 'Wh-types'/transformable.\n\n**A3:** In this paper, we focus on “Wh” or “How” type questions. For other types of questions such as yes/no questions or multiple choice questions, the template mapping is straightforward. We can simply provide candidate options to mask and employ the Parrot model to calculate the probability of each candidate. In contrast, the 'Wh-question' is more challenging because the model must generate answers rather than selecting from a set of candidates.\n\nFor questions that are not well-formed, the transformation process may introduce errors. Future work is needed to develop automatic transformation strategies to address these challenges, which is beyond the scope of this paper. We are happy to discuss them within the paper.\n\n---\n\n**Q4:** Does the baseline also utilize the question transformation during the inference?\n\n**A4:** It depends on whether the baseline receives a question or a masked statement as input during pre-training. For the IR baseline, there isn't a substantial difference between these two inputs. AE-QG receives original questions as input due to the introduction of question generation during its pre-training. Vicunna and ChatGPT also utilize original questions as they are designed as conversational models. Parrot-single employs masked statements as input because its pre-training objective aligns with Parrot.\n\n---\n\n**Q5:** For PARROT-single pre-training, does it still utilize the all the narratives (i.e., 57.4*2K data)?\n\n**A5:** To ensure a fair comparison, both Parrot and Parrot-single utilize 57.4k data for pre-training. We also explored pre-train the Parrot-single using all narratives, but the results were not significantly different. This is because the PARROT-single model only learns to copy from the original document to fill in masks through shallow text matching, which is a straightforward task for its underlying model, T5, to learn, therefore the additional training data does not yield substantial improvements.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The parallel reading strategy which utilizes different renderings to help the understanding each other makes sense.\n2. The zero-shot performance of PARROT outperforms or close to the supervised models and better than larger models. ", "reasons_to_reject": "1. Such strategy requires extra parallel data, which might not exist in many datasets/tasks especially during the pre-training stage. The authors did not consider such cases to propose some cheap ways to acquire such parallel data. Also, utilizing the parallel data for training increase the size of context window, which require larger context window for models and might be expensive.\n2. The question answering requires the template mapping to transform the question into a masked statement, which might cause the poor generalization to questions that are not 'Wh-types'/transformable.\n", "questions_to_authors": "1. Does the baseline also utilize the question transformation during the inference?\n2. For PARROT-single pre-training, does it still utilize the all the narratives (i.e., 57.4*2K data)?\n", "paper_topic_and_main_contributions": "This paper propose a pre-training framework, PARROT, for zero-shot narrative comprehension. Specifically, PARROT utilizes two versions of the same narrative and introduce a parallel reading strategy to predict masked spans (which are selected based on named entities, semantic roles, and verbs and adjective phrases) in one version based on another version during the pre-training to enhance the narrative understanding. Through experiments, their zero-shot performance match the supervised models.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "This looks related to multi-view/consistency regularization work such as:\nSemi-Supervised Sequence Modeling with Cross-View Training\nMulti-View Sequence-to-Sequence Models with Conversational Structure for Abstractive Dialogue Summarization\nUnsupervised Data Augmentation for Consistency Training"}}]

Author rebuttal:

Meta-review: The paper proposes Parrot, a training method to improve narrative reading comprehension, where the reading comprehension task is focused on a story. 

To train, Parrot finds two narratives of the same story, generates questions about the first, then trains the model to answer the questions using the second narrative as context. 

The model is pretrained on parallel narratives from NarraSum, then evaluated "zero-shot" on d FairytaleQA and NarrativeQA. Results show improved results compared to Vicuna-13B and ChatGPT. Results are still worse than fine-tuning on the training set of each dataset (which is not surprising given the zero-shot evaluation), but get closer. 

During the discussion period, the authors augmented their ROUGE-based evaluation with additional human evaluation, which was consistent with the automatic metric.

One thought for future work: the automatic question generation had a lot of moving pieces with engineered rules, makes, templates, question types, named-entity generators... etc. While this seems to work, I would expect all of this machinery to be replaceable with prompting for an LLM, making the method simpler and more robust.
Decision: Accept-Findings