SlowGuess/ACL_2020_OCR huggingface dataset:
with  $M_{y - 1,x - 1} = \beta (s_{x - 1},t_{y - 1})$  .  $1\leq y\leq T$  ; and  $1\leq x\leq n$  and the first column and row of H initialized to  $-y$  and  $-x$  respectively. We perform thetraceback from  $H_{T + 1,n + 1}$  to  $H_{0,0}$  to generate
Figure 4: Visualization of the traceback along the  $H$  matrix in the Needleman-Wunsch alignment approach. Each vertical line for  $s_i$  is the corresponding  $a_i = t_{j:k}$ .
the alignment  $A$  where each  $a \in A$  can be seen as a vertical line in the traced path (Figure 4).
We exclude gap penalties when generating  $H$ , since we want to allow multiple turns to be assigned to a summary chunk and we want to allow a single turn to overlap several summary chunks. We also notice that column-wise normalization on  $M$  reduced the quality of the alignments substantially because large scores can act as an anchor for the algorithm to localize erroneous alignments. It forces the algorithm to 'catch up' or 'pull back' the turn alignments to include the high  $M_{y,x}$  in the final path. Normalization also reduces incentives to keep the path going down a column and heavily favors moving to the next column (summary chunk). We can visualize the improvements in Figure 3(b), where we also notice the algorithm captures turns past  $t_{1833}$  (upto  $t_{1878}$ ) that were previously ignored, leading to higher recall – we manually verified this.
The strong ordering constraint is also the source of some noise. For example, if a summary alignment overshoots the correct turn window by a large margin, it is likely that the subsequent summaries will also be misaligned due to the contiguity constraint. However, the localization effect due to large  $M$  scores help mitigate this. Another source of noise is the forced alignment of the first and last turns in dialogues that continue past the summary.
We also analyze the distribution of the scores along the paths (each path normalized to 1) traced on  $M$  with respect to the nine main players (Table 2). This gives us the distribution of the player contributions to the summaries. Matt's turns contribute most to the summaries since he contributes the most salient narrative points. As the Dungeon Master, he is responsible for world building and the narrative's interaction with the other players. We
5126
Player	β
MATT	0.0307±.0008
ORION	0.0086±.0014
LIAM	0.0083±.0005
TALIESIN	0.0074±.0005
SAM	0.0070±.0004
MARIHSA	0.0058±.0003
TRAVIS	0.0057±.0004
LAURA	0.0056±.0003
ASHLEY	0.0048±.0006
Table 2: Mean (± 0.95 conf. interval) summary contribution scores for each player calculated from the normalized paths traced on  $M$  as determined by the algorithm on  $H$ .
Chunk Size	w/o Filtering	w/ Filtering
2	18569	11124
3	18438	11635
4	18378	11484
can see the other players have much lower mean scores. One explanation for this is that they engage in more non-narrative chit chat than Matt, which leads to a lower mean  $\beta$ .
Data Augmentation Running the Needleman-Wunsch algorithm for a dialogue  $D$  will give us  $N$ $s, a$  pairs. We can extend this by calculating  $S$  as  $S_0 \ldots S_{C-1}$  where  $C$  is the chunk size and  $S_x$  is the shift in the starting point of the contiguous chunking windows. For each of these  $S_x$ , we can then determine an  $A_x$  pair. This method increases our  $s, a$  pairs by a factor of  $C$ . We can go further by running this for different chunk sizes. For our experiment, we chose to run this algorithm for  $C = 2, 3,$  and 4 sentences. We remove dialogues with  $|S| \leq 10$  chunks (since there are some incomplete wikis) and get 55385  $s, a$  pairs. To reduce noise, we also: (1) impose  $2 < |t_{j:k}| \leq 100$ ; and (2) strip out pairs where  $s_i$  contains "Q:" (signifies a differently formatted question answer segment in an episode). We end up with 34243 pairs (Table 3), a substantial increase from the original 159 summary, dialogue pairs. Refer to Figure 1 and to the Appendix for examples of the summaries and examples. These are then split as 26232 training, 3470 validation, and 4541 testing  $s, a$  pairs; refer to Appendix for details.
We calculate precision and recall with respect to the turns on a random sample of 100 pairs from the training split of these 34243 pairs and obtain a precision of 0.8692 and recall of 0.9042. Refer to Appendix for precision and recall calculation
Table 3: number of  $s_i, a_i$  pairs generated for each chunk size with and without filtering.
Summary
“The Mighty Nein make their way up the ladder and through the hatch into the Keystone Pub proper, where they order breakfast. A hooded female wearing a long green cloak covering her left face and side approaches and asks if they’re heading into the swamp today– she’s desperate to go there herself. Calianna apologizes for bothering them, but she couldn’t help but overhear their conversation last night.”
Factoid Question
1. Who was overhearing the Mighty Nein’s conversa-tion the previous night?
Multiple Choice Question
2. What do the Mighty Nein have at the Keystone Pub?
(A) drinks (B) dinner (C) lunch (D) breakfast
Figure 5: Example of questions constructed for a human-written summary chunk aligned to a set of turns.
method. We find precision errors are mostly from extraneous trailing or leading turns attached to the properly aligned set of turns, and almost never from complete misalignment. We find recall errors are from turn sequences that start too late or end too early, and also almost never from complete misalignment. In most cases where  $a$  contains a recall error, we notice the precision for that  $a$  is 1.0, because  $a$  ends up being a subset of the correct  $t_{j:k}$ . We posit this is due to the strong order constraints of the algorithm and our post-alignment filtering, which removes the pairs with the highest risk of complete misalignment.
As a measure of quality of the human written summaries, we also perform a question-answering task on a random sample of  $50s_{i},a_{i}$  pairs from the filtered set. First the questioner records two questions and answers per pair, with the questions and answers coming only from the summaries  $s_i$ . For each pair, there is one factoid question with an open-ended answer and one multiple choice question with four possible answers. The factoid question can be answered by yes—no responses, entity names, or short text. The multiple choice question has at most one correct answer of the four contained in the summary chunks. (Figure 5). The questions are then answered by another person, using only the aligned turns  $a_{i}$  from the pair.
The scores are recorded in Table 4. Out of the 19 incorrect answers, we found that 17 of them were due to summary alignment errors. This is where the correct information was in the dialogue, but not in the aligned set of turns. The other 2 were due to misinterpretation of the question when answering. This indicates, with perfect alignment, all questions
Question Type	Correct	Incorrect	Precision
Free Form	39	11	78%
Multiple Choice	42	8	84%
Total	81	19	81%
could have been answered correctly; meaning what is in the summaries is an accurate reflection of what is in the transcript. However, we recognize all the information in the transcripts is not necessarily in the summaries; for example, out-of-game information. We also notice that multiple choice questions have a higher accuracy due to easier questions and additional context provided by the set of answers themselves, and not due to random guessing. We also found that 12 incorrect answers were due to no answer, meaning the answerer did not feel they had enough information to attempt an answer. For the other 7, the answerer felt that at least some information pertaining to the question was available in the aligned turns.
Unlike ROUGE precision, which relies on word overlap, this evaluation can incorporate latent semantic and contextual information. It is important to note that latent information used when answering varies greatly between people, making this method subjective with respect to the answerer. In future work, it would be interesting to measure variance of accuracy and information in the answers using a large number of people.
5 Summarization Benchmark Results
5.1 Benchmarking Approach
We establish a baseline for abstractive summarization by using the neural summarization architecture introduced by (Chen and Bansal, 2018)<sup>7</sup>. The generated data has noise due to imperfections in the alignment method and due to potentially broken coreference, so we use the model in a semi-supervised fashion.
We choose this architecture as a baseline for several reasons: (1) The paradigm for narrative summarization from noisy dialogue is close to the paradigm assumed by Chen and Bansal. Namely, first extract salient sentences, then abstractively rewrite them with an included copy mechanism to deal with OOV words. (2) The ability to analyze the extractor behavior separately from the abstrac
Table 4: Correct and incorrect answers for the Q&A evaluation method, for measuring precision w.r.t. the human written summaries in the  $s_i$ ,  $a_i$  pairs.
R1	R2	RL	M
Extractive (rnn-ext + RL)
P	20.83±.34	7.34±0.28	18.38±.32
R	44.59±.66	17.42±.62	39.22±.61	16.58
F1	25.20±.34	9.23±.32	22.20±.32
Reported Metrics on CNN/DM
F1	41.47	18.72	37.76	22.35
Abstractive (rnn-ext + abs + RL + rerank)
P	27.38±.34	5.91±.20	25.18±.32
R	22.65±.27	4.75±.16	20.74±.26	8.33
F1	23.35±.23	4.91±.16	21.41±.23
Reported Metrics on CNN/DM
F1	40.88	17.80	38.54	20.38
Table 5: ROUGE (Precision, Recall, F1 ± 0.95 conf. interval) and METEOR (M) metrics on the CRD3 test set using the purely extractive and extractive+abstractive architecture proposed by Chen and Bansal. We show the metrics on the CNN/Daily Mail dataset for the same models as reported by Chen and Bansal.
tor due to the independence of training (before connection by the reinforcement learning mechanism). (3) The speed of training due to the shortened input-target pairs.
We briefly describe the model: First, the model optimizes a sentence extraction module and an abstractive rewrite module independently using maximum-likelihood objectives. Then, end-to-end training is achieved by applying policy gradient methods (due to the "non-differentiable hard extraction" performed by the extractor). The extractor uses a temporal convolutional model to obtain hierarchical sentence representations, then selects sentences using a pointer network. The abstractor is an encoder-aligner-decoder network with a copy mechanism for OOV words. Due to the large amount of non-narrative chit-chat turns between salient turns, we train the extractor on a sequence of turns rather than individual sentences.
5.2 Evaluation and Analysis
We use precision, recall, and F-1 scores of ROUGE-1, 2, and L, along with METEOR (Denkowski and Lavie, 2014) to evaluate the generated summaries (Table 5). We run these metrics on the test set, using both the combined extractive-abstractive model and the purely extractive model for analysis on what turns are considered salient.
The purely extractive model significantly outperforms the combined model in recall and in F-1, due to the much higher recall. In the validation set, we notice the recall measures are improved by the ngrams in summary chunks that have indirect speech ("fjord says", "he says", etc). In the validation
$^{7}$ github.com/ChenRocks/fast_abs_r1
Generated Abstractive Summary
he says he feels worried about trying to learn about these abilities and abilities . he asks if she could try and cheer . the group then heads to the tavern . she asks if she can see jester , and she says she ’s really disturbing .
Figure 6: Extractor+Abstractor output for the dialogue sample in Figure 1
set, the mean ratio of unique overlapping summary n-grams to total unique summary n-grams are: 1- gram  $= {0.679},2 -$  gram  $= {0.336}$  ,and 3-gram  $= {0.205}$  . This high rate of 3-gram overlap motivates changes to the modeling architecture that are more lenient towards phrasal copy instead of just enabling word copy and depending on the learned language model and the word level copy probability.
The grammatical person shift and significant paraphrasing of turns lower the precision of the purely extractive model, leading to a higher precision in the combined model. For example in Figure 1, "beau asks about jester." from the human-authored summary is entirely from turn 1, but the only overlapping word is "jester". From Figure 6, we can see the encoder-decoder model learns the grammatical shift behavior but doesn't include the proper nouns, so the resulting summary misses important speaker information that is included in the human generated summaries. For example, Beau is the character alias for Marisha, which is latent information that was not available to the model at the time of decoding/generation. We also note the encoder-decoder module's learned language model is biased by the narrative elements present in the training dialogue chunks. This causes decoding of similar, but fundamentally different, narrative focused turns to be noisy and nonfactual.
Compared to news summarization metrics with the same model architectures, the dialogue summarization metrics are substantially lower. The disparity in model performance can be attributed to content selection differences between news – where effective summary information is available early in an article (position bias) – and dialogue – where the positional effects are not observed. Other factors include the grammatical and stylistic differences explored earlier. Our findings also confirm the findings of (Kedzie et al., 2018), which compares content selection methods for summarization across various domains (CNN/DM, NYT, DUC, Reddit, AMI, and PubMed). They find a similar disparity in R-2 (recall) and METEOR scores between the
news domain and the AMI meeting dialogue domain. They also include an oracle measurement as a performance ceiling; it achieves a max METEOR score of 17.8 and R-2 recall of 8.7 on the AMI corpus. Though ROUGE and METEOR are more useful for relative measurements than absolute, we find the current evaluation methods in summarization lead to skewed and less informative scores in dialogue domains. The problem is compounded in narrative summarization due to narrative specific lexical information, including speaker aliases. For example, METEOR specifically considers synonyms, paraphrases, and function words; all of which can change a lot from narrative to narrative.
Dialogue understanding and abstractive summarization remain both important and challenging problems for computational linguistics. In this paper, we contribute the Critical Role Dungeons and Dragons Dataset (CRD3), a linguistically rich dataset with dialogue extracted from the unscripted, live-streamed show Critical Role and long, abstractive summaries extracted from the Critical Role Fandom wiki. We provide a data augmentation method to help the community start modeling and evaluation for the dialogue summarization task and discuss the initial modeling benchmark results. We find current paradigms in summarization modeling to have specific failures in capturing semantics and pragmatics, content selection, rewriting, and evaluation in the domain of long, story-telling dialogue. We hope CRD3 offers useful, unique data for the community to further explore dialogue modeling and summarization. We also hope that the dataset can be added to in the future with multi-modal extractions, more granular annotations, and deeper mining of the wiki.
First and foremost, we thank the Critical Role team<sup>8</sup> for creating a fun, entertaining, organized, and growing set of livestreams that we used in this dataset. Next, we thank the CRTranscript team<sup>9</sup> for providing high quality transcripts of the show for the community and we thank all the contributors of the Critical Role Wiki. Finally, we thank Rahul Jha for providing feedback and Oli Bailey for contributing evaluation questions.
$^{8}$ critrole.com/team
<sup>9</sup>crtranscript.tumblr.com/about
Stergos Afantenos, Nicholas Asher, Farah Benamara, Anaïs Cadilhac, Cédric Dégremont, Pascal Denis, Markus Guhe, Simon Keizer, Alex Lascarides, Oliver Lemon, et al. 2012. Developing a corpus of strategic conversation in the settlers of catan.
Rafael E. Banchs. 2012. Movie-DiC: a movie dialogue corpus for research and development. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 203-207, Jeju Island, Korea. Association for Computational Linguistics.
Alan W Black, Susanne Burger, Alistair Conkie, Helen Hastie, Simon Keizer, Oliver Lemon, Nicolas Merigaud, Gabriel Parent, Gabriel Schubiner, Blaise Thomson, et al. 2011. Spoken dialog challenge 2010: Comparison of live and control test results. In Proceedings of the SIGDIAL 2011 Conference, pages 2-7. Association for Computational Linguistics.
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. MultiWOZ - a large-scale multi-domain wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026, Brussels, Belgium. Association for Computational Linguistics.
Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaij, Melissa Kronenthal, Guillaume Lathoud, Mike Lincoln, Agnes Lisowska, Iain McCowan, Wilfried Post, Dennis Reidsma, and Pierre Wellner. 2006. The ami meeting corpus: A pre-announcement. In Proceedings of the Second International Conference on Machine Learning for Multimodal Interaction, MLMI'05, pages 28-39, Berlin, Heidelberg. Springer-Verlag.
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675-686, Melbourne, Australia. Association for Computational Linguistics.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers).
Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation
for any target language. In Proceedings of the ninth workshop on statistical machine translation, pages 376-380.
Alex Djalali, Sven Lauer, and Christopher Potts. 2012. Corpus evidence for preference-driven interpretation. In Proceedings of the 18th Amsterdam Colloquim Conference on Logic, Language and Meaning, AC'11, pages 150-159, Berlin, Heidelberg. Springer-Verlag.
Pierfranca Forchini. 2009. Spontaneity reloaded: American face-to-face and movie conversation compared. In Proceedings of the Corpus Linguistics Conference 2009 (CL2009), page 400.
Spandana Gella, Mike Lewis, and Marcus Rohrbach. 2018. A dataset for telling the stories of social media videos. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 968-974, Brussels, Belgium. Association for Computational Linguistics.
John J. Godfrey, Edward Holliman, and Jan McDaniel. 1992. Switchboard: telephone speech corpus for research and development. [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, 1:517-520 vol.1.
Philip John Gorinski and Mirella Lapata. 2015. Movie script summarization as graph-based scene extraction. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1066-1076, Denver, Colorado. Association for Computational Linguistics.
Philip John Gorinski and Mirella Lapata. 2018. What's this movie about? a joint neural network architecture for movie content analysis. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1770-1781, New Orleans, Louisiana. Association for Computational Linguistics.
Jonathan Gratch, Ning Wang, Jillian Gerten, Edward Fast, and Robin Duffy. 2007. Creating rapport with virtual agents. In IVA.
Janosch Haber, Tim Baumgartner, Ece Takmaz, Lieke Gelderloos, Elia Bruni, and Raquel Fernandez. 2019. The PhotoBook dataset: Building common ground through visually-grounded dialogue. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1895-1910, Florence, Italy. Association for Computational Linguistics.
Karl Moritz Hermann, Tomáš Kočisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS'15, pages 1693-1701, Cambridge, MA, USA. MIT Press.
Zhichao Hu, Michelle Dick, Chung-Ning Chang, Kevin Bowden, Michael Neff, Jean Fox Tree, and Marilyn Walker. 2016. A corpus of gesture-annotated dialogues for monologue-to-dialogue generation from personal narratives. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 3447-3454, Porto-rož, Slovenia. European Language Resources Association (ELRA).
Hayley Hung and Gokul Chittaranjan. 2009. The idiap wolf corpus: exploring group behaviour in a competitive role-playing game. In ACM Multimedia.
Karen Sparck Jones. 1988. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 60:493-502.
Chris Kedzie, Kathleen McKeown, and Hal Daumé III. 2018. Content selection in deep learning models of summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1818-1828, Brussels, Belgium. Association for Computational Linguistics.
Geoffrey Leech. 1992. 100 million words of english: the british national corpus. Language Research, 28(1):1-13.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 986-995, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Pierre Lison and Jörg Tiedemann. 2016. Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 923-929.
Annie Louis and Charles Sutton. 2018. Deep dungeons and dragons: Learning character-action interactions from role-playing game transcripts. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 708-713, New Orleans, Louisiana. Association for Computational Linguistics.
Amita Misra, Pranav Anand, Jean E. Fox Tree, and Marilyn Walker. 2015. Using summarization to discover argument facets in online ideological dialog. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 430-440, Denver, Colorado. Association for Computational Linguistics.