SlowGuess/ACL_2020_OCR huggingface dataset:
Revanth Rameshkumar
reramesh@microsoft.com
Peter Bailey
pbailey@microsoft.com
This paper describes the Critical Role Dungeons and Dragons Dataset (CRD3) and related analyses. Critical Role is an unscripted, live-streamed show where a fixed group of people play Dungeons and Dragons, an open-ended role-playing game. The dataset is collected from 159 Critical Role episodes transcribed to text dialogues, consisting of 398,682 turns. It also includes corresponding abstractive summaries collected from the Fandom wiki. The dataset is linguistically unique in that the narratives are generated entirely through player collaboration and spoken interaction. For each dialogue, there are a large number of turns, multiple abstractive summaries with varying levels of detail, and semantic ties to the previous dialogues. In addition, we provide a data augmentation method that produces 34,243 summary-dialogue chunk pairs to support current neural ML approaches, and we provide an abstractive summarization benchmark and evaluation.
Artificial intelligence applied to human conversation remains an incredibly challenging task in computer science. Task-oriented dialogues, which are more narrowly scoped and information dense than conversational dialogue, have been the focus of recent progress in dialogue understanding (Budzianowski et al., 2018). A difficulty for hypothesis testing on non-task oriented dialogues is a lack of large datasets that are fully representative of the spontaneity and noise of real world conversation, especially in the areas of storytelling and narrative beyond long-form text or monologue. Many potential dialogue processing tasks involve multi-speaker dialogues where narrative elements are conveyed through interaction between two or more speakers. These narrative elements can include changes in the states of narrative objects,
Sample Dialogue Chunk
0 TRAVIS: "i felt like i almost died and i had n't taken care of any of the shit that got me here in the first place . i was so worried about trying to learn about these new abilities that - i felt like i got distracted . i have people i want to find and things i want to remedy .
1 MARIHSA: "yeah . how did jester do ? no offense, but she seems like she 's a little bit more willfully stronger than you are .
2 TRAVIS: "i mean, fuck, it's really disturbing. like, she came out of there like a little kettle of popcorn, just no problem. i mean - can i see jester? is she nearby?"
3 MATT: "jester, are you nearby?"
4 LAURA: "i'm across the bar just fucking dancing alone. -lrb- laughter -rrb-."
5 LIAM: "just sixteen candles-ing it."
6 MARIHSA:"yep."
7 TRAVIS: "i was worried . there were really dark times . i would hear jester singing to herself at night and then she 'd change lyrics , and then my name would be in the lyrics sometimes . every morning, she would try and cheer everybody up that was around her , but she had the muffle ? so i could n't tell if my brain was playing tricks on me , or if she was just - i do n't think there 's much that gets her down . it 's kind of inspiring."
Aligned Summary Chunk
0 "beau asks about jester."
1 "fjord says he is amazed but disturbed at how well jester seems to be doing."
2 "he says jester would try to cheer everyone up and sing, even though her mouth was gagged most of the time."
3 "he looks over to see jester dancing alone by the end of the bar."
Figure 1: A tokenized dialogue chunk and the associated human written summary chunk after the text alignment process. Jester, Beau, and Fjord are the aliases for Laura, Marisha, and Travis respectively.
descriptions of events, or changes in the states of speakers themselves. Some explored sub-tasks for narrative understanding are topic understanding, character state tracking, and abstractive summarization. Though progress has been made in these areas, it has been on datasets where conversation has been constrained to specific topics, constrained by
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5121-5134
medium of communication, or scripted (in the case of television or movies) (Forchini, 2009). With datasets that involve naturally occurring dialogue, the small amount of data per narrative or speaker makes modeling challenging.
1.1 Critical Role Episodes and Wiki
The Critical Role show<sup>1</sup> is a weekly unscripted, live-stream of a fixed group of people playing Dungeons and Dragons, a popular role-playing game. Critical Role is set in a fictional world created by the Dungeon Master (DM) Matthew Mercer.
Separate from Matthew, there are eight other players who participate in his world as role-played characters; whose actions in the game influence the fictional world (as per the DM) along with their own character's state. There are multiple objectives to the game, both hidden and explicitly stated by both parties. For example, the DM might explicitly state a quest for the players to complete or a player's character might have an explicit personal goal that needs to be met. Examples of implicit objectives are non-player characters objectives created by the DM, and a player's character's backstory that influence their actions. This definition and expansion of the fictional world, the interaction with the world, and the development of the narrative is done entirely through unscripted spoken dialogue between the DM and the other players.
Fans have maintained dialogue transcriptions for each episode as well as an online knowledge base (the Fandom  $\mathrm{wiki}^2$ ) where details about the players, characters, world, and game sessions are continuously added to. By extracting dialogues from the Critical Role transcripts, CRD3 aims to provide the community with a narrative-centered dataset that is unscripted, noisy, and spontaneous; while being coherent, consistent in latent speaker attributes and personalities, and considerably longer in dialogue length than similar conversational dialogue datasets. From the wiki, we obtain human-authored, structured summaries for each episode that support tasks of narrative understanding and extraction, topic understanding and segmentation, and summarization from conversational dialogue.
We make five contributions in this paper. First, we produce a cleaned and structured dialogue dataset
extracted from the Critical Role transcripts (CRD3-Dialogues) $^{3}$ . Second, we provide corresponding structured abstractive summaries for each episode, mined from the Fandom wiki (CRD3-Summaries). Third, we analyze the dataset and compare it to similar datasets. Fourth, we describe our method of data augmentation via text alignment to make this data scale-appropriate for neural ML approaches, and provide these summary-dialogue chunk pairs (CRD3-SD-pairs). Finally, we construct an abstractive summarization baseline from these pairs and discuss its evaluation (CRD3-Baseline).
We believe that better abstractive summarization tools to distill information is essential given the ongoing growth of unscripted, multi-person dialogues in entertainment and business scenarios. We hope that CRD3 will support research and development for such tools.
The Critical Role Dungeons and Dragons Dataset is a combination of story-telling dialogues structured around the game-play of Dungeons and Dragons and corresponding abstractive summarizations for each dialogue. As such, it can be compared to existing dialogue datasets and summarization datasets.
2.1 Dialogue Datasets
There are currently many existing dialogue datasets (disregarding machine-to-machine) that can be roughly grouped into task-oriented, conversational, scripted, constrained, and spontaneous dialogues (Serban et al., 2015). Task-oriented datasets address specific tasks and are constrained by an ontology (Budzianowski et al., 2018). If the task is sufficiently constrained, even a human-to-human task-oriented dialogue can lack spontaneity and noise of open domain conversation (Haber et al., 2019), (Vaidyanathan et al., 2018), (Lison and Tiedemann, 2016). Agents trained on such datasets cannot be expected to model spontaneous conversational dialogue. Scripted dialogue datasets are closer to conversational dialogue. Popular scripted dialogues come from TV shows, movies, and novels; sometimes featuring further annotations (Poria et al., 2019a), (Lison and Tiedemann, 2016), (Banchs, 2012). Though the lack of noise can be helpful in training a dialogue system, they do contain artificialities in their linguistic properties (Forchini, 2009). With datasets that do have
<sup>1</sup>critrole.com
2criticalrole.fandom.com
$^{3}$ github.com/RevanthRameshkumar/CRD3
natural conversation, either with provided topics (Rashkin et al., 2019), (Godfrey et al., 1992), (Carletta et al., 2006) or truly naturally occurring (Ritter et al., 2010), (Schrading et al., 2015), (Li et al., 2017), (Leech, 1992), (Misra et al., 2015), the larger scope and noise along with the small amount of data for individual domains, latent speaker attributes, and linguistic attributes make tasks like response generation, abstractive summarization, and speaker personality modeling more difficult (Vinyals and Le, 2015), (Black et al., 2011), (Stent et al., 2005), (Poria et al., 2019b). Story-telling and game-playing dialogues can have properties from both task-oriented and conversational dialogues, as they have specific topics or tasks and are primarily human-to-human (Gratch et al., 2007), (Hung and Chittaranjan, 2009), (Afantenos et al., 2012), (Djalali et al., 2012), (Hu et al., 2016). In storytelling dialogues there is a clear topic constraint and purpose of conveying narratives. In game-play dialogues, there are clear tasks that the speakers try to complete, to either win or progress the game. This helps reduce topic noise and increase information density, but retains natural noise like disfluencies, false starts, fragments, and spontaneity.
CRD3 has extensive storytelling and narrative building through dialogue, as well as game-playing since Dungeons and Dragons is the show's focus. The episodes are unscripted and live-streamed, so the dialogue is naturally occurring and contains a large amount of context-switching and chit-chat. Since it is spoken then transcribed to text, there exists linguistic noise as usually present in naturally spoken dialogue. Finally, the large amount of turns combined with consistent cast and persistent environments make modelling based on latent speaker and linguistic attributes more feasible.
2.2 Abstractive Summarization Datasets
Most of the recent abstractive summarization research is conducted on document datasets (news, scientific papers, and patents) (Hermann et al., 2015), (Cohan et al., 2018), (Sharma et al., 2019). However, the methods used to perform well in these domains are less effective in dialogue (movies, personal-interviews, multi-person dialogues, etc) (Kedzie et al., 2018). As (Narayan et al., 2018) noted, many of the current summarization datasets highly reward extractive approaches due to the large amount of phrasal overlap in document and summary. Dialogue summarization is under
explored in datasets. For abstractive summarization, the most popular spoken dialogue datasets are AMI and Switchboard. Others exist, but are more constrained or purely textual, (Zhou et al., 2018), (Gella et al., 2018), (Misra et al., 2015), (Louis and Sutton, 2018), (Pan et al., 2018). Notably, (Gorinski and Lapata, 2015), (Gorinski and Lapata, 2018) combine movie scripts with Wikipedia plot summaries and other metadata. Though this brings us closer to longer form abstractive dialogue summarization data, there is significant information about the plot conveyed through script notes and descriptions, and not spoken dialogue.
3 Data Collection and Preprocessing
3.1 Dungeons and Dragons
Briefly, Dungeons and Dragons is a popular roleplaying game that is driven by structured storytelling. Players create characters to participate in a fictional world created by the Dungeon Master (DM). They interact with the world entirely through dialogue with the DM and use dice rolls as a way to introduce randomness to the consequences of their actions. Actions can include exploring the environment, talking to fictional characters (role played by the DM), battle, and puzzle solving.4
3.2 Critical Role Video Stream Transcripts
The CRD3 dataset consists of 159 episodes (dialogues) from two campaigns. Campaign 1 has 113 episodes and Campaign 2 has 46 episodes, with new episodes being actively added. The episodes are unscripted and live-streamed, then archived and transcribed; they are usually several hours long. Detailed episode information can be found on the Fandom wiki<sup>5</sup>. The episodes usually start with some out-of-narrative logistics, then proceed to the actual D&D game where the players communicate character action by in-character role-playing or by describing the characters' actions in third person. There is also substantial out of narrative chit-chat and context switching.
For each episode, we extract the names and turns from the dialogue transcript and clean the data as much as possible. We try to resolve the inconsistencies in spelling of speaker names, use of quotes, onomatopoeia, speaker aliases (and character aliases), parse multiple speakers for turns if needed, and others that exist due to the transcripts
dnd.wizards.com/dungeons-and-dragons
criticalrole.fandom.com/wiki/List_of Episodes
Metric	CRD3	MELD	M. WOZ	AMI	CNN	DailyMail
Dialogue Count	159	190	10438	142	92465	219506
Turn Count	398682	13708	143048	79672	3074340	6189038
Total token count in dialogues	5056647	120913	1886018	706803	60476397	154282948
Unique token count in dialogues	42509	6251	20197	9958	341451	596032
Avg. turns per dialogue	2507.4	72.2	13.7	561.1	33.4	28.2
Avg. tokens per turn	12.7	8.82	13.2	8.9	19.7	24.9
Total token count in summaries	327899	-	-	22965	3897045	11308821
Avg. tokens per summary	2062.3	-	-	161.7	42.1	51.5
Avg. summary:dialogue token ratio	0.065	-	-	0.038	0.085	0.087
Table 1: We compare CRD3 with other similar datasets. MELD, Multi-WOZ, and AMI are dialogue datasets. We use the subset of the AMI dialogues with available abstractive summaries. CNN and Daily Mail are abstractive summarization datasets for news articles (we treat an article as a dialogue and a sentence as a turn).
being written over time by fans. We also replace all instances of character aliases in the speaker field with the real speakers' names to reduce noise. Along with the cleaned data, we provide the raw transcription data to document the changes via diff.
3.3 Critical Role Episode Summaries
The summaries for each episode were mined from the Critical Role Fandom wiki. The summaries are unique in that they are structured and offer different levels of summarization. Most episodes have a (1) wiki opening blurb, which offers briefest level of summarization. This is followed by a synopsis section which is (usually) comprised of several parts: (2) pre-show and announcements, where some logistical information is mentioned; (3) recap, where the previous episode is summarized (usually done by Matt in the episode and is narrative focused); and (4) the episode's plot which is the largest part and summarizes the narrative developments of the episode. The plot sections are also usually divided into sub-sections aligned to narrative topics. Sometimes the wiki also has a break and post-episode sections (usually non-narrative), which we include in the dataset.
3.4 Analysis and Comparison
Refer to Table 1 for turn and token count comparisons. CRD3's total turn count, turns per dialogue, and unique token count are substantially larger than MELD (Poria et al., 2019a) (scripted Friends TV show dataset), Multi-WOZ (Budzianowski et al., 2018) (unscripted task-oriented dialogue dataset), and AMI (Carletta et al., 2006) (unscripted meetings dataset). For AMI, we only consider the dialogues with available abstractive summaries  $^{6}$ . Multi-WOZ is dyadic while AMI, MELD, and CRD3 have multiple speakers per dialogue.
We extract 72 total speakers from the entire CRD3 dataset; 9 of which are the main cast (players and DM) and make up  $99.48\%$  of the total turns; the DM alone makes up 111,994 turns. In comparison, the 6 main cast of MELD make up  $83.27\%$  of the total turns. In addition to real (human) speakers, there are also purely in-game characters role-played by the DM. The indication of the DM role-playing through the use of quotes seem to be mostly consistent in the transcripts. As a loose measure of role-playing, we find the turns that contain quotes from the DM ( $\approx 21383$ ) and compare to all other players ( $\approx 2497$ ). A core aspect of the game is players querying the DM, so we also measure the instances of questions from a player (turn ending in ‘?’) followed by a DM response; a mean of 199 per dialogue with 58 standard deviation. Finally, we apply the spaCy English NER model on all dialogues as a loose measure of named entity presence. We get a mean of 1275 entities per dialogue with standard deviation of 344.5.
For the summaries, we measure the token counts per summary and compare to AMI, CNN, and Daily Mail (Table 1). Again, CRD3 is substantially larger (though smaller in total tokens than the news datasets). The news datasets also feature more summary-article pairs, making them more amenable to current neural ML approaches; we address this for CRD3 in Section 4. We also measure the compression of the original text to summary via ratio of tokens per summary to tokens per original text and find they correspond to the ratios of total tokens to unique tokens. Finally, we measure the average token count and standard deviation of each section of the structured summaries for the CRD3 dataset (outlined in Section 3.3): (1) Wiki opening blurb:  $50 \pm 16.7$ ; (2) pre-show and announcements:  $183 \pm 254$ ; (3) recap:  $335 \pm 123.9$ ; and (4) episode plot:  $1544 \pm 1553.7$ .
$^{6}$ github.com/gcunhase/AMICorpusXML
4 Scaling up the Dialogue Summaries
The CRD3 dataset can be applied to many tasks, but we find abstractive dialogue summarization the most compelling task to explore in this paper. Due to the extensive length of the dialogues and summaries, and the frequent context switching and noise, we are presented with challenges that are poorly addressed by the current modeling and evaluation methods:
1. The dataset has relatively few episodes (159); as is, this is not enough samples to train, test, and validate using current neural approaches.
2. The current, most successful summarization approaches do not explicitly attempt to capture coreference, semantics, and pragmatics in very long documents or conversations.
3. Current automatic summarization evaluation methods have specific failures in evaluating narrative summarization.
We do not attempt to propose a solution for either the second or third challenges, as they are beyond the scope of this paper. Instead, we address the first challenge by proposing a novel data augmentation method to dramatically scale up the number of available summary-dialogue turn sequence pairs. That outcome enables the community to start modeling and evaluation for the dialogue summarization task and we discuss initial benchmark results over this augmented set in Section 5.
4.1 Data Augmentation via Text Alignment
We found that the summaries written by fans on the wiki are detailed, mostly ordered with respect to the corresponding episode, and mostly non-repetitive. Due to the large number of sentences in the summaries, we can break up the summaries into chunks and align each chunk to some continuous segment of the dialogue. Formally, given dialogue  $D$  consisting of  $T$  turns  $\{t_i | i \in 1 \dots T\}$  and summary  $S$  split into  $n$  contiguous chunks  $\{s_i | i \in 1 \dots n\}$ , we try to determine  $A = \{a_i | i \in 1 \dots n\}$  where  $a_i$  is a contiguous set of turns from  $D$  ( $a_i = t_{j:k}$ ) and where  $t_j$  and  $t_k$  ( $j \leq k$ ) are the earliest and latest turns in  $D$  to align to  $s_i$ ; refer to Figure 2. To determine  $A$ , we try two approaches.
Greedy Algorithm We make an independence assumption for all  $s$  and  $t$  and try to maximize an alignment score,  $\alpha (A;S,\beta)$ , where  $\beta (s,a)$  calculates an alignment score between a single  $s$  and  $a$ .
Figure 2: Chunking and mapping of  $C$  contiguous summary sentences onto the T turns of the dialogue. The greedy approach (left) has no order or contiguity constraint. The Needleman-Wunsch approach (right) has strict order and contiguity constraints.
$$
\alpha (A; S, \beta) = \sum_ {i = 0} ^ {n} \max  _ { \begin{array}{c} 0 \leq c \leq T \\ 0 \leq w \leq 1 4 \end{array} } \left(\beta (s, t _ {c - w: c + w})\right) \tag {1}
$$
where bounds for  $w$  are determined empirically. For several dialogues, we tested  $0 \leq w \leq T$ , but this had no change in the final assignments  $A$  and greatly increased computation time. To choose  $\beta$ , we tried several scoring functions including variations of ROUGE (Lin, 2004), variations of TF-IDF (Jones, 1988), and other n-gram overlap scorings. We selected a scaled version of ROUGE-F1 score:
$$
\begin{array}{l} \beta (s, a) = | \tau (s) \cap \tau (a) | * R O U G E _ {F 1} \\ = \frac {2 * | \tau (s) \cap \tau (a) | ^ {2}}{| \tau (s) | + | \tau (a) |} \tag {2} \\ \end{array}
$$
where  $\tau$  is a tokenization function for the given text. The scaling via  $|\tau(s) \cap \tau(a)|$  term gives extra importance to the absolute token overlap count.
To calculate the tokens, we found just unigrams and bigrams gave us the least noisy alignments. We also found lemmatization and stop-word removal greatly reduces the alignment quality because of the large number of n-grams  $(\geq 2)$  from the turn windows that are directly used in the summaries.
In Figure 3(a), we plot the turn indices as a function of the summary chunk indices. We notice the greedy alignment approach can largely preserve the order of the summary chunks relative to the dialogue turns, without any ordering constraints. However, there are some issues with this method. First, it allows out-of-order alignments of summary chunks, which we have assessed as almost always erroneous in this dataset. Second, the recall can
Figure 3: (a) Midpoints of turn sequences as a function of the summary chunk indices for campaign 2 ep. 31, determined by the greedy approach. The plot is generally monotonic, with the out of order points verified as misalignments. After assessing many dialogue and summary pairs, we determined a strong monotonic assumption for this dataset. (b) For the same summary sentence chunk indices as in graph (a), we plot the new turn sequence midpoints as determined by the Needleman-Wunsch approach. The plot is now perfectly monotonic due to the ordering constraint and captures previously missed turn sequences.
be low due to early cutoffs at boundaries, generally because of extensive chit-chat in between two salient utterances. Forcing boundaries between  $a_{i}$  and  $a_{i + 1}$  to be contiguous leads to lower precision due to salient utterances being incorrectly assigned near the borders of the turn windows.
Needleman-Wunsch Algorithm The recursive approach to determining  $A$  involves imposing strict order constraints using the sequence alignment algorithm Needleman-Wunsch (Needleman and Wunsch, 1970), similar to (Nelken and Shieber, 2006). The algorithm imposes order by forcing  $a_{i}$  and  $a_{i + 1}$  to be assigned to contiguous turn windows. We can also forgo the maximization over some window  $w$  as the algorithm does this by virtue of its score maximization function. We tried several functions for  $\beta$ , including the TF-IDF function proposed by (Nelken and Shieber, 2006) and found (2) still performs best. To use the algorithm, we first apply  $\beta$  independently for each turn (of size 1) and summary chunk to generate a match-score matrix  $M$  of size  $T \times n$ . We then build an alignment score matrix  $H$  of size  $(T + 1) \times (n + 1)$  using:
$$
H _ {x y} = \max  \left\{ \begin{array}{l} H _ {y - 1, x - 1} + M _ {y - 1, x - 1} \\ H _ {y - 1, x} + M _ {y - 1, x - 1} \\ H _ {y, x - 1} + M _ {y - 1, x - 1} \end{array} \right. \tag {3}
$$