SlowGuess/ACL_2020_OCR huggingface dataset:
Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Dont give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807.
Saul B. Needleman and Christian D. Wunsch. 1970. A general method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of Molecular Biology, 48(3):443 - 453.
Rani Nelken and Stuart M. Shieber. 2006. Towards robust context-sensitive sentence alignment for monolingual corpora. In 11th Conference of the European Chapter of the Association for Computational Linguistics, Trento, Italy. Association for Computational Linguistics.
Haojie Pan, Junpei Zhou, Zhou Zhao, Yan Liu, Deng Cai, and Min Yang. 2018. Dial2desc: End-to-end dialogue description generation. arXiv preprint arXiv:1811.00185.
Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019a. MELD: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 527-536, Florence, Italy. Association for Computational Linguistics.
Soujanya Poria, Navonil Majumder, Rada Mihalcea, and Eduard H. Hovy. 2019b. Emotion recognition in conversation: Research challenges, datasets, and recent advances. IEEE Access, 7:100943-100953.
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5370-5381, Florence, Italy. Association for Computational Linguistics.
Alan Ritter, Colin Cherry, and Bill Dolan. 2010. Unsupervised modeling of twitter conversations. In *Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics*, pages 172-180, Los Angeles, California. Association for Computational Linguistics.
Nicolas Schrading, Cecilia Ovesdotter Alm, Ray Ptucha, and Christopher Homan. 2015. An analysis of domestic abuse discourse on Reddit. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2577-2583, Lisbon, Portugal. Association for Computational Linguistics.
Iulian Vlad Serban, Ryan Lowe, Peter Henderson, Laurent Charlin, and Joelle Pineau. 2015. A survey of available corpora for building data-driven dialogue systems.
Eva Sharma, Chen Li, and Lu Wang. 2019. Bigpatent: A large-scale dataset for abstractive and coherent summarization. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Amanda Stent, Matthew Marge, and Mohit Singhai. 2005. Evaluating evaluation methods for generation in the presence of variation. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 341-351. Springer.
Preethi Vaidyanathan, Emily T. Prud'hommeaux, Jeff B. Pelz, and Cecilia O. Alm. 2018. SNAG: Spoken narratives and gaze dataset. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 132-137, Melbourne, Australia. Association for Computational Linguistics.
Oriol Vinyals and Quoc Le. 2015. A neural conversational model. Proceedings of the International Conference on Machine Learning, Deep Learning Workshop.
Kangyan Zhou, Shrimai Prabhumoye, and Alan W Black. 2018. A dataset for document grounded conversations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 708-713, Brussels, Belgium. Association for Computational Linguistics.
A.1 Summary Dialogue Alignment Precision and Recall Calculation Method
We calculate precision and recall for summary dialogue alignment with respect to the dialogue's turns in Section 4.1. Here, we describe our method for calculating precision and recall.
Precision is expressed as a function of true positives and false positives and recall is expressed as a function of true positives and false negatives. For each alignment  $a_i \in A$ , we classify each of its turns  $t$  as a True Positive (TP), False Positive (FP), or False Negative (FN). We take the counts of all TP, FP, and FN over the entire  $A$  and perform the precision and recall calculations, precision =  $\frac{\text{total}(TP)}{\text{total}(TP) + \text{total}(FP)}$ , recall =  $\frac{\text{total}(TP)}{\text{total}(TP) + \text{total}(FN)}$ .
A.1.1 TP, FP, FN Classifications
We have the following guidelines to classify a turn in  $a_{i}$  as a TP, FP, or FN.
1. First, find the earliest and latest turns in the original dialogue that correspond to the summary chunk  $s_i$ . All alignments  $a \in A$  are a contiguous sequence of turns extracted from
the dialogue. For example, in the summary chunk in Figure 1, the earliest turn in the entire dialogue that corresponds to the summary is (1) in the alignment. The latest turn in the entire dialogue that corresponds to the summary is (7) in the alignment (we verify this by looking at the turns in the original dialogue before and after the sequence presented in the alignment).
2. Any turn in the alignment in between the earliest and latest turns identified in Step 1 (inclusive) is considered a true positive. Any turn in the alignment outside of the earliest and latest turns identified in Step 1 is considered a false positive. In Figure 1, turn (0) would be considered a false positive because it does not correspond to any of the summary sentences (0,1,2,3). Turns (1,2,3,4,5,6,7) are considered true positives since they are between the earliest and latest turns that correspond to the summary sentences in original dialogue.
3. Any turn between the earliest and latest turns identified in Step 1 that is NOT present in the alignment is considered a false negative. In Figure 1, if the turn (7) was not in the alignment, it would be considered a false negative because the turn (7) corresponds to the summary sentence (2) and is between the earliest and latest turns identified in Step 1 (turns 1 and 7 respectively).
A.2 More Examples of Summary-Dialogue Alignments
We give more examples of summary-dialogue alignments  $(s_i, a_i)$  pairs. For the sake of brevity, we chose to show examples that were only 10 turns or smaller. Please refer to the dataset itself for much longer samples.
In Figure 7, we have an alignment with a large recall error. In Figure 8, we have an example of a summary referring to out-of-game turns. We find these types of summaries are typically written for break-times in the show, before the start of a game session, or after the end of a game session. Generally, they seem to make up a smaller portion of the overall summary content. This example in particular is for a Q/A session the team held after their session<sup>10</sup>. In Figure 9, we have a perfect alignment, with the summary explicitly capturing implied information in the turns. There are also examples
10 Attack_on_the_Duergar_Warcamp episode
Recall Error Dialogue Chunk
0	MATT: “End of your turn, it’s going to use two actions to do a wing attack, beating its wings, hitting every creature within 15 feet. You’re out of range, actually, Marisha. Grog, I need you to make a dexterity saving throw.”
1	TRAVIS: “I think I have advantage on this because of rage. I do. 21.”
2	MATT: “21? That unfortunately fails. You take 15 points of bludgeoning damage, and you’re knocked prone. Also, Pike and Vax, you both fail a death saving throw from the bludgeoning winds of the ice dragon’s wings beating downward.”
Aligned Summary Chunk
0	“Scanlan takes a Greater Healing Potion and moves towards Vorugal. He hits him with a Fireball.”
1	“Vorugal uses a wing attack against Grog, hitting both Vax and Pike as well, losing a death save each.”
Figure 7: A (not tokenized) turn sequence and the associated human written summary chunk after the text alignment process. It is clear from the second sentence of the summary chunk, that the turn aligned turns are a subset of the true turn sequence the summary chunk is referring to. In order to capture the turns referred to by the first sentence in the summary, we need to include the additional 29 preceding turns in the dialogue (which are treated as 29 False Negatives).
Out of Game Dialogue Chunk
0	ORION: “Ooh, like Thai food.”
1	LIAM: “I like Indian.”
2	MATT: “Ooh, Indian is good.”
3	ASHLEY: “I really noticed–”
4	ZAC: “Let them know not to order food.”
5	LIAM: “Don’t, that’s a terrible idea.”
6	ORION: “We just had a bunch of chicken.”
7	MARIHSA: “Oh you mean like right now? Yeah, don’t do it right now.”
8	ZAC: “If you tell them what you want, all of a sudden I’ll get a call, like, ”your food is on the way!””
Aligned Summary Chunk
0	“Liam, Matt, Marisha, and Taliesin like Indian food.”
1	“Zac chimes in telling the chat not to order any more food right now.”
of role-playing by Matt in this turn sequence, as he speaks to the other players from the perspective of the in-game character Ripley. This is shown through the use of quotes in turns 0, 4, and 6.
A.3 Train, Validation, Test Split Method
In Section 4.1, we split the aligned 34243 pairs into 26232 training, 3470 validation, and 4541 testing pairs. Here, we briefly describe our method.
We first split the 159 dialogues into an  $(80\%, 10\%, 10\%)$  train, validation, and test split based on
Figure 8: An out-of-game turn sequence and summary chunk. We find a single precision error in this alignment with Orion mentioning Thai food, which is not in this summary chunk.
In Game Dialogue Chunk with Roleplay
0	MATT: “‘I don’t spend my time wondering or curious about her well-being! I just know that she is usually here.’”
1	TALIESIN: “Anna. I’m going to take a leap of faith and believe, contrary to all evidence, that you are a smart woman. I pull out the gun, and I put it to her head. Now. If you were the Briarwoods, where would you put my sister?”
2	LAURA: “‘An important question here, Percy. Are they keeping her, or is she here of her own volition?’”
3	TALIESIN: “‘I don’t know. And if you don’t know, make me believe it.”
4	MATT: “‘I know she’s not allowed anywhere near the ziggurat or near our distillery.’”
5	TALIESIN: “Distillery? I pull the gun away.”
6	MATT: “She breathes a sigh of relief. ‘That’s been largely my project as part of this entire endeavor. All right, so when I was brought in here, I was tasked to experiment with the design and create large amounts of a very, very delicately prepared acidic compound, one that could dissolve the stone of your whitestone and distill it down into pure residuum. This would allow the bulk creation of a very powerful magical essence for use in construction materials that we could instill and use apparently for this ziggurat, as well as other such things. Thus, that was my main reason for being here. We were ahead of schedule, and I completed the bulk of our development weeks ago, and I no longer had much of a purpose here.’”
Aligned Summary Chunk
0	“When asked where she could be, Ripley claims that she prefers not to pay attention to the well-being of others, only that she is usually in her room. Percy then starts to lose his patience.”
1	“Giving in to Percy’s threat, Ripley mentions that Cassandra is not allowed anywhere near the Ziggurat or the “distillery”. ”
2	“He lowers the weapon to allow her to explain.”
Figure 9: An turn sequence and summary chunk with perfect alignment. We observe there is implied information in the turns that is captured more explicitly in the summaries. For example "Giving into Percy's threat, Ripley..." summarizes what happens after turn 1 where Ripley is threatened with the gun and "gives in" by answering Laura's question.
order. This guarantees that episodes from validation will succeed episodes in training, and episodes in testing will succeed episodes in validation. We take all the  $s, a$  pairs from these dialogues and put them into their respective train, validation, test sets. We chose to split by this method so that (1) there will never be an episode that is in more than one train/val/test set; (2) no summary of chunk size  $C_i$  from validation or testing is a subset of summary of chunk size  $C_j$  from the training set where  $i \leq j$ , thus avoiding bias in the final metrics; and (3) we can train on information that happened in the show prior to information we validate or test on, thus better mimicking a real-world scenario where you cannot train on future information.
5133
As new Critical Role episodes and seasons are added, we hope to expand the CRD3 dataset correspondingly. Future work might include splitting the training, validation, and testing sets based on season or some method that guarantees independence between narrative elements from the summaries and turns in the training, validation, and testing sets. Note, as new Critical Role episodes are added, we will keep the original version preserved so as to keep the experiments and analysis reproducible.