SlowGuess/ACL_2020_OCR huggingface dataset:
Hyundong Cho and Jonathan May
Information Sciences Institute University of Southern California
{jcho, jonmay}@isi.edu
Effective dialogue involves grounding, the process of establishing mutual knowledge that is essential for communication between people. Modern dialogue systems are not explicitly trained to build common ground, and therefore overlook this important aspect of communication. Improvisational theater (improv) intrinsically contains a high proportion of dialogue focused on building common ground, and makes use of the yes-and principle, a strong grounding speech act, to establish coherence and an actionable objective reality. We collect a corpus of more than 26,000 yes-and turns, transcribing them from improv dialogues and extracting them from larger, but more sparsely populated movie script dialogue corpora, via a bootstrapped classifier. We fine-tune chit-chat dialogue systems with our corpus to encourage more grounded, relevant conversation and confirm these findings with human evaluations.
For humans, dialogue is fundamentally a collaborative, cooperative process by which partners coordinate via turns or acts to jointly construct a common world state (Bohm and Nichol, 2004). Without coordination, partners may establish different or conflicting world states, leading to solipsism in the best case and conflict in the worst. Clark and Schaefer (1989), describe five dimensions of grounding, by which partners cooperate to establish common ground, or a shared world state. The dimension of "initiation of next relevant contribution" is the most effective of these in expressing understanding of an ongoing dialogue, and yet is the least observed in dialogue systems.
Improvisational theater (improv) is a form of theater in which most or all of what is performed is unscripted, created spontaneously by the actors in real time. Because the performance is not scripted and there is typically little to no scenery or other es
Figure 1: Explicit (top) and implicit (bottom) examples of yes-ands in the SPOLIN corpus. The text highlighted in light blue reflects acceptance of the context established in the prompt ("yes") and the text highlighted in orange initiates a new relevant contribution to the dialogue ("and").
tablished environment, $^{1}$  there is no objective reality that can naturally ground the scene. Hence, actors must mainly rely on dialogue in order to build a coherent scene and progressively establish a common world view. This necessitates accelerated use of the "initiation of next relevant contribution," which in improv is known as the yes-and principle. The yes-and principle is a rule-of-thumb that suggests that a participant should accept the reality of what the other participant has said ("yes") and expand or refine that reality with additional information ("and"). Since actors consciously abide by this principle during improv performances, there is a high proportion of these turns embedded in improv dialogue, which helps ensure scenes are coherent and interesting.
1 except for, on occasion, external stimulus such as a suggestion from the audience
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2398-2413 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Open-domain neural dialogue systems, by contrast, specifically lack coherence and interestingness. They commonly repeat previous utterances (Li et al., 2016c) or generate non-committal, generic statements such as I don't know that are logically coherent as a response but preempt further conversation (Sordoni et al., 2015; Serban et al., 2015; Li et al., 2016a). Either of these developments leads to a conversational black hole and discourages participation in further dialogue turns. This is a critical shortcoming for open-domain dialogue agents, which, unlike task-oriented dialogue systems, are not guided by specific objectives other than entertainment (Huang et al., 2020). It would behoove such systems to adopt the strategies improvisers include by habit in their dialogues and, consequently, incorporating improv acts should be a key focus for the dialogue community.
Yet, to the best of our knowledge, this has not been previously done. There has been work in applying improv to build believable agents that interact with humans (Bruce et al., 2000; Winston and Magerko, 2017) or generate improvised stories (Martin et al., 2016), but development of improvisable systems in the neural era is largely absent, stymied, we suspect, by the lack of substantial corpora. This is unsurprising; while improv speech acts such as yes-and are crucial in all dialogues, they are only highly concentrated in improv dialogues. And improv dialogues are quite difficult to collect; research collections (Busso and Narayanan, 2008) have been far too small to be useful in the modern ML era. The art form has historically been mostly ephemeral, performed live in regional venues on shoestring budgets and rarely recorded. Transcripts are all but absent and mainstream media products are rare. However, the liberalization of high quality audio podcasts since 2014 has enabled the availability of a long tail of niche products, improv included (McHugh, 2016).
Therefore we set our objective as collecting yes-and-type dialogue pairs (yes-ands) to enable their modeling by corpus-driven dialogue systems. We mine podcasts and existing movie script corpora for dialogue that abides by the yes-and principle and extract dialogue pairs from these sources to build the Selected Pairs Of Learnable ImprovisatioN (SPOLIN) corpus. SPOLIN is a collection of more than 26,000 English dialogue turn pairs, each consisting of a prompt and subsequent response, which abide by the yes-and principle, though in diverse manners. Examples of yes-and type dialogue pairs collected for SPOLIN are in Figure 1. The corpus is substantial enough to be usable for fine-tuning existing dialogue models to encourage more yes-and behavior, and beyond that may prove a valuable knowledge base for empirical sociolinguistic studies on this dialogue act.
- We carefully curate Selected Pairs Of Learnable ImprovisatioN (SPOLIN), the first large-scale corpus of yes-and dialogue acts, sourced from improv and movie dialogues.
- We iteratively build a high-precision yes-and classifier, which we use to mine additional yes-ands from dialogue corpora with high volume but low yes-and density.
- We fine-tune existing open-domain conversational models with our corpus and confirm via human evaluations that this approach improves creative grounding.
- We release our models and data for public use, including a 64,000 turn pair extension of the core SPOLIN, at https://justin-cho.com/spolin.
2 Data Collection
Our data collection has five stages:
1. Manually extract yes-ands from a rich corpus of improv to obtain an initial set of yes-ands.
2. Construct a yes-and classifier from the corpus of collected yes-and data and negative examples.
3. Use the classifier from step 2 to automatically extract yes-and candidates from a much larger but sparser dialogue corpus.
2The art form has long roots, extending to the Italian Commedia dell'arte tradition from the 16th century and farces from the Roman era, but we constrain our focus to the post-20th century form developed and championed by e.g. Keith Johnstone (Johnstone, 2017), Del Close (Halpern et al., 1994), and our corpus' namesake, Viola Spolin (Spolin et al., 1986). Spolin was the originator of Theater Games, acting exercises that encourage the development of specific theatrical skills. As our corpus is similarly designed to elicit specific skills, we backronym it in recognition of her influence.
One exception, the long-running TV show Whose Line Is It Anyway, has, despite a large number of episodes, surprisingly little continuous improvised dialogue, due to the rapid-fire nature of the program.
Figure 2: An illustration of the yes-and collection workflow. The core SPOLIN corpus comprises Spontaneanation yes-ands and Cornell yes-ands (in blue boxes). However, SPOLIN can be augmented by including other general-purpose dialogue corpora in place of Cornell in this workflow, as described in Section 5.
Figure 3: Amazon Mechanical Turk interface for transcribing yes-ands from Spontaneanation episodes. Approximate transcriptions with speaker turns and time stamps generated from Amazon Transcribe are provided for additional guidance.
4. If necessary, manually validate candidates before adding them to the yes-and corpus.
5. Repeat from step 2 as needed.
An overview of this process is shown in Figure 2.
2.1 Core yes-and Collection from Spontaneanation
We select the Spontaneanation<sup>4</sup> podcast as a source of concentrated yes-ands for its relatively noise-free recording quality and high-quality volume of broad domain improv dialogue. Each episode of this podcast includes an approximately 30 minute-long improv session performed by professional improvisers. Over its 201 episodes, we identified a total of 43K lines of useful spoken dialogue.
Given the confluence of a lack of objective reality, and uninterrupted multiturn dialogue, the improvisers mostly abide by the yes-and principle, and therefore Spontaneanation is a rich resource for natural, high-quality yes-ands. As it exists only in audio form, and automatic transcription services are too noisy for high quality annotation use, we
ask Amazon Mechanical Turk workers (Turkers) to listen to the improv sessions, view Amazon Transcribe preliminary transcriptions, and re-transcribe all of the yes-ands that they hear using our transcription interface, shown in Figure 3. The interface is based on oTranscribe, an open-source transcription service. Although the quality of transcriptions is poor, we find that including them assists the Turkers in identifying speaker turns and also understanding parts that are sometimes incomprehensible without helping context.
2.1.1 Recruiting Quality Crowdworkers for Difficult Annotation Tasks
One of the main challenges for the data collection process is to recruit competent Turkers who are able to develop a good understanding of the yes-and principle. We actively recruit potential annotators to our task by inviting denizens of the sub-Reddit TurkerNation, rather than simply inviting workers through Amazon's native task posting interface based on HIT approval rate and total number of HITs approved. Our approach enables more human-level engagement, making it easier to determine Turkers' English fluency and experience with improv. To ensure their competence,
4https://www.earwolf.com/show/ spontaneanation-with-paul-f-tompkins/
Iteration	1	2	3	4
Spontaneanation +	10,459	10,459	10,459	10,459
Spontaneanation -	-	-	3,225	5,587
Cornell +	-	3,327	8,464	12,220
Cornell -	10,459	13,786	15,698	17,092
Total Training Samples	20,198	27,572	37,846	45,358
Dev Set Acc. (Spont)	80.9%	73.6%	71.6%	73.0%
Dev Set Acc. (Cornell)	52.2%	56.8%	62.1%	64.5%
Confidence Threshold	95%	70%	50%	50%
New Extraction Volume	12,360	12,802	5,150	3,515
New Proportion of yes-ands	26.9%	44.0%	72.9%	78.4%
Table 1: Iterative data collection results over Cornell. + indicates yes-ands and - indicates non-yes-ands. These counts exclude 500 turns collected from each of Spontaneanation and Cornell to form the validation set. The New Extraction Volume row indicates the new number of yes-and candidates identified with the given confidence threshold, and the New Proportion of yes-and row show as a percentage how many of these candidates were indeed evaluated as yes-ands by Turkers. The proportion of yes-ands increases after each iteration despite the lower confidence threshold used to filter the new predictions with the updated classifier.
Turkers first read yes-and guidelines (in the appendix) then demonstrate their level of understanding through qualification Human Intelligence Tasks (HITs), which test whether the candidates can identify if a yes-and exists in a 30 second audio segment and transcribe it if there is one. s
Even after inviting Turkers for the actual HIT of transcribing yes-ands, we frequently monitor the quality of the data they collect and provide feedback for incorrectly identified yes-ands. Apart from base pay for each episode they work on, we provide incentives for extracting more yes-ands. The pay for our HITs averages well above California minimum wage. From all of the episodes, we extract 10,959 yes-ands, indicating about  $25\%$  of the total number of dialogue turns in Spontaneanation are yes-ands.
2.2 Guided Extraction from the Cornell Movie-Dialogs Corpus
Although larger than any improv corpus, let alone yes-and corpus known to date, we seek to increase our corpus volume from 10,959 turn pairs. The Cornell Movie-Dialogs Corpus (Danescu-Niculescu-Mizil and Lee, 2011, Cornell) contains 304,713 turns, nearly an order of magnitude more than Spontaneanation, and it is one of the closest in domain to improv among existing dialogue datasets. However, a sample annotation of 300 randomly selected turn pairs by Turkers reveal only  $11.1\%$  of pairs are yes-ands. We thus use the already-collected
yes-ands to probe Cornell for likely candidates, to speed the search process. Recent developments of language models pre-trained on massive text data enable the training of high-accuracy models for down-stream tasks even with a small number of samples, by leveraging the contextualized embeddings that these models learn (Devlin et al., 2019; Radford et al., 2019). We thus fine-tune an initial BERT-based sequence classifier based on the implementation of Wolf et al. (2019a) with the yes-ands from the Spontaneanation episodes to determine if a given dialogue pair is a yes-and, using a high threshold (initially, a  $95\%$  probability of being yes-and) to bias for precision. We ask Turkers to validate the turn pairs identified by the classifier and add the validated pairs to our yes-and corpus. This procedure can be iterated.
For the first iteration, we train the classifier with a balanced number of non-yes-ands chosen by random sampling from Cornell, a reasonable assumption due to the relatively low concentration of yes-ands observed. The same Turkers that extracted yes-ands from Spontaneanation are invited to validate the yes-and candidates filtered out by the classifier using the interface shown in Figure 4. In order to ensure consistent annotation standards among Turkers, they are given a small number of overlapping HITs against which we validated. For 90 samples of unfiltered yes-and candidates from Cornell, the two workers yield a reasonably high Cohen's  $\kappa$  value of 0.74. Turkers are paid at rates consistent with their rates on the extraction-from-Spontaneanation task.
After the set of Cornell yes-and candidates are validated, the yes-ands and non-yes-ands are added to the training set to train a new classifier, and the same process is repeated. We hold out 500 dialogue pairs from each subcategory (i.e. Spontaneanation yes-ands) as the development set for monitoring the classifier's performance after each iteration. We incrementally lower the classification threshold for choosing a yes-and candidate as the classifier improved. We set this threshold on each iteration except for the first by retrospective evaluation of the classifier on the actual yes-and candidates' labels from previous iterations. The threshold with the highest F1 score is chosen to filter new yes-and candidates to be validated.
We balance each progressively larger corpus with negative sample turn pairs, which are either randomly selected from Cornell (round 1), selected
No. When you put me on the stand, I'll say your client is catatonic and exhibits classic symptoms of a schizophrenic, sociopathic personality. And he doesn't sleep.
Figure 4: Amazon Mechanical Turk interface for validating yes-and candidates determined by the yes-and classifier. Turkers are asked to correct minor errors in grammar, spelling, and punctuation for qualifying yes-and candidates, which are then categorized as 'Typo/Fix.'
Cargo/Fix
from the rejected-but-extracted turn pairs from Cornell (round 2 and later), or sampled from nonyes-and turn pairs in Spontaneanation formed by random coupling of prompts and responses of the Spontaneanation yes-ands (round 3 and later). The latter forces the classifier to make decisions based on semantic features relevant to a yes-and instead of only stylometric features in Spontaneanation yes-ands. We stop this iterative process after four rounds, when fewer than 5,000 new yes-and candidates are identified by the classifier, yielding a total corpus size of 26,435 yes-ands and 23,938 negative samples. An overview of this iterative process is summarized in Table 1. The negative sampling procedure, while somewhat ad-hoc, ultimately provides a mix of turn pairs from both corpora that is sufficient to allow extraction of yes-ands from new corpora at high precision rates, and is sufficient for our goals.
2.3 Additional Notes on yes-and Criteria
Although the concept of a yes-and is easy to define and understand, there are borderline cases between a yes-and and a non-yes-and that make the validation phase more difficult than originally expected. One of the cases that confused Turkers in the earlier stages of data collection is the case of yes-but's. A yes-but is a yes-and with a response that is coherent with the provided reality, but does not appear to provide an affirmative acceptance of a suggestion given in the prompt. These are different from contradictions that do not align with the previously established reality. In addition, there are instances where the response is a yes-and, but is accepted by a speaker other than the one to whom the prompt is directed. Some yes-and responses initiate a re
pair of a problem encountered while accepting the prompt, due to a confusion or a possible inconsistency, by asking for clarification (Clark and Schaefer, 1989). While these responses may not strictly establish more detail, they provide information for ultimately establishing new information. We elide these edge cases under the umbrella category yes-and in SPOLIN as they further our top-level goal of providing relevant, actionable turn responses. Examples of some of these subtle differences are shown in Table 2.
In order to provide a better understanding on the characteristics of our corpus, we annotate 200 yes-ands and 200 non-yes-ands in SPOLIN's development set to categorize them into specific yes-and or non-yes-and types.
We classify yes-ands into explicit yes-ands, implicit yes-ands, or yes-buts. Only  $15\%$  of all yes-ands are explicit yes-ands, containing phrases such as "Yeah" or "Sure" that reflects agreement. Even with such phrases, identifying explicit yes-ands is not a trivial task because it requires semantic understanding of the relevance of the context established in the prompt and that introduced in the response. In fact, there are non-yes-ands that contain phrases affirming agreement but have no contributions or have new contributions that lack relevance. The majority  $(78\%)$  of yes-ands are implicit yes-ands, meaning that the agreement is implied, often in a subtle manner. The remaining  $7\%$  are yes-buts.
Non-yes-ands are divided into contradictions and others. Most of the non-yes-and were others, as only  $5\%$  of candidates extracted from Cornell are contradictions, which are dialogue pairs with
Indicate whether each of these dialogue exchanges is a "Yes, and..." or not:
Click for Instructions
Type	Example	%
yes-and	Explicit	P: Does this map look homemade to you?R: Yeah, it looks like someone without a grasp of English drew it.	15%
Implicit	P: Alright, pull up that plate so I can take a picture.R: Sorry, the coleslaw is definitely giving off a lot of glare.	78%
yes-but	P: We all must say the chant that we say to the king.R: No, it's too erotic, please don't.	7%
non-yes-and	Contra	P: Hey, hey, aren't you afraid you'll burn out a tonsil?R: Tonsil? Me? No! Me burn a tonsil? My tonsils won't burn - As life's corners I...	5%
Other	P: I feel different right now.R: You wait and see. You're going to marry a big hero!	95%
Table 2: Examples and proportions of yes-and and non-yes-and types from annotations of 200 yes-ands and non-yes-ands in SPOLIN's development set. Determining whether a given dialogue pair is a yes-and or not is a non-trivial task, as the agreement or contradiction of the previous dialogue turn's context is usually implicit.
yes-ands	non-yes-ands
Spontaneanation	10,959	6,087*
Cornell	15,476	18,351
Total	26,435	24,438
Table 3: Composition of SPOLIN, including the development set. yes-ands and non-yes-ands from Cornell are validated by Turkers. *Spontaneanation non-yes-ands are sampled from random combination of prompts and responses in Spontaneanation yes-ands to balance the dataset for training the classifier in the final iteration, as shown in the last column of Table 1.
a response that actively negates the reality in the prompt. Others encompass any dialogue pairs with a response that lacks coherence to the prompt or adds no or minimal contributions. The distribution and examples of different types of yes-ands and non-yes-ands are shown in Table 2.
The main focus of our work is on yes-ands, but we provide non-yes-ands as part of SPOLIN for those interested in training their own classifiers. The negative samples are collected using the methods described in Section 2.2. The composition details of SPOLIN are shown in Table 3.
To evaluate the effect of SPOLIN on generating yes-and responses and thus improving generated dialogue quality, we train a common architecture with a variety of fine-tuning data configurations, both with and without SPOLIN. Specifically, for each data configuration we fine-tune a doublehead GPT-2 model (117M-parameter version based on
the implementation by Wolf et al. (2019b)), which achieved state-of-the-art performance on Persona-chat for the ConvAI-2 dialogue competition (Zhang et al., 2018). We fine-tune the models using two learning objectives, which we weigh equally in calculating loss:
1. Predicting the next word.
2. Predicting the next correct candidate that best fits the dialogue given the dialogue history.
The language modeling component uses pretrained weights from OpenAI, while the candidate classification head is trained from scratch. For evaluation, we use the language modeling component of the fine-tuned model to generate single-turn responses for the yes-and prompts in the development set. We use nucleus sampling (Holtzman et al., 2020) for the decoding step to keep only the top tokens with a cumulative probability that together exceed 0.9, from which the next token is chosen with multinomial sampling.
4.1 Data Configurations
For our experiments, we use several established dialogue datasets as baselines, namely Persona-chat (Zhang et al., 2018), Cornell (Danescu-Niculescu-Mizil and Lee, 2011) (the unfiltered corpus out of which we extract yes-ands, as described in Section 2.2), and DailyDialog (Li et al., 2017b). Each of these is an English-language open-domain casual conversation corpus with  $100\mathrm{k}-300\mathrm{k}$  turns. For each of these datasets, we either simply fine-tune on that dataset, or fine-tune and then further
For each of the questions, rank the given options in order of which is the best "Yes, and" response to the given prompt.
Click for instructions
Q1
Prompt
4	Response1
what do you do for a living?
2	Response2
I feel like you're taking responsibility for everything, you know.
1	Response3
Naw, you didn't let me down. It was a long shot all the way. We gave 'em a good run at it.
3	Response4
You're right. I do feel like I have to make a choice.
Figure 5: Interface used by human evaluators to rank responses based on their quality as a yes-and, where a rank of 1 is most preferred. The correct ranking is shown for this example. The option ranked 1 is a yes-but: it does not reject a reality but rather rejects a suggestion and provides refining information that is most coherent to the prompt.