Introduction
Large repositories of textual communications (e.g. forum and microblog posts) have gained recent popularity as proxies for dialog BIBREF0, BIBREF1, BIBREF2. However, conversations in these settings differ from natural dialog: turns may be sparsely scattered over a large temporal span, contain distinct syntax and vocabulary BIBREF3, and differ greatly in formality and focus BIBREF4. In this paper, we investigate how appropriate such data is for modeling natural dialog, and introduce Interview, a new high-quality large-scale open-domain conversational dataset grounded in interview settings with annotations for specific speaker roles.
We compare the performance of state-of-the-art language models fine-tuned on Interview and other popular conversational datasets, demonstrating that Interview contains more complex dialog and better models the characteristics of natural spoken conversations. Our dataset is an order of magnitude larger than existing high-quality natural dialog datasets and contains speaker role annotations for each turn, facilitating the development of conversational agents and assistive systems for settings involving specific speaker roles, such as doctor-patient interviews or hosted talk shows.
In particular, we explore the tasks of role modeling in media dialog and role change detection on Interview and find that leveraging role information can enable more nuanced, on-topic and natural dialog generation, as well as improve role change classification performance.
In summary, we present Interview, the first large-scale open-domain media dialog dataset. We explore two tasks for which it serves as a promising benchmark dataset: speaker role modeling and speaker change detection. We build simple yet strong models to show quantitatively that role labels from Interview improve performance on such tasks. Interview's scale, spoken origins, role diversity, and complex utterances make it a better source for grounded open-domain conversations.
Related Works
Broadly speaking, dialog and conversation datasets can be classified as constrained (goal-oriented) or open-domain, written or spoken, and scripted or spontaneous BIBREF5. In the realm of written dialog, the closest proxy to natural dialog comes in the form of role-play-style BIBREF6 conversations, featuring two agents instructed to participate in a constrained conversation. This setup has seen recent usage to construct goal-oriented BIBREF7, BIBREF8 and grounded conversations BIBREF9, BIBREF10. These datasets are expensive to collect at scale and are heavily constrained/guided by the instructions given to participants. Several initiatives have recorded and manually transcribed natural conversations occurring in the course of normal life, resulting in small, high-quality natural dialog datasets BIBREF11, BIBREF12, BIBREF13, BIBREF14. We explore an alternative venue for collecting a large-scale dataset of natural dialog: conversations and interviews on public radio.
The US Defense Advanced Research Projects Agency (DARPA) has undertaken efforts to collect broadcast and informal conversation from public and private sources including messaging boards, SMS BIBREF15, and broadcast newswire content BIBREF16, BIBREF17. However, it proves difficult to adopt these datasets as widely available benchmarks on dialog modeling tasks, as they come with a substantial cost ($100-$1000 per dataset/year, covering up to a hundred hours of transcribed conversation). In this vein, we contribute an open-access large-scale corpus of cleanly annotated broadcast media dialog.
BIBREF18 explores the patterns and discourse within media dialog and contrast the associated speaker role dynamics with spontaneous natural conversation. The author manually annotates and investigates 24 hours of Israeli news television programs. We see an opportunity for the investigation of speaker dynamics and significance of speaker roles at scale with our dataset.
Dialog modeling of open-domain chit-chat predicts one turn of dialog from one or many context turn(s). Structured approaches for dialog modeling build on hierarchical RNNs BIBREF19, BIBREF20, BIBREF21, with recent work employing a simple concatenation of dialog history in a transformer-based architecture BIBREF22. We draw inspiration from recent works in dialog generation that model speakers via persistent `personas,' whose representations are learned from a set of grounding facts BIBREF23 or other non-conversational metadata BIBREF24. Our approach eschews external grounding and learns speaker embeddings via dialog modeling, similar to BIBREF25. We, however, propose to learn speaker embeddings for different roles and capture role-dependent lexical profiles in conversation.
Interview Dataset
We collect a novel dataset of 105K multi-party interview transcripts for 7 programs on National Public Radio (NPR) over 20 years (1999–2019), total of 10k hours. These transcripts contain a total of 3M turns comprising 7.5M sentences (127M words) from 184K speakers, of which 287 are hosts. To investigate role-play in media dialog, we curate a subset, Interview 2P, with two roles: a host and a guest, comprising 23K two-party conversations encompassing 455K turns, with 1.24M sentences and 21.7M words.
In these two-party conversations, each speaker takes an average of nine turns per dialog. Guests tend to speak longer on their turns, with 1.6x as many sentences spoken and 2x as many words per turn, and also use a more diverse vocabulary (1.6x size). Meanwhile, hosts ask five times as many questions as guests, with 40% of their dialog turns containing questions. When asking questions, hosts and guests use interrogative forms BIBREF26 at the same rate (65%). We note that the host and guest roles have differing discourse patterns, which support the notion of role modeling.
Interview Dataset ::: Comparison with Other Datasets
To assess how well Interview represents open-domain dialog, we look to two datasets in widespread usage: DailyDialog BIBREF4, 13K short dialogs written to simulate simple conversations from daily life; and CALLHOME BIBREF11, transcriptions from 120 half-hour casual telephone conversations. We measure the language modeling performance of a pre-trained transformer model—117M-parameter GPT2 BIBREF27—both in its original form and versions fine-tuned (FT) on the training splits for Interview, DailyDialog, and CALLHOME. We evaluated the zero-shot performance of these models on the test splits of these datasets, with perplexities shown in tab:datasetcomparison.
While models fine-tuned on the training set performed best on each dataset as expected, we observe that 1) models trained on other datasets obtain relatively poor zero-shot performance on Interview; and 2) the model trained on Interview achieved the best out-of-domain performance on DailyDialog and CALLHOME by large margins. This suggests that language models trained on Interview can learn patterns characteristic of natural open-domain dialog in both simple daily conversation and informal long spoken exchanges. We also investigate DialoGPT, a model pre-trained on 147M Reddit threads as a proxy for dialog BIBREF22. Our results show that while Reddit threads can be used to emulate conversation, they may not resemble natural speech; DialoGPT posts by far the worst zero-shot modeling performance across all test datasets ($>$500 perplexity)—inferior to zero-shot GPT2. These experiments confirm that Interview, a dataset of real, complex conversations, is useful for modeling patterns in natural spoken dialog. We show statistics for Interview compared to other dialog datasets in tab:nprstats.
Tasks and Experiments
We additionally explore two tasks that are facilitated by speaker role annotations in Interview: 1) generating appropriate responses for a specific role given a conversation history (speaker role modeling); and 2) predicting whether a new speaker will interject on the next sentence of a conversation. These tasks are crucial components to building fluent and role-specific dialog systems, for settings such as healthcare and customer service.
Tasks and Experiments ::: Task 1: Role Modeling
We generate a response conditioned on the host speaker role, to specifically model how an interview host speaks and inquires, contrary to speaker-agnostic dialog settings BIBREF28, BIBREF29. Individual guests appear sparsely and their utterances heavily rely on external world knowledge. Thus, we model host responses, which are generally aimed towards moderating the conversation via follow-up questions and acknowledgements. Role-specific generation like this can benefit the development of assistive technologies and role-dependent dialog systems.
We approach speaker role modeling conditional language modeling task: generating the next response $T_{t, \textbf {h}}$ for host $\textbf {h}$ with the highest likelihood given a trace of prior utterances $T_{1\dots t, g}$ and $T_{1\dots t-1, \textbf {h}}$. We use a transformer decoder to generate tokens $T_{1 \dots t}$ from inputs $T_{0 \dots t-1}$, but calculate loss only across the target sequence (gold host response). We mimic the input schema for DialoGPT, concatenating all historical turns with separator tokens, appending the host target response.
Tasks and Experiments ::: Task 1: Role Modeling ::: Conditioning on Speakers
To condition on a speaker role, we prepend each utterance in the dialog history with a role-specific speaker ID. Hosts each have one ID, while guests share a single ID, allowing us to model idiosyncrasies and interviewing patterns for individual hosts:
These role-specific speaker IDs are modeled by a speaker embedding layer of the same dimensions as the transformer hidden state, injected into the transformer input layer. We fine-tune GPT2 (Speaker GPT2) and DialoGPT (Speaker DialoGPT) on our dataset with speaker embeddings. We also finetune (FT) DialoGPT and GPT2 on Interview without speaker information as strong speaker-agnostic baselines for host response generation.
For training and evaluation, we provide our model with up to 512 tokens of non-truncated historical turns. We use an 80-10-10 train/dev/test split with unique conversations in each split.
We use GPT2-small (Transformer with 12 layers, 768 hidden size, 12 heads, and 117M parameters) as the base architecture for all of our models. We perform BPE tokenization with the GPT2Tokenizer. We use the RAdam optimizer BIBREF30 with a learning rate of $10^{-6} \times \text{batch size} \times \text{no. of GPUs}$ to utilize linear scaling in multi-GPU training. Our models are trained to convergence on 8 NVIDIA Tesla V100 GPUs, with a batch size of 5 per GPU. We use teacher-forcing to calculate perplexity for all train/dev/test splits. We avoid modeling salutations and sign-offs (which tend to be formulaic, speaker-independent, and specific to the radio station) by restricting the target turns to those with at least three prior turns and two following turns of conversation, resulting in a target training set of 87K host-only turns and 11K host-only turns for dev and test.
We decode the host response via top-$k$ sampling BIBREF27 with $k=5$. Results across all models on the test set are in tab:metrics.
Tasks and Experiments ::: Task 1: Role Modeling ::: Performance
Speaker-conditioned models generate utterances closer to gold length than speaker-agnostic baselines, with significantly lower perplexity and higher BLEU scores. This indicates that including speaker information promotes the generation of higher fidelity responses. Our speaker models, especially Speaker GPT2, produce the most inquisitive responses (59.4% question-asking rate).
In an interview setting, it is also important for host utterances to be related to the conversation at hand. We evaluate the content similarity between generated responses and the dialog history. We show that our speaker-conditioned models generate responses with the most noun-phrases / topical references. These also overlap the most with topics in the dialog history, indicating topical relatedness. We note that gold responses include more noun phrases with lower historical overlap, possibly due to hosts bringing up new topics.
Tasks and Experiments ::: Task 1: Role Modeling ::: Speaker Role Ranking
To measure the conditioning effect of speaker role profiles on host response generation, we generate a dialog turn with the gold host profile and a dialog history. We then compute the likelihood of generating that response conditioned on the same context but with the gold and nine randomly sampled hosts. As in BIBREF31, we rank the likelihoods for each host and report the host matching accuracy (HMA)—proportion where the gold host is highest ranked—and Mean Reciprocal Rank (MMR) BIBREF32 of the gold host. Our speaker-conditioned models achieve much higher HMA and MRR compared to strong speaker-agnostic baselines, indicating significant conditioning on host profiles.
Tasks and Experiments ::: Task 1: Role Modeling ::: Qualitative Analysis
Our models additionally exhibit several qualitative properties of high-quality and fluent conversation. We present a sample generation in tab:sampleconv (additional samples in the Appendix) that is indicative of broad trends across the test set. None of the models are able to introduce novel information (like Gold), but our speaker-conditioned models produce markedly better inquisitive responses. While GPT2 generates a natural-sounding short question with little relevance to the topic at hand, our Speaker DialoGPT model paraphrases previous turns and refers to existing entities to ask a substantial and coherent question. We further performed a human evaluation on a Likert scale to assess subjective dialog quality, with human raters preferring speaker model responses to speaker-agnostic models 62.5% of the time across 150 pairwise comparisons.
Tasks and Experiments ::: Task 2: Role Change Detection
We also investigate role change detection as a binary classification task for two-party dialogs. As a single turn of dialog may consist of multiple sentences, we aim to use a series of historical sentences and their speakers to classify whether a role change will occur in the next sentence of dialog. In contrast to previous textual speaker change detection tasks BIBREF33, we do not provide the target sentence for which we are predicting the role change. This setting is more realistic for a real-time assistive dialog system and online prediction in general.
We fine-tune BERT BIBREF34 to encode the dialog history, classifying speaker changes with a linear layer over the [CLS] representation. To understand the role of contextual speaker information in this task, we investigate representing the dialog history with and without speaker labels for each turn. This is a difficult task on our dataset, as BERT obtains a 63.2 F1 score without speaker information, struggling to predict role changes substantially better than random. While the task remains difficult, the classifier benefits from the inclusion of speaker labels, learning speaker embeddings and achieving a 66.1 F1 score. We see the potential for further research toward learning speaker representations to predict role changes and infer the structure of dialogs.
Conclusion
We contribute a large-scale media dialog dataset that can act as a benchmark for complex open-domain, role-dependent grounded dialog. We present baseline model for role-conditioned dialog generation and show that they benefit from speaker information when added. In future work, we aim to perform temporal analyses of trends and biases within Interview and take advantage of the news setting to investigate external knowledge grounding in long natural conversations. These directions could potentially lead to more coherent free-form and assistive dialog systems.
Generated Examples
See the following tables for sample dialog histories and generated host responses from each of our baseline and speaker-conditioned dialog models.

Question: Which baselines did they compare to?
Answer: Fine tuned DIaloGPT and GPT2 on Interview without speaker information.

finetune (FT) DialoGPT and GPT2 on Interview without speaker information

two models (GPT2 and DialoGPT) on two datasets (DailyDialog and CALLHOME)

Question: What dialog tasks was it experimented on?
Answer: role modeling in media dialog , role change detection 

1) generating appropriate responses for a specific role given a conversation history (speaker role modeling), 2) predicting whether a new speaker will interject on the next sentence of a conversation

role modeling in media dialog and role change detection on Interview

Question: How was annotation done?
Answer: Unanswerable

annotations for each turn

Unanswerable

Question: Which news outlets did they focus on?
Answer: natural dialog

Unanswerable

NPR

Question: Do the interviews fall under a specific news category? 
Answer: No

Question: Which baselines did they compare to?
Answer: We also finetune (FT) DialoGPT and GPT2 on Interview without speaker information

Fine-tuned DialGPT and GPT2  on Interview without speaker information.

Question: Which dialog tasks did they experiment on?
Answer:  role modeling in media dialog and role change detection on Interview

1) generating appropriate responses for a specific role given a conversation history (speaker role modeling), 2) predicting whether a new speaker will interject on the next sentence of a conversation

Question: Did they use crowdsourcing for annotations?
Answer: Unanswerable

Unanswerable

Question: Were annotations done manually?
Answer: Unanswerable

Unanswerable

Question: Which news sources do the transcripts come from?
Answer: 7 programs on National Public Radio (NPR) over 20 years

 7 programs on National Public Radio (NPR)