SlowGuess/ACL_2020_OCR huggingface dataset:
Alex Boyd*†
alexjb@uci.edu
Raul Puri*
raulp@nvidia.com
Mohammad Shoeybi
mshoeybi@nvidia.com
Mostofa Patwary
mpatwary@nvidia.com
Bryan Catanzaro
bcatanzaro@nvidia.com
Non-goal oriented dialog agents (i.e. chatbots) aim to produce varying and engaging conversations with a user; however, they typically exhibit either inconsistent personality across conversations or the average personality of all users. This paper addresses these issues by controlling an agent's persona upon generation via conditioning on prior conversations of a target actor. In doing so, we are able to utilize more abstract patterns within a person's speech and better emulate them in generated responses. This work introduces the GENERATIVE CONVERSATION CONTROL model, an augmented and fine-tuned GPT-2 language model that conditions on past reference conversations to probabilistically model multi-turn conversations in the actor's persona. We introduce an accompanying data collection procedure to obtain 10.3M conversations from 6 months worth of Reddit comments. We demonstrate that scaling model sizes from 117M to 8.3B parameters yields an improvement from 23.14 to 13.14 perplexity on 1.7M held out Reddit conversations. Increasing model scale yielded similar improvements in human evaluations that measure preference of model samples to the held out target distribution in terms of realism (31% increased to 37% preference), style matching (37% to 42%), grammar and content quality (29% to 42%), and conversation coherency (32% to 40%). We find that conditionally modeling past conversations improves perplexity by 0.47 in automatic evaluations. Through human trials we identify positive trends between conditional modeling and style matching and outline steps to further improve persona control.
Modeling dialog agents, otherwise known as chatbots, has been a longstanding goal within artificial intelligence research. Historically, approaches to this task can be divided into one of the two categories: retrieval and generative. The former is posed as a search problem where an appropriate response for a conversation is selected from a large set of candidate replies, whereas the latter autoregressively samples a reply, thereby potentially creating a response that the model may not have seen before. The flexibility and creativity afforded by not prespecifying every possible response is a major draw for generative based approaches.
In recent years, advances in neural methods have shown promise in effectively modeling this task. Early progress first demonstrated potential with recurrent network based models capable of holding simple conversations (Sordoni et al., 2015). Further architecture optimizations and tweaks improved user experiences; however, they largely experienced issues with the agent exhibiting an inconsistent personality and producing uninteresting comments (Li et al., 2015). Some works have attempted to alleviate this through conditioning on various factors of the conversation through methods such as sentiment or speaker embeddings (Li et al., 2016), but the added data annotation makes these methods not scale well to the gargantuan amounts of data needed to train larger models.
A persona-based conversation task was introduced by Zhang et al. (2018) where a set of Reddit comments and their replies were accompanied by brief descriptions or factoids about the speakers, such as their hobbies and interests. Recent works Wolf et al. (2019) have shown that leveraging this format with pre-trained transformer-based language models yield state-of-the-art (SOTA) performance in generative conversation modeling. How
* First two authors have contributed equally.
$^{\dagger}$  Research conducted during an internship at NVIDIA.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 66-84
Speaker	Conversation Turn
A	They are worried about themes becoming an exploit. It happened multiple times with the 3DS
B	How was themes an exploit on the 3ds
A	You would inject the theme and holding L or R at boot would start the homebrew launcher
B	Thanks i was not aware of that and thought i would learn a new thing
A	I mean, 3DS has been out for some time
B	yeah but i only started playing it in december 2018
(i)
#	(Ref. Parent Comment) → Ref. Reply Comment for Speaker B
1	(n/a) → i once had 100 pings from the same channel and all was because the owner did not know how to make a long comment
2	(You're worse than us.) → And im one of them but i do got skins im just just as bad
3	(n/a) → Oh well im a newbie
4	(n/a) → oh wow i did not see the read it backwards thing so i did not understand only when i scrolled down
Table 1: (i) is a conversation sampled from our GCC-DEC (8.3B) model with (ii) corresponding to the reference material from speaker B (' $n/a$ ' indicates no parent comment for the given reference comment). Reference material for speaker A was not included for brevity. The first three turns in (i) are from a real conversation had within the validation set. The last three turns (italicized) were generated by sampling turns sequentially, alternating the target speaker between B and A. Yellow highlights indicate where the model appropriately transferred a part of style between references and generations for speaker B (e.g. 'i' v. 'I'), boxed words indicate a transfer of content for speaker B, and green highlights indicate consistent style for turns from speaker A (e.g. '3DS' v. '3ds').
ever, in our interactions with these models they produced conversations that adhered to the reference facts, but were devoid of unique personality and instead exhibited a mean average style.
Personality, as seen through text, manifests itself not just through content, but also through a person's tone, grammar, and vernacular. As such, a criticism of prior persona-based solutions is that the "personas" only reflect surface-level characteristics of a person's manner of speaking and can result in banal generations. What does showcase a personality are actual conversation examples from a person. By conditioning on previous, unrelated, conversation turns for a speaker, we generate new replies that utilize more abstract personality traits inherent in the reference examples. We define this as a conditional conversation task.
Emulating this abstract notion of style requires large amount of data and sufficiently powerful model. We propose a data collection procedure that heuristically scrapes user data and comment threads from Reddit to produce conversations that vary widely in content, speakers, and reference histories to condition on. This work also introduces the GENERATIVE CONVERSATION CONTROL (GCC) model, an augmented and fine-tuned GPT-2 language model. We take advantage of
large transformers' ability to model long contexts and dependencies, and successfully model multi-turn and multi-actor conversations that are significantly longer (up to 15 turns) than most prior work.
We find that scaling model sizes from 117M to 8.3B parameters yields an improvement from 23.14 to 13.14 perplexity on 1.7M held out Reddit conversations. Similar improvements from model scaling are found in human evaluations that measure sample preference to the held out target distribution in terms of realism (31% increased to 37% preference), style matching (37% to 42%), grammar and content quality (29% to 42%), and conversation coherency (32% to 40%).
To summarize, our contributions in this work are three-fold:
i We introduce a new conversational task and demonstrate added value over traditional conversation modeling through both better control and response generation.
ii We document the creation of a large, multi-turn, multi-actor conversational dataset and the techniques used to clean it and extract conversations and reference material for style.
iii We demonstrate that by increasing model size from 117M to 8.3B parameters, human evaluations measuring preference of model genera
'https://reddit.com/
ated samples over held out target distribution increase with respect to realism, style matching, grammar, and conversation coherency. Automatic evaluations also showcase similar trends with the largest model leading to significantly lower perplexities.
2 Conversation Modeling
Let  $\mathbf{c}$  represent a multi-turn conversation of variable-length, and let  $\mathbf{x}_j$  represent a single turn that contains a variable-amount of tokens. Mathematically, this is represented as  $\mathbf{c} = (\mathbf{x}_1,\dots ,\mathbf{x}_{|\mathbf{c}|})$  with  $\mathbf{x}_j = (x_{j,1},\ldots ,x_{j,|\mathbf{x}_j|})$ . Every token, in every turn, belongs to the same fixed vocabulary (i.e.  $x_{j,t}\in \mathcal{V}$ ). Assume that  $p_*(\cdot)$  represents the true distribution of content.
2.1 Language Modeling and Dialog
Standard language modeling involves modeling sequences of tokens. After factorizing, the problem is most commonly construed as a next-token prediction problem where  $p_{*}(\mathbf{x})$  is approximated via:
$$
p _ {\theta} (\mathbf {x}) = \prod_ {t = 1} ^ {| \mathbf {x} |} p _ {\theta} \left(x _ {t} \mid x _ {<   t}\right) \tag {1}
$$
where  $\theta$  is optimized over a set of documents,  $\mathcal{D} = \{\mathbf{x}^{(1)},\dots ,\mathbf{x}^{|D|}\}$ , using maximum likelihood estimation:
$$
\mathcal {L} (\theta , \mathcal {D}) = \sum_ {i = 1} ^ {| \mathcal {D} |} \log p _ {\theta} \left(\mathbf {x} ^ {(i)}\right) \tag {2}
$$
Likewise, to model dialog in the same vein requires just a small alteration. Instead of modeling just a single sequence of tokens,  $\mathbf{x}$ , the new objective is to model several sequences of tokens that comprise a conversation,  $\mathbf{c}$ . As such,  $p_{*}(\mathbf{c})$  is approximated via:
$$
\begin{array}{l} p _ {\theta} (\mathbf {c}) = \prod_ {j = 1} ^ {| \mathbf {c} |} p _ {\theta} \left(\mathbf {x} _ {j} \mid \mathbf {x} _ {<   j}\right) \tag {3} \\ = \prod_ {j = 1} ^ {| \mathbf {c} |} \prod_ {t = 1} ^ {| \mathbf {x} _ {j} |} p _ {\theta} \left(x _ {j, t} \mid x _ {j, <   t}, \mathbf {x} _ {<   j}\right) \\ \end{array}
$$
where  $\theta$  is optimized over a set of conversations,  $\mathcal{D} = \{\mathbf{c}^{(1)},\dots ,\mathbf{c}^{|D|}\}$ , using maximum likelihood estimation:
$$
\mathcal {L} (\theta , \mathcal {D}) = \sum_ {i = 1} ^ {| \mathcal {D} |} \log p _ {\theta} \left(\mathbf {c} ^ {(i)}\right) \tag {4}
$$
2.2 Conditioning on Prior Conversations
To have more control over generation and better insight into the distribution of turns within a conversation, it is better to conditionally model  $\mathbf{c}$  instead of modeling it unconditionally as in Equation 3.
For every turn in a particular conversation,  $\mathbf{x}_j \in \mathbf{c}$ , let  $\mathbf{r}_j$  be a corresponding set of reference history tuples. These tuples contain (i) a prior turn of conversation (ii) a turn of conversation spoken by the same agent as  $\mathbf{x}_j$  in response to the first member of the tuple. In the event that (ii) corresponds to the beginning of a conversation (i) is left blank. We stipulate that the turns of  $\mathbf{c}$  and the turns of  $\mathbf{r}_j$  are disjoint. This is defined mathematically as:
$$
\begin{array}{l} \mathbf {r} _ {j} = \left\{\left(\mathbf {x} _ {k - 1}, \mathbf {x} _ {k}\right) \mid \operatorname {a u t h o r} \left(\mathbf {x} _ {k}\right) = \operatorname {a u t h o r} \left(\mathbf {x} _ {j}\right) \right. \\ \left. \wedge \mathbf {x} _ {k} \notin \mathbf {c} \right\} \tag {5} \\ \end{array}
$$
The intention of including previous replies by the same person is to get a better idea of the personality, tone, vernacular, and content of potential responses when predicting next tokens for the given turn. Likewise, the turns that the agent was replying to in  $\mathbf{r}_j$  are also included to get a better idea as to the transition dynamics of how they interact with other agents.
We update our prior equations to reflect this change in modeling objective:
$$
\begin{array}{l} p _ {\theta} (\mathbf {c} | \mathbf {r}) = \prod_ {j = 1} ^ {| \mathbf {c} |} p _ {\theta} \left(\mathbf {x} _ {j} | \mathbf {x} _ {<   j}, \mathbf {r} _ {j}\right) \\ = \prod_ {j = 1} ^ {| \mathbf {c} |} \prod_ {t = 1} ^ {| \mathbf {x} _ {j} |} p _ {\theta} \left(x _ {j, t} \mid x _ {j, <   t}, \mathbf {x} _ {<   j}, \mathbf {r} _ {j}\right) \tag {6} \\ \end{array}
$$
$$
\mathcal {L} (\theta , \mathcal {D}) = \sum_ {i = 1} ^ {| \mathcal {D} |} \log p _ {\theta} (\mathbf {c} ^ {(i)} | \mathbf {r} ^ {(i)})
$$
In order to sufficiently train a model to be able to autoregressively generate turns in a conversation conditioned on prior conversations, we require an ample amount of diverse examples accompanied with plenty of reference material. A suitable source of data for this purpose can be found from comments made on Reddit posts. Thanks to a publicly available archive on pushshift.io, comments are processed from Reddit ranging from October of 2018 to March of 2019 for training, and April of 2019 for validation. The techniques described in
this section can naturally be extended to the full range of Reddit data spanning as far back as 2005; however, we choose to focus on just the 6 months in question for the sake of tractability.
3.1 Extracting Conversations
Comments for a singular post on Reddit naturally exist as a tree structure; however, conversations necessitate a sequence of turns. As such, we obtain conversations by extracting "valid" paths from the comment graph structure. Paths are extracted sequentially from the longest candidates to shortest, and a candidate path is considered valid if and only if it satisfy the following conditions:
1. The path has a minimum of 5 turns
2. The path has a maximum of 15 turns
3. At least one turn has minimum karma score $^2$  of 4 within the path
4. All turns in the path have at least 3 words
5. The path shares a maximum of 2 turns with previously extracted paths
6. No turns in the path originate from a "not safe for work" subreddit
These rules were decided upon to ensure that the model is able to learn multi-turn conversations (rules 1 and 2) with appropriate and meaningful comments being made (3, 4, and 6) while ensuring a diverse set of examples (5) are available.
Due to memory constraints, comments are only processed on a month to month basis so any conversations that span across months are lost; however, this loss is negligible due to the vast amount of data at hand. Furthermore, this technique possibly results in more relevant references than those collected from prior months as the reference data is temporally local to the conversations in question and reflects users' current personas and interests.
After all conversations have been extracted, a reference set of turns (and comments that they were replying to) are collected for every user. We save, at most if available, the top 8 scoring comments for every user. Most users have much more than 8 comments, so an average of 7.1 reference tuples per user are collected, with about half of the tuples containing a parent comment that the user was replying to.
Dataset	Convos	Turns	Users	Refs
Training	10.3M	72.6M	10.5M	73.4M
Validation	1.8M	12.5M	1.9M	13.2M
Table 2: Statistics of the training and validation datasets detailing the amount of different conversations, turns, users, and reference tuples present in each. The training and validation sets were processed from Reddit comment threads spanning from October, 2018 to March, 2019 and April, 2019 respectively.
All models proposed stem from the GPT-2 model architecture as their base design (Radford et al., 2019). The class of models will be defined as GENERATIVE CONVERSATION CONTROL models, GCC We experiment with the number of layers,  $l$ , the hidden size,  $h$ , and the number of attention heads,  $A$  in the GPT-2 model architecture. Modeling conversations with GCC requires three steps: (i) identify a speaker to emulate and obtain their reference history consisting of comments they made on other Reddit posts, (ii) input the reference history and conversation turns into the model, and (iii) retrieve estimated next-token probabilities only associated with turns in the conversation spoken by the target speaker.
4.1 Data Representation
Due to supporting multi-actor conversations present in our dataset, special care is needed for presenting this information to the model. In general, this is accomplished by designating a speaker of interest to model in a conversation.
As visualized in Figure 1, the designated speaker's reference history tokens are gathered and concatenated together, with a parent comment followed by its associated reply (made by the speaker of interest) followed by another parent comment and so forth. Positional embeddings will signal to the model the order of comments being made; however, additional signal is needed to tell the model which comments are made by the speaker of interest and which are not. This is achieved by token type embeddings that get added to the positional and vocabulary embeddings. All tokens in the reference history that belong to the speaker get the same token type embedding, and all others get a different one. This representation choice allows us to naturally handle multi-actor conversation by only making a distinction between the speaking
2"Karma" can be thought of as the net amount of likes and dislikes a comment has, as voted upon by the users of Reddit.
(i) Target Speaker: A
(ii) Target Speaker: C
Figure 1: Illustration of input representation for a conversation from three different speakers  $(A, B, C)$  composed of a sequence of four turns (denoted  $A^1, B, A^2, C$  in the "Tokens" row) when separately modeling two different target speakers ( $A$  and  $C$  for (i) and (ii) respectively). The model receives different reference histories (parent comments  $P_{A/C}^j$  and associated reply  $R_{A/C}^j$  from target speaker  $A/C$  for  $j \in \{1, 2\}$ ) and different placements of token types based on which Target Speaker is modeled (in "Token Types" row,  $P$  denotes parent comment,  $R$  is reply comment,  $S$  is a turn from target speaker, and  $NS$  is a turn not from the target speaker. Losses are computed only for the comments corresponding to the active Target Speaker  $(\hat{A}^j \vee \hat{C})$ . Note that this representation explicitly allows for multi-actor conversation modeling.
user and non speaking users. Reference history sequences larger than 512 are truncated from the end to keep the length within 512 tokens.
The conversation turns are similarly represented by concatenating them together in order of occurrence with a special token at the beginning signifying the start of the conversation. For practicality, all turns after the final turn associated with the target speaker are discarded for a given iteration. Each token in the conversation sequence receives a specific token type embedding if it is associated with the speaker of interest, and receives a different type if not. Note, the conversation and reference history have disjoint sets of token type embeddings to differentiate the different types of content. The max length a conversation sequence can be is 512 tokens with extra tokens truncated from the beginning of the conversation to encourage a variety of conversation lengths. In models that have access to the reference history this leads to a total sequence length of 1024 tokens and 512 tokens otherwise.
There is flexibility in how to model conversations with reference histories due to the turns in a conversation and reference comments being indirectly related, both content and style-wise. As such, the design choices we consider either encode the references separate from the conversation, or together.
Decoder-Only: GCC-DEC The simplest of the three considered models consists of only a transformer for decoding, which is the original configuration for GPT-2. The input consists of the reference history tokens concatenated with the con
versation turn tokens and the corresponding token types. A left-to-right (LR) mask is used across the entire sequence. See Figure 1 for an illustration. Despite it's simplicity we find that this model performs the best.
Seq2Seq Baseline: GCC-S2S For this model the reference material with corresponding token types is encoded in a separate transformer using a bidirectional mask. The conversation turns are then decoded with a LR mask using both self-attention and attention against the final hidden states of the encoded reference. This is representative of the typical formulation for attention-based Seq2Seq models (Vaswani et al., 2017).
Variational Autoencoder Baseline: GCC-VAE This configuration also encodes the reference history and corresponding token types in a separate transformer using a bidirectional mask. The final hidden state of a special classification token is then linearly transformed into the sufficient statistics of a normal latent state which is then sampled. This latent state is then prepended to the embedded inputs of the conversation turns. The final sequence is then decoded using a LR mask in a separate transformer. We explored this method as latent variables are commonly used to control aspects of style across various areas of study.
No Reference Context Baseline: GCC-NRC This version is similar to GCC-DEC except that there are no reference material included when decoding information. This model can be seen as a re-implementation of Olabiyi and Mueller (2019) with the minor differences being that we introduced
token types for multi-actor modeling and we did not utilize their random padding strategy. We found this unnecessary as we did not experience overfitting due to the large amount of training data available. As such, GCC-NRC will largely serve as our previous SOTA baseline to compare against when demonstrating the advantage of conditioning on prior conversations.
It is known that for the language modeling validation perplexity measures using teacher forcing is not the best evaluation of generative capabilities, even if there is correlation between the two. However, it is a commonly used metric for language modeling, and can be parallelized and computed inexpensively without the need for autoregressive sampling of output text. With that in mind, two sets of evaluations were done, the first of which being an architecture search using automatic evaluation with validation perplexity and the second being a qualitative study using Amazon's Mechanical Turk<sup>3</sup> to assess generated samples.
5.1 Automatic Evaluation
All evaluations in this section are done on the validation set (Reddit comments from April, 2019) using perplexity, which is calculated as follows:
$$
\operatorname {P P L} (\theta , \mathcal {D}) = \exp \left\{- \frac {1}{| \mathcal {D} |} \sum_ {i = 1} ^ {| \mathcal {D} |} \log p _ {\theta} \left(\mathbf {c} ^ {(i)} \mid \mathbf {r} ^ {(i)}\right) \right\} \tag {7}
$$
All models are trained using mixed precision arithmetic, a learning rate that linearly increases from 0.0 to  $1.5e - 4$  over the first  $1\%$  of iterations followed by it decaying to 0.0 over the remaining iterations with a cosine annealing schedule, and the Adam optimization algorithm with default hyperparameters (Kingma and Ba, 2014).
Architecture We evaluate three main architectures under two scenarios: similar total number of encoder and decoder parameters, and similar total number of decoder parameters. As such, a 355M parameter version of GCC-DEC is compared to two versions each of GCC-S2S and GCC-VAE. When present, the encoder and decoder transformers shared the same hidden sizes, number of layers, and number of attention heads. Additionally, all
Model	h	l	A	Params	PPL
GCC-S2S	768	18	16	375M	22.09
GCC-VAE	768	20	16	362M	22.43
GCC-DEC	1024	24	16	355M	19.10
GCC-S2S	1024	24	16	810M	19.89
GCC-VAE	1024	24	16	711M	20.49
Table 3: Comparison of model architecture perplexity (PPL) trained from scratch for 200K iterations. The top half of the table are iso-parameter count experiments while the bottom half are iso-architecture experiments. Note that  $h, l$ , and  $A$  define the sizes of the encoder and decoder transformers individually.
models were trained from scratch for 200,000 iterations at a global batch size of 256.