SlowGuess/NAACL_2022_OCR huggingface dataset:
Ramya Ramakrishnan
rramakrishnan@asapp.com
Hashan Buddhika Narangodage
hnarangodage@asapp.com
Mauro Schilman
mschilman@asapp.com
Kilian Q. Weinberger
ASAPP, Cornell
kweinberger@asapp.com
rmcdonald@asapp.com
Current approaches for controlling dialogue response generation are primarily focused on high-level attributes like style, sentiment, or topic. In this work, we focus on constrained long-term dialogue generation, which involves more fine-grained control and requires a given set of control words to appear in generated responses. This setting requires a model to not only consider the generation of these control words in the immediate context, but also produce utterances that will encourage the generation of the words at some time in the (possibly distant) future. We define the problem of constrained long-term control for dialogue generation, identify gaps in current methods for evaluation, and propose new metrics that better measure long-term control. We also propose a retrieval-augmented method that improves performance of long-term controlled generation via logit modification techniques. We show through experiments on three task-oriented dialogue datasets that our metrics better assess dialogue control relative to current alternatives and that our method outperforms state-of-the-art constrained generation baselines.
Despite recent advances in dialogue systems (Serban et al., 2016; Ham et al., 2020), controlling dialogue generation remains a significant challenge. Response generation in dialogue can be controlled towards different topics and styles (Madotto et al., 2020) or towards a set of hard constraints (i.e., lexical control words need to appear in the generated text) (Sha, 2020). We focus on the hard constraint setting, also known as constrained generation, as this provides a more fine-grained method of controlling dialogues.
For example, consider a customer service use case (Figure 1), in which an agent speaks to a
Control words: shirt, refund, credit, oversized, please
Short-term control
Agent: Hi! How may I help you?
Customer: I want a shirt today with credit please.
Agent: Which brand and size?
Customer: Tommy Hilfiger and large size
Agent: What's your address?
Customer: 53 Fennel Creek Drive, Boston, MA 08531
Agent: Great, the order is in.
Customer: Thank you so much for your help!
Long-term control
Customer: I'd like to get a refund please.
Agent: What was the item?
Customer: It's a Nike shirt I bought a week ago.
Agent: What's wrong with it?
Customer: It's oversized and doesn't fit me.
Agent: What was your payment?
Customer: I bought it using my credit card.
Figure 1: Examples of short vs. long-term control for dialogue generation. (Left) In short-term control, many control words are generated initially, but the conversation is led away from the desired future. (Right) In long-term control, responses are generated with the future in mind with words generated at natural points in the conversation.
customer about an issue. The goal is to generate a given set of control words in the responses of one of the speakers (agent or customer). Naive constrained generation approaches (Pascual et al., 2020; Miao et al., 2019) use methods like beam search and stochastic search to force the generation of these control words for short-term control, where control words need to appear in a single utterance or phrase. Because they do not consider the future, these approaches may generate the words all at once in a single response or not generate them at natural places in the conversation (Figure 1, left).
The above example highlights the challenges of applying existing constrained generation methods to long-term dialogue generation. First, since another speaker is involved in the dialogue, the model does not have full control of the generated text. Instead, the model can only control the dialogue indirectly. Second, dialogues can be long and
$^{1}$ Our code is available at https://github.com/a sappresearch/constrained-dialogue-generation
Human Language Technologies, pages 738 - 753
thus, controlling utterances several time steps into the future is non-trivial. In this work, we propose the problem of long-term dialogue control, where the goal is to generate a set of control words over many utterances in a dialogue, which requires appropriately timing the generation of control words (Figure 1, right). To the best of our knowledge, we are the first work to constrain long-term dialogue generation through lexical control words.
We begin by highlighting challenges with evaluation for this problem. Successful long-term control of dialogue can be difficult to measure. We describe current evaluation metrics for constrained text generation and show that these metrics can be gamed by generating all or many control words early in the conversation. To resolve this and measure how natural the control is, we propose a new set of metrics: long-term success rate, which measures the percentage of control words in simulated roll-outs of the conversation, and precision, recall, and F1-score, which compare control words in generated responses to those in reference responses from a historical dataset. The second set of metrics specifically help to capture whether the control words are generated at the right time.
Next, we propose a novel method to explicitly address long-term control. Prior methods are unable to handle this task as the number of possible future sequences is exponential. To alleviate this issue, we retrieve similar conversations from training and condition on them during generation. We first identify similar neighbors using a kNN-based approach and then guide the language model towards generating similar responses, inspired by plug-and-play methods (Madotto et al., 2021; Dathathri et al., 2019; Pascual et al., 2020). The motivation for this is that retrieved conversations guide the model to generate the control words at more natural points in the conversation.
We conduct experiments on multiple task-oriented dialogue datasets and show that our method outperforms several constrained text generation baselines on automated evaluation metrics as well as human evaluation. Specifically, we are able to generate  $30 - 40\%$  more control words on long-term success rate compared with baselines, while preserving fluency (scores of  $\geq 4.3$  out of 5), as measured by human evaluation.
Controllable text generation. Prior work has developed many methods for controllable text generation. These approaches can be categorized into three general areas. The first is altering decoding strategies (Grover et al., 2019; Deng et al., 2020), in which the sampling distribution can be modified (Ghazvininejad et al., 2017; Baheti et al., 2018) or hidden states in the models can be changed (Gu et al., 2017). The second area involves including prompts to guide text generation (Ribeiro et al., 2018; Jiang et al., 2020; Li and Liang, 2021), for example through universal trigger tokens (Wallace et al., 2019; Shin et al., 2020). Finally, finetuning can be used to guide language model outputs through the use of a latent variable (Fan et al., 2018; Peng et al., 2018) or through CTRL codes (Keskar et al., 2019). Our work differs from the broad area of controllable language generation in that 1) we require more fine-grained generation through lexical control words and 2) we focus on dialogue settings where another speaker can also change the course of the conversation.
Constrained text generation. The key difference between constrained text generation and controllable text generation is the focus on hard rather than soft constraints. Typically, there are two general methods for constrained generation: beam search (Hokamp and Liu, 2017; Post and Vilar, 2018; Pascual et al., 2020) and stochastic search (Miao et al., 2019; Sha, 2020). Directed Beam Search (DBS) (Pascual et al., 2020), modifies language model logits to encourage generation of a specified set of "guide words", or control words. A method based on stochastic search (Miao et al., 2019) uses Metropolis-Hastings with the constraint of keyword inclusion. These approaches do not apply to the dialogue setting where these constraints need to hold for many utterances into the future.
Dialogue response generation. While many works develop methods for unconstrained response generation (Budzianowski and Vulic, 2019; Peng et al., 2020; Cao et al., 2020; Hosseini-Asl et al., 2020; Yavuz et al., 2019), there is a subset of work more related to our problem focused on controlling response generation. In one work, transformer models are fine-tuned for dialogue through modifications of the inputs, for example by adding information about the user's persona (Wolf et al., 2019). The work of Lippe et al. (2020) generates utter
ances by paraphrasing templated responses. Several works control generation through exemplarguided methods (Cai et al., 2020; Gupta et al., 2020), which is a different setting from ours since we want to guide generation based on a set of control words rather than through a prototype. One work (Xu et al., 2019) controls response generation through meta-words that include desired attributes of the response (e.g., response length and specificity). Another work controls response generation through control words by adding inductive biases into training to guide generation (Wu et al., 2020). However, this work only controls generation for a single response, rather than controlling several utterances into the future. The closest work to ours is work by (Tang et al., 2019), which proposes a similar problem of long-term control towards a target subject. While the setup is similar, we learn to constrain dialogue responses given a set of control words rather than a target attribute, which also results in a different approach.
Retrieval-augmented generation. Another related area is retrieval-augmented language generation, which inspires our approach of using retrieval to control dialogue generation. REALM (Guu et al., 2020) uses a latent knowledge retriever to identify relevant documents and backpropagates through this retrieval step. In another work (Fan et al., 2020), relevant information is retrieved from an external knowledge base to guide dialogue generation. Several works by Khandelwal et al leverage nearest neighbor approaches to improve performance with no additional training (Khandelwal et al., 2019, 2020). While these works condition on retrieval for uncontrolled generation, we leverage ideas from this space specifically for control in dialogue.
3 Problem definition
We first define the problem of long-term constrained dialogue generation. A conversation  $\mathcal{X} = \{s_1, u_1, s_2, u_2, \dots, s_T, u_T\}$  is defined as a list of utterances generated by two speakers: the system  $s$  that we are trying to control and the user  $u$ , which we don't have explicit control over.  $T$  denotes the total number of turns in the conversation. Given the current dialogue context of a conversation  $x = \{s_1, u_1, \dots, s_t, u_t\}$  up until timestep  $t$  and a set of control words  $\mathcal{W} = \{w_1, w_2, \dots, w_M\}$ , our goal is to generate the remaining responses of the conversation  $\mathcal{R}_{t+1:T} = \{s_{t+1}, \dots, s_T\}$  such that the control words  $\mathcal{W}$  appear in the future generated
responses. We consider a scenario in which someone provides a set of control words to be included in the conversation without assumptions on their order. This means methods need to handle control words given in any order.
We additionally assume access to a historical dataset of conversations  $\mathcal{D} = \{x^{(i)}\}$ ,  $i \in [1, \dots, N]$  and a fine-tuned language model  $M$  on this dataset. We can leverage these inputs in order to control future responses  $\mathcal{R}_{t+1:T}$ . We focus on the plug-and-play setting (Pascual et al., 2020), in which approaches simply guide the given language model  $M$  towards generating the control words without any additional re-training.
4 Proposed metrics for evaluation
Directly evaluating the generated responses in terms of prior evaluation methods can lead to misleading results. Previous works on constrained text generation (Pascual et al., 2020) have used metrics like perplexity to measure fluency and success rate to measure the percentage of control words generated. However, these metrics are more relevant for short-term generation, as they can be gamed in settings where the control words would be naturally distributed across the full conversation. As shown in the left-hand side of Figure 1, when several words are forced into the first response, the conversation may move away from the desired future and control word generation could be inappropriately timed. To better evaluate how well the model generates the right words at the right time, we propose the following new metrics.
The first metric we propose is long-term success rate, which involves simulating conversations with a user language model and computing the percentage of generated control words in the system responses of these simulated roll-outs. Prior work (Ghandeharioun et al., 2019) has used self-play for evaluation, but they do not propose roll-outs as a way to measure dialogue control.
Long-term success rate: Our modified success rate metric is computed as the fraction of control words generated in a full simulated roll-out of the conversation. We compute this as:  $s = \frac{n_w}{|\mathcal{W}|}$ , where  $n_w$  is the number of control words that appear in all of the future system responses  $\mathcal{R}_{t+1:T}$ .
One limitation of long-term success rate is that it doesn't measure the timing of control words in the conversation. So next, we want to evaluate whether the methods generate control words at appropriate
Figure 2: Visualization of FOP-retrieval. First, each conversation in the historical dataset  $\mathcal{D}$  is split into many past-future conversation pairs. The current context  $x$  and the pasts are encoded using language model  $M$ . We use kNN search to identify pasts similar to context  $x$  and then select a desired future with the highest number of control words. The output is the first response in the selected future  $\tilde{s}_{t+1}$ .
points in the conversation. To measure this, we propose computing precision, recall, and F1-score for control words. This particular evaluation is not done in simulation. Instead, we consider each true system response in the evaluation dataset in isolation and generate a response for each, given the conversation history up until that point. We compute the number of generated control words that are correctly predicted, when compared with the control words in the ground truth response in the same time step.
For example, on the right side of Figure 1, when generating the second customer response (given the true conversation history up until then), we would count a "correct" prediction for P/R/F1 as a response that includes the word "shirt" (in any position in the response), as it is a control word that appears in the ground truth response in that time step. It is true that control words can also appear later in the conversation, but this setting is already evaluated by long-term success rate in simulated rollouts. After counting the number of correctly predicted control words for each response individually, we aggregate across all responses.
Precision: Precision is calculated at the corpus-level as the number of correctly predicted control words over the total number of predicted control words  $(p = \frac{|\text{correct}|}{|\text{predicted}|})$ .
Recall: Recall is similarly computed at the corpus-level as the number of correctly predicted control words over the total number of actual control words  $(r = \frac{|correct|}{|actual|})$ .
F1-score: Finally, F1-score combines precision and recall into one metric  $(f1 = \frac{(2*p*r)}{(p + r)})$ .
These metrics penalize models that condense all control words into one response. Instead, we want the models to naturally generate control words when they are relevant. These metrics evaluate whether control words are generated at the appropriate position in a conversation. To introduce some flexibility, an extension could be to compute a soft version of precision, recall, and F1-score that scores utterances based on whether control words appear within N utterances of the ground truth position.
Finally, we use human evaluation to evaluate how realistic and relevant the generated responses are. Specifically, we evaluate each conversation on fluency, consistency of control word generation, relevance, coherence, and diversity.
5 Retrieval-based Control
We now present our proposed approach for constrained dialogue generation. Inspired by work in retrieval-augmented generation (Guu et al., 2020; Fan et al., 2020), we retrieve similar pasts based on the current context  $x$  and use their futures to control dialogue response generation. The key insight here is that by looking at how people have used these control words in similar conversations in the past, we can bias the models towards more natural dialogues. In other words, we use futures of the past conversations to guide the current response generation. To better motivate the use of retrieval in our problem, consider the example conversation in Figure 1. The agent asks which item the customer wants to return, and there are many possible answers (e.g., "I want my pant refunded.", "I want to return gloves I bought yesterday"). Keyword-
Figure 3: Visualization of FOP-guided. Language model logits are first modified using a window-based approach. All words (and similar words based on GloVe vector similarity) within the window are upweighted with a weight decay. Once any word in the window is generated, the window shifts until the full response is generated. After  $N$  generations, a re-ranking step selects the response with the highest number of control words and lowest loss.
based retrieval will surface a response about shirts, a control word, which encourages the model to generate a natural response with that word: "It's a Nike shirt I bought a week ago."
We present two variants of our retrieval-inspired Futures of the Past (FOP) approach: 1) FOP-retrieval: we retrieve the desired future from historical data and simply use the retrieved utterance as the generated response and 2) FOP-guided: we use the utterance from FOP-retrieval as a reference sentence to guide the model towards similar responses.
The simple variant of our approach, FOP-retrieval, is shown in Figure 2. It focuses on identifying what the model should say now that will lead to the control words in the future. The reason we need to determine what to say now is that control words in our problem are distributed across a long dialogue conversation. One possible approach to generate the current response is to run many rollouts of the conversation and select the response that leads to the highest number of control words. However, this brute force approach is computationally expensive and will not be effective for rich, diverse conversations. Instead, we leverage historical conversation data to identify the most relevant futures given the current context and control words. The retrieved futures can guide the model towards what to say now that will lead to the desired future. The guided variant, shown in Figure 3, involves guiding the language model towards generating a response similar to the retrieved utterance.
Our proposed approaches address some of the challenges of long-term control for dialogue generation. First, another speaker can change the course of the conversation, which is why we retrieve a new set of similar past contexts at each time step to re-align with the current context. Second, to control responses many steps into the future, we
retrieve historical conversations with the desired future (high percentage of control words) and gently nudge the conversation in that direction, thus controlling not only the current utterance but also the future of the conversation.
5.1 Retrieval Futures of the Past (FOP-retrieval)
For the retrieval component, the goal is to select futures that have relevant past contexts as well as desired futures based on the control words. To do this, we employ a multi-step approach. First, we split each conversation  $x^{(i)}$  in the historical dataset  $\mathcal{D}$  into a set of past-future conversation pairs  $x^{(i)} = \{(p,f)^{(i,j)}\}$ . We encode the current context  $M(x)$  and each past conversation  $M(p^{(i,j)})$  using the language model  $M$ . Then, we use kNN search based on FAISS, a library for fast nearest neighbor retrieval (Johnson et al., 2019), to identify  $k$  similar pasts from the historical data that closely match the current context  $x$ . We then filter the futures of these past conversations based on which have the highest percentage of control words.
$$
\mathrm {K N N} _ {\mathbf {x}} = \operatorname {f a i s s} (M (x), M (p ^ {(i, j)}), k)
$$
$$
f ^ {*} = \operatorname {a r g m a x} \left(\left[ \operatorname {c o u n t} \left(f ^ {(i, j)}, \mathcal {W}\right) \right] _ {f ^ {(i, j)} \in \mathrm {K N N} _ {\mathrm {x}}}\right)
$$
$$
\tilde {s} _ {t + 1} = f ^ {*} [ 0 ], f ^ {*} = \{s _ {1}, u _ {1}, \dots , s _ {T}, u _ {T} \}
$$
In the above equations, the count function counts the number of control words  $\mathcal{W}$  in the future  $f^{(i,j)}$ . The reference response  $\tilde{s}_{t + 1}$  is simply the first utterance of the retrieved future.
5.2 Guided Futures of the Past (FOP-guided)
Now that we have a candidate reference response  $\tilde{s}_{t + 1}$ , we can guide the language model towards generating a similar response. To do this, we modify the logits from the language model to encourage
generation of the control words or similar words. We start with the first word  $w_{0}$  in  $\tilde{s}_{t + 1}$  and upweight logits in a way similar to DBS (Pascual et al., 2020) using similarity of GloVe vector embeddings:
$$
l _ {i} ^ {\prime} = l _ {i} + \lambda \cdot \min  (0, \cos (\gamma (t _ {i}), \gamma (w _ {j}))) ^ {2},
$$
where  $\gamma$  represents GloVe embeddings,  $t_i$  is the  $i$ th token of the language model's vocabulary  $V$ ,  $w_j$  is the current reference word, and  $\lambda$  is a hyperparameter specifying how much weight to put on generating words similar to  $w_j$ .
With this approach, we observed that sometimes the model got stuck on the first word and never moved on to later words. To enable more flexible control, instead of requiring every word to be generated before moving on to the next word, we include a window of size  $q$  and increase the logits of each word in the window, with a decay multiplier of  $\frac{1}{2^i}$ ,  $i \in q$ . If any of the words in the window have been generated, the window is shifted beginning from the generated word with the same window size of  $q$ . The process repeats until the full response has been generated.
The decay multiplier is used to encourage the model to generate earlier words in the reference response and not skip words unless it's highly likely. We generate  $N$  such responses using this method and include an additional ranking step to select the best one. We first sort by the number of control words in the generated response. If multiple responses generate the highest number of control words, we sort by the loss from the model and select the response with the lowest loss  $l$ :
$$
\begin{array}{l} \tilde {\mathcal {R}} _ {t + 1} = \{M ^ {j} (l ^ {\prime}) | j \in [ 1, \dots , N ] \} \\ s ^ {*} = \max  \left(\left[ \operatorname {c o u n t} (r, \mathcal {W}) \right] _ {r \in \tilde {\mathcal {R}} _ {t + 1}}\right) \\ \end{array}
$$
$$
\begin{array}{l} \hat {\mathcal {R}} _ {t + 1} = \left\{r | \operatorname {c o u n t} (r, \mathcal {W}) = s ^ {*}, r \in \tilde {\mathcal {R}} _ {t + 1} \right\} \\ r _ {t + 1} = \operatorname {a r g m i n} \left(\left[ \operatorname {l o s s} (r) \right] _ {r \in \hat {\mathcal {R}} _ {t + 1}}\right), \\ \end{array}
$$
where  $\tilde{\mathcal{R}}_{t + 1}$  is the set of  $N$  generated responses, using a model with logits  $l^{\prime}$ . The final generated response  $r_{t + 1}$  is selected based on the two-step ranking process. None of the other approaches include this ranking component.
6.1 Task-Oriented Dialogue Datasets