Venue: NAACL 2019

# Text Generation with Exemplar-based Adaptive Decoding

 Hao Peng\({}^{\spadesuit}\) Ankur P. Parikh\({}^{\diamondsuit}\) Manaal Faruqui\({}^{\diamondsuit}\) Bhuwan Dhingra\({}^{\clubsuit}\) Dipanjan Das\({}^{\diamondsuit}\)

\({}^{\spadesuit}\) Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, WA

\({}^{\diamondsuit}\) Google AI Language, New York, NY

\({}^{\clubsuit}\) School of Computer Science, Carnegie Mellon University, Pittsburgh, PA

hapeng@cs.washington.edu, {aparikh,mfaruqui,dipanjand}@google.com, bdhingra@andrew.cmu.edu

+ Work done during internship at Google.

###### Abstract

We propose a novel conditioned text generation model. It draws inspiration from traditional template-based text generation techniques, where the source provides the content (i.e., _what to say_), and the template influences _how to say it_. Building on the successful encoder-decoder paradigm, it first encodes the content representation from the given input text; to produce the output, it retrieves exemplar text from the training data as "soft templates," which are then used to construct an exemplar-specific decoder. We evaluate the proposed model on abstractive text summarization and data-to-text generation. Empirical results show that this model achieves strong performance and outperforms comparable baselines.

## 1 Introduction

Conditioned text generation is the essence of many natural language processing (NLP) tasks, e.g., text summarization [20], machine translation [38], and data-to-text generation [47, 56]. In its common neural sequence-to-sequence formulation [9, 2], an encoder-decoder architecture is used. The decoder generates the text autoregressively, token-by-token, conditioning on the feature representations encoded from the source, typically with attention [1] and copy mechanisms [21, 22]. This paradigm is capable of generating fluent abstractive text, but in an uncontrolled and sometimes unreliable way, often producing degenerate outputs and favoring generic utterances [23, 24].

The encoder-decoder approach differs considerably from earlier template-based methods [2, 25, 26], where the source content is filled into the slots of a handcrafted template. These solutions offer higher generation precision compared to neural approaches [23], but tend to lack the naturalness of neural systems, and are less scalable to open domain settings, where the number of required templates can be prohibitively large.

To sidestep the scalability problems with handcrafted templates, it has been proposed to use similar training samples as **exemplars**, to guide the decoding process [21, 22, 24, 25, 26]. In general, existing methods accomplish this by (a) using traditional information retrieval (IR) techniques for exemplar extraction (e.g., TF-IDF), and then (b) concatenating the exemplar to the source as additional inputs, allowing the decoder to attend over and copy from both.

We propose a different strategy for using exemplars. For motivation, Figure 1 shows a source-target pair together with its exemplar from the Gigaword dataset [26]. The target is a summary of the source sentence, and the exemplar is retrieved from the training set (SS3.2).2 There is word overlap between the exemplar and the desired output, which would be easily captured by an attention/copy mechanism (e.g. _Norway_ and _aid_). Despite this, ideally, the model should also exploit the structural and stylistic aspects to produce an output with a similar sentence structure, even if the words are different.

Footnote 2: We use the _training target_ as the exemplar, whose source is most similar to the current input. §3.2 describes the details.

Indeed, in tradition

is supposed to determine "what to say," while the templates aim to address "how to say it," reminiscent of the classical content selection and surface realization pipeline Reiter and Dale (1997). For instance, an ideal template for this example might look as follows:

```
----grantsaidof----------------to-----
```

In the neural formulation, the "how to say it" aspect is primarily controlled by the decoder.

Inspired by the above intuition, we propose **exemplar-based adaptive decoding**, where a customized decoder is constructed for each exemplar. This is achieved by letting the exemplars to directly influence decoder parameters through a reparameterization step (SS3.1). The adaptive decoder can be used as a drop-in replacement in the encoder-decoder architecture. It offers the potential to better incorporate the exemplars' structural and stylistic aspects into decoding, without excessive increase in the amount of parameters or computational overhead.

We empirically evaluate our approach on abstractive text summarization and data-to-text generation (SS4), on which most of the recent efforts on exemplar-guided text generation have been studied. On three benchmark datasets, our approach outperforms comparable baselines, and achieves performance competitive with the state of the art. The proposed method can be applicable in many other conditioned text generation tasks. Our implementation is available at [https://homes.cs.washington.edu/~hapeng](https://homes.cs.washington.edu/~hapeng).

## 2 Background

This section lays out the necessary background and notations for further technical discussion. We begin with conditioned text generation and the encoder-decoder framework Sutskever et al. (2014); Cho et al. (2014). In the interest of the notation clarity, SS3 will use an Elman network Elman (1990) as a running example for the decoder, which is briefly reviewed in SS3. The proposed technique generalizes to other neural network architectures (SS3.3).

Conditioned text generation and the encoder-decoder architectureOur discussion centers around conditioned text generation, i.e., the model aims to output the target \({\bf y}=y_{1}y_{2}\dots y_{T}\) given the source input \({\bf x}=x_{1}x_{2}\dots x_{S}\), both of which are sequences of tokens. Each token \(x_{i}\), \(y_{i}\) takes one value from a vocabulary \(\mathcal{V}\). \({\bf x}\) and \({\bf y}\) could vary depending on the tasks, e.g., they will respectively be articles and summaries for text summarization; and for data-to-text generation, \({\bf x}\) would be structured data, which can sometimes be linearized Lebret et al. (2016); Wiseman et al. (2018), _inter alia_), and \({\bf y}\) is the output text. We aim to learn a (parameterized) conditional distribution of the target text \({\bf y}\) given the source \({\bf x}\),

\[p\big{(}{\bf y}\mid{\bf x}\big{)}=\prod_{t=1}^{T}p\big{(}y_{t}\mid{\bf y}_{< t},{\bf x}\big{)}, \tag{1}\]

where \({\bf y}_{<t}=y_{1}\dots y_{t-1}\) is the prefix of \({\bf y}\) up to the \((t-1)^{\textrm{th}}\) token (inclusive).

The probability of each target token is usually estimated with a \(\operatorname*{softmax}\) function:

\[p\left(y_{t}\mid{\bf y}_{<t},{\bf x}\right)=\frac{\exp{\mathbf{h}_{t-1}^{ \top}}{\mathbf{w}}_{y_{t}}}{\sum_{y}\exp{\mathbf{h}_{t-1}^{\top}}{\mathbf{ w}}_{y}}. \tag{2}\]

\({\mathbf{w}}_{y}\) denotes a learned vector for token \(y\in\mathcal{V}\). \({\mathbf{h}}_{t-1}\) depends on \({\bf y}_{<t}\) and \({\bf x}\), and is computed by a function which we will describe soon.

A typical implementation choice for computing \({\mathbf{h}}_{t}\) is the encoder-decoder architecture Sutskever et al. (2014). More specifically, an **encoder**\({\mathbf{g}}_{\boldsymbol{\theta}}\) first gathers the feature representations from the source \({\bf x}\); then a **decoder**\({\mathbf{f}}_{\phi}\) is used to compute the \({\mathbf{h}}_{t}\) feature vectors:

\[{\mathbf{h}}_{t}={\mathbf{f}}_{\phi}\Big{(}y_{t},{\mathbf{h}}_{t-1},{\mathbf{ g}}_{\boldsymbol{\theta}}({\bf x})\Big{)}. \tag{3}\]

\(\boldsymbol{\theta}\) and \(\boldsymbol{\phi}\) are, respectively, the collections of parameters for the encoder and the decoder, both of which can be implemented as recurrent neural networks (RNNs) such as LSTMs Hochreiter and Schmidhuber (1997) or GRUs Cho et al. (2014), or the transformer Vaswani et al. (2017). In Sutskever et al. (2014), the dependence of \({\mathbf{f}}_{\phi}\) on \({\mathbf{g}}_{\boldsymbol{\theta}}\) is made by using the last hidden state of the encoder as the initial state of the decoder. Such dependence can be further supplemented with at

Figure 1: A source-target pair from Gigaword training set, along with its exemplar.

 

tention (Bahdanau et al., 2015) and copy mechanisms (Gu et al., 2016; See et al., 2017), as we will do in this work.

SS3 introduces how we use exemplars to inform decoding, by dynamically constructing the decoder's parameters \({\mathbf{\phi}}\). For the notation clarity, we will use the Elman network as a running example, reviewed below.

Elman networksGiven input sequence \({\bf x}\), an Elman network (Ellan, 1990) computes the hidden state at time step \(t\) from the previous one and the current input token by

\[{\bf h}_{t}=\tanh\bigl{(}{\bf Ph}_{t-1}+{\bf Qv}_{t}\bigr{)}, \tag{4}\]

where \({\bf P}\) and \({\bf Q}\) are learned \(d\times d\) parameter matrices (with \(d\) being the hidden dimension), and \({\bf v}_{t}\) is the embedding vector for token \(x_{t}\). We omit the bias term for clarity.

## 3 Method

This section introduces the proposed method in detail. Our aim is to use exemplars to inform the decoding procedure (i.e., _how to say it_). To accomplish this, we reparameterize the decoder's parameters with weighted linear sums, where the coefficients are determined by an exemplar. The decoder is **adaptive**, in the sense that its parameters vary according to the exemplars. The adaptive decoder can be used as a drop-in replacement in the encoder-decoder architecture. Before going into details, let us first overview the high-level generation procedure of our model. Given source text \({\bf x}\), the model generates an output as follows:

1. Run a standard encoder to gather the content representations \({\bf g}_{\theta}({\bf x})\) from the source.
2. Retrieve its exemplar \({\bf z_{x}}\), and compute exemplar-specific coefficients (SS3.2).
3. Construct the adaptive decoder parameters \({\mathbf{\phi}}\) (SS3.1), using the coefficients computed at step 2. Then the output is generated by applying the adaptive decoder followed by a \(\mathrm{softmax}\), just as in any other encoder-decoder architecture.

Aiming for a smoother transition, we will first describe step 3 in SS3.1, and then go back to discuss step 2 in SS3.2. For clarity, we shall assume that the decoder is implemented as an Elman network (Ellan, 1990; Equation 4). The proposed technique generalizes to other neural network architectures, as we will discuss later in SS3.3.

### Reparameterizing the RNN Decoder

At its core, the exemplar-specific adaptive decoder involves a reparameterization step, which we now describe. We focus on the parameters of the Elman network decoder, i.e., \({\bf P}\) and \({\bf Q}\) in Equation 4.

Parameter construction with linear sumsWe aim to reparameterize the pair of matrices \(({\bf P},{\bf Q})\), in a way that they are influenced by the exemplars.

Let us first consider an extreme case, where one assigns a different pair of parameter matrices to each exemplar, _without_ any sharing. This leads to an unreasonably large amount of parameters, which are difficult to estimate reliably.3

Footnote 3: The amount of parameters grows linearly with the number of possible exemplars, which, as we will soon discuss in §3.2, can be as large as the training set.

We instead construct \({\bf P}\) and \({\bf Q}\) from a set of predefined parameters matrices. Take \({\bf P}\) for example, it is computed as the weighted sum of \({\bf P}_{i}\) matrices:

\[{\bf P}=\sum_{i=1}^{r}\lambda_{i}{\bf P}_{i}, \tag{5}\]

where \({\bf P}_{i}\in\mathbb{R}^{d\times d}\), with \(d\) being the size of the hidden states. \(r\) is a hyperparameter, determining the number of \({\bf P}_{i}\) matrices to use.4 The summation is weighted by the coefficients \(\lambda_{i}\), which are computed from the exemplar \({\bf z_{x}}\). For clarity, the dependence of both \({\bf P}\) and \(\lambda_{i}\) on \({\bf z_{x}}\) is suppressed when the context is clear.

Footnote 4: Instead of choosing \(r\) empirically, we set it equal to \(d\) in the experiments. Please see the end of §3.1 for a related discussion.

Equation 5 constructs the decoder's parameter matrix \({\bf P}\) using a linear combination of \(\{{\bf P}_{i}\}_{i=1}^{r}\). The exemplar informs this procedure through the coefficients \(\lambda_{i}\)'s, the detailed computation of which is deferred to SS3.2. The other matrix \({\bf Q}\) can be similarly constructed by \({\bf Q}=\sum_{i}\lambda_{i}{\bf Q}_{i}\).

Rank-1 constraintsIn the above formulation, the number of parameters is still \(r\) times more than a standard Elman network, which can lead to overfitting with a limited amount of training data. Besides, it would be more interesting to compare the adaptive decoder to a standard RNN under a comparable parameter budget. Therefore we want to further limit the amount of parameters. This can be achieved by forcing the ranks of \({\bf P}_{i}\) and \({\bf Q}_{i}\) to be 1, since it then takes \(2d\) parameters to form each of them, instead of \(d^{2}\). More formally, we upper 

bound their ranks by construction:

\[{\bf P}_{i}={\bf u}^{(p)}_{i}\,\otimes\,{\bf v}^{(p)}_{i}. \tag{6}\]

\({\bf a}\otimes{\bf b}={\bf ab}^{\top}\) denotes the outer product of two vectors; \({\bf u}^{(p)}_{i}\) and \({\bf v}^{(p)}_{i}\) are learned \(d\)-dimensional vectors. Each \({\bf Q}_{i}\) can be similarly constructed by a separate set of vectors \({\bf Q}_{i}={\bf u}^{(q)}_{i}\otimes{\bf v}^{(q)}_{i}\).

Let \({\bf U}_{p},{\bf V}_{p}\in\mathbb{R}^{d\times r}\) denote the stack of \({\bf u}^{(p)}_{i}\), \({\bf v}^{(p)}_{i}\) vectors, i.e.,

\[{\bf U}_{p} =\Big{[}{\bf u}^{(p)}_{1},\ldots,{\bf u}^{(p)}_{r}\Big{]}, \tag{7a}\] \[{\bf V}_{p} =\Big{[}{\bf v}^{(p)}_{1},\ldots,{\bf v}^{(p)}_{r}\Big{]}. \tag{7b}\]

Equations 5 and 6 can be compactly written as

\[{\bf P}={\bf U}_{p}\ {\boldsymbol{\Lambda}}\ {\bf V}^{\top}_{p}. \tag{8}\]

where \({\boldsymbol{\Lambda}}\) is the diagonal matrix built from the \(r\)-dimensional coefficient vector \({\boldsymbol{\lambda}}=[\lambda_{1},\ldots,\lambda_{r}]^{\top}\):

\[{\boldsymbol{\Lambda}}={\rm diag}({\boldsymbol{\lambda}})=\begin{bmatrix} \lambda_{1}&&\\ &\ddots&\\ &&&\lambda_{r}\end{bmatrix}. \tag{9}\]

The construction of \({\bf Q}\) is similar, but with a different set of parameters matrices \({\bf U}_{q}\) and \({\bf V}_{q}\):5

Footnote 5: The bias term in the Elman network \({\bf b}\) can be constructed as \({\bf b}={\bf B}{\boldsymbol{\lambda}}\), with \({\bf B}\) being a learned \(d\times r\) matrix.

\[{\bf Q}={\bf U}_{q}\ {\boldsymbol{\Lambda}}\ {\bf V}^{\top}_{q}. \tag{10}\]

Note that, despite their similarities to SVD at a first glance, Equations 8 and 10 are not performing matrix factorization. Rather, we are learning \(\{{\bf U}_{p},{\bf V}_{p},{\bf U}_{q},{\bf V}_{q}\}\) directly; \({\bf P}\), \({\bf Q}\), \(\{{\bf P}_{i}\}\), and \(\{{\bf Q}_{i}\}\) are never explicitly instantiated Peng et al. (2017, 2018c).

To summarize, we reparameterize \({\bf P}\) and \({\bf Q}\) as interpolations of rank-1 matrices. By the fact that \(\mathrm{rank}({\bf A}+{\bf B})\leq\mathrm{rank}({\bf A})+\mathrm{rank}({\bf B})\), the ranks of \({\bf P}\) and \({\bf Q}\) are upper-bounded by \(r\). As pointed out by Krueger and Memisevic (2017), the parameter matrices of a trained RNN tend to have full rank. Therefore, in the experiments, we set \(r\) equal to the hidden size \(d\), aiming to allow the adaptive decoder to use full-rank matrices in the recurrent computation. Yet, if one holds _a priori_ beliefs that the matrices should have lower ranks, using \(r<d\) could be desirable. When \(r=d\), an adaptive RNN constructed by the above approach has \(4d^{2}\) parameters, which is comparable to the \(2d^{2}\) parameters in a standard Elman network.6

Footnote 6: This does not include the bias term, which contributes additional \(d^{2}\) parameters to the former, and \(d\) to the latter.

### Incorporating Exemplars

We now discuss the computation of coefficients \({\boldsymbol{\lambda}}\), through which the exemplars inform the decoder construction (Equations 8 and 10). Before detailing the neural network architecture, we begin by describing the exemplar retrieval procedure.

**Retrieving exemplars \({\mathbf{z}}_{\mathbf{x}}\).** Intuitively, similar source texts should hold similar targets. Therefore, given source input \({\mathbf{x}}\), we use the _training target_ as its exemplar \({\mathbf{z}}_{\mathbf{x}}\), whose source is most similar to \({\mathbf{x}}\).7 To compute the similarities between source texts, we use bag-of-words (BOW) features and cosine similarity. We extract the top-1 exemplar for each instance. This step is part of the preprocessing, and we do _not_ change the exemplars as the training proceeds.

Footnote 7: The source of an exemplar is only used in the retrieval and never fed into the encoder-decoder model. For a training instance, we additionally disallow using its own target as the exemplar.

There are, of course, many other strategies to get the exemplars, e.g., using handcrafted or heuristically created hard templates Reiter et al. (2005); Becker (2002); Foster and White (2004), _inter alia_), randomly sampling _multiple_ training instances Guu et al. (2018), or learning a neuralerranker Cao et al. (2018). Using more sophistically extracted exemplars is definitelly interesting to explore, which we defer to future work.

Computing coefficients.Next we describe the computation of \({\boldsymbol{\lambda}}\), the \(r\)-dimensional coefficient vector, which is used to construct the adaptive decoder (Equations 8 and 10).

Intuitively, the rank-1 matrices (\({\bf P}_{i}\)'s and \({\bf Q}_{i}\)'s in Equation 6 and thereafter) can be seen as capturing different aspects of the generated text. And \({\boldsymbol{\lambda}}\) determines how much each of them contributes to the adaptive decoder construction. A natural choice to calculate \({\boldsymbol{\lambda}}\) is to use the similarities between the exemplar and each of the aspects.

To accomplish this, we run a RNN encoder over \({\mathbf{z}}_{\mathbf{x}}\), and use the last hidden state as its vector representation \({\mathbf{a}}\).8 We further associate each \(({\bf P}_{i},{\bf Q}_{i})\) pair with a learned vector \({\bf c}_{i}\); and then \(\lambda_{i}\) is computed as the similarity between \({\mathbf{a}}\) and \({\bf c}_{i}\), using an inner product \(\lambda_{i}={\mathbf{a}}^{\top}{\bf c}_{i}\). More compactly,

Footnote 8: For clarity, the dependence of \({\mathbf{a}}\) on the exemplar \({\mathbf{z}}_{\mathbf{x}}\) is suppressed, just as \({\boldsymbol{\lambda}}\).

\[{\boldsymbol{\lambda}}={\bf C}{\mathbf{a}}, \tag{11}\]

with \({\bf C}=[{\bf c}_{1},\ldots,{\bf c}_{r}]^{\top}\).

 

Closing this section, Algorithm 1 summarizes the procedure to construct an adaptive decoder.

### Discussion.

Although we've based our discussion on Elman networks so far, it is straightforward to apply this method to its gated variants Hochreiter and Schmidhuber (1997); Cho et al. (2014), _inter alia_), and other quasi-/non-recurrent neural architectures Bradbury et al. (2017); Vaswani et al. (2017); Peng et al. (2018a), _inter alia_). Throughout the experiments, we will be using an adaptive LSTM decoder (SS4). As a drop-in replacement in the encoder-decoder architecture, it introduces a reasonable amount of additional parameters and computational overhead, especially when one uses a small encoder for the exemplar (i.e., the sizes of the \({\bf c}_{i}\) vectors in Equation 11 are small). It can benefit from the highly-optimized GPU implementations, e.g., CuDNN, since it uses the same recurrent computation as a standard nonadaptive RNN.

In addition to the neural networks, the adaptive decoder requires access to the full training set due to the retrieval step. In this sense it is _semi-parametric_.9 The idea to dynamically construct the parameters is inspired by Hypernetworks Ha et al. (2017) and earlier works therein. It proves successful in tasks such as classification Jia et al. (2016); Liu et al. (2017) and machine translation Platanios et al. (2018). Many recent template-based generation models include the exemplars as _content_ in addition to the source, and allow the decoder to attend over and copy from both Gu et al. (2018); Guu et al. (2018); Weston et al. (2018); Pandey et al. (2018); Cao et al. (2018), _inter alia_). We compare to this approach in the experiments, and show that our model offers favorable performance, and that they can potentially be combined to achieve further improvements.

Footnote 9: Nothing prohibits adaptively constructing other components of the model, e.g., the encoder \({\bf g}_{\mathbf{o}}\). Yet, our motivation is to use exemplars to inform _how to say it_, which is primarily determined by the decoder (in contrast, the encoder relates more to selecting the content).

## 4 Experiments

This section empirically evaluates the proposed model on two sets of text generation tasks: abstractive summarization (SS4.2) and data-to-text generation (SS4.3). Before heading into the experimental details, we first describe the architectures of the compared models in SS4.1.

### Compared Models

In addition to previous works, we compare to the following baselines, aiming to control for confounding factors due to detailed implementation choices.

* Seq2seq. The encoder-decoder architecture enhanced with attention and copy mechanisms. The encoder is implemented with a bi-directional LSTM (BiLSTM; Hochreiter and Schmidhuber (1997); Schuster and Paliwal (1997); Graves (2012), and the decoder a uni-directional one. We tie the input embeddings of both the encoder and the decoder, as well as the \(\operatorname{softmax}\) weights Press and Wolf (2017). We use beam search during evaluation, with length penalty Wu et al. (2016).
* AttExp. It is based on Seq2seq. It encodes, attends over, and copies from the exemplars, in addition to the source inputs.

Our model using the adaptive decoder (AdaDec) closely builds upon Seq2seq. It uses a dynamically constructed LSTM decoder, and _does not_ use attention or copy mechanisms over the encoded exemplars. The extracted exemplars are the same as those used by AttExp. To ensure fair comparisons, we use comparable training procedures and regularization techniques for the above models. The readers are referred to the appendix for further details such as hyperparameters.

\begin{table}
\begin{tabular}{l r r r r} \hline \hline  & & **NYT** & **Giga** & **Wikibio** \\ \multirow{5}{*}{\# inst.} & Train & 92K & 3.8M & 583K \\  & Dev. & 9K & 190K & 73K \\  & Test & 9,706 & 1,951 & 73K \\ \cline{1-1} \cline{2-5} Avg. len. & Src. & 939.0 & 31.4 & N/A \\  & Tgt. & 48.6 & 8.2 & 26.0 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Number of instances and average text lengths for the datasets used in the experiments. The lengths are averaged over training instances.

 



### Text Summarization

Datasets.We empirically evaluate our model on two benchmark text summarization datasets:

* Annotated Gigaword corpus (Gigaword; Graff et al., 2003; Napoles et al., 2012). Gigaword contains news articles sourced from various news services over the last two decades. To produce the dataset, we follow the split and preprocessing by Rush et al. (2015), and pair the first sentences and the headlines in the news articles. It results in a 3.8M/190K/1,951 train/dev./test split. The average lengths of the source and target texts are 31.4 and 8.2, respectively.
* New York Times Annotated Corpus (NYT; Sandaus, 2008). It contains news articles published between 1996 and 2007 by New York Times. We use the split and preprocessing by Durrett et al. (2016).10 Following their effort, we evaluate on a smaller portion of the test set, where the gold summaries are longer than 50 tokens. We further randomly sample 9,000 instances from the training data for validation, resulting in a 91,834/9,000/3,452 train/dev./test split. Compared to Gigaword, the inputs and targets in NYT are much longer (averaging 939.0 and 48.6, respectively). Footnote 10: [https://github.com/gregdurrett/berkeley-doc-summarizer](https://github.com/gregdurrett/berkeley-doc-summarizer).

Table 1 summarizes some statistics of the datasets. We note that some recent works use a different split of the NYT corpus (Paulus et al., 2018; Gehrmann et al., 2018), and thus are not comparable to the models in Table 3. We decide to use the one by Durrett et al. (2016) because their preprocessing script is publicly available.

For both datasets, we apply byte-paired encoding (BPE; Sennrich et al., 2016), which proves to improve the generation of proper nouns Fan et al. (2018).

Empirical results.Table 2 compares the models on Gigaword test set in Rouge\(F_{1}\)Lin (2004).11 By using adaptive decoders, our model (AdaDec) improves over Seq2seq by more than 1.1 Rouge scores. Cao et al. (2018b) and the Full model by Cao et al. (2018a) hold the best published results. The former uses extensive handcrafted features and relies on external information extraction and syntactic parsing systems; while the latter uses additional encoding, attention and copy mechanisms over the exemplars extracted using a novel neural reranker. AdaDec achieves better or comparable performance to the state-of-the-art models, _without_ using any handcrafted features or reranking techniques. The Basic model by Cao et al. (2018a) ablates the reranking component from their Full model, and uses the top exemplar retrieved by the IR system. Therefore it is a more comparable baseline to ours. AdaDec outperforms it by more than 1.3 Rouge scores. Surprisingly, we do not observe interesting improvements by AttExp over the sequence-to-sequence baseline. We believe that our model can benefit from better extracted exemplars by, e.g., applying a reranking system. Such exploration is deferred to future work.

Footnote 11: Version 1.5.5 of the official script.

The NYT experimental results are summarized in Table 3. We follow previous works and report limited-length Rouge_recall_ values.12 Durrett et al. (2016) is an extractive model, and Paulus et al. (2018) an abstractive approach based on reinforcement learning. Our AdaDec model outperforms both. We observe similar trends when comparing AdaDec to the Seq2seq and AttExp baselines, with the exception that AttExp_does_ improve over Seq2seq.

Footnote 12: Following Durrett et al. (2016) and Paulus et al. (2018), we truncate the predictions to the lengths of the gold summaries, and evaluate Rouge recall, instead of \(F_{1}\) on full-length predictions.

### Data-to-text Generation

Data-to-text generation aims to generate textual descriptions of structured data, which can be

seen as a table consisting of a collection of records (Liang et al., 2009). For a given entity, each record is an _(attribute, value)_ tuple. Figure 2 shows an example for entity _Jacques-Louis David_. The table specifies the entity's properties with tuples _(born, 30 August 1748)_, _(nationality, French)_, and so forth. The table is paired with a description, which the model is supposed to generate using the table as input. We refer the readers to Lebret et al. (2016) for more details about the task.

Dataset and implementation detailsWe use the Wikibio dataset (Lebret et al., 2016). It is automatically constructed by pairing the tables and the opening sentences of biography articles from English Wikipedia. We follow the split and preprocessing provided along with the dataset, with around 583K/73K/73K train/dev./test instances. Following Lebret et al. (2016), we linearize the tables, such that we can conveniently train the sequence-to-sequence style models described in SS4.1. Table 1 summarizes some statistics of the dataset.

In contrast to the text summarization experiment (SS4.2), we do _not_ apply BPE here. Further, the word embeddings are initialized with GloVe (Pennington et al., 2014; fixed during training), and _not_ tied with the \({\rm softmax}\) weights. In addition to the models introduced in SS4.1, we additionally compare to AdaDec+AttExp, aiming to study whether the adaptive decoder can further benefit from attention and copy mechanisms over the exemplars.

Empirical resultsFollowing Liu et al. (2018) we report Rouge-4 and Bleu scores (Papineni et al., 2002).13 Table 4 summarizes the data-to-text generation results on the Wikibio test set. Overall, we observe similar trends to those in the summarization experiment (SS4.2): by attending over and copying from the exemplars, AttExp improves upon the Seq2seq baseline by around 0.6 absolute scores. Also utilizing exemplar information, our AdaDec model outperforms Seq2seq by a larger margin: 1.3 for Rouge-4 and 1.1 for Bleu. We further study whether we can get further improvements by combining both. AdaDec+AttExp achieves around 0.5 absolute improvements over AdaDec, less than those by AttExp over Seq2seq. This provides evidence that, to some extend, the ways AttExp and AdaDec incorporate exemplar information might be complementary. Wiseman et al. (2018) is a template-motivated model based on a semi-Markov model. Liu et al. (2018) hold the current state-of-the-art results. They encode the table structures by using (a) position and filed embeddings, and (b) structure-aware attention and gating techniques. These techniques are beyond the scope of this work, which focuses mainly on the decoding end.

Figure 2: A training instance from the Wikibio dataset. It consists of a collections of records for Jacques-Louis David (top), and a piece of textual description (bottom).

 



## 5 Analysis

We now qualitatively evaluate our model, by studying how its outputs are affected by using different exemplars. Figure 3 shows two randomly sampled Gigaword development instances. It compares the outputs by AdaDec (i.e., without attention/copy over exemplars; SS4.1) when receiving different exemplars, controlling for the same source inputs. In each example, Exemplar 1 is retrieved by the system (i.e., a training target; SS3.2); while the remaining ones are produced by the authors, by modifying the first one in styles and sometimes introducing distractions in the content.

In the top example, the model includes _people_ into the subject (_Three_ vs. _Three people_) under the influence by Exemplar 2; Exemplar 3 changes the tense and adds some distraction by changing the place from _Britain_ to _Canada_. The model follows the tense switch, but gets confused by the distraction, and decides to let a train in southern Europe collide into North America, which it should not. Looking at the bottom example, the model in general follows the exemplar in using noun adjuncts or prepositional phrases (e.g., _new home sales_ vs. _sales of new homes_), except the first one. Perhaps confused by the distraction in Exemplar 3, the model makes a judgment on the specific amount of growth, but gets it wrong.

## 6 Related Work

Exemplar-based generationPartly inspired by traditional template-based generation Kukich (1983); Reiter Dale (1997), _inter alia_), many recent efforts have been devoted to augmenting text generation models with retrieved exemplars Hodosh et al. (2013); Mason and Charniak (2014); Song et al. (2016); Lin et al. (2017), _inter alia_). Without committing to an explicit slot-filling process, a typical method is to include exemplars as additional inputs to the sequence-to-sequence models Gu et al. (2018); Pandey et al. (2018); Guu et al. (2018), _inter alia_). Wiseman et al. (2018) took a different approach and used a semi-Markov model to learn templates.

Dynamic parameter constructionThe idea of using a smaller network to generate weights for a larger one dues back to Stanley et al. (2009) and Koutnik et al. (2010), mainly under the evolution computing context. It is later revisited with representation learning Moczulski et al. (2015); Fernando et al. (2016); Al-Shedivat et al. (2017), _inter alia_), and successfully applied to classifica

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Model** & **RG-4** & **BLEU** \\ \hline Wiseman et al. (2018) & 38.6 & 34.8 \\ Liu et al. (2018) & 41.7 & 44.7 \\ \hline This work (Seq2seq) & 39.3 & 42.5 \\ \({}^{\dagger}\)This work (AttExp) & 40.0 & 43.1 \\ \hline \({}^{\dagger}\)This work (AdaDec) & 40.6 & 43.6 \\ \({}^{\dagger}\)This work (AdaDec+AttExp) & 41.1 & 44.1 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Data-to-text generation performance in Rouge-4 and Bleu on the Wikibio test set (§4.3). \(\dagger\) indicates the models using retrieved exemplars.

Figure 3: Two randomly sampled Gigaword development instances used for qualitative evaluation (§5). Exemplar 1’s are retrieved by the system (§3.2), while the remaining ones are produced by the authors. Notable exemplars changes are highlighted in bold purple, and output changes in _italic yellow_.



tion (Jia et al., 2016; Liu et al., 2017) and machine translation (Platanios et al., 2018). It also relates to the meta-learning set-up (Thrun and Pratt, 1998).

## 7 Conclusion

We presented a text generation model using exemplar-informed adaptive decoding. It reparameterizes the decoder using the information gathered from retrieved exemplars. We experimented with text summarization and data-to-text generation, and showed that the proposed model achieves strong performance and outperforms comparable baselines on both. The proposed model can be applicable in other conditioned text generation tasks. We release our implementation at [https://homes.cs.washington.edu/~hapeng](https://homes.cs.washington.edu/~hapeng).

## Acknowledgments

We thank Antonios Anastasopoulos, Ming-Wei Chang, Michael Collins, Jacob Devlin, Yichen Gong, Luheng He, Kenton Lee, Dianqi Li, Zhouhan Lin, Slav Petrov, Oscar Tackstrom, Kristina Toutanova, and other members of the Google AI language team for the helpful discussion, and the anonymous reviewers for their valuable feedback.

## References

* Al-Shedivat et al. (2017) Maruan Al-Shedivat, Avinava Dubey, and Eric P. Xing. 2017. Contextual explanation networks. _arXiv:1705.10301_.
* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In _Proc. of ICLR_.
* Becker (2002) Tilman Becker. 2002. Practical, template-based natural language generation with tag. In _Proceedings of the Sixth International Workshop on Tree Adjoining Grammar and Related Frameworks_.
* Bradbury et al. (2017) James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. 2017. Quasi-Recurrent Neural Network. In _Proc. of ICLR_.
* Cao et al. (2018a) Ziqiang Cao, Wenjie Li, Sujian Li, and Furu Wei. 2018a. Retrieve, rerank and rewrite: Soft template based neural summarization. In _Proc. of ACL_.
* Cao et al. (2018b) Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018b. Faithful to the original: Fact aware neural abstractive summarization. In _Proc. of AAAI_.
* Cho et al. (2014) Kyunghyun Cho, Bart Van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In _Proc. of EMNLP_.
* Durrett et al. (2016) Greg Durrett, Taylor Berg-Kirkpatrick, and Dan Klein. 2016. Learning-based single-document summarization with compression and anaphoricity constraints. In _Proc. of ACL_.
* Elman (1990) Jeffrey L. Elman. 1990. Finding structure in time. _Cognitive science_, 14(2):179-211.
* Fan et al. (2018) Angela Fan, David Grangier, and Michael Auli. 2018. Controllable abstractive summarization. In _Proceedings of the 2nd Workshop on Neural Machine Translation and Generation_.
* Fernando et al. (2016) Chrisantha Fernando, Dylan Banarse, Malcolm Reynolds, Frederic Besse, David Pfau, Max Jaderberg, Marc Lanctot, and Daan Wierstra. 2016. Convolution by evolution: Differentiable pattern producing networks. In _Proceedings of the Genetic and Evolutionary Computation Conference 2016_.
* Foster and White (2004) Mary Ellen Foster and Michael White. 2004. Techniques for text planning with xslt. In _Proceedings of the Workshop on NLP and XML: RDF/RDFS and OWL in Language Technology_.
* Gatt and Reiter (2009) Albert Gatt and Ehud Reiter. 2009. SimpleNLG: A realisation engine for practical applications. In _Proceedings of the 12th European Workshop on Natural Language Generation_.
* Gehrmann et al. (2018) Sebastian Gehrmann, Yuntian Deng, and Alexander M Rush. 2018. Bottom-up abstractive summarization. In _Proc. of EMNLP_.
* Graff et al. (2003) David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2003. English Gigaword Second Edition.
* Graves (2012) Alex Graves. 2012. _Supervised Sequence Labelling with Recurrent Neural Networks_, volume 385 of _Studies in Computational Intelligence_. Springer.
* Gu et al. (2016) Jiatao Gu, Zhengdong Lu, Hang Li, and Victor OK Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In _Proc. of ACL_.
* Gu et al. (2018) Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor OK Li. 2018. Search engine guided non-parametric neural machine translation. In _Proc. of AAAI_.
* Gou et al. (2018) Kelvin Gou, Tatsumori B Hashimoto, Yonatan Oren, and Percy Liang. 2018. Generating sentences by editing prototypes. _TACL_, 6:437-450.
* Ha et al. (2017) David Ha, Andrew Dai, and Quoc V Le. 2017. Hypernetworks. In _Proc. of ICLR_.
* He et al. (2015) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep residual learning for image recognition. In _Proc. of CVPR_ 

* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_, 9(8):1735-1780.
* Hodosh et al. (2013) Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013. Framing image description as a ranking task: Data, models and evaluation metrics. _JAIR_, 47(1):853-899.
* Jia et al. (2016) Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V Gool. 2016. Dynamic filter networks. In _Proc. of NeuralIPS_.
* Kingma and Ba (2015) Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _Proc. of ICLR_.
* Kingma et al. (2015) Durk P Kingma, Tim Salimans, and Max Welling. 2015. Variational dropout and the local reparameterization trick. In _Proc. of NeuralIPS_.
* Koehn (2009) Philipp Koehn. 2009. _Statistical machine translation_. Cambridge University Press.
* Koutnik et al. (2010) Jan Koutnik, Faustino Gomez, and Jurgen Schmidhuber. 2010. Evolving neural networks in compressed weight space. In _Proceedings of the Annual Conference on Genetic and Evolutionary Computation_.
* Krueger and Memisevic (2017) David Krueger and Roland Memisevic. 2017. Regularizing rnns by stabilizing activations. In _Proc. of ICLR_.
* Kukich (1983) Karen Kukich. 1983. Design of a knowledge-based report generator. In _Proc. of ACL_.
* Lebret et al. (2016) Remi Lebret, David Grangier, and Michael Auli. 2016. Neural text generation from structured data with application to the biography domain. In _Proc. of EMNLP_.
* Li et al. (2016) Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In _Proc. of NAACL_.
* Liang et al. (2009) Percy Liang, Michael I. Jordan, and Dan Klein. 2009. Learning semantic correspondences with less supervision. In _Proc. of ACL_.
* Lin (2004) Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In _Text Summarization Branches Out: Proceedings of the ACL-04 Workshop_.
* Lin et al. (2017) Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming-ting Sun. 2017. Adversarial ranking for language generation. In _Proc. of NeuralIPS_.
* Liu et al. (2017) Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2017. Dynamic compositional neural networks over tree structure. In _Proc. of IJCAI_.
* Liu et al. (2018) Tianyu Liu, Kexiang Wang, Lei Sha, Baobao Chang, and Zhifang Sui. 2018. Table-to-text generation by structure-aware seq2seq learning. In _Proc. of AAAI_.
* Luong et al. (2015) Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In _Proc. of EMNLP_.
* Mani (1999) Inderjeet Mani. 1999. _Advances in automatic text summarization_. MIT press.
* Mason and Charniak (2014) Rebecca Mason and Eugene Charniak. 2014. Domain-specific image captioning. In _Proc. of CoNLL_.
* McKeown (1992) Kathleen McKeown. 1992. _Text generation_. Cambridge University Press.
* Moczulski et al. (2015) Marcin Moczulski, Misha Denil, Jeremy Appleyard, and Nando de Freitas. 2015. ACDC: A structured efficient linear layer. _arXiv:1511.05946_.
* Napoles et al. (2012) Courtney Napoles, Matthew Gormley, and Benjamin Van Durme. 2012. Annotated gigaword. In _Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction_.
* Pandey et al. (2018) Gaurav Pandey, Danish Contractor, Vineet Kumar, and Sachindra Joshi. 2018. Exemplar encoder-decoder for neural conversation generation. In _Proc. of ACL_.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proc. of ACL_.
* Paulus et al. (2018) Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In _Proc. of ICLR_.
* Peng et al. (2018a) Hao Peng, Roy Schwartz, Sam Thomson, and Noah A. Smith. 2018a. Rational recurrences. In _In Proc. of EMNLP_.
* Peng et al. (2017) Hao Peng, Sam Thomson, and Noah A. Smith. 2017. Deep multitask learning for semantic dependency parsing. In _Proc. of ACL_.
* Peng et al. (2018b) Hao Peng, Sam Thomson, and Noah A. Smith. 2018b. Backpropagating through structured argmax using a spigot. In _Proc. of ACL_.
* Peng et al. (2018c) Hao Peng, Sam Thomson, Swabha Swayamdipta, and Noah A. Smith. 2018c. Learning joint semantic parsers from disjoint data. In _Proc. of NAACL_.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In _Proc. of EMNLP_.
* Platanios et al. (2018) Emmanouil Antonios Platanios, Mrinnaya Sachan, Graham Neubig, and Tom Mitchell. 2018. Contextual parameter generation for universal neural machine translation. In _Proc. of EMNLP_.
* Press and Wolf (2017) Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In _Proc. of EACL_.
* Reiter and Dale (1997) Ehud Reiter and Robert Dale. 1997. Building applied natural language generation systems. _Natural Language Engineering_, 3(1):57-87.

 

Ehud Reiter, Somayajulu Sripada, Jim Hunter, Jin Yu, and Ian Davy. 2005. Choosing words in computer-generated weather forecasts. _Artificial Intelligence_, 167(1-2):137-169.
* Rush et al. (2015) Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In _Proc. of EMNLP_.
* Sandaus (2008) Evan Sandaus. 2008. _The New York Times Annotated Corpus_. LDC corpora. Linguistic Data Consortium.
* Schuster and Paliwal (1997) M. Schuster and K.K. Paliwal. 1997. Bidirectional recurrent neural networks. _Transactions on Signal Proccesing_, 45(11):2673-2681.
* See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In _Proc. of ACL_.
* Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In _Proc. of ACL_.
* Song et al. (2016) Yiping Song, Rui Yan, Xiang Li, Dongyan Zhao, and Ming Zhang. 2016. Two are better than one: An ensemble of retrieval- and generation-based dialog systems. _arXiv:1610.07149_.
* Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. _JMLR_, 15(1):1929-1958.
* Stanley et al. (2009) K. O. Stanley, D. B. D'Ambrosio, and J. Gauci. 2009. A hypercube-based encoding for evolving large-scale neural networks. _Artificial Life_, 15(2):185-212.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In _Proc. of NeuralIPS_.
* Thrun and Pratt (1998) Sebastian Thrun and Lorien Pratt, editors. 1998. _Learning to Learn_. Kluwer Academic Publishers.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Proc. of NeuralIPS_.
* Vinyals and Le (2015) Oriol Vinyals and Quoc Le. 2015. A neural conversational model. In _Proc. of ICML_.
* Weston et al. (2018) Jason Weston, Emily Dinan, and Alexander Miller. 2018. Retrieve and refine: Improved sequence generation models for dialogue. In _Proceedings of the International Workshop on Search-Oriented Conversational AI_.
* Wiseman et al. (2017) Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges in data-to-document generation. In _Proc. of EMNLP_.
* Wiseman et al. (2018) Sam Wiseman, Stuart M Shieber, and Alexander M Rush. 2018. Learning neural templates for text generation. In _Proc. of EMNLP_.
* Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Kriktun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. _arXiv:1609.08144_.