A core step in statistical data-to-text generation concerns learning correspondences between structured data representations (e.g., facts in a database) and paired texts BIBREF0 , BIBREF1 , BIBREF2 . These correspondences describe how data representations are expressed in natural language (content realisation) but also indicate which subset of the data is verbalised in the text (content selection).Although content selection is traditionally performed by domain experts, recent advances in generation using neural networks BIBREF3 , BIBREF4 have led to the use of large scale datasets containing loosely related data and text pairs. A prime example are online data sources like DBPedia BIBREF5 and Wikipedia and their associated texts which are often independently edited. Another example are sports databases and related textual resources. Wiseman et al. wiseman-shieber-rush:2017:EMNLP2017 recently define a generation task relating statistics of basketball games with commentaries and a blog written by fans.In this paper, we focus on short text generation from such loosely aligned data-text resources. We work with the biographical subset of the DBPedia and Wikipedia resources where the data corresponds to DBPedia facts and texts are Wikipedia abstracts about people. Figure 1 shows an example for the film-maker Robert Flaherty, the Wikipedia infobox, and the corresponding abstract. We wish to bootstrap a data-to-text generator that learns to verbalise properties about an entity from a loosely related example text. Given the set of properties in Figure ( 1 a) and the related text in Figure ( 1 b), we want to learn verbalisations for those properties that are mentioned in the text and produce a short description like the one in Figure ( 1 c).In common with previous work BIBREF6 , BIBREF7 , BIBREF8 our model draws on insights from neural machine translation BIBREF3 , BIBREF9 using an encoder-decoder architecture as its backbone. BIBREF7 introduce the task of generating biographies from Wikipedia data, however they focus on single sentence generation. We generalize the task to multi-sentence text, and highlight the limitations of the standard attention mechanism which is often used as a proxy for content selection. When exposed to sub-sequences that do not correspond to any facts in the input, the soft attention mechanism will still try to justify the sequence and somehow distribute the attention weights over the input representation BIBREF10 . The decoder will still memorise high frequency sub-sequences in spite of these not being supported by any facts in the input.We propose to alleviate these shortcomings via a specific content selection mechanism based on multi-instance learning (MIL; BIBREF11 , BIBREF11 ) which automatically discovers correspondences, namely alignments, between data and text pairs. These alignments are then used to modify the generation function during training. We experiment with two frameworks that allow to incorporate alignment information, namely multi-task learning (MTL; BIBREF12 , BIBREF12 ) and reinforcement learning (RL; BIBREF13 , BIBREF13 ). In both cases we define novel objective functions using the learnt alignments. Experimental results using automatic and human-based evaluation show that models trained with content-specific objectives improve upon vanilla encoder-decoder architectures which rely solely on soft attention.The remainder of this paper is organised as follows. We discuss related work in Section "Related Work" and describe the MIL-based content selection approach in Section "Bidirectional Content Selection" . We explain how the generator is trained in Section "Generator Training" and present evaluation experiments in Section "Experimental Setup" . Section "Conclusions" concludes the paper.Previous attempts to exploit loosely aligned data and text corpora have mostly focused on extracting verbalisation spans for data units. Most approaches work in two stages: initially, data units are aligned with sentences from related corpora using some heuristics and subsequently extra content is discarded in order to retain only text spans verbalising the data. belz2010extracting obtain verbalisation spans using a measure of strength of association between data units and words, walter2013corpus extract textual patterns from paths in dependency trees while mrabet:webnlg16 rely on crowd-sourcing. Perez-Beltrachini and Gardent perezbeltrachini-gardent:2016:*SEM learn shared representations for data units and sentences reduced to subject-predicate-object triples with the aim of extracting verbalisations for knowledge base properties. Our work takes a step further, we not only induce data-to-text alignments but also learn generators that produce short texts verbalising a set of facts.Our work is closest to recent neural network models which learn generators from independently edited data and text resources. Most previous work BIBREF7 , BIBREF14 , BIBREF15 , BIBREF16 targets the generation of single sentence biographies from Wikipedia infoboxes, while wiseman-shieber-rush:2017:EMNLP2017 generate game summary documents from a database of basketball games where the input is always the same set of table fields. In contrast, in our scenario, the input data varies from one entity (e.g., athlete) to another (e.g., scientist) and properties might be present or not due to data incompleteness. Moreover, our generator is enhanced with a content selection mechanism based on multi-instance learning. MIL-based techniques have been previously applied to a variety of problems including image retrieval BIBREF17 , BIBREF18 , object detection BIBREF19 , BIBREF20 , text classification BIBREF21 , image captioning BIBREF22 , BIBREF23 , paraphrase detection BIBREF24 , and information extraction BIBREF25 . The application of MIL to content selection is novel to our knowledge.We show how to incorporate content selection into encoder-decoder architectures following training regimes based on multi-task learning and reinforcement learning. Multi-task learning aims to improve a main task by incorporating joint learning of one or more related auxiliary tasks. It has been applied with success to a variety of sequence-prediction tasks focusing mostly on morphosyntax. Examples include chunking, tagging BIBREF26 , BIBREF27 , BIBREF28 , BIBREF29 , name error detection BIBREF30 , and machine translation BIBREF31 . Reinforcement learning BIBREF13 has also seen popularity as a means of training neural networks to directly optimize a task-specific metric BIBREF4 or to inject task-specific knowledge BIBREF32 . We are not aware of any work that compares the two training methods directly. Furthermore, our reinforcement learning-based algorithm differs from previous text generation approaches BIBREF4 , BIBREF32 in that it is applied to documents rather than individual sentences.We consider loosely coupled data and text pairs where the data component is a set ${{\cal P}}$ of property-values $\lbrace p_1:v_1, \cdots , p_{|{{\cal P}}|}:v_{|{{\cal P}}|}\rbrace $ and the related text ${{\cal T}}$ is a sequence of sentences $(s_1, \cdots , s_{|{\cal T}|})$ . We define a mention span $\tau $ as a (possibly discontinuous) subsequence of ${{\cal T}}$ containing one or several words that verbalise one or more property-value from ${{\cal P}}$ . For instance, in Figure 1 , the mention span “married to Frances H. Flaherty” verbalises the property-value $\lbrace Spouse(s)
: Frances \; Johnson \; Hubbard\rbrace $ .In traditional supervised data to text generation tasks, data units (e.g., $p_i:v_i$ in our particular setting) are either covered by some mention span $\tau _j$ or do not have any mention span at all in ${{\cal T}}$ . The latter is a case of content selection where the generator will learn which properties to ignore when generating text from such data. In this work, we consider text components which are independently edited, and will unavoidably contain unaligned spans, i.e., text segments which do not correspond to any property-value in ${{\cal P}}$ . The phrase “from 1914” in the text in Figure ( 1 b) is such an example. Similarly, the last sentence, talks about Frances' awards and nominations and this information is not supported by the properties either.Our model checks content in both directions; it identifies which properties have a corresponding text span (data selection) and also foregrounds (un)aligned text spans (text selection). This knowledge is then used to discourage the generator from producing text not supported by facts in the property set ${{\cal P}}$ . We view a property set ${{\cal P}}$ and its loosely coupled text ${{\cal T}}$ as a coarse level, imperfect alignment. From this alignment signal, we want to discover a set of finer grained alignments indicating which mention spans in ${{\cal T}}$ align to which properties in ${{\cal P}}$ . For each pair $({\cal P},
{\cal T})$ , we learn an alignment set ${\cal A}({\cal P}, {\cal T})$ which contains property-value word pairs. For example, for the properties $spouse$ and $died$ in Figure 1 , we would like to derive the alignments in Table 1 .We formulate the task of discovering finer-grained word alignments as a multi-instance learning problem BIBREF11 . We assume that words from the text are positive labels for some property-values but we do not know which ones. For each data-text pair $({{\cal P}},
{{\cal T}})$ , we derive $|{\cal T}|$ pairs of the form $({{\cal P}},s)$ where $|{\cal T}|$ is the number of sentences in ${\cal T}$ . We encode property sets ${{\cal P}}$ and sentences $s$ into a common multi-modal $h$ -dimensional embedding space. While doing this, we discover finer grained alignments between words and property-values. The intuition is that by learning a high similarity score for a property set ${{\cal P}}$ and sentence pair $s$ , we will also learn the contribution of individual elements (i.e., words and property-values) to the overall similarity score. We will then use this individual contribution as a measure of word and property-value alignment. More concretely, we assume the pair is aligned (or unaligned) if this individual score is above (or below) a given threshold. Across examples like the one shown in Figure ( 1 a-b), we expect the model to learn an alignment between the text span “married to Frances H. Flaherty” and the property-value $|{\cal T}|$0 .In what follows we describe how we encode $({{\cal P}}, s)$ pairs and define the similarity function.In this section we describe the base generation architecture and explain two alternative ways of using the alignments to guide the training of the model. One approach follows multi-task training where the generator learns to output a sequence of words but also to predict alignment labels for each word. The second approach relies on reinforcement learning for adjusting the probability distribution of word sequences learnt by a standard word prediction training algorithm.We follow a standard attention based encoder-decoder architecture for our generator BIBREF3 , BIBREF33 . Given a set of properties $X$ as input, the model learns to predict an output word sequence $Y$ which is a verbalisation of (part of) the input. More precisely, the generation of sequence $Y$ is conditioned on input $X$ :$$P(Y|X) = \prod _{t=1}^{|Y|} P(y_t|y_{1:t-1}, X)$$   (Eq. 12)  The encoder module constitutes an intermediate representation of the input. For this, we use the property-set encoder described in Section "Bidirectional Content Selection" which outputs vector representations $\lbrace  \mathbf {p}_1, \cdots , \mathbf {p}_{|X|} \rbrace $ for a set of property-value pairs. The decoder uses an LSTM and a soft attention mechanism BIBREF33 to generate one word $y_t$ at a time conditioned on the previous output words and a context vector $c_t$ dynamically created:$$P(y_{t+1}|y_{1:t},X) = softmax(g(\mathbf {h}_t, c_t))$$   (Eq. 13) where $g(\cdot )$ is a neural network with one hidden layer parametrised by $\mathbf {W}_o \in \mathbb {R}^{|V| \times d}$ , $|V|$ is the output vocabulary size and $d$ the hidden unit dimension, over $\mathbf {h}_t$ and $c_t$ composed as follows:$$g(\mathbf {h}_t, c_t) = \mathbf {W}_o \; tanh(\mathbf {W}_c [ c_t ; \mathbf {h}_t ] )$$   (Eq. 14) where $\mathbf {W}_c \in \mathbb {R}^{d \times 2d}$ . $\mathbf {h}_t$ is the hidden state of the LSTM decoder which summarises $y_{1:t}$ :$$\mathbf {h}_t = \text{LSTM}(y_t, \mathbf {h}_{t-1})$$   (Eq. 15) The dynamic context vector $c_t$ is the weighted sum of the hidden states of the input property set (Equation ( 16 )); and the weights $\alpha _{ti}$ are determined by a dot product attention mechanism:$$c_t = \sum _{i=1}^{|X|}\alpha _{ti} \, \mathbf {p}_i$$   (Eq. 16) $$\alpha _{ti} = {\text{exp}(\mathbf {h}_{t} \, \mathchoice{\mathbin {\hbox{\scalebox {.5}{$\m@th \displaystyle \bullet $}}}}{}{}{}}{\mathbin {\hbox{\scalebox {.5}{$\m@th \textstyle \bullet $}}}}$$   (Eq. 17)  pi)i exp(ht pi )We initialise the decoder with the averaged sum of the encoded input representations BIBREF34 . The model is trained to optimize negative log likelihood:$${\cal L}_{wNLL} = - \sum _{t=1}^{|Y|} log \, P(y_t|y_{1:t-1}, X)$$   (Eq. 18) We extend this architecture to multi-sentence texts in a way similar to wiseman-shieber-rush:2017:EMNLP2017. We view the abstract as a single sequence, i.e., all sentences are concatenated. When training, we cut the abstracts in blocks of equal size and perform forward backward iterations for each block (this includes the back-propagation through the encoder). From one block iteration to the next, we initialise the decoder with the last state of the previous block. The block size is a hyperparameter tuned experimentally on the development set.The generation of the output sequence is conditioned on the previous words and the input. However, when certain sequences are very common, the language modelling conditional probability will prevail over the input conditioning. For instance, the phrase from 1914 in our running example is very common in contexts that talk about periods of marriage or club membership, and as a result, the language model will output this phrase often, even in cases where there are no supporting facts in the input. The intuition behind multi-task training BIBREF12 is that it will smooth the probabilities of frequent sequences when trying to simultaneously predict alignment labels.Using the set of alignments obtained by our content selection model, we associate each word in the training data with a binary label $a_t$ indicating whether it aligns with some property in the input set. Our auxiliary task is to predict $a_t$ given the sequence of previously predicted words and input $X$ :$$P(a_{t+1}|y_{1:t},X) = sigmoid(g^{\prime }(\mathbf {h}_t, c_t))$$   (Eq. 20) $$g^{\prime }(\mathbf {h}_t, c_t) = \mathbf {v}_a \, \mathchoice{\mathbin {\hbox{\scalebox {.5}{$\m@th \displaystyle \bullet $}}}}{}{}{}$$   (Eq. 21)  tanh(Wc [ ct ; ht ] )where $\mathbf {v}_a \in \mathbb {R}^{d}$ and the other operands are as defined in Equation ( 14 ). We optimise the following auxiliary objective function:$${\cal L}_{aln} = - \sum _{t=1}^{|Y|} log \, P(a_t|y_{1:t-1}, X)$$   (Eq. 22) and the combined multi-task objective is the weighted sum of both word prediction and alignment prediction losses:$${\cal L}_{MTL} = \lambda \, {\cal L}_{wNLL} + (1 - \lambda ) \, {\cal L}_{aln}$$   (Eq. 23) where $\lambda $ controls how much model training will focus on each task. As we will explain in Section "Experimental Setup" , we can anneal this value during training in favour of one objective or the other.Although the multi-task approach aims to smooth the target distribution, the training process is still driven by the imperfect target text. In other words, at each time step $t$ the algorithm feeds the previous word $w_{t-1}$ of the target text and evaluates the prediction against the target $w_t$ .Alternatively, we propose a training approach based on reinforcement learning ( BIBREF13 ) which allows us to define an objective function that does not fully rely on the target text but rather on a revised version of it. In our case, the set of alignments obtained by our content selection model provides a revision for the target text. The advantages of reinforcement learning are twofold: (a) it allows to exploit additional task-specific knowledge BIBREF32 during training, and (b) enables the exploration of other word sequences through sampling. Our setting differs from previous applications of RL BIBREF4 , BIBREF32 in that the reward function is not computed on the target text but rather on its alignments with the input.The encoder-decoder model is viewed as an agent whose action space is defined by the set of words in the target vocabulary. At each time step, the encoder-decoder takes action $\hat{y}_t$ with policy $P_{\pi }(\hat{y}_t|\hat{y}_{1:t-1}, X)$ defined by the probability in Equation ( 13 ). The agent terminates when it emits the End Of Sequence (EOS) token, at which point the sequence of all actions taken yields the output sequence $\hat{Y}=(\hat{y}_1, \cdots ,
\hat{y}_{|\hat{Y}|})$ . This sequence in our task is a short text describing the properties of a given entity. After producing the sequence of actions $\hat{Y}$ , the agent receives a reward $r(\hat{Y})$ and the policy is updated according to this reward.We define the reward function $r(\hat{Y})$ on the alignment set ${\cal A}(X,Y)$ . If the output action sequence $\hat{Y}$ is precise with respect to the set of alignments ${\cal A}(X,Y)$ , the agent will receive a high reward. Concretely, we define $r(\hat{Y})$ as follows:$$r(\hat{Y}) = \gamma ^{pr} \, r^{pr}(\hat{Y})$$   (Eq. 26) where $\gamma ^{pr}$ adjusts the reward value $r^{pr}$ which is the unigram precision of the predicted sequence $\hat{Y}$ and the set of words in ${\cal A}(X,Y)$ .We use the REINFORCE algorithm BIBREF13 to learn an agent that maximises the reward function. As this is a gradient descent method, the training loss of a sequence is defined as the negative expected reward:$${\cal L}_{RL} = -\mathbb {E}_{(\hat{y}_1, \cdots , \hat{y}_{|\hat{Y}|})} \sim P_\pi (\text{·}|X)[r(\hat{y}_1, \cdots ,
\hat{y}_{|\hat{Y}|})] \nonumber $$   (Eq. 28) where $P_\pi $ is the agent's policy, i.e., the word distribution produced by the encoder-decoder model (Equation ( 13 )) and $r(\text{·})$ is the reward function as defined in Equation ( 26 ). The gradient of ${\cal L}_{RL}$ is given by:$$\nabla {\cal L}_{RL} \approx \sum ^{|\hat{Y}|}_{t=1}\nabla \,
\text{log} \, P_{\pi }(\hat{y}_t|\hat{y}_{1:t-1},
X)[r(\hat{y}_{1:|\hat{Y}|})-b_t] \nonumber $$   (Eq. 29) where $b_t$ is a baseline linear regression model used to reduce the variance of the gradients during training. $b_t$ predicts the future reward and is trained by minimizing mean squared error. The input to this predictor is the agent hidden state $\mathbf {h}_t$ , however we do not back-propagate the error to $\mathbf {h}_t$ . We refer the interested reader to BIBREF13 and BIBREF4 for more details.Rather than starting from a state given by a random policy, we initialise the agent with a policy learnt by pre-training with the negative log-likelihood objective BIBREF4 , BIBREF32 . The reinforcement learning objective is applied gradually in combination with the log-likelihood objective on each target block subsequence. Recall from Section "Encoder-Decoder Base Generator" that our document is segmented into blocks of equal size during training which we denote as MaxBlock. When training begins, only the last $\mho $ tokens are predicted by the agent while for the first $(\text{{\small \textsc {MaxBlock}}}-\mho )$ we still use the negative log-likelihood objective. The number of tokens $\mho $ predicted by the agent is incremented by $\mho $ units every 2 epochs. We set $\mho =3$ and the training ends when $(\text{{\small \textsc {MaxBlock}}}-\mho )=0$ . Since we evaluate the model's predictions at the block level, the reward function is also evaluated at the block level.We compared the performance of an encoder-decoder model trained with the standard negative log-likelihood method (ED), against a model trained with multi-task learning (ED $_{\mathrm {MTL}}$ ) and reinforcement learning (ED $_{\mathrm {RL}}$ ). We also included a template baseline system (Templ) in our evaluation experiments.The template generator used hand-written rules to realise property-value pairs. As an approximation for content selection, we obtained the 50 more frequent property names from the training set and manually defined content ordering rules with the following criteria. We ordered personal life properties (e.g., $birth\_date$ or $occupation$ ) based on their most common order of mention in the Wikipedia abstracts. Profession dependent properties (e.g., $position$ or $genre$ ), were assigned an equal ordering but posterior to the personal properties. We manually lexicalised properties into single sentence templates to be concatenated to produce the final text. The template for the property $position$ and example verbalisation for the property-value ${position : defender}$ of the entity zanetti are “ $[$ NAME $]$ played as $[$ POSITION $]$ .” and “ Zanetti played as defender.” respectively.In this paper we focused on the task of bootstrapping generators from large-scale datasets consisting of DBPedia facts and related Wikipedia biography abstracts. We proposed to equip standard encoder-decoder models with an additional content selection mechanism based on multi-instance learning and developed two training regimes, one based on multi-task learning and the other on reinforcement learning. Overall, we find that the proposed content selection mechanism improves the accuracy and fluency of the generated texts. In the future, it would be interesting to investigate a more sophisticated representation of the input BIBREF34 . It would also make sense for the model to decode hierarchically, taking sequences of words and sentences into account BIBREF41 , BIBREF42 .We thank the NAACL reviewers for their constructive feedback. We also thank Xingxing Zhang, Li Dong and Stefanos Angelidis for useful discussions about implementation details. We gratefully acknowledge the financial support of the European Research Council (award number 681760).

Q: What is the multi-instance learning?