Variational Template Machine for Data-to-Text Generation

1 INTRODUCTION . Generating text descriptions from structured data ( data-to-text ) is an important task with many practical applications . Data-to-text has been used to generate different kinds of texts , such as weather reports ( Angeli et al. , 2010 ) , sports news ( Mei et al. , 2016 ; Wiseman et al. , 2017 ) and biographies ( Lebret et al. , 2016 ; Wang et al. , 2018b ; Chisholm et al. , 2017 ) . Figure 1 gives an example of data-to-text task , which takes an infobox 1 as the input and outputs a brief description of the information in the table . There are several recent methods utilizing neural encoder-decoder frameworks to generate text description from data tables ( Lebret et al. , 2016 ; Bao et al. , 2018 ; Chisholm et al. , 2017 ; Liu et al. , 2018 ) . Although current table-to-text models could generate high quality sentences , the diversity of these output sentences are not satisfactory . We find that templates are crucial in increasing the variations of sentence structure . For example , Table 1 gives three descriptions with their templates for the given table input . Different templates control the sentence arrangement , thus vary the generation . Some related work ( Wiseman et al. , 2018 ; Dou et al. , 2018 ) employs hidden semi-Markov hidden model to extract templates from table-text pairs . We argue that templates can be better considered for generating more diverse outputs . First , it is non-trivial to sample different templates for obtaining different output utterances . Directly adopting variational auto-encoders ( VAEs , Kingma & Welling ( 2014 ) ) in table-to-text only enables to sample in the latent space . However , VAEs always generate irrelevant outputs , which may change the table content instead of sampling templates . This may harm the quality of output sentences . To address the above problem , if we can directly sample in the template space , we may get more diverse outputs while keeping the good quality of output sentences . ∗Work done while Rong Ye was a research intern at ByteDance AI Lab . 1An infobox is a table containing attribute-value data about a certain subject . It is mostly used on Wikipedia pages . Second , we can hardly obtain promising sentences by sampling in the template space , if the template space is less informative . Namely , either encoder-decoder models or VAE-based models requires abundant parallel table-text pairs during the training . In such case , constructing high-quality parallel dataset is often labor-intensive . With limited table-sentence pairs , a VAE model can not construct an informative template space . How to fully utilize raw sentences ( without aligned table ) to enrich the latent template space is under study . In this paper , to address the above two problems , we propose the variational template machine ( VTM ) for data-to-text generation , which enables to generate sentences with diverse templates while preserving the high quality . Particularly , we introduce two latent variables , representing template and content , to control the generation . The two latent variables are disentangled , and thus we can generate diverse outputs by directly sampling in the latent space for template . Moreover , we propose a novel approach for semi-supervised learning in the VAE framework , which could fully exploit the raw sentences for enriching the template space . Inspired by back-translation ( Sennrich et al. , 2016 ; Burlot & Yvon , 2018 ; Artetxe et al. , 2018 ) , we design a variational back-translation process . Instead of training a sentence-to-table backward generation model directly , we take the variational posterior of the content latent variable as the backward model to help to train the forward generative model . Auxiliary losses are introduced to ensure the learning of meaningful and disentangled latent variables . Experimental results on Wikipedia biography dataset ( Lebret et al. , 2016 ) and sentence planning NLG dataset ( Reed et al. , 2018 ) show that our model can generate texts with more diversity while keeping a good fluency . Training together with a large amount of raw text , VTM can further improve the generation performance . Besides , VTM is more predominant in the case where sentence-to-table backward model is hard to train . Ablation studies also demonstrate the effects of the auxiliary losses on the disentanglement of template and content spaces . 2 PROBLEM FORMULATION AND NOTATIONS . As a data-to-text task , we have table-text pairs Dp = { ( xi , yi ) } Ni=1 , where xi is the table , and yi is the output sentence . Following the description scheme of Lebret et al . ( 2016 ) , a table x can be viewed as a set of K records of field-position-value triples , i.e. , x = { ( f , p , v ) i } Ki=1 , where f is the field and p is the index of value v in the field f . For example , an item “ Name : John Lennon ” is denoted as two corresponding records : ( Name , 1 , John ) and ( Name , 2 , Lennon ) . For each triple , we first embed field , position and value as d-dim vectors ep , ef , ev ∈ Rd . Then , the dt-dim representation of the record is obtained by hi = tanh ( W [ ef , ep , ev ] T +b ) , i = 1 ... K , whereW ∈ Rdt×3d and b ∈ Rdt are parameters . The final representation of the table , denoted as fenc ( x ) , is obtained by max-pooling over all field-position-value triple records , fenc ( x ) = h = MaxPooli { hi ; i = 1 ... K } . In addition to the table-text pairs , we also have raw texts without table input , denoted as Dr = { yi } Mi=1 . It usually has M N . 3 VARIATIONAL TEMPLATE MACHINE . As shown in the graphical model in Figure 2 , our VTM modifies the vanilla VAE model by introducing two independent latent variables z and c , representing template latent variable and content latent variable respectively . c models the content information in the table , while z models the sentence template information . Target sentence y is generated by both content and template variables . The two latent variables are disentangled , which makes it possible to generate diverse and relevant sentences by sampling template variable and retraining the content variable . Considering pairwise and raw data presented in Figure 1 , their generation process for the content latent variable c is different . • For a given table-text pair ( x , y ) ∈ Dp , the content is observable from table x . As a result , c is assumed to be deterministic given table x , whose prior is defined as a delta distribution p ( c|x ) = δ ( c = fenc ( x ) ) . The marginal log-likelihood is : log pθ ( y|x ) = log ∫ z ∫ c pθ ( y|x , z , c ) p ( z ) p ( c|x ) dcdz = log ∫ z pθ ( y|x , z , c = fenc ( x ) ) p ( z ) dz , ( x , y ) ∈ Dp . ( 1 ) • For raw text y ∈ Dn , the content is unobservable with the absence of table x . As a result , the content latent variable c should be sampled from prior of Gaussian distribution N ( 0 , I ) . The marginal log-likelihood is : log pθ ( y ) = log ∫ z ∫ c pθ ( y|z , c ) p ( z ) p ( c ) dcdz , y ∈ Dr. ( 2 ) In order to make full use of both table-text pair data and raw text data , the above marginal loglikelihood should be optimized jointly : L ( θ ) = E ( x , y ) ∼Dp [ log pθ ( y|x ) ] + Ey∼Dr [ log pθ ( y ) ] . ( 3 ) Directly optimizing Equation 3 is intractable . Following the idea of variational inference ( Kingma & Welling , 2014 ) , a variational posterior qφ ( · ) is constructed as an inference model ( dashed lines in Figure 2 ) to approximate the true posterior . Instead of optimizing the marginal log-likelihood in Equation 3 , we maximize the evidence lower bound ( ELBO ) . In Section 3.1 and 3.2 , the ELBO of table-text pairwise data and raw text data are discussed , respectively . 3.1 LEARNING FROM TABLE-TEXT PAIR DATA . In this section , we will show the learning loss of table-text pair data . According to the aforementioned assumption , the content variable c is observable and follows a delta distribution centred in the hidden representation of the table x. ELBO objective . Assuming that the template variable z only relies on the template of target sentence , we introduce qφ ( z|y ) as an approximation of the true posterior p ( z|y , c , x ) , The ELBO loss of Equation 1 is written as LELBOp ( x , y ) = −Eqφz ( z|y ) log pθ ( y|z , c = fenc ( x ) , x ) +DKL ( qφz ( z|y ) ‖p ( z ) ) , ( x , y ) ∈ Dp . The variational posterior qφz ( z|y ) is assumed as a multivariate Gaussian distribution N ( µφz ( y ) , Σφz ( y ) ) , while the prior p ( z ) is taken as a normal distribution N ( 0 , I ) . Preserving-Template Loss . Without any supervision , the ELBO loss alone does not guarantee to learn a good template representation space . Inspired by the work in style-transfer ( Hu et al. , 2017b ; Shen et al. , 2017 ; Bao et al. , 2019 ; John et al. , 2018 ) , an auxiliary loss is introduced to embed the template information of sentences into template variable z . With table , we are able to roughly align the tokens in sentence with the records in the table . By replacing these tokens with a special token < ent > , we can remove the content information from sentences and get the sketchy sentence template , denote as ỹ . We introduce the preserving-template loss Lpt to ensure that the latent variable z only contains the information of the template . Lpt ( x , y , ỹ ) = −Eqφz ( z|y ) log pη ( ỹ|z ) = −Eqφz ( z|y ) m∑ t=1 log pη ( ỹt|z , ỹ < t ) where m is the length of the ỹ , and η denotes the parameters of the extra template generator . Lpt is trained via parallel data . In practice , due to the insufficient amount of parallel data , template generator pη may not be well-learned . However , experimental results show that this loss is sufficient to provide a guidance for learning a template space . 3.2 LEARNING FROM RAW TEXT DATA . Our model is able to make use of a large number of raw data without table since the content information of table could be obtained by the content latent variable . ELBO objective . According to the definition of generative model in Equation 2 , the ELBO of raw text data is log pθ ( y ) = Eqφ ( z , c|y ) log pθ ( y , z , c ) qφ ( z , c|y ) , y ∈ Dr. With the mean field approximation ( Xing et al. , 2003 ) , qφ ( z , c|x ) can be factorized as : qφ ( z , c|y ) = qφz ( z|y ) qφc ( c|y ) . We have : LELBOr ( y ) =− Eqφz ( z|y ) qφc ( c|y ) log pθ ( y|z , c ) +DKL ( qφz ( z|y ) ||p ( z ) ) +DKL ( qφc ( c|y ) ||p ( c ) ) , y ∈ Dr . In order to make use of template information contained in raw text data effectively , the parameters of generation network pθ ( y|z , c ) and posterior network qφz ( z|y ) are shared for pairwise and raw data . In decoding process , for raw text data , we use content variable c as the table embedding for the missing of table x. Variational posterior for c is deployed as another multivariate Guassian qφc ( c|y ) = N ( µφc ( y ) , Σφc ( y ) ) . Both p ( z ) and p ( c ) are taken as normal distribution N ( 0 , I ) . Preserving-Content Loss . In order to make the posterior qφc ( c|y ) correctly infers the content information , the table-text pairs are used as the supervision to train the recognition network of qφc ( c|y ) . To this end , we add a preserving-content loss Lpc ( x , y ) = −Eqφc ( c|y ) ‖c− h‖ 2 +DKL ( qφc ( c|y ) ||p ( c ) ) , ( x , y ) ∈ Dp , where h = fenc ( x ) is the embedding of table obtained by the table encoder . Minimizing Lpc is also helpful to bridge the gap of c between pairwise ( taking c = h ) and raw training data ( sampling from qφ ( c|y ) ) . Moreover , we find that the first term of Lpc is equivalent to ( 1 ) make the mean of qφ ( c|y ) closer to h ; ( 2 ) minimize the trace of co-variance of qφ ( c|y ) . The second term serves as a regularization . Detailed explanations and proof are referred in supplementary materials . Algorithm 1 Training procedure Input : Model parameters φz , φc , θ , η Table-text pair data Dp = { ( x , y ) i } Ni=1 ; raw text data Dr = { yj } Mj=1 ; M N Procedure TRAIN ( Dp , Dr ) : 1 : Update φz , φc , θ , η by gradient descent on LELBOp + LMI + Lpt + Lpc 2 : Update φz , φc , θ by gradient descent on LELBOr + LMI 3 : Update φz , φc , θ , η by gradient descent on Ltot

Summary: This paper proposes Variational Template Machine (VTM), a generative model to generate textual descriptions from structured data (i.e., tables). VTM is derived from the variational autoencoder, where the input is a row entry from a table and the output is the text associated with this entry. The authors introduce two latent variables to model contents and templates. The content variable is conditioned on the table entry, and generates the textual output together with the template variable. The model is trained on both paired table-to-text examples as well as unpaired (text only) examples. Experiments on the Wiki and SpNLG datasets show that models generate diverse sentences, and the overall performance in terms of BLEU is only slightly below the best baseline Table2Seq model that does not generate diverse sentences. The results also show that additional losses for preserving contents and templates introduced by the authors play an important role in the overall model performance.