SlowGuess/ACL_2020_OCR huggingface dataset:
Architecture and Hyperparameters For all tasks, we use an encoder-decoder LSTM for the generative model. We follow recent state-of-the-art works in parametrizing our encoder, and we use copy attention and dual attention (Gu et al., 2016; Gulcehre et al., 2016; Liu et al., 2018): full model architectures are given in the supplement.
The inference network scores are computed using a BiLSTM. We compute the emission scores  $\phi_{(e)}$  using span embeddings (Wang and Chang, 2016; Kitaev and Klein, 2018; Stern et al., 2017); transition scores  $\phi_{(t)}$  by dot product between embedding vectors for the class labels; lengths  $\phi_{(l)}$  is kept uniform, as in Wiseman et al. (2018). Additional details are in the supplement.
At training time, we use a rate for alleviating posterior collapse in the ELBO: warm-up the ELBO objective by linearly annealing the coefficient on the term  $\sum_{t=1}^{T} \log p_{\theta}(z_t \mid z_{<t}, y_{<t})$  and  $\mathrm{H}[q_{\phi}(z \mid x, y)]$  from 0 to 1, as implemented in Kim et al. (2019). We use the REINFORCE algorithm to do Monte Carlo estimation of the stochastic gradient. We choose the control variate to be the mean of the samples (Mnih and Rezende, 2016).
At decoding time, we only use the generative model. We use beam search with length normaliza
tion to jointly generate both the control states and the sentences. To obtain controlled generation, we observe the control states, and apply constrained beam search to  $p(y \mid x, z)$ .
Baselines For generation on E2E, we compare externally against 4 systems: E2E-BENCHMARK (Dusek and Jurcicek, 2016) is an encoder-decoder network followed by a reranker used as the shared task benchmark; NTEMP, a controllable neuralized hidden semi-Markov model; NTEMP+AR, the product of experts of both a NTemp model and an autoregressive LSTM network (Wiseman et al., 2018); SHEN19 (Shen et al., 2019) is an pragmatically informed model, which is the current state-of-the-art system on E2E dataset.
We also compare internally with ablations of our system: EncDEC is a conditional model  $p(y \mid x)$  trained without control states.  $\mathbf{PC}^0$  is posterior control model with no constraints. It uses structured encoder with the PR coefficient set to 0.  $\mathbf{PC}^\infty$  is our model with hard constraints, which assumes fully-observed control states. These control states are obtained by mapping tokens with lexical overlap to their designated state; otherwise we map to a generic state. We train a seq2seq model  $p(y, z \mid x)$  with full supervision of both control states and target text. Our main model is  $\mathbf{PC}^\lambda$ , which applies PR with coefficient given by hyperparameter  $\lambda$ .
For WikiBio, we compare externally against 5 systems: NTEMP and NTEMP+AR as above; LeBRET16 (Lebret et al., 2016a), which uses copy attention and an NNLM; LIU18 (ENCDEC), which is our base encoder-decoder LSTM model, and LIU18 (Field Gating) which uses a field gating table encoder and a decoder with dual attention (Liu et al., 2018). For internal comparison on WikiBio, we compare between the one-to-one and one-to
Official E2E evaluation scripts available at https://github.com/tuetschek/e2e-metrics
BLEU	NIST	E2E ROUGE	CIDEr	MET
validation
E2E-BENCH*	69.25	8.48	72.6	2.40	47.0
ENCDEC*	70.81	8.37	74.1	2.48	48.0
NTEMP	64.53	7.66	68.6	1.82	42.5
NTEMP+AR	67.70	7.98	69.5	2.29	43.1
PC0	69.10	8.32	72.6	2.35	47.3
PC∞	69.36	8.36	71.3	2.29	46.4
PCλ	72.93	8.63	75.5	2.54	48.4
test
E2E-BENCH*	65.93	8.59	68.5	2.23	44.8
SHEN19*	68.60	8.73	70.8	2.37	45.3
ENCDEC*	66.34	8.55	68.0	2.18	44.3
NTEMP	55.17	7.14	65.7	1.70	41.9
NTEMP+AR	59.80	7.56	65.0	1.95	38.8
PCλ	67.12	8.52	68.7	2.24	45.4
WikiBio
BLEU	NIST	R-4
test
LEBRET16*	34.7	7.98	25.8
LIU18(ENCDEC)*	43.7	-	40.3
LIU18(FieldGating)*	44.9	-	41.2
NTEMP	34.2	7.94	35.9
NTEMP+AR	34.8	7.59	38.6
PCone-to-one	44.7	9.92	43.3
PCone-to-many	44.2	9.59	41.5
Table 3: Automatic metrics for text generation. * marks systems without learned control states. (Left) E2E. Comparison of systems from Dušek and Jurčíček (2016); Wiseman et al. (2018); Shen et al. (2019), our model and ablations. (Right) WikiBio. Comparison of Wiseman et al. (2018); Liu et al. (2018); Lebret et al. (2016a) and our full model.
many constraints in  $\S 5$ .  $\mathbf{PC}_{\mathrm{one-to-one}}^{\lambda}$  applies the One-to-One posterior constraints (left of Table 1).  $\mathbf{PC}_{\mathrm{one-to-many}}^{\lambda}$  applies the One-to-Many posterior constraints (right of Table 1).
Table 3 shows the main results for the E2E and WikiBio, comparing to both standard neural models and controllable systems. On E2E (left), our posterior control model outperforms the neural benchmark system on all validation metrics and most of the test metrics. It also achieves results comparable or better than a specialized encoder-decoder system. It has significantly better performance than the controllable NTemp and NTemp+AR in all metrics on both validation and test. This demonstrates that the PC model provides interpretable and controllable states without sacrificing any representation power or generation performance.
For internal comparison, having soft constraints on the posterior outperforms the system  $\mathrm{PC}^{\infty}$  (forced hard constraints) and  $\mathrm{PC}^{0}$  (no constraints). Anecdotally, we find that if two fields have the same value, then the hard coding system is often forced into the wrong decision. Similarly removing posterior regularization altogether leads to a slightly weaker performance than our controlled model.
On the larger WikiBio dataset (right) our model also significantly outperforms both the controllable NTemp and NTemp+AR baselines in all three met
rics. It gives improvements over Liu et al. (2018)'s strong encoder-decoder style model. The promising result from WikiBio dataset suggests that the method scales to larger datasets and the PR style works well in handling large field spaces. In addition, we find that dynamic constraints are feasible compared with static constraints (we believe this is because the modeling burden on  $\mathrm{PC}_{\mathrm{one-to-many}}^{\lambda}$  is heavier since it also needs to figure out the clustering). Overall, the dynamic framework opens up the possibility of generalizing to work well with a wider set of constraints.
9 Analysis
Qualitative Analysis Table 4 shows how control states (shown by different colors) are used in generated sentences. We use examples generated by the  $\mathrm{PC}^{\lambda}$  system on the WikiBio dataset. We obtain outputs by beam search over control states and words. The first block contains examples with relatively complete coverage by the semantically grounded control states, including name, birth date, death date, occupation and nationality. We note that when a control state is selected, the textual span covered by the control state tend to respect truthfulness by copying from the table. The second block shows a longer example that uses less of the source, but still remain truthful with respect to the table.
Table 5 (left) qualitatively demonstrates the multi-modality of output of the system on E2E
billy ruge -lrb- c. 1885 - 1955 -rrb- was an american film actor .
debra dene barnes is an associate professor of piano studies at miss america 1968.
shaalin zoya -lrb- born 22 february 1997 -rrb- is an indian actress .
carlos albert andrs -lrb- born february 24 , 1978 in madrid, spain -rrb- is a spanish sculptor .
Table  $(x)$  :name[ james horton];birthdate[1850];deathdate[none];birthplace[boston, massachusetts];allegiance[united states of America];branch[united states navy];rank[captain of the top];awards[medal of honor]
REF: james horton -lrb- born 1850 -rrb- was a sailor serving in the united states navy who received the medal of honor for bravery.
PC $^{\lambda}$ : james horton -lrb- born 1850, date of death unknown -rrb- was a united states navy sailor and a recipient of the united states military's highest decoration, the medal of honor.
Table 4: Qualitative examples on WikiBio dataset. (Top) Generated sentences control states highlighted. (Bottom) Full example of content selection with data table and reference. (Best viewed in color.)
dataset. We particularly note how the final system is trained to associate control states with field types. Here we fix the prior on  $z$  to 8 different sequences of class labels shown in different colors, and do constrained beam search on the generative model by holding  $z$  fixed, and decoding from the model  $p_{\theta}(y \mid x, z)$ .
Controllability Next we consider a quantitative experiment on model control. Assuming we have a mapping from control states to fields, ideally, at test time  $z$  should use the right states from the source  $x$ . Let  $\mathcal{S} = \{(i,j,f):z_{i,j} = c,f\in x,\sigma (f) = c\}$  be the field states used by  $z$ . Define the field word overlap between  $x$  and  $y$  as,
$$
\# \text {m a t c h} = \sum_ {(i, j, f) \in \mathcal {S}} \text {u n i g r a m - o v e r l a p} \left(y _ {i: j}, x _ {f}\right)
$$
We can compute precision, recall, and coverage under this metric,
$$
\frac {\# m a t c h}{\sum_ {(i , j , f) \in \mathcal {S}} (j - i)}, \frac {\# m a t c h}{\sum_ {f \in x} | x _ {f} |}, \frac {| \mathcal {S} |}{| c : c \in x |}.
$$
Under these metrics we see the following control metrics on the E2E dataset,
P	R	C
PC∞	0.996	0.895	0.833
PCλ	1.0	0.969	1.0
The PC model with soft posterior constraints performs better than having hard constraints on all three metrics. Having  $P = 1$  means that the control states are a strong signal to copy from the table, and  $C = 1$  means that control states learn to cover all table fields. On WikiBio, the model has a precision of 0.83 on the, meaning that on average, when we generate a good control state,  $83\%$  of the generated tokens will match the table content. Since only a fraction of the source table in WikiBio is used, recall and coverage are less applicable.
Distributional Metrics Table 5 (right) shows distributional metrics related to the optimization of the generative model and the inference network. The reconstruction perplexity, Rec. is much lower than the full perplexity, PPL and the KL divergence between the variational posterior and the conditional prior is highly non-zero. These observations indicate that latent variables are being used in a non-trivial way by the generative model. It also suggests the variational model is not experiencing posterior collapse.
Limitations Given the promise of PR as a technique for inducing control states, it is worth noting some of the current limitations to our specific application of the method. Currently, we use simple rules which do not generalize well to paraphrase. Our weak supervision relies on direct overlap to align states and fails on aligning phrases like less then 10 dollars that are expressed as cheap. Additionally, while at test time, our method is comparable to a standard decoder model, it does require slightly longer to train due to both the dynamic program and the requirement to compute multiple samples.
This work introduces a method for controlling the output of a blackbox neural decoder model to follow weak supervision. The methodology utilizes posterior regularization within a structured variational framework. We show that this approach can induce a fully autoregressive neural model that is as expressive as standard neural decoders but also utilizes meaningful discrete control states. We show this decoder is effective for text generation while inducing meaningful discrete representations.
On E2E dataset, we remove the binary table field, "family friendly" which is never expressed by lexical match.
(1) Clowns is a 5 star coffee shop located near Clare Hall.
(2) Clowns is a coffee shop that serves English food and is near Clare Hall. It is in riverside and has a 5 out of 5 customer rating.
(3) Near Clare Hall in Riverside is coffee shop, Clowns. It serves English food, and has received a customer rating of 5 out of 5.
(4) Near the riverside, Clare Hall is a coffee shop called Clowns that serves English food and has a customer rating of 5 - stars.
(5) Near Clare Hall, Clowns coffee shop has a five star rating and English food.
(6) Clare Hall is a 5 star coffee shop near to Clowns that serves British food.
(7) Clowns coffee shop is near Clare Hall in Riverside. It serves English food and has an excellent customer rating.
(8) 5 star rated restaurant, Clowns coffee shop is located near Clare Hall.
Table  $(x)$  :name[Clowns] eatType[coffee shop] food[English] customerring[5 out of 5] area[riiverside] near[Clare Hall]
Models	Rec. ↓	PPL ↓	KL
E2E
PC0	1.81	3.74	19.8
PCλ	2.35	3.70	12.8
WikiBio
PC0	2.57	3.82	10.69
PCλone-to-one	2.45	4.07	10.19
PCλone-to-many	2.59	4.58	13.07
Table 5: (Left) Example of controlled generation  $p_{\theta}(y \mid x, z)$  on the source entity "Clowns" from E2E dataset. The color represents the class label of the token  $z$ . (Right) Metrics related to the generative model/inference network measured on both E2E and WikiBio. Rec. is reconstruction perplexity based on  $\mathbb{E}_{q(z|x,y)}[\log p_{\theta}(y \mid x,z)]$ . PPL is the perplexity per token estimated by importance sampling.
Induction of grounded control states opens up many possible future directions for this work. These states can be used to provide integration with external rule-based systems such as hard constraints at inference time. They also can be used to provide tools for human-assisted generation. Another direction is to improve the sources of weak supervision and such as interactive new constraints provided by users. One could also explore alternative posterior constraints based on pre-trained models for summarization or paraphrase tasks to induce semantically grounded latent variables. Finally, it would be interesting to explore alternative training methods for these models, such as reducing reliance on hard sampling through better relaxations of structured models.
Thanks to Yoon Kim, Jambay Kinley, and Tristan Yang for ideas and discussion. AMR was supported by NSF CAREER 1845664 and IIS 1901030. XLL was supported by a Sony Research Award. We thank the anonymous reviewers for helpful comments.
Waleed Ammar, Chris Dyer, and Noah A. Smith. 2014. Conditional random field autoencoders for unsupervised structured prediction. CoRR, abs/1411.1147.
Anja Belz and Ehud Reiter. 2006. Comparing automatic and human evaluation of NLG systems. In
11th Conference of the European Chapter of the Association for Computational Linguistics, Trento, Italy. Association for Computational Linguistics.
Hakan Bilen, Marco Pedersoli, and Tinne Tuytelaars. 2014. Weakly supervised detection with posterior regularization. In Proceedings of the British Machine Vision Conference. BMVA Press.
Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. 2019. Controllable paraphrase generation with a syntactic exemplar. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5972-5984, Florence, Italy. Association for Computational Linguistics.
Emilie Colin and Claire Gardent. 2018. Generating syntactic paraphrases. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 937-943, Brussels, Belgium. Association for Computational Linguistics.
Jan Milan Deriu and Mark Cieliebak. 2018. Syntactic manipulation for generating more diverse and interesting texts. In Proceedings of the 11th International Conference on Natural Language Generation, pages 22-34, Tilburg University, The Netherlands. Association for Computational Linguistics.
Ondrej Dusek and Filip Jurcicek. 2016. Sequence-to-sequence generation for spoken dialogue via deep syntax trees and strings. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).
Ondrej Dusek and Filip Jurcicek. 2016. Sequence-to-sequence generation for spoken dialogue via deep syntax trees and strings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 45-51, Berlin, Germany. Association for Computational Linguistics.
M.J.F. Gales and Steve Young. 1993. The theory of segmental hidden markov models.
Kuzman Ganchev, João Graça, Jennifer Gillenwater, and Ben Taskar. 2010. Posterior regularization for structured latent variable models. The Journal of Machine Learning Research, 11:20012049.
Joshua Goodman. 1999. Semiring parsing. Comput. Linguist., 25(4):573-605.
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1631-1640, Berlin, Germany. Association for Computational Linguistics.
Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. 2016. Pointing the unknown words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 140-149, Berlin, Germany. Association for Computational Linguistics.
Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric Xing. 2016a. Harnessing deep neural networks with logic rules. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2410-2420, Berlin, Germany. Association for Computational Linguistics.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017. Toward controlled generation of text.
Zhiting Hu, Zichao Yang, Ruslan Salakhutdinov, and Eric Xing. 2016b. Deep neural networks with massive learned knowledge. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1670-1679, Austin, Texas. Association for Computational Linguistics.
Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1875-1885, New Orleans, Louisiana. Association for Computational Linguistics.
Matthew Johnson, David K Duvenaud, Alex Wiltschko, Ryan P Adams, and Sandeep R Datta. 2016. Composing graphical models with neural networks for structured representations and fast inference. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 2946-2954. Curran Associates, Inc.
Yoon Kim, Alexander M. Rush, Lei Yu, Adhiguna Kuncoro, Chris Dyer, and Gabor Melis. 2019. Unsupervised recurrent neural network grammars. CoRR, abs/1904.03746.
Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR).
Diederik P. Kingma and Max Welling. 2014. Autoencoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.
Nikita Kitaev and Dan Klein. 2018. Constituency parsing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2676-2686, Melbourne, Australia. Association for Computational Linguistics.
Rahul G. Krishnan, Uri Shalit, and David Sontag. 2017. Structured inference networks for nonlinear state space models. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI'17, pages 2101-2109. AAAI Press.
Alon Lavie and Abhaya Agarwal. 2007. Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, StatMT '07, pages 228-231, Stroudsburg, PA, USA. Association for Computational Linguistics.
Rémi Lebret, David Grangier, and Michael Auli. 2016a. Neural text generation from structured data with application to the biography domain. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1203-1213, Austin, Texas. Association for Computational Linguistics.
Rmi Lebret, David Grangier, and Michael Auli. 2016b. Neural text generation from structured data with application to the biography domain. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.
Zhifei Li and Jason Eisner. 2009. First- and second-order expectation semirings with applications to minimum-risk training on translation forests. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 40-51, Singapore.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Tianyu Liu, Kexiang Wang, Lei Sha, Baobao Chang, and Zhifang Sui. 2018. Table-to-text generation by structure-aware seq2seq learning.
Fuli Luo, Damai Dai, Pengcheng Yang, Tianyu Liu, Baobao Chang, Zhifang Sui, and Xu Sun. 2019. Learning to control the fine-grained sentiment for story ending generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6020-6026, Florence, Italy. Association for Computational Linguistics.
Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. 2016. What to talk about and how? selective generation using lstms with coarse-to-fine alignment. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Andriy Mnih and Karol Gregor. 2014. Neural variational inference and learning in belief networks. CoRR, abs/1402.0030.
Andriy Mnih and Danilo Rezende. 2016. Variational inference for monte carlo objectives. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 2188-2196, New York, New York, USA. PMLR.
Amit Moryossef, Yoav Goldberg, and Ido Dagan. 2019. Step-by-step: Separating planning from realization in neural data-to-text generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2267-2277, Minneapolis, Minnesota. Association for Computational Linguistics.
Jekaterina Novikova, Ondrej Dusek, and Verena Rieser. 2017. The E2E dataset: New challenges for end-to-end generation. CoRR, abs/1706.09254.
Shereen Oraby, Lena Reed, Shubhangi Tandon, Sharath T.S., Stephanie Lukin, and Marilyn Walker. 2018. Controlling personality-based stylistic variation with neural natural language generators. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pages 180-190, Melbourne, Australia. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, pages 311-318, Stroudsburg, PA, USA. Association for Computational Linguistics.
Nanyun Peng, Marjan Ghazvininejad, Jonathan May, and Kevin Knight. 2018. Towards controllable story generation. In Proceedings of the First Workshop on Storytelling, pages 43-49, New Orleans, Louisiana. Association for Computational Linguistics.
Ratish Puduppully, Li Dong, and Mirella Lapata. 2019. Data-to-text generation with entity modeling. In