SlowGuess/ACL_2020_OCR huggingface dataset:
We start by comparing the JUGbasic performance with existing work following the original split of the datasets. The results are shown in Table 4. On E2E dataset, we follow previous work to use F1 of slot-values as the measurement for NLU, and BLEU-4 for NLG. For weather dataset, there is only published results for NLG. It can be observed that the JUGbasic model outperforms the previous state-of-the-art NLU and NLG systems on the E2E dataset, and also for NLG on the weather dataset. The results prove the effectiveness of introducing the shared latent variable  $z$  for jointly training NLU and NLG. We will further study the impact of the shared  $z$  in Section 4.4.2.
We also evaluated the three training scenarios of JUG in the semi-supervised setting, with different proportion of labelled and unlabelled data. The results for E2E is presented in Table 5 and 6. We computed both F1 score and joint accuracy (Mrksić
Table 7: NLU results with exact match accuracy  $(\%)$  on weather dataset.
Model / Data	5%	10%	25%	50%	100%
Decoupled	0.632	0.667	0.703	0.719	0.725
Augmentation*	0.635	0.677	0.703	0.727	-
JUGbasic	0.634	0.673	0.701	0.720	0.726
JUGmarginal	0.627	0.671	0.711	0.721	0.722
JUGsemi*	0.670	0.701	0.725	0.733	-
Table 8: NLG results with BLEU on weather dataset.
et al., 2017) of slot-values as a more solid NLU measurement. Joint accuracy is defined as the proportion of test examples whose slot-value pairs are all correctly predicted. For NLG, both BLEU-4 and semantic accuracy are computed. Semantic accuracy measures the proportion of correctly generated slot values in the produced utterances. From the results, we observed that Decoupled can be improved with techniques of generating pseudo data (Augmentation), which forms a stronger baseline. However, all our model variants perform better than the baselines on both NLU and NLG. When using only labelled data, our model JUGmarginal can surpass Decoupled across all the four measurements. The gains mainly come from the fact that the model uses auto-encoding objectives to help learn a shared semantic space. Compared to Augmentation, JUGmarginal also has a 'built-in mechanism' to bootstrap pseudo data on the fly of training (see Section 3.4). When adding extra unlabelled data, our model JUGsemi gets further performance boosts and outperforms all baselines by a significant margin.
With the varying proportion of unlabelled data in
Figure 2: Visualisation of latent variable  $z$ . Given a pair of  $x$  and  $y$ ,  $z$  can be sampled from the posterior  $q(z|x)$  or  $q(z|y)$ , denoted by blue and orange dots respectively.
the training set, we see that unlabelled data is helpful in almost all cases. Moreover, the performance gain is the more significant when the labelled data is less. This indicates that the proposed model is especially helpful for low resource setups when there is a limited amount of labelled training examples but more available unlabelled ones.
The results for weather dataset are presented in Table 7 and 8. In this dataset, NLU is more like a semantic parsing task (Berant et al., 2013) and we use exact match accuracy as its measurement. Meanwhile, NLG is measured by BLEU. The results reveal a very similar trend to that in E2E. The generated examples can be found in Appendix.
In this section we further analyse the impact of the shared latent variable and also the impact of utilising unlabelled data.
4.4.1 Visualisation of Latent Space
As mentioned in Section 2.1, the latent variable  $z$  can be sampled from either posterior approximation  $q(z|x)$  or  $q(z|y)$ . We inspect the latent space in Figure 2 to find out how well the model learns intent sharing. We plot  $z$  with the E2E dataset on 2-dimensional space using t-SNE projection (Maaten and Hinton, 2008).
We observe two interesting properties. First, for each data point  $(x,y)$ , the  $z$  values sampled from  $q(z|x)$  and  $q(z|y)$  are close to each other. This reveals that the meanings of  $x$  and  $y$  are tied in the latent space. Second, there exists distinct clusters in the space of  $z$ . By further inspecting the actual examples within each cluster, we found that a cluster represents a similar meaning composition. For instance, the cluster cen
Model	NLU	NLG
JUGbasic	90.55	0.726
JUGbasic (feed random z)	38.13	0.482
Table 9: A comparative study to evaluate the contribution of the learned latent variable  $z$  in NLU/NLG decoding. Models are trained on the whole weather dataset.
Method	NLU	NLG
Mi	Re	Wr	Mi	Wr
Decoupled	714	256	2382	5714	2317
JUGbasic	594	169	1884	4871	2102
Table 10: Error analysis on E2E dataset. Numbers of missing (Mi), redundant (Re) and wrong (Wr) predictions on slot-value pairs are reported for NLU; numbers of missing or wrong generated slot values are listed for NLG. Lower number indicates the better results. Both models are trained on  $5\%$  of the training data.
tered at (-20, -40) contains {name, foodtype, price, rating, area, near}, while the cluster centered at (45, 10) contains {name, eattype, foodtype, price}. This indicates that the shared latent serves as conclusive global feature representations for NLU and NLG.
4.4.2 Impact of the Latent Variable
One novelty of our model is the introduction of shared latent variable  $z$  for natural language  $x$  and formal representations  $y$ . A common problem in neural variational models is that when coupling a powerful autoregressive decoder, the decoder tends to learn to ignore  $z$  and solely rely on itself to generate the data (Bowman et al., 2016; Chen et al., 2017; Goyal et al., 2017). In order to examine to what extent does our model actually rely on the shared variable in both NLU and NLG, we seek for an empirical answer by comparing the JUGbasic model with a model variant which uses a random value of  $z$  sampled from a normal distribution  $N(0,1)$  during testing. From Table 9, we can observe that there exists a large performance drop if  $z$  is assigned with random values. This suggests that JUG indeed relies greatly on the shared variable to produce good-quality  $x$  or  $y$ .
We further analyse the various sources of errors to understand the cases which  $z$  helps to improve. On E2E dataset, wrong prediction in NLU comes from either predicting not Mention label for certain slots in ground truth semantics; predicting arbitrary values on slots not present in the ground truth semantics; or predicting wrong values com
Method	E2E	Weather
NLU	NLG	NLU	NLG
JUGbasic	60.30	0.685	73.62	0.634
+unlabelled x	62.89	0.765	74.97	0.654
+unlabelled y	59.55	0.815	76.98	0.621
+unlabelled x and y	68.09	0.814	79.19	0.670
Table 11: Comparison on sources of unlabelled data for semi-supervised learning using only utterances  $(x)$ , only semantic representations  $(y)$  or both  $(x$  and  $y)$ . JUGbasic model is trained on  $5\%$  of training data.
paring to ground truth. Three types of error are referred to Missing (Mi), Redundant (Re) and Wrong (Wr) in Table 10. For NLG, semantic errors can be either missing or generating wrong slot values in the given semantics (Wen et al., 2015). Our model makes fewer mistakes in all these error sources comparing to the baseline Decoupled. We believe this is because the clustering property learned in the latent space provides better feature representations at a global scale, eventually benefiting NLU and NLG.
4.4.3 Impact of Unlabelled Data Source
In Section 4.3, we found that the performance of our model can be further enhanced by leveraging unlabelled data. As we used both unlabelled utterances and unlabelled semantic representations together, it is unclear if both contributed to the performance gain. To answer this question, we start with the JUGbasic model, and experimented with adding unlabelled data from 1) only unlabelled utterances  $x$ ; 2) only semantic representations  $y$ ; 3) both  $x$  and  $y$ . As shown in Table 11, when adding any uni-sourced unlabelled data ( $x$  or  $y$ ), the model is able to improve to a certain extent. However, the performance can be maximised when both data sources are utilised. This strengthens the argument that our model can leverage bi-sourced unlabelled data more effectively via latent space sharing to improve NLU and NLG at the same time.
Natural Language Understanding (NLU) refers to the general task of mapping natural language to formal representations. One line of research in the dialogue community aims at detecting slot-value pairs expressed in user utterances as a classification problem (Henderson et al., 2012; Sun et al., 2014; Mrkšić et al., 2017; Vodolán et al., 2017). Another line of work focuses on converting single-turn user utterances to more structured meaning representa
tions as a semantic parsing task (Zettlemoyer and Collins, 2005; Jia and Liang, 2016; Dong and Lapata, 2018; Damonte et al., 2019).
In comparison, Natural Language Generation (NLG) is scoped as the task of generating natural utterances from their formal representations. This is traditionally handled with a pipelined approach (Reiter and Dale, 1997) with content planning and surface realisation (Walker et al., 2001; Stent et al., 2004). More recently, NLG has been formulated as an end-to-end learning problem where text strings are generated with recurrent neural networks conditioning on the formal representation (Wen et al., 2015; Dušek and Jurcicek, 2016; Dušek et al., 2020; Balakrishnan et al., 2019; Tseng et al., 2019).
There has been very recent work which does NLU and NLG jointly. Both Ye et al. (2019) and Cao et al. (2019) explore the duality of semantic parsing and NLG. The former optimises two sequence-to-sequence models using dual information maximisation, while the latter introduces a dual learning framework for semantic parsing. Su et al. (2019) proposes a learning framework for dual supervised learning (Xia et al., 2017) where both NLU and NLG models are optimised towards a joint objective. Their method brings benefits with annotated data in supervised learning, but does not allow semi-supervised learning with unlabelled data. In contrast to their work, we propose a generative model which couples NLU and NLG with a shared latent variable. We focus on exploring a coupled representation space between natural language and corresponding semantic annotations. As proved in experiments, the information sharing helps our model to leverage unlabelled data for semi-supervised learning, which eventually benefits both NLU and NLG.
We proposed a generative model which couples natural language and formal representations via a shared latent variable. Since the two space is coupled, we gain the luxury of exploiting each unpaired data source and transfer the acquired knowledge to the shared meaning space. This eventually benefits both NLU and NLG, especially in a low-resource scenario. The proposed model is also suitable for other translation tasks between two modalities.
As a final remark, natural language is richer and more informal. NLU needs to handle ambiguous
or erroneous user inputs. However, formal representations utilised by an NLG system are more precisely-defined. In future, we aim to refine our generative model to better emphasise this difference of the two tasks.
Bo-Hsiang Tseng is supported by Cambridge Trust and the Ministry of Education, Taiwan. This work has been performed using resources provided by the Cambridge Tier-2 system operated by the University of Cambridge Research Computing Service (http://www.hpc.cam.ac.uk) funded by EPSRC Tier-2 capital grant EP/P020259/1..
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Anusha Balakrishnan, Jinfeng Rao, Kartikeya Upasani, Michael White, and Rajen Subba. 2019. Constrained decoding for neural nlg from compositional representations in task-oriented dialogue. arXiv preprint arXiv:1906.07220.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffith, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178-186, Sofia, Bulgaria. Association for Computational Linguistics.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533-1544, Seattle, Washington, USA. Association for Computational Linguistics.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 10-21.
Ruisheng Cao, Su Zhu, Chen Liu, Jieyu Li, and Kai Yu. 2019. Semantic parsing with dual learning. ACL.
Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. 2017. Variational lossy autoencoder. In 5th International Conference on Learning Representations, ICLR 2017, Toulon,
France, April 24-26, 2017, Conference Track Proceedings.
Marco Damonte, Rahul Goel, and Tagyoung Chung. 2019. Practical semantic parsing for spoken language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Industry Papers), pages 16-23.
Li Dong and Mirella Lapata. 2018. Coarse-to-fine decoding for neural semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 731-742.
Ondrej Dusek and Filip Jurcicek. 2016. Sequence-to-sequence generation for spoken dialogue via deep syntax trees and strings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 45-51.
Ondrej Dušek, Jekaterina Novikova, and Verena Rieser. 2018. Findings of the e2e nlg challenge. In Proceedings of the 11th International Conference on Natural Language Generation, pages 322-328.
Ondrej Dušek, Jekaterina Novikova, and Verena Rieser. 2020. Evaluating the state-of-the-art of end-to-end natural language generation: The e2e nlg challenge. Computer Speech & Language, 59:123-156.
Anirudh Goyal Alias Parth Goyal, Alessandro Sordoni, Marc-Alexandre Côté, Nan Rosemary Ke, and Yoshua Bengio. 2017. Z-forcing: Training stochastic recurrent networks. In Advances in neural information processing systems, pages 6713-6723.
Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. 2016. Dual learning for machine translation. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 820-828. Curran Associates Inc.
Matthew Henderson, Milica Gašić, Blaise Thomson, Pirros Tsiakoulis, Kai Yu, and Steve Young. 2012. Discriminative spoken language understanding using word confusion networks. In 2012 IEEE Spoken Language Technology Workshop (SLT), pages 176-181. IEEE.
Matthew Henderson, Blaise Thomson, and Steve Young. 2014. Word-based dialog state tracking with recurrent neural networks. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 292-299.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12-22.
Juraj Juraska, Panagiotis Karagiannis, Kevin Bowden, and Marilyn Walker. 2018. A deep ensemble model with slot alignment for sequence-to-sequence natural language generation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 152-162.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Diederik P Kingma and Max Welling. 2013. Autoencoding variational bayes. arXiv preprint arXiv:1312.6114.
Dong-Hyun Lee. 2013. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning, ICML, volume 3, page 2.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579-2605.
Nikola Mrkšić, Diarmuid Žáeghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve Young. 2017. Neural belief tracker: Data-driven dialogue state tracking. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1777-1788.
Jekaterina Novikova, Ondrej Dušek, and Verena Rieser. 2017. The E2E dataset: New challenges for end-to-end generation. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 201-206, Saarbrücken, Germany. Association for Computational Linguistics.
Panupong Pasupat, Sonal Gupta, Karishma Mandyam, Rushin Shah, Mike Lewis, and Luke Zettlemoyer. 2019. Span-based hierarchical semantic parsing for task-oriented dialog. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1520-1526, Hong Kong, China. Association for Computational Linguistics.
Ehud Reiter and Robert Dale. 1997. Building applied natural language generation systems. Natural Language Engineering, 3(1):57-87.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96.
Amanda Stent, Rashmi Prasad, and Marilyn Walker. 2004. Trainable sentence planning for complex information presentation in spoken dialog systems. In Proceedings of the 42nd annual meeting on association for computational linguistics, page 79. Association for Computational Linguistics.
Shang-Yu Su, Chao-Wei Huang, and Yun-Nung Chen. 2019. Dual supervised learning for natural language understanding and generation. ACL.
Kai Sun, Lu Chen, Su Zhu, and Kai Yu. 2014. The sjtu system for dialog state tracking challenge 2. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIG-DIAL), pages 318-326.
Bo-Hsiang Tseng, Paweł Budzianowski, Yen-chen Wu, and Milica Gasic. 2019. Tree-structured semantic encoder with knowledge sharing for domain adaptation in natural language generation. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 155–164.
Miroslav Vodolan, Rudolf Kadlec, Jan Kleindienst, and V Parku. 2017. Hybrid dialog state tracker with asr features. EACL 2017, page 205.
Marilyn A Walker, Owen Rambow, and Monica Rogati. 2001. Spot: A trainable sentence planner. In Proceedings of the second meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies, pages 1-8. Association for Computational Linguistics.
Tsung-Hsien Wen, Milica Gasic, Nikola Mrkšić, PeiHao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned LSTM-based natural language generation for spoken dialogue systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1711-1721.
Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256.
Yingce Xia, Tao Qin, Wei Chen, Jiang Bian, Nenghai Yu, and Tie-Yan Liu. 2017. Dual supervised learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 3789-3798. JMLR.org.
Hai Ye, Wenjie Li, and Lu Wang. 2019. Jointly learning semantic parser and natural language generator via dual information maximization. arXiv preprint arXiv:1906.00575.
Steve Young, Milica GaÅqiÄG, Simon Keizer, FranÅgois Mairesse, Jost Schatzmann, Blaise Thomson, and Kai Yu. 2010. The hidden information state model: A practical framework for pomdp-based spoken dialogue management. Computer Speech Language, 24(2):150 - 174.
Tao Yu, Zifan Li, Zilin Zhang, Rui Zhang, and Dragomir Radev. 2018. Typesql: Knowledge-based type-aware neural text-to-sql generation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2, pages 588-594.
Luke S Zettlemoyer and Michael Collins. 2005. Learning to map sentences to logical form: structured classification with probabilistic categorical grammars. In Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence, pages 658-666. AUAI Press.
Luke S Zettlemoyer and Michael Collins. 2012. Learning to map sentences to logical form: Structured classification with probabilistic categorical grammars. arXiv preprint arXiv:1207.1420.
Xiaodong Zhang and Houfeng Wang. 2016. A joint model of intent determination and slot filling for spoken language understanding. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, pages 2993-2999. AAAI Press.
A.1 Derivation of Lower Bounds
We derive the lower bounds for  $\log p(x,y)$  as follows:
$$
\begin{array}{l} \log p (x, y) = \log \int_ {z} p (x, y, z) \\ = \log \int_ {z} \frac {p (x , y , z) q (z | x)}{q (z | x)} \\ = \log \int_ {z} \frac {p (x | z , y) p (y | z , x) p (z) q (z | x)}{q (z | x)} \\ = \log \mathbb {E} _ {q (z | x)} \frac {p (x | z , y) p (y | z , x) p (z)}{q (z | x)} \\ \geq \mathbb {E} _ {q (z | x)} \log \frac {p (x | z , y) p (y | z , x) p (z)}{q (z | x)} \\ = \mathbb {E} _ {q (z | x)} [ \log p (x | z, y) + \log p (y | z, x) ] \\ - \operatorname {K L} [ q (z | x) | | p (z) ] \tag {14} \\ \end{array}
$$
where  $q(z|x)$  represents an approximated posterior. This derivation gives us the Equation 6 in the paper. Similarly we can derive an alternative lower bound in Equation 7 by introducing  $q(z|y)$  instead of  $q(z|x)$ .
For marginal log-likelihood  $\log p(x)$  or  $\log p(y)$ , its lower bound is derived as follows:
$$
\begin{array}{l} \log p (x) = \log \int_ {y} \int_ {z} p (x, y, z) \\ = \log \int_ {y} \int_ {z} \frac {p (x | z , y) p (y) p (z) q (z | x) q (y | z , x)}{q (z | x) q (y | z , x)} \\ = \log \mathbb {E} _ {q (y | z, x)} \mathbb {E} _ {q (z | x)} \frac {p (x | z , y) p (y) p (z)}{q (z | x) q (y | z , x)} \\ \geq \mathbb {E} _ {q (y | z, x)} \mathbb {E} _ {q (z | x)} \log \frac {p (x | z , y) p (y) p (z)}{q (z | x) q (y | z , x)} \\ = \mathbb {E} _ {q (y | z, x)} \mathbb {E} _ {q (z | x)} \log p (x | z, y) \\ - \operatorname {K L} [ q (z | x) | | p (z) ] - \operatorname {K L} [ q (y | x, z) | | p (y) ] \tag {15} \\ \end{array}
$$
Note that the resulting lower bound consists of three terms: a reconstruction of  $x$ , a KL divergence which regularises the space of  $z$ , and also a KL divergence which regularises the space of  $y$ . We have dropped the last term in our optimisation objective in Equation 9, since we do not impose any prior assumption on the output space of the NLU model.
Analogously we can derive the lower bound for  $\log p(y)$ . We also do not impose any prior assumption on the output space of the NLG model, which leads us to Equation 10.
A.2 Generated Examples
Reference of example
x: "for those prepared to pay over £30, giraffe is a restaurant located near the six bells."
y: {name=giraffe, eat_type=restaurant, price_range=more than £30, near=the six bells}
Prediction by Decoupled model
x: "near the six bells, there is a restaurant called giraffe that is children friendly."
y: {name=travelers rest beefeater, price_range=more than £30, near=the six bells} (wrong name, miss eat_type)
Prediction by JUGsemi model
x: "giraffe is a restaurant near the six bells with a price range of more than £30."
y: {name=giraffe, eat_type=restaurant, price_range=more than £30, near=the six bells} (exact match)