SlowGuess/ACL_2020_OCR huggingface dataset:
Bo-Hsiang Tseng $^{1*}$ , Jianpeng Cheng $^{2}$ , Yimai Fang $^{2}$  and David Vandyke $^{2}$
<sup>1</sup>Engineering Department, University of Cambridge, UK
2Apple
bht26@cam.ac.uk
{jianpeng.cheng, yimai_fang, dvandyke}@apple.com
Natural language understanding (NLU) and natural language generation (NLG) are two fundamental and related tasks in building task-oriented dialogue systems with opposite objectives: NLU tackles the transformation from natural language to formal representations, whereas NLG does the reverse. A key to success in either task is parallel training data which is expensive to obtain at a large scale. In this work, we propose a generative model which couples NLU and NLG through a shared latent variable. This approach allows us to explore both spaces of natural language and formal representations, and facilitates information sharing through the latent space to eventually benefit NLU and NLG. Our model achieves state-of-the-art performance on two dialogue datasets with both flat and tree-structured formal representations. We also show that the model can be trained in a semi-supervised fashion by utilising unlabelled data to boost its performance.
Natural language understanding (NLU) and natural language generation (NLG) are two fundamental tasks in building task-oriented dialogue systems. In a modern dialogue system, an NLU module first converts a user utterance, provided by an automatic speech recognition model, into a formal representation. The representation is then consumed by a downstream dialogue state tracker to update a belief state which represents an aggregated user goal. Based on the current belief state, a policy network decides the formal representation of the system response. This is finally used by an NLG module to generate the system response(Young et al., 2010).
It can be observed that NLU and NLG have opposite goals: NLU aims to map natural language
(a) Generation
(c) NLU
(d) NLG
Figure 1: Generation and inference process in our model, and how NLU and NLG are achieved.  $x$  and  $y$  denotes utterances and formal representations respectively;  $z$  represents the shared latent variable for  $x$  and  $y$ .
to formal representations, while NLG generates utterances from their semantics. In research literature, NLU and NLG are well-studied as separate problems. State-of-the-art NLU systems tackle the task as classification (Zhang and Wang, 2016) or as structured prediction or generation (Damonte et al., 2019), depending on the formal representations which can be flat slot-value pairs (Henderson et al., 2014), first-order logical form (Zettlemoyer and Collins, 2012), or structured queries (Yu et al., 2018; Pasupat et al., 2019). On the other hand, approaches to NLG vary from pipelined approach subsuming content planning and surface realisation (Stent et al., 2004) to more recent end-to-end sequence generation (Wen et al., 2015; Dušek et al., 2020).
However, the duality between NLU and NLG has been less explored. In fact, both tasks can be treated as a translation problem: NLU converts
*Work done while the author was an intern at Apple.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1795-1807 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
natural language to formal language while NLG does the reverse. Both tasks require a substantial amount of utterance and representation pairs to succeed, and such data is costly to collect due to the complexity of annotation involved. Although unannotated data for either natural language or formal representations can be easily obtained, it is less clear how they can be leveraged as the two languages stand in different space.
In this paper, we propose a generative model for Joint natural language Understanding and Generation (JUG), which couples NLU and NLG with a latent variable representing the shared intent between natural language and formal representations. We aim to learn the association between two discrete spaces through a continuous latent variable which facilitates information sharing between two tasks. Moreover, JUG can be trained in a semi-supervised fashion, which enables us to explore each space of natural language and formal representations when unlabelled data is accessible. We examine our model on two dialogue datasets with different formal representations: the E2E dataset (Novikova et al., 2017) where the semantics are represented as a collection of slot-value pairs; and a more recent weather dataset (Balakrishnan et al., 2019) where the formal representations are tree-structured. Experimental results show that our model improves over standalone NLU/NLG models and existing methods on both tasks; and the performance can be further boosted by utilising unlabelled data.
Our key assumption is that there exists an abstract latent variable  $z$  underlying a pair of utterance  $x$  and formal representation  $y$ . In our generative model, this abstract intent guides the standard conditional generation of either NLG or NLU (Figure 1a). Meanwhile,  $z$  can be inferred from either utterance  $x$ , or formal representation  $y$  (Figure 1b). That means performing NLU requires us to infer the  $z$  from  $x$ , after which the formal representation  $y$  is generated conditioning on both  $z$  and  $x$  (Figure 1c), and vice-versa for NLG (Figure 1d). In the following, we will explain the model details, starting with NLG.
2.1 NLG
As mentioned above, the task of NLG requires us to infer  $z$  from  $y$ , and then generate  $x$  using
both  $z$  and  $y$ . We choose the posterior distribution  $q(z|y)$  to be Gaussian. The task of inferring  $z$  can then be recast to computing mean  $\mu$  and standard deviation  $\sigma$  of the Gaussian distribution using an NLG encoder. To do this, we use a bi-directional LSTM (Hochreiter and Schmidhuber, 1997) to encode formal representation  $y$ , which is linearised and represented as a sequence of symbols. After encoding, we obtain a list of hidden vectors  $\mathbf{H}$ , with each representing the concatenation of forward and backward LSTM states. These hidden vectors are then average-pooled and passed through two feedforward neural networks to compute mean  $\pmb{\mu}_{y,z}$  and standard deviation  $\sigma_{y,z}$  vectors of the posterior  $q(z|y)$ .
$$
\mathbf {H} = \operatorname {B i - L S T M} (\mathbf {y})
$$
$$
\bar {\mathbf {h}} = \operatorname {P o o l i n g} (\mathbf {H})
$$
$$
\boldsymbol {\mu} _ {y, z} = \mathbf {W} _ {\mu} \bar {\mathbf {h}} + \mathbf {b} _ {\mu} \tag {1}
$$
$$
\sigma_ {y, z} = \mathbf {W} _ {\sigma} \bar {\mathbf {h}} + \mathbf {b} _ {\sigma}
$$
where  $\mathbf{W}$  and  $\mathbf{b}$  represent neural network weights and bias. Then the latent vector  $\mathbf{z}$  can be sampled from the approximated posterior using the re-parameterisation trick of Kingma and Welling (2013):
$$
\epsilon \sim \mathcal {N} (0, \mathbf {I}) \tag {2}
$$
$$
\mathbf {z} = \boldsymbol {\mu} _ {y, z} + \sigma_ {y, z} \boldsymbol {\epsilon}
$$
The final step is to generate natural language  $x$  based on latent variable  $z$  and formal representation  $y$ . We use an LSTM decoder relying on both  $z$  and  $y$  via attention mechanism (Bahdanau et al., 2014). At each time step, the decoder computes:
$$
\mathbf {g} _ {i} ^ {x} = \operatorname {L S T M} \left(\mathbf {g} _ {i - 1} ^ {x}, \mathbf {x} _ {i - 1}\right)
$$
$$
\mathbf {c} _ {i} = \operatorname {a t t e n t i o n} \left(\mathbf {g} _ {i} ^ {x}, \mathbf {H}\right) \tag {3}
$$
$$
p \left(x _ {i}\right) = \operatorname {s o f t m a x} \left(\mathbf {W} _ {\mathbf {v}} \left[ \mathbf {c} _ {i} \oplus \mathbf {g} _ {i} ^ {x} \oplus \mathbf {z} \right] + \mathbf {b} _ {v}\right)
$$
where  $\oplus$  denotes concatenation.  $\mathbf{x}_{i - 1}$  is the word vector of input token;  $\mathbf{g}_i^x$  is the corresponding decoder hidden state and  $p(x_{i})$  is the output token distribution at time step  $i$ .
2.2 NLU
NLU performs the reverse procedures of NLG. First, an NLU encoder infers the latent variable  $z$  from utterance  $x$ . The encoder uses a bi-directional LSTM to convert the utterance into a list of hidden states. These hidden states are pooled and passed through feed-forward neural networks to compute the mean  $\mu_{x,z}$  and standard deviation  $\sigma_{x,z}$  of the posterior  $q(z|x)$ . This procedure follows Equation 1 in NLG.
However, note that a subtle difference between natural language and formal language is that the former is ambiguous while the later is precisely defined. This makes NLU a many-to-one mapping problem but NLG is one-to-many. To better reflect the fact that the NLU output requires less variance, when decoding we choose the latent vector  $\mathbf{z}$  in NLU to be the mean vector  $\mu_{x,z}$ , instead of sampling it from  $q(z|x)$  like Equation 2.
After the latent vector is obtained, the formal representation  $y$  is predicted from both  $z$  and  $x$  using an NLU decoder. Since the space of  $y$  depends on the formal language construct, we consider two common scenarios in dialogue systems. In the first scenario,  $y$  is represented as a set of slot-value pairs, e.g., {food type=British, area=north} in restaurant search domain (Mrkšić et al., 2017). The decoder here consists of several classifiers, one for each slot, to predict the corresponding values. Each classifier is modelled by a 1-layer feed-forward neural network that takes  $z$  as input:
$$
p (y _ {s}) = \operatorname {s o f t m a x} (\mathbf {W} _ {\mathbf {s}} \mathbf {z} + \mathbf {b} _ {\mathbf {s}}) \qquad (4)
$$
where  $p(y_{s})$  is the predicted value distribution of slot  $s$ .
In the second scenario,  $y$  is a tree-structured formal representation (Banarescu et al., 2013). We then generate  $y$  as a linearised token sequence using an LSTM decoder relying on both  $z$  and  $x$  via the standard attention mechanism (Bahdanau et al., 2014). The decoding procedure follows exactly Equation 3.
2.3 Model Summary
One flexibility of the JUG model comes from the fact that it has two ways to infer the shared latent variable  $z$  through either  $x$  or  $y$ ; and the inferred  $z$  can aid the generation of both  $x$  and  $y$ . In this next section, we show how this shared latent variable enables the JUG model to explore unlabelled  $x$  and  $y$ , while aligning the learned meanings inside the latent space.
3 Optimisation
We now describe how JUG can be optimised with a pair of  $x$  and  $y$  (§3.1), and also unpaired  $x$  or
$y$  (§3.2). We specifically discuss the prior choice of JUG objectives in §3.3. A combined objective can be thus derived for semi-supervised learning: a practical scenario when we have a small set of labelled data but abundant unlabelled ones (§3.4).
3.1 Optimising  $p(x,y)$
Given a pair of utterance  $x$  and formal representation  $y$ , our objective is to maximise the log-likelihood of the joint probability  $p(x, y)$ :
$$
\log p (x, y) = \log \int_ {z} p (x, y, z) \tag {5}
$$
The optimisation task is not directly tractable since it requires us to marginalise out the latent variable  $z$ . However, it can be solved by following the standard practice of neural variational inference (Kingma and Welling, 2013). An objective based on the variational lower bound can be derived as
$$
\begin{array}{l} \mathcal {L} _ {x, y} = \mathbb {E} _ {q (z | x)} \log p (y | z, x) + \mathbb {E} _ {q (z | x)} \log p (x | z, y) \\ - \operatorname {K L} [ q (z | x) | | p (z) ] \tag {6} \\ \end{array}
$$
where the first term on the right side is the NLU model; the second term is the reconstruction of  $x$ ; and the last term denotes the Kullback-Leibler divergence between the approximate posterior  $q(z|x)$  with the prior  $p(z)$ . We defer the discussion of prior to Section 3.3 and detailed derivations to Appendix.
The symmetry between utterance and semantics offers an alternative way of inferring the posterior through the approximation  $q(z|y)$ . Analogously we can derive a variational optimisation objective:
$$
\begin{array}{l} \mathcal {L} _ {y, x} = \mathbb {E} _ {q (z | y)} \log p (x | z, y) + \mathbb {E} _ {q (z | y)} \log p (y | z, x) \\ - \operatorname {K L} [ q (z | y) | | p (z) ] \tag {7} \\ \end{array}
$$
where the first term is the NLG model; the second term is the reconstruction of  $y$ ; and the last term denotes the KL divergence.
It can be observed that our model has two posterior inference paths from either  $x$  or  $y$ , and also two generation paths. All paths can be optimised.
3.2 Optimising  $p(x)$  or  $p(y)$
Additionally, when we have access to unlabelled utterance  $x$  (or formal representation  $y$ ), the optimisation objective of JUG is the marginal likelihood  $p(x)$  (or  $p(y)$ ):
$$
\log p (x) = \log \int_ {y} \int_ {z} p (x, y, z) \tag {8}
$$
<sup>1</sup>Note that it is still necessary to compute the standard deviation  $\sigma_{x,z}$  in NLU, since the term is needed for optimisation. See more details in Section 3.
2Each slot has a set of corresponding values plus a special one not Mention.
Note that both  $z$  and  $y$  are unobserved in this case.
We can develop an objective based on the variational lower bound for the marginal:
$$
\begin{array}{l} \mathcal {L} _ {x} = \mathbb {E} _ {q (y | z, x)} \mathbb {E} _ {q (z | x)} \log p (x | z, y) \\ - \operatorname {K L} [ q (z | x) | | p (z) ] \tag {9} \\ \end{array}
$$
where the first term is the auto-encoder reconstruction of  $x$  with a cascaded NLU-NLG path. The second term is the KL divergence which regularizes the approximated posterior distribution. Detailed derivations can be found in Appendix.
When computing the reconstruction term of  $x$ , it requires us to first run through the NLU model to obtain the prediction on  $y$ , from which we run through NLG to reconstruct  $x$ . The full information flow is  $(x \to z \to y \to z \to x)$ . Connections can be drawn with recent work which uses back-translation to augment training data for machine translation (Sennrich et al., 2016; He et al., 2016). Unlike back-translation, the presence of latent variable in our model requires us to sample  $z$  along the NLU-NLG path. The introduced stochasticity allows the model to explore a larger area of the data manifold.
The above describes the objectives when we have unlabelled  $x$ . We can derive a similar objective for leveraging unlabelled  $y$ :
$$
\begin{array}{l} \mathcal {L} _ {y} = \mathbb {E} _ {q (x | z, y)} \mathbb {E} _ {q (z | y)} \log p (y | z, x) \\ - \operatorname {K L} [ q (z | y) | | p (z) ] \tag {10} \\ \end{array}
$$
where the first term is the auto-encoder reconstruction of  $y$  with a cascaded NLG-NLU path. The full information flow here is  $(y\rightarrow z\rightarrow x\rightarrow z\rightarrow y)$
3.3 Choice of Prior
The objectives described in 3.1 and 3.2 require us to match an approximated posterior (either  $q(z|x)$  or  $q(z|y)$ ) to a prior  $p(z)$  that reflects our belief. A common choice of  $p(z)$  in the research literature is the Normal distribution (Kingma and Welling, 2013). However, it should be noted that even if we match both  $q(z|x)$  and  $q(z|y)$  to the same prior, it does not guarantee that the two inferred posteriors are close to each other; this is a desired property of the shared latent space.
To better address the property, we propose a novel prior choice: when the posterior is inferred
from  $x$  (i.e.,  $q(z|x))$ , we choose the parameterised distribution  $q(z|y)$  as our prior belief of  $p(z)$ . Similarly, when the posterior is inferred from  $y$  (i.e.,  $q(z|y))$ , we have the freedom of defining  $p(z)$  to be  $q(z|x)$ . This approach directly pulls  $q(z|x)$  and  $q(z|y)$  closer to ensure a shared latent space.
Finally, note that it is straightforward to compute both  $q(z|x)$  and  $q(z|y)$  when we have parallel  $x$  and  $y$ . However when we have the access to unlabelled data, as described in Section 3.2, we can only use the pseudo  $x - y$  pairs that are generated by our NLU or NLG model, such that we can match an inferred posterior to a pre-defined prior reflecting our belief of the shared latent space.
3.4 Training Summary
In general, JUG subsumes the following three training scenarios which we will experiment with.
When we have fully labelled  $x$  and  $y$ , the JUG jointly optimises NLU and NLG in a supervised fashion with the objective as follows:
$$
\mathcal {L} _ {\text {b a s i c}} = \sum_ {(x, y) \sim (X, Y)} \left(\mathcal {L} _ {x, y} + \mathcal {L} _ {y, x}\right) \tag {11}
$$
where  $(X,Y)$  denotes the set of labelled examples.
Additionally in the fully supervised setting, JUG can be trained to optimise both NLU, NLG and auto-encoding paths. This corresponds to the following objective:
$$
\mathcal {L} _ {\text {m a r g i n a l}} = \mathcal {L} _ {\text {b a s i c}} + \sum_ {(x, y) \sim (X, Y)} \left(\mathcal {L} _ {x} + \mathcal {L} _ {y}\right) \tag {12}
$$
Furthermore, when we have additional unlabelled  $x$  or  $y$ , we optimise a semi-supervised JUG objective as follows:
$$
\mathcal {L} _ {\text {s e m i}} = \mathcal {L} _ {\text {b a s i c}} + \sum_ {x \sim X} \mathcal {L} _ {x} + \sum_ {y \sim Y} \mathcal {L} _ {y} \tag {13}
$$
where  $X$  denotes the set of utterances and  $Y$  denotes the set of formal representations.
We experiment on two dialogue datasets with different formal representations to test the generality of our model. The first dataset is E2E (Novikova et al., 2017), which contains utterances annotated with flat slot-value pairs as their semantic representations. The second dataset is the recent weather dataset (Balakrishnan et al., 2019), where both utterances and semantics are represented in tree structures. Examples of the two datasets are provided in tables 1 and 2.
This information flow requires us to sample both  $z$  and  $y$  in reconstructing  $x$ . Since  $y$  is a discrete sequence, we use REINFORCE (Williams, 1992) to pass the gradient from NLG to NLU in the cascaded NLU-NLG path.
Natural Language
"sousa offers British food in the low price range.
it is family friendly with a 3 out of 5 star rating.
you can find it near the sunshine vegetarian cafe."
Semantic Representation
restaurant_name=sousa, food=english,
price_range=cheap, customerrating=average,
family Friendly=yes, near=sunshine vegetarian cafe
Table 1: An example in E2E dataset.
Natural Language (original)
"[_DG_YES_Yes] , [_DG_INFORM_
[_ARG_DATE_TIME_ [_ARG_COLLOQUIAL today's] ]
forecast is [_ARG_CLOUD_COVERAGEmostly cloudy ]
with [_ARG_CONDITION_light rain showers] ."
Natural Language (processed by removing tree annotations)
"Yes, today's forecast is mostly cloudy with light rain showers."
Semantic Representation
[_DG_YES_ [_ARG_TASK_get_weather_attribute] ]
[_DG_INFORM_ [_ARG_TASK_get_forecast]
[_ARG_CONDITION_light rain showers ]
[_ARG_CLOUD_COVERAGEmostly cloudy ]
[_ARG_DATE_TIME_ [_ARG_COLLOQUIAL today's] ]
4.1 Training Scenarios
We primarily evaluated our models on the raw splits of the original datasets, which enables us to fairly compare fully-supervised JUG with existing work on both NLU and NLG. Statistics of the two datasets can be found in Table 3.
In addition, we set up an experiment to evaluate semi-supervised JUG with a varying amount of labelled training data (5%, 10%, 25%, 50%, 100%, with the rest being unlabelled). Note that the original E2E test set is designed on purpose with unseen slot-values in the test set to make it difficult (Dusek et al., 2018, 2020); we remove the distribution bias by randomly re-splitting the E2E dataset. On the contrary, utterances in the weather dataset contains extra tree-structure annotations which make the NLU task a toy problem. We therefore remove these annotations to make NLU more realistic, as shown in the second row of Table 2.
As described in Section 3.4, we can optimise our proposed JUG model in various ways. We investigate the following approaches:
JUGbasic: this model jointly optimises NLU
Table 2: An example in weather dataset. The natural language in original dataset (first row) is used for training to have a fair comparison with existing methods. The processed utterances (second row) is used in our semi-supervised setting.
Dataset	Train	Valid	Test
E2E	42061	4672	4693
Weather	25390	3078	3121
Table 3: Number of examples in two datasets
E2E NLU	F1
Dual supervised learning (Su et al., 2019)	0.7232
JUGbasic	0.7337
E2E NLG	BLEU
TGEN (Dušek and Jurcicek, 2016)	0.6593
SLUG (Juraska et al., 2018)	0.6619
Dual supervised learning (Su et al., 2019)	0.5716
JUGbasic	0.6855
Weather NLG	BLEU
S2S-CONSTR (Balakrishnan et al., 2019)	0.7660
JUGbasic	0.7768
Table 4: Comparison with previous systems on two datasets. Note that there is no previous system trained for NLU in weather dataset.
and NLG with the objective in Equation 11. This uses labelled data only.
JUGmarginal: jointly optimises NLU, NLG and auto-encoders with only labelled data, per Equation 12.
JUG<sub>semi</sub>: jointly optimises NLU and NLG with labelled data and auto-encoders with unlabelled data, per Equation 13.
We compare our proposed model with some existing methods as shown in Table 4 and two designed baselines as follows:
Decoupled: The NLU and NLG models are trained separately by supervised learning. Both of the individual models have the same encoder-decoder structure as JUG. However, the main difference is that there is no shared latent variable between the two individual NLU and NLG models.
Augmentation: We pre-train Decoupled models to generate pseudo label from the unlabeled corpus (Lee, 2013) in a setup similar to backtranslation (Sennrich et al., 2016). The pseudo data and labelled data are then used together to fine-tune the pre-trained models.
Among all systems in our experiments, the number of units in LSTM encoder/decoder are set to  $\{150, 300\}$  and the dimension of latent space is 150. The optimiser Adam (Kingma and Ba, 2014) is used with learning rate 1e-3. Batch size is set to  $\{32, 64\}$ . All the models are fully trained and the
$^{4}$ Following Balakrishnan et al. (2019), the evaluation code https://github.com/tuetschek/e2e-metrics provided by the E2E organizers is used here for calculating BLEU in NLG.
Model / Data	5%	10%	25%	50%	100%
Decoupled	52.77 (0.874)	62.32 (0.902)	69.37 (0.924)	73.68 (0.935)	76.12 (0.942)
Augmentation*	54.71 (0.878)	62.54 (0.902)	68.91 (0.922)	73.84 (0.935)	-
JUGbasic	60.30 (0.902)	67.08 (0.918)	72.49 (0.932)	74.74 (0.937)	78.05 (0.945)
JUGmarginal	62.96 (0.907)	68.43 (0.920)	73.35 (0.933)	75.74 (0.939)	78.93 (0.948)
JUGsemi*	68.09 (0.921)	70.33 (0.925)	73.79 (0.935)	75.46 (0.939)	-
Table 5: NLU results on E2E dataset. Joint accuracy (%) and F1 score (in bracket) are both reported with varying percentage of labelled training data. Models using unlabelled data are marked with * .
Model / Data	5%	10%	25%	50%	100%
Decoupled	0.693 (83.47)	0.723 (87.33)	0.784 (92.52)	0.793 (94.91)	0.813 (96.98)
Augmentation*	0.747 (84.79)	0.770 (90.13)	0.806 (94.06)	0.815 (96.04)	-
JUGbasic	0.685 (84.20)	0.734 (88.68)	0.769 (93.83)	0.788 (95.11)	0.810 (95.07)
JUGmarginal	0.724 (85.57)	0.775 (93.59)	0.803 (94.99)	0.817 (98.67)	0.830 (99.11)
JUGsemi*	0.814 (90.47)	0.792 (94.76)	0.819 (95.59)	0.827 (98.42)	-
Table 6: NLG results on E2E dataset. BLEU and semantic accuracy  $(\%)$  (in bracket) are both reported with varying percentage of labelled training data. Models using unlabelled data are marked with  $*$
Model / Data	5%	10%	25%	50%	100%
Decoupled	73.46	80.85	86.00	88.45	90.68
Augmentation*	74.77	79.84	86.24	88.69	-
JUGbasic	73.62	80.13	86.15	87.94	90.55
JUGmarginal	74.61	81.14	86.83	89.06	91.28
JUGsemi*	79.19	83.22	87.46	89.17	-
best model is picked by the average of NLU and NLG results on validation set during training.