SlowGuess/ACL_2020_OCR huggingface dataset:
Shang-Yu Su Chao-Wei Huang Yun-Nung Chen
{f05921117,r07922069}@ntu.edu.tw y.v.chen@ieee.org
In modular dialogue systems, natural language understanding (NLU) and natural language generation (NLG) are two critical components, where NLU extracts the semantics from the given texts and NLG is to construct corresponding natural language sentences based on the input semantic representations. However, the dual property between understanding and generation has been rarely explored. The prior work (Su et al., 2019) is the first attempt that utilized the duality between NLU and NLG to improve the performance via a dual supervised learning framework. However, the prior work still learned both components in a supervised manner; instead, this paper introduces a general learning framework to effectively exploit such duality, providing flexibility of incorporating both supervised and unsupervised learning algorithms to train language understanding and generation models in a joint fashion. The benchmark experiments demonstrate that the proposed approach is capable of boosting the performance of both NLU and NLG.
Spoken dialogue systems that assist users to solve complex tasks such as booking a movie ticket have become an emerging research topic in artificial intelligence and natural language processing areas. With a well-designed dialogue system as an intelligent personal assistant, people can accomplish certain tasks more easily via natural language interactions. Nowadays, there are several virtual intelligent assistants, such as Apple's Siri, Google Assistant, Microsoft's Cortana, and Amazon's Alexa.
The recent advance of deep learning has inspired many applications of neural dialogue systems (Wen et al., 2017; Bordes et al., 2017). A typical dialogue system pipeline can be divided into several
components: a speech recognizer that transcribes a user's speech input into texts, a natural language understanding module (NLU) to classify the domain along with domain-specific intents and fill in a set of slots to form a semantic frame (Tur and De Mori, 2011; Hakkani-Tur et al., 2016). A dialogue state tracking (DST) module predicts the current dialogue state according to the multi-turn conversations, then the dialogue policy determines the system action for the next turn given the current dialogue state (Peng et al., 2018; Su et al., 2018a). Finally, the semantic frame indicating the policy is fed into a natural language generation (NLG) module to construct a response utterance to the user (Wen et al., 2015b; Su et al., 2018b).
Generally, NLU is to extract core semantic concepts from the given utterances, while NLG is to construct corresponding sentences based on the given semantic representations. However, the dual property between understanding and generation has been rarely investigated, Su et al. (2019) first introduced the duality into the typical supervised learning schemes to train these two models. Different from the prior work, this paper proposes a general learning framework leveraging the duality between understanding and generation, providing flexibility of incorporating not only supervised but also unsupervised learning algorithms to jointly train NLU and NLG modules. The contributions can be summarized as 3-fold:
- This paper proposes a general learning framework using the duality between NLU and NLG, where supervised and unsupervised learning can be flexibly incorporated for joint training.
- This work is the first attempt to exploits the dual relationship between NLU and NLG towards unsupervised learning.
- The benchmark experiments demonstrate the effectiveness of the proposed framework.
The source code is available at: https://github. com/MiuLab/DuaLUG.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 671-680 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
This paper focuses on modeling the duality between understanding and generation towards unsupervised learning of the two components, related work is summarized below.
Natural Language Understanding In dialogue systems, the first component is a natural language understanding (NLU) module—Parsing user utterances into semantic frames that capture the core meaning (Tur and De Mori, 2011). A typical NLU first determines the domain given input utterances, predicts the intent, and then fill the associated slots (Hakkani-Tur et al., 2016; Chen et al., 2016). However, the above work focused on single-turn interactions, where each utterance is treated independently. To overcome the error propagation and further improve understanding performance, contextual information has been leveraged and shown useful (Chen et al., 2015; Sun et al., 2016; Shi et al., 2015; Weston et al., 2015). Also, different speaker roles provided informative signal for capturing speaking behaviors and achieving better understanding performance (Chen et al., 2017; Su et al., 2018c).
Natural Language Generation NLG is another key component in dialogue systems, where the goal is to generate natural language sentences conditioned on the given semantics from the dialogue manager. As an endpoint of interacting with users, the quality of generated sentences is crucial for better user experience. In spite of robustness and adequacy of the rule-based methods, poor diversity makes talking to a template-based machine unsatisfactory. Furthermore, scalability is an issue, because designing sophisticated rules for a specific domain is time-consuming. Previous work proposed a RNNLM-based NLG that can be trained on any corpus of dialogue act-utterance pairs without hand-crafted features and any semantic alignment (Wen et al., 2015a). The following work based on sequence-to-sequence (seq2seq) models further obtained better performance by employing encoder-decoder structure with linguistic knowledge such as syntax trees (Sutskever et al., 2014; Su et al., 2018b).
Dual Learning Various tasks may have diverse goals, which are usually independent to each other. However, some tasks may hold a dual form, that is, we can swap the input and target of a task to formu
late another task. Such structural duality emerges as one of the important relationship for further investigation. Two AI tasks are of structure duality if the goal of one task is to learn a function mapping from space  $\mathcal{X}$  to  $\mathcal{Y}$ , while the others goal is to learn a reverse mapping from  $\mathcal{Y}$  and  $\mathcal{X}$ . Machine translation is an example (Wu et al., 2016), translation from English to Chinese has a dual task, which is translated from Chinese to English; the goal of automatic speech recognition (ASR) is opposite to the one of text-to-speech (TTS) (Tjandra et al., 2017), and so on. Previous work first exploited the duality of the task pairs and proposed supervised (Xia et al., 2017) and unsupervised (reinforcement learning) (He et al., 2016) learning frameworks. These recent studies magnified the importance of the duality by revealing exploitation of it could boost the learning of both tasks. Su et al. (2019) employed the dual supervised learning framework to train NLU and NLG and improve both models simultaneously. Recently, Shen et al. (2019) improved models for conditional text generation using techniques from computational pragmatics. The techniques formulated language production as a game between speakers and listeners, where a speaker should generate text which a listener can use to correctly identify the original input the text describes.
However, although the duality has been considered into the learning objective, two models in previous work are still trained separately. In contrast, this work proposes a general learning framework that trains the models jointly, so that unsupervised learning methods in this research field can be better explored.
In this section, we describe the problem formulation and the proposed learning framework, which is illustrated in Figure 1.
The problems we aim to solve are NLU and NLG; for both tasks, there are two spaces: the semantics space  $\mathcal{X}$  and the natural language space  $\mathcal{Y}$ . NLG is to generate sentences associated with the given semantics, where the goal is to learn a mapping function  $f:\mathcal{X}\to \mathcal{Y}$  that transforms semantic representations into natural language. On the other hand, NLU is to capture the core meaning of sentences, where the goal is to find a function  $g:\mathcal{Y}\rightarrow \mathcal{X}$  that
Figure 1: Left: The proposed joint dual learning framework, which comprises Primal Cycle and Dual Cycle. The framework is agnostic to learning objectives and the algorithm is detailed in Algorithm 1. Right: In our experiments, the models for NLG and NLU are a GRU unit accompanied with a fully-connected layer.
predicts semantic representations from the given natural language.
Given  $n$  data pairs  $\{(x_i, y_i)\}_{i=1}^n$  i.i.d. sampled from the joint space  $\mathcal{X} \times \mathcal{Y}$ . A typical strategy for the optimization problem is based on maximum likelihood estimation (MLE) of the parameterized conditional distribution by the trainable parameters  $\theta_{x \to y}$  and  $\theta_{y \to x}$  as below:
$$
f(x;\theta_{x\to y}) = \operatorname *{arg  max}_{\theta_{x\to y}}P(y\mid x;\theta_{x\to y}),
$$
$$
g(y;\theta_{y\to x}) = \operatorname *{arg  max}_{\theta_{y\to x}}P(x\mid y;\theta_{y\to x}).
$$
The E2E NLG challenge dataset (Novikova et al., 2017) $^{2}$  is adopted in our experiments, which is a crowd-sourced dataset of 50k instances in the restaurant domain. Each instance is a pair of a semantic frame containing specific slots and corresponding values and a associated natural language utterance with the given semantics. For example, a semantic frame with the slot-value pairs "name[Bibimbap House], food[English], priceRange[moderate], area [riverside], near [Clare Hall]" corresponds to the target sentence "Bibimbap House is a moderately priced restaurant who's main cuisine is English food. You will find this local gem near Clare Hall in the Riverside area". Although the original dataset is for NLG, of which the goal is to generate sentences based on the given slot-value pairs, we further formulate the NLU task as predicting slot-value pair based on the utterances, which can be viewed as a multi-label classification problem and each possible slot-value
pair is treated as an individual label. The formulation is similar to the prior work (Su et al., 2019).
3.2 Joint Dual Learning
Although previous work has introduced the learning schemes that exploit duality of AI tasks, most of it was based on reinforcement learning or standard supervised learning and the models of primal and dual tasks ( $f$  and  $g$  respectively) are trained separately. Intuitively, if the models of primal and dual tasks are optimally learned, a complete cycle of transforming data from the original space to another space then back to the original space should be exactly the same as the original data, which could be viewed as the ultimate goal of a dual problem. In our scenario, if we generate sentences from given semantics  $x$  via the function  $f$  and transform them back to the original semantics perfectly via the function  $g$ , it implies that our generated sentences are grounded to the original given semantics and has the mathematical condition:
$$
g (f (x)) \equiv x.
$$
Therefore, our objective is to achieve the perfect complete cycle of data transforming by training two dual models  $(f$  and  $g)$  in a joint manner.
3.2.1 Algorithm Description
As illustrated in Figure 1, the framework is composed of two parts: Primal Cycle and Dual Cycle. Primal Cycle starts from semantic frames  $x$ , (1) first transforms the semantic representation to sentences by the function  $f$ , (2) then computes the loss by the given loss function  $l_{1}$ , (3) predicts the semantic meaning from the generated sentences,
$^{2}$ http://www.macs.hw.ac.uk/InteractionLab/E2E/
(4) computes the loss by the given loss function  $l_{2}$ , (5) finally train the models based on the computed loss; Dual Cycle starts from utterances and is symmetrically formulated. The learning algorithm is described in Algorithm 1, which is agnostic to types of learning objective. Either a supervised learning objective or an unsupervised learning objective can be conducted at the end of the training cycles, and the whole framework can be trained in an end-to-end manner.
As the language understanding task in our experiments is to predict corresponding slot-value pairs of utterances, which is a multi-label classification problem, we utilized the binary cross entropy loss as the supervised objective function for NLU. Likewise, the cross entropy loss function is used as the supervised objective for NLG. Take NLG for example, the objective of the model is to optimize the conditional probability of predicting word tokens given semantics  $p(y \mid x)$ , so that the difference between the predicted distribution and the target distribution,  $q(y \mid x)$ , can be minimized:
$$
- \sum_ {y} ^ {n} \sum_ {y} q (y \mid x) \log p (y \mid x), \tag {1}
$$
where  $n$  is the number of samples.
On the other hand, we can also introduce the reinforcement learning objective into our framework, the objective aims to maximize the expected value of accumulated reward. In our experiments, we conduct policy gradient (REINFORCE) method (Sutton et al., 2000) for optimization, the gradient
could be written as:
$$
\nabla \mathbb {E} [ r ] = \mathbb {E} [ r (y) \nabla \log p (y \mid x) ], \tag {2}
$$
where the variety of reward  $r$  will be elaborated in the next section. The loss function  $l_{1}$  for both tasks could be (1), (2), and the combination of them.
3.4 Reward Function
Different types of rewards reflect various objectives and would result in different behaviors in the learned policy. Hence, we design various reward functions to explore the model behavior, including explicit and implicit feedback.
3.4.1 Explicit Reward
To evaluate the quality of generated sentences, two explicit reward functions are adopted.
Reconstruction Likelihood In our scenario, if we generate sentences based on given semantics  $x$  by the function  $f$  and could transform them back to the original semantics perfectly by the function  $g$ , it implies our generated sentences ground on the original given semantics. Therefore we use the reconstruction likelihood at the end of the training cycles as a reward function:
$$
\left\{ \begin{array}{l l} \log p (x \mid f (x _ {i}; \theta_ {x \to y}); \theta_ {y \to x}) & \text {P r i m a l}, \\ \log p (y \mid g (y _ {i}; \theta_ {y \to x}); \theta_ {x \to y}) & \text {D u a l}. \end{array} \right.
$$
Automatic Evaluation Score The goal of most NLP tasks is to predict word tokens correctly, so the loss functions used to train these models focus on the word level, such as cross entropy maximizing the continuous probability distribution of the next correct word given the preceding context.
However, the performance of these models is typically evaluated using discrete metrics. For instance, BLEU and ROUGE measure n-gram overlaps between the generated outputs and the reference texts. In order to enforce our NLG to generate better results in terms of the evaluation metrics, we utilize these automatic metrics as rewards to provide the sentence-level information. Moreover, we also leverage F-score in our NLU model to indicate the understanding performance.
3.4.2 Implicit Reward
In addition to explicit signals like reconstruction likelihood and the automatic evaluation metrics, a "softer" feedback signal may be informative. For both tasks, we design model-based methods estimating data distribution in order to provide such soft feedback.
Language Model For NLG, we utilize pretrained language models which estimate the whole data distribution to compute the joint probability of generated sentences, measuring their naturalness and fluency. In this work, we use a simple language model based on RNN (Mikolov et al., 2010; Sundermeyer et al., 2012). The language model is learned by a cross entropy objective in an unsupervised manner:
$$
p (y) = \prod_ {i} ^ {L} p \left(y _ {i} \mid y _ {1}, \dots , y _ {i - 1}; \theta_ {y}\right), \tag {3}
$$
where  $y(\cdot)$  are the words in a sentence  $y$ , and  $L$  is the length of the utterance.
Masked Autoencoder for Distribution Estimation (MADE) For NLU, the output contains a set of discrete labels, which do not fit the sequential model scenarios such as language models. Each semantic frame  $x$  in our work contains the core concept of a certain sentence, furthermore, the slot-value pairs are not independent to others, because they correspond to the same individual utterance. For example, McDonald's would probably be inexpensive; therefore the correlation should be taken into account when estimating the joint distribution.
Following Su et al. (2019), we measure the soft feedback signal for NLU using masked autoencoder (Germain et al., 2015) to estimate the joint distribution. By interrupting certain connections between hidden layers, we could enforce the variable unit  $x_{d}$  to only depend on any specific set of variables, not necessary on  $x_{<d}$ ; eventually we
could still have the joint distribution by product rule:
$$
p (x) = \prod_ {d} ^ {D} p (x _ {d} \mid S _ {d}),
$$
where  $d$  is the index of variable unit,  $D$  is the total number of variables, and  $S_{d}$  is a specific set of variable units. Because there is no explicit rule specifying the exact dependencies between slot-value pairs in our data, we consider various dependencies by ensembles of multiple decomposition by sampling different sets  $S_{d}$  and averaging the results.
3.5 Flexibility of Learning Scheme
The proposed framework provides various flexibility of designing and extending the learning scheme, described as follows.
Straight-Through Estimator In many NLP tasks, the learning targets are discrete, so the goals of most NLP tasks are predicting discrete labels such as words. In practice we perform argmax operations on the output distribution from learned models to select the most possible candidates. However, such operation does not have any gradient value, forbidding the networks be trained via backpropagation. Therefore, it is difficult to directly connect a primal task (NLU in our scenario) and a dual task (NLG in our scenario) and jointly train these two models due to the above issue.
The Straight-Through (ST) estimator (Bengio et al., 2013) is a widely applied method due to its simplicity and effectiveness. The idea of Straight-Through estimator is directly using the gradients of discrete samples as the gradients of the distribution parameters. Because discrete samples could be generated as the output of hard threshold functions or some operations on the continuous distribution, Bengio et al. (2013) explained the estimator by setting the gradients of hard threshold functions to 1. The structure of the Straight-Through estimator is illustrated in Figure 2. In this work, we introduce ST estimator for connecting two models, and therefore the gradient can be estimated and two models can be jointly trained in an end-to-end manner.
Distribution as Input In addition to employing the Straight-Through estimator, an alternative solution is to use continuous distribution as the input of models. For NLU, the inputs are the word tokens from NLG, so we use the predicted distribution over the vocabulary to perform the weighted-sum of word embeddings. For NLG, the model requires
Figure 2: Straight-Through estimator.
semantic frame vectors predicted by NLU as the input condition; in this case, the probability distribution of slot-value pairs predicted by NLU can directly serve as the input vector. By utilizing the output distribution in this way, two models can be trained jointly in an end-to-end fashion.
Hybrid Objective As described before, the proposed approach is agnostic to learning algorithms; in other words, we could apply different learning algorithms at the middle and end of the cycles. For example, we could apply supervised learning on NLU in the first half of Primal Cycle and reinforcement learning on NLG to form a hybrid training cycle. Because two models are trained jointly, the objective applied on one model would potentially impact on the behavior of the other. Furthermore, we could also apply multiple objective functions including supervised or unsupervised ones to formulate multi-objective learning schemes.
Towards Unsupervised Learning Because the whole framework can be trained jointly and propagate the gradients, we could apply only one objective in one learning cycle at the end of it. Specifically, in Algorithm 1, we can apply only  $l_{2}$  in line 8 and only  $l_{1}$  in line 15. Such flexibility potentially enables us to train the models based on unpaired data in a unsupervised manner. For example, sample unpaired data  $x$  and transform the data by function  $f$ , next, feed them into the function  $g$ , then compare the predicted results and the original input to compute the loss. Likewise, we can perform the training cycle symmetrically from  $y$ . It is also possible to utilize limited data and perform
the autoencoding cycle described above to apply semi-supervised learning.
Our models are trained on the official training set and verified on the official testing set of the E2E NLG challenge dataset (Novikova et al., 2017). The data preprocessing includes trimming punctuation marks, lemmatization, and turning all words into lowercase. Each possible slot-value pair is treated as an individual label and the total number of labels is 79. To evaluate the quality of the generated sequences regarding both precision and recall, for NLG, the evaluation metrics include BLEU and ROUGE (1, 2, L) scores with multiple references, while F1 measure is reported for evaluating NLU.
The proposed framework and algorithm are agnostic to model structures. In our experiments, we use a gated recurrent unit (GRU) (Cho et al., 2014) with fully-connected layers at ends of GRU for both NLU and NLG, which are illustrated in the right part of Figure 1. Thus the models may have semantic frame representation as initial and final hidden states and sentences as the sequential input. In all experiments, we use mini-batch Adam as the optimizer with each batch of 64 examples. 10 training epochs were performed without early stop, the hidden size of network layers is 200, and word embedding is of size 50.
The experimental results are shown in Table 1, each reported number is averaged on the official testing set from three turns. Row (a) is the baseline where NLU and NLG models are trained independently and separately by supervised learning. The best performance in Su et al. (2019) is reported in row (b), where NLU and NLG are trained separately by supervised learning with regularization terms exploiting the duality.
To overcome the issue of non-differentiability, we introduce Straight-Through estimator when connecting two tasks. Based on our framework, another baseline for comparison is to train two models jointly by supervised loss and straight-through estimators, of which the performance is reported in row (c). Specifically, the cross entropy loss (1) is utilized in both  $l_{1}$  and  $l_{2}$  in Algorithm 1. Because the models in the proposed framework are trained
Learning Scheme	NU
Micro-F	NLG
BLEU	ROUGE-1	ROUGE-2	ROUGE-L
(a)	Iterative Training (Supervised)	71.14	55.05	55.37	27.95	39.90
(b)	Dual Supervised Learning (Su et al., 2019)	72.32	57.16	56.37	29.19	40.44
(c)	Joint Training (Straight-Through)	71.73	55.19	55.16	27.45	39.33
(d)	(c) + (NLG w/ distribution)	73.22	55.18	55.35	27.81	39.36
(e)	(c) + (NU w/ distribution)	79.19	51.47	53.62	26.17	37.90
(f)	(c) + (NU and NLG w/ distribution)	80.03	55.34	56.17	28.48	39.24
(g)	(f) + RLmid(reconstruction likelihood)	80.07	55.32	56.12	28.07	39.59
(h)	(f) + RLend(reconstruction likelihood)	79.97	55.21	56.15	28.50	39.42
(i)	(f) + RLmid(BLEU+ROUGE, F1)	79.49	56.04	56.61	28.78	39.93
(j)	(f) + RLend(BLEU+ROUGE, F1)	80.35	57.59	56.71	29.06	40.28
(k)	(f) + RLmid(LM, MADE)	81.52	54.13	54.60	26.85	38.90
(l)	(f) + RLend(LM, MADE)	79.52	55.61	55.97	28.57	39.97
Table 1: The NLU performance reported on micro-F and the NLG performance reported on BLEU, ROUGE-1, ROUGE-2, and ROUGE-L of models (\%).