Introduction
As an essential part of a task-oriented dialogue system BIBREF0 , the task of natural language generation (NLG) is to produce a natural language utterance containing the desired information given a semantic representation consisting of dialogue act types with a set of slot-value pairs. Conventional methods using hand-crafted rules often generates monotonic utterances and it requires substantial amount of human engineering work. Recently, various neural approaches BIBREF1 , BIBREF2 , BIBREF3 have been proposed to generate accurate, natural and diverse utterances. However, these methods are typically developed for particular domains. Moreover, they are often data-intensive to train. The high annotation cost prevents developers to build their own NLG component from scratch. Therefore, it is extremely useful to train a NLG model that can be generalized to other NLG domains or tasks with a reasonable amount of annotated data. This is referred to low-resource NLG task in this paper.
Recently, some methods have been proposed for low-resource NLG tasks. Apart from the simple data augmentation trick BIBREF4 , specialized model architectures, including conditional variational auto-encoders (CVAEs, BIBREF3 , BIBREF5 , BIBREF6 ) and adversarial domain adaptation critics BIBREF5 , have been proposed to learn domain-invariant representations. Although promising results were reported, we found that datasets used by these methods are simple which tend to enumerate many slots and values in an utterance without much linguistic variations. As a consequence, over-fitting the slots and values in the low-resource target domain could even outperform those versions trained with rich source domain examples BIBREF6 . Fortunately, there is a new large-scale dialog dataset (MultiWoz, BIBREF7 ) that contains a great variety of domains and linguistic patterns that allows us to conduct extensive and meaningful experimental analysis for low-resource NLG tasks.
In this paper, instead of casting the problem as model-based approaches, we propose a generalized optimization-based meta-learning approach to directly enhance the optimization procedure for the low-resource NLG task. We start by arguing that a recently proposed model-agnostic meta-learning algorithm (MAML, BIBREF8 ) is a nice fit to the low-resource NLG task. Then, we proposed a generalized NLG algorithm called Meta-NLG based on MAML by viewing languages in different domains or dialog act types as separate Meta NLG tasks. Following the essence of MAML, the goal of Meta-NLG is to learn a better initialization of model parameters that facilitates fast adaptation to new low-resource NLG scenarios. As Meta-NLG is model-agnostic as long as the model can be optimized by gradient descent, we could apply it to any existing NLG models to optimize them in a way that adapt better and faster to new low-resource tasks.
The main contribution of this paper is two-fold:
Natural Language Generation (NLG)
Neural models have recently shown promising results in tackling NLG tasks for task-oriented dialog systems. Conditioned on some semantic representation called dialog act (DA), a NLG model decodes an utterance word by word, and the probability of generating an output sentence of length INLINEFORM0 is factorized as below:
DISPLAYFORM0
INLINEFORM0 is the NLG model parameterized by INLINEFORM1 , and INLINEFORM2 is the DA of sentence INLINEFORM3 . For example, INLINEFORM4 is a one-hot representation of a DA “Inform(name=The Oak Bistro, food=British)”. “Inform” (DA type) controls the sentence functionality, and “name” and “food” are two involved slots. A realization utterance INLINEFORM5 could be “There is a restaurant called [The Oak Bistro] that serves [British] food.”. Each sentence might contain multiple DA types. A series of neural methods have been proposed, including HLSTM BIBREF9 , SCLSTM BIBREF1 , Enc-Dec BIBREF10 and RALSTM BIBREF2 .
Low-resource NLG
The goal of low-resource NLG is to fine-tune a pre-trained NLG model on new NLG tasks (e.g., new domains) with a small amount of training examples. BIBREF4 proposed a “data counterfeiting” method to augment the low-resource training data in the new task without modifying the model or training procedure. BIBREF3 proposed a semantically-conditioned variational autoencoder (SCVAE) learn domain-invariant representations feeding to SCLSTM. They shown that it improves SCLSTM in low-resource settings. BIBREF6 adopted the same idea as in BIBREF3 . They used two conditional variational autoencoders to encode the sentence and the DA into two separate latent vectors, which are fed together to the decoder RALSTM BIBREF2 . They later designed two domain adaptation critics with an adversarial training algorithm BIBREF5 to learn an indistinguishable latent representation of the source and the target domain to better generalize to the target domain. Different from these model-based approaches, we directly tackle the optimization issue from a meta-learning perspective.
Meta-Learning
Meta-learning or learning-to-learn, which can date back to some early works BIBREF11 , has recently attracted extensive attentions. A fundamental problem is “fast adaptation to new and limited observation data”. In pursuing this problem, there are three categories of meta-learning methods:
Metric-based: The idea is to learn a metric space and then use it to compare low-resource testing samples to rich training samples. The representative works in this category include Siamese Network BIBREF12 , Matching Network BIBREF13 , Memory-augmented Neural Network (MANN BIBREF14 ), Prototype Net BIBREF15 , and Relation Network BIBREF16 .
Model-based: The idea is to use an additional meta-learner to learn to update the original learner with a few training examples. BIBREF17 developed a meta-learner based on LSTMs. Hypernetwork BIBREF18 , MetaNet BIBREF19 , and TCML BIBREF20 also learn a separate set of representations for fast model adaptation. BIBREF21 proposed an LSTM-based meta-learner to learn the optimization algorithm (gradients) used to train the original network.
Optimization-based: The optimization algorithm itself can be designed in a way that favors fast adaption. Model-agnostic meta-learning (MAML, BIBREF8 , BIBREF22 , BIBREF23 ) achieved state-of-the-art performance by directly optimizing the gradient towards a good parameter initialization for easy fine-tuning on low-resource scenarios. It introduces no additional architectures nor parameters. Reptile BIBREF24 is similar to MAML with only first-order gradient. In this paper, we propose a generalized meta optimization method based on MAML to directly solve the intrinsic learning issues of low-resource NLG tasks.
Meta-Learning for Low-resource NLG
In this section, we first describe the objective of fine-tuning a NLG model on a low-resource NLG task in Section 3.1. Then, we describe how our Meta-NLG algorithm encapsulates this objective into Meta NLG tasks and into the meta optimization algorithm to learn better low-resource NLG models.
Fine-tune a NLG model
Suppose INLINEFORM0 is the base NLG model parameterized by INLINEFORM1 , and we have an initial INLINEFORM2 pre-trained with DA-utterance pairs INLINEFORM3 from a set INLINEFORM4 of high-resource source tasks. When we adapt INLINEFORM5 to some low-resource task INLINEFORM6 with DA-utterance pairs INLINEFORM7 , the fine-tuning process on INLINEFORM8 can be formulated as follows: DISPLAYFORM0
The parameter INLINEFORM0 will be used for initialization, and the model is further updated by new observations INLINEFORM1 . The size of INLINEFORM2 in low-resource NLG tasks is very small due to the high annotation cost, therefore, a good initialization parameter INLINEFORM3 learned from high-resource source tasks is crucial for the adaptation performance on new low-resource NLG tasks.
Meta NLG Tasks
To learn a INLINEFORM0 that can be easily fine-tuned on new low-resource NLG tasks, the idea of our Meta-NLG algorithm is to repeatedly simulate auxiliary Meta NLG tasks from INLINEFORM1 to mimic the fine-tuning process in Eq.( EQREF9 ). Then, we treat each Meta NLG task as a single meta training sample/episode, and utilize the meta optimization objective in the next section to directly learn from them.
Therefore, the first step is to construct a set of auxiliary Meta NLG tasks INLINEFORM0 to simulate the low-resource fine-tuning process. We construct a Meta NLG task INLINEFORM1 by:
DISPLAYFORM0
INLINEFORM0 and INLINEFORM1 of each INLINEFORM2 are two independent subsets of DA-utterance pairs from high-resource source data INLINEFORM3 . INLINEFORM4 and INLINEFORM5 correspond to meta-train (support) and meta-test (query) sets of a typical meta-learning or few-shot learning setup, and INLINEFORM6 is often referred to as a training episode. This meta setup with both INLINEFORM7 and INLINEFORM8 in one Meta NLG task allows our Meta-NLG algorithm to directly learn from different Meta NLG tasks. The usage of them will be elaborated later. Meta NLG tasks are constructed with two additional principles:
Task Generalization: To generalize to new NLG tasks, Meta NLG tasks follow the same modality as the target task. For example, if our target task is to adapt to DA-utterance pairs in a new domain, then DA-utterance pairs in each INLINEFORM0 are sampled from the same source domain. We also consider adapting to new DA types in later experiments. In this case, DA-utterance pairs in each INLINEFORM1 have the same DA type. This setting merges the goal of task generalization.
Low-resource Adaptation: To simulate the process of adapting to a low-resource NLG task, the sizes of both subsets INLINEFORM0 and INLINEFORM1 , especially INLINEFORM2 , are set small. Therefore, when the model is updated on INLINEFORM3 as a part of the later meta-learning steps, it only sees a small amount of samples in that task. This setup embeds the goal of low-resource adaptation.
Meta Training Objective
With the Meta NLG tasks defined above, we formulate the meta-learning objective of Meta-NLG as below: DISPLAYFORM0 DISPLAYFORM1
The optimization for each Meta NLG task INLINEFORM0 is computed on INLINEFORM1 referring to INLINEFORM2 . Firstly, the model parameter INLINEFORM3 to be optimized is updated on INLINEFORM4 by Eq.( EQREF14 ). This step mimics the process when INLINEFORM5 is adapted to a new low-resource NLG task INLINEFORM6 with low-resource observations INLINEFORM7 . We need to note that Eq.( EQREF14 ) is an intermediate step, and it only provides an adapted parameter ( INLINEFORM8 ) to our base model INLINEFORM9 to be optimized in each iteration. Afterwards, the base model parameterized by the updated parameter ( INLINEFORM10 ) is optimized on INLINEFORM11 using the meta objective in Eq.( EQREF13 ). This meta-learning optimization objective directly optimizes the model towards generalizing to new low-resource NLG tasks by simulating the process repeatedly with Meta NLG tasks in Eq.( EQREF13 ).
The optimization of Eq.( EQREF13 ) can be derived in Eq.( EQREF15 ). It involves a standard first-order gradient INLINEFORM0 as well as a gradient through another gradient INLINEFORM1 . Previous study BIBREF8 shows that the second term can be approximated for computation efficiency with marginal performance drop. In our case, we still use the exact optimization in Eq.( EQREF15 ) as we do not encounter any computation difficulties even on the largest NLG dataset so far. The second-order gradient is computed by a Hessian matrix INLINEFORM2 .
DISPLAYFORM0
DISPLAYFORM0
To better understand the meta objective, we include a standard multi-task learning (MTL) objective in Eq.( EQREF17 ). MTL learns through individual DA-utterance pairs from different high-resource NLG tasks INLINEFORM0 , and it does not explicitly learn to adapt to new low-resource NLG tasks. Figure FIGREF16 visually illustrates the differences with three high-resource source tasks INLINEFORM1 with optimal parameters INLINEFORM2 for each task. INLINEFORM3 is learned from individual DA-utterance pairs in INLINEFORM4 , while Meta-NLG repeatedly constructs auxiliary Meta NLG tasks INLINEFORM5 from INLINEFORM6 and directly learns INLINEFORM7 from them. As a result, INLINEFORM8 is closer to INLINEFORM9 and INLINEFORM10 (the optimal parameters of some new low-resource tasks, e.g, INLINEFORM11 and INLINEFORM12 ) than INLINEFORM13 . As we will see soon later, our meta optimization scheme results in a substantial gain in the final performance.
Algorithm 1 illustrates the process to learn INLINEFORM0 from INLINEFORM1 . We note that batches are at the level of Meta NLG tasks, not DA-utterances pairs. Fine-tuning Meta-NLG on a new low-resource NLG task with annotated DA-utterance pairs INLINEFORM2 uses the same algorithm parameterized by ( INLINEFORM3 ).
[1]
INLINEFORM0 INLINEFORM1
Initialize INLINEFORM0
INLINEFORM0 not converge
Simulate a batch of Meta NLG tasks INLINEFORM0 INLINEFORM1
Compute INLINEFORM0 in Eq.( EQREF14 )
Meta update INLINEFORM0 in Eq.( EQREF15 )
Return INLINEFORM0
Meta-NLG( INLINEFORM0 )
Baselines and Model Settings
We utilized the well-recognized semantically conditioned LSTM (SCLSTM BIBREF1 ) as the base model INLINEFORM0 . We used the default setting of hyperparameters (n_layer = 1, hidden_size = 100, dropout = 0.25, clip = 0.5, beam_width = 5). We implemented Meta-NLG based on the PyTorch SCLSTM implementation from BIBREF7 . As Meta-NLG is model-agnostic, it is applicable to many other NLG models.
We included different model settings as baseline:
Scratch-NLG: Train INLINEFORM0 with only low-resource target task data, ignoring all high-resource source task data.
MTL-NLG: Train INLINEFORM0 using a multi-task learning paradigm with source task data, then fine-tune on the low-resource target task.
Zero-NLG: Train INLINEFORM0 using multi-task learning (MTL) with source task data, then directly test on a target task without a fine-tuning step. This corresponds to a zero-shot learning scenario.
Supervised-NLG: Train INLINEFORM0 using MTL with full access to high-resource data from both source and target tasks. Its performance serves an upper bound using multi-task learning without the low-resource restriction.
Meta-NLG(proposed): Use Algorithm 1 to train INLINEFORM0 on source task data, then fine-tune on the low-resource target task.
For Meta-NLG, we set batch size to 5, and INLINEFORM0 and INLINEFORM1 . A single inner gradient update is used per meta update with Adam BIBREF25 . The size of a Meta NLG task is set to 400 with 200 samples assigned to INLINEFORM2 and INLINEFORM3 because the minimum amount of target low-resource samples is 200 in our later experiments. During fine-tuning on a low-resource target task, early-stop is conducted on a small validation set with size 200. The model is then evaluated on other DA-utterance pairs in the target task.
As in earlier NLG researches, we use the BLEU-4 score BIBREF26 and the slot error rate (ERR) as evaluation metrics. ERR is computed by the ratio of the sum of the number of missing and redundant slots in a generated utterance divided by the total number of slots in the DA. We randomly sampled target low-resource task five times for each experiment and reported the average score.
MultiWoz Dataset for NLG
We used a recently proposed large-scale multi-domain dialog dataset (MultiWOZ, BIBREF7 ). It is a proper benchmark for evaluating NLG components due to its domain complexity and rich linguistic variations. A visualization of DA types in different domains are given in Figure FIGREF25 , and slots in different domains are summarized in Table TABREF26 . The average utterance length is 15.12, and almost 60% of utterances have more than one dialogue act types or domains. A total of 69,607 annotated utterances are used, with 55,026, 7,291, 7,290 utterances for training, validation, and testing respectively.
Domain Adaptation
In this section, we tested when a NLG model is adapted to two types (near and far) of low-resource language domains. Experiment follows a leave-one-out setup by leaving one target domain for low-resource adaptation, while using the remainder domains as high-resource source training data. A target domain is a near-domain if it contains no domain-specific DA type but only domain-specific slots compared to the remainder domains. In contrast, a target domain containing both domain-specific DA types and slots is considered as a far-domain. Adapting to near-domains requires to capture unseen slots, while adapting to far-domains requires to learn new slots as well as new language patterns. Adaptation size is the number of DA-utterance pairs in the target domain used to fine-tune the NLG model. To test different low-resource degrees, we considered different adaptation sizes (1,000, 500, 200) in subsequent experiments.
Near-domain Adaptation: Figure FIGREF25 and Table TABREF26 show that “Attraction”, “Hotel”, “Restaurant”, and “Taxi”, are four near-domains compared to remainder domains. Only results for “Attraction” and “Hotel” are included due to page limit. The other two domains are also simpler with only one domain-specific slot. Several observations can be noted from results in Table TABREF27 . First, Using only source or target domain samples does not produce competitive performance. Using only source domain samples (Zero-NLG) performs the worst. It obtains very low BLEU-4 scores, indicating that the sentences generated do not match the linguistic patterns in the target domain. Using only low-resource target domain samples (Scratch-NLG) performs slightly better, yet still much worse than MTL-NLG and Meta-NLG. Second, Meta-NLG shows a very strong performance for this near-domain adaptation setting. It consistently outperforms MTL-NLG and other methods with very remarkable margins in different metrics and adaptation sizes. More importantly, it even works better than Supervised-NLG which is trained on high-resource samples in the target domain. Third, Meta-NLG is particularly strong in performance when the adaptation size is small. As the adaptation size decreases from 1,000 to 200, the performance of Scratch-NLG and MTL-NLG drops quickly, while Meta-NLG performs stably well. Both BLEU-4 and ERR even increase in “Hotel” domain when the adaptation size decreases from 500 to 200.
Far-domain Adaptation: In this experiment, we tested the performance when adapting to two low-resource far-domains (“Booking” and “Train”). Again, we can see that Meta-NLG shows very strong performance on both far-domains with different adaptation sizes. Similar observations can be made as in the previous near-domain adaptation experiments. Because far-domain adaptation is more challenging, Meta-NLG does not outperform Supervised-NLG, and the performance of Meta-NLG drops more obviously as the adaptation size decreases. Noticeably, “Train” is more difficult than “Booking” as the former contains more slots, some of which can only be inferred from the smallest “Taxi” domain. The improvement margin of Meta-NLG over MTL-NLG and other methods is larger on the more difficult “Train” domain than on the “Booking” domain.
Dialog Act (DA) Type Adaptation
It is also important and attractive for a task-oriented dialog system to adapt to new functions, namely, supporting new dialog acts that the system has never observed before. To test this ability, we left certain DA types out for adaptation in a low-resource setting. We chose “Recommend”, “Book” as target DA types, and we mimic the situation that a dialog system needs to add a new function to make recommendations or bookings for customers with a few number of annotated DA-utterance pairs. As presented in Table TABREF31 , results show that Meta-NLG significantly outperforms other baselines. Therefore, we can see that Meta-NLG is also able to adapt well to new functions that a dialog system has never observed before.
Adaptation Curve Analysis
To further investigate the adaptation process, we presented in Figure FIGREF34 the performance curves of MTL-NLG and Meta-NLG as fine-tuning epoch proceeds on the most challenging “Train” domain. The effect of meta-learning for low-resource NLG can be observed by comparing the two solid curves against the corresponding dashed curves. First, Meta-NLG adapts faster than MTL-NLG. We can see that the ERR of Meta-NLG (red-solid) decreases much more rapidly than that of MTL-NLG (red-dashed) , and the BLEU-4 score of Meta-NLG (purple-solid) also increases more quickly. The optimal BLEU-4 and ERR that MTL-NLG converges to can be obtained by Meta-NLG within 10 epochs. Second, Meta-NLG adapts better than MTL-NLG. As it can be seen, Meta-NLG achieves a much lower ERR and a higher BLEU-4 score when it converges, indicating that it found a better INLINEFORM0 of the base NLG model to generalize to the low-resource target domain.
Manual Evaluation
To better evaluate the quality of the generated utterances, we performed manual evaluation.
Metrics: Given a DA and a reference utterance in a low-resource target domain with adaptation size 500, two responses generated by Meta-NLG and MTL-NLG were presented to three human annotators to score each of them in terms of informativeness and naturalness (rating out of 3), and also indicate their pairwise preferences (Win-Tie-Lose) on Meta-NLG against MTL-NLG. Informativeness is defined as whether the generated utterance captures all the information, including multiple slots and probably multiple DA types, specified in the DA. Naturalness measures whether the utterance is plausibly generated by a human.
Annotation Statistics: Cases with identical utterances generated by two models were filtered out. We obtained in total 600 annotations on each individual metric for each target domain. We calculated the Fleiss’ kappa BIBREF27 to measure inter-rater consistency. The overall Fleiss’ kappa values for informativeness and naturalness are 0.475 and 0.562, indicating “Moderate Agreement”, and 0.637 for pairwise preferences, indicating “Substantial Agreement”.
Results: Scores of informativeness and naturalness are presented in Table TABREF36 . Meta-NLG outscores MTL-NLG in terms of both metrics on all four domains. Overall, Meta-NLG received significantly (two-tailed t-test, INLINEFORM0 ) higher scores than MTL-NLG. Results for pairwise preferences are summarized in Table TABREF37 . Even though there are certain amount of cases where the utterances generated by different models are nearly the same (Tie) to annotators, Meta-NLG is overall significantly preferred over MTL-NLG (two-tailed t-test, INLINEFORM1 ) across different target domains.
Case Study
Table TABREF32 shows three examples in the “Train” domain. The first sample shows that MTL-NLG fails to generate the domain-specific slot “Ticket”, instead, it mistakenly generates slots (“Leave” and “Arrive”) that are frequently observed in the low-resource adaptation set. In the second example, MTL-NLG failed to generate the domain-specific slot `Id” and another rare slot “Dest”, while Meta-NLG succeeded both. The last example shows similar results on a domain-specific dialog act type “Offer_Booked”, in which Meta-NLG successfully captured two domain-specific slots and a rare slot.
Conclusion
We propose a generalized optimization-based meta-learning approach Meta-NLG for the low-resource NLG task. Meta-NLG utilizes Meta NLG tasks and a meta-learning optimization procedure based on MAML. Extensive experiments on a new benchmark dataset (MultiWoz) show that Meta-NLG significantly outperforms other training procedures, indicating that our method adapts fast and well to new low-resource settings. Our work may inspire researchers to use similar optimization techniques for building more robust and scalable NLG components in task-oriented dialog systems.

Question: what are the evaluation metrics used?
Answer: BLEU-4, slot error rate

informativeness , naturalness 

BLEU-4, slot error rate, informativeness, naturalness

Question: what other training procedures were explored?
Answer: Scratch-NLG, MTL-NLG, Zero-NLG, Supervised-NLG

Scratch-NLG, MTL-NLG, Zero-NLG, Supervised-NLG