SlowGuess/ACL_2020_OCR huggingface dataset:
Libo Qin†, Xiao Xu†, Wanxiang Che†*, Yue Zhang‡, Ting Liu†
$^{\dagger}$ Research Center for Social Computing and Information Retrieval  
 $^{\dagger}$ Harbin Institute of Technology, China
$^{\ddagger}$ School of Engineering, Westlake University, China
$^{\ddagger}$ Institute of Advanced Technology, Westlake Institute for Advanced Study
$\dagger$  {lbqin, xxu, car,tliu} @ir.hit.edu.cn, yue.zhang@wias.org.cn
Recent studies have shown remarkable success in end-to-end task-oriented dialog system. However, most neural models rely on large training data, which are only available for a certain number of task domains, such as navigation and scheduling. This makes it difficult to scalable for a new domain with limited labeled data. However, there has been relatively little research on how to effectively use data from all domains to improve the performance of each domain and also unseen domains. To this end, we investigate methods that can make explicit use of domain knowledge and introduce a shared-private network to learn shared and specific knowledge. In addition, we propose a novel Dynamic Fusion Network (DF-Net) which automatically exploit the relevance between the target domain and each domain. Results show that our model outperforms existing methods on multi-domain dialogue, giving the state-of-the-art in the literature. Besides, with little training data, we show its transferability by outperforming prior best model by  $13.9\%$  on average.
Task-oriented dialogue systems (Young et al., 2013) help users to achieve specific goals such as restaurant reservation or navigation inquiry. In recent years, end-to-end methods in the literature usually take the sequence-to-sequence (Seq2Seq) model to generate a response from a dialogue history (Eric and Manning, 2017; Eric et al., 2017; Madotto et al., 2018; Wen et al., 2018; Gangi Reddy et al., 2019; Qin et al., 2019b; Wu et al., 2019a). Taking the dialogue in Figure 1 as an example, to answer the driver's query about the "gas station", the end-to-end dialogue system directly generates system response given the query and a corresponding knowledge base (KB).
Knowledge Base (KB)
Address	Distance	POI type	POI	Traffic info
5672 barringer street	5 miles	certain address	5672 barringer street	no traffic
200 Alester Ave	2 miles	gas station	Valero	road block nearby
899 Ames Ct	5 miles	hospital	Stanford Childrens Health	moderate traffic
481 Amaranta Ave	1 miles	parking garage	Palo Alto Garage R	moderate traffic
Dialogue
Driver	Address to the gas station.
Car	Valero is located at 200 Alester Ave.
Driver	OK, please give me directions via a route that avoids all heavy traffic.
Car	Since there is a road block nearby, I found another route for you and I sent it on your screen.
Figure 1: Example of a task-oriented dialogue that incorporates a knowledge base (KB) from the SMD dataset (Eric et al., 2017). Words with the same color refers queried entity from the KB. Better viewed in color.
Though achieving promising performance, end-to-end models rely on a considerable amount of labeled data, which limits their usefulness for new and extended domains. In practice, we cannot collect rich datasets for each new domain. Hence, it is important to consider methods that can effectively transfer knowledge from a source domain with sufficient labeled data to a target domain with limited or little labeled data.
Existing work can be classified into two main categories. As shown in Figure 2(a), the first strand of work (Eric and Manning, 2017; Eric et al., 2017; Madotto et al., 2018; Wu et al., 2019a) simply combines multi-domain datasets for training. Such methods can implicitly extract the shared features but fail to effectively capture domain-specific knowledge. As shown in Figure 2(b), The second strand of work (Wen et al., 2018; Qin et al., 2019b) trains model separately for each domain, which can better capture domain-specific features. However, those methods ignore shared knowledge between different domains (e.g. the location word exists in both schedule domain and navigation domain).
We consider addressing the limitation of existing work by modeling knowledge connections between domains explicitly. In particular, a simple baseline
*Email corresponding.
6344
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6344-6354
(d)
Figure 2: Methods for multi-domain dialogue. Previous work either trains a general model on mixed multi-domain mixed datasets (a), or on each domain separately (b). The basic shared-private framework is shown (c). Our proposed extension with dynamic fusion mechanism is shown (d).
to incorporate domain-shared and domain-private features is shared-private framework (Liu et al., 2017; Zhong et al., 2018; Wu et al., 2019b). Shown in Figure 2(c), it includes a shared module to capture domain-shared feature and a private module for each domain. The method explicitly differentiates shared and private knowledge. However, this framework still has two issues: (1) given a new domain with extremely little data, the private module can fail to effectively extract the corresponding domain knowledge. (2) the framework neglects the fine-grained relevance across certain subsets of domains. (e.g. schedule domain is more relevant to the navigation than to the weather domain.)
To address the above issues, we further propose a novel Dynamic Fusion Network (DF-Net), which is shown in Figure 2 (d). In contrast to the shared-private model, a dynamic fusion module (see §2.3) is further introduced to explicitly capture the correlation between domains. In particular, a gate is leveraged to automatically find the correlation between a current input and all domain-specific models, so that a weight can be assigned to each domain for extracting knowledge. Such a mechanism is adopted for both the encoder and the decoder, and also a memory module to query knowledge base features. Given a new domain with little or no training data, our model can still make the best use of existing domains, which cannot be achieved by the baseline model.
We conduct experiments on two public benchmarks, namely SMD (Eric et al., 2017) and MultiWOZ 2.1 (Budzianowski et al., 2018). Results show that our framework consistently and significantly outperforms the current state-of-the-art methods. With limited training data, our framework outperforms the prior best methods by  $13.9\%$  on average.
To our best of knowledge, this is the first work to effectively explore shared-private framework in
multi-domain end-to-end task-oriented dialog. In addition, when given a new domain which with few or zero shot data, our extended dynamic fusion framework can utilize fine-grained knowledge to obtain desirable accuracies, which makes it more adaptable to new domains.
All datasets and code are publicly available at: https://github.com/LooperXX/DF-Net.
2 Model Architecture
We build our model based on a seq2seq dialogue generation model (§2.1), as shown in Figure 3(a). To explicitly integrate domain awareness, as shown in Figure 3(b) we first propose to use a shared-private framework (§2.2) to learn shared and the corresponding domain-specific features. Next, we further use a dynamic fusion network (§2.3) to dynamically exploit the correlation between all domains for fine-grained knowledge transfer, which is shown in Figure 3(c). In addition, adversarial training is applied to encourage shared module generate domain-shared feature.
2.1 Seq2Seq Dialogue Generation
We define the Seq2Seq task-oriented dialogue generation as finding the system response  $Y$  according to the input dialogue history  $X$  and KB  $B$ . Formally, the probability of a response is defined as
$$
p (Y \mid X, B) = \prod_ {t = 1} ^ {n} p \left(y _ {t} \mid y _ {1}, \dots , y _ {t - 1}, X, B\right), \tag {1}
$$
where  $y_{t}$  represents an output token. In a vanilla Seq2Seq task-oriented dialogue system (Eric and Manning, 2017), a long short-term Memory network (LSTM, Hochreiter and Schmidhuber (1997)) is used to encode the dialogue history  $X = (x_{1}, x_{2}, \dots, x_{T})$  ( $T$  is the number of tokens in the dialogue history) to produce shared context-sensitive
6345
Figure 3: Workflow of our baseline and our proposed model.
hidden states  $\pmb {H} = (h_1,h_2,\dots,h_T)$  ..
$$
\boldsymbol {h} _ {i} = \operatorname {B i L S T M} _ {\text {e n c}} \left(\phi^ {e m b} \left(x _ {i}\right), \boldsymbol {h} _ {i - 1}\right), \tag {2}
$$
where  $\phi^{emb}(\cdot)$  represents the word embedding matrix. LSTM is also used to repeatedly predict outputs  $(y_{1},y_{2},\dots,y_{t - 1})$  by the decoder hidden states  $(h_{\mathrm{dec},1},h_{\mathrm{dec},2},\dots,h_{\mathrm{dec},t})$ . For the generation of  $y_{t}$ , the model first calculates an attentive representation  $h_{\mathrm{dec},t}^{\prime}$  of the dialogue history over the encoding representation  $H$ . Then, the concatenation of  $h_{\mathrm{dec},t}$  and  $h_{\mathrm{dec},t}^{\prime}$  is projected to the vocabulary space  $\mathcal{V}$  by  $U$ :
$$
\boldsymbol {o} _ {t} = \boldsymbol {U} \left[ h _ {\mathrm {d e c}, t}, h _ {\mathrm {d e c}, t} ^ {\prime} \right], \tag {3}
$$
where  $\mathbf{o}_t$  is the score (logit) for the next token generation. The probability of next token  $y_t \in \mathcal{V}$  is finally calculated as:
$$
p \left(y _ {t} \mid y _ {1}, \dots , y _ {t - 1}, X, B\right) = \operatorname {S o f t m a x} \left(\boldsymbol {o} _ {t}\right). \tag {4}
$$
Different from typical text generation with Seq2seq model, the successful conversations for task-oriented dialogue system heavily depend on accurate knowledge base (KB) queries. We adopt
the global-to-local memory pointer mechanism (GLMP) (Wu et al., 2019a) to query the entities in KB, which has shown the best performance. An external knowledge memory is proposed to store knowledge base (KB)  $B$  and dialogue history  $X$ . The KB memory is designed for the knowledge source while the dialogue memory is used for directly copying history words. The entities in external knowledge memory are represented in a triple format and stored in the memory module, which can be denoted as  $M = [B;X] = (m_{1},\ldots ,m_{b + T})$ , where  $m_{i}$  is one of the triplet of  $M$ ,  $b$  and  $T$  denotes the number of KB and dialog history respectively. For a  $k$ -hop memory network, the external knowledge is composed of a set of trainable embedding matrices  $C = (C^1,\dots,C^{k + 1})$ . We can query knowledge both in encoder and decoder process to enhance model interaction with knowledge module.
Query Knowledge in Encoder We adopt the last hidden state as the initial query vector:
$$
\boldsymbol {q} _ {\text {e n c}} ^ {1} = \boldsymbol {h} _ {T}. \tag {5}
$$
In addition, it can loop over  $k$  hops and compute the attention weights at each hop  $k$  using
$$
\boldsymbol {p} _ {i} ^ {k} = \operatorname {S o f t m a x} \left(\left(\boldsymbol {q} _ {\mathrm {e n c}} ^ {k}\right) ^ {\top} \boldsymbol {c} _ {i} ^ {k}\right), \tag {6}
$$
where  $\pmb{c}_i^k$  is the embedding in  $i^{th}$  memory position using the embedding matrix  $C^k$ . We obtain the global memory pointer  $G = (g_{1},\ldots ,g_{b + T})$  by applying  $g_{i}^{k} = \mathrm{Sigmoid}((\pmb{q}_{\mathrm{enc}}^{k})^{\top}\pmb{c}_{i}^{k})$ , which is used to filter the external knowledge for relevant information for decoding.
Finally, the model reads out the memory  $\pmb{o}^k$  by the weighted sum over  $c^{k + 1}$  and updates the query vector  $q_{\mathrm{enc}}^{k + 1}$ . Formally,
$$
\boldsymbol {o} _ {\text {e n c}} ^ {k} = \sum_ {i} \boldsymbol {p} _ {i} ^ {k} \boldsymbol {c} _ {i} ^ {k + 1}, \quad \boldsymbol {q} _ {\text {e n c}} ^ {k + 1} = \boldsymbol {q} _ {\text {e n c}} ^ {k} + \boldsymbol {o} _ {\text {e n c}} ^ {k}. \tag {7}
$$
$q_{\mathrm{enc}}^{k + 1}$  can be seen as the encoded KB information, and is used to initialized the decoder.
Query Knowledge in Decoder we use a sketch tag to denote all the possible slot types that start with a special token. (e.g., @address stands for all the Address). When a sketch tag is generated by Eq. 4 at  $t$  timestep, we use the concatenation of the hidden states  $h_{\mathrm{dec},t}$  and the attentive representation  $h_{\mathrm{dec},t}'$  to query knowledge, which is similar with
6346
Figure 4: The dynamic fusion layer for fusing domain-shared feature and domain-specific feature.
the process of querying knowledge in the encoder:
$$
\boldsymbol {q} _ {\mathrm {d e c}} ^ {1} = \left[ \boldsymbol {h} _ {\mathrm {d e c}, t}, \boldsymbol {h} _ {\mathrm {d e c}, t} ^ {\prime} \right], \tag {8}
$$
$$
p _ {i} ^ {k} = \operatorname {S o f t m a x} \left(\left(\boldsymbol {q} _ {\mathrm {d e c}} ^ {k}\right) ^ {\top} \boldsymbol {c} _ {i} ^ {k} g _ {i} ^ {k}\right). \tag {9}
$$
Here, we can treat  $P_{t} = (p_{1}^{k},\ldots ,p_{b + T}^{k})$  as the probabilities of queried knowledge, and select the word with the highest probability from the query result as the generated word.
2.2 Shared-Private Encoder-Decoder Model
The model in section 2.1 is trained over mixed multi-domain datasets and the model parameters are shared across all domains. We call such model as shared encoder-decoder model. Here, we propose to use a shared-private framework including a shared encoder-decoder for capturing domain-shared feature and a private model for each domain to consider the domain-specific features explicitly. Each instance  $X$  goes through both the shared and its corresponding private encoder-decoder.
Enhancing Encoder Given an instance along with its domain, the shared-private encoder-decoder generates a sequence of encoder vectors denoted as  $H_{\mathrm{enc}}^{(s,d)}$ , including shared and domain-specific representation from corresponding encoder:
$$
\begin{array}{l} \boldsymbol {H} _ {\text {e n c}} ^ {\{s, d \}} = \left(\boldsymbol {h} _ {\text {e n c}, 1} ^ {\{s, d \}}, \dots , \boldsymbol {h} _ {\text {e n c}, T} ^ {\{s, d \}}\right) \tag {10} \\ = \operatorname {B i L S T M} _ {\text {e n c}} ^ {\{s, d \}} (X). \\ \end{array}
$$
The final shared-specific encoding representation  $H_{\mathrm{enc}}^{f}$  is a mixture:
$$
\boldsymbol {H} _ {\text {e n c}} ^ {f} = \boldsymbol {W} _ {2} \left(\text {L e a k y R e L U} \left(\boldsymbol {W} _ {1} \left[ \boldsymbol {H} _ {\text {e n c}} ^ {s}, \boldsymbol {H} _ {\text {e n c}} ^ {d} \right]\right)\right). \tag {11}
$$
For ease of exposition, we define the shared-specific fusion function as:
$$
\operatorname {s h p r i v a t e}: \left(\boldsymbol {H} _ {\text {e n c}} ^ {s}, \boldsymbol {H} _ {\text {e n c}} ^ {d}\right)\rightarrow \boldsymbol {H} _ {\text {e n c}} ^ {f}. \tag {12}
$$
In addition, self-attention has been shown useful for obtaining context information (Zhong et al., 2018). Finally, we follow Zhong et al. (2018) to use self-attention over  $H_{\mathrm{enc}}^{f}$  to get context vector  $c_{\mathrm{enc}}^{f}$ . We replace  $h_{T}$  with  $c_{\mathrm{enc}}^{f}$  in Eq. 5. This makes our query vector combine the domain-shared feature with domain-specific feature.
Enhancing Decoder At  $t$  step of the decoder, the private and shared hidden state is:
$$
\boldsymbol {h} _ {\text {d e c}, t} ^ {\{s, d \}} = \operatorname {L S T M} _ {\text {d e c}, t} ^ {\{s, d \}} (X). \tag {13}
$$
We also apply the shared-specific fusion function to the hidden states and the mixture vector is:
$$
\operatorname {s h p r i v a t e}: \left(\boldsymbol {h} _ {\mathrm {d e c}, t} ^ {s}, \boldsymbol {h} _ {\mathrm {d e c}, t} ^ {d}\right)\rightarrow \boldsymbol {h} _ {\mathrm {d e c}, t} ^ {f}. \tag {14}
$$
Similarly, we obtain the fused attentive representation  $h_{\mathrm{dec},t}^{f'}$  by applying attention from  $h_{\mathrm{dec},t}^{f}$  over  $H_{\mathrm{enc}}^{f}$ . Finally, we replace  $[h_{\mathrm{dec},t}, h_{\mathrm{dec},t}^{'}]$  in Eq. 8 with  $[h_{\mathrm{dec},t}^{f}, h_{\mathrm{dec},t}^{f'}]$  which incorporates shared and domain-specific features.
2.3 Dynamic Fusion for Querying Knowledge
The shared-private framework can capture the corresponding specific feature, but neglects the fine-grained relevance across certain subsets of domains. We further propose a dynamic fusion layer to explicitly leverage all domain knowledge, which is shown in Figure 4. Given an instance from any domain, we first put it to multiple private encoder-decoder to obtain domain-specific features from all domains. Next, all domain-specific features are fused by a dynamic domain-specific feature fusion module, followed by a shared-specific feature fusion for obtaining shared-specific features.
Dynamic Domain-Specific Feature Fusion Given domain-specific features from all domains, a Mixture-of-Experts mechanism (MoE) (Guo et al., 2018) is adapted to dynamically incorporate all domain-specific knowledge for the current input in both encoder and decoder. Now, we give a detailed description on how to fuse the timestep  $t$  of decoding and the fusion process is the same to encoder. Given all domain feature representations in  $t$  decoding steps:  $\{h_{\mathrm{dec},t}^{d_i}\}_{i = 1}^{\mathbb{D}}$  where  $|\mathbb{D}|$  represents the number of domains, an expert gate  $E$  takes  $\{h_{\mathrm{dec},t}^{d_i}\}$  as input and outputs a softmax score  $\alpha_{t,i}$  that represents the degree of correlation between each domain and the
6347
current input token. We achieve this by a simple feedforward layer:
$$
\boldsymbol {\alpha} _ {t} = \operatorname {S o f t m a x} \left(\boldsymbol {W} * \boldsymbol {h} _ {\mathrm {d e c}, t} ^ {d} + \boldsymbol {b}\right). \tag {15}
$$
The final domain-specific feature vector is a mixture of all domain outputs, dictated by the expert gate weights  $\alpha_{t} = (\alpha_{t,1},\dots ,\alpha_{t,|\mathbb{D}|})$  , which can be written as  $h_{\mathrm{dec},t}^{d_f} = \sum_i\alpha_{t,i}h_{\mathrm{dec},t}^{d_i}$
During training, take the decoder for example, we apply the cross-entropy loss  $L_{\mathrm{dec}}^{moe}$  as the supervision signal for the expert gate to predict the domain of each token in the response, where the expert gate output  $\alpha_{t}$  can be treated as the  $t^{th}$  token's predicted domain probability distribution by multiple private decoder. Hence, the more accurate the domain prediction is, the more correct expert gets:
$$
L _ {\mathrm {d e c}} ^ {m o e} = - \sum_ {t = 1} ^ {n} \sum_ {i = 1} ^ {| \mathbb {D} |} \left(e _ {i} \cdot \log \left(\alpha_ {t, i} \mid \boldsymbol {\theta} _ {\boldsymbol {s}}, \boldsymbol {\theta} _ {\boldsymbol {d e c}} ^ {\boldsymbol {m}}\right)\right), \tag {16}
$$
where  $\theta_{s}$  represents the parameters of encoder-decoder model,  $\theta_{dec}^{m}$  represents the parameters of the MoE module (Eq. 15) in the decoder and  $e_i\in \{0,1\}$  represents whether the response with  $n$  tokens belongs to the domain  $d_{i}$ . Similarly, we can get the  $L_{\mathrm{enc}}^{moe}$  for the encoder and sum up them as:  $\mathcal{L}_{moe} = L_{\mathrm{enc}}^{moe} + L_{\mathrm{dec}}^{moe}$ .
$\mathcal{L}_{moe}$  is used to encourage samples from a certain source domain to use the correct expert, and each expert learns corresponding domain-specific features. When a new domain has little or no labeled data, the expert gate can automatically calculate the correlation between different domains with the target domain and thus better transfer knowledge from different source domains in both encoder and decoder module.
Shared-Specific Feature Fusion We directly apply shprivate operation to fuse shared and final domain-specific feature:
$$
\operatorname {s h p r i v a t e}: \left(\boldsymbol {h} _ {\mathrm {d e c}, t} ^ {s}, \boldsymbol {h} _ {\mathrm {d e c}, t} ^ {d _ {f}}\right)\rightarrow \boldsymbol {h} _ {\mathrm {d e c}, t} ^ {f}. \tag {17}
$$
Finally, we denote the dynamic fusion function as dynamic  $(\pmb{h}_{\mathrm{dec},t}^{s},\{\pmb{h}_{\mathrm{dec},t}^{d_{i}}\}_{i = 1}^{\lvert\mathbb{D}\rvert})$  . Similar to Section 2.2, we replace  $[\pmb{h}_{\mathrm{dec},t},\pmb{h}_{\mathrm{dec},t}^{\prime}]$  in Eq. 8 with  $[\pmb{h}_{\mathrm{dec},t}^{f},\pmb{h}_{\mathrm{dec},t}^{f^{\prime}}]$  . The other components are kept the same as the shared-private encoder-decoder framework.
Dataset	Domains	Train	Dev	Test
SMD	Navigate, Weather, Schedule	2,425	302	304
Multi-WOZ 2.1	Restaurant, Attraction, Hotel	1,839	117	141
Adversarial Training To encourage the model to learn domain-shared features, we apply adversarial learning on the shared encoder and decoder module. Following Liu et al. (2017), a gradient reversal layer (Ganin and Lempitsky, 2014) is introduced after the domain classifier layer. The adversarial training loss is denoted as  $\mathcal{L}_{adv}$ . We follow Qin et al. (2019a) and the final loss function of our Dynamic fusion network is defined as:
$$
\mathcal {L} = \gamma_ {b} \mathcal {L} _ {\text {b a s i c}} + \gamma_ {m} \mathcal {L} _ {\text {m o e}} + \gamma_ {a} \mathcal {L} _ {\text {a d v}}, \tag {18}
$$
where  $\mathcal{L}_{\text {basic }}$  keep the same as GLMP (Wu et al., 2019a),  $\gamma_{b}, \gamma_{m}$  and  $\gamma_{a}$  are hyper-parameters. More details about  $\mathcal{L}_{\text {basic }}$  and  $\mathcal{L}_{\text {adv }}$  can be found in appendix.
Two publicly available datasets are used in this paper, which include SMD (Eric et al., 2017) and an extension of Multi-WOZ 2.1 (Budzianowski et al., 2018) that we equip the corresponding KB to every dialogue. $^{1}$  The detailed statistics are also presented in Table 1. We follow the same partition as Eric et al. (2017), Madotto et al. (2018) and Wu et al. (2019a) on SMD and (Budzianowski et al., 2018) on Multi-WOZ 2.1.
The dimensionality of the embedding and LSTM hidden units is 128. The dropout ratio we use in our framework is selected from  $\{0.1, 0.2\}$  and the batch size from  $\{16, 32\}$ . In the framework, we adopt the weight typing trick (Wu et al., 2019a). We use Adam (Kingma and Ba, 2015) to optimize the parameters in our model and adopt the suggested hyper-parameters for optimization. All hyper-parameters are selected according to validation set. More details about hyper-parameters can be found in Appendix.
We compare our model with the following state-of-the-art baselines.
<sup>1</sup>The constructed datasets will be publicly available for further research.
6348
SMD	Multi-WOZ 2.1
Model	BLEU	F1	Navigate F1	Weather F1	Calendar F1	BLEU	F1	Restaurant F1	Attraction F1	Hotel F1
Mem2Seq (Madotto et al., 2018)	12.6	33.4	20.0	32.8	49.3	6.6	21.62	22.4	22.0	21.0
DSR (Wen et al., 2018)	12.7	51.9	52.0	50.4	52.1	9.1	30.0	33.4	28.0	27.1
KB-retriever (Qin et al., 2019b)	13.9	53.7	54.5	52.2	55.6	-	-	-	-	-
GLMP (Wu et al., 2019a)	13.9	60.7	54.6	56.5	72.5	6.9	32.4	38.4	24.4	28.1
Shared-Private framework (Ours)	13.6	61.7	56.3	56.5	72.8	6.6	33.8	39.8	26.0	28.3
Dynamic Fusion framework (Ours)	14.4*	62.7*	57.9*	57.6*	73.1*	9.4*	35.1*	40.9*	28.1*	30.6*
Table 2: Main results. The numbers with * indicate that the improvement of our framework over all baselines is statistically significant with  $p < 0.05$  under t-test.
Model	Entity F1 (%)
Test	Δ
Full model	62.7	-
w/o Domain-Shared Knowledge Transfer	59.0	3.7
w/o Dynamic Fusion Mechanism	60.9	1.8
w/o Multi-Encoder	61.0	1.7
w/o Multi-Decoder	58.9	3.8
w/o Adversarial Training	61.6	1.1
Table 3: Ablation tests on the SMD test set.
- Mem2Seq (Madotto et al., 2018): the model takes dialogue history and KB entities as input and uses a pointer gate to control either generating a vocabulary word or selecting an input as the output.
- DSR (Wen et al., 2018): the model leverages dialogue state representation to retrieve the KB implicitly and applies copying mechanism to retrieve entities from knowledge base while decoding.
- KB-retriever (Qin et al., 2019b): the model adopts a retriever module to retrieve the most relevant KB row and filter the irrelevant information for the generation process.
- GLMP (Wu et al., 2019a): the framework adopts the global-to-local pointer mechanism to query the knowledge base during decoding and achieve state-of-the-art performance.
For Mem2Seq, DSR, KB-retriever  $^{2}$ , we adopt the reported results from Qin et al. (2019b) and Wu et al. (2019a). For GLMP, we rerun their public code to obtain results on same datasets. $^{3}$