SlowGuess/ACL_2020_OCR huggingface dataset:
Yangming Li\*,, Han Li, Kaisheng Yao and Xiaolong Li
$\spadesuit$ Ant Financial Services Group, Alibaba Group
$\checkmark$ Harbin Institute of Technology
{pangmao.lym,kaisheng.yao,xl.li}  $@$  antfin.com
One great challenge in neural sequence labeling is the data sparsity problem for rare entity words and phrases. Most of test set entities appear only few times and are even unseen in training corpus, yielding large number of out-of-vocabulary (OOV) and low-frequency (LF) entities during evaluation. In this work, we propose approaches to address this problem. For OOV entities, we introduce local context reconstruction to implicitly incorporate contextual information into their representations. For LF entities, we present delexicalized entity identification to explicitly extract their frequency-agnostic and entity-type-specific representations. Extensive experiments on multiple benchmark datasets show that our model has significantly outperformed all previous methods and achieved new start-of-the-art results. Notably, our methods surpass the model fine-tuned on pre-trained language models without external resource.
In the context of natural language processing (NLP), the goal of sequence labeling is to assign a categorical label to each entity word or phrase in a text sequence. It is a fundamental area that underlies a range of applications including slot filling and named entity recognition. Traditional methods use statistical models. Recent approaches have been based on neural networks (Collobert et al., 2011; Mesnil et al., 2014; Ma and Hovy, 2016; Strubell et al., 2017; Li et al., 2018; Devlin et al., 2018; Liu et al., 2019a; Luo et al., 2020; Xin et al., 2018) and they have made great progresses in various sequence labeling tasks.
However, a great challenge to neural-network-based approaches is from the data sparsity problem (Augenstein et al., 2017). Specifically in the context of sequence labeling, the majority of entities
Frequency	Number	Percentage
= 0 (OOV)	1611	65.1%
= 1 (Low)	191	7.7%
< 10 (Low)	635	25.7%
> 20 (High)	117	4.7%
≥ 0 (Total)	2475	100.0%
Table 1: Number of occurrences of test set entities in the training set. OOV entities are those that have no occurrence (Frequency = 0) in the training set. Low frequency entities are those with fewer than ten occurrences (Frequency < 10). Percentages of entity occurrences are also shown. Data source is CoNLL-03.
in test dataset may occur in training corpus few times or are absent at all. In this paper, we refer this phenomenon particularly to rare entity problem. It is different from other types of data sparsity problems such as the lack of training data for low-resource language (Lin et al., 2018), as this rare entity problem is more related to a mismatch of entity distributions between training and test, rather than the size of training data. We present an example of the problem in Table 1. It shows that less than  $5\%$  of test set entities are frequently observed in the training set, and about  $65\%$  of test set entities are absent from the training set.
The rare entities can be categorized into two types: out-of-vocabulary (OOV) for those test set entities that are not observed in the training set, and low frequency (LF) for those entities with low frequency (e.g., fewer than 10) occurrences in the training set. Without proper processing, rare entities can incur the following risks when building a neural network. Firstly, OOV terms may act as noise for inference, as they lack lexical information from training set (Bazzi, 2002). Secondly, it is hard to obtain high-quality representations on LF entities (Gong et al., 2018). Lastly, high occurrences of OOV and LF entities expose distribution discrepancy.
6441
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6441-6451 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
ancy between training and test, which mostly leads to poor performances during test.
In general, there are two existing strategies attempting to mitigate the above issues: external resource and transfer learning. The external resource approach, for example (Huang et al., 2015; Li et al., 2018), uses external knowledge such as part-of-speech tags for NER or additional information from intent detection for slot filling. However, external knowledge such as part-of-speech tag is not always available for practical applications and open source taggers such as (Manning et al., 2014) may perform poorly for cross-domain annotations. Character or n-gram feature are mainly designed to deal with morphologically similar OOV words. The transfer learning approach, such as using ELMo (Peters et al., 2018) and BERT (Devlin et al., 2018), fine-tunes pre-trained models on the downstream task (Liu et al., 2019a). Nevertheless, it is not directly addressing problems such as entity distribution discrepancy between training and test. Moreover, our proposed methods surpass these methods without resorting to external resources nor large pre-trained language models.
This paper proposes novel techniques that enable sequence labeling models to achieve state-of-the-art performances without using external resource nor transfer learning. These are
- local context reconstruction (LCR), which is applied on OOV entities, and
- delexicalized entity identification (DEI), which is applied on LF entities.
Local context reconstruction enables OOV entities to be related to their contexts. One key point is applying variational autoencoder to model this reconstruction process that is typically a one-to-many generation process. Delexicalized entity identification aims at extracting frequency-agnostic and entity-type-specific representation, therefore reducing the reliance on high-frequency occurrence of entities<sup>1</sup>. It uses a novel adversarial training technique to achieve this goal. Both methods use an effective random entity masking strategy.
We evaluate the methods on sequence labeling tasks on several benchmark datasets. Extensive experiments show that the proposed methods significantly outperform previous models by a large margin. Detailed analysis indicates that the proposed
methods indeed alleviate the rare entity problem. Notably, without using any external knowledge nor pre-trained models, the proposed methods surpass the model that uses fine-tuned BERT.
Given an input sequence  $X = [x_{1}, x_{2}, \dots, x_{N}]$  with  $N$  tokens, the sequence labeling task aims at learning a functional mapping to obtain a target label sequence  $Y = [y_{1}, y_{2}, \dots, y_{N}]$  with equal length. In the following, we briefly introduce a typical method for sequence labeling and review related techniques we use in deriving our model.
2.1 Bidirectional RNN + CRF
Recurrent neural network (RNN) (Hochreiter and Schmidhuber, 1997) has been widely used for sequence labeling. The majority of high performance models use bidirectional RNN (Schuster and Paliwal, 1997) to encode input sequence  $X$  and conditional random field (CRF) (Lafferty et al., 2001) as a decoder to output  $Y$ .
The bidirectional RNN firstly embeds observation  $x_{i}$  at each position  $i$  to a continuous space  $\mathbf{x}_i$ . It then applies forward and backward operations on the whole sequence time-recursively as
$$
\left\{ \begin{array}{l} \overrightarrow {\mathbf {h}} _ {i} = \overrightarrow {f} (\mathbf {x} _ {i}, \overrightarrow {\mathbf {h}} _ {i - 1}) \\ \overleftarrow {\mathbf {h}} _ {i} = \overleftarrow {f} (\mathbf {x} _ {i}, \overleftarrow {\mathbf {h}} _ {i + 1}) \end{array} . \right. \tag {1}
$$
CRF computes the probability of a label sequence  $Y$  given  $X$  as
$$
\left\{ \begin{array}{c} \log p (Y | X) \propto \sum_ {i} \left(\mathbf {g} _ {i} \left[ y _ {i} \right] + \mathbf {G} \left[ y _ {i}, y _ {i + 1} \right]\right) \\ \mathbf {g} _ {i} = \mathbf {W} * (\overrightarrow {\mathbf {h}} _ {i} \oplus \overleftarrow {\mathbf {h}} _ {i}) \end{array} , \right. \tag {2}
$$
where  $\oplus$  denotes concatenation operation.  $\mathbf{G}$  and  $\mathbf{W}$  are learnable matrices. The sequence with the maximum score is the output of the model, typically obtained using Viterbi algorithm.
We use bidirectional RNN + CRF model, in particular, Bi-LSTM+CRF (Huang et al., 2015), as the baseline model in our framework and it is referred in the bottom part of Figure 1.
2.2 Variational Autoencoder
The above model, together with other encoder-decoder models (Sutskever et al., 2014; Bahdanau et al., 2014), learn deterministic and discriminative functional mappings. The variational auto-encoder (VAE) (Kingma and Welling, 2015; Rezende et al.,
This paper refers slots in slot filling tasks as entities for brevity, although their definitions are not equivalent.
6442
Figure 1: Overall framework to use local context reconstruction and delexicalized entity identification for neural sequence labeling. "[SOS]" and "[EOS]" are used for marking sequence beginning and sequence ending, respectively. The local context reconstruction is applied between any two successive entities, including the special entities. The delexicalized entity identification is applied for all entities except for the special entities.
2014; Bowman et al., 2015), on the other hand, is stochastic and generative.
Using VAE, we may assume a sequence  $\mathbf{x} = [\mathbf{x}_1,\mathbf{x}_2,\dots ,\mathbf{x}_N]$  is generated stochastically from a latent global variable  $\mathbf{z}$  with a joint probability of
$$
p (\mathbf {x}, \mathbf {z}) = p (\mathbf {x} | \mathbf {z}) * p (\mathbf {z}). \tag {3}
$$
where  $p(\mathbf{z})$  is the prior probability of  $\mathbf{z}$ , generally a simple Gaussian distribution, to keep the model from generating  $\mathbf{x}$  deterministically.  $p(\mathbf{x}|\mathbf{z})$  represents a generation density, usually modeled with a conditional language model with initial state of  $\mathbf{z}$ .
Maximum likelihood training of a model for Eq. (3) involves computationally intractable integration of  $\mathbf{z}$ . To circumvent this, VAE uses variational inference with variational distribution of  $\mathbf{z}$  coming from a Gaussian density  $q(\mathbf{z}|\mathbf{x}) = \mathcal{N}(\mu, \mathrm{diag}(\sigma^2))$ , with vector mean  $\mu$  and diagonal matrix variance  $\mathrm{diag}(\sigma^2)$  parameterized by neural networks. VAE also uses reparameterization trick to obtain latent variable  $\mathbf{z}$  as follows:
$$
\mathbf {z} = \mu + \sigma \odot \epsilon , \tag {4}
$$
where  $\epsilon$  is sampled from standard Gaussian distribution and  $\odot$  denotes element-wise product.
The evidence lower bound (ELBO) of the likelihood  $p(\mathbf{x})$  is obtained using Jensen's inequality  $E_{q(\mathbf{z}|\mathbf{x})}\log p(\mathbf{x},\mathbf{z})\leq \log p(\mathbf{x})$  as follows:
$$
\begin{array}{l} \mathcal {L} ^ {\mathrm {v a e}} (\mathbf {x}) = - \mathrm {K L} (q (\mathbf {z} | \mathbf {x}) | | p (\mathbf {z})) \\ - \mathrm {C E} (q (\mathbf {z} | \mathbf {x}) | p (\mathbf {x} | \mathbf {z})) ^ {\prime} \\ \end{array}
$$
where  $\mathrm{KL}(q||p)$  and  $\mathrm{CE}(q|p)$  respectively denote the Kullback-Leibler divergence and the cross-entropy between distribution  $q$  and  $p$ . ELBO can
be optimized by alternating between optimizations of parameters of  $q(\mathbf{z}|\mathbf{x})$  and  $p(\mathbf{x}|\mathbf{z})$ .
We apply VAE for local context reconstruction from slot/entity tags in Figure 1. This is a generation process that is inherently one-to-many. We observe that VAE is superior to the deterministic model (Bahdanau et al., 2014) in learning representations of rare entities.
2.3 Adversarial Training
Adversarial training (Goodfellow et al., 2014), originally proposed to improve robustness to noise in image, is later extended to NLP tasks such as text classification (Miyato et al., 2015, 2016) and learning word representation (Gong et al., 2018).
We apply adversarial training to learn better representations of low frequency entities via delexicalized entity identification in Figure 1. It has a discriminator to differentiate representations from the original low-frequency entities and the representations of the delexicalized entities. Training aims at obtaining representations that can fool the discriminator, therefore achieving frequency-agnosticities and entity-type-specificity.
3 The Model
We illustrate the overall framework of the proposed model in Figure 1. Its baseline sequence labeling module is described in Section 2.1. We describe the details of local context reconstruction in Sec. 3.1 and delexicalized entity identification in Sec. 3.2, together with an example to illustrate them in Figure 2. We denote parameters in Sec. 2.1 as  $\theta^{\mathrm{rnn}}$  and  $\theta^{\mathrm{emb}}$ , respectively, for its RNN and matrix to obtain
6443
(a) VAE for local context reconstruction.
(b) AT for delexicalized entity identification.
Figure 2: An example to illustrate local context reconstruction and delexicalized entity identification.
embedding. Parameters in Sec. 3.1 and Sec. 3.2 are each denoted as  $\theta^{\mathrm{lcr}}$  and  $\theta^{\mathrm{D}}$ .
3.1 Local Context Reconstruction
Contrary to the conventional methods that explicitly provide abundant lexical features from external knowledge, we implicitly enrich word representations with contextual information by training them to reconstruct their local contexts.
Masking Every entity word  $x_{i}$  in  $X$ , which is defined to be not associated with non-entity label "O", in sequence  $X$  is firstly randomly masked with OOV symbol "[UNK]" as follows:
$$
x _ {i} ^ {u} = \left\{ \begin{array}{c c} {[ \mathrm {U N K} ] ”} & {\text {i f} y _ {i} \neq “ \mathrm {O} ” \cap \epsilon > p} \\ x _ {i} & \text {o t h e r w i s e} \end{array} , \right. \tag {6}
$$
where constant  $p$  is a threshold and  $\epsilon$  is uniformly sampled between 0 and 1.
Forward Reconstruction In the forward reconstruction process, the forward pass of Eq. (1) is firstly applied on sequence  $X^{u} = [x_{1}^{u}, x_{2}^{u}, \dots, x_{N}^{u}]$  to obtain hidden states  $\vec{h}_{i}^{u}$ . Then, a forward span representation,  $\mathbf{m}_{jk}^{f}$ , of the local context between position  $k$  and  $j$  is obtained using RNN-minus feature (Wang and Chang, 2016) as follows:
$$
\mathbf {m} _ {j k} ^ {f} = \overrightarrow {\mathbf {h}} _ {k} ^ {u} - \overrightarrow {\mathbf {h}} _ {j} ^ {u}. \tag {7}
$$
To apply VAE to reconstruct the local context, the mean  $\mu$  and log-variance  $\log \sigma$  are firstly computed from the above representation as follows:
$$
\left\{ \begin{array}{c} \mu_ {j k} ^ {f} = \mathbf {W} _ {1} ^ {\mu} \tanh  \left(\mathbf {W} _ {0} ^ {\mu} \mathbf {m} _ {j k} ^ {f}\right) \\ \log \sigma_ {j k} ^ {f} = \mathbf {W} _ {1} ^ {\sigma} \tanh  \left(\mathbf {W} _ {0} ^ {\sigma} \mathbf {m} _ {j k} ^ {f}\right) \end{array} , \right. \tag {8}
$$
where  $\mathbf{W}_*$  are all learnable matrices. Then, the reparameterization trick in Eq. (4) is applied on  $\mu_{jk}^{f}$  and  $\sigma_{jk}^{f} = \exp (\log \sigma_{jk}^{f})$  to obtain a global latent variable  $\mathbf{z}_{jk}^{f}$  for the local context.
To generate the  $i$ -th word in the local context sequence  $[x_{j+1}, x_{j+2}, \dots, x_{k-1}]$ , we first apply a RNN-decoder with its initial hidden state from the latent variable  $\mathbf{z}_{jk}^{f}$  and the first observation from the embedding of “[SOS” symbol to recursively obtain hidden state  $\vec{\mathbf{r}}_i^f$  as follows:
$$
\overrightarrow {\mathbf {r}} _ {i} ^ {f} = \overrightarrow {f} (\mathbf {x} _ {i}, \overrightarrow {\mathbf {r}} _ {i - 1} ^ {f}), \tag {9}
$$
This RNN-decoder specifically does parameter sharing with the forward pass RNN-encoder in Eq. (1). We then use softmax to compute the distribution of word at position  $l$  as
$$
\vec {P} _ {i} ^ {\text {v a e}} = \operatorname {S o f t m a x} \left(\mathbf {W} _ {g} ^ {f} * \mathbf {r} _ {i} ^ {f}\right), \tag {10}
$$
where  $\mathbf{W}_q^f$  is a learnable matrix.
Lastly, we compute KL distance and cross-entropy for length- $L$  local context sequence in Eq. (5) as follows:
$$
\left\{ \begin{array}{l} \mathrm {K L} _ {j k} ^ {f} = \sum_ {d} \zeta \left(\mu_ {j k} ^ {f} [ d ], \sigma_ {j k} ^ {f} [ d ]\right), \\ \mathrm {C E} _ {j k} ^ {f} = - \frac {1}{L} \sum_ {i} \log \left(\overrightarrow {P} _ {i} ^ {\text {v a e}} [ x _ {i} ]\right), \\ \overrightarrow {\mathcal {L}} _ {j k} ^ {\text {v a e}} = - \mathrm {K L} _ {j k} ^ {f} - \mathrm {C E} _ {j k} ^ {f}, \end{array} \right. \tag {11}
$$
where  $d$  denotes hidden dimension index and the closed form KL divergence  $\zeta$  is defined as
$$
\zeta (\mu , \sigma) = \mu^ {2} + \sigma - (1 + \log \sigma). \tag {12}
$$
Backward Reconstruction Same as the forward reconstruction, the backward reconstruction is applied on non-adjacent successive entities. The backward pass of Eq. (1) is firstly applied on the entity-masked sequence  $X^u$ . Once the backward span representation,  $\mathbf{m}_{kj}^b$ , of the local context between position  $k$  and  $j$  is obtained as  $\mathbf{m}_{kj}^b = \overleftarrow{\mathbf{h}}_j^u - \overleftarrow{\mathbf{h}}_k^u$ , the same procedures of the above described forward reconstruction are conducted, except using
the backward RNN-encoder  $\overleftarrow{f} (\cdot)$  in lieu of the forward RNN-encoder in Eq. (9).
The objective for local context reconstruction is
$$
\mathcal {J} ^ {\mathrm {v a e}} (X; \theta^ {\mathrm {l c r}}, \theta^ {\mathrm {r n n}}) = \max  _ {\theta^ {\mathrm {l c r}}, \theta^ {\mathrm {r n n}}} \sum_ {j k} \overrightarrow {\mathcal {L}} _ {j k} ^ {\mathrm {v a e}} + \overleftarrow {\mathcal {L}} _ {j k} ^ {\mathrm {v a e}}, \tag {13}
$$
which is to maximize the ELBO w.r.t. parameters  $\theta^{\mathrm{lcr}}$  and  $\theta^{\mathrm{rn}}$ .
3.2 Delexicalized Entity Identification
For low-frequency entities, the delexicalized entity identification aims at obtaining frequency-agnostic and entity-type-specific representations.
Delexicalization We first randomly substitute entity words in input sequence  $X$  with their corresponding labels as
$$
x _ {i} ^ {d} = \left\{ \begin{array}{l l} y _ {i} & \text {i f} y _ {i} \neq “ \mathrm {O} ” \cap \epsilon > p \\ x _ {i} & \text {o t h e r w i s e} \end{array} , \right. \tag {14}
$$
where  $p$  is a threshold and  $\epsilon$  is uniformly sampled from [0, 1]. We refer this to delexicalization (Wen et al., 2015), but insert randomness in it.
Representation for Identification To obtain representation to identify whether an entity has been delexicalized to its label, we first use forward and backward RNN-encoders in Eq. (1) on the sentence  $X^{d} = [x_{1}^{d}, x_{2}^{d}, \dots, x_{N}^{d}]$  and obtain hidden states  $\overleftarrow{\mathbf{h}}_{i}^{d}$  and  $\overrightarrow{\mathbf{h}}_{i}^{d}$  for each position  $i$ . Their concatenation is  $\mathbf{h}_{i}^{d} = \overleftarrow{\mathbf{h}}_{i}^{d} \oplus \overrightarrow{\mathbf{h}}_{i}^{d}$ . For position  $i$  in the original sequence without delexicalization, its concatenated hidden state  $\mathbf{h}_{i} = \overleftarrow{\mathbf{h}}_{i} \oplus \overrightarrow{\mathbf{h}}_{i}$ .
For an entity with a span from position  $j$  to  $k$ , its representation  $\mathbf{e}_{jk}^{d}$  is obtained from the following average pooling
$$
\mathbf {e} _ {j k} ^ {d} = \frac {1}{k - j + 1} \sum_ {i} \mathbf {h} _ {i} ^ {d}. \tag {15}
$$
Average pooling is also applied on  $\mathbf{h}_i$ s to obtain  $\mathbf{e}_{jk}$  for the original entity with that span.
Discriminator A multi-layer perceptron (MLP) based discriminator with parameter  $\theta^{\mathrm{D}}$  is employed to output a confidence score in [0, 1], indicating the probability of the lexicalization of an entity; i.e.,
$$
\left\{ \begin{array}{l} p _ {j k} ^ {d} = \sigma \left(\mathbf {v} _ {d} ^ {T} \tanh  \left(\mathbf {W} _ {d} * \mathbf {e} _ {j k} ^ {d}\right)\right) \\ p _ {j k} = \sigma \left(\mathbf {v} _ {d} ^ {T} \tanh  \left(\mathbf {W} _ {d} * \mathbf {e} _ {j k}\right)\right), \end{array} \right. \tag {16}
$$
where parameters  $\mathbf{v}_d$  and  $\mathbf{W}_d$  are learnable and  $\sigma(x)$  is Sigmoid function  $\frac{1}{1 + \exp(-x)}$ .
Following the principle of adversarial training, we develop the following minimax objective to train RNN model  $\theta^{\mathrm{rn}}$  and the discriminator  $\theta^{\mathrm{D}}$ :
$$
\mathcal {J} ^ {\mathrm {a t}} (X, Y; \theta^ {\mathrm {D}}, \theta^ {\mathrm {r n n}}) = \tag {17}
$$
$$
\min  _ {\theta^ {\mathrm {D}}} \max  _ {\theta^ {\mathrm {r n n}}} \sum_ {j k} \log (p _ {j k}) + \log (1 - p _ {j k} ^ {d}),
$$
which aims at fooling a strong discriminator  $\theta^{\mathrm{D}}$  with parameter  $\theta^{\mathrm{rnn}}$  optimized, leading to frequency-agnosticities.
4 Training Algorithm
Notice that the model has three modules with their own objectives. We update their parameters jointly using Algorithm 1. The algorithm first improves discriminator  $\theta^{\mathrm{D}}$  to identify delexicalized items. It then updates  $\theta^{\mathrm{lcr}}$  and  $\theta^{\mathrm{rn}}$  with joint optimization  $\mathcal{I}^{\mathrm{vae}}$  and  $\mathcal{J}^{\mathrm{at}}$  to improve  $\theta^{\mathrm{rn}}$  to fool the discriminator. As VAE optimization of  $\mathcal{I}^{\mathrm{vae}}$  has posterior collapse problem, we adopt KL cost annealing strategy and word dropout techniques (Bowman et al., 2015). Finally, the algorithm updates both of  $\theta^{\mathrm{rnn}}$  and  $\theta^{\mathrm{emb}}$  in Bi-LSTM+CRF by gradient ascent according to Eq. (2). Note that  $\theta^{\mathrm{lcr}}$  shares the same parameters with  $\theta^{\mathrm{rnn}}$  and  $\theta^{\mathrm{emb}}$ .
During experiments, we also find it is beneficial to have a few epochs of pretraining of parameters  $\theta^{\mathrm{rnn}}$  and  $\theta^{\mathrm{emb}}$  with optimization of Eq. (2).
This section compares the proposed model against state-of-the-art models on benchmark datasets.
Slot Filling We use available ATIS dataset (Turel et al., 2010) and SNIPS dataset (Coucke et al.,
6445
Models	ATIS	SNIPS	CoNLL-03
Lample et al. (2016)	Bi-LSTM + CRF w/ char	95.17	93.71	90.94
Liu et al. (2018)	LM-LSTM-CRF	95.33	94.07	91.24
Liu et al. (2019a)	GCDT	95.98	95.03	91.96
Qin et al. (2019)	Stack-propagation†	95.9	94.2	-
Liu et al. (2019b)	CM-Net†	95.82	97.15	-
This Work	Bi-LSTM + CRF	95.02	93.37	90.11
w/ external resources‡	95.67	94.76	91.04
w/ BERT fine-tuned embedding*	95.94	96.15	92.53
w/ Proposed Methods	96.01	97.20	92.67
Table 2: Sequence labeling test results of baselines and the proposed model on benchmark datasets. * refers to fine tuning on pretrained large models. † refers to using multi-task learning. ‡ refers to adopting external resources. The improvements over all prior methods are statistically significant with  $p < 0.01$  under t-test.
2018). Meanwhile, we follow the same setup as (Goo et al., 2018; Qin et al., 2019).
NER We use the public CoNLL-03 dataset (Sang and Meulder, 2003) as in (Huang et al., 2015; Lample et al., 2016; Liu et al., 2019a). The dataset is tagged with four named entity types, including PER, LOC, ORG, and MISC.
Baselines We compare the proposed model with five types of methods: 1) strong baseline (Lample et al., 2016) use character embedding to improve sequence tagger; 2) recent state-of-the-art models for slot filling (Qin et al., 2019; Liu et al., 2019b) that utilize multi-task learning to incorporate additional information from intent detection; 3) recent state-of-the-art models, including Liu et al. (2018) and Liu et al. (2019a), for NER; 4) Bi-LSTM + CRF model augmented with external resources, (i.e., POS tagging using Stanford  $\mathrm{Parser}^2$ ); and 5) Bi-LSTM + CRF model with word embedding from fine-tuned BERT $_{\text{LARGE}}$  (Devlin et al., 2018). Results are reported in F1 scores.
We follow most of the baseline performances reported in (Lample et al., 2016; Liu et al., 2019b; Qin et al., 2019; Liu et al., 2019a) and rerun the open source toolkit NCRFpp<sup>3</sup>, LM-LSTM-CRF<sup>4</sup>, and GCDT<sup>5</sup> on slot filling tasks<sup>6</sup>.
Implementation Details We use the same configuration setting for all datasets. The hidden dimensions are set as 500. We apply dropout to hid