Venue: NAACL 2018

# Label-aware Double Transfer Learning for Cross-Specialty

Medical Named Entity Recognition

 Zhenghui Wang\({}^{\dagger}\), Yanru Qu\({}^{\dagger}\), Liheng Chen\({}^{\dagger}\), Jian Shen\({}^{\dagger}\), Weinan Zhang\({}^{\dagger}\)1

**Shaodian Zhang\({}^{\dagger\ddagger}\), Yimei Gao\({}^{\dagger}\), Gen Gu\({}^{\dagger}\), Ken Chen\({}^{\dagger}\), Yong Yu\({}^{\dagger}\) \({}^{\dagger}\)APEX Data and Knowledge Management Lab, Shanghai Jiao Tong University \({}^{\ddagger}\)Synyi LLC. {felixwzh,wnzhang,shaodian}@apex.sjtu.edu.cn chen.ken@synyi.com**

Footnote 1: Weinan Zhang is the corresponding author.

###### Abstract

We study the problem of named entity recognition (NER) from electronic medical records, which is one of the most fundamental and critical problems for medical text mining. Medical records which are written by clinicians from different specialties usually contain quite different terminologies and writing styles. The difference of specialties and the cost of human annotation makes it particularly difficult to train a universal medical NER system. In this paper, we propose a label-aware double transfer learning framework (LaDTL) for cross-speciality NER, so that a medical NER system designed for one specialty could be conveniently applied to another one with minimal annotation efforts. The transferability is guaranteed by two components: (i) we propose label-aware MMD for feature representation transfer, and (ii) we perform parameter transfer with a theoretical upper bound which is also label aware. We conduct extensive experiments on 12 cross-specialty NER tasks. The experimental results demonstrate that La-DTL provides consistent accuracy improvement over strong baselines. Besides, the promising experimental results on non-medical NER scenarios indicate that La-DTL is potential to be seamlessly adapted to a wide range of NER tasks.

## 1 Introduction

The development of hospital information system and medical informatics drives the leverage of various medical data for a more efficient and intelligent medical care service. Among many kinds of medical data, electronic health records (EHRs) are one of the most valuable and informative data as they contain detailed information about the patients and the clinical practices. EHRs are essential to many intelligent clinical applications, such as hospital quality control and clinical decision support systems Wu et al. (2015). Most of EHRs are recorded in an unstructured form, i.e., natural language. Hence, extracting structured information from EHRs using natural language processing (NLP), e.g., named entity recognition (NER) and entity linking, plays a fundamental role in medical informatics Zhang and Elhadad (2013). In this paper, we focus on medical NER from EHRs, which is a fundamental task and is widely studied in the research community Nadeau and Sekine (2007); Uzuner et al. (2011).

In practice, the difficulty of building a universally robust and high-performance medical NER system lies in the variety of medical terminologies and expressions among different departments of specialties and hospitals. However, building separate NER systems for so many specialties comes with a prohibitively high cost. The data privacy issue further discourages the sharing of the data across departments or hospitals, making it more difficult to train a canonical NER system to be applied everywhere. This raises a natural question: if we have sufficient annotated EHRs data in one _source_ specialty, can we distill the knowledge and transfer it to help training models in a related _target_ specialty with few annotations? By transferring the knowledge we can achieve higher performance in target specialties with lower annotation cost and bypass the data sharing concerns. This is commonly referred to as _transfer learning_Pan and Yang (2010).

Current state-of-the-art transfer learning methods for NER are mainly based on deep neural networks, which perform an end-to-end training to distill sequential dependency patterns in the natural language Ma and Hovy (2016); Lample et al. (2016). These transfer learning methods include (i) feature representation transfer Peng and Dredze (2017); Kulkarni et al. (2016), which normally lever

ages deep neural networks to learn a close feature mapping between the source and target domains, and (ii) parameter transfer Murthy et al. (2016); Yang et al. (2017), which performs parameter sharing or joint training to get the target-domain model parameters close to those of the source-domain model. To the best of our knowledge, there is no previous literature working on transfer learning for NER in the medical domain, or even in a larger scope, i.e., medical natural language processing.

In this paper, we propose a novel NER transfer learning framework, namely label-aware double transfer learning (La-DTL): (i) We leverage bidirectional long-short term memory (Bi-LSTM) network Graves and Schmidhuber (2005) to automatically learn the text representations, based on which we perform a label-aware feature representation transfer. We propose a variant of maximum mean discrepancy (MMD) Gretton et al. (2012), namely label-aware MMD (La-MMD), to explicitly reduce the domain discrepancy of feature representations of tokens with the same label between two domains. (ii) Based on the learned feature representations from Bi-LSTM, two conditional random field (CRF) models are performed for sequence labeling for source and target domain separately, where parameter transfer learning is performed. Specifically, an upper bound of KL divergence between the source and target domain's CRF label distributions is added over the emission and transition matrices across the source and target CRF models to explore the shareable parts of the parameters. Both (i) and (ii) have a label-aware characteristic, which will be discussed later. We further argue that label-aware characteristic is crucial for transfer learning in sequence labeling problems, e.g., NER, because only when the corresponding labels are matched, can the "similar" contexts (i.e. feature representation) and model parameters be efficiently borrowed to improve the label prediction.

Extensive experiments are conducted on 12 cross-specialty medical NER tasks with real-world EHRs. The experimental results demonstrate that La-DTL provides consistent accuracy improvement over strong baselines, with overall 2.62% to 6.70% absolute F1-score improvement over the state-of-the-art methods. Besides, the promising experimental results on other two non-medical NER scenarios indicate that La-DTL has the potential to be seamlessly adapted to a wide range of NER tasks.

## 2 Related Works

**Named Entity Recognition** (NER) is fundamental in information extraction area which aims at automatic detection of named entities (e.g., person, organization, location and geo-political) in free text Marrero et al. (2013). Many high-level applications such as entity linking Moro et al. (2014) and knowledge graph construction Hachey et al. (2011) could be built on top of an NER system. Traditional high-performance approaches include conditional random fields models (CRFs) Lafferty et al. (2001), maximum entropy Markov models (MEMMs) McCallum et al. (2000) and hidden Markov models (HMMs). Recently, many neural network-based models have been proposed Collobert et al. (2011); Chiu and Nichols (2016); Ma and Hovy (2016); Lample et al. (2016), in which few feature engineering works are needed to train a high-performance NER system. The architecture of those neural network-based models are similar, where different neural networks (LSTMs, CNNs) at different levels (char- and word-level) are applied to learn feature representations, and on top of neural networks, a CRF model is employed to make label predictions.

**Transfer Learning** distills knowledge from a source domain to help create a high-performance learner for a target domain. Transfer learning algorithms are mainly categorized into three types, namely instance transfer, feature representation transfer and parameter transfer Pan and Yang (2010). Instance transfer normally samples or reweights source-domain samples to match the distribution of the target domain Chen et al. (2011); Chu et al. (2013). Feature representation transfer typically learns a feature mapping which projects source and target domain data simultaneously onto a common feature space following similar distributions Zhuang et al. (2015); Long et al. (2015); Shen et al. (2017). Parameter transfer normally involves a joint or constrained training for the models on source and target domains, usually introduce connections between source target parameters via sharing Srivastava and Salakhutdinov (2013), initialization Perlich et al. (2014), or intermodel parameter penalty schemes Zhang et al. (2016).

**Transfer Learning for NER** Training a high-performance NER system requires expensive and 

time-consuming manually annotated data. But sufficient labeled data is critical for the generalization of an NER system, especially for neural network-based models. Thus, transfer learning for NER is a practically important problem. The first group of methods focuses on sharing model parameters but they differ in the training schemes. He and Sun (2017) proposed to train the parameter-shared model with source and target data jointly, while the learning rates for sentences from source domain are re-weighted by the similarity with target domain corpus. Yang et al. (2017) proposed a family of frameworks which share model parameters in hierarchical recurrent networks to handle cross-application, cross-lingual, and cross-domain transfer in sequence labeling tasks. Differently, Lee et al. (2017) first trained the model with source domain data and then fine-tuned the model with little annotated target domain data.

Domain adaptation method has been well studied in NER scenarios such as using distributed word representations (Kulkarni et al., 2016) and leveraging rule-based annotators (Chiticariu et al., 2010). Multi-task learning has also been studied to improve performance in multiple NER tasks by transferring meaningful knowledge from other tasks (Collobert et al., 2011; Peng and Dredze, 2016). To take the advantages of both domain adaptation and multi-task learning, Peng and Dredze (2017) proposed a multi-task domain adaptation model.

## 3 Preliminaries

This section briefly introduces bidirectional LSTM, conditional random field and maximum mean discrepancy, which are the building blocks of our transfer learning framework.

**Bidirectional LSTM** Recurrent neural networks (RNNs) are widely used in NLP tasks for their great capability to capture contextual information in sequence data. A widely used variant of RNNs is long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997), which incorporates input and forget gates to capture both long and short term dependencies. Furthermore, it will be beneficial if we process the sequence in not only a forward but also a backward way. Thus, bidirectional LSTM (Bi-LSTM) was employed in many previous works (Chiu and Nichols, 2016; Ma and Hovy, 2016; Lample et al., 2016) to capture bidirectional information in a sequence. More specifically, for token \({\bf x}_{t}\) (embedding vector) at timestep \(t\) in sequence \({\bf X}=({\bf x}_{1},{\bf x}_{2},...,{\bf x}_{n})\), the \(\theta_{b}\)-parameterized Bi-LSTM recurrently updates hidden vectors \({\bf h}^{\rightarrow}_{t}=G^{f}_{\theta_{b}}({\bf X},{\bf h}^{\leftarrow}_{ t-1})\) and \({\bf h}^{\leftarrow}_{t}=G^{b}_{\theta_{b}}({\bf X},{\bf h}^{\leftarrow}_{t+1})\) produced by a forward LSTM and a backward one, respectively. Then we concatenate \({\bf h}^{\rightarrow}_{t}\) and \({\bf h}^{\leftarrow}_{t}\) to \({\bf h}_{t}\) as the final hidden vector produced by Bi-LSTM:

\[{\bf h}_{t}={\bf h}^{\rightarrow}_{t}\oplus{\bf h}^{\leftarrow}_{t}.\]

The representations learned from Bi-LSTM for sequence \({\bf X}\) is thus denoted as \({\bf H}=({\bf h}_{1},{\bf h}_{2},...,{\bf h}_{n})\).

**Conditional Random Field** The goal of NER is to detect named entities in a sequence \({\bf X}\) by predicting a sequence of labels \({\bf y}=(y_{1},y_{2},...,y_{n})\). Conditional random field (CRF) is widely used to make joint labeling of the tokens in a sequence (Lafferty et al., 2001).

Recently, Lample et al. (2016) proposed to build a CRF layer on top of a Bi-LSTM so that the automatically learned feature representation \({\bf H}=({\bf h}_{1},{\bf h}_{2},...,{\bf h}_{n})\) of the sequence can be directly fed into the CRF for sequence labeling. For a sequence of labels \({\bf y}\), given the hidden vector sequence \({\bf H}\), we define its \(\theta_{c}\)-parametrized score function \(s_{\theta_{c}}({\bf H},{\bf y})\) as:

\[s_{\theta_{c}}({\bf H},{\bf y})=\sum_{i=1}^{n}{\bf E}_{i,y_{i}}+\sum_{i=1}^{n-1 }{\bf A}_{y_{i},y_{i+1}},\]

where \({\bf E}\) is the emission score matrix of size \(n\times m\) (\(m\) is the number of unique labels), and is computed by \({\bf E}={\bf HW}\) where \({\bf W}\) is the label emission parameter matrix; \({\bf A}\) is the label transition parameter matrix; thus \(\theta_{c}=\left\{{\bf W},{\bf A}\right\}\). We then define the conditional probability of label sequence \({\bf y}\) given \({\bf H}\) by a softmax over all possible label sequences in set \({\cal Y}({\bf H})\) as:

\[p_{\theta_{c}}({\bf y}|{\bf H})= \exp\{s_{\theta_{c}}({\bf H},{\bf y})\}/Z({\bf H}) \tag{1}\] \[= \exp\{s_{\theta_{c}}({\bf H},{\bf y})\}\Big{/}\!\sum_{{\bf y}^{ \prime}\in{\cal Y}({\bf H})}\exp\{s_{\theta_{c}}({\bf H},{\bf y}^{\prime})\},\]

where \(\theta_{c}\) is omitted for simplification in the following part. The training objective in the CRF layer is to maximize the log-likelihood \(\max_{\theta_{c}}\log p({\bf y}|{\bf H})\). In the label prediction phase, we give the output label sequence \({\bf y}^{*}\) with the highest conditional probability \({\bf y}^{*}=\arg\max_{{\bf y}^{\prime}\in{\cal Y}({\bf H})}p({\bf y}^{\prime}|{ \bf H})\) by dynamic programming (Sutton et al., 2012).

**Maximum Mean Discrepancy** Maximum Mean Discrepancy (Gretton et al., 2012) is a non 

parametric test statistic to measure the distribution discrepancy in terms of the distance between the kernel mean embeddings of two distributions \(p\) and \(q\). The MMD is defined in particular function spaces that witness the difference in distributions

\[\text{MMD}(\mathcal{F},p,q)=\sup_{f\in\mathcal{F}}(\mathbb{E}_{x\sim p}[f(x)]- \mathbb{E}_{y\sim q}[f(y)]).\]

By defining the function class \(\mathcal{F}\) as the unit ball in a universal Reproducing Kernel Hilbert Space (RKHS), denoted by \(\mathcal{H}\), it holds that \(\text{MMD}[\mathcal{F},p,q]=0\) if and only if \(p=q\). And then given two sets of samples \(X=\left\{x_{1},...,x_{m}\right\}\) and \(Y=\left\{y_{1},...,y_{n}\right\}\) independently and identically distributed (i.i.d.) from \(p\) and \(q\) on the data space \(\mathcal{X}\), the empirical estimate of MMD can be written as the distance between the empirical mean embeddings after mapping to RKHS

\[\text{MMD}(X,Y)=\Big{\|}\frac{1}{m}\sum_{i=1}^{m}\phi(x_{i})-\frac{1}{n}\sum_{ j=1}^{n}\phi(y_{j})\Big{\|}_{\mathcal{H}}, \tag{2}\]

where \(\phi(\cdot):\mathcal{X}\rightarrow\mathcal{H}\) is the nonlinear feature mapping that induces \(\mathcal{H}\).

## 4 Methodology

In this section, we present a label-aware double transfer learning (La-DTL) framework and discuss its rationale.

### Framework Overview

Figure 1 gives an overview of La-DTL for NER. From bottom up, each input sentence is converted into a sequence of embedding vectors, which are then fed into a Bi-LSTM to sequentially encode contextual information into fixed-length hidden vectors. The embedding and Bi-LSTM layers are shared among source/target domains. With label-aware maximum mean discrepancy (La-MMD) to reduce the feature representation discrepancy between two domains, the hidden vectors are directly fed into source/target domain specific CRF layers to predict the label sequence. We use domain constrained CRF layers to enhance the target domain performance.

More formally, let \(\mathcal{D}_{s}=\left\{(\mathbf{X}_{i}^{s},\mathbf{y}_{i}^{s})\right\}_{i=1}^{ N^{s}}\) be the training set of \(N^{s}\) samples from the source domain and \(\mathcal{D}_{t}=\left\{(\mathbf{X}_{i}^{t},\mathbf{y}_{i}^{t})\right\}_{i=1}^ {N^{t}}\) be the training set of \(N^{t}\) samples from the target domain, with \(N^{t}\ll N^{s}\). Bi-LSTM encodes a sentence \(\mathbf{X}=(\mathbf{x}_{1},\mathbf{x}_{2},...,\mathbf{x}_{n})\) to hidden vectors \(\mathbf{H}=(\mathbf{h}_{1},\mathbf{h}_{2},...,\mathbf{h}_{n})\). We occasionally use \(\mathbf{H}(\mathbf{X})\) to denote the corresponding hidden vectors when feeding \(\mathbf{X}\) into the Bi-LSTM. CRF decodes hidden vectors \(\mathbf{H}\) to a label sequence \(\hat{\mathbf{y}}=(\hat{y}_{1},\hat{y}_{2},...,\hat{y}_{n})\). Our goal is to improve label prediction accuracy on the target domain \(\mathcal{D}_{t}\) by utilizing the knowledge from the source domain \(\mathcal{D}_{s}\):

\[p(\mathbf{y}|\mathbf{X})= p(\mathbf{y}|\mathbf{H}(\mathbf{X})),\] \[\log p(\mathbf{y}|\mathbf{H})= \sum_{i=1}^{n}\mathbf{E}_{i,y_{i}}+\sum_{i=1}^{n-1}\mathbf{A}_{ y_{i},y_{i+1}}-\log Z(\mathbf{H}). \tag{3}\]

Thus training a transferable model \(p(\mathbf{y}|\mathbf{X})\) requires both \(\mathbf{H}(\mathbf{X})\) and \(p(\mathbf{y}|\mathbf{H})\) to be transferable.

We use share word embedding and Bi-LSTM by approaching the feature representation distributions \(p(\mathbf{h}|\mathcal{D}_{s})\) and \(p(\mathbf{h}|\mathcal{D}_{t})\), i.e., the distributions of Bi-LSTM hidden vectors at each timestep of the sentences from the source and target domains respectively. The rationale behind it lies on the insufficiency of labeled target data. Even though LSTM has high capacity, its generalization ability highly relies on viewing "sufficient" data. Otherwise, LSTM is very likely to overfit the data. Training on both source and target data, the Bi-LSTM is expected to learn feature representations with high quality. Yosinski et al. (2014) provided a justification of this solution that sharing bottom layers is promising for transfer learning in practice.

With the sentences projected onto the same hidden space, the conditional distribution \(p(\mathbf{h}^{s}|\mathcal{D}_{s})\) and \(p(\mathbf{h}^{t}|\mathcal{D}_{t})\), however, may be distant because

Figure 1: La-DTL framework overview: embedding and Bi-LSTM layers are shared across domains, predictors in red (upper) boxes are task-specific CRFs, with label-aware MMD and L2 constraints to perform feature representation transfer and parameter transfer.

 

LSTM hidden vectors contain contextual information which is different across domains. In order to reduce source/target discrepancy, we refine MMD (Gretton et al., 2012) with label constraints, i.e., label-aware MMD (La-MMD). Using La-MMD, the source/target hidden states are pushed to similar distributions to make the feature representation \({\bf H}({\bf X})\) transfer feasible.

Based on the hidden vectors from Bi-LSTM, we adopt independent CRF layers for each domain. The rationale lies in the hypotheses that (i) the target domain predictor can better capture target data distribution which could be very unique; (ii) a good predictor trained on the source domain directly could be leveraged to assist the target domain predictor without directly borrowing the source domain training data to bypass the data privacy issue. With respect to the emission and transition score matrices \(\sum{\bf E}_{i,y_{i}}\) and \(\sum{\bf A}_{y_{i},y_{i+1}}\), we adopt an upper bound between source/target domains, which helps the target domain predictor to be guided by the source domain predictor. Thus \(p({\bf y}|{\bf H})\) is also transferable.

There are also other transfer methods, including fine-tuning, sharing parameter directly (without constraints) He and Sun (2017); Lee et al. (2017); Yang et al. (2017), etc. However, simply sharing models may dismiss target specific instances.

### Learning Objective

The learning objective is to minimize the following loss \(\mathcal{L}\) with respect to parameters \(\boldsymbol{\Theta}=\{\theta_{b},\theta_{c}\}\):

\[\mathcal{L}=\mathcal{L}_{c}+\alpha~{}\mathcal{L}_{\text{La-MMD}}+\beta~{} \mathcal{L}_{p}+\gamma~{}\mathcal{L}_{r},\]

where \(\mathcal{L}_{c}\) is the CRF loss, \(\mathcal{L}_{\text{La-MMD}}\) is the La-MMD loss, \(\mathcal{L}_{p}\) is the parameter similarity loss on CRF layers, and \(\mathcal{L}_{r}\) is the regularization term, with \(\alpha,\beta,\gamma\) as hyperparameters to balance loss terms.

The CRF loss is our ultimate objective predicting the label sequence given the input sentence, i.e., we minimize the negative log-likelihood of training samples from both source/target domains:

\[\mathcal{L}_{c}=-\frac{\varepsilon}{N^{s}}\sum_{i=1}^{N^{s}}\log p({\bf y}_{i }^{*}|{\bf H}_{i}^{*})-\frac{1-\varepsilon}{N^{t}}\sum_{i=1}^{N^{t}}\log p({ \bf y}_{i}^{*}|{\bf H}_{i}^{t}),\]

where \({\bf H}\) are hidden vectors obtained from Bi-LSTM, \(\varepsilon\) is the balance coefficient. The La-MMD loss \(\mathcal{L}_{\text{La-MMD}}\) and parameter similarity loss \(\mathcal{L}_{p}\) are discussed in Section 4.3 and 4.4, respectively. The regularization term is to generally control overfitting:

\[\mathcal{L}_{r}=\lVert\theta_{b}\rVert_{2}^{2}+\lVert\theta_{c}\rVert_{2}^{2}.\]

We will provide the model convergence and hyperparameter study in Section 5.1.

### Bi-LSTM Feature Representation Transfer

To learn transferable feature representations, the maximum mean discrepancy (MMD) which measures the distance between two distributions, has been widely used in domain adaptation scenarios Long et al. (2015); Rozantsev et al. (2016). Almost all these works focus on reducing the _marginal_ distribution distance between different domain features in an unsupervised manner to make them indistinguishable. However, considering a word is not evenly distributed conditioning on different labels, it may result in that the discriminative property of features from different domains may not be similar, which means that close source and target samples may not have the same label. Different from previous works, we propose label-aware MMD (La-MMD) in Eq. (5) to explicitly reduce the discrepancy between hidden representations with the same label, i.e., the linear combination of the MMD for each label. For each label class \(y\in\mathcal{Y}_{v}\), where \(\mathcal{Y}_{v}\) is the set of matched labels in two domains, we compute the squared population MMD between the hidden representations of source/target samples with the same label \(y\):

Figure 2: Illustration for La-MMD. MMD-\(y\) is computed between two domains’ hidden representations with the same ground truth label \(y\). A linear combination is then applied to each label-wise MMD to form La-MMD and the coefficient is set as \(\mu_{y}=1\).

 

\[{\rm MMD}^{2}(\mathcal{R}_{y}^{s},\mathbf{R}_{y}^{t})= \frac{1}{(N_{f}^{s})^{2}}\sum_{i,j=1}^{N_{y}^{s}}k(\mathbf{h}_{i}^ {s},\mathbf{h}_{j}^{s})+\frac{1}{(N_{f}^{s})^{2}}\sum_{i,j=1}^{N_{y}^{s}}k( \mathbf{h}_{i}^{s},\mathbf{h}_{j}^{t})\] \[-\frac{2}{N_{g}^{s}N_{y}^{s}}\sum_{i,j=1}^{N_{y}^{s},N_{y}^{t}}k( \mathbf{h}_{i}^{s},\mathbf{h}_{j}^{t}), \tag{4}\]

where \(\mathcal{R}_{y}^{s}\) and \(\mathcal{R}_{y}^{t}\) are sets of hidden representation \(\mathbf{h}^{s}\) and \(\mathbf{h}^{t}\) with corresponding number \(N_{y}^{s}\) and \(N_{y}^{t}\). Eq. (4) can be easily derived by casting Eq. (2) into inner product form and applying \(\langle\phi(x),\phi(y)\rangle_{\mathcal{H}}=k(x,y)\) where \(k\) is the reproducing kernel function (Gretton et al., 2012). For each label class, we compute the MMD loss in a normal manner. After that, we define the La-MMD loss as:

\[\small\mathcal{L}_{\text{La-MMD}}=\sum_{y\in\mathcal{Y}_{v}}\mu_{ y}\cdot{\rm MMD}^{2}(\mathcal{R}_{y}^{s},\mathcal{R}_{y}^{t}), \tag{5}\]

where \(\mu_{y}\) is the corresponding coefficient. The illustration of La-MMD is shown in Figure 2.

Once we have applied this La-MMD to our representations learned from Bi-LSTM, the representation distribution of instances with the same label from different domains should be close. Then the standard CRF layer which has a simple linear structure takes these similar representations as input and is likely to give a more transferable label decision for instances with the same label.

### CRF Parameter Transfer

Simply sharing the CRF layer is non-promising when source/target data are diversely distributed. According to probability decomposition in Eq. (3), in order to transfer on source/target CRF layers, more specifically, \(p(\mathbf{y}|\mathbf{H})\), we reduce the KL divergence from \(p^{t}(\mathbf{y}|\mathbf{H})\) to \(p^{s}(\mathbf{y}|\mathbf{H})\). But directly reducing \(D_{\text{KL}}(p^{s}(\mathbf{y}|\mathbf{H})||p^{t}(\mathbf{y}|\mathbf{H}))\) is intractable, we tend to reduce its upper bound:

\[D_{\text{KL}}(p^{s}(\mathbf{y}|\mathbf{H})||p^{t}(\mathbf{y}| \mathbf{H}))\] \[= \sum_{\mathbf{y}\in\mathcal{Y}(\mathbf{H})}p^{s}(\mathbf{y}| \mathbf{H})\log(\frac{p^{s}(\mathbf{y}|\mathbf{H})}{p^{s}(\mathbf{y}|\mathbf{ H})})\] \[= -H(p^{s}(\mathbf{y}|\mathbf{H}))-\sum_{\mathbf{y}\in\mathcal{Y}( \mathbf{H})}p^{s}(\mathbf{y}|\mathbf{H})\log p^{t}(\mathbf{y}|\mathbf{H})\] \[\leq c(||\mathbf{W}^{s}-\mathbf{W}^{t}||_{2}^{2}+||\mathbf{A}^{s}- \mathbf{A}^{t}||_{2}^{2})^{\frac{1}{2}}, \tag{6}\]

where \(H(\cdot)\) is the entropy of distribution \((\cdot)\) and \(c\) is a constant. The detailed proof is provided in Appendix A.1. Since \(c(\|\mathbf{W}^{s}-\mathbf{W}^{t}\|_{2}^{2}+||\mathbf{A}^{s}-\mathbf{A}^{t}\| _{2}^{2})\) is the upper bound of \(D_{\text{KL}}(p^{s}(\mathbf{y}|\mathbf{H})\|p^{t}(\mathbf{y}|\mathbf{H}))\), we conduct CRF parameter transfer by minimizing

\[\small\mathcal{L}_{p}=\left\lVert\mathbf{W}^{s}-\mathbf{W}^{t} \right\rVert_{2}^{2}+\left\lVert\mathbf{A}^{s}-\mathbf{A}^{t}\right\rVert_{2}^ {2}.\]

It turns out that a similar regularization term is applied in our CRF parameter transfer method and the regularization framework (RF) for domain adaptation (Lu et al., 2016). However, RF is proposed to generalize the feature augmentation method in (Daume III, 2007), and these two methods are only discussed from a perspective of the parameter. There is no guarantee that two models having similar parameters yields similar output distributions. In this work, we discuss the model behavior in CRF conditions, and we successfully prove that two CRF models having similar parameters (in Euclidean space) yields similar output distributions. In another word, our method guarantees transferability in the model behavior level, while previous works are limited in parameter level.

The CRF parameter transfer is illustrated in Figure 3, which is also label-aware since the L2 constraint is added over parameters corresponding to the same label in two domains, e.g., \(\mathbf{W}_{O}^{s}\) and \(\mathbf{W}_{O}^{t}\).

### Training

We train La-DTL in an end-to-end manner with mini-batch AdaGrad (Duchi et al., 2011). One mini-batch contains training samples from both domains, otherwise the computation of \(\mathcal{L}_{\text{La-MMD}}\) can not be performed. During training, word (and character) embeddings are fine-tuned to adjust real data distribution. During both training and decoding (testing) of CRF layers, we use dynamic programming to compute the normalizer in Eq. (1) and infer the label sequence.

Figure 3: Illustration for CRF parameter transfer.

 



## 5 Experiments

In this section, we evaluate La-DTL1 and other baseline methods on 12 cross-specialty NER problems based on real-world datasets. The experimental results show that La-DTL steadily outperforms other baseline models in all tasks significantly. We also conduct further ablation study and robustness study. We evaluate La-DTL on two more non-medical NER transfer tasks to validate its general efficacy over a wide range of applications.

Footnote 1: [https://github.com/felixwzh/La-DTL](https://github.com/felixwzh/La-DTL)

### Cross-Speciality NER

**Datasets** We collected a Chinese medical NER (_CM-NER_) corpus for our experiments. This corpus contains 1600 de-identified EHRs of our affiliated hospital from four different specialties in four departments: Cardiology (500), Respiratory (500), Neurology (300) and Gastroenterology (300), and the research had been reviewed and approved by the ethics committee. Named entities are annotated in the BIOES format (Begin, Inside, Outside, End and Single), with 30 types in total. The statistics of _CM-NER_ is shown in Table 1.

**Baselines** The following methods are compared. For a fair comparison, we implement La-DTL and baselines with the same base model introduced in Lample et al. (2016) but with different transfer techniques.

* **Non-transfer** uses the target domain labeled data only.
* **Domain mask** and **Linear projection** belong to the same framework proposed by Peng and Dredze (2017) but have different implementations at the projection layer, which aims to produce shared feature representations among different domains through a linear transformation.
* **Re-training** is proposed by Lee et al. (2017), where an artificial neural networks (ANNs) is first trained on the source domain and then re-trained on the target domain.
* **Joint-training** is a transfer learning method proposed by Yang et al. (2017) where different tasks are trained jointly.
* **CD-learning** is a cross-domain learning method proposed by He and Sun (2017), where each source domain training example's learning rate is re-weighted.

Experimental SettingsWe use 23,217 unlabeled clinical records to train the word embeddings (word2vec) at 128 dimensions using skipgram model Mikolov et al. (2013). The hidden state size is set to be 200 for word-level Bi-LSTM. We evaluate La-DTL for cross-specialty NER with _CM-NER_ in 12 transfer tasks, results shown in Table 2. For each task, we take the whole source domain training set \({\cal D}_{s}\) and 10% sentences of the target domain training set \({\cal D}_{t}\) as training data. We use the development set in target domain to search hyper-parameters including training epochs. We then take the models to make the prediction in target domain test set and use F1-score as the evaluation metric. Statistical significance has been determined using a randomization version of the paired sample t-test Cohen (1995).

Results and DiscussionFrom the results of 12 cross-specialty NER tasks shown in Table 2, we find that La-DTL outperforms all the strong baselines in all the 12 cross-specialty transfer learning tasks, with 2.62% to 6.70% F1-score lift over state-of-the-art baseline methods. Meanwhile, Linear projection and Domain mask Peng and Dredze (2017) do not perform as good as other three baselines, which may be because such linear transformation methods are likely to weaken the representations. While other three baseline methods all share the whole model between source/target domains but differ in the training schemes and performance.

To better understand the transferability of La-DTL, we also evaluate three variants of La-DTL: La-MMD, CRF-L2, and MMD-CRF-L2. La-MMD and CRF-L2 have the same networks and loss function as La-DTL but with different building blocks: La-MMD has \(\beta=0\), while CRF-L2 has \(\alpha=0\). In MMD-CRF-L2, we replace La-MMD loss \({\cal L}_{\text{La-MMD}}\) in La-DTL with a vanilla MMD loss:

\[{\cal L}_{\text{MMD}}=\text{MMD}^{2}({\cal R}^{s},{\cal R}^{t}),\]

\begin{table}
\begin{tabular}{l|c c c} \hline \hline Department & \# Train & \# Dev & \# Test \\ \hline Cardiology & 3,004 & 601 & 601 \\ Respiratory & 3,025 & 605 & 606 \\ Neurology & 932 & 187 & 187 \\ Gastroenterology & 1,517 & 303 & 304 \\ \hline Sum & 8,478 & 1,696 & 1,698 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Sentence numbers for _CM-NER_ corpus.



where \({\cal R}^{s}\) and \({\cal R}^{t}\) are sets of hidden representation from source and target domain. Results in Table 2 show that: (i) Using La-MMD alone does achieve satisfactory performance since it outperforms the best baseline Joint-training Yang et al. (2017) in 7 of 12 tasks. And it has a significant improvement over Domain mask and Linear projection methods Peng and Dredze (2017), which indicates that using La-MMD to reduce the domain discrepancy of feature representations in sequence tagging tasks is promising. (ii) CRF-L2 is also a promising method when transferring between NER tasks, and it improves the La-MMD method significantly when these two methods are combined to form La-DTL. (iii) Label-aware characteristic is important in sequence labeling problems because there is an obvious performance drop when La-MMD is replaced with a vanilla MMD in La-DTL. But MMD-CRF-L2 still has very competitive performance compared to all the baseline methods. This shows positive empirical evidence that transferring knowledge at both BiLSTM feature representation level and CRF parameter level for NER tasks is better than transferring knowledge at only one of these two levels, as discussed in Section 4.1.

**Robustness to Target Domain Data Sparsity** We further study the sparsity problem (target domain) of La-DTL in C\(\rightarrow\)R task comparing to Joint-training Yang et al. (2017) and Non-transfer method. We evaluate La-DTL with different data volume (sampling rate: 10%, 25%, 50%, 100%) on the target domain training set. Results are shown in Figure 4(a). We observe that La-DTL outperforms Joint-training and Non-transfer results under all circumstances, and the improvement of La-DTL is more significant when the sampling rate is lower.

To show La-DTL's convergence and significant improvement over Joint-training, we repeat the 10% sampling rate experiment for 10 times with 10 random seeds. The F1-score on the target domain development set for two methods with a 95% confidence interval is shown in Figure 4(b) where La-DTL outperforms Joint-training method significantly.

**Hyperparameter Study** We study the influence of three key hyperparameters in La-DTL: \(\alpha,\beta,\) and \(\varepsilon\) in C\(\rightarrow\)R task with 10% target domain sampling rate. We first apply a rough grid search for the three hyperparameters, and the result is (\(\alpha=0.02,\beta=0.03,\varepsilon=0.3\)). We then fix two hyperparameters and test the third one in a finer granularity. The results in Figure 5 indicate that setting \(\alpha\in[0.01,0.04]\) could better leverage La-MMD and further setting \(\beta\in[0.03,0.12]\) and \(\varepsilon\in[0.3,0.4]\) yields the best empirical perfor

Figure 4: (a) F1-score of La-DTL, Joint-training and Non-transfer method in C\(\rightarrow\)R task with different sampling rate. (b) The learning curve of La-DTL and Joint-training in C\(\rightarrow\)R task.

Figure 5: Hyperparameter study for \(\alpha,\beta,\) and \(\varepsilon.\)



mance. This shows that we need to balance the learning objective of the source and target domains for better transferability.

### NER Transfer Experiment on Non-medical Corpus

To show La-DTL could be applied in a wide range of NER transfer learning scenarios, we make experiments on two non-medical NER tasks. Corpora's details are shown in Table 3.

_WeiboNER Transfer_ Following He and Sun (2017); Peng and Dredze (2017), we transfer knowledge from _SighanNER_ (MSR corpus of the sixth SIGHAN Workshop on Chinese language processing) to _WeiboNER_ (a social media NER corpus) Peng and Dredze (2015). Results in Table 4 show that La-DTL outperforms all the baseline methods in Chinese social media domain.

_TwitterNER Transfer_ Following Yang et al. (2017) we transfer knowledge from CoNLL 2003 English NER Tjong Kim Sang and De Meulder (2003) to _TwitterNER_ Ritter et al. (2011). Since the entity types in these two corpora cannot be exactly matched, La-DTL and Joint-training Yang et al. (2017) can be applied directly in this case while other baselines can not. Because the CRF parameter transfer of La-DTL is label-aware, and Joint-training simply leverages two independent CRF layers. The results are shown in Table 5, where La-DTL again outperforms Joint-training, indicating that La-DTL could be applied seamlessly to transfer learning scenarios with mismatched label sets and languages like English.

## 6 Conclusions

In this paper, we propose La-DTL, a label-aware double transfer learning framework, to conduct both Bi-LSTM feature representation transfer and CRF parameter transfer with label-aware constraints for cross-speciality medical NER tasks. To our best knowledge, this is the first work on transfer learning for medical NER in cross-specialty scenario. Experiments on 12 cross-specialty NER tasks show that La-DTL provides consistent performance improvement over strong baselines. We further perform a set of experiments on different target domain data size, hyperparameter study and other non-medical NER tasks, where La-DTL shows great robustness and wide efficacy. For future work, we plan to jointly perform NER and entity linking for better cross-specialty media structural information extraction.

## Acknowledgments

The work done by SJTU is sponsored by Synyi-SJTU Innovation Program, National Natural Science Foundation of China (61632017, 61702327, 61772333) and Shanghai Sailing Program (17YF1428200).

## References

* Chen et al. (2011) Minmin Chen, Kilian Q Weinberger, and John Blitzer. 2011. Co-training for domain adaptation. In _Advances in Neural Information Processing Systems 24_, pages 2456-2464. Curran Associates, Inc.
* Chiticariu et al. (2010) Laura Chiticariu, Rajasekar Krishnamurthy, Yunya Li, Frederick Reiss, and Shivakumar Vaithyanathan. 2010. Domain adaptation of rule-based annotators for named-entity recognition tasks. In _Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing_, pages 1002-1012, Cambridge, MA. Association for Computational Linguistics.
* Chorrado et al. (2019)

\begin{table}
\begin{tabular}{l|c} \hline \hline Method & F1-score \\ \hline Non-transfer & 54.78 \\ Linear projection Peng and Dredze (2017)\({}^{*}\) & 56.40 \\ Linear projection Peng and Dredze (2017)\({}^{*}\) & 56.99 \\ Domain mask Peng and Dredze (2017)\({}^{*}\) & 56.80 \\ CD-learning He and Sun (2017)\({}^{*}\) & 52.05 \\ CD-learning He and Sun (2017) & 56.46 \\ Re-training Lee et al. (2017) & 55.36 \\ Joint-training Yang et al. (2017) & 56.80 \\ \hline La-DTL & **57.74** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Results (F1-score %) of _WeiboNER_ transfer.

\(*\) indicates the result reported in the corresponding reference.

\begin{table}
\begin{tabular}{l|c c} \hline \hline Corpus & \# Train & \# Dev & \# Test \\ \hline SighanNER & 23,182 & - & 4,636 \\ WeiboNER & 1,350 & 270 & 270 \\ CoNLL 2003 & 14,987 & 3,466 & 3,684 \\ TwitterNER & 1,900 & 240 & 254 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Sentence numbers for non-medical corpora.

\begin{table}
\begin{tabular}{l|p{142.3pt}} \hline \hline Method & F1-score \\ \hline Non-transfer & 34.65 \\ Joint-training Yang et al. (2017)\({}^{*}\) & 43.24 \\ \hline La-DTL & **45.71** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Results (F1-score %) of _TwitterNER_ transfer.

 

Jason Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional lstm-cnns. _Transactions of the Association for Computational Linguistics_, 4:357-370.
* Chu et al. (2013) Wen-Sheng Chu, Fernando De la Torre, and Jeffery F Cohn. 2013. Selective transfer machine for personalized facial action unit detection. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 3515-3522.
* Cohen (1995) Paul R Cohen. 1995. _Empirical methods for artificial intelligence_, volume 139. MIT press Cambridge, MA.
* Collobert et al. (2011) Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. _J. Mach. Learn. Res._, 12:2493-2537.
* Daume III (2007) Hal Daume III. 2007. Frustratingly easy domain adaptation. In _Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics_, pages 256-263. Association for Computational Linguistics.
* Duchi et al. (2011) John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. _J. Mach. Learn. Res._, 12:2121-2159.
* Graves and Schmidhuber (2005) Alex Graves and Jurgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional lstm and other neural network architectures. _Neural Networks_, 18(5):602-610.
* Gretton et al. (2012) Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Scholkopf, and Alexander Smola. 2012. A kernel two-sample test. _J. Mach. Learn. Res._, 13:723-773.
* Hachey et al. (2011) Ben Hachey, Will Radford, and James R. Curran. 2011. Graph-based named entity linking with wikipedia. In _Proceedings of the 12th International Conference on Web Information System Engineering_, WISE'11, pages 213-226, Berlin, Heidelberg. Springer-Verlag.
* He and Sun (2017) Hangfeng He and Xu Sun. 2017. A unified model for cross-domain and semi-supervised named entity recognition in chinese social media. In _AAAI_, pages 3216-3222.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_, 9(8):1735-1780.
* Kulkarni et al. (2016) Vivek Kulkarni, Yashar Mehdad, and Troy Chevalier. 2016. Domain adaptation for named entity recognition in online media with word embeddings. _arXiv preprint arXiv:1612.00148_.
* Lafferty et al. (2001) John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In _Proceedings of the Eighteenth International Conference on Machine Learning_, ICML '01, pages 282-289, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
* Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 260-270, San Diego, California. Association for Computational Linguistics.
* Lee et al. (2017) Ji Young Lee, Franck Dernoncourt, and Peter Szolovits. 2017. Transfer learning for named-entity recognition with neural networks. _arXiv preprint arXiv:1705.06273_.
* Long et al. (2015) Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. 2015. Learning transferable features with deep adaptation networks. In _Proceedings of the 32nd International Conference on Machine Learning_, volume 37 of _Proceedings of Machine Learning Research_, pages 97-105, Lille, France. PMLR.
* Lu et al. (2016) Wei Lu, Hai Leong Chieu, and Jonathan Lofgren. 2016. A general regularization framework for domain adaptation. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 950-954, Austin, Texas. Association for Computational Linguistics.
* Ma and Hovy (2016) Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-crf. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1064-1074, Berlin, Germany. Association for Computational Linguistics.

* McCallum et al. (2000) Andrew McCallum, Dayne Freitag, and Fernando C. N. Pereira. 2000. Maximum entropy markov models for information extraction and segmentation. In _Proceedings of the Seventeenth International Conference on Machine Learning_, ICML '00, pages 591-598, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In _Advances in Neural Information Processing Systems 26_, pages 3111-3119. Curran Associates, Inc.
* Moro et al. (2014) Andrea Moro, Alessandro Raganato, and Roberto Navigli. 2014. Entity linking meets word sense disambiguation: a unified approach. _Transactions of the Association for Computational Linguistics_, 2:231 

V Murthy, Mitesh Khapra, Pushpak Bhattacharyya, et al. 2016. Sharing network parameters for crosslingual named entity recognition. _arXiv preprint arXiv:1607.00198_.
* Nadeau and Sekine (2007) David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. _Lingvisticae Investigationes_, 30(1):3-26.
* Pan and Yang (2010) Sinno Jialin Pan and Qiang Yang. 2010. A survey on transfer learning. _IEEE Trans. on Knowl. and Data Eng._, 22(10):1345-1359.
* Peng and Dredze (2015) Nanyun Peng and Mark Dredze. 2015. Named entity recognition for chinese social media with jointly trained embeddings. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 548-554, Lisbon, Portugal. Association for Computational Linguistics.
* Peng and Dredze (2016) Nanyun Peng and Mark Dredze. 2016. Improving named entity recognition for chinese social media with word segmentation representation learning. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 149-155, Berlin, Germany. Association for Computational Linguistics.
* Peng and Dredze (2017) Nanyun Peng and Mark Dredze. 2017. Multi-task domain adaptation for sequence tagging. In _Proceedings of the 2nd Workshop on Representation Learning for NLP_, pages 91-100, Vancouver, Canada. Association for Computational Linguistics.
* Perlich et al. (2014) Claudia Perlich, Brian Dalessandro, Troy Raeder, Ori Stitelman, and Foster Provost. 2014. Machine learning for targeted display advertising: Transfer learning in action. _Mach. Learn._, 95(1):103-127.
* Ritter et al. (2011) Alan Ritter, Sam Clark, Mausam, and Oren Etzioni. 2011. Named entity recognition in tweets: An experimental study. In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_, pages 1524-1534, Edinburgh, Scotland, UK. Association for Computational Linguistics.
* Rozantsev et al. (2016) Artem Rozantsev, Mathieu Salzmann, and Pascal Fua. 2016. Beyond sharing weights for deep domain adaptation. _arXiv preprint arXiv:1603.06432_.
* Shen et al. (2017) Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. 2017. Wasserstein distance guided representation learning for domain adaptation. _arXiv preprint arXiv:1707.01217_.
* Srivastava and Salakhutdinov (2013) Nitish Srivastava and Ruslan R Salakhutdinov. 2013. Discriminative transfer learning with tree-based priors. In _Advances in Neural Information Processing Systems 26_, pages 2094-2102. Curran Associates, Inc.
* Sutton et al. (2012) Charles Sutton, Andrew McCallum, et al. 2012. An introduction to conditional random fields. _Foundations and Trends(r) in Machine Learning_, 4(4):267-373.
* Tsung et al. (2003) Erik F. Tsung Kim Sang and Feien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_, pages 142-147.
* Uzuner et al. (2011) Ozlem Uzuner, Brett R South, Shuying Shen, and Scott L DuVall. 2011. 2010 i2b2/va challenge on concepts, assertions, and relations in clinical text. _Journal of the American Medical Informatics Association_, 18(5):552-556.
* Wu et al. (2015) Yonghui Wu, Min Jiang, Jianbo Lei, and Hua Xu. 2015. Named entity recognition in chinese clinical text using deep neural network. _Studies in health technology and informatics_, 216:624.
* Yang et al. (2017) Zhilin Yang, Ruslan Salakhutdinov, and William W Cohen. 2017. Transfer learning for sequence tagging with hierarchical recurrent networks. In _ICLR_.
* Volume 2_, NIPS'14, pages 3320-3328, Cambridge, MA, USA. MIT Press.
* Zhang and Elhadad (2013) Shaodian Zhang and Noemie Elhadad. 2013. Unsupervised biomedical named entity recognition: Experiments with clinical and biological texts. _Journal of biomedical informatics_, 46(6):1088-1098.
* Zhang et al. (2016) Weinan Zhang, Ulrich Paquet, and Katja Hofmann. 2016. Collective noise contrastive estimation for policy transfer learning. In _AAAI_, pages 1408-1414.
* Zhuang et al. (2015) Furzhen Zhuang, Xiaohu Cheng, Ping Luo, Sinno Jialin Pan, and Qing He. 2015. Supervised representation learning: Transfer learning with deep autoencoders. In _Proceedings of the 24th International Conference on Artificial Intelligence_, IJCAI'15, pages 4119-4125. AAAI Press.

 

Appendix

### Detailed Proof

Recall the bound as in Eq. (6):

**Lemma A.1**.: \(c_{1}(\|\mathbf{W}^{s}-\mathbf{W}^{t}\|_{2}^{2}+\|\mathbf{A}^{s}-\mathbf{A}^{t} \|_{2}^{2})\) _is the upper bound of \((s^{s}(\mathbf{H},\mathbf{y})-s^{t}(\mathbf{H},\mathbf{y}))^{2}\)._

Proof of Lemma a.1..: \(\otimes\) refers to convolutional product, \(\mathbf{H}^{W},\mathbf{H}^{A}\) are mask matrices corresponding to the given hidden vectors \(\mathbf{H}\), and \(c_{1}\) is a constant. We have:

\[(s^{s}(\mathbf{H},\mathbf{y})-s^{t}(\mathbf{H},\mathbf{y}))^{2}\] \[= (\sum_{i=1}^{n}\mathbf{E}_{i,y_{i}}^{s}+\sum_{i=1}^{n-1}\mathbf{A }_{y_{i},y_{i+1}}^{s}-\sum_{i=1}^{n}\mathbf{E}_{i,y_{i}}^{t}-\sum_{i=1}^{n-1} \mathbf{A}_{y_{i},y_{i+1}}^{t})^{2}\] \[= (\mathbf{W}^{s}\otimes\mathbf{H}^{W}+\mathbf{A}^{s}\otimes \mathbf{H}^{A}-\mathbf{W}^{t}\otimes\mathbf{H}^{W}-\mathbf{A}^{t}\otimes \mathbf{H}^{A})^{2}\] \[= ((\mathbf{W}^{s}-\mathbf{W}^{t})\otimes\mathbf{H}^{W}+(\mathbf{A} ^{s}-\mathbf{A}^{t})\otimes\mathbf{H}^{A})^{2}\] \[\leq 2((\mathbf{W}^{s}-\mathbf{W}^{t})\otimes\mathbf{H}^{W})^{2}+2(( \mathbf{A}^{s}-\mathbf{A}^{t})\otimes\mathbf{H}^{A})^{2}\] \[= 2(\sum_{i,j}(\mathbf{W}^{s}-\mathbf{W}^{t})_{i,j}\cdot\mathbf{H} _{i,j}^{W})^{2}+2(\sum_{p,q}(\mathbf{A}^{s}-\mathbf{A}^{t})_{p,q}\cdot \mathbf{H}_{p,q}^{A})^{2}\] \[\leq 2(\sum_{i,j}(\mathbf{W}^{s}-\mathbf{W}^{t})_{i,j}^{2}\cdot\sum_{ i,j}(\mathbf{H}_{i,j}^{W})^{2})+2(\sum_{p,q}(\mathbf{A}^{s}-\mathbf{A}^{t})_{ p,q}^{2}\cdot\sum_{p,q}(\mathbf{H}_{p,q}^{A})^{2})\] \[= 2(\|\mathbf{W}^{s}-\mathbf{W}^{t}\|_{2}^{2}\cdot\|\mathbf{H}^{ W}\|_{2}^{2})+2(\|\mathbf{A}^{s}-\mathbf{A}^{t}\|_{2}^{2}\cdot\|\mathbf{ H}^{A}\|_{2}^{2})\] \[\leq c_{1}(\|\mathbf{W}^{s}-\mathbf{W}^{t}\|_{2}^{2}+\|\mathbf{A}^{ s}-\mathbf{A}^{t}\|_{2}^{2}).\]

**Lemma A.2**.: \(c(\|\mathbf{W}^{s}-\mathbf{W}^{t}\|_{2}^{2}+\|\mathbf{A}^{s}-\mathbf{A}^{t} \|_{2}^{2})^{\frac{1}{2}}\) _is the upper bound of \(D_{KL}(p^{s}(\mathbf{y}|\mathbf{H})||p^{t}(\mathbf{y}|\mathbf{H}))\)._

Proof of Lemma a.2..: With Lemma. (A.1), we set \(\varepsilon=(c_{1}(\|\mathbf{W}^{s}-\mathbf{W}^{t}\|_{2}^{2}+\|\mathbf{A}^{s} -\mathbf{A}^{t}\|_{2}^{2}))^{\frac{1}{2}}\geq 0\) and \(c=2c_{1}^{\frac{1}{2}}\), and we have:

\[s^{s}(\mathbf{H},\mathbf{y})-\varepsilon\leq s^{t}(\mathbf{H}, \mathbf{y})\leq s^{s}(\mathbf{H},\mathbf{y})+\varepsilon, \tag{7}\]

\[\log\{\sum_{\mathbf{y}^{\prime}\in\mathcal{Y}(\mathbf{H})}\exp[s^{ s}(\mathbf{H},\mathbf{y}^{\prime})]\}-\varepsilon\leq\log\{\sum_{\mathbf{ y}^{\prime}\in\mathcal{Y}(\mathbf{H})}\exp[s^{t}(\mathbf{H},\mathbf{y}^{ \prime})]\}\leq\log\{\sum_{\mathbf{y}^{\prime}\in\mathcal{Y}(\mathbf{H})}\exp[ s^{s}(\mathbf{H},\mathbf{y}^{\prime})]\}+\varepsilon.\] (8) 

With Eq. (7) and Eq. (8), we can derive

\[-\sum_{\mathbf{y}\in\mathcal{Y}(\mathbf{H})}p^{s}(\mathbf{y}|\mathbf{ H})\log p^{t}(\mathbf{y}|\mathbf{H})\] \[= -\sum_{\mathbf{y}\in\mathcal{Y}(\mathbf{H})}p^{s}(\mathbf{y}| \mathbf{H})\log\frac{\exp[s^{t}(\mathbf{H},\mathbf{y})]}{\sum_{\mathbf{y}^{ \prime}\in\mathcal{Y}(\mathbf{H})}\exp[s^{t}(\mathbf{H},\mathbf{y}^{\prime})]}\] \[= -\sum_{\mathbf{y}\in\mathcal{Y}(\mathbf{H})}p^{s}(\mathbf{y}| \mathbf{H})\big{\{}s^{t}(\mathbf{H},\mathbf{y})-\log\{\sum_{\mathbf{y}^{\prime }\in\mathcal{Y}(\mathbf{H})}\exp[s^{t}(\mathbf{H},\mathbf{y}^{\prime})]\}\big{\}}\] \[\leq -\sum_{\mathbf{y}\in\mathcal{Y}(\mathbf{H})}p^{s}(\mathbf{y}| \mathbf{H})\big{\{}s^{s}(\mathbf{H},\mathbf{y})-\varepsilon-\log\{\sum_{ \mathbf{y}^{\prime}\in\mathcal{Y}(\mathbf{H})}\exp[s^{s}(\mathbf{H},\mathbf{ y}^{\prime})]\}-\varepsilon\big{\}}\] \[= -\sum_{\mathbf{y}\in\mathcal{Y}(\mathbf{H})}p^{s}(\mathbf{y}| \mathbf{H})\big{\{}\log\frac{\exp[s^{s}(\mathbf{H},\mathbf{y})]}{\sum_{ \mathbf{y}^{\prime}\in\mathcal{Y}(\mathbf{H})}\exp[s^{s}(\mathbf{H},\mathbf{y} ^{\prime})]}-2\varepsilon\big{\}}\] \[= -\sum_{\mathbf{y}\in\mathcal{Y}(\mathbf{H})}p^{s}(\mathbf{y}| \mathbf{H})\big{\{}\log p^{s}(\mathbf{y}|\mathbf{H})-2\varepsilon\big{\}}\] \[= H(p^{s}(\mathbf{y}|\mathbf{H}))+2\varepsilon.\]

Finally, we have

\[D_{\text{KL}}(p^{s}(\mathbf{y}|\mathbf{H})||p^{t}(\mathbf{y}| \mathbf{H}))\] \[= \sum_{\mathbf{y}\in\mathcal{Y}(\mathbf{H})}p^{s}(\mathbf{y}| \mathbf{H})\log(\frac{p^{s}(\mathbf{y}|\mathbf{H})}{p^{t}(\mathbf{y}|\mathbf{ H})})\] \[= -H(p^{s}(\mathbf{y}|\mathbf{H}))-\sum_{\mathbf{y}\in\mathcal{Y}( \mathbf{H})}p^{s}(\mathbf{y}|\mathbf{H})\log p^{t}(\mathbf{y}|\mathbf{H})\] \[\leq -H(p^{s}(\mathbf{y}|\mathbf{H}))+H(p^{s}(\mathbf{y}|\mathbf{H}))+2\varepsilon\] \[= c(\|\mathbf{W}^{s}-\mathbf{W}^{t}\|_{2}^{2}+\|\mathbf{A}^{s}- \mathbf{A}^{t}\|_{2}^{2})^{\frac{1}{2}}.\]

### Case Analysis

In clinical practice, patients with specific diseases would be assigned to different departments, and specialist doctors in their department may pay more attention to the specific disease. When writing a medical chart, these specific diseases and related clinical findings would have a more detailed description. Therefore, some medical terms would have enriched meanings in different departments accordingly. For example, patients with rheumatic heart disease are often treated in the department of Cardiology. The term, "rheumatic", a modifier, describes and limits the type of "heart disease". In English, "rheumatic" is an adjective modifying "heart disease". However, in Chinese, "rheumatic heart disease" can be regarded as two diseases, "rheumatism" and "heart disease". In the department of Cardiology, "rheumatic heart disease" is usually mentioned as a single term. While in other departments, "rheumatism" and "heart disease" are mostly two independent named entities in annotated datasets. As such, it is difficult to train an NER model to capture the relationship between "rheumatism" and "heart disease", and band them as a whole. In the training set of our study, the diagnostic term "rheumatic heart disease" (including synonym) is mentioned for 17 times in Dept. Cardiology, 16 times in Dept. Respiratory, none in Dept. Neurology and 3 times in Dept. Gastroenterology. We use the data from the first 3 departments as source domain training set respectively, and the data from Dept. Gastroenterology as the target domain training set. We test our models on the test set from Dept. Gastroenterology, where "rheumatic heart disease" is mentioned 3 times, and compare the results across models 

with/without transfer learning. As expected, models with source training data from Dept. Cardiovascular and Respiration correctly predict all these entities, but the model using source data from Dept. Neurology fails and so does a model without transfer learning.

Patients with pulmonary heart disease were often referred to Dept. Respiratory and Dept. Cardiology. In our training set, "pulmonary heart disease" (including synonym) is labeled for 24 times in Dept. Respiratory and 4 times in Dept. Cardiology. In English, "pulmonary" modified "heart disease". In Chinese, "pulmonary heart disease" contains body structure "lung" and disease name "heart disease". The model trained with the source set from both from department of respiratory and cardiology could correctly recognize the relation between lung and heart disease and predict the entity in the test set from Dept. Gastroenterology.

Similarly, "coronary atherosclerotic heart disease" contains two disease names, "coronary atherosclerosis" and "heart disease". Training model using source set from a department where the terms are enriched could improve the performance of recognizing the whole entity.

### Medical Experiments Details

The 30 entity types for medical domain are: Symptom, Disease, Examination, Treatment, Laboratory index, Products, Body structure, Frequency, Negative word, Value, Trend, Modification, Temporal word, Noun of locality, Degree modifier, Probability, Object, Organism, Location, Person, Pronoun, Privacy information, Accident, Action, Header, Instrument and material, Non-physiological structure, Dosage, Scale, and Preposition.

### Non-medical Experiments Details

_WeiboNER Transfer_

Both _SighanNER_ and _WeiboNER_ are annotated in the BIO format (Begin, Inside and Outside), but there is one more entity type (geo-political) in _WeiboNER_. For a fair comparison, we follow Peng and Dredze (2017); He and Sun (2017) to merge geo-political entities and locations in _WeiboNER_, to match different labeling schemes between _WeboNER_ and _SighanNER_. We use the inconsistencies fixed second version of _WeiboNER_ data and word embeddings provided by _WeiboNER_'s developers (Peng and Dredze, 2015)2 in this experiment.

Footnote 2: [https://github.com/hltcee/golden-horse](https://github.com/hltcee/golden-horse)

#### TwitterNER Transfer

To show that La-DTL could be applied in transfer learning for NER scenario with mismatched

\begin{table}
\begin{tabular}{c|c|c|c|c|c|c} \hline \multirow{2}{*}{Diseasease} & Transfer & \# disease term in & \# disease term in & \# disease term in & \# accurate \\  & Task & source domain & target domain & target domain & labeling & labeling \\  & & training set & training set & test set & without transfer & with transfer \\ \hline \multirow{2}{*}{rheumatic heart disease} & C→G & 17 & \multirow{2}{*}{0} & \multirow{2}{*}{3} & \multirow{2}{*}{0} & 3 \\  & N→G & 0 & & & & 0 \\  & R→G & 16 & & & & 3 \\ \hline \multirow{2}{*}{pulmonary heart disease} & C→G & \multirow{2}{*}{0} & \multirow{2}{*}{2} & \multirow{2}{*}{0} & 2 \\  & N→G & 0 & & & & 0 \\  & R→G & 24 & & & & 2 \\ \hline coronary & G→N & 5 & \multirow{2}{*}{0} & \multirow{2}{*}{15} & \multirow{2}{*}{10} & 3 \\ atherosclerotic & C→N & 136 & & & & 15 \\ heart disease & R→N & 23 & & & & 11 \\ \hline \end{tabular}
\end{table}
Table 6: Case analysis for cross-specialty medical NER tasks. C, R, N, G are short for department of Cardiology, Respiratory, Neurology, and Gastroenterology, respectively.

Figure 6: La-DTL framework for language like English.

 

named entity types and languages like English, we conduct this experiment transfer from CoNLL 2003 English NER to _TwitterNER_. The four entity types in CoNLL 2003 English NER are LOC, PER, ORG, and MISC. The ten entity types in _TwitterNER_ are company, facility, geo-loc, movie, musicartist, other, person, product, sportsteam, and tvshow.

The Joint-training method Yang et al. (2017) separates the CRF layers for each domain to bypass the label mismatch problem. Since our La-DTL is label-aware, we match four pairs of named entities between two CoNLL 2003 English NER and _TwitterNER_: LOC with geo-loc, PER with person, ORG with company and MISC with other to compute \(\mathcal{L}_{\text{La-MMD}}\) and \(\mathcal{L}_{p}\), and leave six named entities unmatched. Following Yang et al. (2017), We leverage char-level Bi-LSTM to generate better word representations, concatenate it with pre-trained word embeddings and feed concatenated embeddings to the word-level Bi-LSTM. The framework used for language like English is illustrated in Figure 6.

We also convert all characters to lowercase and use the same word embeddings provided by Yang et al. (2017)3. Also, we concatenate the training set and the development set for both domains and sample the same 10% from _TwitterNER_ as Yang et al. (2017) to be target domain training data. Since Yang et al. (2017) merge training and development set into training data, both Yang et al. (2017) and we report the best performance in the target domain test set.